DNS failover for cloudflare with monit

We use Cloudflare for our sites and for a while i been wanting to have a server switchover when my primary servers fails. Cloudflare says they are working on a feature for it but i really can’t wait for that. The good thing about cloudflare is that the clients always get the Cloudflare ips and thus by doing a DNS failover all cloudflare has to do is sync it internally. I noticed the TTL averages around 1-5 min depending on the time.

This is what we need:

  • Backup server.
  • Sync between the primary server and the backup so content stays the same if you run a dynamic website.
  • Small server to run monit on.
  • The site configurated with Cloudflare and acceleration activated. You can get the API key in the Account tab.

To reduce cost you could run monit on the actual backup server but its best if the monit is ran independatly. I use a small cloud server from leaseweb for it. The configuration on both primary and backup server stays untouched. We’ll only be working on our monit server.

I use CentOS 6.2 for my monit server but any Linux distro that supports monit should work. Guide is made for CentOS but should be easy to adjust for any other linux distro.

We start by installing Monit. Monit is a watchdog that can keep watch on any service or even monitor urls for their uptime. This is what we’ll be using to monitor the sites.

yum install monit -y

After this is done i did some small changes to the monit.conf found in /etc/monit.conf

set daemon  5
allow admin:password

We can now put the config in /etc/monit.d, i called in main.conf but you can name it whatever you want.

nano /etc/monit.d/main.conf

The monit config, its configured to check with pings and http requests. Change values to your config:

check host SITENAME with address YOURIP
alert YOUREMAIL
if failed icmp type echo count 3 with timeout 1 seconds for 4 cycles then exec "/bin/bash -c /root/cloudflare/tobackup.sh"
else if succeeded for 20 cycles then exec "/bin/bash -c /root/cloudflare/tomain.sh"
if failed url http://YOURSITE with timeout 2 seconds and retry 3 for 15 cycles then exec "/bin/bash -c /root/cloudflare/tobackup.sh"
else if succeeded for 20 cycles then exec "/bin/bash -c /root/cloudflare/tomain.sh"

Thats it for monit, lets drop our switchover scripts in the respective folders. I use /root/cloudflare.

mkdir /root/cloudflare
mkdir /root/cloudflare/logs
nano /root/cloudflare/tobackup.sh

tobackup.sh:

nano /root/cloudflare/tobackup.sh
#!/bin/bash

wget -q --tries=1 --timeout=3 http://www.google.com -O /tmp/index.google >> /dev/null
if [ ! -s /tmp/index.google ];then
       echo `date` No interwebz? Not switching over! >> /root/cloudflare/switch.log
	exit 0
else
#Start check if in backup mode
if [ -f /root/cloudflare/backupactive ];
then
    echo "already in backup mode!"
    exit 0
fi
cd /root/cloudflare

key=YOUR-CLOUDFLARE-API-KEY
user=CLOUDFLARE-EMAIL/USER
mainip=MAINSERVER-IP
backupip=BACKUP-IP
# Also supports multiple A site records, comma separated. Example: www.google.com,google.com,google.de,www.google.de
site=SITE-HOSTNAME

echo "Switching to backup..."
touch /root/cloudflare/backupactive

curl https://www.cloudflare.com/api_json.html -d "a=DIUP" -d "email=$user" -d "tkn=$key" -d "ip=$backupip" -d "hosts=$site" | python -mjson.tool

echo `date` going down! >> /root/cloudflare/switch.log
fi
exit 0

The script to switch back once the primary server is back up:

nano /root/cloudflare/tomain.sh

tomain.sh:

#!/bin/bash
if [ ! -f /root/cloudflare/backupactive ];
then
	echo "Server is not in backup mode, can't switch back to main" && exit 0
fi
cd /root/cloudflare

key=YOUR-CLOUDFLARE-API-KEY
user=CLOUDFLARE-EMAIL/USER
mainip=MAINSERVER-IP
backupip=BACKUP-IP
# Also supports multiple A site records, comma separated. Example: www.google.com,google.com,google.de,www.google.de
site=SITE-HOSTNAME

curl https://www.cloudflare.com/api_json.html -d "a=DIUP" -d "email=$user" -d "tkn=$key" -d "ip=$mainip" -d "hosts=$site" | python -mjson.tool

echo `date` going up! >> /root/cloudflare/switch.log
rm -f /root/cloudflare/backupactive

Lets set permissions and boot up monit and set it to start on boot.

chmod +x /root/cloudflare/tobackup.sh
chmod +x /root/cloudflare/tomain.sh
chkconfig monit on
/etc/init.d/monit start

You can tweak the settings and timing with the monit config to be more or less strict by changing the cycles it takes before it executed the switchover script. The failover script first checks if the monit server has internet before switching over. You should also be able to go to the monit web panel by using a web browser and go to YOURIP:2812. The user is set by the value in monit.conf (allow USERNAME:PASSOWRD)

Categorized: Technical

16 comments on “DNS failover for cloudflare with monit

  1. Hi, interesting read. I’m still pretty confused about your infrastructure, though. It seems that on one hand, you have some quite powerfull servers doing the grunt work (http://booru.org/servers), on the other hand you mention that you use a CDN (cloudfare) for booru. Do the CDN servers hosted by cloudfare “mirror” the whole content that you put up on your own, dedicated servers (say, nanoha or vivio)? Or do they act like some kind of “proxy”?

  2. The solution works. Very cost effective solution to switch DNS records when on CloudFlare.

    To the author of this blog,

    If you happen to visit the Philippines, please send me an email to this email address. I owe you dinner and a barrel of beer. You just saved me a lot of work.

  3. I don’t quite think I am getting this. I tested downing my main webserver “mymainsite” but it did not failover to the backup, these are identical sites. does my mymainsite, mybackupsite and monit server all have to be on the same network? currently my mymainsite and mybackupsite server are on the same network and my monit server has internet access to that network at all times.
    Also, in my tomain.sh my “site=mymainsite.whatever.com” and in my tobackup.sh my “site=mybackupsite.whatever.com ?

    • Site urls should be the same in both scripts (since you only want to change ips). The code is a bit messy but I’m sure some programmer can rewrite it so its smaller and less dirty.

  4. If you don’t mind, could you put an open source license on these scripts? I made a few modifications for my own use and would like to publish the changes in my server’s documentation. I’m not allowed to do this unless you slap a GPL or MIT or CC or Apache license on it. Of course, if you do, I will give credit where credit it due.

    Thanks!

  5. Hello!
    It seems that CloudFlare has changed their API so I wonder if you have a new script setup?
    If not I’d love to help you find a solution that will work with the latest CloudFlare API as well.

  6. Great blog right here! Additionally your site
    quite a bit up fast! What host are you the use of? Can I am getting your associate link
    in your host? I desire my website loaded up as fast as yours
    lol

  7. Right here is the right site for anybody who wants to understand this
    topic. You realize so much its almost hard to argue with you (not that I personally would want to…HaHa).
    You definitely put a brand new spin on a subject that’s been written about for
    years. Excellent stuff, just wonderful!

  8. Hi blogger, i must say you have high quality posts here.
    Your website can go viral. You need initial traffic boost only.
    How to get it? Search for; Mertiso’s tips go viral

Leave a Reply

Your email address will not be published. Required fields are marked *