DRBD (Distributed Replicated Block Device) Installation
The Distributed Replicated Block Device (DRBD) is a Linux Kernel module that constitutes a distributed storage system. You can use DRBD to share block devices between Linux servers and, in turn, share file systems and data.
When used with MySQL, DRBD can be used to ensure availability in the event of a failure. MySQL is configured to store information on the DRBD block device, with one server acting as the primary and a second machine available to operate as an immediate replacement in the event of a failure.
For automatic failover support, you can combine DRBD with the Linux Heartbeat project, which manages the interfaces on the two servers and automatically configures the secondary (passive) server to replace the primary (active) server in the event of a failure. You can also combine DRBD with MySQL Replication to provide both failover and scalability within your MySQL environment.
Additional Resources and documentation
http://www.drbd.org/users-guide/s-initial-full-sync.html
http://www.drbd.org/users-guide/s-use-online-verify.html
http://www.drbd.org/users-guide/s-configure-sync-rate.html
http://www.drbd.org/users-guide/s-configure-congestion-policy.html
Installation of DRBD on DB cluster servers
First we need to instal ELREPO extra package repository from where DRBD can be installed,
as the package DRBD does not come in the default yum repositories.
[root@db1 ~]# rpm -Uvh http://elrepo.org/elrepo-release-6-4.el6.elrepo.noarch.rpm Retrieving http://elrepo.org/elrepo-release-6-4.el6.elrepo.noarch.rpm warning: /var/tmp/rpm-tmp.dIHerV: Header V4 DSA/SHA1 Signature, key ID baadae52: NOKEY Preparing... ########################################### [100%] 1:elrepo-release ########################################### [100%]
Install DRBD 8.4.1 (latest stable version at the time of this writting)
yum install drbd84-utils kmod-drbd84
- CentOS 6.3 comes with a firewall installed by default.
- We need to open ports 7788 through 7799 in the internal firewall of both servers, for DRBD to operate correctly between both servers db1 and db2:
Edit the file /etc/sysconfig/iptables with your favorite text editor
[root@db1 ~]# emacs /etc/sysconfig/iptables
and add the following lines to the rules:
-A INPUT -p tcp -m state --state NEW -m tcp --dport 7788:7799 -j ACCEPT -A OUTPUT -p tcp -m tcp --dport 7788:7799 -j ACCEPT
Here is the final version wit the extra rules added:
# Firewall configuration written by system-config-firewall # Manual customization of this file is not recommended. *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT -A INPUT -p icmp -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 7788:7799 -j ACCEPT -A INPUT -j REJECT --reject-with icmp-host-prohibited -A FORWARD -j REJECT --reject-with icmp-host-prohibited -A OUTPUT -p tcp -m tcp --dport 7788:7799 -j ACCEPT COMMIT
Then restart the firewall:
[root@db1 ~]# service iptables restart iptables: Flushing firewall rules: [ OK ] iptables: Setting chains to policy ACCEPT: filter [ OK ] iptables: Unloading modules: [ OK ] iptables: Applying firewall rules: [ OK ] [root@db1 ~]#
if you want to confirm, iptables-save should display following info:
[root@db1 ~]# iptables-save # Generated by iptables-save v1.4.7 on Thu Sep 20 20:53:03 2012 *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [14:1480] -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A INPUT -p icmp -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 7788:7799 -j ACCEPT -A INPUT -j REJECT --reject-with icmp-host-prohibited -A FORWARD -j REJECT --reject-with icmp-host-prohibited -A OUTPUT -p tcp -m tcp --dport 7788:7799 -j ACCEPT COMMIT # Completed on Thu Sep 20 20:53:03 2012
INSTALL NTP DAEMON TO KEEP SERVER TIME UPDATED
[root@db1 etc]# yum install ntp
ENABLE NTPD ON BOOT
[root@db1 etc]# chkconfig ntpd on
[root@db1 etc]# chkconfig --list | grep ntp ntpd 0:off 1:off 2:on 3:on 4:on 5:on 6:off ntpdate 0:off 1:off 2:off 3:off 4:off 5:off 6:off
INSTALL VIM EDITOR
yum install vim-enhanced
CHECKING THE HOSTNAME IS CORRECT
[root@db1 etc]# cat /etc/sysconfig/network NETWORKING=yes HOSTNAME=db1.snarvaez.poweredbygnulinux.com GATEWAY=192.168.2.1
CHECKING NETWORK CARDS
[root@db1 etc]# cat /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE="eth0" HWADDR="00:1B:21:AD:8E:8A" NM_CONTROLLED="yes" ONBOOT="yes" IPADDR="172.16.15.51" NETMASK="255.255.255.0" [root@db1 etc]# cat /etc/sysconfig/network-scripts/ifcfg-eth1 DEVICE="eth1" HWADDR="00:25:90:32:7B:90" NM_CONTROLLED="yes" ONBOOT="yes" IPADDR="192.168.10.51" NETMASK="255.255.255.0" [root@db1 etc]# cat /etc/sysconfig/network-scripts/ifcfg-eth2 DEVICE="eth2" HWADDR="00:25:90:32:7B:91" NM_CONTROLLED="yes" ONBOOT="yes" IPADDR="192.168.2.51" NETMASK="255.255.255.0"
[root@db1 etc]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:1B:21:AD:8E:8A
inet addr:172.16.15.51 Bcast:172.16.30.255 Mask:255.255.255.0
inet6 addr: fe80::21b:21ff:fead:8e8a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:8964 errors:0 dropped:0 overruns:0 frame:0
TX packets:29 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1077250 (1.0 MiB) TX bytes:2090 (2.0 KiB)
Interrupt:28 Memory:fba20000-fba40000
eth1 Link encap:Ethernet HWaddr 00:25:90:32:7B:90
inet addr:192.168.10.51 Bcast:192.168.10.255 Mask:255.255.255.0
inet6 addr: fe80::225:90ff:fe32:7b90/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:195794 errors:0 dropped:0 overruns:0 frame:0
TX packets:17 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:28376444 (27.0 MiB) TX bytes:1334 (1.3 KiB)
Interrupt:16 Memory:fbce0000-fbd00000
eth2 Link encap:Ethernet HWaddr 00:25:90:32:7B:91
inet addr:192.168.2.51 Bcast:192.168.2.255 Mask:255.255.255.0
inet6 addr: fe80::225:90ff:fe32:7b91/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:72313 errors:0 dropped:0 overruns:0 frame:0
TX packets:24468 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:70167216 (66.9 MiB) TX bytes:2066116 (1.9 MiB)
Interrupt:17 Memory:fbde0000-fbe00000
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:12 errors:0 dropped:0 overruns:0 frame:0
TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1064 (1.0 KiB) TX bytes:1064 (1.0 KiB)
START NTPD SERVICE
[root@db1 etc]# service ntpd start Starting ntpd: [ OK ]
Configure DRBD
[root@db1 etc]# cat drbd.conf # You can find an example in /usr/share/doc/drbd.../drbd.conf.example include "drbd.d/global_common.conf"; include "drbd.d/*.res";
SAVE THE ORIGINAL GLOBAL CONFIGURATION FILE ON BOTH SERVERS:
mv /etc/drbd.d/global_common.conf /etc/drbd.d/global_common.sample
You can read full example of config file here:
/usr/share/doc/drbd84-utils-8.4.1/drbd.conf.example
[root@db1 drbd84-utils-8.4.1]# uname -n db1.snarvaez.poweredbygnulinux.com
Create file: /etc/drbd.d/global_common.conf
global {
usage-count no;
# minor-count dialog-refresh disable-ip-verification
}
common {
handlers {
pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
# fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
# split-brain "/usr/lib/drbd/notify-split-brain.sh root";
# out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
# before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
# after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
}
startup {
# wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb
}
options {
# cpu-mask on-no-data-accessible
}
disk {
# size max-bio-bvecs on-io-error fencing disk-barrier disk-flushes
# disk-drain md-flushes resync-rate resync-after al-extents
# c-plan-ahead c-delay-target c-fill-target c-max-rate
# c-min-rate disk-timeout
}
net {
# protocol timeout max-epoch-size max-buffers unplug-watermark
# connect-int ping-int sndbuf-size rcvbuf-size ko-count
# allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri
# after-sb-1pri after-sb-2pri always-asbp rr-conflict
# ping-timeout data-integrity-alg tcp-cork on-congestion
# congestion-fill congestion-extents csums-alg verify-alg
# use-rle
}
}
Create file: /etc/drbd.d/mysqldata.res
# @see /usr/share/doc/drbd84-utils-8.4.1/drbd.conf.example
# http://www.drbd.org/users-guide/
resource mysqldata {
options {
on-no-data-accessible suspend-io;
}
net {
cram-hmac-alg "sha1";
shared-secret "put_a_random_string_here";
protocol C;
}
on db1.snarvaez.poweredbygnulinux.com {
address 192.168.2.51:7788;
device /dev/drbd0;
disk /dev/mysqlVol/mysqldata0;
meta-disk internal;
}
on db2.snarvaez.poweredbygnulinux.com {
address 192.168.2.52:7788;
device /dev/drbd0;
disk /dev/mysqlVol/mysqldata0;
meta-disk internal;
}
}
Ensure both servers domain names can be reached by each other by manually
adding the IPs in the /etc/hosts
[root@db2 etc]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.2.51 db1.snarvaez.poweredbygnulinux.com 192.168.2.52 db2.snarvaez.poweredbygnulinux.com
Now we have to create the metadata on both servers:
[root@db1 ~]# drbdadm create-md mysqldata Writing meta data... initializing activity log NOT initializing bitmap New drbd meta data block successfully created.
START THE DRBD SERVICE ON BOTH SERVERS:
service drbd start
Starting DRBD resources: [
create res: mysqldata
prepare disk: mysqldata
adjust disk: mysqldata
adjust net: mysqldata
]
DRBD’s startup script waits for the peer node(s) to appear.
In case this node was already a degraded cluster before the reboot the timeout is 0 seconds. [degr-wfc-timeout] If the peer was available before the reboot the timeout will expire after 0 seconds. [wfc-timeout] (These values are for resource 'mysqldata'; 0 sec -> wait forever) To abort waiting enter 'yes' [ 16]:
Check Initial status
Connected means both servers can see each other, and the disk resources are ready.
[root@db1 ~]# service drbd status drbd driver loaded OK; device status: version: 8.4.1 (api:1/proto:86-100) GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by phil@Build64R6, 2012-04-17 11:28:08 m:res cs ro ds p mounted fstype 0:mysqldata Connected Secondary/Secondary Inconsistent/Inconsistent C
[root@db2 drbd.d]# service drbd status drbd driver loaded OK; device status: version: 8.4.2 (api:1/proto:86-101) GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by dag@Build64R6, 2012-09-06 08:16:10 m:res cs ro ds p mounted fstype 0:mysqldata Connected Secondary/Secondary Inconsistent/Inconsistent C
THE INITIAL DEVICE SYNCHRONIZATION
Select an initial sync source. If you are dealing with newly-initialized, empty
disk, this choice is entirely arbitrary. If one of your nodes already has
valuable data that you need to preserve, however, it is of crucial importance
that you select that node as your synchronization source. If you do initial
device synchronization in the wrong direction, you will lose that data. Exercise
caution.
Start the initial full synchronization. This step must be performed on only one
node, only on initial resource configuration, and only on the node you selected
as the synchronization source. To perform this step, issue this command:
drbdadm primary --force resource
[root@db1 ~]# drbdadm primary --force mysqldata [root@db1 ~]# service drbd status drbd driver loaded OK; device status: version: 8.4.1 (api:1/proto:86-100) GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by phil@Build64R6, 2012-04-17 11:28:08 m:res cs ro ds p mounted fstype 0:mysqldata SyncSource Primary/Secondary UpToDate/Inconsistent C ... sync'ed: 0.1% (2516464/2516504)M
Check that devices are syncing
[root@db2 drbd.d]# service drbd status drbd driver loaded OK; device status: version: 8.4.2 (api:1/proto:86-101) GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by dag@Build64R6, 2012-09-06 08:16:10 m:res cs ro ds p mounted fstype 0:mysqldata SyncTarget Secondary/Primary Inconsistent/UpToDate C ... sync'ed: 0.2% (2513984/2516504)M
[root@db1 ~]# drbd-overview
0:mysqldata/0 SyncSource Primary/Secondary UpToDate/Inconsistent C r-----
[>....................] sync'ed: 1.0% (2492104/2516504)M
BASIC MANUAL FAIL-OVER
# umount /dev/drbd/by-res/<resource> # drbdadm secondary <resource>
Now on the node we wish to make primary promote the resource and mount the device.
# drbdadm primary <resource> # mount /dev/drbd/by-res/<resource> <mountpoint>