Service high availability using open-source tools (part 2)

[Introduction]

This article is a continuation of "Service high availability using open-source tools" . Here we will describe the process of relatively complex cluster configuration with several nodes (more than two).

[Cluster architecture]

Description

Typically, a cluster consists of four nodes. Let’s name them:
Node_A
Node_B
Node_C
Node_D

There are several services that will run on this cluster:
1. HTTP service 1
VIP with Apache service.
2. HTTP service 2
VIP with Nginx service
3. HTTP service 3
VIP with Apache service
4. MySQL service
We will just manage MySQL VIP. MySQL Replication should be set manually and MySQL services should run manually.
5. DRBD Storage
In order to not use rsync to copy the content from one server to another, we will use DRBD for data replication.
6. NFS Server and NFS CLient
The NFS Server will export the DRBD storage mount which will be used by NFS Clients on all remaining modes.

Which service will run on which node in cluster normal state:
- Node_A: NFS Client, MySQL Server
- Node_B: HTTP Service 1, NFS Client
- Node_C: DRBD Storage (Primary), NFS Server, HTTP Service 2, MySQL VIP, MySQL Server
- Node_D: DRBD Storage (Secondary), HTTP Service 3

A service from each node should be allowed to failover only to one node. What service to what node should failover:
Node_A:
Nothing.
Node_B:
HTTP Service 1 -> Node_A
Node_C:
DRBD Storage ( Primary ) -> Node_D
NFS Server -> Node_D
HTTP Service 2 -> Node_D
MySQL VIP -> Node_A
Node_D:
HTTP Service 3 -> Node_C

At this point, the reader will be asking: “Why on earth is this structure so complicated?” The answer is:
1.A cluster had to made from four different servers, each with different specs:
Node_A and Node_B with RHEL 4, Four Cores, 8G RAM, RAID1, and small discs on Node_B.
Node_C and Node_D with RHEL 5, Twelve Cores, 12G RAM, RAID 10, and Node_C with large discs.
2. In this way, it can be shown how a complicated cluster with many failure states can be built using good open source software.
Here is a cluster schema:

[Cluster configuration]

Choose cluster software
Pacemaker/Heartbeat 3.0. Mon will not be used anymore as Pacemaker/Heartbeat 3.0 has the ability to monitor resources and perform the needed actions (migrate resource, restart resource, etc).

Configure Heartbeat
First, install Pacemaker 1.x and Heartbeat 3.0 from "http://clusterlabs.org".
Create /etc/ha.d/ha.cf with following content:

# cluster name
cluster SuperCluster

# logging
debug 0
coredumps true
logfacility daemon
logfile /var/log/ha-log

# nodes
node Node_A Node_B Node_C Node_D

# HA detection
bcast eth0 eth1

# general
keepalive 1
warntime 20
deadtime 40
udpport 694
baud 19200
crm respawn
auto_failback no
respawn hacluster /usr/lib64/heartbeat/ipfail

# API auth
apiauth cibmon uid=hacluster

# monitoring
ping 192.168.1.1

# communication with nodes
ucast eth0 192.168.1.193
ucast eth0 192.168.1.194
ucast eth1 10.10.1.141
ucast eth1 10.10.1.142

The code can be adjusted as needed. Then Heartbeat should be started.

Configure Cluster Resources

HTTP Service 1
Resources: HTTP VIP and HTTP Service (Apache). Both resources will be monitored for failures and restarted if needed. Apache will be monitored by accessing "/server-status" page via VIP.<

primitive VIP_192.168.1.10 ocf:heartbeat:IPaddr \
         params ip="192.168.1.10" nic="eth0" cidr_netmask="255.255.255.0" \
         op monitor interval="5s" timeout="20s" start-delay="1s" 
primitive http_1_apache ocf:heartbeat:apache \ 
         params port="80" statusurl="192.168.1.10/server-status" 
         configfile="/etc/httpd/conf/httpd.conf" \ 
         op monitor interval="15" timeout="20" on-fail="restart" start-delay="15"

HTTP Service 2
Resources: HTTP VIP and HTTP Service (Nginx).
Both resources will be monitored for failures and restarted if needed. Nginx will be monitored by LSB script only because currently, there is no monitoring script for Nginx.

primitive VIP_192.168.1.11 ocf:heartbeat:IPaddr \
	params ip="192.168.1.11" nic="eth0" cidr_netmask="255.255.255.0" \
	op monitor interval="5s" timeout="20s" start-delay="1s"
primitive http_2_nginx nginx lsb:nginx \
	op monitor interval="5" timeout="15" on-fail="restart" start-delay="15"

HTTP Service 3
Resources: HTTP VIP and HTTP Service (Apache). Both resources will be monitored for failures and restarted if needed. Apache will be monitored by accessing "/server-status" page via VIP.

primitive VIP_192.168.1.13 ocf:heartbeat:IPaddr \
	params ip="192.168.1.13" nic="eth0" cidr_netmask="255.255.255.0" \
	op monitor interval="5s" timeout="20s" start-delay="1s"
primitive http_2_apache ocf:heartbeat:apache \
	params port="80" statusurl="192.168.1.13/server-status"
	configfile="/etc/httpd/conf/httpd.conf" \
	op monitor interval="15" timeout="20" on-fail="restart" start-delay="15"

MySQL Service
Resources: MySQL VIP.

primitive VIP_10.1.1.11 ocf:heartbeat:IPaddr \
	params ip="10.1.1.11" nic="eth1" cidr_netmask="255.255.255.0" \
	op monitor interval="5s" timeout="20s" start-delay="1s"

DRBD Storage
Resources: DRBD process and DRBD mount.
Both resources will be monitored for failure and restarted if needed.

primitive drbd_ps heartbeat:drbddisk \
	params 1="data" \
	op monitor interval="20" timeout="40" on-fail="restart" start-delay="10"
primitive fs_var_www ocf:heartbeat:Filesystem \
	params device="/dev/drbd0" directory="/var/www" fstype="ext3" options="noatime"
	statusfile_prefix=".fs_nfs_server_ha" \
	op monitor interval="20" timeout="60" on-fail="restart" start-delay="10"

NFS Server
Resources: NFS VIP, NFS server service and NFS Lock service. All resources will be monitored for failure and restarted if needed.

primitive VIP_10.1.1.10 ocf:heartbeat:IPaddr \
	params ip="10.1.1.10" nic="eth1" cidr_netmask="255.255.255.0" \
	op monitor interval="5s" timeout="20s" start-delay="1s"
primitive nfs lsb:nfs \
	op monitor interval="10" timeout="15" on-fail="restart" start-delay="15"
primitive nfslock lsb:nfslock \
	op monitor interval="5" timeout="15" on-fail="restart" start-delay="15"

NFS Client
Resources: NFS Lock service and FS mount. Both resources will be monitored for failure and restarted if needed.

primitive nfslock_client lsb:nfslock \
	op monitor interval="5" timeout="15" on-fail="restart" start-delay="15"
primitive fs_var_www_nfs_client ocf:heartbeat:fs_nfs_client \
	params server="10.10.1.143" server_directory="/var/www" directory="/var/www" statusfile_prefix=".fs_nfs_client_ha" options="rw,soft,async,noatime,proto=udp" \
	op monitor interval="30" timeout="60" on-fail="restart" start-delay="10"
	OCF_CHECK_LEVEL="20" \
	meta target-role="Started" is-managed="true"

Configure resource Groups
Resource Groups are an alternative to Resource Collocation constraints with the ability to order resources management. In order to reduce the complexity of configuration some resources will be regrouped.
Both, VIP_192.168.1.10 and http_1_apache resources should be present on the same node. A group of them will be created.

group gr_http_1_apache VIP_192.168.1.10 http_1_apache \
        meta ordered="true" collocated="true"

Both, VIP_192.168.1.11 and http_2_nginx resources should be present on same node, we will create a group of them.


group gr_http_2_nginx VIP_192.168.1.11 http_2_nginx \
        meta ordered="true" collocated="true"

Both, VIP_192.168.1.13 and http_3_apache resources should be present on same node, we will create a group of them.


group gr_http_3_apache VIP_192.168.1.13 http_3_apache \
        meta ordered="true" collocated="true"

DRBD process, DRBD mount, NFS VIP, NFS server service and NFS Lock service should be present on same node, we will create a group of them.


group gr_storage_server VIP_10.1.1.10 drbd_pp fs_var_www nfslock nfs \
	meta ordered="true" collocated="true"

Both, NFS Lock service and FS mount should be present on the same node. Therefore, we will create a group of them.


group gr_storage_client fs_var_www_nfs_client nfslock_client \
	meta ordered="true" collocated="true"

Configure resource Dependencies
NFS/DRBD Storage client is dependent on NFS DRBD Storage Server.


order ord_storage inf: gr_storage_server gr_storage_client

All HTTP service depends on NFS/DRBD Storage Server and CLient groups.


	order ord_www inf: gr_storage_server ( gr_http_1_apache gr_http_2_nginx gr_http_3_apache )
	order ord_www2 inf: gr_storage_client ( gr_http_1_apache gr_http_2_nginx gr_http_3_apache )

Configure Resource Locations
Almost each resource or resource group will have its default location and failover location.

HTTP Service 1
By default this resource group should be located on Node_B. In case of failure, it should be moved to Node_A.

location loc_gr_http_1_apache_default gr_http_1_apache 100: #uname eq Node_B
location loc_gr_http_1_apache_failover gr_http_1_apache 50: #uname eq Node_A

HTTP Service 2
By default this resource group should be located on Node_C. In case of failure, it should be moved to Node_D.

location loc_gr_http_2_nginx_default gr_http_2_nginx 100: #uname eq Node_C
location loc_gr_http_2_nginx_failover gr_http_2_nginx 50: #uname eq Node_D

HTTP Service 3
By default this resource group should be located on Node_D. In case of failure, it should be moved to Node_C.

location loc_gr_http_3_apache_default gr_http_3_apache 100: #uname eq Node_D
location loc_gr_http_3_apache_failover gr_http_1_apache 50: #uname eq Node_C

MySQL Service
By default this resource group should be located on Node_C. In case of failure it should be moved to Node_A.

location loc_gr_http_3_apache_default gr_http_3_apache 100: #uname eq Node_D
location loc_gr_http_3_apache_failover gr_http_1_apache 50: #uname eq Node_C

DRBD/NFS Server Storage

location loc_gr_storage_server_default gr_storage_server 100: #uname eq Node_C
location loc_gr_storage_server_failover gr_storage_server 50: #uname eq Node_D

Configure Resource Collocations
Resource groups gr_storage_server and gr_storage_client can not be located on same node.

colocation colo_gr_storage -inf: gr_storage_client gr_storage_server

Configure Resource Clones
Resource Group gr_storage_client should run on three nodes.

clone clone_gr_storage_client gr_storage_client \
	meta globally-unique=false clone-max=3 clone-node-max=1

[Conclusion]

Pacemaker/Heartbeat 3.0 is a good, flexible, customizable OpenSource Cluster resource management solution to build complex active/active and active/passive or mixed clusters, like in our case.