rendered paste bodydrbd.conf:
=========
global { usage-count yes; }
common { syncer { rate 10M; } }
resource r0 {
protocol C;
net {
cram-hmac-alg sha1;
shared-secret "FooFunFactory";
}
on wt-vm30 {
device /dev/drbd0;
disk /dev/sda5;
address 192.168.0.3:7788;
meta-disk internal;
}
on wt-vm31 {
device /dev/drbd0;
disk /dev/sda5;
address 192.168.0.4:7788;
meta-disk internal;
}
}
ha.cf:
======
debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0
keepalive 2
deadtime 30
warntime 10
initdead 120
udpport 694
bcast eth1
auto_failback on
node wt-vm30
node wt-vm31
ping 10.3.9.254
#respawn userid /path/name/to/run
respawn hacluster /usr/lib/heartbeat/ipfail
#apiauth client-name gid=gidlist uid=uidlist
#apiauth ipfail gid=haclient uid=hacluster
#crm on
haresources:
============
wt-vm30 \
drbddisk::r0 \
Filesystem::/dev/drbd0::/drbd::ext3 \
apache2::/etc/apache2/httpd.conf \
IPaddr2::10.3.9.250/24/eth0
ha-log from wt-vm31:
====================
tail -f /var/log/ha-log
heartbeat[18236]: 2007/05/30_16:23:26 WARN: Core dumps could be lost if multiple dumps occur
heartbeat[18236]: 2007/05/30_16:23:26 WARN: Consider setting /proc/sys/kernel/core_uses_pid (or equivalent) to 1 for maximum supportability
heartbeat[18236]: 2007/05/30_16:23:26 info: Version 2 support: false
heartbeat[18236]: 2007/05/30_16:23:26 WARN: Logging daemon is disabled --enabling logging daemon is recommended
heartbeat[18236]: 2007/05/30_16:23:26 info: **************************
heartbeat[18236]: 2007/05/30_16:23:26 info: Configuration validated. Starting heartbeat 2.0.9
heartbeat[18237]: 2007/05/30_16:23:26 info: heartbeat: version 2.0.9
heartbeat[18237]: 2007/05/30_16:23:26 info: Heartbeat generation: 17
heartbeat[18237]: 2007/05/30_16:23:26 info: G_main_add_TriggerHandler: Added signal manual handler
heartbeat[18237]: 2007/05/30_16:23:26 info: G_main_add_TriggerHandler: Added signal manual handler
heartbeat[18237]: 2007/05/30_16:23:26 info: Removing /var/run/heartbeat/rsctmp failed, recreating.
heartbeat[18237]: 2007/05/30_16:23:26 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1
heartbeat[18237]: 2007/05/30_16:23:26 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1
heartbeat[18237]: 2007/05/30_16:23:26 info: glib: ping heartbeat started.
heartbeat[18237]: 2007/05/30_16:23:26 info: G_main_add_SignalHandler: Added signal handler for signal 17
heartbeat[18237]: 2007/05/30_16:23:26 info: Local status now set to: 'up'
heartbeat[18237]: 2007/05/30_16:23:27 info: Link 10.3.9.254:10.3.9.254 up.
heartbeat[18237]: 2007/05/30_16:23:27 info: Status update for node 10.3.9.254: status ping
heartbeat[18237]: 2007/05/30_16:23:27 info: Link wt-vm30:eth1 heartbeat[18237]: 2007/05/30_16:23:54 info: Link wt-vm31:eth1 up.
heartbeat[18237]: 2007/05/30_16:23:54 info: Status update for node wt-vm31: status up
harc[18245]: 2007/05/30_16:23:55 info: Running /etc/ha.d/rc.d/status status
heartbeat[18237]: 2007/05/30_16:23:55 ERROR: HBDoMsg_T_REPNODES: Attempt to delete node 10.3.9.254
heartbeat[18237]: 2007/05/30_16:23:55 info: Comm_now_up(): updating status to active
heartbeat[18237]: 2007/05/30_16:23:55 info: Local status now set to: 'active'
heartbeat[18237]: 2007/05/30_16:23:55 info: Starting child client "/usr/lib/heartbeat/ipfail" (90,90)
heartbeat[18256]: 2007/05/30_16:23:55 info: Starting "/usr/lib/heartbeat/ipfail" as uid 90 gid 90 (pid 18256)
heartbeat[18237]: 2007/05/30_16:23:55 info: Status update for node wt-vm31: status active
harc[18257]: 2007/05/30_16:23:56 info: Running /etc/ha.d/rc.d/status status
heartbeat[18237]: 2007/05/30_16:24:06 info: local resource transition completed.
heartbeat[18237]: 2007/05/30_16:24:06 info: Initial resource acquisition complete (T_RESOURCES(us))
heartbeat[18267]: 2007/05/30_16:24:06 info: Local Resource acquisition completed.
harc[18300]: 2007/05/30_16:24:06 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-resp
ip-request-resp[18300]: 2007/05/30_16:24:06 received ip-request-resp drbddisk::r0 OK yes
ResourceManager[18315]: 2007/05/30_16:24:06 info: Acquiring resource group: wt-vm30 drbddisk::r0 Filesystem::/dev/drbd0::/drbd::ext3 apache2 # apache2::/etc/apache2/httpd.conf IPaddr2::10.3.9.250/24/eth0
ResourceManager[18315]: 2007/05/30_16:24:06 info: Running /etc/ha.d/resource.d/drbddisk r0 start
heartbeat[18237]: 2007/05/30_16:24:06 info: remote resource transition completed.
Filesystem[18380]: 2007/05/30_16:24:06 INFO: Resource is stopped
ResourceManager[18315]: 2007/05/30_16:24:06 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /drbd ext3 start
Filesystem[18425]: 2007/05/30_16:24:06 INFO: Running start for /dev/drbd0 on /drbd
Filesystem[18420]: 2007/05/30_16:24:06 INFO: Success
ResourceManager[18315]: 2007/05/30_16:24:06 info: Running /etc/init.d/apache2 start
ResourceManager[18315]: 2007/05/30_16:24:09 ERROR: Cannot locate resource script #
ResourceManager[18315]: 2007/05/30_16:24:09 ERROR: Cannot locate resource script #
ResourceManager[18315]: 2007/05/30_16:24:09 ERROR: Cannot locate resource script #
ResourceManager[18315]: 2007/05/30_16:24:09 CRIT: Giving up resources due to failure of #
ResourceManager[18315]: 2007/05/30_16:24:09 info: Releasing resource group: wt-vm30 drbddisk::r0 Filesystem::/dev/drbd0::/drbd::ext3 apache2 # apache2::/etc/apache2/httpd.conf IPaddr2::10.3.9.250/24/eth0
ResourceManager[18315]: 2007/05/30_16:24:09 info: Running /etc/ha.d/resource.d/IPaddr2 10.3.9.250/24/eth0 stop
IPaddr2[18596]: 2007/05/30_16:24:09 ERROR: [/usr/lib/heartbeat/findif -C] failed
IPaddr2[18573]: 2007/05/30_16:24:09 ERROR: Illegal argument
ResourceManager[18315]: 2007/05/30_16:24:09 ERROR: Return code 2 from /etc/ha.d/resource.d/IPaddr2
ResourceManager[18315]: 2007/05/30_16:24:10 info: Retrying failed stop operation [IPaddr2::10.3.9.250/24/eth0]
ResourceManager[18315]: 2007/05/30_16:24:10 info: Running /etc/ha.d/resource.d/IPaddr2 10.3.9.250/24/eth0 stop
IPaddr2[18671]: 2007/05/30_16:24:10 ERROR: [/usr/lib/heartbeat/findif -C] failed
IPaddr2[18648]: 2007/05/30_16:24:10 ERROR: Illegal argument
ResourceManager[18315]: 2007/05/30_16:24:10 ERROR: Return code 2 from /etc/ha.d/resource.d/IPaddr2
ResourceManager[18315]: 2007/05/30_16:24:11 info: Retrying failed stop operation [IPaddr2::10.3.9.250/24/eth0]
ResourceManager[18315]: 2007/05/30_16:24:11 info: Running /etc/ha.d/resource.d/IPaddr2 10.3.9.250/24/eth0 stop
IPaddr2[18746]: 2007/05/30_16:24:12 ERROR: [/usr/lib/heartbeat/findif -C] failed
IPaddr2[18723]: 2007/05/30_16:24:12 ERROR: Illegal argument
ResourceManager[18315]: 2007/05/30_16:24:12 ERROR: Return code 2 from /etc/ha.d/resource.d/IPaddr2
ResourceManager[18315]: 2007/05/30_16:24:13 info: Retrying failed stop operation [IPaddr2::10.3.9.250/24/eth0]
ResourceManager[18315]: 2007/05/30_16:24:13 info: Running /etc/ha.d/resource.d/IPaddr2 10.3.9.250/24/eth0 stop
IPaddr2[18821]: 2007/05/30_16:24:13 ERROR: [/usr/lib/heartbeat/findif -C] failed
IPaddr2[18798]: 2007/05/30_16:24:13 ERROR: Illegal argument
ResourceManager[18315]: 2007/05/30_16:24:13 ERROR: Return code 2 from /etc/ha.d/resource.d/IPaddr2
ResourceManager[18315]: 2007/05/30_16:24:14 info: Retrying failed stop operation [IPaddr2::10.3.9.250/24/eth0]
ResourceManager[18315]: 2007/05/30_16:24:14 info: Running /etc/ha.d/resource.d/IPaddr2 10.3.9.250/24/eth0 stop
IPaddr2[18896]: 2007/05/30_16:24:14 ERROR: [/usr/lib/heartbeat/findif -C] failed
IPaddr2[18873]: 2007/05/30_16:24:14 ERROR: Illegal argument
ResourceManager[18315]: 2007/05/30_16:24:14 ERROR: Return code 2 from /etc/ha.d/resource.d/IPaddr2
up.
wt-vm30:~ # tail -f /var/log/ha-log
heartbeat[22455]: 2007/05/30_16:27:51 info: G_main_add_TriggerHandler: Added signal manual handler
heartbeat[22455]: 2007/05/30_16:27:51 info: Removing /var/run/heartbeat/rsctmp failed, recreating.
heartbeat[22455]: 2007/05/30_16:27:51 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1
heartbeat[22455]: 2007/05/30_16:27:51 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1
heartbeat[22455]: 2007/05/30_16:27:51 info: glib: ping heartbeat started.
heartbeat[22455]: 2007/05/30_16:27:51 info: G_main_add_SignalHandler: Added signal handler for signal 17
heartbeat[22455]: 2007/05/30_16:27:51 info: Local status now set to: 'up'
heartbeat[22455]: 2007/05/30_16:27:52 info: Link 10.3.9.254:10.3.9.254 up.
heartbeat[22455]: 2007/05/30_16:27:52 info: Status update for node 10.3.9.254: status ping
heartbeat[22455]: 2007/05/30_16:27:52 info: Link wt-vm30:eth1 up.
heartbeat[22455]: 2007/05/30_16:28:07 info: Link wt-vm31:eth1 up.
heartbeat[22455]: 2007/05/30_16:28:07 info: Status update for node wt-vm31: status up
harc[22464]: 2007/05/30_16:28:07 info: Running /etc/ha.d/rc.d/status status
heartbeat[22455]: 2007/05/30_16:28:07 ERROR: HBDoMsg_T_REPNODES: Attempt to delete node 10.3.9.254
heartbeat[22455]: 2007/05/30_16:28:07 info: Comm_now_up(): updating status to active
heartbeat[22455]: 2007/05/30_16:28:07 info: Local status now set to: 'active'
heartbeat[22455]: 2007/05/30_16:28:07 info: Starting child client "/usr/lib/heartbeat/ipfail" (90,90)
heartbeat[22475]: 2007/05/30_16:28:07 info: Starting "/usr/lib/heartbeat/ipfail" as uid 90 gid 90 (pid 22475)
heartbeat[22455]: 2007/05/30_16:28:08 info: Status update for node wt-vm31: status active
harc[22476]: 2007/05/30_16:28:08 info: Running /etc/ha.d/rc.d/status status
ip-request-resp[22519]: 2007/05/30_16:28:19 received ip-request-resp drbddisk::r0 OK yes
ResourceManager[22534]: 2007/05/30_16:28:19 info: Acquiring resource group: wt-vm30 drbddisk::r0 Filesystem::/dev/drbd0::/drbd::ext3 apache2::/etc/apache2/httpd.conf IPaddr::10.3.9.250/24/eth0
ResourceManager[22534]: 2007/05/30_16:28:19 info: Running /etc/ha.d/resource.d/drbddisk r0 start
Filesystem[22599]: 2007/05/30_16:28:19 INFO: Resource is stopped
ResourceManager[22534]: 2007/05/30_16:28:19 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /drbd ext3 start
Filesystem[22644]: 2007/05/30_16:28:19 INFO: Running start for /dev/drbd0 on /drbd
Filesystem[22639]: 2007/05/30_16:28:19 INFO: Success
ResourceManager[22534]: 2007/05/30_16:28:19 info: Running /etc/init.d/apache2 /etc/apache2/httpd.conf start
ResourceManager[22534]: 2007/05/30_16:28:19 ERROR: Return code 1 from /etc/init.d/apache2
ResourceManager[22534]: 2007/05/30_16:28:19 CRIT: Giving up resources due to failure of apache2::/etc/apache2/httpd.conf
ResourceManager[22534]: 2007/05/30_16:28:19 info: Releasing resource group: wt-vm30 drbddisk::r0 Filesystem::/dev/drbd0::/drbd::ext3 apache2::/etc/apache2/httpd.conf IPaddr::10.3.9.250/24/eth0
ResourceManager[22534]: 2007/05/30_16:28:19 info: Running /etc/ha.d/resource.d/IPaddr 10.3.9.250/24/eth0 stop
IPaddr[22777]: 2007/05/30_16:28:20 INFO: Success
ResourceManager[22534]: 2007/05/30_16:28:20 info: Running /etc/init.d/apache2 /etc/apache2/httpd.conf stop
ResourceManager[22534]: 2007/05/30_16:28:20 ERROR: Return code 1 from /etc/init.d/apache2
ResourceManager[22534]: 2007/05/30_16:28:21 info: Retrying failed stop operation [apache2::/etc/apache2/httpd.conf]
ResourceManager[22534]: 2007/05/30_16:28:21 info: Running /etc/init.d/apache2 /etc/apache2/httpd.conf stop
ResourceManager[22534]: 2007/05/30_16:28:21 ERROR: Return code 1 from /etc/init.d/apache2
ResourceManager[22534]: 2007/05/30_16:28:22 info: Retrying failed stop operation [apache2::/etc/apache2/httpd.conf]
ResourceManager[22534]: 2007/05/30_16:28:22 info: Running /etc/init.d/apache2 /etc/apache2/httpd.conf stop
ResourceManager[22534]: 2007/05/30_16:28:22 ERROR: Return code 1 from /etc/init.d/apache2
ResourceManager[22534]: 2007/05/30_16:28:23 info: Retrying failed stop operation [apache2::/etc/apache2/httpd.conf]
ResourceManager[22534]: 2007/05/30_16:28:23 info: Running /etc/init.d/apache2 /etc/apache2/httpd.conf stop
ResourceManager[22534]: 2007/05/30_16:28:23 ERROR: Return code 1 from /etc/init.d/apache2
ResourceManager[22534]: 2007/05/30_16:28:24 info: Retrying failed stop operation [apache2::/etc/apache2/httpd.conf]
ResourceManager[22534]: 2007/05/30_16:28:25 info: Running /etc/init.d/apache2 /etc/apache2/httpd.conf stop
ResourceManager[22534]: 2007/05/30_16:28:25 ERROR: Return code 1 from /etc/init.d/apache2
ResourceManager[22534]: 2007/05/30_16:28:26 info: Retrying failed stop operation [apache2::/etc/apache2/httpd.conf]
ResourceManager[22534]: 2007/05/30_16:28:26 info: Running /etc/init.d/apache2 /etc/apache2/httpd.conf stop
ResourceManager[22534]: 2007/05/30_16:28:26 ERROR: Return code 1 from /etc/init.d/apache2
ResourceManager[22534]: 2007/05/30_16:28:27 info: Retrying failed stop operation [apache2::/etc/apache2/httpd.conf]
ResourceManager[22534]: 2007/05/30_16:28:27 info: Running /etc/init.d/apache2 /etc/apache2/httpd.conf stop
ResourceManager[22534]: 2007/05/30_16:28:27 ERROR: Return code 1 from /etc/init.d/apache2
ResourceManager[22534]: 2007/05/30_16:28:28 info: Retrying failed stop operation [apache2::/etc/apache2/httpd.conf]
ResourceManager[22534]: 2007/05/30_16:28:28 info: Running /etc/init.d/apache2 /etc/apache2/httpd.conf stop
ResourceManager[22534]: 2007/05/30_16:28:28 ERROR: Return code 1 from /etc/init.d/apache2
ResourceManager[22534]: 2007/05/30_16:28:32 ERROR: Return code 1 from /etc/init.d/apache2
ResourceManager[22534]: 2007/05/30_16:28:32 ERROR: Resource script for apache2::/etc/apache2/httpd.conf probably not LSB-compliant.
ResourceManager[22534]: 2007/05/30_16:28:32 WARN: it (apache2::/etc/apache2/httpd.conf) MUST succeed on a stop when already stopped
ResourceManager[22534]: 2007/05/30_16:28:32 WARN: Machine reboot narrowly avoided!
ResourceManager[22534]: 2007/05/30_16:28:32 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /drbd ext3 stop
Filesystem[23449]: 2007/05/30_16:28:32 INFO: Running stop for /dev/drbd0 on /drbd
Filesystem[23449]: 2007/05/30_16:28:32 INFO: Trying to unmount /drbd
Filesystem[23449]: 2007/05/30_16:28:32 INFO: unmounted /drbd successfully
Filesystem[23444]: 2007/05/30_16:28:32 INFO: Success
ResourceManager[22534]: 2007/05/30_16:28:32 info: Running /etc/ha.d/resource.d/drbddisk r0 stop
hb_standby[23527]: 2007/05/30_16:29:02 Going standby [foreign].
heartbeat[22455]: 2007/05/30_16:29:02 info: wt-vm30 wants to go standby [foreign]
heartbeat[22455]: 2007/05/30_16:29:03 info: standby: wt-vm31 can take our foreign resources
heartbeat[23538]: 2007/05/30_16:29:03 info: give up foreign HA resources (standby).
heartbeat[23538]: 2007/05/30_16:29:03 info: foreign HA resource release completed (standby).
heartbeat[22455]: 2007/05/30_16:29:03 info: Local standby process completed [foreign].
heartbeat[22455]: 2007/05/30_16:29:04 WARN: 1 lost packet(s) for [wt-vm31] [43:45]
heartbeat[22455]: 2007/05/30_16:29:04 info: remote resource transition completed.
heartbeat[22455]: 2007/05/30_16:29:04 info: No pkts missing from wt-vm31!
heartbeat[22455]: 2007/05/30_16:29:04 info: Other node completed standby takeover of foreign resources.