rendered paste bodyheartbeat[3512]: 2007/05/03_21:46:11 WARN: node wt-vm30: is dead
heartbeat[3512]: 2007/05/03_21:46:11 WARN: No STONITH device configured.
heartbeat[3512]: 2007/05/03_21:46:11 WARN: Shared disks are not protected.
heartbeat[3512]: 2007/05/03_21:46:11 info: Resources being acquired from wt-vm30.
heartbeat[3512]: 2007/05/03_21:46:11 info: Link wt-vm30:eth1 dead.
harc[3590]: 2007/05/03_21:46:11 info: Running /etc/ha.d/rc.d/status status
heartbeat[3591]: 2007/05/03_21:46:12 info: No local resources [/usr/lib/heartbeat/ResourceManager listkeys wt-vm31] to acquire.
mach_down[3601]: 2007/05/03_21:46:12 info: Taking over resource group IPaddr::10.3.9.250/24/eth0
ResourceManager[3630]: 2007/05/03_21:46:12 info: Acquiring resource group: wt-vm30 IPaddr::10.3.9.250/24/eth0 drbddisk::drbd0 Filesystem::/dev/drbd0::/srv::ext3 httpd
IPaddr[3654]: 2007/05/03_21:46:12 INFO: Resource is stopped
ResourceManager[3630]: 2007/05/03_21:46:12 info: Running /etc/ha.d/resource.d/IPaddr 10.3.9.250/24/eth0 start
IPaddr[3730]: 2007/05/03_21:46:12 INFO: Using calculated netmask for 10.3.9.250: 255.255.255.0
IPaddr[3730]: 2007/05/03_21:46:12 DEBUG: Using calculated broadcast for 10.3.9.250: 10.3.9.255
IPaddr[3730]: 2007/05/03_21:46:13 INFO: eval /sbin/ifconfig eth0:0 10.3.9.250 netmask 255.255.255.0 broadcast 10.3.9.255
IPaddr[3730]: 2007/05/03_21:46:13 DEBUG: Sending Gratuitous Arp for 10.3.9.250 on eth0:0 [eth0]
IPaddr[3709]: 2007/05/03_21:46:13 INFO: Success
ResourceManager[3630]: 2007/05/03_21:46:13 info: Running /etc/ha.d/resource.d/drbddisk drbd0 start
Filesystem[3861]: 2007/05/03_21:46:13 INFO: Resource is stopped
ResourceManager[3630]: 2007/05/03_21:46:13 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /srv ext3 start
Filesystem[3902]: 2007/05/03_21:46:13 INFO: Running start for /dev/drbd0 on /srv
Filesystem[3899]: 2007/05/03_21:46:13 INFO: Success
ResourceManager[3630]: 2007/05/03_21:46:14 info: Running /etc/init.d/httpd start
ResourceManager[3630]: 2007/05/03_21:46:14 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[3630]: 2007/05/03_21:46:14 CRIT: Giving up resources due to failure of httpd
ResourceManager[3630]: 2007/05/03_21:46:14 info: Releasing resource group: wt-vm30 IPaddr::10.3.9.250/24/eth0 drbddisk::drbd0 Filesystem::/dev/drbd0::/srv::ext3 httpd
ResourceManager[3630]: 2007/05/03_21:46:15 info: Running /etc/init.d/httpd stop
ResourceManager[3630]: 2007/05/03_21:46:15 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[3630]: 2007/05/03_21:46:16 info: Retrying failed stop operation [httpd]
ResourceManager[3630]: 2007/05/03_21:46:16 info: Running /etc/init.d/httpd stop
ResourceManager[3630]: 2007/05/03_21:46:16 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[3630]: 2007/05/03_21:46:17 info: Retrying failed stop operation [httpd]
ResourceManager[3630]: 2007/05/03_21:46:17 info: Running /etc/init.d/httpd stop
ResourceManager[3630]: 2007/05/03_21:46:17 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[3630]: 2007/05/03_21:46:18 info: Retrying failed stop operation [httpd]
ResourceManager[3630]: 2007/05/03_21:46:18 info: Running /etc/init.d/httpd stop
ResourceManager[3630]: 2007/05/03_21:46:19 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[3630]: 2007/05/03_21:46:20 info: Retrying failed stop operation [httpd]
ResourceManager[3630]: 2007/05/03_21:46:20 info: Running /etc/init.d/httpd stop
ResourceManager[3630]: 2007/05/03_21:46:20 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[3630]: 2007/05/03_21:46:21 info: Retrying failed stop operation [httpd]
ResourceManager[3630]: 2007/05/03_21:46:21 info: Running /etc/init.d/httpd stop
ResourceManager[3630]: 2007/05/03_21:46:21 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[3630]: 2007/05/03_21:46:22 info: Retrying failed stop operation [httpd]
ResourceManager[3630]: 2007/05/03_21:46:22 info: Running /etc/init.d/httpd stop
ResourceManager[3630]: 2007/05/03_21:46:22 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[3630]: 2007/05/03_21:46:23 info: Retrying failed stop operation [httpd]
ResourceManager[3630]: 2007/05/03_21:46:24 info: Running /etc/init.d/httpd stop
ResourceManager[3630]: 2007/05/03_21:46:24 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[3630]: 2007/05/03_21:46:25 info: Retrying failed stop operation [httpd]
ResourceManager[3630]: 2007/05/03_21:46:25 info: Running /etc/init.d/httpd stop
ResourceManager[3630]: 2007/05/03_21:46:25 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[3630]: 2007/05/03_21:46:26 info: Retrying failed stop operation [httpd]
ResourceManager[3630]: 2007/05/03_21:46:26 info: Running /etc/init.d/httpd stop
ResourceManager[3630]: 2007/05/03_21:46:26 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[3630]: 2007/05/03_21:46:27 info: Retrying failed stop operation [httpd]
ResourceManager[3630]: 2007/05/03_21:46:27 info: Running /etc/init.d/httpd stop
ResourceManager[3630]: 2007/05/03_21:46:28 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[3630]: 2007/05/03_21:46:28 ERROR: Resource script for httpd probably not LSB-compliant.
ResourceManager[3630]: 2007/05/03_21:46:28 WARN: it (httpd) MUST succeed on a stop when already stopped
ResourceManager[3630]: 2007/05/03_21:46:28 WARN: Machine reboot narrowly avoided!
ResourceManager[3630]: 2007/05/03_21:46:28 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /srv ext3 stop
Filesystem[4457]: 2007/05/03_21:46:28 INFO: Running stop for /dev/drbd0 on /srv
Filesystem[4457]: 2007/05/03_21:46:28 INFO: Trying to unmount /srv
Filesystem[4457]: 2007/05/03_21:46:28 INFO: unmounted /srv successfully
Filesystem[4454]: 2007/05/03_21:46:28 INFO: Success
ResourceManager[3630]: 2007/05/03_21:46:28 info: Running /etc/ha.d/resource.d/drbddisk drbd0 stop
ResourceManager[3630]: 2007/05/03_21:46:28 info: Running /etc/ha.d/resource.d/IPaddr 10.3.9.250/24/eth0 stop
IPaddr[4565]: 2007/05/03_21:46:29 INFO: /sbin/ifconfig eth0:0 10.3.9.250 down
IPaddr[4544]: 2007/05/03_21:46:29 INFO: Success
mach_down[3601]: 2007/05/03_21:46:29 info: /usr/lib/heartbeat/mach_down: nice_failback: foreign resources acquired
mach_down[3601]: 2007/05/03_21:46:29 info: mach_down takeover complete for node wt-vm30.
heartbeat[3512]: 2007/05/03_21:46:29 info: mach_down takeover complete.
hb_standby[4613]: 2007/05/03_21:46:59 Going standby [foreign].
heartbeat[3512]: 2007/05/03_21:46:59 info: wt-vm31 wants to go standby [foreign]
heartbeat[3512]: 2007/05/03_21:47:10 WARN: No reply to standby request. Standby request cancelled.
heartbeat[3512]: 2007/05/03_21:48:46 CRIT: Cluster node wt-vm30 returning after partition.
heartbeat[3512]: 2007/05/03_21:48:46 info: For information on cluster partitions, See URL: http://linux-ha.org/SplitBrain
heartbeat[3512]: 2007/05/03_21:48:46 WARN: Deadtime value may be too small.
heartbeat[3512]: 2007/05/03_21:48:46 info: See FAQ for information on tuning deadtime.
heartbeat[3512]: 2007/05/03_21:48:46 info: URL: http://linux-ha.org/FAQ#heavy_load
heartbeat[3512]: 2007/05/03_21:48:46 info: Link wt-vm30:eth1 up.
heartbeat[3512]: 2007/05/03_21:48:46 WARN: Late heartbeat: Node wt-vm30: interval 184360 ms
heartbeat[3512]: 2007/05/03_21:48:46 info: Status update for node wt-vm30: status active
harc[4625]: 2007/05/03_21:48:46 info: Running /etc/ha.d/rc.d/status status
heartbeat[3512]: 2007/05/03_21:48:46 ERROR: Ignored standby message 'other' from wt-vm30 in state 0
heartbeat[3512]: 2007/05/03_21:48:48 info: Heartbeat shutdown in progress. (3512)
heartbeat[4635]: 2007/05/03_21:48:48 info: Giving up all HA resources.
ResourceManager[4645]: 2007/05/03_21:48:48 info: Releasing resource group: wt-vm30 IPaddr::10.3.9.250/24/eth0 drbddisk::drbd0 Filesystem::/dev/drbd0::/srv::ext3 httpd
ResourceManager[4645]: 2007/05/03_21:48:48 info: Running /etc/init.d/httpd stop
ResourceManager[4645]: 2007/05/03_21:48:48 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[4645]: 2007/05/03_21:48:49 info: Retrying failed stop operation [httpd]
ResourceManager[4645]: 2007/05/03_21:48:49 info: Running /etc/init.d/httpd stop
ResourceManager[4645]: 2007/05/03_21:48:49 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[4645]: 2007/05/03_21:48:50 info: Retrying failed stop operation [httpd]
ResourceManager[4645]: 2007/05/03_21:48:50 info: Running /etc/init.d/httpd stop
ResourceManager[4645]: 2007/05/03_21:48:51 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[4645]: 2007/05/03_21:48:52 info: Retrying failed stop operation [httpd]
ResourceManager[4645]: 2007/05/03_21:48:52 info: Running /etc/init.d/httpd stop
ResourceManager[4645]: 2007/05/03_21:48:52 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[4645]: 2007/05/03_21:48:53 info: Retrying failed stop operation [httpd]
ResourceManager[4645]: 2007/05/03_21:48:53 info: Running /etc/init.d/httpd stop
ResourceManager[4645]: 2007/05/03_21:48:53 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[4645]: 2007/05/03_21:48:54 info: Retrying failed stop operation [httpd]
ResourceManager[4645]: 2007/05/03_21:48:54 info: Running /etc/init.d/httpd stop
ResourceManager[4645]: 2007/05/03_21:48:54 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[4645]: 2007/05/03_21:48:55 info: Retrying failed stop operation [httpd]
ResourceManager[4645]: 2007/05/03_21:48:56 info: Running /etc/init.d/httpd stop
ResourceManager[4645]: 2007/05/03_21:48:56 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[4645]: 2007/05/03_21:48:57 info: Retrying failed stop operation [httpd]
ResourceManager[4645]: 2007/05/03_21:48:57 info: Running /etc/init.d/httpd stop
ResourceManager[4645]: 2007/05/03_21:48:57 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[4645]: 2007/05/03_21:48:58 info: Retrying failed stop operation [httpd]
ResourceManager[4645]: 2007/05/03_21:48:58 info: Running /etc/init.d/httpd stop
ResourceManager[4645]: 2007/05/03_21:48:58 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[4645]: 2007/05/03_21:48:59 info: Retrying failed stop operation [httpd]
ResourceManager[4645]: 2007/05/03_21:48:59 info: Running /etc/init.d/httpd stop
ResourceManager[4645]: 2007/05/03_21:49:00 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[4645]: 2007/05/03_21:49:01 info: Retrying failed stop operation [httpd]
ResourceManager[4645]: 2007/05/03_21:49:01 info: Running /etc/init.d/httpd stop
ResourceManager[4645]: 2007/05/03_21:49:01 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[4645]: 2007/05/03_21:49:01 ERROR: Resource script for httpd probably not LSB-compliant.
ResourceManager[4645]: 2007/05/03_21:49:01 WARN: it (httpd) MUST succeed on a stop when already stopped
ResourceManager[4645]: 2007/05/03_21:49:01 WARN: Machine reboot narrowly avoided!
ResourceManager[4645]: 2007/05/03_21:49:01 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /srv ext3 stop
Filesystem[5133]: 2007/05/03_21:49:01 INFO: Running stop for /dev/drbd0 on /srv
Filesystem[5130]: 2007/05/03_21:49:01 INFO: Success
ResourceManager[4645]: 2007/05/03_21:49:01 info: Running /etc/ha.d/resource.d/drbddisk drbd0 stop
ResourceManager[4645]: 2007/05/03_21:49:02 info: Running /etc/ha.d/resource.d/IPaddr 10.3.9.250/24/eth0 stop
IPaddr[5203]: 2007/05/03_21:49:02 INFO: Success
heartbeat[4635]: 2007/05/03_21:49:02 info: All HA resources relinquished.
heartbeat[3512]: 2007/05/03_21:49:03 WARN: 1 lost packet(s) for [wt-vm30] [52:54]
heartbeat[3512]: 2007/05/03_21:49:03 info: No pkts missing from wt-vm30!
heartbeat[3512]: 2007/05/03_21:49:03 info: killing /usr/lib/heartbeat/ipfail process group 3532 with signal 15
heartbeat[3512]: 2007/05/03_21:49:04 info: killing HBFIFO process 3515 with signal 15
heartbeat[3512]: 2007/05/03_21:49:04 info: killing HBWRITE process 3516 with signal 15
heartbeat[3512]: 2007/05/03_21:49:04 info: killing HBREAD process 3517 with signal 15
heartbeat[3512]: 2007/05/03_21:49:04 info: killing HBWRITE process 3518 with signal 15
heartbeat[3512]: 2007/05/03_21:49:04 info: killing HBREAD process 3519 with signal 15
heartbeat[3512]: 2007/05/03_21:49:04 info: Core process 3515 exited. 5 remaining
heartbeat[3512]: 2007/05/03_21:49:04 info: Core process 3516 exited. 4 remaining
heartbeat[3512]: 2007/05/03_21:49:04 info: Core process 3517 exited. 3 remaining
heartbeat[3512]: 2007/05/03_21:49:04 info: Core process 3518 exited. 2 remaining
heartbeat[3512]: 2007/05/03_21:49:04 info: Core process 3519 exited. 1 remaining
heartbeat[3512]: 2007/05/03_21:49:04 info: wt-vm31 Heartbeat shutdown complete.
heartbeat[3512]: 2007/05/03_21:49:04 info: Heartbeat restart triggered.
heartbeat[3512]: 2007/05/03_21:49:04 info: Restarting heartbeat.
heartbeat[3512]: 2007/05/03_21:49:04 info: Performing heartbeat restart exec.
heartbeat[3512]: 2007/05/03_21:49:35 WARN: Logging daemon is disabled --enabling logging daemon is recommended
heartbeat[3512]: 2007/05/03_21:49:35 info: **************************
heartbeat[3512]: 2007/05/03_21:49:35 info: Configuration validated. Starting heartbeat 2.0.8
heartbeat[5240]: 2007/05/03_21:49:35 info: heartbeat: version 2.0.8
heartbeat[5240]: 2007/05/03_21:49:35 info: Heartbeat generation: 42
heartbeat[5240]: 2007/05/03_21:49:35 info: G_main_add_TriggerHandler: Added signal manual handler
heartbeat[5240]: 2007/05/03_21:49:35 info: G_main_add_TriggerHandler: Added signal manual handler
heartbeat[5240]: 2007/05/03_21:49:35 info: Removing /var/run/heartbeat/rsctmp failed, recreating.
heartbeat[5240]: 2007/05/03_21:49:35 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1
heartbeat[5240]: 2007/05/03_21:49:35 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1
heartbeat[5240]: 2007/05/03_21:49:35 info: glib: ping heartbeat started.
heartbeat[5240]: 2007/05/03_21:49:35 info: G_main_add_SignalHandler: Added signal handler for signal 17
heartbeat[5240]: 2007/05/03_21:49:35 info: Local status now set to: 'up'
heartbeat[5240]: 2007/05/03_21:49:36 info: Link wt-vm30:eth1 up.
heartbeat[5240]: 2007/05/03_21:49:36 info: Status update for node wt-vm30: status active
heartbeat[5240]: 2007/05/03_21:49:36 info: Link 10.3.9.254:10.3.9.254 up.
heartbeat[5240]: 2007/05/03_21:49:36 info: Status update for node 10.3.9.254: status ping
heartbeat[5240]: 2007/05/03_21:49:36 info: Link wt-vm31:eth1 up.
harc[5248]: 2007/05/03_21:49:36 info: Running /etc/ha.d/rc.d/status status
heartbeat[5240]: 2007/05/03_21:49:37 info: Comm_now_up(): updating status to active
heartbeat[5240]: 2007/05/03_21:49:37 info: Local status now set to: 'active'
heartbeat[5240]: 2007/05/03_21:49:37 info: Starting child client "/usr/lib/heartbeat/ipfail" (500,500)
heartbeat[5259]: 2007/05/03_21:49:37 info: Starting "/usr/lib/heartbeat/ipfail" as uid 500 gid 500 (pid 5259)
heartbeat[5240]: 2007/05/03_21:49:37 info: remote resource transition completed.
heartbeat[5240]: 2007/05/03_21:49:37 info: remote resource transition completed.
heartbeat[5240]: 2007/05/03_21:49:37 info: Local Resource acquisition completed. (none)
heartbeat[5240]: 2007/05/03_21:49:37 info: Initial resource acquisition complete (T_RESOURCES(them))