rendered paste body[ 8390.456752] LustreError: 1793:0:(socklnd_cb.c:1945:ksocknal_find_timed_out_conn()) Timed out TX to 12345-172.16.3.21@tcp1 Q 0 ffff81007a708000 172.16.3.21
[ 8390.473288] LustreError: 1793:0:(socklnd_cb.c:1977:ksocknal_check_peer_timeouts()) Timeout out conn->12345-172.16.3.21@tcp1 ip 172.16.3.21:988
[ 8390.488590] LustreError: A timeout occurred sending data to 172.16.3.22; the network or that node may be down.
[ 8390.490967] BUG: soft lockup detected on CPU#0!
[ 8390.490970] CPU 0:
[ 8390.490972] Modules linked in: nfsd exportfs osc llite lov mdc ksocklnd ptlrpc lnet obdclass lvfs libcfs bluesmoke_k8 bluesmoke_mc mptspi mptscsih mptbase bcm5700 w83627hf e
[ 8390.490988] Pid: 4653, comm: nfsd Not tainted 2.6.15.7 #1
[ 8390.490990] RIP: 0010:[<ffffffff801b5bd4>] <ffffffff801b5bd4>{get_num_ver+296}
[ 8390.490997] RSP: 0018:ffff810056a5b668 EFLAGS: 00000246
[ 8390.491000] RAX: 0000000000000000 RBX: ffffffff8012a0fe RCX: 0000000000000000
[ 8390.491003] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff81001a264030
[ 8390.491005] RBP: ffff810056a5ba98 R08: 00000000ffffffff R09: 0000000000000000
[ 8390.491008] R10: ffff81001a264000 R11: ffff810056a5bc58 R12: 0000000000000008
[ 8390.491011] R13: ffff810056a5b778 R14: 0000000000000001 R15: ffff81007a082c00
[ 8390.491015] FS: 00002aaaaae00680(0000) GS:ffffffff80469800(0000) knlGS:00000000556a82a0
[ 8390.491017] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8390.491020] CR2: 00002aaaab459000 CR3: 0000000010502000 CR4: 00000000000006e0
[ 8390.491022]
[ 8390.491022] Call Trace:<ffffffff801b5bd4>{get_num_ver+296} <ffffffff801b718d>{fix_nodes+2039}
[ 8390.491034] <ffffffff801c2ae3>{reiserfs_paste_into_item+272} <ffffffff801acad3>{reiserfs_add_entry+727}
[ 8390.491071] <ffffffff80329293>{__up_wakeup+53} <ffffffff801acd92>{reiserfs_create+262}
[ 8390.491090] <ffffffff8017c5b8>{vfs_create+182} <ffffffff885f7f78>{:nfsd:nfsd_create_v3+791}
[ 8390.491115] <ffffffff885fdce8>{:nfsd:nfsd3_proc_create+308} <ffffffff885f2697>{:nfsd:nfsd_dispatch+219}
[ 8390.491140] <ffffffff803198c4>{svc_process+1011} <ffffffff8012a0ad>{default_wake_function+0}
[ 8390.491149] <ffffffff885f2267>{:nfsd:nfsd+0} <ffffffff885f242d>{:nfsd:nfsd+454}
[ 8390.491169] <ffffffff8010e746>{child_rip+8} <ffffffff885f2267>{:nfsd:nfsd+0}
[ 8390.491182] <ffffffff885f2267>{:nfsd:nfsd+0} <ffffffff8010e73e>{child_rip+0}
[ 8390.491197]
[ 8390.725330] LustreError: 1793:0:(socklnd_cb.c:1945:ksocknal_find_timed_out_conn()) Timed out TX to 12345-172.16.3.22@tcp1 Q 0 ffff81003b09c000 172.16.3.22
[ 8390.741871] LustreError: 1793:0:(socklnd_cb.c:1977:ksocknal_check_peer_timeouts()) Timeout out conn->12345-172.16.3.22@tcp1 ip 172.16.3.22:988
[ 8390.757169] LustreError: A timeout occurred sending data to 172.16.3.23; the network or that node may be down.
[ 8390.769147] LustreError: 1793:0:(socklnd_cb.c:1945:ksocknal_find_timed_out_conn()) Timed out TX to 12345-172.16.3.23@tcp1 Q 0 ffff81003b204000 172.16.3.23
[ 8390.785697] LustreError: 1793:0:(socklnd_cb.c:1977:ksocknal_check_peer_timeouts()) Timeout out conn->12345-172.16.3.23@tcp1 ip 172.16.3.23:988
[ 8390.800993] LustreError: A timeout occurred sending data to 172.16.3.24; the network or that node may be down.
[ 8390.812984] LustreError: 1793:0:(socklnd_cb.c:1945:ksocknal_find_timed_out_conn()) Timed out TX to 12345-172.16.3.24@tcp1 Q 0 ffff81003b200000 172.16.3.24
[ 8390.829520] LustreError: 1793:0:(socklnd_cb.c:1977:ksocknal_check_peer_timeouts()) Timeout out conn->12345-172.16.3.24@tcp1 ip 172.16.3.24:988
[ 8390.844848] Lustre: 1793:0:(router.c:184:lnet_notify()) Upcall: NID 172.16.3.21@tcp1 is dead
[ 8390.855288] Lustre: 9:0:(linux-debug.c:96:libcfs_run_upcall()) Invoked portals upcall /usr/lib/lustre/lnet_upcall ROUTER_NOTIFY,172.16.3.21@tcp1,down,1152188191
[ 8394.440878] LustreError: A timeout occurred sending data to 172.16.3.253; the network or that node may be down.
[ 8394.452970] LustreError: 1793:0:(socklnd_cb.c:1945:ksocknal_find_timed_out_conn()) Timed out TX to 12345-172.16.3.253@tcp1 Q 0 ffff81007a6d0000 172.16.3.253
[ 8394.469717] LustreError: 1793:0:(socklnd_cb.c:1977:ksocknal_check_peer_timeouts()) Timeout out conn->12345-172.16.3.253@tcp1 ip 172.16.3.253:988
[ 8394.485234] Lustre: 1793:0:(router.c:184:lnet_notify()) Upcall: NID 172.16.3.253@tcp1 is dead
[ 8394.495446] Lustre: 1793:0:(router.c:184:lnet_notify()) previously skipped 3 similar messages