rendered paste body[2017-05-03 20:18:01.619691] I [MSGID: 100030] [glusterfsd.c:2454:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.8.11 (args: /usr/sbin/glusterfs --volfile-server=srvpve2g --volfi
le-id=datastore2 /mnt/pve/datastore2)
[2017-05-03 20:18:01.625747] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2017-05-03 20:18:01.625785] E [socket.c:2309:socket_connect_finish] 0-glusterfs: connection to 192.168.102.2:24007 failed (Connection refused)
[2017-05-03 20:18:01.625807] E [glusterfsd-mgmt.c:1908:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: srvpve2g (Transport endpoint is not connected)
[2017-05-03 20:18:01.625816] I [glusterfsd-mgmt.c:1926:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers
[2017-05-03 20:18:01.625976] W [glusterfsd.c:1327:cleanup_and_exit] (-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0xda) [0x7fe21a4bf59a] -->/usr/sbin/glusterfs(+0x1013e) [0x55d7ddf8a13e] -->/u
sr/sbin/glusterfs(cleanup_and_exit+0x57) [0x55d7ddf83967] ) 0-: received signum (1), shutting down
[2017-05-03 20:18:01.626002] I [fuse-bridge.c:5788:fini] 0-fuse: Unmounting '/mnt/pve/datastore2'.
[2017-05-04 13:03:21.997503] I [MSGID: 100030] [glusterfsd.c:2454:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.8.11 (args: /usr/sbin/glusterfs --volfile-server=srvpve3g --volfi
le-id=datastore2 /mnt/pve/datastore2)
[2017-05-04 13:03:22.005099] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2017-05-04 13:03:22.014008] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2017-05-04 13:03:22.014584] I [MSGID: 114020] [client.c:2356:notify] 0-datastore2-client-0: parent translators are ready, attempting connect on transport
[2017-05-04 13:03:22.014896] I [MSGID: 114020] [client.c:2356:notify] 0-datastore2-client-1: parent translators are ready, attempting connect on transport
[2017-05-04 13:03:22.015079] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-datastore2-client-0: changing port to 49157 (from 0)
[2017-05-04 13:03:22.015112] I [MSGID: 114020] [client.c:2356:notify] 0-datastore2-client-2: parent translators are ready, attempting connect on transport
Final graph:
+------------------------------------------------------------------------------+
1: volume datastore2-client-0
2: type protocol/client
3: option ping-timeout 42
4: option remote-host srvpve2g
5: option remote-subvolume /data/brick2/brick
6: option transport-type socket
7: option transport.address-family inet
8: option send-gids true
9: end-volume
10:
11: volume datastore2-client-1
12: type protocol/client
13: option ping-timeout 42
14: option remote-host srvpve3g
15: option remote-subvolume /data/brick2/brick
16: option transport-type socket
17: option transport.address-family inet
18: option send-gids true
19: end-volume
20:
21: volume datastore2-client-2
22: type protocol/client
23: option ping-timeout 42
24: option remote-host srvpve1g
25: option remote-subvolume /data/brick2/brick
26: option transport-type socket
27: option transport.address-family inet
28: option send-gids true
29: end-volume
30:
31: volume datastore2-replicate-0
32: type cluster/replicate
33: option arbiter-count 1
34: subvolumes datastore2-client-0 datastore2-client-1 datastore2-client-2
35: end-volume
36:
37: volume datastore2-dht
38: type cluster/distribute
39: option lock-migration off
40: subvolumes datastore2-replicate-0
41: end-volume
42:
43: volume datastore2-write-behind
44: type performance/write-behind
45: subvolumes datastore2-dht
46: end-volume
47:
48: volume datastore2-read-ahead
49: type performance/read-ahead
50: subvolumes datastore2-write-behind
51: end-volume
52:
53: volume datastore2-readdir-ahead
54: type performance/readdir-ahead
55: subvolumes datastore2-read-ahead
56: end-volume
57:
58: volume datastore2-io-cache
59: type performance/io-cache
60: subvolumes datastore2-readdir-ahead
61: end-volume
62:
63: volume datastore2-quick-read
64: type performance/quick-read
65: subvolumes datastore2-io-cache
66: end-volume
67:
68: volume datastore2-open-behind
69: type performance/open-behind
70: subvolumes datastore2-quick-read
71: end-volume
72:
73: volume datastore2-md-cache
74: type performance/md-cache
75: subvolumes datastore2-open-behind
76: end-volume
77:
78: volume datastore2
79: type debug/io-stats
80: option log-level INFO
81: option latency-measurement off
82: option count-fop-hits off
83: subvolumes datastore2-md-cache
84: end-volume
85:
86: volume meta-autoload
87: type meta
88: subvolumes datastore2
89: end-volume
90:
+------------------------------------------------------------------------------+
[2017-05-04 13:03:22.015602] I [MSGID: 114057] [client-handshake.c:1440:select_server_supported_programs] 0-datastore2-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-05-04 13:03:22.015885] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-datastore2-client-0: Connected to datastore2-client-0, attached to remote volume '/data/brick2/brick'.
[2017-05-04 13:03:22.015896] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-datastore2-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2017-05-04 13:03:22.015928] I [MSGID: 108005] [afr-common.c:4387:afr_notify] 0-datastore2-replicate-0: Subvolume 'datastore2-client-0' came back up; going online.
[2017-05-04 13:03:22.015975] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-datastore2-client-0: Server lk version = 1
[2017-05-04 13:03:22.015989] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-datastore2-client-2: changing port to 49155 (from 0)
[2017-05-04 13:03:22.016024] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-datastore2-client-1: changing port to 49153 (from 0)
[2017-05-04 13:03:22.016585] I [MSGID: 114057] [client-handshake.c:1440:select_server_supported_programs] 0-datastore2-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-05-04 13:03:22.016780] I [MSGID: 114057] [client-handshake.c:1440:select_server_supported_programs] 0-datastore2-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-05-04 13:03:22.016905] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-datastore2-client-2: Connected to datastore2-client-2, attached to remote volume '/data/brick2/brick'.
[2017-05-04 13:03:22.016914] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-datastore2-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2017-05-04 13:03:22.017039] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-datastore2-client-2: Server lk version = 1
[2017-05-04 13:03:22.018221] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-datastore2-client-1: Connected to datastore2-client-1, attached to remote volume '/data/brick2/brick'.
[2017-05-04 13:03:22.018236] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-datastore2-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2017-05-04 13:03:22.023226] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-datastore2-client-1: Server lk version = 1
[2017-05-04 13:03:22.023340] I [fuse-bridge.c:4147:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.23
[2017-05-04 13:03:22.023382] I [fuse-bridge.c:4832:fuse_graph_sync] 0-fuse: switched to graph 0
[2017-05-04 13:03:22.024323] I [MSGID: 108031] [afr-common.c:2157:afr_local_discovery_cbk] 0-datastore2-replicate-0: selecting local read_child datastore2-client-0
[2017-05-04 13:08:12.034040] W [socket.c:590:__socket_rwv] 0-glusterfs: readv on 192.168.102.3:24007 failed (No data available)
[2017-05-04 13:08:12.034079] E [glusterfsd-mgmt.c:1908:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: srvpve3g (No data available)
[2017-05-04 13:08:12.034096] I [glusterfsd-mgmt.c:1926:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers
[2017-05-04 13:08:23.028587] E [socket.c:2309:socket_connect_finish] 0-glusterfs: connection to 192.168.102.3:24007 failed (Connection refused)
[2017-05-04 13:08:35.897005] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2017-05-04 13:10:09.183071] W [glusterfsd.c:1327:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x8064) [0x7f7ac2490064] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x563aeb3b9af5] -->/
usr/sbin/glusterfs(cleanup_and_exit+0x57) [0x563aeb3b9967] ) 0-: received signum (15), shutting down
[2017-05-04 13:10:09.183098] I [fuse-bridge.c:5788:fini] 0-fuse: Unmounting '/mnt/pve/datastore2'.
[2017-05-04 13:10:09.220777] I [fuse-bridge.c:5080:fuse_thread_proc] 0-fuse: unmounting /mnt/pve/datastore2
[2017-05-04 13:15:40.464333] I [MSGID: 100030] [glusterfsd.c:2454:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.8.11 (args: /usr/sbin/glusterfs --volfile-server=srvpve3g --volfi
le-id=datastore2 /mnt/pve/datastore2)
[2017-05-04 13:15:40.471061] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2017-05-04 13:15:40.479485] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2017-05-04 13:15:40.480057] I [MSGID: 114020] [client.c:2356:notify] 0-datastore2-client-0: parent translators are ready, attempting connect on transport
[2017-05-04 13:15:40.480371] I [MSGID: 114020] [client.c:2356:notify] 0-datastore2-client-1: parent translators are ready, attempting connect on transport
[2017-05-04 13:15:40.480577] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-datastore2-client-0: changing port to 49157 (from 0)
[2017-05-04 13:15:40.480639] I [MSGID: 114020] [client.c:2356:notify] 0-datastore2-client-2: parent translators are ready, attempting connect on transport
Final graph:
+------------------------------------------------------------------------------+
1: volume datastore2-client-0
2: type protocol/client
3: option ping-timeout 42
4: option remote-host srvpve2g
5: option remote-subvolume /data/brick2/brick
6: option transport-type socket
7: option transport.address-family inet
8: option send-gids true
9: end-volume
10:
11: volume datastore2-client-1
12: type protocol/client
13: option ping-timeout 42
14: option remote-host srvpve3g
15: option remote-subvolume /data/brick2/brick
16: option transport-type socket
17: option transport.address-family inet
18: option send-gids true
19: end-volume
20:
21: volume datastore2-client-2
22: type protocol/client
23: option ping-timeout 42
24: option remote-host srvpve1g
25: option remote-subvolume /data/brick2/brick
26: option transport-type socket
27: option transport.address-family inet
28: option send-gids true
29: end-volume
30:
31: volume datastore2-replicate-0
32: type cluster/replicate
33: option arbiter-count 1
34: subvolumes datastore2-client-0 datastore2-client-1 datastore2-client-2
35: end-volume
36:
37: volume datastore2-dht
38: type cluster/distribute
39: option lock-migration off
40: subvolumes datastore2-replicate-0
41: end-volume
42:
43: volume datastore2-write-behind
44: type performance/write-behind
45: subvolumes datastore2-dht
46: end-volume
47:
48: volume datastore2-read-ahead
49: type performance/read-ahead
50: subvolumes datastore2-write-behind
51: end-volume
52:
53: volume datastore2-readdir-ahead
54: type performance/readdir-ahead
55: subvolumes datastore2-read-ahead
56: end-volume
57:
58: volume datastore2-io-cache
59: type performance/io-cache
60: subvolumes datastore2-readdir-ahead
61: end-volume
62:
63: volume datastore2-quick-read
64: type performance/quick-read
65: subvolumes datastore2-io-cache
66: end-volume
67:
68: volume datastore2-open-behind
69: type performance/open-behind
70: subvolumes datastore2-quick-read
71: end-volume
72:
73: volume datastore2-md-cache
74: type performance/md-cache
75: subvolumes datastore2-open-behind
76: end-volume
77:
78: volume datastore2
79: type debug/io-stats
80: option log-level INFO
81: option latency-measurement off
82: option count-fop-hits off
83: subvolumes datastore2-md-cache
84: end-volume
85:
86: volume meta-autoload
87: type meta
88: subvolumes datastore2
89: end-volume
90:
+------------------------------------------------------------------------------+
[2017-05-04 13:15:40.481282] I [MSGID: 114057] [client-handshake.c:1440:select_server_supported_programs] 0-datastore2-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-05-04 13:15:40.481481] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-datastore2-client-2: changing port to 49155 (from 0)
[2017-05-04 13:15:40.481616] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-datastore2-client-1: changing port to 49153 (from 0)
[2017-05-04 13:15:40.481811] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-datastore2-client-0: Connected to datastore2-client-0, attached to remote volume '/data/brick2/brick'.
[2017-05-04 13:15:40.481823] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-datastore2-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2017-05-04 13:15:40.481870] I [MSGID: 108005] [afr-common.c:4387:afr_notify] 0-datastore2-replicate-0: Subvolume 'datastore2-client-0' came back up; going online.
[2017-05-04 13:15:40.481928] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-datastore2-client-0: Server lk version = 1
[2017-05-04 13:15:40.482039] I [MSGID: 114057] [client-handshake.c:1440:select_server_supported_programs] 0-datastore2-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-05-04 13:15:40.482333] I [MSGID: 114057] [client-handshake.c:1440:select_server_supported_programs] 0-datastore2-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-05-04 13:15:40.482402] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-datastore2-client-2: Connected to datastore2-client-2, attached to remote volume '/data/brick2/brick'.
[2017-05-04 13:15:40.482411] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-datastore2-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2017-05-04 13:15:40.482537] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-datastore2-client-2: Server lk version = 1
[2017-05-04 13:15:40.483155] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-datastore2-client-1: Connected to datastore2-client-1, attached to remote volume '/data/brick2/brick'.
[2017-05-04 13:15:40.483162] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-datastore2-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2017-05-04 13:15:40.487983] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-datastore2-client-1: Server lk version = 1
[2017-05-04 13:15:40.488107] I [fuse-bridge.c:4147:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.23
[2017-05-04 13:15:40.488124] I [fuse-bridge.c:4832:fuse_graph_sync] 0-fuse: switched to graph 0
[2017-05-04 13:15:40.488930] I [MSGID: 108031] [afr-common.c:2157:afr_local_discovery_cbk] 0-datastore2-replicate-0: selecting local read_child datastore2-client-0