2008/02/24

Red Hat Cluster: CentOS 5.1上でRHCSを使ってみる・その8・クラスタ動作の試験

前回までで、まったくサービスが乗っていないクラスタ、cDomUsを生成した。正常に動作するかどうかを試験する。『Red Hat Cluster: kernel-xenにバグか?』で報告したようなバグがないか、各ノードdc[123]を数度フェンスを実行する。
フェンス操作は、Luci管理画面からもコマンドラインからも実行できる。

Luci管理画面からのフェンス
まず、Luci管理画面から実行する。ブラウザからLuciにアクセスしログインする。[Luci Homebase]画面から、[cluster]タブ→[cDomUs]→[Nodes]と選択しcDomUsの管理画面を表示する。対象ノード(ここではdc2)のリストボックスから[Fence this node]を選択し、その右の[Go]ボタンを押下する。

確認を求めるダイアログに対して、[OK]を選択すると、進行状況が表示される。

すぐに元の画面に戻る。

このとき、フェンスされたノードdc2は即時に停止され(おそらく、fs1上のfence_xvmdによって、xm destroy dc2相当の操作がなされている)、再起動される。
Starting HAL daemon: [  OK  ]
Starting oddjobd: [  OK  ]
Starting Cluster Module - cluster monitor: Setting verbosity level to LogBasic
[  OK  ]
Starting Cluster Service Manager: [  OK  ]
dlm: lockspace 30003 from 1 type 1 not found
dlm: lockspace 30003 from 3 type 1 not found
dlm: lockspace 30003 from 1 type 1 not found
Starting ricci: [  OK  ]

CentOS release 5 (Final)
Kernel 2.6.18-53.1.13.el5xen on an i686

dc2.xencluster login:
一方、ノードdc3では、以下のようなログが出力される。
[root@dc3 ~]# tail -f /var/log/messages
Feb 24 11:20:46 dc3 fence_node[31277]: Fence of "dc2.xencluster" was successful
Feb 24 11:20:53 dc3 openais[1358]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 24 11:20:53 dc3 openais[1358]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 24 11:20:53 dc3 openais[1358]: [TOTEM] Transmit multicast socket send buffer size (219136 bytes).
Feb 24 11:20:53 dc3 openais[1358]: [TOTEM] entering GATHER state from 2.
dlm: closing connection to node 2
Feb 24 11:20:58 dc3 openais[1358]: [TOTEM] entering GATHER state from 0.
Feb 24 11:20:58 dc3 openais[1358]: [TOTEM] Saving state aru 4a high seq received 4a
Feb 24 11:20:58 dc3 openais[1358]: [TOTEM] Storing new sequence id for ring 38
Feb 24 11:20:58 dc3 openais[1358]: [TOTEM] entering COMMIT state.
Feb 24 11:20:58 dc3 openais[1358]: [TOTEM] entering RECOVERY state.
Feb 24 11:20:58 dc3 openais[1358]: [TOTEM] position [0] member 192.168.140.7:
Feb 24 11:20:58 dc3 openais[1358]: [TOTEM] previous ring seq 52 rep 192.168.140.7
Feb 24 11:20:58 dc3 openais[1358]: [TOTEM] aru 4a high delivered 4a received flag 1
Feb 24 11:20:58 dc3 openais[1358]: [TOTEM] position [1] member 192.168.140.9:
Feb 24 11:20:58 dc3 openais[1358]: [TOTEM] previous ring seq 52 rep 192.168.140.7
Feb 24 11:20:58 dc3 openais[1358]: [TOTEM] aru 4a high delivered 4a received flag 1
Feb 24 11:20:58 dc3 openais[1358]: [TOTEM] Did not need to originate any messages in recovery.
Feb 24 11:20:58 dc3 openais[1358]: [CLM  ] CLM CONFIGURATION CHANGE
Feb 24 11:20:58 dc3 openais[1358]: [CLM  ] New Configuration:
Feb 24 11:20:58 dc3 openais[1358]: [CLM  ]      r(0) ip(192.168.140.7)
Feb 24 11:20:58 dc3 openais[1358]: [CLM  ]      r(0) ip(192.168.140.9)
Feb 24 11:20:58 dc3 openais[1358]: [CLM  ] Members Left:
Feb 24 11:20:58 dc3 openais[1358]: [CLM  ]      r(0) ip(192.168.140.8)
Feb 24 11:20:58 dc3 openais[1358]: [CLM  ] Members Joined:
Feb 24 11:20:58 dc3 openais[1358]: [CLM  ] CLM CONFIGURATION CHANGE
Feb 24 11:20:58 dc3 openais[1358]: [CLM  ] New Configuration:
Feb 24 11:20:58 dc3 openais[1358]: [CLM  ]      r(0) ip(192.168.140.7)
Feb 24 11:20:58 dc3 openais[1358]: [CLM  ]      r(0) ip(192.168.140.9)
Feb 24 11:20:58 dc3 openais[1358]: [CLM  ] Members Left:
Feb 24 11:20:58 dc3 openais[1358]: [CLM  ] Members Joined:
Feb 24 11:20:58 dc3 openais[1358]: [SYNC ] This node is within the primary component and will provide service.
Feb 24 11:20:58 dc3 kernel: dlm: closing connection to node 2
Feb 24 11:20:58 dc3 openais[1358]: [TOTEM] entering OPERATIONAL state.
Feb 24 11:20:58 dc3 openais[1358]: [CLM  ] got nodejoin message 192.168.140.7
Feb 24 11:20:58 dc3 openais[1358]: [CLM  ] got nodejoin message 192.168.140.9
Feb 24 11:20:58 dc3 openais[1358]: [CPG  ] got joinlist message from node 3
Feb 24 11:20:58 dc3 openais[1358]: [CPG  ] got joinlist message from node 1
Feb 24 11:20:58 dc3 fenced[1374]: dc2.xencluster not a cluster member after 0 sec post_fail_delay
Feb 24 11:21:15 dc3 openais[1358]: [TOTEM] entering GATHER state from 11.
Feb 24 11:21:15 dc3 openais[1358]: [TOTEM] Saving state aru 21 high seq received 21
Feb 24 11:21:15 dc3 openais[1358]: [TOTEM] Storing new sequence id for ring 3c
Feb 24 11:21:15 dc3 openais[1358]: [TOTEM] entering COMMIT state.
Feb 24 11:21:15 dc3 openais[1358]: [TOTEM] entering RECOVERY state.
Feb 24 11:21:15 dc3 openais[1358]: [TOTEM] position [0] member 192.168.140.7:
Feb 24 11:21:15 dc3 openais[1358]: [TOTEM] previous ring seq 56 rep 192.168.140.7
Feb 24 11:21:15 dc3 openais[1358]: [TOTEM] aru 21 high delivered 21 received flag 1
Feb 24 11:21:15 dc3 openais[1358]: [TOTEM] position [1] member 192.168.140.8:
Feb 24 11:21:15 dc3 openais[1358]: [TOTEM] previous ring seq 56 rep 192.168.140.8
Feb 24 11:21:15 dc3 openais[1358]: [TOTEM] aru 1 high delivered 0 received flag 1
Feb 24 11:21:15 dc3 openais[1358]: [TOTEM] position [2] member 192.168.140.9:
Feb 24 11:21:15 dc3 openais[1358]: [TOTEM] previous ring seq 56 rep 192.168.140.7
Feb 24 11:21:15 dc3 openais[1358]: [TOTEM] aru 21 high delivered 21 received flag 1
Feb 24 11:21:15 dc3 openais[1358]: [TOTEM] Did not need to originate any messages in recovery.
Feb 24 11:21:15 dc3 openais[1358]: [CLM  ] CLM CONFIGURATION CHANGE
Feb 24 11:21:15 dc3 openais[1358]: [CLM  ] New Configuration:
Feb 24 11:21:15 dc3 openais[1358]: [CLM  ]      r(0) ip(192.168.140.7)
Feb 24 11:21:15 dc3 openais[1358]: [CLM  ]      r(0) ip(192.168.140.9)
Feb 24 11:21:15 dc3 openais[1358]: [CLM  ] Members Left:
Feb 24 11:21:15 dc3 openais[1358]: [CLM  ] Members Joined:
Feb 24 11:21:15 dc3 openais[1358]: [CLM  ] CLM CONFIGURATION CHANGE
Feb 24 11:21:15 dc3 openais[1358]: [CLM  ] New Configuration:
Feb 24 11:21:15 dc3 openais[1358]: [CLM  ]      r(0) ip(192.168.140.7)
Feb 24 11:21:15 dc3 openais[1358]: [CLM  ]      r(0) ip(192.168.140.8)
Feb 24 11:21:15 dc3 openais[1358]: [CLM  ]      r(0) ip(192.168.140.9)
Feb 24 11:21:15 dc3 openais[1358]: [CLM  ] Members Left:
Feb 24 11:21:15 dc3 openais[1358]: [CLM  ] Members Joined:
Feb 24 11:21:15 dc3 openais[1358]: [CLM  ]      r(0) ip(192.168.140.8)
Feb 24 11:21:15 dc3 openais[1358]: [SYNC ] This node is within the primary component and will provide service.
Feb 24 11:21:15 dc3 openais[1358]: [TOTEM] entering OPERATIONAL state.
Feb 24 11:21:15 dc3 openais[1358]: [CLM  ] got nodejoin message 192.168.140.7
Feb 24 11:21:15 dc3 openais[1358]: [CLM  ] got nodejoin message 192.168.140.8
Feb 24 11:21:15 dc3 openais[1358]: [CLM  ] got nodejoin message 192.168.140.9
Feb 24 11:21:15 dc3 openais[1358]: [CPG  ] got joinlist message from node 3
Feb 24 11:21:15 dc3 openais[1358]: [CPG  ] got joinlist message from node 1
dlm: connecting to 2
dlm: got connection from 2
Feb 24 11:21:20 dc3 kernel: dlm: connecting to 2
Feb 24 11:21:20 dc3 kernel: dlm: got connection from 2
このログから、cDomUsのメンバdc3がフェンス操作を(仮想マシンフェンスfVMによって)指示していることがわかる。
クラスタの状態を見てみる。
[root@dc2 ~]# cman_tool services
type             level name       id       state
fence            0     default    00010003 none
[1 2 3]
dlm              1     clvmd      00020003 none
[1 2 3]
dlm              1     rgmanager  00030003 none
[1 2 3]
[root@dc2 ~]#
この[state]の部分が[none]になっていることに注意。

コマンドラインからのフェンス
次に、コマンドラインからフェンスする。対象クラスタcDomUsのメンバ(この場合はdc2)から次のように実行する。引数のノード名は、FQDNで指定しなければならない。
[root@dc2 ~]# fence_node dc3.xencluster
[root@dc2 ~]#
フェンスされたノードは即時に停止され、再起動される。
＜＜略＞＞
Starting HAL daemon: [  OK  ]
Starting oddjobd: [  OK  ]
Starting Cluster Module - cluster monitor: Setting verbosity level to LogBasic
[  OK  ]
Starting Cluster Service Manager: [  OK  ]
dlm: lockspace 30003 from 2 type 1 not found
Starting ricci: [  OK  ]

CentOS release 5 (Final)
Kernel 2.6.18-53.1.13.el5xen on an i686

dc3.xencluster login:
他のノードのログは次の通り。
[root@dc2 ~]# tail -f /var/log/messages
Feb 24 15:39:47 dc2 fence_node[22750]: Fence of "dc3.xencluster" was successful
Feb 24 15:39:53 dc2 openais[1377]: [TOTEM] entering GATHER state from 12.
dlm: closing connection to node 1
Feb 24 15:39:58 dc2 openais[1377]: [TOTEM] entering GATHER state from 11.
Feb 24 15:39:58 dc2 openais[1377]: [TOTEM] Saving state aru 39 high seq received 39
Feb 24 15:39:58 dc2 openais[1377]: [TOTEM] Storing new sequence id for ring 40
Feb 24 15:39:58 dc2 openais[1377]: [TOTEM] entering COMMIT state.
Feb 24 15:39:58 dc2 openais[1377]: [TOTEM] entering RECOVERY state.
Feb 24 15:39:58 dc2 openais[1377]: [TOTEM] position [0] member 192.168.140.7:
Feb 24 15:39:58 dc2 openais[1377]: [TOTEM] previous ring seq 60 rep 192.168.140.7
Feb 24 15:39:58 dc2 openais[1377]: [TOTEM] aru 39 high delivered 39 received flag 1
Feb 24 15:39:58 dc2 openais[1377]: [TOTEM] position [1] member 192.168.140.8:
Feb 24 15:39:58 dc2 openais[1377]: [TOTEM] previous ring seq 60 rep 192.168.140.7
Feb 24 15:39:58 dc2 openais[1377]: [TOTEM] aru 39 high delivered 39 received flag 1
Feb 24 15:39:58 dc2 openais[1377]: [TOTEM] Did not need to originate any messages in recovery.
Feb 24 15:39:58 dc2 openais[1377]: [CLM  ] CLM CONFIGURATION CHANGE
Feb 24 15:39:58 dc2 openais[1377]: [CLM  ] New Configuration:
Feb 24 15:39:58 dc2 openais[1377]: [CLM  ]      r(0) ip(192.168.140.7)
Feb 24 15:39:58 dc2 openais[1377]: [CLM  ]      r(0) ip(192.168.140.8)
Feb 24 15:39:58 dc2 openais[1377]: [CLM  ] Members Left:
Feb 24 15:39:58 dc2 openais[1377]: [CLM  ]      r(0) ip(192.168.140.9)
Feb 24 15:39:58 dc2 openais[1377]: [CLM  ] Members Joined:
Feb 24 15:39:58 dc2 openais[1377]: [CLM  ] CLM CONFIGURATION CHANGE
Feb 24 15:39:58 dc2 openais[1377]: [CLM  ] New Configuration:
Feb 24 15:39:58 dc2 openais[1377]: [CLM  ]      r(0) ip(192.168.140.7)
Feb 24 15:39:58 dc2 openais[1377]: [CLM  ]      r(0) ip(192.168.140.8)
Feb 24 15:39:58 dc2 openais[1377]: [CLM  ] Members Left:
Feb 24 15:39:58 dc2 openais[1377]: [CLM  ] Members Joined:
Feb 24 15:39:58 dc2 openais[1377]: [SYNC ] This node is within the primary component and will provide service.
Feb 24 15:39:58 dc2 openais[1377]: [TOTEM] entering OPERATIONAL state.
Feb 24 15:39:58 dc2 openais[1377]: [CLM  ] got nodejoin message 192.168.140.7
Feb 24 15:39:58 dc2 openais[1377]: [CLM  ] got nodejoin message 192.168.140.8
Feb 24 15:39:58 dc2 kernel: dlm: closing connection to node 1
Feb 24 15:39:58 dc2 openais[1377]: [CPG  ] got joinlist message from node 3
Feb 24 15:39:59 dc2 openais[1377]: [CPG  ] got joinlist message from node 2
Feb 24 15:39:59 dc2 fenced[1393]: dc3.xencluster not a cluster member after 0 sec post_fail_delay
Feb 24 15:40:17 dc2 openais[1377]: [TOTEM] entering GATHER state from 11.
Feb 24 15:40:17 dc2 openais[1377]: [TOTEM] Saving state aru 1f high seq received 1f
Feb 24 15:40:17 dc2 openais[1377]: [TOTEM] Storing new sequence id for ring 44
Feb 24 15:40:17 dc2 openais[1377]: [TOTEM] entering COMMIT state.
Feb 24 15:40:17 dc2 openais[1377]: [TOTEM] entering RECOVERY state.
Feb 24 15:40:17 dc2 openais[1377]: [TOTEM] position [0] member 192.168.140.7:
Feb 24 15:40:17 dc2 openais[1377]: [TOTEM] previous ring seq 64 rep 192.168.140.7
Feb 24 15:40:17 dc2 openais[1377]: [TOTEM] aru 1f high delivered 1f received flag 1
Feb 24 15:40:17 dc2 openais[1377]: [TOTEM] position [1] member 192.168.140.8:
Feb 24 15:40:17 dc2 openais[1377]: [TOTEM] previous ring seq 64 rep 192.168.140.7
Feb 24 15:40:17 dc2 openais[1377]: [TOTEM] aru 1f high delivered 1f received flag 1
Feb 24 15:40:17 dc2 openais[1377]: [TOTEM] position [2] member 192.168.140.9:
Feb 24 15:40:17 dc2 openais[1377]: [TOTEM] previous ring seq 64 rep 192.168.140.9
Feb 24 15:40:17 dc2 openais[1377]: [TOTEM] aru 9 high delivered 9 received flag 1
Feb 24 15:40:17 dc2 openais[1377]: [TOTEM] Did not need to originate any messages in recovery.
Feb 24 15:40:17 dc2 openais[1377]: [CLM  ] CLM CONFIGURATION CHANGE
Feb 24 15:40:17 dc2 openais[1377]: [CLM  ] New Configuration:
Feb 24 15:40:17 dc2 openais[1377]: [CLM  ]      r(0) ip(192.168.140.7)
Feb 24 15:40:17 dc2 openais[1377]: [CLM  ]      r(0) ip(192.168.140.8)
Feb 24 15:40:17 dc2 openais[1377]: [CLM  ] Members Left:
Feb 24 15:40:17 dc2 openais[1377]: [CLM  ] Members Joined:
Feb 24 15:40:17 dc2 openais[1377]: [CLM  ] CLM CONFIGURATION CHANGE
Feb 24 15:40:17 dc2 openais[1377]: [CLM  ] New Configuration:
Feb 24 15:40:17 dc2 openais[1377]: [CLM  ]      r(0) ip(192.168.140.7)
Feb 24 15:40:17 dc2 openais[1377]: [CLM  ]      r(0) ip(192.168.140.8)
Feb 24 15:40:17 dc2 openais[1377]: [CLM  ]      r(0) ip(192.168.140.9)
Feb 24 15:40:17 dc2 openais[1377]: [CLM  ] Members Left:
Feb 24 15:40:17 dc2 openais[1377]: [CLM  ] Members Joined:
Feb 24 15:40:17 dc2 openais[1377]: [CLM  ]      r(0) ip(192.168.140.9)
Feb 24 15:40:17 dc2 openais[1377]: [SYNC ] This node is within the primary component and will provide service.
Feb 24 15:40:17 dc2 openais[1377]: [TOTEM] entering OPERATIONAL state.
Feb 24 15:40:17 dc2 openais[1377]: [CLM  ] got nodejoin message 192.168.140.7
Feb 24 15:40:17 dc2 openais[1377]: [CLM  ] got nodejoin message 192.168.140.8
Feb 24 15:40:17 dc2 openais[1377]: [CLM  ] got nodejoin message 192.168.140.9
Feb 24 15:40:17 dc2 openais[1377]: [CPG  ] got joinlist message from node 2
Feb 24 15:40:17 dc2 openais[1377]: [CPG  ] got joinlist message from node 3
dlm: connecting to 1
Feb 24 15:40:21 dc2 kernel: dlm: connecting to 1
クラスタの状態を確認する。
[root@dc2 ~]# cman_tool services
type             level name       id       state
fence            0     default    00010003 none
[1 2 3]
dlm              1     clvmd      00020003 none
[1 2 3]
dlm              1     rgmanager  00030003 none
[1 2 3]
[root@dc2 ~]#


その1・概要
その2・Dom0 fsXのインストール
その3・DomU dcXのインストール
その4・DomU bs1のインストール
その5・Luciの準備
その6・クラスタcDom0sの生成
その7・クラスタcDomUsの生成
その8・クラスタ動作の試験
ラベル: , , , , , , , , , , ,

0 件のコメント:

コメントを投稿

登録: コメントの投稿 (Atom)