[SOLVED][arch newbie] Installing corosync/openais + ocfs2

CarmineM · 2011-06-21 15:25:29

Hi to everyone,

After having followed all the howtos', guides and hints over the respective lists, i'm resorting to
bothering you all ( ) with this very annoying problem i'm facing.

First and foremost i'm an arch newbie and just installed two VMs to test a cluster infrastructure
i'm about to deploy.

The objective is to run a two node Active/Active cluster with corosync/openais + DRBD + OCFS2.

Following the guidelines on clusterlabs.org, after a little of struggling, i've managed to compile and
run a very basic cluster.

Problem arose once time arrived to install ocfs2.

I have downloaded source tarballs from oracle (tried with both version 1.4.4 and 1.6.4).
I managed to get everything compiled, then configured the cluster and got completely stuck
with following errors:

Running "ocfs2_controld.pcmk -D" produces:

ocfs2_controld[5295]: 2011/06/21_17:15:51 info: init_ais_connection: Creating connection to our AIS plugin
ocfs2_controld[5295]: 2011/06/21_17:15:51 info: init_ais_connection: AIS connection established
ocfs2_controld[5295]: 2011/06/21_17:15:51 info: get_ais_nodeid: Server details: id=2254612672 uname=archnode1 cname=pcmk
ocfs2_controld[5295]: 2011/06/21_17:15:51 info: crm_new_peer: Node archnode1 now has id: 2254612672
ocfs2_controld[5295]: 2011/06/21_17:15:51 info: crm_new_peer: Node 2254612672 is now known as archnode1
1308669351 setup_stack@168: Cluster connection established.  Local node id: -2040354624
1308669351 setup_stack@172: Added Pacemaker as client 1 with fd 5
1308669351 setup_ckpt@609: Initializing CKPT service (try 1)
1308669351 setup_ckpt@615: Connected to CKPT service with handle 0x327b23c600000000
1308669351 call_ckpt_open@160: Opening checkpoint "ocfs2:controld:8662a8c0" (try 1)
1308669351 call_ckpt_open@170: Opened checkpoint "ocfs2:controld:8662a8c0" with handle 0x6633487300000000
1308669351 call_section_write@340: Writing to section "daemon_max_protocol" on checkpoint "ocfs2:controld:8662a8c0" (try 1)
1308669351 call_section_create@292: Creating section "daemon_max_protocol" on checkpoint "ocfs2:controld:8662a8c0" (try 1)
1308669351 call_section_create@300: Created section "daemon_max_protocol" on checkpoint "ocfs2:controld:8662a8c0"
1308669351 call_section_write@340: Writing to section "ocfs2_max_protocol" on checkpoint "ocfs2:controld:8662a8c0" (try 1)
1308669351 call_section_create@292: Creating section "ocfs2_max_protocol" on checkpoint "ocfs2:controld:8662a8c0" (try 1)
1308669351 call_section_create@300: Created section "ocfs2_max_protocol" on checkpoint "ocfs2:controld:8662a8c0"
1308669351 start_join@588: Starting join for group "ocfs2:controld"
1308669351 start_join@592: cpg_join succeeded
1308669351 loop@975: setup done
ocfs2_controld[5295]: 2011/06/21_17:15:51 info: ais_dispatch: Membership 188: quorum still lost
ocfs2_controld[5295]: 2011/06/21_17:15:51 ERROR: crm_int_helper: Conversion of 2254612672 was clipped: 2147483647
ocfs2_controld[5295]: 2011/06/21_17:15:51 info: crm_get_peer: Node archnode1 now has id: 2147483647
ocfs2_controld[5295]: 2011/06/21_17:15:51 info: crm_update_peer: Node archnode1: id=2147483647 state=member (new) addr=r(0) ip(192.168.98.134)  (new) votes=1 (new) born=0 seen=188 proc=00000000000000000000000000111312 (new)
1308669351 confchg_cb@495: confchg called
1308669351 daemon_change@398: ocfs2_controld (group "ocfs2:controld") confchg: members 1, left 0, joined 1
1308669351 cpg_joined@909: CPG is live, we are the first daemon
1308669351 call_ckpt_open@160: Opening checkpoint "ocfs2:controld" (try 1)
1308669351 call_ckpt_open@170: Opened checkpoint "ocfs2:controld" with handle 0x2ae8944a00000001
1308669351 call_section_write@340: Writing to section "daemon_protocol" on checkpoint "ocfs2:controld" (try 1)
1308669351 call_section_create@292: Creating section "daemon_protocol" on checkpoint "ocfs2:controld" (try 1)
1308669351 call_section_create@300: Created section "daemon_protocol" on checkpoint "ocfs2:controld"
1308669351 call_section_write@340: Writing to section "ocfs2_protocol" on checkpoint "ocfs2:controld" (try 1)
1308669351 call_section_create@292: Creating section "ocfs2_protocol" on checkpoint "ocfs2:controld" (try 1)
1308669351 call_section_create@300: Created section "ocfs2_protocol" on checkpoint "ocfs2:controld"
1308669351 cpg_joined@923: Daemon protocol is 1.0
1308669351 cpg_joined@925: fs protocol is 1.0
1308669351 cpg_joined@927: Connecting to dlm_controld
1308669351 setup_dlmcontrol@335: Unable to connect to dlm_controld: Connection refused
1308669351 start_leave@613: leaving group "ocfs2:controld"
1308669351 start_leave@626: cpg_leave succeeded
1308669351 exit_cpg@760: closing cpg connection
1308669351 call_ckpt_close@240: Closing checkpoint "ocfs2:controld:8662a8c0" (try 1)
1308669351 call_ckpt_close@246: Closed checkpoint "ocfs2:controld:8662a8c0"
1308669351 exit_ckpt@643: Disconnecting from CKPT service (try 1)
1308669351 exit_ckpt@647: Disconnected from CKPT service
1308669351 exit_stack@144: closing pacemaker connection
ocfs2_controld[5295]: 2011/06/21_17:15:51 notice: terminate_ais_connection: Disconnected from AIS

I've checked that UDEV made the required device available:

[root@archnode1 ~]# ls -la /dev/misc
total 0
drwxr-xr-x  2 root root    120 Jun 21 16:58 .
drwxr-xr-x 16 root root   5320 Jun 21 16:58 ..
lrwxrwxrwx  1 root root     14 Jun 21 16:47 dlm-control -> ../dlm-control
lrwxrwxrwx  1 root root     14 Jun 21 16:47 dlm-monitor -> ../dlm-monitor
lrwxrwxrwx  1 root root     12 Jun 21 16:47 dlm_plock -> ../dlm_plock
crw-rw----  1 root root 10, 56 Jun 21 16:58 ocfs2_control

I got puzzled by the fact that ocfs2_controld complains about not being able to connect to
dlm_controld, so i checked with "ps" whether there was dlm_controld running or not.
First time no dlm_controld process was running.

I tried manually launching dlm_controld and soon after "ocfs2_controld.pcmk -D" which
show a different output:

ocfs2_controld[5706]: 2011/06/21_17:18:45 info: init_ais_connection: Creating connection to our AIS plugin
ocfs2_controld[5706]: 2011/06/21_17:18:45 info: init_ais_connection: AIS connection established
ocfs2_controld[5706]: 2011/06/21_17:18:45 info: get_ais_nodeid: Server details: id=2254612672 uname=archnode1 cname=pcmk
ocfs2_controld[5706]: 2011/06/21_17:18:45 info: crm_new_peer: Node archnode1 now has id: 2254612672
ocfs2_controld[5706]: 2011/06/21_17:18:45 info: crm_new_peer: Node 2254612672 is now known as archnode1
1308669525 setup_stack@168: Cluster connection established.  Local node id: -2040354624
1308669525 setup_stack@172: Added Pacemaker as client 1 with fd 5
1308669525 setup_ckpt@609: Initializing CKPT service (try 1)
1308669525 setup_ckpt@615: Connected to CKPT service with handle 0x327b23c600000000
1308669525 call_ckpt_open@160: Opening checkpoint "ocfs2:controld:8662a8c0" (try 1)
1308669525 call_ckpt_open@170: Opened checkpoint "ocfs2:controld:8662a8c0" with handle 0x6633487300000000
1308669525 call_section_write@340: Writing to section "daemon_max_protocol" on checkpoint "ocfs2:controld:8662a8c0" (try 1)
1308669525 call_section_create@292: Creating section "daemon_max_protocol" on checkpoint "ocfs2:controld:8662a8c0" (try 1)
1308669525 call_section_create@300: Created section "daemon_max_protocol" on checkpoint "ocfs2:controld:8662a8c0"
1308669525 call_section_write@340: Writing to section "ocfs2_max_protocol" on checkpoint "ocfs2:controld:8662a8c0" (try 1)
1308669525 call_section_create@292: Creating section "ocfs2_max_protocol" on checkpoint "ocfs2:controld:8662a8c0" (try 1)
1308669525 call_section_create@300: Created section "ocfs2_max_protocol" on checkpoint "ocfs2:controld:8662a8c0"
1308669525 start_join@588: Starting join for group "ocfs2:controld"
1308669525 start_join@592: cpg_join succeeded
1308669525 loop@975: setup done
ocfs2_controld[5706]: 2011/06/21_17:18:45 info: ais_dispatch: Membership 188: quorum still lost
ocfs2_controld[5706]: 2011/06/21_17:18:45 ERROR: crm_int_helper: Conversion of 2254612672 was clipped: 2147483647
ocfs2_controld[5706]: 2011/06/21_17:18:45 info: crm_get_peer: Node archnode1 now has id: 2147483647
ocfs2_controld[5706]: 2011/06/21_17:18:45 info: crm_update_peer: Node archnode1: id=2147483647 state=member (new) addr=r(0) ip(192.168.98.134)  (new) votes=1 (new) born=0 seen=188 proc=00000000000000000000000000111312 (new)
1308669525 confchg_cb@495: confchg called
1308669525 daemon_change@398: ocfs2_controld (group "ocfs2:controld") confchg: members 1, left 0, joined 1
1308669525 cpg_joined@909: CPG is live, we are the first daemon
1308669525 call_ckpt_open@160: Opening checkpoint "ocfs2:controld" (try 1)
1308669525 call_ckpt_open@170: Opened checkpoint "ocfs2:controld" with handle 0x2ae8944a00000001
1308669525 call_section_write@340: Writing to section "daemon_protocol" on checkpoint "ocfs2:controld" (try 1)
1308669525 call_section_create@292: Creating section "daemon_protocol" on checkpoint "ocfs2:controld" (try 1)
1308669525 call_section_create@300: Created section "daemon_protocol" on checkpoint "ocfs2:controld"
1308669525 call_section_write@340: Writing to section "ocfs2_protocol" on checkpoint "ocfs2:controld" (try 1)
1308669525 call_section_create@292: Creating section "ocfs2_protocol" on checkpoint "ocfs2:controld" (try 1)
1308669525 call_section_create@300: Created section "ocfs2_protocol" on checkpoint "ocfs2:controld"
1308669525 cpg_joined@923: Daemon protocol is 1.0
1308669525 cpg_joined@925: fs protocol is 1.0
1308669525 cpg_joined@927: Connecting to dlm_controld
1308669525 cpg_joined@934: Opening control device
1308669525 cpg_joined@938: Error opening control device: I/O error on channel
1308669525 exit_dlmcontrol@363: Closing dlm_controld connection
1308669525 start_leave@613: leaving group "ocfs2:controld"
1308669525 start_leave@626: cpg_leave succeeded
1308669525 exit_cpg@760: closing cpg connection
1308669525 call_ckpt_close@240: Closing checkpoint "ocfs2:controld:8662a8c0" (try 1)
1308669525 call_ckpt_close@246: Closed checkpoint "ocfs2:controld:8662a8c0"
1308669525 exit_ckpt@643: Disconnecting from CKPT service (try 1)
1308669525 exit_ckpt@647: Disconnected from CKPT service
1308669525 exit_stack@144: closing pacemaker connection
ocfs2_controld[5706]: 2011/06/21_17:18:45 notice: terminate_ais_connection: Disconnected from AIS

This time the complaint was about "Error opening control device: I/O error on channel".

crm_mon shows:

============
Last updated: Tue Jun 21 17:23:16 2011
Stack: openais
Current DC: archnode1 - partition WITHOUT quorum
Version: 1.1.1-cc0e4d295e298510dad994ad5f9f6a8ffbf9f9ff
2 Nodes configured, 2 expected votes
6 Resources configured.
============

Online: [ archnode1 ]
OFFLINE: [ archnode2 ]

 ClusterIP    (ocf::heartbeat:IPaddr2):    Started archnode1
 Master/Slave Set: TestDataClone
     Masters: [ archnode1 ]
     Stopped: [ TestData:1 ]
 TestFS    (ocf::heartbeat:Filesystem):    Started archnode1
 vsftpd    (ocf::frigocaserta:vsftpd):    Started archnode1
 Clone Set: dlm-clone
     Started: [ archnode1 ]
     Stopped: [ dlm:1 ]

Judging by the previous output, i tend to assume that dlm is working (then why
there isn't any dlm_controld process running?).

This is my actual configuration:

node archnode1 \
    attributes standby="off"
node archnode2 \
    attributes standby="off"
primitive ClusterIP ocf:heartbeat:IPaddr2 \
    params ip="192.168.98.136" cidr_netmask="32" \
    op monitor interval="30s"
primitive TestData ocf:linbit:drbd \
    params drbd_resource="r0" \
    op monitor interval="60s"
primitive TestFS ocf:heartbeat:Filesystem \
    params device="/dev/drbd1" directory="/mnt/r0" fstype="ext3"
primitive dlm ocf:pacemaker:controld \
    params daemon="dlm_controld" \
    op monitor interval="120s"
primitive o2cb ocf:pacemaker:o2cb \
    op monitor interval="120s" \
    meta is-managed="true" target-role="Started"
primitive vsftpd ocf:frigocaserta:vsftpd \
    params conffile="/etc/vsftpd.conf" \
    op monitor interval="10s" \
    meta target-role="Started"
ms TestDataClone TestData \
    meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true" target-role="Started"
clone dlm-clone dlm \
    meta interleave="true"
clone o2cb-clone o2cb \
    meta interleave="true" is-managed="true" target-role="Started"
location cli-prefer-vsftpd vsftpd \
    rule $id="cli-prefer-rule-vsftpd" inf: #uname eq archnode1
colocation fs_on_drbd inf: TestFS TestDataClone:Master
colocation ftp_with_clusterip inf: vsftpd ClusterIP
colocation o2cb-with-dlm inf: o2cb-clone dlm-clone
order fs_after_testdata inf: TestDataClone:promote TestFS:start
order ftp_after_clusterip inf: ClusterIP vsftpd
order o2cb-after-dlm inf: dlm-clone:promote o2cb-clone:start
property $id="cib-bootstrap-options" \
    dc-version="1.1.1-cc0e4d295e298510dad994ad5f9f6a8ffbf9f9ff" \
    cluster-infrastructure="openais" \
    expected-quorum-votes="2" \
    stonith-enabled="false" \
    no-quorum-policy="ignore" \
    last-lrm-refresh="1308576570"
rsc_defaults $id="rsc-options" \
    resource-stickiness="100"

What's that I am doing wrong?

Thanks in advance for your help.
Regards,
Carmine

Last edited by CarmineM (2011-06-24 08:57:21)

CarmineM · 2011-06-24 08:58:26

Should anyone be interested into how i got ridden of the "Error opening control device: I/O error on channel", please
take a look at the post i made over the linux-ha mailing list.

http://lists.linux-ha.org/pipermail/lin … 43328.html

Cheers,
Carmine

Arch Linux

#1 2011-06-21 15:25:29

[SOLVED][arch newbie] Installing corosync/openais + ocfs2

#2 2011-06-24 08:58:26

Re: [SOLVED][arch newbie] Installing corosync/openais + ocfs2

Board footer