Hi All,

Multipathd is not working.

getting below error
-----------------------
[root@node1 ~]# /etc/init.d/multipathd restart
ux_socket_connect: Connection refused
Stopping multipathd daemon: [FAILED]
Starting multipathd daemon: [ OK ]
[root@node1 ~]# /etc/init.d/multipathd start
Starting multipathd daemon: [ OK ]
[root@node1 ~]# /etc/init.d/multipathd status
multipathd dead but pid file exists
-----------------------

[root@node1 ~]# rpm -qa|grep multipath
device-mapper-multipath-0.4.9-87.el6.x86_64
device-mapper-multipath-libs-0.4.9-87.el6.x86_64

[root@node1 log]# lsb_release -a
LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarchrinting-4.0-amd64rinting-4.0-noarch
Distributor ID: CentOS
Description: CentOS release 6.5 (Final)
Release: 6.5
Codename: Final

[root@node1 log]# uname -r
2.6.32-431.el6.x86_64

In Centos, you should use service multipathd start.
Is there anything in /var/log/messages (or simply dmesg)?
There are a few web pages that match your error, did you check them?

By the way, while it's probably unrelated to your error, Centos 6.5 is outdated. Upgrade to 6.7 or 7.

Will try that and let you know

Try to run multipathd manually and attach the output here:

Hi Robert,

below is output for /sbin/multipathd -d -v4

[root@localhost ~]# /sbin/multipathd -d -v4
Oct 28 09:36:14 | --------start up--------
Oct 28 09:36:14 | read /etc/multipath.conf
Oct 28 09:36:14 | activate request-based multipathing mode (driver >= v1.1.0)
Oct 28 09:36:14 | No configuration dir '/etc/multipath/conf.d'
Oct 28 09:36:14 | loading /lib64/multipath/libcheckdirectio.so checker
Oct 28 09:36:14 | loading /lib64/multipath/libprioconst.so prioritizer
Oct 28 09:36:14 | Discover device /sys/block/ram0
Oct 28 09:36:14 | ram0: device node name blacklisted
Oct 28 09:36:14 | Discover device /sys/block/ram1
Oct 28 09:36:14 | ram1: device node name blacklisted
Oct 28 09:36:14 | Discover device /sys/block/ram2
Oct 28 09:36:14 | ram2: device node name blacklisted
Oct 28 09:36:14 | Discover device /sys/block/ram3
Oct 28 09:36:14 | ram3: device node name blacklisted
Oct 28 09:36:14 | Discover device /sys/block/ram4
Oct 28 09:36:14 | ram4: device node name blacklisted
Oct 28 09:36:14 | Discover device /sys/block/ram5
Oct 28 09:36:14 | ram5: device node name blacklisted
Oct 28 09:36:14 | Discover device /sys/block/ram6
Oct 28 09:36:14 | ram6: device node name blacklisted
Oct 28 09:36:14 | Discover device /sys/block/ram7
Oct 28 09:36:14 | ram7: device node name blacklisted
Oct 28 09:36:14 | Discover device /sys/block/ram8
Oct 28 09:36:14 | ram8: device node name blacklisted
Oct 28 09:36:14 | Discover device /sys/block/ram9
Oct 28 09:36:14 | ram9: device node name blacklisted
Oct 28 09:36:14 | Discover device /sys/block/ram10
Oct 28 09:36:14 | ram10: device node name blacklisted
Oct 28 09:36:14 | Discover device /sys/block/ram11
Oct 28 09:36:14 | ram11: device node name blacklisted
Oct 28 09:36:14 | Discover device /sys/block/ram12
Oct 28 09:36:14 | ram12: device node name blacklisted
Oct 28 09:36:14 | Discover device /sys/block/ram13
Oct 28 09:36:14 | ram13: device node name blacklisted
Oct 28 09:36:14 | Discover device /sys/block/ram14
Oct 28 09:36:14 | ram14: device node name blacklisted
Oct 28 09:36:14 | Discover device /sys/block/ram15
Oct 28 09:36:14 | ram15: device node name blacklisted
Oct 28 09:36:14 | Discover device /sys/block/loop0
Oct 28 09:36:14 | loop0: device node name blacklisted
Oct 28 09:36:14 | Discover device /sys/block/loop1
Oct 28 09:36:14 | loop1: device node name blacklisted
Oct 28 09:36:14 | Discover device /sys/block/loop2
Oct 28 09:36:14 | loop2: device node name blacklisted
Oct 28 09:36:14 | Discover device /sys/block/loop3
Oct 28 09:36:14 | loop3: device node name blacklisted
Oct 28 09:36:14 | Discover device /sys/block/loop4
Oct 28 09:36:14 | loop4: device node name blacklisted
Oct 28 09:36:14 | Discover device /sys/block/loop5
Oct 28 09:36:14 | loop5: device node name blacklisted
Oct 28 09:36:14 | Discover device /sys/block/loop6
Oct 28 09:36:14 | loop6: device node name blacklisted
Oct 28 09:36:14 | Discover device /sys/block/loop7
Oct 28 09:36:14 | loop7: device node name blacklisted
Oct 28 09:36:14 | Discover device /sys/block/sr0
Oct 28 09:36:14 | sr0: device node name blacklisted
Oct 28 09:36:14 | Discover device /sys/block/sda
Oct 28 09:36:14 | sda: not found in pathvec
Oct 28 09:36:14 | sda: mask = 0x3f
Oct 28 09:36:14 | sda: dev_t = 8:0
Oct 28 09:36:14 | sda: size = 41943040
Oct 28 09:36:14 | sda: subsystem = scsi
Oct 28 09:36:14 | sda: vendor = VMware,
Oct 28 09:36:14 | sda: product = VMware Virtual S
Oct 28 09:36:14 | sda: rev = 1.0
Oct 28 09:36:14 | sda: h:b:t:l = 2:0:0:0
Oct 28 09:36:14 | sda: serial =
Oct 28 09:36:14 | sda: get_state
Oct 28 09:36:14 | loading /lib64/multipath/libcheckreadsector0.so checker
Oct 28 09:36:14 | sda: path checker = readsector0 (config file default)
Oct 28 09:36:14 | sda: checker timeout = 180000 ms (sysfs setting)
Oct 28 09:36:14 | sda: state = running
Oct 28 09:36:14 | sda: state = 3
Oct 28 09:36:14 | sda: getuid = /lib/udev/scsi_id --replace-whitespace --whiteli sted --device=/dev/n (config file default)
Oct 28 09:36:14 | reading events from udev socket.
Oct 28 09:36:14 | /lib/udev/scsi_id exitted with 1
Oct 28 09:36:14 | error calling out /lib/udev/scsi_id --replace-whitespace --whi telisted --device=/dev/sda
Oct 28 09:36:14 | sda: state = running
Oct 28 09:36:14 | /lib/udev/scsi_id exitted with 1
Oct 28 09:36:14 | error calling out /lib/udev/scsi_id --replace-whitespace --whi telisted --device=/dev/sda
Oct 28 09:36:14 | sda: detect_prio = 1 (config file default)
Oct 28 09:36:14 | sda: prio = const (config file default)
Oct 28 09:36:14 | sda: const prio = 1
Oct 28 09:36:14 | Discover device /sys/block/dm-0
Oct 28 09:36:14 | dm-0: device node name blacklisted
Oct 28 09:36:14 | Discover device /sys/block/dm-1
Oct 28 09:36:14 | dm-1: device node name blacklisted
Oct 28 09:36:14 | Discover device /sys/block/dm-2
Oct 28 09:36:14 | dm-2: device node name blacklisted
Oct 28 09:36:14 | Discover device /sys/block/dm-3
Oct 28 09:36:14 | dm-3: device node name blacklisted
Oct 28 09:36:14 | Discover device /sys/block/dm-4
Oct 28 09:36:14 | dm-4: device node name blacklisted
Oct 28 09:36:14 | Discover device /sys/block/dm-5
Oct 28 09:36:14 | dm-5: device node name blacklisted
Oct 28 09:36:14 | Discover device /sys/block/dm-6
Oct 28 09:36:14 | dm-6: device node name blacklisted
Oct 28 09:36:14 | Discover device /sys/block/sdb
Oct 28 09:36:14 | sdb: not found in pathvec
Oct 28 09:36:14 | sdb: mask = 0x3f
Oct 28 09:36:14 | sdb: dev_t = 8:16
Oct 28 09:36:14 | sdb: size = 1024000
Oct 28 09:36:14 | sdb: subsystem = scsi
Oct 28 09:36:14 | sdb: vendor = IET
Oct 28 09:36:14 | sdb: product = VIRTUAL-DISK
Oct 28 09:36:14 | sdb: rev = 0001
Oct 28 09:36:14 | sdb: h:b:t:l = 3:0:0:1
Oct 28 09:36:14 | sdb: tgt_node_name = CentOS:target1
Oct 28 09:36:14 | sdb: serial = beaf11
Oct 28 09:36:14 | sdb: get_state
Oct 28 09:36:14 | sdb: path checker = readsector0 (config file default)
Oct 28 09:36:14 | sdb: checker timeout = 30000 ms (sysfs setting)
Oct 28 09:36:14 | sdb: state = running
Oct 28 09:36:14 | sdb: state = 3
Oct 28 09:36:14 | sdb: getuid = /lib/udev/scsi_id --replace-whitespace --whiteli sted --device=/dev/%n (config file default)
Oct 28 09:36:14 | sdb: uid = 1IET_00010001 (callout)
Oct 28 09:36:14 | sdb: state = running
Oct 28 09:36:14 | sdb: detect_prio = 1 (config file default)
Oct 28 09:36:14 | sdb: prio = const (config file default)
Oct 28 09:36:14 | sdb: const prio = 1
Oct 28 09:36:14 | Discover device /sys/block/sdc
Oct 28 09:36:14 | sdc: not found in pathvec
Oct 28 09:36:14 | sdc: mask = 0x3f
Oct 28 09:36:14 | sdc: dev_t = 8:32
Oct 28 09:36:14 | sdc: size = 1024000
Oct 28 09:36:14 | sdc: subsystem = scsi
Oct 28 09:36:14 | sdc: vendor = IET
Oct 28 09:36:14 | sdc: product = VIRTUAL-DISK
Oct 28 09:36:14 | sdc: rev = 0001
Oct 28 09:36:14 | sdc: h:b:t:l = 4:0:0:1
Oct 28 09:36:14 | sdc: tgt_node_name = CentOS:target1
Oct 28 09:36:14 | sdc: serial = beaf11
Oct 28 09:36:14 | sdc: get_state
Oct 28 09:36:14 | sdc: path checker = readsector0 (config file default)
Oct 28 09:36:14 | sdc: checker timeout = 30000 ms (sysfs setting)
Oct 28 09:36:14 | sdc: state = running
Oct 28 09:36:14 | sdc: state = 3
Oct 28 09:36:14 | sdc: getuid = /lib/udev/scsi_id --replace-whitespace --whiteli sted --device=/dev/%n (config file default)
Oct 28 09:36:14 | sdc: uid = 1IET_00010001 (callout)
Oct 28 09:36:14 | sdc: state = running
Oct 28 09:36:14 | sdc: detect_prio = 1 (config file default)
Oct 28 09:36:14 | sdc: prio = const (config file default)
Oct 28 09:36:14 | sdc: const prio = 1
Oct 28 09:36:14 | Discovery status 0
Oct 28 09:36:14 | sdb: ownership set to 1IET_00010001
Oct 28 09:36:14 | sdb: not found in pathvec
Oct 28 09:36:14 | sdb: mask = 0xc
Oct 28 09:36:14 | sdb: get_state
Oct 28 09:36:14 | sdb: state = running
Oct 28 09:36:14 | sdb: state = 3
Oct 28 09:36:14 | sdb: state = running
Oct 28 09:36:14 | sdb: const prio = 1
Oct 28 09:36:14 | sdc: ownership set to 1IET_00010001
Oct 28 09:36:14 | sdc: not found in pathvec
Oct 28 09:36:14 | sdc: mask = 0xc
Oct 28 09:36:14 | sdc: get_state
Oct 28 09:36:14 | sdc: state = running
Oct 28 09:36:14 | sdc: state = 3
Oct 28 09:36:14 | sdc: state = running
Oct 28 09:36:14 | sdc: const prio = 1
Oct 28 09:36:14 | 1IET_00010001: verified path sdb dev_t 8:16
Oct 28 09:36:14 | 1IET_00010001: verified path sdc dev_t 8:32
Oct 28 09:36:14 | 1IET_00010001: pgfailback = -2 (config file default)
Oct 28 09:36:14 | 1IET_00010001: pgpolicy = multibus (config file default)
Oct 28 09:36:14 | 1IET_00010001: selector = round-robin 0 (internal default)
Oct 28 09:36:14 | 1IET_00010001: features = 0 (internal default)
Oct 28 09:36:14 | 1IET_00010001: hwhandler = 0 (internal default)
Oct 28 09:36:14 | 1IET_00010001: rr_weight = 2 (config file default)
Oct 28 09:36:14 | 1IET_00010001: minio = 1 rq (config file default)
Oct 28 09:36:14 | 1IET_00010001: no_path_retry = -1 (config file default)
Oct 28 09:36:14 | pg_timeout = NONE (internal default)
Oct 28 09:36:14 | 1IET_00010001: retain_attached_hw_handler = 1 (config file def ault)
Oct 28 09:36:14 | 1IET_00010001: reload_readwrite = 1 (config file default)
Oct 28 09:36:14 | delay_watch_checks = DISABLED (internal default)
Oct 28 09:36:14 | delay_wait_checks = DISABLED (internal default)
Oct 28 09:36:14 | 1IET_00010001: set ACT_CREATE (map does not exist)
Segmentation fault (core dumped)

This should be the reason why your daemon doesn't run.
Either a bug (not so likely) or some version mismatch with a library or so.

Thanks robertjinx, berndbausch for valuable comments and solutions.

multipathd is now working properly.

Refer below logs:-

/etc/init.d/multipathd restart
ok
Stopping multipathd daemon: [ OK ]
Starting multipathd daemon: [ OK ]

/etc/init.d/multipathd status
multipathd (pid 16559) is running...

multipath -ll
1IET_00010001 dm-7 IET,VIRTUAL-DISK
size=500M features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 3:0:0:1 sdb 8:16 active ready running
`- 4:0:0:1 sdc 8:32 active ready running

I checked /var/log/messages and found below log, this log was not there earlier.
=======================
kernel: multipathd[15518]: segfault at 0 ip (null) sp 00007fffb5f52018 error 14 in multipathd[400000+10000]
=======================

so checked google and found the solution

root cause is "The server has latest package of device-mapper-multipath, but a very old version of device-mapper and glibc"

Solution:-
yum update device-mapper glibc -y
service multipathd restart
multipath -ll

Regards,
Gaurav Bhatkar

Glad it works now. Good learning for me, too.

Cool, good for you.