[dm-devel] rdac priority checker changing priorities

Discussion:

Lucas Brasilino

2009-04-29 22:34:54 UTC

Hi

I don't know if I'm misundertanding something. I've got an DS4700 and I'm
switching from RDAC[1] to multipath, since it's natively supported in
the distribution I use
here (SLES 10 SP2).

Since RDAC[1] works perfect, I'm trying to use 'rdac' priority in multipath.

My /etc/multiconf.conf is quite tiny, since I'm building it step-by-step :-) :

blacklist {
devnode "^sda[0-9]*"
}

defaults {
user_friendly_names yes
prio rdac
path_checker tur
}

multipaths {
multipath {
wwid 3600a0b8000327b900000107549f85224
alias mpath0
}
}

I think that using 'prio rdac' makes multipath to use 'mpath_prio_rdac' tool.

# multipath -v2 -ll
mpath0 (3600a0b8000327b900000107549f85224) dm-0 IBM,1814 FAStT
[size=140G][features=1 queue_if_no_path][hwhandler=1 rdac]
\_ round-robin 0 [prio=6][active]
\_ 9:0:0:0 sdb 8:16 [active][ready]
\_ round-robin 0 [prio=1][enabled]
\_ 10:0:0:0 sdc 8:32 [active][ghost]

So the first path has priority 6, as I can confirm:

# mpath_prio_rdac /dev/sdb
6
# mpath_prio_rdac /dev/sdc
1

After the first path (prio=6) failure I get:

# multipath -v2 -ll
sdb: rdac prio: inquiry command indicates error
mpath0 (3600a0b8000327b900000107549f85224) dm-0 IBM,1814 FAStT
[size=140G][features=1 queue_if_no_path][hwhandler=1 rdac]
\_ round-robin 0 [prio=0][enabled]
\_ 9:0:0:0 sdb 8:16 [failed][faulty]
\_ round-robin 0 [prio=1][enabled]
\_ 10:0:0:0 sdc 8:32 [active][ghost]

Ok.. working great, activating the second path. But after the faulty
path is restored:

# multipath -v2 -ll
mpath0 (3600a0b8000327b900000107549f85224) dm-0 IBM,1814 FAStT
[size=140G][features=1 queue_if_no_path][hwhandler=1 rdac]
\_ round-robin 0 [prio=2][enabled]
\_ 9:0:0:0 sdb 8:16 [active][ghost]
\_ round-robin 0 [prio=5][active]
\_ 10:0:0:0 sdc 8:32 [active][ready]

Second path is now priority!!! And of course does not fails back! By
the way, my LUN is configured in
DS4700 in sort a way that the first path *is* the path to preferred controller.

I think path priorities should not change. If so first path goes back
to 'active' status.
Am I misunderstanding something ? Or messing things up?

By the way, here comes the default 'multipath.conf':

#defaults {
# udev_dir /dev
# polling_interval 10
# selector "round-robin 0"
# path_grouping_policy multibus
# getuid_callout "/lib/udev/scsi_id -g -u -s /block/%n"
# prio const
# path_checker directio
# rr_min_io 100
# max_fds 8192
# rr_weight priorities
# failback immediate
# no_path_retry fail
# user_friendly_names no
#}
#blacklist {
# wwid 26353900f02796769
# devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
# devnode "^hd[a-z][[0-9]*]"
# devnode "^cciss!c[0-9]d[0-9]*[p[0-9]*]"
# device {
# vendor DEC.*
# product MSA[15]00
# }
#}
[...]
#devices {
# device {
# vendor "COMPAQ "
# product "HSV110 (C)COMPAQ"
# path_grouping_policy multibus
# getuid_callout "/lib/udev/scsi_id -g -u -s /block/%n"
# path_checker directio
# path_selector "round-robin 0"
# hardware_handler "0"
# failback 15
# rr_weight priorities
# no_path_retry queue
# rr_min_io 100
# product_blacklist LUN_Z
# }
# device {
# vendor "COMPAQ "
# product "MSA1000 "
# path_grouping_policy multibus
# }
#}

Thanks a lot in advance
Lucas Brasilino

[1] http://www.lsi.com/rdac/ds4000.html

Hannes Reinecke

2009-04-30 06:25:02 UTC

Permalink

Hi Lucas,

Post by Lucas Brasilino
Hi
I don't know if I'm misundertanding something. I've got an DS4700 and I'm
switching from RDAC[1] to multipath, since it's natively supported in
the distribution I use
here (SLES 10 SP2).
Since RDAC[1] works perfect, I'm trying to use 'rdac' priority in multipath.
blacklist {
devnode "^sda[0-9]*"
}
defaults {
user_friendly_names yes
prio rdac
path_checker tur
}
multipaths {
multipath {
wwid 3600a0b8000327b900000107549f85224
alias mpath0
}
}
I think that using 'prio rdac' makes multipath to use 'mpath_prio_rdac' tool.
# multipath -v2 -ll
mpath0 (3600a0b8000327b900000107549f85224) dm-0 IBM,1814 FAStT
[size=140G][features=1 queue_if_no_path][hwhandler=1 rdac]
\_ round-robin 0 [prio=6][active]
\_ 9:0:0:0 sdb 8:16 [active][ready]
\_ round-robin 0 [prio=1][enabled]
\_ 10:0:0:0 sdc 8:32 [active][ghost]
# mpath_prio_rdac /dev/sdb
6
# mpath_prio_rdac /dev/sdc
1
# multipath -v2 -ll
sdb: rdac prio: inquiry command indicates error
mpath0 (3600a0b8000327b900000107549f85224) dm-0 IBM,1814 FAStT
[size=140G][features=1 queue_if_no_path][hwhandler=1 rdac]
\_ round-robin 0 [prio=0][enabled]
\_ 9:0:0:0 sdb 8:16 [failed][faulty]
\_ round-robin 0 [prio=1][enabled]
\_ 10:0:0:0 sdc 8:32 [active][ghost]
Ok.. working great, activating the second path. But after the faulty
# multipath -v2 -ll
mpath0 (3600a0b8000327b900000107549f85224) dm-0 IBM,1814 FAStT
[size=140G][features=1 queue_if_no_path][hwhandler=1 rdac]
\_ round-robin 0 [prio=2][enabled]
\_ 9:0:0:0 sdb 8:16 [active][ghost]
\_ round-robin 0 [prio=5][active]
\_ 10:0:0:0 sdc 8:32 [active][ready]
Second path is now priority!!! And of course does not fails back! By
the way, my LUN is configured in
DS4700 in sort a way that the first path *is* the path to preferred controller.
I think path priorities should not change. If so first path goes back
to 'active' status.
Am I misunderstanding something ? Or messing things up?

You are using an old version of multipathing for SLES10 SP2.
This had a bug triggering priority inversion on RDAC.
Please update to the latest version.

Cheers,

Hannes

--
Dr. Hannes Reinecke zSeries & Storage
***@suse.de +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)

Chandra Seetharaman

2009-04-30 18:05:43 UTC

Permalink

Hannes,

I think we need to revisit the priority value we provide for preferred
path(4) relative to active path (2) and non-preferred(1).

Consider the following scenario:

Access to a lun thru 2 preferred and 2 non-preferred path. Lets call
path group with preferred paths as pg1 and with non-preferred paths as
pg2.

Initially pg1 has priority of 8 and pg2 has priority of 2. pg1 is chosen
and I/O goes thru pg1, all good.

Both the paths in pg1 fails, pg2 has been made the active path group and
I/O is sent thru that path and since it became "active", its priority
raises to 6 ( 2 path times (active + non-preferred)).

When one of the paths in pg1 comes back, one would expect the failback
to happen. It doesn't happen as pg1's priority (4) is smaller than that
of pg2 (6). Which is not correct.

We can do the same exercise with more than 4 paths also, like 6, 8 etc.,
and the results are worse.

So, IMO we need to give the disproportionately large number for
preferred path w.r.t active and non-preferred. What do you think ?

chandra

Post by Hannes Reinecke
Hi Lucas,

You are using an old version of multipathing for SLES10 SP2.
This had a bug triggering priority inversion on RDAC.
Please update to the latest version.
Cheers,
Hannes

Hannes Reinecke

2009-05-04 10:43:50 UTC

Permalink

Hi Chandra,

Post by Chandra Seetharaman
Hannes,
I think we need to revisit the priority value we provide for preferred
path(4) relative to active path (2) and non-preferred(1).
Access to a lun thru 2 preferred and 2 non-preferred path. Lets call
path group with preferred paths as pg1 and with non-preferred paths as
pg2.
Initially pg1 has priority of 8 and pg2 has priority of 2. pg1 is chosen
and I/O goes thru pg1, all good.
Both the paths in pg1 fails, pg2 has been made the active path group and
I/O is sent thru that path and since it became "active", its priority
raises to 6 ( 2 path times (active + non-preferred)).
When one of the paths in pg1 comes back, one would expect the failback
to happen. It doesn't happen as pg1's priority (4) is smaller than that
of pg2 (6). Which is not correct.

Is this really a valid case?
This means we'll have a setup like this:

rdac
pg1
sda failed
sdb failed
pg2
sdc active
sdd active

Correct?
So, given your assumptions, the proposed scenario would be represented
like this:

rdac
pg1
sda active
sdb failed
pg2
sdc active
sdd active

So it is really a good idea to switch paths in this case? The 'sdb'
path would not be reachable here, so any path switch command wouldn't
have been received, either. I'm not sure _what_ is going to happen
when we switch paths now and sdb comes back later; but most likely
the entire setup will be messed up then:
sda (pref & owned) 6
sdb 0
sdc (sec) 1
sdd (sec & owned) 3
and we'll be getting the path layout thoroughly jumbled then.
So I don't really like this idea. We should only be switching
paths when _all_ paths of a path group become available again.
Providing not all paths have failed in the active group, of course.
Then we should be switching paths regardless.

Cheers,

Hannes

--
Dr. Hannes Reinecke zSeries & Storage
***@suse.de +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)

Chandra Seetharaman

2009-05-04 17:30:28 UTC

Permalink

Hi Hannes
On Mon, 2009-05-04 at 12:43 +0200, Hannes Reinecke wrote:

<snip>

Post by Hannes Reinecke
Is this really a valid case?

Yes, Having more than one patch to a (or both) controller(s) is a valid
case.

Post by Hannes Reinecke
rdac
pg1
sda failed
sdb failed
pg2
sdc active
sdd active
Correct?

correct.

Post by Hannes Reinecke
So, given your assumptions, the proposed scenario would be represented

It is not an assumption, it is the behavior I have seen :)

Post by Hannes Reinecke
rdac
pg1
sda active
sdb failed
pg2
sdc active
sdd active
So it is really a good idea to switch paths in this case? The 'sdb'

Yes. We need to switch for two reasons
- since there is a preferred path available we _should_ use it
(otherwise it will throw off the load balancing the admin has made
in the storage).
- To be consistent with multipath's state before the access to the
preferred controller failed. i.e if multipath has configured a dm
device in this state, multipath _does_ make pg1 the active path
group.

Post by Hannes Reinecke
path would not be reachable here, so any path switch command wouldn't
have been received, either. I'm not sure _what_ is going to happen

Since, both paths are leading to the same controller, mode select sent
for sda would have made sdb also the active controller. But, as you
mentioned it is not seen by dm-multipath.

Post by Hannes Reinecke
when we switch paths now and sdb comes back later; but most likely

The patch I re-submitted last week (Handle multipath paths in a path
group properly during pg_init :
http://marc.info/?l=dm-devel&m=124094710300894&w=2) handles this
situation correctly, by sending an activate during reinstate.

Post by Hannes Reinecke
sda (pref & owned) 6
sdb 0
sdc (sec) 1
sdd (sec & owned) 3

No, this will not be the case. As soon as the access to sdb comes back
it will be seen as pref and owned and hence will get a priority value of
6.

Also, as soon as sda has been made active, sdd will become
passive/ghost, and hence will have the priority value of 1.

Post by Hannes Reinecke
and we'll be getting the path layout thoroughly jumbled then.
So I don't really like this idea. We should only be switching
paths when _all_ paths of a path group become available again.
Providing not all paths have failed in the active group, of course.
Then we should be switching paths regardless.

Here are the details:

===========================================================
(1) Initial configuration (all are good):
pg1
sda (pref and active) - 6
sdb (pref and active) - 6
pg2
sdc (sec and passive) - 1
sdd (sec and passive) - 1
------
(2) Access to sdb goes down
------
pg1
sda (pref and active) - 6
sdb (not there) - 0
pg2
sdc (sec and passive) - 1
sdd (sec and passive) - 1
------
(3) Access to sda goes down, path group switches
------
pg1
sda (not there) - 0
sdb (not there) - 0
pg2
sdc (sec and active) - 3
sdd (sec and active) - 3
------
(4) sda comes back, path group switch _should_ happen here.
to be consistent with (1). If the path group switch happens, sda
will have a priority of 6 and sdc/sdd will have priority
of 1 each (as they will become passive).
Path switch can happen only if the priority we give for preferred
path is lot more than the sum of all priorities of all the paths
in the other path group.
------
pg1
sda (pref and passive)- 4
sdb (not there) - 0
pg2
sdc (sec and active) - 3
sdd (sec and active) - 3

Hope it is clear now.

Post by Hannes Reinecke
Cheers,
Hannes

Chandra Seetharaman

2009-06-23 00:47:24 UTC

Permalink

Hi Hannes,

Please see the attached file for the real example.

Can I go ahead and generate a patch to increase the priority of the
preferred path to say, 50 ?

chandra

Post by Hannes Reinecke
Hi Chandra,

Is this really a valid case?
rdac
pg1
sda failed
sdb failed
pg2
sdc active
sdd active
Correct?
So, given your assumptions, the proposed scenario would be represented
rdac
pg1
sda active
sdb failed
pg2
sdc active
sdd active
So it is really a good idea to switch paths in this case? The 'sdb'
path would not be reachable here, so any path switch command wouldn't
have been received, either. I'm not sure _what_ is going to happen
when we switch paths now and sdb comes back later; but most likely
sda (pref & owned) 6
sdb 0
sdc (sec) 1
sdd (sec & owned) 3
and we'll be getting the path layout thoroughly jumbled then.
So I don't really like this idea. We should only be switching
paths when _all_ paths of a path group become available again.
Providing not all paths have failed in the active group, of course.
Then we should be switching paths regardless.
Cheers,
Hannes

Hannes Reinecke

2009-06-23 06:20:49 UTC

Permalink

Post by Chandra Seetharaman
Hi Hannes,
Please see the attached file for the real example.
Can I go ahead and generate a patch to increase the priority of the
preferred path to say, 50 ?

No. That's just wrong and we'll run into the same problem once
someone increases the number of paths to 50.

What we should do here is to modify the priority value, or
rather the way priority is used.

We should be splitting the current priority value into two
fields, pg priority and number of paths in a pg.
pg priority is the priority of a _single_ path here, and,
by definition as we're using group_by_prio, the priority
of each path in the pg.

Then we should be modifying the algorithm to choose the
next pg to do something like this:

-> Choose the pg with the highest priority
-> If two pgs have the same priority choose the pg
with the highest path count. Maybe we could
even use the highest _valid_ path count here,
depending if we have the information at that point.

This algorithm would solve the problem we're having
now once and for all.

Just adding the priorities of the individual paths will
always lead to these type of problems.

I see if I can find some time to draw up a patch.

Cheers,

Hannes

--
Dr. Hannes Reinecke zSeries & Storage
***@suse.de +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)

Lucas Brasilino

2009-05-05 17:59:39 UTC

Permalink

Well, maybe it's a newbie opinion, but looks like that's easier if
each path has a
'fixed value' priority, isn't it ? I've made a little code here that returns
a choosen fixed value to paths and everything goes fine.....

regards
Lucas Brasilino

Post by Hannes Reinecke
Hi Lucas,

You are using an old version of multipathing for SLES10 SP2.
This had a bug triggering priority inversion on RDAC.
Please update to the latest version.
Cheers,
Hannes

--
dm-devel mailing list
https://www.redhat.com/mailman/listinfo/dm-devel