Looks like sg_map falls down with flat address method (and maybe more..)
brianatpurestorage opened this issue · comments
I have been doing some testing with Linux and LUN numbers above 255. Two problems happen, one is fixable and one is not.
There is this check:
(((my_idlun->dev_id >> 8) & 0xff) == sidp->lun)
1 != 16385, etc.
This can't pass with flat addressing since 16384 is added to the LUN number and it is compared with only numbers 255 and lower. This is fixable by anding the LUN with 0x3ff and then it will work but only up to LUN 255 in flat addressing. LUNs higher than 255 continue to fail because of the kernel ioctl, SCSI_IOCTL_GET_IDLUN. This ioctl can only give values 255 and lower for the LUN number so the higher LUNs can never match.
I don't know what the plan is for flat address method handling in LUN numbers but it is becoming more interesting in the world of virtualization. No one used to have more than 255 LUNs on a host but now it is showing up more.
Actually there's not much we can do.
Ultimately it's down to the SCSI HBA how many LUNs it can address, and which addressing scheme is supported.
Clearly it'll be pointless trying to ask for LUNs larger than 16 on a SCSI parallel HBA, and similarly some HBAs have limitations on the max LUN it can support; some have fixed-size fields for the LUN number, which occasionally is smaller than 64 bytes.
And we have to take into account the LUN number supported by the target; there has been a shift in meaning between SCSI-2, SPC, and SPC-2 (and later).
So, sadly, we cannot assume that anything beyond 255 is actually supported. Hence I'd rather have sg_map stop at LUN 255, and emit a warning if we encounter anything beyond that.
Experience shows that only IBM DS supports LUN numbers higher than 255; most of the other arrays have a hard cap at 255 LUNs per host for precisely this reason.
As a storage vendor it is difficult to find consensus among initiators and their expectations past LUN 255. In flat addressing there is access to LUNs 16383 but it seems like many initiators just consume the first two bytes of the 8 bytes on the LUN number and use that. This is clearly wrong by the spec but we have found that initiators like ESXi, Hitachi VSP, and Oracle VM server will continue to work with peripheral addressing past LUN 255. They just steal bits from the bus identifier and are happy.
By default what we do is present LUNs 256 and higher in flat addressing while LUNs 255 and below are presented as peripheral. That is also what IBM does, but even that is sort of spec compliant in the sense that it probably makes more sense to present all the LUNs in flat addressing if the highest LUN exceeds 255 so we don't get the gap showing up between 255 and 16385. That would involve though changing addressing on already discovered LUNs which we weren't convinced would work.
It would be nice is there was a standard way of doing this that initiators and target could agree on when LUN 255 is exceeded since the virtual world of today has this appearing more often.
The biggest problem here is that it's anyones guess how LUN 256 will be presented. So it's virtually impossible to predict what the storage array will accept for that LUN.
This is why I restricted the linux implementation for linear scans to LUNs < 256; if larger LUNs are required please use REPORT LUNs.
I guess this is my point. I would like to see a collaboration in this area and get unified way of doing this. Why not start with Linux since it is the most likely to react and change?
sg_map is an old utility kept alive mostly for backward compatibility. Have you looked at my lsscsi utility? 'lsscsi -g' will show the mapping between the primary block level device name and the sg driver device name. And lsscsi should do a better job with "larger" LUNs.
Doug,
Yes I can use 'lsscsi' for sure. My concern really is around how storage vendors should present LUNs above 255 to Linux. The ioctl that sg_map is using will not work for that since it will have collisions above 255. The Linux kernel doesn't seem to deal with peripheral vs. flat address space at all. It just presents flat addresses as 16384 + the actual intended LUN number. What vendors, us included, seem to do is for LUNs above 255 is to switch the LUN address mode from peripheral, which has a 255 cap, to flat, where we can have access to more bits. This leads to LUNs 0-255 showing as peripheral and higher LUNs appearing as flat. I guess that is OK but there doesn't seem to be initiator consensus around how LUNs above 255 should be displayed. Some OS's prefer just continuing in peripheral addressing and step on bits that don't belong to the LUN number itself.
I was mainly looking for some thoughts from those with an opinion about high numbered LUNs. I think in our virtualized world it will become more common.
Well; from the initiator side there is nothing to 'deal' with wrt LUN numbers.
We actually need to use them as is as the target might derive some routing information from the LUN number; hence we can't just arbitrarily strip the address method bits from the LUN and hope that everything works out okay.
So from the initiator side the LUNs are opaque IDs; that some LUNs are consecutive is actually considered legacy, and whenever possible we try to rely on the 'REPORT LUNS' SCSI command to get the list of available LUN numbers.
After all, the target might present us with a LUN 0 in peripheral addressing and a LUN 0 in flat addressing mode; if both show up in REPORT LUNS then they are two distinct LUNs (irrespective of the fact the both are actually LUN 0), and we need to treat them as two devices.
From the target side, however, things are different. Here you need to make a choice on how to present higher LUN numbers to the host.
What I would be doing is to present LUN numbers below 256 as peripheral (for compability), and use 'flat' or 'extended flat' addressing for anything higher than 255.
You could switch the addressing scheme completely to 'extended flat' depending on the host type, but then you fall foul on the infamous 'LUN 0 must always be present' SCSI paradigm, which essentially binds you to using peripheral addressing for at least LUN 0 :-(
But do avoid LUN aliasing; a particular LUN should always have a unique LUN number.
Apart from using lsscsi (which uses sysfs) help is on the way (slowly) in the sg driver interface. I'm trying to get a major driver update into the kernel (lots of back and forth required, not helped by the size of the update). In particular ioctl(SG_GET_SCSI_ID) is being extended, in a backward compatible way, to additionally yield the scsi_lun which is an 8 byte array in T10's preferred LUN representation (until they change their minds). This is done by adding an anonymous union at the end of struct sg_scsi_id, taking advantage of the 'int unused[2]' currently there.