I'm using fc14 and the SG driver to test some SCSI (SAS) targets. In doing so, I'm bumping up against what appears to be a 512KB maximum transfer size per command. Transfers up to 4MB sometimes work, but often they result in ENOMEM or EINVAL returned from the write() function in the SG driver.

I could not find any good documentation on how the SCSI system in Linux works so I've been studying the source for drivers in drivers/scsi. I see that there is a scsi_device struct that contains a request_queue struct that contains a queue_limits struct that contains an element called max_sectors. The SG driver seems to use this to limit the size of the reserve buffer it is willing to create. I see that there are several constants used to initialize max_sectors to 1024 which would result in the 512KB limit I see (with targets having 512 byte sectors).

At this point I have several questions:
1) When the open() function for the sg driver gets called, who initializes the scsi_device struct with the default values?
2) Can I merely change the limits struct to arbitrary values after initialization and cause the SG ioctls to set the reserve buffer to allow greater values?
3) What are the likely limits to what size the reserve buffer can be? My SAS HBA says that it's maximum transfer size is 32MB. I'd like to be able to take full advantage of that.
4) As I mentioned earlier, the SG driver seems to be able to perform I/Os with transfer sizes greater than its reserve buffer size. How does it do that? What imposes the limitation on transfer size if it's not the reserve buffer?

Thanks in advance to all of you who took the time to read this entire post and ponder whether you can answer any of my questions.

Marc

The limit is due to the scatter-gather list made up for each I/O. In the worst case there is one element for each page to be read or written. See drivers/scsi/scsi_lib.c

The constant MAX_PHYS_SEGMENTS limits the scatter-gather list size.

The reason it isn't larger is that there is no reason to do so. The overhead per command is very small by the time you get to 512K transfers, so making it larger would have no effect on performance under normal conditions. You can still issue larger reads or writes from an application, the scsi subsystem will just break them up into multiple scsi commands.

I suspect you can issue larger transfers if you have your data in a contiguous buffer so that you don't need many list elements and you are doing direct I/O. I/O through cache will always be broken up into pages.

1 members found this post helpful.

Thanks for the tidbit of information. I will explore making the SG lists bigger as well. Can you think of what the next limitation I'll hit is? I can add physical memory to my tests systems if necessary. I just need to understand how it will be allocated and if doing so will make more memory available to the kernel for allocation of buffers for the SG driver.

Marc