Linux - ServerThis forum is for the discussion of Linux Software used in a server related context.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
I am trying to understand the difference between the below parameters on Centos6/7, aside from the fact that scsi_mod is built-in with the kernel, and lpfc_mod is a loadable module when using specific HBA adapter.
i.e., if lpfc module is loaded (modinfo lpfc), what is the relevance of scsi_mod?
+++ to give a background on the question +++
The server has multipathing and there are connection through 03:00 which apparently is a Broadcom HBA, and 04:00 which is an Emulex HBA.
Code:
[root@c6server~]lspci -v -s 03:00
03:00.3 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
Subsystem: Hewlett-Packard Company Ethernet 1Gb 4-port 331FLR Adapter
Flags: bus master, fast devsel, latency 0, IRQ 36
Memory at f6a60000 (64-bit, prefetchable) [size=64K]
Memory at f6a50000 (64-bit, prefetchable) [size=64K]
Memory at f6a40000 (64-bit, prefetchable) [size=64K]
[virtual] Expansion ROM at f4140000 [disabled] [size=256K]
Capabilities: [48] Power Management version 3
Capabilities: [50] Vital Product Data
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Capabilities: [a0] MSI-X: Enable- Count=17 Masked-
Capabilities: [ac] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [13c] Device Serial Number 00-00-a0-d3-c1-00-13-1b
Capabilities: [150] Power Budgeting <?>
Capabilities: [160] Virtual Channel
Kernel driver in use: tg3
Kernel modules: tg3
<more output>
[root@c6server~]lspci -v -s 04:00
04:00.0 Fibre Channel: Emulex Corporation Saturn-X: LightPulse Fibre Channel Host Adapter (rev 03)
Subsystem: Emulex Corporation Saturn-X: LightPulse Fibre Channel Host Adapter
Physical Slot: 1
Flags: bus master, fast devsel, latency 0, IRQ 40
Memory at f7ff0000 (64-bit, non-prefetchable) [size=4K]
Memory at f7fe0000 (64-bit, non-prefetchable) [size=16K]
I/O ports at 6000 [size=256]
[virtual] Expansion ROM at f4300000 [disabled] [size=256K]
Capabilities: [58] Power Management version 3
Capabilities: [60] MSI: Enable- Count=1/16 Maskable+ 64bit+
Capabilities: [78] MSI-X: Enable+ Count=32 Masked-
Capabilities: [84] Vital Product Data
Capabilities: [94] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [12c] Power Budgeting <?>
Kernel driver in use: lpfc
Kernel modules: lpfc
So c6server was rebooted and for some reason, some LUNs were no longer visible, and the following lines showed up in /var/log/messages:
Code:
c6server kernel: scsi: On host 3 channel 0 id 0 only 511 (max_report_luns) of 569 luns reported, try increasing max_report_luns.
c6server kernel: scsi: On host 4 channel 0 id 0 only 511 (max_report_luns) of 569 luns reported, try increasing max_report_luns
That pointed me to increase the following params to 1024 by editing the grub line.
Rebooted the server, now all the LUNs were visible.
That all went fine until i ran through another server with the same setup (multipathing through 2 HBAs which are Broadcom HBA on 03:00 and Emulex HBA on 04:00).
Notice that there are 1023 actual LUNs but max_luns was only 512. How was that possible? The weird thing is that, it is capped at 1023 but in reality there are more LUNs presented to c6server22 and it's not reading them all. Where did 1023 come from?
I know I have to keep increasing /sys/module/scsi_mod/parameters/max*luns but I would like to understand how things work and this is all confusing for me now. I would really appreciate any guidance on this.
lpfc and scsi_mod are independent layers. All disk commands go through the SCSI layer (including fibre channel and SATA). Only commands going to Emulex fibre channel connected devices also go through lpfc. Each has its own settings for max supported LUNs. If you want to have 1024 LUNs on one SCSI ID, then you need to set max luns in both the SCSI layer and the Emulex layer.
The error you are getting says it is from the 'scsi:' driver.
Code:
c6server kernel: scsi: On host 3 channel 0 id 0 only 511 (max_report_luns) of 569 luns reported, try increasing max_report_luns.
Not sure what the Ethernet adapter has to do with any of this. lspci addresses have nothing to do with host adapter numbers. It's more likely that host3 and host4 are two Emulex cards. Try the commands:
lpfc and scsi_mod are independent layers. All disk commands go through the SCSI layer (including fibre channel and SATA). Only commands going to Emulex fibre channel connected devices also go through lpfc. Each has its own settings for max supported LUNs. If you want to have 1024 LUNs on one SCSI ID, then you need to set max luns in both the SCSI layer and the Emulex layer.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
