I'm running a 2.6.30.9 kernel on a SuperMicro/Intel chipset, and I'm writing a module for a PCIe device that uses MSI. The first time I load the module using insmod, interrupts work correctly, and the ISR is run as it should. When I do a rmmod and another insmod, interrupts are not caught by my module.
I've included the code from the init and exit functions below, but I have followed the directions in the MSI HOW-TO.txt file to call pci_enable_msi then request_irq in the init function. In the exit function, I call free_irq, then pci_disable_msi.
When interrupts are working (first insmod after reboot), /proc/interrupts shows the following after running a test program:
Code:
53: 33289 566723 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge RMA Interrupt
When interrupts aren't working after reloading the module, /proc/interrupts show the following after running the same test program:
Code:
53: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge RMA Interrupt
I know that the problem isn't with the device because a reboot will fix the problem without restarting the device. Additionally, with a signal tap on the FPGA, the IRQ request signal remains high for the time when the first interrupt should come. The acknowledge signal isn't coming back to the device because the module isn't receiving the interrupts and going through the ISR.
Both when interrupts are working, and when they aren't, lspci -vv show the following:
Code:
05:00.0 Class ff00: Altera Corporation Unknown device 0004 (rev 01)
Subsystem: Altera Corporation Unknown device 0004
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 53
Region 0: Memory at 680000000 (64-bit, prefetchable) [size=1024M]
Capabilities: [50] Message Signalled Interrupts: 64bit+ Queue=0/2 Enable+
Address: 00000000fee00000 Data: 40ba
Capabilities: [68] MSI-X: Enable- Mask- TabSize=2
Vector table: BAR=0 offset=00000400
PBA: BAR=0 offset=00000500
Capabilities: [78] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [80] Express Endpoint IRQ 0
Device: Supported: MaxPayload 2048 bytes, PhantFunc 0, ExtTag-
Device: Latency L0s <64ns, L1 <1us
Device: AtnBtn- AtnInd- PwrInd-
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
Device: RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
Device: MaxPayload 256 bytes, MaxReadReq 512 bytes
Link: Supported Speed unknown, Width x4, ASPM L0s, Port 1
Link: Latency L0s unlimited, L1 unlimited
Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
Link: Speed unknown, Width x4
Here is my source code for the init and exit functions for loading and unloading the module:
Code:
static int __init ssrma_init(void){
int retval;
ssrma_DEV_T *rma_dev;
irq_handler_t irq_handler = &ssrma_pciIsr;
rma_dev = &install_device;
/* Register the driver with the PCI core */
retval = pci_register_driver(&pci_driver);
if(retval) return retval;
/* Setup the cdev portion of the rma_device struct */
retval = rma_setup_cdev(rma_dev);
if(retval){
goto registered;
}
/* Verify the device is found on a bus */
rma_dev->pci_dev = pci_get_device(ssrma_VENDOR_ID, ssrma_DEVICE_ID, NULL);
if(!rma_dev->pci_dev){
retval = -ENODEV; // No such device
goto cleanup;
}
/* Add the device to the device structure */
pci_dev_put(rma_dev->pci_dev);
/* Enable the MSI interrupts */
if(pci_enable_msi(rma_dev->pci_dev)){
retval = -1;
goto cleanup;
}
/* Setup the interrupt ISR */
if(request_irq(rma_dev->pci_dev->irq, irq_handler, IRQF_DISABLED, "RMA Interrupt", rma_dev)){
retval = -1;
goto msi_cleanup;
}
return 0;
ioremap_cleanup:
/* Unmap the mapped memory space */
iounmap(rma_dev->virt_start_addr);
/* Free the requested IRQ */
free_irq(rma_dev->pci_dev->irq, rma_dev);
msi_cleanup:
/* Disable the MSI interrupts */
pci_disable_msi(rma_dev->pci_dev);
cleanup:
/* Remove the device */
cdev_del(&rma_dev->cdev);
registered:
/* Unregister the driver */
pci_unregister_driver(&pci_driver);
return retval;
} /* end rma_init */
static void __exit ssrma_exit(void){
ssrma_DEV_T *rma_dev;
rma_dev = &install_device;
/* Unmap the mapped space */
iounmap(rma_dev->virt_start_addr);
/* Free the requested IRQ */
free_irq(rma_dev->pci_dev->irq, rma_dev);
/* Disable the MSI interrupts */
pci_disable_msi(rma_dev->pci_dev);
/* Remove the device */
cdev_del(&rma_dev->cdev);
/* Unregister the driver */
pci_unregister_driver(&pci_driver);
} /* end rma_exit */
If you have had a similar problem, or know what I'm doing wrong, please let me know. If you need more information, I would be happy to provide it. It is frustrating to have to reboot everytime I change the module code, and it's something that needs to be fixed. Thank you.
Reloading Module stops MSIs
