DMA Out of SW-IOMMU space

propofol · 05-29-2019, 02:29 AM

I am not sure if this is the correct place for it but either way. I have a Debian Stretch headless server which have become unresponsive lately and it seems to be related to this error:

Code:

DMA: Out of SW-IOMMU space for 8 bytes at device 0000:00:1d.0

This PC also function as a wifi access point with 2 adapters attached to the USB ports, a RT2870/RT3070 & Atheros AR9271 Wireless Adapter. The modules used are ath9k_htc, rt2800usb & rt2x00usb. Since my upgrade to the 4.9.0-9-amd64 stock kernel the PC seems to become unresponsive after a few days and the samba & nfs shares times out. On reboot, /var/log/syslog & kern.log is filled with this message. >5000000 lines worth.

The device in question seems to be the USB 2.0 controller hub:
00:1d.0 USB controller: Intel Corporation 7 Series/C216 Chipset Family USB Enhanced Host Controller #1 (rev 04)

I have been able to find similar error messages on Google relating to much older kernels & one suggestion was to set swiommu memory limit higher by adding a option to kernel boot line in grub. I am just wondering if this is the correct solution.

Regards,
Stefan

business_kid · 05-29-2019, 08:34 AM

When you get that, what's the o/p of 'free -h'?

Have you any clue what address they're writing to?

There is a thing in the kernel all right. It's relevant on 32 bit boxes, I think. The 32 bit address limits your memory at 4 Gigs. Now basically all processors since the z80 have been able to address twice the data bus width by a double data width instruction. So a 32 bit cpu can pair internal registers, and store 64 bit numbers, manipulate them them, or write them, so you're able to higher, but in the past it probably required a little gymnastics. There's piles of IOMMU handlers in the kernel. Find your kernel config in /boot and grep it for IOMMU; check your chipset is supported.

This shouldn't be an obstacle on a debian stock kernel.

propofol · 05-29-2019, 02:41 PM

Quote:

Originally Posted by business_kid

When you get that, what's the o/p of 'free -h'?

Unfortunately I do not know. I was unable to ssh the server & when I attached a screen & keyboard there was no response so I had to power cycle the computer. I only saw the messages after the restart. I do have enough memory for the server:

Code:

              total        used        free      shared  buff/cache   available
Mem:           7.5G        624M        4.6G         14M        2.3G        6.6G
Swap:          7.5G          0B        7.5G

Quote:

Originally Posted by business_kid

Have you any clue what address they're writing to?

There is a thing in the kernel all right. It's relevant on 32 bit boxes, I think. The 32 bit address limits your memory at 4 Gigs. Now basically all processors since the z80 have been able to address twice the data bus width by a double data width instruction. So a 32 bit cpu can pair internal registers, and store 64 bit numbers, manipulate them them, or write them, so you're able to higher, but in the past it probably required a little gymnastics. There's piles of IOMMU handlers in the kernel. Find your kernel config in /boot and grep it for IOMMU; check your chipset is supported.

This shouldn't be an obstacle on a debian stock kernel.

I am using a stock kernel & I was under the impression the Intel chipset on the motherboard is fully supported. At least I have not previously had problems - everything was essentially plug & play.

grep IOMMU /boot/config-4.9.0-9-amd64:

Code:

CONFIG_GART_IOMMU=y
CONFIG_CALGARY_IOMMU=y
CONFIG_CALGARY_IOMMU_ENABLED_BY_DEFAULT=y
CONFIG_IOMMU_HELPER=y
CONFIG_VFIO_IOMMU_TYPE1=m
# CONFIG_VFIO_NOIOMMU is not set
CONFIG_IOMMU_API=y
CONFIG_IOMMU_SUPPORT=y
# Generic IOMMU Pagetable Support
CONFIG_IOMMU_IOVA=y
CONFIG_AMD_IOMMU=y
CONFIG_AMD_IOMMU_V2=y
CONFIG_INTEL_IOMMU=y
CONFIG_INTEL_IOMMU_SVM=y
# CONFIG_INTEL_IOMMU_DEFAULT_ON is not set
CONFIG_INTEL_IOMMU_FLOPPY_WA=y
# CONFIG_IOMMU_DEBUG is not set
# CONFIG_IOMMU_STRESS is not set

Thanks for replying,
Stefan

business_kid · 05-30-2019, 05:20 AM

Ok. Scratch the IOMMU issue, I'm sure you're ok.

If you had to reboot the box, it tells us nothing except that it crashed. Check the logs. You'll have to go at this the hard way. If you an update and a dist-upgrade, it might sort itself. I'd try a memtest, and get Debian's mind on it, posting a bug if necessary.

propofol · 06-03-2019, 01:57 AM

Quote:

Originally Posted by business_kid

Ok. Scratch the IOMMU issue, I'm sure you're ok.

If you had to reboot the box, it tells us nothing except that it crashed. Check the logs. You'll have to go at this the hard way. If you an update and a dist-upgrade, it might sort itself. I'd try a memtest, and get Debian's mind on it, posting a bug if necessary.

I have done an update & dist-upgrade and I have plugged the wifi adapters into usb 3.0 ports. So far the error has not showed up again in either syslog or kernel.log. Funny thing is that this PC has been running with up times of several months and after the last kernel update it crashed twice so I don't think it is a hardware issue.

Regards,
Stefan

PS This is the output of

Code:

sudo grep "SW-IOMMU" /var/log/kern.log | sed -e 's/\[[0-9]*\.[0-9]*\]//g' | sort | uniq -c | sort -n | tail

Code:

   4502 May 27 05:27:43 ISKServer kernel:  DMA: Out of SW-IOMMU space for 8 bytes at device 0000:00:1d.0
   4503 May 27 05:16:49 ISKServer kernel:  DMA: Out of SW-IOMMU space for 8 bytes at device 0000:00:1d.0
   4509 May 27 05:34:08 ISKServer kernel:  DMA: Out of SW-IOMMU space for 8 bytes at device 0000:00:1d.0
   4510 May 27 05:30:49 ISKServer kernel:  DMA: Out of SW-IOMMU space for 8 bytes at device 0000:00:1d.0
   4512 May 27 05:42:01 ISKServer kernel:  DMA: Out of SW-IOMMU space for 8 bytes at device 0000:00:1d.0
   4515 May 27 05:06:08 ISKServer kernel:  DMA: Out of SW-IOMMU space for 8 bytes at device 0000:00:1d.0
   4516 May 27 05:24:56 ISKServer kernel:  DMA: Out of SW-IOMMU space for 8 bytes at device 0000:00:1d.0
   4520 May 27 04:50:56 ISKServer kernel:  DMA: Out of SW-IOMMU space for 8 bytes at device 0000:00:1d.0
   4538 May 27 04:47:07 ISKServer kernel:  DMA: Out of SW-IOMMU space for 8 bytes at device 0000:00:1d.0
   4575 May 27 04:41:54 ISKServer kernel:  DMA: Out of SW-IOMMU space for 8 bytes at device 0000:00:1d.0
   4664 May 27 05:32:31 ISKServer kernel:  DMA: Out of SW-IOMMU space for 8 bytes at device 0000:00:1d.0

The error appears over about one hour & occasionally > 4000x per second.

business_kid · 06-03-2019, 12:50 PM

Sounds Good. Broke itself, fixed itself…… Lets hope it stays. Saves us thinking.