How to know absolute address of memory mapped device on PCIE card?

kaza · 05-31-2020, 12:52 AM

Hello!

I'm trying to test an FPGA with memory mapped registers which is installed on a PCIE
card in a PC with Fedora 32 installed on it.

After some googling I saw that I'll need something like "busybox"
and will probably need to recompile the kernel to disable the "CONFIG_STRICT_DEVMEM"
or add a boot switch "iomem=relaxed" but before all this I need to know
what is the absolute address at which the FPGA is seen by the PCIE.
I don't even know if it's mapped throught the IO address space of memory
address space.

I tried using "lspci" and saw that the card (Xilinx) is seen:

Code:

<leonb-lap>[root].../root>lspci
00:00.0 Host bridge: Intel Corporation 82946GZ/PL/GL Memory Controller Hub (rev 02)
00:01.0 PCI bridge: Intel Corporation 82946GZ/PL/GL PCI Express Root Port (rev 02)
00:02.0 VGA compatible controller: Intel Corporation 82946GZ/GL Integrated Graphics Controller (rev 02)
00:1b.0 Audio device: Intel Corporation NM10/ICH7 Family High Definition Audio Controller (rev 01)
00:1c.0 PCI bridge: Intel Corporation NM10/ICH7 Family PCI Express Port 1 (rev 01)
00:1c.1 PCI bridge: Intel Corporation NM10/ICH7 Family PCI Express Port 2 (rev 01)
00:1d.0 USB controller: Intel Corporation NM10/ICH7 Family USB UHCI Controller #1 (rev 01)
00:1d.1 USB controller: Intel Corporation NM10/ICH7 Family USB UHCI Controller #2 (rev 01)
00:1d.2 USB controller: Intel Corporation NM10/ICH7 Family USB UHCI Controller #3 (rev 01)
00:1d.3 USB controller: Intel Corporation NM10/ICH7 Family USB UHCI Controller #4 (rev 01)
00:1d.7 USB controller: Intel Corporation NM10/ICH7 Family USB2 EHCI Controller (rev 01)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev e1)
00:1f.0 ISA bridge: Intel Corporation 82801GB/GR (ICH7 Family) LPC Interface Bridge (rev 01)
00:1f.2 IDE interface: Intel Corporation NM10/ICH7 Family SATA Controller [IDE mode] (rev 01)
00:1f.3 SMBus: Intel Corporation NM10/ICH7 Family SMBus Controller (rev 01)
01:00.0 Memory controller: Xilinx Corporation Device 8038
04:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetLink BCM5786 Gigabit Ethernet PCI Express (rev 02)

and some more info specifically about it:

Code:

<leonb-lap>[root].../root>lspci -s 01:00.0 -v
01:00.0 Memory controller: Xilinx Corporation Device 8038
        Subsystem: Xilinx Corporation Device 0007
        Flags: bus master, fast devsel, latency 0, IRQ 11
        Memory at d0200000 (32-bit, non-prefetchable) [size=1M]
        Memory at d0100000 (32-bit, non-prefetchable) [size=1M]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [c0] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [300] Secondary PCI Express

The FPGA registers address space is 64K, does the two shown memories
0xd0200000, 0xd0100000 one of them is the FPGA base? Or they're just
internal PCIE address spaces?
Can it be the access to FPGA is through a series of writes/reads to some
specific PCIE registers?

So far I haven't yet recompipled the kernel nor did I add "iomem=relaxed"
boot kernel argument so running absolute address reads like:

busybox devmem <address>

is limited to the lowest 1MB which leaves values like
0xd0200000, 0xd0100000 outside the permitted range.

I'm mostly seeking feedback that I'm generally going in the right
direction (or totaly wrong one?).

TIA,
kaza.

business_kid · 05-31-2020, 10:31 AM

Ok, 225 reviews and no replies. Let me have a go.

FPGAs I know. But handling FPGAs under PCIE is not for the faint hearted. FPGAs are for the most part, non X86 cpus with other random stuff nobody knows about. It's just your dirty thoughts in there, and of course you're forbidden to beathe a word about it, because your employer is hyper-paranoid about industrial espionage. In fact he's paranoid about me saying that

. There's a relevant Irish song https://www.youtube.com/watch?v=6Pns4Aq-RNs so

IIRC from my own hardware days, FPGAs have little external or internal EEProms re-programming them on powerup. If your company invested in some PCIE card there will have to be instructions, specs, & data, and some notes on software. Hard addressing is passé at any rate, I was led to believe; position independent stuff is preferred so as not to be a stationary target for hackers. Xilinx,anyhow are full of documentation, and I think they did ISA cards, then pci cards, so it must be pcie cards now. There's Gigabytes of documentation on it, and you're going to (sadly) read some of it. They can't fire you for reading docs. But all it will give is outputs into some hardware monitor so you can see your FPGA misbehave on a screen instead of a 'scope. Unless you need to debug specific pulse trains, the card may slow you down as much as it speeds you up. I lookes at the cards around the ISA --> PCI era, and decided they weren't worth the time investment.

sp331yi · 05-31-2020, 01:17 PM

Thanks for your input and THE LINK!
{i didn't say that} LOL

kaza · 06-01-2020, 12:27 AM

Thanks for the reply, business_kid! I'll listen to the link at home :-)

The board is Xilinx evaluation board and the FPGA code is ours.
From what you wrote it seems the access is some indirect
writing/reading of some specific addresses (or IO addresses?)
that supply the address/data/controls from which, eventually,
a bus access is created for the FPGA. And as you mentioned,
there are probably some translation tables to read too
to calculate the correct address. So far we used a Windows app
for it and I thought why not try doing it under LINUX.
It seems not as easy as I thought but I'll try searching for docs
and figuring out how can it be done.

Thanks
kaza.

business_kid · 06-01-2020, 04:52 AM

It's done via pointers in a program, afaik.In it's simplest terms

The program says:'put these (NN) pointers somewhere pls'
The kernel says: 'OK, I've tucked them $THERE.'

So the program has it's pointers starting from $THERE to $THERE+NN, and making sense of it is very difficult. This is much more secure than the fixed I/O and hard coded adresses used in the bad old ISA days.