physical memory access in linux

mashhur · 01-26-2012, 06:03 AM

i have a cpu and a fpga(Spartan 6) in board. im gonna write and read fpga address data through EBI.
Now the questions suffering me are:
1. Does CPU see my FPGA memory (no any driver or im not able to write a driver for it)?
2. i tried to open "/dev/mem" and applied my physical addr for mmap(), nothing happened.
let's say im gonna write "0xFFFF" to address of 0xE0300000.
how to implement my issue?

pls help!
thanks in advance.

paulsm4 · 01-27-2012, 12:52 AM

The whole point of a virtual-memory operating system like Linux or Windows is to *insulate* user space programs from machine details like "physical memory addresses". And the whole point of kernel-level modules like "device drivers" is to act as the "middle man" between the virtual environment a user-space process runs in, and the kernel.

So yes, typically all direct access to an arbitrary device connected to the OS would be done through a device driver.

And typically, you would already have a device driver available to you: either from your vendor, or from the open source community. The driver would include details on how to program the device.

Q: Isn't your FPGA on a board? And the *board* doesn't run Linux, does it?

Q: Don't you just use Linux (or Windows) as a development platform for your FPGA device? Or to connect to the device (e.g. through an RS-232 or USB connection)?

Q: Have you downloaded and had a chance to play with the XiLinx "ISE Webpack" development software?

http://www.xilinx.com/products/desig...se-webpack.htm

mashhur · 01-27-2012, 02:04 AM

Quote:

Originally Posted by paulsm4

The whole point of a virtual-memory operating system like Linux or Windows is to *insulate* user space programs from machine details like "physical memory addresses". And the whole point of kernel-level modules like "device drivers" is to act as the "middle man" between the virtual environment a user-space process runs in, and the kernel.

So yes, typically all direct access to an arbitrary device connected to the OS would be done through a device driver.

And typically, you would already have a device driver available to you: either from your vendor, or from the open source community. The driver would include details on how to program the device.

Q: Isn't your FPGA on a board? And the *board* doesn't run Linux, does it?

Q: Don't you just use Linux (or Windows) as a development platform for your FPGA device? Or to connect to the device (e.g. through an RS-232 or USB connection)?

Q: Have you downloaded and had a chance to play with the XiLinx "ISE Webpack" development software?

http://www.xilinx.com/products/desig...se-webpack.htm

hi there,
thanks trying to help me.
A: Yes, it is, FPGA is attached to the board. Board has another CPU that runs Linux on it.

A: I did serial connections by opening the device drivers, no problem working fine.

A: I used twice Xilinx "ISE Webpack" and 100% believe that there is nothing to do with my problem now.

Now, I will tel you what is wrong with board.
Board has also EBI (as you EBI supports bus and data addresses). Though the EBI I should read and write FPGA memory in Linux kernel.
ex: EBI data address (16pins) is 0xE0300320.
how to write 0xFFFF(16bits) to that address?
thanks.

paulsm4 · 01-27-2012, 11:09 AM

Hi -

Short answer: "No".

Longer answer: "It can be done, but it will take a bit of work".

This link gives you the gist of how you'd need to proceed:

http://www.at91.com/forum/viewtopic.php/f,12/t,20598/

Nominal Animal · 01-27-2012, 03:34 PM

If the memory is accessible to the kernel as normal RAM, then this program should work:

Code:

#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <errno.h>

#include <string.h>
#include <stdio.h>

/*
 * API:
*/

/* Read bytes of physical memory starting at address into destination.
*/
int mem_read(const unsigned long address, void *const destination, const size_t bytes);

/* Write bytes from source to physical memory starting at address.
*/
int mem_write(const unsigned long address, const void *const source, const size_t bytes);



/*
 * Internal implementation:
*/

static int mem_device = -1;

static int mem_open(void)
{
    int result;

    if (mem_device != -1)
        return errno = EBADF;

    do {
        result = open("/dev/mem", O_RDWR);
    } while (result == -1 && errno == EINTR);
    if (result == -1)
        return errno;

    mem_device = result;
    return 0;
}

int mem_read(const unsigned long address, void *const destination, const size_t bytes)
{
    unsigned char   *d = (unsigned char *)destination;
    off_t            o = (off_t)address;
    size_t           n = bytes;
    ssize_t          r;

    while (n) {

        r = pread(mem_device, d, n, o);
        if (r == (ssize_t)n) {
            return 0;

        } else
        if (r >= (ssize_t)0) {
            d += r;
            n -= r;
            o += r;

        } else
        if (r != (ssize_t)-1) {
            return errno = EIO;

        } else
        if (errno == EBADF) {
            if (mem_open())
                return errno;

        } else
        if (errno != EINTR && errno != EAGAIN && errno != EWOULDBLOCK)
            return errno;
    }

    return 0;
}

int mem_write(const unsigned long address, const void *const source, const size_t bytes)
{
    unsigned char   *s = (unsigned char *)source;
    off_t            o = (off_t)address;
    size_t           n = bytes;
    ssize_t          w;

    while (n) {

        w = pread(mem_device, s, n, o);
        if (w == (ssize_t)n) {
            return 0;

        } else
        if (w >= (ssize_t)0) {
            s += w;
            n -= w;
            o += w;

        } else
        if (w != (ssize_t)-1) {
            return errno = EIO;

        } else
        if (errno == EBADF) {
            if (mem_open())
                return errno;

        } else
        if (errno != EINTR && errno != EAGAIN && errno != EWOULDBLOCK)
            return errno;
    }

    return 0;
}

/*
 * Example main().
*/

int main(void)
{
    unsigned long   addr = 0xE0300000UL;
    unsigned short  word = 0xFFFFU;

    if (mem_write(addr, &word, sizeof(word)) {
        fprintf(stderr, "Cannot write %d bytes to physical memory address 0x%08lX: %s.\n",
                        (int)sizeof(word), addr, strerror(errno));
        return 1;
    }

    if (mem_read(addr, &word, sizeof word)) {
        fprintf(stderr, "Cannot read %d bytes from physical memory address 0x%08lX: %s.\n",
                        (int)sizeof(word), addr, strerror(errno));
        return 2;
    }

    printf("Physical memory address 0x%08lX = 0x%04X (%d bytes).\n",
           addr, (int)word, (int)sizeof(word));
    return 0;
}

Save it as e.g. 0xe0300000.c and compile and run it using e.g.

Code:

gcc 0xe0300000.c -Wall -O3 -o 0xe0300000
sudo ./0xe0300000

Because it accesses /dev/mem, it requires superuser rights (root).

The above will not work for you, if the FPGA is not already memory-mapped by the kernel and/or hardware.

In that case, you need a simple character device driver you can access from userspace. I have no idea how the EBI bus is handled by your hardware, it might be on a separate bus chip connected via USB for all I know. There are two possibilities, however:

Directly hardware accessible by the CPU (memory-mapped)
In this case the character device driver would just need to enable the hardware access (set the chip select pins, if necessary), and map the memory to the userspace.
Bit-banging, or other serial/parallel connection
In this case the situation is a bit more complicated.
1. Providing a memory map
  The memory map resides in normal RAM, and is only periodically (partially) flushed to the FPGA. This can be either kernel-assisted, so that only one page is used, and whenever the userspace accesses a different page, the changes to the old page are flushed first, then the mapping changed. The overhead is significant on most architectures.
  Certain embedded display devices like e-ink which support partial screen refreshes use a similar approach, but with shared memory maps in userspace.
2. Accessing the device as a file, using lseek()/read()/write() or pread()/pwrite
  Like files, the character device has a "position". lseek()/read()/write() set and modify this position. pread()/pwrite() specify the position as a parameter, ignoring the device position.
  The device driver uses this position, the index into the "file", as the address relative to the start of the FPGA. Since each data byte is copied to/from user space, the device driver can bit-bang or transmit each byte quite easily.
  From the application point of view, the FPGA no longer looks like memory, but a file; this may be a bit confusing to some developers. It is, however, often more efficient than playing with memory maps, unless the hardware can handle all the details (the initial case above).

If you can tell exactly which hardware you have, I might be a bit more specific. (Unless, of course, it is custom or in-house hardware, not publicly accessible; in that case I obviously cannot help you any further.)

mashhur · 01-28-2012, 03:42 AM

thanks guys for helping, im gonna try what you mentioned above and share the result.
Thanks again!

mashhur · 01-29-2012, 07:35 PM

Quote:

Originally Posted by Nominal Animal

If the memory is accessible to the kernel as normal RAM, then this program should work:

Code:

#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <errno.h>

#include <string.h>
#include <stdio.h>

/*
 * API:
*/

/* Read bytes of physical memory starting at address into destination.
*/
int mem_read(const unsigned long address, void *const destination, const size_t bytes);

/* Write bytes from source to physical memory starting at address.
*/
int mem_write(const unsigned long address, const void *const source, const size_t bytes);



/*
 * Internal implementation:
*/

static int mem_device = -1;

static int mem_open(void)
{
    int result;

    if (mem_device != -1)
        return errno = EBADF;

    do {
        result = open("/dev/mem", O_RDWR);
    } while (result == -1 && errno == EINTR);
    if (result == -1)
        return errno;

    mem_device = result;
    return 0;
}

int mem_read(const unsigned long address, void *const destination, const size_t bytes)
{
    unsigned char   *d = (unsigned char *)destination;
    off_t            o = (off_t)address;
    size_t           n = bytes;
    ssize_t          r;

    while (n) {

        r = pread(mem_device, d, n, o);
        if (r == (ssize_t)n) {
            return 0;

        } else
        if (r >= (ssize_t)0) {
            d += r;
            n -= r;
            o += r;

        } else
        if (r != (ssize_t)-1) {
            return errno = EIO;

        } else
        if (errno == EBADF) {
            if (mem_open())
                return errno;

        } else
        if (errno != EINTR && errno != EAGAIN && errno != EWOULDBLOCK)
            return errno;
    }

    return 0;
}

int mem_write(const unsigned long address, const void *const source, const size_t bytes)
{
    unsigned char   *s = (unsigned char *)source;
    off_t            o = (off_t)address;
    size_t           n = bytes;
    ssize_t          w;

    while (n) {

        w = pread(mem_device, s, n, o);
        if (w == (ssize_t)n) {
            return 0;

        } else
        if (w >= (ssize_t)0) {
            s += w;
            n -= w;
            o += w;

        } else
        if (w != (ssize_t)-1) {
            return errno = EIO;

        } else
        if (errno == EBADF) {
            if (mem_open())
                return errno;

        } else
        if (errno != EINTR && errno != EAGAIN && errno != EWOULDBLOCK)
            return errno;
    }

    return 0;
}

/*
 * Example main().
*/

int main(void)
{
    unsigned long   addr = 0xE0300000UL;
    unsigned short  word = 0xFFFFU;

    if (mem_write(addr, &word, sizeof(word)) {
        fprintf(stderr, "Cannot write %d bytes to physical memory address 0x%08lX: %s.\n",
                        (int)sizeof(word), addr, strerror(errno));
        return 1;
    }

    if (mem_read(addr, &word, sizeof word)) {
        fprintf(stderr, "Cannot read %d bytes from physical memory address 0x%08lX: %s.\n",
                        (int)sizeof(word), addr, strerror(errno));
        return 2;
    }

    printf("Physical memory address 0x%08lX = 0x%04X (%d bytes).\n",
           addr, (int)word, (int)sizeof(word));
    return 0;
}

Save it as e.g. 0xe0300000.c and compile and run it using e.g.

Code:

gcc 0xe0300000.c -Wall -O3 -o 0xe0300000
sudo ./0xe0300000

Because it accesses /dev/mem, it requires superuser rights (root).

The above will not work for you, if the FPGA is not already memory-mapped by the kernel and/or hardware.

In that case, you need a simple character device driver you can access from userspace. I have no idea how the EBI bus is handled by your hardware, it might be on a separate bus chip connected via USB for all I know. There are two possibilities, however:

Directly hardware accessible by the CPU (memory-mapped)
In this case the character device driver would just need to enable the hardware access (set the chip select pins, if necessary), and map the memory to the userspace.
Bit-banging, or other serial/parallel connection
In this case the situation is a bit more complicated.
1. Providing a memory map
  The memory map resides in normal RAM, and is only periodically (partially) flushed to the FPGA. This can be either kernel-assisted, so that only one page is used, and whenever the userspace accesses a different page, the changes to the old page are flushed first, then the mapping changed. The overhead is significant on most architectures.
  Certain embedded display devices like e-ink which support partial screen refreshes use a similar approach, but with shared memory maps in userspace.
2. Accessing the device as a file, using lseek()/read()/write() or pread()/pwrite
  Like files, the character device has a "position". lseek()/read()/write() set and modify this position. pread()/pwrite() specify the position as a parameter, ignoring the device position.
  The device driver uses this position, the index into the "file", as the address relative to the start of the FPGA. Since each data byte is copied to/from user space, the device driver can bit-bang or transmit each byte quite easily.
  From the application point of view, the FPGA no longer looks like memory, but a file; this may be a bit confusing to some developers. It is, however, often more efficient than playing with memory maps, unless the hardware can handle all the details (the initial case above).

If you can tell exactly which hardware you have, I might be a bit more specific. (Unless, of course, it is custom or in-house hardware, not publicly accessible; in that case I obviously cannot help you any further.)

Hi again,

Thanks to you Nominal Animal (ur name is cool

) to post.
Ur code is really awesome working fine on NAND Flash& SDRAM and others. However, when I applied EBI bus address it cannot read or write.
Could pls share me ur ideas about this situation.
Im thinking the problems wud be setting, i mean i should set chip select and other read enable or write enable parameters. Second, kernel does not see the EBI memory (or EBI bus address). As u said it would be memory mapping problem (how to map before read/write?).
Im using S5PC100 cpu and FPGA is Spartan-6 (I guess FPGA is not big deal) and kernel is 2.6.28.6. The board designed by my coworker.

I appreciate ur helps guys. Thanks.

Nominal Animal · 01-29-2012, 11:55 PM

The S5PC100 is an ARM Cortex A8 processor made by Samsung. Knowing that makes the situation a lot clearer. The link paulsm4 pointed to in post #4 contains the outline of a solution; it is very similar to yours.

Quote:

Originally Posted by mashhur

you Nominal Animal (ur name is cool

I dunno about cool, but it does describe me well.

Quote:

Originally Posted by mashhur

working fine on NAND Flash& SDRAM and others. However, when I applied EBI bus address it cannot read or write.

You do need a simple character device driver. For background, look at Linux Device Drivers, 3rd edition.

I suspect all the low-level stuff is already done in the s5p100 platform driver in the kernel, so all your driver needs to do is to request_mem_region() it when opened, release_mem_region() when closed, ioremap_nocache() in the read() and write() hooks so you can copy_to_user() and copy_from_user() from the region ( iounmap()'ing before returning to the userspace), and io_remap_pfn_range() when mmapped() (this mapping is released when the region is released).

If you pick only read()/write() or mmap(), and not both, the driver will be even simpler.

Here is an example of how simple a mmap'ing driver is. In your case, you would also need .open and .close handlers to request_mem_region() and release_mem_region(), but that shouldn't amount to more than a dozen lines of code. I suspect the platform code has already all the necessary stuff wired in, so unless you need to fiddle with the EBI interface, that is really about it.

You might wish to look at the carma FPGA driver in the Linux kernel. It is much more complex than what you might need, but it might be useful to at least look at a relatively similar fully-featured driver already in the kernel. In particular, you might consider if there are parts like protecting certain areas on the FPGA from modification, that should be done on the kernel driver instead.

mashhur · 02-01-2012, 06:06 PM

Quote:

Originally Posted by Nominal Animal

The S5PC100 is an ARM Cortex A8 processor made by Samsung. Knowing that makes the situation a lot clearer. The link paulsm4 pointed to in post #4 contains the outline of a solution; it is very similar to yours.

I dunno about cool, but it does describe me well.

You do need a simple character device driver. For background, look at Linux Device Drivers, 3rd edition.

I suspect all the low-level stuff is already done in the s5p100 platform driver in the kernel, so all your driver needs to do is to request_mem_region() it when opened, release_mem_region() when closed, ioremap_nocache() in the read() and write() hooks so you can copy_to_user() and copy_from_user() from the region ( iounmap()'ing before returning to the userspace), and io_remap_pfn_range() when mmapped() (this mapping is released when the region is released).

If you pick only read()/write() or mmap(), and not both, the driver will be even simpler.

Here is an example of how simple a mmap'ing driver is. In your case, you would also need .open and .close handlers to request_mem_region() and release_mem_region(), but that shouldn't amount to more than a dozen lines of code. I suspect the platform code has already all the necessary stuff wired in, so unless you need to fiddle with the EBI interface, that is really about it.

You might wish to look at the carma FPGA driver in the Linux kernel. It is much more complex than what you might need, but it might be useful to at least look at a relatively similar fully-featured driver already in the kernel. In particular, you might consider if there are parts like protecting certain areas on the FPGA from modification, that should be done on the kernel driver instead.

good day guys again

im really pissed off cuz of my system to get worked.
Nominal Animal, you are exactly right on this problem and im doing as you guide.
Now another problem im experiencing that S5PC100's NAND Flash memory (which file system loaded) is READ-ONLY i guess.
I wrote a simple device driver, and tried to put it in kernel. when I create a driver (by typing 'mknod 770 /dev/dev_driver_name c major_num minor_num') in /dev/ folder it gives me "mknod failed: Read-only file system" error. ('insmod module' works fine)
So, logically, there is another way to implement it that i should manually register driver to kernel before uploading it to the nand flash. But, somehow i might be wrong, i mean the problem would be on other side and wanna share with you guys.

im wondering if any one could help me please.
any help would be highly appreciated.thanks.

Nominal Animal · 02-01-2012, 06:33 PM

Most likely the filesystem is just mounted read-only. See if mount -o remount,rw / as root helps.

Mounting the filesystem read-only is a very good idea, because it avoid writing to the filesystem. Normally, metadata changes -- most often file access timestamps -- generate a surprising amount of writes, causing wear on the flash. Mounting the filesystem read-only avoids that. After updates, you can do mount -o remount,ro / to remount it read-only again.

The mount command will list you the filesystems you have mounted. Some of them, like /proc, /sys and /dev/pts are special filesystems: actually, kernel interfaces. The third field in the mount output tells you where the filesystem is mounted, and the things in parentheses at the end of the line tells you the options. If it contains ro, then that filesystem is mounted read-only, and you can try the command I listed above. (In many embedded devices there is really just the root filesystem, and the special filesystems, so the remount command above should work. In other cases, the filesystems are separated to allow splitting them on multiple devices etc. You will need to tell the mount command the location of the mount to remount it; a directory within the mount will not work.)

mashhur · 02-02-2012, 07:57 AM

Quote:

Originally Posted by Nominal Animal

Most likely the filesystem is just mounted read-only. See if mount -o remount,rw / as root helps.

The mount command will list you the filesystems you have mounted. Some of them, like /proc, /sys and /dev/pts are special filesystems: actually, kernel interfaces. The third field in the mount output tells you where the filesystem is mounted, and the things in parentheses at the end of the line tells you the options. If it contains ro, then that filesystem is mounted read-only, and you can try the command I listed above. (In many embedded devices there is really just the root filesystem, and the special filesystems, so the remount command above should work. In other cases, the filesystems are separated to allow splitting them on multiple devices etc. You will need to tell the mount command the location of the mount to remount it; a directory within the mount will not work.)

hi Nominal Animal,
i did as u explained above but still no result. i mean still the file system is read-only.
u r true that there is the root filesystem.
here is my log:

[root@Samsung]# mount -o remount,rw /dev/root
[root@Samsung]# mount
rootfs on / type rootfs (rw)
/dev/root on / type cramfs (ro)
ramfs on /etc type ramfs (rw)
tmpfs on /tmp type tmpfs (rw)
/proc on /proc type proc (rw)
/proc/bus/usb on /proc/bus/usb type usbfs (rw)
/sys on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,mode=600)

don't know what to do... desperated...

and, also i have written a simple loadable kernel module and put it into kernel/device/ directory with its .ko & Makefile files. made a zImage file and uploaded to board, still no result, where is the module, is it loading(lsmod doesn't show) i don't have any idea.

If u have any ideas let me know pls, thanks.

Nominal Animal · 02-02-2012, 06:07 PM

Your actual root is a cramfs provided by device /dev/root; see the mount output. Try mount -o remount,rw /dev/root
(I should have mentioned this initially, because this is very typical, very usual, on embedded devices. Sorry.)

As to the module, add the name (without the .ko) to /etc/modules and it should be loaded at boot time.

Don't be desperate: there are just a lot of options, and we only need to find out a good, working one. For example, it is possible that adding the module name to /etc/modules is not enough, you might have to add it to the initial ramdisk (I believe you use). That is not difficult at all, but exactly how it is done, depends on the way you build the system image.

mashhur · 02-03-2012, 11:13 PM

Quote:

Originally Posted by Nominal Animal

Your actual root is a cramfs provided by device /dev/root; see the mount output. Try mount -o remount,rw /dev/root
(I should have mentioned this initially, because this is very typical, very usual, on embedded devices. Sorry.)

As to the module, add the name (without the .ko) to /etc/modules and it should be loaded at boot time.

For example, it is possible that adding the module name to /etc/modules is not enough, you might have to add it to the initial ramdisk (I believe you use). That is not difficult at all, but exactly how it is done, depends on the way you build the system image.

hi Nominal Animal,
I have a good news today. suddenly, in my brain the idea came out "why i don't use rw file systems" and inserted module (LKM) on tmpfs (/tmp/module) and as u explained i did request to memory in order to map and witnessed the oscilloscope shows me chip select signal when i address to FPGA memory.
A lot thanks to you to support me. the rest of problems regarding to linux I/O or others i will solve.
and finally, i realized the problem was not with any hardware or software complexity but was with me, i mean it is the first time im handling such h/w and s/w together, no any experience on embedded

. however, im in an optimistic view that im gonna implement it

.
take care~~