Mmapping Not Honored By User Space

jackfrye11 · 02-22-2018, 02:22 PM

I apologize for the length of this post, but I have been trying to debug this for weeks and I have found all of these details necessary to understanding the problem. Also, if this is too advanced for the Newbie forum, please inform me so I can move it to a more appropriate location.

I have a platform device that creates two character devices of the same class. One is supposed to be used as a transmit DMA channel and the other as a receive DMA channel. The problem is that in my user space application, when I try to mmap into the second character device I generated, corresponding to one of the channels, data is not mapped properly. I will change the mapped data in the userspace, and nothing will happen to the corresponding kernel data when IOCTL is called.

Here is the flow of set up.
1. Create data structures corresponding to the DMA Channels, which contains a struct dma_proxy_interface that will be used to map into the userspace to set length of tx/rx buffer and data sent/received.

Code:

struct dma_proxy_channel {
	struct dma_proxy_channel_interface *interface_p;	/* user to kernel space interface */
	dma_addr_t interface_phys_addr;			
	
	struct device *proxy_device_p;				/* character device support */
	struct device *dma_device_p;
	dev_t dev_node;
	struct cdev cdev;
	struct class *class_p;

	struct dma_chan *channel_p;				/* dma support */
	struct completion cmp;
	dma_cookie_t cookie;
	dma_addr_t dma_handle;
	u32 direction;						/* DMA_MEM_TO_DEV or DMA_DEV_TO_MEM */
};
#define TEST_SIZE 1024

struct dma_proxy_channel_interface {
        unsigned char buffer[TEST_SIZE];
        enum proxy_status { PROXY_NO_ERROR = 0, PROXY_BUSY = 1, PROXY_TIMEOUT = 2, PROXY_ERROR = 3 } status;
        unsigned int length;
        bool ready;
};

Create two of these channels, each will correspond to the send or receive dma channel and have its own character device.

Code:

static struct dma_proxy_channel tx_proxy_channel;
static struct dma_proxy_channel rx_proxy_channel;

2. Use the platform driver probe, which is mapped to DMA-related data in the device tree to create the DMAEngine channels

Code:

int rfx_axidmatest_driver_probe(struct platform_device *pdev)
{

	create_channel(&rx_proxy_channel, "_rx", DMA_DEV_TO_MEM, pdev);	
	create_channel(&tx_proxy_channel, "_tx", DMA_MEM_TO_DEV, pdev);


}

3. Create the character device class corresponding to a DMA Channel

Code:

static int cdevice_init(struct dma_proxy_channel *pchannel_p, char *name)
{

	int rc;
	char device_name[32] = "dma_proxy";
	static struct class *local_class_p = NULL;

	/* Allocate a character device from the kernel for this 
	 * driver
	 */ 
	rc = alloc_chrdev_region(&pchannel_p->dev_node, 0, 1, "dma_proxy");

	if (rc) {
		printk(KERN_INFO "unable to get a char device number\n");
		return rc;
	}

	/* Initialize the ter device data structure before
	 * registering the character device with the kernel
	 */ 
	printk("pchannel_p->cdev addr: %p\n", (void*)&pchannel_p->cdev);
	cdev_init(&pchannel_p->cdev, &dm_fops);
	pchannel_p->cdev.owner = THIS_MODULE;
	rc = cdev_add(&pchannel_p->cdev, pchannel_p->dev_node, 1);

	if (rc) {
		printk(KERN_INFO "unable to add char device\n");
		goto init_error1;
	}

	/* Only one class in sysfs is to be created for multiple channels,
 	 * create the device in sysfs which will allow the device node 
	 * in /dev to be created 
	 */
	if (!local_class_p) {
		local_class_p = class_create(THIS_MODULE, DRIVER_NAME);
		
		if (IS_ERR(pchannel_p->dma_device_p->class)) {
			printk(KERN_INFO, "unable to create class\n");
			rc = ERROR;
			goto init_error2;
		}	
	}

	pchannel_p->class_p = local_class_p;

	/* Create the device node in /dev so the device is accessible 
	 * as a character device
	 */
 	strcat(device_name, name);
	pchannel_p->proxy_device_p = device_create(pchannel_p->class_p, NULL,
							pchannel_p->dev_node, NULL, device_name);

	if (IS_ERR(pchannel_p->proxy_device_p)) {
		printk(KERN_INFO "unable to create the device\n");
		goto init_error3;
	}

	return 0;

init_error3:
	class_destroy(pchannel_p->class_p);

init_error2:
	cdev_del(&pchannel_p->cdev);

init_error1:
	unregister_chrdev_region(pchannel_p->dev_node, 1);
	return rc;
}

/* Create a DMA channel by getting a DMA channel from the DMA Engine and then setting
 * up the channel as a character device to allow user space control.
 */
static int create_channel(struct dma_proxy_channel *pchannel_p, 
			char *name, 
			u32 direction,
			struct platform_device *pdev)
{
	int rc;
	
	if(name == "_tx")
	{
		pchannel_p->channel_p = dma_request_slave_channel(&pdev->dev, "axidma0");
		if (!pchannel_p->channel_p) {
			printk(KERN_INFO "DMA channel request error\n");
			return ERROR;
		}	
	}
	else
	{
		pchannel_p->channel_p = dma_request_slave_channel(&pdev->dev, "axidma1");
		if (!pchannel_p->channel_p) {
			printk(KERN_INFO "DMA channel request error\n");
			return ERROR;
		}		
	}

	/* Request the DMA channel from the DMA engine and then use the device from
 	 * the channel for the proxy channel also.
 	 */	
	pchannel_p->dma_device_p = &pchannel_p->channel_p->dev->device;

	/* Initialize the character device for the dma proxy channel 
 	 */
	rc = cdevice_init(pchannel_p, name);
	if (rc) {
		printk(KERN_INFO "Device init failed");
		return rc;
	}

	pchannel_p->direction = direction;

	/* Allocate memory for the proxy channel interface for the channel as either
 	 * cached or non-cache depending on input parameter. Use the managed 
 	 * device memory when possible but right now there's a bug that's not understood
 	 * when using devm_kzalloc rather than kzalloc, so stay with kzalloc.
 	 */
	pchannel_p->interface_p = (struct dma_proxy_channel_interface *)
				kzalloc(sizeof(struct dma_proxy_channel_interface),  
	                     	GFP_KERNEL);
	printk(KERN_INFO "Allocating cached memory at 0x%08X\n", 
				(unsigned int)pchannel_p->interface_p);
	if (!pchannel_p->interface_p) {
		printk(KERN_INFO "DMA allocation error\n");
		return ERROR;
	}
	return 0;
}

Here is the flow of each character device and the userspace application.
1. From userspace, open each device, the send and receive channel. MMap the data into a corresponding struct dma_proxy_channel_interface that will be used to change the kernel data from the user space. Run some quick tests using the function provded underneath to peek/poke at mmapped data in the kernel.

Code:

int main(int argc, char *argv[])
{	
	int i;
	int dummy;
	pthread_t tid;


	/* Step 1, open the DMA proxy device for the transmit and receive channels with
 	 * read/write permissions
 	 */ 	

	tx_proxy_fd = open("/dev/dma_proxy_tx", O_RDWR);

	if (tx_proxy_fd < 1) {
		printf("Unable to open DMA proxy device file\n");
		return -1;
	}

	rx_proxy_fd = open("/dev/dma_proxy_rx", O_RDWR);
	if (rx_proxy_fd < 1) {
		printf("Unable to open DMA proxy device file\n");
		return -1;
	}

	/* Step 2, map the transmit and receive channels memory into user space so it's accessible
 	 */
	rx_proxy_interface_p = (struct dma_proxy_channel_interface *)mmap(NULL, sizeof(struct dma_proxy_channel_interface),
									PROT_READ | PROT_WRITE, MAP_SHARED, rx_proxy_fd, 0);



	tx_proxy_interface_p = (struct dma_proxy_channel_interface *)mmap(NULL, sizeof(struct dma_proxy_channel_interface), 
									PROT_READ | PROT_WRITE, MAP_SHARED, tx_proxy_fd, 0);


    	if ((rx_proxy_interface_p == MAP_FAILED) || (tx_proxy_interface_p == MAP_FAILED)) {
        	printf("Failed to mmap\n");
        	return -1;
    	}

	tx_proxy_interface_p->buffer[0] = 0x0;
	
	
	//rx_proxy_interface_p->length = TEST_SIZE;	
	/*for (i = 0; i < TEST_SIZE; i++) {
		rx_proxy_interface_p->buffer[i] = 0;
	}*/
	printf("RX Transfer Test\n");	
	debug_kernel_channel(rx_proxy_interface_p, rx_proxy_fd);

	/* Create the thread for the transmit processing and then wait a second so the printf output is not 
 	 * intermingled with the receive processing
	 */
	printf("TX Transfer Test\n");
	debug_kernel_channel(tx_proxy_interface_p, tx_proxy_fd);
//	pthread_create(&tid, NULL, tx_thread, NULL);	

	ioctl(rx_proxy_fd, 0, &dummy);
	ioctl(tx_proxy_fd, 0, &dummy);	

	if (rx_proxy_interface_p->status != PROXY_NO_ERROR)
		printf("Proxy rx transfer error\n");

	/* Verify the data recieved matchs what was sent (tx is looped back to tx)
 	 */
	for (i = 0; i < TEST_SIZE; i++) {
//		printf("tx: %d\trx: %d\n", tx_proxy_interface_p->buffer[i], rx_proxy_interface_p->buffer[i]);
   	}

	/* Unmap the proxy channel interface memory and close the device files before leaving
	 */
	munmap(tx_proxy_interface_p, sizeof(struct dma_proxy_channel_interface));
	munmap(rx_proxy_interface_p, sizeof(struct dma_proxy_channel_interface));

	while(!test_done)
	{

	}

	close(tx_proxy_fd);
	close(rx_proxy_fd);
	return 0;
}

Function to peek/poke at data in the kernel

Code:

void debug_kernel_channel(struct dma_proxy_channel_interface *channel, int fd)
{
	
	char a[60];
	printf("Initial kernel values\n");
	usleep(100000);
 	read(fd, a, 50);

	printf("Before SEG FAULT???\n");
	usleep(100000);
	printf("addr of channel %p\n", (void*)channel);
	usleep(100000);
	printf("length addr: %p\n", (void*)&(channel->length));	
	usleep(100000);
	channel->length = TEST_SIZE;
	printf("Length set");
	usleep(100000);
	int i;	
	for (i = 0; i < TEST_SIZE; i++) {
		channel->buffer[i] = 0;
	}
	printf("Buffer set\n");
	usleep(100000);
	channel->buffer[0] = 0x20;
	printf("buffer[0] set\n");
	usleep(100000);
	channel->ready = true;
	printf("ready set\n");

	printf("user space values\n");
	printf("length: %d\n", channel->length);
	printf("buffer size: %d\n", sizeof(channel->buffer));
	printf("buffer[0]: %d\n", channel->buffer[0]);
	msync(channel, sizeof(*channel), MS_SYNC);
	usleep(3000000);
	printf("New Kernel Values\n");
	usleep(100000);
	read(fd, a, 50);
	printf("\n\n");
	//test_done = 1;

}

Back to the driver. Here are the actual file operations for the character device.

Open sets the characters device's private data to be the dma_proxy_channel corresponding to the struct dma_proxy_channel that corresponds to that device (transmit or receive).

Code:

static int local_open(struct inode *ino, struct file *file)
{

        file->private_data = container_of(ino->i_cdev, struct dma_proxy_channel, cdev);
	printk(KERN_INFO "ino->i_cdev addr: %p\n", (void*)ino->i_cdev);

	return 0;
}

Mmap takes the address of the file private_data, that has previously been set to the value of the corresponding channel structure in open() and maps it to a corresponding virtual memory area in the user space.

Code:

static int mmap(struct file *file_p, struct vm_area_struct *vma)
{
	printk(KERN_INFO "mmap\n");

        struct dma_proxy_channel *pchannel_p = (struct dma_proxy_channel *)file_p->private_data;
	printk("Poking buffer[0] manually from mmap\n");
	pchannel_p->interface_p->buffer[0] = 0xA0;

	/* The virtual address to map into is good, but the page frame will not be good since
 	 * user space passes a physical address of 0, so get the physical address of the buffer
 	 * that was allocated and convert to a page frame number.
 	 */

	printk(KERN_INFO "PAGE SHIFT: %d\n", PAGE_SHIFT);	
	if (remap_pfn_range(vma, vma->vm_start, 
				virt_to_phys((void *)pchannel_p->interface_p)>>PAGE_SHIFT, 
			    	vma->vm_end - vma->vm_start, vma->vm_page_prot)) {
		printk(KERN_INFO "remap1\n");
		return -EAGAIN;
	}

	return 0;
		
}

ioctl will actually start the DMA (which I have disabled since the values of the proxy_channel_interface are not being set properly, which are needed to feed the DMAEngine.

Code:

static long ioctl(struct file *file, unsigned int unused , unsigned long arg)
{
	printk(KERN_INFO "ioctl kernel space\n");	

        struct dma_proxy_channel *pchannel_p = (struct dma_proxy_channel *)file->private_data;
	printk(KERN_INFO "Peeking buffer[0] in ioctl %x\n", pchannel_p->interface_p->buffer[0]);

//	transfer(pchannel_p); 

	return 0;
}

Lastly, I implemented a read function which I have used simply for debug purposes

Code:

ssize_t read(struct file *filp, char *buf, size_t bytes, loff_t *off)
{
	printk("Kernel Space Read\n");
        struct dma_proxy_channel *pchannel_p = (struct dma_proxy_channel *)filp->private_data;
        printk(KERN_INFO "length in read: %d\n", pchannel_p->interface_p->length);
	printk(KERN_INFO "buffer size: %d\n", sizeof(pchannel_p->interface_p->buffer));
	printk(KERN_INFO "buffer[0]: %x\n", pchannel_p->interface_p->buffer[0]);
	printk(KERN_INFO "tx buffer[0]: %x\n", tx_proxy_channel.interface_p->buffer[0]);
	printk(KERN_INFO "rx buffer[0]: %x\n", rx_proxy_channel.interface_p->buffer[0]);

	return 0;
}

Here is the output of the userspace application, which contains print from both the application and the driver.

RX Transfer Test
Initial kernel values
Kernel Space Read
length in read: 1024
buffer size: 1024
buffer[0]: 0
tx buffer[0]: 0
rx buffer[0]: 0
Before SEG FAULT???
addr of channel 0xb6f73000
length addr: 0xb6f73404
Length setBuffer set
buffer[0] set
ready set
user space values
length: 1024
buffer size: 1024
buffer[0]: 32
New Kernel Values
Kernel Space Read
length in read: 1024
buffer size: 1024
buffer[0]: 20
tx buffer[0]: 0
rx buffer[0]: 20

TX Transfer Test
Initial kernel values
Kernel Space Read
length in read: 0
buffer size: 1024
buffer[0]: 0
tx buffer[0]: 0
rx buffer[0]: 20
Before SEG FAULT???
addr of channel 0xb6f71000
length addr: 0xb6f71404
Length setBuffer set
buffer[0] set
ready set
user space values
length: 1024
buffer size: 1024
buffer[0]: 32
New Kernel Values
Kernel Space Read
length in read: 0
buffer size: 1024
buffer[0]: 0
tx buffer[0]: 0
rx buffer[0]: 20

Note the following:
1. In the probe, the rx channel character device is initialized first. The output suggests that mmap is working only for this data. The length of the tx channel is not being changed
2. When the order of character device initialization is reversed, values of the rx channel in the kernel cannot be changed but values of the tx channel can.

I have been trying to debug this for weeks. I thought the character device API was supposed to be simple. What is going on here? Is there something about caching I missed?

rtmistler · 02-23-2018, 07:10 AM

I have no experience with mmap, however do realize it is supposed to be for DMA.

My experiences with deep, long standing problems that befuddle me is to get fundamental.

Test a fundamental nmap() call, such as the most basic example
Expand that to cover the varieties of customizations you are adding to it, one by one
Eventually you will find what you are doing wrong and likely it will not be related to a misuse of nmap() but instead an interpretation of what is really being done by that, or another system call along the way
I also add TONS of debug, inline logs or printf/printk calls to remark "got here", "and here", etc
Next I also pay 200% attention to EVERY system call, the outcome, ERRNO, and print all of that. Many is the time that I've coded far along, all sorts of assumptions, "after all, I've used open() only 10,000 times or more!", only to find that exactly that was my problem and it was the hardware resource, the file type, or whatever, that treated it differently in some very unpredicted, subtle, manner.

Moving this to the Programming forum to gain it some better exposure.

I do realize that you feel you need to explain it all, however I do feel you may benefit by trying the fundamental example and modifying up from there. After all, nmap() does work, so something you are doing is causing your problem.

jackfrye11 · 02-23-2018, 08:57 AM

Thanks for the response. You are the first one from a number of forums to comment on this issue.

Note that currently this should not be a DMA issue. All DMA functionality is disabled. Even the threading capability that is supposed to sync the DMA channels is disabled. I am simply trying to create two mmap-able structures in the kernel and map to them from the userspace, then poke them to ensure the mmaps are holding and the data is synced between user space and kernel space. Even that is not working at the moment. This is especially confusing because this should be fairly basic. The only DMA call that is made is to get the channels in the create_channel call with dma_request_slave_channel(). I cannot see how this would explain the issue I am having though.

NevemTeve · 02-23-2018, 12:50 PM

My first question: do you wish to mmap a character device? Is that supposed to work?

jackfrye11 · 02-23-2018, 01:35 PM

"do you wish to mmap a character device?"
No I am not mmapping to a character device.

If you look at the struct dma_proxy_channel, it has members cdev and struct dma_proxy_channel_interface.
I am registering the device as a device node firstly.

1. When the device is opened, the private data of the device is set to the dma_proxy_channel that is the container(struct containing) of the cdev, which is the i_cdev property of the file struct which is being referred to from the userspace open function. Based on the way the file was registered, its i_cdev property is the cdev child of the struct dma_proxy_channel instance.

In cdevice_init()

Code:

cdev_init(&pchannel_p->cdev, &dm_fops);
pchannel_p->cdev.owner = THIS_MODULE;
rc = cdev_add(&pchannel_p->cdev, pchannel_p->dev_node, 1);

2. In the mmap function, the vma area struct is mapped to the physical location of the dma_proxy_channel_interface property of the file->private data that was set to be the dma_proxy_channel back in the open function. A virtual pointer to this memory is returned to the userspace.

3. Lastly, in the IOCtl function, this dma_proxy_channel_interface property is fetched from file->private_data which has been updated from the userspace based on the mapping in 2. This data is (going to be, it is commented out now), passed to the DMAEngine for DMA transfer but that is a step ahead.

"Is that supposed to work?"
Yes.

astrogeek · 02-23-2018, 01:46 PM

I am not mmap proficient, and have not digested your example code, but I think NevemTeve has touched the right nerve - that you can't do that with a character device. That just sounds right and a few quick searches and thumbing through my Linux Device Drivers print copy seems to confirm that.

*** EDIT - we were typing at the same time, ignore if not applicable to your case.

jackfrye11 · 02-23-2018, 01:48 PM

So you are sort of just saying that I should bag the whole idea, regardless of whether or not I can get the mapping of memory to work since DMA makes everything a nightmare from the userspace?

Having reread your comment, are you saying that private_data cannot be remapped to another memory area? I am confused.

What exactly cant I not do with a character device that I am doing? astrogeek or NevemTree, please explain to me. I am still a real newbie at this kind of development. I will also look at ORielly again for reference.

It appears what I am doing is actually working, but just for one of the character devices.

Xilinx seems to think it can be done, but this is somewhat outdated.
http://forums.xilinx.com/xlnx/attach...ace-public.pdf

astrogeek · 02-23-2018, 03:35 PM

Quote:

Originally Posted by jackfrye11

So you are sort of just saying that I should bag the whole idea, regardless of whether or not I can get the mapping of memory to work since DMA makes everything a nightmare from the userspace?

No, not at all - I did not mean to discourage you! I only meant that within the limited perspective I gave, mmaping a character device appeared problematic, and investigating that view might open a new path out of your impasse.

It still looks problematic to me, but the xilinx PDF you linked is interesting - page/slide 12-17 in particular look applicable to the case you have described.

jackfrye11 · 02-23-2018, 04:20 PM

On the DMA-Proxy device driver side, I find it slightly disturbing that no one in the Xilinx world has done since a much older version of the kernel. I am confident it has been done and can be done.

On the general kernel side which is more applicable to this thread, isn't mmap a common operation to implement in the file_operations of a character driver? The mmap/IOCtl combo is how /dev/mem works, which is a HUGELY important device for embedded Linux folks.