I hope maybe some knowledgeable Linux USB device driver expert or programmer can help me with this; I guess I can wish for David Brownell's help but that's wishing too much
Anyways, here's my situation. I'm doing embedded Linux programming on an OMAP3 (ARM Cortex 8) development board called the BeagleBoard. The board is using a USB gadget (peripheral) driver for network communication; the module is g_ether.ko. For my project, I need to modify that driver (specifically the driver file ether.c) to be able to handle some "vendor specific" traffic on Control Endpoint 0 (ep0). Normally, control endpoints only process standard USB requests (USB_TYPE_STANDARD); for my project, I need to modify it to also process vendor-specific USB request (USB_TYPE_VENDOR). I was able to track the processing in (drivers/usb/gadget) down to the composite.c module and all endpoint request goes through the composite_setup function, which currently only support USB standard requests (such as get USB descriptors, set USB configuration, set USB interface, etc). I was able to modify composite.c to call a vendor callback function I added to ether.c if the type is USB_TYPE_VENDOR.
In the vendor callback function I added to ether.c, I was able to send the USB request data up to a userspace application via a character driver interface. However, a "GET" request requires this vendor callback function to "wait" for the userspace app to send the response down to it via the character driver. It seems to work when I sent a vendor specific USB control message from the host to the target BeagleBoard; the Get request made it through the g_ether driver to the vendor specific callback function and up to the userspace application. However, this is where the driver crashes (waiting for the GET request response data so that it can send it back to the host).
I know I'm violating all kinds of rules but please forgive me as I'm new to Linux kernel programming. I tried searching the web for help or driver examples on how to implement this but I can't find any. I tried to use the libusb API but that is just for "host" programming, not "peripheral" programming. Can someone please provide guidance, advice, links, examples, etc on how I can properly implement this vendor specific control endpoint processing, which will forward the vendor-specific requests from the g_ether driver up to a userspace application and then "waits" for the response data from it so that it can return the data back to the host on the other end?