There is no single answer to that question. Device managment is done in the kernel for some, userspace for others. While the kernel so most of the job detecting devices and loading the correct drive, udev will take care of population the /dev directory, hosting virtual sockets to access IO based devices. Then there is Xorg, aslo managing devices at an higher level, so is FUSE and libusb to some extents. Application themselves can manage devices if the physical port is handled in /dev or libusb. An example would be CUPS, the printer device manager or a serial (COM1/RS232) device terminal.
That list everything you need, you have to figure out the details if you are not done yet.