I've been setting up a bluetooth PAN between my phone and my desktop and while everything seems to work, the lantency on the connection is sometimes as high as 2000ms! This didnt seem right, so I tried the same setup inside a Windows XP VM I had, with windows bluetooth drivers, and the latency was only a fraction (~400ms) of what it was in lunux. Throughput was similar, only the latency was different.

What could be causing such a big difference in times?

I use a USB Bluetooth adapter with the btusb driver in 2.6.29-gentoo-r1 x64

lsusb for bluetooth adapter:
Bus 004 Device 005: ID 0a12:0001 Cambridge Silicon Radio, Ltd Bluetooth Dongle (HCI mode)