Sounds like you may just have a hotplug issue. The easiest way would be to look into where hotplug is started and, well, not have it start

It's not very elegant, but it should do the trick.
You could also play with the hotplug agents and make sure that your phone/USB doesn't initiate any attempts at module/application loading.
If you could include some of your dmesg output when this error happens then someone may be able to offer a more direct solution.