Anytime anything (job, car, girlfriend, boyfriend, soundcard, you name it) becomes more trouble than it's worth, it's time to move on. Life is short - why put up with a lot of constant stress and hassles.
Obviously if you are using any of the other functions on that card like the modem you would lose that by tossing the card but of course you can easily buy a replacement for that as well. Naturally this approach would depend on your budget and also the number of available PCI slots you've got. Good luck with it -- J.W.