"my question is if i dont patch it will it make my system slower or anything?"
There is a flaw in the basic design of PCs that there are not enough IRQs. Linux (and any other operating system) fixes the problem by stacking more than one device per IRQ and then sorting out the interrupts as they occur. If your IRQ routing is working then I advise you not to open that can of worms. Any performance enhancements that you might get would be small, if in fact performance did not go the wrong way.
Be prepared. Create a LifeBoat CD.