Not a yum user but I think it's "yum remove <pkg>". You might want to check the options enabled for your current php version before doing that, so you can compile the next one using the same exact configuration switches. You can use phpinfo() for example if you are not sure, just put a file called test.php in your server with this:
Code:
<?php
phpinfo();
?>
It shows lots of info, but the most important bit for this purpose is the "configure command", which is on the top.
