I'm not entirely sure how to uninstall if you are compiling from source, you might try:
However, the PHP authors may not have provided an uninstall version.
It may be enough to simply download the source code of PHP 5.4 into a different directory and try the same configure/make/make install commands there.
On the other hand, compiling from source is sometimes tricky if you don't know what configure command to use. If you are using linux, you can use a package manager to auto-install such things.
On a debian-based system, you could do something like this:
apt-get install php5 php5-xmlrpc php5-gd php5-mysql
If you're on a red hat based system (e.g., Fedora or CentOS) then the command would be yum
instead of apt-get and the package names would be different.