1) are you sure your hardware supports AIX 5.2 ?
2) Instead of hitting F5 during the boot, try someting else :
a) While AIX 4.3 is running, type "bootlist -m normal cd0 hdisk XX"
(where XX is the name of your bootlist, juste type lspv and see where rootvg is).
b) put the 5.2 CD-ROM and type "shutdown -Fr", it should boot directly from the CD.
I guess that your graphic terminal is not supported, maybe you should connect a dumb terminal at the serial1 plug, at least during the first step of the install.
By the way, before doing what I say, you should make a "smit mksysb" tape backup. So you can go back in case of trouble.