the easiest way to do it is to reinstall with japanese selected as one of the languages but I guess you wouldn't want to do it.
As you are running mandrake, go to mandrake control center ->software management ->install. Search for wnn and install the two things that show up (freewnn and kinput2-something). After this, go to mcc->system->drakxservices and make sure jserver is checked to start at boot. This should basically allow you to use japanese on your system.
Reboot. Add a new user, and set the locale for that user to japan and all should be fine. If you don't want a new user, but just the ability to input japanese (you should be display japanese w/o problems), copy the /etc/sysconfig/i18n over to ~/.i18n and then make the following changes:
(add the last two). This should get you japanese input by pressing shift-space. You can quit it by pressing ctrl-\. If it doesn't work, try making the changes to /etc/sysconfig/i18n (back it up first!) and reboot.