Hi.
figuring out RAM usage is seldom an easy and straightforward task. Having memory maxed at 98% is suspicious although it doesn't have to mean something's wrong. Depends how is the memory subsystem tuned.
My first approach would probably be running topas and noting:
1. how much memory is computational and how much non-computational
2. is paging space used / is the system paging at the moments (check the figures for Steals and PgspIn / PgspOut)
Then try vmstat -v. There you can find how many pages of certain types are there in the memory, what are their limits, etc...
Another somewhat useful command is svmon. Running svmon -P should give you figures of used memory per process (note the output can be pretty long). Don't forget the memory is shared. Simply adding numbers of used pages process by process will not give you anything relevant.
Perhaps you have a persistent metrics recording enabled, try running topasrec -l, this should show you where are the recordings located. If you'd like to know more about this, try checking man pages of topasout and google for nmon analyzer.
If you have a Comfort Line maintenance agreement with IBM, open a call and request perfpmr analysis.
There is a great page on topic Overview of AIX page replacement here
http://www.ibm.com/developerworks/aix/library/au-vmm/
