Well I would probably start by talking to a couple of the system manufacturers and see who can meet your needs from a hardware standpoint.. you've listed some pretty beefy specs there, and being the first Linux system you are planning to manage SUPPORT from the manufacturer would be a good idea.
So on to manufacturers.. start with the Obvious
IBM, Dell, HP.
IBM many different platforms, supports RedHat Enterprise Linux and Suse Enterprise Linux
Dell, Pretty much Intel only, supports RedHat Enterprise Linux and Suse Enterprise Linux
HP many different platforms, supports RedHat Enterprise Linux and Suse Enterprise Linux and Debian Linux.
Being a Debian fan I would probably look at the HP route pretty closely. Debian will also run on more varied processor architectures than RedHat or Suse.
But then again looking at your requirements ifort 9.1 supports the following..
# Red Hat Enterprise Linux 3, 4
# SUSE Linux 9.1 Professional
# SUSE Linux Enterprise Server* 9
Which brings you back to RH or SUSE as the preferred OS for your project.
Looking at the supported OS's for the PGI compiler brings us back to Suse and RedHat as well..
64-bit - SuSE SLES 8 SP2, SLES 9/10, SuSE 9.0/9.1/9.2/9.3/10.0/10.1,
RHEL 3.0/4.0, Fedora Core 2/3/4/5 (AMD64, Xeon EM64T)
Since you are looking for a MPP environment you will be purchasing multiple servers with their own memory and storage in each server, to basically build a cluster computing environment.
The sales reps from the hardware manufacturers can be a good resource for you, if you explain your needs, get quotes from several manufacturers to keep them all honest and locate any weaknesses in your proposed design. The Sales reps have Sales Engineers at their beck and call, that will design the system per your specs, so don't be afraid to make use of those resources.
You may find out that IBM or HP provided the hardware for the model you are viewing, so they may be more aware of what you need than you are..