I have an HPC cluster, I recently had to reboot one of the submit node VMs. Now the submit VM is no longer mounted to the head node. The other submit VMs are mounted fine.

When I try to mount I get mount: mount to NFS server "server ip" failed: timed out (retrying)

I restarted NFS on the head node, and still did not have any luck. Any ideas would be great

Just stepping back a little - can you confirm a few things?

1) Can you definitely still ping the NFS server from your client on the network you're trying to mount across?

2) Assuming you are running NFS on standard ports - can you try:
telnet <server IP> 2049

Does this let you connect or does this refuse you? If it refuses you then perhaps there is a firewall on the server (or even client) that is preventing access.

3) Can you test mounting a share from another NFS server?

4) What happens if you unmount the NFS share on one of the other clients, can they re-mount it ok?

Hope this helps.

I can ssh and ping to the server.

I had this issue on another environment a long time ago...

seems to me I had to do something with /var/lib/nfs rmtab

But can you connect to the system using port 2049?