This is handled by the shell you are logged into. That shell probably has a startup script in your home directory which you can edit to your liking. For instance, if you use bash, this file is called .bashrc (so the path is ~/.bashrc). You can add or edit the "PS1" line to read something like
Repeat this for each of the machines you ssh into.