Please suggest a workflow for working on a remote machine.

SegFault1 · 07-23-2020, 04:35 AM

My organization has a very powerful machine that I use to run my ML models. It runs on some version of Ubuntu. Due to this ongoing pandemic, I have to work from home. This machine is not public facing and therefore, I VPN into my organization network using OpenFortiVPN. And then I SSH into that machine.

The problem is that my internet is not very good and/because it a mobile network that I tether on my machine (and that's why I get a dynamic IP evertime I am connected to the network). However, that machine at my organization is connected to very good network. The datasets that I work with are hundreds of GBs. I cannot download the dataset and work on my machine. The dataset and computing power reside on the remote machine, not on my machine.

Suggest a workflow for this setup. The problems I face are,

When I SSH into the machine, I only get one terminal. I need many terminal---to edit my code for the model, to run my model, to examine datasets, etc. I don't want to exit my editor every time I want to AWK into my dataset or, run my code. The models take a lot of time to train and run, I don't want to sit and look at my cursor, I'd rather look at my code. I can finally turn to TMUX or Screen, but I want to keep them as a last resort.
My dataset also contains images, and while examining dataset, I sometimes need GNUPLOT. SSHing into the machine doesn't give me a X.
I tried mounting the remote machine using SSHFS to my local machine. That solves some problem, but now my local machine becomes slow due to a slow connection to the remote machine. And viewing images on that mountpoint downloads the entire directory which my network cannot support.
Due to my choppy internet, that key press has a noticeable lag in the SSH session. As someone who doesn't even use a DE (only tiling-WMs or sometimes even without X) to work fast, this lag gets on my nerve.
I don't know why, but if I leave my SSH terminal for too long without activity, the terminal hangs. Maybe due to bad connection, but not sure.

Please don't suggest me Jupyter (or alike) as I like to work in a terminal with my own editor (that has been vimrc-ed to hell) and also, my python scripts are dependencies everywhere that Jupyter doesn't handle well. I need my bare bones python.

Admittedly, I'm new to SSH, and VPNs and networking so, if there is something obvious that I'm missing in my setup, please tell me.

pan64 · 07-23-2020, 04:41 AM

without GUI the best you can do is tmux or screen or something similar.
ssh can be used with -X (or -Y) if you wish to use GUI. But probably not the best solution for you.
You can try VNC to have a local display somewhere and you will only connect to that display (or actually you can try any other remote desktop solution).

We cannot help on the lagging, that should be solved by (or inside) your company and ISP.

Turbocapitalist · 07-23-2020, 05:02 AM

Quote:

Originally Posted by SegFault1

When I SSH into the machine, I only get one terminal. I need many terminal---to edit my code for the model, to run my model, to examine datasets, etc. I don't want to exit my editor every time I want to AWK into my dataset or, run my code. The models take a lot of time to train and run, I don't want to sit and look at my cursor, I'd rather look at my code. I can finally turn to TMUX or Screen, but I want to keep them as a last resort.
My dataset also contains images, and while examining dataset, I sometimes need GNUPLOT. SSHing into the machine doesn't give me a X.

I'm quite a fan of tmux and would encourage getting comfortable with it. With even just a few basic activities scripted or commited to memory, it becomes very, very useful. It can even be launched with pre-defined windows and panes, and programs within those panes, if you need that.

You can also reconnect SSH and tmux automatically if you are authenticating to the remote machine with keys or certificates. Here is an example of reconnecting automatically using a key and from there reconnecting to an existing tmux session named 'models' or create it if it does not already exist.

Code:

ssh-add ~/.ssh/somekey;
while ! ssh -i ~/.ssh/somekey segfault@xx.yy.zz.aa \
    'tmux attach-session -t models -d || tmux new-session -s models'; 
do 
        sleep 2;
done;

You can also put quite a bit into the SSH client's configuration file, ~/.ssh/config, as far as keys and user names go.

Code:

Host model2
        Hostname model.example.com
        User segfault1
        IdentitiesOnly yes
        IdentityFile ~/.ssh/somekey
        ForwardX11 yes
        RemoteCommand tmux a || tmux 

Host *.example.com
        ServerAliveCountMax 3
        ServerAliveInterval 30

That would give you a shortcut 'ssh model2' for those settings.

Code:

while ! ssh model2; 
do
       sleep 2;
done;

See "man ssh_config" for the details.

If you need to run a graphical program on the remote host yet have it displayed locally, throw in -X to the above formula. However, there can be a bit of lag when working interactively that way.

pan64 · 07-23-2020, 05:15 AM

Oh yes, TurboCapitalist already explained, here is a possible solution to the timeout (last point): https://patrickmn.com/aside/how-to-k...-ssh-sessions/

michaelk · 07-23-2020, 05:20 AM

ssh does have an idle time out and will automatically disconnect but looks like it just hangs. You can adjust some settings in the ssh_config file as posted

As posted tmux or screen are your only options with out a GUI. Any remote desktop would likely increase bandwidth needs and increase lagging. Your only solution is to get a better personal network connection if possible.

Turbocapitalist · 07-23-2020, 05:22 AM

(Oops. I am slow in typing. )

Quote:

Originally Posted by pan64

Oh yes, TurboCapitalist already explained, here is a possible solution to the timeout (last point): https://patrickmn.com/aside/how-to-k...-ssh-sessions/

Useful link, as it covers the server side too.

Quote:

Originally Posted by SegFault1

Due to my choppy internet, that key press has a noticeable lag in the SSH session. As someone who doesn't even use a DE (only tiling-WMs or sometimes even without X) to work fast, this lag gets on my nerve.
I don't know why, but if I leave my SSH terminal for too long without activity, the terminal hangs. Maybe due to bad connection, but not sure.

The lag is probably due to the VPN. If it can be removed and the inner machine accessed via an SSH jump host instead, it would be more secure (according to the cipher choices at least) and faster with less latency.

You could then connect to the inner machine via the jump host using the ProxyJump option.

Code:

ssh -J jump.example.com model.example.com

Some call it a bastion host.

As for the connection dropping, that too might be the VPN. If you have the SSH client send both a TCP and SSH heartbeat maybe that will help as mentioned. From the tail end of ~/.ssh/config

Code:

Host *.example.com
        ServerAliveCountMax 3
        ServerAliveInterval 30

Host * 
        TCPKeepAlive yes

SegFault1 · 07-23-2020, 05:24 AM

Quote:

Originally Posted by Turbocapitalist

I'm quite a fan of tmux and would encourage getting comfortable with it. With even just a few basic activities scripted or commited to memory, it becomes very, very useful. It can even be launched with pre-defined windows and panes, and programs within those panes, if you need that.

This is interesting. I've tried tmux before but didn't stick to it since AwesomeWM+tmux felt redundant. And while working with tmux, my AwesomeWM's muscle memory kicks in and instead of moving around in different panes, I do something weird. Also, CTRL+B (like the ESC counterpart of Vi) is a bit weird, I tried to map it to backslash, but it didn't work for some reason. And copying text in tmux is weird since, it copies pane borders and text from other panes. If you can point me to a link that fixes these problems, that'd be great. Thank you.

Turbocapitalist · 07-23-2020, 05:53 AM

Changing the prefix key is easy. Here it is changed to ctrl-backslash, but it could be set to anything else just as well:

Code:

tmux set -g prefix C-\\
tmux unbind-key C-b
tmux bind-key C-\\ send-prefix

The mouse is a more challenging problem since the mouse is on the local machine and the tmux session is on the remote machine. If you connect to the remote machine using X11 forwarding using the -X option and have xclip installed there as well, then the following formula for .tmux.conf works:

Code:

set -g mouse on
bind -n WheelUpPane if-shell -F -t = "#{mouse_any_flag}" "send-keys -M" "if -Ft= '#{pane_in_mode}' 'send-keys -M' 'select-pane -t=; copy-mode -e; send-keys -M'"bind -n WheelDownPane select-pane -t= \; send-keys -M
bind -n C-WheelUpPane select-pane -t= \; copy-mode -e \; send-keys -M
bind -T copy-mode-vi    C-WheelUpPane   send-keys -X halfpage-up
bind -T copy-mode-vi    C-WheelDownPane send-keys -X halfpage-down
bind -T copy-mode-emacs C-WheelUpPane   send-keys -X halfpage-up
bind -T copy-mode-emacs C-WheelDownPane send-keys -X halfpage-down

# To copy, left click and drag to highlight text in yellow,
# once you release left click yellow text will disappear and will automatically be available in clibboard                                                       # # Use vim keybindings in copy mode
setw -g mode-keys vi
# Update default binding of `Enter` to also use copy-pipe
unbind -T copy-mode-vi Enter
bind-key -T copy-mode-vi Enter send-keys -X copy-pipe-and-cancel "xclip -selection c"                                                                           bind-key -T copy-mode-vi MouseDragEnd1Pane send-keys -X copy-pipe-and-cancel "xclip -in -selection clipboard"

from https://unix.stackexchange.com/quest...ouse-with-tmux

Then you can highligh the text, press Enter, and it will be in the local machine's clipboard.

Remember that Enter and Return are not the same.

SegFault1 · 07-23-2020, 06:16 AM

Thank you, Turbocapitalist for taking time to help me and showing around. Guess its time to read man pages.