LinuxQuestions.org - [SOLVED] write a bash script

Page 1 of 2

Show 50 post(s) from this thread on one page

- Linux - General (https://www.linuxquestions.org/questions/linux-general-1/)

- - write a bash script (https://www.linuxquestions.org/questions/linux-general-1/write-a-bash-script-4175660773/)

Pedroski

09-12-2019 03:49 AM

write a bash script

I would like to make a bash script. I know a little about Python, but I have no experience with bash scripts.

I have a homework webpage. The students fill in the gaps or choose the right word. I managed to make a php file which saves their answers in a folder on my webhost. Works well.

I can get the files from the webhost with ssh. Also works well.

Step 1:

Before I get the files I need to move any previously existing files from

/home/pedro/winter2019-20/18BE/studiAnswers/

to

/home/pedro/winter2019-20/18BE/oldstudiAnswers/

Step 2:

Then run:

sftp -i ~/.ssh/godaddy5_rsa -b ~/.ssh/goDaddy.sftpHW.batch myusername@123.456.789.123

Step 3:

Maybe print "All done!" at the end.

I have not yet mastered using ssh in Python and I think that is unnecessary. I think a straight up bash script would do.

If I can add the path of the script to $PATHS, I could just call it any time.

Any tips for this amateur, please?

Turbocapitalist

09-12-2019 04:22 AM

You could probably start with Part 4 of The Linux Command Line. If you are really starting out from the beginning, buying a paper copy might be useful since you need to have a good familiarity with at least some system utilities before delving into automating them with a script.

There are also:

Greg's Bash Pitfalls
Greg's Bash FAQ
Greg's Bash Guide

Bash is the default interactive shell on many systems and nearly all GNU/Linux systems. However there are others, like Zsh and Ksh. There is also plain POSIX sh. Most of the principles for Bash will apply to the others, just watch out for bash-only extensions. They can be useful but need to be used with awareness.

michaelk

09-12-2019 05:34 AM

In addition you can run remote commands via ssh

ssh -i my_key user@host "remote_command"

berndbausch

09-12-2019 06:22 AM

Step 1 is a simple mv command:

Code:

mv /home/pedro/winter2019-20/18BE/studiAnswers/* /home/pedro/winter2019-20/18BE/oldstudiAnswers/

For step 2, I would use scp. sftp is designed for interactive usage. I don't know what's in the batch file, though. On the other hand, if your sftp command works, just use it verbatim.

Quote:

Maybe print "All done!" at the end.

Do you really need help with that?

Turbocapitalist

09-12-2019 06:51 AM

Quote:

Originally Posted by berndbausch (Post 6035988)

For step 2, I would use scp. sftp is designed for interactive usage. I don't know what's in the batch file, though. On the other hand, if your sftp command works, just use it verbatim.

SFTP would work well in batch mode, as is proposed by the OP, so that would be the way to do it. Though maybe rysnc would be another viable option.

Either way, scp is considered deprecated as of OpenSSH 8.0 and should be avoided, especially in new endeavors.

Pedroski

09-12-2019 06:14 PM

Thanks!

The reason I want this to work is just: up to now I use php to send the students' answers to an email, then use python to get the emails, check the answers and put the results in my timetable. I think this will be a bit quicker, that's all. And it is an alternative to email.

My batch file is below. It works, to my amazement!

Quote:

lcd /home/pedro/winter2019-20/18BE/studiAnswers/
cd public_html/18BE/php/uploads/
pwd
ls -l
mget *
rm *
ls
exit

2 questions if you have time:

1. This is what I have written. Does it look OK?
2. I see many scripts have an ending .sh Is that necessary? I called it getHW.sh, put it in /home/pedro/myBashscripts/

Quote:

#! /bin/bash
echo "This script should move some files then get the latest files from the webpage."
mv /home/pedro/winter2019-20/18BE/studiAnswers/* /home/pedro/winter2019-20/18BE/oldstudiAnswers/
sftp -i ~/.ssh/godaddy5_rsa -b ~/.ssh/goDaddy.sftpHW.batch myusername@123.456.789.123
echo "All done!"

Firerat

09-12-2019 06:50 PM

well, the OS and bash doesn't really care what a script ends with
But

it tells us humans it is a shell script
it will tell bash-completion is is a shell script

use [code] instead of [quote]

Quote:

Code:

lcd /home/pedro/winter2019-20/18BE/studiAnswers/

cd public_html/18BE/php/uploads/

pwd

ls -l

mget *

rm * # I would be worried about this

# what if the mget failed for some reason?

ls

exit

I would have to check if sftp can do some logic to work out if files had been copied successfully
if it were like bash

Code:

mget * || echo error && rm *

but I have no idea if sftp can do that

regards the script that moves to old, does it matter if it overwrites files?
would it make sense to tar up a batch and archive instead of leaving loose ?

Edit:

I didn't dig deep, but found this in sftp manpage

Quote:

A batchfile of ‘-’ may be used to indicate standard input. sftp will abort if any of the following commands fail:
get, put, reget, reput, rename, ln, rm, mkdir, chdir, ls, lchdir, chmod, chown, chgrp, lpwd, df,
symlink, and lmkdir.

I assume sftp would exit non-zero
so the trick would be to grab stdout ( or just stderr? ) work out what went wrong and act accordingly
it might be that you get a nice exit code

Code:

sftp ...... 

sftpexitcode=$?

echo $sftpexitcode

or you could just send stderr to a log

Code:

sftp .... 2>stderr.log > stdout.log

and grep them later

Pedroski

09-13-2019 12:12 AM

Thank for the suggestions.

I will find a workaround.

Good idea to hold up on the rm * until I've marked all the answers and see who "forgot", check if there is a file which was not downloaded, then delete. I have about 150 students.

My python program only gets unread messages. Only after I have marked everything and checked do I run another little program to mark all the emails as seen, but, of course, the emails are still there.

Maybe I can make a little php program to copy all the files to another directory on the server, then run my bash script. I just don't want this weeks HW getting mixed up with next weeks HW.

The files are all just tiny text files, they don't take up much space. I only use about 5% of the webspace I have.

Just had an idea: Back them up immediately!

This is the bit of the php which writes the text files, I just added a second path and file. Each file is unique, because they are marked with the student number and week number like 1825010132_18BEwW4 I just doubled up!

Code:

$path = "/home/myusername/public_html/18BE/php/uploads/";

$path2 = "/home/myusername/public_html/18BE/php/uploads/oldfiles/";

$fp = fopen($path . $newname, 'w');

$fp2 = fopen($path2 . $newname, 'w');

fwrite($fp, $body);

fwrite($fp2, $body);

fclose($fp);

echo ' Your data has been saved in a text file!  <br> ';

Turbocapitalist

09-13-2019 12:34 AM

Quote:

Originally Posted by Pedroski (Post 6036302)

I will find a workaround.

Good idea to hold up on the rm * until I've marked all the answers and see who "forgot", check if there is a file which was not downloaded, then delete. I have about 150 students.

A safer way to do that would be to use rsync instead. It has the option to --remove-source-files once they have been successfully transferred. Be sure to read the manual page about that and the other options.

Code:

$path1="/home/myusername/public_html/18BE/php/uploads/";

$path2="/home/myusername/public_html/18BE/php/uploads/oldfiles/";

$server="123.123.123.123"



ssh -i ~/.ssh/godaddy5_rsa username@$server "rsync -av --remove-source-files $path1 $path2"



rsync -av -e 'ssh -i ~/.ssh/godaddy5_rsa' "username@123.123.123.123/$path2" /some/local/directory/

That will move the files more or less safely from one directory to another and then download them. Again, be sure to read the manual page about rsync the options you are using.

Pedroski

09-13-2019 05:07 PM

Thanks again, I will definitely look into rsync. Maybe I can get it to work, I am not too good with this stuff!

Do these 2 need to go together, or are they completely separate commands?

Quote:

ssh -i ~/.ssh/godaddy5_rsa username@$server "rsync -av --remove-source-files $path1 $path2"

Quote:

rsync -av -e 'ssh -i ~/.ssh/godaddy5_rsa' "username@123.123.123.123/$path2" /some/local/directory/

I think the second one will synchronise a folder on the host with a folder on my local computer. That would be great, because there are always a few students who "forget". A javascript disables the send button, so if they are late, they can't send. Sometimes I let them do it the next day by changing the disable time. Then getting just the files I do not have would be good!

I would like to see the --progress. Is this right?

Quote:

rsync -av -e 'ssh -i --progress ~/.ssh/godaddy5_rsa' "username@123.123.123.123/$path2" /some/local/directory/

Firerat

09-13-2019 06:31 PM

Quote:

Code:

ssh -i ~/.ssh/godaddy5_rsa username@$server "rsync -av --remove-source-files $path1 $path2"

that is connecting via ssh and running the command on the remote

same as

Code:

ssh -i ~/.ssh/godaddy5_rsa username@$server "mv -av $path1 $path2"

Quote:

I would like to see the --progress. Is this right?

Code:

rsync -av -e 'ssh -i --progress ~/.ssh/godaddy5_rsa' "username@123.123.123.123/$path2" /some/local/directory/

no the -e is the remote shell
this is what you want

Code:

rsync -av --progress -e 'ssh -i ~/.ssh/godaddy5_rsa' "username@123.123.123.123/$path2" /some/local/directory/

I think there is some confusion with local and remote

If I understand it correctly you

want to archive oldanswers on the local machine
retrive newanswers from remote
remove newanswers from remote

Untested

Code:

#!/bin/bash

RemotePath="public_html/18BE/php/uploads"



Answers="/home/pedro/winter2019-20/18BE/studiAnswers"

ArchiveDir="/home/pedro/winter2019-20/18BE/oldstudiAnswers"

ArchiveFile="${ArchiveDir}/${date +%F}_studiAnswers.txz"





#TODO test we have files in Answers and ?

# clear them?

# ask what to do with them ?

# exit with error message ?



# /!\ It has been a while since I used rsync,

# I may have this wrong 



rsync \

  -av \

  --progress \

  --remove-source-files \

  -e 'ssh -i ~/.ssh/godaddy5_rsa' \

  username@123.123.123.123:${RemotePath}/ "${Answers}"



#TODO add logic to test archive pre-existing



tar cfJ "${ArchiveFile}" ${Answers}/*



#TODO test archive

I'm not sure how rsync would behave if I protect spaces in RemotePath
but I assume it wouldn't have any ( that is a lazy assumption , I should test rsync to find out how to protect spaces properly )

I don't know what your workflow is, here I archived them straight away
leaving originals alone
They would need to be move/removed before next script run

Pedroski

09-13-2019 08:53 PM

Thank you all for your advice. I am nearly done!

I've been trying to make rsync work. I seem to have got the moving files on the server bit working.

Quote:

1. ssh -i ~/.ssh/godaddy5_rsa $user@$server "rsync -av --remove-source-files $path1 $path2"

and this gets the files to my local machine:

Quote:

2. rsync -av --progress --remove-source-files -e 'ssh -i ~/.ssh/godaddy5_rsa' $user@$server:${RemotePath}/ "${Answers}"

I'm just wondering how to not call ssh 2 times?

How to build the second line into the first line?? I've tried, but then it fails.

Firerat

09-13-2019 09:37 PM

Well,

(1) -- is moving the files on the remote server from one folder to another
(2) -- is fetching files from the remove server, and then removing them

I don't see any need for (1)

Firerat

09-13-2019 09:49 PM

Tip with ssh

~/.ssh/config

Code:

Host exampleservername.com

#        HostName ExampleServerName

        HostName exampleservername.com

        User ExampleUser

        IdentityFile ~/.ssh/godaddy5_rsa

edit: fix HostName not being a *real* address
Host can be an alias

Code:

#instead of 

rsync -av --progress --remove-source-files -e 'ssh -i ~/.ssh/godaddy5_rsa' $user@$server:${RemotePath}/ "${Answers}" 

# you can have

rsync -av --progress --remove-source-files -e ssh $server:${RemotePath}/ "${Answers}"

Turbocapitalist

09-13-2019 11:18 PM

Code:

Host exampleservername.com

        HostName ExampleServerName

        User ExampleUser

        IdentityFile ~/.ssh/godaddy5_rsa

        IdentitiesOnly yes

Using the client's configuration file is one of the best tips there is. I'd only add that IdentitiesOnly is good to use once you start having more that one or two keys at a time in the agent.

Edit2: fixed misformatted excerpt

All times are GMT -5. The time now is 08:51 PM.

Page 1 of 2

Show 50 post(s) from this thread on one page