Issue with Bash script

Mo_ · 12-24-2022, 03:35 PM

Hi, community,
I'm having some weird issues with the script. I'm not sure it's the right place to post it.
We have a script in the server to pull data from elastic search and save it in server. There are three types of files so we have to run the script 3 times. There can be 3,4 million files or more. Two types of files are in larger quantities and one is less. For example, the first file type takes 8 hours, the second one 3 hours, and the third one 6 hours. This is how I run the script manually:

Code:

./script.sh admit 20221201 20221207 01

01 is the elastic search node. That script is fine and runs it manually every week three times for each file type.
I created another script to run that script and I scheduled it from crontab to run every 15 minutes on weekends. The second script run and checks if no script is running then it checks the first directory and sees if it's empty. If it's empty then run to export that file type. If the first directory is not empties then check the second so on. Once all are exported then it won't do anything.

The issue I'm noticing is that it's missing some files from the very first file type. When comparing it with elasticSearch files. I can see a couple of thousands of files have not been copied. I changed the order to copy the second files first but having the same issue. It only happens when it runs the very first time.
If remove the files from that directory and it runs again then it runs properly. I don't have an idea what is the issue. Any can help and see if something wrong with the script? Thank you for your help!

Code:

#!/bin/bash
date=$(date)
startDate=$(date -d "7 day ago" '+%Y%m%d')
endDate=$(date -d "1 day ago" '+%Y%m%d')
log="/opt/reports/weekly/export.log"
msg1="/opt/reports/weekly/output/Admit"
msg2="/opt/reports/weekly/output/Discharge"
msg3="/opt/reports/weekly/output/Transfer"
path="/opt/reports/weekly/weeklyscript.sh"
script=$(ps aux | grep "/bin/bash /opt/reports/weekly/weeklyscript.sh" | grep -v grep)
echo $date >> $log
if [ "$script" == "" ]
then
echo "No script running" >> $log
if [ -z "$(ls -A -- "$msg1")" ]
then
echo "Admit directory is empty and starting admit script" >> $log
$path admit $startDate $endDate 01 &
elif [ -z "$(ls -A -- "$msg2")" ]
then
echo "Discharge directory is empty and discharge script started" >> $log
$path discharge $startDate $endDate 01 &
elif [ -z "$(ls -A -- "$msg3")" ]
then
echo "Transfer directory is empty and transfer script is started" >> $log
$path tranfer $startDate $endDate 01 &
else
echo "Directory is not empty, either files already exported or a script is in progress" >> $log
fi
else
echo "Script is still running, nothing to do at this time" >> $log
fi

MadeInGermany · 12-25-2022, 03:47 AM

Your ps grep (why not pgrep?) detects any of the pull scripts.
And the if-elif chain starts just one type.
So I expect that only the fist empty one in the if-elif chain is running.

What is your goal?
Always have one script of each type running?

Mo_ · 12-25-2022, 02:38 PM

There are three types of files in the database. We want to export one file type at a time and copy it to that file type's directory. So each file type goes to its own directory. Let me explain the process which I run each week. I run "./weeklyscript.sh admit 20221201 20221207 01" and monitor it, once it's completed then I run "./weeklyscript.sh discharge 20221201 20221207 01", once the second one is completed then I run the last one by typing "./weeklyscript.sh transfer 20221201 20221207 01". admit files get saved in "/opt/reports/weekly/output/Admit" and discharge files get saved in "/opt/reports/weekly/output/Discharge" and transfer files in "/opt/reports/weekly/output/Transfer". Since these are types of files and use for different purposes.
My goal is to automate the process so that I don't have to run the script manually all the time and I can schedule it to run on weekends when there isn't much activity. Initially, it checks if the script is already running then it won't run the script again. Once the script copy all admit types of files and stop. When no script running then it checks the admit directory, if files are already copied there then it no run the script for admit and moves to the discharge directory if there aren't files then it runs the script for that and does the same for transfer files. Once all directories have files then it won't run the script.

Just want to clarify that there are two scripts the first one copies the files. I don't touch the script. The second one I created to run that script instead of me running it all the time. Hope my explanation make sense.

I never used pgrep but I'll test it.

Thank you for your help.

MadeInGermany · 12-25-2022, 05:24 PM

Consider

Code:

pid=$( pgrep -f "/bin/bash $path )

and

Code:

if [ -z "$pid" ]

According to your description you want to run all 3 scripts at each invocation, one after the other.
Then it is
fi; if
instead of
elif
And no & back-grounding.

Mo_ · 12-26-2022, 02:02 PM

I'm not sure I understood properly. I made changes according to my understanding. I ran the script and it started three processes for all three files type. I want to export one file type at a time. For example, the logic I'm looking for is to copy A type of file and copy to A directory. Once A files are copied then run the script again and copy B files and the same with C files. I don't want to start three scripts since we don't have that many resources and it causes issues.

Quote:

#!/bin/bash
date=$(date)
startDate=$(date -d "7 day ago" '+%Y%m%d')
endDate=$(date -d "1 day ago" '+%Y%m%d')
log="/opt/reports/weekly/export.log"
msg1="/opt/reports/weekly/output/Admit"
msg2="/opt/reports/weekly/output/Discharge"
msg3="/opt/reports/weekly/output/Transfer"
path="/opt/reports/weekly/weeklyscript.sh"
pid=$( pgrep -f "/bin/bash $path" )
echo $date >> $log
if [ -z "$pid" ]
then
echo "No script running" >> $log
if [ -z "$(ls -A -- "$msg1")" ]
then
echo "Admit directory is empty and starting admit script" >> $log
$path admit $startDate $endDate 01 &
fi;

if [ -z "$(ls -A -- "$msg2")" ]
then
echo "Discharge directory is empty and discharge script started" >> $log
$path discharge $startDate $endDate 01 &
fi;

if [ -z "$(ls -A -- "$msg3")" ]
then
echo "Transfer directory is empty and transfer script is started" >> $log
$path tranfer $startDate $endDate 01 &
else
echo "Directory is not empty, either files already exported or a script is in progress" >> $log
fi;

else
echo "Script is still running, nothing to do at this time" >> $log
fi;

michaelk · 12-26-2022, 03:47 PM

Why not just run your script as :

Code:

#!/bin/bash
date=$(date)
startDate=$(date -d "7 day ago" '+%Y%m%d')
endDate=$(date -d "1 day ago" '+%Y%m%d')
log="/opt/reports/weekly/export.log"
msg1="/opt/reports/weekly/output/Admit"
msg2="/opt/reports/weekly/output/Discharge"
msg3="/opt/reports/weekly/output/Transfer"
path="/opt/reports/weekly/weeklyscript.sh"
echo $date >> $log
$path admit $startDate $endDate 01
$path discharge $startDate $endDate 01
$path tranfer $startDate $endDate 01

Without the & the script runs serially i.e. discharge will run after admit completes and so on.

Mo_ · 12-26-2022, 06:26 PM

Quote:

Originally Posted by michaelk

Why not just run your script as :

Code:

#!/bin/bash
date=$(date)
startDate=$(date -d "7 day ago" '+%Y%m%d')
endDate=$(date -d "1 day ago" '+%Y%m%d')
log="/opt/reports/weekly/export.log"
msg1="/opt/reports/weekly/output/Admit"
msg2="/opt/reports/weekly/output/Discharge"
msg3="/opt/reports/weekly/output/Transfer"
path="/opt/reports/weekly/weeklyscript.sh"
echo $date >> $log
$path admit $startDate $endDate 01
$path discharge $startDate $endDate 01
$path tranfer $startDate $endDate 01

Without the & the script runs serially i.e. discharge will run after admit completes and so on.

It starts with three processes one for each file type. Running three processes at the same time taking more resources and taking more time to complete the export. The goal is run start one and wait till first gets complete then start the second one and so on. However in your logic it starts all three in the at the same time.

michaelk · 12-26-2022, 07:05 PM

No, my script does not run all three at the same time. Only if there were &s at the end which would run them in the background.

Mo_ · 12-26-2022, 07:25 PM

Quote:

Originally Posted by michaelk

No, my script does not run all three at the same time. Only if there were &s at the end which would run them in the background.

Yes, without including the &, it works. Just curious without &, the process won't be killed after a while? There can be 5 million files for one file type. I was looking at old logs when I ran it manually and one of the file types took about 8 hours to complete. I am afraid that the process stops after a while if I won't use screen and &. If & and screen session is not needed then I can use as you mentioned.

michaelk · 12-26-2022, 07:33 PM

If you close the terminal window it will stop the process. However, if you run it from cron the process isn't attached to a terminal window and basically runs in the background anyway so it doesn't matter( but still only one type at a time).

Mo_ · 12-26-2022, 08:00 PM

Perfect, I'll update the script and schedule it this weekend and see how it goes. Thank you so much.

MadeInGermany · 12-27-2022, 01:39 AM

You can still have the if conditions.

Code:

#!/bin/bash
date=$(date)
startDate=$(date -d "7 day ago" '+%Y%m%d')
endDate=$(date -d "1 day ago" '+%Y%m%d')
log="/opt/reports/weekly/export.log"
msg1="/opt/reports/weekly/output/Admit"
msg2="/opt/reports/weekly/output/Discharge"
msg3="/opt/reports/weekly/output/Transfer"
path="/opt/reports/weekly/weeklyscript.sh"
pids=$( pgrep -f "/bin/bash $path" )
{
# stdout redirected to $log
echo $date
if [ -z "$pids" ]
then
  echo "No script running"
  if [ -z "$(ls -A -- "$msg1")" ]
  then
    echo "Admit directory is empty and starting admit script"
    $path admit $startDate $endDate 01
  fi
  if [ -z "$(ls -A -- "$msg2")" ]
  then
    echo "Discharge directory is empty and discharge script started"
    $path discharge $startDate $endDate 01
  fi
  if [ -z "$(ls -A -- "$msg3")" ]
  then
    echo "Transfer directory is empty and transfer script is started"
    $path tranfer $startDate $endDate 01
  fi
else
  echo "Script is still running, nothing to do at this time" 
fi
} >> $log

michaelk · 12-27-2022, 09:30 AM

One of the possible problems with running hte script multiple times is if the date changes. The OP posted that one of the file types may take more then 8 hours to complete. It depends on how the cron job is configured to run.

Mo_ · 12-28-2022, 01:32 PM

I'll try both. The actual issue was with my script that when it ran and it never copied all the files for the first file type. There were a couple of thousands were missing out of 5 million or so. However, it didn't have issues with other files type. When it completed the first one and started the second one if I deleted files from the first one and when it ran the second time it copied all the files. I know there isn't an issue with the main script because I have been running it for months manually with no issues. When I wrote the second script to run the first script then I'm noticing missing files when running for the very first time. I'll test it one of the methods this weekend and see how it goes and I'll update you here if there'll be an issue. Thank you both.

Mo_ · 01-13-2023, 10:09 PM

I tried both scripts both are working fine. I added an extra step to get notified whenever the script runs successfully or fails. It runs every Saturday when I'm not working, instead of checking it manually I thought it is better to have an email notification. The only issue is if it fails it sends me a fail notification but does not send me the error. I would like to have an error in the email. Is that possible to get an error in the email?
Here is my working code without error.

Quote:

email="test@mail.com"
if [ $? -eq 0 ]
then
echo -e "Script started successfully" | mail -s "Script ran successfully" $email
else
echo "Script failed, there must be an issue" | mail -s "Script failed to run" $email
fi

This is not complete code, it's an example. How can include the error, for example, invalid date, directory exists, etc? I don't what variable that error gets to save or if there is another way to do it.
Thank you for your help.