Stuck in loop and unable to find answer

blason · 03-27-2023, 01:56 PM

Hi Team,

I need to write a code where I have certain urls entered in file. This I am taking as a loop and wanted to download those files and keep it separate with name. Once downloaded I can then concatenate those. Below are my codes but dang I am not able to store them in different file.

PHP Code:



cat $URL_FILE | while read line; do 
 curl_cli -o /tmp/file --user $_USER:$_PASS -s --cacert $DIR/conf/ca-bundle.crt --retry 10 --retry-delay 60 $line | dos2unix | grep -vE '^$' 
       done

So in this example my $URL_FILE contains

http://download.example.com/file1.txt
http://download.example.com/file2.txt
http://download.example.com/file3.txt
http://download.example.com/file4.txt

And in that loop I want to keep output those files in separate file; so that at the end of loop I can then

cat file1.txt file2.txt file3.txt file4.txt | dos2unix | sort -n | uniq > /tmp/finalfile

However I am not able to achieve it; can someone please help?

TIA
Blason R

business_kid · 03-27-2023, 02:44 PM

Quote:

Originally Posted by blason

However I am not able to achieve it;

Hi, not a coder, but that bit gets no replies, because coders need to see exactly what went wrong so they can solve it.
Do better, look at the errors you get, and try stuff.

michaelk · 03-27-2023, 03:12 PM

Code:

-o /tmp/file

With just a quick look your writing to file but then wanting to pipe the output?
Something like:

Code:

while read -r line; do
  base=$(basename $line)
  curl ... | dos2unix | grep -vE '^$' > /tmp/$base
done < "$URL_FILE"
cat /tmp/*.txt | sort -n | uniq > /tmp/finalfile

ntubski · 03-27-2023, 05:00 PM

https://manned.org/curl.1

Code:

       -O, --remote-name
              Write output to a local file named like the remote file we get.
              (Only the file part of the remote file is used, the path is cut
              off.)

Alternatively, drop the -o entirely and put the pipe after the loop.

michaelk · 03-27-2023, 05:30 PM

To elaborate on ntubski's post you do not need to save them separately.

Code:

while read -r line
do
  curl ... $line
done < "$URL_FILE" | dos2unix | grep -vE '^$' | sort -n | uniq > "/tmp/finalfile"

MadeInGermany · 03-29-2023, 08:30 AM

And

Code:

grep -vE '^$'

can become

Code:

grep .

Consider having each $variable in quotes: "$line"

blason · 04-01-2023, 10:40 PM

Thanks folks that resolved my issue,