[SOLVED] wget question, how to combine -i foo.txt and -O name.jpg

lleb · 07-24-2021, 09:58 AM

I currently use a text file labeled just grab.txt to grab a series of images from the schools site.

The images are just 1.jpg, 2.jpg, etc.... so that makes them easy to grab.

After I run a simple script to rename them to class.date.####.jpg

If there is only a small number of images to grab I'll do them by hand with:

wget URL -O class.date.####.jpg

many days there are between 10 - 1000 images to grab, thus the -i option with the grab.txt file.

Is there a way to combine the -i with -O in one step without having to edit and run the renaming script?

Thank you.

shruggy · 07-24-2021, 10:13 AM

Looks like curl will be a better fit for this.

lleb · 07-24-2021, 10:49 AM

shruggy,

Thank you for the link, i have a question on the output

Code:

curl "URL.[1-100].jpg" -o class.date.[0001-0100].jpg

created a single file named lass.date.[0001-0100].jpg and that contained just the first file. I did see the output from curl grabbing 1-100.jpg , just didnt do as expected.

shruggy · 07-24-2021, 10:52 AM

You should look up description of the -o option in the curl man page. It's

Code:

curl "URL/[1-100].jpg" -o "class.date.#1.jpg"

lleb · 07-24-2021, 11:04 AM

Quote:

Originally Posted by shruggy

You should look up description of the -o option in the curl man page. It's

Code:

curl "URL/[1-100].jpg" -o "class.date.#1.jpg"

Thank you that is better, but instead of class.date.0001.jpg through class.date.1000.jpg or what ever the actual range is for the day, today its 0001-0011, i get outputs with ## instead of leading 0's.

Attempting to replace ###1 with 0001 results in a single output again.

shruggy · 07-24-2021, 11:37 AM

curl doesn't work this way. You cannot zero pad output name variables. You have to rename the files afterwards.

Code:

curl "URL/[1-100].jpg" -o "class.date.#1.jpg"
rename class.date. class.date.000 class.date.?.jpg
rename class.date. class.date.00 class.date.??.jpg
rename class.date. class.date.0 class.date.???.jpg

lleb · 07-24-2021, 11:46 AM

in that case, im back to my original process of wget -i grab.txt and afterwards running my renaming script.

shruggy · 07-24-2021, 11:52 AM

Well, at least, you can

Code:

curl \
-o "class.date.000#1.jpg" "URL/[1-9].jpg" \
-o "class.date.00#1.jpg" "URL/[10-99].jpg" \
-o "class.date.0#1.jpg" "URL/[100-999].jpg"

teckk · 07-24-2021, 11:57 AM

Quote:

Is there a way to combine the -i with -O in one step without having to edit and run the renaming script?

Simple example:

cat testfile.txt

Code:

http://www.a.com/one.jpg
http://www.a.com/two.jpg
http://www.a.com/three.jpg
http://www.a.com/four.jpg
http://www.a.com/five.jpg
http://www.a.com/six.jpg
http://www.a.com/seven.jpg
http://www.a.com/eight.jpg
http://www.a.com/nine.jpg
http://www.a.com/ten.jpg

Code:

mylist=($(while read line; do echo "$line"; done < testfile.txt))
agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0"

num=1
for i in "${mylist[@]}"; do
    echo "wget -U "$agent" "$i" -O class.date."$num".jpg"
    num=$(($num +1))
done

wget -U Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0 http://www.a.com/one.jpg -O class.date.1.jpg
wget -U Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0 http://www.a.com/two.jpg -O class.date.2.jpg
wget -U Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0 http://www.a.com/three.jpg -O class.date.3.jpg
wget -U Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0 http://www.a.com/four.jpg -O class.date.4.jpg
wget -U Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0 http://www.a.com/five.jpg -O class.date.5.jpg
wget -U Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0 http://www.a.com/six.jpg -O class.date.6.jpg
wget -U Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0 http://www.a.com/seven.jpg -O class.date.7.jpg
wget -U Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0 http://www.a.com/eight.jpg -O class.date.8.jpg
wget -U Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0 http://www.a.com/nine.jpg -O class.date.9.jpg
wget -U Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0 http://www.a.com/ten.jpg -O class.date.10.jpg

boughtonp · 07-24-2021, 12:14 PM

Why not simply

Code:

for f in {1..1000}
do
   zf="0000$f"
   echo curl "url.$f.jpg" -o "class.date.${zf: -4}.jpg"
done

?

lleb · 07-24-2021, 12:47 PM

Quote:

Originally Posted by boughtonp

Why not simply

Code:

for f in {1..1000}
do
   zf="0000$f"
   echo curl "url.$f.jpg" -o "class.date.${zf: -4}.jpg"
done

?

Running that with correct URL & 1...#, I get the echo output, but the command does not execute? would it not just be curl instead of echo curl inside the for loop?

edit to update:

So I removed the echo command and it works perfect. Thank you all for both the help and guidance.

boughtonp · 07-24-2021, 03:08 PM

Yep, the echo was so I could confirm I hadn't typed something wrong, but I didn't want to make requests / cleanup files.

Also I'll note that space after the colon is important - without it the substring expansion would instead set a default value if the parameter was unset or null - more details in the docs.

lleb · 07-24-2021, 05:31 PM

this is what i ended with

Code:

#!/bin/bash

# file to grab multiple web images/files and rename in local directory
# July 24, 2021

# f = total number of images to grab
# zf = listed number for output with leading zeros pre-padded in zf="0000$f"
# url = URL to grab files from

#url=https://foo/
#for f in {1..1000}
#do
#   zf="0000${f}"
#   curl "{url}${f}.jpg" -o "class.date.${zf: -4}.jpg"
#done

# copy/paste the URL in url= location
# adjust 1..1000 to match correct # of files to grab
# replace NN. with name of output file

url=
for f in {1..1000}
do
   zf="0000${f}"
   curl "${url}${f}.jpg" -o "NN.${zf: -4}.jpg"
done

Yes I intentionally have the raw code commented out. Helps me remember when I go back after time to read the code and figure out what its doing.