[SOLVED] BASH - script to loop through files in size order descending

GPGAgent · 04-16-2023, 10:07 AM

I can write a simple for loop to loop through files in name order.

Code:

 for ff in *.mp4; do HandBrakeCLI -i "$ff" -o "X$ff";done

This works fine with a command in the do section to process the $ff file

But I want the list in size order descending so I did this

Code:

for ff in "$( ls *.mp4 | sort -r )"; do echo "$ff"; done

which results in this

Code:

$ for ff in "$( ls *.mp4 | sort -r )"; do echo "$ff"; done;
X BIG.mp4
CMID.mp4
BMID.mp4
ASMALL.mp4

Fine I thought, sorted in size order escending, just need to put my command in the do section

Code:

for ff in "$( ls *.mp4 | sort -r )"; do HandBrakeCLI -i "$ff" -o "X$ff"; done

But I was wrong, there is a /n hidden in $ff whcih lets it display but not much use sending the whole list of files to HandBrakeCLI in one hit.
The result

Code:

$ for ff in "$( ls *.mp4 | sort -r )"; do HandBrakeCLI -i "$ff" -o "X$ff"; done;
[16:03:52] hb_init: starting libhb thread
[16:03:52] thread 7fbf823a0700 started ("libhb")
HandBrake 1.3.1 (2020032300) - Linux x86_64 - https://handbrake.fr
16 CPUs detected
Opening X BIG.mp4
CMID.mp4
BMID.mp4
ASMALL.mp4...
[16:03:52] CPU: Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz
[16:03:52]  - logical processor count: 16
[16:03:52] hb_scan: path=X BIG.mp4
CMID.mp4
BMID.mp4
ASMALL.mp4, title_index=1
disc.c:323: failed opening UDF image X BIG.mp4
CMID.mp4
BMID.mp4
ASMALL.mp4
disc.c:424: error opening file BDMV/index.bdmv
disc.c:424: error opening file BDMV/BACKUP/index.bdmv
....
..
......

How do I seperate the list into individual movies to be processed squentially?

Note: Some files have spaces in their names

GPGAgent · 04-16-2023, 10:39 AM

I've tried something else, I've made a file of the movie names in descending size order, simple enough

Code:

$ ls *.mp4 | sort -r > x.x
$ cat x.x
X BIG.mp4
CMID.mp4
BMID.mp4
ASMALL.mp4
$

So now just read that file with a while loop

Code:

$ while read ff; do echo $ff; done < x.x
X BIG.mp4
CMID.mp4
BMID.mp4
ASMALL.mp4
$

And that works fine, so now put the HandBrakeCLI command in

Code:

$ while read ff; do HandBrakeCLI -i "$ff" -o "X$ff" --stop-at seconds:20; done < x.x
[16:36:34] hb_init: starting libhb thread
[16:36:34] thread 7f55a33bd700 started ("libhb")
HandBrake 1.3.1 (2020032300) - Linux x86_64 - https://handbrake.fr
16 CPUs detected
Opening X BIG.mp4...
[16:36:34] CPU: Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz
[16:36:34]  - logical processor count: 16
[16:36:34] hb_scan: path=X BIG.mp4, title_index=1
....
..
x264 [info]: Weighted P-Frames: Y:1.0% UV:0.0%
x264 [info]: ref P L0: 63.7%  7.1% 21.8%  7.3%  0.1%
x264 [info]: ref B L0: 87.1% 10.3%  2.6%
x264 [info]: ref B L1: 97.7%  2.3%
x264 [info]: kb/s:1401.61
[16:36:46] mux: track 0, 600 frames, 3507325 bytes, 1399.20 kbps, fifo 1024
[16:36:46] mux: track 1, 939 frames, 5702 bytes, 2.27 kbps, fifo 1024
[16:36:46] Finished work at: Sun Apr 16 16:36:46 2023

[16:36:46] libhb: work result = 0

Encode done!

HandBrake has exited.
$

But only processes the first line!!!!

I thought I'd cracked it!

elgrandeperro · 04-16-2023, 11:03 AM

My guess is that handbrake is sending signals which is killing the shell.

Try this:

Code:

(while read ff; do HandBrakeCLI -i "$ff" -o "X$ff" --stop-at seconds:20; done)< x.x

GPGAgent · 04-16-2023, 11:09 AM

Quote:

Originally Posted by elgrandeperro

My guess is that handbrake is sending signals which is killing the shell.
Try this:

Code:

(while read ff; do HandBrakeCLI -i "$ff" -o "X$ff" --stop-at seconds:20; done)< x.x

That was my thought, but now your suggestion also stopped after processing the first line.

I know I will kick myself when I work it out or someone else does

MadeInGermany · 04-16-2023, 11:38 AM

Code:

for ff in "$( ls *.mp4 | sort -r )"; do

This does not work because $( ) becomes a string that is protected by the quotes.
Without the quotes the string is subject to word splitting (partly desired) and filename generation (not desired).
The best solution is a while-read loop

Code:

ls *.mp4 | sort -r | while IFS= read -r ff; do

Each read reads one line. IFS="" and -r turn
off some magic in read.
Commands in the loop body that can read from stdin would compete with the read, and should be redirected </dev/null
If HandBrakeCLI is such a command:

Code:

</dev/null HandBrakeCLI ...

Regarding
ls *.mp4 | sort -r
This is not by size but reverse alphabetical, like
ls -r *.mp4

GPGAgent · 04-16-2023, 11:57 AM

Quote:

Originally Posted by MadeInGermany

Code:

for ff in "$( ls *.mp4 | sort -r )"; do

This does not work because $( ) becomes a string that is protected by the quotes.
Without the quotes the string is subject to word splitting (partly desired) and filename generation (not desired).
The best solution is a while-read loop

Code:

ls *.mp4 | sort -r | while IFS= read -r ff; do

Each read reads one line. IFS="" and -r turn
off some magic in read.
Commands in the loop body that can read from stdin would compete with the read, and should be redirected </dev/null
If HandBrakeCLI is such a command:

Code:

</dev/null HandBrakeCLI ...

Regarding
ls *.mp4 | sort -r
This is not by size but reverse alphabetical, like
ls -r *.mp4

You're absolutely correct about it NOT sorted by size, my mistake. Thanks for pointing that out.

Anyway I trid it like this

Code:

ls *.mp4 | sort -r | while IFS= read -r ff; do HandBrakeCLI -i "$ff" -o "X$ff" --stop-at seconds:20 ; done

but yet again it stopped after processing the first file!

MadeInGermany · 04-16-2023, 12:08 PM

Try the </dev/null

Code:

ls *.mp4 | sort -r | while IFS= read -r ff; do </dev/null HandBrakeCLI -i "$ff" -o "X$ff" --stop-at seconds:20 ; done

GPGAgent · 04-16-2023, 12:16 PM

Quote:

Originally Posted by MadeInGermany

Try the </dev/null

Code:

ls *.mp4 | sort -r | while IFS= read -r ff; do </dev/null HandBrakeCLI -i "$ff" -o "X$ff" --stop-at seconds:20 ; done

Cheers that worked, can you explain why the </dev/null kept it in the loop please?

Next step is to feed it a list by size, I think this might do it

Code:

ls *.mp4 -S

Don't forget some of the file names have spaces!

And yes this worked just fine

Code:

ls *.mp4 -S | sort -r | while IFS= read -r ff; do </dev/null HandBrakeCLI -i "$ff" -o "Y$ff" --stop-at seconds:20 ; done

teckk · 04-16-2023, 01:02 PM

Lots of ways of doing that with bash and friends.

Few Examples:

Sort files by size.

Code:

find *.mp4 -type f -exec du -h {} + | sort -rh
find *.mp4 -type f -exec ls -lS {} +

Code:

files1=(*.mp4)
files2=($(du -a <<< "$files1" | sort -nr))

printf '%s\n' "${files2[@]}"

for i in "${files1[@]}"; do
    echo "$i"
    sleep 1
done

for i in "${files2[@]}"; do
    echo ${i##*/}
    sleep 1
done

#First will fail, the rest will be ok
for i in "${files2[@]}"; do
    echo "HandBrakeCLI -i ${i##*/} -o New_${i##*/}"
    sleep 1
done

MadeInGermany · 04-16-2023, 01:57 PM

If ls can sort it, do not re-sort it by piping to sort!
Perhaps you want to reverse the order, then you must tell the ls to do it:

Code:

ls *.mp4 -rS | while ...

Regarding the </dev/null, I tried to explain it.
The loop's stdin is fed by the pipe.
Not only the read can read a line from the stdin.
Obviously the HandBrakeCLI reads from the stdin, too. And "steals" it, so the read doesn't get it.

GPGAgent · 04-16-2023, 03:55 PM

Quote:

Originally Posted by MadeInGermany

If ls can sort it, do not re-sort it by piping to sort!
Perhaps you want to reverse the order, then you must tell the ls to do it:

Code:

ls *.mp4 -rS | while ...

Regarding the </dev/null, I tried to explain it.
The loop's stdin is fed by the pipe.
Not only the read can read a line from the stdin.
Obviously the HandBrakeCLI reads from the stdin, too. And "steals" it, so the read doesn't get it.

Sorry, I did read your original post, but it didn't register in my 72yr old brain!!

GPGAgent · 04-16-2023, 03:55 PM

Thanks all, I've learn't a bit more now, enough to be dangerous!!