LinuxQuestions.org - [SOLVED] BASH - script to loop through files in size order descending

- Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)

- - BASH - script to loop through files in size order descending (https://www.linuxquestions.org/questions/linux-newbie-8/bash-script-to-loop-through-files-in-size-order-descending-4175724133/)

GPGAgent

04-16-2023 10:07 AM

BASH - script to loop through files in size order descending

I can write a simple for loop to loop through files in name order.

Code:

for ff in *.mp4; do HandBrakeCLI -i "$ff" -o "X$ff";done

This works fine with a command in the do section to process the $ff file

But I want the list in size order descending so I did this

Code:

for ff in "$( ls *.mp4 | sort -r )"; do echo "$ff"; done

which results in this

Code:

$ for ff in "$( ls *.mp4 | sort -r )"; do echo "$ff"; done;

X BIG.mp4

CMID.mp4

BMID.mp4

ASMALL.mp4

Fine I thought, sorted in size order escending, just need to put my command in the do section

Code:

for ff in "$( ls *.mp4 | sort -r )"; do HandBrakeCLI -i "$ff" -o "X$ff"; done

But I was wrong, there is a /n hidden in $ff whcih lets it display but not much use sending the whole list of files to HandBrakeCLI in one hit.
The result

Code:

$ for ff in "$( ls *.mp4 | sort -r )"; do HandBrakeCLI -i "$ff" -o "X$ff"; done;

[16:03:52] hb_init: starting libhb thread

[16:03:52] thread 7fbf823a0700 started ("libhb")

HandBrake 1.3.1 (2020032300) - Linux x86_64 - https://handbrake.fr

16 CPUs detected

Opening X BIG.mp4

CMID.mp4

BMID.mp4

ASMALL.mp4...

[16:03:52] CPU: Intel(R) Xeon(R) CPU          E5620  @ 2.40GHz

[16:03:52]  - logical processor count: 16

[16:03:52] hb_scan: path=X BIG.mp4

CMID.mp4

BMID.mp4

ASMALL.mp4, title_index=1

disc.c:323: failed opening UDF image X BIG.mp4

CMID.mp4

BMID.mp4

ASMALL.mp4

disc.c:424: error opening file BDMV/index.bdmv

disc.c:424: error opening file BDMV/BACKUP/index.bdmv

....

..

......

How do I seperate the list into individual movies to be processed squentially?

Note: Some files have spaces in their names

GPGAgent

04-16-2023 10:39 AM

I've tried something else, I've made a file of the movie names in descending size order, simple enough

Code:

$ ls *.mp4 | sort -r > x.x

$ cat x.x

X BIG.mp4

CMID.mp4

BMID.mp4

ASMALL.mp4

$

So now just read that file with a while loop

Code:

$ while read ff; do echo $ff; done < x.x

X BIG.mp4

CMID.mp4

BMID.mp4

ASMALL.mp4

$

And that works fine, so now put the HandBrakeCLI command in

Code:

$ while read ff; do HandBrakeCLI -i "$ff" -o "X$ff" --stop-at seconds:20; done < x.x

[16:36:34] hb_init: starting libhb thread

[16:36:34] thread 7f55a33bd700 started ("libhb")

HandBrake 1.3.1 (2020032300) - Linux x86_64 - https://handbrake.fr

16 CPUs detected

Opening X BIG.mp4...

[16:36:34] CPU: Intel(R) Xeon(R) CPU          E5620  @ 2.40GHz

[16:36:34]  - logical processor count: 16

[16:36:34] hb_scan: path=X BIG.mp4, title_index=1

....

..

x264 [info]: Weighted P-Frames: Y:1.0% UV:0.0%

x264 [info]: ref P L0: 63.7%  7.1% 21.8%  7.3%  0.1%

x264 [info]: ref B L0: 87.1% 10.3%  2.6%

x264 [info]: ref B L1: 97.7%  2.3%

x264 [info]: kb/s:1401.61

[16:36:46] mux: track 0, 600 frames, 3507325 bytes, 1399.20 kbps, fifo 1024

[16:36:46] mux: track 1, 939 frames, 5702 bytes, 2.27 kbps, fifo 1024

[16:36:46] Finished work at: Sun Apr 16 16:36:46 2023



[16:36:46] libhb: work result = 0



Encode done!



HandBrake has exited.

$

But only processes the first line!!!!

I thought I'd cracked it!

elgrandeperro

04-16-2023 11:03 AM

My guess is that handbrake is sending signals which is killing the shell.

Try this:

Code:



(while read ff; do HandBrakeCLI -i "$ff" -o "X$ff" --stop-at seconds:20; done)< x.x

GPGAgent

04-16-2023 11:09 AM

Quote:

Originally Posted by elgrandeperro (Post 6424980)

My guess is that handbrake is sending signals which is killing the shell.
Try this:

Code:

(while read ff; do HandBrakeCLI -i "$ff" -o "X$ff" --stop-at seconds:20; done)< x.x

That was my thought, but now your suggestion also stopped after processing the first line.

I know I will kick myself when I work it out or someone else does

MadeInGermany

04-16-2023 11:38 AM

Code:

for ff in "$( ls *.mp4 | sort -r )"; do

This does not work because $( ) becomes a string that is protected by the quotes.
Without the quotes the string is subject to word splitting (partly desired) and filename generation (not desired).
The best solution is a while-read loop

Code:

ls *.mp4 | sort -r | while IFS= read -r ff; do

Each read reads one line. IFS="" and -r turn
off some magic in read.
Commands in the loop body that can read from stdin would compete with the read, and should be redirected </dev/null
If HandBrakeCLI is such a command:

Code:

</dev/null HandBrakeCLI ...

Regarding
ls *.mp4 | sort -r
This is not by size but reverse alphabetical, like
ls -r *.mp4

GPGAgent

04-16-2023 11:57 AM

Quote:

Originally Posted by MadeInGermany (Post 6424991)

Code:

for ff in "$( ls *.mp4 | sort -r )"; do

Code:

ls *.mp4 | sort -r | while IFS= read -r ff; do

Code:

</dev/null HandBrakeCLI ...

Regarding
ls *.mp4 | sort -r
This is not by size but reverse alphabetical, like
ls -r *.mp4

You're absolutely correct about it NOT sorted by size, my mistake. Thanks for pointing that out.

Anyway I trid it like this

Code:

ls *.mp4 | sort -r | while IFS= read -r ff; do HandBrakeCLI -i "$ff" -o "X$ff" --stop-at seconds:20 ; done

but yet again it stopped after processing the first file!

MadeInGermany

04-16-2023 12:08 PM

Try the </dev/null

Code:

ls *.mp4 | sort -r | while IFS= read -r ff; do </dev/null HandBrakeCLI -i "$ff" -o "X$ff" --stop-at seconds:20 ; done

GPGAgent

04-16-2023 12:16 PM

Quote:

Originally Posted by MadeInGermany (Post 6425002)

Try the </dev/null

Code:

ls *.mp4 | sort -r | while IFS= read -r ff; do </dev/null HandBrakeCLI -i "$ff" -o "X$ff" --stop-at seconds:20 ; done

Cheers that worked, can you explain why the </dev/null kept it in the loop please?

Next step is to feed it a list by size, I think this might do it

Code:

ls *.mp4 -S

Don't forget some of the file names have spaces!

And yes this worked just fine

Code:

ls *.mp4 -S | sort -r | while IFS= read -r ff; do </dev/null HandBrakeCLI -i "$ff" -o "Y$ff" --stop-at seconds:20 ; done

teckk

04-16-2023 01:02 PM

Lots of ways of doing that with bash and friends.

Few Examples:

Sort files by size.

Code:

find *.mp4 -type f -exec du -h {} + | sort -rh

find *.mp4 -type f -exec ls -lS {} +

Code:

files1=(*.mp4)

files2=($(du -a <<< "$files1" | sort -nr))



printf '%s\n' "${files2[@]}"



for i in "${files1[@]}"; do

    echo "$i"

    sleep 1

done



for i in "${files2[@]}"; do

    echo ${i##*/}

    sleep 1

done



#First will fail, the rest will be ok

for i in "${files2[@]}"; do

    echo "HandBrakeCLI -i ${i##*/} -o New_${i##*/}"

    sleep 1

done

MadeInGermany

04-16-2023 01:57 PM

If ls can sort it, do not re-sort it by piping to sort!
Perhaps you want to reverse the order, then you must tell the ls to do it:

Code:

ls *.mp4 -rS | while ...

Regarding the </dev/null, I tried to explain it.
The loop's stdin is fed by the pipe.
Not only the read can read a line from the stdin.
Obviously the HandBrakeCLI reads from the stdin, too. And "steals" it, so the read doesn't get it.

GPGAgent

04-16-2023 03:55 PM

Quote:

Originally Posted by MadeInGermany (Post 6425039)

If ls can sort it, do not re-sort it by piping to sort!
Perhaps you want to reverse the order, then you must tell the ls to do it:

Code:

ls *.mp4 -rS | while ...

Sorry, I did read your original post, but it didn't register in my 72yr old brain!!

GPGAgent

04-16-2023 03:55 PM

Thanks all, I've learn't a bit more now, enough to be dangerous!!

All times are GMT -5. The time now is 10:41 AM.