LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 09-01-2013, 05:32 AM   #1
Corsari
Member
 
Registered: Oct 2004
Posts: 54

Rep: Reputation: 15
create file list: SED inline vs SED standalone, enormous speed difference


Hello to all the community

I've created this one line command, script, to create files lists, that works, though

Code:
ls -R1 /rootpathname/ | while read l; do case $l in *:) d=${l%:};; "") d=;; *) echo "$d/$l";; esac; done > /tmp/filelistname.txt
since I do use it with disks mounted under /media (ubuntu), the files-lists increases in size since at the begin of every line, there is the
Code:
/media/volumename
string

so I've added some sed, but the speed slowed down in a terrible manner, hundreds times slower

here it is the code, though... and it works

Code:
ls -R1 /media/MAC01/ | while read l; do case $l in *:) d=${l%:};; "") d=;; *) echo $d/$l | sed 's/\/media\/MAC01//';; esac; done >  /tmp/MAC01-file-list.txt
but this command will take ages against doing it into two steps

Code:
ls -R1 /media/MAC01/ | while read l; do case $l in *:) d=${l%:};; "") d=;; *) echo "$d/$l";; esac; done > /tmp/MAC01-file-list.txt
followed by

Code:
sed 's/\/media\/MAC01//' /tmp/MAC01-file-list.txt > /tmp/MAC01-file-list-cleaned.txt
literally some seconds against many minutes (this drive contains more than 400.000 files, which mean that its files lists' length, is more than 400.000 lines)

Do any of you have an technical explanation about this enormous speed difference?

Have I placed the sed command in a wrong position?

Thank you for hinting

Cor

Last edited by Corsari; 09-01-2013 at 05:33 AM.
 
Old 09-01-2013, 05:56 AM   #2
Corsari
Member
 
Registered: Oct 2004
Posts: 54

Original Poster
Rep: Reputation: 15
Maybe the command substitution technique could help increasing the speed,

since they write it extracts the stdout of a command, then assigns it to a variable using the = operator.

but I don't find any reference that I could understand to apply it to the above "slow" script.

Last edited by Corsari; 09-01-2013 at 05:57 AM.
 
Old 09-01-2013, 11:17 AM   #3
GazL
Senior Member
 
Registered: May 2008
Posts: 4,081
Blog Entries: 1

Rep: Reputation: 1526Reputation: 1526Reputation: 1526Reputation: 1526Reputation: 1526Reputation: 1526Reputation: 1526Reputation: 1526Reputation: 1526Reputation: 1526Reputation: 1526
Seems like a awfully complicated way of producing a file list.
Can't you just use 'find'?

Code:
find "/media/MAC01/" -type f -printf "%P\n" | sort > filelist.txt
(the %P format code will strip off the /media/MAC01/ prefix, so no need for any sed'ing)


The reason your code is so slow is because you placed the call to sed within the body of the loop, so you're asking it to start 400,000 instances of sed, one after the other.
 
1 members found this post helpful.
Old 09-01-2013, 05:59 PM   #4
Corsari
Member
 
Registered: Oct 2004
Posts: 54

Original Poster
Rep: Reputation: 15
thank you GazL

it works and it is fast

searching I've also found this command

Code:
find . > filelist.txt
it must be run from the root of the tree you want to create the filelist.

to only count the files in a tree , the output could be piped to wc, like this find . | wc -l

Cor

Last edited by Corsari; 09-01-2013 at 06:14 PM.
 
Old 09-02-2013, 04:01 AM   #5
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,243

Rep: Reputation: 2684Reputation: 2684Reputation: 2684Reputation: 2684Reputation: 2684Reputation: 2684Reputation: 2684Reputation: 2684Reputation: 2684Reputation: 2684Reputation: 2684
Please remember to mark as SOLVED once you have a solution
 
  


Reply

Tags
sed bash


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] sed help to run sed command against multiple different file names bkone Programming 2 04-16-2012 01:27 PM
How to create a script to add sed command into a file Runicer Programming 4 04-08-2012 04:00 PM
[SOLVED] sed 's/Tb05.5K5.100/Tb229/' alone but doesn't work in sed file w/ other expressions Radha.jg Programming 6 03-03-2011 08:59 AM
suing sed to remove certain list of strings from a file ??? HuMan-BiEnG Linux - Distributions 11 10-13-2010 10:04 AM
Get a list of delimited filenames from a text file (sed?) Ksearch Linux - Newbie 14 06-30-2009 05:51 PM


All times are GMT -5. The time now is 08:43 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration