[SOLVED] Sorting a Recursive Directory Listing by time without dividing into subdirectories

scorchgeek · 09-05-2011, 03:28 PM

I am trying to get a listing of all the files in my home directory by the time I last modified them. So I have:

Code:

ls -lRt ~ | less

But this displays a listing like so:

Code:

/home/soren:
drwxrwxrwt  3 root  root     100 2011-09-05 12:50 Downloads
drwxr-xr-x  3 soren soren   4096 2011-09-04 22:48 Desktop
...

/home/soren/Downloads:
...

/home/soren/Desktop:
...

So I wind up with many different lists of files by date modified, one for each subdirectory. Is there a way to get a list with similar details (and still showing the full path to each file, so removing the entries with sed and resorting won't cut it), but all in one list?

Larry James · 09-05-2011, 03:59 PM

Quote:

Originally Posted by scorchgeek

I am trying to get a listing of all the files in my home directory by the time I last modified them. So I have:

Code:

ls -lRt ~ | less

But this displays a listing like so:

Code:

/home/soren:
drwxrwxrwt  3 root  root     100 2011-09-05 12:50 Downloads
drwxr-xr-x  3 soren soren   4096 2011-09-04 22:48 Desktop
...

/home/soren/Downloads:
...

/home/soren/Desktop:
...

So I wind up with many different lists of files by date modified, one for each subdirectory. Is there a way to get a list with similar details (and still showing the full path to each file, so removing the entries with sed and resorting won't cut it), but all in one list?

Try:

Code:

ls -lRt1

The files are still sorted per directory, but you get a full path for each file. Also each directory is time sorted. Of course this will meet your sed criterion.

-- L. James

--
L. D. James
ljames@apollo3.com
www.apollo3.com/~ljames

colucix · 09-05-2011, 04:07 PM

You can achieve something similar using find with -printf, then pipe the output to sort:

Code:

find ~ -type f -printf "%M %u %g %s %TY%Tm%Td %TH%TM %p\n" | sort -k5n -k6n

For better formatted output, pipe to awk and finally to column:

Code:

find ~ -type f -printf "%M %u %g %s %TY%Tm%Td %TH%TM %p\n" | sort -k5n -k6n  | awk '{$6=(substr($6,1,2) ":" substr($6,3,2))}1' | column -t

scorchgeek · 09-05-2011, 05:42 PM

Quote:

Originally Posted by colucix

You can achieve something similar using find with -printf, then pipe the output to sort:

Code:

find ~ -type f -printf "%M %u %g %s %TY%Tm%Td %TH%TM %p\n" | sort -k5n -k6n

For better formatted output, pipe to awk and finally to column:

Code:

find ~ -type f -printf "%M %u %g %s %TY%Tm%Td %TH%TM %p\n" | sort -k5n -k6n  | awk '{$6=(substr($6,1,2) ":" substr($6,3,2))}1' | column -t

Excellent, thank you very much. The first command works, but it's the opposite order of what I would like. How would I go about reversing the order of the output so the newest one is on the top? Also, is there a way to exclude certain directories (like cache, etc)? And finally, the columns don't line up neatly--I assume that's what you were trying to do with the second command?

The second command resulted in unreadable output, though--it fragmented filenames all over the place. Maybe some of my filenames were too long?

(And Larry, your command simply didn't work-- 1 didn't give a full path for each entry.)

colucix · 09-05-2011, 06:36 PM

Quote:

Originally Posted by scorchgeek

Excellent, thank you very much. The first command works, but it's the opposite order of what I would like. How would I go about reversing the order of the output so the newest one is on the top?

Just reverse the output of the sort command using the r modifier:

Code:

find ~ -type f -printf "%M %u %g %s %TY%Tm%Td %TH%TM %p\n" | sort -k5nr -k6nr

Quote:

Originally Posted by scorchgeek

Also, is there a way to exclude certain directories (like cache, etc)?

Yes, you have to refine the search criteria of the find command. What exactly are the directories you want to exclude? All the hidden ones or some specific subfolder like the browser cache one?

Quote:

Originally Posted by scorchgeek

And finally, the columns don't line up neatly--I assume that's what you were trying to do with the second command? The second command resulted in unreadable output, though--it fragmented filenames all over the place. Maybe some of my filenames were too long?

The awk part just puts a colon in the time field, the column -t command formats the output. What do you mean by fragmented filenames all over the place? Please, can you post an example of the wrong part of the output?

scorchgeek · 09-05-2011, 06:57 PM

I couldn't get the output to cut and paste correctly, so here's a screenshot. The problem isn't terminal size--I increased the width to something way more than enough to display the entire line. However, if I let it run for a little while the next part of the list looks excellent.

I would like to exclude the hidden folders, and also a single folder called "VirtualBox VMs."

Thanks a ton for your help!

colucix · 09-06-2011, 02:39 AM

Let's begin from the excluding hidden directories part. I use the -regex predicate of the find command like this:

Code:

find ~ \( ! -regex '.*/\..*' \) -type f

In a similar way you can exclude any other directory name based on a pattern/regexp. Another approach is to use -prune but if the -regex does its job, you can spare time and maybe an headache for now.

Regarding the long lines problem, it occurs where file names have spaces. The column command uses spaces as separator by default, so that a file with blank spaces is split over multiple columns. To avoid that you might uses another separator, e.g. TAB but you have to specify it in all the commands. Let's try:

Code:

find ~ \( ! -regex '.*/\..*' \) -type f -printf "%M\t%u\t%g\t%s\t%TY%Tm%Td\t%TH%TM\t%p\n" | sort -t$'\t' -k5nr -k6nr | awk 'BEGIN{FS=OFS="\t"}{$6=(substr($6,1,2) ":" substr($6,3,2))}1' | column -s$'\t' -t

The parts in blue take care of the TAB separator.

However I noticed a problem using this approach. It appears that the column command has a limit on the number of rows he can manage or maybe on the size of the input. If I run the command above on my machine, I get 18665 files, whereas if I remove the column command from the pipe I get all my 125890 files. I need further investigation about this issue.

colucix · 09-06-2011, 03:10 AM

An alternative to the column command is the usage of printf in the awk statement. Since we have a fixed number of fields to print out (we decide them in the find command) we can establish a fixed format for each field, e.g.

Code:

find ~ \( ! -regex '.*/\..*' \) -type f -printf "%M\t%u\t%g\t%s\t%TY%Tm%Td\t%TH%TM\t%p\n" | sort -t$'\t' -k5nr -k6nr | awk 'BEGIN{FS="\t"}{$6=(substr($6,1,2) ":" substr($6,3,2))}{printf "%10s %-10s %-10s %11d  %8d %5s  %s\n",$1,$2,$3,$4,$5,$6,$7}'

This gives the following output and the number of files is correct:

Code:

-rw-r--r-- colucix    users            10326  20110906 09:00  /home/colucix/ADRIAROMS/COSMO_ANALYSES/analysis.log
-rw-rw-r-- colucix    users         28893098  20110906 06:35  /home/colucix/ADRIAROMS/COSMO_ANALYSES/20110906_analysis.grb.gz
-rw-rw-r-- colucix    users         29381248  20110905 08:39  /home/colucix/ADRIAROMS/COSMO_ANALYSES/20110904_analysis.grb.gz
-rw-rw-r-- colucix    users         29352263  20110905 06:35  /home/colucix/ADRIAROMS/COSMO_ANALYSES/20110905_analysis.grb.gz
-rw-rw-r-- colucix    users         29088766  20110903 06:35  /home/colucix/ADRIAROMS/COSMO_ANALYSES/20110903_analysis.grb.gz
<omitted>
-rw-r--r-- colucix    users             4810  19950224 16:04  /home/colucix/mfiles/roms_tools/tools/contoursurf.m
-rw-r--r-- colucix    users             5442  19950224 16:04  /home/colucix/mfiles/roms_tools/tools/extclabel.m
-rw-rw-rw- colucix    users            42009  19940127 11:39  /home/colucix/odvmp/samples/EPSFIG.STY
-rw-rw-rw- colucix    users             8559  19930512 10:43  /home/colucix/odvmp/samples/EPSF.STY
-rw-rw-rw- colucix    users              268  19930506 01:00  /home/colucix/odvmp/samples/EPSF.TEX

scorchgeek · 09-06-2011, 02:46 PM

Excellent, that does just what I wanted. Thanks a lot!

colucix · 09-07-2011, 08:45 AM

You're welcome!