Linux - NewbieThis Linux forum is for members that are new to Linux.
Just starting out and have a question?
If it is not in the man pages or the how-to's this is the place!
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
I don't think either of the two pipelines work. You create a long listing and pass filenames like drwxrwxrwx or admin to the rm command. Try without the -l option.
EDIT: The rm -f option prevents error messages to be printed, so that it should actually work (though it's not pretty).
Last edited by berndbausch; 10-16-2019 at 05:14 AM.
ls -tl /share/CACHEDEV1_DATA/Backup/NetBakData/User
list the files including the full path?
no
your first example works ( with lots of errors ) as the full path eventually hits
Code:
drwxrwxrwx 3 admin administ 4096 Oct 8 09:59 2019-10-08-10-00-00/
drwxrwxrwx 3 admin administ 4096 Oct 7 09:59 2019-10-07-10-00-00/
the ones in red produce error ( they do not exist )
the green do exist, they get deleted.
pipe trains are awful., don't use them
ls is for humans, don't use them in scripts
I'm going to ignore "files", and only look at the "dirs", taking advantage of the sort order to delete the oldest
This assumes your dirnames will always have that "date format"
Code:
#!/bin/bash
LC_ALL=C
# set the locale to ensure sort order is consistent
DirPath=/share/CACHEDEV1_DATA/Backup/NetBakData/User
Keep=4
Dirs=( ${DirPath%/}/*/ )
for (( i=0 ; i < $(( ${#Dirs[@]} - $Keep )) ; i++ ))
do
echo rm -rf \"${Dirs[$i]}\"
done
Pourquoi?
For me, it is just following the *nix philosophy of using a simple tool for a single task, then linking those tasks together.
did that ideology work for the OP ?
did I achieve what the OP wanted to without using pipe trains?
pipe trains are slow, and ugly
They do have a use in reconnaissance while you explore something.. but once you begin writing a script you need to stop using those trains. especially the ones with UUOC and UUOE , awful scripts, truly awful.
No, because the commands weren't used correctly. That's not really what I'd call an "ideological" issue.
Quote:
Originally Posted by Firerat
did I achieve what the OP wanted to without using pipe trains?
Perhaps, perhaps not:
Quote:
Originally Posted by Firerat
Code:
Dirs=( ${DirPath%/}/*/ )
The above will break if the string containing all file names in ${DirPath} exceeds the OS line length limit. This limit can be anything from a few kilobytes (Cygwin...) to a few megabytes, so this is a very real issue. For that reason the practice of using shell expansion directly in scripts is discouraged.
Quote:
Originally Posted by Firerat
pipe trains are slow, and ugly
Whether they are slower than using a different approach depends very much on the task being performed.
As for "ugliness", a list of piped commands are usually exceedingly easy to understand. Of course, you could also make a list of commands look like a jumbled mess, but that's true for any method or tool used in scripting or programming. "A poor craftsman blames his tools."
(Edit: BTW, did you try the code before posting it? It doesn't do what you think it does, and certainly not what the OP asked for.)
No, because the commands weren't used correctly. That's not really what I'd call an "ideological" issue.
Perhaps, perhaps not:
The above will break if the string containing all file names in ${DirPath} exceeds the OS line length limit. This limit can be anything from a few kilobytes (Cygwin...) to a few megabytes, so this is a very real issue. For that reason the practice of using shell expansion directly in scripts is discouraged.
Whether they are slower than using a different approach depends very much on the task being performed.
As for "ugliness", a list of piped commands are usually exceedingly easy to understand. Of course, you could also make a list of commands look like a jumbled mess, but that's true for any method or tool used in scripting or programming. "A poor craftsman blames his tools."
That's very interesting! I knew this worked in Linux, but what I did not know was that it's part of the POSIX specification, and as such it should work basically everywhere.
I still maintain that it's not exactly "best practices" to use what basically amounts to a trick that only works with builtins. If used in any other context than assigning values to variables, it will break if the command isn't a builtin, which in many cases depends entirely on the shell being used.
You could end up with a perfectly generic looking command that would turn out to be a bashism under certain non-obvious circumstances.
Quote:
Originally Posted by Firerat
an alternative
Code:
LC_ALL=C
DirPath=/share/CACHEDEV1_DATA/Backup/NetBakData/User
Keep=4
Dirs=( ${DirPath%/}/*/ )
for (( i=0 ; i < $(( ${#Dirs[@]} - $Keep )) ; i++ ))
do
printf "%s\n" "${Dirs[$i]}"
done | xargs echo rm -r
This works fine, but the process of reading directories into an array slows the whole script down. Try this with a few thousand directory names and see what happens.
Simply generating a list of directories and piping them directly to rm skips that step entirely, and by using the correct command to fetch the directory names, the list could arrive pre-sorted.
yes, I threw the xargs in to deal with huge numbers of dirs
ah bless, another one without the basics
xargs is specifically to get round ARG_MAX issue
that was never a problem with my original script
ok, with lots of dir individual rm -r from the for loop is going to be slow, but the actual unlink is going to take much longer than iterating over the array
man xargs
Code:
NAME
xargs - build and execute command lines from standard input
SYNOPSIS
xargs [options] [command [initial-arguments]]
DESCRIPTION
This manual page documents the GNU version of xargs. xargs reads items from the standard input, delimited by blanks (which can be pro-
tected with double or single quotes or a backslash) or newlines, and executes the command (default is /bin/echo) one or more times with
any initial-arguments followed by items read from standard input. Blank lines on the standard input are ignored.
The command line for command is built up until it reaches a system-defined limit (unless the -n and -L options are used). The speci-
fied command will be invoked as many times as necessary to use up the list of input items. In general, there will be many fewer invo-
cations of command than there were items in the input. This will normally have significant performance benefits. Some commands can
usefully be executed in parallel too; see the -P option.
Because Unix filenames can contain blanks and newlines, this default behaviour is often problematic; filenames containing blanks and/or
newlines are incorrectly processed by xargs. In these situations it is better to use the -0 option, which prevents such problems.
When using this option you will need to ensure that the program which produces the input for xargs also uses a null character as a sep-
arator. If that program is GNU find for example, the -print0 option does this for you.
If any invocation of the command exits with a status of 255, xargs will stop immediately without reading any further input. An error
message is issued on stderr when this happens.
yes, I threw the xargs in to deal with huge numbers of dirs
If so, you did it wrong. The xargs command never receives any data and just sits there, because it expects data to arrive via a pipe, something you're not using. A quick test run would have told you that.
No matter how xargs is invoked, it must in fact be invoked before anything happens, and anything you supply as "initial-arguments" (that would be ${Dirs[@]} in this case) are obviously subject to ARG_MAX.
This is laughably easy to test. (Spoiler: The error message from xargs is "xargs: can not fit single argument within argument list size limit".)
Quote:
Originally Posted by Firerat
that was never a problem with my original script
No, besides the potentially memory/time hogging array, your original script contained some odd escaping that resulted in something I've honestly never seen before. You should remove just the echo statement and try it, it's actually completely safe.
Quote:
Originally Posted by Firerat
ok, with lots of dir individual rm -r from the for loop is going to be slow, but the actual unlink is going to take much longer than iterating over the array
It's generating the array that takes time, and lots of it, for no benefit at all.
There surely must exist scenarios where it makes sense to read everything into memory before processing the data, but this just isn't one.
Quote:
Originally Posted by Firerat
ah bless, another one without the basics
Quote:
Originally Posted by Firerat
please educate yourself
You don't think it's a bit arrogant to make statements such as these, when you haven't even checked to see that perhaps you might be in the wrong? Which in this case, you were. (And even if you weren't, why be rude?)
that is completely offtopic here, but anyway yes, it is ugly and slow.
But usually that does not count, because the machine is more or less idle and has time to execute a lot of programs simultaneously. Furthermore 0.1ms or 15ms or anything similar (as execution time) is virtually the same, all are acceptable.
But in a production environment, where there are a plenty of users running hundreds of tasks this is just waste of time, ram and cpu.
That pipe chain [most of them] can be easily replaced by a single awk/perl/python/java/whatever script, which will be much faster and more efficient. You can find examples in this forum too.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
