Bash - Testing directories and Files where one folder is unknown?
I want to do the following testing where I will not know one of the folders
[[ -f /folder/*/folder3/file.txt ]] && ehco "success" [[ -d /folder/*/folder4/folder5 ]] && echo "success" The above fails so I tried to escape the * [[ -f /folder/\*/folder3/file.txt ]] && ehco "success" but this fails aslo.. the reasion to do this is to speed up a script, where the echo "success" are replaced with find commands. Whats the best way to do this? Cheers Hdaz |
Definitely not optimized but why don't you 'find /folder -type d -name folder3' and use that output to "anchor" your test use 'find /folder | grep -q "folder3/file.txt' && doSomething'?
|
Hi unSpawn,
Thanks for your reply, sure I can do that but it seems a little OTT and painful to do this just to test the condition. The script i am trying to speed up could have a lot of these within so it becomes painful working out each any every folder structure just to test for a file or folder. This is 90% there hmmm [[ -d `/folder/*/folder3/` ]] && ehco "success" -bash: /folder/folder2/folder3/: is a directory which would be the correct result as * is not a directory... [[ -d `/folder/\*/folder3/` ]] && ehco "success" -bash: /folder/*/folder3/: No such file or directory If this works why does the 90% version above not hmm [[ -n `ls -lrth /folder/*/folder3/` ]] && echo "success" success Just to answer my own questions hmm cat folder.sh #!/usr/bin/env bash [[ -d `/folder//.*/folder3/` ]] && ehco "success" bash -x folder.sh ++ '/folder//.*/folder3/' folder.sh: line 2: /folder//.*/folder3/: No such file or directory + [[ -d '' ]] hdaz |
Quote:
Quote:
Quote:
Code:
[ -d /folder/*/folder3 ] && echo "OK" Code:
readlink -f /folder/*/folder3 && echo "OK" Code:
[ -d $(readlink -f /folder/*/folder3) ] && echo "OK" |
Thanks for the reply unSpawn...
I was just testing and talking out aloud with all the different attempts.. The last one does it nicely :) don't think I have every used readlink before interesting.. :) wonders why does this fail in double [[ ? [[ -d $(readlink -f /folder/*/folder3) ]] && echo "OK" [ -d $(readlink -f /folder/*/folder3) ] && echo "OK" OK Have a good evening Cheers hdaz |
Quote:
cat /file|grep == grep /file cat /file|sed > /somefile == sed -i /file ps|grep something == pgrep something etc, etc. Script more and you'll find out more. Quote:
http://www.tldp.org/HOWTO/Bash-Prog-Intro-HOWTO.html http://www.tldp.org/LDP/Bash-Beginne...tml/index.html http://www.gnu.org/software/bash/man...ode/index.html http://www.grymoire.com/Unix/Sh.html http://www.tldp.org/LDP/abs/html/ http://mywiki.wooledge.org/BashFAQ http://mywiki.wooledge.org/BashPitfalls |
Hey UnSapwn,
Unfortunitly after a little testing [ -d $(readlink -f /folder/*/folder3) ] && echo "OK" does not work and here is why... ls -lrth /folder/folder2/folder3 total 0 -rw-r--r-- 1 root root 0 Apr 26 09:04 file [ -f $(readlink -f /folder/*/folder3/file) ] && echo "OK" OK [ -f $(readlink -f /folder/*/folder3/fie) ] && echo "OK" OK [ -d $(readlink -f /folder/*/foler3) ] && echo "OK" OK bash -x foldertest.sh ++ readlink -f /folder/folder2/folder3 + '[' -d /folder/folder2/folder3 ']' + echo OK OK [root@hostingtest ~]# bash -x foldertest.sh ++ readlink -f '/folder/*/foder3' + '[' -d ']' + echo OK OK [ -d $(readlink -ve /folder/*/foler3) ] && echo "OK" readlink: /folder/*/foler3: No such file or directory OK [ -d $(readlink -vf /folder/*/foler3) ] && echo "OK" readlink: /folder/*/foler3: No such file or directory OK Close but not quite... |
Yeah, my bad, that's because it simply takes the exit value of 'readlink'.
Code:
# The problem with this ITEM is that you DO NOT want to have the script expand it unless necessary. |
Hey unSpawn,
I just re-read my own posts and realised I answered my own question doh :) Just use [[ -n $(ls -lrth /folder/*/folder3/) ]] && echo "success" which works correctly when the folder does not exist. Maybe this should be an improvement within the handing of bash for [[ -d or -f ]] directives... unless we are just doing something wrong.. I agree the script could probably could do with restructuring but there are reasons why I wrote it the way I did, I wanted it as concise and compact as possible.... hence looking at commands that would one speed things up and two keep things compact.. unfortunately the script contains a lot of sensitive data so its not really possible to upload it here. hdaz |
Please use ***[code][/code]*** tags around your code and data, to preserve the original formatting and to improve readability. Do not use quote tags, bolding, colors, "start/end" lines, or other creative techniques.
I think we need to start by getting a little more info. First, are you sure you're using bash? What does the shebang on the top line look like? If it says #!/bin/sh, then you're using the system's posix-based shell, which may be something like dash instead. That could be why you aren't able to use things like [[..]]. Always use #!/bin/bash if you want support for bash-specific built ins. See here for the main differences between bash and posix shells: http://mywiki.wooledge.org/Bashism Actually, I see you using things like "bash -x foldertest.sh". That does specify the script as bash, but it's such a sloppy way to do it. As long as the scripts have proper shebangs, just run them directly with "/path/to/foldertest.sh" and similar. Second, what do the actual filenames/paths look like? Could you show us an example directory tree and explain more clearly what needs to be tested about it? The first problem, as UnSpawn mentioned above, is that a globbing pattern of "*/" will expand into a list of all subdirectories inside the one specified. So using "/folder/*/folder3/file" could end up giving you: /folder/folder2a/folder3/file /folder/folder2b/folder3/file /folder/folder2c/folder3/file You can't run a single test on output like that. Perhaps you could capture the results of the glob expansion into an array, then loop over that to test for possible values. But again, it's going to depend on the details of what you have and what you want. From what's been posted so far, I agree that the whole code flow probably needs re-writing, particularly since you say that it "contains a lot of sensitive data". A script has any kind of data hard-coded into it all (outside of perhaps a few default variable values at the top) is by definition a poorly written script. I suggest you go through the script to redact (replace with dummy values) the sensitive parts and then post what you can. We need to see more of the code in context before we can really get down to fixing it. PS: I see a lot of typos in the above posts; 'foder' instead of 'folder', etc. You'll have to tell us whether this is important to the script or just mistakes in explaining it here. |
Dave, Many thanks for the reply sorry you find my output a little sloppy..
Quote:
Code:
[ -d $(readlink -f /folder/*/folder3) ] && echo "OK" My original thought and question was why following fails if the files and folders exists Code:
[[ -f /folder/*/folder3/file.txt ]] && ehco "success" Code:
[[ -f Code:
[[ -d As can be seen on this attempt bash knows the files structure but still fails i.e. the continued commands in this case "ehco "success"" never gets run. Code:
[[ -d `/folder/*/folder3/` ]] && ehco "success" Code:
[[ -n $(ls /folder/*/folder3/) ]] && echo "success" Cheers Hdaz |
Code:
[[ -d `/folder/*/folder3/` ]] && ehco "success" 2. there's no such cmd as 'ehco', so that'll fail always. You must ensure exact correct spelling at all times. |
Hi Chris,
none of these makes any diffence the second command never gets run. Code:
ls -lrth /folder/folder2/folder3/file Cheers Hdaz |
Code:
[[ -d '/folder/*/folder3/' ]] && echo "success" In short you must have an actual, fixed, directory path before you can test for a directory. In any case, you didn't address any of the important points from my last post. 1) Since you're using double-bracket tests, I'm assuming the availability of bash or ksh, is that correct? Or does this have to be a posix-compatible script, limiting the availability to its less-advanced features? 2) Can you guarantee that the globbing pattern will always expand into a single file or directory? If not, and I maintain that you can't, you'll have to decide what to do if you come across more than one entry. But whatever the case, to do it safely the globbing patterns should be run through a loop that tests each possible value individually. 3) As I mentioned, good coding practice says that code and data should be kept as separate as possible. File and directory names, and any text strings that could change, should be set in variables at the top of the script, or even imported into it from elsewhere. Keep the code clean. This includes the paths you are trying to test for, so could you please explain clearly what should and shouldn't exist? So I'm going to ask again for some more background on what you are trying to accomplish. It's still not completely clear to me exactly what you need to test for, and what those tests need to accomplish. Could you please explain the context of your needs in some more detail? What files and directories need to exist, what should not exist, and what should happen if conditions aren't what you expect them to be (e.g. there are extra entries)? And again, if you could post at least the relevant sections of the actual code, with any sensitive data removed or altered, I'd be happy to help you re-write it to be cleaner and more robust. |
Hi David,
1, bash or ksh - keeping things portable might be useful but at the moment 99% of all vms/servers are CentOS. Code:
cat folder.sh That's perfect as I want to catch each any every entry Code:
&& echo " Code:
for in $(find -iname /DIR/path/xyzfile);do xyz;done 3, thats a great point, I do like to try and keep things a separated as possible but in this case not really reinvent. I have a much better bash project that might be more interesting an useful if you would like to help with that? Cheers Hdaz |
Ok, now we're getting somewhere. I'd still like to have a little more detail about the context of your script; the exact matching criteria you want to use, and what you intend to do with the matches, but at least we can work with this.
Since you do indeed expect to test multiple entries, yes, a loop is what you need. But do not use a for loop if you intend to use an external command like find. Code:
while IFS='' read -d '' -r dname; do Using -print0 (null separators) and the corresponding settings in read, makes it possible to safely handle all file names. Notice also the syntax of find. You need to give it one or more starting directories, followed by your matching options, and finally one or more actions to perform on them (-print is the default). The -name/-iname options use globbing patterns, so you have to specify something that will match the entire file name (and don't forget to quote it to protect it from shell expansion). The input is fed into the loop with a bash/ksh style process substitution, so it's not posix portable. There are more portable ways do handle it if needed. On the other hand, a for loop is just fine if you use a simple globbing pattern. If you know what directory level to search and don't need to do recursive searching, then this is probably what you really want. Code:
shopt -s dotglob nullglob Actually, you could use '*/' as the globbing pattern to expand directories only and skip the other file types. And again, you'll still have to decide exactly what to do with whatever it detects. You could add them to arrays, for example, for later use. Code:
if [[ -d $name ]]; then Code:
find "${dirarray[@]}" <searchoptions> <actions> You can also use a globbing pattern directly in the startdir part of find, BTW, as long as it would expand into a list of directories. Here are a couple of good links about using find: http://mywiki.wooledge.org/UsingFind http://www.grymoire.com/Unix/Find.html Quote:
|
Hi David,
Many thanks for your time and the detailed reply, the information looks very useful to myself and many others that might glance this post. I have 40 for loops so at some time will rewrite them to see how much performance difference I might gain from using while loops instead. It's more of a habit than anything else. I certainlly will post the other project if I don't find what I am seeking (doing a little research before posting :) Cheers Hdaz |
The use of while vs for loops isn't really a performance issue, but comes from the fact that they have different functions.
A for loop iterates over a fixed list of individual word tokens, whereas a while loop runs for as long as some condition is true. When the while loop is combined with read it can be used to parse arbitrary input text, from both files and other commands. The trouble usually comes from trying to use a for loop on command and variable substitutions, as shown in the link I provided. As long as the expansion results in a simple word list it's not a big problem, unless the list is very large, but trying to use it on things like filenames is very risky, due to the shell word-splitting and pathname expansion operations that follow the substitution. for loops are, however, recommended for use on file names generated by direct globbing expansion. The risk only comes when the list is generated indirectly by another command. It's therefore a good idea to simply remember to use while+read loops when the input comes from a text file or command, and for loops on globbing, arrays, brace expansion and other lists of simple elements. The things you should really focus on if you want to reduce script overhead are to: 1) Eliminate as many external command calls as possible, usually by using built-in string manipulations, instead. Also, learn to run a command once and save its output for future use, instead of calling it over and over every time you need it (date often tends to be abused that way). As a rule of thumb, bulk operations that have to scan large amounts of text are often better handled by an external tool like sed or awk, but once a text string is stored in a variable, it's usually better to use built-in shell operations on it. 2) Design your code flow to eliminate as many redundant operations as possible. e.g. use a single case statement instead of a series of if..elif..else tests, and printf instead of print loops. If your script has 40+ loops, I imagine you can probably combine the operations of at least some of them together. You should also consider creating functions for often-called operations. This may or may not save on redundancy, but it can at the very least make the code cleaner. |
All times are GMT -5. The time now is 02:09 AM. |