LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 04-24-2013, 09:22 AM   #1
hyperdaz
Member
 
Registered: Sep 2004
Location: UK
Distribution: CentOS 5.5
Posts: 44

Rep: Reputation: 1
Smile Bash - Testing directories and Files where one folder is unknown?


I want to do the following testing where I will not know one of the folders

[[ -f /folder/*/folder3/file.txt ]] && ehco "success"
[[ -d /folder/*/folder4/folder5 ]] && echo "success"

The above fails so I tried to escape the *

[[ -f /folder/\*/folder3/file.txt ]] && ehco "success"

but this fails aslo..

the reasion to do this is to speed up a script, where the echo "success" are replaced with find commands.

Whats the best way to do this?

Cheers
Hdaz
 
Old 04-24-2013, 05:23 PM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,310
Blog Entries: 54

Rep: Reputation: 2858Reputation: 2858Reputation: 2858Reputation: 2858Reputation: 2858Reputation: 2858Reputation: 2858Reputation: 2858Reputation: 2858Reputation: 2858Reputation: 2858
Definitely not optimized but why don't you 'find /folder -type d -name folder3' and use that output to "anchor" your test use 'find /folder | grep -q "folder3/file.txt' && doSomething'?
 
Old 04-25-2013, 04:25 AM   #3
hyperdaz
Member
 
Registered: Sep 2004
Location: UK
Distribution: CentOS 5.5
Posts: 44

Original Poster
Rep: Reputation: 1
Hi unSpawn,

Thanks for your reply, sure I can do that but it seems a little OTT and painful to do this just to test the condition.

The script i am trying to speed up could have a lot of these within so it becomes painful working out each any every folder structure just to test for a file or folder.

This is 90% there hmmm
[[ -d `/folder/*/folder3/` ]] && ehco "success"
-bash: /folder/folder2/folder3/: is a directory


which would be the correct result as * is not a directory...
[[ -d `/folder/\*/folder3/` ]] && ehco "success"
-bash: /folder/*/folder3/: No such file or directory

If this works why does the 90% version above not hmm
[[ -n `ls -lrth /folder/*/folder3/` ]] && echo "success"
success


Just to answer my own questions hmm
cat folder.sh
#!/usr/bin/env bash
[[ -d `/folder//.*/folder3/` ]] && ehco "success"

bash -x folder.sh
++ '/folder//.*/folder3/'
folder.sh: line 2: /folder//.*/folder3/: No such file or directory
+ [[ -d '' ]]

hdaz

Last edited by hyperdaz; 04-25-2013 at 04:45 AM.
 
Old 04-25-2013, 03:24 PM   #4
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,310
Blog Entries: 54

Rep: Reputation: 2858Reputation: 2858Reputation: 2858Reputation: 2858Reputation: 2858Reputation: 2858Reputation: 2858Reputation: 2858Reputation: 2858Reputation: 2858Reputation: 2858
Quote:
Originally Posted by hyperdaz View Post
it seems a little OTT and painful to do this just to test the condition.
Sure, I never said it was anything other than crappy ;-p


Quote:
Originally Posted by hyperdaz View Post
The script i am trying to speed up could have a lot of these within so
Sounds more like restructuring is in order...


Quote:
Originally Posted by hyperdaz View Post
This is 90% there hmmm
Code:
[[ -d `/folder/*/folder3/` ]] && ehco "success"
-bash: /folder/folder2/folder3/: is a directory
Don't confuse backticks with single quotes (which you don't need either) and mind your typos:
Code:
[ -d /folder/*/folder3 ] && echo "OK"
or maybe use
Code:
readlink -f /folder/*/folder3 && echo "OK"
or
Code:
[ -d $(readlink -f /folder/*/folder3) ] && echo "OK"

Last edited by unSpawn; 04-25-2013 at 03:26 PM.
 
1 members found this post helpful.
Old 04-25-2013, 04:01 PM   #5
hyperdaz
Member
 
Registered: Sep 2004
Location: UK
Distribution: CentOS 5.5
Posts: 44

Original Poster
Rep: Reputation: 1
Thanks for the reply unSpawn...

I was just testing and talking out aloud with all the different attempts..

The last one does it nicely don't think I have every used readlink before interesting..

wonders why does this fail in double [[ ?

[[ -d $(readlink -f /folder/*/folder3) ]] && echo "OK"
[ -d $(readlink -f /folder/*/folder3) ] && echo "OK"
OK

Have a good evening
Cheers
hdaz
 
Old 04-26-2013, 01:32 AM   #6
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,310
Blog Entries: 54

Rep: Reputation: 2858Reputation: 2858Reputation: 2858Reputation: 2858Reputation: 2858Reputation: 2858Reputation: 2858Reputation: 2858Reputation: 2858Reputation: 2858Reputation: 2858
Quote:
Originally Posted by hyperdaz View Post
The last one does it nicely don't think I have every used readlink before interesting..
There's common things you should not need to like for example:
cat /file|grep == grep /file
cat /file|sed > /somefile == sed -i /file
ps|grep something == pgrep something
etc, etc.

Script more and you'll find out more.


Quote:
Originally Posted by hyperdaz View Post
wonders why does this fail in double [[ ?
Let's hope one of our resident Bash evangelists like David The H. will be here RSN to answer that. In the meanwhile maybe have a look at some Bash scripting guides:
http://www.tldp.org/HOWTO/Bash-Prog-Intro-HOWTO.html
http://www.tldp.org/LDP/Bash-Beginne...tml/index.html
http://www.gnu.org/software/bash/man...ode/index.html
http://www.grymoire.com/Unix/Sh.html
http://www.tldp.org/LDP/abs/html/
http://mywiki.wooledge.org/BashFAQ
http://mywiki.wooledge.org/BashPitfalls
 
Old 04-26-2013, 03:37 AM   #7
hyperdaz
Member
 
Registered: Sep 2004
Location: UK
Distribution: CentOS 5.5
Posts: 44

Original Poster
Rep: Reputation: 1
Hey UnSapwn,

Unfortunitly after a little testing [ -d $(readlink -f /folder/*/folder3) ] && echo "OK" does not work and here is why...

ls -lrth /folder/folder2/folder3
total 0
-rw-r--r-- 1 root root 0 Apr 26 09:04 file

[ -f $(readlink -f /folder/*/folder3/file) ] && echo "OK"
OK
[ -f $(readlink -f /folder/*/folder3/fie) ] && echo "OK"
OK
[ -d $(readlink -f /folder/*/foler3) ] && echo "OK"
OK

bash -x foldertest.sh
++ readlink -f /folder/folder2/folder3
+ '[' -d /folder/folder2/folder3 ']'
+ echo OK
OK

[root@hostingtest ~]# bash -x foldertest.sh
++ readlink -f '/folder/*/foder3'
+ '[' -d ']'
+ echo OK
OK

[ -d $(readlink -ve /folder/*/foler3) ] && echo "OK"
readlink: /folder/*/foler3: No such file or directory
OK

[ -d $(readlink -vf /folder/*/foler3) ] && echo "OK"
readlink: /folder/*/foler3: No such file or directory
OK

Close but not quite...

Last edited by hyperdaz; 04-26-2013 at 03:43 AM.
 
Old 04-27-2013, 03:45 AM   #8
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,310
Blog Entries: 54

Rep: Reputation: 2858Reputation: 2858Reputation: 2858Reputation: 2858Reputation: 2858Reputation: 2858Reputation: 2858Reputation: 2858Reputation: 2858Reputation: 2858Reputation: 2858
Yeah, my bad, that's because it simply takes the exit value of 'readlink'.
Code:
# The problem with this ITEM is that you DO NOT want to have the script expand it unless necessary.
# Using single quotes keeps BASH from expanding it:
ITEM='directory/*/directory3'

# ... at the expense of violating the rule to avoid 'eval'. The problem is the expanded ITEM can
# potentially hold MULTIPLE values (as in 'mkdir -p directory/{1,2,3}/directory{1,2,3}
#  ; mkdir -p directory/dir\ {1,2,3}/directory{1,2,3}')
# further hampered by IFS splitting.
ITEM=($(eval echo "${ITEM}"))
echo "${#ITEM[@]}"
9

# So the correct way IMHO still would be to anchor it
find directory -type d -name directory3 | while read ITEM; do
 doSomething "${ITEM}"
done

# Then again one of our true BASH evangelists may know a better solution...
I think the essential problem with your script is that it needs structural change and I suggest you post it in full.
 
Old 04-27-2013, 06:01 AM   #9
hyperdaz
Member
 
Registered: Sep 2004
Location: UK
Distribution: CentOS 5.5
Posts: 44

Original Poster
Rep: Reputation: 1
Hey unSpawn,

I just re-read my own posts and realised I answered my own question doh

Just use [[ -n $(ls -lrth /folder/*/folder3/) ]] && echo "success" which works correctly when the folder does not exist.

Maybe this should be an improvement within the handing of bash for [[ -d or -f ]] directives...
unless we are just doing something wrong..

I agree the script could probably could do with restructuring but there are reasons why I wrote it the way I did, I wanted it as concise and compact as possible.... hence looking at commands that would one speed things up and two keep things compact..

unfortunately the script contains a lot of sensitive data so its not really possible to upload it here.

hdaz
 
Old 04-29-2013, 11:00 AM   #10
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947
Please use ***[code][/code]*** tags around your code and data, to preserve the original formatting and to improve readability. Do not use quote tags, bolding, colors, "start/end" lines, or other creative techniques.

I think we need to start by getting a little more info.

First, are you sure you're using bash? What does the shebang on the top line look like?

If it says #!/bin/sh, then you're using the system's posix-based shell, which may be something like dash instead. That could be why you aren't able to use things like [[..]]. Always use #!/bin/bash if you want support for bash-specific built ins.

See here for the main differences between bash and posix shells:
http://mywiki.wooledge.org/Bashism

Actually, I see you using things like "bash -x foldertest.sh". That does specify the script as bash, but it's such a sloppy way to do it. As long as the scripts have proper shebangs, just run them directly with "/path/to/foldertest.sh" and similar.


Second, what do the actual filenames/paths look like? Could you show us an example directory tree and explain more clearly what needs to be tested about it?


The first problem, as UnSpawn mentioned above, is that a globbing pattern of "*/" will expand into a list of all subdirectories inside the one specified. So using "/folder/*/folder3/file" could end up giving you:

/folder/folder2a/folder3/file
/folder/folder2b/folder3/file
/folder/folder2c/folder3/file

You can't run a single test on output like that. Perhaps you could capture the results of the glob expansion into an array, then loop over that to test for possible values. But again, it's going to depend on the details of what you have and what you want.


From what's been posted so far, I agree that the whole code flow probably needs re-writing, particularly since you say that it "contains a lot of sensitive data". A script has any kind of data hard-coded into it all (outside of perhaps a few default variable values at the top) is by definition a poorly written script.

I suggest you go through the script to redact (replace with dummy values) the sensitive parts and then post what you can. We need to see more of the code in context before we can really get down to fixing it.


PS: I see a lot of typos in the above posts; 'foder' instead of 'folder', etc. You'll have to tell us whether this is important to the script or just mistakes in explaining it here.
 
1 members found this post helpful.
Old 04-29-2013, 12:06 PM   #11
hyperdaz
Member
 
Registered: Sep 2004
Location: UK
Distribution: CentOS 5.5
Posts: 44

Original Poster
Rep: Reputation: 1
Dave, Many thanks for the reply sorry you find my output a little sloppy..

Quote:
PS: I see a lot of typos in the above posts; 'foder' instead of 'folder', etc. You'll have to tell us whether this is important to the script or just mistakes in explaining it here.
These are not typos they are showing that if the "*/" is used on
Code:
 [ -d $(readlink -f /folder/*/folder3) ] && echo "OK"
the code always gives a positive "OK" even if the folders are not created, maybe I could of made that clearer, I thought showing the full "bash -x" was quite clear.


My original thought and question was why following fails if the files and folders exists

Code:
[[ -f /folder/*/folder3/file.txt ]] && ehco "success"
[[ -d /folder/*/folder4/folder5 ]] && echo "success"
To me it seems logical that the file directive
Code:
  [[ -f
and the directory directive
Code:
 [[ -d
test conditions should be able to expand and test with a folder structure simular to the following "folder/*/folder3/".

As can be seen on this attempt bash knows the files structure but still fails i.e. the continued commands in this case "ehco "success"" never gets run.

Code:
 [[ -d `/folder/*/folder3/` ]] && ehco "success"
-bash: /folder/folder2/folder3/: is a directory
I can use the following this does gives the correct results if the file/folder exist or does not exist which was the original objective with -f and -d
Code:
 [[ -n $(ls /folder/*/folder3/) ]] && echo "success"
To me its a simple question adding detailed script to the question would probably make the question more complicated.

Cheers
Hdaz

Last edited by hyperdaz; 04-29-2013 at 12:08 PM.
 
Old 04-29-2013, 09:21 PM   #12
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,269

Rep: Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028
Code:
 [[ -d `/folder/*/folder3/` ]] && ehco "success"
1. you've used backquotes there, so its trying to interpret that as a cmd(!). Use single quotes instead
2. there's no such cmd as 'ehco', so that'll fail always.
You must ensure exact correct spelling at all times.
 
Old 04-30-2013, 05:23 AM   #13
hyperdaz
Member
 
Registered: Sep 2004
Location: UK
Distribution: CentOS 5.5
Posts: 44

Original Poster
Rep: Reputation: 1
Hi Chris,

none of these makes any diffence the second command never gets run.

Code:
ls -lrth /folder/folder2/folder3/file
-rw-r--r-- 1 root root 0 Apr 26 09:04 /folder/folder2/folder3/file

[[ -d '/folder/*/folder3/' ]] && echo "success"

[[ -d `/folder/*/folder3/` ]] && echo "success"
-bash: /folder/folder2/folder3/: is a directory

[[ -d "/folder/*/folder3/" ]] && echo "success"
If it ever got to ehco then it would output "-bash: ehco: command not found" but thanks I did not spot that on the original test above.

Cheers
Hdaz
 
Old 05-05-2013, 07:21 AM   #14
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947
Code:
[[ -d '/folder/*/folder3/' ]] && echo "success"
It doesn't really matter what you use here because globbing patterns do not expand inside [[..]] brackets. You're always going to get a negative result because it will try to test the literal string, and there is no directory named "*" in that position. You can use command substitutions, but that just means you have to be sure that it will expand into a string that the test can recognize anyway.

In short you must have an actual, fixed, directory path before you can test for a directory.


In any case, you didn't address any of the important points from my last post.

1) Since you're using double-bracket tests, I'm assuming the availability of bash or ksh, is that correct? Or does this have to be a posix-compatible script, limiting the availability to its less-advanced features?

2) Can you guarantee that the globbing pattern will always expand into a single file or directory? If not, and I maintain that you can't, you'll have to decide what to do if you come across more than one entry. But whatever the case, to do it safely the globbing patterns should be run through a loop that tests each possible value individually.

3) As I mentioned, good coding practice says that code and data should be kept as separate as possible. File and directory names, and any text strings that could change, should be set in variables at the top of the script, or even imported into it from elsewhere. Keep the code clean.

This includes the paths you are trying to test for, so could you please explain clearly what should and shouldn't exist?


So I'm going to ask again for some more background on what you are trying to accomplish. It's still not completely clear to me exactly what you need to test for, and what those tests need to accomplish. Could you please explain the context of your needs in some more detail? What files and directories need to exist, what should not exist, and what should happen if conditions aren't what you expect them to be (e.g. there are extra entries)?

And again, if you could post at least the relevant sections of the actual code, with any sensitive data removed or altered, I'd be happy to help you re-write it to be cleaner and more robust.
 
2 members found this post helpful.
Old 05-06-2013, 03:36 PM   #15
hyperdaz
Member
 
Registered: Sep 2004
Location: UK
Distribution: CentOS 5.5
Posts: 44

Original Poster
Rep: Reputation: 1
Hi David,

1, bash or ksh - keeping things portable might be useful but at the moment 99% of all vms/servers are CentOS.
Code:
 cat folder.sh
#!/usr/bin/env bash
2, more than one entry
That's perfect as I want to catch each any every entry
Code:
 && echo "
is replaced with
Code:
 for in $(find -iname /DIR/path/xyzfile);do xyz;done
As I said I wanted to reduce the time it took for find to do it's work on TB of data in a lot of directories.

3, thats a great point, I do like to try and keep things a separated as possible but in this case not really reinvent.

I have a much better bash project that might be more interesting an useful if you would like to help with that?

Cheers
Hdaz
 
  


Reply

Tags
bash scripting


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
BASH Unzip (and Search) Many Files in Directories dougp23 Programming 4 03-09-2011 01:45 PM
BASH script optimization for testing large number of files instag Programming 24 09-26-2010 11:40 PM
BASH script that removes sub-directories (not just files) but keeps the two newest. kennc Linux - Newbie 2 04-23-2009 01:09 PM
[BASH] how to put .htaccess files automatic in directories sinister1 Programming 16 06-25-2008 09:58 AM
bash ls not coloring files, directories etc mr_smith Red Hat 2 08-30-2006 07:05 AM


All times are GMT -5. The time now is 07:53 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration