[SOLVED] Partial list with ls-l in bash script run in cron but full list run from command line

redgshost · 12-31-2010, 01:54 PM

Quote:

Originally Posted by catkin

It may be that dir is detecting what its stdout is going to with something like the isatty() system call.

I am very sorry but I have a very limited knowledge of linux and I don't know what you mean. Could you explain a little more? Do I need to change what I am using as the command?

Thanks

Red

catkin · 01-01-2011, 01:02 AM

Mysterious. Is echo * output complete?

redgshost · 01-01-2011, 06:06 PM

Catkin,

Thank you for the reply. I am really showing my ignorance now, but how do I implement the echo * command? Do I add it as another line after the ls -l command?

Thanks and sorry for the basic question.

Red

catkin · 01-01-2011, 10:08 PM

Quote:

Originally Posted by redgshost

I am very sorry but I have a very limited knowledge of linux and I don't know what you mean. Could you explain a little more? Do I need to change what I am using as the command?

OK. It's in the depths of programming; the isatty function when applied to stdout (standard output) asks the kernel whether it is connected to a terminal or not. The ls command probably uses it before finding out the current screen width which it uses when formatting its output. Somewhat esoteric but it is an example of a possible reason "why this part is different in the different situations".

It was a theoretical answer of no practical significance for your issue.

catkin · 01-01-2011, 10:27 PM

You could try echo * at the command line. In your script you can use it instead of ls -l or as well as ls -l (as another line after). This suggestion is to investigate the problem, not to workaround it.

The * is a shell "filename expansion" expression. It will be expanded by the shell and replaced by the names of all files and directories not beginning with . The echo command will write those names to stdout which you can

The reason for asking you to try it is to try something more basic than dir or ls to show whether the files are visible.

If your ls command has options to change the order files are listed in (like -t, -S, -r) then you could experiment with them and maybe see whether the problem is that ls does not "see" the files or whether its output is is truncated.

At this stage, we are varying the action in the hope of gaining a clearer understanding of the problem.

It's particularly strange that only two of the four very similar "Any Human Heart_<nnnnnn>.rec" files are shown. And ls is pretty robust ...

Can you try ls -l > lsl.$$.out when running from cron? That would show whether the problem is with ls itself or somehow in the post-processing of its output. Can you post the part of the script that runs ls, up to emailing its output?

catkin · 01-01-2011, 10:39 PM

Quote:

Originally Posted by redgshost

The ls-l command leads to a seemingly random list of a subset of the files in the directory. Whilst the subset does appear random it is consistent from one run to the next.

As shown in your later post, the subset may be random but it is also the smallest files, with 2137460736 being the size of the largest file listed and 2171054080 being the size of the smallest file not listed.

redgshost · 01-03-2011, 05:11 AM

Catkin

what a fantastic observation! I moved some files around and let it run in cron again. The following is the output from the script. Again, only the files with a size below the cut-off you identified are listed with ls -l whilst dir-l lists all files. I confirmed this is all files by ftp'ing to check.

I have a new question now, which is why is there this cut off and is there a way around it?

Quote:

Using dir-l

total 6712704
-rwxr-xr-x 1 root root 2328432640 Jan 3 09:55 Dr\ Seuss'\ Horton\ Hears\ a\ Who!_101218.rec -rwxr-xr-x 1 root root 777426944 Nov 26 18:21 Master\ and\ Commander_101126.rec -rwxr-xr-x 1 root root 3767941120 Jan 3 08:56 The\ Chronicles\ of\ Narnia\ Prince\ Caspian_101224.rec
drwxr-xr-x 1 root root 4096 Aug 12 00:23 avi
drwxr-xr-x 1 root root 4096 Jun 21 2009 mpg

Using '/bin/ls-ld *'

drwxr-xr-x 1 root root 4096 Jan 3 09:55 /mnt/nas1/Films/keep
-rwxr-xr-x 1 root root 777426944 Nov 26 18:21 Master and Commander_101126.rec
drwxr-xr-x 1 root root 4096 Aug 12 00:23 avi
drwxr-xr-x 1 root root 4096 Jun 21 2009 mpg

Thank you very much

Red

redgshost · 01-11-2011, 03:08 PM

Catkin

apologies for the pause I was away for a couple of days

Here is the text of the script that does the listing. I've added the parts you suggested but it is due to run in cron later. The text files that are created get turned into the body of an email and sent to me.

Thank you for your help.

Red

Code:

#Get list of files on NAS1

# note time did list at
TIME1=`date "+%Y/%m/%d %H:%M:%S"`

# preamable text
echo "Files in NAS1 '/mnt/nas1/Films/keep' " > /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log
echo "List start time = $TIME1" >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log

echo "Using 'cd /mnt/nas1/Films/keep' " >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log
cd /mnt/nas1/Films/keep

# Check what is in NAS1 Keep and what files are active
echo "Using dir-l" >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log
dir -l /mnt/nas1/Films/keep >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log

echo "Using 'ls-l' " >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log
ls -l /mnt/nas1/Films/keep >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log

echo "Using 'echo *' " >> /public/ftop2nas/nas1keep.log
cd /mnt/nas1/Films/keep
echo >> /public/ftop2nas/nas1keep.log
echo * /mnt/nas1/Films/keep >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log

echo "Using '/bin/ls-l' " >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log
/bin/ls -l /mnt/nas1/Films/keep >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log

echo "Using 'ls -l > lsl.$$.out' " >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log
ls -l > lsl.$$.out /mnt/nas1/Films/keep >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log

# find out the number of the lsl file
echo "Using 'ls-l' to find out number of 'lsl.$$.out file' " >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log
ls -l >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log

catkin · 01-12-2011, 10:51 AM

Hello Red

Quote:

Originally Posted by redgshost

I have a new question now, which is why is there this cut off and is there a way around it?

That I do not know and cannot investigate because I do not use unslung nslu2 but I get the impression it is a cut down version of Linux so may have restrictions. 2 to the power 31 is 214748365 so that may be the cutoff file size but I have no ideas about why ls behaves differently when run from cron or command prompt ... ?

Quote:

Originally Posted by redgshost

apologies for the pause I was away for a couple of days

Here is the text of the script that does the listing. I've added the parts you suggested but it is due to run in cron later. The text files that are created get turned into the body of an email and sent to me.

No worries, I've been busy and not been on LQ much myself recently.

The ls -l > lsl.$$.out /mnt/nas1/Films/keep >> /public/ftop2nas/nas1keep.log line should put nothing in the log because it should all go to lsl.$$.out. In case ls is producing any error messages you could try ls -l > lsl.$$.out /mnt/nas1/Films/keep 2>> /public/ftop2nas/nas1keep.log

redgshost · 01-14-2011, 02:27 PM

Catkin thanks to the thought about 2^31

I modfified the script to use echo *. This lists all the files that dir pulls up.

I used ls.$$$.out and output the output to the text file. It is included below and behaves as anticipated.

I also modified the script so that ls sorted the files by size and in reverse size order also. Essentially it makes no difference.

It looks like ls is broken on unslung.

Thank you very much for all your help trying to understand what is going on.

Red

Quote:

Using dir-l

total 10226168
-rwxr-xr-x 1 root root 2138609043 Apr 3 2009 Eloge\ de\ l'Amour_080210.mpg
-rwxr-xr-x 1 root root 3309570048 Jan 14 02:25 Oliver_110101.rec
-rwxr-xr-x 1 root root 2871412736 Jan 14 01:07 The\ Adventures\ of\ Baron\ Munchausen_101222.rec
-rwxr-xr-x 1 root root 2151995392 Nov 15 2009 The\ Sixth\ Sense.rec
drwxr-xr-x 1 root root 4096 Aug 12 00:23 avi
drwxr-xr-x 1 root root 4096 Jun 21 2009 mpg

Using 'ls-l'

-rwxr-xr-x 1 root root 2138609043 Apr 3 2009 Eloge de l'Amour_080210.mpg
drwxr-xr-x 1 root root 4096 Aug 12 00:23 avi
drwxr-xr-x 1 root root 4096 Jun 21 2009 mpg

Using 'echo *'

Eloge de l'Amour_080210.mpg Oliver_110101.rec The Adventures of Baron Munchausen_101222.rec The Sixth Sense.rec avi mpg /mnt/nas1/Films/keep

Using 'ls -l > lsl.621.out'

-rwxr-xr-x 1 root root 2138609043 Apr 3 2009 Eloge de l'Amour_080210.mpg
drwxr-xr-x 1 root root 4096 Aug 12 00:23 avi
-rwxr-xr-x 1 root root 0 Jan 14 02:25 lsl.621.out
drwxr-xr-x 1 root root 4096 Jun 21 2009 mpg

Using 'ls-lS'

-rwxr-xr-x 1 root root 2138609043 Apr 3 2009 Eloge de l'Amour_080210.mpg
drwxr-xr-x 1 root root 4096 Aug 12 00:23 avi
drwxr-xr-x 1 root root 4096 Jun 21 2009 mpg
-rwxr-xr-x 1 root root 0 Jan 14 02:25 lsl.621.out

Using 'ls-lSr'

-rwxr-xr-x 1 root root 0 Jan 14 02:25 lsl.621.out
drwxr-xr-x 1 root root 4096 Jun 21 2009 mpg
drwxr-xr-x 1 root root 4096 Aug 12 00:23 avi
-rwxr-xr-x 1 root root 2138609043 Apr 3 2009 Eloge de l'Amour_080210.mpg

redgshost · 01-14-2011, 03:16 PM

I have concluded this is probably a unslung slug based problem so I have moved on to an unslung slug specific forum at

http://tech.groups.yahoo.com/group/n...neral/messages

Thank you to catkin for all the help.

Red

cin_ · 01-14-2011, 03:31 PM

Perhaps it is an environment thing... where your standard and cron call different environments.
Test it by writing some scripts with different environments and then having the cron() call these scripts instead.

Code:

#!/bin/bash
ls -l

Code:

#!/bin/sh
ls -l

Code:

#!/bin/csh
ls -l

Code:

#!/bin/tcsh
ls -l

Code:

#!/bin/zsh
ls -l

Code:

#!/bin/ksh
ls -l

catkin · 01-15-2011, 08:08 AM

In case anyone wants to follow, here's the specific nslu2 thread. Apparently nslu2 has a /bin/ls with the 2 GB limit and a /opt/bin/ls which does not so this problem is probably a cron $PATH issue.

redgshost · 01-16-2011, 04:06 AM

Catkin and cin_

thank you very much. I was dropping back to tidy up with the answer and I saw it had already been posted.

The issue is as cin_ suggested and as catkin outlines. Using /opt/bin/ls in the script sees it work perfectly.

Thank you all very much

Red

catkin · 01-16-2011, 12:14 PM

Glad you found a solution in the end

We might have got there if we had focussed more on the cron/terminal aspect than on the bizarre 2 GB limit.

Threads can be marked SOLVED via the Thread Tools menu.