LinuxQuestions.org
Did you know LQ has a Linux Hardware Compatibility List?
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 12-31-2010, 01:54 PM   #16
redgshost
LQ Newbie
 
Registered: Dec 2010
Posts: 15

Original Poster
Rep: Reputation: 0

Quote:
Originally Posted by catkin View Post
It may be that dir is detecting what its stdout is going to with something like the isatty() system call.
I am very sorry but I have a very limited knowledge of linux and I don't know what you mean. Could you explain a little more? Do I need to change what I am using as the command?

Thanks

Red
 
Old 01-01-2011, 01:02 AM   #17
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,551
Blog Entries: 28

Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
Mysterious. Is echo * output complete?
 
1 members found this post helpful.
Old 01-01-2011, 06:06 PM   #18
redgshost
LQ Newbie
 
Registered: Dec 2010
Posts: 15

Original Poster
Rep: Reputation: 0
Catkin,

Thank you for the reply. I am really showing my ignorance now, but how do I implement the echo * command? Do I add it as another line after the ls -l command?

Thanks and sorry for the basic question.

Red
 
Old 01-01-2011, 10:08 PM   #19
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,551
Blog Entries: 28

Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
Quote:
Originally Posted by redgshost View Post
I am very sorry but I have a very limited knowledge of linux and I don't know what you mean. Could you explain a little more? Do I need to change what I am using as the command?
OK. It's in the depths of programming; the isatty function when applied to stdout (standard output) asks the kernel whether it is connected to a terminal or not. The ls command probably uses it before finding out the current screen width which it uses when formatting its output. Somewhat esoteric but it is an example of a possible reason "why this part is different in the different situations".

It was a theoretical answer of no practical significance for your issue.

Last edited by catkin; 01-01-2011 at 10:28 PM.
 
1 members found this post helpful.
Old 01-01-2011, 10:27 PM   #20
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,551
Blog Entries: 28

Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
You could try echo * at the command line. In your script you can use it instead of ls -l or as well as ls -l (as another line after). This suggestion is to investigate the problem, not to workaround it.

The * is a shell "filename expansion" expression. It will be expanded by the shell and replaced by the names of all files and directories not beginning with . The echo command will write those names to stdout which you can

The reason for asking you to try it is to try something more basic than dir or ls to show whether the files are visible.

If your ls command has options to change the order files are listed in (like -t, -S, -r) then you could experiment with them and maybe see whether the problem is that ls does not "see" the files or whether its output is is truncated.

At this stage, we are varying the action in the hope of gaining a clearer understanding of the problem.

It's particularly strange that only two of the four very similar "Any Human Heart_<nnnnnn>.rec" files are shown. And ls is pretty robust ... Can you try ls -l > lsl.$$.out when running from cron? That would show whether the problem is with ls itself or somehow in the post-processing of its output. Can you post the part of the script that runs ls, up to emailing its output?
 
1 members found this post helpful.
Old 01-01-2011, 10:39 PM   #21
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,551
Blog Entries: 28

Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
Quote:
Originally Posted by redgshost View Post
The ls-l command leads to a seemingly random list of a subset of the files in the directory. Whilst the subset does appear random it is consistent from one run to the next.
As shown in your later post, the subset may be random but it is also the smallest files, with 2137460736 being the size of the largest file listed and 2171054080 being the size of the smallest file not listed.
 
1 members found this post helpful.
Old 01-03-2011, 05:11 AM   #22
redgshost
LQ Newbie
 
Registered: Dec 2010
Posts: 15

Original Poster
Rep: Reputation: 0
Catkin

what a fantastic observation! I moved some files around and let it run in cron again. The following is the output from the script. Again, only the files with a size below the cut-off you identified are listed with ls -l whilst dir-l lists all files. I confirmed this is all files by ftp'ing to check.

I have a new question now, which is why is there this cut off and is there a way around it?

Quote:
Using dir-l

total 6712704
-rwxr-xr-x 1 root root 2328432640 Jan 3 09:55 Dr\ Seuss'\ Horton\ Hears\ a\ Who!_101218.rec -rwxr-xr-x 1 root root 777426944 Nov 26 18:21 Master\ and\ Commander_101126.rec -rwxr-xr-x 1 root root 3767941120 Jan 3 08:56 The\ Chronicles\ of\ Narnia\ Prince\ Caspian_101224.rec
drwxr-xr-x 1 root root 4096 Aug 12 00:23 avi
drwxr-xr-x 1 root root 4096 Jun 21 2009 mpg


Using '/bin/ls-ld *'

drwxr-xr-x 1 root root 4096 Jan 3 09:55 /mnt/nas1/Films/keep
-rwxr-xr-x 1 root root 777426944 Nov 26 18:21 Master and Commander_101126.rec
drwxr-xr-x 1 root root 4096 Aug 12 00:23 avi
drwxr-xr-x 1 root root 4096 Jun 21 2009 mpg
Thank you very much

Red
 
Old 01-11-2011, 03:08 PM   #23
redgshost
LQ Newbie
 
Registered: Dec 2010
Posts: 15

Original Poster
Rep: Reputation: 0
Catkin

apologies for the pause I was away for a couple of days

Here is the text of the script that does the listing. I've added the parts you suggested but it is due to run in cron later. The text files that are created get turned into the body of an email and sent to me.

Thank you for your help.

Red


Code:
#Get list of files on NAS1

# note time did list at
TIME1=`date "+%Y/%m/%d %H:%M:%S"`

# preamable text
echo "Files in NAS1 '/mnt/nas1/Films/keep' " > /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log
echo "List start time = $TIME1" >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log

echo "Using 'cd /mnt/nas1/Films/keep' " >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log
cd /mnt/nas1/Films/keep

# Check what is in NAS1 Keep and what files are active
echo "Using dir-l" >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log
dir -l /mnt/nas1/Films/keep >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log

echo "Using 'ls-l' " >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log
ls -l /mnt/nas1/Films/keep >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log

echo "Using 'echo *' " >> /public/ftop2nas/nas1keep.log
cd /mnt/nas1/Films/keep
echo >> /public/ftop2nas/nas1keep.log
echo * /mnt/nas1/Films/keep >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log

echo "Using '/bin/ls-l' " >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log
/bin/ls -l /mnt/nas1/Films/keep >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log

echo "Using 'ls -l > lsl.$$.out' " >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log
ls -l > lsl.$$.out /mnt/nas1/Films/keep >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log

# find out the number of the lsl file
echo "Using 'ls-l' to find out number of 'lsl.$$.out file' " >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log
ls -l >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log
echo >> /public/ftop2nas/nas1keep.log
 
Old 01-12-2011, 10:51 AM   #24
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,551
Blog Entries: 28

Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
Hello Red

Quote:
Originally Posted by redgshost View Post
I have a new question now, which is why is there this cut off and is there a way around it?
That I do not know and cannot investigate because I do not use unslung nslu2 but I get the impression it is a cut down version of Linux so may have restrictions. 2 to the power 31 is 214748365 so that may be the cutoff file size but I have no ideas about why ls behaves differently when run from cron or command prompt ... ?
Quote:
Originally Posted by redgshost View Post
apologies for the pause I was away for a couple of days

Here is the text of the script that does the listing. I've added the parts you suggested but it is due to run in cron later. The text files that are created get turned into the body of an email and sent to me.
No worries, I've been busy and not been on LQ much myself recently.

The ls -l > lsl.$$.out /mnt/nas1/Films/keep >> /public/ftop2nas/nas1keep.log line should put nothing in the log because it should all go to lsl.$$.out. In case ls is producing any error messages you could try ls -l > lsl.$$.out /mnt/nas1/Films/keep 2>> /public/ftop2nas/nas1keep.log
 
1 members found this post helpful.
Old 01-14-2011, 02:27 PM   #25
redgshost
LQ Newbie
 
Registered: Dec 2010
Posts: 15

Original Poster
Rep: Reputation: 0
Catkin thanks to the thought about 2^31

I modfified the script to use echo *. This lists all the files that dir pulls up.

I used ls.$$$.out and output the output to the text file. It is included below and behaves as anticipated.

I also modified the script so that ls sorted the files by size and in reverse size order also. Essentially it makes no difference.

It looks like ls is broken on unslung.

Thank you very much for all your help trying to understand what is going on.

Red

Quote:
Using dir-l

total 10226168
-rwxr-xr-x 1 root root 2138609043 Apr 3 2009 Eloge\ de\ l'Amour_080210.mpg
-rwxr-xr-x 1 root root 3309570048 Jan 14 02:25 Oliver_110101.rec
-rwxr-xr-x 1 root root 2871412736 Jan 14 01:07 The\ Adventures\ of\ Baron\ Munchausen_101222.rec
-rwxr-xr-x 1 root root 2151995392 Nov 15 2009 The\ Sixth\ Sense.rec
drwxr-xr-x 1 root root 4096 Aug 12 00:23 avi
drwxr-xr-x 1 root root 4096 Jun 21 2009 mpg


Using 'ls-l'

-rwxr-xr-x 1 root root 2138609043 Apr 3 2009 Eloge de l'Amour_080210.mpg
drwxr-xr-x 1 root root 4096 Aug 12 00:23 avi
drwxr-xr-x 1 root root 4096 Jun 21 2009 mpg


Using 'echo *'

Eloge de l'Amour_080210.mpg Oliver_110101.rec The Adventures of Baron Munchausen_101222.rec The Sixth Sense.rec avi mpg /mnt/nas1/Films/keep


Using 'ls -l > lsl.621.out'

-rwxr-xr-x 1 root root 2138609043 Apr 3 2009 Eloge de l'Amour_080210.mpg
drwxr-xr-x 1 root root 4096 Aug 12 00:23 avi
-rwxr-xr-x 1 root root 0 Jan 14 02:25 lsl.621.out
drwxr-xr-x 1 root root 4096 Jun 21 2009 mpg


Using 'ls-lS'

-rwxr-xr-x 1 root root 2138609043 Apr 3 2009 Eloge de l'Amour_080210.mpg
drwxr-xr-x 1 root root 4096 Aug 12 00:23 avi
drwxr-xr-x 1 root root 4096 Jun 21 2009 mpg
-rwxr-xr-x 1 root root 0 Jan 14 02:25 lsl.621.out


Using 'ls-lSr'

-rwxr-xr-x 1 root root 0 Jan 14 02:25 lsl.621.out
drwxr-xr-x 1 root root 4096 Jun 21 2009 mpg
drwxr-xr-x 1 root root 4096 Aug 12 00:23 avi
-rwxr-xr-x 1 root root 2138609043 Apr 3 2009 Eloge de l'Amour_080210.mpg
 
Old 01-14-2011, 03:16 PM   #26
redgshost
LQ Newbie
 
Registered: Dec 2010
Posts: 15

Original Poster
Rep: Reputation: 0
I have concluded this is probably a unslung slug based problem so I have moved on to an unslung slug specific forum at

http://tech.groups.yahoo.com/group/n...neral/messages

Thank you to catkin for all the help.

Red
 
Old 01-14-2011, 03:31 PM   #27
cin_
Member
 
Registered: Dec 2010
Posts: 266

Rep: Reputation: 23
Environments

Perhaps it is an environment thing... where your standard and cron call different environments.
Test it by writing some scripts with different environments and then having the cron() call these scripts instead.

Code:
#!/bin/bash
ls -l
Code:
#!/bin/sh
ls -l
Code:
#!/bin/csh
ls -l
Code:
#!/bin/tcsh
ls -l
Code:
#!/bin/zsh
ls -l
Code:
#!/bin/ksh
ls -l

Last edited by cin_; 01-14-2011 at 03:39 PM.
 
Old 01-15-2011, 08:08 AM   #28
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,551
Blog Entries: 28

Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
In case anyone wants to follow, here's the specific nslu2 thread. Apparently nslu2 has a /bin/ls with the 2 GB limit and a /opt/bin/ls which does not so this problem is probably a cron $PATH issue.
 
Old 01-16-2011, 04:06 AM   #29
redgshost
LQ Newbie
 
Registered: Dec 2010
Posts: 15

Original Poster
Rep: Reputation: 0
Catkin and cin_

thank you very much. I was dropping back to tidy up with the answer and I saw it had already been posted.

The issue is as cin_ suggested and as catkin outlines. Using /opt/bin/ls in the script sees it work perfectly.

Thank you all very much

Red
 
Old 01-16-2011, 12:14 PM   #30
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,551
Blog Entries: 28

Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
Glad you found a solution in the end

We might have got there if we had focussed more on the cron/terminal aspect than on the bizarre 2 GB limit.

Threads can be marked SOLVED via the Thread Tools menu.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
'E:Malformed line 54 in source list /etc/apt/sources.list (dist parse), E:The list of vsssuccess@gmail.com Linux - Desktop 1 11-17-2010 08:17 AM
Bash script run via cron not executing MYSQL command mackstar Linux - Server 4 04-23-2009 05:01 AM
Bash script can't handle errors when run by cron rosv Programming 4 01-08-2009 07:50 PM
Run bash commands script as cron anjanesh Linux - General 4 04-03-2007 06:25 AM
Script to Run the Same Commands on a List of Files Chryzmo Programming 1 10-03-2004 08:34 AM


All times are GMT -5. The time now is 05:32 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration