LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 05-07-2017, 10:09 AM   #1
NevemTeve
Senior Member
 
Registered: Oct 2011
Location: Budapest
Distribution: Debian/GNU/Linux, AIX
Posts: 3,300

Rep: Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006
Frequently Repeated Question/Request/Problems


Well, it isn't about a problem, it is a FAQ. (But if it offends someone, or breaks some rule, please ask a moderator to delete it).
 
Old 05-07-2017, 10:23 AM   #2
NevemTeve
Senior Member
 
Registered: Oct 2011
Location: Budapest
Distribution: Debian/GNU/Linux, AIX
Posts: 3,300

Original Poster
Rep: Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006
Q: How could I debug my script? It produces no output; it runs from cron/at/icon/batch/etc! It runs in one context, fails in another.
A: Yes, you should debug your script, here is a simple way: add these lines into the beginning of your script (after the very first line):
Code:
exec >>/tmp/debug.$$.log 2>&1
set -xv

ulimit -a
locale
date
printf 'SHELL=%s\n' "$SHELL"
printf 'SHELL-FLAGS=%s\n' "$-"
printf 'PATH=%s\n' "$PATH"
printf 'PWD=%s\n' "$PWD"
Whenever you run you script, you will get a file in /tmp that contains every executed commands (twice, to be precise: before and after execution) Also add additional debug-prints to find out more about the context your script runs in.

Plus, be careful when you set the first line: '/bin/sh' and '/bin/bash' are different programs, they are (most likely) incompatible.

Note: You might not understand every bits (eg SHELL-FLAGS, locale or ulimit), but you can read manuals or do web-search about them; or, if you ask for help, you can at least quote these bits of information.

Another note: For non-interactive background processes you might want to prevent terminal-related features. One way could be to add this command into your script:
Code:
unset TERM
Edit: there is also shopt that might change your script's behaviour.

Last edited by NevemTeve; 05-11-2017 at 12:35 PM.
 
1 members found this post helpful.
Old 05-07-2017, 10:32 AM   #3
NevemTeve
Senior Member
 
Registered: Oct 2011
Location: Budapest
Distribution: Debian/GNU/Linux, AIX
Posts: 3,300

Original Poster
Rep: Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006
Q: I could do something with a specific file, but I have to process multiple files!
A: That's what find(1) is good for, examples:
simple command, that supports multiple parameters:
Code:
find . -name '*.jpg' -type f -print0 | xargs -0 chmod 0644
simple command, that doesn't supports multiple parameters:
Code:
find . -name '*.lst' -type f -exec gzip {} \;
something difficult:
Code:
find . -maxdepth 3 -name '*avi' -type f | while read -r FileName; do
    DirName="$(dirname "$FileName")"
    BaseName="$(basename "$FileName")"
    printf 'Now please do something with %s/%s\n' "$DirName" "$BaseName"
done

Last edited by NevemTeve; 05-16-2017 at 03:51 PM.
 
1 members found this post helpful.
Old 05-07-2017, 01:20 PM   #4
dugan
LQ Guru
 
Registered: Nov 2003
Location: Canada
Distribution: Slackware
Posts: 7,620

Rep: Reputation: 2948Reputation: 2948Reputation: 2948Reputation: 2948Reputation: 2948Reputation: 2948Reputation: 2948Reputation: 2948Reputation: 2948Reputation: 2948Reputation: 2948
I'd suggest reproducing this on the LQ Wiki, once we have some good answers.
 
1 members found this post helpful.
Old 05-08-2017, 05:36 AM   #5
NevemTeve
Senior Member
 
Registered: Oct 2011
Location: Budapest
Distribution: Debian/GNU/Linux, AIX
Posts: 3,300

Original Poster
Rep: Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006
Q: How can I perform trivial set-operations on files such as union or intersection?
A: Your files have to be sorted, without duplications (sets don't have duplicated elements). Also sort-order depends on LC_COLLATE, so use the same LC_COLLATE in every operations.

Code:
A U B: sort -u a.txt b.txt >c.txt
A ∩ B: comm -12 a.txt b.txt >c.txt
A \ B: comm -23 a.txt b.txt >c.txt
B \ A: comm -13 a.txt b.txt >c.txt
Q: What about non-trivial operations (e.g. when there's key and data fields in the files)?
A: I suggest you use an actual programming language to solve this; perhaps a script-language like AWK, Perl, PHP, Python etc.

Last edited by NevemTeve; 05-08-2017 at 05:37 AM.
 
1 members found this post helpful.
Old 06-22-2017, 09:20 AM   #6
NevemTeve
Senior Member
 
Registered: Oct 2011
Location: Budapest
Distribution: Debian/GNU/Linux, AIX
Posts: 3,300

Original Poster
Rep: Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006
Q: I got a file from Dos/Windows, and I cannot properly use it.
A: That might be because of the line-ends: it is CRLF (\r\n or 0D0A) in WinDos; single LF (\n 0A) in Unix.

Q: So how do I remove the CR characters?
A: There are programs for that like dos2unix or rmcr (here is one), or just use tr:
Code:
tr -d '\r' <windos.file >unix.file
 
1 members found this post helpful.
Old 06-22-2017, 12:26 PM   #7
scasey
Member
 
Registered: Feb 2013
Location: Tucson, AZ, USA
Distribution: CentOS 7.4
Posts: 380

Rep: Reputation: 126Reputation: 126
Quote:
Originally Posted by NevemTeve View Post
Plus, be careful when you set the first line: '/bin/sh' and '/bin/bash' are different programs, they are (most likely) incompatible.
Generally speaking these are good ideas and might serve well in a FAQ, and, yes, sh may not be the same as bash, but:
Code:
$ which bash
/bin/bash
$ which sh
/bin/sh
$ ls -l /bin/sh
lrwxrwxrwx 1 root root 4 Oct  1  2014 /bin/sh -> bash
$
I don't know if that's true everywhere, but it is on my CentOS server.
 
1 members found this post helpful.
Old 10-21-2017, 01:29 AM   #8
NevemTeve
Senior Member
 
Registered: Oct 2011
Location: Budapest
Distribution: Debian/GNU/Linux, AIX
Posts: 3,300

Original Poster
Rep: Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006
Note: Method 3 in post#3 is not suitable if you wish to modify files or create/remove files. In this case, I suggest putting the new/modified files into a different directory. An example:
Code:
#!/bin/sh

FromDir=fromhere
ToDir=tothere

if [ ! -d "$FromDir" ]; then exit; fi

(cd -- "$FromDir"; find . -type f -name '*.java') |
while read RelName; do
    RelDir=$(dirname -- "$RelName")
    InDname="$FromDir/$RelDir"
    OutDname="$ToDir/$RelDir"
    Fname=$(basename -- "$RelName")

    mkdir -p -- "$OutDname"
    printf -- '%s/%s ---> %s/%s\n' "$InDname" "$Fname" "$OutDname" "$Fname"
    sed 's/ /@/g' "$InDname/$Fname" >"$OutDname/$Fname"
done

Last edited by NevemTeve; 10-22-2017 at 08:30 AM. Reason: Corrections suggested by MadeInGermany
 
Old 10-21-2017, 02:17 PM   #9
MadeInGermany
Member
 
Registered: Dec 2011
Location: Simplicity
Posts: 503

Rep: Reputation: 235Reputation: 235Reputation: 235
The find will go terribly wrong if the cd failed.
Simple fix, a && or an if-then-fi
Code:
cd -- "$FromDir" && find ...
Code:
if cd -- "$FromDir"; then find ...; fi
--
Code:
find . -name '*.jpg' -type f -print0 | xargs -0 chmod 0644
is old (GNU find and xargs) style.
A recent find efficiently handles the same with
Code:
find . -name '*.jpg' -type f -exec chmod 0644 {} +

Last edited by MadeInGermany; 10-21-2017 at 02:20 PM.
 
  


Reply

Tags
faq


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Question: Incremental backups of large files that change frequently haertig Linux - Software 8 09-21-2013 09:27 PM
Begging: Repeated Argus question... how do I get it installed properly? Kayone Linux - Newbie 9 11-19-2009 09:55 AM
Repeated Disk Failure Question bercikr Linux - Hardware 2 11-25-2008 07:52 PM
Repeated RAID 5 Remount Problems After Power Failure HalNineThousand Linux - General 3 01-03-2006 02:18 PM
Repeated request to log in librano LQ Suggestions & Feedback 7 12-18-2005 09:55 AM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 09:05 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration