LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 05-07-2017, 09:09 AM   #1
NevemTeve
Senior Member
 
Registered: Oct 2011
Location: Budapest
Distribution: Debian/GNU/Linux, AIX
Posts: 3,503

Rep: Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080
Frequently Repeated Question/Request/Problems


Well, it isn't about a problem, it is a FAQ. (But if it offends someone, or breaks some rule, please ask a moderator to delete it).
 
Old 05-07-2017, 09:23 AM   #2
NevemTeve
Senior Member
 
Registered: Oct 2011
Location: Budapest
Distribution: Debian/GNU/Linux, AIX
Posts: 3,503

Original Poster
Rep: Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080
Q: How could I debug my script? It produces no output; it runs from cron/at/icon/batch/etc! It runs in one context, fails in another.
A: Yes, you should debug your script, here is a simple way: add these lines into the beginning of your script (after the very first line):
Code:
exec >>/tmp/debug.$$.log 2>&1
set -xv

ulimit -a
locale
date
printf 'SHELL=%s\n' "$SHELL"
printf 'SHELL-FLAGS=%s\n' "$-"
printf 'PATH=%s\n' "$PATH"
printf 'PWD=%s\n' "$PWD"
Whenever you run you script, you will get a file in /tmp that contains every executed commands (twice, to be precise: before and after execution) Also add additional debug-prints to find out more about the context your script runs in.

Plus, be careful when you set the first line: '/bin/sh' and '/bin/bash' are different programs, they are (most likely) incompatible.

Note: You might not understand every bits (eg SHELL-FLAGS, locale or ulimit), but you can read manuals or do web-search about them; or, if you ask for help, you can at least quote these bits of information.

Another note: For non-interactive background processes you might want to prevent terminal-related features. One way could be to add this command into your script:
Code:
unset TERM
Edit: there is also shopt that might change your script's behaviour.

Last edited by NevemTeve; 05-11-2017 at 11:35 AM.
 
1 members found this post helpful.
Old 05-07-2017, 09:32 AM   #3
NevemTeve
Senior Member
 
Registered: Oct 2011
Location: Budapest
Distribution: Debian/GNU/Linux, AIX
Posts: 3,503

Original Poster
Rep: Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080
Q: I could do something with a specific file, but I have to process multiple files!
A: That's what find(1) is good for, examples:
simple command, that supports multiple parameters:
Code:
find . -name '*.jpg' -type f -print0 | xargs -0 chmod 0644
simple command, that doesn't supports multiple parameters:
Code:
find . -name '*.lst' -type f -exec gzip {} \;
something difficult:
Code:
find . -maxdepth 3 -name '*avi' -type f | while read -r FileName; do
    DirName="$(dirname "$FileName")"
    BaseName="$(basename "$FileName")"
    printf 'Now please do something with %s/%s\n' "$DirName" "$BaseName"
done

Last edited by NevemTeve; 05-16-2017 at 02:51 PM.
 
1 members found this post helpful.
Old 05-07-2017, 12:20 PM   #4
dugan
LQ Guru
 
Registered: Nov 2003
Location: Canada
Distribution: Slackware
Posts: 8,228

Rep: Reputation: 3343Reputation: 3343Reputation: 3343Reputation: 3343Reputation: 3343Reputation: 3343Reputation: 3343Reputation: 3343Reputation: 3343Reputation: 3343Reputation: 3343
I'd suggest reproducing this on the LQ Wiki, once we have some good answers.
 
1 members found this post helpful.
Old 05-08-2017, 04:36 AM   #5
NevemTeve
Senior Member
 
Registered: Oct 2011
Location: Budapest
Distribution: Debian/GNU/Linux, AIX
Posts: 3,503

Original Poster
Rep: Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080
Q: How can I perform trivial set-operations on files such as union or intersection?
A: Your files have to be sorted, without duplications (sets don't have duplicated elements). Also sort-order depends on LC_COLLATE, so use the same LC_COLLATE in every operations.

Code:
A U B: sort -u a.txt b.txt >c.txt
A ∩ B: comm -12 a.txt b.txt >c.txt
A \ B: comm -23 a.txt b.txt >c.txt
B \ A: comm -13 a.txt b.txt >c.txt
Q: What about non-trivial operations (e.g. when there's key and data fields in the files)?
A: I suggest you use an actual programming language to solve this; perhaps a script-language like AWK, Perl, PHP, Python etc.

Last edited by NevemTeve; 05-08-2017 at 04:37 AM.
 
1 members found this post helpful.
Old 06-22-2017, 08:20 AM   #6
NevemTeve
Senior Member
 
Registered: Oct 2011
Location: Budapest
Distribution: Debian/GNU/Linux, AIX
Posts: 3,503

Original Poster
Rep: Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080
Q: I got a file from Dos/Windows, and I cannot properly use it.
A: That might be because of the line-ends: it is CRLF (\r\n or 0D0A) in WinDos; single LF (\n 0A) in Unix.

Q: So how do I remove the CR characters?
A: There are programs for that like dos2unix or rmcr (here is one), or just use tr:
Code:
tr -d '\r' <windos.file >unix.file
 
1 members found this post helpful.
Old 06-22-2017, 11:26 AM   #7
scasey
Senior Member
 
Registered: Feb 2013
Location: Tucson, AZ, USA
Distribution: CentOS 7.4
Posts: 1,204

Rep: Reputation: 404Reputation: 404Reputation: 404Reputation: 404Reputation: 404
Quote:
Originally Posted by NevemTeve View Post
Plus, be careful when you set the first line: '/bin/sh' and '/bin/bash' are different programs, they are (most likely) incompatible.
Generally speaking these are good ideas and might serve well in a FAQ, and, yes, sh may not be the same as bash, but:
Code:
$ which bash
/bin/bash
$ which sh
/bin/sh
$ ls -l /bin/sh
lrwxrwxrwx 1 root root 4 Oct  1  2014 /bin/sh -> bash
$
I don't know if that's true everywhere, but it is on my CentOS server.
 
1 members found this post helpful.
Old 10-21-2017, 12:29 AM   #8
NevemTeve
Senior Member
 
Registered: Oct 2011
Location: Budapest
Distribution: Debian/GNU/Linux, AIX
Posts: 3,503

Original Poster
Rep: Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080
Note: Method 3 in post#3 is not suitable if you wish to modify files or create/remove files. In this case, I suggest putting the new/modified files into a different directory. An example:
Code:
#!/bin/sh

FromDir=fromhere
ToDir=tothere

if [ ! -d "$FromDir" ]; then exit; fi

(cd -- "$FromDir"; find . -type f -name '*.java') |
while read RelName; do
    RelDir=$(dirname -- "$RelName")
    InDname="$FromDir/$RelDir"
    OutDname="$ToDir/$RelDir"
    Fname=$(basename -- "$RelName")

    mkdir -p -- "$OutDname"
    printf -- '%s/%s ---> %s/%s\n' "$InDname" "$Fname" "$OutDname" "$Fname"
    sed 's/ /@/g' "$InDname/$Fname" >"$OutDname/$Fname"
done

Last edited by NevemTeve; 10-22-2017 at 07:30 AM. Reason: Corrections suggested by MadeInGermany
 
Old 10-21-2017, 01:17 PM   #9
MadeInGermany
Member
 
Registered: Dec 2011
Location: Simplicity
Posts: 783

Rep: Reputation: 341Reputation: 341Reputation: 341Reputation: 341
The find will go terribly wrong if the cd failed.
Simple fix, a && or an if-then-fi
Code:
cd -- "$FromDir" && find ...
Code:
if cd -- "$FromDir"; then find ...; fi
--
Code:
find . -name '*.jpg' -type f -print0 | xargs -0 chmod 0644
is old (GNU find and xargs) style.
A recent find efficiently handles the same with
Code:
find . -name '*.jpg' -type f -exec chmod 0644 {} +

Last edited by MadeInGermany; 10-21-2017 at 01:20 PM.
 
Old 12-28-2017, 11:52 PM   #10
NevemTeve
Senior Member
 
Registered: Oct 2011
Location: Budapest
Distribution: Debian/GNU/Linux, AIX
Posts: 3,503

Original Poster
Rep: Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080
Q: In shell, what is the compatible format of 'echo'?
A: It's 'printf':
Code:
printf -- '%s\n' "String: Literal and/or $Variable"
Q: Why the '--' part?
A: To prevent getopt[_long] interpreting the string as option. (Your printf might use GNU!getopt, which might search for options even after a non-option like '%s'.)

Last edited by NevemTeve; 12-29-2017 at 02:27 AM.
 
Old 02-12-2018, 05:59 AM   #11
NevemTeve
Senior Member
 
Registered: Oct 2011
Location: Budapest
Distribution: Debian/GNU/Linux, AIX
Posts: 3,503

Original Poster
Rep: Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080
Another method to post #8: Work in two passes: pass1: collect the commands you will execute in pass2; pass2: execute them.
Example: replace symbolic links with the pointed files.

Code:
#!/bin/bash

find . -type l -xtype f | while read -r l; do
    Dir="$(dirname "$l")";     DirSec="$(sed 's;["\\];\\&;g'    <<<"$Dir")"
    Base="$(basename "$l")";   BaseSec="$(sed 's;["\\];\\&;g'   <<<"$Base")"
    Target=$(readlink "$l");   TargetSec="$(sed 's;["\\];\\&;g' <<<"$Target")"
##  printf 'Dir=%s Base=%s Target=%s\n' "$Dir" "$Base" "$Target"
    printf '(cd -- "%s"; rm -- "%s"; cp -p -- "%s" "%s")\n' "$DirSec" "$BaseSec" "$TargetSec" "$BaseSec"
done >pass2.sh

. pass2.sh

Last edited by NevemTeve; 02-12-2018 at 10:41 PM.
 
Old 02-12-2018, 01:30 PM   #12
NoStressHQ
Member
 
Registered: Apr 2010
Location: Geneva - Switzerland ( Bordeaux - France / Montreal - QC - Canada)
Distribution: Slackware 14.2 - 32/64bit
Posts: 606

Rep: Reputation: 218Reputation: 218Reputation: 218
Quote:
Originally Posted by scasey View Post
Generally speaking these are good ideas and might serve well in a FAQ, and, yes, sh may not be the same as bash, but:
I don't know if that's true everywhere, but it is on my CentOS server.
As man says, bash behaves differently depending on which name is used to call it... If it's called "as" sh, it runs in "sh" compatibility mode.
 
2 members found this post helpful.
Old 02-13-2018, 06:41 AM   #13
NevemTeve
Senior Member
 
Registered: Oct 2011
Location: Budapest
Distribution: Debian/GNU/Linux, AIX
Posts: 3,503

Original Poster
Rep: Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080
Q: Speaking of bash, what determines whether or not I get error messages about SIGPIPE?
A: Two things: bash compile-time setting DONT_REPORT_SIGPIPE and run-time setting pipefail.

Q: By the way, what is pipefail good for?
A: It will help with errexit, if you have a redirection to 'tee' in effect:
Code:
set -o errexit
set -o pipefail
make all 2>&1 | tee log.make.all
make install 2>&1 | tee log.make.install
echo 'We have reached this point, so both make was successfull'
 
Old 02-13-2018, 03:04 PM   #14
scasey
Senior Member
 
Registered: Feb 2013
Location: Tucson, AZ, USA
Distribution: CentOS 7.4
Posts: 1,204

Rep: Reputation: 404Reputation: 404Reputation: 404Reputation: 404Reputation: 404
Quote:
Originally Posted by NoStressHQ View Post
As man says, bash behaves differently depending on which name is used to call it... If it's called "as" sh, it runs in "sh" compatibility mode.
Ahh. Makes sense, thinking about it, 'cause bash is an "extended" (Born Again) sh.

Thank for the elucidation!
 
Old 03-16-2018, 07:55 AM   #15
NevemTeve
Senior Member
 
Registered: Oct 2011
Location: Budapest
Distribution: Debian/GNU/Linux, AIX
Posts: 3,503

Original Poster
Rep: Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080Reputation: 1080
Q: Back to shell-script: $* and "$*" and "$@" are synonyms?
A: Certainly not. Here's a little test-script:
Code:
#!/bin/sh

for i in  $* ; do printf -- '[%s]\n' "$i"; done
for i in "$*"; do printf -- '[%s]\n' "$i"; done
for i in "$@"; do printf -- '[%s]\n' "$i"; done
Results:
Code:
a$ ./echotest.sh 'a b c' "alfa beta"
-- $* --
[a]
[b]
[c]
[alfa]
[beta]

-- "$*" --
[a b c alfa beta]

-- "$@" --
[a b c]
[alfa beta]
So usually you will need "$@" to get the parameters one-by-one (even if there's spaces in them).

Q: I often see scripts like this:
Code:
for File in `ls`; do ... done
Can that be wrong, too?
A: Yes, it is an anti-pattern. Instead:
Code:
for File in *; do ... done

Last edited by NevemTeve; 03-16-2018 at 08:08 AM.
 
  


Reply

Tags
faq


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Question: Incremental backups of large files that change frequently haertig Linux - Software 8 09-21-2013 08:27 PM
Begging: Repeated Argus question... how do I get it installed properly? Kayone Linux - Newbie 9 11-19-2009 08:55 AM
Repeated Disk Failure Question bercikr Linux - Hardware 2 11-25-2008 06:52 PM
Repeated RAID 5 Remount Problems After Power Failure HalNineThousand Linux - General 3 01-03-2006 01:18 PM
Repeated request to log in librano LQ Suggestions & Feedback 7 12-18-2005 08:55 AM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 03:35 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration