LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 10-18-2017, 04:03 AM   #1
ostene
Member
 
Registered: Jul 2006
Location: Lidingo, Sweden
Distribution: Kubuntu
Posts: 54

Rep: Reputation: 1
Use gawk to clean up records fails


Hello,

Have a file (infile.txt) with 500 records,
that consists of mail addresses with records
and ; as fields separators lika this:

;name1@gmail.com;
;name2@gmail.com;

The results should go to an outfile.txt

Tried to make a script that remove the
fields separators. It "should" be an easy
thing to get working, but it seems
impossibly for me to get the script working.
Have read manuals and testing for hours
but no luck. Really feel like a newbie.
I hope someone can write a small script
for me that do the work.

Many thanks for that!
ostene
 
Old 10-18-2017, 04:22 AM   #2
AwesomeMachine
LQ Guru
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora; Mint
Posts: 5,513

Rep: Reputation: 1009Reputation: 1009Reputation: 1009Reputation: 1009Reputation: 1009Reputation: 1009Reputation: 1009Reputation: 1009
It looks like a job for sed. Just escape the ; like this: \;
 
Old 10-18-2017, 04:25 AM   #3
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 19,713

Rep: Reputation: 3549Reputation: 3549Reputation: 3549Reputation: 3549Reputation: 3549Reputation: 3549Reputation: 3549Reputation: 3549Reputation: 3549Reputation: 3549Reputation: 3549
If it is simply removing the semi-colons, as mentioned sed will do fine, but so will awk.
Show us what you tried so we can help with learning gawk maybe.
 
Old 10-18-2017, 05:12 AM   #4
ostene
Member
 
Registered: Jul 2006
Location: Lidingo, Sweden
Distribution: Kubuntu
Posts: 54

Original Poster
Rep: Reputation: 1
Use gawk to clean up records fails

My last try is:

gawk '{-v OFS="\r" -v FS=";" ; print $1 > "outfile.txt" }' infile.txt

and that resulted in this outfile with only 2 records of 500:

;name1@gmail.com;
;name2@gmail.com;

I want to use gawk because after reading on web I think
it is the strongest text manipulating tool.

ostene
 
Old 10-18-2017, 05:57 AM   #5
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 19,713

Rep: Reputation: 3549Reputation: 3549Reputation: 3549Reputation: 3549Reputation: 3549Reputation: 3549Reputation: 3549Reputation: 3549Reputation: 3549Reputation: 3549Reputation: 3549
There is a lot to learn, even in that snippet. That couldn't work, your results are from a prior attempt.
You certainly don't want to use "\r" (a teletype carriage return) as an output separator. That will generate plenty of angst if you have more than one output field.
If you have a field separator as first character, $1 will be null - hence your lack of expected output.
OFS and FS are known variables to gawk - if you set them, they need to be done outside the code block, or in a BEGIN block. Not using -v which is used to pass in shell variables.

Others will have recommendations for tutorials.
 
Old 10-18-2017, 06:02 AM   #6
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,615
Blog Entries: 3

Rep: Reputation: 2871Reputation: 2871Reputation: 2871Reputation: 2871Reputation: 2871Reputation: 2871Reputation: 2871Reputation: 2871Reputation: 2871Reputation: 2871Reputation: 2871
Here is a good tutorial to start with:

http://www.grymoire.com/Unix/Awk.html

Corrected, the line you like you are trying to make is this:

Code:
gawk '{ print $1 }' FS=';' OFS="\n" infile.txt > outfile.txt
However, as mentioned that might not do what you are aiming for.
 
Old 10-28-2017, 04:38 PM   #7
ostene
Member
 
Registered: Jul 2006
Location: Lidingo, Sweden
Distribution: Kubuntu
Posts: 54

Original Poster
Rep: Reputation: 1
What is wrong with this really short script?

Code:
#!/bin/sh

gawk ' {total += $6} Test_file.txt 
END {print total}
'
I got this output:
Code:
gawk: cmd. line:1:  {total += $6} Test_file.txt 
gawk: cmd. line:1:                         ^ syntax error
The file Test_file.txt contains number in the 6th field
in the records, and I want to get the sum of all numbers.

ostene

Last edited by ostene; 11-09-2017 at 08:31 AM.
 
Old 10-28-2017, 06:41 PM   #8
astrogeek
Moderator
 
Registered: Oct 2008
Distribution: Slackware [64]-X.{0|1|2|37|-current} ::12<=X<=14, FreeBSD_12{.0|.1}
Posts: 5,619
Blog Entries: 11

Rep: Reputation: 3657Reputation: 3657Reputation: 3657Reputation: 3657Reputation: 3657Reputation: 3657Reputation: 3657Reputation: 3657Reputation: 3657Reputation: 3657Reputation: 3657
The input file name must be the last argument, outside the quoted gawk program text.

Please place your code snippets inside [CODE]...[/CODE] tags for better readability. You may type those yourself or click the "#" button in the edit controls.

Last edited by astrogeek; 10-28-2017 at 06:45 PM.
 
1 members found this post helpful.
Old 10-29-2017, 03:47 AM   #9
MadeInGermany
Senior Member
 
Registered: Dec 2011
Location: Simplicity
Posts: 1,744

Rep: Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791
(g)awk reads from stdin unless you give input files as additional argument(s).
Code:
awk '
{total += $6}
END {print total}
' Test_file.txt
The first argument in 'ticks' is the awk program, the following argument is the input file.

Last edited by MadeInGermany; 10-29-2017 at 03:49 AM.
 
1 members found this post helpful.
Old 11-09-2017, 08:12 AM   #10
ostene
Member
 
Registered: Jul 2006
Location: Lidingo, Sweden
Distribution: Kubuntu
Posts: 54

Original Poster
Rep: Reputation: 1
Solved

Hello,

After more googling I found:
Code:
www.gnu.org/software/gawk/manual/gawk.html
This superior tutorial got me a clear idea of writing scripts in awk, with infile and outfiles e.g.

Code:
#!/bin/sh

awk 'BEGIN { format = "%-10s %s\n"
             printf format, "Name", "Number" > "Header.txt"
             printf format, "----", "------" > "Underline"}
        { printf format, $1, $2 > "Newfile.txt"}' mail-list
Many thanks guys for the help!
ostene

Last edited by ostene; 11-09-2017 at 08:28 AM.
 
Old 11-09-2017, 11:47 AM   #11
MadeInGermany
Senior Member
 
Registered: Dec 2011
Location: Simplicity
Posts: 1,744

Rep: Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791
Here is a solution to the pitfall in my last post (#9):
if the input file is empty then the variable total is not initialized, it will be treated as an empty string rather than a zero integer.
If you want to force a zero integer you can either initialize it
Code:
awk '
  BEGIN {total=0}
  {total += $6}
  END {print total}
' Test_file.txt
or use printf with an integer format
Code:
awk '
  {total += $6}
  END {printf "%u\n", total}
' Test_file.txt
Or add a zero
Code:
awk '
  {total += $6}
  END {print total+0}
' Test_file.txt
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Clean install boot fails into a shell initramfs JuergenUbuntu Linux - Kernel 2 12-21-2014 05:34 AM
Puppy Lucid 5.2.5 Full -clean- install grub loader fails. WARemick Linux - Newbie 3 05-25-2011 08:32 PM
[SOLVED] MX Records / A Records / CNAME Records - Advice Please fusion1275 Linux - Newbie 15 01-18-2011 04:06 AM
compile fails since clean with mrproper Kristian2 Slackware 1 05-26-2004 07:07 PM
OpenBSD - make clean fails - Newbie DwightDE *BSD 5 11-25-2003 09:42 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 06:55 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration