Use gawk to clean up records fails

ostene · 10-18-2017, 04:03 AM

Hello,

Have a file (infile.txt) with 500 records,
that consists of mail addresses with records
and ; as fields separators lika this:

;name1@gmail.com;
;name2@gmail.com;

The results should go to an outfile.txt

Tried to make a script that remove the
fields separators. It "should" be an easy
thing to get working, but it seems
impossibly for me to get the script working.
Have read manuals and testing for hours
but no luck. Really feel like a newbie.
I hope someone can write a small script
for me that do the work.

Many thanks for that!
ostene

AwesomeMachine · 10-18-2017, 04:22 AM

It looks like a job for sed. Just escape the ; like this: \;

syg00 · 10-18-2017, 04:25 AM

If it is simply removing the semi-colons, as mentioned sed will do fine, but so will awk.
Show us what you tried so we can help with learning gawk maybe.

ostene · 10-18-2017, 05:12 AM

My last try is:

gawk '{-v OFS="\r" -v FS=";" ; print $1 > "outfile.txt" }' infile.txt

and that resulted in this outfile with only 2 records of 500:

;name1@gmail.com;
;name2@gmail.com;

I want to use gawk because after reading on web I think
it is the strongest text manipulating tool.

ostene

syg00 · 10-18-2017, 05:57 AM

There is a lot to learn, even in that snippet. That couldn't work, your results are from a prior attempt.
You certainly don't want to use "\r" (a teletype carriage return) as an output separator. That will generate plenty of angst if you have more than one output field.
If you have a field separator as first character, $1 will be null - hence your lack of expected output.
OFS and FS are known variables to gawk - if you set them, they need to be done outside the code block, or in a BEGIN block. Not using -v which is used to pass in shell variables.

Others will have recommendations for tutorials.

Turbocapitalist · 10-18-2017, 06:02 AM

Here is a good tutorial to start with:

http://www.grymoire.com/Unix/Awk.html

Corrected, the line you like you are trying to make is this:

Code:

gawk '{ print $1 }' FS=';' OFS="\n" infile.txt > outfile.txt

However, as mentioned that might not do what you are aiming for.

ostene · 10-28-2017, 04:38 PM

What is wrong with this really short script?

Code:

#!/bin/sh

gawk ' {total += $6} Test_file.txt 
END {print total}
'

I got this output:

Code:

gawk: cmd. line:1:  {total += $6} Test_file.txt 
gawk: cmd. line:1:                         ^ syntax error

The file Test_file.txt contains number in the 6th field
in the records, and I want to get the sum of all numbers.

ostene

astrogeek · 10-28-2017, 06:41 PM

The input file name must be the last argument, outside the quoted gawk program text.

Please place your code snippets inside [CODE]...[/CODE] tags for better readability. You may type those yourself or click the "#" button in the edit controls.

MadeInGermany · 10-29-2017, 03:47 AM

(g)awk reads from stdin unless you give input files as additional argument(s).

Code:

awk '
{total += $6}
END {print total}
' Test_file.txt

The first argument in 'ticks' is the awk program, the following argument is the input file.

ostene · 11-09-2017, 08:12 AM

Hello,

After more googling I found:

Code:

www.gnu.org/software/gawk/manual/gawk.html

This superior tutorial got me a clear idea of writing scripts in awk, with infile and outfiles e.g.

Code:

#!/bin/sh

awk 'BEGIN { format = "%-10s %s\n"
             printf format, "Name", "Number" > "Header.txt"
             printf format, "----", "------" > "Underline"}
        { printf format, $1, $2 > "Newfile.txt"}' mail-list

Many thanks guys for the help!
ostene

MadeInGermany · 11-09-2017, 11:47 AM

Here is a solution to the pitfall in my last post (#9):
if the input file is empty then the variable total is not initialized, it will be treated as an empty string rather than a zero integer.
If you want to force a zero integer you can either initialize it

Code:

awk '
  BEGIN {total=0}
  {total += $6}
  END {print total}
' Test_file.txt

or use printf with an integer format

Code:

awk '
  {total += $6}
  END {printf "%u\n", total}
' Test_file.txt

Or add a zero

Code:

awk '
  {total += $6}
  END {print total+0}
' Test_file.txt