not getting desired output with awk

linux_project · 08-01-2015, 01:18 AM

I am not getting desired output with below awk command. I want to cut column number 4 using awk with below but it is not fetching the right column. As per my knowledge it should pick one or spaces in a same way . But while Pasting my file content over here . I find *(stars) that i cannot see in vi and notepad++. Any idea for the issue ? How can i find in future as well

2 *ip * * * * *0x12 * * * *10.153.98.246 * * 129.144.130.130 * * * 6666 * * * * 514 * * *539583 * 9.72
3 *ip * * * * *0x06 * * * *10.227.108.31 * * 10.198.184.4 * * * * * * 216 * * * 48719 * * *531099 * 9.57
4 *ip * * * * *0x08 * * * *10.217.97.237 * * 10.198.180.84 * * * * * *216 * * * 64942 * * *333256 * 6.00%
5 *ip * * * * *0x06 * * * *10.226.230.218 * *10.198.184.4 * * * * * * 216 * * * 50049 * * *307549 * 5.54%
6 *ip * * * * *0x32 * * * *160.34.15.60 * * *216.131.130.12 * * * * 46972 * * * 34501 * * *232317 * 4.18%
7 *ip * * * * *0x06 * * * *10.227.48.133 * * 10.198.184.4 * * * * * * 216 * * * 62924 * * *187629 * 3.38%
8 *ip * * * * *0x06 * * * *10.227.93.49 * * *10.198.184.4 * * * * * * 216 * * * 42644 * * *180628 * 3.25%
9 *ip * * * * *0x32 * * * *160.34.31.60 * * *216.131.130.12 * * * * 46972 * * * 34503 * * *172353 * 3.10%
10 *ip * * * * *0x32 * * * *160.34.47.60 * * *216.131.130.12 * * * * 46972 * * * 34516 * * * 90544 * 1.63%

[root@test ~]# cat /tmp/ip | awk -F " " '{print $4}'

10.153.98.246
10.227.108.31
10.217.97.237
10.226.230.218
160.34.15.60 *
10.227.48.133
10.227.93.49 *
160.34.31.60 *
160.34.47.60

grail · 08-01-2015, 04:33 AM

Well my first question back would be, are you showing us the same file you are using??

When i use the data shown and your awk, all i get is 9 lines of asterisks:

Code:

$ awk -F " " '{print $4}' file
*
*
*
*
*
*
*
*
*

So I would need to know how you get any of the ip addresses you are showing as I do not.

I am also not following the, "I find *(stars) that i cannot see in vi and notepad++"??

AnanthaP · 08-02-2015, 01:08 AM

I am sure there is a preprocess (prior programme) which you have not mentioned, where the sequence of one or more stars is replaced by a space and the output has been put in /tmp/ip Also you have pasted the original file (input to the prior programme).

OK

hickmz · 08-02-2015, 05:20 AM

Hi I have been trying 4 hours now i have some text as follows HASH####7hAZ1BoLTM#### all i want is the 7hAZ1BoLTM i have tried countless variations but i either get nothing or the whole line just prints again not just the test that i want
awk '/####/{a=1};a;/####/{exit}' test.txt

grail · 08-02-2015, 06:01 AM

Firstly, please do not hijack someone else's question and simply ask your own. Try setting the field separator appropriately. See [url=http://www.gnu.org/software/gawk/manual/html_node/index.html]here[\url] for more details.

tronayne · 08-02-2015, 07:34 AM

Your file, as shown in your post, appears to have non printing characters embedded in it (that you don't want embedded).

You can use the od utility to dump the file in octal or hex or ASCII (read the manual page for the utility) which will help you identify what those asterisks are so you can get rid of them -- you probably want to look at whatever is writing the file first as it is probably inserting values in your file (those values will be non printing characters but will be embedded in the file). You can also invoke vi so that it will show you non printing characters but I just can't remember right now how to do that (I'll keep looking).

It might be useful to attach your file to a post so that we can take a look at it and perhaps identify the problem (you could also give a link to your system if that's possible).

Hope this helps some.

linux_project · 08-04-2015, 02:50 AM

Thanks tronayne. I am attaching the file. PFA

millgates · 08-04-2015, 03:38 AM

The columns in your file seem to be separated by sequences "20c2a0", where 20 is a space and "c2a0" is a utf-8-encoded non-breaking space. Try:

Code:

awk -F"[ \xc2\xa0]+" '{print $4}' filename

linux_project · 08-04-2015, 04:51 AM

Thanks millgates. This worked... But how could i find this in vi . what command can be used in future to find it in the vi editor itself .

millgates · 08-04-2015, 05:36 AM

You may find this article interesting. Displaying the non-breaking spaces only seems to work in vim, though. That said, if I suspected rogue characters in my files, I'd inspect them with a hex editor.

tronayne · 08-04-2015, 07:07 AM

Here's a slightly easier way to see what's in a file (any file, by the way): a little utility called cdump (as in, character dump).

Save the attached file, cdump.txt, as cdmup.c somewhere then

Code:

make cdump

move that to somewhere on your PATH (such as /usr/local/bin).

Use it

Code:

cdump file_spcl_char.txt
000000:  32 20 c2 a0 69 70 20 c2  a0 20 c2 a0 20 c2 a0 20   2...ip..........
000010:  c2 a0 20 c2 a0 30 78 31  32 20 c2 a0 20 c2 a0 20   .....0x12.......
000020:  c2 a0 20 c2 a0 31 30 2e  31 35 33 2e 39 38 2e 32   .....10.153.98.2
000030:  34 36 20 c2 a0 20 c2 a0  20 31 32 39 2e 31 34 34   46.......129.144
000040:  2e 31 33 30 2e 31 33 30  20 c2 a0 20 c2 a0 20 c2   .130.130........
000050:  a0 20 20 20 36 36 36 36  20 c2 a0 20 c2 a0 20 c2   ....6666........
000060:  a0 20 35 31 34 20 c2 a0  20 c2 a0 20 c2 a0 35 33   ..514.........53
000070:  39 35 38 33 20 c2 a0 20  39 2e 37 32 25 0d 0a 33   9583....9.72%..3

000080:  20 c2 a0 69 70 20 c2 a0  20 c2 a0 20 c2 a0 20 c2   ...ip...........
000090:  a0 20 c2 a0 30 78 30 36  20 c2 a0 20 c2 a0 20 c2   ....0x06........
0000a0:  a0 20 c2 a0 31 30 2e 32  32 37 2e 31 30 38 2e 33   ....10.227.108.3
0000b0:  31 20 c2 a0 20 c2 a0 20  31 30 2e 31 39 38 2e 31   1.......10.198.1
0000c0:  38 34 2e 34 20 c2 a0 20  c2 a0 20 c2 a0 20 c2 a0   84.4............
0000d0:  20 c2 a0 20 c2 a0 20 32  31 36 20 c2 a0 20 c2 a0   .......216......
0000e0:  20 c2 a0 20 34 38 37 31  39 20 c2 a0 20 c2 a0 20   ....48719.......
0000f0:  c2 a0 35 33 31 30 39 39  20 c2 a0 20 39 2e 35 37   ..531099....9.57

(the above is abbreviated output).

The offending characters (c2) show up.

Then, use @millgates method for doing what you want with AWK (or an editor to replace them with a space character).

Hope this helps some.

tronayne · 08-04-2015, 08:27 AM

Oops!

Just noticed that the last line of the file cdup.txt is missing (why I do not know) and I can't figure out how to delete the file in the post above so here's the corrected file file.

Sorry about that.

linux_project · 08-05-2015, 10:53 AM

Thanks millgates. I have gone through the link. I am able to identify the special characters but still not able to remove special characters in vim editor.
I ran `:set nolist` but it just remove the special character but doesn't remove special character permanently.