[SOLVED] Interesting awk

sysmicuser · 03-17-2012, 02:23 AM

Fellas,

I trust and completly believe that there are many Unix Gurus on this forum and I am just beginner !

hoping to reach there soon

I need small help.

Code:

[user01@centos-flash rel-01]$find . -type f -print |xargs p4 have|awk 'not on {print $1}'
./hf_rel-01.0.14073_cca.sql - file(s) not on client.
./hf_rel-01.0.29.14656_ccr.sql.commit - file(s) not on client.
./hf_rel-01.0.28_ccr.sql - file(s) not on client.
./hf_rel-01.0.27_ccr.sql - file(s) not on client.
./hf_rel-01.0.29.14074_cca.sql.commit - file(s) not on client.
./hf_rel-01.0.29.14191_ccr.sql - file(s) not on client.
./hf_rel-01.0.29.14085_ccr.sql - file(s) not on client.
./hf_rel-01.0.29.12459_ccr.sh - file(s) not on client.
./hf_rel-01.0.12_cca.sql - file(s) not on client.
./hf_rel-01.0.29.12701_ccr.sql - file(s) not on client.
./hf_rel-01.0.29.14013_cca.sql.commit - file(s) not on client.
./14074_CCCB2PRD_20110514.out - file(s) not on client.
./hf_rel-01.0.12_cca.sh - file(s) not on client.
./hf_rel-01.0.29.14180_ccr.sql.commit - file(s) not on client.

I need to extract only name of file say hf_rel-01.0.14073_cca.sql so on and so forth..

I even did

Code:

find . -type f -print |xargs p4 have|awk 'not on {print $1}'|grep -v "not on client"

Code:

find . -type f -print |xargs p4 have|grep "not on"|awk -F'-' '{print $1}'

but had no success

How should I go about this?

druuna · 03-17-2012, 03:10 AM

Hi,

You've chosen a certain solution for your problem and are stuck with that, could you explain what it is you are trying to accomplish?

Quote:

Originally Posted by sysmicuser

$ find . -type f -print | xargs p4 have | awk 'not on {print $1}'

You seem to be looking for normal files in the current and subdirectories (the find . -type f -print part), the rest of the pipe (xargs and the awk part) doesn't make any sense.

Please tell us what your starting point is and what you want as end-result. Depending on what it is you want to accomplish the find command might not be needed at all.

Hope this helps.

sysmicuser · 03-17-2012, 03:42 AM

@drunna

What I trying to accomplish is a good question.

My intention is to look for all files which are not in perforce.
First part as your rightly said is find all files in current and sub directories

second part xargs p4 have means executing on each and every file found a simple comand called p4 have

Once we get output from p4 have all I do is search for that file which is not in perforce client.

It would give typical output as follows is file is not in perforce client

Code:

./hf_rel-01.0.14073_cca.sql - file(s) not on client.

But I need to further filter in such a way that I need to get only filename of those file which are not in client.

Hope I have now clarified?

So I want output such as

Code:

hf_rel-01.0.14073_cca.sql

druuna · 03-17-2012, 04:16 AM

Hi,

If I focus on the output of this part find . -type f -print |xargs p4 have, which I believe could produce lines like this ./hf_rel-01.0.14073_cca.sql - file(s) not on client. (of which you only want to print the filename) I would do something like this:

Code:

$ find . -type f -print | xargs p4 have | awk -F"[ /]" '/not on client/ { print $2 }'

The bold part looks for lines that contain not on client and will print the filename. You can use multiple field separators in awk (the -F"[ /]" part), in this case a space and the forward slash are the field separators.

Hope this helps.

sysmicuser · 03-17-2012, 05:09 AM

@drunna bro.

No effect it is still the same

Code:

[user01@centos-flash rel-01]$find . -type f -print | xargs p4 have | awk -F"[ /]" '/not on client/ { print $2 }'
./hf_rel-01.0.14073_cca.sql - file(s) not on client.
./hf_rel-01.0.29.14656_ccr.sql.commit - file(s) not on client.
./hf_rel-01.0.28_ccr.sql - file(s) not on client.
./hf_rel-01.0.27_ccr.sql - file(s) not on client.
./hf_rel-01.0.29.14074_cca.sql.commit - file(s) not on client.
./hf_rel-01.0.29.14191_ccr.sql - file(s) not on client.
./hf_rel-01.0.29.14085_ccr.sql - file(s) not on client.
./hf_rel-01.0.29.12459_ccr.sh - file(s) not on client.
./hf_rel-01.0.12_cca.sql - file(s) not on client.

I am also looking let us see.

sysmicuser · 03-17-2012, 05:23 AM

One more interesting this is that if I use stderr and stdout I cannot bloody use in all of the above commands

2>&1 | tee -a 1.log

here 1.log file after execution is of "0" size

That's not so cool..

Code:

[user01@centos-flash rel-01]$find . -type f -print | xargs p4 have | awk -F"[ /]" '/not on client/ { print $2 }' 2>&1 | tee -a 1.log
[user01@centos-flash rel-01]$ls -l 1.log 
-rw-r--r-- 1 user01 user01 0 Mar 17 21:21 1.log
[user01@centos-flash rel-01]$

also if I use instead of print $2, print FILENAME that doesn't work either....

druuna · 03-17-2012, 05:34 AM

Hi,

I can only assume that the output generated by p4 have might have something to do with it. It might not print the output to the stdout (and or stderr).

Without knowing how the p4 program/script works and how it generates output this will be hard to trouble-shoot.

Does the following work? (please post output of commands, within code tags):

Code:

$ find . -type f -print | xargs p4 have > outfile
$ head -3 outfile
$ file outfile
$ od -c outfile | head

BTW: You are working on a linux box and are using bash as shell?

sysmicuser · 03-17-2012, 05:46 AM

Yes working on a linux box and using bash as shell.

sysmicuser · 03-17-2012, 05:48 AM

I am trying to do google but what the heck -perm -200 means??

---------- Post added 03-17-12 at 09:48 PM ----------

Yeah getting output for each command and sending it your private email.

druuna · 03-17-2012, 05:50 AM

Please post the output I asked for in this thread! I'm not going to do a 1 on 1 session!

sysmicuser · 03-17-2012, 06:20 AM

I don't want either private 1 on 1 session, email was only to keep our security freaks happy!

I shall post on thread anyways!

sysmicuser · 03-17-2012, 07:29 AM

Output of ccommand 1.

Quote:

find . -type f -print | xargs p4 have > outfile

Code:

[user01@centos-flash 1.9.0.4.6.1]$find . -type f -print | xargs p4 have > outfile
./hf_1.9.0.4.6.1.0.14073_cca.sql - file(s) not on client.
./hf_1.9.0.4.6.1.0.29.14656_ccr.sql.commit - file(s) not on client.
./hf_1.9.0.4.6.1.0.28_ccr.sql - file(s) not on client.
./hf_1.9.0.4.6.1.0.27_ccr.sql - file(s) not on client.
./hf_1.9.0.4.6.1.0.29.14074_cca.sql.commit - file(s) not on client.
./hf_1.9.0.4.6.1.0.29.14191_ccr.sql - file(s) not on client.
./hf_1.9.0.4.6.1.0.29.14085_ccr.sql - file(s) not on client.
./hf_1.9.0.4.6.1.0.29.12459_ccr.sh - file(s) not on client.
./hf_1.9.0.4.6.1.0.12_cca.sql - file(s) not on client.
./hf_1.9.0.4.6.1.0.29.12701_ccr.sql - file(s) not on client.
./hf_1.9.0.4.6.1.0.29.14013_cca.sql.commit - file(s) not on client.

Output of command 2

Quote:

head -3 outfile

Code:

[user01@centos-flash 1.9.0.4.6.1]$head -3 outfile
//glide/main/scm/hfscripts/1.9.0.4.6.1/hf_1.9.0.4.6.1.0.24_int.sh#1 - /app/scm/p4ws/scmbuild/scmtools/hfscripts/1.9.0.4.6.1/hf_1.9.0.4.6.1.0.24_int.sh
//glide/main/scm/hfscripts/1.9.0.4.6.1/hf_1.9.0.4.6.1.0.22_ccr.sh#1 - /app/scm/p4ws/scmbuild/scmtools/hfscripts/1.9.0.4.6.1/hf_1.9.0.4.6.1.0.22_ccr.sh
//glide/main/scm/hfscripts/1.9.0.4.6.1/hf_1.9.0.4.6.1.0.18_iop.sh#1 - /app/scm/p4ws/scmbuild/scmtools/hfscripts/1.9.0.4.6.1/hf_1.9.0.4.6.1.0.18_iop.sh
[user01@centos-flash 1.9.0.4.6.1]$

Output of comamnd 3

Quote:

file outfile

[code]
[user01@centos-flash 1.9.0.4.6.1]$file outfile
outfile: ASCII text
[user01@centos-flash 1.9.0.4.6.1]$
[code]

Output of command 4

Quote:

od -c outfile | head

Code:

[user01@centos-flash 1.9.0.4.6.1]$od -c outfile | head
0000000   /   /   g   l   i   d   e   /   m   a   i   n   /   s   c   m
0000020   /   h   f   s   c   r   i   p   t   s   /   1   .   9   .   0
0000040   .   4   .   6   .   1   /   h   f   _   1   .   9   .   0   .
0000060   4   .   6   .   1   .   0   .   2   4   _   i   n   t   .   s
0000100   h   #   1       -       /   a   p   p   /   s   c   m   /   p
0000120   4   w   s   /   s   c   m   b   u   i   l   d   /   s   c   m
0000140   t   o   o   l   s   /   h   f   s   c   r   i   p   t   s   /
0000160   1   .   9   .   0   .   4   .   6   .   1   /   h   f   _   1
0000200   .   9   .   0   .   4   .   6   .   1   .   0   .   2   4   _
0000220   i   n   t   .   s   h  \n   /   /   g   l   i   d   e   /   m
[user01@centos-flash 1.9.0.4.6.1]$

---------- Post added 03-17-12 at 11:30 PM ----------

Interesting to note that output and logfile are completely different !

druuna · 03-17-2012, 07:37 AM

Hi,

The output you see in post #1 and post #5 are not send to stdout by the p4 program. The output from command 1 and 2 show this (outfile is filled with other information).

Without knowing what p4 does I cannot help you. It (p4) probably manipulates file descriptors for its multiple output.

I would suggest changing p4 so that the output that is now send to screen (the ./hf_1.9.0.4.6.1.0.14073_cca.sql - file(s) not on client. lines) are send to a seperate file, which you can then use.

Hope this helps.

sysmicuser · 03-17-2012, 08:23 AM

I was also thinking say for example if get output in stdout by means of script command and then exit and later parsing the file.

You are correct p4 doesn't send output to stdout,
I ran script and later parsed file and it gives exact file names !

Code:

awk -F"[ /]" '/not on client/ { print $2 }' <file_name>

certainly works.

Well I would like to ask you the most idiot question anyone would have ever asked. Sorry about my incompetent shell/unix skills

what is stdout and stderr

When any error is thrown does it means stderr and general output as stdout? what is exactly difference between them? how does unix os in intself what is error and what is output.

The reason I ask is whenever we do deployments I always capture any command out to a logfile by following command

2>&1 | tee -a <log_file>

does this means whatever is displayed on screen/terminal throw that stuff in <log_file> if thats the case it even throws build failed error in logfile should it not only throw stdout?

Please help me understand this.

Many Thanks.

druuna · 03-17-2012, 08:55 AM

Hi,

By default stdout (1) is used for normal output and stderr (2) is used for outputting error related messages. Both, by default, print their messages to screen.

A user or a script/program can manipulate this. If you use 2>&1 you tell bash to redirect the stderr (the 2) messages to stdout (the 1). Messages from both are now going through the same pipe.

Expanding on that: Why would you need/want that? When using this 2>&1 | tee -a <log_file>, redirecting to one pipe is needed because the tee command only captures stdout (man tee). If you would omit the 2>&1 part, normal output would be send to <log_file> and errors would go to the screen (stderr's default).

In the case of p4 I suspect one or more extra file descriptors ("pipes") to be created/used, which send output to screen (stderr and stdout are used for other output).

Here's the whole story: I/O Redirection