LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 08-17-2012, 11:19 AM   #1
neoanomally
LQ Newbie
 
Registered: Nov 2010
Posts: 13

Rep: Reputation: 0
Cat command I/O intensive question.


Hello all,

I'm still pretty new to Linux servers. I'm working on a project for a web server, and one of my team members said that we need to cut back on I/O intensive commands. He told me to start at looking at all the scripts that use the cat command because they are I/O intensive.

I wasn't aware of till he mentioned it that the cat command was I/O intensive. What could I do that's more efficient. Should I write something in perl, awk, c++? Or is the cat command more efficient?

Why is the cat command so I/O intensive? ( so that I have a better understanding)

Thank you!
 
Old 08-17-2012, 12:07 PM   #2
PoleStar
Member
 
Registered: Jul 2010
Posts: 165

Rep: Reputation: 2
I think its the file size what matter.
We had a guy who would less/vi on multi-GB files almost crashed machines every time.

Play with iostat/vmstat while you are opening file or doing other task, that will give you idea according to your
environment.

you can do
Quote:
>watch iostat
> watch vmstat

Last edited by PoleStar; 08-17-2012 at 12:09 PM.
 
Old 08-17-2012, 12:30 PM   #3
neoanomally
LQ Newbie
 
Registered: Nov 2010
Posts: 13

Original Poster
Rep: Reputation: 0
Thanks for the watch iostat command. I'll see what I can do...

The problem is the person who designed the program on the web server used the cat command for a lot of the scripts. it frequently uses the cat command to parse through a file. I haven't looked through all the perl scripts yet, but the developer used the cat command a lot to read files in. I don't know how inefficient that can be.. Like I said I'm still new. So I'm trying to figure out how to make things more efficient
 
Old 08-17-2012, 12:50 PM   #4
ntubski
Senior Member
 
Registered: Nov 2005
Distribution: Arch
Posts: 3,013

Rep: Reputation: 1225Reputation: 1225Reputation: 1225Reputation: 1225Reputation: 1225Reputation: 1225Reputation: 1225Reputation: 1225Reputation: 1225
Quote:
Originally Posted by neoanomally View Post
The problem is the person who designed the program on the web server used the cat command for a lot of the scripts. it frequently uses the cat command to parse through a file.
Useless use of cat is inefficient, but I don't think it would increase disk IO: it just means that the file gets copied into cat's memory and then copied to the invoking program's memory. So extra memory copying, not extra disk reading.
 
Old 08-17-2012, 01:37 PM   #5
qweeak
LQ Newbie
 
Registered: Jan 2012
Posts: 24

Rep: Reputation: 2
neoanomally, use command like sed or awk for file manipulation. They are more effient with files. Perl also does a good job if the job is too complicated. Regarding io r/w iotop gives a good information
 
Old 08-19-2012, 08:40 PM   #6
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.8, Centos 5.10
Posts: 17,240

Rep: Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324
It sounds like its more a case of just using too much bash scripting in general, which tends to invoke a lot of external cmds/utils like cat.
You should be able to re-write them all in eg native Perl.
Just make sure you use the internal Perl cmds; don't start making lots of external calls like some people do

Useful links
http://perldoc.perl.org/
http://www.perlmonks.org/?node=Tutorials
http://www.tizag.com/perlT/index.php
 
Old 08-20-2012, 09:16 AM   #7
neoanomally
LQ Newbie
 
Registered: Nov 2010
Posts: 13

Original Poster
Rep: Reputation: 0
That's what I figured I would do. I just wasn't sure how much the external calls were going to affect the processes. After looking at many of the processes that are currently written, it looks like the previous programmer used external script calls whenever possible. So it looks like I have my work cut out for me.

Also, thanks for the links!!! tizag, has been the most helpful perl information online I've seen so far!
 
Old 08-20-2012, 10:00 PM   #8
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.8, Centos 5.10
Posts: 17,240

Rep: Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324
Been There, Done That
Worked at a place that had an old bash/sed/awk etc script that took ~2 days to run on a small amt of data, in fact they stopped running it.
I re-wrote in Perl and used a lot(!) more data and got it down to ~1.5 hrs

That's also why you should not keep firing off the same cronjob every minute... My rule of thumb is that if you want that every 5 mins or more frequently, just write a daemon and wait at the bottom of the loop as reqd.
Also stops it tripping over other copies of itself.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Cat 6 Cat 5e cable question babbab Linux - Networking 1 04-07-2010 03:33 PM
CAT Command ouncya Linux - Newbie 11 08-26-2007 12:48 AM
[Question] Cat command with sudo hbinded Programming 4 12-18-2006 02:53 AM
cat Command Moeses Linux - General 1 06-20-2002 04:27 AM


All times are GMT -5. The time now is 01:11 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration