Hello all,

I'm still pretty new to Linux servers. I'm working on a project for a web server, and one of my team members said that we need to cut back on I/O intensive commands. He told me to start at looking at all the scripts that use the cat command because they are I/O intensive.

I wasn't aware of till he mentioned it that the cat command was I/O intensive. What could I do that's more efficient. Should I write something in perl, awk, c++? Or is the cat command more efficient?

Why is the cat command so I/O intensive? ( so that I have a better understanding)

Thank you!

I think its the file size what matter.
We had a guy who would less/vi on multi-GB files almost crashed machines every time.

Play with iostat/vmstat while you are opening file or doing other task, that will give you idea according to your
environment.

you can do

Thanks for the watch iostat command. I'll see what I can do...

The problem is the person who designed the program on the web server used the cat command for a lot of the scripts. it frequently uses the cat command to parse through a file. I haven't looked through all the perl scripts yet, but the developer used the cat command a lot to read files in. I don't know how inefficient that can be.. Like I said I'm still new. So I'm trying to figure out how to make things more efficient

Useless use of cat is inefficient, but I don't think it would increase disk IO: it just means that the file gets copied into cat's memory and then copied to the invoking program's memory. So extra memory copying, not extra disk reading.

neoanomally, use command like sed or awk for file manipulation. They are more effient with files. Perl also does a good job if the job is too complicated. Regarding io r/w iotop gives a good information

It sounds like its more a case of just using too much bash scripting in general, which tends to invoke a lot of external cmds/utils like cat.
You should be able to re-write them all in eg native Perl.
Just make sure you use the internal Perl cmds; don't start making lots of external calls like some people do

Useful links
http://perldoc.perl.org/
http://www.perlmonks.org/?node=Tutorials
http://www.tizag.com/perlT/index.php

That's what I figured I would do. I just wasn't sure how much the external calls were going to affect the processes. After looking at many of the processes that are currently written, it looks like the previous programmer used external script calls whenever possible. So it looks like I have my work cut out for me.

Also, thanks for the links!!! tizag, has been the most helpful perl information online I've seen so far!

Been There, Done That
Worked at a place that had an old bash/sed/awk etc script that took ~2 days to run on a small amt of data, in fact they stopped running it.
I re-wrote in Perl and used a lot(!) more data and got it down to ~1.5 hrs

That's also why you should not keep firing off the same cronjob every minute... My rule of thumb is that if you want that every 5 mins or more frequently, just write a daemon and wait at the bottom of the loop as reqd.
Also stops it tripping over other copies of itself.