Minimize /reduce disk reading/ writing in BASH scripting

lithos · 11-05-2011, 05:30 PM

Hi

I'd like to minimize disk operations in BASH script when executing my freshly written script, which depends on writing outputs to file and then processing with grep, awk..

It is possible to read all the output from commands to the variables, BUT (yes, there's always one

)
then the data in variables doesn't retain the formatting.

I'm attaching my script and outputs that I get in the files for better explanation what the script does.

I would like the script not to make so much writing to disk if the RAID fails (because at that moment it's fairly high disk usage getting Rebuilding the RAID set)
and get the output to a variable and process then all from memory, not the file.

huh, I hope for some explanation if anyone could lead me

Thank you

catkin · 11-06-2011, 01:20 AM

Can you post a simple example illustrating how the data in variables doesn't retain the formatting?

syg00 · 11-06-2011, 01:40 AM

Use a tmpfs mount (/tmp ?) and write the file out to real disk (if needed) as a final operation in the script.
I would have thought it should be retained in cache anyway ...

David the H. · 11-06-2011, 03:53 AM

A parameter will store any character except the null-byte, which is discarded. Whatever characters you put into a variable will stay there as-is until you declare a new value for it or unset it. This includes spaces, tabs, newlines, and other control characters.

If your formatting isn't being preserved, it's almost certainly due to the shell processing whitespace after the variable expansion. Always be sure to double-quote your variables to keep this from happening.

http://mywiki.wooledge.org/Arguments
http://mywiki.wooledge.org/WordSplitting
http://mywiki.wooledge.org/Quotes

jlinkels · 11-06-2011, 09:01 AM

Quote:

Originally Posted by lithos

I'd like to minimize disk operations in BASH script when executing my freshly written script, which depends on writing outputs to file and then processing with grep, awk..

I admit I am too lazy to read throught your script on the disk usage. How much disk usage do you call a lot? Hard disks now have a few MB of cache, and unless your script is creating bulk output, script output is usually measured in kB. You won't notice.

jlinkels

catkin · 11-07-2011, 02:02 AM

Quote:

Originally Posted by jlinkels

I admit I am too lazy to read throught your script on the disk usage. How much disk usage do you call a lot? Hard disks now have a few MB of cache, and unless your script is creating bulk output, script output is usually measured in kB. You won't notice.

jlinkels

+1 to that.

Linux buffers file system writes (unless it is configured not to) and then the storage devices buffer their writes.

The only time I had an issue with script logging was on an IBM geo-cluster (maybe not the right term -- it was a long time ago) which was configured for maximum file system data retention so any file system write was immediately sent over the network to the remote mirroring server.