LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)
-   -   Minimize /reduce disk reading/ writing in BASH scripting (https://www.linuxquestions.org/questions/linux-newbie-8/minimize-reduce-disk-reading-writing-in-bash-scripting-912079/)

lithos 11-05-2011 06:30 PM

Minimize /reduce disk reading/ writing in BASH scripting
 
2 Attachment(s)
Hi

I'd like to minimize disk operations in BASH script when executing my freshly written script, which depends on writing outputs to file and then processing with grep, awk..

It is possible to read all the output from commands to the variables, BUT (yes, there's always one :) )
then the data in variables doesn't retain the formatting.

I'm attaching my script and outputs that I get in the files for better explanation what the script does.

I would like the script not to make so much writing to disk if the RAID fails (because at that moment it's fairly high disk usage getting Rebuilding the RAID set)
and get the output to a variable and process then all from memory, not the file.


huh, I hope for some explanation if anyone could lead me

Thank you

catkin 11-06-2011 01:20 AM

Can you post a simple example illustrating how the data in variables doesn't retain the formatting?

syg00 11-06-2011 01:40 AM

Use a tmpfs mount (/tmp ?) and write the file out to real disk (if needed) as a final operation in the script.
I would have thought it should be retained in cache anyway ...

David the H. 11-06-2011 04:53 AM

A parameter will store any character except the null-byte, which is discarded. Whatever characters you put into a variable will stay there as-is until you declare a new value for it or unset it. This includes spaces, tabs, newlines, and other control characters.

If your formatting isn't being preserved, it's almost certainly due to the shell processing whitespace after the variable expansion. Always be sure to double-quote your variables to keep this from happening.

http://mywiki.wooledge.org/Arguments
http://mywiki.wooledge.org/WordSplitting
http://mywiki.wooledge.org/Quotes

jlinkels 11-06-2011 10:01 AM

Quote:

Originally Posted by lithos (Post 4516842)
I'd like to minimize disk operations in BASH script when executing my freshly written script, which depends on writing outputs to file and then processing with grep, awk..

I admit I am too lazy to read throught your script on the disk usage. How much disk usage do you call a lot? Hard disks now have a few MB of cache, and unless your script is creating bulk output, script output is usually measured in kB. You won't notice.

jlinkels

catkin 11-07-2011 03:02 AM

Quote:

Originally Posted by jlinkels (Post 4517251)
I admit I am too lazy to read throught your script on the disk usage. How much disk usage do you call a lot? Hard disks now have a few MB of cache, and unless your script is creating bulk output, script output is usually measured in kB. You won't notice.

jlinkels

+1 to that.

Linux buffers file system writes (unless it is configured not to) and then the storage devices buffer their writes.

The only time I had an issue with script logging was on an IBM geo-cluster (maybe not the right term -- it was a long time ago) which was configured for maximum file system data retention so any file system write was immediately sent over the network to the remote mirroring server.


All times are GMT -5. The time now is 11:17 AM.