LinuxQuestions.org
Latest LQ Deal: Linux Power User Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 11-05-2011, 06:30 PM   #1
lithos
Senior Member
 
Registered: Jan 2010
Location: SI : 45.9531, 15.4894
Distribution: CentOS, OpenNA/Trustix, testing desktop openSuse 12.1 /Cinnamon/KDE4.8
Posts: 1,144

Rep: Reputation: 217Reputation: 217Reputation: 217
Minimize /reduce disk reading/ writing in BASH scripting


Hi

I'd like to minimize disk operations in BASH script when executing my freshly written script, which depends on writing outputs to file and then processing with grep, awk..

It is possible to read all the output from commands to the variables, BUT (yes, there's always one )
then the data in variables doesn't retain the formatting.

I'm attaching my script and outputs that I get in the files for better explanation what the script does.

I would like the script not to make so much writing to disk if the RAID fails (because at that moment it's fairly high disk usage getting Rebuilding the RAID set)
and get the output to a variable and process then all from memory, not the file.


huh, I hope for some explanation if anyone could lead me

Thank you
Attached Files
File Type: txt raid_check.sh.txt (2.4 KB, 12 views)
File Type: txt raid_check_output.txt (1.3 KB, 8 views)

Last edited by lithos; 11-05-2011 at 06:33 PM.
 
Old 11-06-2011, 01:20 AM   #2
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Debian
Posts: 8,576
Blog Entries: 31

Rep: Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195
Can you post a simple example illustrating how the data in variables doesn't retain the formatting?
 
Old 11-06-2011, 01:40 AM   #3
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 15,997

Rep: Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219
Use a tmpfs mount (/tmp ?) and write the file out to real disk (if needed) as a final operation in the script.
I would have thought it should be retained in cache anyway ...
 
Old 11-06-2011, 04:53 AM   #4
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian + kde 4 / 5
Posts: 6,837

Rep: Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981
A parameter will store any character except the null-byte, which is discarded. Whatever characters you put into a variable will stay there as-is until you declare a new value for it or unset it. This includes spaces, tabs, newlines, and other control characters.

If your formatting isn't being preserved, it's almost certainly due to the shell processing whitespace after the variable expansion. Always be sure to double-quote your variables to keep this from happening.

http://mywiki.wooledge.org/Arguments
http://mywiki.wooledge.org/WordSplitting
http://mywiki.wooledge.org/Quotes
 
Old 11-06-2011, 10:01 AM   #5
jlinkels
Senior Member
 
Registered: Oct 2003
Location: Bonaire, Leeuwarden
Distribution: Debian /Jessie/Stretch/Sid, Linux Mint DE
Posts: 4,728

Rep: Reputation: 754Reputation: 754Reputation: 754Reputation: 754Reputation: 754Reputation: 754Reputation: 754
Quote:
Originally Posted by lithos View Post
I'd like to minimize disk operations in BASH script when executing my freshly written script, which depends on writing outputs to file and then processing with grep, awk..
I admit I am too lazy to read throught your script on the disk usage. How much disk usage do you call a lot? Hard disks now have a few MB of cache, and unless your script is creating bulk output, script output is usually measured in kB. You won't notice.

jlinkels
 
Old 11-07-2011, 03:02 AM   #6
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Debian
Posts: 8,576
Blog Entries: 31

Rep: Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195
Quote:
Originally Posted by jlinkels View Post
I admit I am too lazy to read throught your script on the disk usage. How much disk usage do you call a lot? Hard disks now have a few MB of cache, and unless your script is creating bulk output, script output is usually measured in kB. You won't notice.

jlinkels
+1 to that.

Linux buffers file system writes (unless it is configured not to) and then the storage devices buffer their writes.

The only time I had an issue with script logging was on an IBM geo-cluster (maybe not the right term -- it was a long time ago) which was configured for maximum file system data retention so any file system write was immediately sent over the network to the remote mirroring server.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Writing to and reading from a socket from bash script. vouser Programming 6 05-10-2012 06:13 PM
Writing to and reading from a socket from bash script. vouser Linux - Networking 2 03-12-2010 11:07 PM
Reading a bash variable in bash scripting problem freeindy Programming 3 11-27-2008 03:29 AM
Reading and writing a file with bash XeroXer Linux - Newbie 7 06-02-2008 09:08 AM
Find out what hard disk is reading/writing to newbuyer17 Linux - General 9 05-22-2008 02:49 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 06:32 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration