LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices



Reply
 
Search this Thread
Old 12-12-2012, 06:10 PM   #1
TheOnlyQ
Member
 
Registered: Apr 2011
Posts: 33

Rep: Reputation: 0
Sleep inside for loop inside while loop causing issues.


Hi,

I've been trying to make a script which gives me a bandwidth rate by doing some maths, sleeping for a second, doing maths again, and then basic arithmetic to give me the difference, which I can do more arithmetic with to get the kbps.

It works beautifully, but I'm running into a big issue, the script is not for a single node, it is supposed to find the bandwidth rate of a bunch of containers (virtualization) - The issue is that the script is formatted like this :

while (infinite while)
do
for containers in $list (list is a list of the containers)
do
command to grab dataset1

sleep 1

command to grab dataset2

if datasets are not equal to 0

do the math (dataset 1 - dataset2 /1 /1024)

echo a set of stats (for all of the $containers)

----

That works, but as you may well realize, the list generated is one by one, which means it takes a second to print out the data for each container, when we're working with 5 containers this is fine, but this will not work when we're working with 500 and we need to grab stats quickly on the spot with only a few seconds delay.

I've reworked it, broken it, switched it up, done EVERYTHING I can possibly think of to fix this - but since the sleep NEEDS to be in between the dataset1 and dataset2 the only way to get all stats to print out together basically breaks the math so its useless anyway.

If anyone here could be a wizard and shout some ideas at me I would absolutely love you.

Thank you.
 
Old 12-12-2012, 06:24 PM   #2
jlinkels
Senior Member
 
Registered: Oct 2003
Location: Bonaire
Distribution: Debian Wheezy/Jessie/Sid, Linux Mint DE
Posts: 4,241

Rep: Reputation: 545Reputation: 545Reputation: 545Reputation: 545Reputation: 545Reputation: 545
I would propose:

Code:
for container in $list do
   grab dataset1
   record time stamp
   put both values into array element[$container]
done
sleep 1
for container in $list do
   grab dataset2
   record time stamp
   add both values into array element[$container]
done
for container in $list do
   get the timestamps from element[$container]
   substract the timestamps
   do the math on array element[$container]
done
If I am not mistaken date (at least the GNU version) is able to record in nanoseconds. Which probable are not nanoseconds but provide a resolution much better than a second anyway.

jlinkels
 
Old 12-12-2012, 07:28 PM   #3
TheOnlyQ
Member
 
Registered: Apr 2011
Posts: 33

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by jlinkels View Post
I would propose:

Code:
for container in $list do
   grab dataset1
   record time stamp
   put both values into array element[$container]
done
sleep 1
for container in $list do
   grab dataset2
   record time stamp
   add both values into array element[$container]
done
for container in $list do
   get the timestamps from element[$container]
   substract the timestamps
   do the math on array element[$container]
done
If I am not mistaken date (at least the GNU version) is able to record in nanoseconds. Which probable are not nanoseconds but provide a resolution much better than a second anyway.

jlinkels
If there is still a sleep between the math, how could the output be grouped?
 
Old 12-13-2012, 07:31 AM   #4
TheOnlyQ
Member
 
Registered: Apr 2011
Posts: 33

Original Poster
Rep: Reputation: 0
Any idea? Hate to bump but I know this might have been thrown to the bottom of the list
 
Old 12-14-2012, 05:16 AM   #5
jlinkels
Senior Member
 
Registered: Oct 2003
Location: Bonaire
Distribution: Debian Wheezy/Jessie/Sid, Linux Mint DE
Posts: 4,241

Rep: Reputation: 545Reputation: 545Reputation: 545Reputation: 545Reputation: 545Reputation: 545
I didn't see your reply before.

I don't understand your question. Can you please elaborate? You have 2 sets of data. One on T-1 and one on T. You calculate the difference for each container and then you have the throughput for roughly 1 second for all containers.

jlinkels
 
Old 12-17-2012, 01:14 AM   #6
TheOnlyQ
Member
 
Registered: Apr 2011
Posts: 33

Original Poster
Rep: Reputation: 0
Wink

Quote:
Originally Posted by jlinkels View Post
I didn't see your reply before.

I don't understand your question. Can you please elaborate? You have 2 sets of data. One on T-1 and one on T. You calculate the difference for each container and then you have the throughput for roughly 1 second for all containers.

jlinkels
Hi,

Sorry for taking some time to response.

You must have misunderstood my original post. My issue is that, the output must contain statistics for all containers, at once.

However, since we are in a for loop, the sleep is executed once for every container, but the sleep NEEDS to be in the for loop because the mathematics are inside of the for loop, since we need roughly one second between the first dataset and the second to do the maths. If you don't understand this, I'm not sure how else I can put it - The best thing I can think of is to use something other than a for loop to execute commands on all containers to grab data, but what exactly I have no idea.

The output, in case you are wondering, is like this :

CONTAINER 1 USING XX KB/S AT 6:09:13 AM
*one second*
CONTAINER 2 USING XX KB/S AT 6:09:14 AM
*one second*
CONTAINER 3 USING XX KB/S AT 6:09:15 AM
*one second*
CONTAINER 3 USING XX KB/S AT 6:09:16 AM
...

and it needs to be :

CONTAINER 1 USING XX KB/S AT 6:09:13 AM
CONTAINER 2 USING XX KB/S AT 6:09:13 AM
CONTAINER 3 USING XX KB/S AT 6:09:13 AM
..

As you can see, grouped. The commands to grab data need to be executed and the result stored for each container, the way I do that is with a for loop, but since a sleep is inside this for loop, we're also repeating the sleep for each container. The only way I can think of is to minimize our sleeps by 10x (0.1second) or even lower, and edit the arithmatic.. but this would still have a delay, whether less noticeable or not (1second/10containers).

HELP???
 
Old 12-17-2012, 04:28 PM   #7
jlinkels
Senior Member
 
Registered: Oct 2003
Location: Bonaire
Distribution: Debian Wheezy/Jessie/Sid, Linux Mint DE
Posts: 4,241

Rep: Reputation: 545Reputation: 545Reputation: 545Reputation: 545Reputation: 545Reputation: 545
I think I did understand your question. You want to collect statistics for a lot of containers, but to avoid that for each container you have to perform the sleep sequentially. So for 600 containers you won't end up with 600 seconds total of sleep time.

Therefore I proposed that you record the start count for each container, including time stamp. Then wait one second, and record the end count for each container. So after roughly one second you can calculate for each container the end count and start count. By using the recorded time stamps you can accurately determine the elapsed time between the start and end count. Because you are querying all 600 containers in sequence, the correct time stamps are relevant. You cannot assume that the exact interval for measurements is 1 second due to differences in retrieval time.

So the total time you need is [retrieval all start values] + 1 second + [retrieval all end values]. You end up with all throughputs in approximately 1 second.

What else would you want or why doesn't this solution serve your purpose?

jlinkels

Last edited by jlinkels; 12-17-2012 at 04:31 PM.
 
Old 12-17-2012, 08:37 PM   #8
TheOnlyQ
Member
 
Registered: Apr 2011
Posts: 33

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by jlinkels View Post
I think I did understand your question. You want to collect statistics for a lot of containers, but to avoid that for each container you have to perform the sleep sequentially. So for 600 containers you won't end up with 600 seconds total of sleep time.

Therefore I proposed that you record the start count for each container, including time stamp. Then wait one second, and record the end count for each container. So after roughly one second you can calculate for each container the end count and start count. By using the recorded time stamps you can accurately determine the elapsed time between the start and end count. Because you are querying all 600 containers in sequence, the correct time stamps are relevant. You cannot assume that the exact interval for measurements is 1 second due to differences in retrieval time.

So the total time you need is [retrieval all start values] + 1 second + [retrieval all end values]. You end up with all throughputs in approximately 1 second.

What else would you want or why doesn't this solution serve your purpose?

jlinkels
I'm just not sure I understand your addition at all. How will these time stamp help me? The script will still take 1 second per container, the problem still exists?

Maybe I'm missing something?
 
Old 12-18-2012, 01:52 AM   #9
jlinkels
Senior Member
 
Registered: Oct 2003
Location: Bonaire
Distribution: Debian Wheezy/Jessie/Sid, Linux Mint DE
Posts: 4,241

Rep: Reputation: 545Reputation: 545Reputation: 545Reputation: 545Reputation: 545Reputation: 545
Please read my previous two posts and read carefully. I explained it once by pseudo code and once in text. It does not take once second per container. It takes 1 second in total plus retrieval time for all containers. I also explained why the time stamp is needed.

jlinkels
 
Old 12-18-2012, 02:09 AM   #10
TheOnlyQ
Member
 
Registered: Apr 2011
Posts: 33

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by jlinkels View Post
Please read my previous two posts and read carefully. I explained it once by pseudo code and once in text. It does not take once second per container. It takes 1 second in total plus retrieval time for all containers. I also explained why the time stamp is needed.

jlinkels
I have re-read your posts multiple times, but the slight inaccuracy of the extra time is not of my concern, nor what I am trying to fix. And I fail to see how your addition would fix the problem that I have. It is taking 1 second per container, that isn't of question, and your pseudo code nor the idea I read in text doesn't change that at all - so I'm confused.

If you're doing :

for i in 1 2 3 4 5
do
echo $i
sleep 1
done

you're going to have an script that sleeps for 5 seconds, imagine what you really want is for it to echo $i all 5 times and then sleep for a second, and for whatever reason you cannot place the sleep anywhere other than inside the for loop (in my case, because of arithmetic) - that's the most simple explanation of my issue I can possibly give, one of us is getting the wrong end of the stick and I hope I'm not frustrating you by reinforcing that.
 
Old 12-18-2012, 04:13 AM   #11
jlinkels
Senior Member
 
Registered: Oct 2003
Location: Bonaire
Distribution: Debian Wheezy/Jessie/Sid, Linux Mint DE
Posts: 4,241

Rep: Reputation: 545Reputation: 545Reputation: 545Reputation: 545Reputation: 545Reputation: 545
PLEASE read my pseudo code again.
The sleep is outside both for loops.
Apparently you are not reading.

jlinkels
 
Old 12-18-2012, 01:59 PM   #12
TheOnlyQ
Member
 
Registered: Apr 2011
Posts: 33

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by jlinkels View Post
I didn't see your reply before.

I don't understand your question. Can you please elaborate? You have 2 sets of data. One on T-1 and one on T. You calculate the difference for each container and then you have the throughput for roughly 1 second for all containers.

jlinkels
Now I have another issue. Given that I'm assigning the output of a command (the command that grabs the data) to a variable, we can assume $var contains this data.

The issue is that now the actions are seperated between for loops, this var name is basically overwritten - which means it is useless.

I tried to use the container ID, convert it using letter/number cypher (0-9 become a-j) and append this to the end of the variable name var so that it would become vara for example if the container ID was 0, and so on. But this didn't work - you can't specify var$cypher (var is the output of command to grab dataset.. cypher the output of the conversion) because bash treats this as a command in itself ignoring var and trying to execute (if $cypher is equal to 0 and output of command to grab data was 1) $0=1 which obviously doesn't exist.

So, now my next thought, (and as you originally said) store the output of the command in an array, but then my next issue is how to sort through this array and use it easily when doing the maths.. it would seem like an awfully long and ugly process which I really would like to avoid.

So while writing this I decided to try arrays - I spent a good hour or two on it - end result. Not good. Broken in more ways than one and just generally dirty as hell. Hopefully you can save me again and give me some other method.

sleep 28800
 
Old 12-18-2012, 04:22 PM   #13
jlinkels
Senior Member
 
Registered: Oct 2003
Location: Bonaire
Distribution: Debian Wheezy/Jessie/Sid, Linux Mint DE
Posts: 4,241

Rep: Reputation: 545Reputation: 545Reputation: 545Reputation: 545Reputation: 545Reputation: 545
It is an array, so you address the elements by index. It is one of the most logical and elegant methods to use.

jlinkels
 
Old 12-19-2012, 01:59 PM   #14
TheOnlyQ
Member
 
Registered: Apr 2011
Posts: 33

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by jlinkels View Post
It is an array, so you address the elements by index. It is one of the most logical and elegant methods to use.

jlinkels
I knew that.. but something was breaking the array, found my mistake.

Had to change quite a lot around on top of that to get it working, last for became a while, had to get the array size (amount of elements) and then use that to count through the dataset array (so that I can identify which element belongs to which container in the list of containers) and then go through that from 0 to $amountofelements and echo my output.. a few other things here and there but overall it works!

Thank you a lot, I appreciate your persistence

solved.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
nohup inside loop santhosh_cv Programming 1 10-01-2012 05:42 AM
stream editing inside the while loop m4rtin Programming 7 03-11-2010 04:26 AM
bash: variables inside while loop J_Szucs Programming 5 07-19-2009 09:24 AM
trouble with expr inside a while loop farkus888 Linux - General 5 04-06-2007 04:19 AM
for loop inside select box opioid Programming 1 03-17-2005 03:22 PM


All times are GMT -5. The time now is 09:21 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration