[Python] for loop question

jonnybinthemix · 02-10-2017, 04:36 PM

Hey Guys,

So, you'll be glad to know the the post I've marked as resolved. I have a solution which works. I think I was trying to get perfect data back from the remote server. I've searched and searched and maybe Python just isn't as clever as I thought it was. Who knows..

Anyway, I'm just returning all data (which in the real world will be a lot!):

Code:

#!/usr/bin/python
#

from subprocess import *

ssh = Popen(["ssh", "root@boot", 'xtprocadmin'],
            shell=False,
            stdout=PIPE,
            stderr=PIPE)
result = ssh.stdout.readlines()

lines = result.splitlines()
for line in lines[1:]:
    columns = line.split()
    status = columns[3]
    nodes = columns[2]
    if status == "down":
        print "%s is down" % nodes

So, I've tested the above with some tests, so I'm pretty sure that the theory is solid (ish) - But could you help with my loop?

I'm trying to take the data returned from the ssh session, and break it into lines.. then into columns. Then find the columns that contain the word down.. then print (for now) the node which is down.

I thought the above would work, but clearly I'm missing something simple.

Thanks in advance,
Jon

Myk267 · 02-10-2017, 09:14 PM

I've emboldened the code which I think you're misunderstanding.

Code:

#!/usr/bin/python
#

from subprocess import *

ssh = Popen(["ssh", "root@boot", 'xtprocadmin'],
            shell=False,
            stdout=PIPE,
            stderr=PIPE)
result = ssh.stdout.readlines()

lines = result.splitlines()
for line in lines[1:]:
    columns = line.split()
    status = columns[3]
    nodes = columns[2]
    if status == "down":
        print "%s is down" % nodes

jonnybinthemix · 02-11-2017, 03:34 AM

Quote:

Originally Posted by Myk267

I've emboldened the code which I think you're misunderstanding.

Code:

#!/usr/bin/python
#

from subprocess import *

ssh = Popen(["ssh", "root@boot", 'xtprocadmin'],
            shell=False,
            stdout=PIPE,
            stderr=PIPE)
result = ssh.stdout.readlines()

lines = result.splitlines()
for line in lines[1:]:
    columns = line.split()
    status = columns[3]
    nodes = columns[2]
    if status == "down":
        print "%s is down" % nodes

Hi, and many thanks for responding.

My understanding of it is (which is likely wrong, as I'm learning):

Loop through the lines of the data and separate the words in that line into columns. But I added [1:] as I don't want the headings. I will try to emphasize my understanding of what I've written.

Thanks
Jon

jonnybinthemix · 02-11-2017, 03:50 AM

Ahhhhhh! I see....

Code:

lines = result.splitlines()

for line in result[1:]:

I see that I was trying to split the lines in the output and then loop through them and split them again into columns.

I changed it for:

Code:

from subprocess import *

ssh = Popen(["ssh", "root@boot", 'xtprocadmin'],
            shell=False,
            stdout=PIPE,
            stderr=PIPE)
result = ssh.stdout.readlines()

for line in result[1:]:
    columns = line.split()
    status = columns[3]
    nodes = columns[2]
    if status == "down":
        print "%s is down" % nodes

This seems to work actually, but I need to do some more testing to make sure it's solid.

Thank you for highlighting the area which needed attention.

I wouldn't say that I FULLY understand what I've done, it's a bit of a fudge.. and I'm pretty sure it's not the best way of doing it.

Thanks,
Jon

pan64 · 02-11-2017, 03:51 AM

Quote:

I thought the above would work, but clearly I'm missing something simple.

what kind of problem did you experience?

I would try to add some debug line to the code to see what's happening, later you can remove them:

Code:

for line in lines[1:]:
    print "line=" + line
    columns = line.split()
    status = columns[3]
    print "status=" + status
    nodes = columns[2]
    print "nodes=" + nodes
    if status == "down":
        print "%s is down" % nodes

but obviously you can modify it as you wish to make it really usable (=that means it will give you information about what's going on)

Myk267 · 02-11-2017, 03:56 AM

Quote:

Originally Posted by jonnybinthemix

Hi, and many thanks for responding.

My understanding of it is (which is likely wrong, as I'm learning):

Loop through the lines of the data and separate the words in that line into columns. But I added [1:] as I don't want the headings. I will try to emphasize my understanding of what I've written.

Thanks
Jon

In that case, we both get to be wrong. Fantastic.

See, I had figured that what you really wanted was to drop the first column (just my guess), which was throwing your column indexing off, but now we know that you didn't want the header. So now all you've got to do is fix up your error in indexing into the columns list. Python lists start counting from 0.

Have a nice day

jonnybinthemix · 02-11-2017, 05:00 AM

Quote:

Originally Posted by pan64

what kind of problem did you experience?

I would try to add some debug line to the code to see what's happening, later you can remove them:

Code:

for line in lines[1:]:
    print "line=" + line
    columns = line.split()
    status = columns[3]
    print "status=" + status
    nodes = columns[2]
    print "nodes=" + nodes
    if status == "down":
        print "%s is down" % nodes

but obviously you can modify it as you wish to make it really usable (=that means it will give you information about what's going on)

Great advice regarding extra steps for debugging. Not only will that help me to debug the code, even in a working example it will help me to fully understand what I'm doing

I'll remember to do that more

It works now, kind of as expected. But there are a couple of other little things which aren't quite right. But I think it's going to be tough to get exactly what I want. This particular example works okay, but that's because my VM Test environment is a fudge. 'xtprocadmin' is a custom tool that we use which responds with the status of each node in the cluster. For development purposes I have written a bash one liner to cat out a file which has 15 or so lines of an xtprocadmin output. In the real world, when I use this script.. xtprocadmin will respond with nearly 7000 nodes. Which is in turn a lot more data for it to manage. But having said that, thinking while I'm typing; 7000 lines really is not a lot for python to deal with, nor is it really too much for ssh to deal with since it's plain text. So I'm likely worrying over nothing.

jonnybinthemix · 02-11-2017, 05:08 AM

Quote:

Originally Posted by Myk267

In that case, we both get to be wrong. Fantastic.

See, I had figured that what you really wanted was to drop the first column (just my guess), which was throwing your column indexing off, but now we know that you didn't want the header. So now all you've got to do is fix up your error in indexing into the columns list. Python lists start counting from 0.

Have a nice day

I don't think I need to fix the column indexing, as it works as I'd expect it to (in a sense). I was aware of the column indexing being 0 relative

- I did a Python course online yesterday lol.

This is my code so far, I have put some error checking in for the ssh session, and added some more to the loop for displaying another node status type.

Code:

#!/usr/bin/python
#

from subprocess import *
import sys

ssh = Popen(["ssh", "root@boot", 'xtprocadmin'],
            shell=False,
            stdout=PIPE,
            stderr=PIPE)
result = ssh.stdout.readlines()

if result == []:
    error = ssh.stderr.readlines()
    print >>sys.stderr, "ERROR: %s" % error
else:
        for line in result[1:]:
            columns = line.split()
            status = columns[3]
            nodes = columns[2]
            if status == "down":
                print "%s is down" % nodes
            if status == "admindown":
                print "%s is admindown" % nodes

The only gripe I have so far is with ordering the output. At the moment the output is as such:

Code:

[root@jons-server ~]# ./test.py
c2-0c2s4n1 is down
c2-0c1s4n2 is down
c2-0c1s15n0 is down
c2-0c1s8n3 is admindown
c2-0c2s2n1 is down
c2-0c1s6n1 is admindown
c2-0c1s15n3 is down
c2-0c1s1n3 is down
c2-0c1s6n1 is admindown

I don't like how it mixes down and admindown. I guess it's being displayed in the order it is in the test file. However it would be nice to order it by columns[3] - I'm sure it's possibly and probably straight forward. However it's not critical, more of a nice to know... as the plans for this would be to now put the node names into another process/function and search log files using the node name as a search criteria and then filter results according to keywords to look out for, like CATERR or MCE or CORRECTABLE etc.

In the course I did online there was a section on opening files, and reading them line by line.. so hopefully that will get me started on this.

This is not something that needs to be written at all, I have written shell scripts which do all of this.. but I want to learn Python and thought what better way than to give myself a challenge.

pan64 · 02-11-2017, 05:17 AM

there are sort and filter functions available in python, so theoretically you can do something like this:

Code:

for line in filtered result[1:]:
- or
for line in sorted result[1:]:

but it is not real python code at this moment, you need to specify how to sort/filter. If you really wish to go into details just tell us....

jonnybinthemix · 02-11-2017, 05:21 AM

Quote:

Originally Posted by pan64

there are sort and filter functions available in python, so theoretically you can do something like this:

Code:

for line in filtered result[1:]:
- or
for line in sorted result[1:]:

but it is not real python code at this moment, you need to specify how to sort/filter. If you really wish to go into details just tell us....

I initially just wanted to have all the "down" together and all the "admindown" together. It's not all that important though.. it's more just for my understanding

jonnybinthemix · 02-11-2017, 02:00 PM

Quote:

Originally Posted by jonnybinthemix

I initially just wanted to have all the "down" together and all the "admindown" together. It's not all that important though.. it's more just for my understanding

This it not really relevant.. as I don't really want to display the down nodes at all yet in the script.. it was merely for my own knowledge.

I have been working on what to do with the list of nodes after I have them, and that would be to collate them into a list and/or variable or something, to be able to loop through them and check for their existence in a set of log files.

I have played with the following:

Code:

#!/usr/bin/python
#

from subprocess import *
import sys

ssh = Popen(["ssh", "root@boot", 'xtprocadmin'],
            shell=False,
            stdout=PIPE,
            stderr=PIPE)
result = ssh.stdout.readlines()

if result == []:
    error = ssh.stderr.readlines()
    print >>sys.stderr, "ERROR: %s" % error
else:
        for line in result[1:]:
            columns = line.split()
            status = columns[3]
            nodes = columns[2]
            if status == "down":
                #print "%s is down" % nodes
                down_nodes = nodes
            if status == "admindown":
                #print "%s is admindown" % nodes
                admin_down_nodes = nodes

print down_nodes+admin_down_nodes
#down = down_nodes+admin_down_nodes

However, when I test the two variables down_nodes and admin_down_nodes I just see the first line. I guess this is because it's overwriting the variables on each iteration of the loop, so in actual fact I'm probably seeing the last line? You can see I experimented with the ability to take those two other variables and create a single variable with all node names... however, is there a way that I can tidy this up by creating a variable in which each iteration of the loop for both instances (down and admindown) will be appended to the variable, and as such I will end up with one variable with a simple list of node names?

jonnybinthemix · 02-11-2017, 02:14 PM

I have tried things like this:

Code:

            if status == "down":
                #print "%s is down" % nodes
                down_nodes = down_nodes.append(nodes)
            if status == "admindown":
                #print "%s is admindown" % nodes
                down_nodes = down_nodes.append(nodes)

It feels like I'm starting to think the right way, but still new enough to mess up all the time. Am I close?

EDIT:

The error I was getting with the above was "down_nodes not defined" so I added down_nodes = "" to the top, in a vein attempt to declare it as a variable before the loop runs. Which did change the error message to: AttributeError: 'str' object has no attribute 'append'

EDIT AGAIN:

I think I have just worked out myself that append() is just for lists, and not for strings. Hmmm... thinking......

jonnybinthemix · 02-11-2017, 02:31 PM

Code:

#!/usr/bin/python
#

from subprocess import *
import sys


ssh = Popen(["ssh", "root@boot", 'xtprocadmin'],
            shell=False,
            stdout=PIPE,
            stderr=PIPE)
result = ssh.stdout.readlines()

down_nodes=[]

if result == []:
    error = ssh.stderr.readlines()
    print >>sys.stderr, "ERROR: %s" % error
else:
        for line in result[1:]:
            columns = line.split()
            status = columns[3]
            nodes = columns[2]
            if status == "down":
                down_nodes = down_nodes.append(nodes)
            if status == "admindown":
                down_nodes = down_nodes.append(nodes)

print down_nodes

I thought with down_nodes=[] I would be declaring it as a list (albeit empty) - and then with down_nodes = down_nodes.append(nodes) I would be appending the list. But maybe.............

....as I was typing, I realised.. I'm thinking too much like a shell script!! var = var+something!

I changed it to:

Code:

 if status == "down":
                down_nodes.append(nodes)
            if status == "admindown":
                down_nodes.append(nodes)

So now it works, when I print it out.. it's not very attractive, but it doesn't need to be.

Sorry, I probably don't need to post any of this, but I figure if someone else embarks on the same or similar journey as I am, my thoughts along the way may be helpful to them.

pan64 · 02-12-2017, 03:19 AM

additionally instead of:

Code:

            if status == "down":
                down_nodes.append(nodes)
            if status == "admindown":
                down_nodes.append(nodes)

you can write something like this (that is the more pythonish way):

Code:

status_list = ["down", "admindown"]
....
           if status in status_list:
                down_nodes.append(nodes)

jonnybinthemix · 02-12-2017, 03:35 AM

Quote:

Originally Posted by pan64

additionally instead of:

Code:

            if status == "down":
                down_nodes.append(nodes)
            if status == "admindown":
                down_nodes.append(nodes)

you can write something like this (that is the more pythonish way):

Code:

status_list = ["down", "admindown"]
....
           if status in status_list:
                down_nodes.append(nodes)

Ah great, thanks. That's much better