LinuxQuestions.org - python pickle dump and load

- Programming (https://www.linuxquestions.org/questions/programming-9/)

- - python pickle dump and load (https://www.linuxquestions.org/questions/programming-9/python-pickle-dump-and-load-707557/)

python pickle dump and load

Hello,

I have a python question. I am new to python and to programing.

I wrote a program and ran it.
I have a list List_1[]. In List_1[] I store values(x,y) tuple. Over time the list grows. I store 1200 of these (x,y) values in the list. As the list grows to 1200 I dump, using pickle, the first 200 values into a file on the hard drive and remove the first 200 (x,y)'s from the list. The list grows again to 1200 and again I dump the first 200 into the file on the hard drive and remove 200 from the list. After hours of this You would think that there would be more than 200 values in the file on the hard drive, but there is not.

while the program was running, I watched as the file size grew on the hard drive as it was being dumped to, thinking I was appending to the file. Obviously pickle is not appending to the dump file just because I opened the file with 'ab'. But if it were overwriting the file it would seem that the file size on disk would not grow while being dumped to.

Code:

CH_1 = []



if len(CH_1) == 1200:

    list = open('pointlist', 'ab')

    pickle.dump(CH_1[0:200], list)

    list.close()

    del CH_1[0:200]

How do I read 200 out of CH_1[] and append that data to a file on the hard drive for later use?
Do I need to create separate files every time 200 is dumped and then later on rebuild all the files?

are you using pickle properly. check the documentation for examples on how to use pickle.

ghostdog74,
Thanks for reading my post.

Just because I said

Quote:

I am new to python and to programing.

Does not mean that I am new.

I have read the docs. Been reading them for some time and I either missed something or missunderstood something. I do not think pickle is the way to go for what I am trying to do but I could be wrong. That is why I asked.

Quote:

How do I read 200 out of CH_1[] and append that data to a file on the hard drive for later use?
Do I need to create separate files every time 200 is dumped and then later on rebuild all the files?

With all due respect, If you aint got anything better to say than go read the doc's dont bother answering.

Quote:

Obviously pickle is not appending to the dump file just because I opened the file with 'ab'. But if it were overwriting the file it would seem that the file size on disk would not grow while being dumped to.

Pickle should be appending data values to the file - it's just that it will append in blocks of 201 values so in effect the file will store values for several lists of length 201. When you open and load values back they will be assigned in blocks of lists 201 values long.

If you wish to restore all values you should organize a loop to load values and possibly list.extend() onto the end of a list to end up with one long list. You could implement a try: except EOFError: loop to load all values and test for EOF.

bgeddy,

Thanks for your reply.
What you say makes a great deal of sense. Earlier, When I rebuilt the data and ploted it out it would plot out the first 200 points, and that was it. I then set up the program to dump 400, collected some points and it ploted 400. Always the first set of points. Anyway I will be working on a loop to solve this like you say. Thanks again for your response.

P.S.

Say Hi to my brother Stewart next time you get to Wantage.

This is basically the kind of thing I meant to restore the values in a loop:

Code:

df=open("pointlist","rb")

CH_1=[]

while 1:

  try: CH_1.extend(pickle.load(df))

  except EOFError: break

Should give you some pointers.