LinuxQuestions.org
Did you know LQ has a Linux Hardware Compatibility List?
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 02-25-2009, 07:10 PM   #1
jabfinger
Member
 
Registered: Jan 2004
Location: USA, Pa
Distribution: Gentoo, Fedora Core 3, Mepis, Vector
Posts: 83

Rep: Reputation: 15
python pickle dump and load


Hello,

I have a python question. I am new to python and to programing.

I wrote a program and ran it.
I have a list List_1[]. In List_1[] I store values(x,y) tuple. Over time the list grows. I store 1200 of these (x,y) values in the list. As the list grows to 1200 I dump, using pickle, the first 200 values into a file on the hard drive and remove the first 200 (x,y)'s from the list. The list grows again to 1200 and again I dump the first 200 into the file on the hard drive and remove 200 from the list. After hours of this You would think that there would be more than 200 values in the file on the hard drive, but there is not.

while the program was running, I watched as the file size grew on the hard drive as it was being dumped to, thinking I was appending to the file. Obviously pickle is not appending to the dump file just because I opened the file with 'ab'. But if it were overwriting the file it would seem that the file size on disk would not grow while being dumped to.



Code:
CH_1 = []

if len(CH_1) == 1200:
    list = open('pointlist', 'ab')
    pickle.dump(CH_1[0:200], list)
    list.close()
    del CH_1[0:200]

How do I read 200 out of CH_1[] and append that data to a file on the hard drive for later use?
Do I need to create separate files every time 200 is dumped and then later on rebuild all the files?
 
Old 02-25-2009, 07:36 PM   #2
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,695
Blog Entries: 5

Rep: Reputation: 240Reputation: 240Reputation: 240
are you using pickle properly. check the documentation for examples on how to use pickle.
 
Old 02-25-2009, 08:50 PM   #3
jabfinger
Member
 
Registered: Jan 2004
Location: USA, Pa
Distribution: Gentoo, Fedora Core 3, Mepis, Vector
Posts: 83

Original Poster
Rep: Reputation: 15
ghostdog74,
Thanks for reading my post.

Just because I said

Quote:
I am new to python and to programing.
Does not mean that I am new.

I have read the docs. Been reading them for some time and I either missed something or missunderstood something. I do not think pickle is the way to go for what I am trying to do but I could be wrong. That is why I asked.

Quote:
How do I read 200 out of CH_1[] and append that data to a file on the hard drive for later use?
Do I need to create separate files every time 200 is dumped and then later on rebuild all the files?
With all due respect, If you aint got anything better to say than go read the doc's dont bother answering.
 
Old 02-25-2009, 11:07 PM   #4
bgeddy
Senior Member
 
Registered: Sep 2006
Location: Liverpool - England
Distribution: slackware64 13.37 and -current, Dragonfly BSD
Posts: 1,810

Rep: Reputation: 227Reputation: 227Reputation: 227
Quote:
Obviously pickle is not appending to the dump file just because I opened the file with 'ab'. But if it were overwriting the file it would seem that the file size on disk would not grow while being dumped to.
Pickle should be appending data values to the file - it's just that it will append in blocks of 201 values so in effect the file will store values for several lists of length 201. When you open and load values back they will be assigned in blocks of lists 201 values long.

If you wish to restore all values you should organize a loop to load values and possibly list.extend() onto the end of a list to end up with one long list. You could implement a try: except EOFError: loop to load all values and test for EOF.
 
Old 02-26-2009, 12:29 AM   #5
jabfinger
Member
 
Registered: Jan 2004
Location: USA, Pa
Distribution: Gentoo, Fedora Core 3, Mepis, Vector
Posts: 83

Original Poster
Rep: Reputation: 15
Smile

bgeddy,

Thanks for your reply.
What you say makes a great deal of sense. Earlier, When I rebuilt the data and ploted it out it would plot out the first 200 points, and that was it. I then set up the program to dump 400, collected some points and it ploted 400. Always the first set of points. Anyway I will be working on a loop to solve this like you say. Thanks again for your response.

P.S.

Say Hi to my brother Stewart next time you get to Wantage.
 
Old 02-26-2009, 07:02 AM   #6
bgeddy
Senior Member
 
Registered: Sep 2006
Location: Liverpool - England
Distribution: slackware64 13.37 and -current, Dragonfly BSD
Posts: 1,810

Rep: Reputation: 227Reputation: 227Reputation: 227
This is basically the kind of thing I meant to restore the values in a loop:

Code:
df=open("pointlist","rb")
CH_1=[]
while 1:
   try: CH_1.extend(pickle.load(df))
   except EOFError: break
Should give you some pointers.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
python update - Unable to load GTK2 Python bindings: No module named gtk itpedersen Linux - Software 2 10-03-2008 03:44 AM
Stack dump during rsync -e ssh ... heavy load! holst Linux - Software 1 09-27-2007 06:41 PM
Load firefox with url - Python Penguin of Wonder Programming 1 02-25-2006 10:29 PM
Resolution Pickle! Please Help! Geepak Linux - Newbie 4 02-08-2005 09:20 AM
Python 2.3 IDLE won't load... arpanet1969 Linux - Software 2 12-31-2004 10:15 AM


All times are GMT -5. The time now is 10:22 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration