LinuxQuestions.org - partial serialize/deserialize a structure

- Programming (https://www.linuxquestions.org/questions/programming-9/)

- - partial serialize/deserialize a structure (https://www.linuxquestions.org/questions/programming-9/partial-serialize-deserialize-a-structure-486963/)

partial serialize/deserialize a structure

Hello everyone,

I have a large number of strcture instances, the I can not have all of them in memory. So I have the idea of serialize/deserialize partial of them to/from disk. The operation I need on these structure instances is to enumerate elements one by one, and I want to provide applications which use this interface to enumerate elements be transparent to how and which parts of structure instances is written/read to/from disk -- and the application only will feel read from memory.

I am wondering whether there are any samples/tutorials to make a reference?

thanks in advance,
George

Sounds like something that i was thinking of doing for my cd program. Only way that i can think of is to dump some to disk using a fwrite function and then freeing the memory that is pointed to by some of your structures. but to show you an example would require large amount of code and ive yet to preform this action sucessfully my self :{

Quote:

Originally Posted by exvor

Only way that i can think of is to dump some to disk using a fwrite function and then freeing the memory that is pointed to by some of your structures.

I think you need to do something like this, however you shouldn't directly export the data from memory (i.e. by casting structs to 'unsigned char*' and copying); you should have 'save' and 'load' functions for your objects so that they can safely save/load. This will prevent pointer/dynamic allocation errors and memory leaks.
ta0kira

Thank you exvor,

Quote:

Originally Posted by exvor

I have the same idea. If you have found some samples, please let me know.

regards,
George

Thank you ta0kira,

Quote:

Originally Posted by ta0kira

I think your idea is feasible and practical. Do you have any samples to make a reference?

regards,
George

I have acutally gotten my code to work properly now but im not sure its really what your trying to do. In my example it only works with a struct called datanode that gets passed to my read and write functions. Im not working with a list of stuff.

anyway here is my write function (it will be totaly dismantaled and rewritten so that It has more functionality.) in the new version it wont look for the correct place anymore but rather take a file postion from my list function and just write data using that file postion. hope this gets your ideas going tho.

Code:

 



#include<stdio.h>

#include<stdlib.h>

#include"cddat.h"





int addcd(DATANODE *infoadd,const char *database,int option)

{

    FILE *cddat;

    int number;

    int loop = 0;

    DATANODE *chek;  



    if ((cddat = fopen(database,"r+")) == NULL)

      {

      printf("\nError writeing to file!. func addcd\n");

      return 1;

      }



    number = infoadd->cdnum; 

    

    if (option == 1)

    { 

        





        fwrite(infoadd,sizeof(DATANODE),1,cddat);

    }

    else if (option == 2) 

      {

        

        while(feof(cddat) == 0)

          { 

            chek = malloc(DATANODE); 

            fseek(cddat,loop * sizeof(DATANODE),SEEK_SET); 

            fread(chek,sizeof(DATANODE),1,cddat); 

            if (chek->cdnum == number)

                    { 

                      free(chek); 

                      fwrite(infoadd,sizeof(DATANODE),1,cddat); 

                      return 0; 

                    } 

            else 

                { 

                  loop++; 

                  free(chek);  

                } 

            }

      return 2; /* This only occurs if the element is not in the file */ 



      }

    else 

      { fclose(cddat); 

        return 1; 

      }

  

  if (fclose(cddat))

            { 

        printf("\nError closeing file. func addcd\n"); 

        return 1; 

      }

  

  else 

    return 0; 

}

as you can see checking for memory issues is not built in yet.

here is the lister by the way it acutally returns file pos now

Code:

 

#include<stdio.h> 

#include<stdlib.h> 

#include"cddat.h"





long int listfile(const char *basename)

{

  FILE *basefile; 

  DATANODE *temphold; 

  int discnum;

  int check;  

  

  int looper = 0; 

  

  if ((basefile = fopen(basename,"r")) != NULL)

 { 

  



  temphold = malloc(sizeof(DATANODE));

  printf("\nWhat disc would you like to access?\n"); 

  printf("DISK#="); 

  scanf("%d",&discnum); 

      

    

  do {

    fseek(basefile,looper * sizeof(DATANODE),SEEK_SET);  

  fread(temphold,sizeof(DATANODE),1,basefile);

  check = temphold->cdnum; 

  printf("\ndiscnum %d, check %d\n",discnum,check); 

  if (discnum == check)

      {



  

      

      

          printf("\nRecord # %d\n",temphold->cdnum); 

          printf("+----------+\n"); 

          printf("| DISC NAME| %s\n",temphold->cdname); 

          printf("+----------+\n");

          printf("| TYPE    | "); 

          switch(temphold->disktype)

            {

            case 1: printf("CD"); 

            break; 

            case 2: printf("DVD"); 

            break; 

            case 3: printf("Data CD"); 

            break; 

            case 4: printf("Data DVD"); 

            break;

            default: printf("UNDEFINED"); 

            break; 

            }



          printf("\n+----------+\n");

          printf("| CONDITION| "); 

          if (temphold->condition == 1)

            printf("BAD");

          else 

            printf("GOOD");

          

          printf("\n+----------+\n"); 

          printf("| LISTFILE | ");

          printf("Note at this time listfiles are disabled");

          printf("\n+----------+\n"); 

          fclose(basefile); 

          free(temphold); 

          return (looper * sizeof(DATANODE)); 

        }

  looper++;

    

  }while(feof(basefile) == 0);



  printf("End of file reached record not found\n"); 

  fclose(basefile); 

  free(temphold); 

  return 0; 

  }

  fclose(basefile); 

  free(temphold); 

  return 1; 

}

[

Quote:

Originally Posted by George2

I think your idea is feasible and practical. Do you have any samples to make a reference?

1) Are you using C or C++?
2) Do your structures hold pointers?
3) (if C++) Do your structures allocate memory or hold objects which do (such as strings)?

If you tell me those things, I should be able to come up with an example (I don't know of any off hand, but I've been working on this type of thing lately.)

If your structures have pointers to other objects that will also be "set aside" then that complicates things quite a bit, but doesn't make it impossible. If you're using C, you can always handle export/import in C++ and provide 'extern "C"' functions for the C portion.

Have you considered using a database library? I'd suggest using that if possible, but you'd still have to create functions to safely extract/insert the data from the structures. You'd just have those functions write and query the database rather than coming up with your own db-type file scheme.
ta0kira

PS Writing and reading are the easy parts; the hard parts are 1) finding the correct stored structure, and 2) making sure it can deal with a memory state that varies from when it was stored.

I think you should simply require a certain amount of VM, and let the kernel do the swapping for you. Swap is almost guaranteed to be faster than trying to save/restore the stuff yourself, especially since the kernel knows what is being used at that specific moment better than you do.

Not tying to hijack the thread but will i have issues with what your saying here.

2) making sure it can deal with a memory state that varies from when it was stored.

if its a structure it should always have the same size when writeing it from disk ?

If one structure holds a pointer to something else and the structure is "saved" then removed from memory (it would have to be for this whole thing to be useful), that which it pointed to might not be there when it gets loaded back into memory. This could happen if the pointer is to another structure (or a component of it) that was subsequently saved + removed. That's why I asked about pointers to other structures; if they're used, then each structure "might" require a "serial number" independent of the pointer which will have to be saved along with it, and those pointers to others will have to be converted to serial numbers, also (all of which would need to be managed by a cache of some sort.)
ta0kira

I agree that it is generally wise to let the virtual-memory subsystem be your "very large disk file." Just be sensible about it. Use data-structures that allow the system to zoom in on a particular key with a minimum of memory accesses. The OS will automatically handle memory management for you and will do it very well.

Ohh my program doesent have pointers to other structures in the structures. but for the original question i can see how that would become a problem as a linked list has pointers to the other elements. When i was condisering doing something like what you were going to do i was going to save all the structures to the disk file and then rebuild the linked list from the data. I never got that out of the concept stage tho so there was probably alot of issues i never thought to work out.

Quote:

Originally Posted by tuxdev

I think you should simply require a certain amount of VM

I agree. Even if the end user has no VM, the program could take whatever space it was going to use as a cache and loop it as swap space.
ta0kira

Hi ta0kira,

Quote:

Originally Posted by ta0kira

My answers,

1. Yes, C++;
2. Not currently. But may use pointer to other data structures. But better to be extensible to hold pointers;
3. What do you mean structures allocate memory? Structures are just static things, how could they allocate memory? :-)

My structures only contain basic types, like int/char/char[], something like these.

regards,
George

Thank you tuxdev!

Quote:

Originally Posted by tuxdev

My first purpose is to limit the amonut of memory my application will use, so that the application will not impact other applications. I think if we use OS built-in functions, which are transparent to users, to swap materials from/to memory/disk, my application may consume too much memory ...

regards,
George