LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 11-10-2006, 08:45 PM   #1
dideas
LQ Newbie
 
Registered: Nov 2006
Posts: 11

Rep: Reputation: 0
writing and reading at same time


Hi...

i got a file in which i'm writing sql sentences continuosly.
i want each minute a script read that file and execute them, and therefore, delete them, but meanwhile new sentences can be coming again.

how can i do it?
if i read the file and then i empty it, between that 2 process new sentences can be inserted and i would lose them.

if i "block" the file for writing, the process that insert new sentences would give an error... and i would lose sql's also...

help please
 
Old 11-10-2006, 10:38 PM   #2
kr4m3r
LQ Newbie
 
Registered: Nov 2006
Location: san.rr.com
Distribution: Gentoo 2006.0
Posts: 9

Rep: Reputation: 0
Perhaps you should rotate files? Have one being writen to and then being read, after the read file is read, start writing to it and read from the other.

If you don't want to keep the sql statments around then you could just write and read from a buffer.

Last edited by kr4m3r; 11-10-2006 at 10:39 PM.
 
Old 11-11-2006, 04:46 AM   #3
dideas
LQ Newbie
 
Registered: Nov 2006
Posts: 11

Original Poster
Rep: Reputation: 0
Hi:

1) if i rotate files, how can i do it ? the problem is that while i do something with the file (emptying it, storing or rotating it, ect...) new sentences can be coming in that "short" space of time...

2) in that case (don't need to store them)... tell me about the buffer... i do not know that option...

tip: the reason about executing that script (reading the sql sentences) each minute is to save time opening/closeing mysql connection, because it's a huge load of sentences.

thanx

Last edited by dideas; 11-11-2006 at 05:05 AM.
 
Old 11-11-2006, 05:04 AM   #4
kr4m3r
LQ Newbie
 
Registered: Nov 2006
Location: san.rr.com
Distribution: Gentoo 2006.0
Posts: 9

Rep: Reputation: 0
Well, if you allocated a space in memory you could write to it until 1 minute has passed, then copy the contents to another location in memory, clear the first buffer, and then, on another thread if you like, execute them. That should be fast enough to retain all of your sql infomation.

To do that you would probably have to use a programming language instead of scripting.
 
Old 11-11-2006, 05:21 AM   #5
dideas
LQ Newbie
 
Registered: Nov 2006
Posts: 11

Original Poster
Rep: Reputation: 0
yeah, i'm doing in C... do you know any example i can see ?
any tutorial of this ?

i'd appreciate it !!!

Although... i'm wondering... between your steps "then copy the contents to another location in memory" and "clear the first buffer" , if other sentences goes to the buffer to be written... would they be lost ?

Last edited by dideas; 11-11-2006 at 05:22 AM.
 
Old 11-11-2006, 08:00 AM   #6
jlinkels
Senior Member
 
Registered: Oct 2003
Location: Bonaire
Distribution: Debian Wheezy/Jessie/Sid, Linux Mint DE
Posts: 4,188

Rep: Reputation: 523Reputation: 523Reputation: 523Reputation: 523Reputation: 523Reputation: 523
Well, the pure nature of a file is such that it is a bit difficult what you want. Isn't possible for the program that generates the SQL statements to redirect the output to another progran?

In that you could simply write a piece of C which read from stdin, puts statements in a ring buffer, and read out the ring buffer periodically.

jlinkels
 
Old 11-11-2006, 08:06 AM   #7
Wim Sturkenboom
Senior Member
 
Registered: Jan 2005
Location: Roodepoort, South Africa
Distribution: Slackware 10.1/10.2/12, Ubuntu 12.04, Crunchbang Statler
Posts: 3,786

Rep: Reputation: 282Reputation: 282Reputation: 282
As there are two independent processes, you have to implement a synchronisation mechanism.

For files, you can look at file locking (I don't have man pages available on this machine to check the exact usage and terms (try man flock and man fcntl)). I used it before for a C program, I don't know if it can be done with a shell script. If the latter is the case, it can set a lock on the file so your C-program can't write to it till the lock is released.
 
Old 11-11-2006, 08:46 AM   #8
dideas
LQ Newbie
 
Registered: Nov 2006
Posts: 11

Original Poster
Rep: Reputation: 0
Wim Sturkenboom: if i do that, while locking, the program that writes new sql sentences to that file couldn't do it, and i would lose sql queries.

--------

jlinkels:
Quote:
Isn't possible for the program that generates the SQL statements to redirect the output to another progran?
Yes, it's possible...

Quote:
In that you could simply write a piece of C which read from stdin, puts statements in a ring buffer, and read out the ring buffer periodically.
hmm, i do not understand that part exactly... ring buffer ?

please, explain me, ty in advance !
 
Old 11-11-2006, 12:47 PM   #9
dideas
LQ Newbie
 
Registered: Nov 2006
Posts: 11

Original Poster
Rep: Reputation: 0
is this ?
http://en.wikipedia.org/wiki/Ring_Buffer
http://elonen.iki.fi/code/misc-notes...fer/index.html

what i do not understand is how to implement in my case
 
Old 11-11-2006, 04:34 PM   #10
kr4m3r
LQ Newbie
 
Registered: Nov 2006
Location: san.rr.com
Distribution: Gentoo 2006.0
Posts: 9

Rep: Reputation: 0
I beleive a circular buffer could look something like this:

Code:
char * pPointer, pRead, pWrite = (char *) malloc(27);
Writing:
Code:
while (!feof())
{
  while (pWrite == pRead); // Buffer is full
  *pWrite = fgetc(stdin);
  pWrite = pPointer + (((pWrite - pPointer) + 1) % sizeof(pPointer));
}
Reading:
Code:
while (true)
{
  while (pRead == pWrite) if (feof(stdin)) break; // Buffer is empty
  fprintf(stderr,"%c",*pRead);
  pRead = pPointer + (((pRead - pPointer) + 1) % sizeof(pPointer));
}
Both of these loops would be running at the same time, and of course, you would be reading in a sql statment and writing it into the buffer, and instead of printing out the letters you would be executing the statments.

I would think that a linear buffer would be more appropriate for your case, unless you want to proccess sql statments as they come in.

Sorry, now I have a question:
How do you create threads in linux?

Last edited by kr4m3r; 11-11-2006 at 05:21 PM.
 
Old 11-11-2006, 08:27 PM   #11
dideas
LQ Newbie
 
Registered: Nov 2006
Posts: 11

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by kr4m3r
I think that a linear buffer would be more appropriate for your case, unless you want to proccess sql statments as they come in.
No, precisely... i'm doing all this stuff because i want to execute sql's in groups of "1 minute", i mean, to execute all the "INSERT's" of the last minute, because in that way i "save" time opening/closing mysql connection just 1 time per minute.

That's because i'm looking a way to "buffer" sql's and process one time each minute, but that "buffer" is always receiving new stuff, that's because i sill have the problem

Quote:
Originally Posted by kr4m3r
Sorry, now I have a question:
How do you create threads in linux?
I do not what u're talking about... threads? i don't know!
 
Old 11-12-2006, 02:08 AM   #12
kr4m3r
LQ Newbie
 
Registered: Nov 2006
Location: san.rr.com
Distribution: Gentoo 2006.0
Posts: 9

Rep: Reputation: 0
You could read all of the statments one at a time and execute them one at a time until there are no more, and do that again in one minute.

If you have to read them all at once and you dont know the total size then you will just have to use a large linear buffer.

If you were using a file on an ext2 file system you would be able to store 4GB of sql data. So if you are expecting a LARGE amount of data files would probly be a better idea.

How are you reading the sql data into your c program?
 
Old 11-12-2006, 05:13 AM   #13
dideas
LQ Newbie
 
Registered: Nov 2006
Posts: 11

Original Poster
Rep: Reputation: 0
Not reading them yet, that's i'm looking for a nice way here

Yes, i want to load all data from "the last minute" and execute them... How can i do that "linear" buffer then?

I'm not expecting too much size, because the data in the buffer is tiny "compressed" in a way that i generate the sql's from them, i mean, i do not send all "insert" sql, but just the values ordered, and i create the sql after.

anyway, i need a way to load "the last minute" of that "sql feed", deleting it and meanwhile the feed does not stop to receive more.
 
Old 11-12-2006, 07:06 AM   #14
Wim Sturkenboom
Senior Member
 
Registered: Jan 2005
Location: Roodepoort, South Africa
Distribution: Slackware 10.1/10.2/12, Ubuntu 12.04, Crunchbang Statler
Posts: 3,786

Rep: Reputation: 282Reputation: 282Reputation: 282
Quote:
Originally Posted by dideas
Wim Sturkenboom: if i do that, while locking, the program that writes new sql sentences to that file couldn't do it, and i would lose sql queries.
Sorry, maybe I misunderstood you. I assumed that the program that is writing to the file is written by you in C and that the program that reads is a script written by you.

If the first is the case, you can modify the code to wait till the lock is cleared and in the meantime buffer the data.
 
Old 11-12-2006, 01:02 PM   #15
dideas
LQ Newbie
 
Registered: Nov 2006
Posts: 11

Original Poster
Rep: Reputation: 0
let's see... all is written by me in C, that's right

I got a program in C that (after doing stuff from the web page) generates necessary SQL (inserts, updates, deletes, etc...). Now i'm in the point of just print the sql's in stdout...

Now, because it's a huge ammount of sql, i decided to execute them by "groups" instead executing one by one (changing the printf by mysql_query function :P )

I want to do it in this way because i will optimize and get more performance with sql statements doing by groups... so it will be just one connection, a lot of queries nice optimized, and then close connection.

I decided to do it each minute, so in any way i have to save that sql's in a file, buffer, whatever, to read it with a another program (in C by me, with your help ). The problem as i described is that while i'm reading and emptying that sql's from the "place", the first program is still writing.

that's the point :P

any help ?
 
  


Reply

Tags
buffer


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
reading and writing floppydisks anatman Linux - Newbie 3 05-22-2006 01:32 AM
writing and reading from a file! sahel Programming 1 12-27-2005 02:33 PM
C File reading and writing AbhishekSamuel Programming 3 05-03-2005 04:59 PM
Help reading and writing to file xiste Programming 1 04-15-2005 01:43 AM
Reading and Writing DVD media elsabio Fedora 2 03-06-2005 07:25 PM


All times are GMT -5. The time now is 08:45 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration