LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 06-05-2020, 06:51 PM   #31
Skaperen
Senior Member
 
Registered: May 2009
Location: WV, USA
Distribution: Xubuntu, Slackware, Amazon Linux
Posts: 2,028

Original Poster
Blog Entries: 20

Rep: Reputation: 128Reputation: 128

Quote:
Originally Posted by Geist View Post
Even if you don't have access to this library, you have access to your own code.
So if the library wants to write a file immediately, then simply don't call the library functions until you want to.
That should work, right?

Keep your own copy of whatever you want to ultimately write in memory and only call the library function when you want to.
when my code does, eventually, call that library code, there will still be a period of time between when it calls open() and when it calls the first write().

i'm thinking to create an empty file distinct to the target file, though not the same exact name as the target file, and lock on that file to avoid concurrently doing this part in my code. that will at least be a start. locks will need to be added to other code, too.
 
Old 06-06-2020, 12:23 AM   #32
Geist
Member
 
Registered: Jul 2013
Distribution: Slackware 14 / current
Posts: 394

Rep: Reputation: Disabled
Quote:
Originally Posted by Skaperen View Post
when my code does, eventually, call that library code, there will still be a period of time between when it calls open() and when it calls the first write().

i'm thinking to create an empty file distinct to the target file, though not the same exact name as the target file, and lock on that file to avoid concurrently doing this part in my code. that will at least be a start. locks will need to be added to other code, too.
I don't see how you will be able to do anything then.
This makes it seem like you do one call to the library, which then doesn't return until it writes the file, which entails opening it first for some time while it does some operation that takes some time.

I can't think of a reliable, non crazy way to use a second thread to somehow balance the time between the library call starts (and creates the file) and finishes its prepwork to write to it.

There's so many variables.
Why does it even matter?

Security?
Well then encrypt the data first before you send it to the library and then decrypt it in a post processing step.
 
Old 06-06-2020, 12:59 PM   #33
Skaperen
Senior Member
 
Registered: May 2009
Location: WV, USA
Distribution: Xubuntu, Slackware, Amazon Linux
Posts: 2,028

Original Poster
Blog Entries: 20

Rep: Reputation: 128Reputation: 128
yeah, there's no hard definitive solution under the I/O model Posix/Unix/Linux uses. there's only designing around the problem to be sure no issues can happen in the first place. and that might mean having some way to lock against double uses of the file before some like first write or file completion.

there is a related problem. it's related because these programs will often take all day to complete the file due to the nature of what they are doing. consider opening a file for appending. now consider 2 processes opening the very same file for appending.
 
Old 06-12-2020, 09:43 PM   #34
Skaperen
Senior Member
 
Registered: May 2009
Location: WV, USA
Distribution: Xubuntu, Slackware, Amazon Linux
Posts: 2,028

Original Poster
Blog Entries: 20

Rep: Reputation: 128Reputation: 128
if the kernel (of any POSIX OS) were changed to work as i described, what would break? any applications? any libraries? is it something that would be harder to implement had the kernel done it this way since its first creation?
 
Old 06-12-2020, 11:13 PM   #35
EdGr
Member
 
Registered: Dec 2010
Location: California, USA
Distribution: I run my own OS
Posts: 368

Rep: Reputation: 164Reputation: 164
Zero-length files are valid and useful. They indicate the absence of something. The "test -s" command checks for non-zero-length files. I have used it many times.

I am still not sure why you want to avoid zero-length files.
Ed
 
Old 06-14-2020, 01:55 PM   #36
Skaperen
Senior Member
 
Registered: May 2009
Location: WV, USA
Distribution: Xubuntu, Slackware, Amazon Linux
Posts: 2,028

Original Poster
Blog Entries: 20

Rep: Reputation: 128Reputation: 128
zero length files have value where intended. i am trying to avoid them where they are not intended.
 
Old 06-14-2020, 08:33 PM   #37
EdGr
Member
 
Registered: Dec 2010
Location: California, USA
Distribution: I run my own OS
Posts: 368

Rep: Reputation: 164Reputation: 164
We posted the solution: create a temporary file, write to it, and then rename the temporary file to the desired name.
Ed
 
1 members found this post helpful.
Old 06-15-2020, 01:20 AM   #38
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 7.7 (?), Centos 8.1
Posts: 17,774

Rep: Reputation: 2537Reputation: 2537Reputation: 2537Reputation: 2537Reputation: 2537Reputation: 2537Reputation: 2537Reputation: 2537Reputation: 2537Reputation: 2537Reputation: 2537
... and create it under /tmp or similar ie somewhere that gets cleaned out by the OS if it gets rebooted.
 
Old 06-15-2020, 05:59 AM   #39
GazL
LQ Guru
 
Registered: May 2008
Posts: 5,497
Blog Entries: 14

Rep: Reputation: 3328Reputation: 3328Reputation: 3328Reputation: 3328Reputation: 3328Reputation: 3328Reputation: 3328Reputation: 3328Reputation: 3328Reputation: 3328Reputation: 3328
...or, as I said back in post #9, open() it with O_TMPFILE: though that is a Linux specific solution and is not portable.
 
1 members found this post helpful.
Old 06-15-2020, 06:17 PM   #40
Skaperen
Senior Member
 
Registered: May 2009
Location: WV, USA
Distribution: Xubuntu, Slackware, Amazon Linux
Posts: 2,028

Original Poster
Blog Entries: 20

Rep: Reputation: 128Reputation: 128
Quote:
Originally Posted by EdGr View Post
We posted the solution: create a temporary file, write to it, and then rename the temporary file to the desired name.
Ed
the contrary case is something that runs for hours and hours collecting some statistical data that needs to be reviewed periodically all day. and this data needs to not be lost if the system reboots, eliminating the use of /tmp.
 
Old 06-15-2020, 06:17 PM   #41
Skaperen
Senior Member
 
Registered: May 2009
Location: WV, USA
Distribution: Xubuntu, Slackware, Amazon Linux
Posts: 2,028

Original Poster
Blog Entries: 20

Rep: Reputation: 128Reputation: 128
Quote:
Originally Posted by GazL View Post
...or, as I said back in post #9, open() it with O_TMPFILE: though that is a Linux specific solution and is not portable.
you sure did. and i overlooked it thinking it was something different. that does look like a doable approach. i'll have to make sure there is a lock to prevent duplicate attempts. but this is certainly a new, good, perspective on the problem.
 
Old 06-16-2020, 06:35 AM   #42
GazL
LQ Guru
 
Registered: May 2008
Posts: 5,497
Blog Entries: 14

Rep: Reputation: 3328Reputation: 3328Reputation: 3328Reputation: 3328Reputation: 3328Reputation: 3328Reputation: 3328Reputation: 3328Reputation: 3328Reputation: 3328Reputation: 3328
Well, it's an option, but after reading your later posts (#20 specifically), IMO you're approaching the problem from the wrong end. Catch SIGTERM/SIGINT and give it a handler that will remove an empty file before exiting; then you can simply rely on open(filename, O_CREAT|O_EXCL) semantics to provide serialisation.
 
Old 06-19-2020, 07:54 PM   #43
Skaperen
Senior Member
 
Registered: May 2009
Location: WV, USA
Distribution: Xubuntu, Slackware, Amazon Linux
Posts: 2,028

Original Poster
Blog Entries: 20

Rep: Reputation: 128Reputation: 128
i'm trying to work out a way where an empty file doesn't matter. i can't reveal the code at that level.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Automatic User Creation with SSH Key creation kjsubbu Linux - Security 5 07-21-2014 09:15 AM
LXer: Fully open sourced JavaFX delayed LXer Syndicated Linux News 0 10-22-2012 03:31 PM
[SOLVED] Prevent subdirectory creation while allowing file creation bweddell Linux - Security 5 07-31-2011 08:54 PM
LXer: Open sourcing of Canonical's Launchpad delayed LXer Syndicated Linux News 0 07-20-2009 03:11 PM
System user creation with mysql user creation moos3 Programming 1 08-11-2007 08:01 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 01:30 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration