LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 05-13-2020, 09:30 PM   #1
Skaperen
Senior Member
 
Registered: May 2009
Location: WV, USA
Distribution: Xubuntu, Slackware, Amazon Linux
Posts: 2,028
Blog Entries: 20

Rep: Reputation: 128Reputation: 128
open with delayed creation


i would like to be able to open a file without actually creating it until the first write is performed. is there a way to do that without having a 2nd process doing that, yet get the file descriptor?
 
Old 05-14-2020, 12:11 AM   #2
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 7.7 (?), Centos 8.1
Posts: 17,774

Rep: Reputation: 2537Reputation: 2537Reputation: 2537Reputation: 2537Reputation: 2537Reputation: 2537Reputation: 2537Reputation: 2537Reputation: 2537Reputation: 2537Reputation: 2537
Can you explain a bit why you are trying to do this?

If you are looking at 'hiding' it from other processes until it has some data in it, there are various solns depending on the rules/circumstance that apply.
 
Old 05-14-2020, 02:42 PM   #3
NevemTeve
Senior Member
 
Registered: Oct 2011
Location: Budapest
Distribution: Debian/GNU/Linux, AIX
Posts: 4,051

Rep: Reputation: 1469Reputation: 1469Reputation: 1469Reputation: 1469Reputation: 1469Reputation: 1469Reputation: 1469Reputation: 1469Reputation: 1469Reputation: 1469
> is there a way to do that without having a 2nd process doing that

How could a second process help in your problem?
 
Old 05-15-2020, 09:55 AM   #4
dugan
LQ Guru
 
Registered: Nov 2003
Location: Canada
Distribution: distro hopper
Posts: 9,597

Rep: Reputation: 4278Reputation: 4278Reputation: 4278Reputation: 4278Reputation: 4278Reputation: 4278Reputation: 4278Reputation: 4278Reputation: 4278Reputation: 4278Reputation: 4278
Quote:
Originally Posted by Skaperen View Post
i would like to be able to open a file without actually creating it until the first write is performed. is there a way to do that without having a 2nd process doing that, yet get the file descriptor?
No, you can't have a file descriptor without a corresponding file (using the broad *nix definition of "file").

And note that you typically don't keep open-for-writing file handles open anyway. Typically, you'd build the file contents in memory, open the file for writing, write immediately, close immediately.

Last edited by dugan; 05-15-2020 at 09:58 AM.
 
2 members found this post helpful.
Old 05-15-2020, 06:50 PM   #5
Skaperen
Senior Member
 
Registered: May 2009
Location: WV, USA
Distribution: Xubuntu, Slackware, Amazon Linux
Posts: 2,028

Original Poster
Blog Entries: 20

Rep: Reputation: 128Reputation: 128
Quote:
Originally Posted by NevemTeve View Post
> is there a way to do that without having a 2nd process doing that

How could a second process help in your problem?
a 2nd process could have a pipe connected to it and delay the open() until there is data to write. BTDT.
 
Old 05-15-2020, 06:52 PM   #6
Skaperen
Senior Member
 
Registered: May 2009
Location: WV, USA
Distribution: Xubuntu, Slackware, Amazon Linux
Posts: 2,028

Original Poster
Blog Entries: 20

Rep: Reputation: 128Reputation: 128
Quote:
Originally Posted by chrism01 View Post
Can you explain a bit why you are trying to do this?

If you are looking at 'hiding' it from other processes until it has some data in it, there are various solns depending on the rules/circumstance that apply.
so there is never a time when the file exists but is empty.
 
Old 05-15-2020, 06:59 PM   #7
scasey
Senior Member
 
Registered: Feb 2013
Location: Tucson, AZ, USA
Distribution: CentOS 7.8.2003
Posts: 4,829

Rep: Reputation: 1764Reputation: 1764Reputation: 1764Reputation: 1764Reputation: 1764Reputation: 1764Reputation: 1764Reputation: 1764Reputation: 1764Reputation: 1764Reputation: 1764
Quote:
Originally Posted by dugan View Post
And note that you typically don't keep open-for-writing file handles open anyway. Typically, you'd build the file contents in memory, open the file for writing, write immediately, close immediately.
Yes. Simply don’t open the file until you’re ready to write to it...
 
1 members found this post helpful.
Old 05-15-2020, 07:02 PM   #8
Skaperen
Senior Member
 
Registered: May 2009
Location: WV, USA
Distribution: Xubuntu, Slackware, Amazon Linux
Posts: 2,028

Original Poster
Blog Entries: 20

Rep: Reputation: 128Reputation: 128
Quote:
Originally Posted by dugan View Post
No, you can't have a file descriptor without a corresponding file (using the broad *nix definition of "file").

And note that you typically don't keep open-for-writing file handles open anyway. Typically, you'd build the file contents in memory, open the file for writing, write immediately, close immediately.
i have never written any code in C that builds the whole file content in memory before writing it. but i have had to do that a few times in Python. my C programs always did a appropriate write when the data to be written was available. i often has cases where data was buffered and no more that one buffer was written (small file) so in those cases, sure, it did end up collection the whole file. but it could have had large data and written many buffers.
 
Old 05-16-2020, 04:12 AM   #9
GazL
LQ Guru
 
Registered: May 2008
Posts: 5,497
Blog Entries: 14

Rep: Reputation: 3328Reputation: 3328Reputation: 3328Reputation: 3328Reputation: 3328Reputation: 3328Reputation: 3328Reputation: 3328Reputation: 3328Reputation: 3328Reputation: 3328
One common approach is to first create the file with a temporary name, then rename it once it is "ready". You often see this with file downloaders, where they'll add a .partial suffix until the file is complete.

Failing that, you can look at the O_TMPFILE flag of open(2).
 
1 members found this post helpful.
Old 05-16-2020, 11:00 AM   #10
dugan
LQ Guru
 
Registered: Nov 2003
Location: Canada
Distribution: distro hopper
Posts: 9,597

Rep: Reputation: 4278Reputation: 4278Reputation: 4278Reputation: 4278Reputation: 4278Reputation: 4278Reputation: 4278Reputation: 4278Reputation: 4278Reputation: 4278Reputation: 4278
Quote:
Originally Posted by Skaperen View Post
i have never written any code in C that builds the whole file content in memory before writing it. but i have had to do that a few times in Python. my C programs always did a appropriate write when the data to be written was available. i often has cases where data was buffered and no more that one buffer was written (small file) so in those cases, sure, it did end up collection the whole file. but it could have had large data and written many buffers.
Well I'm sorry to hear that you haven't written C since the 80s, but...

Is your current target platform one where memory constraints would dictate an approach like this?

Would your current target platform perform better with a few large buffers or many small ones? Hint: does the target platform have a CPU cache?

Also keep in mind that on some platforms, sporadically writing many small files would create more disk fragmentation than writing a single large file in one operation.

Finally, if you're keeping the file open and locked throughout the lifetime of the application, well, that's not the way you're supposed to do it on *nix. Locking is supposed to be done only when necessary.

Last edited by dugan; 05-16-2020 at 11:11 AM.
 
Old 05-16-2020, 11:27 AM   #11
EdGr
Member
 
Registered: Dec 2010
Location: California, USA
Distribution: I run my own OS
Posts: 371

Rep: Reputation: 165Reputation: 165
I have written code that creates a temporary file, writes to it, closes it, and renames the temporary file over an existing file. The key feature is that the update appears atomic. I am guessing that Skaperen really wants atomic update.
Ed
 
Old 05-16-2020, 02:48 PM   #12
Skaperen
Senior Member
 
Registered: May 2009
Location: WV, USA
Distribution: Xubuntu, Slackware, Amazon Linux
Posts: 2,028

Original Poster
Blog Entries: 20

Rep: Reputation: 128Reputation: 128
Quote:
Originally Posted by scasey View Post
Yes. Simply donít open the file until youíre ready to write to it...
which means i will need to modify the library i will be calling to do the processing and output. i don't have source. it's not free.
 
Old 05-17-2020, 03:49 AM   #13
NevemTeve
Senior Member
 
Registered: Oct 2011
Location: Budapest
Distribution: Debian/GNU/Linux, AIX
Posts: 4,051

Rep: Reputation: 1469Reputation: 1469Reputation: 1469Reputation: 1469Reputation: 1469Reputation: 1469Reputation: 1469Reputation: 1469Reputation: 1469Reputation: 1469
Could you please explain X in this XY-problem?
 
1 members found this post helpful.
Old 05-17-2020, 04:48 AM   #14
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 18,887

Rep: Reputation: 3205Reputation: 3205Reputation: 3205Reputation: 3205Reputation: 3205Reputation: 3205Reputation: 3205Reputation: 3205Reputation: 3205Reputation: 3205Reputation: 3205
I though that was my line ...
 
1 members found this post helpful.
Old 05-17-2020, 06:11 AM   #15
MadeInGermany
Senior Member
 
Registered: Dec 2011
Location: Simplicity
Posts: 1,392

Rep: Reputation: 624Reputation: 624Reputation: 624Reputation: 624Reputation: 624Reputation: 624
In the application you can have a framework that opens the file at the first writing. For example awk:
Code:
awk -F: '$7~/\/zsh/ { print $1 > "outfile" }' /etc/passwd
This writes all users with a zsh login shell to outfile.
If no user got a zsh then outfile is not created at all.

Last edited by MadeInGermany; 05-17-2020 at 06:14 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Automatic User Creation with SSH Key creation kjsubbu Linux - Security 5 07-21-2014 09:15 AM
LXer: Fully open sourced JavaFX delayed LXer Syndicated Linux News 0 10-22-2012 03:31 PM
[SOLVED] Prevent subdirectory creation while allowing file creation bweddell Linux - Security 5 07-31-2011 08:54 PM
LXer: Open sourcing of Canonical's Launchpad delayed LXer Syndicated Linux News 0 07-20-2009 03:11 PM
System user creation with mysql user creation moos3 Programming 1 08-11-2007 08:01 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 10:18 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration