LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Programming (https://www.linuxquestions.org/questions/programming-9/)
-   -   open with delayed creation (https://www.linuxquestions.org/questions/programming-9/open-with-delayed-creation-4175675186/)

Skaperen 05-13-2020 09:30 PM

open with delayed creation
 
i would like to be able to open a file without actually creating it until the first write is performed. is there a way to do that without having a 2nd process doing that, yet get the file descriptor?

chrism01 05-14-2020 12:11 AM

Can you explain a bit why you are trying to do this?

If you are looking at 'hiding' it from other processes until it has some data in it, there are various solns depending on the rules/circumstance that apply.

NevemTeve 05-14-2020 02:42 PM

> is there a way to do that without having a 2nd process doing that

How could a second process help in your problem?

dugan 05-15-2020 09:55 AM

Quote:

Originally Posted by Skaperen (Post 6122689)
i would like to be able to open a file without actually creating it until the first write is performed. is there a way to do that without having a 2nd process doing that, yet get the file descriptor?

No, you can't have a file descriptor without a corresponding file (using the broad *nix definition of "file").

And note that you typically don't keep open-for-writing file handles open anyway. Typically, you'd build the file contents in memory, open the file for writing, write immediately, close immediately.

Skaperen 05-15-2020 06:50 PM

Quote:

Originally Posted by NevemTeve (Post 6123026)
> is there a way to do that without having a 2nd process doing that

How could a second process help in your problem?

a 2nd process could have a pipe connected to it and delay the open() until there is data to write. BTDT.

Skaperen 05-15-2020 06:52 PM

Quote:

Originally Posted by chrism01 (Post 6122710)
Can you explain a bit why you are trying to do this?

If you are looking at 'hiding' it from other processes until it has some data in it, there are various solns depending on the rules/circumstance that apply.

so there is never a time when the file exists but is empty.

scasey 05-15-2020 06:59 PM

Quote:

Originally Posted by dugan (Post 6123521)
And note that you typically don't keep open-for-writing file handles open anyway. Typically, you'd build the file contents in memory, open the file for writing, write immediately, close immediately.

Yes. Simply don’t open the file until you’re ready to write to it...

Skaperen 05-15-2020 07:02 PM

Quote:

Originally Posted by dugan (Post 6123521)
No, you can't have a file descriptor without a corresponding file (using the broad *nix definition of "file").

And note that you typically don't keep open-for-writing file handles open anyway. Typically, you'd build the file contents in memory, open the file for writing, write immediately, close immediately.

i have never written any code in C that builds the whole file content in memory before writing it. but i have had to do that a few times in Python. my C programs always did a appropriate write when the data to be written was available. i often has cases where data was buffered and no more that one buffer was written (small file) so in those cases, sure, it did end up collection the whole file. but it could have had large data and written many buffers.

GazL 05-16-2020 04:12 AM

One common approach is to first create the file with a temporary name, then rename it once it is "ready". You often see this with file downloaders, where they'll add a .partial suffix until the file is complete.

Failing that, you can look at the O_TMPFILE flag of open(2).

dugan 05-16-2020 11:00 AM

Quote:

Originally Posted by Skaperen (Post 6123676)
i have never written any code in C that builds the whole file content in memory before writing it. but i have had to do that a few times in Python. my C programs always did a appropriate write when the data to be written was available. i often has cases where data was buffered and no more that one buffer was written (small file) so in those cases, sure, it did end up collection the whole file. but it could have had large data and written many buffers.

Well I'm sorry to hear that you haven't written C since the 80s, but...

Is your current target platform one where memory constraints would dictate an approach like this?

Would your current target platform perform better with a few large buffers or many small ones? Hint: does the target platform have a CPU cache?

Also keep in mind that on some platforms, sporadically writing many small files would create more disk fragmentation than writing a single large file in one operation.

Finally, if you're keeping the file open and locked throughout the lifetime of the application, well, that's not the way you're supposed to do it on *nix. Locking is supposed to be done only when necessary.

EdGr 05-16-2020 11:27 AM

I have written code that creates a temporary file, writes to it, closes it, and renames the temporary file over an existing file. The key feature is that the update appears atomic. I am guessing that Skaperen really wants atomic update.
Ed

Skaperen 05-16-2020 02:48 PM

Quote:

Originally Posted by scasey (Post 6123675)
Yes. Simply donít open the file until youíre ready to write to it...

which means i will need to modify the library i will be calling to do the processing and output. i don't have source. it's not free.

NevemTeve 05-17-2020 03:49 AM

Could you please explain X in this XY-problem?

syg00 05-17-2020 04:48 AM

I though that was my line ... :p

MadeInGermany 05-17-2020 06:11 AM

In the application you can have a framework that opens the file at the first writing. For example awk:
Code:

awk -F: '$7~/\/zsh/ { print $1 > "outfile" }' /etc/passwd
This writes all users with a zsh login shell to outfile.
If no user got a zsh then outfile is not created at all.


All times are GMT -5. The time now is 05:08 AM.