LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 05-17-2020, 03:40 PM   #16
Skaperen
Senior Member
 
Registered: May 2009
Location: WV, USA
Distribution: Xubuntu, Slackware, Amazon Linux
Posts: 2,028

Original Poster
Blog Entries: 20

Rep: Reputation: 128Reputation: 128

Quote:
Originally Posted by dugan View Post
Well I'm sorry to hear that you haven't written C since the 80s, but...
i wrote one last month.

Quote:
Originally Posted by dugan View Post
Is your current target platform one where memory constraints would dictate an approach like this?
operability constraints would. if you had to collect a day-long stream of messages from a network connection, would you collect it in memory and write it all at the end of the day? this may be an exception. what is to be gained by your approach?

Quote:
Originally Posted by dugan View Post
Would your current target platform perform better with a few large buffers or many small ones? Hint: does the target platform have a CPU cache?
today's operating systems do equally well in both situations. caching takes care of this.

Quote:
Originally Posted by dugan View Post
Also keep in mind that on some platforms, sporadically writing many small files would create more disk fragmentation than writing a single large file in one operation.
how did this become a many small files situation? i didn't bring up this topic. if you want to discuss it, start a new thread and PM me the URL.

Quote:
Originally Posted by dugan View Post
Finally, if you're keeping the file open and locked throughout the lifetime of the application, well, that's not the way you're supposed to do it on *nix. Locking is supposed to be done only when necessary.
how did this become a file lock situation? there is no need to lock a file to write it sequentially. there are often many alternatives, depending on the application that typically gets implemented with locking.
 
Old 05-17-2020, 03:52 PM   #17
Skaperen
Senior Member
 
Registered: May 2009
Location: WV, USA
Distribution: Xubuntu, Slackware, Amazon Linux
Posts: 2,028

Original Poster
Blog Entries: 20

Rep: Reputation: 128Reputation: 128
Quote:
Originally Posted by MadeInGermany View Post
In the application you can have a framework that opens the file at the first writing. For example awk:
Code:
awk -F: '$7~/\/zsh/ { print $1 > "outfile" }' /etc/passwd
This writes all users with a zsh login shell to outfile.
If no user got a zsh then outfile is not created at all.
awk is designed to do it this way. C isn't. Even Python isn't. when the tool lets you output to a not-yet-opened reference, such as a string with the name of the target file, the issue is solved. i have done many projects in awk, but many others need way beyond its capability. but, at least awk is not buffering the whole contents in memory before actually asking the system to write some of it.
 
Old 05-17-2020, 03:56 PM   #18
rnturn
Senior Member
 
Registered: Jan 2003
Location: Illinois (SW Chicago 'burbs)
Distribution: Currently: openSUSE, Raspbian, Slackware. Formerly: CentOS, MacOS, Red Hat. Other: Solaris, Tru64
Posts: 1,920

Rep: Reputation: 286Reputation: 286Reputation: 286
Quote:
Originally Posted by Skaperen View Post
i have never written any code in C that builds the whole file content in memory before writing it. but i have had to do that a few times in Python. my C programs always did a appropriate write when the data to be written was available.
I remember the days when I had to demonstrate to users that it was actually not faster to process data by reading in an entire file into memory and then write it out all at once after processing the data. Especially on a multi-user system where, because of quotas, you simply will not typically have permission to have access to all available memory. Displaying wall clock time and the post-execution resource utilization statistics accumulated during program execution was the eye-opener for many (this was, I think, easier to do then than it is today on Linux). Sucking an entire dataset into memory cost big time in paging activity and increased run time immensely. And with many users' taking the advice to not allocate swap space, a large dataset will likely have you reaching for the Big Red Switch. (There are times when I think that anyone writing software should be forced to write code on a small memory system for a while---it forces you to think about the problem at hand a bit and work within a finite set of resources.)

Just my $0.02.

Later...

Last edited by rnturn; 05-17-2020 at 03:59 PM.
 
Old 06-01-2020, 10:47 AM   #19
dogpatch
Member
 
Registered: Nov 2005
Location: Central America
Distribution: Mepis, Android
Posts: 413
Blog Entries: 4

Rep: Reputation: 181Reputation: 181
You seem to be saying you never want an empty file. If you want to make sure the file has at least some data in it, then follow the advice above: create the file by opening it in 'w'rite mode when you have some data. Immediately after opening / creating the file, you write the data, and close the file. Subsequent writes, open in 'a'ppend mode, write the data, close the file.

If you want to make sure the file is complete before you create it, then do the above, but with a temporary file name. When it is complete, rename the file to its permanent name.
 
Old 06-01-2020, 02:48 PM   #20
Skaperen
Senior Member
 
Registered: May 2009
Location: WV, USA
Distribution: Xubuntu, Slackware, Amazon Linux
Posts: 2,028

Original Poster
Blog Entries: 20

Rep: Reputation: 128Reputation: 128
i want to replicate a behavior i have seen in a different OS which is equivalent to the file won't exist if the system is shutdown between open and the first write AND if another process attempts to open the same file name for exclusive creation after that first open by the first process before its first write, then it would fail. that kernel logic seems to be that it locks the name when the first successful open is done and actually creates the file in the file system when the first write is done.
 
Old 06-01-2020, 04:03 PM   #21
dogpatch
Member
 
Registered: Nov 2005
Location: Central America
Distribution: Mepis, Android
Posts: 413
Blog Entries: 4

Rep: Reputation: 181Reputation: 181
In that case, create and lock the file. Then create a temporary filename, and right after writing data to it, unlock the permanent file and rename the temp file to its permanent name.

Last edited by dogpatch; 06-01-2020 at 04:06 PM.
 
Old 06-01-2020, 04:03 PM   #22
SoftSprocket
Member
 
Registered: Nov 2014
Posts: 297

Rep: Reputation: Disabled
If you call open, immediately followed by unlink the file is deleted when it is closed i.e. close is called or the program exits.

You might be able to use that behavior to achieve what you're after.
 
Old 06-01-2020, 06:19 PM   #23
Skaperen
Senior Member
 
Registered: May 2009
Location: WV, USA
Distribution: Xubuntu, Slackware, Amazon Linux
Posts: 2,028

Original Poster
Blog Entries: 20

Rep: Reputation: 128Reputation: 128
@dogpatch you mean have 2 files, one to lock the name (created first), and the other to write to and eventually replace the first? hmmm, that might work.

Last edited by Skaperen; 06-01-2020 at 06:21 PM.
 
Old 06-01-2020, 06:26 PM   #24
Skaperen
Senior Member
 
Registered: May 2009
Location: WV, USA
Distribution: Xubuntu, Slackware, Amazon Linux
Posts: 2,028

Original Poster
Blog Entries: 20

Rep: Reputation: 128Reputation: 128
@SoftSprocket the unlink makes that name go away. while it's gone, some other process might open a file at that name. and something needs to be done to bring the file name back and the only ways to do that either create an all new empty file or need to reference an existing name.
 
Old 06-02-2020, 08:40 AM   #25
SoftSprocket
Member
 
Registered: Nov 2014
Posts: 297

Rep: Reputation: Disabled
Ah, quite so - I should have tested first. As my penance I scribbled something that does work:

Code:
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <fcntl.h>
#include <signal.h>
#include <sys/stat.h>

char* reserve_fn;
void delete_on_close () {
	printf ("atexit %s\n", reserve_fn);
	if (reserve_fn != NULL) {
		if (unlink (reserve_fn) > 0) {
			perror ("unlink");
		}
	}
}

int main () {
	reserve_fn = "reserved.txt";

	int fd = creat (reserve_fn, O_RDWR);

	if (fd < 0) {
		perror ("open");
		exit (EXIT_FAILURE);
	}
	fchmod (fd, S_IRUSR | S_IWUSR);
	
	signal(SIGINT, delete_on_close);
	atexit (delete_on_close);

	printf ("sleeping\n");
	sleep (10);

	printf ("Using fd\n");

	reserve_fn = NULL;	

	write (fd, &fd, sizeof fd);

	close (fd);

	return 0;
}
Exit on interupt or when exit is called and the file is removed - unless the reserve_fn is NULL. Not exactly elegant but it does work.
 
Old 06-02-2020, 02:54 PM   #26
Skaperen
Senior Member
 
Registered: May 2009
Location: WV, USA
Distribution: Xubuntu, Slackware, Amazon Linux
Posts: 2,028

Original Poster
Blog Entries: 20

Rep: Reputation: 128Reputation: 128
what's the point of delete_on_vlose() when the final desire is to have the fire? the whole point is to have the file never be empty. you need a way to block the name from being used while it doesn't exist.
 
Old 06-02-2020, 02:59 PM   #27
SoftSprocket
Member
 
Registered: Nov 2014
Posts: 297

Rep: Reputation: Disabled
Your original ask was to prevent the name from being used but delete the file if it isn't used. That is what the example will do. While the program is in the sleep you can interrupt the program and the file will be deleted. However if you don't the file will be preserved. Presuming real work going on before the write, an exit (say after a failed system call) will also remove the file. Did you try the program? If you don't interrupt during the sleep the file will be there.
 
Old 06-04-2020, 01:49 AM   #28
Skaperen
Senior Member
 
Registered: May 2009
Location: WV, USA
Distribution: Xubuntu, Slackware, Amazon Linux
Posts: 2,028

Original Poster
Blog Entries: 20

Rep: Reputation: 128Reputation: 128
what the kernel could do for this kind of open is have an internal set of which object names are open with delayed creation. this can be included in the test for "already exists" so another process cannot open it. but nothing is saved on disk, yet. then when the first write happens, the creation is completed. things like permission need to be tested at open time. the write could fail due to out of space. the file may be created empty if its parent directory does not need another block for it. this might need to be suppressed or reverted if it really is intended to avoid an empty file.

decades ago i worked on IBM mainframes. empty files were "impossible". i never was concerned about why or how enough to investigate (i did have the source code).
 
Old 06-04-2020, 07:51 AM   #29
SoftSprocket
Member
 
Registered: Nov 2014
Posts: 297

Rep: Reputation: Disabled
You might be able to use FUSE to do something like that. https://www.kernel.org/doc/html/late...tems/fuse.html. In addition to being used to write a filesystem in user space I thnk it can also be used as a filter of some sort.

I got my start on IBM mainframes. My main memory was the frustration of waiting while what I typed traveled down tie lines and back. Everything went through a switch that managed priorities and I as the low rung. Mercifully they put me on PCs where, even with 9 inch floppies and no hard drives, the experience was far more satisfying.
 
Old 06-05-2020, 05:04 PM   #30
Geist
Member
 
Registered: Jul 2013
Distribution: Slackware 14 / current
Posts: 394

Rep: Reputation: Disabled
Even if you don't have access to this library, you have access to your own code.
So if the library wants to write a file immediately, then simply don't call the library functions until you want to.
That should work, right?

Keep your own copy of whatever you want to ultimately write in memory and only call the library function when you want to.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Automatic User Creation with SSH Key creation kjsubbu Linux - Security 5 07-21-2014 09:15 AM
LXer: Fully open sourced JavaFX delayed LXer Syndicated Linux News 0 10-22-2012 03:31 PM
[SOLVED] Prevent subdirectory creation while allowing file creation bweddell Linux - Security 5 07-31-2011 08:54 PM
LXer: Open sourcing of Canonical's Launchpad delayed LXer Syndicated Linux News 0 07-20-2009 03:11 PM
System user creation with mysql user creation moos3 Programming 1 08-11-2007 08:01 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 01:59 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration