LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software > Linux - Kernel
User Name
Password
Linux - Kernel This forum is for all discussion relating to the Linux kernel.

Notices


Reply
  Search this Thread
Old 03-15-2016, 07:04 PM   #1
tullo
LQ Newbie
 
Registered: Feb 2016
Posts: 3

Rep: Reputation: Disabled
close() system call takes long time to finish


I am testing performance using sendfile() to copy big files under Linux 6.4.

My code follows the following logic:

offset = 0;
read_fd = open (argv[1], O_RDONLY);
fstat(read_fd, &stat_buf);
write_fd = open (argv[2], O_WRONLY | O_CREAT, stat_buf.st_mode);

left_to_write = stat_buf.st_size;
while (left_to_write > 0) {
written=sendfile (write_fd, read_fd, &offset, stat_buf.st_size);
if (written == -1)
return -1;
else {
left_to_write -= written;
bytes_done=stat_buf.st_size-left_to_write;
printf("%ld bytes written, %ld bytes left to write\n", written, left_to_write);
}
}

close(read_fd);
close(write_fd); /* this takes minutes */

the sendfile() call is very fast, i can see it's writting chunks of 2GB about ever 5 seconds.

When the while loop is over, it stays at close(output) for minutes before finish successfully.

Why close(output) takes so long to run?

Last edited by tullo; 03-16-2016 at 05:34 PM. Reason: add real code
 
Old 03-16-2016, 03:35 AM   #2
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 11,350

Rep: Reputation: 3418Reputation: 3418Reputation: 3418Reputation: 3418Reputation: 3418Reputation: 3418Reputation: 3418Reputation: 3418Reputation: 3418Reputation: 3418Reputation: 3418
probably sendfile writes only into a buffer and close will save the file.
 
1 members found this post helpful.
Old 03-16-2016, 07:35 AM   #3
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 9,078
Blog Entries: 4

Rep: Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170
Yes, that would be it. File-writes are "lazy." The data is written to a buffer, then written to disk "in the near future." The close() system call, among others, is responsible for flushing the data physically to wherever it needs to go. On a network filesystem, it can take even longer.
 
Old 03-16-2016, 05:33 PM   #4
tullo
LQ Newbie
 
Registered: Feb 2016
Posts: 3

Original Poster
Rep: Reputation: Disabled
I want to add this behavior happens when I copy from local disk to an NFS share. If I copy from local disk to local disk, close() doesn't run long. Also if i copy and an NFS share to local drive, close() runs fast too. The file I copy is 12GB.
generally my code using sendfile() is slower than cp command. I am under impression that sendfile() is faster that read()+write().

Any ideas?
 
Old 03-16-2016, 08:22 PM   #5
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,021

Rep: Reputation: 1769Reputation: 1769Reputation: 1769Reputation: 1769Reputation: 1769Reputation: 1769Reputation: 1769Reputation: 1769Reputation: 1769Reputation: 1769Reputation: 1769
On a local file, close() isn't even waiting for the data to be written to the disk. Over NFS, it does wait for the data to be transmitted (but still not written to disk on the remote system).
 
Old 03-18-2016, 08:56 AM   #6
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 9,078
Blog Entries: 4

Rep: Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170
Ahh ... and what about the implementation of sync()? Does that actually assure a physical commit? Is it actually possible to "assure a physical commit" on any network file system?
 
Old 03-18-2016, 11:28 AM   #7
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,021

Rep: Reputation: 1769Reputation: 1769Reputation: 1769Reputation: 1769Reputation: 1769Reputation: 1769Reputation: 1769Reputation: 1769Reputation: 1769Reputation: 1769Reputation: 1769
Quote:
Originally Posted by sundialsvcs View Post
Ahh ... and what about the implementation of sync()? Does that actually assure a physical commit?
Not really. On a local disk, it ensures that the data has been sent to the drive. How long it takes the drive to work through any queued operations and write the data to the medium is another matter. That could be surprisingly long if the drive happened to be busy retrying a failing read operation on a wonky sector.

I'm not sure that sync does anything useful with a networked filesystem.
 
Old 03-18-2016, 02:27 PM   #8
tullo
LQ Newbie
 
Registered: Feb 2016
Posts: 3

Original Poster
Rep: Reputation: Disabled
back to my original purpose of the question, what is the quickest way to copy big files in linux?
local to local or local to NFS share
I thought sendfile() could be faster than cp command but that's not what I observed
 
Old 03-18-2016, 03:31 PM   #9
rtmistler
Moderator
 
Registered: Mar 2011
Location: MA, USA
Distribution: MINT Debian, Angstrom, SUSE, Ubuntu, Debian
Posts: 7,141
Blog Entries: 12

Rep: Reputation: 2632Reputation: 2632Reputation: 2632Reputation: 2632Reputation: 2632Reputation: 2632Reputation: 2632Reputation: 2632Reputation: 2632Reputation: 2632Reputation: 2632
People may not agree, however this is what I did long ago and it worked for me.

We had a data collector just using the EXT2 file system and open(), write(), close(). Also a lot of data, but not Gigabytes, Megabytes, still it was streaming up to the point where the user's activity was done and within seconds they could dock the device and then wish to grab files. They got zero length files or incomplete files because the file system was not synced. In issuing a sync() call, it would take seconds, or more, maybe even as much as 20-30 and that was rather obnoxious.

So, somewhere along the lines I got the impression that when I forked a child and completed that child's functions and exited, that all file handles owned by the child had to be resolved. Therefore I made my files in my top level process. Then forked a child and used the rename() function to rename the file. Then I exited my child process and once the parent received the termination signal, all was well. It literally took what always seemed like zero time, but really just it happened very fast. We never had any problems with indeterminate files ever again.

I recall writing this once ere long ago and people jumped all over it either explaining it more deeply, or contradicting it. All I got is that I do a lot of exactly this architecture: real-time embedded boards collecting fast data and sending it to Linux serially. Linux correlates these files, conditionally depending on the top level application activities at the time, this all results in several files per field job activity and a very common action is the field guy never stops their activity, however when they shove the device back into the dock, this completes all jobs. Meanwhile the face that the docking has occurred, the file system is now available to the dock and thus the master computer in the vehicle, and that starts grabbing files it hasn't seen before. And as I say, this "method" I employed, has always worked for me.
 
Old 03-21-2016, 04:02 AM   #10
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 11,350

Rep: Reputation: 3418Reputation: 3418Reputation: 3418Reputation: 3418Reputation: 3418Reputation: 3418Reputation: 3418Reputation: 3418Reputation: 3418Reputation: 3418Reputation: 3418
I would say the speed of the transfer will only depend on the underlying hardware/network/whatever, there is no way to influence it (especially the happenings on the remote/nfs side). So you can speed it up if you could somehow lower the amount of data (like rsync) to transfer, or find another trick.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] 64-current / update-mime-database takes *long* time every system start cthibal Slackware 6 04-27-2015 05:23 PM
madvise system call with MADV_SEQIENTIAL option taking too long to finish a52362@trbvm.com Programming 5 08-11-2014 06:34 AM
fclose() operation takes very long time to execute on ext4 file system rudreshsm Linux - Kernel 4 09-13-2012 08:31 AM
linux OS takes time to close some ports? xeon123 Linux - Newbie 2 02-29-2012 05:28 PM
GDM takes a *long* time to start first time grumpybuffalo Linux From Scratch 2 09-09-2007 12:17 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software > Linux - Kernel

All times are GMT -5. The time now is 07:43 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration