LinuxQuestions.org
Latest LQ Deal: Complete CCNA, CCNP & Red Hat Certification Training Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 03-08-2018, 06:57 PM   #1
ans1
LQ Newbie
 
Registered: Mar 2018
Posts: 12

Rep: Reputation: Disabled
How do I preserve "crtime" (creation/birth time) when copying from Windows NTFS to Linux EXT4?


Hello and thanks for the help.

I'm attempting to move from Windows-7 to Debian testing.

How do I preserve "crtime" (creation/birth time) when copying from Windows NTFS to Linux EXT4?

I'm trying to copy files from a Windows-7 NTFS computer to a Debian/testing/Buster EXT4 computer (kernel version 4.14).
Unfortunately, the original "creation/birth time" gets lost during this copy.
Instead, the "creation/birth time" of the copied files gets changed to the time of copying.

Below is what I've tried:
On Windows-7 computer, copy files (including creation/birth time, "crtime") to USB stick:
<plug-in NTFS USB stick, E: drive>
<format USB stick to NTFS>
robocopy C:\TEST E:\TEST /MIR /COPYALL /DCOPY:T /SECFIX /TIMFIX
<use "Windows Explorer" to verify correct file creation time ("crtime") on NTRF USB stick>
<eject NTFS USB stick>
On Debian/testing/Buster computer, copy files (including creation/birth time, "crtime") from USB stick:
<plug-in NTFS USB stick>
rsync -avzhHEA --delete <NTFS USB mount point>/TEST ~
Check "crtime" on a file copied from NTFS to EXT4 using rsync:
sudo debugfs -R 'stat /home/<user>/TEST/file' <device>
"debugfs" reports that "crtime" (creation/birth time), "ctime", and "atime" are all changed when copying to EXT4. They are not their Windows NTFS values.
Only "mtime" (last modify time) is the correct Windows NTFS value.

So this procedure failed to copy the original NTFS "crtime".
Instead, it changed "crtime" to the time the file was copied to EXT4. (Additionally, "ctime" and "atime" were also changed when copied to EXT4. Only "mtime" was preserved.)

Further investigations:
I replaced "rsync" with "cp -p". "cp -p" also fails to preserve "crtime". So the problem is not unique to "rsync".

Next I looked into the file systems. Both NTFS & EXT4 file systems support "creation timestamp" (https://en.wikipedia.org/wiki/Compar...f_file_systems).
So the file systems are not the problem.

Next I looked into the kernel:
2010: discussions on adding "crtime" to kernel/stat were put on hold for 5 years, until 2015 (http://lists.gnu.org/archive/html/co.../msg00047.html).
2016: "crtime" (creation/birth time) added to kernel/stat version 4.11 (https://git.kernel.org/pub/scm/linux...3e1bd296c8493f).
So the kernel should handle "crtime" today, 2018.

Summary:
The ability to read the "crtime" was added to kernel version 4.11 in 2016.
It's now 2018, and I'm using kernel version 4.14.
So the ability to access "crtime" should be present within the kernel.
Both file systems (NTFS and EXT4) support "crtime".
I've verified that "crtime" is present and correct on the files contained on the NTFS USB stick.
I've instructed both "rsync" and "cp" to copy all of the file parameters to the EXT4 Linux computer.
Yet files copied to EXT4 fail to retain their original creation/birth times (crtime).

What am I doing wrong?
Is there a kernel module or package I need to install?
How do I preserve "crtime" (creation/birth time) when copying from NTFS to EXT4?

Thanks for the help.
 
Old 03-09-2018, 07:38 PM   #2
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 16,749

Rep: Reputation: 2475Reputation: 2475Reputation: 2475Reputation: 2475Reputation: 2475Reputation: 2475Reputation: 2475Reputation: 2475Reputation: 2475Reputation: 2475Reputation: 2475
New syscalls don't just magically appear in (old) code, the tools have to be re-written to use them - in this case xstat. I see no evidence rsync on my machine uses it yet; nor for that matter the userland stat command which is why we still have to use the debugfs rigmarole.

It's possible there are patched versions of rsync for Linux around (I see a patch for Mac), but I don't know of any.
 
Old 03-09-2018, 08:37 PM   #3
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 3,943

Rep: Reputation: 1719Reputation: 1719Reputation: 1719Reputation: 1719Reputation: 1719Reputation: 1719Reputation: 1719Reputation: 1719Reputation: 1719Reputation: 1719Reputation: 1719
Even if the tools were rewritten, there is simply no mechanism for preserving crtime. It gets set to the time when the inode is assigned to the file, and there is no system call for altering it. There is similarly no way to manipulate ctime.

(OK, you can use debugfs or hexedit to directly modify the device that holds the filesystem and change anything. That's not even remotely within the area of "normal use".)
 
Old 03-09-2018, 10:58 PM   #4
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 16,749

Rep: Reputation: 2475Reputation: 2475Reputation: 2475Reputation: 2475Reputation: 2475Reputation: 2475Reputation: 2475Reputation: 2475Reputation: 2475Reputation: 2475Reputation: 2475
In the interests of a little knowledge, I have dropped a kernel trace on the entry for statx - quiet laptop, nothing in the first few minutes, in contrast to, for example, statfs. Did some rsync, stat commands, no effect.
I'll let it run for a few hours.
 
Old 03-10-2018, 01:13 AM   #5
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 16,749

Rep: Reputation: 2475Reputation: 2475Reputation: 2475Reputation: 2475Reputation: 2475Reputation: 2475Reputation: 2475Reputation: 2475Reputation: 2475Reputation: 2475Reputation: 2475
Not a thing, not even one.
 
Old 03-10-2018, 04:46 AM   #6
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 16,749

Rep: Reputation: 2475Reputation: 2475Reputation: 2475Reputation: 2475Reputation: 2475Reputation: 2475Reputation: 2475Reputation: 2475Reputation: 2475Reputation: 2475Reputation: 2475
Quote:
Originally Posted by rknichols View Post
... there is simply no mechanism for preserving crtime. It gets set to the time when the inode is assigned to the file, and there is no system call for altering it.
I have a lot of sympathy for the OP - there is a case for being able to set this at initial file creation.
For me, the only files I care about this sort of thing is photos, and generally (not always) this info is available in exif meta-data. There really is no reason why extended metadata can't be accessed - and in need set on initial creation, even if it can't subsequently be reset.
 
Old 03-10-2018, 10:07 AM   #7
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 3,943

Rep: Reputation: 1719Reputation: 1719Reputation: 1719Reputation: 1719Reputation: 1719Reputation: 1719Reputation: 1719Reputation: 1719Reputation: 1719Reputation: 1719Reputation: 1719
crtime tells when the file was created in this filesystem. It was never intended to indicate when the content might have been created somewhere else. For a file containing the words, "To be or not to be, that is the question.", the content creation time really should be some date in the early 17th century. That's not the sort of metadata you would expect the filesystem to be carrying.

Sympathy for the OP's use case echoed here. There is sometimes no substitute for just mounting a bit-for-bit image of the original filesystem.

Last edited by rknichols; 03-10-2018 at 10:12 AM. Reason: Added sympathy.
 
Old 03-10-2018, 07:42 PM   #8
ans1
LQ Newbie
 
Registered: Mar 2018
Posts: 12

Original Poster
Rep: Reputation: Disabled
syg00 and rknichols, thank you for your comments and investigations.

I don't have your depth of knowledge on the inner workings of Linux.
But I've gleaned from your postings that: if I drop Microsoft and move my files onto Debian Linux, then my files will loose their creation/birth times.

I could preserve this timestamp information by making a backup of the NTFS drive.
Unfortunately, such a backup adds significant complexity when accessing/processing this timestamp information in the future.

So I tried a different approach: I've examined two other *NIX variants, TrueOS/FreeBSD and Apple.

TrueOS is BSD UNIX using the ZFS filesystem (which supports "creation time"). So I installed and tested TrueOS. Unfortunately, TrueOS gave similar results to Debian testing EXT4 (original "crtime" lost).

Apple provides a ray of hope. Apples version of "rsync" has an option to preserve crtime: "-N, --crtimes" (https://www.manpagez.com/man/1/rsync/).
Unfortunately, I can't find a patch which adds this capability to the Debian version of "rsync".
So I've sent requests to the Debian package managers of "rsync" (please add Apples "rsync -N") and "core-utils" (please include "crtime" in "cp -p").

At this point, I'm out of ideas.

Again, thank you for your information. It's better to know than not. I would have floundered for a long time without your help.
 
Old 03-10-2018, 08:53 PM   #9
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 3,943

Rep: Reputation: 1719Reputation: 1719Reputation: 1719Reputation: 1719Reputation: 1719Reputation: 1719Reputation: 1719Reputation: 1719Reputation: 1719Reputation: 1719Reputation: 1719
Quote:
Originally Posted by ans1 View Post
So I tried a different approach: I've examined two other *NIX variants, TrueOS/FreeBSD and Apple.

TrueOS is BSD UNIX using the ZFS filesystem (which supports "creation time"). So I installed and tested TrueOS. Unfortunately, TrueOS gave similar results to Debian testing EXT4 (original "crtime" lost).

Apple provides a ray of hope. Apples version of "rsync" has an option to preserve crtime: "-N, --crtimes" (https://www.manpagez.com/man/1/rsync/).
Unfortunately, I can't find a patch which adds this capability to the Debian version of "rsync".
So I've sent requests to the Debian package managers of "rsync" (please add Apples "rsync -N") and "core-utils" (please include "crtime" in "cp -p").
It's simply a feature that the kernel does not support. No amount of patching to programs like rsync can do it.

You might have better luck by making a case to the kernel developers at kernel.org. No major distribution is likely to make such a change to a basic kernel function.
 
Old 03-11-2018, 01:40 AM   #10
polpak
Member
 
Registered: Jan 2011
Location: Planet Earth, Australia, NSW
Distribution: GNOME on openSUSE Leap 42.3 (x86_64) & Ubuntu16.04
Posts: 137

Rep: Reputation: 12
Linux is particular about when a file last modified then saved, this also changes the Folders last contents modified timestamp.

Try test copy some files with their "creation/birth time" into a separate MS-DOS format partition on new systems HDrive.


Terminal Command: cp --preserve --recursive /wherestored/FoldersFilesToSave /where2copy/SavedFoldersFiles


Preserve saves last changed timestamp, not sure about CreationDateTime.


IF works, have time to work out which if any application or conversion format you will use with them.


.

.
 
Old 03-12-2018, 08:40 PM   #11
ans1
LQ Newbie
 
Registered: Mar 2018
Posts: 12

Original Poster
Rep: Reputation: Disabled
Hello polpak. Thanks for your response;

I tried a variation of your idea:
1) Using Windows 7: created an NTFS USB drive
2) Using Windows 7: copied a Windows folder containing test files onto this NTFS drive
3) Using Debian: used both "cp -pR" and "rsync -avzhHEA" to duplicate this folder within the same Windows partition (not across file systems as before).
4) Using Windows 7: verified the duplicated folders against original folder.
This approach:
- uses the same tool to examine the original and final results (Windows 7 Explorer), eliminating possible *NIX examination issues.
- uses the same file system to hold the original and the duplicate directories (NTFS on one USB drive), eliminating file system transitions issues.
- only uses Debian testing to create the duplicates, narrowing the area of possible causes.
Specifically:
Using Windows-7:
<Format a USB drive to NTFS as drive E:>
robocopy robocopy C:\TEST E:\TEST /MIR /COPYALL /DCOPY:T /SECFIX /TIMFIX
<verified "crtime" of files matches on "C:\TEST" and "E:TEST" on NTRS drive>
<eject NTFS drive>
Installed fresh copy of Debian testing:
<auto mount NTFS drive>
cd <NTFS mount point>
rsync -avzhHEA TEST TESTrsync
cp -pR TEST TESTcp
<eject NTFS drive>
Using Windows-7:
<check crtime of duplicated folders "TESTrsync" and "TESTcd" against original "TEST">
Results:
The duplicated "crtime" values were all reset to the copy time.
I also repeated this test, replacing Debian with TrueOS (BSD kernel), and got similar results.
So the problem exists in both Linux and Unix.

These results are consistent with the postings of syg00 and rknichols.
When using Debian testing, the original "crtime" information is not being preserved by "rsync" and "cp".

My understanding is that while the Linux and Unix kernels are different, the tools ("cp" and "rsync") are similar.
So the problem is likely with the tools, not the kernels.
While the ability to access "crtime" was added into the Linux kernel in 2016, the tools have yet to be updated to include this information.

I've found others who are struggling with this issue.
The consensus seems to be that "crtime" preservation is coming. Since the capabilities are now in the kernel, it's just a matter of time until the user commands are updated.
Also, some users are taking minor steps on their own. For example: a temporary script which adds creation/birth time to the user "stat" command is available from "github.com/bernd-wechner/Linux-Tools/blob/master/xstat". It only works for ext4, but it's something.
 
Old 03-16-2018, 06:48 PM   #12
polpak
Member
 
Registered: Jan 2011
Location: Planet Earth, Australia, NSW
Distribution: GNOME on openSUSE Leap 42.3 (x86_64) & Ubuntu16.04
Posts: 137

Rep: Reputation: 12
In Linux/Unix systems no 'creation date' record for files is saved. < My Incorrect Assumption.

Search for duplicate files with terminal command fdupes --recursive PathToFolderName find changed names and/or datestamps, but with no change to the files content.

look at two duplicate files using terminal command stat %w PathAndNameOfFile

the result will show you dates for Access, Modify, Change

Terminal command man stat
does show these


%w time of file birth, human-readable; - if unknown
%W time of file birth, seconds since Epoch; 0 if unknown


So far only seen blank results for Birth=

Perhaps another can explain why ? LOL gave up, went to look up something else, and there was information !

This is the "birth" time of a specific file - the moment when it was created on the file system. This attribute is new to ext4 and is also known as crtime or btime

https://unix.stackexchange.com/quest...-empty-on-ext4



.

Last edited by polpak; 03-16-2018 at 07:15 PM. Reason: correction
 
Old 03-16-2018, 07:13 PM   #13
polpak
Member
 
Registered: Jan 2011
Location: Planet Earth, Australia, NSW
Distribution: GNOME on openSUSE Leap 42.3 (x86_64) & Ubuntu16.04
Posts: 137

Rep: Reputation: 12
Admit while NON-Techncial I did enjoy reading how this is being fixed, so Birth soon can be read and used.

https://unix.stackexchange.com/quest...-empty-on-ext4

very pleased to read
Note that xstat() has eventually been added to Linux, so it's only a matter of time before the GNU libc and find add support for it. – Stéphane Chazelas Jun 3 '17 at 9:35
1




Another thanks to all those linux developers : - )


.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
crtime, ext4 and the 'find' command Shm00nix Linux - General 1 06-22-2014 05:38 AM
Copying a set of files through CIFS from Linux to Windows 7 to preserve the timestamp Rupadhya Linux - Software 2 08-28-2013 08:33 PM
How to get file creation time (Ext4, ntfs-3g)? isync Linux - Software 10 06-19-2010 05:49 PM
Help - My Samba share changes the timestamp when copying files to it.. Preserve Time. hheejj Linux - Software 2 06-28-2005 09:34 AM
Preserve file permissions when copying from NTFS to a Samba share? somedude Linux - Software 4 06-11-2004 09:20 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 05:18 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration