LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 07-01-2014, 01:02 PM   #1
Rygir
LQ Newbie
 
Registered: Apr 2014
Posts: 15

Rep: Reputation: Disabled
How to best test timestamp equality using (Ba)sh if?


For a bash script (not necessarily bash shell, but from what I can tell, bash compatible) I need to compare dates.

Using if you can test if a file is older or newer than another file like so :
[ FILE1 -nt FILE2 ] True if FILE1 has been changed more recently than FILE2, or if FILE1 exists and FILE2 does not.
[ FILE1 -ot FILE2 ] True if FILE1 is older than FILE2, or is FILE2 exists and FILE1 does not.
(source http://www.tldp.org/LDP/Bash-Beginne...ect_07_01.html )

My question is now, how do I test if FILE1 has been changed at EXACTLY the same time as FILE2. I.e. it's probably the same file.

Aside from grepping and parsing commandline output which I wanted to avoid, how would you best do this, with the highest possible accuracy?

Anything to worry about if FILE1 is not in the same file system as FILE2? (for example a mounted NAS?)
 
Old 07-01-2014, 01:06 PM   #2
smallpond
Senior Member
 
Registered: Feb 2011
Location: Massachusetts, USA
Distribution: Fedora
Posts: 4,138

Rep: Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263
If you want to test whether two names point to the same file compare inode numbers.
 
Old 07-01-2014, 01:18 PM   #3
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,573

Rep: Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142
There are much better ways of testing for file equality, like "diff -q".

If you really are interested in the timestamps though, you could try testing both -nt and -ot, if they both fail or are both true (not sure if they're > or >=) then it must be the same, assuming you check the usual caveats (both files exist, etc.)

As for the different filesystem, absolutely yes that would cause problems, one of the many reasons why testing for file equality using timestamps is not a good idea. Different filesystems have different time resolution. ext3, for example, only has integer second resolution, while ext4 has nanosecond resolution. Even the exact same file would fail a timestamp comparison when crossing between these two filesystems.

Last edited by suicidaleggroll; 07-01-2014 at 01:21 PM.
 
Old 07-01-2014, 01:20 PM   #4
Habitual
LQ Veteran
 
Registered: Jan 2011
Location: Abingdon, VA
Distribution: Catalina
Posts: 9,374
Blog Entries: 37

Rep: Reputation: Disabled
Code:
FILESTAT1=$(stat -c%Z /path/to/file1)
FILESTAT2=$(stat -c%Z /path/to/file2)
if [ "$FILESTAT1" -ne "$FILESTAT2" ] ; then echo diff ; else echo same ; fi
"best" method is subjective.
boxers vs. briefs
Mr. Pibb vs. Dr Pepper
...

See http://mywiki.wooledge.org/BashFAQ/099

Last edited by Habitual; 07-01-2014 at 01:29 PM.
 
Old 07-01-2014, 04:11 PM   #5
dugan
LQ Guru
 
Registered: Nov 2003
Location: Canada
Distribution: distro hopper
Posts: 11,220

Rep: Reputation: 5309Reputation: 5309Reputation: 5309Reputation: 5309Reputation: 5309Reputation: 5309Reputation: 5309Reputation: 5309Reputation: 5309Reputation: 5309Reputation: 5309
(Ba)sh?

There's a Bourne Shell variant called ash, so why not make it B(a(sh)).

Last edited by dugan; 07-01-2014 at 04:13 PM.
 
Old 07-02-2014, 05:57 AM   #6
Rygir
LQ Newbie
 
Registered: Apr 2014
Posts: 15

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by smallpond View Post
If you want to test whether two names point to the same file compare inode numbers.
They wouldn't be on the same volume. So this can't work.
 
Old 07-02-2014, 06:01 AM   #7
Rygir
LQ Newbie
 
Registered: Apr 2014
Posts: 15

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by suicidaleggroll View Post
There are much better ways of testing for file equality, like "diff -q".
I know, it's the date that I must check. The file contents may or may not be identical.

Quote:
Originally Posted by suicidaleggroll View Post
If you really are interested in the timestamps though, you could try testing both -nt and -ot, if they both fail or are both true (not sure if they're > or >=) then it must be the same, assuming you check the usual caveats (both files exist, etc.)
Are you sure that would be correct? I considered it but I wanted a check to make sure.

Quote:
Originally Posted by suicidaleggroll View Post
As for the different filesystem, absolutely yes that would cause problems, one of the many reasons why testing for file equality using timestamps is not a good idea. Different filesystems have different time resolution. ext3, for example, only has integer second resolution, while ext4 has nanosecond resolution. Even the exact same file would fail a timestamp comparison when crossing between these two filesystems.
That's why I'm looking for a way to take this into account, so I can check if the difference is caused by the filesystem.
 
Old 07-02-2014, 06:10 AM   #8
Rygir
LQ Newbie
 
Registered: Apr 2014
Posts: 15

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by dugan View Post
(Ba)sh?

There's a Bourne Shell variant called ash, so why not make it B(a(sh)).
Because there are many more and I was trying to stick to just sh but found that simple things like [[double brackets]] in bash made things much easier and were well supported.
 
Old 07-02-2014, 10:29 AM   #9
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,573

Rep: Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142
Quote:
Originally Posted by Rygir View Post
I know, it's the date that I must check. The file contents may or may not be identical.
If the file contents aren't identical, then they're not the same file, in which case the time stamp will never match on any recent filesystem (it may on ext3 and older due to the integer second resolution). I do hesitate to say "never" here, but I honestly can't imagine any scenario in which two non-identical files could have identical modification times to the nanosecond.

Quote:
Originally Posted by Rygir View Post
Are you sure that would be correct? I considered it but I wanted a check to make sure.
No, but you could test it and see.

Quote:
Originally Posted by Rygir View Post
That's why I'm looking for a way to take this into account, so I can check if the difference is caused by the filesystem.
Take it into account how? What's your end goal here? The only way to "take it into account" would be to lose the accurate timestamping of any recent filesystem and drop to the least common denominator (or just fall back to integer second regardless), in which case you can follow Habitual's suggestion.

Maybe it would help if you clarified what you're really trying to do here. I guess I just don't see the value in comparing whether or not two timestamps are identical on non-identical files that can cross filesystems with different timestamping resolution. What would you be able to glean from that result?
 
Old 07-02-2014, 11:03 AM   #10
Rygir
LQ Newbie
 
Registered: Apr 2014
Posts: 15

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by suicidaleggroll View Post
If the file contents aren't identical, then they're not the same file, in which case the time stamp will never match on any recent filesystem (it may on ext3 and older due to the integer second resolution). I do hesitate to say "never" here, but I honestly can't imagine any scenario in which two non-identical files could have identical modification times to the nanosecond.
I do : The goal is to verify a synchronization process. If the copy was incomplete or corrupt, the file contents would differ but the date would not. In fact it may happen later on, that the data has become corrupted. The goal is long term storage.

Quote:
No, but you could test it and see.
I was going to

Quote:
Take it into account how?
Well I was planning on testing if they were exact matches, if not, see what the difference is. If the difference happened to fall within a certain margin of error, it could be ignored and logged as a file system difference.

Quote:
What's your end goal here?
Verification that dates have been preserved as well as possible in backups and/or archives.
Quote:
The only way to "take it into account" would be to lose the accurate timestamping of any recent filesystem and drop to the least common denominator (or just fall back to integer second regardless)
That would be perfect.
Quote:
, in which case you can follow Habitual's suggestion.
I'm still looking into that, haven't yet had the time to fully understand it. It's looking promising though.

Quote:
Maybe it would help if you clarified what you're really trying to do here. I guess I just don't see the value in comparing whether or not two timestamps are identical on non-identical files that can cross filesystems with different timestamping resolution. What would you be able to glean from that result?
At risk of letting the thread go off topic : I have a linux phone that can't be recharged, so I have to get the data off within a battery charge or keep track of how much I've done.
To aid in this process, I've got it to connect to the wifi, mount a NAS drive and start copying files. Which is great, but I want to check after each copy if the file was treated correctly by verifying that reading the data yields the same result (instead of blindingly trusting some result variable interpretation). This should make sure the copy was a success and the file can now be deleted.

The goal is to move the whole file system as intact as possible to a NAS.

The way I'm approaching the script is as a general robust file copier/mover that can be reused for similar issues.
 
Old 07-02-2014, 01:39 PM   #11
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,573

Rep: Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142
Quote:
Originally Posted by Rygir View Post
I do : The goal is to verify a synchronization process. If the copy was incomplete or corrupt, the file contents would differ but the date would not. In fact it may happen later on, that the data has become corrupted. The goal is long term storage.
Ok, so with that in mind, what information would a matching or non-matching timestamp provide? It sounds to me, from the sentence quoted above, that the timestamp does not provide the information you need to verify a backup.

Quote:
Originally Posted by Rygir View Post
To aid in this process, I've got it to connect to the wifi, mount a NAS drive and start copying files. Which is great, but I want to check after each copy if the file was treated correctly by verifying that reading the data yields the same result (instead of blindingly trusting some result variable interpretation). This should make sure the copy was a success and the file can now be deleted.
So if it's the file content you care about, why are you comparing the timestamp instead of the content?

Say you're copying a file onto the backup system. A lot of different things can go wrong here, but there are only four end results:
1 - date does not match, content does not match
2 - date does not match, content matches
3 - date matches, content does not match
4 - date matches, content matches

What would you want the script to do in these four instances? For a backup, it seems to me that #2 and #4 you would do nothing, #1 and #3 you would re-copy the file, in which case the timestamp is meaningless and all you care about is if the files match. #2 could cause some irregularities in the backup, but the majority of the time this is going to be caused by filesystem mismatches and there isn't anything you can do about it. The only time #2 could happen without it being a filesystem timestamp resolution issue, I would think, is if you run your copy command without the necessary flags to preserve timestamps, but once this is scripted that should be a non-issue.

Last edited by suicidaleggroll; 07-02-2014 at 01:41 PM.
 
Old 07-02-2014, 02:15 PM   #12
Rygir
LQ Newbie
 
Registered: Apr 2014
Posts: 15

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by suicidaleggroll View Post
Ok, so with that in mind, what information would a matching or non-matching timestamp provide? It sounds to me, from the sentence quoted above, that the timestamp does not provide the information you need to verify a backup.
It does, it verifies that the timestamp has been backed up correctly.



Quote:
So if it's the file content you care about, why are you comparing the timestamp instead of the content?
Because I do care about the timestamp, it's a piece of data I wish to preserve. I also care about the file content obviously, but that has been taken care of with cmp command elsewhere.

Quote:
Say you're copying a file onto the backup system. A lot of different things can go wrong here, but there are only four end results:
1 - date does not match, content does not match
2 - date does not match, content matches
3 - date matches, content does not match
4 - date matches, content matches

What would you want the script to do in these four instances?
If the date matches, check the content, if the content matches as well delete original.
If the date doesn't match or the content doesn't match, log an error for manual review because I'm not sure how often that would happen, I hope almost never.

Quote:
For a backup, it seems to me that #2 and #4 you would do nothing, #1 and #3 you would re-copy the file, in which case the timestamp is meaningless and all you care about is if the files match. #2 could cause some irregularities in the backup, but the majority of the time this is going to be caused by filesystem mismatches and there isn't anything you can do about it.
Actually it can be because of a permissions issue and/or a file system issue that I can resolve (format target drive differently) and/or I copied it wrong and forgot to synchronize the dates and/or the nas protocol won't accept dates.
Quote:
The only time #2 could happen without it being a filesystem timestamp resolution issue, I would think, is if you run your copy command without the necessary flags to preserve timestamps, but once this is scripted that should be a non-issue.
I hope so, but better to be safe than sorry. I wouldn't have to check if the content was identical if everything always went perfectly .

All in all, what I want should be just one line in a shell script; "if datesource=datetarget set dates are equal flag".
 
Old 07-02-2014, 04:11 PM   #13
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,573

Rep: Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142
Then I would follow Habitual's recommendation, but note that you'll probably get false errors on some files when crossing between "old" and "new" filesystems due to roundoff/truncation. You may need to take the absolute value of the difference between the two and check if it's less than some threshold instead.

Last edited by suicidaleggroll; 07-02-2014 at 04:13 PM.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
script to timestamp files with timestamp from directory eRJe Programming 4 11-13-2013 06:52 PM
Is equality of the sexes immoral? jiml8 General 21 06-11-2009 09:40 AM
perl: search for equality pattern kpachopoulos Programming 2 06-02-2007 09:47 PM
Distros Equality Question. rvijay Linux - Newbie 3 08-24-2003 08:21 AM
Equality MaddMaxx Linux - General 2 04-26-2002 09:28 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 05:09 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration