LinuxQuestions.org
Latest LQ Deal: Linux Power User Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 07-13-2009, 03:47 PM   #1
7565
LQ Newbie
 
Registered: Jul 2009
Distribution: Fedora
Posts: 3

Rep: Reputation: 0
cp adds exclamation points when copying very large text files


I am a graduate student in computational chemistry who has been using Linux for about a year and a half.

For my research I have some very large files that are basically millions of lines of ten columns of numbers. These files can be up to 5 GB in size.

Recently I noticed that when I made a copy of one of my files, some exclamation points appeared in it where there should not be any: in front of random numbers throughout the file. Making another copy of the file would result in exclamation points in front of different numbers in different parts of the file. Doing this many times has given me up to four exclamation points in different parts of the file. Sometimes the file copies just fine without producing any extraneous exclamation points.

Additionally, I have occasionally seen a "^K" where there should be a newline (the data that should have been on the next line was instead on the previous line with a ^K in front of it) in copies that I have made of my files. I don't know if this is related or not.

To have reliable results in my research I need to have exact copies of my files. Can anyone tell me what is going on?
 
Old 07-13-2009, 03:53 PM   #2
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,067
Blog Entries: 11

Rep: Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910
Hi, welcome to LQ!


Which distro are you using, what kernel version is it running, and
what's your hardware? Chances are you're hitting a bug in your
machines DMA chip or in one of the lower level kernel drivers.



Cheers,
Tink
 
Old 07-13-2009, 04:24 PM   #3
7565
LQ Newbie
 
Registered: Jul 2009
Distribution: Fedora
Posts: 3

Original Poster
Rep: Reputation: 0
I am running Fedora 10, kernel version 2.6.27.25-170.2.72.fc10.x86_64

My machine has an Intel Core2 Quad @ 2.40 GHz, two Seagate 1.5 TB harddrives, the motherboard is an Intel DP35DP (I think?), it has 8 GB of memory, I am not sure about the type/brand. Is there any other hardware info that would help?
 
Old 08-13-2009, 02:11 PM   #4
7565
LQ Newbie
 
Registered: Jul 2009
Distribution: Fedora
Posts: 3

Original Poster
Rep: Reputation: 0
I continue to see the behavior described above. I have since updated to kernel 2.6.27.29-170.2.78. Does anyone have any suggestions about where a solution might lie? Or about how to prevent file corruption in general? I have some control over the size of my files. Is there a maximum size that I should stick to to avoid copy errors or increasing my risk of corruption?
 
Old 08-13-2009, 02:22 PM   #5
karamarisan
Member
 
Registered: Jul 2009
Location: Illinois, US
Distribution: Fedora 11
Posts: 374

Rep: Reputation: 55
Could be a hardware problem, too. Well, It's not a fix, but do you know about md5sum? It'll take a few minutes to run on a file that big, but you can use it to make sure that each new copy hasn't changed since the last.
 
Old 08-13-2009, 02:23 PM   #6
John VV
LQ Muse
 
Registered: Aug 2005
Location: A2 area Mi.
Posts: 17,090

Rep: Reputation: 2474Reputation: 2474Reputation: 2474Reputation: 2474Reputation: 2474Reputation: 2474Reputation: 2474Reputation: 2474Reputation: 2474Reputation: 2474Reputation: 2474
odd i have not seen that with the 5 gig+ files i use ( imaging data sets from Cassini and MRO mostly)
but then again they are in a ascii image format or as raw, or .v ( vips)

I did skip using fedora 10 and am using fedora 11 with ext4 formatted drive
and CentOS5.3 with ext3 partitions

Is your system ONLY running fedora 10 or is there a Windows or mac install ?
If more than ONE os on the computer is the partition that the data is being copied to and from a ext3 or a ( MS)ntfs format .
 
Old 08-13-2009, 04:03 PM   #7
ddffnn
LQ Newbie
 
Registered: Aug 2009
Posts: 3

Rep: Reputation: 0
You could try dd instead of cp
 
Old 08-13-2009, 06:04 PM   #8
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.9, Centos 7.3
Posts: 17,357

Rep: Reputation: 2367Reputation: 2367Reputation: 2367Reputation: 2367Reputation: 2367Reputation: 2367Reputation: 2367Reputation: 2367Reputation: 2367Reputation: 2367Reputation: 2367
Well, Fedora is a bleeding edge distro. Nothing wrong with that, but I'd stick to Centos for serious stuff.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Copying large numbers of files dman65 Linux - General 8 02-26-2009 07:32 PM
Copying Large Files on the Network bichonfrise74 Linux - Newbie 7 02-20-2009 12:48 PM
using mmap for copying large files rohanak Programming 4 05-18-2008 06:28 AM
Copying large files to FAT32 sketch7 Linux - General 11 10-22-2007 10:13 AM
Samba hangs up at copying many large files ogcu Linux - Networking 1 05-19-2004 01:29 AM


All times are GMT -5. The time now is 07:55 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration