LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 12-07-2011, 01:01 PM   #1
loluengo
LQ Newbie
 
Registered: Dec 2011
Posts: 2

Rep: Reputation: Disabled
Question Prepend a header without copying the whole file


That's it. I want to prepend a header without copying the whole file. I want to avoid this because file are rather big, (several gigs) and a
Quote:
cat header file > newfile
would do the trick, but i don't want to make a copy of the file (slooooow!), just insert the header.

Any ideas on this?
 
Old 12-07-2011, 01:24 PM   #2
tronayne
Senior Member
 
Registered: Oct 2003
Location: Northeastern Michigan, where Carhartt is a Designer Label
Distribution: Slackware 32- & 64-bit Stable
Posts: 3,541

Rep: Reputation: 1065Reputation: 1065Reputation: 1065Reputation: 1065Reputation: 1065Reputation: 1065Reputation: 1065Reputation: 1065
What sort of header are you talking about? A header found in /usr/include, some other type?
 
Old 12-07-2011, 03:34 PM   #3
loluengo
LQ Newbie
 
Registered: Dec 2011
Posts: 2

Original Poster
Rep: Reputation: Disabled
Well by a header I mean any binary portion of a file.

i.e

let the header be XXXX and my file is ZZZZZZZ I want to modify my file to be XXXXZZZZZZZ but to avoid copying ZZZZZZZ
 
Old 12-08-2011, 10:52 AM   #4
Nominal Animal
Senior Member
 
Registered: Dec 2010
Location: Finland
Distribution: Xubuntu, CentOS, LFS
Posts: 1,723
Blog Entries: 3

Rep: Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948
The only way to insert data into the beginning or middle of a file is to copy or rewrite the file. Copy-replace is trivial, and almost always the best choice:
Code:
cat header file > newfile && mv -f newfile file
One could write a "prepend" utility to rewrite the file in place, at least in C99 or GNU C, using low-level I/O routines (from unistd.h). The trick is to use an output buffer large enough to fit both the input buffer and the prepended data, and to always refill the input buffer to the brim before flushing the output buffer, so you will never overwrite yet-unread data in the file. It would not be faster than using cat, but it would not need/use the extra disk space, not even temporarily. The problem is that if you happen to have an I/O error midway through, the file will be mangled. Using cat, you still have the original file.
 
Old 12-08-2011, 11:01 AM   #5
tronayne
Senior Member
 
Registered: Oct 2003
Location: Northeastern Michigan, where Carhartt is a Designer Label
Distribution: Slackware 32- & 64-bit Stable
Posts: 3,541

Rep: Reputation: 1065Reputation: 1065Reputation: 1065Reputation: 1065Reputation: 1065Reputation: 1065Reputation: 1065Reputation: 1065
The only way I can think of that you can do that is if your file is marked in some way that allows you to search for a beginning mark and copy to an ending mark. That means you'd have to section that file using unique patterns that can be found by whatever utility you want or roll your own C, C++, whatever, whatever program to do the extraction. Even if you knew the beginning and ending addresses in the file, you could extract that way (but you'd have to write a program to do it).

Another way might be a DBMS, such as MySQL, PostgreSQL, etc. Put your header stuff in a table with an index so you can extract it on-the-fly using a where ind_val = some_pattern.

This is only one approach, perhaps you can think of others.

Hope this helps some.
 
Old 12-09-2011, 02:23 PM   #6
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: Rocky Linux
Posts: 4,776

Rep: Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212
In some cases it may be acceptable to generate the concatenated stream and pass it directly to the receiving program without storing the data on disk again. For example, to pass a stream with two concatenated files to a program called "zorb":
Code:
zorb <(cat headerFile reallyBigFile)
Note that this is not replacing the command's standard input. The argument list will contain the name of a pseudo-file in /proc, which the program will need to open by name to see the content, as it would with an ordinary file.

Disadvantages include
  1. Since input is coming from a pipe, the program won't be able to do seeks or see the metadata for the actual input files.
  2. If the program wants to complain about something in the file, the reported file name will be something useless like, "/proc/self/fd/63".
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] [tar] Do not prepend subdir name to each file? littlebigman Linux - Software 6 06-25-2011 07:05 AM
How to check missing header files included from another header file adisan82 Linux - Software 1 01-28-2011 03:57 AM
[SOLVED] Prepend to ASCII text file in C MS3FGX Programming 4 06-22-2010 10:09 PM
want to prepend to a text file, not append urentity Programming 7 08-03-2009 12:45 AM
How to prepend header to sendfile? vargadanis Programming 0 08-07-2008 02:04 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 01:18 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration