LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 08-07-2009, 07:47 AM   #1
zhjim
Senior Member
 
Registered: Oct 2004
Distribution: Debian Squeeze x86_64
Posts: 1,748
Blog Entries: 11

Rep: Reputation: 233Reputation: 233Reputation: 233
Looking for a programm that changes head of large file


Hi folks,

I'm looking for a program that can change the first few lines of a big file > 1G (in this case 32G).

Backgroung:
Right now I'm setting up a replica of our mysql server. Having one giant file of all the db's containing the sql statements to create the tables and put in the data. So first I csplited the big one to give me some smaller per DB files. After this I wanted to change the name of the DB the tables should be created in. Normaly it would read

Code:
-- Current Database: `Profiler`
--

CREATE DATABASE /*!32312 IF NOT EXISTS*/ `Profiler` /*!40100 DEFAULT CHARACTER SET latin1 COLLATE latin1_german1_ci */;

USE `Profiler`;

--
-- Table structure for table
But i like it to read
Code:
Use Profiler_20090403
The ususal vi just dumped me cause the files is to big. Takes ages to load. I finally got it going with
Code:
sed 's/USE `Profiler`/Use Profiler_20090403/g' ./profiler.sql
and piped this over to the mysql process. But for some uncertain reasons I don't like that approach.
Anyone knows of a programm that can just change the first few lines of a file?

Regards Zhjim
 
Old 08-07-2009, 08:23 AM   #2
GrapefruiTgirl
LQ Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 555Reputation: 555Reputation: 555Reputation: 555Reputation: 555Reputation: 555
I think you're on the right track, using SED. But why not do an in-place edit of the file using sed? Check the man page (yes, I know it's big) for details, but I'd wager an experienced 'sedder' will pop in sooner or later with a nice simple way for you to try.

Sasha
 
Old 08-07-2009, 10:20 AM   #3
rn_
Member
 
Registered: Jun 2009
Location: Orlando, FL, USA
Distribution: Suse, Redhat
Posts: 127
Blog Entries: 1

Rep: Reputation: 25
Pretty much any program you use, even if it's a custom program, you will have to read the whole file and write it out again, if that's what your concern is.

If the new value was the same size as the old one, then a program that could seek to that location and change the bytes in place would have eliminated the need to create a new file and thus would be more efficient.

However, in your case, you should probably leave the file as is and do what you did before; sed and pipe whenever you need to change the value. If you'd rather save the file, you could do it 'in-place' using the -i option on sed (sed --help) as GrapefruiTgirl suggested (or perl -pi -e, or awk, or other progs), however, that's just gonna create a temp file with your changes and then rename it to the original once it's done; all it saves you is to have to rename the file at the end.


HTH.
-RN.
 
Old 08-10-2009, 02:54 AM   #4
zhjim
Senior Member
 
Registered: Oct 2004
Distribution: Debian Squeeze x86_64
Posts: 1,748

Original Poster
Blog Entries: 11

Rep: Reputation: 233Reputation: 233Reputation: 233
Thanks the both of you for your answers.
I also had the idea with the --in-place option of sed but I wanted to be eager and just have it run for a few seconds and then abort it. But like m_ explained this did not work cause not the original file was changed.

Quote:
Originally Posted by rn_ View Post
Pretty much any program you use, even if it's a custom program, you will have to read the whole file and write it out again, if that's what your concern is.
Thats exactly the point. I have one big ass of a backup file. It's just 35G of pure sql statements. The rest is no problem to come by or just change before doing anything with it. But that little thingy just too big .
As I'm having some unused characters in the file I'm thinking about a bit of C code to do the job. But this also depends if it just changes the part I want or also writes back the whole file. I'm not so deep into C and filesystems to know for sure. I guess I'll just give it a try.

Thanks
Zhjim
 
Old 08-10-2009, 11:45 PM   #5
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 7.7 (?), Centos 8.1
Posts: 17,903

Rep: Reputation: 2614Reputation: 2614Reputation: 2614Reputation: 2614Reputation: 2614Reputation: 2614Reputation: 2614Reputation: 2614Reputation: 2614Reputation: 2614Reputation: 2614
You can use seek etc in Perl, which is easier than C.
http://perldoc.perl.org/functions/seek.html
 
Old 08-11-2009, 04:20 AM   #6
zhjim
Senior Member
 
Registered: Oct 2004
Distribution: Debian Squeeze x86_64
Posts: 1,748

Original Poster
Blog Entries: 11

Rep: Reputation: 233Reputation: 233Reputation: 233
As I don't know both languages very well it might be easier in perl
Next week I'll see if I can do it in c.

Heres a link to a blog witht the perl skript

http://www.linuxquestions.org/blog/z...sql-dump-2175/
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
How window in programm can be resized by another programm. rexStart Programming 1 09-23-2008 02:47 AM
[quick] trying to split a large file but linux says it's to large steve51184 Linux - General 16 05-06-2008 07:40 AM
File too large (script is too large to execute) DeuceNegative Linux - General 1 05-09-2007 12:10 AM
how to store a pixmap image created by gdk programm into a new file in linux m.harshavardhan Linux - General 0 05-06-2005 12:57 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 02:18 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration