Looking for a programm that changes head of large file

zhjim · 08-07-2009, 07:47 AM

Hi folks,

I'm looking for a program that can change the first few lines of a big file > 1G (in this case 32G).

Backgroung:
Right now I'm setting up a replica of our mysql server. Having one giant file of all the db's containing the sql statements to create the tables and put in the data. So first I csplited the big one to give me some smaller per DB files. After this I wanted to change the name of the DB the tables should be created in. Normaly it would read

Code:

-- Current Database: `Profiler`
--

CREATE DATABASE /*!32312 IF NOT EXISTS*/ `Profiler` /*!40100 DEFAULT CHARACTER SET latin1 COLLATE latin1_german1_ci */;

USE `Profiler`;

--
-- Table structure for table

But i like it to read

Code:

Use Profiler_20090403

The ususal vi just dumped me cause the files is to big. Takes ages to load. I finally got it going with

Code:

sed 's/USE `Profiler`/Use Profiler_20090403/g' ./profiler.sql

and piped this over to the mysql process. But for some uncertain reasons I don't like that approach.
Anyone knows of a programm that can just change the first few lines of a file?

Regards Zhjim

GrapefruiTgirl · 08-07-2009, 08:23 AM

I think you're on the right track, using SED. But why not do an in-place edit of the file using sed? Check the man page (yes, I know it's big) for details, but I'd wager an experienced 'sedder' will pop in sooner or later with a nice simple way for you to try.

Sasha

rn_ · 08-07-2009, 10:20 AM

Pretty much any program you use, even if it's a custom program, you will have to read the whole file and write it out again, if that's what your concern is.

If the new value was the same size as the old one, then a program that could seek to that location and change the bytes in place would have eliminated the need to create a new file and thus would be more efficient.

However, in your case, you should probably leave the file as is and do what you did before; sed and pipe whenever you need to change the value. If you'd rather save the file, you could do it 'in-place' using the -i option on sed (sed --help) as GrapefruiTgirl suggested (or perl -pi -e, or awk, or other progs), however, that's just gonna create a temp file with your changes and then rename it to the original once it's done; all it saves you is to have to rename the file at the end.

HTH.
-RN.

zhjim · 08-10-2009, 02:54 AM

Thanks the both of you for your answers.
I also had the idea with the --in-place option of sed but I wanted to be eager and just have it run for a few seconds and then abort it. But like m_ explained this did not work cause not the original file was changed.

Quote:

Originally Posted by rn_

Pretty much any program you use, even if it's a custom program, you will have to read the whole file and write it out again, if that's what your concern is.

Thats exactly the point. I have one big ass of a backup file. It's just 35G of pure sql statements. The rest is no problem to come by or just change before doing anything with it. But that little thingy just too big

.
As I'm having some unused characters in the file I'm thinking about a bit of C code to do the job. But this also depends if it just changes the part I want or also writes back the whole file. I'm not so deep into C and filesystems to know for sure. I guess I'll just give it a try.

Thanks
Zhjim

chrism01 · 08-10-2009, 11:45 PM

You can use seek etc in Perl, which is easier than C.
http://perldoc.perl.org/functions/seek.html

zhjim · 08-11-2009, 04:20 AM

As I don't know both languages very well it might be easier in perl

Next week I'll see if I can do it in c.

Heres a link to a blog witht the perl skript

http://www.linuxquestions.org/blog/z...sql-dump-2175/