gzip using too much CPU

chovy · 01-20-2006, 03:03 PM

So I have this problem with my virtual host provider.
Gzipping a large file 260meg results in 99% CPU usage, and my session gets killed.

Any ideas around this?

I wanted to gzip a 260mb .sql file (a dump from mysqldump).

'nice' doesn't seem to work even at --adjustment=19.

ilikejam · 01-20-2006, 04:23 PM

You could get the PID of the gzip, and issue a SIGSTOP to it (with kill -STOP), then sleep for a second, then issue a SIGCONT to it (with kill -CONT), then sleep for a second, then issue a SIGSTOP to it.......

That should effectively half your processor time. Adjust the sleeps to taste.

Dave

chovy · 01-20-2006, 04:26 PM

So there's no real coreutils utility to do this for me?

Is there any way to control the pipe or buffer, whatever gzip uses?

$gzip bigfile.sql smallfile.sql.gz

This will eat up all the memory on the server, I'm surprised if there's no way to control it and play "nicely".

ilikejam · 01-20-2006, 04:41 PM

I don't know of any core utils which will control processor usage (except for nice). Most people like their computers to run as fast as possible, so it's not an oft-requested feature to slow down processes arbitrarily.

gzip usually doesn't eat up RAM (it allocates and deallocates quite small chunks as it goes along), so I wouldn't worry about that. (I just tried gzip'ing a 400MB file, and memory usage never went above 0.1% on a 512MB RAM machine).

Dave

chovy · 01-20-2006, 04:45 PM

When I run gzip on a 260mb I get kicked off, even other ssh connections to the ISP server.

The mem usage goes up to 99.9%

chovy · 01-20-2006, 04:46 PM

aiyyy...it IS CPU usage that goes up to 99.9% not ram.

chovy · 01-20-2006, 04:47 PM

I tried
$ nice --adjustment=19 gzip test.sql

and i get kicked everytime.

ilikejam · 01-21-2006, 07:24 PM

Hi again.

Have you tried the STOP-sleep-CONT thing in post #2?

Dave

chovy · 01-21-2006, 07:40 PM

I think it was a combination of gzip --fast and ssh -o TCPKeepAlive=yes

that fixed the problem, at least temporarily for me.