Brace yourselves for this one, I'm totally clueless.

At random, my server will stop responding (PHP errors out, sites don't load, MySQL refuses logins, etc). It'll sit in that coma state for about 10-15 minutes, then resume working like nothing ever went wrong.

After that mess is over, a "df -h" shows that 8-10GB of space went MIA. It literally goes MIA, the partition size doesn't change, and I can't locate anything on it consuming that space. Logs don't show anything either. No logs for anything even show that things went bad at all, hence I'm left totally clueless.

I've checked for weird services, signs of being hacked, etc, but nothing comes up.

This has happened twice now, 13 days exactly apart, but at different times of the day. Next time this happens it'll be out of space.

Suggestions?

I don't know why it's happening, but the du command should help you find where the files are being written. For example, run du -sh /* to get the space usage at the top of your file system. If, for example, there was an unusually large usage under /var, you could use du -sh /var/* and keep going like that until you find the files that were created.

Alternatively, if you know when it's happening, you could use the find command and look for files less than a particular age..

I know what days things went bad on, any way to list what files were created/modified that day?

The find command will do that - have a look at the -amin, -atime, -mmin and -mtime options. For example, the following will list the files that were accessed at least 100 minutes ago in the /tmp directory:

Thanks. Seems like logs were the cause of this mess. Logs of a DDoS

Have you looked at logrotate before? It lets you cycle your log files by date or size. That way you can control how large the /var/log directory gets.

Looks like the right tool for the job