Sabinou

[Log in to get rid of this advertisement]

Hello folks !

I have a weird problem : it's been a week that three of my /var/log files sometimes suddenly grow VERY large and make me have zero free bytes on /

Those files are syslog, bandwith, and /kernel/info, each of them suddenly getting 400++ megabytes large.

Sometimes my Mandriva 2006 runs fine, but the next boot there is 0 free bytes on /
Sometimes my computer has been running for a few hours and suddenly partmon (of that thingy in the KDE traybar telling you the free space on the partitions) brings a popup telling there's no more free space on /

And when I check, those 3 files in /var/log are back, fullsize.

On other threads I was given a script to monitor the file sizes in /var/log during boottime or with cron, but the problem is different : the file sizes will explose while Mandriva is running !

Would you have any idea why that could happen ?
I didn't see any common point between all the times that it happened, i've been using various programs each time...

And would there be a way to manually define the maximum size a log file is allowed to use ? Setting max 100 megabytes would sure be useful ^^

Emmanuel_uk

look into logrotate

by the way, this could be the sign of some kind of attack? maybe

IS this a server?

What is in the logfile? I mean what does take all that room

Something is very wrong

Sabinou

Thanks Emmanuel

So, as a reply : I recently installed webmin and proftpd in order to try, by curiosity, to setup a server. I haven't been running this yet, but who knows maybe the server's set as active by default.
However, I'm behind a router and haven't manually added any port other than aMule's ports, so I doubt that could be an attack.

Concerning logrotate : if I'm not mistaken, that will only consider "cleaning" the log files during reboot time, and the problem is that now log files are filling up even while mandriva is running.

And as far as the content of the log files is concerned, I'm very much confused, I didn't think of looking what was written into them, i'm ashamed ^^
I'm replying from work, so I cannot open the log files, Emmanuel.
Well, I guess i'll have to open syslog, bandwith and info, when I'm back home.
Would you know if there are log-browsing softwares that would be more efficient than a simple text editor like nano and kwrite (especially for 400 MB files ) ?

The results of my log browsing will come within a few hours -- that is, IF my log files will fill up. Sometimes they won't fill for 4 hours, sometimes they'll fill once per hour ^^

Emmanuel_uk

No worries, you are welcome

http://www.die.net/doc/linux/man/man8/logrotate.8.html
"It will not modify a log multiple times in one day unless the criterium for that log is based on the log's size and logrotate is being run multiple times each day".
I would think you can control the size with logrotate from the man page
(no first hand experience, i.e. my logrotates fine on my 2005 LE)

Using cron as well you might be able to do sthg
(i.e call logrotate more often, or tail x number of line and delete the log)
I am just guessing

Looking at your log try
tail /var/log/syslog
tail -n300 /var/log/syslog
pipe it to less or more if you want

maybe?
vim /var/log/syslog
less /var/log/syslog

which file goes huge exactly? or all three?

service webmin stop
service proftpd stop
stop amule and anything like that

look into your settings of shorewall / firewall as well

Sabinou

So.
Finally webmin was already uninstalled, and I also uninstalled proftpd.

I don't have plenty of time, family is waiting, so I just had time for one thing : checking the contents of the files.

And here's the output : normal beginning of file (of course), but soon enough tons of always the very same garbage.

I copied the text of those 3 moments : the beginning, the moment it went crazy, and the last lines of this, here are the outputs into text files :
http://sabin.free.fr/tmp/syslog.txt
http://sabin.free.fr/tmp/bandwidth.txt
http://sabin.free.fr/tmp/info.txt

Each of them gets exactly the same size (total free space divided by three), when they grow from scratch (after a previous deletion).

I'll check the rest later on, good evening ^_^

Emmanuel_uk

Quick look:
You have a problem with bind (edit I meant may have]
Stop running bind

service named stop
then drakxservices disable it for good

not sure what mdv kernel: BANDWIDTH is about

what's your /etc/shorewall/rules like?

going in SRC=213.186.33.37 DST=192.168.1.2
etc
lots of crazy traffic

It is like you are logging every packet

Sabinou

I tried that after reading you, Emmanuel, without success, but thanks ^^

That problem's taking place right now, each of the log files gains, basically, 300 kb every second. I noted that running any extensive internet using application, such as bittorrent or amule (both right now), will generate growth of the three log files.

I went into MCC and stopped, one after the other, every service related to internet, and that never stopped the log files' size growth.
Only stopping syslog itself will prevent this, but who'd want to completely stopping system messages from being logged >_<

Concerning /etc/shorewall/rules, the only lines not starting with # are :
ACCEPT net fw udp 3666,3672,6429 -
ACCEPT net fw tcp 6881:6999,3663,6419 -
REDIRECT loc 3128 tcp www -

But you know what ?
I just RIGHT NOW found a "temporary patch" while browsing the MCC : deactivating the system's firewall stops logging of all network events (it was configured to allow bittorrent and run in interactive mode to prevent port scanning). Even deactivating the interactive mode and port scanning won't stop log files growth, it requires total firewall deactivation

The last line of shorewall/rules has been removed by mandriva (REDIRECT loc 3128 tcp www -)

The problem is temporarily removed, but it sucks, having to deactivate the firewall
Being behind a router isn't enough security, I think.

Emmanuel_uk

DO NOT stop your firewall

I had a problem while/ shortly aftet browsing your link, I do not know
if it was related. It was the first time ever
see
http://www.linuxquestions.org/questi...d.php?t=437672

if you stop BT the log stops growing?

>>interactive mode
not sure. There is some type of interactive / reactive that
is I read "dangerous" because it makes all this log happened

Is your default policy DROP?

where is the=is BANDWIDTH coming from (I mean shorewall or other?)

[edit again what is your draksec level?]

Sabinou

Woah, don't worry about that weird martian story ^^ A small search gave me those results : Packets that have source addresses with no known route are referred to as "martians". For example, if you have two different subnets plugged into the same hub, the routers on each end will see each other as martians. In other words those martians would be badly adressed or "incomplete" packets. Another source explains that log_martians file is simply a switch to log packets which will be dropped. (source : http://archives.neohapsis.com/archiv...0-q4/0157.html )
So I don't feel it's a security compromission, rather more likely a network problem.

As for me, hmmm...

- aMule or BT or any extensive-use net application will make my log files grow. But singly firefox for instance will also have lines added into the log files.

- the log's increasing wether interactive mode is active or not

- concerning the default policy, I have no idea where that is defined ???
Maybe that's the point, in /etc/shorewall/policy, there's written
loc net ACCEPT
loc fw ACCEPT
fw loc ACCEPT
fw net ACCEPT
net all DROP info
all all REJECT info

- draksec's security level is default, average.

_ About the martians, in /etc/shorewall/shorewall.conf there is LOG_MARTIANS=No

- about the choice of the log files, in /etc/syslog.conf thre is
# Various entry
*.*;auth,authpriv.none -/var/log/syslog
# Explanations from Mandrake Linux configuration tools
kern.=debug -/var/log/bandwidth
# Kernel logging
kern.=debug;kern.=info;kern.=notice -/var/log/kernel/info

Whatever all that may mean

Emmanuel_uk

Thanks for the martian infos. I am not too concerned, but it is humorous in a way.

>>But singly firefox for instance will also have lines added into the log files
Interesting
>>the log's increasing wether interactive mode is active or not
Interesting as well, things are narrowing

>>concerning the default policy, I have no idea where that is defined ???
>>Maybe that's the point, in /etc/shorewall/policy, there's written
It is. And I believe this would be better (just try)
all all drop

>>security level is default, average.
I think high would be better if you are a server…

BTW, I know nothing about bandwidth

Sabinou

I don't understand why, but the problem is gone, the log files have stopped increasing madly in size o_O
And yet, I didn't set the policy to "all drop".

I'll give up on the problem, then... it's weird...

Thanks for your time, Emmanuel

Emmanuel_uk

Hum, good and not good.
Keep an eye on your log. This was not normal.

If you never used -f, have a look in terminal at this
tail -f /var/log/syslog
so you can keep an eye realtime on what is happening,
or notice as soon as it restarts

My guess is that it was something to do with P2P

the drop all policy should be the default one.
I mean it is the recommanded one for better security
(allow only what is needed, deny by default)

Look also at /etc/hosts or /etc/hosts.allow configuration

I would be more paranoid if I were you...
Mabe try azureus for P2p?

gannas

Ok, so I realize that this info is a bit late (almost a year), but... I was having the same problem until I noticed two things in this thread we have in common.

1. You have installed webmin. And I am willing to bet the problem started when you uninstalled it.

2. Your syslog.conf also has kernel debug output going to /var/log/bandwidth

It would appear that when you remove webmin it removes a script called rotate.pl that takes care of automagically rotating the /var/log/bandwidth file. I actually removed webmin because the machine was taking too much CPU time running rotate.pl, but surprisingly the logging didn't stop when webmin was removed using the RPM command. So assuming that rotate.pl isn't running you'll need to remove [edit: or better yet comment it out with a preceeding "#"] the line in /etc/syslog.conf that reads similar to "kern.=debug -/var/log/bandwidth" then you have to restart the syslog service with something like "/sbin/service syslog restart". Before you make this change you can "tail -f /var/log/bandwidth" in another console and when you change the syslog.conf and restart the service the file will stop growing.

What a fun bug this was. This was happening on my mythtv backend and the more we watched TV the faster it would fill up. It actually got to the point where when multiple frontends were running the file was growing to fill 5Gb in under an hour and my "rm -f /var/log/bandwidth" hourly cron job stopped fixing the problem.

So yeah, maybe reconsider the installation of webmin on high-traffic high-load machines. I never took the time to realize how much it can affect performance.

Regards,
Cory Zerwas

Sabinou

Fascinating.

Congratulations on finding the origin of the problem, I hope this can help people who get this bug in the future

ernie

gannas,

You should report this bug on bugzilla (the Mandriva Bug tracking system) so the Mandriva team can fix it (unless it has been fixed in a later version). This looks like a oversight in the RPM removal scripts.