[SOLVED] Rather huge IPtables chain, iptables: Memory allocation problem.

Gangrif · 09-10-2009, 03:20 PM

I'm trying to import a rather gigantic list of IP ranges into iptables.

It's 22 thousand lines.

I get to about 17K, and iptables starts spitting out:
iptables: Memory allocation problem.

I assume this is because i've exhausted some memory limit in iptables.

Is there a method of getting around this?

Thanks!

rayfordj · 09-10-2009, 06:34 PM

I'm not sure exactly how you are attempting to import the rules, but suspect that a change in approach or breaking the import into smaller "chunks" may prove better results.

Are you able to create multiple distinct imports?

Gangrif · 09-10-2009, 09:55 PM

Basically, i've taken a pre-existing list, and converted it to a series of "iptables -A ...." commands.

Each of these 22000 lines is another command similar to "iptables -A chainname -m iprange --src-range 1.2.3.4-5.6.7.8 -j DROP

Gangrif · 09-11-2009, 07:10 AM

this actually brings me to another question....

Is there a better way to add rules via a script?

Also, some more information.
That error was produced when i was testing the import on my workstation. Which is rather beefy, quad core, 8gb of memory. Running FC11.

The target of this whole project is a much lower end machine, running Smoothwall Express 3. I have not tried the import there yet.

rayfordj · 09-11-2009, 07:30 AM

hmmm, odd. I wrote a loop that created a single rule for IPs x.y.1-254.1-254 on a test box that made it from x.y.1-129.z without issue when I last checked. It is a RHEL5.x install, single cpu, maybe 512MB RAM... 128*254 would put last count above 32512 rules. I will check in later and update.

Have you tried creating 2 or 3 scripts out of your rules and running them individually?

rayfordj · 09-11-2009, 08:41 AM

Looks like I hit a snag. Not able to create a rule beyond rule 55399 on the filter table. I can still write to the nat table though... I just wrote a single rule for each IP x.y.1-32.1-254 as a test without any problems on nat PREROUTING chain while filter chains will not accept any more. I guess it is possible that you are hitting a similar ceiling, but sooner...

Hope this helps.

Any way to consolidate some of the rules?

Gangrif · 09-11-2009, 09:13 AM

Quote:

Originally Posted by rayfordj

Any way to consolidate some of the rules?

I'm looking into that now.
The original list, has every singe entry listed as a range. Even if it's one IP. I think that at the very least, i should be able to hack out the non-range ranges, and replace them with single entry rules. I may also be able to take the ranges, and convert them from a range format, to an ip/subnet format. Say, change 1.2.3.0-1.2.3.255 to 1.2.3.0/24

If i change that around, i should be able to add the rules without the -m iprange option. Which may (or... may not) help with resources.

Gangrif · 09-11-2009, 10:06 AM

working on my importer a bit, and i realized something.
I was off by a bit on my list length. A quick look at the number of lines, made me think that i was looking at 22K lines. I looked more closely today, and it's actually... 226K lines.

I have, however, made some improvements, and i'm no longer using the iprange module. I'm testing the import now, we'll see what happens.

Gangrif · 09-11-2009, 12:34 PM

tried breaking the adds up into multiple lists, it was no help. is it possible to increase the available memory to iptables?

Gangrif · 09-11-2009, 03:55 PM

it was all in my method.
i seem to have gotten around the issue.
It wasnt iptables causing the problem, rather, bash.

I was adding all of these rules via a bash script.

I chagned my importer to output an iptables-restore formatted file, and now it imports no sweat.

Thanks for the input!

rayfordj · 09-11-2009, 04:30 PM

Thanks for the followup!