[SOLVED] Why does iptables rule doesn't get set after a reboot once in a while

On my network device, iptables firewall rules service script is started by the init.rc scripts during boot time.

One of the firewall INPUT chain rules doesn't get configured in the iptables firewall configuration once in 50 reboots.

In the following rule present in the iptables script:

#SYNFLOOD DEFEND 10 pps
$IPTABLES -A INPUT -p tcp --syn -m limit --limit 10/s --limit-burst 5 -j ACCEPT
$IPTABLES -A INPUT-p tcp --syn -j LOG --log-level 4 \
--log-prefix "iptables: synflood dropped packet:"
$IPTABLES -A INPUT -p tcp --syn -j DROP

Here in the above sequence, ACCEPT rule doesn't get configured once in a while after a reboot,
iptables -L command from the console shows that ACCEPT rule doesn't get configured and DROP rule drops all the TCP packets.

Verified dmesg and /var/log/messages for kernel logs, but error logs doesn't get logged to find the reason due to which the rule got missed.

The Rule doesn't get set in the firewall INPUT chain configuration with out any error

Is there any way to enable the kernel debug messages to find out the reason of the failure?

Has any one faced a similar problem, or could someone throw some light on how to enable the diagnostic logs for the iptables rules in kernel?

~ # iptables -v
iptables v1.4.11.1: no command specified
Try `iptables -h' or 'iptables --help' for more information.

Is there any way to enable the kernel debug messages to find out the reason of the failure?

Gprathap, if it was the kernel...why doesn't anybody else have this problem? We all use the same kernel.

It is found that randomly one or the other rule gets missed after reboot. This problem is not seen when sleep of 1 second is added before each and every iptables rule is added. Seems to be there is timing issue for the iptables rules configuration in the netfilter kernel module.
Is net-filter module handling iptables re-entrant? If we have a more number of rules to be added (more than 30) do we need to add sleep before adding the rule?

Not as far as I know however I don't have a server that's running more than a 15 or 20 separate rules.

started by the init.rc scripts during boot time.

I don't know which distro you're running but if your script is nested in init.rc, you could be initiating the rule prior to the instantiation of the service, which would cause the rule to fail. You might see that in dmesg.
If you have an rc.local/ directory, put it in there in the last startup level and see if that doesn't fix the problem.

What distro? What release of that distro?

Hi dijetlo,

Debugging further it is found that setsockopt() in /iptables/libiptc/libiptc.c is failing with the ERROR EAGAIN.

As per the below thread, setsockopt may fail if there are any concurrent rules being set by the user.
http://www.spinics.net/lists/netfilt.../msg27570.html

Testing by applying the fix patch provided in URL, to use do While until the setsockopt() succeeds,
it is found that the call fails continuously and never succeed.

Are there any fixes in the iptables libiptc.c area in the recent for TC_COMMIT function?

With Regards
Prathap G

The problem seems to be, concurrency, GP. You might try to create wrapper for your iptables call to have it set a lock in var and while the lock is set any subsequent calls begin with a loop checking for the removal of the lock.

Hi dijetlo,

Thanks for the inputs.
I will try to create a wrapper for iptables call.
What do you mean by set a lock in var?


Thanks
Prathap

Create a flag in var/lock/subsys/ for example, when the call for setsocket instantiates, it checks for a flag (call it, setsock), if setsock is present, it waits a second or two and tries again, as soon as setsocket has been cleared, it sets the flag, sets the socket/runs the iptables command, and then removes the flag. This forces the iptables rules to access the setsocket call one rule at a time without multiple rules trying to set socket at the same time.

Thanks dijetlo.
I will try to implement as per the inputs.
I have further debugged the kernel netfilter code for REPLACE command that is sent in setsockopt and found that EAGAIN is returned by the function tagged in RED color.
num_counter that is sent by the application is not matching with the xt_table list -> number that is stored.


Should this part of code be locked using a MUTEX which might be getting changed due to concurrency?

struct xt_table_info *
xt_replace_table(struct xt_table *table,
unsigned int num_counters,
struct xt_table_info *newinfo,
int *error)
{
struct xt_table_info *private;
int ret;

printk(KERN_DEBUG "ip_tables.c: PRATHAP_DEBUG :Kernel xt_replace_table pid=%d\n",task_pid_nr(current));
ret = xt_jumpstack_alloc(newinfo);
if (ret < 0) {
printk(KERN_DEBUG "ip_tables.c: PRATHAP_DEBUG :Kernel xt_replace_table err ret = %d, pid=%d\n",ret,task_pid_nr(current));
*error = ret;
return NULL;
}

/* Do the substitution. */
local_bh_disable();
private = table->private;

/* Check inside lock: is the old number correct? */
if (num_counters != private->number) {
pr_debug("num_counters != table->private->number (%u/%u)\n",
num_counters, private->number);
local_bh_enable();
*error = -EAGAIN;
return NULL;
}

table->private = newinfo;
newinfo->initial_entries = private->initial_entries;

/*
* Even though table entries have now been swapped, other CPU's
* may still be using the old entries. This is okay, because
* resynchronization happens because of the locking done
* during the get_counters() routine.

local_bh_enable();

#ifdef CONFIG_AUDIT
if (audit_enabled) {
struct audit_buffer *ab;

ab = audit_log_start(current->audit_context, GFP_KERNEL,
AUDIT_NETFILTER_CFG);
if (ab) {
audit_log_format(ab, "table=%s family=%u entries=%u",
table->name, table->af,
private->number);
audit_log_end(ab);
}
}
#endif

printk(KERN_DEBUG "ip_tables.c: PRATHAP_DEBUG :Kernel xt_replace_table return SUCCESS pid=%d\n",task_pid_nr(current));
return private;
}

Prathap, I would caution you about modifying kernel code on production machines without a lot of testing

Sure, will ensure that.
I have found in the latest iptables release 1.4.21, there is a new command called 'wait' added to the list.
http://www.netfilter.org/projects/ip...downloads.html

Does this solve concurrency problem?

Habitual is right, at the very least, for reference purposes it would be helpful for the next guy to know which distro you're working on here.
From man iptables

iptables -w --wait.. Wait for the xtables lock. To prevent multiple instances of the program from running concurrently, an attempt will be made to obtain an exclusive lock at launch. By default, the program will exit if the lock cannot be obtained. This option will make the program wait until the exclusive lock can be obtained.

So yeah, good call, that definitely looks like the way to go here.

Sorry for missing to mention the distro.
I am working on Android version 4.3 device, having elinux and running iptables 1.4.11.1 version.
Ported the wait command fix from 1.4.21 version and found that the concurrency issue mentioned is not reproducible again.

Thanks for the inputs.