To be clear , The traffic incident to the interface can be Established comming back traffic
OR
New traffic being generated and send to interface "koki"


############################################################

ip rule add from all fwmark 100 lookup koki
iptables -t mangle -A INPUT -i koki -j MARK --set-mark 100
iptables -t mangle -A PREROUTING -i koki -j MARK --set-mark 100
#######################################################

ip route show table koki
default via 10.219.203.2 dev koki
###################################


iptables -t mangle -L -n -v
Chain PREROUTING (policy ACCEPT 69M packets, 66G bytes)
pkts bytes target prot opt in out source destination
44053 2417K MARK all -- koki * 0.0.0.0/0 0.0.0.0/0 MARK set 0x64

Chain INPUT (policy ACCEPT 69M packets, 66G bytes)
pkts bytes target prot opt in out source destination
2715 2921K MARK all -- koki * 0.0.0.0/0 0.0.0.0/0 MARK set 0x64

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 74M packets, 72G bytes)
pkts bytes target prot opt in out source destination

Chain POSTROUTING (policy ACCEPT 74M packets, 72G bytes)
pkts bytes target prot opt in out source destination


#########################################


BASED ON above , i can see the traffic is marked to 0x64 and the rule 0x64 shoud have default route back over default via 10.219.203.2 dev koki

But in production , We cant see any traffic being routed based on the rule 0x64 .

So basically its matched in the mangle table , But for some reason i may be missing something somewhere else .


Hope to help.

Thanks