RIM Blackberry Proxy Servers Cannot Access My Server

dflyguy · 06-20-2012, 09:02 AM

Hi Guys,
I am a DIY administrator and not to familiar with linux networking - I have a basic setup of Nginx and Fpm, and for some reason the site is not accessible from the native blackberry browsers on blackberry mobile phones (Gives timeout errro, 10005), This is mainly because requests from the blackberry browser is routed through the RIM proxy servers in canada (for compression and speed), The problem here is that the RIM servers usually times out when trying to get content from my server - The request do not even get to the Nginx, somehow I think the request stop at the TCP transport layer, I have tried disabling my CSF and iptables firewall couple of times to see if this was blocking connections from RIM servers, but the didnt have any effect. - So I am stuck at the moment, I do not know how to detect what is causing the connection from blackberry servers to time out when connecting to my server, any heads up on this would be very helpful.

I think the issue is from my server because all other sites browse fine on my blackberry phone using same routing, I do not know how to use netstate to debug this and find out where and what is causing the connection to drop.

thanks

unSpawn · 06-21-2012, 03:49 AM

- What shows if you 'tcptraceroute' from your server to one of the RIM proxy servers?
- Is the server in your own domain or hosted somewhere else?
- Does your server have a FQDN and proper reverse record?
- What port does your service run on if not TCP/80?
- Is the service located behind your own (reverse) proxy?
- Do you deploy any (active) IP address blocking?
- Can you at least use "-j LOG" rules in your firewall rule set to ensure really no traffic at all from RIM proxy servers hits it?

dflyguy · 06-21-2012, 04:14 AM

Hi unSpawn,
Thanks for the response --> I have temporarily routed most of my mobile traffic via Maxcdn network before I get a solution to this.

So this is what you asked me to do:

tcptraceroute shows:

PHP Code:



 1  xx.xx.xx.xx  0.225 ms  0.220 ms  0.141 ms 
 2  xx.xx.xx.xx  0.257 ms  0.295 ms  0.188 ms 
 3  xe-8-0-0.bar1.Tampa1.Level3.net (4.53.172.1)  0.240 ms  0.246 ms  0.158 ms 
 4  ae-0-11.bar2.Tampa1.Level3.net (4.69.137.110)  0.275 ms  0.242 ms  0.213 ms 
 5  ae-12-12.ebr1.Dallas1.Level3.net (4.69.137.118)  23.823 ms  23.821 ms  23.853 ms 
 6  ae-14-14.ebr2.Chicago2.Level3.net (4.69.151.117)  42.961 ms  42.961 ms  44.016 ms 
 7  ae-2-52.edge4.Chicago3.Level3.net (4.69.138.166)  42.988 ms  43.046 ms  69.948 ms 
 8  RESEARCH-IN.edge4.Chicago3.Level3.net (4.53.98.38)  43.447 ms  43.436 ms  43.621 ms

Is the server in your own domain or hosted somewhere else?
The server is hosted remotely in the states

- Does your server have a FQDN and proper reverse record?
Not sure how to check reverse record? - it does however have a FQDN on the /etc/hostname --> and shows from the hostname command - How do I see if the reverse is correct?

What port does your service run on if not TCP/80?
Standard port 80 (Totally accessible from desktop and other non-RIM-routed-mobile-browsers, without any problem)

Do you deploy any (active) IP address blocking?
Nothing, just standard house keeping rules to block portscan e.t.c

- Can you at least use "-j LOG" rules in your firewall rule set to ensure really no traffic at all from RIM proxy servers hits it?
How do i set this up? As I previously said, I have once stopped iptables and that didnt make any difference but if you tell me how to log it, it would be nice to see. (the problem is not at domain level either, whenever blackberry phone gives me that error, I do try typing in the IP address directly and its same error).

thanks for the help thus far.

dflyguy · 06-21-2012, 10:36 AM

and a quick update too:

- Does your server have a FQDN and proper reverse record?
Yes, Checked and double checked, all works fine and resolves well.

- Is the service located behind your own (reverse) proxy?
No

unSpawn · 06-21-2012, 06:17 PM

Quote:

Originally Posted by dflyguy

How do i set this up? As I previously said, I have once stopped iptables and that didnt make any difference but if you tell me how to log it, it would be nice to see. (the problem is not at domain level either, whenever blackberry phone gives me that error, I do try typing in the IP address directly and its same error).

If you post your rule set then I'll help you intersperse it with logging rules. That said I've found two interesting threads on the subject: http://supportforums.blackberry.com/...ed/td-p/963785 and http://nsxprime.com/forum/showthread.php?t=157371, the latter leading to http://support.cloudflare.com/kb/tro...y-10005-error:
- You aren't using Cloudflare aren't you? (would be too easy ;-p)
- Does your web server in any way differentiate between User Agents when deciding what content to serve (if any)?
- With FQDN I mean your site has a domain name and a PTR record which properly resolves back.

dflyguy · 06-22-2012, 11:36 AM

- You aren't using Cloudflare aren't you? (would be too easy ;-p)

No, I dont use cloudflare, I have been to those link during my times googgling but no much joy, I am surprised a big company like cloudflare is suggesting such an hack to get around a very obvious problem like this, I would have expected a better solution with their size and calibre of engineers, but again - how many average blackberry users can be bothered with going to find service books when most popular sites load just fine?, Not a good solution I would think.

- Does your web server in any way differentiate between User Agents when deciding what content to serve (if any)?
None that i know of, just out of the box centos setup, wack up my typical Nginx, fpm - nothing out of the ordinary, - I am very sure this could be something at the system level or network level.

- With FQDN I mean your site has a domain name and a PTR record which properly resolves back.
Yea, I figured that out and made sure alls in order - but unfortunately BB still acting up.

See my current Iptables

Quote:

# Firewall configuration written by system-config-firewall
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -s 192.168.10.0/24 -m state --state NEW -m tcp -p tcp -j ACCEPT
-A INPUT -p tcp --dport 8080 -j ACCEPT
-A INPUT -p tcp --dport 12000:12100 -j ACCEPT

# Attempt to block portscans
# Anyone who tried to portscan us is locked out for an entire day.
-A INPUT -m recent --name portscan --rcheck --seconds 86400 -j DROP
-A FORWARD -m recent --name portscan --rcheck --seconds 86400 -j DROP

# Once the day has passed, remove them from the portscan list
-A INPUT -m recent --name portscan --remove
-A FORWARD -m recent --name portscan --remove

# These rules add scanners to the portscan list, and log the attempt.
-A INPUT -p tcp -m tcp --dport 139 -m recent --name portscan --set -j LOG --log-prefix "Portscan:"
-A INPUT -p tcp -m tcp --dport 139 -m recent --name portscan --set -j DROP

-A FORWARD -p tcp -m tcp --dport 139 -m recent --name portscan --set -j LOG --log-prefix "Portscan:"
-A FORWARD -p tcp -m tcp --dport 139 -m recent --name portscan --set -j DROP

#Multicast-adresses.
-A INPUT -s 224.0.0.0/4 -j DROP
-A INPUT -d 224.0.0.0/4 -j DROP
-A INPUT -s 240.0.0.0/5 -j DROP
-A INPUT -d 240.0.0.0/5 -j DROP
-A INPUT -s 0.0.0.0/8 -j DROP
-A INPUT -d 0.0.0.0/8 -j DROP
-A INPUT -d 239.255.255.0/24 -j DROP
-A INPUT -d 255.255.255.255 -j DROP

# Drop all invalid packets
-A INPUT -m state --state INVALID -j DROP
-A FORWARD -m state --state INVALID -j DROP
-A OUTPUT -m state --state INVALID -j DROP

-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

My question - I know this is abit of a situation, But for the right solution, I am willing to shed some dollars - if you can offer some of your time to go through this on my server and see if there is something wrong - I once did some kernel tweaking back then on the sysctl file, but nothing major - It just beats my imagination that this only happens when the request goes through the RIM servers (206.51.26.0/24) that is supposedly meant to compress and make things faster - but also this does not happen to bigger sites like blogspot or LQ.

I will appreciate help on this, we can PM if you want

unSpawn · 06-22-2012, 02:38 PM

Quote:

Originally Posted by dflyguy

None that i know of

Still it would be good to check and ensure.

Quote:

Originally Posted by dflyguy

See my current Iptables

Here's your rule set cleaned up, re-prioritized and with (rate/recent)-limited logging enabled:

Code:

*raw
:PREROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
# Bogon ranges can be updated from www.team-cymru.org:
-A PREROUTING -i eth0 -s 0.0.0.0/8 -j DROP
-A OUTPUT -d 0.0.0.0/8 -o eth0 -j DROP
COMMIT

*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -i lo -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -m conntrack --ctstate INVALID -m limit --limit 30/minute --limit-burst 5 -j LOG --log-prefix "INVALID "
-A INPUT -m conntrack --ctstate INVALID -j REJECT --reject-with icmp-admin-prohibited
-A INPUT -s 192.168.10.0/24 -m state --state NEW -m tcp -p tcp -j ACCEPT
# Log RIM HTTP(S) per IP / 300 sec and allow:
-A INPUT -m state --state NEW -m tcp -m multiport --dports 80,443 -s 206.51.26.0/24 -J RIMHTTP
-A RIMHTTP -m recent --rcheck --seconds 300 --name RIMHTTP --rsource -j LOG --log-prefix "RIMHTTP "
-A RIMHTTP -m recent --set --name RIMHTTP --rsource -j ACCEPT
# Collapse 5 rules:
-A INPUT -m state --state NEW -m tcp -m multiport --dports 80,443,8080,22,12000:12100 -j ACCEPT
# Removed portscan: instead use Snort, PSAD, etc. Removed ^.*cast and let it go. For now.
-A INPUT -m limit --limit 30/minute --limit-burst 5 -j LOG --log-prefix "REJECT "
-A INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT

or if you want something more basic with simple RIM range logging only:

Code:

*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -i lo -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -m conntrack --ctstate INVALID -j REJECT --reject-with icmp-admin-prohibited
-A INPUT -s 192.168.10.0/24 -m state --state NEW -m tcp -p tcp -j ACCEPT
-A INPUT -m state --state NEW -m tcp -m multiport --dports 80,443 -s 206.51.26.0/24 -m limit --limit 30/minute --limit-burst 5 -j LOG --log-prefix "RIMHTTP "
-A INPUT -m state --state NEW -m tcp -m multiport --dports 80,443,8080,22,12000:12100 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT

Whichever rule set you choose I can't emphasize enough you should test the rule set on a staging machine or virtual host before using it in production. Pay extra care if you're running OpenVZ for the kernel module problems it brings.

Quote:

Originally Posted by dflyguy

My question - I know this is abit of a situation, But for the right solution, I am willing to shed some dollars - if you can offer some of your time to go through this on my server and see if there is something wrong

I'll support you as much as I can but I'm sorry to say that option is not open to me. If we manage to determine what the issue is you're free to donate to LQ afterwards or become a contributing LQ member but you're under no obligation to do so. If you manage to determine the issue yourself I however do expect you to reciprocate and post the fix.

dflyguy · 06-22-2012, 03:50 PM

Hi Unspawn,
thanks for the updates -

I applied the first set of iptable rules, but got this error:

Quote:

iptables: Applying firewall rules: iptables-restore v1.4.7: multiport needs `-p tcp', `-p udp', `-p udplite', `-p sctp' or `-p dccp'
Error occurred at line: 20

I hope we can eventually cause the root cause of this issue, as there is not a single thread on google that have this solution yet.

unSpawn · 06-24-2012, 03:11 PM

Quote:

Originally Posted by dflyguy

I applied the first set of iptable rules, but got this error:

As it lists the solution I hope you applied it, yes?

dflyguy · 06-24-2012, 05:54 PM

No, Note sure how it fits into the rules - any example will help

dflyguy · 06-25-2012, 05:04 AM

Yes so I applied it --- Added the instruction as per the error...

Although I think I might have found the cause of this problem .... It was something related tot the sysctl tweaking, connections getting dropped.

I applied instructions as per on this post:

http://systembash.com/content/tweaki...fers-on-linux/

and blackberry browsers have been playing fine ever since (yesterday).

Will monitor for few more days to be sure this is the cause, but so far it seems so!

dflyguy · 06-25-2012, 06:04 AM

Hello What are these two lines supposed to do in the IPtables?

Quote:

-A INPUT -m conntrack --ctstate INVALID -m limit --limit 30/minute --limit-burst 5 -j LOG --log-prefix "INVALID "

-A INPUT -m limit --limit 30/minute --limit-burst 5 -j LOG --log-prefix "REJECT "

thanks

unSpawn · 06-25-2012, 10:32 AM

Quote:

Originally Posted by dflyguy

Although I think I might have found the cause of this problem .... It was something related tot the sysctl tweaking, connections getting dropped.

Neat. Which setting did you change before tweaking it now?

Quote:

Originally Posted by dflyguy

I applied instructions as per on this post:
http://systembash.com/content/tweaki...fers-on-linux/

I'm asking because choosing the "right" congestion control algorithm, enlarging the local port range, enabling timestamps, window scaling, SACK (and dsack and fack) makes sense but from kernel 2.4 on net.core.{r,w}mem do not need any tweaking. Documentation in /usr/src/[kernelversion]/Documentation should show those values are set by the kernel on boot according to how much RAM is available.

Quote:

Originally Posted by dflyguy

and blackberry browsers have been playing fine ever since (yesterday).

Good to hear.

Quote:

Originally Posted by dflyguy

Hello What are these two lines supposed to do in the IPtables?

Code:

-A INPUT -m conntrack --ctstate INVALID -m limit --limit 30/minute --limit-burst 5 -j LOG --log-prefix "INVALID "

In the filter table INPUT chain, use connection tracking to filter for any packet that hasn't got a valid state (see http://www.faqs.org/docs/iptables/statemachine.html) and log the message but limit logging to 30 per minute.

Code:

-A INPUT -m limit --limit 30/minute --limit-burst 5 -j LOG --log-prefix "REJECT "

In the filter table INPUT chain log a message for any packet that hits this rule but limit logging to 30 per minute.

dflyguy · 06-25-2012, 11:49 AM

Neat. Which setting did you change before tweaking it now?

These four:
net.core.rmem_default = 256960
net.core.rmem_max = 256960
net.core.wmem_default = 256960
net.core.wmem_max = 256960

I'm asking because choosing the "right" congestion control algorithm, enlarging the local port range, enabling timestamps, window scaling, SACK (and dsack and fack) makes sense but from kernel 2.4 on net.core.{r,w}mem do not need any tweaking. Documentation in /usr/src/[kernelversion]/Documentation should show those values are set by the kernel on boot according to how much RAM is available.
Well I think increasing the buffers is certainly the answer to the problem, not sure on the specifics of how that works out, but according to this documentation:
http://wwwx.cs.unc.edu/~sparkst/howt...ork_tuning.php
Those settings dictate max buffer that goes in or out of the system, am sure the RIM servers are using something out of the ordinary range? and again the connection hasn't dropped since then. Do you think otherwise on this?

Guess I need to do some more readings on IPtables