406 error trying to connect to harryshearer.com

RandomTroll · 12-08-2019, 02:28 PM

2 different domains, on 2 different hosts, blocked my attempts to connect to harryshearer.com (Spinal Tap's bassist, voice of many characters on The Simpsons), returning

Quote:

'Generally a 406 error is caused because a request has been blocked by Mod Security. If you believe that your request has been blocked by mistake please contact the web site owner.'

frankbell · 12-08-2019, 08:18 PM

For what it's worth, I had no problems connecting either tonight or several times earlier this week, so it's not a system-wide issue.

RandomTroll · 12-08-2019, 09:50 PM

I figured it out: harryshearer.com rejects connections by lynx. At home I change the user-agent header to fool them. If I do it at those sites I can fool harryshearer.com.

frankbell · 12-09-2019, 07:41 PM

Thanks for the update! Nice job of detective work.

I just checked. It accepts connections from w3m and links.

RandomTroll · 12-10-2019, 06:04 PM

I should have suspected. I'm a long-time member of the lynx mailing list, this is a common problem, I long ago changed my user-agent header at home. I don't know why I didn't think of it first thing elsewhere.

frankbell · 12-10-2019, 07:14 PM

Why would they ban lynx and not other text browsers?

RandomTroll · 12-10-2019, 10:51 PM

Quote:

Originally Posted by frankbell

Why would they ban lynx and not other text browsers?

Lynxsters tell me lynx is the most popular browser to 'scrape' websites. It may be smaller and/or faster; it may merely be older. washingtonpost.com blocks lynx. I notice that many websites don't count lynx accesses. I can access nytimes.com all I like without a subscription. newyorker.com does count, but it's exceptional.