Taken out by Duct Tape!
Posted 07-30-2015 at 05:31 PM by Habitual
Every Week for the last half-year, we have had some kind of issue with ping/icmp-related zabbix checks. We couldn't nail it down.
Finally, in the end, I disabled ping checks via zabbix and (not-so-)silently declared icmp checks in zabbix as 'fluff'. Useless.
This has been ok up until this week. We hit our cap Hard and Fast on Wednesday morning.
It took out my zabbix and Kibana dashboards and caused another client's app portal to stop working.
We found the culprit by examining the logs on our DDoS front-end.
99% of the traffic was headed for a sole IP. We asked the client and he said they ran a promo for "Duct Tape"
The article was titled "Duct Tape Prom Outfits" and it was featured on yahoo.com.
The cause, near as I can figure is pulling 92 pictures from our host at 1.x megs apiece
We have since instructed the client to reduce the size of the graphics and this seems to have helped.
I never would have guessed "Duct Tape Prom Outfits" to be so popular.
What a crazy world.
Finally, in the end, I disabled ping checks via zabbix and (not-so-)silently declared icmp checks in zabbix as 'fluff'. Useless.
This has been ok up until this week. We hit our cap Hard and Fast on Wednesday morning.
It took out my zabbix and Kibana dashboards and caused another client's app portal to stop working.
We found the culprit by examining the logs on our DDoS front-end.
99% of the traffic was headed for a sole IP. We asked the client and he said they ran a promo for "Duct Tape"
The article was titled "Duct Tape Prom Outfits" and it was featured on yahoo.com.
The cause, near as I can figure is pulling 92 pictures from our host at 1.x megs apiece
We have since instructed the client to reduce the size of the graphics and this seems to have helped.
I never would have guessed "Duct Tape Prom Outfits" to be so popular.
What a crazy world.
Total Comments 2
Comments
-
Of all the "Oprah'd" websites I've heard of, this has to be one of the more interesting topics haha. Duct tape, indeed.
Should advise them next time to hire a CDN for image distribution before nuking your network =)Posted 07-30-2015 at 05:47 PM by rocket357 -
"Developers", go figure.
Posted 07-31-2015 at 12:46 PM by Habitual