An unexpected turn of events when Jeff Geerling posted “I’m hosting my website on a FARM!”
Posted by jpluimers on 2023/07/06
Some links on the unexpected turn of events after [Archive] Jeff Geerling (@geerlingguy) / Twitter posted
- [Archive] I’m hosting my website on a FARM! – YouTube
- [Wayback/Archive] Hosting this website on a farm – or anywhere | Jeff Geerling
First his site got more traffic because of the post, then within an hour traffic exploded because of a DDoS overflowing both his Raspberry Pi cluster and his mobile data capacity.
Jeff will likely do blog posts on these and update the underlying GitHub repository at [Wayback/Archive] geerlingguy/turing-pi-2-cluster: Turing Pi 2 Cluster , but until then (since his Tweets were not threaded), this is what happened on 20220209 as it taught me a few bits:
all times in UTC; all tweets by Jeff Geerling unless otherwise noted.
- -00:05 – [Archive] Guess what the problem was. Just guess. Yes, it was DNS. Luckily CoreDNS started behaving again after a simple reboot.
DNS: the core of all problems
- 00:28 – [Archive] Tomorrow… #TuringPi2 …
- 15:01 – [Archive] I’m hosting my website on a FARM!? … #TuringPi2 #RaspberryPi
- 15:12 – [Archive] My little #TuringPi2 cluster is seeing a bump after posting today’s new blog post on the off-grid K3s cluster: …
- 15:45 – [Archive] image
caption: http://www.jeffgeerling.com 69 current visitors
- 15:58 – [Archive] #ad I’ve had people ask about the solar panels and battery from @EcoFlowTech — the 400W panels are extremely durable (and big!), and survived falling over in 20 mph wind a few times on the day of the shoot. The battery is currently running the cluster (16 hours and counting): …
- 16:28 – [Archive] “Cattle not pets” “Why not both?” From the end of my latest video, on hosting my website on a #TuringPi2 #RaspberryPi cluster on a farm: …
- 16:26 – [Archive] Now we’re seeing that HN bump… cluster load average (4x Pi 8GB Lite nodes) is still hovering around 1. No big issues. #TuringPi2 …
- 16:40 – [Archive] …and
JeffGeerling.comis down. Predictable, maybe someone actually hit something important :P - 16:47 – [Archive] Pretty sure someone has decided they’d like to blast the backend with a real DoS/DDoS, so might have to wrap up testing for the day… seeing 5-7 Mbps of traffic now (which is past what LTE can safely handle) …
Jeff was reluctant to go the Cloudflare way…
- 17:01 – [Archive] So yeah… someone is basically flooding the server with requests right now :( Fun times. …
- 17:02 – [Archive] My poor little Pi cluster with it’s 9 Mbps connection was trying to handle that. Seriously! …
- 17:10 – [Archive] Thousands and thousands of requests from
164.90.211.4that break through cache (even if it results in a 301 / 400, it broke cache).164.90.211.4 - - [09/Feb/2022:11:08:52 -0600] "POST jeffgeerling.com HTTP/1.1" 301 194 "-" "undefined" "-"I’ve turned on Cloudflare. Sigh.- 17:13 – [Archive] Matthew DiCecca on Twitter: “@geerlingguy Consider reporting this to
abuseipdb.comwww.abuseipdb.com/check/164.90.211.4Currently there is no record for the IP, but I guarantee if they are doing it to you they’ve done or will do it to others” / Twitter - 17:17 – [Archive] Patryk Szczygłowski on Twitter: “@geerlingguy Next time, for faster recovery, don’t change NS away from Cloudflare. Just go to dashboard and disable proxying on DNS records you are interested in.” / Twitter
- 17:18 – [Archive] Jeff Geerling on Twitter: “@epatryk I never had it on Cloudflare (until a few minutes ago)… now have to wait for DNS to propagate, and trying to mitigate direct hits for now…” / Twitter
- 17:18 – [Archive] Patryk Szczygłowski on Twitter: “@geerlingguy Ah, I see. Don’t hesitate to ping me if you have any CF questions.” / Twitter
- 17:22 – [Archive] Patryk Szczygłowski on Twitter: “@geerlingguy Because your origin IP is already known to the attacker, if the attack doesn’t move with a DNS record, consider changing your origin IP and/or setting up Cloudflare Tunnel.” / Twitter
- 17:23 – [Archive] Jeff Geerling on Twitter: “@epatryk Yeah… I might end up having to do that. Attack just started again :(” / Twitter
- 17:29 – [Archive] Patryk Szczygłowski on Twitter: “@geerlingguy I’ve taken the liberty of flushing DNS cache for
1.1.1.1and8.8.8.8. Now your site is in a redirect loop. Adjust your SSL encryption mode.dash.cloudflare.com/?to=/:account/:zone/ssl-tls” / Twitter - 17:30 – [Archive] Jeff Geerling on Twitter: “@epatryk D’oh, thought I was missing something. Thanks” / Twitter
- 17:31 – [Archive] Patryk Szczygłowski on Twitter: “@geerlingguy You welcome. As I mentioned, don’t hesitate asking me about CF stuff.” / Twitter
- 17:32 – [Archive] Jeff Geerling on Twitter: “@epatryk Gah, every time I start Nginx again, load goes to 25+ and nothing gets served regardless… =” / Twitter
- 17:35 – [Archive] Patryk Szczygłowski on Twitter: “@geerlingguy Some tips on dealing with DDoS attack:
support.cloudflare.com/hc/en-us/articles/200170196-Responding-to-DDoS-attacksI also recommend either allowlisting only CF IPs on your firewall (cloudflare.com/ips) or closing the port altogether and setting up Cloudflare Tunnel instead.” / Twitter - 17:43 – [Archive] @epatryk I’m doing that currently, but nginx seems to hate me right now :P
- 18:02 – [Archive] Patryk Szczygłowski on Twitter: “@geerlingguy Can’t help with that, but as a quick resource relief, I can also recommend set a Page Rule to cache everything. e.g. URL = your whole site Setting: Cache Level = Cache Everything https://t.co/Y3Szic9cpb” / Twitter
- 18:03 – [Archive] @epatryk Will do — heh… I forgot that’s not enabled by default! (Been a couple years since I set up Cloudflare on a domain, and I wasn’t ‘under attack’ at that time)
- 15:10 (the day after) – [Archive] Jeroen Wiert Pluimers on Twitter: “@geerlingguy @epatryk About CloudFlare for a domain: do you have a video about it? It’s a skill I need to acquire too and I like how you deliver knowledge.” / Twitter
- 17:20 – [Archive] Daryll Swer on Twitter: “@geerlingguy Rate limit on the origin server on layer 3 for certain traffic types, then rate limit on layer 7 for the webserver and it should handle just fine.” / Twitter
- 17:22 – [Archive] @DaryllSwer First thing I wanted to do was pop everything over to Cloudflare (just to get back above water). It was difficult to discern much (even the patterns) since there seemed to be over 100 requests per second, and I never set up any logging analysis on my personal server :P
- 17:24 – [Archive] @geerlingguy For reasons such as multi-gigabit DDoS, I do not host my site on my home devices or something. It’s cool and all to run it on a Pi until 10Gbps drops on your fibre door. Rather shift it to the cloud where there’s DDoS mitigation among other things. And yup, use Cloudlfare.
- 17:22 – [Archive] @DaryllSwer First thing I wanted to do was pop everything over to Cloudflare (just to get back above water). It was difficult to discern much (even the patterns) since there seemed to be over 100 requests per second, and I never set up any logging analysis on my personal server :P
- 17:44 – [Archive] Jeroen Wiert Pluimers on Twitter: “@geerlingguy I hear a new blog post and video are coming up discussing cloudflare.” / Twitter
- 17:45 – [Archive] @jpluimers lol… the sad thing is if it weren’t a targeted attack, I’d be happily able to deal with normal traffic spikes. Sadly, people are jerks.
- 17:13 – [Archive] Matthew DiCecca on Twitter: “@geerlingguy Consider reporting this to
- 17:55 – [Archive] Heh… I accidentally locked myself out of the site too. Will have to fix that.
- 17:59 – [Archive] So far Cloudflare has blocked 6 million POST requests in the past 30 minutes… so yeah, no way my little Pi Cluster could handle that …
- 18:59 – [Archive] Here are the IP addresses in the 10,000+ request range (some may be valid like CloudFlare fronting requests… but that top one looks to be a german DO address, doing most of the damage): …
- 20:15 – [Archive] Hehe
- 22:34 – [Archive] I would like to know, wise YouTube commenter, how it would possible to build an off-grid web server without a connection to the Internet? (In response to a comment that you can’t call something ‘off-grid’ if it has an Internet connection)
In the aftermath, Jeff is contemplating. Some nice responses too about how “off-grid” things can be. Recommended reading!
Some mentioned links:
- [Wayback/Archive] Configure DO VPS to toggle from proxying cluster to local site · Issue #14 · geerlingguy/turing-pi-2-cluster
- [Wayback/Archive] Document the ‘fun’ morning blocking a DDoS attack · Issue #141 · geerlingguy/jeffgeerling-com
- dash.cloudflare.com/?to=/:account/:zone/ssl-tls Adjust your SSL encryption mode.
- [Wayback/Archive] Responding to DDoS attacks – Cloudflare Help Center
- [Wayback/Archive] Overview
- [Wayback/Archive] Step 1: Enable Under Attack Mode
- [Wayback/Archive] Step 2: Enable the Web Application Firewall (WAF)
- [Wayback/Archive] Step 3: Challenge or block traffic via the Firewall app
- [Wayback/Archive] Step 4: Mitigate DDoS Ransom Campaigns
- [Wayback/Archive] Step 5: Contact Cloudflare Support
- [Wayback/Archive] Related resources
- [Wayback/Archive] www.cloudflare.com/ips
IP Ranges
Last updated: April 8, 2021Some applications or host providers might find it handy to know about Cloudflare’s IPs. This page is intended to be the definitive source of Cloudflare’s current IP ranges.You can also use the Cloudflare API to access this listIPv4
- •103.21.244.0/22
- •103.22.200.0/22
- •103.31.4.0/22
- •104.16.0.0/13
- •104.24.0.0/14
- •108.162.192.0/18
- •131.0.72.0/22
- •141.101.64.0/18
- •162.158.0.0/15
- •172.64.0.0/13
- •173.245.48.0/20
- •188.114.96.0/20
- •190.93.240.0/20
- •197.234.240.0/22
- •198.41.128.0/17
–jeroen
https://www.youtube.com/watch?v=OLRldZjty_s
Leave a Comment