The ~entire internet went down.
Nope, it only happened with sites that integrate Cloudflare, not all sites down. Not sure which things might be changed in the future, with the forum, and coming Epochtalk forum too.
With regret, I am (for now) admitting defeat on the DDoS front, and we will soon be using using Cloudflare to protect against DDoS attacks. This change is in progress, and will take ~24 hours for everyone to see.
I really don't believe in willingly putting a man-in-the-middle in your HTTPS like this, but my homebrew DDoS mitigation has been one of my biggest time sinks for the last 6 months or so, and the necessary servers are still pretty expensive. If I had more manpower, then I would prioritize maintaining our own DDoS protection, but with me as the only sysadmin and current-software developer, it's become unsustainable.
I especially dislike Cloudflare, which I'm almost certain is basically owned by US intelligence agencies. I considered several alternatives to Cloudflare, but the smaller ones (eg. Stackpath and OVH) didn't strike me as reputable/competent enough, and the enterprise-targeted ones like Incapsula and Akamai are around $3500/month. Even though $3500/month seems absolutely ridiculous to me, I was seriously considering Incapsula due to its pretty good reputation, but then they were having all sorts of technical issues while I was trying to set it up. So I gave up for now and went with Cloudflare.
The Internet is seriously flawed if everyone needs to huddle behind these huge centralized anti-DDoS companies in order to survive...
The security implications are that Cloudflare can read everything you send to or receive from the server, including your cleartext password and any PMs you send or look at. They can't access the database arbitrarily, though: they can only see data that passes over the Internet.
Tor users and benevolent-bot operators: please wait a couple of days for the current DDoS to subside, and then post your complaints here. I am able and willing to tune Cloudflare to be minimally annoying. Not every Cloudflare site has to do that "Using Tor? Here's an impossible captcha" thing.
The Internet is fundamentally broken. We
need DDoS protection at the network layer, or else you're going to continue seeing 99% of the Internet hiding behind a few centralized third-parties. It's absolutely ridiculous. Realize also that Cloudflare can see all traffic unencrypted. They're almost certainly an NSA honeypot already, but even if not, their many screwups make them unworthy of this kind of trust. (Their Argo tunnel doesn't fix this trust issue at all, BTW.) However, since the Internet is broken fundamentally, mitigating it is too difficult for it to be a good idea for me to devote resources to it at this time.
I don't have time to work on this at all, but if someone created a non-profit dedicated to producing decentralized anti-DDoS solutions, I'd donate to it. On github I see two very immature projects in this area:
-
gatekeeper is intended for large organizations, and blocks attacks at the network/transport layer. However, I've found that SYNPROXY gateways plus upstream UDP blocking is sufficient for this on bitcointalk.org's scale, and gatekeeper also requires access to BGP, which isn't common unless you're pretty big.
-
AntiDDOS works at layer 7, which is where
my homebrew DDoS protection broke down. But it doesn't have a good IP classification system, and it's based on (and assumes the existence of) a single final application server.
(BTW, this problem is an example of centralization being used as an ever-increasing crutch for systems that are technologically flawed. It has parallels to scaling of cryptocurrencies and other supposed-to-be-decentralized systems.)
