Cloudflare - Pain In The Ass

[AB de Royse777]

Please enable cookies.
Error 1023 Ray ID: 4f011fa609786b71 • 2019-07-02 13:58:46 UTC
Could not find host
What happened?

You've requested a page on a website (bitcointalk.org) that is on the Cloudflare network. Cloudflare is currently unable to resolve your requested domain (bitcointalk.org). There are two potential causes of this:

    Most likely: if the owner just signed up for Cloudflare it can take a few minutes for the website's information to be distributed to our global network.
    Less likely: something is wrong with this site's configuration. Usually this happens when accounts have been signed up with a partner organization (e.g., a hosting provider) and the provider's DNS fails.

Cloudflare Ray ID: 4f011fa609786b71 • Your IP: 31.220.0.225 • Performance & security by Cloudflare

Why Cloudflare? Can we not run this service without this third party?

[subSTRATA]

some time ago bitcointalk was going through a lot of ddos attacks, and the homebrew protection designed by theymos just wasnt cutting it anymore. you can read his post here if you really want to. https://bitcointalk.org/index.php?topic=2485318.msg25449826#msg25449826

[AB de Royse777]

some time ago bitcointalk was going through a lot of ddos attacks, and the homebrew protection designed by theymos just wasnt cutting it anymore. you can read his post here if you really want to. https://bitcointalk.org/index.php?topic=2485318.msg25449826#msg25449826

Yes, I was aware about it. I think it's time for us to create our own security algorithm. It's been over one and half years now. We are paying this people and possibly compromising our privacy. I hope theymos do something about it.

[The Sceptical Chymist]

Yeah, I just got the error message on bitcointalk and another website as well.  Fortunately both got back to normal within a minute or so, so no major inconvenience from my end.  This isn't the first time it's happened, and I'm sure it won't be the last.

We are paying this people and possibly compromising our privacy. I hope theymos do something about it.

Don't know about you, but I never paid anyone in charge of the forum for anything.  But yeah, it'd be nice if Theymos changed some things.

[AB de Royse777]

~snip~

We are paying this people and possibly compromising our privacy. I hope theymos do something about it.

Don't know about you, but I never paid anyone in charge of the forum for anything.  But yeah, it'd be nice if Theymos changed some things.

LOL I did not mean us (you and me or other) literally. I meant theymos paying for the service and I considered theymos as a part of the community.  Sorry about the confusion. Hope we are clear now unless it was a sarcasm from you :-P

[subSTRATA]

some time ago bitcointalk was going through a lot of ddos attacks, and the homebrew protection designed by theymos just wasnt cutting it anymore. you can read his post here if you really want to. https://bitcointalk.org/index.php?topic=2485318.msg25449826#msg25449826

Yes, I was aware about it. I think it's time for us to create our own security algorithm. It's been over one and half years now. We are paying this people and possibly compromising our privacy. I hope theymos do something about it.

theymos did put out a post somewhere for the tech savvy members of the forum to devise a solution, along with a few requirements. I don't believe anything came of it in the end.

Managed to actually find the post:
https://bitcointalk.org/index.php?topic=2497008.msg25572747#msg25572747

The first major flaw with my setup is that it wasn't easy to change. My setup would grab a few configuration details (eg. the origin server IP) from VPC-local DNS records that I would set, but if I wanted to make deeper changes, I'd have to modify one of the instances, convert that into a new AMI, terminate all of the other instances, and then start new instances again. If I wanted to change the number of gates, I'd have to start/stop them manually and change the DNS records myself. A good solution would never require this much manual work, and would use things like auto scaling groups and CloudFormation to simplify it. It should only take a couple of minutes to add a new iptables rule, for example.

The second major flaw with my setup is that it lacked a good, systematic way of classifying IPs as good/bad/neutral. All of the gates should collect long-term stats on every IP which connects to them and contribute it to a central database. Using some sort of model over the data in the central IP database, it should then be able to determine whether an IP address is probably good (because it's been acting like a normal person browsing the site for a long time), probably bad (because it eg. just started requesting tons of pages), or unknown/neutral. Then based on that classification plus an idea of how busy the site currently is, it can block an IP, allow an IP, or insert a Cloudflare-style captcha challenge for an IP. If you pass the challenge, the system sets a cookie on you which whitelists you for several days.

For the forum to go back to a homebrew solution from Cloudflare, the above two pieces would need to be very-well-satisfied.

[tranthidung]

Yeah, I just got the error message on bitcointalk and another website as well.

Exactly! It seems that massive issues occured with sites that use Cloudflare, not only the forum. It occured around one hour ago and lasted around 15 minutes.

[theymos]

The Internet is fundamentally broken. We need DDoS protection at the network layer, or else you're going to continue seeing 99% of the Internet hiding behind a few centralized third-parties. It's absolutely ridiculous. Realize also that Cloudflare can see all traffic unencrypted. They're almost certainly an NSA honeypot already, but even if not, their many screwups make them unworthy of this kind of trust. (Their Argo tunnel doesn't fix this trust issue at all, BTW.) However, since the Internet is broken fundamentally, mitigating it is too difficult for it to be a good idea for me to devote resources to it at this time.

I don't have time to work on this at all, but if someone created a non-profit dedicated to producing decentralized anti-DDoS solutions, I'd donate to it. On github I see two very immature projects in this area:
 - gatekeeper is intended for large organizations, and blocks attacks at the network/transport layer. However, I've found that SYNPROXY gateways plus upstream UDP blocking is sufficient for this on bitcointalk.org's scale, and gatekeeper also requires access to BGP, which isn't common unless you're pretty big.
 - AntiDDOS works at layer 7, which is where my homebrew DDoS protection broke down. But it doesn't have a good IP classification system, it's based on (and assumes the existence of) a single final application server, and it's too simple/incomplete overall.

(BTW, this problem is an example of centralization being used as an ever-increasing crutch for systems that are technologically flawed. It has parallels to scaling of cryptocurrencies and other supposed-to-be-decentralized systems.)

[TECSHARE]

Yeah, I just got the error message on bitcointalk and another website as well.

Exactly! It seems that massive issues occured with sites that use Cloudflare, not only the forum. It occured around one hour ago and lasted around 15 minutes.

I can confirm this, I was using Bitchute and the forum as they both stopped responding at exactly the same time going to a 502.

[bernardos]

So it was actually Cloudflare that caused the downtime today. I didnt think too much about it. I had some issues on my end today because my anti virus software stopped working, after uninstalling it I couldnt connect to the internet at all. I guess the Windows firewall blocked everything. After I got my internet connection up and running I still couldnt access the forum.

[jackg]

The Internet is fundamentally broken. We need DDoS protection at the network layer, or else you're going to continue seeing 99% of the Internet hiding behind a few centralized third-parties. It's absolutely ridiculous. Realize also that Cloudflare can see all traffic unencrypted. They're almost certainly an NSA honeypot already, but even if not, their many screwups make them unworthy of this kind of trust. (Their Argo tunnel doesn't fix this trust issue at all, BTW.) However, since the Internet is broken fundamentally, mitigating it is too difficult for it to be a good idea for me to devote resources to it at this time.

Is there never a thing about making a new domain that certain trusted users are able to access?

I'd be able to buy a domain name easy enough to remember if you could give a bypass to me and a couple of others (other trusted legendaries that I consider trustworthy and the legendaries they consider trustworthy)


Remind me in a year about the decentralised forwarding thing, I'd need something for a disseration probably if I don't look into it before then. I plan on fiddling a lot with networking at some point so I'll try and take a look but it might be difficult since youc an spoof IP and MAC addresses as much as you like...

[TryNinja]

Is there never a thing about making a new domain that certain trusted users are able to access?

I'd be able to buy a domain name easy enough to remember if you could give a bypass to me and a couple of others (other trusted legendaries that I consider trustworthy and the legendaries they consider trustworthy)

That would be nice. This remembered me of a thread by ChipMixer made more than 1 year ago.

As I said before (https://bitcointalk.org/index.php?topic=2485318.msg26028401#msg26028401) we would like to buy access to BitcoinTalk forum that bypasses Cloudflare.

Anyone else is willing to pay to use BitcoinTalk?

theymos' answer to it:

It's a good idea, but I'm not sure how I would set that up. I'd need either one unique server IP per user or some method of segregating users so that you can't just pay the fee, find the IP address of the "pro" forum, and attack that.

I could use the CF API to whitelist IPs for a fee, but most people don't browse from a static IP. Maybe it's possible to use CF page rules to whitelist certain cookies; I'm not sure.

[Initscri]

I honestly don't blame Theymos for switching to Cloudflare. I've been in similar situations guarding larger*ish* sites. It certainly makes thing a hell of a lot simpler.

It's a good idea, but I'm not sure how I would set that up. I'd need either one unique server IP per user or some method of segregating users so that you can't just pay the fee, find the IP address of the "pro" forum, and attack that.

I could use the CF API to whitelist IPs for a fee, but most people don't browse from a static IP. Maybe it's possible to use CF page rules to whitelist certain cookies; I'm not sure.

I looked, not certain there is any feature YET at CF that allows this. There should be though, it would definitely help.