Cloudflare - Pain In The Ass

[AB de Royse777]

Please enable cookies.
Error 1023 Ray ID: 4f011fa609786b71 • 2019-07-02 13:58:46 UTC
Could not find host
What happened?

You've requested a page on a website (bitcointalk.org) that is on the Cloudflare network. Cloudflare is currently unable to resolve your requested domain (bitcointalk.org). There are two potential causes of this:

    Most likely: if the owner just signed up for Cloudflare it can take a few minutes for the website's information to be distributed to our global network.
    Less likely: something is wrong with this site's configuration. Usually this happens when accounts have been signed up with a partner organization (e.g., a hosting provider) and the provider's DNS fails.

Cloudflare Ray ID: 4f011fa609786b71 • Your IP: 31.220.0.225 • Performance & security by Cloudflare

Why Cloudflare? Can we not run this service without this third party?

[subSTRATA]

some time ago bitcointalk was going through a lot of ddos attacks, and the homebrew protection designed by theymos just wasnt cutting it anymore. you can read his post here if you really want to. https://bitcointalk.org/index.php?topic=2485318.msg25449826#msg25449826

[AB de Royse777]

some time ago bitcointalk was going through a lot of ddos attacks, and the homebrew protection designed by theymos just wasnt cutting it anymore. you can read his post here if you really want to. https://bitcointalk.org/index.php?topic=2485318.msg25449826#msg25449826

Yes, I was aware about it. I think it's time for us to create our own security algorithm. It's been over one and half years now. We are paying this people and possibly compromising our privacy. I hope theymos do something about it.

[The Sceptical Chymist]

Yeah, I just got the error message on bitcointalk and another website as well.  Fortunately both got back to normal within a minute or so, so no major inconvenience from my end.  This isn't the first time it's happened, and I'm sure it won't be the last.

We are paying this people and possibly compromising our privacy. I hope theymos do something about it.

Don't know about you, but I never paid anyone in charge of the forum for anything.  But yeah, it'd be nice if Theymos changed some things.

[AB de Royse777]

~snip~

We are paying this people and possibly compromising our privacy. I hope theymos do something about it.

Don't know about you, but I never paid anyone in charge of the forum for anything.  But yeah, it'd be nice if Theymos changed some things.

LOL I did not mean us (you and me or other) literally. I meant theymos paying for the service and I considered theymos as a part of the community.  Sorry about the confusion. Hope we are clear now unless it was a sarcasm from you :-P

[subSTRATA]

some time ago bitcointalk was going through a lot of ddos attacks, and the homebrew protection designed by theymos just wasnt cutting it anymore. you can read his post here if you really want to. https://bitcointalk.org/index.php?topic=2485318.msg25449826#msg25449826

Yes, I was aware about it. I think it's time for us to create our own security algorithm. It's been over one and half years now. We are paying this people and possibly compromising our privacy. I hope theymos do something about it.

theymos did put out a post somewhere for the tech savvy members of the forum to devise a solution, along with a few requirements. I don't believe anything came of it in the end.

Managed to actually find the post:
https://bitcointalk.org/index.php?topic=2497008.msg25572747#msg25572747

The first major flaw with my setup is that it wasn't easy to change. My setup would grab a few configuration details (eg. the origin server IP) from VPC-local DNS records that I would set, but if I wanted to make deeper changes, I'd have to modify one of the instances, convert that into a new AMI, terminate all of the other instances, and then start new instances again. If I wanted to change the number of gates, I'd have to start/stop them manually and change the DNS records myself. A good solution would never require this much manual work, and would use things like auto scaling groups and CloudFormation to simplify it. It should only take a couple of minutes to add a new iptables rule, for example.

The second major flaw with my setup is that it lacked a good, systematic way of classifying IPs as good/bad/neutral. All of the gates should collect long-term stats on every IP which connects to them and contribute it to a central database. Using some sort of model over the data in the central IP database, it should then be able to determine whether an IP address is probably good (because it's been acting like a normal person browsing the site for a long time), probably bad (because it eg. just started requesting tons of pages), or unknown/neutral. Then based on that classification plus an idea of how busy the site currently is, it can block an IP, allow an IP, or insert a Cloudflare-style captcha challenge for an IP. If you pass the challenge, the system sets a cookie on you which whitelists you for several days.

For the forum to go back to a homebrew solution from Cloudflare, the above two pieces would need to be very-well-satisfied.

[notocactus]

Yeah, I just got the error message on bitcointalk and another website as well.

Exactly! It seems that massive issues occured with sites that use Cloudflare, not only the forum. It occured around one hour ago and lasted around 15 minutes.

[theymos]

The Internet is fundamentally broken. We need DDoS protection at the network layer, or else you're going to continue seeing 99% of the Internet hiding behind a few centralized third-parties. It's absolutely ridiculous. Realize also that Cloudflare can see all traffic unencrypted. They're almost certainly an NSA honeypot already, but even if not, their many screwups make them unworthy of this kind of trust. (Their Argo tunnel doesn't fix this trust issue at all, BTW.) However, since the Internet is broken fundamentally, mitigating it is too difficult for it to be a good idea for me to devote resources to it at this time.

I don't have time to work on this at all, but if someone created a non-profit dedicated to producing decentralized anti-DDoS solutions, I'd donate to it. On github I see two very immature projects in this area:
 - gatekeeper is intended for large organizations, and blocks attacks at the network/transport layer. However, I've found that SYNPROXY gateways plus upstream UDP blocking is sufficient for this on bitcointalk.org's scale, and gatekeeper also requires access to BGP, which isn't common unless you're pretty big.
 - AntiDDOS works at layer 7, which is where my homebrew DDoS protection broke down. But it doesn't have a good IP classification system, it's based on (and assumes the existence of) a single final application server, and it's too simple/incomplete overall.

(BTW, this problem is an example of centralization being used as an ever-increasing crutch for systems that are technologically flawed. It has parallels to scaling of cryptocurrencies and other supposed-to-be-decentralized systems.)

[TECSHARE]

Yeah, I just got the error message on bitcointalk and another website as well.

Exactly! It seems that massive issues occured with sites that use Cloudflare, not only the forum. It occured around one hour ago and lasted around 15 minutes.

I can confirm this, I was using Bitchute and the forum as they both stopped responding at exactly the same time going to a 502.

[bernardos]

So it was actually Cloudflare that caused the downtime today. I didnt think too much about it. I had some issues on my end today because my anti virus software stopped working, after uninstalling it I couldnt connect to the internet at all. I guess the Windows firewall blocked everything. After I got my internet connection up and running I still couldnt access the forum.

[jackg]

The Internet is fundamentally broken. We need DDoS protection at the network layer, or else you're going to continue seeing 99% of the Internet hiding behind a few centralized third-parties. It's absolutely ridiculous. Realize also that Cloudflare can see all traffic unencrypted. They're almost certainly an NSA honeypot already, but even if not, their many screwups make them unworthy of this kind of trust. (Their Argo tunnel doesn't fix this trust issue at all, BTW.) However, since the Internet is broken fundamentally, mitigating it is too difficult for it to be a good idea for me to devote resources to it at this time.

Is there never a thing about making a new domain that certain trusted users are able to access?

I'd be able to buy a domain name easy enough to remember if you could give a bypass to me and a couple of others (other trusted legendaries that I consider trustworthy and the legendaries they consider trustworthy)


Remind me in a year about the decentralised forwarding thing, I'd need something for a disseration probably if I don't look into it before then. I plan on fiddling a lot with networking at some point so I'll try and take a look but it might be difficult since youc an spoof IP and MAC addresses as much as you like...

[TryNinja]

Is there never a thing about making a new domain that certain trusted users are able to access?

I'd be able to buy a domain name easy enough to remember if you could give a bypass to me and a couple of others (other trusted legendaries that I consider trustworthy and the legendaries they consider trustworthy)

That would be nice. This remembered me of a thread by ChipMixer made more than 1 year ago.

As I said before (https://bitcointalk.org/index.php?topic=2485318.msg26028401#msg26028401) we would like to buy access to BitcoinTalk forum that bypasses Cloudflare.

Anyone else is willing to pay to use BitcoinTalk?

theymos' answer to it:

It's a good idea, but I'm not sure how I would set that up. I'd need either one unique server IP per user or some method of segregating users so that you can't just pay the fee, find the IP address of the "pro" forum, and attack that.

I could use the CF API to whitelist IPs for a fee, but most people don't browse from a static IP. Maybe it's possible to use CF page rules to whitelist certain cookies; I'm not sure.

[Initscri]

I honestly don't blame Theymos for switching to Cloudflare. I've been in similar situations guarding larger*ish* sites. It certainly makes thing a hell of a lot simpler.

It's a good idea, but I'm not sure how I would set that up. I'd need either one unique server IP per user or some method of segregating users so that you can't just pay the fee, find the IP address of the "pro" forum, and attack that.

I could use the CF API to whitelist IPs for a fee, but most people don't browse from a static IP. Maybe it's possible to use CF page rules to whitelist certain cookies; I'm not sure.

I looked, not certain there is any feature YET at CF that allows this. There should be though, it would definitely help.

[UmerIdrees]

Why Cloudflare? Can we not run this service without this third party?

A question that was asked a few years back, i didn't know how badly it could affect the whole Internet as i have witnessed it today. This is really a concern that one or two service providers can bring down the whole Internet. Yeah, the sites use the services of Cloudflare to prevent themselves from DDOS attacks but if the hackers can bring that service down, this would have a much bigger impact 

[The Sceptical Chymist]

Why Cloudflare? Can we not run this service without this third party?

A question that was asked a few years back, i didn't know how badly it could affect the whole Internet as i have witnessed it today. This is really a concern that one or two service providers can bring down the whole Internet. Yeah, the sites use the services of Cloudflare to prevent themselves from DDOS attacks but if the hackers can bring that service down, this would have a much bigger impact 

Gah!!!!  You tricked me into meriting the OP by necrobumping this thread (my fault for not reading the date, of course).  And I can see that I've already posted here a few years ago and that not only have things not changed, they've likely gotten worse--and I'm talking in a global sense, not just about our beloved forum here.

I wonder how many of you remember what the internet was like circa 1996-2000ish.  For those of you who weren't old enough to have experienced the birth and growth of it, I am genuinely saddened.  What the internet turned into was a continent-long stretch of urban sprawl where once it used to be a jaw-dropping scenic route that made you not want to stop to pee.

[NotATether]

A question that was asked a few years back, i didn't know how badly it could affect the whole Internet as i have witnessed it today. This is really a concern that one or two service providers can bring down the whole Internet. Yeah, the sites use the services of Cloudflare to prevent themselves from DDOS attacks but if the hackers can bring that service down, this would have a much bigger impact 

It's quite annoying. You survive an AWS outage and then you go down because Cloudflare gets hit.

The worst part of it is that there was nothing you can do as a customer to prevent the Cloudflare errors in the first place.

DDoS attacks are frustrating. I concur with theymos here, but I have no idea how anyone can protect themselves from an attack except by the ASNs themselves implementing this protection. So they can blackhole the attackers' IP addresses with BGP.

[Mpamaegbu]

Gah!!!!  You tricked me into meriting the OP by necrobumping this thread (my fault for not reading the date, of course).

I expect this thread to be reawakened a few years down the line again, except it's locked.

DDoS attacks are frustrating. I concur with theymos here, but I have no idea how anyone can protect themselves from an attack except by the ASNs themselves implementing this protection. So they can blackhole the attackers' IP addresses with BGP.

How many IP addresses can anyone blackhole? New ones will spring up if existing ones are restricted. Attacks won't cease, from my little experience in this space, I can say this.

It was actually frustrating two days ago when cloudflare got activated again and lots of users couldn't assess this forum. I was directly affected. I believe this won't be the end to it.

[coupable]

Gah!!!!  You tricked me into meriting the OP by necrobumping this thread (my fault for not reading the date, of course).

I expect this thread to be reawakened a few years down the line again, except it's locked.

There are already at least two other threads discussing the same topic about forum and cloudflare. Also topics in local boards.

DDoS attacks are frustrating. I concur with theymos here, but I have no idea how anyone can protect themselves from an attack except by the ASNs themselves implementing this protection. So they can blackhole the attackers' IP addresses with BGP.

How many IP addresses can anyone blackhole? New ones will spring up if existing ones are restricted. Attacks won't cease, from my little experience in this space, I can say this.

It was actually frustrating two days ago when cloudflare got activated again and lots of users couldn't assess this forum. I was directly affected. I believe this won't be the end to it.

Of course it won't be the end for such a sabotage attacks. However, from my little experience in this space, I can say that no one have noticed the forum infected by ddos attacks since the integration with cloudflare. I can't tell how much the service is reliable but after the recent incident, everybody should think about an alternative.

[arhipova]

Why Cloudflare? Can we not run this service without this third party?

I have not seen Cloudflare going down in so many years. It stopped working for few minutes and you want it be removed. Have you considered the protection it has provided from different types of attacks like DDOS attacks. Had it not been used on this forum, there might have been multiple instances of the forum going down because of such attacks. So, now you decide, which situation is better for you ?

[noormcs5]

Why Cloudflare? Can we not run this service without this third party?

I have not seen Cloudflare going down in so many years. It stopped working for few minutes and you want it be removed. Have you considered the protection it has provided from different types of attacks like DDOS attacks. Had it not been used on this forum, there might have been multiple instances of the forum going down because of such attacks. So, now you decide, which situation is better for you ?

Using a service for many years doesn't make it foolproof. Yeah, it is a very good service, protecting against the DDOS attacks but then what we saw recently is even more severe. I know Cloudfare is doing a good work, but does it make sense to depend on a single service, a single point of failure  

I heard people saying that sometime in the future there will be a complete internet outage globally, I never believed it as how can the whole of the internet in the world can go down all at once, well, now I know it's possible