Bitcoin Forum
December 14, 2017, 01:41:54 PM *
News: Latest stable version of Bitcoin Core: 0.15.1  [Torrent].
 
   Home   Help Search Donate Login Register  
Pages: [1]
  Print  
Author Topic: How my custom DDoS protection worked, and how it could've been improved  (Read 452 times)
theymos
Administrator
Legendary
*
Offline Offline

Activity: 2870


View Profile
December 01, 2017, 07:51:51 PM
 #1

About a year ago I created my own homebrew DDoS protection. Here's how it worked:

Set up several smallish EC2 instances. Each one acts as a reverse proxy to the origin server:

Code:
client1-->gate1\
client2-->gate2->--->origin server
client3-->gate3/

It's just an nginx reverse proxy. Pass the real IP in the X-Real-IP header, etc. Easy.

Each gate will have iptables and nginx rules to detect easy attacks (eg. rate limiting). Importantly, they all must have SYNPROXY rules, a feature of modern Linux kernels. Having SYNPROXY iptables rules over several gateways like this completely defeats all SYN flood attacks.

The gateways need to be in an AWS VPC set to block all UDP traffic in the VPC's stateless traffic settings. This completely blocks all UDP flood attacks. If you instead block UDP traffic in the gates' security groups, then very large UDP floods can still affect you. It has to be at the VPC level.

I found that the best way to set up the DNS to distribute traffic was like this. Assume that you have 4 gates, g1 through g4. Then using Route 53's weighted record feature, you would have the DNS return at random one of the following 5 pairs of IPs, each with a TTL of 5 minutes:
g1&g2
g2&g3
g3&g4
g4&g1

This seems to work better than just putting all of the gate IPs into one A record. I think that the randomization that should happen in that case actually gets cached at some points, and so whichever record is returned first at <wherever it's cached> gets hit harder.

Additionally, I had a system of classifying and blocking malicious-looking IPs, but it failed to work well enough in the end, so I'm not going to describe it in detail.

So that's my homebrew DDoS protection that we were using for the last year or so. It worked impressively well against many attacks which you might think would require something like Cloudflare, but failed in the end against attackers with thousands of IPs, making full TCP connections, who can blend into the legit traffic too well. A more complete solution which could replace Cloudflare etc. in many ways would look more like this:

-----

The first major flaw with my setup is that it wasn't easy to change. My setup would grab a few configuration details (eg. the origin server IP) from VPC-local DNS records that I would set, but if I wanted to make deeper changes, I'd have to modify one of the instances, convert that into a new AMI, terminate all of the other instances, and then start new instances again. If I wanted to change the number of gates, I'd have to start/stop them manually and change the DNS records myself. A good solution would never require this much manual work, and would use things like auto scaling groups and CloudFormation to simplify it. It should only take a couple of minutes to add a new iptables rule, for example.

The second major flaw with my setup is that it lacked a good, systematic way of classifying IPs as good/bad/neutral. All of the gates should collect long-term stats on every IP which connects to them and contribute it to a central database. Using some sort of model over the data in the central IP database, it should then be able to determine whether an IP address is probably good (because it's been acting like a normal person browsing the site for a long time), probably bad (because it eg. just started requesting tons of pages), or unknown/neutral. Then based on that classification plus an idea of how busy the site currently is, it can block an IP, allow an IP, or insert a Cloudflare-style captcha challenge for an IP. If you pass the challenge, the system sets a cookie on you which whitelists you for several days.

For the forum to go back to a homebrew solution from Cloudflare, the above two pieces would need to be very-well-satisfied.

Another point is that you could design the system such that it does not require looking into HTTPS traffic. It can just work at the TCP layer and pass the encrypted HTTPS traffic verbatim. I'm not sure how exactly you would tunnel the real data to the real server (I previously thought that GRE tunnels would work, but somebody told me that this might not be the appropriate tool), but it should definitely be possible. The upside to this is that you can use a very powerful service like AWS without trusting them too much. The downside is that you cannot use layer 7 data for IP classification, and you cannot insert a challenge; it's either block or allow. The ideal anti-DDoS solution would give you the option of whether you want to give the gates access to your HTTPS or not.

1NXYoJ5xU91Jp83XfVMHwwTUyZFK64BoAD
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1513258914
Hero Member
*
Offline Offline

Posts: 1513258914

View Profile Personal Message (Offline)

Ignore
1513258914
Reply with quote  #2

1513258914
Report to moderator
stevegee58
Hero Member
*****
Offline Offline

Activity: 836



View Profile
December 01, 2017, 07:56:31 PM
 #2

That was actually pretty interesting.  At least you learned a lot from the experience!
I will say the site is very snappy now.

You are in a maze of twisty little passages, all alike.
bluefirecorp_
Member
**
Offline Offline

Activity: 84


View Profile
December 02, 2017, 02:53:57 AM
 #3

>The first major flaw with my setup is that it wasn't easy to change. My setup would grab a few configuration details (eg. the origin server IP) from VPC-local DNS records that I would set, but if I wanted to make deeper changes, I'd have to modify one of the instances, convert that into a new AMI, terminate all of the other instances, and then start new instances again. If I wanted to change the number of gates, I'd have to start/stop them manually and change the DNS records myself. A good solution would never require this much manual work, and would use things like auto scaling groups and CloudFormation to simplify it. It should only take a couple of minutes to add a new iptables rule, for example.


Look into some automation tooling for this. I don't think it'd be that hard to roll out changes automatically. Locking the forums into a system like cloudformation might not be the best idea.

Check into terraform to deploy on different infrastructure. AWS is neat, but linode could be much more cost effective to mitigate a DDoS.

>Another point is that you could design the system such that it does not require looking into HTTPS traffic. It can just work at the TCP layer and pass the encrypted HTTPS traffic verbatim. I'm not sure how exactly you would tunnel the real data to the real server (I previously thought that GRE tunnels would work, but somebody told me that this might not be the appropriate tool), but it should definitely be possible. The upside to this is that you can use a very powerful service like AWS without trusting them too much. The downside is that you cannot use layer 7 data for IP classification, and you cannot insert a challenge; it's either block or allow. The ideal anti-DDoS solution would give you the option of whether you want to give the gates access to your HTTPS or not.


Like a standard firewall? Layer 4 blocking?
mprep
Diamond Member
Global Moderator
Legendary
*
Offline Offline

Activity: 1722


In a world of peaches, don't ask for apple sauce


View Profile
December 02, 2017, 04:49:30 AM
 #4

<...>AWS is neat, but linode could be much more cost effective to mitigate a DDoS.<...>
A bit off-topic, but Linode doesn't really have a good rep in the Bitcoin community: https://arstechnica.com/information-technology/2012/03/bitcoins-worth-228000-stolen-from-customers-of-hacked-webhost/


Advertisment:
           ▄▄███████▄▄
        ▄███▀▀
▄▄▄▄    ▀▄
     ▄▄█████████████▄▄  ▀▄
  ▄▀▀██▀           ▀▀██▄▄▀▄
▄▀  ██                 ▀██
  ██       ▀▀█▀▀         █
█▀        █ █ █        ▄█▀▄
▀▄         █ █ █       ▄█  █
 ██         █▄▄▄█      ▄█  ▄▀
  ██▄                ▄█▀  ▄▀
  ▀▄▀██▄▄          ▄█▀  ▄▀
   ▀▄ ▀▀███▄▄▄▄▄▄█████▀▀
     ▀▀▄▄▄▄▄▄▀▀▀▀▀▀▀
UTRUST ▀████████▄
  ▀███████▄
    ▀██████▄
      ▀██████
       ▀█████
        ▀████▄
         █████
          ▀███
           ███
           ▀██
            ██
             █
             █
●  Download WHITEPAPER  ●
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ ▼ ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
facebook      twitter      slack
▀████████▄
  ▀███████▄
    ▀██████▄
      ▀██████
       ▀█████
        ▀████▄
         █████
          ▀███
           ███
           ▀██
            ██
             █
             █
Unofficial Bitcointalk API - get posts, boards, topics
Bitcointalk Post Iconizer - custom post editing buttons
|  Services advertised here are not endorsed by me
|  Advertise in my signature
bluefirecorp_
Member
**
Offline Offline

Activity: 84


View Profile
December 02, 2017, 12:53:08 PM
 #5

<...>AWS is neat, but linode could be much more cost effective to mitigate a DDoS.<...>
A bit off-topic, but Linode doesn't really have a good rep in the Bitcoin community: https://arstechnica.com/information-technology/2012/03/bitcoins-worth-228000-stolen-from-customers-of-hacked-webhost/

That's strange. I didn't notice they were compromised in 2012. I'm betting their security measures have increased many times after an attack like that. However, I was just pointing out leveraging multiple providers instead of putting all your eggs into the same basket. Sure AWS functions great now, but why not also leverage OVH, GCP, Azure, DigitalOcean or any other provider. AWS is stupid expensive iirc; paying for them to mitigate a DDoS attack might not really be cost effective.
saddampbuh
Legendary
*
Online Online

Activity: 1204



View Profile
December 02, 2017, 01:41:10 PM
 #6

someone do me a favour and let the folks at kraken know about this

Be radical, have principles, be absolute, be that which the bourgeoisie calls an extremist: give yourself without counting or calculating, don't accept what they call ‘the reality of life' and act in such a way that you won't be accepted by that kind of ‘life', never abandon the principle of struggle.
Pages: [1]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!