[ANN Mt.Gox] It’s been an epic few days: What happened?

[MPOE-PR]

Looky what I found:



Anyone recall?

[1715528839]

1715528839

[1715528839]

1715528839

[1715528839]

1715528839

[Zaih]

Cheers for the update. Ignore the haters 

[human]

the attack can also happen in order to STABILIZE bitcoin. the more people are unsatisfied with mtgox, the more they will flock to other exchanges and STABILIZE the bitcoin ecosystem. we do not need one huge centralized exchange. remember this!

[MPOE-PR]

Also MtGox could take a position on my UDP streams idea, which could be any of the following without commitment:

a) Great idea, we haven't thought of it, and you're right, it would totally get information out immune to DDoS, we'll consider it but like anything else will take time
b) Great idea, but we don't agree it would work as well as you think it will, or for (specific technical reason) won't work on our platform
c) We haven't got a clue as to what this means
d) I don't have a clue what this means because I'm not a developer or tech guy myself, but I have relayed your suggestion to someone more technical, and he says (response).  (Hopefully this suggestion is more valuable than to merely forward it blindly like the latest facebook meme, since MtGox's reputation is suffering and this will actually solve the claimed issue at hand)

Just to be clear, using UDP to broadcast ticker data would be, for all intents and purposes, IMMUNE from DDoS attacks, because such a stream consists solely of outbound traffic which is not influenced by inbound traffic.  Unlike a normal stream, there is no inbound overhead for packets to acknowledge or to keep the connection in sync, packets which can be drowned out in a DDoS attack.  UDP is much more like a point-to-point radio broadcast: the signal gets sent from point A to B even if nobody's listening

I don't think you understand how DDoS works on this level. Your UDP stream would have to have a source, which would have to have an IP, which then would get flooded to crap. It's the routers that pop, not the machines.

[nexus 6]

I propose two ideas, i dont know if good ideas:

1-since the attackers are operating in order to get benefits by buy/sell, couldn´t gox proceed to identify them or putting under suspect some accounts?

2-stop the site while ddos occurs?

[deeplink]

I don't think you understand how DDoS works on this level. Your UDP stream would have to have a source, which would have to have an IP, which then would get flooded to crap. It's the routers that pop, not the machines.

The point is that when merely using UDP, unlike TCP, the source can block ALL incoming traffic which makes it immune to DDoS. As casascius points out, UDP is like a radio broadcast signal. TCP is like the postal service with delivery confirmation.

[MPOE-PR]

The point is that when merely using UDP, unlike TCP, the source can block ALL incoming traffic which makes it immune to DDoS. As casascius points out, UDP is like a radio broadcast signal. TCP is like the postal service with delivery confirmation.

At what level do you propose blocking the incoming traffic?

[phelix]

Why the lag? It's really not that many transactions. It has to be 90% the procedure / architecture of your engine.

Also +1 to UDP

[sega01]

Thank you for the explanation and terrific post.

What can be done?
Believe it or not, there is pretty much nothing that can be done. Large companies are frequently victims of these kinds of attacks. Even though we are using one of the best companies to help us fight against these DDoS attacks, we are still being affected.

This is absolutely true. Some attacks are more application level (synflood, real HTTP requests), but others are of such a volume where the pipe is saturated. You'd pretty much have to have anycasted datacenters and massive pipes to the Internet to absorb large enough attacks. I've seen 3 Gbit/s attacks to minor sites for no apparent reason. I can't imagine what MTGox gets on a regular basis.

Re, the UDP suggestion. That might not be a bad idea at first glance. It'd work if MTGox advertised the price from some mostly unknown IPs and out different routers, out to a list of subscribers. Another option would be to put this data in DNS, maybe in a TXT or SRV record with a TTL of 60. Then the DNS servers might be attacked, which could be a new problem.

In my opinion, MTGox runs a great site. It's a bit tricky to get onto and the interface isn't as sleek as some sites, but ultimately, MTGox has single handedly encouraged a massive growth of adoption. I think ideally trades should be distributed by nature, but MTGox is still (and probably always will be) the benchmark site for Bitcoin trading, especially in bulk.

My hat is off to these guys for how thorough they are, dealing with the past through days, and 57,000 signups in one month. Those are some real challenges.

[prof7bit]

Just to be clear, using UDP to broadcast ticker data would be, for all intents and purposes, IMMUNE from DDoS attacks

I guess there simply would not be be anything to broadcast during the times when the engine is down or when nobody can make any trades. It would broadcast only silence. Maybe it could broadcast "Help!!! Help!!! We're under attack!".

[MPOE-PR]

I've seen 3 Gbit/s attacks to minor sites for no apparent reason. I can't imagine what MTGox gets on a regular basis.

They say reading enriches the imagination.

Very high spikes. While baseline remained for the entire duration in the 5-10 Gbit range, I've seen spikes as high as 100Gbit and I'm not even sure I've actually measured the highest ones.

Re, the UDP suggestion. That might not be a bad idea at first glance. It'd work if MTGox advertised the price from some mostly unknown IPs and out different routers, out to a list of subscribers. Another option would be to put this data in DNS, maybe in a TXT or SRV record with a TTL of 60. Then the DNS servers might be attacked, which could be a new problem.

You are rubbing sticks together trying to solve already solved problems. The many ways available for talking to MPEx.

I guess there simply would not be be anything to broadcast during the times when the engine is down or when nobody can make any trades. It would broadcast only silence. Maybe it could broadcast "Help!!! Help!!! We're under attack!".

Of course the clueless knowitalls haven't yet answered the

At what level do you propose blocking the incoming traffic?

[casascius]

The point is that when merely using UDP, unlike TCP, the source can block ALL incoming traffic which makes it immune to DDoS. As casascius points out, UDP is like a radio broadcast signal. TCP is like the postal service with delivery confirmation.

At what level do you propose blocking the incoming traffic?

Before it comes within miles of the host sending it.  After not informing the public who the UDP is coming from.

The UDP sending address doesn't have to be public knowledge, since not anyone can necessarily subscribe to it.  It would be a private UDP feed only offered to specific known sites.  The UDP feed would be used to drive the services of other sites who currently get it via websocket now, who in turn could provide that data to other downstream TCP websocket clients.

[gweedo]

The point is that when merely using UDP, unlike TCP, the source can block ALL incoming traffic which makes it immune to DDoS. As casascius points out, UDP is like a radio broadcast signal. TCP is like the postal service with delivery confirmation.

At what level do you propose blocking the incoming traffic?

Before it comes within miles of the host sending it.  After not informing the public who the UDP is coming from.

The UDP sending address doesn't have to be public knowledge, since not anyone can necessarily subscribe to it.  It would be a private UDP feed only offered to specific known sites.  The UDP feed would be used to drive the services of other sites who currently get it via websocket now, who in turn could provide that data to other downstream TCP websocket clients.

UDP isn't going to solve this problem, instead it make it harder for bots to trade. If they really wanted to solve this issue it is so simple. The trading engine should be ran completely offline, and use a database, like redis to store all information so the rest api can still have access to the information. Also the trade engine should be written in java or python.

[casascius]

...which makes it immune to DDoS...

UDP isn't going to solve this problem, instead it make it harder for bots to trade.

I am not sure that trading bots and DDoS are considered the same problem from the view of the consensus here.

[gweedo]

...which makes it immune to DDoS...

UDP isn't going to solve this problem, instead it make it harder for bots to trade.

I am not sure that trading bots and DDoS are considered the same problem from the view of the consensus here.

Do you know what UDP is? If they switched to UDP now all PHP/Python/java scripts would be useless and take now even more code to just connect. Also trading bots need reliable connections which UDP doesn't support by any means. UDP is great for the same information broadcast over and over, like said above a radio. Trading shouldn't use UDP it makes it very unreliable. They just need either a better network architecture clearly.

[casascius]

Do you know what UDP is? If they switched to UDP now all PHP/Python/java scripts would be useless and take now even more code to just connect. Also trading bots need reliable connections which UDP doesn't support by any means. UDP is great for the same information broadcast over and over, like said above a radio. Trading shouldn't use UDP it makes it very unreliable. They just need either a better network architecture clearly.

Yes, UDP stands for Useless Deprecated Protocol, it's that protocol that automatically detects and cunningly removes any information and redundancy you add to transmissions to help an application using it detect and recover from losses of datagrams, thereby making it useless as an upstream data source for scripts.  It is also a protocol that is specialized in conveying data that, by virtue of having traveled via UDP, becomes impossible to republish on a TCP stream for the benefit of being consumed by scripts, to help increase the workload of script writers who of course will want to implement a UDP listener in their scripts directly.  It is commonly known as an unreliable protocol, and it gets this reputation by sneakily altering important data in such a manner where it cannot be made reliable through other methods.  In fact, it even warps the minds of people considering using it, such that they cease to even know what UDP is!

[mp420]

Also the trade engine should be written in java or python.

Why?

I would like the trade engine to be written in Haskell, Erlang or some flavor of Lisp. Just because a functional language makes it easier to keep the code clean of unintended consequences. Also, Java is very bloaty, not a good thing for high performance code.

However, I understand sometimes compromise is necessary.

[MPOE-PR]

Before it comes within miles of the host sending it.  After not informing the public who the UDP is coming from.

The UDP sending address doesn't have to be public knowledge, since not anyone can necessarily subscribe to it.  It would be a private UDP feed only offered to specific known sites.  The UDP feed would be used to drive the services of other sites who currently get it via websocket now, who in turn could provide that data to other downstream TCP websocket clients.

This may work, but you will have to make arrangements with the DC hosting, which is something they may or may not be able to do (from the looks of it, more like not).

Yes, UDP stands for Useless Deprecated Protocol, it's that protocol that automatically detects and cunningly removes any information and redundancy you add to transmissions to help an application using it detect and recover from losses of datagrams, thereby making it useless as an upstream data source for scripts.  It is also a protocol that is specialized in conveying data that, by virtue of having traveled via UDP, becomes impossible to republish on a TCP stream for the benefit of being consumed by scripts, to help increase the workload of script writers who of course will want to implement a UDP listener in their scripts directly.  It is commonly known as an unreliable protocol, and it gets this reputation by sneakily altering important data in such a manner where it cannot be made reliable through other methods.  In fact, it even warps the minds of people considering using it, such that they cease to even know what UDP is!

Ahaha can I have that engraved on a medallion? 

[SomeWhere]

Also the trade engine should be written in java or python.

Why?

I would like the trade engine to be written in Haskell, Erlang or some flavor of Lisp. Just because a functional language makes it easier to keep the code clean of unintended consequences. Also, Java is very bloaty, not a good thing for high performance code.

However, I understand sometimes compromise is necessary.

A high performance trade engine should be written in C/C++, simple as that.

[Dr3AM$cAp3]

the attack can also happen in order to STABILIZE bitcoin. the more people are unsatisfied with mtgox, the more they will flock to other exchanges and STABILIZE the bitcoin ecosystem. we do not need one huge centralized exchange. remember this!

These are my thoughts to an extent.
Down with Walmart. *ren and stimpy stinky face*
Anarchy!