Bitcoin Forum
August 14, 2018, 11:49:49 AM *
News: Latest stable version of Bitcoin Core: 0.16.2  [Torrent].
 
   Home   Help Search Donate Login Register  
Pages: [1] 2 3 »  All
  Print  
Author Topic: [ANN Mt.Gox] It’s been an epic few days: What happened?  (Read 3824 times)
Mt.Gox_Alex
Member
**
Offline Offline

Activity: 112
Merit: 10



View Profile WWW
April 04, 2013, 02:52:05 AM
 #1

Dear Mt.Gox users and Bitcoiners,

It’s been an epic few days on Bitcoin, with prices going up as high as $142 per BTC. We all hope that this is just the beginning!

However, there are many who will try to take advantage of the system. The past few days were a reminder of this sad truth.

Mt.Gox has been suffering from its worst trading lag ever, 502 errors, and at one point some users were not able to log in their account. The culprit is a major DDoS attack against Mt.Gox.

Since yesterday, we are continuing to experience a DDoS attack like we have never seen. While we are being protected by companies like Prolexic, the sheer volume of this DDoS left us scrambling to fine-tune the system every few hours to make sure that things don’t go beyond a few 502 error pages and trading lag.

Why has Mt.Gox become the target of a DDoS attack?
It is not yet clear who is behind this DDoS and we may never know, but these actions seem to have two major purposes:

1.   Destabilize Bitcoin in general.
It is not a secret Mt.Gox is the largest Bitcoin exchange with more than 80% of all USD trades and more than 70% of all currencies. Mt.Gox is an easy target for anyone that wants to hurt Bitcoin in general.

2.   Abuse the system for profit.
Attackers wait until the price of Bitcoins reaches a certain value, sell, destabilize the exchange, wait for everybody to panic-sell their Bitcoins, wait for the price to drop to a certain amount, then stop the attack and start buying as much as they can. Repeat this two or three times like we saw over the past few days and they profit.

What can be done?
Believe it or not, there is pretty much nothing that can be done. Large companies are frequently victims of these kinds of attacks. Even though we are using one of the best companies to help us fight against these DDoS attacks, we are still being affected.

There are a few things that we can implement to help fight the attacks, such as disconnecting the trade engine backend from the Internet. By separating the data center from the Mt.Gox website, we will continue to be able to trade.

What can you do?
Like our favorite author here at Tibanne says… Don’t Panic!

“Panic-selling is a wide-scale selling of an investment which causes a sharp decline in prices. Specifically, an investor wants to get out of an investment with little regard of the price obtained. The selling activity is problematic because the investor is selling in reaction to emotion and fear, rather than evaluating the fundamentals.” (Source: Wikipedia)

I understand that many of you have a lot at stake here, but remember that Bitcoin, despite being designed to have its value increase over time, will always be the victim of people trying to abuse the system, or even the value of Bitcoin decreasing occasionally. These are not new phenomena and have been present since the beginning of time when humans first started trading.

Trade Engine Lags
Lag affects everyone, not only us, but also major, world-renowned exchanges like the NASDAQ and NYSE. We can fix lag, but we cannot eradicate lag. Only small exchanges with low volume and liquidity are immune to lag.

Does this mean that we are giving up fighting lag? Hell, no. We are working on it by creating a new trade engine that will solve many problems, but it’s not a magic bullet. We can always try to scale our servers, but we cannot predict what happens from external sources: DDoS, panic selling, immediate increase of buyers, etc. Lag will always be there, but our mission is to make lag as small as possible.

Account Verification
As if a major DDoS attack was not enough, we at Mt.Gox are victim of our own success!

Last year, Mt.Gox saw an average of 9,000 to 10,000 new accounts created every month. This number doubled in January, tripled in February, and sextupled in March. In this month alone, March, over 57,000 new accounts were created!

Our support and account verification team went from four people in January 2012 to twenty-two people working every day of the week. We are now hiring even more people to solve this problem by finalizing some deals with external companies.

Remember that even if you are waiting for your account to be verified, you can still deposit or withdraw funds via our Japanese account and make your trades! (Only accounts that we pro-actively required to be verified are limited to deposits and trade only.)

Finally
We have seen a significant amount of comments on the web (various forums, Reddit, etc.) that portray Mt.Gox as a company held by “idiots” and other rather rude words, complaining about inability to deal with lag and other system issues, without understanding the magnitude of work and attacks we are facing every day.

I understand the frustration many of you feel. We hate this situation as well. Since we took over Mt.Gox, we have been through Hell and back and we are still here. We are still the largest exchange with over 420,000 trades per month and  USD $121 million monthly trade volume. We have worked our way through all the requirements needed to run our exchange legally.

Now, there are some things we can improve, but so far we are doing an incredible job that no other exchange has been able to do so far. While I understand a certain amount of frustration, realize what we have accomplished. I appreciate all the work you are doing everyday to push things forward and to help secure the future of Bitcoin

And to all of you who are supporting us on a daily basis, thank you! We could not have done any of this without your help!


Mt.Gox : The Leading International Bitcoin Exchange.
Mt.Gox Merchant Solutions :   Now Available!
1534247389
Hero Member
*
Offline Offline

Posts: 1534247389

View Profile Personal Message (Offline)

Ignore
1534247389
Reply with quote  #2

1534247389
Report to moderator
1534247389
Hero Member
*
Offline Offline

Posts: 1534247389

View Profile Personal Message (Offline)

Ignore
1534247389
Reply with quote  #2

1534247389
Report to moderator
1534247389
Hero Member
*
Offline Offline

Posts: 1534247389

View Profile Personal Message (Offline)

Ignore
1534247389
Reply with quote  #2

1534247389
Report to moderator
You can see the statistics of your reports to moderators on the "Report to moderator" pages.
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
anzy
Sr. Member
****
Offline Offline

Activity: 276
Merit: 250


I like helping people


View Profile
April 04, 2013, 02:54:18 AM
 #2

Thanks for the update

I like building things
justusranvier
Legendary
*
Offline Offline

Activity: 1400
Merit: 1006



View Profile WWW
April 04, 2013, 02:58:40 AM
 #3

Finally
We have seen a significant amount of comments on the web (various forums, Reddit, etc.) that portray Mt.Gox as a company held by “idiots” and other rather rude words, complaining about inability to deal with lag and other system issues, without understanding the magnitude of work and attacks we are facing every day.
This thread is a great example of the communication that goes a long way towards defusing those kinds of reactions.

When your customers don't know what's going on they tend to assume the worst.
TheButterZone
Legendary
*
Offline Offline

Activity: 2212
Merit: 1008


Pay with SegWit!


View Profile WWW
April 04, 2013, 03:02:26 AM
 #4

Do you have enough reserves to suspend trading and meet payroll/vendor obligations until the new trade engine is ready for prime time (and maybe the verification queue is beaten to within an inch of its life, as well)? Not sure how many strikes against Gox there are going to have to be, before most smart people say "you're out!" and look somewhere else.

Personally, Namecheap is being denied 4 years of domain transfer/renewals of mine because of this insane high-low. I was just about to execute my bitcoin deposit to them before the latest of seemingly countless DDOSs. Bit-Pay loses out on its fee as well.

NBA Store coupons here!
Saying that you don't trust someone because of their behavior is completely valid.
casascius
Mike Caldwell
VIP
Legendary
*
Offline Offline

Activity: 1386
Merit: 1039


The Casascius 1oz 10BTC Silver Round (w/ Gold B)


View Profile WWW
April 04, 2013, 03:06:05 AM
 #5

Thanks for the excellent feedback.

There is one thing you can do that's relatively easy.  You guys should consider doing a UDP blast of ticker information to perhaps a dozen volunteer sites who can use this information to relay it into an independently-provided web socket feed so that it doesn't have to go through you.  By UDP, just in case this isn't clear, I mean User Datagram Protocol, the sessionless protocol supported by TCP/IP, the kind of one-way outbound packets you can send from your facilities that shouldn't be affected by an inbound DDoS flood.

Companies claiming they got hacked and lost your coins sounds like fraud so perfect it could be called fashionable.  I never believe them.  If I ever experience the misfortune of a real intrusion, I declare I have been honest about the way I have managed the keys in Casascius Coins.  I maintain no ability to recover or reproduce the keys, not even under limitless duress or total intrusion.  Remember that trusting strangers with your coins without any recourse is, as a matter of principle, not a best practice.  Don't keep coins online. Use paper or hardware wallets instead.
SomeWhere
Member
**
Offline Offline

Activity: 71
Merit: 10


View Profile
April 04, 2013, 03:08:25 AM
 #6

I applaud you for this update and must admit that I was one of the people to go a little overboard.

This kind of communication goes a long way and is something you should do a lot more.

I however maintain that you had a *lot* of time over the past months to get a more efficient trading engine up and - given that it should be *the* top priority - the progress on that front is simply not satisfactory.

I, for one, will be moving to another exchange until Mt.Gox meets the standards that I expect from the leader of the industry.

P.S. Just to have some basis to my claims: I have a strong background in computer engineering, especially when it comes to big & performance intensive systems. I can assure you, I know the challenges that come with hosting infrastructure, a good trading engine, massively parallel architectures. Still, if prioritized the right way, results should have been there a long time ago.
repentance
Hero Member
*****
Offline Offline

Activity: 840
Merit: 1000


View Profile
April 04, 2013, 03:09:51 AM
 #7

Alex, you mentioned the verification issue.

It's really unclear to me what is going to happen with this when US/Canadian users get transferred to CoinLab.  Will the accounts which are currently awaiting verification still be verified by MtGox or will US/Canadian customers be verified by CoinLab following the transition (ie, will those US/Canadian customers currently awaiting verification by MtGox need to start the verification process again with CoinLab)?  

The lack of recent information about the transition is worrying as customers were initially told that they needed to agree to CoinLab's ToS if they wanted to use MtGox following the transition but the information on the CoinLab site says that the funds of US/Canadian customers will automatically be transferred to the US.  

This implies that people will need to withdraw their funds from MtGox before the transition if they don't want those funds transferred to the US bank (which may present difficulties for unverified customers given the backlog on verification) or they'll have to register as a CoinLab user whether they want to or not in order to withdraw their funds.

It was previously stated that MtGox user data would not be transferred to CoinLab without user agreement (ie, MtGox users accepting CoinLab's ToS), but if the funds are being transferred automatically then how will users be able to access them without using CoinLab?

I know you have a lot going on, but the transition to CoinLab has to the potential to be disastrous if you don't keep your users fully informed and - quite frankly - you don't need any more things happening right now which undermine people's confidence in you.


All I can say is that this is Bitcoin. I don't believe it until I see six confirmations.
Bitobsessed
Sr. Member
****
Offline Offline

Activity: 291
Merit: 250



View Profile
April 04, 2013, 03:10:49 AM
 #8

Dear Mt.Gox users and Bitcoiners,

It’s been an epic few days on Bitcoin, with prices going up as high as $142 per BTC. We all hope that this is just the beginning!


Thought the top was $147?
Mt.Gox_Alex
Member
**
Offline Offline

Activity: 112
Merit: 10



View Profile WWW
April 04, 2013, 03:15:56 AM
 #9

Dear Mt.Gox users and Bitcoiners,

It’s been an epic few days on Bitcoin, with prices going up as high as $142 per BTC. We all hope that this is just the beginning!


Thought the top was $147?

You are right (last 24hrs), I was stuck on the past 12hrs data

Mt.Gox : The Leading International Bitcoin Exchange.
Mt.Gox Merchant Solutions :   Now Available!
moni3z
Hero Member
*****
Offline Offline

Activity: 886
Merit: 1000



View Profile
April 04, 2013, 03:50:58 AM
 #10

Prolexic sucks, they are resellers. Go with Black Lotus and make them agree to guaranteed uptime protection they'll do it. You could also clone a backup elastic cloud image on Amazon that ssh's to your secure db and switch to it whenever you get hammered beyond 100Gbps
Mt.Gox_Alex
Member
**
Offline Offline

Activity: 112
Merit: 10



View Profile WWW
April 04, 2013, 03:52:48 AM
 #11

Alex, you mentioned the verification issue.

It's really unclear to me what is going to happen with this when US/Canadian users get transferred to CoinLab.  Will the accounts which are currently awaiting verification still be verified by MtGox or will US/Canadian customers be verified by CoinLab following the transition (ie, will those US/Canadian customers currently awaiting verification by MtGox need to start the verification process again with CoinLab)?  

The lack of recent information about the transition is worrying as customers were initially told that they needed to agree to CoinLab's ToS if they wanted to use MtGox following the transition but the information on the CoinLab site says that the funds of US/Canadian customers will automatically be transferred to the US.  

This implies that people will need to withdraw their funds from MtGox before the transition if they don't want those funds transferred to the US bank (which may present difficulties for unverified customers given the backlog on verification) or they'll have to register as a CoinLab user whether they want to or not in order to withdraw their funds.

It was previously stated that MtGox user data would not be transferred to CoinLab without user agreement (ie, MtGox users accepting CoinLab's ToS), but if the funds are being transferred automatically then how will users be able to access them without using CoinLab?

I know you have a lot going on, but the transition to CoinLab has to the potential to be disastrous if you don't keep your users fully informed and - quite frankly - you don't need any more things happening right now which undermine people's confidence in you.


We, Coinlab & Mt.Gox, will announce something on this matter soon.

Mt.Gox : The Leading International Bitcoin Exchange.
Mt.Gox Merchant Solutions :   Now Available!
Mt.Gox_Alex
Member
**
Offline Offline

Activity: 112
Merit: 10



View Profile WWW
April 04, 2013, 03:55:20 AM
 #12

Prolexic sucks, they are resellers. Go with Black Lotus and make them agree to guaranteed uptime protection they'll do it. You could also clone a backup elastic cloud image on Amazon that ssh's to your secure db and switch to it whenever you get hammered beyond 10Gbps

We were using Black Lotus and runaway from them... And 10Gbps is pretty much nothing for us, we have this on weekly basis and EC2 doesn't have enough CPU/memory to handle our db on a single instance

Mt.Gox : The Leading International Bitcoin Exchange.
Mt.Gox Merchant Solutions :   Now Available!
repentance
Hero Member
*****
Offline Offline

Activity: 840
Merit: 1000


View Profile
April 04, 2013, 04:01:29 AM
 #13


We, Coinlab & Mt.Gox, will announce something on this matter soon.

I really hope so because to date both services have fallen well short of the mark in keeping users informed.

All I can say is that this is Bitcoin. I don't believe it until I see six confirmations.
Mt.Gox_Alex
Member
**
Offline Offline

Activity: 112
Merit: 10



View Profile WWW
April 04, 2013, 04:05:09 AM
 #14


We, Coinlab & Mt.Gox, will announce something on this matter soon.

I really hope so because to date both services have fallen well short of the mark in keeping users informed.

I agree with you, but let's say that the FinCEN announcement delayed a few things.

Mt.Gox : The Leading International Bitcoin Exchange.
Mt.Gox Merchant Solutions :   Now Available!
Wekkel
Legendary
*
Offline Offline

Activity: 1932
Merit: 1040


yes


View Profile
April 04, 2013, 06:39:55 AM
 #15

Thanks for updating the community  Cheesy

           ▀██▄ ▄██▀
            ▐█████▌
           ▄███▀███▄
         ▄████▄  ▀███▄
       ▄███▀ ▀██▄  ▀███▄
     ▄███▀  ▄█████▄  ▀███▄
   ▄███▀  ▄███▀ ▀███▄  ▀███▄
  ███▀  ▄████▌   ▐████▄  ▀███
 ███   ██▀  ██▄ ▄██  ▀██   ███
███   ███  ███   ███  ███   ███
███   ███   ███████   ███   ███
 ███   ███▄▄       ▄▄███   ███
  ███▄   ▀▀█████████▀▀   ▄███
   ▀████▄▄           ▄▄████▀
      ▀▀███████████████▀▀
DeepOnion
.Anonymous and Untraceable.
ANN  Whitepaper  Facebook  Twitter  Telegram  Discord 





      ▄▄██████████▄▄
    ▄███▀▀      ▀▀█▀   ▄▄
   ███▀              ▄███
  ███              ▄███▀   ▄▄
 ███▌  ▄▄▄▄      ▄███▀   ▄███
▐███  ██████   ▄███▀   ▄███▀
███▌ ███  ███▄███▀   ▄███▀
███▌ ███   ████▀   ▄███▀
███▌  ███   █▀   ▄███▀  ███
▐███   ███     ▄███▀   ███
 ███▌   ███  ▄███▀     ███
  ███    ██████▀      ███
   ███▄             ▄███
    ▀███▄▄       ▄▄███▀
      ▀▀███████████▀▀
.
MPOE-PR
Hero Member
*****
Offline Offline

Activity: 756
Merit: 500



View Profile
April 04, 2013, 10:20:32 AM
 #16

Thanks for the excellent feedback.

If this is excellent I bet nobody eats what you cook.

If you were to make the list of "events which might have killed Bitcoin", MtGox's hack is indisputably at #1, above the Pirate heist, above the recent unexpected fork, above everything else. This is MtGox's legacy: they're a historical threat to Bitcoin' continued existence and a permanent nuisance in the day to day life of it. To further illustrate this point:

Quote
Mar 04 19:15:17 blu3gr1ffon ;;goxlag
Mar 04 19:15:22 gribble 106.133771 seconds

Mar 06 04:46:15 Ukto ;;goxlag
Mar 06 04:46:15 gribble 78.10124 seconds

Mar 06 05:17:03 Chaang-Noi ;;goxlag
Mar 06 05:17:05 gribble 456.476647 seconds

Mar 06 11:41:29 dub ;;goxlag
Mar 06 11:41:46 gribble 245.37453 seconds

Mar 07 01:58:34 kakobrekla ;;goxlag
Mar 07 01:58:37 gribble 471.638897 seconds

Mar 11 09:35:00 jurov ;;goxlag
Mar 11 09:35:01 gribble 2025.557307 seconds
Mar 11 09:35:10 jurov what is the record?

Mar 12 03:53:17 dub ;;goxlag
Mar 12 03:53:18 gribble 38.016469 seconds

Mar 12 04:12:36 smickles ;;goxlag
Mar 12 04:12:36 gribble 157.044926 seconds

Mar 12 04:23:10 gesell ;;goxlag
Mar 12 04:23:11 gribble 199.519792 seconds

Let's skip to recent times:

Quote
Mar 29 00:34:07 Bitesaak ;;goxlag
Mar 29 00:34:07 gribble 288.72481 seconds

Mar 29 00:40:41 thestringpuller ;;goxlag
Mar 29 00:40:41 gribble 167.450794 seconds

It's so bad we have a special command to query it! And it's so funny people have taken to embellishing it:

Quote
Apr 03 09:24:23 ThickAsThieves ;;goxlag
Apr 03 09:24:24 gribble MtGox lag is 219.524796 seconds. During this time, light travels 0.439925233262 AU. You could have sent a bitcoin from the Sun to Mercury (0.39 AU).

Apr 03 11:50:16 dub ;;goxlag
Apr 03 11:50:16 gribble MtGox lag is 468.536678 seconds. During this time, light travels 0.938942256714 AU. You could have sent a bitcoin from the Sun to Earth (1 AU).

Apr 03 17:52:42 TomServo ;;goxlag
Apr 03 17:52:42 gribble MtGox lag is 6048.679827 seconds. During this time, light travels 12.1214866489 AU. You could have sent a bitcoin from the Sun to Saturn (9.54 AU).

Apr 03 18:03:07 thestringpuller ;;goxlag
Apr 03 18:03:07 gribble MtGox lag is 5455.751923 seconds. During this time, light travels 10.933265768 AU. You could have sent a bitcoin from the Sun to Saturn (9.54 AU).

The notion that you're running an exchange with multisecond lag is ridiculous on its face. I'm not even sure why this has to be spelled out, it's beyond ridiculous. It's like Monty Python's cheese shop, it's like Monty Python's "self defense classes", it's like a comedy routine. This is what we use you for, MtGox, comedic relief. You're not an exchange, okay? You're Bitcoin's very own Comedy Central.

So now, armed with this basic understanding of what's what and where we're standing, let us dissect MtGox's most recent load of bullcrap (continued).

My Credentials  | THE BTC Stock Exchange | I have my very own anthology! | Use bitcointa.lk, it's like this one but better.
Zooey
Sr. Member
****
Offline Offline

Activity: 241
Merit: 250


Time you enjoy wasting is not wasted time.


View Profile
April 04, 2013, 12:16:50 PM
 #17


Mt.Gox has been suffering from its worst trading lag ever, 502 errors, and at one point some users were not able to log in their account.


Have to admit, I too am a sucker for folks who fess up to their failings, so props to you for doing that head-on without any crap; it affords you a level of credibility that ought to be a sober example for others who lack it in lumps, like the bullshitting BFL brigade.


✘ www.NOTFORSALECAMPAIGN.org ✘
Human trafficking enslaves 30 million people: Join the movement to re-abolish slavery. | A 05:49s Vimeo |
Isokivi
Hero Member
*****
Offline Offline

Activity: 910
Merit: 1000


Items flashing here available at btctrinkets.com


View Profile WWW
April 04, 2013, 01:01:22 PM
 #18

One thing Mt.Gox could do is release what information they have on the attack(s). We all know it's a botnet of infected computers, but I would not count out the bitcoin community out from doing detective work and more.

Bitcoin trinkets now on my online store: btc trinkets.com <- Bitcoin Tiepins, cufflinks, lapel pins, keychains, card holders and challenge coins.
DougTanner
Sr. Member
****
Offline Offline

Activity: 280
Merit: 250



View Profile
April 04, 2013, 01:10:19 PM
 #19

I didn't see anything in there about adjusting the trade fee schedule.

It used to be < $160,000 to get to 0.3% (and probably a lot lower), now it's upwards of $1,370,000, a factor of ten. When will you be adjusting it to make it in line with current prices?

Help pay for my BTCCharts.com subscription: 1DTannerQHQD7qjSmA3abLQaDkVFLnBGma
casascius
Mike Caldwell
VIP
Legendary
*
Offline Offline

Activity: 1386
Merit: 1039


The Casascius 1oz 10BTC Silver Round (w/ Gold B)


View Profile WWW
April 04, 2013, 01:11:40 PM
 #20

Also MtGox could take a position on my UDP streams idea, which could be any of the following without commitment:

a) Great idea, we haven't thought of it, and you're right, it would totally get information out immune to DDoS, we'll consider it but like anything else will take time
b) Great idea, but we don't agree it would work as well as you think it will, or for (specific technical reason) won't work on our platform
c) We haven't got a clue as to what this means
d) I don't have a clue what this means because I'm not a developer or tech guy myself, but I have relayed your suggestion to someone more technical, and he says (response).  (Hopefully this suggestion is more valuable than to merely forward it blindly like the latest facebook meme, since MtGox's reputation is suffering and this will actually solve the claimed issue at hand)

Just to be clear, using UDP to broadcast ticker data would be, for all intents and purposes, IMMUNE from DDoS attacks, because such a stream consists solely of outbound traffic which is not influenced by inbound traffic.  Unlike a normal stream, there is no inbound overhead for packets to acknowledge or to keep the connection in sync, packets which can be drowned out in a DDoS attack.  UDP is much more like a point-to-point radio broadcast: the signal gets sent from point A to B even if nobody's listening

Companies claiming they got hacked and lost your coins sounds like fraud so perfect it could be called fashionable.  I never believe them.  If I ever experience the misfortune of a real intrusion, I declare I have been honest about the way I have managed the keys in Casascius Coins.  I maintain no ability to recover or reproduce the keys, not even under limitless duress or total intrusion.  Remember that trusting strangers with your coins without any recourse is, as a matter of principle, not a best practice.  Don't keep coins online. Use paper or hardware wallets instead.
Pages: [1] 2 3 »  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!