A Possible Explanation on the BTC Withdrawal Delays at MtGox

[thecomputerscientist]

Myself, and many others I know, know that BTC withdrawals from MtGox can get stuck. It appears to be quite random. You can withdraw 10 BTC just fine (they immediately show up in blockchain) and when you withdraw another 10 BTC they get stuck in limbo. So what the heck is going on?  Well, I _think_ I know the answer, but it is going to be quite technical.

Most Bitcoin exchange software is based on the original software (Github bitcoin.org). I bet this includes MtGox. It is quite complicated to write a client from scratch and whenever changes are made on the main trunk, you want to be able to merge those changes into your own custom branch. This is the reason why most people use the original client as a starting point.

The first thing you need to do with the original client is to replace the wallet code. That code isn't well suited for a full blown exchange/wallet because it uses a rather primitive database. Most companies want to use SQL or MongoDB to store their wallets in. Therefore, you basically replace most of the code in wallet.cpp with a custom version.

In the original client the wallet code is quite robust. All the transactions to/from the wallet are guarded by mutex code so that no race conditions can occur. For those with programming skills can take a look at CWallet::CreateTransaction and see "LOCK2(cs_main, cs_wallet)". However, the original client is quite complicated when it comes to select which outputs from transactions to use as funds for a transaction. This happens in a sub-function called SelectCoins() which stochastically choose random transaction outputs until the desired sum is reached.

When changing this code base to use a real database it is extremely important that that the operation SelectCoins() is atomic, or otherwise a race condition will occur and there’s a danger that two or more persons making BTC withdrawals at the same time will end up using the same transaction outputs.

Fortunately, the blockchain is all about preventing double spending, which means that only the first transaction will go through and the rest will be stuck in limbo. The only way to resolve this is to cancel those transactions, which MtGox presumably does manually. Once the transactions are cancelled, they will retry transferring the funds, but the same problem can happen again, which implies that they may have to cancel those transactions as well, etc.

However, if many are doing withdrawals, the greater the chance for these race conditions. MtGox MUST solve this problem ASAP or the technical support tickets being created will go to the roof and the user confidence will quickly evaporate into quantum foam.

Are there any bitcoin developers out there that could chime in on my theory?

UPDATE: It is likely that these race conditions started to occur when they upgraded hardware from one server to many for load balancing. (This happened approximately one year ago.)

[1715317026]

1715317026

[1715317026]

1715317026

[DieJohnny]

MtGox MUST solve this problem ASAP or the technical support tickets being created will go to the roof and the user confidence will quickly evaporate into quantum foam.

Did you mean to post this topic last February???

[thecomputerscientist]

Did you mean to post this topic last February???

What do you mean?

I only started to look at this problem when it happened to myself (and some of my friends).
When I did some research I saw that others had this problem as well, so I started to dig deeper into this.

The problem is much worse now than in February, because the BTC outflow from MtGox is much higher now, so the problem is very frequent. Most likely, it happens to all MtGox users (once in a while).

[mailaddy]

I suggested it sounded like a race condition several days ago in https://bitcointalk.org/index.php?topic=429769.msg4738455#msg4738455 - this was dropped into #mtgox and also sent to MagicalTux, but nobody got in touch.

Obviously we have no idea how their code is working though, and i would think that they know about race conditions and how to fix them anyway, because their trade matching engine needs to have locking.

[thecomputerscientist]

I suggested it sounded like a race condition several days ago in https://bitcointalk.org/index.php?topic=429769.msg4738455#msg4738455 - this was dropped into #mtgox and also sent to MagicalTux, but nobody got in touch.

Obviously we have no idea how their code is working though, and i would think that they know about race conditions and how to fix them anyway, because their trade matching engine needs to have locking.

Ah, at least someone else observing the same thing with the same conclusion! Thanks for notifying! I'm pretty convinced it has to be a race condition in their code and it is most likely the SelectCoins() function that is not properly guarded.

[mailaddy]

Yes but others have said they think it's related to dust transactions and TX sizes.  I don't understand how it would take more than a day to find the problem and resolve it, whatever it is.  I was affected by this at the end of last year also, but the transaction was resent and made good within 12 hours.. it is now 6 days for this one and nothing.  I directly lost probably $2k because of this.

[tvbcof]

Looks like you guys were not around when Mt.Gox tossed some bunch of BTC into the ether never to be seen again via a fuck-up to the scripting engine.  In short, it's not big news that Mark (or someone of questionable competence) has puttered around with the code.

I never ran across any convincing arguments of what Mt.Gox was trying to achieve with their re-write, but also never really looked that hard.  Could have been just a fun side project and one which the implementer doesn't want to let go of.

I was also wondering recently if there could be some sort of an exploit/inside job against Gox's stash which manifested itself as a lot of double-spends.  Or similarly, if strategic double-spends in conjunction with an information stream to mining pool might be a way to steer mining profits toward a co-conspirator.

[thecomputerscientist]

Looks like you guys were not around when Mt.Gox tossed some bunch of BTC into the ether never to be seen again via a fuck-up to the scripting engine.  In short, it's not big news that Mark (or someone of questionable competence) has puttered around with the code.

I never ran across any convincing arguments of what Mt.Gox was trying to achieve with their re-write, but also never really looked that hard.  Could have been just a fun side project and one which the implementer doesn't want to let go of.

I was also wondering recently if there could be some sort of an exploit/inside job against Gox's stash which manifested itself as a lot of double-spends.  Or similarly, if strategic double-spends in conjunction with an information stream to mining pool might be a way to steer mining profits toward a co-conspirator.

No, I was not around at that time. I joined Bitcoin around January 2013 (although I read about in 2011). If they implemented the scripting instructions themselves (and how they are assembled for transactions), it is likely that MtGox has written the entire client from scratch. It seems rather complicated though, because you would like to merge with the latest changes from the official client, especially when there are hard forks.

Trying to exploit double spends is a dead end though. I doubt that the double spends are intentional.

[tvbcof]

If they implemented the scripting instructions themselves (and how they are assembled for transactions), it is likely that MtGox has written the entire client from scratch. It seems rather complicated though, because you would like to merge with the latest changes from the official client, especially when there are hard forks.

I'm not so sure it would be that problematic to perform the necessary merging.  The scripts should be largely forward compatible, and if one is just running some core business function a lot of the extensions would be unnecessary.  IOW, I would think that a simple script engine would require very few updates, and they would be of the type which would be fairly straightforward to add for the guy who did the initial implementation.

Again, it would be interesting to know what Mark (or whoever) was trying to achieve...but not actually interesting enough to spend much time trying to find out.  Back before Mark was on my shit-list for ripping me off for $5k, I though perhaps he was trying to do some transaction optimizations to benefit the network.  There is no reason why him ripping me off should change my estimate of Mt. Gox's care about the network health, but I cannot help it.

[xorred]

same situation here. Hash and address return 0 in blockchain. this sucks, these were ALL my coins!!!!!!!!!!!

[skyrunner]

So any conclusion? Is Gox fucking with us or there might be a technical problem

[tvbcof]

...
I was also wondering recently if there could be some sort of an exploit/inside job against Gox's stash which manifested itself as a lot of double-spends.  ...

I ponder no longer.

I had imagined that Gox was taking their own database as a source of truth for accounting purposes and that was probably at the root of the problem.  This seems to be true.

Looks like it probably was not an inside job as I had hypothesized although the potential exists for insiders to have been re-issue payments even after understanding that it would cause a loss.  That's not necessary to explain the observations though.

It would be fun to know how much Mt. Gox (and probably others) lost to this simplistic form of attack.

One way or another, it is pretty sleazy for Karpeles to try to blame the issue on the Bitcoin protocol when they tried to write a processor client without even troubling themselves to ponder the information described on the wiki page.