Bitcoin Forum

Bitcoin => Development & Technical Discussion => Topic started by: Stammer on February 26, 2014, 05:52:37 AM



Title: Mt.Gox technical autopsy
Post by: Stammer on February 26, 2014, 05:52:37 AM
After browsing the threads about Mt.Gox and the malleability issue here are a couple of questions.

Q1. Is there any way to ascertain that the Mt.Gox bankrupcy was indeed induced by transaction malleability?

I understand that the malleability issue is real. Indeed it has been real and known for more than two years. I still fail to understand how it could be realistically harnessed to heist hundreds  of thousands of bitcoins.

Q2. Are we sure that the whole thing wasn't orchestrated with the complicity of Mt.Gox management? Can we ever be sure?

Q3. Consider a Bitcoin maketplace model where major exchanges  mysteriously go bankrupt and handwave their alleged losses to some technical issue, previously regarded as minor.  Is such a model viable in the long tem?


Title: Re: Mt.Gox technical autopsy
Post by: Cor2 on February 26, 2014, 09:13:34 AM
After browsing the threads about Mt.Gox and the malleability issue here are a couple of questions.

Q1. Is there any way to ascertain that the Mt.Gox bankrupcy was indeed induced by transaction malleability?

I understand that the malleability issue is real. Indeed it has been real and known for more than two years. I still fail to understand how it could be realistically harnessed to heist hundreds  of thousands of bitcoins.

Q2. Are we sure that the whole thing wasn't orchestrated with the complicity of Mt.Gox management? Can we ever be sure?

Q3. Consider a Bitcoin maketplace model where major exchanges  mysteriously go bankrupt and handwave their alleged losses to some technical issue, previously regarded as minor.  Is such a model viable in the long tem?
How do you answer rhetorical questions?...


Title: Re: Mt.Gox technical autopsy
Post by: Stammer on February 26, 2014, 10:08:36 AM
Quote
How do you answer rhetorical questions?..

I doubt my questions are rhetorical. Let me sketch some tentative answers, although I would obviously prefer to hear something from outside my head

A:Q1+Q2. That largely depends on Mt.Gox. If they release all the data they hold, including all their code and records, we may get an idea of what happened, which techniques were used and how transaction malleability was used in the heist.

I personally believe that insider support must have  played a role, but I am less sanguine about wilful involvement of top management. Anyways, IMO there are lessons to be learnt here, both at the technical and at the management level.

A:Q3. Probably not, although some amount fraud may be tolerable. In the current ecosystem, as far as I understand, major exchanges are trust repositories. if trust repositories are necessary, then they should be fully accountable entities. But perhaps trust repositories are not necessary and the Bitcoin ecosystem should move away from them.


Title: Re: Mt.Gox technical autopsy
Post by: behindtext on February 26, 2014, 01:39:25 PM
After browsing the threads about Mt.Gox and the malleability issue here are a couple of questions.

Q1. Is there any way to ascertain that the Mt.Gox bankrupcy was indeed induced by transaction malleability?

I understand that the malleability issue is real. Indeed it has been real and known for more than two years. I still fail to understand how it could be realistically harnessed to heist hundreds  of thousands of bitcoins.

i am not normally one to speak positively of regulation, but this is one of several reasons that money transmitters and MSBs require licenses in the US: too many unscrupulous people have started up such businesses, then there is a "theft" of funds and all the customers are missing money.

there are numerous ways a tx malleability exploit should have been stopped at mtgox. incompetence is a decent defense in the BTC markets when you consider how little most people know about BTC.

Quote
Q2. Are we sure that the whole thing wasn't orchestrated with the complicity of Mt.Gox management? Can we ever be sure?

Q3. Consider a Bitcoin maketplace model where major exchanges  mysteriously go bankrupt and handwave their alleged losses to some technical issue, previously regarded as minor.  Is such a model viable in the long tem?

if BTC 600K or more has been "stolen", which has not been reliably confirmed by any means, i would bet a large amount of money on it being an inside job. you don't just end up having your cold wallets emptied and not know something is up, even if you're as incompetent as MK.


Title: Re: Mt.Gox technical autopsy
Post by: Loozik on February 26, 2014, 06:12:32 PM
After browsing the threads about Mt.Gox and the malleability issue here are a couple of questions.

Q1. Is there any way to ascertain that the Mt.Gox bankrupcy was indeed induced by transaction malleability?

Let's say person X (attacker) withdrew e.g. BTC 666.696969 from Gox (or any other exchange). The same person X needed to claim exactly the same amount (BTC 666.696969) from Gox a week or two weeks later, right?

What could be done is to run a query on blockchain data to identify such transaction pairs, initiated from addresses that once had a fairly high value of ''total received'' (indicating they were exchange address) and sent the same amount (BTC 666.696969) twice within a certain period of time.

If someone identifies such pairs, then we might at least get the idea of the maximum possible malleability threshold that went on the Bitcoin network.


Q2. Are we sure that the whole thing wasn't orchestrated with the complicity of Mt.Gox management? Can we ever be sure?

We are not sure and cannot be sure at the moment. The malleability can very well be a real thing or just an excuse. Someone brainy needs to investigate this through blockchain data analysis.


Title: Re: Mt.Gox technical autopsy
Post by: npiguet on February 26, 2014, 08:01:18 PM
Let's say person X (attacker) withdrew e.g. BTC 666.696969 from Gox (or any other exchange). The same person X needed to claim exactly the same amount (BTC 666.696969) from Gox a week or two weeks later, right?

What could be done is to run a query on blockchain data to identify such transaction pairs, initiated from addresses that once had a fairly high value of ''total received'' (indicating they were exchange address) and sent the same amount (BTC 666.696969) twice within a certain period of time.

If someone identifies such pairs, then we might at least get the idea of the maximum possible malleability threshold that went on the Bitcoin network.

You couldn't do that, because only one of the transactions would actually be in the block chain. So you would never actually see the pair of transactions. You can only catch this stuff while neither of the equivalent transactions has entered a valid block.


Title: Re: Mt.Gox technical autopsy
Post by: roy7 on February 26, 2014, 08:04:52 PM
i am not normally one to speak positively of regulation, but this is one of several reasons that money transmitters and MSBs require licenses in the US: too many unscrupulous people have started up such businesses, then there is a "theft" of funds and all the customers are missing money.

I'm not either and because of the regulations I would never set up my own exchange of any sort. However, I do benefit from it by only using coinbase for my transactions since they do follow the US regulations and are a US company. I feel much more comfortable doing bank transfers to coinbase for BTC purchases that I would with other random exchanges (be it BTC-E, Cryptsy, Coinex, or someone else). Also getting cash back from them by selling BTC is easy and direct to my bank account.


Title: Re: Mt.Gox technical autopsy
Post by: cr1776 on February 26, 2014, 08:10:29 PM
Let's say person X (attacker) withdrew e.g. BTC 666.696969 from Gox (or any other exchange). The same person X needed to claim exactly the same amount (BTC 666.696969) from Gox a week or two weeks later, right?

What could be done is to run a query on blockchain data to identify such transaction pairs, initiated from addresses that once had a fairly high value of ''total received'' (indicating they were exchange address) and sent the same amount (BTC 666.696969) twice within a certain period of time.

If someone identifies such pairs, then we might at least get the idea of the maximum possible malleability threshold that went on the Bitcoin network.

You couldn't do that, because only one of the transactions would actually be in the block chain. So you would never actually see the pair of transactions. You can only catch this stuff while neither of the equivalent transactions has entered a valid block.

Both transactions would be in the blockchain - the initial withdrawal and then the re-withdrawl request several weeks later which mtgox would have allowed.  This is different than the initial gox withdrawal which had the tx id mutated and eventually made it into the blockchain albeit with this mutated tx id.





Title: Re: Mt.Gox technical autopsy
Post by: Loozik on February 26, 2014, 08:44:22 PM
You couldn't do that, because only one of the transactions would actually be in the block chain.

That would mean that valid transactions do not appear on blockchain - I consider this rubbish.


Both transactions would be in the blockchain - the initial withdrawal and then the re-withdrawl request several weeks later which mtgox would have allowed.

I agree with this.

One needs to pair those two transactions (find them in the blockchain). There would be a few non-complex criteria for finding such malleable-looking transactions:
- amount of both must match
- date range should not be broad, e.g. the second transaction cannot be older than 1 day / week / month than the first one
- both transactions must be spent from an exchange address (not sure how to find such addresses but my guess is that such addresses generated a huge number of transactions for a short time)
- others

------------------------------

It is a perfect research project for applicants who want grants from TBF.


Title: Re: Mt.Gox technical autopsy
Post by: kjj on February 27, 2014, 03:09:45 AM
After browsing the threads about Mt.Gox and the malleability issue here are a couple of questions.

Q1. Is there any way to ascertain that the Mt.Gox bankrupcy was indeed induced by transaction malleability?

Let's say person X (attacker) withdrew e.g. BTC 666.696969 from Gox (or any other exchange). The same person X needed to claim exactly the same amount (BTC 666.696969) from Gox a week or two weeks later, right?

What could be done is to run a query on blockchain data to identify such transaction pairs, initiated from addresses that once had a fairly high value of ''total received'' (indicating they were exchange address) and sent the same amount (BTC 666.696969) twice within a certain period of time.

If someone identifies such pairs, then we might at least get the idea of the maximum possible malleability threshold that went on the Bitcoin network.

This assumes that they were replacing transactions, not balances.  This assumption may not be valid.


Title: Re: Mt.Gox technical autopsy
Post by: Loozik on February 27, 2014, 03:53:07 AM
After browsing the threads about Mt.Gox and the malleability issue here are a couple of questions.

Q1. Is there any way to ascertain that the Mt.Gox bankrupcy was indeed induced by transaction malleability?

Let's say person X (attacker) withdrew e.g. BTC 666.696969 from Gox (or any other exchange). The same person X needed to claim exactly the same amount (BTC 666.696969) from Gox a week or two weeks later, right?

What could be done is to run a query on blockchain data to identify such transaction pairs, initiated from addresses that once had a fairly high value of ''total received'' (indicating they were exchange address) and sent the same amount (BTC 666.696969) twice within a certain period of time.

If someone identifies such pairs, then we might at least get the idea of the maximum possible malleability threshold that went on the Bitcoin network.

This assumes that they were replacing transactions, not balances.  This assumption may not be valid.

Okay, this assumption may not be valid, but at least we might start to try assessing the damage done to exchanges from here.

I am not the expert on bitcoin. I think a geek should conduct a study on this subject to arrive at a methodology of assessing the scale of malleability exploit.

If a good methodology is devised and a proper study run, someone might at least be able to confirm or debunk claims that were written in the ''leaked'' ''document'' that says +700k coins are gone.


Title: Re: Mt.Gox technical autopsy
Post by: Tonka Branded Truck on February 27, 2014, 05:23:01 AM
Malleability exploit uses double spend right? Why not mtgox software cannot detect double spend? I know cryptsy software will automatically freezes the trade and withdrawal once the double spend is detected on the blockchain. I don't know why mtgox did not develop this since this issue is known two years ago. They  have millions of money they can hire the brightest programmers available. Or go blind about it and make it as a scapegoat.


Title: Re: Mt.Gox technical autopsy
Post by: Stammer on February 27, 2014, 06:14:45 AM
After browsing the threads about Mt.Gox and the malleability issue here are a couple of questions.

Q1. Is there any way to ascertain that the Mt.Gox bankrupcy was indeed induced by transaction malleability?

Let's say person X (attacker) withdrew e.g. BTC 666.696969 from Gox (or any other exchange). The same person X needed to claim exactly the same amount (BTC 666.696969) from Gox a week or two weeks later, right?

What could be done is to run a query on blockchain data to identify such transaction pairs, initiated from addresses that once had a fairly high value of ''total received'' (indicating they were exchange address) and sent the same amount (BTC 666.696969) twice within a certain period of time.

If someone identifies such pairs, then we might at least get the idea of the maximum possible malleability threshold that went on the Bitcoin network.

This assumes that they were replacing transactions, not balances.  This assumption may not be valid.

I would like to read more about this. AFAIU transaction malleability is still an open wound in the Bitcoin protocol. Understanding its implications is important.

I found this thread (https://bitcointalk.org/index.php?topic=471705.0) , Maged's post  about the Mt.Gox mess in particular, quite instructive.

By the way, Maged writes:

...
This means that it's not possible for both transactions to be in the blockchain, so people/organizations affected by this won't have to go through the blockchain to find people who double-withdrew, right?
Yes. Only one can ultimately exist in the blockchain.


Title: Re: Mt.Gox technical autopsy
Post by: kjj on February 27, 2014, 12:23:24 PM
I would like to read more about this. AFAIU transaction malleability is still an open wound in the Bitcoin protocol. Understanding its implications is important.

I found this thread (https://bitcointalk.org/index.php?topic=471705.0) , Maged's post  about the Mt.Gox mess in particular, quite instructive.

By the way, Maged writes:

...
This means that it's not possible for both transactions to be in the blockchain, so people/organizations affected by this won't have to go through the blockchain to find people who double-withdrew, right?
Yes. Only one can ultimately exist in the blockchain.

He mentions that SR2 was a scam using this as cover.  It is quite possible that gox is the same.

This is my understanding, the best that I've been able to put together.

Gox's custom software doesn't always strip leading zero bytes from signature values.  Sipa estimated (on IRC) that only ~1% of gox transactions had excessive padding.

A while back, this was no big deal because the network would happily relay both forms, so any attempt at changing the transaction would result in a race.  Later, the default node behavior changed to not relay the padded version.  Once this changed, the original would pretty much always lose the race, if there was one.

Next, someone made a bot to "fix" transactions on the fly.  Quite possibly this person was sick of waiting for their money and/or was just being helpful.

So, we have 3 eras.

First, all transactions confirm normally.
Second, 1% of transactions never confirm.
Third, 1% of transactions confirm in modified form.

Only the third era is vulnerable, but it is a relatively short era.  I don't think anyone knows exactly how long it has been going on, but it can't have been years because the second era isn't that old either.

Now, to exploit this, you'd need to get lucky, or you'd need to keep up a massive circular flow out of and then back into gox.  Remember that each transaction has about a 1% chance.  So, 99% of the time that you withdraw your balance, it works fine.  And you have to wait 6 or 7 confirmations (minimum) before you can try again.  You'd get about 85 chances per year, so if we assume the third era was about a year long, figure the average attacker could have doubled their money about once in that time.

It just doesn't add up.  Someone is lying, or there are very important things that we don't know yet.


Title: Re: Mt.Gox technical autopsy
Post by: Loozik on February 27, 2014, 12:45:45 PM
kjj,

Thanks for your thoughts. I can't fully understand them (I have a language barrier and no knowledge of bitcoin mechanics / technicalities / jargon). But at least you came up with some grounds for methodology of calculating / estimating the level of malleability exploit used to trick exchanges.

I have funds at Gox (or ''had'' in case they are gone). I really want to find out what is going on / what was going on. While trying to find it out I want NOT to rely on leaked recovery plans (which may or may not represent the reality).

Any help will be appreciated that will lead to discovery of the amount that was extracted from exchanges through malleability bug.


Title: Re: Mt.Gox technical autopsy
Post by: mistfpga on February 27, 2014, 05:01:07 PM
Hi,

This is my understanding of how bitcoin works. If i am misunderstanding how transactions work, please someone correct me. Also this is only about mtgox... not how malleability works in general.

I would like to read more about this. AFAIU transaction malleability is still an open wound in the Bitcoin protocol. Understanding its implications is important.

I agree that more users should be fully aware as to the implications of malleability, but they are documented and somewhat limited. Developers have no excuse for not being fully aware.

I think "an open wound" is a bit excessive. this is not bitcoins fault. bitcoin gives you the rope to hang yourself, it is up to you if you do that or not.  irreversible transactions and  accidentally large transaction fees are other examples of things that could be seen by some as a 'bug' or flaw in the bitcoin code. they are not.

Trusting unsigned inputs (like transaction ID, whilst not directly controllable it is certainly modifiable.) as signed inputs is such a basic mistake - I mean really really basic.  like not pulling your trousers and underwear down before you go for a shit.

Block chain bloat, overly complex legacy that needs to be supported, almost obfuscated reference code and a very steep learning curve are open wounds. (that are healing, slowly...)


...
This means that it's not possible for both transactions to be in the blockchain, so people/organizations affected by this won't have to go through the blockchain to find people who double-withdrew, right?
Yes. Only one can ultimately exist in the blockchain.
[/quote]

Yeah, but that is missing the point, there has to be 3 transactions for gox's story to hold true (even though two of them spend the same funds/are the same transaction)
- the initial transaction,
- the double spend (which must be mined before transaction 1.)
- the reissue of funds in transaction 1 by the original issuer.  it is this reissue that would set alarm bells off. (and why not reissue the _same_ funds they claim not to have received? maybe you'd have to run a multimillion dollar business and have a track record for successful technical ventures to check for that sort of thing.)

makes me wonder why the accountants did not flag the discrepancies between the sale of coins, transaction reissues and actual stock levels. they must have been doing zero stock checking for their story to hold true. (or the 'attack' was spotted very quickly) This is not even a technical thing, it is a bean counter thing. (the stock checks will not show what the issue is, just their is a major problem... _everything_ is accountable it is all right there in the block chain - bitcoin is an accountants wet dream.)

Quote
He mentions that SR2 was a scam using this as cover.  It is quite possible that gox is the same.

Not in the thread you linked, he even specifically states they are not connected. interesting idea though. (you would probably make more money just selling drugs...)

Quote
Depends on the "issue" you are referring to. You see, in the past week, the three major stories you've heard in the media (MtGox, SR 2.0, DDoS) were all unrelated.

Quote
Gox's custom software doesn't always strip leading zero bytes from signature values.  Sipa estimated (on IRC) that only ~1% of gox transactions had excessive padding.

I think the not relaying of transactions is a distraction technique. same for the unsticking of transactions, anyone could verbatim supply the transaction to a pool they know will mine it.  Mining rules are different to passing of transaction rules.  There is no reason not to mine nonstandard transactions if you see them.

Quote
A while back, this was no big deal because the network would happily relay both forms, so any attempt at changing the transaction would result in a race.  Later, the default node behavior changed to not relay the padded version.  Once this changed, the original would pretty much always lose the race, if there was one.

maybe, however even now it shouldnt be a big deal. gox's custom code would relay the transactions, so gox would have to be not connected to the major mining pools... this seems absurd at best. the padded transactions will be included in the next block. there is no reason not to mine them, just not to transmit them.

The best an attacker could hope for would be to connect to the same pools, but they are still massively disadvantaged because they have to see the transaction second, modify it, and get it into the transaction pool ahead of the transaction they got from the transaction pool...  for gox's account to be true the attacker would have to see the broadcast before the pools. That is until anyone can up the fees on a transaction... but that is a different can of worms. (and a really bad idea.)

So the attacker is left with having to see the transaction in memory at a major pool then modify it and pass it to another pool that gox is not connected too and hope that pool hits a block first. Unless the pools decide not to mine nonstandard transactions (not rational)

gox would never have a transaction propagation issue no matter how nonstandard yet valid their transactions are.  However this changes when we get the ability to pay for fees on other peoples transactions, making the attack easier. (assuming higher fees are more likely to appear in blocks)

Quote
Next, someone made a bot to "fix" transactions on the fly.  Quite possibly this person was sick of waiting for their money and/or was just being helpful.

I would be tempted to think this was Gox implementing a 'fix'.  A good willed person does not need to modify the transaction to be helpful, just relay it to the pools that gox doesnt talk too (okay if you arent up on bitcoin attacks, unless gox hasnt heard of a sybil attack https://en.bitcoin.it/wiki/Weaknesses they are connected to well over 75% of the mining power, hence me banging on about it.  if 1% have an issue, it is an infinitesimally small chance you will be able to out race it.) you dont need to strip the zeros.

Quote
So, we have 3 eras.

First, all transactions confirm normally.
Second, 1% of transactions never confirm.
Third, 1% of transactions confirm in modified form.

Only the third era is vulnerable, but it is a relatively short era.  I don't think anyone knows exactly how long it has been going on, but it can't have been years because the second era isn't that old either.

I am not sure what you mean by era's but the issue isnt as passive as it seems. there is a difference between the rules for relaying transactions and the rules for including a transaction in a block. the attack relies on the attacker being able to send the modified transaction to the mining pools before they see the original transaction. the network propagation rules only apply to people without custom software to craft the transactions and direct connections to the major pools. technically it could have been going on the whole time.  the original transaction most likely never made it to the network if they really were attacked.

Quote
Now, to exploit this, you'd need to get lucky, or you'd need to keep up a massive circular flow out of and then back into gox.  Remember that each transaction has about a 1% chance.  So, 99% of the time that you withdraw your balance, it works fine.  And you have to wait 6 or 7 confirmations (minimum) before you can try again.  You'd get about 85 chances per year, so if we assume the third era was about a year long, figure the average attacker could have doubled their money about once in that time.

It just doesn't add up.  Someone is lying, or there are very important things that we don't know yet.

I agree completely. the story as told (have gox given an official response? I saw gesticulating and general flapping on their blog but nothing concrete.) cannot be possible.

There is a lot less than 1% chance. so much so it puts it out of the hands of even medium players.

There are other methods to take advantage of the coding error that I can think of, but they all need more access...  the only semi reasonable answer is either someone
1) MITM the connections to the major pools and rewrote every transaction, and dropping the original ones thus guaranteeing your transaction is seen first.
2) Own the transaction server, rewriting every transaction and dropping the original before it is even broadcast.

I cant wait for them to have thier solution implemented (it would seem that they have passed the trust to block explorer and block explorer say yay or nay. lol. idiots.)

Hope this helps? sorry if it goes on a bit.  but as far as I know you cannot 'update' a transaction.


Title: Re: Mt.Gox technical autopsy
Post by: Stammer on February 28, 2014, 04:58:39 AM
Compare

You couldn't do that, because only one of the transactions would actually be in the block chain.

That would mean that valid transactions do not appear on blockchain - I consider this rubbish.


Both transactions would be in the blockchain - the initial withdrawal and then the re-withdrawl request several weeks later which mtgox would have allowed.

I agree with this.

One needs to pair those two transactions (find them in the blockchain). ...

and

...
This means that it's not possible for both transactions to be in the blockchain, so people/organizations affected by this won't have to go through the blockchain to find people who double-withdrew, right?
Yes. Only one can ultimately exist in the blockchain.

My understanding is that indeed only one transaction gets registered in the blockchain, but there seem to be different takes on this issue.


Title: Re: Mt.Gox technical autopsy
Post by: kjj on February 28, 2014, 06:07:53 AM
Oh, right.  I forgot about another confounding factor.

Apparently gox's software didn't properly age coins before spending them.  Not normally a problem, but when the hot wallet is low, things can get interesting.


Title: Re: Mt.Gox technical autopsy
Post by: Stammer on February 28, 2014, 07:49:13 AM
Surmising a brief sum up.

a) Only one transaction gets registered in the blockchain -> no blockchain-based autopsy is possible.

b) There have been some  interesting contributions discussing the Gox software. I wonder whether it is available and can be inspected. Perhaps it would be reasonable to require exchanges to make their software available to public scrutiny. If a chunk of the Bitcoin ecosystem is a black hole it's hardly surprising that it swallows money.


Title: Re: Mt.Gox technical autopsy
Post by: nanonano on February 28, 2014, 12:08:12 PM
My understanding is that indeed only one transaction gets registered in the blockchain, but there seem to be different takes on this issue.

This is incorrect. Maged is describing what happens to competent exchanges with the malleability bug in the quote -- and it seems MtGox was not one of those.

The claimed explanation for MtGox was:
* MtGox sends a transaction
* attacker modifies the txid
* modified transaction goes in the blockchain, and bitcoin ownership changes
* attacker complains MtGox "I didn't receive my coins"
* MtGox searches for original txid in blockchain, doesn't find it  (<-- this is the bug)
* MtGox thinks they haven't paid, apologizes and sends a second, totally separate transaction of the same value

So there will be two transactions in the blockchain (Maged even says so in his post "Under no circumstance should you just issue a completely new transaction like MtGox did. That is how they lost some bitcoins"). Identifying the pairs could be very difficult as the sending and receiving addresses might be different.


Title: Re: Mt.Gox technical autopsy
Post by: jamal26 on February 28, 2014, 01:58:09 PM
Hello, this is a topic I've been wondering about.  Please excuse me if my knowledge of bitcoin transaction details are a not up to speed at this point.

The thing that keeps sticking in my mind is how Gox went for 2 years without people noticing and screaming about it.  If Gox balances didn't agree with legit transactions balances, how do that go unnoticed?  First of all there are 2 different situations that are relevant to this issue.

1. Individual depositors transactions appear on the block chain.
2. Gox aggregated individual depositors transactions so they really only have a commitment from Gox that their bitcoins are real.

From what I've heard #2 is the way it works, and that other "exchanges" work the same way.  I find this very much against the bitcoin philosophy of a redundant trust system.  You're trusting Gox, not the redundant block chain, and maybe that's the fundamental problem with it.

But let's look at the idea that a long term attack could have gone unnoticed under these 2 different conditions.

1. Individual depositors would have recognized right away that their bitcoins were missing from their wallets if they saw they didn't appear correctly on the block chain.  Very hard to believe that they wouldn't.

2. Gox reconciliations should have picked up missing bitcoins from the block chain.  Did they not do this simple reconciliation?  This is also very hard to believe, but since it's based on inept management from a dysfunctional organization, it's easier to believe than #1.

My view on this is that the ecosystem is immature at this point and is suffering from it being primarily a fiat investment device.  People want non-sustainable things from it like excessive returns and fiat anonymity which makes people want to use exchanges and become vulnerable to trust problems with the exchange.

Another issue I have is the size of the block chain and the fact that it's not distributed, as fiat money is.  Think about if there were a block chain for dollars or euros.  It would be absolutely unmanageable and a huge breach of privacy.  These are issues that are fundamental to the viability of the bitcoin model if you ask me.


Title: Re: Mt.Gox technical autopsy
Post by: bitserve on February 28, 2014, 03:10:54 PM
Very interesting explanations about the malleability "issue" and its limited practical impact on mtgox assets.

The "funny" thing here is that MK has deluded himself into thinking that the "It's gone" will suffice as an explanation without offering any proofs in an environment so full of them.

Sooner or later he will have to provide internal accounting records and then it will be very easy to finally know what (and when) really happenned.

One of most important accounting records he will have to provide is the list of all the cold wallet addresses that they were using at each moment in time. Being the most valuable assets of the company (apart from bank accounts) all of them should be perfectly identified in company reports showing periodical balance.

Then we will have the full picture of what was happenning and since when.... the same full picture that Karpeles had in front of him all the time... the same that would automatically show a discrepance as soon as it started between mtgox internal BTC balances and real ("blockchained") addresses under their control.

After that basic accounting is done then it will be easy to check which explanation fits better into this case... and since when was MTgox knowingly operating in insolvency without reporting it, which I supposse its a criminal act in Japan as much as everywhere else.

Maybe then the malleability "issue" and all the bunch of excuses will be insignificant and no matter anymore even if it had any additional impact on the already criminal operation of MtGOX.

... Or maybe the next twist of this plot will be to execute another "It's gone!" this time to the accounting records? That would be even more funny.

TL;DR: The answer is in the accounting records which he will have to provide to prove the insolvency and how it developed over time. It won't be easy to fake them as they must match and be coherent with the public blockchain.


Title: Re: Mt.Gox technical autopsy
Post by: wheatstone on February 28, 2014, 03:49:11 PM
Seriously, this nonsense must stop. Understanding of the transaction malleability issue is completely wrong throughout this thread.

Transaction malleability does not result in two transactions in the blockchain.

This cannot be stressed enough, apparently.

nanonano pretty much had it right with his list.

The claimed explanation for MtGox was:
* MtGox sends a transaction
* attacker modifies the txid
* modified transaction goes in the blockchain, and bitcoin ownership changes
* attacker complains MtGox "I didn't receive my coins"
* MtGox searches for original txid in blockchain, doesn't find it  (<-- this is the bug)
* MtGox thinks they haven't paid, apologizes and sends a second, totally separate transaction of the same value

Nothing here is incorrect, as such, but the conclusions drawn are incorrect.

When mtgox, or anyone else, submits a transaction it does not immediately enter the blockchain. This only happens once a block containing the transaction is mined, resulting in what is referred to as a confirmation.

Transaction malleability relies on the fact that while the inputs and outputs (source and destination) are fixed by cryptographic signatures, the transaction id (txid) is not. In other words, anyone can submit an identical transaction (same source and destination, same amounts etc), just with a different txid.

The way mining works, if a miner already has the "original" transaction in their memory, they will discard the modified transaction. However, the miners' perception of which is the original depends only on which one they see first.

In other words, exploiting transaction malleability relies on submitting modified transactions in the very small window before the original transaction reaches all miners... and then hoping that the modified variant is the one that gets mined first.

Only the transaction that gets mined first gets in the blockchain (with the small caveat that sometimes blocks are "orphaned" if two miners solve a block almost at the same time).

To see which transactions had multiple versions would thus require "live" access to the miners' memory pools. If no logging of these discarded transactions (whichever reached a miner last) happened, there's no way to see it after the fact.


Title: Re: Mt.Gox technical autopsy
Post by: bitserve on February 28, 2014, 04:58:19 PM

To see which transactions had multiple versions would thus require "live" access to the miners' memory pools. If no logging of these discarded transactions (whichever reached a miner last) happened, there's no way to see it after the fact.

Thanks for the detailed explanation.

Do you have any idea of how high is the percentage of discarded transations under "normal" conditions? Would it be feasible for some miners to be logging those "incidents"?

Anyway, one thing is that we can't detect those "malleability attacks" afterwards and other very different that mtgox can't reconcile its internal accounting with the blockchain and detect each and all discrepancies.

Please let me know if theres some reason that a process like this would not work:

1- Balances for all accounts with no withdrawals are ok (at least from a "malleability issue" point of view)
2- Balances for all accounts with no failed withdrawal + reissue are also OK
3- Accounts with failed withdrawal transactions lets check like this:
    - For each failed transaction lets search if the transaction went through. Obviously not using the txid, but ammount, input, output. If it did, then flag the account as "abuser" and take note of the "leaked" ammount.

Now you should know exactly how much the malleability "issue" is to blame for the "missing" BTC's and also who were the users behind it, and when/where the leaked money did go to.

Is there any technical reason for this to not be a simple process to run against mtgox records or maybe the only reason they havent made this figure public is because they already know that total sum would just be negligible and that was not the real problem?

P.S.: I am aware that there have been some comments saying that one of the faults of mtgox was that maybe they were doing the reissue using different inputs...

Well, that's what makes difficult/impossible to locate the "dupes" using only the blockchain, but mtgox accounting records should have the missing information, i.e. inputs/output/ammount for each attempted transaction, original or reissued, so that it would be a matter of going through that records rechecking everything against the blockchain.



Title: Re: Mt.Gox technical autopsy
Post by: jamal26 on February 28, 2014, 05:11:11 PM
Quote
In other words, exploiting transaction malleability relies on submitting modified transactions in the very small window before the original transaction reaches all miners... and then hoping that the modified variant is the one that gets mined first.

Are you saying that someone can observe a transaction being submitted to one node and then quickly submit a bogus transaction to other redundant nodes?  If not, how is the original transaction information obtained?  If so, then what happens when the different nodes disagree?

I have an accounting background, so I'm still very confused about why there appears to have been no reconciliations done.


Title: Re: Mt.Gox technical autopsy
Post by: wheatstone on February 28, 2014, 05:56:29 PM
Are you saying that someone can observe a transaction being submitted to one node and then quickly submit a bogus transaction to other redundant nodes?

Correct. To do so on any consistent basis, one would need good (low latency) access to multiple nodes (preferably the one mtgox submits to plus another with a lot of hashing power).

Quote
If so, then what happens when the different nodes disagree?

Hashing power rules all...

Nodes never "agree" in the sense that their set of transactions is identical. Mining pools are completely free to include whatever transactions they wish and the same for relaying transactions. Certain transactions are not supposed to be relayed, including modified transactions. Assuming everyone behaves (and they pretty much do), exploiting transaction malleability becomes a race to reach the greatest amount of hashing power.

Even if the attacker only reaches 10% of all hashing power (and he could likely reach much more with good connections to the major pools), he would still get the modified transaction into the blockchain about 10% of the time.

There is also another way "nodes" can disagree, but I don't think that is what you were talking about. I alluded to it a few times earlier in my previous reply (and again two sentences ago). The issue here is that if two mining pools find a valid block at very close to the same time, the nodes will relay the version of the blockchain they received first (just like with the modified transactions earlier). The result is that for a short while, the blockchain has actually forked. Each individual mining pool works on the one they received first (or are supposed to, anyway, see "51% attack"). At some point, the next block in one of the chains will be found and all the miners will switch to the longest chain.


Title: Re: Mt.Gox technical autopsy
Post by: ilpirata79 on February 28, 2014, 06:00:50 PM
You couldn't do that, because only one of the transactions would actually be in the block chain.

That would mean that valid transactions do not appear on blockchain - I consider this rubbish.


Both transactions would be in the blockchain - the initial withdrawal and then the re-withdrawl request several weeks later which mtgox would have allowed.

I agree with this.

One needs to pair those two transactions (find them in the blockchain). There would be a few non-complex criteria for finding such malleable-looking transactions:
- amount of both must match
- date range should not be broad, e.g. the second transaction cannot be older than 1 day / week / month than the first one
- both transactions must be spent from an exchange address (not sure how to find such addresses but my guess is that such addresses generated a huge number of transactions for a short time)
- others

------------------------------

It is a perfect research project for applicants who want grants from TBF.


There is no need for equal transactions (apart from the txid) to be present in the blockchain:

1) I ask for a withdraw of some amount of bitcoins
2) I alter the txid
4) I wait for the altered transaction to get confirmed
3) I ask Mtgox to credit those bitcoins back to my account because the transaction was not successful
4) go to 1

Best regards,
ilpirata79



Title: Re: Mt.Gox technical autopsy
Post by: wheatstone on February 28, 2014, 06:22:23 PM
I have an accounting background, so I'm still very confused about why there appears to have been no reconciliations done.

There's no arguing with this sentiment. It is truly baffling if the ledgers were not reconciled on a regular basis.


Title: Re: Mt.Gox technical autopsy
Post by: wheatstone on February 28, 2014, 06:34:37 PM
Do you have any idea of how high is the percentage of discarded transations under "normal" conditions?

I'm not sure what "normal" conditions are. If you mean no-one-trying-to-abuse-anything situation, then 0%. Not that every single modified txid would be an attempt to abuse, but there doesn't seem to be any good reason for doing it and certainly none of the major clients support it.

Quote
Would it be feasible for some miners to be logging those "incidents"?

Yes. In fact, you wouldn't even need to be a miner. Anyone relaying transactions (e.g. running the full bitcoin client) could, in theory, log these "incidents".

Someone actually did this to an extent after mtgox made their initial announcement. From the results he posted to this forum, probably only "counts" were logged (not each individual transaction). Their count was generally quite low, with a few spikes here and there (and a 24 hour break on Chinese New Year, if I recall correctly).


Title: Re: Mt.Gox technical autopsy
Post by: bitserve on February 28, 2014, 06:53:18 PM
I have an accounting background, so I'm still very confused about why there appears to have been no reconciliations done.

There's no arguing with this sentiment. It is truly baffling if the ledgers were not reconciled on a regular basis.

Do you really think that it is really possible that all the BTC's were leaked out of gox by taking advantage of the malleability issue?

And that, even if no periodic balance reconciliation were done (which might be criminal negligence in itself), they woulnt notice something very wrong was happenning each time they had to load another of their deep cold wallet address because the previous ones had all been emptied without any logical reason?

I find easier to believe that all the time they knew they were running a fractional reserve, probably risking the remaining BTCs in an attempt to recover solvency and losing them in unfortunate trades, until they couldnt even fulfill its pending withdrawals... and then blaming some old known issue which may or may not have had an additional but negligible impact in its balance.




Title: Re: Mt.Gox technical autopsy
Post by: bitserve on February 28, 2014, 07:08:36 PM
Do you have any idea of how high is the percentage of discarded transations under "normal" conditions?

I'm not sure what "normal" conditions are. If you mean no-one-trying-to-abuse-anything situation, then 0%. Not that every single modified txid would be an attempt to abuse, but there doesn't seem to be any good reason for doing it and certainly none of the major clients support it.

Then there is a reason why such collisions should be considered an "incident" and logged separately (for forensic purposes), especially if the volume is not that big as to make it impractical. Pity it wasn't being already done on a regular basis, at least on some "strategical" points of the network.

Quote
Quote
Would it be feasible for some miners to be logging those "incidents"?

Yes. In fact, you wouldn't even need to be a miner. Anyone relaying transactions (e.g. running the full bitcoin client) could, in theory, log these "incidents".

Someone actually did this to an extent after mtgox made their initial announcement. From the results he posted to this forum, probably only "counts" were logged (not each individual transaction). Their count was generally quite low, with a few spikes here and there (and a 24 hour break on Chinese New Year, if I recall correctly).

Again a pity it wasn't being logged in full even after the mtgox announcement. But anyways, that information could imply that the "attack" was not so generalised as to justify the ammount of damage mtgox wants everyone to believe. Maybe it even was a smoke screen to give some credibility to the announcement.

Also, being this type of attack some sort of race condition (combined with mtgox negligencies), the volume should have probably been huge in comparison to the sucessful "exploits", and this doesnt seem to have been the case.





Title: Re: Mt.Gox technical autopsy
Post by: wheatstone on February 28, 2014, 07:44:30 PM
Do you really think that it is really possible that all the BTC's were leaked out of gox by taking advantage of the malleability issue?

And that, even if no periodic balance reconciliation were done (which might be criminal negligence in itself), they woulnt notice something very wrong was happenning each time they had to load another of their deep cold wallet address because the previous ones had all been emptied without any logical reason?

There seems to be no shortage of speculation on this topic around here without me contributing.

Let me simply state that it would have been quite interesting to have access to a full log from one or more relaying/mining nodes, were one such to exist. Preferably from day 1.


Title: Re: Mt.Gox technical autopsy
Post by: wheatstone on February 28, 2014, 08:07:43 PM
Please let me know if theres some reason that a process like this would not work:

1- Balances for all accounts with no withdrawals are ok (at least from a "malleability issue" point of view)
2- Balances for all accounts with no failed withdrawal + reissue are also OK
3- Accounts with failed withdrawal transactions lets check like this:
    - For each failed transaction lets search if the transaction went through. Obviously not using the txid, but ammount, input, output. If it did, then flag the account as "abuser" and take note of the "leaked" ammount.

...

Well, that's what makes difficult/impossible to locate the "dupes" using only the blockchain, but mtgox accounting records should have the missing information, i.e. inputs/output/ammount for each attempted transaction, original or reissued, so that it would be a matter of going through that records rechecking everything against the blockchain.

Any reasonable audit would require access to inside information (i.e. from mtgox).

Every single point you list above requires record keeping on the part of mtgox. Although one might reasonably presume that such records existed, I do not know what records mtgox did in fact keep.

The logic in your points seems sound, however.

As for the inputs used in a given payment, it seems reasonable that the exchange would simply queue up the outgoing transaction using the normal process for doing so (which would result in an entirely new payment, with a (possibly) new set of inputs).

With regards to what tentative analysis could be done on the blockchain, one would need a reasonably exhaustive list of addresses / outputs that belonged to mtgox in order to even get started. Such a list does not exist, publicly at least. Since mtgox used a custom wallet to create these transactions, it is possible that they could in some way be identified if they had some characteristic that was different from other transactions (but I doubt it).


Title: Re: Mt.Gox technical autopsy
Post by: ilpirata79 on February 28, 2014, 08:16:19 PM
Get to know the abusers is not difficult. Just check the transactions that got confirmed with a different txid. For this there is need of the authentic Mtgox logs.

This helps locating the stolen bitcoins. Off course it does not help to locate the thieves because they for sure used false documents to register and an anonymous connection.

Best regards,
ilpirata79


Title: Re: Mt.Gox technical autopsy
Post by: bitserve on February 28, 2014, 09:08:51 PM
Thanks for the detailed explanations. Yes, I meant using mtgox internal accounting records, which obviously mtgox should have and so (being this forensic reconciliation possible) could/should do this audit themselves and offer it as proof in the process as defense of the accusation of internal theft (which is also already in the air).

Each attempted transaction should have been logged, not only because of its forensic value, but because one couldnt know it would become a "failed" one, you need to log it to be able to track the outcome. In no way they could justify that failed transactions were simple wiped from the logs afterwards.

It's the same as they should have a detailed log of the order book at any point in time.

But now that I am thinking about the malleability issue, and after reading the explanations, some thing has come to my mind:

Withdrawal transactions are put in queue and then a transaction with many inputs and many outputs (all the withdrawal destinations) is created... so... for each sucessful malleability attack, not only the "attacker" withdrawal would be reissued, but all the ones in the same "failed" transaction would. So many people besides the attacker would have received duplicate bitcoin transactions for each sucessful attack. (Unless I am wrong in my understanding of how that process works)

At least if we come to believe the explanation that it was an AUTOMATED reissue process and not a manual one after opening a ticket asking for the reissue.

The more I think about it, the more it all seems totally BS to me.






Title: Re: Mt.Gox technical autopsy
Post by: nanonano on February 28, 2014, 09:43:45 PM
Get to know the abusers is not difficult. Just check the transactions that got confirmed with a different txid. For this there is need of the authentic Mtgox logs.

So have you been logging those transactions that never ended up in the block chain? Because I sure haven't.


Title: Re: Mt.Gox technical autopsy
Post by: wheatstone on February 28, 2014, 09:46:02 PM
Off course it does not help to locate the thieves because they for sure used false documents to register and an anonymous connection.

No false documents necessary. Karpales has stated that the double payouts were from unverified accounts. Until recently, mtgox did not require verified accounts for btc withdrawal.


Title: Re: Mt.Gox technical autopsy
Post by: ilpirata79 on February 28, 2014, 09:51:50 PM
Get to know the abusers is not difficult. Just check the transactions that got confirmed with a different txid. For this there is need of the authentic Mtgox logs.

So have you been logging those transactions that never ended up in the block chain? Because I sure haven't.

I haven't, but Mtgox should have. As I said we need their authentic logs (authentic in the sense that they are not faked)...

Best regards,
ilpirata79


Title: Re: Mt.Gox technical autopsy
Post by: nanonano on February 28, 2014, 09:56:07 PM
Get to know the abusers is not difficult. Just check the transactions that got confirmed with a different txid. For this there is need of the authentic Mtgox logs.
So have you been logging those transactions that never ended up in the block chain? Because I sure haven't.
I haven't, but Mtgox should have. As I said we need their authentic logs (authentic in the sense that they are not faked)...
Ok, then we agree. The only reason I asked is that I thought you said the opposite in the post I quoted: that we would not need MtGox logs for this.


Title: Re: Mt.Gox technical autopsy
Post by: bigasic on February 28, 2014, 09:57:47 PM
I know for a fact that karpales (sp) has been investigated by us authorities for months. One federal agent told me that they had a close eye on him and that he was a crook. I was told this late last summer. So, its pretty obvious that this whole mtgox mess was caused by them (mtgox or mk) by either just by plain old theft or trying to manipulate bitcoin someway..

Either way, Im glad mtgox is out of the equation and let the healing begin... Hopefully we will be up to 1k by summer...


Title: Re: Mt.Gox technical autopsy
Post by: wheatstone on February 28, 2014, 10:19:38 PM
But now that I am thinking about the malleability issue, and after reading the explanations, some thing has come to my mind:

Withdrawal transactions are put in queue and then a transaction with many inputs and many outputs (all the withdrawal destinations) is created... so... for each sucessful malleability attack, not only the "attacker" withdrawal would be reissued, but all the ones in the same "failed" transaction would. So many people besides the attacker would have received duplicate bitcoin transactions for each sucessful attack. (Unless I am wrong in my understanding of how that process works)

At least if we come to believe the explanation that it was an AUTOMATED reissue process and not a manual one after opening a ticket asking for the reissue.

I very much doubt multiple withdrawals were happening per transaction. Presumably, they used some algorithm to determine which input(s) were best spent on a given withdrawal.


Title: Re: Mt.Gox technical autopsy
Post by: bitserve on February 28, 2014, 10:42:14 PM

I very much doubt multiple withdrawals were happening per transaction. Presumably, they used some algorithm to determine which input(s) were best spent on a given withdrawal.

I see. I thought due to its (presumably) high number of transactions maybe they were doing the same "merging" of multiple inputs(funds)/outputs(payments) into one big transaction that satoshidice was using... but probably they didnt have the same need to combine so many small ammounts into a bigger transaction to cut on fees.

What you say makes more sense and is more consistent with this whole "transactions" issue.

Well, lets see if at some point gox publish the data needed to back their vague explanations, until then there's not much more to think besides speculation.


Title: Re: Mt.Gox technical autopsy
Post by: BurtW on February 28, 2014, 11:19:51 PM
If the attack vector is "my coins never arrived" followed by GOX either returning the coins to the users account or issuing a second transaction then everything needed to track down the culprits is in the help records because every report that "my coins never arrived" would have to go to the help desk.

So, simply scan through all the help desk records and find out who was reporting lost transactions.

Now, if all the attacker had to do was open a separate new unverified account for each "lost transaction", and they were smart about it we will never know exactly who did it.  But the extent of the fraud would be easily known from those records.


Title: Re: Mt.Gox technical autopsy
Post by: bitserve on March 01, 2014, 01:42:33 PM
If the attack vector is "my coins never arrived" followed by GOX either returning the coins to the users account or issuing a second transaction then everything needed to track down the culprits is in the help records because every report that "my coins never arrived" would have to go to the help desk.

So, simply scan through all the help desk records and find out who was reporting lost transactions.

Now, if all the attacker had to do was open a separate new unverified account for each "lost transaction", and they were smart about it we will never know exactly who did it.  But the extent of the fraud would be easily known from those records.

That's the point. It's so easy (for mtgox) to audit the whole issue up to the last satoshi that the vague excuses just dont match.

Also, any time I try think of a situation in which this happenned from a long time ago (ability to withdraw without verification) its almost impossible that it wasn't detected on time before a HUGE hole, and unthinkable that it wouldnt be detected afterwards any time during the past year.

I hope that when the criminal charges press him, he will give some better and more detailed explanations of WHAT (and HOW)  REALLY happenned.

Also, the "fact" that we can't follow the traces up to a certain identity is a common myth. Given enough (internal) data about the whole issue, and considering the INMENSE ammount of BTC we are talking about and that humans make mistakes, I am very confident that with mtgox colaboration (if they really are not into it) it would be possible to follow the traces of the MANY "leaks" to individual entitities.

But the vague explanations, trying to blame theoretical vulnerabilities without giving ANY proof of it actual impact, etc... makes me think the answer its much more simple than all that.


P.S.: Also, I want to point some thing:

The reason for having a hot/cold/deep wallets system is because when you run an online exchange you can't trust whatever your online databases (ie: BALANCES) says, because it can be hacked, manipulated, etc...

I mean, if you take advantage of a vulnerability that makes the online system belive you have a balance of 1000 BTC you do some checks before withdrawal, or, at the very least, you risk your hot wallet to be emptied and ANY time that happens, BEFORE loading one of the cold wallets to replenish the hot wallet, you reconciliate the balances to check that everything is ok and you arent being fooled by altered data.

Not doing so, would be the equal of not having a cold/hot wallet protection at all, and you could simply be putting all your balance on a hot wallet anyways.

So no, not only saying they didnt periodically reconcicle the balances IS criminal negligence... it is also *FALSE*.


Title: Re: Mt.Gox technical autopsy
Post by: bitserve on March 01, 2014, 08:17:17 PM
And here it is:

http://www.reddit.com/user/WeAreMtGox

WeAreMtGox 301 puntos 10 meses atrás

NO. Everything is accounted for (BTC and money). Fractional reserve is absolutely against our principles. In fact 90~95% of BTC are held in cold storage.

[–]WeAreMtGox 10 puntos 8 meses atrás

Absolutely not true. We do not operate a fractional reserve exchange. 100% of deposits and Bitcoins are accounted for at all times.


That was 8/10 months ago... Were they lying then or now? Pick your choice, because the two just doesn't match.