Help a newbie; why is hashing not done once but twice during Bitcoin transaction

[NdaMk]

Hash SHA256 are said to be a good ID numbers (TXID) since they can accept any long type of data string and generate a short but unique data.
This TXID which can also be use when an output is required to be use for a transaction input.

My question now is since   during hashing bytes sequence are required what happens when one forgets to pack the hexadecimal strings to bytes?

Secondly why is hashing not done once but twice in Bitcoin?

Thirdly can two transactions have same TXID?

[PX-Z]

I don' have enough technical knowledge about the two, but I can give an answer for this

Thirdly can two transactions have same TXID?

Bitcoin already has +700 millions transactions^[1] since then yet there's no single case of the same txid happened (someone correct me if im wrong). Although there's no centralized database to check if there is already existing txid but with the hashing method used, there's almost zero chance that a transaction will have the same txid.

[1] https://www.blockchain.com/charts/n-transactions-total

[BlackHatCoiner]

My question now is since   during hashing bytes sequence are required what happens when one forgets to pack the hexadecimal strings to bytes?

I'll need you to elaborate. What you do mean one forgets to pack strings to bytes?

Secondly why is hashing not done once but twice in Bitcoin?

That is a good question. It turns out that it has to do with the so-called length extension attack. I have never seen this been discussed back in the early days, though, so there may be another reason.

Thirdly can two transactions have same TXID?

Theoretically, there could be a collision. Realistically effectively, it can't. It's 1 in 2^256 which is a stupidly large range of numbers.

[hosseinimr93]

Theoretically, there could be a collision. Realistically effectively, it can't. It's 1 in 2^256 which is a stupidly large range of numbers.

Even in theory, that's impossible. BIP30 doesn't allow a block to include a transaction with a same ID as an existing a not-fully-spent transaction.
(The post has been edited. Thanks o_e_l_e_o for the correction.)

Blocks are not allowed to contain a transaction whose identifier matches that of an earlier, not-fully-spent transaction in the same chain.

Before implementing BIP30, transactions could have the same ID.

The coinbase transactions on blocks number 91,722 and 91,880 have the same transaction ID.
The coinbase transactions on blocks number 91,812 and 91,842 have the same transaction ID.

For more information, visit learnmeabitcoin.

[NdaMk]

My question now is since   during hashing bytes sequence are required what happens when one forgets to pack the hexadecimal strings to bytes?

I'll need you to elaborate. What you do mean one forgets to pack strings to bytes?

Thanks for concern.
My question is since the hash function accepts data in binary form, as I read it should be in bytes. What happens when one forgets to change the strings in hexadecimal to its respective bytes, will the result still be the same

My question now is since   during hashing bytes sequence are required what happens when one forgets to pack the hexadecimal strings to bytes?

I'll need you to elaborate. What you do mean one forgets to pack strings to bytes?

Secondly why is hashing not done once but twice in Bitcoin?

That is a good question. It turns out that it has to do with the so-called length extension attack. I have never seen this been discussed back in the early days, though, so there may be another reason.

Thirdly can two transactions have same TXID?

Theoretically, there could be a collision. Realistically effectively, it can't. It's 1 in 2^256 which is a stupidly large range of numbers.

Secondly why is hashing not done once but twice in Bitcoin?

That is a good question. It turns out that it has to do with the so-called length extension attack. I have never seen this been discussed back in the early days, though, so there may be another reason.

Thirdly can two transactions have same TXID?

Theoretically, there could be a collision. Realistically effectively, it can't. It's 1 in 2^256 which is a stupidly large range of numbers.
[/quote]

[o_e_l_e_o]

BIP30 doesn't allow a block to include a transaction with a same ID as an existing transaction in the blockchain.

Not quite. BIP30 doesn't allow a transaction to have the same ID as a not-fully-spent transaction. So if all the outputs from a given transaction are spent, then that transaction ID can indeed be duplicated in the future. While it is still incredibly unlikely that such an event would ever happen, without this wording it would be impossible to run pruned nodes, since all nodes would have keep a record of every transaction ever made to ensure any current transaction was not duplicating a previous one.

[LoyceV]

Before implementing BIP30, transactions could have the same ID.

The coinbase transactions on blocks number 91,722 and 91,880 have the same transaction ID.
The coinbase transactions on blocks number 91,812 and 91,842 have the same transaction ID.

I didn't know this. Does this mean the 50 Bitcoin created by the second transaction basically don't exist?

[hosseinimr93]

Does this mean the 50 Bitcoin created by the second transaction basically don't exist?

Both blocks are valid and both transactions exist in the blockchain.
The blocks number 91842 and 91880 were considered as exceptions and bitcoins generated in them should be spendable.  

Visit the link below for more information.
Apply BIP30 checks to all blocks except the two historic violations.

[NdaMk]

Thank you all for your contributions.
The other unanswered question is that if an hexadecimal string is inputted to the hash function due to forgetfulness will it accept it? and will there be a difference to the result since the function requires the string in bytes?

[o_e_l_e_o]

Does this mean the 50 Bitcoin created by the second transaction basically don't exist?

Both blocks are valid and both transactions exist in the blockchain.
The blocks number 91842 and 91880 were considered as exceptions and bitcoins generated in them should be spendable.

The combination of these two duplicate transactions has effectively removed 100 BTC from the supply permanently.

When you create a bitcoin transaction, you reference the UTXOs you want to use as inputs based on the transaction ID of the transaction which created those UTXOs. Considering one of the pairs listed above, then since those transactions have identical transaction IDs, then there is no way for you to reference them separately. The second transaction therefore effectively overwrote the first one. As far as the blockchain is concerned, there is only one output of 50 BTC in each case. So given that it has happened twice, 100 BTC has been destroyed permanently.

[BlackHatCoiner]

What happens when one forgets to change the strings in hexadecimal to its respective bytes, will the result still be the same

No.

The hash of a string is different than the hash of the bytes of the string. As you said, hash function takes the input in binary format. If you hash the hex "abcdef", you'll get:

Code:

Hex: abcdef
Bin: 101010111100110111101111
SHA256: 995da3cf545787d65f9ced52674e92ee8171c87c7a4008aa4349ec47d21609a7

Respectively, to hash the string "abcdef" you'll have to convert it first to binary.

Code:

UTF-8: abcdef
Hex: 616263646566
Bin: 011000010110001001100011011001000110010101100110
SHA256: bef57ec7f53a6d40beb640a780a639c83bc29ac8a9816f1fc6c5c6dcd93c4721

[larry_vw_1955]

The coinbase transactions on blocks number 91,722 (https://www.blockchain.com/btc/block/91722) and 91,880 (https://www.blockchain.com/btc/block/91880) have the same transaction ID.
The coinbase transactions on blocks number 91,812 (https://www.blockchain.com/btc/block/91812) and 91,842 (https://www.blockchain.com/btc/block/91842) have the same transaction ID.

people always say there is no known collision with sha256 but i guess they are keeping this matter secret so as not to alarm anyone.
not one but 2 collisions. sha256 is broken for sure if it's already happened twice.

When you create a bitcoin transaction, you reference the UTXOs you want to use as inputs based on the transaction ID of the transaction which created those UTXOs. Considering one of the pairs listed above, then since those transactions have identical transaction IDs, then there is no way for you to reference them separately. The second transaction therefore effectively overwrote the first one. As far as the blockchain is concerned, there is only one output of 50 BTC in each case. So given that it has happened twice, 100 BTC has been destroyed permanently.

i don't know about that. from just looking at the blockchain how would you know which miner got the 50btc reward since the coinbase transaction is dated by the first transaction not the second. but one thing is for sure, someone lost 50btc. overwriting the blockchain is a very serious bug. too bad 2 people had to pay the price for that. one of the miners got deleted out of existence in each pair of transactions. i'm sure they wondered what the heck just happened. very serious bug!

[pooya87]

people always say there is no known collision with sha256 but i guess they are keeping this matter secret so as not to alarm anyone.
not one but 2 collisions. sha256 is broken for sure if it's already happened twice.

This is NOT a collision, these are same EXACT transactions that produce the same EXACT hash.

It used to be possible because coinbase script didn't have to include the block height so 2 coinbase transactions would have same inputs and everything else (version, locktime, intput/output count, output) could be also similar hence producing the same exact transaction with same exact hash.
But now after BIP34 activation, all coinbase transactions must contain the block height which makes it impossible to produce same hash since finding a SHA256 collision is impossible.

[LoyceV]

people always say there is no known collision with sha256 but i guess they are keeping this matter secret so as not to alarm anyone.
not one but 2 collisions. sha256 is broken for sure if it's already happened twice.

I took a sha256sum of the above text:
34da38ff7f462890cc5b6b950db52dc97c8d80f7364075fdec91077f34c3ea50
Wait, let me do it again:
34da38ff7f462890cc5b6b950db52dc97c8d80f7364075fdec91077f34c3ea50
It's the same That doesn't mean sha256sum is broken, it means it does exactly what it's supposed to do. If you can get the same hash from a different input, then you can call it broken.

[o_e_l_e_o]

from just looking at the blockchain how would you know which miner got the 50btc reward since the coinbase transaction is dated by the first transaction not the second. but one thing is for sure, someone lost 50btc.

Considering either one of the pairs of transactions being discussed, there are not two miners - it is the same miner mining both blocks and costing themselves 50 BTC. The transactions are identical in both cases, and they pay the same 50 BTC to the same address.

overwriting the blockchain is a very serious bug.

They've overwritten a transaction, not the blockchain. Both transactions are still easily visible on the blockchain.

As discussed above, this was a bug in how transactions were created which has been long fixed, and has nothing to do with SHA256 being broken.

[LoyceV]

Bitcoin already has +700 millions transactions^[1] since then yet there's no single case of the same txid happened (someone correct me if im wrong).

Let's put this to the test: I took Bitcoin block data (665 GB): inputs, outputs and transactions, and extracted all txids from all "outputs" blocks. This gave 724,446,069 transactions (data up to April 9, 2022). As expected, there were only 2 duplicates:

Code:

d5d27987d2a3dfc724e359870c6644b40e497bdc0589a033220fe15429d88599
e3bf3d07d4b0375638d5f1db5255fe07ba2c4cb067cd81b84ee974b6585fb468

I wanted to post this yesterday, but processing the data took longer.

[larry_vw_1955]

This is NOT a collision, these are same EXACT transactions that produce the same EXACT hash.

thats strange.

It used to be possible because coinbase script didn't have to include the block height so 2 coinbase transactions would have same inputs and everything else (version, locktime, intput/output count, output) could be also similar hence producing the same exact transaction with same exact hash.

and no one said anything about it or noticed it was possible?

But now after BIP34 activation, all coinbase transactions must contain the block height which makes it impossible to produce same hash since finding a SHA256 collision is impossible.

so they squashed another bug. hopefully there's no more of those...

They've overwritten a transaction, not the blockchain. Both transactions are still easily visible on the blockchain.

that's not what happened. what happened is, the first coinbase transaction was logged on to the blockchain but not the 2nd one. so that's how it happened.

if a transaction got overwritten then what the miner could have done is spend his 50btc and then get another 50btc reward when his first coinbase transaction got overwritten. not what happened though. there was no 2nd coinbase transaction and you can't see it anywhere.

[odolvlobo]

Thank you all for your contributions.
The other unanswered question is that if an hexadecimal string is inputted to the hash function due to forgetfulness will it accept it? and will there be a difference to the result since the function requires the string in bytes?

It depends on the tool used for hashing. Binary and hex are just different representations and the tool may or may not convert the input to the desired representation.

There are many possibilities. For example, the tool may detect a hex string and convert it to binary data before hashing, or it may expect binary data and hash the string regardless of what it contains, or it may expect a hex string and be unable to hash binary data because it would be invalid hex.

[pooya87]

and no one said anything about it or noticed it was possible?

It first had to be detected. When it did, BIP30 was created.

that's not what happened. what happened is, the first coinbase transaction was logged on to the blockchain but not the 2nd one. so that's how it happened.

if a transaction got overwritten then what the miner could have done is spend his 50btc and then get another 50btc reward when his first coinbase transaction got overwritten. not what happened though. there was no 2nd coinbase transaction and you can't see it anywhere.

Nothing is being overwritten in the blockchain. The blockchain is immutable and can not change. What happens is that the first transaction's output was placed in UTXO set after it was created, then the second one came along and replaced it in the UTXO set. So if the second one is spent, it will be removed from the set and the first one that was already replaced no longer exists in the UTXO set to be spent.

[larry_vw_1955]

and no one said anything about it or noticed it was possible?

It first had to be detected. When it did, BIP30 was created.

what do you mean it first had to be detected? with all the intellectually powerful people in bitcoin, you would think that one would be a gimme. and they would have saw it coming a mile a way but i guess not. an inflation bug is one thing but something like this. satoshi should have realized that could happen. suprised it didn't happen to him as many blocks as he mined!

Nothing is being overwritten in the blockchain.

Looking at the time stamp of the coinbase transaction proves that. Nothing got overwritten.

The blockchain is immutable and can not change. What happens is that the first transaction's output was placed in UTXO set after it was created, then the second one came along and replaced it in the UTXO set.

how exactly does that happen?

So if the second one is spent, it will be removed from the set and the first one that was already replaced no longer exists in the UTXO set to be spent.

maybe in your view that's how it worked but by looking at the two coinbase transaction time stamps they go by the first one. so it's like the 2nd coinbase transaction never existed. if you don't believe that then how do you expain the scenario of if the miner spent his 50 btc before the 2nd coinbase transaction came along? you can't. unless you agree that the 2nd coinbase transaction was just ignored when it saw that a transaction id already existed with the same id. you can't go back and alter the blockchain so...