6 questions about the protocol

[ignacio_buendia]

My primary motivation is to precisely understand one step-by-step attempt of generating one proof of work - regardless of success -, and illustrate the steps in a flow chart. If you are interested in discussing technology related topics about the basics of the Bitcoin protocol, I would be glad if you could answer the followings questions.:


MERKLE TREE AND HASH

1) Preparation of a merkle structure.:

Source: ‘The body of the block contains the transactions. These are hashed only indirectly through the Merkle root. Because transactions aren't hashed directly, hashing a block with 1 transaction takes exactly the same amount of effort as hashing a block with 10,000 transactions.’

Question: ‘hashing a block with 1 transaction takes exactly the same amount of effort as hashing a block with 10,000 transactions.’ This statement wouldn’t be better like this? - Finding a proof of work of a block with 1 transaction takes exactly the same amount of effort as hashing a block with 10,000 transactions.

Question b: The preparation of the merkle tree - hashing the transactions into a block - is also the task of a miner, right?


2) The time paradox of the coinbase transaction.:

If I understand it correctly, the coinbase transaction is always the first in the body of a block. When a miner organizes the transactions in a merkle structure it starts the order of transactions with his own coinbase transaction. Hence when the next block is generated, the coinbase transaction of the ‘transaction sequence’ of the previous block is the first one, despite the fact, that chronologically it was the “last one”.

Question: Is that correct?


3) Hash as a constant variable of 1 proof of work?.:

When an ASIC is working at 15 TH/s, it means that it is doing this 15 000 000 000 000 times a second. I assume that if the value of nonce is about 4 million (e.g.) than it has to mean that incrementing the nonce with one requires a tremendous amount of additional hashes.

Question: Why is that? How many hash calculations are required / incrementing the nonce with one, or put it differently: How many hashing steps are required to make 1 proof of work attempt, regardless of success.


NONCE

4) ExtraNonce field.:

Source.: 'It’s best to avoid adding the extraNonce until the nonce is exhausted, because any change to the extraNonce changes the merkle tree. This requires extra computation in order to propagate the change upwards until a new root of the merkle tree is calculated.'

That would mean, that the parts of the merkle tree has to be recalculated. Not the entire structure though, because by changing the extraNonce, only the coinbase transaction has to be changed. Yellows are the changes which changes with the new coinbase transaction.:

https://docs.google.com/spreadsheets/d/12rGTdd4DyD3SIt9HVTTxSGAxnr5ZZNq6lzoImP9hZI4/edit#gid=0

Question: Is that logic correct?:


5) Same nonce value, different miners.:

Let’s assume the existence of two ‘twin’ miner, which means that they have started the actual proof of work process together and have the same hashing power.:

Miner1 input message with nonce 2345 looks like this.:

02000000|632b33406d89309e916545870727cabf498631a3a135230f0000000000000000|
56d851eb7bcee82e4020649c6bc6405878659290524c830e8d8a22939013a9db|98739054|747B1B18|55161AC0

Miner2 input message with nonce 2345 looks like this.:

02000000|632b33406d89309e916545870727cabf498631a3a135230f0000000000000000|
2e99f445c007a9158207cc30cebad2b3d26c45fdab2ebdf50d261335fc00d92c|98739054|747B1B18|55161AC0

Theoretically speaking the following values are also the same, because the miners are ‘twins’ and both of them work on the same proof of work.:

    - the version (self-evidently)
    - the previous proof of work (632b33406d89309e916545870727cabf498631a3a135230f0000000000000000)
    - the timestamp (they are twin miners, so both of them reached the nonce level ‘1427512000’ at the same time)

Question: Is that correct?


6) Nonce: 0.:

Question: Is it obligatory to start the computation with a nonce field of 0?


It would already be a great help to answer a single question. Thank you!

[1714790437]

1714790437

[1714790437]

1714790437

[1714790437]

1714790437

[1714790437]

1714790437

[pooya87]

Finding a proof of work of a block with 1 transaction takes exactly the same amount of effort as hashing a block with 10,000 transactions.

finding proof of work is the same as hashing the block [header] so it makes no difference.

The preparation of the merkle tree - hashing the transactions into a block - is also the task of a miner, right?

that is correct.
the miner chooses any valid transaction from its memory pool and places them in the block it is mining then computes its merkle root hash to use in the header to then repeatedly change and hash to find the desired hash.

Hence when the next block is generated, the coinbase transaction of the ‘transaction sequence’ of the previous block is the first one, despite the fact, that chronologically it was the “last one”.

i'm not sure what you are asking here. the next block doesn't  care about previous block's coinbase transaction or its index inside the block. there is no chronology either, all tranasctions in that block are the same and the only thing Bitcoin cares about is that the first translation aka coinbase can't be spent until it reaches maturity.

Why is that? How many hash calculations are required / incrementing the nonce with one, or put it differently: How many hashing steps are required to make 1 proof of work attempt, regardless of success.

i believe the hashrate reported by ASICs is the speed at which the machine computes hash of the header at each nonce not each round of SHA256. that means computing SHA256(SHA256(80byte)) or 3 SHA256 block compressions.
each round the nonce is changed (0, 1, 2, ..., 0xffffffff=4,294,967,295) and the hash is computed again.

4) ExtraNonce field.:
Question: Is that logic correct?:

yes.

    - the version (self-evidently)
    - the previous proof of work (632b33406d89309e916545870727cabf498631a3a135230f0000000000000000)
    - the timestamp (they are twin miners, so both of them reached the nonce level ‘1427512000’ at the same time)

out of the 6 block header fields only 2 must be fixed for all miners: previous block header hash and the target.
the rest are flexible:
version has to be bigger than a certain value (changing with upgrades to the protocol)
merke root hash is always different for each miner (or pool of miners) since it is always calculated based on different set of transactions in that block
time has to be accurate with a 2 hour room for mistake
nonce can be anything and most probably is different even if 2 miners succeed at finding the same block.

Is it obligatory to start the computation with a nonce field of 0?

no.

[nc50lc]

4) ExtraNonce field.:

That would mean, that the parts of the merkle tree has to be recalculated. Not the entire structure though, because by changing the extraNonce, only the coinbase transaction has to be changed. Yellows are the changes which changes with the new coinbase transaction.:

Question: Is that logic correct?:

Yes, only the yellow ones - but that's if your miner is capable of keeping a cache of those previous "green" hashes whenever it needed to be concatenated with the changed hashes.
Because in order to get the "yellow" hash of the next level, it will still need the paired "green" hash of the previous level; if your miner previously discarded those "green" hashes, it'll have to compute them again.

5) Same nonce value, different miners.:
Let’s assume the existence of two ‘twin’ miner, which means that they have started the actual proof of work process together and have the same hashing power.:

-snip-

Question: Is that correct?

Yes, that must have happened or may happen at one point but I can't find the logical reason for asking this specific question.
If the block header is valid and its hash is lower than the target, then there's no problem.

[ignacio_buendia]

Thank you for the super helpful answers! However the estimated hash amount / 1 attempt remain somewhat chaotic.:


The logic of hashings in the case of 1 proof of work attempt.:

Statements:

1) hashing a block with 1 transaction takes exactly the same amount of effort as hashing a block with 10,000 transactions. Source: https://en.bitcoin.it/wiki/Block_hashing_algorithm

2) The preparation of the merkle tree - hashing the transactions into a block - is also the task of a miner.

3) When an ASIC is working at 15 TH/s, it means that it make 9,000,000,000,000,000 hashes in ten minutes.

4) The average nonce value of the previous 16 blocks is: 1,845,583,671, with a minimum value of 107,730,458 and a maximum value of 3,930,999,388. (I dig deeper, but the pattern shows, that the nonce falls between this rough interval.)


Conclusions:

i) In practice the miners calculate the nonce from 0.

that means computing SHA256(SHA256(80byte)) or 3 SHA256 block compressions.

ii) If 1-3 hash means 1 attempt of proof of work, then one miner with a 15 TH/s mining power has to find the golden nonce in a tiny fraction of a second which is impossible, hence the following possibilities:


Possible explanations:

A) The hashing process starts with constructing a merkle tree and after every attempt the merkle recalculation has to start over. Seems logical, but that would mean, that the statement 1) is false.

B) After using up the original 4 byte size of the nonce field, the usage of the extraNonce field starts to cause the coinbase transaction to change. And after that threshold every attempt require to recalculate the entire merkle structure. That wouldn’t make the statement 1) entirely false, only inaccurate.

C) A miner can work on a proof of work simultaneously.

D) After 1 hash, there is a vast amount of processing - or any other derivation work - , which a miner has to done. Knowing the simple nature of ASIC devices, it is a very unlikely scenario, but couldn’t be ruled out using simple logic.

E) Every time the timestamp - or anything else in the header - is modified, the nonce can be restarted and can be counted from 0 again.

F) The vast majority of valid proof of works are made by large mining pools. There is a secret mechanism behind the administration of community mining which in beyond my current knowledge.


As I was writing I was almost certain, that the E) is the correct answer. That would also explain the somewhat deterministic and narrow range of the nonce values. However it would lead to other questions.:

Q1: At what rate the timestamp usually changes in the block header?

Q2: Is it possible that the recalculation of a merkle tree causes the change in the header? Because there is 2 hour ‘concession’ it is also very unlikely that timestamp is frequently (within the 10 minutes) modified.

Q3: If not the timestamp, than the recalculation of the merkle root is the main cause of the narrow range of the nonces. Then explanation A) can be also true, right?

Q4: Do you have any source which illustrated the working principle of an ASIC miner?

Yes, that must have happened or may happen at one point but I can't find the logical reason for asking this specific question.

I tried to see the mechanics of the entire process in a different perspective. The only reason was complete understanding.

[pooya87]

1) hashing a block with 1 transaction takes exactly the same amount of effort as hashing a block with 10,000 transactions. Source: https://en.bitcoin.it/wiki/Block_hashing_algorithm

this means whether you have 1 tx or 10k txs in the end to find the correct block hash for proof of work you are still hashing the same fixed size byte (the 80-byte block header) instead of hashing the entire block for example.

A) The hashing process starts with constructing a merkle tree and after every attempt the merkle recalculation has to start over. Seems logical, but that would mean, that the statement 1) is false.

it is not false as i explained above but another way of looking at this is that the mining process (hashing the block header) and computing merkle tree are performed by 2 separate threads and nowadays it is even on 2 different machines.
one is the ASIC that is only hashing the header
another is the node on a computer using its CPU to verify transactions, construct the blocks, update blocks, handle memory pool, compute merkle roots,...

additionally we can't really call computation of merkle root "effort" in comparison to computing the block header hash. that is at worst a couple of thousand hashes with a lot less when updating it versus 4.2 billion hashes for each round.

B) After using up the original 4 byte size of the nonce field, the usage of the extraNonce field starts to cause the coinbase transaction to change. And after that threshold every attempt require to recalculate the entire merkle structure.

not the entire structure but only parts of it as you already posted in your OP and that google doc.

Q1: At what rate the timestamp usually changes in the block header?

it is up to the miner but it should be every couple of seconds.
basically the miner has the choice to change the fields i mentioned above. some are even playing with the version field and change that (that could have other reasons but it would go off-topic) but generally they work with nonce then extra nonce and update time every couple of seconds.

[gmaxwell]

Yes, only the yellow ones - but that's if your miner is capable of keeping a cache of those previous "green" hashes whenever it needed to be concatenated with the changed hashes.
Because in order to get the "yellow" hash of the next level, it will still need the paired "green" hash of the previous level; if your miner previously discarded those "green" hashes, it'll have to compute them again.

FWIW,  Bitmain S9 (and presumably later devices) has the controller upload the coinbase transaction and the log2(txn)  left-side-of-the-tree hashes to the FPGA so the fpga can roll the extra-nonce on its own.

It's unclear to me exactly how much of the motivation there was just being able to use lots of hash boards on a really slow control processor vs because it was required for the fpga to do covert asicboost... but regardless, thats how that works.

It's also kind of a bummer because it makes it much more likely some change in bitcoin behaviour will break a ton of miners.

[NotATether]

A) The hashing process starts with constructing a merkle tree and after every attempt the merkle recalculation has to start over. Seems logical, but that would mean, that the statement 1) is false.

it is not false as i explained above but another way of looking at this is that the mining process (hashing the block header) and computing merkle tree are performed by 2 separate threads and nowadays it is even on 2 different machines.
one is the ASIC that is only hashing the header
another is the node on a computer using its CPU to verify transactions, construct the blocks, update blocks, handle memory pool, compute merkle roots,...

Are there not ASICs or at least design ideas that are specialized for computing merkle roots and other adaptations for each functionality you mentioned above, to skip the overhead of CPU? the idea of ASICs works very well with computing SHA256d hashes and creating proofs of work.

[pooya87]

A) The hashing process starts with constructing a merkle tree and after every attempt the merkle recalculation has to start over. Seems logical, but that would mean, that the statement 1) is false.

it is not false as i explained above but another way of looking at this is that the mining process (hashing the block header) and computing merkle tree are performed by 2 separate threads and nowadays it is even on 2 different machines.
one is the ASIC that is only hashing the header
another is the node on a computer using its CPU to verify transactions, construct the blocks, update blocks, handle memory pool, compute merkle roots,...

Are there not ASICs or at least design ideas that are specialized for computing merkle roots and other adaptations for each functionality you mentioned above, to skip the overhead of CPU? the idea of ASICs works very well with computing SHA256d hashes and creating proofs of work.

i believe the modern ASICs have the capability to also compute the merkle hashes, but it still is going to be a very small "effort" in comparison to the mining itself.

[ignacio_buendia]

"...one is the ASIC that is only hashing the header
another is the node on a computer using its CPU to verify transactions, construct the blocks, update blocks, handle memory pool, compute merkle roots,...

additionally we can't really call computation of merkle root "effort" in comparison to computing the block header hash. that is at worst a couple of thousand hashes with a lot less when updating it versus 4.2 billion hashes for each round."

That was the exact answer I was looking for! Much appreciated.

Last questions / request:

that is at worst a couple of thousand hashes with a lot less when updating it versus 4.2 billion hashes for each round.

Q1: Could you please explain this?

Q2: I would like to buy a book to understand these nuances. Can you recommend one?

[pooya87]

that is at worst a couple of thousand hashes with a lot less when updating it versus 4.2 billion hashes for each round.

Q1: Could you please explain this?

it is just the total number of nonces (from 0x00000000 to 0xFFFFFFFF) which is about 4.2 billion versus the total number of transactions in a block which is about 2000-3000 that are used in computing merkle root and since in each round you only compute part of the tree it won't be more than a couple of thousands even if some transactions change in the block.

[ignacio_buendia]

Clear, thanks!

[TheArchaeologist]

Q2: I would like to buy a book to understand these nuances. Can you recommend one?

This is an excellent book to get you covered: https://github.com/bitcoinbook/bitcoinbook. It's for free on github or if you want you can buy it in print.

[ignacio_buendia]

Both of the recommendations looks promising, I surely check them out!