Bitcoin Blockchain

[tf10]

Hi, I wanted to ask if my understanding of how bitcoin works was correct.

A transaction is made
The transaction gets sent across the miners network
The miners validate that transaction and it gets added to the transactions pool
From there some miner will pick up that tx and add to his block
Once the miners block is filled he will start calculating the value for nonce required to validate that block
Once nonce is achieved, the block gets added to the blockchain
In the meantime, if someone else managed to publish a new block then this block is cancelled and the miner then again picks up new transactions from the tx pool.

Is this all right? Or are there any additions to it?

[HeRetiK]

Close, but not quite.

The miners validate that transaction and it gets added to the transactions pool
From there some miner will pick up that tx and add to his block

There's no such thing as a global pool of transaction, each miners keeps their own mempool. They will be mostly the same, but may differ to due differences in mining software, configuration and network connectivity.

Transaction validation doesn't happen until the miner tries to build a block.

Once the miners block is filled he will start calculating the value for nonce required to validate that block

Miners don't care about filling blocks.

Sure, they try filling them as good as possible as to reap transaction fees. But validating transactions takes time and if they happen to find a valid nonce before the block is fully filled they just sent the block to the other nodes in the network, lest some other miners beats them to it. Keep in mind that miners always keep hashing in the background, even as they are just collecting and validating transactions for a new block.

Once nonce is achieved, the block gets added to the blockchain
In the meantime, if someone else managed to publish a new block then this block is cancelled and the miner then again picks up new transactions from the tx pool.

Each miner keeps their own copy of the blockchain. So they add the block to their copy of the blockchain and then tell the rest of the network about the update. Usually their blockchain is the longest one, because they just added a new block. As such, all other miners follow suit.

However if for some reason they missed a couple blocks, their copy of the blockchain will be shorter than the copies of the rest of the network, despite the newly added block. In this case they are out of luck and have to follow the longest blockchain, losing their block reward as their copy of the blockchain is no longer valid. This also makes each transaction this miner had put in their block unconfirmed, unless some other miner already included it in a block (for example the one that our miner missed).

If that's the case, then the miner drops their copy of the blockchain in favour of the longer one and starts building a new block, picking up and validating transactions from their own copy of the mempool as they go, just as mentioned above (ie. not really caring about making the block full and when in doubt releasing an only partially filled block).


TLDR; Miners don't necessarily fill blocks to its maximum because they validate as they go; Each node has their own copy of the mempool and the blockchain as there is no central transaction pool to pick up new transactions from.

[tf10]

Thank you for the reply man. I just have one more question. Every individual miner is responsible for validating the transaction that they include in their block, right? How do they do that? How do they make sure that a tx is valid? And not insert fake transactions of their own in there?

[jnano]

The rules are known to everyone, and are also based on the previous content of the blockchain. Each miner or user on the network can validate transactions. That's one of the core ideas: everyone can validate everything, and many do.

Invalid transactions in the mempool will be rejected and won't be relayed further.

Invalid transactions in a block will lead to miners ignoring it and continuing to build the blockchain as if that bad block did not exist. Whoever sent that block into the wild will have gained nothing.

[DannyHamilton]

A transaction is made

So far so good.

The transaction gets sent across the miners network

The miners don't have their own network.  It's the same network that everyone else uses.

The transaction sender sends the transaction to the nodes he is connected to.  Those nodes validate the transaction. If it is invalid, they ignore it and it doesn't get any farther. If it is valid, then they add it to their own mempool, and then relay it to the nodes that they are connected to.  Those then validate the transaction. If it is invalid, they ignore it and it doesn't get any farther. If it is valid, then they add it to their own mempool, and then relay it to the nodes that they are connected to.  And so on.  Eventually, some of the connected nodes that hear about the transaction are solo-miners or mining pools. They also validate the transaction. If it is invalid, they ignore it and it doesn't get any farther. If it is valid, then they add it to their own mempool, and then relay it to the nodes that they are connected to.

The miners validate that transaction and it gets added to the transactions pool

Not just the miners.  Everyone that is running a full node validates the transaction and adds it to their own mempool if it is valid before they relay it to anyone else.

From there some miner will pick up that tx and add to his block

Solo-miners and mining pools select unconfirmed transactions from their own mempool.

The protocol allows the solo-miner or mining pool to use any criteria they like to add transactions to their blocks.  They can choose to include the transaction, or they can choose to ignore it if they want. Since most miners and mining pools are profit motivated, they will continue to add transactions to the blocks that they are working on so long as there is EITHER still room in the block for more transactions, OR they find a new transaction that pays a higher fee per weight unit than the lowest fee transaction currently in their block (in which case they will remove that low fee transaction to make room for the higher fee transaction.

Once the miners block is filled he will start calculating the value for nonce required to validate that block

The miners are pretty much ALWAYS calculating the value for the nonce required.  If they finish a block, or hear about a finished block, then they immediately start working on the next block. They may add transactions, remove transactions, change the timestamp, or modify the extranonce as needed while they are at it, but they never stop working to wait for transactions.

Once nonce is achieved, the block gets added to the blockchain

They will immediately add it to their own blockchain, and they will relay it to the nodes that they are connected to.  Each of those nodes will validate the block, and if it is invalid they will ignore it and it won't get any farther.  If it is valid, then they will add it to their own blockchain and will relay it the nodes that they are connected to.  Each of those nodes will validate the block, and if it is invalid they will ignore it and it won't get any farther.  If it is valid, then they will add it to their own blockchain and will relay it the nodes that they are connected to. And do on.

In the meantime, if someone else managed to publish a new block then this block is cancelled and the miner then again picks up new transactions from the tx pool.

Whenever a miner receives a block from a peer that they are connected to, they check to make sure it is valid.  If it is valid, then they add it to their own blockchain, relay it to all the nodes they are connected to, and start a new block that uses the block they just received as the "previous block hash".

Transaction validation doesn't happen until the miner tries to build a block.

That is not true.  A miner (or any other node) will NOT add an invalid transaction to their own mempool, nor will they relay it to anyone else.  Transaction validation happens as soon as the transaction is received. The miner does NOT wait until he is ready to use the transaction in the block before he validates it.

Every individual miner is responsible for validating the transaction that they include in their block, right?

Every full node is responsible for validating EVERY transaction AND EVERY block.  A node will not accept into their own mempool or relay an invalid transaction.  A node will not accept into their own blockchain or relay an invalid block.

Because of this fact, it would be VERY foolish for a solo-miner or mining pool to include a transaction into their own block without having validated it.  If they do, they risk wasting a lot of time, money, and resources creating an invalid block that will be rejected by the entire network.

How do they do that? How do they make sure that a tx is valid?

They check that each of the inputs exists and is not yet spent.
They check that all of the scripts process properly.
They check that the sum of the outputs does not exceed the sum of the inputs.
They check that the formatting is all correct.
They check the locktimes.

Invalid transactions in the mempool will be rejected and won't be relayed further.

Invalid transactions will not even make it into a node's mempool in the first place.  They are rejected before they get that far.

[HeRetiK]

Transaction validation doesn't happen until the miner tries to build a block.

That is not true.  A miner (or any other node) will NOT add an invalid transaction to their own mempool, nor will they relay it to anyone else.  Transaction validation happens as soon as the transaction is received. The miner does NOT wait until he is ready to use the transaction in the block before he validates it.

I stand corrected, thank you for clearing this one up. Come to think of it, it does make more sense and makes the network less prone to DOS attacks.

Do you by any chance know which part of adding transactions to a block the time consuming one is? That is, the reason why sometimes new nonces are quicker found than transactions added? Is it because transactions need to be-rechecked to ensure that they are still valid, ie. not spent by the previous block that has just been freshly found?

[DannyHamilton]

Do you by any chance know which part of adding transactions to a block the time consuming one is?

When a mining pool first receives a block from a peer, they need to make sure that the block is valid.  That means that they need to check every transaction in that block to make sure they are all valid, they need to generate the merkle root, to make sure it is correct, they need to check the timestamp, previous block hash, and the difficulty in the block header, and they need to hash the block header to make sure that the result is lower than the current difficulty target. Then they need to remove all the transactions that are in the block from their mempool so that they don't accidentally try to include any of them in the next block they build, and they need to update their UTXO with any new inputs or outputs that are included in the block which they've never seen before. During all of this, any transactions in the mempool which are invalidated by a transaction in the block also needs to be removed from the mempool.

Once they've done that, they can create new work for the pool participants.  Rather than have all the participants sit idle while all this is going on, the pool will typically prepare a set of empty blocks and send the work for those out to the pool participants.  That way they'll have something they can work on while the pool is getting ready to build new blocks with transactions. 

[HeRetiK]

Thank you for the detailed response Danny, very much appreciated.

When a mining pool first receives a block from a peer, they need to make sure that the block is valid.  That means that they need to check every transaction in that block to make sure they are all valid, they need to generate the merkle root, to make sure it is correct, they need to check the timestamp, previous block hash, and the difficulty in the block header, and they need to hash the block header to make sure that the result is lower than the current difficulty target.

Validating the freshly mined blocks from the other miners is the step that I completely overlooked.

What happens if a new block is found before the previous block has been validated? Obviously the block will be empty, as transactions have yet to be re-checked, but will a miner speculatively relay the new block, hoping that the previous block is indeed valid? Basically a hail mary of sorts? Or is this a case where a miner will refrain from relaying the new block until they have validated the previous block?

Then they need to remove all the transactions that are in the block from their mempool so that they don't accidentally try to include any of them in the next block they build, and they need to update their UTXO with any new inputs or outputs that are included in the block which they've never seen before. During all of this, any transactions in the mempool which are invalidated by a transaction in the block also needs to be removed from the mempool.

...and between validating the previous block and updating the local mempool, empty to near-empty blocks happen, got it.

(ignoring other potential implementation / configuration / connectivity issues that can lead to near-empty blocks)

[DannyHamilton]

What happens if a new block is found before the previous block has been validated? Obviously the block will be empty, as transactions have yet to be re-checked, but will a miner speculatively relay the new block, hoping that the previous block is indeed valid? Basically a hail mary of sorts? Or is this a case where a miner will refrain from relaying the new block until they have validated the previous block?

The protocol doesn't attempt to enforce anything on that matter.

A pool could continue with the old blocks while validating the new block (taking a chance on wasting effort on a stale block if this new block is valid).
A pool could immediately mine on top of the new block while validating it (taking a chance on wasting effort on an invalid block if this new block is invalid).
A pool could switch to some other coin while validating the new block (possibly reducing the revenue if that other coin doesn't pay as well).
A pool could halt all hashing until is has finished validating the new block (wasting precious time during which they otherwise might have solved a block).
A pool could do any combination of the above.
A pool could come up with another idea I haven't thought of.

In the end, it's up to the pool operator to decide how he wants his pool to behave when receiving a new block.

Also, depending on how the pool is designed, this may not be a single threaded process.  The validation of the block, and the updating of a UTXO and mempool could be happening simultaneously across multiple processors.

If the pool (or solo miner) completes the proof-of-work on a block before they've finished verifying the received block, there is no benefit in waiting to broadcast.  If the block is good, then peers will accept it. If it is not good, then peers will reject it and it won't get very far. Avoiding broadcasting it just increases the chance that your block will be orphaned if it turns out to be good.

[butka]

The miners are pretty much ALWAYS calculating the value for the nonce required.  If they finish a block, or hear about a finished block, then they immediately start working on the next block. They may add transactions, remove transactions, change the timestamp, or modify the extranonce as needed while they are at it, but they never stop working to wait for transactions.

The nonce, as I understand it, is just a random number added to the input of the hashing function trying to produce a hash of desired properties.

While working on the next block, the miner has to include the entire input, including a random nonce, the collected transactions, and the other necessary data to produce the output (the hash).

If the miner starts computing the hash for the next block from the very beginning, could it happen that they produce an empty block or a block nearly void of transactions?

Sorry about my ignorance, but this is something that has been puzzling me all the time.

[DannyHamilton]

The nonce, as I understand it, is just a random number added to the input of the hashing function trying to produce a hash of desired properties.

The block header is 80 bytes long.  The nonce is the last 4 bytes of that header.

While working on the next block, the miner has to include the entire input, including a random nonce, the collected transactions, and the other necessary data to produce the output (the hash).

The solo-miner (or mining pool) builds an 80 byte block header that includes:

• 4 byte version number
• 32 byte hash of the previous block in the chain
• 32 byte merkle root calculated from the transactions which were selected for this block
• 4 byte timestamp
• 4 byte indication of the current difficulty target
• 4 byte nonce

The transactions need to be hashed to generate the merkle root, but then only the merkle root is used as input for the mining hashing. The miner rapidly tries all possible nonce values until they find one that works.  If they've tried all possible nonce values, and they haven't found one that works, then they need to change something else in the header so they can try all the nonce values again with a new header. Some things in the header that generally be changed are:

• timestamp (as long as you don't exceed the allowable range)
• merkle root (by changing the transaction selection, or the coinbase transaction, a new merkle root can be calculated)

If the miner starts computing the hash for the next block from the very beginning, could it happen that they produce an empty block or a block nearly void of transactions?

Yes. It happens occasionally.

[butka]

The transactions need to be hashed to generate the merkle root, but then only the merkle root is used as input for the mining hashing. The miner rapidly tries all possible nonce values until they find one that works.  If they've tried all possible nonce values, and they haven't found one that works, then they need to change something else in the header so they can try all the nonce values again with a new header.

This explains a lot! Thank you very much, I appreciate the clarification.

[DannyHamilton]

If they've tried all possible nonce values, and they haven't found one that works . . .

Here are some fun and interesting facts...

A 4 byte nonce means that there are only 4,294,967,296 (nearly 4.3 billion) possible nonce values to try.  In the beginning of bitcoin, back in 2009 with CPU mining, it took more than 10 minutes for a single CPU miner to try all 4294967296 values, and the difficulty was such that with several CPU miners running at the same time throughout the world, there was a pretty good chance that one of the miners could find a solution without needing to change anything else in their block header other than the nonce.

However, the modern S9 ASIC miner can churn through 14,000,000,000,000 (14 trillion) hashes every second.  That means that a single S9 ASIC will churn through more than 3,255 different block headers every second.

The current global hash rate is estimated to be about  25,307,686,182,000,000,000 (25.3 quintillion) hashes every second.  That means that globally, the entire network is attempting approximately 5,892,404,863 (5.89 billion) different block headers every second.  Since it takes an average of 10 minutes to find block header that hashes to a value that satisfies the current difficulty, that means that globally an average of 3,535,442,900,000 (3.5 trillion) headers are attempted and discarded for every 1 block header where a solution is found.

[tf10]

Thank you for all the great information man. If I may just ask one more thing. When a transaction happens, how's the entry point to the bitcoin network figured out? Suppose, I make a transaction on my machine. This transaction is turned into a data packet, and sent to my machines default gateway. Where does it go to from there? Where are the packets from my machine routed to? Because Bitcoin is a decentralized network, there's no central point of entry( or is there? ) I'm just confused with how this part works

[jnano]

Peer discovery:
https://bitcoin.org/en/developer-guide#p2p-network

[pebwindkraft]

The transaction gets sent across the miners network

The miners don't have their own network.  It's the same network that everyone else uses.

Hint: there is the FIBRE network, a low latency network (bitcoinfibre.org). I am not sure if this is a specific miners network, but I think I read somewhere in a post, they use it a lot. From the webpage:

The Fast Internet Bitcoin Relay Engine (FIBRE) is a protocol and implementation designed to relay blocks within a network of nodes with almost no delay beyond the speed of light through fiber. Its design is based on several years' experience operating and studying the Bitcoin Relay Network and functions incredibly well even when faced with suboptimal internet conditions.

[ranochigo]

If I may just ask one more thing. When a transaction happens, how's the entry point to the bitcoin network figured out? Suppose, I make a transaction on my machine. This transaction is turned into a data packet, and sent to my machines default gateway. Where does it go to from there? Where are the packets from my machine routed to? Because Bitcoin is a decentralized network, there's no central point of entry( or is there? ) I'm just confused with how this part works

They are routed to your peers. With Bitcoin, there are loads of nodes that form the backbone of the network. They are basically the fundemental aspect of Bitcoin and without them Bitcoin wouldn't exist.

Once you send the transaction to your peers/nodes, they will verify the validity of your transaction and relay your transaction across the network. Same goes for miners who has mined a block. Every node is technically an 'entry-point' where you can send information to them for it to be broadcasted onto the network.

[hundi.online]

The rules are known to everyone, and are also based on the previous content of the blockchain. Each miner or user on the network can validate transactions. That's one of the core ideas: everyone can validate everything, and many do.

Invalid transactions in the mempool will be rejected and won't be relayed further.

Invalid transactions in a block will lead to miners ignoring it and continuing to build the blockchain as if that bad block did not exist. Whoever sent that block into the wild will have gained nothing.

Well explained!! Thanks a lot

[RodeoX]

Good question and good discussion. Merit for all! lol