Fast parallel block validation without UTXO-index

[gmaxwell]

This in contrast to Core's model which needs sequential reads and writes to the UTXO index, even if all scripts are already verified.

This is not how Bitcoin Core works. No access to the disk normally at all happens when a block is verified except for transactions that have not yet been seen, reading or writing.

I am not talking about disk reads and writes, I am talking about UTXO index reads and writes.

There are no UTXO index reads or writes in the normal case (transaction recently seen.) in Bitcoin Core.

[1713296653]

1713296653

[1713296653]

1713296653

[piotr_n]

I think what tomtomtom7 is trying to say is that the common approach (used not only by core) is that even though the node verifies the transactions before a block is announced, the actual moment when the UTXO database is updated (and the changes committed to it)  is during the block's validation.

Whilst his solution updates (appends)  TXO database while verifying a transaction in the first place. Later,  during the block's validation, it's just logging  the information of which of the outputs from the TXO database has been spent by the block.

[gmaxwell]

I think what tomtomtom7 is trying to say is that the common approach (used not only by core) is that even though the node verifies the transactions before a block is announced, the actual moment when the UTXO database is updated (and the changes committed to it)  is during the block's validation.

That is what I understand him to be saying, but it isn't true for Bitcoin Core. For Core, the moment when the UTXO database is updated/and changes committed is _not_ during block validation.  It's after in separate batch steps which apply big batches of changes at a time; specifically to mostly get interacting with the database out of the critical path.

This is also why if you shut your node off uncleanly you'll see it going and reprocessing dozens of blocks-- their effects hadn't been applied to the database yet.

[cryptoanarchist]

I think what tomtomtom7 is trying to say is that the common approach (used not only by core) is that even though the node verifies the transactions before a block is announced, the actual moment when the UTXO database is updated (and the changes committed to it)  is during the block's validation.

That is what I understand him to be saying, but it isn't true for Bitcoin Core. For Core, the moment when the UTXO database is updated/and changes committed is _not_ during block validation.  It's after in separate batch steps which apply big batches of changes at a time; specifically to mostly get interacting with the database out of the critical path.

This is also why if you shut your node off uncleanly you'll see it going and reprocessing dozens of blocks-- their effects hadn't been applied to the database yet.

I don't understand the specifics of the code, so can you enlighten me? What you're saying sounds like semantics: "It's after in separate batch steps" sounds like it is still done when a block validates, just maybe not part of a certain function. I mean, it has to be done before the next block anyway. Is the UTXO updated before the node officially validates the block??

It sounds like this new way validates the TX as they happen, and this saves time later. I wish people could be a little more specific in a tech sub-forum -like with code snippets.

[gmaxwell]

I don't understand the specifics of the code, so can you enlighten me? What you're saying sounds like semantics: "It's after in separate batch steps" sounds like it is still done when a block validates,

It's not semantics. When validating blocks the vast majority of the time the UTXO set is not used by bitcoin core; because the UTXOs are in the cache already (due to mempool) and nor are the result of the block's changes written out-- they're buffered until the node flushes-- and very often never get written at all because the newly created outputs are spent by a later block before the system flushes them.

OP suggests that the structure he proposes reduces the delay at the time a block is processed compared to Bitcoin Core's UTXO database, but it shouldn't-- at least not for that reason-- because the database is only used at block processing times in the exceptional case where a transaction shows up by surprise.

[tomtomtom7]

I don't understand the specifics of the code, so can you enlighten me? What you're saying sounds like semantics: "It's after in separate batch steps" sounds like it is still done when a block validates,

It's not semantics. When validating blocks the vast majority of the time the UTXO set is not used by bitcoin core; because the UTXOs are in the cache already (due to mempool) and nor are the result of the block's changes written out-- they're buffered until the node flushes-- and very often never get written at all because the newly created outputs are spent by a later block before the system flushes them.

OP suggests that the structure he proposes reduces the delay at the time a block is processed compared to Bitcoin Core's UTXO database, but it shouldn't-- at least not for that reason-- because the database is only used at block processing times in the exceptional case where a transaction shows up by surprise.

This is semantics. I am aware that Core uses a cache to make most UTXO read and writes faster, but I would still call them UTXO read writes (or CoinView read/writes?), and as I understand, they still need to be handled sequentially.

Any reasonable database will attempt to use RAM for frequently used items, regardless of whether this is managed by the application, by a dedicated database component or the OS.

[gmaxwell]

This is semantics.

It isn't-- because the speedups you conjecture creating already exist.

You propose that your design moves accessing the UTXO data out of the critical path for block validation, on the basis that you will have already validated the transactions and so don't need to access that data.

Bitcoin Core does not have UTXO accesses in the critical path in the same case and doesn't write in the critical path either. So it would not see the benefit you expect for this reason. (not that it wouldn't potentially have benefit, just not the benefit you are suggesting at least not for the reason you suggest)

Cheers