How does Bitcoin block index work?

[g2com]

Hi, I can find few information on Bitcoin's storage. So I understand that blkindex.dat is for the block index and blk00x.dat for transactions. However I don't understand how is this index generated. I understand an index file speeds up data lookup. However LevelDB states it doesn't support index so how is this index used along with LevelDB data? Also, a transaction is a nested structure that contains multiple inputs and outputs. LevelDB is a key-value storage engine so how does it recognize the hierarchy of each transaction?

[1715188717]

1715188717

[achow101]

The blk.dat file stores raw blocks, not just transactions. It only stores blocks and the transactions included within those blocks. AFAIK the index is an index of offsets for each block, basically where each block is located in which file and where in that file so that it can be read from the disk and processed if need be.

[g2com]

The blk.dat file stores raw blocks, not just transactions. It only stores blocks and the transactions included within those blocks. AFAIK the index is an index of offsets for each block, basically where each block is located in which file and where in that file so that it can be read from the disk and processed if need be.

So, the whole block is read into the memory when Bitcoin accesses it? How does Bitcoin lookup a transaction from a block file?

[achow101]

So, the whole block is read into the memory when Bitcoin accesses it?

I believe so.

How does Bitcoin lookup a transaction from a block file?

Bitcoin Core normally does not need to lookup transactions. It can only do that if you enable the transaction index (-txindex CLI option) in which case it does basically the same thing as for blocks, it makes the database and has each value point to the location in a file of where that transaction is located.

For transactions that are in your own wallet, it stores the transaction data in your wallet.dat file.

The databases also include other things like the current chainstate and the UTXO set.

[g2com]

I saw that the serialized transaction is just a lump of data with a key, so how does Bitcoin find transactions associated with a single private key? Does it need to travserse all unspent tx to get the total balance?

[g2com]

So, the whole block is read into the memory when Bitcoin accesses it?

I believe so.

How does Bitcoin lookup a transaction from a block file?

Bitcoin Core normally does not need to lookup transactions. It can only do that if you enable the transaction index (-txindex CLI option) in which case it does basically the same thing as for blocks, it makes the database and has each value point to the location in a file of where that transaction is located.

For transactions that are in your own wallet, it stores the transaction data in your wallet.dat file.

The databases also include other things like the current chainstate and the UTXO set.

I see. So both wallet.dat and chainstate store unspent transaction data, but only those in wallet.dat are spendable right?

[achow101]

I saw that the serialized transaction is just a lump of data with a key,

There is no key associated with a transaction.

so how does Bitcoin find transactions associated with a single private key? Does it need to travserse all unspent tx to get the total balance?

Yes. It goes through the UTXO set and finds which UTXOs can be spent by you.

I see. So both wallet.dat and chainstate store unspent transaction data, but only those in wallet.dat are spendable right?

Only those in the wallet.dat can be spent by you specifically. Nearly all the UTXOs can be spent by someone.

[g2com]

I saw that the serialized transaction is just a lump of data with a key,

There is no key associated with a transaction.

so how does Bitcoin find transactions associated with a single private key? Does it need to travserse all unspent tx to get the total balance?

Yes. It goes through the UTXO set and finds which UTXOs can be spent by you.

I see. So both wallet.dat and chainstate store unspent transaction data, but only those in wallet.dat are spendable right?

Only those in the wallet.dat can be spent by you specifically. Nearly all the UTXOs can be spent by someone.

Thanks for the answer. The key I meant the transaction hash, since you need to supply a key to retrieve a value.

[g2com]

I've looked into the Tx index. It contains three columns, nFile, nBlockPos and nTxPos. The last two variables are integers indicating where the transaction can be found in a file. I don't understand how does Bitcoin retrieve one transaction. Does the Bitcoin client read the entire index into memory, then create a map that establishes the transaction and its position?