Verifying blocks and transactions.

[Remember remember the 5th of November]

I am writing my own blockchain parser, I wanted to export the UTXOs, I asked around in the bitcoin-dev IRC channel, but my questions got ignored(sort of).

What must I do to verify the blockchain myself? I want to for instance verify that some transaction can actually spend some funds, or that the block in which this transaction is, is not an orphan block and is part of the longest difficulty-wise chain etc.
I was suggested to use "bitcoind", but I am writing my own parser, I want to be able to do these things independently.

[readerbtc]

It seems that you'll have to reproduce in your parser the same rules bitcoind implements. Maybe you are asking for a document describing those rules in detail, so you don't need to read the source code?

[DannyHamilton]

I was suggested to use "bitcoind", but I am writing my own parser, I want to be able to do these things independently.

Where are you getting the blockchain from?  Are you re-creating the communication protocol so you can connect to peers and acquire blocks from them, or are you simply going to supply your parser with a static blockchain file to parse?

[deepceleron]

The details of understanding are a lot less work than recoding, although the coding can be a path to deeper understanding.

Valid blocks are just the longest chain, it is easy to follow the hashes and the difficulties of blocks through and make the same decision about block difficulty. You'll have to use the full and exact rules that bitcoind does when examining every block and transaction though if you aren't using a vetted-by-bitcoin blockchain. If you have a network client that doesn't do any deep inspection, it could easily follow a low-difficulty, invalid rule chain. From the last fork, we see even the smallest details of implementation have to be replicated.

If you want to verify transactions and hold the UXTO set, then you need to build a database, and work through every transaction ever. You'll have to verify the ECDSA signature, and run and validate the scripts. Rules changed at particular blocks, you'll need to reproduce BIP checkpoints and the code that checks them.

You can trust what's in a saved-by-bitcoin blockchain and just parse it. The next step in complexity is called "rewrite Bitcoin".

[danneu]

I am writing my own blockchain parser, I wanted to export the UTXOs, I asked around in the bitcoin-dev IRC channel, but my questions got ignored(sort of).

What must I do to verify the blockchain myself? I want to for instance verify that some transaction can actually spend some funds, or that the block in which this transaction is, is not an orphan block and is part of the longest difficulty-wise chain etc.
I was suggested to use "bitcoind", but I am writing my own parser, I want to be able to do these things independently.

Note: Recreating the reference client's blockchain validation is the most nebulous part about implementing bitcoin. And the reasons why it's hard range from the lack of hand-holding documentation, the bugs that you must recreate, and - most particularly - the implications of getting it all wrong.

First of all, I'd start at the hardest part first: https://en.bitcoin.it/wiki/Script, which will have you implement https://en.bitcoin.it/wiki/OP_CHECKSIG. It will also have you implement DER encode/decode and the menagerie of utility functions you'll need along the way.

Once you're at a place where you can (valid-tx-input? x) and it will (eval-script (concat input-script output-script)), return a boolean, and handle OP_CHECKSIG - even if it's a naive implementation - then you're making good progress.

My naive approach for handling the most-valid-branch of the blockchain in the simplest way is to always append incoming blocks to my blockchain db if they validate (inputs validate, difficulty is correct, prev-outputs are unspent on this branch, block-hash is correct, etc). A fork is represented when multiple blocks point back to the same prev-block. For example, there may be multiple blocks at depth 42.

I calculate (and cache) the most-valid-branch (branch with hardest summed difficulty) with a function that I apply to the genesis block. It returns a set of those blocks in the most-valid-branch. So if Tx ABC exists in two blocks at depth 42, I need a way to get the most valid one, and it's just a matter of intersecting its parent-block with the most-valid block set.

Just know that in a couple months you might find yourself going "What the fuck am I doing?", but that's just weakness leaving the body.

[danneu]

You can trust what's in a saved-by-bitcoin blockchain and just parse it. The next step in complexity is called "rewrite Bitcoin".

Apparently bitcoind writes blocks to blk*.dat files so long as they demonstrate sufficient proof-of-work. Beyond that, you're left to validating the transactions and discerning the most valid chain yourself.

http://bitcoin.stackexchange.com/questions/17801/does-bitcoind-download-bad-blocks/17804#17804

Bitcoin will store a block on disk as soon as it has verified its proof-of-work.

Blocks are stored append-only in the block files, and never deleted or overwritten. You cannot judge from the block files which are considered valid, or which are considered the currently best chain - that information is kept in the block index database.