Hey everyone, the site's gonna be down for a few hours while I import some more math into my math db.

It's low on math.

I'm also adding an index to the `input` column of my sha256_hashes table which should speed things† up.

† The math

/cringe

OP can't even be bothered to write an original debut post on top of all that.

Can only imagine their source code must be one big quilt stitched together from hotscripts.com snippets and Stack Overflow answers.

You definitely want to stay away from customers that aren't even willing to spend $5 on a Digital Ocean VPS.

Nice to see it finally launch, man.

There are more variable fields than just `nonce`.

timestamp, "extra"-nonces, and the selected combination of transactions in the block are also variable.

View it in hexadecimal for a better representation of the data.

If bootstrap.dat is a concatenation of what you find in block.dat files, then the data is a repeating sequence of:

• Magic bytes: uint32 little endian
• Byte-count of the block: uint32 little endian
• Block: BlockCodec

Good stuff.

The reference Bitcoin implementation obfuscates the core simplicity of transaction validation by oddly referring to scripts as "scriptPubKey" and "scriptSig" instead of "inputScript" and "outputScript".

Realizing that "owning bitcoin" just means that you can provide a stack of instructions that evaluates to true when appended to another arbitrary instruction stack is what inspired me to hack on Bitcoin in the first place.

Some feedback:

• It's worth pointing out that "true" is really just "any non-zero value" rather than a boolean literal. In other words, a script is logically valid as long as the top stack value is not 0 and as long as the evaluation isn't short-circuited by an instruction like OP_RETURN.
• The significance of instructions like OP_RETURN and OP_VERIFY is that they short-circuit the evaluation. Otherwise, the script will evaluate in full and the final top stack value will determine the state of the stack. That's why you can append arbitrary data after OP_RETURN -- it's so that the data won't be evaluated.

You basically want to parse the blockchain into a database like MySQL/Postgres so that you can answer that question by executing a query against it.

For example, I parsed it into a database called Datomic and that query looks like this:



The blocks are stored in the blkXXXXX.dat files found in the ~/.bitcoin directory.

Each blkdat file is a sequence of block data where each block can be parsed like this:

• Magic bytes: uint32-le
• Byte-count of the block: uint32-le
• Block: BlockCodec

The file may be padded at the end with null-bytes. You also need to know that blkdat files are not a validated blockchain, just the blocks that bitcoind received over the network. The easiest thing to do is just add all blocks to the database and assume that the longest chain of `latestBlock->prevBlock->prevBlock->...->genesisBlock` is the one you want to query.

Database schema looks something like this:

- Block has many Txns
- Block has one PreviousBlock
- Txn has many TxOuts
- Txn has many TxIns

The pseudocode for it all is pretty simple and parsing is straight forward.

Word of warning: I went so far as to get a local blockexplorer working (https://github.com/danneu/chaingun), but of course I wandered off into the dragon lair of reimplementing blockchain validation and ran out of free time. It became evident that I was in fact climbing a large mountain when I passed by Jon Krakauer's corpse frozen in the snow. At which point I realized I wasn't climbing a mountain at all but slowly sliding down one slippery slope. And when I reached the bottom and turned back around, I realized it wasn't even a slope but instead a heap of my own yak shavings.

Well, the reality is that we're forced to reverse-engineer the specs as with most software since it unfortunately wasn't pre-engineered by a liability-hardened steering committee but rather evolved from a buggy ad-hoc pseudonymous side project by a few volunteers.

Most of the topical work has been done, but we're left with edge-cases and idiosyncrasies and arbitrary implementation details with a high cost for reimplementing it wrong.

But what's new?

If anybody really cared to ever tune up the state of documentation in software projects, then documentation wouldn't be persistently relegated to a thankless chore dependent on the goodwill gruntwork of the few volunteers capable of writing it.

I found it easiest to naively commit blockdats into the database as I parse them.

The simplest way to derive the longest branch for me was to represent it as a function applied to the tree which returns a sequence of block entities on the branch. Then I can just cache it, intersect it with blocks to see if they're on it, or apply it to any sub-tree (like the tree at 50 confirmations) when I append new blocks.

Output scripts only happen to usually conform to some common footprints that let you extract addresses and pubkeys, so just store the binary blob in the database which I'm sure you're doing. If I recall correctly, one of the few assumptions you can make about blockdat data is that the reference client does ensure it parses into op-codes, so at least there's that!

The road to robustness is definitely a humbling journey. Hilarious, even.

Keep us updated. I have a lot gotchas etched into comments around my codebase, but "surely I'll blog about them someday!" turns out only to be a compulsive lie I tell myself to feel benevolent.

Here're my Java crypto:


Are you sure you're working with the actual bytes that the hex represents and not, for example, the bytes of the chars that make up the hex string?

Apparently bitcoind writes blocks to blk*.dat files so long as they demonstrate sufficient proof-of-work. Beyond that, you're left to validating the transactions and discerning the most valid chain yourself.

http://bitcoin.stackexchange.com/questions/17801/does-bitcoind-download-bad-blocks/17804#17804

Note: Recreating the reference client's blockchain validation is the most nebulous part about implementing bitcoin. And the reasons why it's hard range from the lack of hand-holding documentation, the bugs that you must recreate, and - most particularly - the implications of getting it all wrong.

First of all, I'd start at the hardest part first: https://en.bitcoin.it/wiki/Script, which will have you implement https://en.bitcoin.it/wiki/OP_CHECKSIG. It will also have you implement DER encode/decode and the menagerie of utility functions you'll need along the way.

Once you're at a place where you can (valid-tx-input? x) and it will (eval-script (concat input-script output-script)), return a boolean, and handle OP_CHECKSIG - even if it's a naive implementation - then you're making good progress.

My naive approach for handling the most-valid-branch of the blockchain in the simplest way is to always append incoming blocks to my blockchain db if they validate (inputs validate, difficulty is correct, prev-outputs are unspent on this branch, block-hash is correct, etc). A fork is represented when multiple blocks point back to the same prev-block. For example, there may be multiple blocks at depth 42.

I calculate (and cache) the most-valid-branch (branch with hardest summed difficulty) with a function that I apply to the genesis block. It returns a set of those blocks in the most-valid-branch. So if Tx ABC exists in two blocks at depth 42, I need a way to get the most valid one, and it's just a matter of intersecting its parent-block with the most-valid block set.

Just know that in a couple months you might find yourself going "What the fuck am I doing?", but that's just weakness leaving the body.

blk*.dat format is a repetition of:

- magic (uint32-le)
- size  (uint32-le) - bytes count of following block
- block (https://en.bitcoin.it/wiki/Protocol_specification#block)

The block spec points out that `bits` is a uint32-le.

`bits` represents difficulty in compact form and you can see how to expand it here https://en.bitcoin.it/wiki/Difficulty.


The wiki (https://en.bitcoin.it/wiki/Protocol_specification) is pretty comprehensive.

Disregarding all other problems with your proposal, you're just passing the buck to your "secondary" private key.

If you're so careful with your "secondary" private key, then why wouldn't you just be that careful with your "primary" private key? Why play it so loose at all?

Why would you wait out some arbitrary self-invoked penalty on your business transactions when you can sign them with your offline super secure "secondary" key and broadcast them immediately?

In other words, this sounds like indirection that leads right back to the same fatal crux you started with: securing a private key.

The difference between bitcoin and a $1.99/min phone-call is that your phone number is already linked to you through a central service. Your bank account is attached to it.

And if you're using a prepaid phone or a burner, then you can't call 1-900 numbers and our analogy is back to square one.

So when you say this:


It sounds like you're expressing some sort of "reverse address lookup", but I'm not convinced that you actually want that.


No, your original question was "I'm building <X> because [I think] it's how to solve <Y>. Help." Yet you only seem interested validating your assumptions instead of reconsidering them.

The bottom line is that, in the examples you mention, you'd control some sort of rendezvous point for your customers.

A simple example would be a form on a checkout page where they type in their shipping address and it generates a unique bitcoin address for them to pay to. Any time a form is submitted, you insert into your database (A) the user info, (B) the bitcoin keys that you'll start watching, and (C) the order so you know how to fulfill. From there you could let them generate a username and password just like every other ecommerce website.


It's no different than having to actually go to amazon.com to buy their products, or click a google result to go to a website, or type in buyhatsforcats.com from a newspaper ad to buy your cat a hat.

Anyways, the best way to get help is to outline exactly what you actually want because that's the critical info that this thread is missing.

Here's my code. It's a function named `->address` that accepts `pubkey` data, concats together one sequence of `version+hash160+checksum` bytes, and base58 encodes it.


The version-byte is actually determined by the network:


Also, if you base58-encode a byte-array with leading 0-bytes, each 0-byte gets converted into a "1".

From bitcoin's test suite, "00000000000000000000" (hex) gets base58-encoded into "1111111111" (string).

In general, if the wiki doesn't help you, then check out any of the 50 partial bitcoin implementations and cross-examine them.

I don't think you're going to find a simpler one-stop-shop implementation of curve arithmetic than vbuterin's code.


I can't tell if you're referring to http://en.wikipedia.org/wiki/Elliptic_curve_point_multiplication or if you're confusing point-multiplication with `*` in `6 * 2 = 12`.

`base10_multiply` is recursively applying the `base10_add` function. It's all pretty much laid out there for you, although it helps to know that `inv` is an inverse mod function.

If you're having difficulty understanding the math, then it probably means you need to zoom out and get a higher level understanding of elliptic curves and some dependent nodes along that talent tree. If you're using Mathematica, then surely there are examples of curve math out there already described for you in the semantics of Mathematica so you don't have to translate Python.

If an honest peer sees that you're on a weaker fork, it responds by catching you up with the strongest fork that it knows.

So if you possess zero verification ability because you only have a library that implements the wire protocol, I guess you're left to hoping that you're connected to mostly honest nodes and can cross-examine their messages to infer a consensus.

https://en.bitcoin.it/wiki/Thin_Client_Security

It appears to only implement the wire protocol (sending/receiving messages).