Right, blocks themselves are certainly also compressible. A custom serializer could remove a lot of redundancy too, already: signatures in 64 instead of 72 bytes, public keys compressed to 33 bytes, not wasting 4 bytes for version, locktime, nsequence, ...


Sure, I just chose 1 file per block now because it was clean and easy to implement. Also for efficiency combining at least some blocks per file would be beneficial. It's two files per block right now even, actually, as the undo data is stored in a separate file.


Not necessarily. There are cashes at several levels, and when many files are created per second, I'm sure the OS batches them together in a single or only a few writes.


Sure, things like that are possible. Note that if they are really zip files, querying a block would require reading and parsing the ZIP index to locate the data. Just a binary concatenation of (potentially compressed) blocks, with the start byte position stored in the index is probably easier.

(cross-post from the mailing list)

Hello all,

I've implemented a new block/transaction validation system for the reference client, called "ultraprune".

Traditionally, pruning for bitcoin is considered to be the ability to delete full transactions from storage once all their outputs are spent, and they are buried deeply enough in the chain. That is not what this is about.

Given that almost all operations performed on the blockchain do not require full previous transactions, but only their unspent outputs, it seemed wasteful to use the fully indexed blockchain for everything. Instead, we keep a database with only the unspent transaction outputs. After some effort to write custom compact serializations for these, I could reduce the storage required for such a dataset to less than 70 MB. This is kept in a BDB database file (coins.dat), and with indexing and overhead, and takes around 130 MB.

Now, this is not enough. You cannot have a full node wit just these outputs. In particular, it cannot undo block connections, cannot rescan for wallet transactions, and cannot serve the chain to other nodes. These are, however, quite infrequent operations. To compensate, we keep non-indexed but entire blocks (just each block in a separate file, for now), plus "undo" information for connected blocks around in addition to coins.dat. We also need a block index with metadata about all stored blocks, which takes around 40 MB for now (though that could easily be reduced too). Note that this means we lose the ability to find an actual transaction in the chain from a txid, but this is not necessary for normal operation. Such an index could be re-added later, if necessary.

Once you have this, the step to pruning is easily made: just delete block files and undo information for old blocks. This isn't implemented for now, but there shouldn't be a problem. It simply means you cannot rescan/reorg/server those old blocks, but once those are deep enough (say a few thousand blocks), we can tolerate that.

So, in summary, it allows one to run a full node (now) with:

• 130 MB coins.dat
• 40 MB chain.dat
• the size of the retained blocks, +12% of that for undo information.

Oh, it's also faster. I benchmarked a full import of the blockchain (187800 blocks) on my laptop (2.2GHz i7, 8 GiB RAM, 640 GB spinning harddisk) in 22 minutes. That was from a local disk, and not from network (which has extra overhead, and is limited by bandwidth constraints). That is with some tweaking though.

If people want to experiment with it, see my "ultraprune" branch on github:  https://github.com/sipa/bitcoin/tree/ultraprune.

Note that this is experimental, and has some disadvantages:

• you cannot find a (full) transaction from just its txid.
• if you have transactions that depend on unconfirmed transactions, those will not get rebroadcasted
• only block download and reorganization are somewhat tested; use at your own risk
• less consistency checks are possible on the database, and even fewer are implemented

Also note that this is not directly related to the recent pruning proposals that use an alt chain with an index of unspent coins (and addresses), merged mined with the main chain. This could be a step towards such a system, however.

Be VERY careful when using private keys that are derived directly from a passphrase. The Bitcoin network does over 2^42 SHA256 operations per second, which is the equivalent to the number of all 8-character alphanumeric passwords every 17 seconds. Any decent miner on the Bitcoin network has enough power to bruteforce any short or easy passphrase-based private key (although deriving the address from a private key is slower than generating it in the first place).

If you want a safe(r) way for deriving keys from text, use repeated hashing using a standard such as PBKDF2 (which is repeated HMAC-SHA1) or Scrypt.

The current implementation keeps all transaction around, yes, but Bitcoin is designed from the start to allow "pruning", removing transactions which are old and fully spent already. This is described in Satoshi's paper. It's certainly not true that fully verifying nodes need to keep every transaction around forever.

You cannot have global real-time consensus. Just the fact that the speed of light is finite prevents communication from being instantaneous. Whatever limitation you build on accounts or transactions or wallets or addresses, you'll always be able to create two conflicting transactions, and only tell part of the world about each.

Bitcoin does not have "accounts". It has coins, and each transaction consumes, merges, splits and reassigns coins.

And you haven't provided an answer for what happens when someone releases two conflicting transactions simultaneously. Transfer time is not instantaneous, so not every node will hear about the same one first. They will disagree about which was first. Which one is considered valid?

The whole point of the block chain is to create a globally-agreed-upon order of transactions. If not for the possibility of double spends, we wouldn't need a block chain. In Bitcoin, initial introduction of new currency is tied to the production of blocks, but I'd call that a nice side effect rather than its core function.

Assume I was an attacker, and create two conflicting transactions (spending the same input coin), and I emit both in different places of the network. Some nodes would hear about one first, and others would hear the other first. Obviously they cannot agree, but how will they get to an agreement about which was first/valid/true?

There are hundreds of possible micro-improvements possible to the protocol, of course. Very few are worth breaking backward compatibility for.

I think it is wrong to advertize Bitcoin as being anonymous. It provides more privacy than most other electronic forms of payment, but certainly less than cash.

With the subtitle "become your own bank", maybe you're looking for "independent" rather than anonymous?

In a perfect world, where everyone has a nice and well-defined and reachable single IP address, without NATs and bridges and IPv4/IPv6 split, or the necessity for onion routing, ..., yes, TCP/IP would suffice, and you could determine the source IP directly from the IP socket layer.

Reality is more complex. Some nodes are firewalled and only can or only want to make outgoing connections. Some may have an address they can be reached on that differs from the one they connect with. From the other direction, nodes may be behind a NAT and not know their own public IP, even though they are reachable.

As to why Satoshi did things the way he did - we don't know, and we can't ask. Until quite recently, the values in addrMe and addrYou in version messages were entirely unused. That changed, as they can be useful for discovering which of your (perhaps several) detected addresses people connect on, and as a way to advertize your own address. Whether this was what Satoshi intended to be done with them? No idea...

If I recall correctly, testnet bitcoins have been traded in the past for 10000 tnBTC / 3 BTC. That was before the reset, though.

You can specify -externalip more than once.

261

You may want to look at the new -externalip option. It tries to be smart by default, but if you have several addresses in the same network, it can fail.

The upgrade process will not touch the blockchain database or your wallet.

That said, always make backups of wallet.dat.

A hard forking change is one that causes an infinite split between old and new nodes, and is in no way comparable to any change we've ever done.

It is only a hack when considered from the viewpoint of the current infrastructure. There is no reason why you'd leave the hashing of the merkle root on the server instead of the client, especially as it is exactly the same operation (double SHA256).


As fas as I know, there has not been a single hard fork in Bitcoin's history. The changes for BIP16 and BIP30 were "soft forks", that only made a backward-compatible change (meaning only making some existing rules more strict). Even the much more severe bug fixes in juli-august 2010 (see CVEs) were in fact only soft forks. Soft forks are safe as soon as a majority of mining power enforces them.

Changing the serialization format of blocks or transactions, or introducing a new version for these, however does require a hard fork. Other changes that require a hard fork are changing the maximum block size, changing the precision of transaction amounts, or changing the mining subsidy function. All these need a much much higher level of consensus, as these require basically the entire Bitcoin network to upgrade (not just a majority, and not just miners). Everyone using an old client after the switch will get stuck in a sidechain with old miners (even if that is just 1% of the hashing power). If we ever do a hard fork (and we may need to), it will have to be planned, implemented and agreed upon years in advance.


You are. Bitcoin is based on consensus, but you can only reach consensus as long as you can convince enough people there is a problem, and I personally still see this is a minor implementation inconvenience rather than a problem that will limit our growth.

Because of the nonce, we only need to recalculate the merkle root once every 4 billion hashes, instead of for every hash. In the current infrastructure, that merkle root is calculated on the server (typically) while the hashes are calculated by the miner. This means that there is an interaction between server and miner every 4 billion hashes. But the actual calculation per merkle root is nothing compared to the 4 billion hashes (see my post above) the miner already does. If the requesting of work becomes the bottleneck, the work generation can simply be moved to the miner.

No, this is not an issue. No, there is no need to increase the nonce size. Yes, 64 bit nonces would have things slightly easier for us, but all is required is a slightly more complex mining infrastructure and this inconvenience is nothing compared to doing a hard fork.

Let us look at this from a theoretical point of view, rather than from what current infrastructure provides. The first pool that is still operational started in december 2010 - two years ago. By the time we can pull of a block format change, we are at least two years away anyway. At that time, I'm sure the Bitcoin infrastructure will be very different from now.

Assume blocks reach their maximum size: 1000000 bytes, all the time. The smallest typical transactions are 227 bytes (1 input from a compressed pubkey, 2 outputs to addresses). That means a maximum of 4405 transactions.

In a merkle tree with 4405 elements, the leaves are 13 levels deep. That means generation of a new piece of work (for 4 billion double-SHA256 operations worth of calculation), you need to increase the extranonce in the first transaction (the coinbase) 's output, and hash your way up to the merkle root. This requires 13 double-SHA256 operations. If this is offloaded to the GPU's/FPGA's/ASIC's/QuantumCPU's/... that are *already* doing precisely that hashing operation for the block header anyway, they get a 0.0000003% overhead. The only thing they'd require is an occasional (at 4405 transactions per 10 minutes, 7 times per second) update of the list of transactions. A phone connection with a modem suffices for that kind of traffic.

In short: nothing to worry about. The only problem is with the current infrastructure, which will evolve.

Every bitcoin release to date has been beta, and that will remain the case for a while.


Certainly not in 0.6.x. We're working on 0.7 which will have many new features and improvements, but 0.6.x only gets bugfixes now. About coincontrol itself: there are still a few implementation problems with it.


That shouldn't be. Have you reported this problem on the issue tracker?