A soft and hard for are not comparable.

In a soft fork, it is about getting the _majority_ of _miners_ behind the rule. Every piece of old software keeps working. Depending on the change, it may be advisable for merchants to upgrade to get the extra rules enforced, but those who don't just get dropped to SPV-level security. Nothing will break as long as somewhat more than 50% of hash power enforces the new rule.

In a hard fork, it is about getting _all_ of _everyone_ to change the rule at exactly the same time. Doing a hard fork where not everyone is on the same side, is an outright disaster. Every coin that existed before the fork will be spendable once on every side of the chain. If this happens, it is economic suicide for the system. Sure it may recover after a short while, when people realize to pick the side that most others chose, but it is not something I want to see happening.

The only way a hard fork can be done, is when there is reasonable certainty that all players in the network agree.

First of all, my opinion: I'm in favor of increasing the block size limit in a hard fork, but very much against removing the limit entirely. Bitcoin is a consensus of its users, who all agreed (or will need to agree) to a very strict set of rules that would allow people to build global decentralized payment system. I think very few people understand a forever-limited block size to be part of these rules.

However, with no limit on block size, it effectively becomes miners who are in control of _everyone_'s block size. As a non-miner, this is not something I want them to decide for me. Perhaps the tragedy of the commons can be avoided, and long-term rational thinking will kick in, and miners can be trusted with choosing an appropriate block size. But maybe not, and if just one miner starts creating gigabyte blocks, while all the rest agrees on 10 MiB blocks, ugly block-shunning rules will be necessary to avoid such blocks from filling everyone's hard drive (yes, larger block's slower relay will make them unlikely to be accepted, but it just requires one lucky fool to succeed...).

I think retep raises very good points here: the block size (whether voluntarily or enforced) needs to result in a system that remains verifiable for many. What those many are will probably change gradually. Over time, more and more users will probably move to SPV nodes (or more centralized things like e-wallet sites), and that is fine. But if we give up the ability for non-megacorp entities to be able to verify the chain, we might as well be using those a central clearinghouse. There is of course wide spectrum between "I can download the entire chain on my phone" and "Only 5 bank companies in the world can run a fully verifying node", but I think it's important that we choose what point in between there is acceptable.

My suggestion would be a one-time increase to perhaps 10 MiB or 100 MiB blocks (to be debated), and after that an at-most slow exponential further growth. This would mean no for-eternity limited size, but also no way for miners to push up block sizes to the point where they are in sole control of the network. I realize that some people will consider this an arbitrary and unnecessary limit, but others will probably consider it dangerous already. In any case, it's a compromise and I believe one will be necessary.


Then I think you misunderstand what a hard fork entails. The only way a hard fork can succeed is when _everyone_ agrees to it. Developers, miners, merchants, users, ... everyone. A hard fork that succeeds is the ultimate proof that Bitcoin as a whole is a consensus of its users (and not just a consensus of miners, who are only given authority to decide upon the order of otherwise valid transactions).

Realize that Bitcoin's decentralization only comes from very strict - and sometimes arbitrary - rules (why this particular 50/25/12.5 payout scheme, why ECDSA, why only those opcodes in scripts, ...) that were set right from the start and agreed upon by everyone who ever used the system. Were those rules "central planning" too?

If you see the reindexing finding orphans over and over, it usually means one block couldn't be found, but laters blocks can be. Since one block in their ancestry can't be connected, it is reported as an orphan and skipped. It's not an infinite loop though, it will just go over all block files. We've had some bugs in pre-releases that could cause this, but for someone first trying 0.8.0rc1, my guess is that you had a few blocks on disk that actually were corrupted.

How did you fix it? If my assumption is correct, the only solution would be waiting a few minutes for it to skip all blocks you already have, and then waiting for it to start downloading from before the invalid block.

Because the block database(s) are LevelDB now. The BDB databases use a "database environment" (a directory shared by all databases) for journalling and consistency checks. To support moving the database files around, the files were "detached" from the environment at shutdown. Originally, this was always done. Since 0.6.1 this was made optional for the block chain databases (wallets were still always detached at shutdown), and -detachdb was introduced to re-enable it.

LevelDB databases use an entire directory by themselves, and are independent. So there is no environment to detach from anymore. The wallet in 0.8 is still BDB, and is still always detached.


No, it's not. But if you don't intend to go back to 0.7.x or earlier, you can delete blkindex.dat, blk0001.dat and blk0002.dat from the datadir (don't delete anything inside the blocks/ subdirectory though).

Here, in the v0.7.2 source code, is the function that calculates the reward per block at a given height. The 21M limit comes from adding all the resulting rewards.

Please try v0.8.0rc1.

You want the client to validate itself, and tell you has verified it is authentic?

Do you think that someone who distributes a malicious version won't just make it skip that check?

v0.8.0 is not released yet, there's only a release candidate.

Not really. None of this has anything to do with the wallet implementation. Wallets always track all transactions relevant to the user. This has not (or hardly) changed between 0.7 and 0.8.

What has changed, is how the block and transaction validation works. Previously, we stored

• the full blocks (blk000?.dat)
• the (byte) position of every block and transaction in it (blkindex.dat)
• for every transaction output, whether and where is was spent (also blkindex.dat)

This required an ever-growing index, and fast access to the full (ever growing) block data. This was slow.

The new system stores:

• the full blocks (blocks/blk000??.dat)
• the (byte) position of every block in it (but not every transaction!) (blocks/index/*)
• a separate database with a copy of the unspent transaction outputs (so not an index with byte positions for into the block chain, just a copy of not even the full transactions, but only the part that may be relevant in the future) (chainstate/*)
• an undo log for the chain state, so we can go back in time for reorganisations (blocks/rev000??.dat).

The big advantage is that we now only need fast access to the chain state (around 150 MB), instead of to the full blocks and the full index (several GB).

Although I initially called this new database/validation system "ultraprune". This is a very confusing name, as there is no actual pruning going on: we still keep all blocks/transactions around. The block data is still necessary for rescanning, reorganising, serving to other nodes, and the getrawtransaction RPC call. The code that resulted in the new database system actually started as an experiment about how small the chain state (aka UTXO set) could be represented, by pruning it as hard as possible. This is where the name comes from.

No problem - rescan just goes through all old blocks again, one by one. Those are still available in 0.8. The only thing that changes is that no information in the index is kept about spent transactions, so without txindex, it's impossible to find a transaction given just its txid. In practice, really the only thing affected is the getrawtransaction RPC.

There are known problems with the current  block synchronisation mechanism. Not exactly a bug, but just a crappy implementation. It's certainly intended to be changed, but it's not a trivial thing to do. Using the bootstrap.dat torrent, or using -connect=<ip> to a known fast node (not necessarily a trusted one) can speed up the initial download a lot. Not really nice solutions, but for now, it's all we have.


Block compression is certainly possible, but it's hard to get significant improvements. With a lot of effort and a very specifically designed compressed format, I expect we may get to something like 40%. Unfortunately, the blocks contain lots of hashes and cryptographic signatures, which are essentially uncompressible. Right now, I don't think these are a priority. Fixing the download mechanism will have a much larger impact.


Yes, you can if you don't intend to downgrade to 0.7.x (or below) code anymore. Note that the old and the new block files are hardlinked on supported platforms/filesystems, so you won't actually gain any space by deleting them. You can however delete blkindex.dat.


There's no simple answer to this, as it's not just a change from BDB to LevelDB: the actual layout and organisation of the databases changed as well. The actual blocks haven't changed (they are in different, more and smaller files, but the data is the same). The old blkindex.dat (BDB, ~1650 MiB) was replaced by:

• the block index (LevelDB, $DATADIR/blocks/index, ~ 30 MiB)
• the chain state (LevelDB, $DATADIR/chainstate, ~ 170 MiB)
• the undo data (custom format, $DATADIR/blocks/rev*.dat, ~ 700 MiB)

So, roughly, the old index data now takes half as much space, but it's not entirely a fair comparison (we actually do store less information now).

Yes, it does. It uses the same mechanism the block synchronization mechanism uses to find forks: the wallet stores a series of hashes: the last one, the 9 direct ancestors of the last one, and then the hashes of blocks with exponentially increasing distance back.

-checkblocks=0 actually means "check all blocks" (because of compatibility with the pre-0.6 -checkblocks (without parameter) that checked everything instead of just the last 2500 blocks).

-rescan is a wallet option, which rescans the block chain for transactions missing from the wallet. It shouldn't be necessary in normal operation since 0.3.21 (since then, wallet auto rescan from the last block they 'saw').

The primary reason for -checkblocks (a feature that has existed for as long as I remember, but has only been made configurable in 0.6) is preventing accidental disk corruption of the block chain file, which could result in rejecting the best chain. It is certainly not a protection against an attacker that is able to write to your block files (though it does make such an attack harder).

Also note that both the meaning of the check levels as the default number of blocks to scan has been changed in git head (only 288 blocks, but with a far more thorough check). This is safe, as block data corruption is no longer a chain forking risk in the new database layout (only the unspent transaction output database, aka coins database, aka chainstate matters).

Here's an explanation of the new -checklevel meanings (for 0.8 ):

• 0: Validate all block headers + compare (by hash) to blocks on disk for the last -checkblocks blocks
• 1: In addition, verify (standalong) validity of those -checkblocks blocks
• 2: In addition, verify that undo data matches checksums
• 3: In addition, check that the current chainstate matches can reasonably be the result of the last N blocks, where N is limited by both -checkblocks and the amount of in-memory cache (-dbcache); typically it's around 150.
  
• 4: In addition, for the last N blocks (see above), do full validation (including signature checks).

Note that the new default (3) is in fact a stronger check than the previous highest level (6).

Interesting - thanks for reporting, in that case.

I'm quite sure your computer freezing up is not caused by the CPU needed for verifying signatures, but by the database I/O blocking your OS and other applications from accessing your hard disk. Scheduling the task at low priority will not help in that case.

The next release (0.8 ) should decrease I/O requirements a lot. If you're interested, feel free to try the test builds

BTW: I'm not saying that a checkbox to lower CPU priority (or even a default) is a bad idea - I just don't think it will help preventing freezes.

Current versions of the reference client indeed causes excessive disk I/O. The next version (0.8, not yet released) should improve upon this significantly (it uses a more modern database engine, and a new database layout).

Of course. I'll point them to bootstrap.dat now too. But that doesn't mean we don't have to fix the real problem (the fact that the built-in block download code is very crappy). If things were working as they should, we wouldn't need to tell new users about bootstrap.dat (although it may still be useful).

My attitude? I don't want to be responsible for a network fork. The purpose of the reference client is implementing a full node, and that means zero trust: it will verify all incoming data. If you can't live with the resource requirements that are needed for that, and consider trusting some instance to do checking for you, you're better off using a lightweight client (which only checks block headers - see SPV mode in Satoshi's paper), as those don't need trust in the data source and don't risk creating a network fork if the trusted data is wrong.

I think everything that had to be said, is said now. You may disagree with me, but if so, please take it elsewhere. I won't respond anymore to messages here which are not about getting 0.8 tested.

You make the wrong assumption that because the file format is the same, the concatenation will be binary identical. In the 0.8 code, the files are pre-allocated in blocks of 16 MiB, to avoid high fragmentation. That means that at the end of those files, you'll have some zero bytes. So the concatenated files will have these zeroes too, and you'll get a slightly larger bootstrap.dat. Both the loading code in 0.7 and 0.8 should be able to deal with holes in the files, though.

See, I didn't lie.

-checkblocks is about how much verification you want at startup about the data you already have. I find disabling checks when importing a bootstrap.dat very risky - someone distributing "trusted" files which have an accidental (or deliberate!) error in them is enough to cause a block chain split, which in case of widespread deployment could be devastating for the bitcoin economy. If there was something significant to be gained, it could be made optional with a flag with a big warning on that, but I doubt it will help much (I don't have numbers, and you're welcome to do benchmarks yourself if you want to help), as most time is maintaining the database and not doing checks.


Yes, thank you very much for those numbers. I know there are performance improvements possible, I've told you a zillion times now. This thread is about doing tests, so we can get 0.8 out as soon as possible, because 0.7.x's performance is terrible, and it won't be released before it is stable.

Can we please stop this discussion, and get this thread back to test results?