The old files are limited to (almost) 2GB. The new files are limited to 128MiB. Apart from that, they are exactly the same format, they just store fewer blocks per file. If you concatenate either (in the right order), you get a usable bootstrap.dat


Obviously you can use it for whatever you like while Bitcoin isn't running. I just mean that compatibility with external applications was not a design priority (except for the block files - they could have been made smaller as well, but we chose not to, in order not to break external applications).


You're sort of hijacking this thread, ranting about things that are not priorities now.

In 0.7.x, block verification WAS the bottleneck (except when you were on SSD or a tmpfs), and that is the reason for all changes that will be in 0.8, and what this thread exists for. I'm just saying that after those, it isn't. Block downloading and signature verification are the bottleneck. And yes, using bootstrap.dat for initialization avoids the problem with slow block downloading, but that's not what you want to be telling new users. The priority should be making automatic syncup usable again.

I'm not sure what you mean. The current downloading mechanism often takes hours just downloading (not including any verification or storage), a lot of which is waiting and not doing anything. Compare that to (order of magnitude) an hour of verification to reach block 210000 (without signature checks). After block 210000, signature checking becomes the bottleneck. So if we want to improve the experience for new users, the first step should be speeding up the download itself, and the signature verification. After that, we can have a look at further improvements for the verification itself.


It's being replaced to improve performance. The file format of the block files (blk000?.dat before, blocks/blk000??.dat now) hasn't changed: it's a binary concatenation of blocks in network format. The coins directory is a LevelDB database in a cusitom compact format that represents the set of unspent transaction output. See the source code (specifically: main.h above, class CCoins) for the exact specification. The rev* files contain data to be able to disconnect blocks afterwards in case of reorganisation. Bitcoin needs exclusive access to the database anyway, so everything except the block files isn't intended to be used by third party code. If you need access to the raw blockchain/utxo data, use the getblock, getrawtransaction and gettxout RPC commands.


My priority is correctness/completeness of the verification, consistency of the database on disk, and minimizing the working set size. That on itself leads to improved performance, but there are certainly a lot of things possible to improve it. As said, not a priority now. Yes if you do everything in RAM using a hash map, I'm sure it will be faster (it should!).


I'm very aware of the slowness, but changing in the reference code things takes time (to implement, to review and to test) - this  slows development down, but I'm sure the community wouldn't want even a blazingly fast release with subtle bugs in it that lead to a split of the network after a large part of the community adopts it. We do what we can, and most of us (including me) just do this in our free time. The real bottleneck is testing, so thanks for helping out here (though I really can't do much about a segfault without more details...).

Oh and by the way, use `qmake RELEASE=1` to build optimized binaries.

That is exactly the same thing as: making a cursory inspection of an address followed by a detailed inspection of the last few characters of the address (as an address can be considered to be the pubkeyhash with its own checksum appended).

Hard limit: 101 confirmations, i.e. created 100 blocks before (network rule)
Soft limit: 120 confirmations, i.e. created 119 blocks before (client policy, which is probably overly conservative)

0.7.x has indeed a bug that it needs the ability to seek to positions in block files being imported. This limits it to 2 GB on some systems.

The current git head code doesn't need this, and can import blocks from arbitrary-sized files.

I have never claimed otherwise. That is not the bottleneck now, however (crappy block download and signature validation are).


It should report block connect/validation speeds in debug.log.

I have no clue at all about GUI stuff. Wladimir wrote that.

In such a setup I'd advise you to disable listening, as that will reduce the amount of nodes trying to fetch block data from you significantly.

Oh, that's not a change I made, though I should have caught it.

The -static was intended to only be added for Windows.

The error is harmless. It uses git to determine the current version, but it has a fallback in case that fails.

The db_cxx error means you don't have libdb4.8++-dev or libdb5.1++-dev installed. See doc/build-unit.txt for instructions.

Yes, it is hardcoded. 0.7.x have a latest checkpoint at 193000. The current development code has an extra one at 210000.

It will load it, but only the first 193000 blocks will be done without signature checking.

Also, I think 0.7.x has a bug where no more than 2G of bootstrap.dat is loaded anyway. You can use -loadblock=file1 -loadblock=file2, ... though.

I sure, I fully agree. In terms of CPU usage, signatures validation outweighs everything. But before the last checkpoint (=exactly what is stored in bootstrap.dat), signature checking is disabled anyway (and this hasn't changed in 0..

Also, I assume that importing bootstrap.dat on your hardware will take more than 5 minutes. Done completely in RAM, it's probably possible, but the client also maintains a database on disk, and does synchronous writes at times to make sure block data and index/coins remain consistent with eachother.

You can't build an UTXO set (needed for validation) without doing most of the verification anyway (everything except signature checking, basically). If you mean trusting someone to build a pre-indexed blockchain for you, you're far better off running an SPV node - it's faster, needs less disk, needs less bandwidth to maintain, and doesn't require trust in a centralized source for your block data.

The current git head (pre-0. code still has some problems that cause initial syncup to be slow (in particular, the block fetching algorithm is crappy, and gets very easily confused to the point where it stops doing anything for several minutes). From reports, it also seems it's significantly slower at verifying on Windows systems. Still, if combined with parallel+optimized signature verification code (which may or may not make it into 0.8 release), and the block fetching system gets fixed, I hope we can get close to a sync time of 1 hour on reasonably recent systems.

In the future - at some point (near or far) - it's not reasonable to expect that every end user will run a fully verifying bitcoin node. If they use a desktop client at all (I assume much more will happen on mobile devices, for example), it will probably be with an SPV client or even lighter. At this point in the future, "reasonable real-time behaviour for the bitcoin-client" will only apply to those that have to (miners, payment processors, merchants) or volunteer to. I expect such people to run a node 24/7, and time to do a sync will only apply once.

That doesn't mean we should tell everyone now to move away (even though many people are, unfortunately but understandably) for fully validating clients. While the economy grows, we need a network to sustain it. In this respect, you are right, and we should do efforts to keep performance as good as possible (and I believe I do what I can for that). But as retep says, correctness is of much higher importance - if there is a problem with the code, it can be a disaster (lost coins, forked network, ...).

In particular, I have reasonable confidence in the 25% speedup optimization for secp256k1 after having spent time on it myself and tested it by comparing the output of the optimized operations with the non-optimized OpenSSL ones. I'd still like some profession cryptographer to look at the code still.

GPU code to do ECDSA verification would in the first place be an experiment, in my opinion. If it indeed helps in significantly faster verification, some may consider it. But as things stand now, I don't think there's an actual need.

There was a syntactic error in the bitcoin-qt.pro file (which was fixed before, but it seems the fix got lost). Newer versions of lrelease didn't break on it, but older ones did. It should be fixed now.

I'm pretty sure that a good OpenCL implementation of ECDSA verification, running on high-end ATI gpu's, will be significantly faster than what OpenSSL (or any implementation) can do on a CPU. People have already written EC addition in OpenCL, reaching speeds of tens of millions of operations per second. EC multiplication is much slower and more complex to implement efficiently, but I expect it would reach tens of thousands of operations per second at least.

Someone just needs to write it...

The secp256k1-specific optimization has been implemented, and there's a pull request to integrate it in the reference client. It does indeed achieve some 20% improvement.

What smtp asked was about the parallel batch verification, with potentially much higher speedsup (Hal mentioned x4). The problem is that this is a much more invasive change, and the risk of getting crypto stuff wrong is extremely high. It's also not certain which degree of speedup is possible, as we need to do recovery of R before batch verification is possible (and changing this requires a hard fork basically, as the signatures are part of the scripting rules).

That said, if someone is interested in investigating, and potentially supplying a patch that is well testable... by all means, do so.

That's why I add that builds in gitian show the same problem, so I confirm there is a problem here. I just haven't spent the time investigating why.

Building works fine on my laptop, but I believe I hit the same problem when trying to build release binaries for Linux in gitian. It may require the package 'qt4-linguist-tools'.


I've certainly seen more than that - 6000tx/s is over 12000txins/s.