Experimental pre-0.8 builds for testing

[smtp]

Oh, that's not a change I made, though I should have caught it.
The -static was intended to only be added for Windows.

Sorry, that is fixed in the last revision of the stack-protector pull now .

Sorry, I deduced (wrongly) from the fact that if it are your files (URL https://github.com/sipa/bitcoin/tree/turbo) then at least you were responseable for this "-static" patch, sipa.


Please remove this total misleading message:

fatal: Not a git repository (or any parent up to mount parent )
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set)

from the build-process, especially tragic if a different error occurs later which could have been founded (for an outsider) on this reason.

And one more proposal for the next official bitcoin-release:

Mention in the doc/build-unix.txt the possibility to customize theses 6 variables
# Dependency library locations can be customized with:
#    BOOST_INCLUDE_PATH, BOOST_LIB_PATH, BDB_INCLUDE_PATH,
#    BDB_LIB_PATH, OPENSSL_INCLUDE_PATH and OPENSSL_LIB_PATH respectively
in the bitcoin-qt.pro file.

BTW: If noone made errors, nobody could learn.

smtp

[1713949860]

1713949860

[1713949860]

1713949860

[smtp]

Hi

Last night I only started this bitcoin-qt with an empty $HOME/.bitcoin to see whether it principle worked.
Now the bitcoin-qt 0.7.99 is running first time seriously on my hardware (2 GHz Athlon-64 3200+ stepping 00 with a total of 4 GB ECC-Ram).

Firstly (an experiment) I had set two softlinks to the existing blk0001.dat and blk0002.dat and started with "./bitcoin-qt  -reindex -detachdb", but it immediately crashed without error-message (no bootstrap.dat exists).

Next I built the new bootstrap.dat  of size 4883237032 (block height 216283), deleting theses links to blk00*.dat and started
"/usr/bin/time ./bitcoin-qt -detachdb -benchmark" Now it runs
and seems to process all these blocks. After 15 min CPU-time consumed, it still had ~ 59000 left . I wonder why it is so slow, for signature-checkng too fast
but still uses mostly 98% CPU-time, some drops from time to time, but I estimate still > 90% CPU-time on average. Periodcially disk-writes occurs, but because it has taken 3.3 GB system-cache this doesn't really matter to real-time behaviour. Currently, I just look, at remaining ~ 31600 blocks, it still
processes as about 10 - 20 blocks /sec (difficult to estimate). I wonder what it is doing internally with such a high cpu-usage. Disk-reads are fairly sparse in real-time.


... to be continued (now 39 min CPU-time used and 23400 blocks left to be processed).

BTW: System is a flavor of Linux 2.6.32-gentoo-r7 x86_64.

smtp

[smtp]

block height 21000 was reached after 82 min cpu-time, currently it is at block height 214045 needed 128 min Cpu-time and got 93 % CPU-time in the real-time. And I discovered that the new blocks are stored in the sub-directory blocks together with rev*.dat files. I still have the feeling no signature-verification is done . It procesess currently with 439 tx/sec (no txin-count availible in debug.log). I think it will finish (reaching block height 216283) in the next 20 minutes when I'm at lunch.

smtp

[smtp]

Finished:

/usr/bin/time ./bitcoin-qt -detachdb -benchmark
Application asked to unregister timer 0x3a000004 which is not registered in this thread. Fix application.
8680.81user 338.18system 4:18:23elapsed 58%CPU

Elapsed time is much too high, because it waits 1.5 hours until I finished it manually - 2 h 45 min real-time for loading the bootstrap.dat,
93.8 % CPU time looks right and about 12 minutes I/O-time until finish loading.
It only needs 1-2 seconds to finish bitcoin-qt after I have clicked the Quit-Button (-detachdb no longer needed?).
Total size of directory "blocks" is 5417516 kB containing 2*37 files (about 11% overhead, old blkindex.dat had 32% overhead compared to the blk00*.dat).
Directory "coins" has size 164324 kB.

I think there is still a lot of room for speed-optimization of the load of the initial bootstrap.dat. Moreoever I see no effect of this "-benchmark" option (maybe this ".....Fix application." message?)

BTW: I run it with "maxconnections=0" set in the bitcoin.conf file such that at the end of the load no further action was done.

smtp

[Pieter Wuille]

I think there is still a lot of room for speed-optimization of the load of the initial bootstrap.dat.

I have never claimed otherwise. That is not the bottleneck now, however (crappy block download and signature validation are).

Moreoever I see no effect of this "-benchmark" option (maybe this ".....Fix application." message?)

It should report block connect/validation speeds in debug.log.

[smtp]

Hi

There is highly probable a bug in the "-reindex" option called code.

After updating with the bitcoin-qt-0.7.99 to the current main blockchain , this works fine, I restarted it again:

./bitcoin-qt -maxconnections=0 -reindex -dbcche=500 -logtimestamps -benchmark

and it crashed after a few seconds without any error message.
Here is the tail of the debug.log

2013-01-14 20:37:15 Opened LevelDB successfully
2013-01-14 20:37:15  block index              51ms
2013-01-14 20:37:15 Loading wallet...
2013-01-14 20:37:15 nFileVersion = 79900
2013-01-14 20:37:16  wallet                  976ms
2013-01-14 20:37:16 Loading addresses...
2013-01-14 20:37:16 Reindexing block file blk00000.dat...
2013-01-14 20:37:16 Loaded 10094 addresses from peers.dat  40ms
2013-01-14 20:37:16 mapBlockIndex.size() = 0
2013-01-14 20:37:16 nBestHeight = -1
2013-01-14 20:37:16 setKeyPool.size() = 100
2013-01-14 20:37:16 mapWallet.size() = 0
2013-01-14 20:37:16 mapAddressBook.size() = 1
2013-01-14 20:37:16 Done loading
2013-01-14 20:37:16 refreshWallet

Maybe you get an idea.
BTW: Could you explain the new format of these blk*.dat files (and if easy, the rev*.dat) files or is this anywhere already documented?

Appendum: I just want to give you more debugging info and therefore started "gdb ./bitcoin-qt" and then inside
run "-maxconnections=0 -reindex -dbcche=500 -logtimestamps -benchmark"

and it runs ...
:-/

smtp

[Diapolo]

You mean -dbcache, right? I also observed a crash while reindexing with current master, but I didn't yet take a closer look at it.

Dia

[smtp]

You mean -dbcache, right? I also observed a crash while reindexing with current master, but I didn't yet take a closer look at it.

I don't know, but I believe
I used the same options as

bitcoin-qt.exe -reindex -par=4 -dbcche=500 -logtimestamps -benchmark

except this -par=4

smtp

[smtp]

and it runs ...
:-/

2013-01-14 22:38:44 Reindexing finished
 98:20 min CPU-time until block-height 216823  and 99:16 min until 21650 height (the maxium no of blocks in the blk*dat files).

I lookup up in the debug.log : Start at 2013-01-14 20:42:55 which means 116:11 min total real-time or 85.4 % CPU-time.

Compare this with the bootstrap.dat of 216823 blocks using 150 min CPU-time. It should be about equal - but I did a simple trick. *gg*
I recompiled the turbo-bitcoin/sources with the g++ additional options -march=athlon64 -O2
:-)

Nearly a boost of 50% in speed up! I wonder why there is also in the official sources not at least a -O2 option for the g++-compiler.
Everything will profit from -O2.

BTW: This "./bitcoin-qt -maxconnections=0 -reindex -dbcche=500 -logtimestamps -benchmark"  (also with -dbcache inplace of -dbcche) crashes
the client reproducable after start. Sadly not, if invoked with gdb. :-(

smtp

[smtp]

I just notice this reindex-bug is a really nasty bug. If I restart bitcoin-qt next (only with -maxconnection=0) it starts immediately with reindexing. :-(
I quite it and start it again, it starts reindexing ...

Lots of lines
"ProcessBlock: ORPHAN BLOCK, prev=0000000000000640dd32c57bbda456631968a2d57ccc3d40190c767ab2ce7e99"
now writen to debug.log.

Ups! debug.log was truncated by bitcoin-qt -- only 77030 lines of these "ProcessBlock: ORPHAN BLOCK, prev=...."  inside now.

Well, I better should throw all away now .... and wait for Pieter to debug it.

smtp

[2112]

Sadly not, if invoked with gdb. :-(

I just wanted to remind you about "ulimit -c unlimited" to be able to meaningfully use the debugger after the crash. You're probably running with the default of "ulimit -c 0".

[smtp]

Sadly not, if invoked with gdb. :-(

I just wanted to remind you about "ulimit -c unlimited" to be able to meaningfully use the debugger after the crash. You're probably running with the default of "ulimit -c 0".

Sorry, but you remind me only dark fuzzy remembers decades ago that such a command exist(ed).

1) my ulimit has no -c option (only a -f option) on Linux (not only on my gentoo, but also on Ubuntu 10.2 I just checked)
2) called "ulimit" gives the default of unlimited
but most important
3) There occurs no crash in gdb, thus it is not meaningfull (in my context) to talk about "use the debugger after the crash".

BTW: I just discovered the ulimit with many options on SunOS. "ulimit -c" sets the "core" limit of the process there. I wonder what core precisely is meant.

Anyway, thanks for your thoughts
smtp

[smtp]

I think there is still a lot of room for speed-optimization of the load of the initial bootstrap.dat.

I have never claimed otherwise. That is not the bottleneck now, however (crappy block download and signature validation are).

Sorry, but I think I should disagree strongly! This might to be an (or better the!) important bottleneck for each newbie who want to start to use bitcoin-qt.
You are true, it is no bottleneck for you and probably for > 95% of all forums posters here who has done already the initial download of the blockchain data. "crappy block download and signature validation" becomes the next bottle neck AFTER you have your personal block chain in a bitcoin-client-conform format!

I also dislike to see it to be replaced  (the currently undocumented "blkindex.dat"-format) by new formats in rev*dat, blk*dat, coins-directory (and  blktree-diretory?), if these stay undocumented or to be over-complex.  Again making it impossible to use directly this raw block chain data. I think I have shown with my block-parser, that conversion of this data into a bitcoin-qt usable structure can be done only in a few minutes on a typical hardware and not in more than 20 times real time for bitcoin-turbo and checking everything inside the main block chain except the script-verification! Yes, before, 0.7.?, it were more than 200-times of real time, so a great achievement of your new experimental 0.7.99 version.

Please remember: I only started activity in this Bitcoin Forum, because the VERY SLOW initialization with the main block chain by the (most spread) bitcoin-qt client might put off increasingly many want-to-be starters into the bitcoin world.


smtp

[smtp]

3) There occurs no crash in gdb, thus it is not meaningfull (in my context) to talk about "use the debugger after the crash".

I got another idea. You could compile the source with the "-g" flag and then start your client as normal with the "-reindex"-option hoping still(!) the crash occurs and force the OS to produce a core-file (image of all his allocated memory into a file). Then you could investigate this core-file with gdb afterwards.

But I think, before realizing these thoughts, at first sipa must tell us his oppinion in this "-reindex" case.

smtp

[2112]

I wonder what core precisely is meant.

I apologize I wrote like everyone knows and understands the postmortem debugging.

$ cat bitcoind.c
#include <stdio.h>
int main() { return printf("Hello %s!\n",2112); }
$ make bitcoind
$ ulimit -a
core file size           (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
$ ulimit -c unlimited
$ ulimit -a
core file size           (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
$ ./bitcoind
Segmentation fault (core dumped)
$ ls core*
core
$ gdb ./bitcoind core
GNU gdb (GDB) 7.1-ubuntu
Reading symbols from /home/2112/bitcoind...(no debugging symbols found)...done.
Reading symbols from /lib/libc.so.6...done.
Core was generated by `./bitcoind'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f4f24cd7e32 in _IO_vfprintf_internal (s=0x7f4f2500c780,
    format=<value optimized out>, ap=0x7fffcba92680) at vfprintf.c:1617
1617   vfprintf.c: No such file or directory.
   in vfprintf.c
(gdb) where
#0  0x00007f4f24cd7e32 in _IO_vfprintf_internal (s=0x7f4f2500c780,
    format=<value optimized out>, ap=0x7fffcba92680) at vfprintf.c:1617
#1  0x00007f4f24cdf65a in __printf (
    format=0xffffe906 <Address 0xffffe906 out of bounds>) at printf.c:35
#2  0x000000000040053f in main ()
(gdb)

So on a reasonable OS program doesn't have to be compiled with debugging and/or run under the debugger to approximately locate the place where it crashed.

I think that it is a terrible legacy of Microsoft (and/or Apple) that various Linux distributions switched to "disable core dumps by default". The old "remove core dumps older than a week by default" was much better, because nobody would use the "program crashes, but not under the debugger" excuse.

Again, I want to apologize for writing like everyone already knows and just needed a reminder. Hopefully there will be at least one reader who wasn't familiar with the postmortem concept who now will learn how to override the default in their Linux installation to re-enable to perform good old core dumps. Nowadays core memory can probably be found only in museums, but the core dumps are still useful. Lets debug like it is 1970 again!

[smtp]

I wonder what core precisely is meant.

I apologize I wrote like everyone knows and understands the postmortem debugging.

No need to apologize - at least not apologize to me.

1) I did not grasp your idea, that I must/should force the OS to produce this core-file.

2) You seem to have a different environment (I even don't know your OS), where ulimit has this -c option.

3) We have the  rare case, that a program which crashes when invoked by the shell doesn't crash when invoked by gdb.

BTW: I still remember well,  when , about 20 years ago, nearly every crash of a program left a core file.

Thanks,
smtp

[smtp]

I think that it is a terrible legacy of Microsoft (and/or Apple) that various Linux distributions switched to "disable core dumps by default". The old "remove core dumps older than a week by default" was much better, because nobody would use the "program crashes, but not under the debugger" excuse.

The main problem is people don't express precisely what they are talking of or assume environments/dependencies others don't have/use. :-/

When checking for "ulimit" on my sytem/environment , I did "man ulimit" and later "which ulimit". So I  got my informations (also for ubuntu & SunOS) from the man-pages and from the which/PATH-search. So, should I apologize for not doing "whence ulimit", but "which ulimit", don't I?

ulimit (as typed in the command-line) is a shell-built-in (if you use an appropriate shell!) with all these many options, including option -c for size of core-limit, even in (my) Linux (if you use sh or ksh or bash, but not e.g. in csh or tcsh).

Appendum: There exists also the (obsolete) system/library-call ulimit, see "man ulimit 3".

smtp

[Pieter Wuille]

Sorry, but I think I should disagree strongly! This might to be an (or better the!) important bottleneck for each newbie who want to start to use bitcoin-qt.
You are true, it is no bottleneck for you and probably for > 95% of all forums posters here who has done already the initial download of the blockchain data. "crappy block download and signature validation" becomes the next bottle neck AFTER you have your personal block chain in a bitcoin-client-conform format!

I'm not sure what you mean. The current downloading mechanism often takes hours just downloading (not including any verification or storage), a lot of which is waiting and not doing anything. Compare that to (order of magnitude) an hour of verification to reach block 210000 (without signature checks). After block 210000, signature checking becomes the bottleneck. So if we want to improve the experience for new users, the first step should be speeding up the download itself, and the signature verification. After that, we can have a look at further improvements for the verification itself.

I also dislike to see it to be replaced  (the currently undocumented "blkindex.dat"-format) by new formats in rev*dat, blk*dat, coins-directory (and  blktree-diretory?), if these stay undocumented or to be over-complex.  Again making it impossible to use directly this raw block chain data.

It's being replaced to improve performance. The file format of the block files (blk000?.dat before, blocks/blk000??.dat now) hasn't changed: it's a binary concatenation of blocks in network format. The coins directory is a LevelDB database in a cusitom compact format that represents the set of unspent transaction output. See the source code (specifically: main.h above, class CCoins) for the exact specification. The rev* files contain data to be able to disconnect blocks afterwards in case of reorganisation. Bitcoin needs exclusive access to the database anyway, so everything except the block files isn't intended to be used by third party code. If you need access to the raw blockchain/utxo data, use the getblock, getrawtransaction and gettxout RPC commands.

I think I have shown with my block-parser, that conversion of this data into a bitcoin-qt usable structure can be done only in a few minutes on a typical hardware and not in more than 20 times real time for bitcoin-turbo and checking everything inside the main block chain except the script-verification! Yes, before, 0.7.?, it were more than 200-times of real time, so a great achievement of your new experimental 0.7.99 version.

My priority is correctness/completeness of the verification, consistency of the database on disk, and minimizing the working set size. That on itself leads to improved performance, but there are certainly a lot of things possible to improve it. As said, not a priority now. Yes if you do everything in RAM using a hash map, I'm sure it will be faster (it should!).

Please remember: I only started activity in this Bitcoin Forum, because the VERY SLOW initialization with the main block chain by the (most spread) bitcoin-qt client might put off increasingly many want-to-be starters into the bitcoin world.

I'm very aware of the slowness, but changing in the reference code things takes time (to implement, to review and to test) - this  slows development down, but I'm sure the community wouldn't want even a blazingly fast release with subtle bugs in it that lead to a split of the network after a large part of the community adopts it. We do what we can, and most of us (including me) just do this in our free time. The real bottleneck is testing, so thanks for helping out here (though I really can't do much about a segfault without more details...).

Oh and by the way, use `qmake RELEASE=1` to build optimized binaries.

[smtp]

Hi

I should lead-in by saying, "sorry that I pressed you to spend so much real time in this reply - only for me".  :-/

Sorry, but I think I should disagree strongly! This might to be an (or better the!) important bottleneck for each newbie who want to start to use bitcoin-qt.
You are true, it is no bottleneck for you and probably for > 95% of all forums posters here who has done already the initial download of the blockchain data. "crappy block download and signature validation" becomes the next bottle neck AFTER you have your personal block chain in a bitcoin-client-conform format!

I'm not sure what you mean. The current downloading mechanism often takes hours just downloading (not including any verification or storage), a lot of which is waiting and not doing anything. Compare that to (order of magnitude) an hour of verification to reach block 210000 (without signature checks). After block 210000, signature checking becomes the bottleneck. So if we want to improve the experience for new users, the first step should be speeding up the download itself, and the signature verification. After that, we can have a look at further improvements for the verification itself.

And I'm not sure what is (principlely) unclear in my sentences. :-/
You (the newbie) can get the download with maximal speed (his band-width limitation) by simply download a trusted bootstrap.dat! It can't be faster. It is our responsibility to provide in regular (short) time distances these block chains. E.g. each week. So simply!

BTW: I did load bootstrap.dat of the main blockchain until height 216283 once again with your 0.7.99 release (and compiled with -march=athlon64 -O2)
and got 36.5 minutes cpu-time until height 210000 and 96.5 minutes cpu-time until height 216283 which uses a total of 108 min real-time.
(My testing of this -reindex-bug had finally corrupted my blockchain data).
The last 6283 blocks needed exactly 1 h cpu-time or 0.57 sec/block. But better comparing transaction counts not block counts. The first 9344206 tx until block height 210000, could be loaded from bootstrap.dat into the bitcoin-qt data structures with  4266 tx/sec the remaining 1731667 transactions with 481 tx/sec (due to verification checks). Thus, if there would have been a new bootstrap.dat until block height 216283, total real-time would have been only 43 min. Well, my blockparser needs less than 2 min (without sig-checks yields > 92000 tx/sec) , but maybe your deletion of redeemed deposits is expensive (but I doubt). So I guess there is surely a factor 20 (or more) which could be improved (I did no aggressive coding in my parser to reduce cpu or real time).
And I already praised you -- you are faster already by a factor 10 (or even a bit more) compared to version 0.7.1.

BTW: Writing the data new to disk, needs neglectable cpu-time and no further real-time (I see it at the running 0.7.99 because it is done in parallel) , but this I can't do as long as I neither know the structure of blkindex.dat nor of your new structured files in 0.7.99 (I asked for). (A following item).
 
Sorry, it becomes too late in the night. I will continue tomorrow, discussing the/your following items/responses.

Regards,
smtp

[smtp]

2nd part:

I also dislike to see it to be replaced  (the currently undocumented "blkindex.dat"-format) by new formats in rev*dat, blk*dat, coins-directory (and  blktree-diretory?), if these stay undocumented or to be over-complex.  Again making it impossible to use directly this raw block chain data.

It's being replaced to improve performance. The file format of the block files (blk000?.dat before, blocks/blk000??.dat now) hasn't changed: it's a binary concatenation of blocks in network format. The coins directory is a LevelDB database in a cusitom compact format that represents the set of unspent transaction output. See the source code (specifically: main.h above, class CCoins) for the exact specification. The rev* files contain data to be able to disconnect blocks afterwards in case of reorganisation. Bitcoin needs exclusive access to the database anyway, so everything except the block files isn't intended to be used by third party code. If you need access to the raw blockchain/utxo data, use the getblock, getrawtransaction and gettxout RPC commands.

I did (already yesterday) a  "cmp $HOME/.bitcoin.0.7.99/blocks/blk00000.dat $HOME/.bitcoin/blk0001.dat" which results in the (first) difference at byte 134214364. So they can not be simply concatenated to get the block-chain back, right? But maybe I have discovered a bug in the data structure/files.

Well, "everything except the block files isn't intended to be used by third party code", together with "It's being replaced to improve performance" this both sounds like old Mircosoft-windows argumentation. :-/ "Bitcoin needs exclusive access to the database" Huch? only if it is run - but this line of argumentation matches the previous.

BTW: Why do I sound so negatively? Because I must think of my horrible long initialiation of the block chain which lasts far more than 23 h real-time last december and you tell me that it is no bottleneck. Sorry, to be frank.

smtp