Experimental pre-0.8 builds for testing

[smtp]

The bottlenecks are now the crappy block download mechanism (which sometimes stops for several minutes) and signature verification (which is needed in practice, and is several orders of magnitude slower than all the rest combined).

The last point can only be critical, if CPU-usage tends to 100% not the waiting time of the bitcoin client. The only checking my blockparser doesn't do is the merkeltree-hash of each block (I was still too lazy to do the coding) and this OP_CHECKSIG (resp. OP_CHECKMULTISIG), but I had read and understood its detailed description on https://en.bitcoin.it/wiki/OP_CHECKSIG. Thus I can't judge how cpu-intensive this OP_CHECKSIG evaluation really is.

Today, the first time my blockparser could verify the current block chain (until block height 216049) completely. (I removed a few bugs in the documentation at https://en.bitcoin.it/wiki/OP_CHECKSIG - but I did not implement  the hashtype SIGHASH_SINGLE and therefore did not check its documentation (because it is currently not used in the main block chain.) Thus I can now judge your "several orders of magnitude slower".
Indeed, roughly 99% of the CPU time is used for evaluation of all these OP_CHECKSIG (about 8 hours real time on my 2 GHz Athlon64-PC) when checking the total block chain.

My (first) guess: Either we need a much faster implementation of  the openssl function ECDSA_verify  or abandon this OPCHECK_SIG. E.g. a hash computation (and its verification) only should be of equal security, but it is much faster than verifying a signature. Already some bitcoin-transactions in the block chain have used a "OP_HASH160 <14> OP_EQUAL"-script as txout-script.

smtp

[Pieter Wuille]

Using hashes for signatures, where you reveal a secret to spend an output that contained it hash, you give the world the ability to steal your coins as soon as you spend them. For example, any miner could change the transaction to send it them self by using your revealed secret. This is an inherent problem with symmetric cryptography. You need some secret that only you know to prove your identity.

I have seen systems validate over 5000 tx/s (on a 6-core, hyperthreading machine with hal's optimized ECDSA code), so I think in the near future, we're good (at least until the block size limit would be lifted). Not every machine will be a full node, but if hardware to keep up and sync at least at over 1 block/s, is available to consumers, we probably don't have that much to worry about.

[smtp]

Using hashes for signatures, where you reveal a secret to spend an output that contained it hash, you give the world the ability to steal your coins as soon as you spend them. For example, any miner could change the transaction to send it them self by using your revealed secret. This is an inherent problem with symmetric cryptography. You need some secret that only you know to prove your identity.

I c, Thx. But to spend something I need NOT to prove my identity principially. The problem in the bitcoin-model are the miners, because only this special subcommunity can put my transactions (or what ever they want) into blocks. Thus I must trust them not to change my transaction or need another algorithm (signature) to check/prove that theses did not change my transaction. This I had forgoten. :-/ I always feel uncomfortable by relaying on these miners in this bitcoin-world - I mean, to be unable to write my wanted transactions by myself into the block chain.

I have seen systems validate over 5000 tx/s (on a 6-core, hyperthreading machine with hal's optimized ECDSA code), so I think in the near future, we're good (at least until the block size limit would be lifted). Not every machine will be a full node, but if hardware to keep up and sync at least at over 1 block/s, is available to consumers, we probably don't have that much to worry about.

Fine, and what about the guys still have half the CPU-power (or less) than me? I did a bit preciser estimation: I needed 160 sec to read the block chain up to height 21500 (and get all data in place), but then needed 1031 sec (the wrongly 1191 sec was including the 160 secs) to read and verify the next 1283 blocks (which contain a total of 838243 OP_CHECKSIG). Which shows a speed of 813 tx/s (not 704 tx/s) (script-verifications) -- far more realistic/typical than your 6-core hyperthreading parallel machine, IMHO.

BTW: I believe the most cpu-time consuming part of the signature-verification is the modulo-n arithmetic (with n a 256-bit number).

smtp

[Pieter Wuille]

Thus I must trust them not to change my transaction or need another algorithm (signature) to check/prove that theses did not change my transaction.

Not just miners. Anyone relaying your transaction could modify it without invalidating it, with hash-locked transactions.

Fine, and what about the guys still have half the CPU-power (or less) than me? I did a bit preciser estimation: I needed 160 sec to read the block chain up to height 21500 (and get all data in place), but then needed 1191 sec to read and verify the next 1283 blocks (which contain a total of 838243 OP_CHECKSIG). Which shows a speed of 704 tx/s (script-verifications) -- far more realistic/typical than your 6-core hyperthreading parallel machine, IMHO.

I was talking about the near future - not about this very moment. The fact that the hardware exists today means it is accessible for consumers, which is what matters (i.e., one doesn't need a big investment to run/maintain a validating node). Also, not everyone needs to run a validating node (but the more accessible we can make it to do so, the better).

[smtp]

I've uploaded a new set of builds of bitcoind/Bitcoin-Qt to my builds page. The new "turbo" build is based on current master on git, plus some more experimental patches (Hal Finney's optimized secp256k1 code, and parallel script verification). The source code for these can be found here (.tgz archive here).

I just have downloaded the current source version (1869678 bytes), "tar -xf bitcoin-turbo.tar.gz" it and call in the new created directory bitcoin-turbo the "qmake" . But gets lots of error messages about being unable to find all these local language files:

Project MESSAGE: Building with UPNP support
Project MESSAGE: Building with UPNP support
lrelease warning: Met no 'TRANSLATIONS' entry in project file '/home/achim/Downloads/bitcoin-turbo/bitcoin-qt.pro'
RCC: Error in 'src/qt/bitcoin.qrc': Cannot find file 'locale/bitcoin_bg.qm'
RCC: Error in 'src/qt/bitcoin.qrc': Cannot find file 'locale/bitcoin_ca_ES.qm'
RCC: Error in 'src/qt/bitcoin.qrc': Cannot find file 'locale/bitcoin_cs.qm'
....
RCC: Error in 'src/qt/bitcoin.qrc': Cannot find file 'locale/bitcoin_zh_CN.qm'
RCC: Error in 'src/qt/bitcoin.qrc': Cannot find file 'locale/bitcoin_zh_TW.qm'

So, what are the exact preconditions/dependencies for compiling this turbo-bitcoin source successfully? (Are there linux-binaries availible?).

I want to make a speed measurement of blockchain verification with this bitcoin-turbo release compared to my own blockparser.
Currently I'm able to make 939.88 script-verification/sec with my 2 GHz AMD Athlon(tm) 64 Processor 3200+ stepping 00 processor (a few years old)
using gcc 4.4.5 with (my default) options -march=athlon64 -O2.
Thus I wonder, a new, say 3 GHz cpu with 6 cores should even be able to acomplish 10000 script-verification / sec or more today ... not to mention optimized secp256k1 code.

Appendum: "qmake  -v" results in   
QMake version 2.01a
Using Qt version 4.7.2 in /usr/lib64/qt4

smtp

[Pieter Wuille]

I just have downloaded the current source version (1869678 bytes), "tar -xf bitcoin-turbo.tar.gz" it and call in the new created directory bitcoin-turbo the "qmake" . But gets lots of error messages about being unable to find all these local language files:

Project MESSAGE: Building with UPNP support
Project MESSAGE: Building with UPNP support
lrelease warning: Met no 'TRANSLATIONS' entry in project file '/home/achim/Downloads/bitcoin-turbo/bitcoin-qt.pro'
RCC: Error in 'src/qt/bitcoin.qrc': Cannot find file 'locale/bitcoin_bg.qm'
....

So, what are the exact preconditions/dependencies for compiling this turbo-bitcoin source successfully? (Are there linux-binaries availible?).

Building works fine on my laptop, but I believe I hit the same problem when trying to build release binaries for Linux in gitian. It may require the package 'qt4-linguist-tools'.

Thus I wonder, a new, say 3 GHz cpu with 6 cores should even be able to acomplish 10000 script-verification / sec or more today ... not to mention optimized secp256k1 code.

I've certainly seen more than that - 6000tx/s is over 12000txins/s.

[smtp]

Building works fine on my laptop, but I believe I hit the same problem when trying to build release binaries for Linux in gitian. It may require the package 'qt4-linguist-tools'.

You deduce from your environment to other people's environment.
Seriously, when calling "qmake" in the directory $HOME/tmp/bitcoins/bitcoin-0.7.1-linux/src I get no errors but the message

Project MESSAGE: Building with UPNP support
Project MESSAGE: Building with UPNP support
Removed plural forms as the target language has less forms.
If this sounds wrong, possibly the target language is not set or recognized.


I don't know whether this should sound wrong or not, but obviously something has changed regarding "bitcoin-qt.pro" compared to the old release 0.7.1.

smtp

[Pieter Wuille]

Building works fine on my laptop, but I believe I hit the same problem when trying to build release binaries for Linux in gitian. It may require the package 'qt4-linguist-tools'.

You deduce from your environment to other people's environment.

That's why I add that builds in gitian show the same problem, so I confirm there is a problem here. I just haven't spent the time investigating why.

[Pieter Wuille]

That's why I add that builds in gitian show the same problem, so I confirm there is a problem here. I just haven't spent the time investigating why.

There was a syntactic error in the bitcoin-qt.pro file (which was fixed before, but it seems the fix got lost). Newer versions of lrelease didn't break on it, but older ones did. It should be fixed now.

[smtp]

There was a syntactic error in the bitcoin-qt.pro file (which was fixed before, but it seems the fix got lost). Newer versions of lrelease didn't break on it, but older ones did. It should be fixed now.

A typo -- a single lost ". Unbelievable that this did not trigger a syntax error message in qmake.

Thanks Pieter, and sorry that I have pressed you effectively to debug it for me. :-/

smtp

[smtp]

Thanks Pieter, and sorry that I have pressed you effectively to debug it for me. :-/

I wonder whether I should point you to my next problem? "qmake" seems to finish fine and then I type "make" resulting in:

....
make[1]: Leaving directory `/home/achim/Downloads/bitcoin-turbo/src/leveldb'
cd /home/achim/Downloads/bitcoin-turbo; /bin/sh share/genbuild.sh /home/achim/Downloads/bitcoin-turbo/build/build.h
fatal: Not a git repository (or any parent up to mount parent )
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
....

and a bit later it aborts in src/db.h:14:20: Error: db_cxx.h was not found. So I presume this bundled bitcoin-turbo.tar.gz assumes still some Git-structure/environment/dependency to work correctly. /usr/include/db4.8/db_cxx.h exists - but this is not on the /home filesystem and I dislike to test it on the (readonly) /usr filesystem.

BTW: If you dislike that I'm able to compile your pre-release without Git-knowledge, no problem for me if you thus refuse to support me.

Appendum: Meantime I have set/create an environment variable GIT_DISCOVERY_ACROSS_FILESYSTEM=1 , but this has no effect. Same failure when calling "make".

smtp

[Pieter Wuille]

The error is harmless. It uses git to determine the current version, but it has a fallback in case that fails.

The db_cxx error means you don't have libdb4.8++-dev or libdb5.1++-dev installed. See doc/build-unit.txt for instructions.

[smtp]

The error is harmless. It uses git to determine the current version, but it has a fallback in case that fails.

The db_cxx error means you don't have libdb4.8++-dev or libdb5.1++-dev installed. See doc/build-unit.txt for instructions.

Sorry, that I like to disagree. The error message is from the compiler or preprocessor (not linker) that it don't find the (source) file db_cxx.h (which is in /usr/include/db4.8 as I wrote).

Meanwhile I discovered in bitcoin-qt.pro these comments:

# Dependency library locations can be customized with:
#    BOOST_INCLUDE_PATH, BOOST_LIB_PATH, BDB_INCLUDE_PATH,
#    BDB_LIB_PATH, OPENSSL_INCLUDE_PATH and OPENSSL_LIB_PATH respectively
and after inserting
BDB_INCLUDE_PATH=/usr/include/db4.8

directly after these comments lines. It worked.
Now it stops later complaining about

src/net.cpp:18:32: Error: miniupnpc/miniwget.h:

So it looks like I must check where all the needed PATHS really are and write it into bitcoin-qt.pro .... wondering that the default include PATH is not searched by qmake. Well, thanks to only 3 different used libraries I need hopefully at most to customize 6 paths.

smtp

[smtp]

Dear Pieter

I'm a very little bit upset due to your quick and dirty patching of your current bitcoin-turbo/bitcoin-qt.pro file.


Why?

After disabling UPNP-support via the line
UPNP=-
in the bitcoin-qt.pro file everything works fine until the final linkage. There I got the error message:

g++ -static -fstack-protector-all -Wl,-O1 -Wl,-rpath,/usr/lib64/qt4 -o bitcoin-qt build/bitcoin.o build/bitcoingui.o ......
......  -L/usr/X11R6/lib -lQtCore -lgthread-2.0 -lrt -lglib-2.0 -lpthread

/usr/lib/gcc/x86_64-pc-linux-gnu/4.4.5/../../../../x86_64-pc-linux-gnu/bin/ld: cannot find -lboost_system

But these library seems to be in the standard /usr/lib directory. Thus I thought either the line was too long (2486 characters) or some kind of order
problem with the -L$LIBPATH -llibrary was the reason. After about a quarter hour -- I couldn't believe why g++ suddenly doesn't see these libraries --  my eyes
catched at the beginning of the line: "-static " -- and remembered the diff of the new made bitcoin-qt.pro file from today and its predecessor in my downloaded *.tar.gz file. You have inserted

# use static linking
QMAKE_LFLAGS *= -static

in the bitcoin-qt.pro  newly!

I removed the "-static" from the above g++ line and no problem to do the default dynamic library linkage, getting a 13 MB huge dynamical linked 64bit-executable bitcoin-qt! You should tell everybody, that you now expect all libraries to be statically linked! A new feature of the upcoming 0.8.0-release?

BTW: Of course I like to encourage (dispite of all my critic) to continue your valuable work.

Best wishes,
achim

[Pieter Wuille]

Oh, that's not a change I made, though I should have caught it.

The -static was intended to only be added for Windows.

[Pieter Wuille]

Running full node means major consumption of upload bandwidth. With my 32 kB total upload, after 30+ nodes connect to me I can
hardly surf the Internet, even very simple websites. Few people I know have even bigger problem because they actualy pay per GB
transfered in any direction, so they stopped running full node.

In such a setup I'd advise you to disable listening, as that will reduce the amount of nodes trying to fetch block data from you significantly.

[smtp]

Oh, that's not a change I made, though I should have caught it.
The -static was intended to only be added for Windows.

Because the bitcoin-turbo.tar.gz file was not changed which I downloaded after you just had announced the remove of the syntax error in bitcoin-qt.pro  today in this thread, I looked into your git-repository and copied the new changed bitcoin-qt.pro and made a diff to my downloaded in the tar-ball.

A last question for today (it got very late and I must get up tomorrow early): What is the x11-resource name of the color which backgrounds the "This is a pre-release ...." warning in the wallet-window and which controls the red of "(not synchron)" , such that I can change these specific colors?

smtp

[Pieter Wuille]

A last question for today (it got very late and I must get up tomorrow early): What is the x11-resource name of the color which backgrounds the "This is a pre-release ...." warning in the wallet-window and which controls the red of "(not synchron)" , such that I can change these specific colors?

I have no clue at all about GUI stuff. Wladimir wrote that.

[Diapolo]

That's why I add that builds in gitian show the same problem, so I confirm there is a problem here. I just haven't spent the time investigating why.

There was a syntactic error in the bitcoin-qt.pro file (which was fixed before, but it seems the fix got lost). Newer versions of lrelease didn't break on it, but older ones did. It should be fixed now.

So it was related to that single " I mentioned a few days ago on Github, HA .

Dia

[Diapolo]

Oh, that's not a change I made, though I should have caught it.

The -static was intended to only be added for Windows.

Sorry, that is fixed in the last revision of the stack-protector pull now .

Dia