libbitcoin

[genjix]

fullnode example using the python bindings:

https://gitorious.org/libbitcoin-bindings/libbitcoin-bindings/blobs/master/tests/fullnode.py

also if you expose libbitcoin through an RPC interface like Apache Thrift, and publish new confirmed blocks and new validated unconfirmed txs using ZeroMQ, then you get a nice piece of server middleware:

https://github.com/genjix/qlayer/tree/master/src

I've made this before in C++: https://github.com/genjix/query but I think it'd better be served being done in Python.

Also if you have an bunch of dumb slaves with their own blockchain, and they just synchronise with the master (see https://github.com/genjix/slave ) then it's a really fast way to serve the blockchain to tons of people. Since everything runs in its own process too, it's resilient and stable (parts can go offline and everything still functions). On the front you have a load balancer selecting which slave to distribute requests to.

We are porting this to a Python libbitcoin and Twisted (or maybe gevent) architecture. Not sure if we'll stick with Thrift/ZeroMQ.

Code:

import sys

from bitcoin import threadpool, hosts, handshake, network, protocol, session
from bitcoin import leveldb_blockchain, poller, transaction_pool, create_session_params
from bitcoin import hash_transaction
import time

def print_block(block):
    time.ctime(block.timestamp), block.merkle.encode('hex')

class fullnode(object):
    def __init__(self):
        self._net_pool = threadpool(1)
        self._disk_pool = threadpool(1)
        self._mem_pool = threadpool(1)
        self._hosts = hosts(self._net_pool)
        self._handshake = handshake(self._net_pool)
        self._network = network(self._net_pool)
        self._protocol = protocol(self._net_pool,
                                  self._hosts,
                                  self._handshake,
                                  self._network)
        self._chain = leveldb_blockchain(self._disk_pool)
        self._poller = poller(self._mem_pool, self._chain)
        self._txpool = transaction_pool(self._mem_pool, self._chain)
        pars = create_session_params(self._handshake,
				     self._protocol,
				     self._chain,
				     self._poller,
				     self._txpool)
        self._session = session(self._net_pool, pars)
        print "[fullnode] ok"

    def start(self):
        self._protocol.subscribe_channel(self.monitor_tx)
        self._chain.start('database', self.on_chain_start)
        self._chain.subscribe_reorganize(self.on_reorganize)
        self._txpool.start()
        self._session.start(self.on_session_start)
        print "[fullnode.start] ok"

    def stop(self):
        self._session.stop(self.on_session_stop)
        self._net_pool.stop()
        self._disk_pool.stop()
        self._mem_pool.stop()
        self._net_pool.join()
        self._disk_pool.join()
        self._mem_pool.join()
        self._chain.stop()
        print "[fullnode.stop] ok"

    def on_chain_start(self, ec):
        print "[fullnode.chain] started", ec

    def on_session_stop(self, ec):
        print "[fullnode.session] stopped", ec

    def on_session_start(self, ec):
        print "[fullnode.session] started", ec
        if ec:
            self.stop()
            sys.exit(1)

    def on_reorganize(self, ec, height, arrivals, replaced):
        print '[fullnode.reorganize]', height, str(ec), len(arrivals), len(replaced)
        if len(arrivals):
            print ' arrival', print_block(arrivals[0])
        if len(list2):
            print ' replaced', print_block(arrivals[1])
        self._chain.subscribe_reorganize(self.on_reorganize)

    def monitor_tx(self, node):
        print "(fullnode.tx)", node
        node.subscribe_transaction(lambda ec, tx: self.recv_tx(node, tx, ec))
        self._protocol.subscribe_channel(self.monitor_tx)

    def handle_confirm(self, ec):
        print "(fullnode.store) confirm", ec

    def recv_tx(self, node, tx, ec):
        print "(fullnode.recv_tx)", ec, tx
        if ec:
            print "error", ec
            return
        print ' *', len(tx.inputs), len(tx.outputs)
        self._txpool.store(tx, self.handle_confirm, lambda _ec, u: self.new_unconfirm_valid_tx(node, tx, u, _ec))
        node.subscribe_transaction(lambda _ec, _tx: self.recv_tx(node, _tx, _ec))

    def new_unconfirm_valid_tx(self, node, tx, unconfirmed, ec):
        print "(fullnode.valid_tx)", ec, tx, unconfirmed
        tx_hash = hash_transaction(tx)
        if ec:
            print "Error", ec
        else:
            print "Accepted transaction"
            print unconfirmed.__class__
            if not unconfirmed.empty():
               print "Unconfirmed"
               for idx in unconfirmed: 
                   print ' ', idx
            print tx_hash


if __name__ == '__main__':
    app = fullnode()
    app.start()
    raw_input()
    app.stop()
 

[pilotniq]

I'm thinking about making a Bitcoin client that deletes blocks from its blockchain database if they are say more than about 6 blocks deeper than the latest block AND all of its transactions have been spent. I figure this way I can still validate new blocks, ensure that transactions to my addresses are valid, and save lots of disk space.

Can I do this with libbitcoin, but without modifying its source? How hard would it be?

In your initial posts of this thread, you mention MySQL, but I see no trace of MySQL in the latest source I downloaded from Github, just leveldb and bdb. What's up with that?

You also mention the blockchain database as 'append only', does that apply that I can't delete from it?

Thanks for any tips!

[genjix]

Yeah you can do that if you rebuild the database. That means creating a new database (see examples/initchain.cpp), then importing the old blocks into the new one. However block headers are only 80 bytes so you should keep those. In Bitcoin, this is what pruning outputs means (libbitcoin by default doesn't do that). I personally believe that disk space is not a big deal (current unmodified chain is 15 Gb). I'm more worried about fetch speed (not writes) and scalability.

I had a PostgreSQL database in the beginning (hence the bitcoin.sql file in the repo) but eventually moved to bdb. The postgres backend became unmaintained, the API changed a lot and it fell out of development before I dropped it entirely. SQL is too slow for blockchains.

There is a method to delete from the blockchain directly, but I haven't exposed that. It's used for blockchain reorgs. You can use that same code, but make sure you stop the blockchain object before modifying the database. You can then reopen it after:

Code:

leveldb_blockchain* chain = new leveldb_blockchain(pool);
chain->start(dbpath, blockchain_started);
// ... do stuff
// stop/join threadpools using the blockchain.
chain->stop();
delete chain;
// open database, delete whatever you want.
leveldb_blockchain* chain = new leveldb_blockchain(pool);
chain->start(dbpath, blockchain_started);
// continue from where you left off...

something like that.

see the method called "end_slice"

https://github.com/spesmilo/libbitcoin/blob/master/src/blockchain/leveldb/leveldb_chain_keeper.cpp

It's deleting a bunch of blocks from the end of the blockchain.

Also `bool leveldb_chain_keeper::clear_transaction_data(leveldb_transaction_batch& batch, const transaction_type& remove_tx)' (same file) might interest you too. It's deleting specific transactions from the database.

[pilotniq]

I personally believe that disk space is not a big deal (current unmodified chain is 15 Gb). I'm more worried about fetch speed (not writes) and scalability.

I'm thinking about building a device for secure Bitcoin storage based on the Raspberry Pi. It doesn't have to be very fast, but if I want to support a full blockchain for a number of years into the future, the storage would get prohibitive (already is). That's why I'm interested in a pruned blockchain.

I'd much rather use (and possibly contribute to) an existing library than writing my own stuff.

SQL is too slow for blockchains.

Could you elaborate on that? What are the operations that require high speed of database access? I am considering using sqlite for a pruned blockchain, do you think that would not work?

Could I write my own implementation of blockchain which automatically pruned itself without rebuilding, and stored the database (blockchain, transactions, addresses maybe) in its own database? Would it create problems for the rest of libbitcoin?

Yeah you can do that if you rebuild the database.
That means creating a new database (see examples/initchain.cpp), then importing the old blocks into the new one.

Sounds expensive.

I wish there were a bit more comments in the code to explain the concepts, like slices, what the chain keeper, chain organizer do, etc.

There is a method to delete from the blockchain directly, but I haven't exposed that. It's used for blockchain reorgs. You can use that same code, but make sure you stop the blockchain object before modifying the database. You can then reopen it after:

Code:

leveldb_blockchain* chain = new leveldb_blockchain(pool);
chain->start(dbpath, blockchain_started);
// ... do stuff
// stop/join threadpools using the blockchain.
chain->stop();
delete chain;
// open database, delete whatever you want.
leveldb_blockchain* chain = new leveldb_blockchain(pool);
chain->start(dbpath, blockchain_started);
// continue from where you left off...

something like that.

see the method called "end_slice"

https://github.com/spesmilo/libbitcoin/blob/master/src/blockchain/leveldb/leveldb_chain_keeper.cpp

It's deleting a bunch of blocks from the end of the blockchain.

Thanks for the pointers.

Why does the database need to be deleted and then opened in order to delete data from it?

Also `bool leveldb_chain_keeper::clear_transaction_data(leveldb_transaction_batch& batch, const transaction_type& remove_tx)' (same file) might interest you too. It's deleting specific transactions from the database.

Where can I find a description of what's in the database? I'm used to SQL databases with tables, but I understand leveldb only stores key-value pairs? What objects are in the database? Is there for instance a way to find unspent transactions?

[genjix]

I think the Raspberry Pi uses the ARM processor. If you manage to get libbitcoin compiled and running on there, I'll happily add you a remove(tx_hash) method.

SQL is too slow when validating the blocks. Loading the entire blockchain into SQL will take a very long time. However there's a ready made schema there for you if you like.

libbitcoin uses a blockchain interface with different backends. As long as you implement the interface, you should be fine. If you're not too worried about validating blocks, then it should be quite easy (a week(end) of work).

I should comment that part. There's lots of comments in the API, but less so in the deeper parts of the internal machinery. I will get around to that.

You need to close the database handle before writing to it from an external context. That's dangerous otherwise. There should be a single environment open with LevelDB, and you don't want other async writes to conflict while you're doing stuff to the DB.

Finding unspent outputs means doing a linear scan through every output and seeing which ones do not have a corresponding spend entry. The comments for what the tables (called databases in LevelDB) do is in leveldb_blockchain.hpp

[genjix]

Experimental C wrapper: https://github.com/genjix/libbitcoin-c-wrapper/blob/master/fullnode.c

[RoadTrain]

Hello genjix!
I'm curious if your library has already implemented BIP 0037?
I'd like to experiment with this feature.

[genjix]

Lots of new things coming. I'm at a basic milestone where I'm content with the overall internal architecture of the library now.

https://github.com/spesmilo/libbitcoin/commits/master

Next steps is Python bindings, and a blockchain query API. The Python bindings were done before but were experimental. We had some issues with asynchronous calls and the GIL. I'm thinking to simply turn all async calls into sync blocking ones for Python anyway (keep it simple).

The query layer is something really nice:
https://github.com/genjix/query-layer
(see the tutorial.py)
You can move around and examine it using logical Python constructs. I want to expand upon this and also give access to new blocks and verified transactions in the memory pool. Then create an Electrum server protocol implementation chained to this.

Also subvertx (command line tools for Bitcoin using libbitcoin) is a bit out of date and not actively developed. I would like to take that to something serious too but it needs time (if someone wants to volunteer, this is a cool project) to firstly create how the tools should look and then implement them. I can help someone with that if they know C++.
https://github.com/genjix/subvertx
https://bitcointalk.org/index.php?topic=50721.0

Finally I'm working on an Android wallet. It's not using libbitcoin currently, but eventually that will be needed in the future.

There's some other various small tools I've made too. One is a fast-monitor tool which sits in the Bitcoin network and looks at transaction outputs in new blocks and unconfirmed transactions for an address so it can tell a website of a new deposit. The tool doesn't validate the new blocks or transactions and is only meant to be a lightweight thing. I don't see it having too many uses except where a website has TONS of Bitcoin addresses and can't afford to poll the balance for all of them or something.

Most of these goals have been reached.

query API: https://github.com/genjix/obelisk

subvertx (now sx): https://sx.dyne.org/

Next stage is I need a final class for watching an address and giving credit/debit updates, and adding multisig to sx tools.

We're moving into wallet development soon, and other exciting projects to push the boundaries of Bitcoin development.

[Arto]

Just wanted to point out here that only one of the Individual Seat candidates for the ongoing Bitcoin Foundation elections has mentioned libbitcoin so far. That was Joerg Platzer, in the Bitcoin Foundation Candidate Debate hosted by Let's Talk Bitcoin the other day. Find the 7-minute recording of Joerg's Q&A answers at https://soundcloud.com/mindtomatter/bf-debate-joerg-platzer

[2weiX]

Just wanted to point out here that only one of the Individual Seat candidates for the ongoing Bitcoin Foundation elections has mentioned libbitcoin so far. That was Joerg Platzer, in the Bitcoin Foundation Candidate Debate hosted by Let's Talk Bitcoin the other day. Find the 7-minute recording of Joerg's Q&A answers at https://soundcloud.com/mindtomatter/bf-debate-joerg-platzer

+inf


Joerg Platzer needs to be on the board of the Bitcoin Foundation (yes, if you're a member, you can vote for him!)... to show these guys how this is an actual factual thing that happens ON THE STREETS.

Nothing for 'sters and guys in suits.


"This guy is in finance..."

Well THIS guy is ON THE STREET actually WORKING with Bitcoin.

/spamrant

[genjix]

libbitcoin now has a blockchain reorg testing tool!

https://github.com/spesmilo/libbitcoin/tree/master/test/fakechain

Run the bash script run.sh

You can customise the test by editing step.hpp

You can test anything by customising step.hpp. The default test now tries to create 3 chains, handle reorganisations, double spending, spending transactions from alternate branches and duplicate transactions.

$ cd libbitcoin/tests/fakechain/
$ ./run.sh
$ ../../examples/display-last

It works by mining the blocks from the genesis block. It submits a chain of 100 blocks then does the reorg test on top. This is because you cannot spend coinbase transactions that are less than COINBASE_MATURITY (100 blocks).

[phelix]

Trying to build on windows/MinGW for Zeroreserve (https://bitcointalk.org/index.php?topic=295930)

Installed MinGW with gcc 4.8.1

What I got so far:

Code:

mingw-get install autoconf
mingw-get install automake
mingw-get install libtool
in configure.ac: AM_PROG_AR  --> AC_CHECK_TOOL([AR], [ar], [false])     [no idea what I am doing here but seems to work]

---> "autoreconf -i" works ok

Now I am stuck with "./configure --enable-leveldb" --> "could not detect boost libraries"
I have boost installed and can compile Bitcoin but I can not convince libbitcoin configure...   any ideas?

Has anybody ever compiled libbitcoin on windows?

Tried $BOOST_ROOT, --with-boost, --with-bost-libdir and various combinations, e.g.:
./configure --enable-leveldb --with-boost=./libs/boost_1_54_0 --with-boost-libdir=./libs/boost_1_54_0/stage/lib
./configure --enable-leveldb --with-boost=./libs/boost_1_54_0 --with-boost-libdir=./libs/boost_1_54_0/stage
./configure --enable-leveldb --with-boost=./libs/boost_1_54_0 --with-boost-libdir=./libs/boost_1_54_0

What parts of boost should I compile? (chrono, filesystem, program-options, system, thread is what I got now; static linking)

Using boost 1.54.0

[genjix]

Trying to build on windows/MinGW for Zeroreserve (https://bitcointalk.org/index.php?topic=295930)

Installed MinGW with gcc 4.8.1

What I got so far:

Code:

mingw-get install autoconf
mingw-get install automake
mingw-get install libtool
in configure.ac: AM_PROG_AR  --> AC_CHECK_TOOL([AR], [ar], [false])     [no idea what I am doing here but seems to work]

---> "autoreconf -i" works ok

Now I am stuck with "./configure --enable-leveldb" --> "could not detect boost libraries"
I have boost installed and can compile Bitcoin but I can not convince libbitcoin configure...   any ideas?

Has anybody ever compiled libbitcoin on windows?

Tried $BOOST_ROOT, --with-boost, --with-bost-libdir and various combinations, e.g.:
./configure --enable-leveldb --with-boost=./libs/boost_1_54_0 --with-boost-libdir=./libs/boost_1_54_0/stage/lib
./configure --enable-leveldb --with-boost=./libs/boost_1_54_0 --with-boost-libdir=./libs/boost_1_54_0/stage
./configure --enable-leveldb --with-boost=./libs/boost_1_54_0 --with-boost-libdir=./libs/boost_1_54_0

What parts of boost should I compile? (chrono, filesystem, program-options, system, thread is what I got now; static linking)

Using boost 1.54.0

I'll message you via email and we'll sort this out.

[genjix]

Time for a new libbitcoin release courtesy of unSYSTEM!

Shout outs to everyone involved in no particular order: Tristan, Pablo
Castellano, caedes, Santi (d3), Vitalik, Robert, Jaromil, Grazcoin and
others.

Links:

  http://libbitcoin.dyne.org/download/libbitcoin-1.4.tar.bz2

  http://libbitcoin.dyne.org/download/obelisk-0.2.tar.bz2

  http://sx.dyne.org/download/sx-0.2.tar.bz2

This is the first public release of obelisk (scalable blockchain query
infrastructure) and sx (empower the Bitcoin sysadmin) to the world!

Safe journeys fellow travellers!

[virtualcodewarrior]

I just installed Ubuntu 13.10 and rebuild libbitcoin after getting the latest changes from github.
It didn't give any errors as far as I could see but when I tried to build the examples I got the following errors :

Code:

/usr/local/lib/libbitcoin.so: undefined reference to `EC_KEY_free'
/usr/local/lib/libbitcoin.so: undefined reference to `curl_easy_perform'
/usr/local/lib/libbitcoin.so: undefined reference to `d2i_ECPrivateKey'
/usr/local/lib/libbitcoin.so: undefined reference to `BN_add'
/usr/local/lib/libbitcoin.so: undefined reference to `EC_KEY_get0_group'
/usr/local/lib/libbitcoin.so: undefined reference to `BN_set_negative'
/usr/local/lib/libbitcoin.so: undefined reference to `EC_POINT_free'
/usr/local/lib/libbitcoin.so: undefined reference to `BN_sub'
/usr/local/lib/libbitcoin.so: undefined reference to `RIPEMD160_Final'
/usr/local/lib/libbitcoin.so: undefined reference to `curl_easy_init'
/usr/local/lib/libbitcoin.so: undefined reference to `BN_CTX_free'
/usr/local/lib/libbitcoin.so: undefined reference to `BN_set_word'
/usr/local/lib/libbitcoin.so: undefined reference to `BN_CTX_new'
/usr/local/lib/libbitcoin.so: undefined reference to `RIPEMD160'
/usr/local/lib/libbitcoin.so: undefined reference to `SHA1'
/usr/local/lib/libbitcoin.so: undefined reference to `EC_KEY_set_public_key'
/usr/local/lib/libbitcoin.so: undefined reference to `BN_cmp'
/usr/local/lib/libbitcoin.so: undefined reference to `SHA256_Final'
/usr/local/lib/libbitcoin.so: undefined reference to `BN_rand'
/usr/local/lib/libbitcoin.so: undefined reference to `SHA256_Update'
/usr/local/lib/libbitcoin.so: undefined reference to `BN_new'
/usr/local/lib/libbitcoin.so: undefined reference to `BN_clear_free'
/usr/local/lib/libbitcoin.so: undefined reference to `EC_POINT_get_affine_coordinates_GFp'
/usr/local/lib/libbitcoin.so: undefined reference to `EC_POINT_new'
/usr/local/lib/libbitcoin.so: undefined reference to `CRYPTO_free'
/usr/local/lib/libbitcoin.so: undefined reference to `BN_mpi2bn'
/usr/local/lib/libbitcoin.so: undefined reference to `RIPEMD160_Init'
/usr/local/lib/libbitcoin.so: undefined reference to `BN_get_word'
/usr/local/lib/libbitcoin.so: undefined reference to `BN_bin2bn'
/usr/local/lib/libbitcoin.so: undefined reference to `EC_GROUP_new_by_curve_name'
/usr/local/lib/libbitcoin.so: undefined reference to `EC_POINT_mul'
/usr/local/lib/libbitcoin.so: undefined reference to `BN_copy'
/usr/local/lib/libbitcoin.so: undefined reference to `SHA256_Init'
/usr/local/lib/libbitcoin.so: undefined reference to `EC_KEY_copy'
/usr/local/lib/libbitcoin.so: undefined reference to `curl_easy_cleanup'
/usr/local/lib/libbitcoin.so: undefined reference to `BN_div'
/usr/local/lib/libbitcoin.so: undefined reference to `i2o_ECPublicKey'
/usr/local/lib/libbitcoin.so: undefined reference to `ECDSA_size'
/usr/local/lib/libbitcoin.so: undefined reference to `ECDSA_sign'
/usr/local/lib/libbitcoin.so: undefined reference to `BN_mul'
/usr/local/lib/libbitcoin.so: undefined reference to `BN_num_bits'
/usr/local/lib/libbitcoin.so: undefined reference to `EC_POINT_set_affine_coordinates_GFp'
/usr/local/lib/libbitcoin.so: undefined reference to `BN_bn2hex'
/usr/local/lib/libbitcoin.so: undefined reference to `i2d_ECPrivateKey'
/usr/local/lib/libbitcoin.so: undefined reference to `curl_easy_setopt'
/usr/local/lib/libbitcoin.so: undefined reference to `BN_bn2bin'
/usr/local/lib/libbitcoin.so: undefined reference to `BN_bn2mpi'
/usr/local/lib/libbitcoin.so: undefined reference to `EC_KEY_generate_key'
/usr/local/lib/libbitcoin.so: undefined reference to `RIPEMD160_Update'
/usr/local/lib/libbitcoin.so: undefined reference to `SHA256'
/usr/local/lib/libbitcoin.so: undefined reference to `EC_KEY_get0_private_key'
/usr/local/lib/libbitcoin.so: undefined reference to `BN_lshift'
/usr/local/lib/libbitcoin.so: undefined reference to `ECDSA_verify'
/usr/local/lib/libbitcoin.so: undefined reference to `EC_KEY_dup'
/usr/local/lib/libbitcoin.so: undefined reference to `EC_KEY_set_private_key'
/usr/local/lib/libbitcoin.so: undefined reference to `EC_GROUP_get_order'
/usr/local/lib/libbitcoin.so: undefined reference to `o2i_ECPublicKey'
/usr/local/lib/libbitcoin.so: undefined reference to `EC_GROUP_free'
/usr/local/lib/libbitcoin.so: undefined reference to `BN_init'
/usr/local/lib/libbitcoin.so: undefined reference to `BN_free'
/usr/local/lib/libbitcoin.so: undefined reference to `EC_KEY_new_by_curve_name'

I hope someone can help me with this.
Did I forget some step or am I missing some library?

[phelix]

Any news on the windows build? 

[Mike Hearn]

virtualcodewarrior, looks like you're missing OpenSSL.

[virtualcodewarrior]

openssl is installed as far as I can tell (did a reinstall to make sure) and it seems that "curl_easy_perform" should be in libcurl which is also installed but that function is also listed as undefined .

ldconfig has the libraries listed and the "libssl.so.1.0.0" and "libcrypto.so.1.0.0" both exist in "/lib/x86_64-linux-gnu/" directory.

using nm to see the exports from libcrypto.so.1.0.0 does list EC_KEY_free as being exported from it

and libbitcoin.pc has Libs.private: -lcrypto
listed

I am new to using c++ on linux so I might have missed something obvious.

Update:
This problem is caused by changes made between Ubuntu 13.04 and 13.10.
I tried a clean install of both versions of Ubuntu in virtualbox and on 13.04 it will build fine
while on 13.10 it will fail with the undefined references.

I will try and see if I can spot any differences between the two versions.

[genjix]

Hmm strange, anyway I'm upgrading to latest Ubuntu 13.10. Will post a fix here afterwards.

Windows/Mac: in around 1 month (next release). I've got both platforms working before but unsupported.

[genjix]

Fixed.

https://github.com/spesmilo/libbitcoin/commit/e543599b3ac37e4f30370bbf7c1323de1c71fba2

I patched this on the website tarball too.