python-bitcoinlib: Comprehensive bitcoin library for python

[jgarzik]

GitHub URL: https://github.com/jgarzik/python-bitcoinlib
Repository: git://github.com/jgarzik/python-bitcoinlib.git

The python library for pynode has matured sufficiently to have a home of its own.  The python-bitcoinlib project attempts to present a lightweight, modular, a la carte interface to bitcoin data structures and network protocols.

Features:

• Easy object interface to all bitcoin core data structures: block, transaction, addresses, ...
• Full transaction script engine
• Fully verifies main and testnet block chains (via pynode)
• ECDSA verification (OpenSSL wrapper)
• Object interface to all known network messages
• Binary encoding/decoding (serialization) for full bitcoin protocol interoperability
• Passes many of the tests shipped with the bitcoin reference client (bitcoind/Bitcoin-Qt)

Like pynode, this library is currently a developer-only release, not recommended for highly secure production sites.

Pull requests, comments, questions and donations always welcome.

[jgarzik]

Updated with several bug fixes, and some new modules (bloom filter).

[Peter Todd]

Updated with several bug fixes, and some new modules (bloom filter).

Note that I only implemented CBloomFilter, but not CMerkleBlock, which means that while you can set a bloom filter you can't actually yet use the filtered blocks that your peers give you.

But as always, pull-req's welcome.

[jgarzik]

Merged an important bug fix, ensuring bug-for-bug compatibility with reference implementation's SignatureHash()

https://github.com/jgarzik/python-bitcoinrpc/ was also merged into this library, so that RPC support does not require a separate library.

Thanks to Peter Todd for both updates.

[Peter Todd]

FWIW I'm working on making python-bitcoinlib more pythonic: https://github.com/petertodd/python-bitcoinlib/tree/pythonize

Right now it's mainly the core transaction and scripting classes that I've modified; you can now write code like:


proxy = bitcoin.rpx.Proxy()

unspent = proxy.listunspent()

(some steps left out deliberately because I haven't tested this code!)

change_out = CTxOut(FIXME, proxy.getnewaddress().to_scriptPubKey())
msg_out = CTxOut(0, CScript([OP_RETURN, 'Hello World!']))

tx = CTransaction(vins, [change_out, msg_out])

r = proxy.signrawtransaction(tx)

if r['complete']:
    print(proxy.sendrawtransaction(r['tx'])
else:
    print("Didn't work!")


Jeff Garzik has said that if I modify pynode to be compatible with my changes he will likely merge them into bitcoinlib proper.

[knowitnothing]

FWIW I'm working on making python-bitcoinlib more pythonic: https://github.com/petertodd/python-bitcoinlib/tree/pythonize

Right now it's mainly the core transaction and scripting classes that I've modified; you can now write code like:


proxy = bitcoin.rpx.Proxy()

unspent = proxy.listunspent()

Hey there retep. Regarding the RPC, have you considered making their calls async ? I have rolled my own implementation that does that using the futures package, it might make everything a bit harder for the conventional user but there are situations where you simply cannot use blocking RPC. I will publish it later somewhere.

[Peter Todd]

Hey there retep. Regarding the RPC, have you considered making their calls async ? I have rolled my own implementation that does that using the futures package, it might make everything a bit harder for the conventional user but there are situations where you simply cannot use blocking RPC. I will publish it later somewhere.

Patches welcome, though I'll warn you I don't actually know if bitcoind itself has the performance to make async RPC worthwhile.

I'm working on classes to do things like manage the blockchain information (eg. the set of all know blocks and block headers) which may lead to a model where it makes more sense to just do it all in bitcoinlib based on a internal database of blockchain information.

FWIW my personal motivation for working on all this stuff is to make it easier to prototype and experiment at a fairly low level for Bitcoin developers. For instance one of the first things I used python-bicoinlib for was to test some DoS attacks against the Bloom filter code and patches defeating those attacks. (that's why I wrote the CBloomFilter class, but haven't written the corresponding CMerkleBlock class needed to make it useful to SPV clients)

[knowitnothing]

Hey there retep. Regarding the RPC, have you considered making their calls async ? I have rolled my own implementation that does that using the futures package, it might make everything a bit harder for the conventional user but there are situations where you simply cannot use blocking RPC. I will publish it later somewhere.

Patches welcome, though I'll warn you I don't actually know if bitcoind itself has the performance to make async RPC worthwhile.

I'm working on classes to do things like manage the blockchain information (eg. the set of all know blocks and block headers) which may lead to a model where it makes more sense to just do it all in bitcoinlib based on a internal database of blockchain information.

FWIW my personal motivation for working on all this stuff is to make it easier to prototype and experiment at a fairly low level for Bitcoin developers. For instance one of the first things I used python-bicoinlib for was to test some DoS attacks against the Bloom filter code and patches defeating those attacks. (that's why I wrote the CBloomFilter class, but haven't written the corresponding CMerkleBlock class needed to make it useful to SPV clients)

That is great, I appreciate the library.

Now, regarding the earlier comment, here is some code that does async RPC: https://gist.github.com/anonymous/6627745

The motivation is not to send many commands at once to saturate (or some other better term) bitcoind, but just to not block the call. When combined with other services, I simply cannot make a blocking call and that is my motivation for doing it like that. Note that in this code, the important pieces (for these non-blocking calls) are the lines 74, 79, and the function generic_cb for handling the results from the async call. This is not a patch but maybe you (or someone else) could consider it (modified or not) for inclusion in the current code or another branch of it.

[Peter Todd]

That is great, I appreciate the library.

Now, regarding the earlier comment, here is some code that does async RPC: https://gist.github.com/anonymous/6627745

The motivation is not to send many commands at once to saturate (or some other better term) bitcoind, but just to not block the call. When combined with other services, I simply cannot make a blocking call and that is my motivation for doing it like that. Note that in this code, the important pieces (for these non-blocking calls) are the lines 74, 79, and the function generic_cb for handling the results from the async call. This is not a patch but maybe you (or someone else) could consider it (modified or not) for inclusion in the current code or another branch of it.

Your code uses the futures library - one of my goals with what I'm doing with python-bitcoinlib is to eventually be able to either port the performance critical parts to Cython, or even better, turn the "consensus critical" parts of the reference node codebase into a separate library that python-bitcoinlib could in turn use; possibly both approaches will be used.

With that in mind I'd rather that bitcoinlib have as few external dependencies as possible, in particular ones that tie code to a particular way of operating. Similarly with the pynode code I'm working on removing dependencies on the Twisted library so that an implementation of a node can use whatever threading model is desired. (possibly none at all)

FWIW Maaku also has a fork of python-bitcoinlib that he is working on and has used for his UTXO ledger prototyping, but he has taken the exact opposite approach and embraced using features that can only be done in interpreted Python.


EDIT: A good approach for your use-case might be to make a async layer library for the python-bitcoinlib library. Basically you would take the bitcoin.rpc.Proxy class and make a sub-class of it that maintained a thread pool and distributed work as required to those threads. If you need to modify the underlying JSON-RPC code as part of this effort I can work with you to make it possible for the jsonrpc module that the Proxy class uses to be replaced too.

[jgarzik]

Would prefer that the direction of python-bitcoinlib head in the Cython-ish direction, while continuing general pythonization.

Pull requests welcome, including the pythonize branch once its mature.

[knowitnothing]

That is great, I appreciate the library.

Now, regarding the earlier comment, here is some code that does async RPC: https://gist.github.com/anonymous/6627745

The motivation is not to send many commands at once to saturate (or some other better term) bitcoind, but just to not block the call. When combined with other services, I simply cannot make a blocking call and that is my motivation for doing it like that. Note that in this code, the important pieces (for these non-blocking calls) are the lines 74, 79, and the function generic_cb for handling the results from the async call. This is not a patch but maybe you (or someone else) could consider it (modified or not) for inclusion in the current code or another branch of it.

Your code uses the futures library - one of my goals with what I'm doing with python-bitcoinlib is to eventually be able to either port the performance critical parts to Cython, or even better, turn the "consensus critical" parts of the reference node codebase into a separate library that python-bitcoinlib could in turn use; possibly both approaches will be used.

...

EDIT: A good approach for your use-case might be to make a async layer library for the python-bitcoinlib library. Basically you would take the bitcoin.rpc.Proxy class and make a sub-class of it that maintained a thread pool and distributed work as required to those threads. If you need to modify the underlying JSON-RPC code as part of this effort I can work with you to make it possible for the jsonrpc module that the Proxy class uses to be replaced too.

Well, it does use the futures package and it was explicitly mentioned in my first reply. It is using futures because (it is easy to use and) it is available on Python 3.2 and newer, so it is not an external dependency there. It also doesn't involve any other large frameworks, like Twisted you mentioned.

Also, I'm surprised to hear there are critical parts requiring Cython here. What kind of usage do you have in mind that actually has a benefit with Cython ? Why not PyPy then ? I would think that the RPC calls alone are a much greater bottleneck than anything else, and the other relatively heavy code are already coded in C.

Your EDIT sounds reasonable, I will take a look into that later and post some code whenever I find some time. Thanks for looking at it.

[Peter Todd]

Would prefer that the direction of python-bitcoinlib head in the Cython-ish direction, while continuing general pythonization.

Pull requests welcome, including the pythonize branch once its mature.

Speaking of, if you have a change to my pythonize branch, feel free to submit a pull-req directly to it as well; I don't plan to rebase that branch prior to merging it with master.

[Peter Todd]

Well, it does use the futures package and it was explicitly mentioned in my first reply. It is using futures because (it is easy to use and) it is available on Python 3.2 and newer, so it is not an external dependency there. It also doesn't involve any other large frameworks, like Twisted you mentioned.

For now Jeff and I want to maintain support for both 3.x and 2.7

Also, I'm surprised to hear there are critical parts requiring Cython here. What kind of usage do you have in mind that actually has a benefit with Cython ? Why not PyPy then ? I would think that the RPC calls alone are a much greater bottleneck than anything else, and the other relatively heavy code are already coded in C.

Well remember that python-bitcoinlib is meant to be a complete library, not just one meant to be used with RPC. Jeff's full node implementation, pynode, would benefit greatly from faster code for the core Bitcoin data structures.

Your EDIT sounds reasonable, I will take a look into that later and post some code whenever I find some time. Thanks for looking at it.

Thanks. Keep me posted when you have some code I can look at.

[knowitnothing]

Thanks. Keep me posted when you have some code I can look at.

Hello again. I just looked into bitcoin/rpc.py more carefully, so before continuing with this I have to say that using HTTP connections that persist for the duration of the object is not an improvement over the "traditional" method. My reasons for that:

• Suppose you just did a proxy = rpc.Proxy(serviceurl) and decided to stop bitcoind at some remote place. It will hang there, waiting for your proxy to die.
• It makes supporting async calls impossible. You will very often hit a ResponseNotReady exception since you might not have read the response of an earlier call before making another request.

Also, the current way this is coded, I no longer see a simple way to subclass Proxy and make the calls async. Maybe the easiest way would involve firing a new thread for each call and let it handle in the standard blocking way. The futures package makes this a lot easier (and remember it is just a backport of an existing package in the standard library at Python 3.2+), although the code at rpc.py would need to be rewritten. This feature is decisive for me in order to use the package, but it might just be me.