Python implementations

[joecascio]

Now that the compute-intensive parts of mining are being farmed out to ASICs and other specialty hardware, is there any thought being given to developing the rest of the server code and the client using something higher level than C++, like Python? The productivity and quality gains might be significant.

If so, are there any Python implementations that seem to be gaining favor that I can get on board with?

Thank you!!

[1714027141]

1714027141

[1714027141]

1714027141

[1714027141]

1714027141

[maaku]

https://github.com/jgarzik/pynode

https://github.com/monetizeio/python-bitcoin

[joecascio]

Thank you!!

[Sukrim]

An interesting project might be writing a Bitcoin client tailored to the needs of miners, setting custom fee schemes etc. might be something that could really benefit them. You could also think of some things that they might immediately like e.g. payout transactions that are signed and transmitted from other pools get included in blocks for free.

Edit:
A big part of Armory is also written in Python.

[Red Emerald]

Sadly "higher-level" is usually synonymous with "slower."  Especially when it comes to some of the crypto stuff that bitcoin uses so much of.  This means that there will probably always be some c or c++ in the client.

There are quiet a few python implementations being worked on.  Armory and electrum are two that I have been using for quite awhile now.  They are both beta, but both work well.

Armory is designed for security conscious users.

Electrum uses a client/server model designed for speed and usability.  The stratum mining protocol by slush which claims support for 18 EHash/s from a single TCP connection uses an electrum server.  I'm not sure if the mining part of the server is published yet, but I think it is.

[joecascio]

Sadly "higher-level" is usually synonymous with "slower."  Especially when it comes to some of the crypto stuff that bitcoin uses so much of.  This means that there will probably always be some c or c++ in the client.

There are quiet a few python implementations being worked on.  Armory and electrum are two that I have been using for quite awhile now.  They are both beta, but both work well.

Armory is designed for security conscious users.

Electrum uses a client/server model designed for speed and usability.  The stratum mining protocol by slush which claims support for 18 EHash/s from a single TCP connection uses an electrum server.  I'm not sure if the mining part of the server is published yet, but I think it is.

Thanks for the recommendations on Armory and Electrum.

The python pycrypto package uses python-wrapped C implementations for the performance-intensive classes, if my cursory perusal is correct. What I'd consider higher-level applications, like for instance, escrow, refundable deposits or micropayments are not typically that performance sensitive, and happen at "human speed". So as long as the real crypto-crunching hashes and public key algorithms are encapsulated in classes implemented in C, python or ruby are good choices to knit these lower level functions into bigger, more human-relevant applications.

[Red Emerald]

The python pycrypto package uses python-wrapped C implementations for the performance-intensive classes, if my cursory perusal is correct. What I'd consider higher-level applications, like for instance, escrow, refundable deposits or micropayments are not typically that performance sensitive, and happen at "human speed". So as long as the real crypto-crunching hashes and public key algorithms are encapsulated in classes implemented in C, python or ruby are good choices to knit these lower level functions into bigger, more human-relevant applications.

Yeah definitely.  Just wanted to make sure you knew.  Sometimes people like to implement things in pure-python just for the fun of it.

[jgarzik]

The python pycrypto package uses python-wrapped C implementations for the performance-intensive classes, if my cursory perusal is correct. What I'd consider higher-level applications, like for instance, escrow, refundable deposits or micropayments are not typically that performance sensitive, and happen at "human speed". So as long as the real crypto-crunching hashes and public key algorithms are encapsulated in classes implemented in C, python or ruby are good choices to knit these lower level functions into bigger, more human-relevant applications.

Sadly, this is not enough.  pynode is a "full node" implementation.  It fully verifies all bitcoin blockchain data, including scripts and ECDSA signatures.

pynode uses OpenSSL/hashlib for all crypto and ECDSA, which are, of course, C-based python wrapper modules.

CPU profiling of pynode indicates that the vast majority of CPU time -- a significant, noticeable slowdown -- occurs in simple Python data structure copying.  Bitcoin's SignatureHash function requires copying an entire CTransaction, modifying it a bit, then hashing the serialized result.

This would not be noticeable in a lightweight client, but it is the #1 cause of slowdown when verifying a bitcoin block.

[maaku]

CPU profiling of pynode indicates that the vast majority of CPU time -- a significant, noticeable slowdown -- occurs in simple Python data structure copying.  Bitcoin's SignatureHash function requires copying an entire CTransaction, modifying it a bit, then hashing the serialized result.

This would not be noticeable in a lightweight client, but it is the #1 cause of slowdown when verifying a bitcoin block.

This could be mostly eliminated by refactoring how serialization occurs in transaction validation (serialize directly to sighash format without intermediary copying). It's something I'd been planning to add, but haven't gotten around to yet

[Peter Todd]

Sadly, this is not enough.  pynode is a "full node" implementation.  It fully verifies all bitcoin blockchain data, including scripts and ECDSA signatures.

pynode uses OpenSSL/hashlib for all crypto and ECDSA, which are, of course, C-based python wrapper modules.

CPU profiling of pynode indicates that the vast majority of CPU time -- a significant, noticeable slowdown -- occurs in simple Python data structure copying.  Bitcoin's SignatureHash function requires copying an entire CTransaction, modifying it a bit, then hashing the serialized result.

Have you looked into using ''.join() or StringIO/FileIO rather than building up strings with simple concatenation? Python is a lot better at avoiding the n^2 naive immutable string concatenation slowdown than it used to be - the r += foo for instance is handled as an append in place most of the time - but given that the serialization code in pynode is creating strings with code spread out over multiple functions and modules you may be defeating that optimization. In particular I took a quick look at the 2.7 internals and while I may be misunderstanding things it seems that if a string is created in a different module the resize-in-place optimization is disabled for some reason.


My other suggestion would be to look into using Cython. Roughly speaking it's a Python compiler, however really you have to write Cython code rather than pure-Python to get the benefits. The compiler outputs C-code targeting the Python C API. If you know what you're doing this can be every bit as fast as pure C-code while still being written in Python. In addition regular Python code can use a Cython-compiled module just like any other pure-Python module so interoperability isn't a problem. Even just writing part of Pynode in Cython would be perfectly practical. Windows is well supported BTW; the SciPy scientific python project uses Cython extensively.

In addition while the syntax is pretty ugly Cython even supports a decorator-based way of writing Cython. Thus when a Cython install isn't available you can use the provided pure-Python stub module that simply ignores the decorators allowing you to run your code without Cython available. I haven't used this feature myself - as I say the syntax is ugly - but I hear in some cases it can be quite useful.

[joecascio]

My other suggestion would be to look into using Cython. Roughly speaking it's a Python compiler, however really you have to write Cython code rather than pure-Python to get the benefits. The compiler outputs C-code targeting the Python C API.

zoiks! I'm reminded of cfront.

[Peter Todd]

My other suggestion would be to look into using Cython. Roughly speaking it's a Python compiler, however really you have to write Cython code rather than pure-Python to get the benefits. The compiler outputs C-code targeting the Python C API.

zoiks! I'm reminded of cfront.

Targeting C and the Python C API is a very appropriate choice for Cython. The Python C API is very good and easy to write for. The code that Cython produces is also very readable, so you can get a good understanding of what's going on by reading it. If Cython compiled directly to assembly understanding what's going on under the hood would be far more difficult.

The other thing to keep in mind is that one of the big use-cases for Cython is to interface existing C-code to Python, especially for writing wrappers around C libraries. You can basically write C-code with Python syntax if you want, including calling C API's directly: http://docs.cython.org/src/userguide/external_C_code.html Again, if Cython compiled to assembly debugging such extension modules would be far more difficult.

[jgarzik]

pynode pull requests accepted and encouraged!

Right now, I've been busy with the bitcoin C library implementation and client (libccoin and picocoin), which bumped pynode down on the priority list.

[Peter Todd]

pynode pull requests accepted and encouraged!

Right now, I've been busy with the bitcoin C library implementation and client (libccoin and picocoin), which bumped pynode down on the priority list.

I'll see if I can get a chance to take a look at it. pynode is a really good candidate for Cython as you've written it essentially just like a C program anyway. Basically I'd just have to add static type declarations to the performance critical stuff, and add declarations so performance-critical class instances are backed by staticly-laid-out C structs rather than dictionaries. Once you've done that what Cython emits is pretty much straight-forward C code. With some care performance should be pretty similar to the Satoshi client.

Don't get your hopes up though; I work at a startup. I'm also still plodding away on that coinbase time-stamping project I mentioned awhile back, although rather slowly right now due to my job being busy.

Heck, I initially started writing OpenTimestamps in Cython, before I realized I was being an idiot: merkle tree timestamping is embarrassingly scalable...

[sirk390]

You can also have a look at my library and client:
https://github.com/sirk390/coinpy
However, I'm not yet thinking about taking contributions.

[etotheipi]

Armory is based around a suite of C++ tools that are made accessible to Python through SWIG.  Things like scanning the blockchain would be devastatingly slow if it was done in pure python, but are quite efficient in C++.  Armory tries to do as much processing on data as it can in bulk, in the C++ code, so there is minimal opportunity for python to bloat the process. 

However, I made the decision early on to use crypto++ because I didn't really know much about the different crypto suites.  In hindsight, I wish I'd used OpenSSL, because it looks to be about 10x faster than Crypto++ for ECDSA operations.  I wouldn't do full validation with it (maybe only 200-400 ECDSA signature verifications per sec), but it's more than sufficient to deal with all the other operations you need (verifying only transactions relevant to you, signing lots of inputs to create new transactions, etc).

If you clone Armory from github, checkout the extras/sample_armory_code.py which will show you a few ways to access things through pure C++.  However, I recently converted Armory from single-threaded to multi-threaded, so those samples may not be perfect anymore -- they may require slight modification.  Please PM or email if you have any questions.