Bitcoin Forum
November 30, 2021, 12:49:12 AM *
News: Latest Bitcoin Core release: 22.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: [1]
  Print  
Author Topic: Python implementations  (Read 1562 times)
joecascio
Full Member
***
Offline Offline

Activity: 137
Merit: 100

Semi-retired software developer, tech consultant


View Profile WWW
November 30, 2012, 04:23:55 PM
 #1

Now that the compute-intensive parts of mining are being farmed out to ASICs and other specialty hardware, is there any thought being given to developing the rest of the server code and the client using something higher level than C++, like Python? The productivity and quality gains might be significant.

If so, are there any Python implementations that seem to be gaining favor that I can get on board with?

Thank you!!

Joe Cascio
Python/Django & Android developer
Twitter: @joecascio
1638233352
Hero Member
*
Offline Offline

Posts: 1638233352

View Profile Personal Message (Offline)

Ignore
1638233352
Reply with quote  #2

1638233352
Report to moderator
1638233352
Hero Member
*
Offline Offline

Posts: 1638233352

View Profile Personal Message (Offline)

Ignore
1638233352
Reply with quote  #2

1638233352
Report to moderator
1638233352
Hero Member
*
Offline Offline

Posts: 1638233352

View Profile Personal Message (Offline)

Ignore
1638233352
Reply with quote  #2

1638233352
Report to moderator
"Apparently, so I am told, there exist "people" who prefer to wipe sitting down. From the front. Initial research indicates it could be up to half the population." -- benjamindees
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1638233352
Hero Member
*
Offline Offline

Posts: 1638233352

View Profile Personal Message (Offline)

Ignore
1638233352
Reply with quote  #2

1638233352
Report to moderator
1638233352
Hero Member
*
Offline Offline

Posts: 1638233352

View Profile Personal Message (Offline)

Ignore
1638233352
Reply with quote  #2

1638233352
Report to moderator
1638233352
Hero Member
*
Offline Offline

Posts: 1638233352

View Profile Personal Message (Offline)

Ignore
1638233352
Reply with quote  #2

1638233352
Report to moderator
maaku
Legendary
*
expert
Offline Offline

Activity: 905
Merit: 1006


View Profile
November 30, 2012, 07:10:24 PM
 #2

https://github.com/jgarzik/pynode

https://github.com/monetizeio/python-bitcoin

I'm an independent developer working on bitcoin-core, making my living off community donations.
If you like my work, please consider donating yourself: 13snZ4ZyCzaL7358SmgvHGC9AxskqumNxP
joecascio
Full Member
***
Offline Offline

Activity: 137
Merit: 100

Semi-retired software developer, tech consultant


View Profile WWW
November 30, 2012, 09:42:28 PM
 #3

Thank you!!

Joe Cascio
Python/Django & Android developer
Twitter: @joecascio
Sukrim
Legendary
*
Offline Offline

Activity: 2604
Merit: 1003


View Profile
December 02, 2012, 09:41:26 PM
 #4

An interesting project might be writing a Bitcoin client tailored to the needs of miners, setting custom fee schemes etc. might be something that could really benefit them. You could also think of some things that they might immediately like e.g. payout transactions that are signed and transmitted from other pools get included in blocks for free.

Edit:
A big part of Armory is also written in Python.

https://www.coinlend.org <-- automated lending at various exchanges.
https://www.bitfinex.com <-- Trade BTC for other currencies and vice versa.
Red Emerald
Hero Member
*****
Offline Offline

Activity: 742
Merit: 500



View Profile WWW
December 04, 2012, 05:55:12 PM
 #5

Sadly "higher-level" is usually synonymous with "slower."  Especially when it comes to some of the crypto stuff that bitcoin uses so much of.  This means that there will probably always be some c or c++ in the client.

There are quiet a few python implementations being worked on.  Armory and electrum are two that I have been using for quite awhile now.  They are both beta, but both work well.

Armory is designed for security conscious users.

Electrum uses a client/server model designed for speed and usability.  The stratum mining protocol by slush which claims support for 18 EHash/s from a single TCP connection uses an electrum server.  I'm not sure if the mining part of the server is published yet, but I think it is.

joecascio
Full Member
***
Offline Offline

Activity: 137
Merit: 100

Semi-retired software developer, tech consultant


View Profile WWW
December 04, 2012, 09:03:42 PM
 #6

Sadly "higher-level" is usually synonymous with "slower."  Especially when it comes to some of the crypto stuff that bitcoin uses so much of.  This means that there will probably always be some c or c++ in the client.

There are quiet a few python implementations being worked on.  Armory and electrum are two that I have been using for quite awhile now.  They are both beta, but both work well.

Armory is designed for security conscious users.

Electrum uses a client/server model designed for speed and usability.  The stratum mining protocol by slush which claims support for 18 EHash/s from a single TCP connection uses an electrum server.  I'm not sure if the mining part of the server is published yet, but I think it is.


Thanks for the recommendations on Armory and Electrum.

The python pycrypto package uses python-wrapped C implementations for the performance-intensive classes, if my cursory perusal is correct. What I'd consider higher-level applications, like for instance, escrow, refundable deposits or micropayments are not typically that performance sensitive, and happen at "human speed". So as long as the real crypto-crunching hashes and public key algorithms are encapsulated in classes implemented in C, python or ruby are good choices to knit these lower level functions into bigger, more human-relevant applications.

Joe Cascio
Python/Django & Android developer
Twitter: @joecascio
Red Emerald
Hero Member
*****
Offline Offline

Activity: 742
Merit: 500



View Profile WWW
December 04, 2012, 09:07:55 PM
 #7

The python pycrypto package uses python-wrapped C implementations for the performance-intensive classes, if my cursory perusal is correct. What I'd consider higher-level applications, like for instance, escrow, refundable deposits or micropayments are not typically that performance sensitive, and happen at "human speed". So as long as the real crypto-crunching hashes and public key algorithms are encapsulated in classes implemented in C, python or ruby are good choices to knit these lower level functions into bigger, more human-relevant applications.
Yeah definitely.  Just wanted to make sure you knew.  Sometimes people like to implement things in pure-python just for the fun of it.

jgarzik
Legendary
*
qt
Offline Offline

Activity: 1596
Merit: 1036


View Profile
December 04, 2012, 09:42:12 PM
 #8

The python pycrypto package uses python-wrapped C implementations for the performance-intensive classes, if my cursory perusal is correct. What I'd consider higher-level applications, like for instance, escrow, refundable deposits or micropayments are not typically that performance sensitive, and happen at "human speed". So as long as the real crypto-crunching hashes and public key algorithms are encapsulated in classes implemented in C, python or ruby are good choices to knit these lower level functions into bigger, more human-relevant applications.

Sadly, this is not enough.  pynode is a "full node" implementation.  It fully verifies all bitcoin blockchain data, including scripts and ECDSA signatures.

pynode uses OpenSSL/hashlib for all crypto and ECDSA, which are, of course, C-based python wrapper modules.

CPU profiling of pynode indicates that the vast majority of CPU time -- a significant, noticeable slowdown -- occurs in simple Python data structure copying.  Bitcoin's SignatureHash function requires copying an entire CTransaction, modifying it a bit, then hashing the serialized result.

This would not be noticeable in a lightweight client, but it is the #1 cause of slowdown when verifying a bitcoin block.


Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.
Visit bloq.com / metronome.io
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
maaku
Legendary
*
expert
Offline Offline

Activity: 905
Merit: 1006


View Profile
December 05, 2012, 02:51:49 AM
 #9

CPU profiling of pynode indicates that the vast majority of CPU time -- a significant, noticeable slowdown -- occurs in simple Python data structure copying.  Bitcoin's SignatureHash function requires copying an entire CTransaction, modifying it a bit, then hashing the serialized result.

This would not be noticeable in a lightweight client, but it is the #1 cause of slowdown when verifying a bitcoin block.
This could be mostly eliminated by refactoring how serialization occurs in transaction validation (serialize directly to sighash format without intermediary copying). It's something I'd been planning to add, but haven't gotten around to yet Sad

I'm an independent developer working on bitcoin-core, making my living off community donations.
If you like my work, please consider donating yourself: 13snZ4ZyCzaL7358SmgvHGC9AxskqumNxP
Peter Todd
Legendary
*
expert
Offline Offline

Activity: 1106
Merit: 1059


View Profile
December 05, 2012, 07:14:17 PM
 #10

Sadly, this is not enough.  pynode is a "full node" implementation.  It fully verifies all bitcoin blockchain data, including scripts and ECDSA signatures.

pynode uses OpenSSL/hashlib for all crypto and ECDSA, which are, of course, C-based python wrapper modules.

CPU profiling of pynode indicates that the vast majority of CPU time -- a significant, noticeable slowdown -- occurs in simple Python data structure copying.  Bitcoin's SignatureHash function requires copying an entire CTransaction, modifying it a bit, then hashing the serialized result.

Have you looked into using ''.join() or StringIO/FileIO rather than building up strings with simple concatenation? Python is a lot better at avoiding the n^2 naive immutable string concatenation slowdown than it used to be - the r += foo for instance is handled as an append in place most of the time - but given that the serialization code in pynode is creating strings with code spread out over multiple functions and modules you may be defeating that optimization. In particular I took a quick look at the 2.7 internals and while I may be misunderstanding things it seems that if a string is created in a different module the resize-in-place optimization is disabled for some reason.


My other suggestion would be to look into using Cython. Roughly speaking it's a Python compiler, however really you have to write Cython code rather than pure-Python to get the benefits. The compiler outputs C-code targeting the Python C API. If you know what you're doing this can be every bit as fast as pure C-code while still being written in Python. In addition regular Python code can use a Cython-compiled module just like any other pure-Python module so interoperability isn't a problem. Even just writing part of Pynode in Cython would be perfectly practical. Windows is well supported BTW; the SciPy scientific python project uses Cython extensively.

In addition while the syntax is pretty ugly Cython even supports a decorator-based way of writing Cython. Thus when a Cython install isn't available you can use the provided pure-Python stub module that simply ignores the decorators allowing you to run your code without Cython available. I haven't used this feature myself - as I say the syntax is ugly - but I hear in some cases it can be quite useful.

joecascio
Full Member
***
Offline Offline

Activity: 137
Merit: 100

Semi-retired software developer, tech consultant


View Profile WWW
December 05, 2012, 07:19:56 PM
 #11


My other suggestion would be to look into using Cython. Roughly speaking it's a Python compiler, however really you have to write Cython code rather than pure-Python to get the benefits. The compiler outputs C-code targeting the Python C API.

zoiks! I'm reminded of cfront.

Joe Cascio
Python/Django & Android developer
Twitter: @joecascio
Peter Todd
Legendary
*
expert
Offline Offline

Activity: 1106
Merit: 1059


View Profile
December 05, 2012, 07:37:03 PM
 #12


My other suggestion would be to look into using Cython. Roughly speaking it's a Python compiler, however really you have to write Cython code rather than pure-Python to get the benefits. The compiler outputs C-code targeting the Python C API.

zoiks! I'm reminded of cfront.


Targeting C and the Python C API is a very appropriate choice for Cython. The Python C API is very good and easy to write for. The code that Cython produces is also very readable, so you can get a good understanding of what's going on by reading it. If Cython compiled directly to assembly understanding what's going on under the hood would be far more difficult.

The other thing to keep in mind is that one of the big use-cases for Cython is to interface existing C-code to Python, especially for writing wrappers around C libraries. You can basically write C-code with Python syntax if you want, including calling C API's directly: http://docs.cython.org/src/userguide/external_C_code.html Again, if Cython compiled to assembly debugging such extension modules would be far more difficult.

jgarzik
Legendary
*
qt
Offline Offline

Activity: 1596
Merit: 1036


View Profile
December 05, 2012, 08:00:08 PM
 #13

pynode pull requests accepted and encouraged!

Right now, I've been busy with the bitcoin C library implementation and client (libccoin and picocoin), which bumped pynode down on the priority list.


Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.
Visit bloq.com / metronome.io
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
Peter Todd
Legendary
*
expert
Offline Offline

Activity: 1106
Merit: 1059


View Profile
December 05, 2012, 09:28:55 PM
 #14

pynode pull requests accepted and encouraged!

Right now, I've been busy with the bitcoin C library implementation and client (libccoin and picocoin), which bumped pynode down on the priority list.


I'll see if I can get a chance to take a look at it. pynode is a really good candidate for Cython as you've written it essentially just like a C program anyway. Basically I'd just have to add static type declarations to the performance critical stuff, and add declarations so performance-critical class instances are backed by staticly-laid-out C structs rather than dictionaries. Once you've done that what Cython emits is pretty much straight-forward C code. With some care performance should be pretty similar to the Satoshi client.

Don't get your hopes up though; I work at a startup. Smiley I'm also still plodding away on that coinbase time-stamping project I mentioned awhile back, although rather slowly right now due to my job being busy.

Heck, I initially started writing OpenTimestamps in Cython, before I realized I was being an idiot: merkle tree timestamping is embarrassingly scalable...

sirk390
Full Member
***
Offline Offline

Activity: 153
Merit: 100


View Profile
December 05, 2012, 11:55:38 PM
 #15

You can also have a look at my library and client:
https://github.com/sirk390/coinpy
However, I'm not yet thinking about taking contributions.
etotheipi
Legendary
*
expert
Offline Offline

Activity: 1428
Merit: 1048


Core Armory Developer


View Profile WWW
December 06, 2012, 06:03:07 AM
 #16

Armory is based around a suite of C++ tools that are made accessible to Python through SWIG.  Things like scanning the blockchain would be devastatingly slow if it was done in pure python, but are quite efficient in C++.  Armory tries to do as much processing on data as it can in bulk, in the C++ code, so there is minimal opportunity for python to bloat the process. 

However, I made the decision early on to use crypto++ because I didn't really know much about the different crypto suites.  In hindsight, I wish I'd used OpenSSL, because it looks to be about 10x faster than Crypto++ for ECDSA operations.  I wouldn't do full validation with it (maybe only 200-400 ECDSA signature verifications per sec), but it's more than sufficient to deal with all the other operations you need (verifying only transactions relevant to you, signing lots of inputs to create new transactions, etc).

If you clone Armory from github, checkout the extras/sample_armory_code.py which will show you a few ways to access things through pure C++.  However, I recently converted Armory from single-threaded to multi-threaded, so those samples may not be perfect anymore -- they may require slight modification.  Please PM or email if you have any questions.

Founder and CEO of Armory Technologies, Inc.
Armory Bitcoin Wallet: Bringing cold storage to the average user!
Only use Armory software signed by the Armory Offline Signing Key (0x98832223)

Please donate to the Armory project by clicking here!    (or donate directly via 1QBDLYTDFHHZAABYSKGKPWKLSXZWCCJQBX -- yes, it's a real address!)
Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!