Bitcoin Forum

Github URL: https://github.com/jgarzik/pynode

pynode is a simple bitcoin P2P client, originally based on ArtForz' half-a-node, which maintains a blockchain database and TX memory pool.  All bitcoin-related code is separated into a "bitcoin" module, permitting you to use core features such as CBlock or P2P message serialization, without actually using the node itself.

Therefore, pynode has two goals:

1. As a generic "python-bitcoin" library, for any project.  Update: Now a separate project, https://github.com/jgarzik/python-bitcoinlib (https://github.com/jgarzik/python-bitcoinlib)
2. As a P2P node that may be used as a base for other projects, such as a network monitoring node or P2P proxy.

At present, it is mainly for developers and not end users.  Here is what pynode does not do:

• key management (wallet)

Here are the limited things that pynode can do at present:

• Outgoing TCP connection to a single remote P2P node
• Initial blockchain download
• Maintain a transaction memory pool, with TX's removed as they are confirmed in blocks
• Maintain an indexed blockchain database
• Respond to JSON-RPC requests via an internal HTTP server

Over time it will become more capable, but this is an alpha quality release for developers at present.

Reserved.

pynode now successfully verifies all transaction scripts on mainnet.  testnet3 verification still a WIP.

Great work @jgarzik, this is critically important stuff. Even if nobody seems to notice ;)

I noticed it! I am waiting for it to get a little more stable, cause this would make a perfect proxy, have a bitcoind, turn off the RPC and use this so even if the rpc name, and password are figured out, they can't steal any coins.

Definitely interested in this and running it now. Is it normal to see a bunch of blocks labeled as orphan at the first startup?

 Also the example lists /tmp/chaindb for the database location. Perhaps that should be .pynode/ if it will be helpful later to keep the database?

Updated status:

pynode now verifies all known scripts on mainnet and testnet3, except for the OP_IF opcodes.  OP_IF is not present on mainnet right now, so this should not be a problem for testing.

pynode now includes a JSON-RPC server like bitcoind.  Only a few RPC API methods are implemented right now.  Check back in a few days, to see more RPC API calls implemented, such as parts of the raw transaction API.

pynode is not officially stable yet, as script verification is disabled (run testscript.py instead) and chain reorg is not implemented yet.


Possibly yes.  Chain reorganization is not yet implemented.  Small 1-2 block forks happen on mainnet every few days.  If the chain forks, pynode will not understand this, and get stuck.  Known problem, will be fixed.

The workaround is to delete your chain database, and re-download it.  If you know python, you could save download time by writing a simple script that deleted the last 100 blocks.

You may ignore orphans, as long as you see new network blocks being stored in the database, with 'height' increasing.


I trust people can figure this one out on their own :)  They will need avoid using RPC username "XXXX" and password "YYYY" also :)

Updated OP to indicate the "python-bitcoin" nature of the project.  Programmers may use some or all of pynode's library, without using the node itself.

pynode's script engine now successfully verifies all scripts in the mainnet and testnet3 chains.

This is awesome!

Looking forward to seeing chain reorg support.

Because pynode startup may take a while, here is a set of pre-indexed database files you may download:

     http://gtf.org/garzik/bitcoin/chaindb.tar.bz2  (2.3 gigabytes)
     SHA1 a8e093ce7eedb3434a3a97570be8c3b855fa7529

This was generated by pynode git HEAD (commit b93b91b1bb659b0477eeff8b0a6771bdb595f0f4).

If you wish to treat this as untrusted data, you may verify hashes and TX connectivity with the dbck.py script.  Runtime should be under 1 hour in most cases.

If you wish to verify scripts, this will take much longer, possibly 10-12 hours.  testscript.py will verify scripts in the database.

Just pushed chain reorg code to github.  It seems to survive a static test... let's see how it fares on the live network.

Script verification is now enabled by default.

Unfortunately it is quite slow, for an unexpected reason: Python is very slow copying data structures.  Therefore, I strongly urge users to download the chaindb.tar.bz2 archive mentioned earlier in the thread.  Otherwise the block download could take a day or two.

Plans to mitigate this include:

• Do not verify scripts earlier than a checkpoint (reference client does this)
• Store database of only unspent CTxOut (sipa's reference client ultraprune branch does this)
• Swap out slow GDBM database for new pagedb database (https://github.com/jgarzik/pagedb)

In addition to the fully validating node and python-bitcoin module, pynode also maintains a branch that contains a non-validating node calling a "mini-node", naturally found in the "mini-node" branch of pynode.git:

      https://github.com/jgarzik/pynode/tree/mini-node (https://github.com/jgarzik/pynode/tree/mini-node)

This is essentially ArtForz' half-a-node, with the minimum needed to work on mainnet.

Forks have already started appearing.  blkmond inside pushpoold (https://github.com/jgarzik/pushpool) existed for a long time.  And bitcoin network sniffer (https://bitcointalk.org/index.php?topic=102740.0) fork also just appeared.

Thanks jgarzik for pointing this out, I haven't seen this before, so I went and create my own... :(

Thanks to socrates1024, pynode got a much needed upgrade from python's asyncore to the more modern gevent. 

This opens the door to easy multi-peer operation, so expect full node status very soon.

Not sure how this one slipped under my radar ...

Wondering if an Electrum client might be able to modified to talk to a pynode ... or vice versa?

Very nice tool. Am I correct that this could be the basis of a tool to monitor large numbers of addresses for activity?

preserved

What does Electrum use to talk to bitcoind?   pynode provides a JSON-RPC HTTP server interface, just like bitcoind...

Sure.  It could monitor whatever metrics you need, on the bitcoin network.

It uses the Stratum protocol ... but I don't have detailed knowledge of that, yet.

there are 2 electrum applications: the electrum client and the electrum server. The electrum server talks to bitcoind using the JSON-RPC interface but stores it's own blockchain. The electrum client talks to the electrum server using the stratum protocol.

Thanks (again) to socrates1024, pynode has switched to using the leveldb (http://code.google.com/p/leveldb/) database, plus a flat file format for raw blockchain data.

This database upgrade is wholly incompatible with the previous GDBM-based database.

When signature checking is disabled, the entire blockchain may be downloaded in 34 minutes.

When signature checking is enabled, the blockchain download process takes many hours (6-12), due to slowness of python.

The on-disk block file format was just updated to match the network's "block" message format.

This is an incompatible change.  You will need to delete, and re-download, your block databases.

pynode is very much alive and well.  It is becoming popular enough that we can branch off the internal "bitcoin" sub-directory into a proper python library, python-bitcoinlib.

Stay tuned for details.

Library forked, and moved to a new home and forum thread (https://bitcointalk.org/index.php?topic=150436.0):

     https://github.com/jgarzik/python-bitcoinlib (https://github.com/jgarzik/python-bitcoinlib)

Pull requests and donations welcome, as always.

Hello I am a bit new to programming but I would love to experiment around with pynode and python-bitcoinlib are there any guides or source code examples available or would I need to figure that stuff out myself? (I promise not to store anyone's life savings using it until I know the node and libs inside and out inside and out)

pynode and assorted utilities dbck.py and mkbootstrap.py are essentially source code examples for python-bitcoinlib.  Unfortunately that's it.

Contributed examples are welcome in both projects, however.

Thanks for the quick reply. As soon as I figure stuff out I shall contribute anything useful I do back.

This received several bug fixes, by way of its use of python-bitcoinlib.

Merged "addnodes" support, so that additional nodes may be added via configuration file.

Nice. This is a fun project to play around with.

Is pynode still being updated? Or is there a follow up?