[ANN] pynode: Simple bitcoin P2P node

[jgarzik]

Github URL: https://github.com/jgarzik/pynode

pynode is a simple bitcoin P2P client, originally based on ArtForz' half-a-node, which maintains a blockchain database and TX memory pool.  All bitcoin-related code is separated into a "bitcoin" module, permitting you to use core features such as CBlock or P2P message serialization, without actually using the node itself.

Therefore, pynode has two goals:

1. As a generic "python-bitcoin" library, for any project.  Update: Now a separate project, https://github.com/jgarzik/python-bitcoinlib
2. As a P2P node that may be used as a base for other projects, such as a network monitoring node or P2P proxy.

At present, it is mainly for developers and not end users.  Here is what pynode does not do:

• key management (wallet)

Here are the limited things that pynode can do at present:

• Outgoing TCP connection to a single remote P2P node
• Initial blockchain download
• Maintain a transaction memory pool, with TX's removed as they are confirmed in blocks
• Maintain an indexed blockchain database
• Respond to JSON-RPC requests via an internal HTTP server

Over time it will become more capable, but this is an alpha quality release for developers at present.

[jgarzik]

Reserved.

[jgarzik]

pynode now successfully verifies all transaction scripts on mainnet.  testnet3 verification still a WIP.

[maaku]

Great work @jgarzik, this is critically important stuff. Even if nobody seems to notice

[gweedo]

Great work @jgarzik, this is critically important stuff. Even if nobody seems to notice

I noticed it! I am waiting for it to get a little more stable, cause this would make a perfect proxy, have a bitcoind, turn off the RPC and use this so even if the rpc name, and password are figured out, they can't steal any coins.

[weex]

Definitely interested in this and running it now. Is it normal to see a bunch of blocks labeled as orphan at the first startup?

 Also the example lists /tmp/chaindb for the database location. Perhaps that should be .pynode/ if it will be helpful later to keep the database?

[jgarzik]

Updated status:

pynode now verifies all known scripts on mainnet and testnet3, except for the OP_IF opcodes.  OP_IF is not present on mainnet right now, so this should not be a problem for testing.

pynode now includes a JSON-RPC server like bitcoind.  Only a few RPC API methods are implemented right now.  Check back in a few days, to see more RPC API calls implemented, such as parts of the raw transaction API.

pynode is not officially stable yet, as script verification is disabled (run testscript.py instead) and chain reorg is not implemented yet.

Definitely interested in this and running it now. Is it normal to see a bunch of blocks labeled as orphan at the first startup?

Possibly yes.  Chain reorganization is not yet implemented.  Small 1-2 block forks happen on mainnet every few days.  If the chain forks, pynode will not understand this, and get stuck.  Known problem, will be fixed.

The workaround is to delete your chain database, and re-download it.  If you know python, you could save download time by writing a simple script that deleted the last 100 blocks.

You may ignore orphans, as long as you see new network blocks being stored in the database, with 'height' increasing.

Also the example lists /tmp/chaindb for the database location. Perhaps that should be .pynode/ if it will be helpful later to keep the database?

I trust people can figure this one out on their own  They will need avoid using RPC username "XXXX" and password "YYYY" also

[jgarzik]

Updated OP to indicate the "python-bitcoin" nature of the project.  Programmers may use some or all of pynode's library, without using the node itself.

[jgarzik]

pynode's script engine now successfully verifies all scripts in the mainnet and testnet3 chains.

[Ferroh]

This is awesome!

Looking forward to seeing chain reorg support.

[jgarzik]

Because pynode startup may take a while, here is a set of pre-indexed database files you may download:

     http://gtf.org/garzik/bitcoin/chaindb.tar.bz2  (2.3 gigabytes)
     SHA1 a8e093ce7eedb3434a3a97570be8c3b855fa7529

This was generated by pynode git HEAD (commit b93b91b1bb659b0477eeff8b0a6771bdb595f0f4).

If you wish to treat this as untrusted data, you may verify hashes and TX connectivity with the dbck.py script.  Runtime should be under 1 hour in most cases.

If you wish to verify scripts, this will take much longer, possibly 10-12 hours.  testscript.py will verify scripts in the database.

[jgarzik]

Just pushed chain reorg code to github.  It seems to survive a static test... let's see how it fares on the live network.

[jgarzik]

Script verification is now enabled by default.

Unfortunately it is quite slow, for an unexpected reason: Python is very slow copying data structures.  Therefore, I strongly urge users to download the chaindb.tar.bz2 archive mentioned earlier in the thread.  Otherwise the block download could take a day or two.

Plans to mitigate this include:

• Do not verify scripts earlier than a checkpoint (reference client does this)
• Store database of only unspent CTxOut (sipa's reference client ultraprune branch does this)
• Swap out slow GDBM database for new pagedb database

[jgarzik]

In addition to the fully validating node and python-bitcoin module, pynode also maintains a branch that contains a non-validating node calling a "mini-node", naturally found in the "mini-node" branch of pynode.git:

      https://github.com/jgarzik/pynode/tree/mini-node

This is essentially ArtForz' half-a-node, with the minimum needed to work on mainnet.

Forks have already started appearing.  blkmond inside pushpoold existed for a long time.  And bitcoin network sniffer fork also just appeared.

[sebicas]

In addition to the fully validating node and python-bitcoin module, pynode also maintains a branch that contains a non-validating node calling a "mini-node", naturally found in the "mini-node" branch of pynode.git:

      https://github.com/jgarzik/pynode/tree/mini-node

This is essentially ArtForz' half-a-node, with the minimum needed to work on mainnet.

Forks have already started appearing.  blkmond inside pushpoold existed for a long time.  And bitcoin network sniffer fork also just appeared.

Thanks jgarzik for pointing this out, I haven't seen this before, so I went and create my own...

[jgarzik]

Thanks to socrates1024, pynode got a much needed upgrade from python's asyncore to the more modern gevent. 

This opens the door to easy multi-peer operation, so expect full node status very soon.

[marcus_of_augustus]

Not sure how this one slipped under my radar ...

Wondering if an Electrum client might be able to modified to talk to a pynode ... or vice versa?

[FactoredPrimes]

Very nice tool. Am I correct that this could be the basis of a tool to monitor large numbers of addresses for activity?

[Severian]

preserved

[jgarzik]

Wondering if an Electrum client might be able to modified to talk to a pynode ... or vice versa?

What does Electrum use to talk to bitcoind?   pynode provides a JSON-RPC HTTP server interface, just like bitcoind...