Title: Proposal: New RPC interface for bitcoind Post by: QuinnHarris on June 15, 2013, 08:26:21 PM This is the beginning of a proposal for a new RPC interface for bitcoind to query and receive notifications about the daemon status, block chain or specific transactions. This is incomplete but I am seeking comments if this is a good idea and worth the effort to implement.
This is intended to address the following issues
The current JSON-RPC interface does not provide a mechanism for real time updates or to request specific transactions matching some criteria notably spendable by a certain keys or having a specific scriptPubKey. This proposal is primarily a description of messages between the client and server where the server is bitcoind. Google protocol buffers will be used here though it would be possible to provide similar functionality with JSON, XML or some other format. This proposal does not define the exact mechanism for sending these messages. Accordingly it can be integrated as new calls to the existing interface though this proposal does overlap with some of the existing RPC functions and has distinctly different conventions. This interface allows a client to create, query and get notifications for a context containing a list of public keys, scriptPubKeys or transactions ids the client is interested in. A context is identified by an arbitrary byte string (should it be fixed or limited length) context identifier that could be derived from the keys in the context. This allows multiple clients to share the same context. A single Context message can represent the full state of a context. A notification Context message contains a subset of a full Context message containing all new data. The message is structured so all changes can be represented by either updating a fixed field or adding a new message to a repeated field. An object is removed from a repeated field by adding it to another repeated field that is mutually exclusive from the original. The same message is also used to specify which aspects of the context the client will need notifications for. This enables a client to request just the required information for example allowing a client to receive an account balance without receiving the specific transactions. In addition to client defined contexts each bitcoind will have single Global context containing connected peers, general block chain information or anything else not predicated on something specified by the client. Code: service Bitcoin { Examples Setup context for a set of keys to match a current wallet implementation Send the follow message to AddToContext RPC. Code: Context: { Setup server to send notifications to the client by sending the following message to the NotifyContext RPC. The presence of any value in a field signifies a notification is desired. Code: Notification { Typical notification message would look like Code: Notification { Notifications on new best blocks, similar functionality to -blocknotify Set the following notifications with SetNotify Code: Notification { Notifications will be sent like below for each new best block Code: Notification { Ecommerce site using Hierarchical Deterministic Wallets without any private keys. Only needs to know when coins are sent to specific accounts and how many confirmations. Send the follow message to AddToContext RPC. Code: Context: { Set notifications with SetNotify Code: Notification { Typical notification message would look like Code: Notification { bitcoin library I think it would be advantageous to develop a bitcoin library that uses an interface to the daemon like this in addition to providing tools to manage the private keys either through a local file store or another RPC interface to a private key daemon (TPM secured process, specialized hardware). Ideally this library would be the preferred interface to bitcoin. 0mq This could be implemented over 0mq. It isn't evident to me if there are real world use cases to broadcast (PUB-SUB) notifications for specific contexts (small number of specific transactions). I see there is a 0mq pull request that publishes blocks, transactions and new host notifications and uses REQ-REP to provide access to the JSON-RPC. An ideal 0mq implementation would probably be capable of sending the same context notification messages to multiple hosts but would this ever be needed? My current inclination is to implement this with 0mq and google protocol buffers. The result would be harder to understand on the wire than something like JSON-RPC but would be a little easier to implement and run faster. I would expect such an implementation would scale to anything needed. That said, authentication and encryption would be more important than just about anything offered by 0mq alone and will be a easier with an openssl secured TCP socket just like the current JSON-RPC. Title: Re: Proposal: New RPC interface for bitcoind Post by: 2112 on June 15, 2013, 10:01:59 PM Three quick comments:
1) Your proposal doesn't seem to handle chain reorganizations in any way, i.e. negative confirmations (or de-confirmations) of blocks and transactions. 2) Your proposal doesn't solve the most glaring misfeature of the current bitcoind/bitcoin-qt, i.e. inversion of control related to the ThreadSafeAskFee() function. 3) I don't see how your proposal will interoperate with any of the existing distributed transaction coordinators, e.g. Tuxedo, MSFT DTC, just anything really that would allow Bitcoin software to correctly execute three-phase commit protocol. I'll admit that I haven't throroughly audited your proposal, all I did is just a quick skimming looking for the same problems exhibited in all previous proposals floated here. Thanks. Title: Re: Proposal: New RPC interface for bitcoind Post by: rupy on July 17, 2013, 07:50:28 AM We definitely need to add a long-poll option to listsinceblock, so it can block if result is empty.
Title: Re: Proposal: New RPC interface for bitcoind Post by: jgarzik on July 17, 2013, 09:00:04 AM See the zeromq pull request.
A REST interface has also gained interest in some areas. Title: Re: Proposal: New RPC interface for bitcoind Post by: QuinnHarris on July 17, 2013, 05:18:58 PM Is there agreement that ideally a bitcoin daemon handing the public block chain and a bitcoin GUI should have a complete and well defined interface between them that could work over a network and allow different daemon and client implementations to work together?
Is it worthwhile to separate the daemon and GUI part of bitcoin-qt so the daemon operates in one process and the GUI in another and communicate over a well defined interface? In this case the daemon would not handle private keys in any way, but the GUI would use the interface to observer transactions for specific public keys. I am especially interested in this because I have recently implemented accepting bitcoins on my website and found the current JSON-RPC interface lacking. Notably I do not want to store private keys on my web server yet I saw no way to use bitcoind to observe the relevant transactions when they came in short of implementing the bitcoin protocol. I think a good solution to this problem would be a library with bindings for many languages that connects to bitcoind providing an interface where I can supply a public key and chain code for HD wallets and it will provide new keys in the chain and notification when transactions show up matching these keys. This library/interface could provide all functionality needed by a bitcoin GUI. This should significantly reduce the programming effort to implement new software using bitcoin which I think is essential for wide spread adoption for ecommerce in a decentralized way. I realize BitPay and other services make this process easier which is great but we have an opportunity to make handling bitcoins much easier without relying on a central service. While I realize this is a significant undertaking I would consider this an important feature. Is anyone else working on this? Is there a better approach or a reason this is a bad idea? Are there other clearly more important features? I noticed an email on the bitcoin mailing list on 5/16 titled "Modularizing Bitcoin" that seems to suggest something along this lines. Title: Re: Proposal: New RPC interface for bitcoind Post by: jgarzik on July 17, 2013, 05:40:33 PM Is there agreement that ideally a bitcoin daemon handing the public block chain and a bitcoin GUI should have a complete and well defined interface between them that could work over a network and allow different daemon and client implementations to work together? I think there is agreement that is a long term goal for bitcoind, yes. Title: Re: Proposal: New RPC interface for bitcoind Post by: rupy on July 18, 2013, 04:32:09 AM See the zeromq pull request. A REST interface has also gained interest in some areas. Can you explain? Is there a way to add zeromq to bitcoind? Can you add a step-by-step tutorial? Title: Re: Proposal: New RPC interface for bitcoind Post by: jgarzik on July 18, 2013, 11:59:03 AM See the zeromq pull request. A REST interface has also gained interest in some areas. Can you explain? Is there a way to add zeromq to bitcoind? Can you add a step-by-step tutorial? A pull request is a source code change to bitcoind at https://github.com/bitcoin/bitcoin/ Someone has already written zeromq support for bitcoind. |