[RFC] monitor JSON-RPC api (push instead of poll)

[Gavin Andresen]

I've been reworking my old 'monitorreceived' patch to catch up with the latest JSON api changes, and I'm looking for feedback.

New methods I've already implemented:

• monitorblocks <url> [monitor=true]: POSTs a JSON-RPC notification to <url> when new blocks are accepted.
• listmonitored : returns list of URLs that are monitoring new blocks
• getblock <depth> : Returns information about block at depth <depth>

getblock/monitorblocks give this information (this is one of the -testnet blocks):

Code:

{
 "hash":"000000002eb339613fd83ea65f3620cc85a8247893ea7f1f85e40fc9632db50f",
 "blockcount":21109,
 "version":1,
 "merkleroot":"c0efb898417b55dbec645eeda3e5a3c092c22e21e17f423876e858bc223e721c",
 "time":1294269726,
 "nonce":595884571,
 "difficulty":4.81431771,
 "tx":[
   "ea214bb68aeca12eea6e8467b3b72dcf4c3aef0de015e5d21b51d63ed9fba1a9",
   "90727f2409ea326fcb5e218c1c4213608bf3f2e9d18b3191e52fff86ccda7701"
  ],
 "hashprevious":"0000000002889316c2e34614eadcafea44cf0899945dde0da0fa7a765058aca6"
}

The monitor JSON-RPC notification wraps that information with a call to "monitorblock" -- see http://gavinpostbin.appspot.com/15depef for exactly what a notification looks like.

I'm thinking about adding notification for 0-confirmation wallet transactions, too; something like:

monitortx <url> [monitor=true]  : POST to url when wallet transactions (sends and receives) are accepted.

Information posted would be the same as you get from calling gettransaction, and I'll change listmonitored to return lists of { "category" : block/tx, "url" : url }.

Possible reasons NOT to add this to mainline bitcoin:

1. I'm using boost::xpressive (regular expression library) to parse the urls.  Bitcoin is already dependent on lots of other pieces of Boost, and xpressive is compiled as a header-only dependency (no changes to the Makefiles)... but I wouldn't be surprised if using xpressive causes problems on SOME compiler somewhere.

2. POSTing to https: URLs won't work if you're running on Windows (any windows/mingw experts want to take another crack at getting full openssl working?).

3. Related to https/ssl:  if you POST transactions to a non-ssl url, somebody eavesdropping on your packets will be able to figure out which bitcoin addresses belong to you.  This is a potential privacy issue.

As always, feedback, encouragement, and reality-checks are welcome.

[1714903033]

1714903033

[1714903033]

1714903033

[1714903033]

1714903033

[1714903033]

1714903033

[jgarzik]

+1, I think this feature should go upstream

Not sure what's going on with OpenSSL on Windows.  OpenSSL SSL_* C API works for me, when using Fedora 13/14 + their mingw installation, where openssl DLLs come pre-built.  cpuminer's Windows installer (which runs 100% on Linux) ships openssl as a CURL dependency. I know that data point is not very helpful, but well, there it is.

[davout]

Maybe this could be somehow modular, I assume that the people who will find the most use to this kind of feature are running the client on a server anyway, I'd also assume a Unix flavor in most cases.

Maybe a windows client should include the GUI + whats necessary to run an external miner but not the advanced features such as this one.

[jgarzik]

I think we should go ahead with 'monitorblock', because it is simple and immediately useful.  monitortx requires a more complex API, that specifes how bitcoin should select (match) which transactions to POST.

Suggested RPC API:

     monitorblocks add URL [senddata]

          Add URL to list of those monitoring incoming blocks.
          senddata=true: bitcoin will send block contents in JSON format
          senddata=false (default): bitcoin will indicate new block chain height to URL, and nothing else

          Returns: numeric id, represented URL just added to monitor list

     monitorblocks del id

          Deletes URL [id] from monitor list.

          Returns: success / failure

     monitorblocks clear

          Clears monitor list, and stops all monitoring.

          Returns: success / failure

[midnightmagic]

A client that connects up, subscribes, and then gets updates as they happen might be an easier start. Plus you don't have to worry about stale URLs, the subscription is cleared if the connection breaks, you don't have to have any retry logic directly in the server, you can leave out additional Boost dependencies, you can rely on the client having received prior updates (and can thus send only that which has actually changed), and in general it's less surprising for users who might be wondering after a while why some bitcoin daemon keeps plugging away at them.

A subscription client can just as easily act as the glue which sends the monitored updates to something else, you don't have to be the reason why someone's bitcoins just got stolen, and I expect things would be much simpler overall..?

[jgarzik]

A client that connects up, subscribes, and then gets updates as they happen might be an easier start

This is true, and in fact, that was my plan for a push-mining implementation.   hmmmm.

[jon_smark]

+1

In another thread I've already expressed my desire for something like getblock, so you know I'll be happy with that feature...

As for the Push API, of course that jgarzik is right concerning the need for the API to also handle removal and listing of listeners. However, I also agree with midnightmagic concerning the potential pitfalls of Push.  I've expressed those same concerns in a previous thread (see http://bitcointalk.org/index.php?topic=3092.0 for the gory details).

How will the Push API handle the case of the listener being temporarily unavailable, or what happens if the listener gets the message but dies before it has a chance of processing it?  In the latter case, careful listeners will note there is discrepancy in the block chain and will have to issue a getblock anyway...

[jgarzik]

How will the Push API handle the case of the listener being temporarily unavailable, or what happens if the listener gets the message but dies before it has a chance of processing it?  In the latter case, careful listeners will note there is discrepancy in the block chain and will have to issue a getblock anyway...

When monitoring blocks, it is obvious when you have missed blocks from the data.  A block chain reorg is something to think about, though.  And when monitoring for work, it is ok to miss a 'getwork' unit and simply retreive a fresh one.

Monitoring transactions is, as noted, more difficult from this perspective as well...

[Gavin Andresen]

A client that connects up, subscribes, and then gets updates as they happen might be an easier start.

Connects via another port?

Or would you teach bitcoin's minimal http implementation to keep the connection open?  (and service multiple connections at once)

I've got a use case (google App Engine) where a persistent connection to bitcoin isn't possible (App Engine apps can fetch URLs, and can act as 'web hooks', but can't just open a socket and listen).

[jgarzik]

A client that connects up, subscribes, and then gets updates as they happen might be an easier start.

Connects via another port?

Or would you teach bitcoin's minimal http implementation to keep the connection open?  (and service multiple connections at once)

As discussed on IRC, bitcoin's JSON-RPC httpd should be converted to use select(2) (or poll or epoll) as is done now in ThreadSocketHandler2(), which enables HTTP/1.1 persistent connections.

But let's leave that aside for a second.

However, for monitorblocks and push mining ("pushwork"), I would urge consideration of running a TCP server on a third port (8331?) with a simple protocol designed for long-running TCP connections, and data broadcasting.

Server (bitcoind) simplified pseudocode for monitorblocks:

Code:

processed_new_block_event(block):
    for each incoming TCP connection on port 8331:
        if (connection mask & BROADCAST_BLOCK)
            write(socket fd, block)

Client simplified pseudocode for monitorblocks:

Code:

TCP connect to host:8331
send curtime, protocol version, client version string, username, [possibly empty] list of options, sha256(all previous args + password)
wait for server response ("auth ok, here are my capabilities", or "rejected, goodbye")
send "send me new blocks" message

while (true)
     select/poll for new input from remote server (bitcoind)
     read message from server ("hi, I'm a new block!")
     take client-specific action based on message...

In this manner, bitcoind's logic is simple because you don't have to keep track of a list of monitored URLs, nor care about retrying a monitorblocks HTTP POST if server is down, etc.

This same client connection model can be used for "push" mining, where miners are automatically delivered new work if bitcoind receives a new TX or block from the P2P network.

Server (bitcoind) simplified pseudocode for push mining:

Code:

processed_new_block_event(block):
    for each incoming TCP connection on port 8331:
        if (connection mask & BROADCAST_WORK)
            work = RPC.getwork()
            write(socket fd, work)

And client miners would behave similarly (warning to efficiency nuts: this is a simplified example):

Code:

TCP connect to host:8331
send curtime, protocol version, client version string, username, [possibly empty] list of options, sha256(all previous args + password)
wait for server response ("auth ok, here are my capabilities", or "rejected, goodbye")
send "send me new work" message

start proof-of-work GPU/CPU mining threads in background:
     while (true)
          send "get work" message
          read work from server
          execute GPU/CPU miner core
          if solution found:
               send "found solution" message

while (true)
     select/poll for new input from remote server (bitcoind)
     read message from server ("hi, I'm a new work unit!")
     interrupt GPU/CPU miner core, restart with new work

Or something to that effect.

The general idea is simply a binary protocol and connection model that's efficient for data broadcasting tasks such as: monitorblocks, push mining, and a future monitortx feature.

[sandos]

A client that connects up, subscribes, and then gets updates as they happen might be an easier start.

Connects via another port?

Or would you teach bitcoin's minimal http implementation to keep the connection open?  (and service multiple connections at once)

I've got a use case (google App Engine) where a persistent connection to bitcoin isn't possible (App Engine apps can fetch URLs, and can act as 'web hooks', but can't just open a socket and listen).

There is channel API:

http://code.google.com/appengine/docs/python/channel/overview.html

[slush]

Code:

[quote author=jgarzik link=topic=2647.msg46868#msg46868 date=1297379631]
[code]
TCP connect to host:8331
send curtime, protocol version, client version string, username, [possibly empty] list of options, sha256(all previous args + password)
wait for server response ("auth ok, here are my capabilities", or "rejected, goodbye")

[/quote]

We already discussed this on IRC and I think we didn't listen each other. Maybe because I'm not native speaker and my capabilities to explain anything are extremely limited .

I don't have any major problem with your proposal. Skipping HTTP overhead is good idea mainly for high traffic deployment and including some kind of notifications for external applications would be nice. But if I'll have a choice, I'll prefer stateless protocol. I don't see first auth request as necessary, there  is not any problem with sending username/password with every request, but it make code on both sides more transparent.
[/code]

[jgarzik]

I don't have any major problem with your proposal. Skipping HTTP overhead is good idea mainly for high traffic deployment and including some kind of notifications for external applications would be nice. But if I'll have a choice, I'll prefer stateless protocol. I don't see first auth request as necessary, there  is not any problem with sending username/password with every request, but it make code on both sides more transparent.

Highly redundant HTTP headers are probably >50% of your overall bandwidth usage.  My proposal eliminates that entirely.

The cost of stateless is huge.  Think about it -- those HTTP request and response headers are sent, uncompressed, over and over and over again.

My proposal cuts 'getwork' down to a single network packet.  That is maximum network efficiency.

[slush]

Highly redundant HTTP headers are probably >50% of your overall bandwidth usage.

I cannot agree more! I'm not talking about HTTP based stateless protocol.

The cost of stateless is huge.

If you're talking about stateless calls over TCP (maybe binary, but I don't like it, as we discussed in protocol forum thread), there is only little overhead (probably only in sending username/login on every request), but it make thing little easier on server side and also open some possibilities like reusing the same TCP connection for many workers (GPU threads). Currently the GPU miners are asking for more getworks at once, which needs (and probably will need with state protocol) more independent TCP connections from app to server (It's hard to say now, because we are not discussing getwork internals).

But I think this is just a cosmetic thing and I won't fight for this.