JSON-RPC Multiple Invocations

[lachesis]

I noticed from a hint that Satoshi dropped in the JSON-RPC password thread about so-called "Multiple Invocation" support in Bitcoin's JSON-RPC.

As I run a site that polls bitcoind for payments to a large number of addresses twice a minute, I was intrigued. First of all, this isn't JSON-RPC 2.0's "Batch" support, where requests are submitted in an array and responses are received the same way:

Code:

request = [
        {"jsonrpc": "2.0", "method": "sum", "params": [1,2,4], "id": "1"},
        {"jsonrpc": "2.0", "method": "subtract", "params": [42,23], "id": "2"},
        {"jsonrpc": "2.0", "method": "get_data", "id": "9"} 
    ]
response = [
        {"jsonrpc": "2.0", "result": 7, "id": "1"},
        {"jsonrpc": "2.0", "result": 19, "id": "2"},
        {"jsonrpc": "2.0", "result": ["hello", 5], "id": "9"}
    ]

Instead, it's something different, and I can't figure out how to parse the responses in Python. Here's a screen capture of a telnet session to the Bitcoin RPC server:

Code:

$ telnet localhost 8332
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
POST / HTTP/1.1
Content-Type: text/plain
Content-Length: 97

{"params":[],"id":1,"method":"getconnectioncount"}
{"params":[],"id":2,"method":"getdifficulty"}
HTTP/1.1 200 OK
Connection: close
Content-Length: 33
Content-Type: application/json
Date: Sat, 08 Jul 2006 12:04:08 GMT
Server: json-rpc/1.0

{"result":8,"error":null,"id":1}
HTTP/1.1 200 OK
Connection: close
Content-Length: 49
Content-Type: application/json
Date: Sat, 08 Jul 2006 12:04:08 GMT
Server: json-rpc/1.0

{"result":181.5432893640505,"error":null,"id":2}
Connection closed by foreign host.

As you can see, the server replies with two complete HTTP 200 responses instead of (as I would have expected) one response with the two lines concatenated as I did in the request.

I can't figure out how to parse that with anything at all semi-automated in Python. urllib2 and httplib both return after the first response and drop the second one on the floor.

Has anyone encountered this problem before? Does anyone know of a Python library that can handle this strange multi-request behaviour?

[1715175008]

1715175008

[1715175008]

1715175008

[1715175008]

1715175008

[1715175008]

1715175008

[lachesis]

Alright, I know it hasn't been more than an hour since I last posted about this, but I decided to solve this problem myself.

If there's a better solution out there, I'd be glad to hear it, but here's my version 0.1 attempt to make this work:
http://www.alloscomp.com/bitcoin/btcjsonrpc.pys

Let me know if you like it, hate it, or see any obvious flaws with it. There's some getting started docs in the source, but here's the code of the test suite:

Code:

    from btcjsonrpc import Service
    s = Service()
    print 'preparing getbalance; id:',s.getbalance() # Each call returns its ID so you can find it later in the results
    print 'preparing getdifficulty; id:',s.getdifficulty() 
    print 'preparing listreceivedbyaddress; id:',s.listreceivedbyaddress(10000) # Call with a parameter
    print 'preparing getbalance; id:',s.getbalance(id='getbalance 2') # You can also specify your own ID
    print '\nexecuting call\n\nresults:'
    results = s() # Get the results by calling the Service object
    for id,value in results.iteritems():
        print id,value
    # If you'd prefer to work directly with the JSON responses instead of a dict of IDs, then access the list Service.responses.
    print '\njson responses'
    print s.responses

And here's the output, including the docstr:

Code:

$ ./btcjsonrpc.py 
 Socket-based, Bitcoin-compatible JSON-RPC v1.0 client.
 By: Eric Swanson (http://www.alloscomp.com/bitcoin)
 Version 0.1, July 21, 2010
Don't use this for one-off request->response pairs. Use something like the reference python-jsonrpc library, or urllib2 + json. This client is hackish, but it works for me (and has sped up my JSON-RPC accesses tremendously).

For details of WHY exactly I felt the need to redo python-jsonrpc using a raw socket, check out the follow forum post: http://bitcointalk.org/index.php?topic=528.0

Usage is fairly straightforward, and a code sample can be found below the library code (in the if __name__=='__main__' clause). 

preparing getbalance; id: jss-1
preparing getdifficulty; id: jss-2
preparing listreceivedbyaddress; id: jss-3
preparing getbalance; id: getbalance 2

executing call

results:
jss-2 181.543289364
jss-3 []
getbalance 2 2345.94
jss-1 2345.94

json responses
[{u'id': u'jss-2', u'result': 181.54328936405051, u'error': None}, {u'id': u'jss-3', u'result': [], u'error': None}, {u'id': u'getbalance 2', u'result': 2345.9400000000001, u'error': None}, {u'id': u'jss-1', u'result': 2345.9400000000001, u'error': None}]

[satoshi]

Obviously it's a bug that it repeats the header.

I was trying to follow the 1.0 spec: http://json-rpc.org/wiki/specification   It called for multiple invocation.

I think they mean it's like this, but I'm not sure:

Post:
{"method": "postMessage", "params": ["Hello all!"], "id": 99}
{"method": "postMessage", "params": ["I have a question:"], "id": 101}

Reply:
{"result": 1, "error": null, "id": 99}
{"result": 1, "error": null, "id": 101}

I can't remember where I think I saw that it's supposed to send back HTTP status 500 for an error reply.  If it contains multiple responses and one is an error, I wonder if that makes the status 500 for the whole thing, I guess so.  Maybe it should always return 200.  I think someone sounded like the 500 might be causing a problem.

This probably gets fixed after 0.3.3.  Until then, just use single invocation.  I wonder if any JSON-RPC package even supports multiple invocation, probably not.

It would be nice if we could pin down better how multiple-invocation is supposed to work, if at all, before trying to fix it, and whether returning HTTP status 500 for error response is right.

[lachesis]

Yeah the 1.0 spec is a bit strange. JSON-RPC 2.0 pins it down quite nicely: a list of requests begets a list of responses. Afaik, none of the JSON-RPC client libraries handle the error code. They all just look at the "error" field of the JSON-RPC response. At the very least, I would return 200 if ANY of the requests succeed.