Bitcoin Forum
December 09, 2016, 02:19:31 AM *
News: To be able to use the next phase of the beta forum software, please ensure that your email address is correct/functional.
 
   Home   Help Search Donate Login Register  
Pages: [1]
  Print  
Author Topic: Expert needed: Multithreading JSON-RPC  (Read 1486 times)
Luke-Jr
Legendary
*
expert
Offline Offline

Activity: 2086



View Profile
April 27, 2011, 12:45:06 AM
 #1

I wrote a patch to give JSON-RPC connections their own threads. However, it still has some issues that I can't seem to figure out. If anyone can advise on how to fix it, this would be a nice feature.
  • Occasional crashes (SIGABRT)
  • Segfault if -DUSE_SSL
  • Timed out connections aren't closed

http://luke.dashjr.org/programs/bitcoin/w/bitcoind/luke-jr.git/commitdiff/355e162fba35810978742f6f1e6f2e413cdc78e9

Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1481249971
Hero Member
*
Offline Offline

Posts: 1481249971

View Profile Personal Message (Offline)

Ignore
1481249971
Reply with quote  #2

1481249971
Report to moderator
xf2_org
Member
**
Offline Offline

Activity: 70


View Profile
April 27, 2011, 02:37:35 AM
 #2

Cute, but definitely not that easy.  If it was, we would have already done that by now.  There are way too many "there is only one RPC thread" assumptions built into the code at the present time.
Luke-Jr
Legendary
*
expert
Offline Offline

Activity: 2086



View Profile
April 27, 2011, 03:42:55 AM
 #3

Well, care to help? I basically "need" this functionality to setup my pool how I want it-- a 'getwork' request depends on being able to make sub-RPC calls.

Here's a new version, that mostly works fine without -DUSE_SSL... Though I can still crash it if I keep a constant load of 100+ RPC calls at once, it's probably "stable enough" for my pool to just restart it when it dies.

http://luke.dashjr.org/programs/bitcoin/w/bitcoind/luke-jr.git/commitdiff/0071352b3347f92faa266dfd1db18aea88b45c54

Luke-Jr
Legendary
*
expert
Offline Offline

Activity: 2086



View Profile
April 27, 2011, 03:52:04 AM
 #4

What if the connection threads grabbed a lock, and the "real" multithreading only occurs when the RPC function itself releases that lock?

xf2_org
Member
**
Offline Offline

Activity: 70


View Profile
April 27, 2011, 05:40:48 AM
 #5

It's such a mess that I thought about ripping out boost SSL, and doing threading with straight OpenSSL C API.  That eliminates a few of the assumptions, but not all by a long shot.

If you really do manage to fix them all, your pull request will get merged faster than you can say "tonal"

Luke-Jr
Legendary
*
expert
Offline Offline

Activity: 2086



View Profile
April 27, 2011, 03:53:48 PM
 #6

Well, locking helped the non-SSL code. I can no longer crash it. Not sure I care enough to figure out the SSL problem.

jordanlewis
Newbie
*
Offline Offline

Activity: 5


View Profile
April 29, 2011, 06:27:06 PM
 #7

As I understand it, it's not actually all that beneficial to spin off a new request thread for every incoming connection, at least in standard webservers. The general approach is to have a few worker threads processing requests, but each of them asynchronously. So one thread would be able to handle a few incoming connections at once by removing the sequential connection processing code.

I've had a look at the RPC server code and it doesn't seem like it would be that hard to get basic asynchronous IO going, which would help solve any timeout issues by itself. Of course you'd still need to put a lock around all of the actual bitcoin calls until that "one rpc thread" assumption is removed, but nonetheless I think it would be an improvement.

Does that sound like it could be a helpful patch? I'd like to get some feedback before doing any more work in this direction.
jordanlewis
Newbie
*
Offline Offline

Activity: 5


View Profile
April 29, 2011, 06:34:43 PM
 #8

I forgot to mention that the code already uses Boost/ASIO enough that making the RPC server code asynchronous is a pretty natural direction to go in.
Luke-Jr
Legendary
*
expert
Offline Offline

Activity: 2086



View Profile
April 29, 2011, 11:40:44 PM
 #9

Asynchronous wouldn't solve it for me. I have JSON-RPC requests blocking on loopback JSON-RPC requests. My current code is working so long as I build without SSL support, so I guess it's good enough for now.

xf2_org
Member
**
Offline Offline

Activity: 70


View Profile
April 30, 2011, 12:26:55 AM
 #10

As I understand it, it's not actually all that beneficial to spin off a new request thread for every incoming connection, at least in standard webservers. The general approach is to have a few worker threads processing requests, but each of them asynchronously. So one thread would be able to handle a few incoming connections at once by removing the sequential connection processing code.

I've had a look at the RPC server code and it doesn't seem like it would be that hard to get basic asynchronous IO going, which would help solve any timeout issues by itself. Of course you'd still need to put a lock around all of the actual bitcoin calls until that "one rpc thread" assumption is removed, but nonetheless I think it would be an improvement.

Does that sound like it could be a helpful patch? I'd like to get some feedback before doing any more work in this direction.

Definitely!  We would love to see a patch that updated the RPC HTTP server to use async I/O.

Pages: [1]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!