[ATTN: POOL OPERATORS] PoolServerJ - scalable java mining pool backend

[eleuthria]

Right now BTC Guild is forced to do some very awkward things to get pushpoold to scale to 1 TH/s per server.  I'm very interested in this project, but until the source is out I won't be able to give it any real-world testing since my DB Schema is very customized for faster reads/multi-server support.

[1714022293]

1714022293

[1714022293]

1714022293

[Furyan]

Right now BTC Guild is forced to do some very awkward things to get pushpoold to scale to 1 TH/s per server.  I'm very interested in this project, but until the source is out I won't be able to give it any real-world testing since my DB Schema is very customized for faster reads/multi-server support.

+1.  My DB is heavily customized for much the same purposes.  The addition of bits of information can reduce future lookups by millions.

[phorensic]

+1.  My DB is heavily customized for much the same purposes.  The addition of bits of information can reduce future lookups by millions.

Adding one index or changing one line of code on our webpages can cause our mysql server to go from 15 million rows scanned per minute to 5 million.  It's nuts!

[Furyan]

+1.  My DB is heavily customized for much the same purposes.  The addition of bits of information can reduce future lookups by millions.

Adding one index or changing one line of code on our webpages can cause our mysql server to go from 15 million rows scanned per minute to 5 million.  It's nuts!

The joys of database administration and design

[shads]

Well I just needed a shove I guess.  I was going to wait a few weeks while I finished some other thing but since there's demand for it I'm refactoring the code to split out the parts that I'm not ready to release yet (which you won't need anyway).  So I should be able to publish the source very soon.  I'd really like to see the results of other people's tests so I'm prioritising this.

As part of the refactoring I'm splitting out some parts to engine classes that can be easily overridden for the things that are most likely to need customisation.  Namely: Worker db lookups, authentication, share database writes.  That should hopefully minimize the need to fork the code so you won't get clobbered by every update.

I'll do this all as part of release 0.2.3 which is a pretty major improvement.

- fixed memory hog issues so memory usage has stabilised.
- switched duplicate maps and work-worker maps to trove THashMap implementation and added customised hashing strategy which has yielded about a 50% boost to getwork speed
- implemented several bulkloader engines for share db writes which has boosted db write speed by 5-20 times depending on which engine is used.
- partial implementation of mining extensions (reject-reason complete, rollntime and noncerange still work in progress).

[shads]

0.2.3 released now with source code available. 

This is the first release I'd consider usable in a production environment.  Memory leak issues have been resolved and some hefty speed increases have been gained on both the getwork side and the share submit side.


[0.2.3]

- fix: Memory usage issues for work-worker map and duplicate map
- switched duplicate maps and work-worker maps to trove THashMap implementation and added customised hashing strategy which has yielded about a 50% boost to getwork speed
- share database flushing now paused delayed for 4 seconds after block change to free up resources for fullfilling longpoll requests
- implemented several bulkloader engines for share db writes which has boosted db write speed by 5-20 times depending on which engine is used.
- partial implementation of mining extensions (reject-reason complete, rollntime and noncerange still work in progress).
- implemented rudimentary plugin interface to allow customized engine for Authentication, worker fetching, share logging (to database only, file logging is still hardcoded)
- major refactoring and reorganization of packages in preparation for source release.

Known issues
 - Under heavy load shutdown may fail to trigger on the first try.  Sending a 2nd shutdown signal
 seems to reliably complete shutdown.
 - All new bulkloaders are not considered stable.  Use at your own risk.

[BurningToad]

I think I will try testing this soon to see if it will work for me.  My pool is running up against the 450-500 GH/s pushpool barrier.

I would also be very interested to hear what you are doing about it eleuthria, if you have a single pushpool instance scaling up to 1,100 GH/s per server, or, it sounds like you might have 3 instances running on the same server, I guess with routing/port stuff that you have to set up to do that?

[shads]

Source now published to a repo:

https://bitbucket.org/shadders/bitcoin-poolserverj/

[shads]

Architecture diagram is now on the site.  Also for those of you who would need to customise the code to match you database structure I've written a guide to building DB and authentication plugin modules.

http://poolserverj.org/documentation/

[jamesg]

Definitely interested in this.

[shads]

0.2.4 source now pushed to repo.  This primarily improved the plugin mechanism which is only of interest to people who will build from source so no binary has been published on the main site.

[eyeoh]

I'm testing it out now. Maybe I have a setting in the config wrong but I keep getting the good old "Problems communicating with bitcoin RPC" error and I've verified bitcoind is up and happy and all the IP/login/pass/port settings are correct. Hitting it with Firefox gives me "Problem accessing /. Reason HTTP method GET is not supported by this URL" and I'm seeing "33022 [main-con-qtp-22] WARN org.eclipse.jetty.util.log - / java.lang.NullPointerException" in the error log with debugging turned on. Any hints?

[shads]

Can you pm me the full stacktrace. (need to set logStacktraces=true in properties file).

The message you're getting from the brower is correct.  JSON-RPC should not accept GET requests.  The request has to have content so it has to be a POST.  Although some mining clients use it for LP so the longpoll servlet will allow it.

If you've enable the management interface which is enabled in the sample properties file try hitting
http://localhost:8997/?method=getsourcestats from your browser.  That should at least tell if it's up and if it's communicating with the daemon

if you want to test an actual request then try in one browser tab hit the longpoll url, you'll need to include the user:pass in the url:
http://user:pass@mylongpollurl

Then in another browser tab hit:
http://localhost:8997/?method=fireblockchange

This should force your longpoll request to finish and at least give you some kind of status message.

[shads]

BTW if the nullpointer stack trace looks like this one:

Code:

36142 [main-con-qtp-18] WARN org.eclipse.jetty.util.log - /
java.lang.NullPointerException
        at com.shadworld.poolserver.WorkProxy.handleRequest(WorkProxy.java:412)
        at com.shadworld.poolserver.servlet.PoolServerJServlet.getResponse(PoolServerJServlet.java:26)
        at com.shadworld.poolserver.servlet.AbstractJsonRpcServlet.doPost(AbstractJsonRpcServlet.java:176)

Someone else found this issue last night... The problem is caused by client not setting 'id' field in json-rpc request.  I've updated json-rpc lib to fallback to a default '-1' instead of throwing a nullpointer and commited to repo.  Will try an put up a new binary dist today.

if you need a workaround, use a miner that does json-rpc according to spec.  Phoenix seems to do it right...

[shads]

I'm testing it out now. Maybe I have a setting in the config wrong but I keep getting the good old "Problems communicating with bitcoin RPC" error and I've verified bitcoind is up and happy and all the IP/login/pass/port settings are correct. Hitting it with Firefox gives me "Problem accessing /. Reason HTTP method GET is not supported by this URL" and I'm seeing "33022 [main-con-qtp-22] WARN org.eclipse.jetty.util.log - / java.lang.NullPointerException" in the error log with debugging turned on. Any hints?

0.2.5 is available on downloads page now.  If you're was the problem I was describing it should be fixed now.

[0.2.5]

 - update to jsonrpc lib - force JsonRpcRequest.getId() to return a default value (-1)
 if not set.  Some clients don't implement spec properly and this was causing a nullpointerException.
 - Cleaned up directory layout for distribution.
 - Fix: classpath problem for windows service so no longer need to build into one monolithic jar
 - added lib/plugins directory which is in the default classpath so plugins no longer require classpath changes.
 - update procrunsrv to 1.07 to support wildcard classpaths

[0.2.4] - unreleased

 - synchronised share flush to database to prevent concurrent attempts.
 - set server header to PoolServerJ(v<version>)
 - implemented resource pools for recycling objects used in WorkProxy.chooseSourceForWork()
 - Added reset(args) methods to a number of heavily used classes for recycling
 - added example AnyPasswordWorkerAuthenticator class as used in plugin build guide: http://poolserverj.org/documentation/plugin-guide/

[Dusty]

@shads:
Do you thing it would be possible to have a pure java implementation of getWork() without the need of calling the bitcoin daemon, using for example the bitcoinj library?
Could this speed up things?

[shads]

@shads:
Do you thing it would be possible to have a pure java implementation of getWork() without the need of calling the bitcoin daemon, using for example the bitcoinj library?
Could this speed up things?

Dusty I looked at doing this but Mike Hearn pretty much flat out told me that bitcoinj was miles off being ready to serve getworks.  It's not really a problem at the moment though, patched bitcoind's are fast and if you do hit the wall you can always hook up several separate bitcoinds to feed poolserverj and get a big speed boost that way.  The biggest bottleneck I can see is writing to databases.  The bulkloaders I added in 0.2.3 speed that up by somewhere between 500%-2000% depending on which engine you use.

When bitcoinj is ready though I definately will incorporate it.

[eyeoh]

Just tried 0.2.5 and ran into this:

Code:

:/usr/src/poolserverj-0.2.5# /usr/bin/java -jar bin/poolserverj.jar start config.cfg      
Args - [2]: start config.cfg 
PoolServerJ Service Starting Sat Aug 13 16:49:33 MST 2011
java.lang.NoClassDefFoundError: javax/servlet/http/HttpServlet
        at com.shadworld.poolserver.servicelauncher.PoolServerService.start(PoolServerService.java:88)
        at com.shadworld.poolserver.servicelauncher.PoolServerService.windowsService(PoolServerService.java:48)
        at com.shadworld.poolserver.servicelauncher.PoolServerService.main(PoolServerService.java:18)
Caused by: java.lang.ClassNotFoundException: javax.servlet.http.HttpServlet
        at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
        ... 3 more

Anything I can do?

[shads]

Ah sorry the startup command has changed a bit, I've updated http://poolserverj.org/quick-start/

The command is uglier but makes things a lot more flexible.  Probably best to put it in a script...

Code:

java -classpath poolserverj.jar:../lib/*:../lib/lib_non-maven/*:../lib/plugins com.shadworld.poolserver.servicelauncher.PoolServerService start config.cfg

if you create poolserverj.sh

Code:

#!/bin/bash

java -classpath poolserverj.jar:../lib/*:../lib/lib_non-maven/*:../lib/plugins com.shadworld.poolserver.servicelauncher.PoolServerService $1 $2 $3

then it's a bit less ugly:

Code:

chmod 777 poolserverj.sh
./poolserverj.sh start config.cfg

Currently has to be started from the bin dir...

[eyeoh]

Just want to let you know that worked! Up and running with 0.2.5 and Guiminer, i'm about to start testing Diablo and Phx and all the others and stress test it.