[ATTN: POOL OPERATORS] PoolServerJ - scalable java mining pool backend

[shads]

Pushpoold Vs PoolServerJ performance test results now published

As a necessary component to a larger project I've had to write a complete pool server in java.  The rest of the project is still in progress but the pool server is ready to release into beta.  The key problem I needed to solve and part of the reason I rolled my own rather than just using pushpoold is scalability.  Pushpoold is great at what it does but the difference I needed was to get around the bitcoind bottleneck.  PoolServerJ solves that problem rather neatly by caching work and also allowing the pool operator to run as many bitcoin daemons as they want to and feed them all into the poolserver.

A few of the features of PoolServerJ:

    * Native merged mining support.  Will work with standard bitcoind 0.5
    * SCrypt blockchain support (e.g litecoin)
    * Customisable coinbase transaction (pay block rewards to any wallet perhaps an offline secure wallet)
    * Internal work generation (more than 10x faster than rpc getwork)
    * Work caching - caches work from bitcoind or internal WorkMaker to handle short term spikes in requests
    * Database resilience.  Server can keep running if database goes down and shares will be sent to database when connection is reestablished
    * DoS resilience with QoS support to ensure workers who've submitted valid work are serviced with priority
    * Capable of pulling work from multiple bitcoin daemons to get around the getwork bottleneck and also to provide some redundancy
    * Notify of block change via HTTP to a user configurable URL to support event triggered share processing
    * Cached database handling (optional) to reduce round trips.
    * Supports Mysql, Postgresql, sqlite3 (JDBC based so others can be added easily - currently only tested on mysql)
    * Runs as a Windows service
    * Longpolling support
    * Integrated block monitoring using all available bitcoin daemon work sources
    * Dump shares to Database, log file or stdout
    * Safe restart - shares issued to client are dumped to file on shutdown and reloaded on startup so your miners won't get stale shares if you restart a live server
    * Only dependency is a Java 6+ JDK.

Performance

Extensive performance testing against pushpoold has shown PoolServerJ capable of more than 4 times the capacity for share submits and 3 times the capacity for getwork requests.  This is the limit of what could tested on limited available hardware however it's architecture allows for significant scaling and theoretically these limits could be significantly exceeded.

update: Since the release of 0.4.0 WorkMaker Edition getwork capacity has improved by 4 times and share submit capacity by 2 times over the original performance test results.

Please give it a go, feel free to send me any feedback, rip into it if you like.

You can find it here

The webserver may be a bit slow, I'm testing out EC2 micros to see how they handle basic web server duties so sorry about that.  Will have a test pool up and running soon (as soon as I finish the front end). <-- update: this never happened and probably won't... I'm too busy working on the software and there's plenty of pools using it now so I've got enough guinea pigs

[1714161298]

1714161298

[1714161298]

1714161298

[1714161298]

1714161298

[1714161298]

1714161298

[1714161298]

1714161298

[Sukrim]

No Derivative Works — You may not alter, transform, or build upon this work.

Might be a bit hindering... a lot of pools have rewritten parts of pushpoold for example.

Some ideas I had for pool backends:

• Shares evaluation (stale/valid/"winning") done by backend, not bitcoind and only shares that are >=current_difficutly actually get relayed to bitcoind
• "metapool"/"inceptpool" mode that requests getworks from other pools like a miner and relays them
• certain share counts of users as event trigger

[talpan]

Wow,

i definitely will test it.
Although I don't care much about java, this seems to have some very unique features .

What about the stales, do you have some data about that?


regards, talpan

[shads]

No Derivative Works — You may not alter, transform, or build upon this work.

Might be a bit hindering... a lot of pools have rewritten parts of pushpoold for example.

As it says on the download page I plan to fully open source it in the near future, as soon as the larger project is released.

Some ideas I had for pool backends:

• Shares evaluation (stale/valid/"winning") done by backend, not bitcoind and only shares that are >=current_difficutly actually get relayed to bitcoind
• "metapool"/"inceptpool" mode that requests getworks from other pools like a miner and relays them
• certain share counts of users as event trigger

backend currently does evaluate share status, until a real live instance has found a winning block I've got it set to force all shares upstream just in case but that is just a switch and when it's proved it's evaluating a winning solution I will turn it off.

Event triggers are easy to implement, the question is the notification mechanism.  Currently it's just a http request to a URL of your choice but there's got to be a less ugly way.  Open to suggestions.

Re: metapool mode, actually it can already do that.  I just haven't released the doco for it yet.  Coming soon...

[shads]

Although I don't care much about java, this seems to have some very unique features .

What about the stales, do you have some data about that?

Java's not so much of a feature, just a fact.  Although it does give the benefit of a single binary for all platforms.

I haven't done any solid testing on stales but it delivers work back to longpoll requests as fast as it can get them in from the work sources after a block change.  Using multiple daemons should speed this up.  The only instances of stales I saw were in testing when I was bringing the server up and down constantly so the server was losing it's work-worker map and the miner kept humming along with old work.  After I implemented the work-worker map dump to disk on shutdown I haven't seen one.

[Sukrim]

Wow, sounds great so far!

What about Namecoin support? Then you could even quite fast test if your "winning solution relay" works (15 times faster than Bitcoin...).

[shads]

What about Namecoin support?

Haven't tried it yet, just trying to work out if it is now: http://dot-bit.org/forum/viewtopic.php?f=6&t=263&p=1499#p1499

[shads]

Shares evaluation (stale/valid/"winning") done by backend, not bitcoind and only shares that are >=current_difficutly actually get relayed to bitcoind[/li][/list]

BTW I'm pretty sure pushpool does that.  I probably got the idea when I was peering at pushpool source (and getting a headache because I've never learned C).

[phorensic]

Bitcoind (with pushpoold) has been patched to allow 1,000GH/s (on a nice server) and solve the too many open connections problem by Jeff Garzik - http://forum.bitcoin.org/index.php?topic=22585.0 .  It has other nice added features as well.  You say you released this mostly due to scalability.  What is your theoretical limit with your setup?

[shads]

Bitcoind (with pushpoold) has been patched to allow 1,000GH/s (on a nice server) and solve the too many open connections problem by Jeff Garzik - http://forum.bitcoin.org/index.php?topic=22585.0 .  It has other nice added features as well.  You say you released this mostly due to scalability.  What is your theoretical limit with your setup?

By the time the patched bitcoind was announced I was already well into the development process... However I still think there's some significant scalability benefits.  I've just a run a test to try and get some metrics.  The difficult part is eliminating the bottleneck (I don't have a patched bitcoind atm).

Here's the test setup:

Ubuntu 10.4 64 bit running inside vmware on winXP 64 host - Dual core 3Ghz - 8gb ram (3 gb available to ubuntu).
2 * bitcoind, one running on host and one on ubuntu vm.
mysql running in ubuntu vm

To avoid the bottleneck I set the cache size to 20000 getworks per source and waited until the caches were 1/2 full so all works were served from memory.  But the server was still pulling work from the bitcoind's in the background while the tests were running.

Set the difficult to below easy, average 2 hashes required to solve a block.

Client test also run inside ubuntu vm.  
50 concurrent clients continuously issuing getworks.  
Each second 10 of those solve a block (using CPU only) and submit.
Clients shared 10 different sets of worker credentials.

Result:

Avg rate of work delivery was about 1200/sec until the cache ran out then it dropped dramatically due to not being able to get work from daemon fast enough.
about 200 works were submitted and flushed to database.

With 30 block solve attempts/sec:
Avg work delivery 1000/sec
580 works flushed to database

At this point I remembered starcraft 2 was running in the background and killed it...

With 67 block solve attempts per second
Avg work delivery 1200/sec
1000 works flushed to database

To increase the number of block solves/sec anymore I'd need to rewrite some of the test code to synchronize it, due to the way each iteration decides whether to solve a block it becomes unreliable and have no real way of seeing how many block solves it's attempting per sec.

There's some obvious weakness in the test

Unrealistic advantages:
all network communication was local

Unrealistic disadvantages:
Everything (including clients and block hashing) running on same machine.

Hopefully someone who's got more real world experience with pushpool can give those numbers some context.

[phorensic]

I'm sorry, Joel Katz did the patching, not Jeff Garzik, oops!

Unrealistic advantages:
all network communication was local

I'm glad you mentioned the above.  While latency and bandwidth is usually not the bottleneck of a pool server, virtual networking is almost unlimited compared to a typical WAN link.  Not to downplay your development and testing or anything!  Very good info here!

I'm a little bit confused as to the new bottleneck you got to.  Are you saying bitcoind is your bottleneck?  If so, that's why Joel Katz hacked on it.  A month ago the large pool operators were screaming at pushpoold as the bottleneck, including Eleuthria with his awesome experiments and communication with his members.  Then some people started to discover it was actually bitcoind, hence the patches.  With all due respect, I hope you aren't trying to scale the wrong side of the puzzle.  It may be faster than pushpoold, but all for naught if bitcoind can't feed the monster.

[shads]

[0.2.1]

- fix: maxCacheSize was being ignored and hardcoded to 50
- fix: thread pool size for work fetchers not being set correctly.
- enabled allowedManagementAddresses property
- added source.local.<n>.disabled=true option so you don’t have to comment out every line of a source to disable it.

[Sukrim]

2 more features off the top of my head:

* Google app engine compatibility
* Hash offloading to a GPU (I have no idea if that is a major bottleneck of your program though) - I recently read something about doing SSL calculations on a GPU, getwork verifications etc. should also be an easy task.

[shads]

2 more features off the top of my head:

* Google app engine compatibility
* Hash offloading to a GPU (I have no idea if that is a major bottleneck of your program though) - I recently read something about doing SSL calculations on a GPU, getwork verifications etc. should also be an easy task.

I looked at google app engine but it has a couple of gotcha's. 

- Time per request: max 30 sec

Which effectively rules out longpolling.

- Memory cap 128MB.

Depends on how hard you are running it but if you run a large pool you may find this restrictive.  A smallish pool will comfortably run in about 50mb but once you start scaling you are consuming memory two ways, first is caching work.  Second is mapping work delivered to workers for current block and last block.  I can't actually remember why I did this, it was partly to ensure no duplicate work but there was one other reason. 

Another option for scalable hosting would be Amazon elastic beanstalk.  It would require some refactoring to strip out the embedded server and turn it into a fully compliant webapp, I think it's backend is Jetty (have to double check), if it that's good because poolserverj uses some of jettys advanced features (which are outside the servlet spec) to avoid thread saturation and handle long polling.  If there's a reasonable demand then I'll look into doing it.

Interesting idea about the GPU calc but I wonder if in this scenario it would really yield much benefit?  Under a mining scenario the overhead of translating the text to a byte array, sending to GPU and starting calculation happens once then the gpu does many billions of calcs.  In this scenario that overhead would be incurred and the GPU would do a single calculation.  The text-byte array conversion happens anyway I suppose but I still wonder whether the net gain would be comparable to overheard?  I'm not a GPU programmer so it's hard to say.

[shads]

I'm a little bit confused as to the new bottleneck you got to.  Are you saying bitcoind is your bottleneck?  If so, that's why Joel Katz hacked on it.  A month ago the large pool operators were screaming at pushpoold as the bottleneck, including Eleuthria with his awesome experiments and communication with his members.  Then some people started to discover it was actually bitcoind, hence the patches. 

Yes it was bitcoind that was the bottleneck, I built the patched version yesterday and it was a vast improvement. 

With all due respect, I hope you aren't trying to scale the wrong side of the puzzle.  It may be faster than pushpoold, but all for naught if bitcoind can't feed the monster.

I don't know if it's faster than pushpoold.  It quite different in a lot ways.  It just handles some things differently, the key difference being it can use multiple bitcoinds if they are proving to be a bottleneck.  So under some scenarios it will probably be faster and under others it might be slower.

[shads]

Due to the issues noted in the changelog I’ve decided it was probably premature to call PoolServerJ a beta release.  Until those issues are resolved it is reverted to alpha status.

[0.2.2]

- fix: safe restart throwing concurrent modification exceptions due to requests still being serviced during shutdown
- added property enableSafeRestart to allow disabling
- trimmed unnecessary data out of work-worker map entries, reduced overall memory usage by 80%
- enable sharesToStdout and requestsToStdout options
- added useEasiestDifficulty property to enabled stress testing of work submits

Known issues
- Under extreme load database flush thread can become overwhelmed particularly if the server
is started under high load conditions and doesn’t have time for JIT compiler to kick in.
This is currently managed by throttling work submit connections if cache entries exceeds
maxCacheSize.  Need to implement concurrent database writes.
- Block change notification currently doesn’t take into account the state of the database
cache.  This should wait until all shares from the previous block have been flushed to
DB before notifying of block change.
- Cache flushing should be temporarily suspended after a block change to allow resources
to focusing on completing longpoll requests.
- Memory usage.  work-worker map entries size has been reduced by 80% but it still presents
a theoretical limit.  With 768mb max heap size the server can handle about 1.2 million
cached works requests before OOM error occurs.  At about 1 million performance seriously
degrades due to frequent garbage collection.  The map contains shares issue in current
block and previous block.  So this will only be an issue for very busy servers.  Current
workaround is to assign more memory to JVM.  Fix will require a revisit of the work map
strategy to either trim the map entries even more dramatically or find a safe method of
pruning the map.

[shads]

It has other nice added features as well.  You say you released this mostly due to scalability.  What is your theoretical limit with your setup?

I can answer your question properly now.  On the test setup I used it achieved a max long term avg of 2154GH/s.  

Bear in mind that the clients were running on the same machine and performing CPU hashes (approx 2 per submit).  The test was pretty clearly CPU bound from all the observations I made so I'm quite sure if the client wasn't running on the same machine it could have been close to double that.  The server performs 1 hash/submit.  So with additional overhead it's probably a reasonable guess that they were using about 50% of the cpu each (which ps indicated as well).  The next hurdle is database write speed.  Which seemed to on the borderline of becoming the next bottleneck.  I'm going to test concurrent writes to see if that yields an improvement.

[phorensic]

Oh wow, that's impressive.  You might want to contact the big pool owners to see if they are interested.  Currently the new cap on pushpoold seems to be 1,000GH/s on a fully tweaked setup.

[shads]

Oh wow, that's impressive.  You might want to contact the big pool owners to see if they are interested.  Currently the new cap on pushpoold seems to be 1,000GH/s on a fully tweaked setup.

So the bottleneck seem to be submissions rather that getworks?  Can you point me to where this is discussed?  Particularly the 1000gh/s limit?

[phorensic]

Check it out - https://bitcointalk.org/index.php?topic=22585.0