Bitcoin Forum
December 04, 2016, 12:05:51 AM *
News: To be able to use the next phase of the beta forum software, please ensure that your email address is correct/functional.
 
   Home   Help Search Donate Login Register  
Pages: « 1 [2] 3 4 5 6 7 8 9 10 11 12 »  All
  Print  
Author Topic: [ATTN: POOL OPERATORS] PoolServerJ - scalable java mining pool backend  (Read 28220 times)
eleuthria
Legendary
*
Offline Offline

Activity: 1750


BTC Guild Owner


View Profile WWW
August 08, 2011, 05:44:37 PM
 #21

Right now BTC Guild is forced to do some very awkward things to get pushpoold to scale to 1 TH/s per server.  I'm very interested in this project, but until the source is out I won't be able to give it any real-world testing since my DB Schema is very customized for faster reads/multi-server support.

R.I.P. BTC Guild, 2011 - 2015.
BTC Guild Forum Thread
1480809951
Hero Member
*
Offline Offline

Posts: 1480809951

View Profile Personal Message (Offline)

Ignore
1480809951
Reply with quote  #2

1480809951
Report to moderator
1480809951
Hero Member
*
Offline Offline

Posts: 1480809951

View Profile Personal Message (Offline)

Ignore
1480809951
Reply with quote  #2

1480809951
Report to moderator
1480809951
Hero Member
*
Offline Offline

Posts: 1480809951

View Profile Personal Message (Offline)

Ignore
1480809951
Reply with quote  #2

1480809951
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1480809951
Hero Member
*
Offline Offline

Posts: 1480809951

View Profile Personal Message (Offline)

Ignore
1480809951
Reply with quote  #2

1480809951
Report to moderator
1480809951
Hero Member
*
Offline Offline

Posts: 1480809951

View Profile Personal Message (Offline)

Ignore
1480809951
Reply with quote  #2

1480809951
Report to moderator
1480809951
Hero Member
*
Offline Offline

Posts: 1480809951

View Profile Personal Message (Offline)

Ignore
1480809951
Reply with quote  #2

1480809951
Report to moderator
Furyan
Full Member
***
Offline Offline

Activity: 175



View Profile
August 09, 2011, 05:22:40 PM
 #22

Right now BTC Guild is forced to do some very awkward things to get pushpoold to scale to 1 TH/s per server.  I'm very interested in this project, but until the source is out I won't be able to give it any real-world testing since my DB Schema is very customized for faster reads/multi-server support.

+1.  My DB is heavily customized for much the same purposes.  The addition of bits of information can reduce future lookups by millions.
phorensic
Hero Member
*****
Offline Offline

Activity: 630



View Profile
August 09, 2011, 08:03:16 PM
 #23

+1.  My DB is heavily customized for much the same purposes.  The addition of bits of information can reduce future lookups by millions.
Adding one index or changing one line of code on our webpages can cause our mysql server to go from 15 million rows scanned per minute to 5 million.  It's nuts!
Furyan
Full Member
***
Offline Offline

Activity: 175



View Profile
August 09, 2011, 09:25:01 PM
 #24

+1.  My DB is heavily customized for much the same purposes.  The addition of bits of information can reduce future lookups by millions.
Adding one index or changing one line of code on our webpages can cause our mysql server to go from 15 million rows scanned per minute to 5 million.  It's nuts!

Cheesy The joys of database administration and design Smiley
shads
Full Member
***
Offline Offline

Activity: 224


View Profile WWW
August 10, 2011, 01:19:55 AM
 #25

Well I just needed a shove I guess.  I was going to wait a few weeks while I finished some other thing but since there's demand for it I'm refactoring the code to split out the parts that I'm not ready to release yet (which you won't need anyway).  So I should be able to publish the source very soon.  I'd really like to see the results of other people's tests so I'm prioritising this.

As part of the refactoring I'm splitting out some parts to engine classes that can be easily overridden for the things that are most likely to need customisation.  Namely: Worker db lookups, authentication, share database writes.  That should hopefully minimize the need to fork the code so you won't get clobbered by every update.

I'll do this all as part of release 0.2.3 which is a pretty major improvement.

- fixed memory hog issues so memory usage has stabilised.
- switched duplicate maps and work-worker maps to trove THashMap implementation and added customised hashing strategy which has yielded about a 50% boost to getwork speed
- implemented several bulkloader engines for share db writes which has boosted db write speed by 5-20 times depending on which engine is used.
- partial implementation of mining extensions (reject-reason complete, rollntime and noncerange still work in progress).

PoolServerJ Home Page - High performance java mining pool engine

1LezqRatQz7MeNoCVziYwcdwtqeEbvrdAq - http://payb.tc/shads

Quote from: Matthew N. Wright
Stop wasting the internet.
shads
Full Member
***
Offline Offline

Activity: 224


View Profile WWW
August 10, 2011, 02:12:20 PM
 #26

0.2.3 released now with source code available. 

This is the first release I'd consider usable in a production environment.  Memory leak issues have been resolved and some hefty speed increases have been gained on both the getwork side and the share submit side.


[0.2.3]

- fix: Memory usage issues for work-worker map and duplicate map
- switched duplicate maps and work-worker maps to trove THashMap implementation and added customised hashing strategy which has yielded about a 50% boost to getwork speed
- share database flushing now paused delayed for 4 seconds after block change to free up resources for fullfilling longpoll requests
- implemented several bulkloader engines for share db writes which has boosted db write speed by 5-20 times depending on which engine is used.
- partial implementation of mining extensions (reject-reason complete, rollntime and noncerange still work in progress).
- implemented rudimentary plugin interface to allow customized engine for Authentication, worker fetching, share logging (to database only, file logging is still hardcoded)
- major refactoring and reorganization of packages in preparation for source release.

Known issues
 - Under heavy load shutdown may fail to trigger on the first try.  Sending a 2nd shutdown signal
 seems to reliably complete shutdown.
 - All new bulkloaders are not considered stable.  Use at your own risk.

PoolServerJ Home Page - High performance java mining pool engine

1LezqRatQz7MeNoCVziYwcdwtqeEbvrdAq - http://payb.tc/shads

Quote from: Matthew N. Wright
Stop wasting the internet.
BurningToad
Full Member
***
Offline Offline

Activity: 207


View Profile
August 10, 2011, 05:26:52 PM
 #27

I think I will try testing this soon to see if it will work for me.  My pool is running up against the 450-500 GH/s pushpool barrier.

I would also be very interested to hear what you are doing about it eleuthria, if you have a single pushpool instance scaling up to 1,100 GH/s per server, or, it sounds like you might have 3 instances running on the same server, I guess with routing/port stuff that you have to set up to do that?


shads
Full Member
***
Offline Offline

Activity: 224


View Profile WWW
August 11, 2011, 05:16:58 AM
 #28

Source now published to a repo:

https://bitbucket.org/shadders/bitcoin-poolserverj/

PoolServerJ Home Page - High performance java mining pool engine

1LezqRatQz7MeNoCVziYwcdwtqeEbvrdAq - http://payb.tc/shads

Quote from: Matthew N. Wright
Stop wasting the internet.
shads
Full Member
***
Offline Offline

Activity: 224


View Profile WWW
August 12, 2011, 06:19:31 AM
 #29

Architecture diagram is now on the site.  Also for those of you who would need to customise the code to match you database structure I've written a guide to building DB and authentication plugin modules.

http://poolserverj.org/documentation/


PoolServerJ Home Page - High performance java mining pool engine

1LezqRatQz7MeNoCVziYwcdwtqeEbvrdAq - http://payb.tc/shads

Quote from: Matthew N. Wright
Stop wasting the internet.
jamesg
VIP
Legendary
*
Offline Offline

Activity: 1330


AKA: gigavps


View Profile
August 12, 2011, 10:49:34 AM
 #30

Definitely interested in this.
shads
Full Member
***
Offline Offline

Activity: 224


View Profile WWW
August 12, 2011, 02:12:46 PM
 #31

0.2.4 source now pushed to repo.  This primarily improved the plugin mechanism which is only of interest to people who will build from source so no binary has been published on the main site.

PoolServerJ Home Page - High performance java mining pool engine

1LezqRatQz7MeNoCVziYwcdwtqeEbvrdAq - http://payb.tc/shads

Quote from: Matthew N. Wright
Stop wasting the internet.
eyeoh
Jr. Member
*
Offline Offline

Activity: 31



View Profile
August 13, 2011, 02:32:03 AM
 #32

I'm testing it out now. Maybe I have a setting in the config wrong but I keep getting the good old "Problems communicating with bitcoin RPC" error and I've verified bitcoind is up and happy and all the IP/login/pass/port settings are correct. Hitting it with Firefox gives me "Problem accessing /. Reason HTTP method GET is not supported by this URL" and I'm seeing "33022 [main-con-qtp-22] WARN org.eclipse.jetty.util.log - / java.lang.NullPointerException" in the error log with debugging turned on. Any hints?
shads
Full Member
***
Offline Offline

Activity: 224


View Profile WWW
August 13, 2011, 04:01:05 AM
 #33

Can you pm me the full stacktrace. (need to set logStacktraces=true in properties file).

The message you're getting from the brower is correct.  JSON-RPC should not accept GET requests.  The request has to have content so it has to be a POST.  Although some mining clients use it for LP so the longpoll servlet will allow it.

If you've enable the management interface which is enabled in the sample properties file try hitting
http://localhost:8997/?method=getsourcestats from your browser.  That should at least tell if it's up and if it's communicating with the daemon

if you want to test an actual request then try in one browser tab hit the longpoll url, you'll need to include the user:pass in the url:
http://user:pass@mylongpollurl

Then in another browser tab hit:
http://localhost:8997/?method=fireblockchange

This should force your longpoll request to finish and at least give you some kind of status message.

PoolServerJ Home Page - High performance java mining pool engine

1LezqRatQz7MeNoCVziYwcdwtqeEbvrdAq - http://payb.tc/shads

Quote from: Matthew N. Wright
Stop wasting the internet.
shads
Full Member
***
Offline Offline

Activity: 224


View Profile WWW
August 13, 2011, 04:13:10 AM
 #34

BTW if the nullpointer stack trace looks like this one:

Code:
36142 [main-con-qtp-18] WARN org.eclipse.jetty.util.log - /
java.lang.NullPointerException
        at com.shadworld.poolserver.WorkProxy.handleRequest(WorkProxy.java:412)
        at com.shadworld.poolserver.servlet.PoolServerJServlet.getResponse(PoolServerJServlet.java:26)
        at com.shadworld.poolserver.servlet.AbstractJsonRpcServlet.doPost(AbstractJsonRpcServlet.java:176)

Someone else found this issue last night... The problem is caused by client not setting 'id' field in json-rpc request.  I've updated json-rpc lib to fallback to a default '-1' instead of throwing a nullpointer and commited to repo.  Will try an put up a new binary dist today.

if you need a workaround, use a miner that does json-rpc according to spec.  Phoenix seems to do it right...

PoolServerJ Home Page - High performance java mining pool engine

1LezqRatQz7MeNoCVziYwcdwtqeEbvrdAq - http://payb.tc/shads

Quote from: Matthew N. Wright
Stop wasting the internet.
shads
Full Member
***
Offline Offline

Activity: 224


View Profile WWW
August 13, 2011, 08:18:21 AM
 #35

I'm testing it out now. Maybe I have a setting in the config wrong but I keep getting the good old "Problems communicating with bitcoin RPC" error and I've verified bitcoind is up and happy and all the IP/login/pass/port settings are correct. Hitting it with Firefox gives me "Problem accessing /. Reason HTTP method GET is not supported by this URL" and I'm seeing "33022 [main-con-qtp-22] WARN org.eclipse.jetty.util.log - / java.lang.NullPointerException" in the error log with debugging turned on. Any hints?

0.2.5 is available on downloads page now.  If you're was the problem I was describing it should be fixed now.

[0.2.5]

 - update to jsonrpc lib - force JsonRpcRequest.getId() to return a default value (-1)
 if not set.  Some clients don't implement spec properly and this was causing a nullpointerException.
 - Cleaned up directory layout for distribution.
 - Fix: classpath problem for windows service so no longer need to build into one monolithic jar
 - added lib/plugins directory which is in the default classpath so plugins no longer require classpath changes.
 - update procrunsrv to 1.07 to support wildcard classpaths

[0.2.4] - unreleased

 - synchronised share flush to database to prevent concurrent attempts.
 - set server header to PoolServerJ(v<version>)
 - implemented resource pools for recycling objects used in WorkProxy.chooseSourceForWork()
 - Added reset(args) methods to a number of heavily used classes for recycling
 - added example AnyPasswordWorkerAuthenticator class as used in plugin build guide: http://poolserverj.org/documentation/plugin-guide/

PoolServerJ Home Page - High performance java mining pool engine

1LezqRatQz7MeNoCVziYwcdwtqeEbvrdAq - http://payb.tc/shads

Quote from: Matthew N. Wright
Stop wasting the internet.
Dusty
Hero Member
*****
Offline Offline

Activity: 722


Libertas a calumnia


View Profile WWW
August 13, 2011, 09:11:59 AM
 #36

@shads:
Do you thing it would be possible to have a pure java implementation of getWork() without the need of calling the bitcoin daemon, using for example the bitcoinj library?
Could this speed up things?

Articoli bitcoin: Il portico dipinto
shads
Full Member
***
Offline Offline

Activity: 224


View Profile WWW
August 13, 2011, 10:38:04 AM
 #37

@shads:
Do you thing it would be possible to have a pure java implementation of getWork() without the need of calling the bitcoin daemon, using for example the bitcoinj library?
Could this speed up things?

Dusty I looked at doing this but Mike Hearn pretty much flat out told me that bitcoinj was miles off being ready to serve getworks.  It's not really a problem at the moment though, patched bitcoind's are fast and if you do hit the wall you can always hook up several separate bitcoinds to feed poolserverj and get a big speed boost that way.  The biggest bottleneck I can see is writing to databases.  The bulkloaders I added in 0.2.3 speed that up by somewhere between 500%-2000% depending on which engine you use.

When bitcoinj is ready though I definately will incorporate it.

PoolServerJ Home Page - High performance java mining pool engine

1LezqRatQz7MeNoCVziYwcdwtqeEbvrdAq - http://payb.tc/shads

Quote from: Matthew N. Wright
Stop wasting the internet.
eyeoh
Jr. Member
*
Offline Offline

Activity: 31



View Profile
August 13, 2011, 11:50:18 PM
 #38

Just tried 0.2.5 and ran into this:
Code:
:/usr/src/poolserverj-0.2.5# /usr/bin/java -jar bin/poolserverj.jar start config.cfg     
Args - [2]: start config.cfg
PoolServerJ Service Starting Sat Aug 13 16:49:33 MST 2011
java.lang.NoClassDefFoundError: javax/servlet/http/HttpServlet
        at com.shadworld.poolserver.servicelauncher.PoolServerService.start(PoolServerService.java:88)
        at com.shadworld.poolserver.servicelauncher.PoolServerService.windowsService(PoolServerService.java:48)
        at com.shadworld.poolserver.servicelauncher.PoolServerService.main(PoolServerService.java:18)
Caused by: java.lang.ClassNotFoundException: javax.servlet.http.HttpServlet
        at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
        ... 3 more

Anything I can do?
shads
Full Member
***
Offline Offline

Activity: 224


View Profile WWW
August 14, 2011, 12:05:54 AM
 #39

Ah sorry the startup command has changed a bit, I've updated http://poolserverj.org/quick-start/

The command is uglier but makes things a lot more flexible.  Probably best to put it in a script...

Code:
java -classpath poolserverj.jar:../lib/*:../lib/lib_non-maven/*:../lib/plugins com.shadworld.poolserver.servicelauncher.PoolServerService start config.cfg

if you create poolserverj.sh

Code:
#!/bin/bash

java -classpath poolserverj.jar:../lib/*:../lib/lib_non-maven/*:../lib/plugins com.shadworld.poolserver.servicelauncher.PoolServerService $1 $2 $3

then it's a bit less ugly:

Code:
chmod 777 poolserverj.sh
./poolserverj.sh start config.cfg

Currently has to be started from the bin dir...

PoolServerJ Home Page - High performance java mining pool engine

1LezqRatQz7MeNoCVziYwcdwtqeEbvrdAq - http://payb.tc/shads

Quote from: Matthew N. Wright
Stop wasting the internet.
eyeoh
Jr. Member
*
Offline Offline

Activity: 31



View Profile
August 14, 2011, 02:16:57 AM
 #40

Just want to let you know that worked! Up and running with 0.2.5 and Guiminer, i'm about to start testing Diablo and Phx and all the others and stress test it.
Pages: « 1 [2] 3 4 5 6 7 8 9 10 11 12 »  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!