Multipool - the pool mining pool (with source code)

[Grinder]

It would be nice with a status report, nick5429. What's up with all the pending shares, and when will you start paying out the rewards?

[niooron]

You guys should just run a local pool, its easy to setup since all dependencies are at CPAN, and you keep everything.

[mf]

I asked the original poster what license was the code under, as I started rewriting bits of it to be more.. Perlish.. and wanted to be able to release my modifications under the proper license.
I ended up ripping off completely the part that distributes to accounts, and had been running it for a while locally. All was good.
Unfortunately I kept getting a high number of stales/rejections due to the network code that deals with sending proof of work, which is a bit awkward..

I had been thinking of writing something similar using more modern modules like AnyEvent, but lacked the time. Will likely do this if there is enough want.

On the other hand, I went instead to a simpler system which periodically checks which is the best pool to switch to, kills the miner and has it switch to the chosen pool.

I don't think everybody needs both the pool-hopping *and* the shares distribution: I'd be creating the pool hopping thing only, for one to use with their own miners.

I am not entirely sure if the idea of having multipool sites "up" would create problems for the pools, as a number of GHash suddenly start hopping from one to the next.. with what has been happening as of late with DDOSes on the pools..

Still, if anybody is running multipool locally and needs a hand with some small fixes to the current multipool code, I'd be happy to oblige -- PM me for info.

[Folax]

You guys should just run a local pool, its easy to setup since all dependencies are at CPAN, and you keep everything.

Would love to, need some linux skills to do so it seems, which I lack.

[niooron]

You guys should just run a local pool, its easy to setup since all dependencies are at CPAN, and you keep everything.

Would love to, need some linux skills to do so it seems, which I lack.

Install virtual box, a linux distro, download the dependencies from CPAN, and youre done. I have a fedora 14 vm with 384 MB of RAM and it doesn't even use 10% cpu.

[niooron]

I asked the original poster what license was the code under, as I started rewriting bits of it to be more.. Perlish.. and wanted to be able to release my modifications under the proper license.
I ended up ripping off completely the part that distributes to accounts, and had been running it for a while locally. All was good.
Unfortunately I kept getting a high number of stales/rejections due to the network code that deals with sending proof of work, which is a bit awkward..

I had been thinking of writing something similar using more modern modules like AnyEvent, but lacked the time. Will likely do this if there is enough want.

On the other hand, I went instead to a simpler system which periodically checks which is the best pool to switch to, kills the miner and has it switch to the chosen pool.

I don't think everybody needs both the pool-hopping *and* the shares distribution: I'd be creating the pool hopping thing only, for one to use with their own miners.

I am not entirely sure if the idea of having multipool sites "up" would create problems for the pools, as a number of GHash suddenly start hopping from one to the next.. with what has been happening as of late with DDOSes on the pools..

Still, if anybody is running multipool locally and needs a hand with some small fixes to the current multipool code, I'd be happy to oblige -- PM me for info.

We need to add more pools to pool.conf. I have no idea how to parse the pools API.

[mf]

We need to add more pools to pool.conf. I have no idea how to parse the pools API.

The file pools.conf is quite straightforward; for example:

Code:

 slush
api.bitcoin.cz:8332
https://mining.bitcoin.cz/stats/json/ 
$time=~s/.*round_duration": "//; $time=~s/".*//; $time=parse_time($time); $rate=~s/.*ghashes_ps": "//; $rate=~s/(\.\d*)?".*//; $shares=~s/.*shares": //s; $shares=~s/,.*//s;
utility_2

The first line of a stanza is the pool's name; second is the RPC port (the one the miners use), the third is a URL which the Multipool.pl will fetch in order to gather statistics, then some code whose purpose is to get the time since last block found, total shares and total rate in GHash, and lastly the "utility" function to be used to calculate the utility for a pool.

The line of code is messy, but the idea is that it is executed in a context in which $_ (the topic variable) is aliased to the data received from the URL above, and it needs to "output" three variables:
* $time, in seconds, indicating how long ago the last block was found;
* $rate, in GHashes/s, indicating the pool's speed in GH/s
* $shares, indicating how many shares the pool has currently mined.

An example, for mineco.in:

The stats are in JSON:

Code:

  $ curl https://mineco.in/stats.json
  {"hashrate":86362247950,"users":568,"blocks_found":16,"shares_this_round":2102867,"last_block":{"found":1309779953,"user":"mau","shares":1461964,"duration":79213}}

The info we're looking for is the number after "hashrate", but it's in hashes per second so it needs to be divided by 10^6; the number after "duration" and the number after "shares_this_round".

The way you can test the line of code that goes in the config is via:

Code:

  curl URL | perl -lne' $time=$shares=$rate=$_; 
     THAT LINE OF CODE HERE;
     print "Rate: $rate"; print "Time: $time"; print "Shares: $shares";   '

so for example:

Code:

  $ curl https://mineco.in/stats.json | perl -lne'
     $time=$shares=$rate=$_; 
     $time=~s/^.*"duration":(\d+).*$/$1/;
     $shares=~s/^.*"shares_this_round":(\d+).*/$1/;
     $rate=~s/^.*"hashrate":(\d+).*/$1/;$rate/=10**9;
     print "Rate: $rate"; print "Time: $time"; print "Shares: $shares";   '

  Rate: 86.069951565
  Time: 79213
  Shares: 2097252

There you go

The stanza then becomes:

Code:

minecoin
mineco.in:3000
https://mineco.in/stats.json
$time=~s/^.*"duration":(\d+).*$/$1/; $shares=~s/^.*"shares_this_round":(\d+).*/$1/; $rate=~s/^.*"hashrate":(\d+).*/$1/;$rate/=10**9;
utility_1

Check out the different utility_ functions, but usually utility_1 seems to work fine.

If you want me to do others do not hesitate to PM me or try doing them yourselves; above I have explained how.
(edited: wrapped in code blocks)

[MiningBuddy]

Thank you mf, excellent explanation!

[gno]

I never did get the perl CPAN dependencies squared away on two different CENTOS boxes.  Not sure where I went wrong but ran out of patience at the time.  Never could resolve whatever dependency caused

Can't locate JSON/RPC/Client.pm

I appreciate the gentleman running Multiclone but I'd like to run one myself for my own use.  I guess I need to try again sometime, perhaps with a newer distro than Centos 5

[Folax]

You guys should just run a local pool, its easy to setup since all dependencies are at CPAN, and you keep everything.

Would love to, need some linux skills to do so it seems, which I lack.

Install virtual box, a linux distro, download the dependencies from CPAN, and youre done. I have a fedora 14 vm with 384 MB of RAM and it doesn't even use 10% cpu.

I have some Fedora experience but CPAN dependencies are new to me, and would probably result in a bucketload of newb-questions coming your way.

Would it be possible to 7zip up the virtual machine and upload using a Torrent? I am pretty sure people would send you a coin for the download, I would for sure...

[mf]

I never did get the perl CPAN dependencies (...)
Can't locate JSON/RPC/Client.pm
I appreciate the gentleman running Multiclone but I'd like to run one myself for my own use.
I guess I need to try again sometime, perhaps with a newer distro than Centos 5

Why not today?

Use cpanm (http://search.cpan.org/~miyagawa/App-cpanminus-1.4008/lib/App/cpanminus.pm) to install modules -locally- rather than as root.

Example, for Multipool I created a new user, disabled password and all, got the source and then installed the dependencies in a "local lib" with cpanm.

I'll re-do bits of it for you, so you can get comfortable with the process

I'll first create a new user:

Code:

root@xxx:~# adduser --disabled-password teensy
Adding user `teensy' ...
...

Switch to it:

Code:

$ su - teensy

Fetch cpanm:

Code:

teensy@xxx:~$ curl -L http://cpanmin.us > cpanm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  163k  100  163k    0     0  78241      0  0:00:02  0:00:02 --:--:--  377k
teensy@xxx:~$ chmod +x cpanm

Tell it to install JSON::RPC::Client in the local ./lib directory:

Code:

teensy@xxx:~$ ./cpanm -l ./lib JSON::RPC::Client
--> Working on JSON::RPC::Client
Fetching http://search.cpan.org/CPAN/authors/id/M/MA/MAKAMAKA/JSON-RPC-0.96.tar.gz ... OK
Configuring JSON-RPC-0.96 ... OK
Building and testing JSON-RPC-0.96 ... OK
Successfully installed JSON-RPC-0.96
1 distribution installed

Now, if you look under lib/ you will see there are a bunch of directories which you need added to Perl's path for libraries.
In my case, these are: lib/lib/perl5, and lib/lib/perl5/x86_64-linux-gnu-thread-multi.

To see it worked,

Code:

$ perl -MJSON::RPC::Client -e1
Can't locate JSON/RPC/Client.pm in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.10.1 /usr/local/share/perl/5.10.1 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.10 /usr/share/perl/5.10 /usr/local/lib/site_perl .).
BEGIN failed--compilation aborted.
$ PERL5LIB="lib/lib/perl5/x86_64-linux-gnu-thread-multi:lib/lib/perl5" perl -MJSON::RPC::Client -e1

So, what you then can do is either:
* use that PERL5LIB= line before launching Multipool.pl, a la: $ PERL5LIB=xxxx perl Multilib.pl, or
* properly install cpanm and local::lib (search for tutorials on the web, or ask me for more info), or
* modify Multilib.pl to have " use lib './lib/lib/perl5', 'lib/lib/perl5/x86_64-linux-gnu-thread-multi'; " at its beginning after "use strict"

There you go

[mf]

Would it be possible to 7zip up the virtual machine and upload using a Torrent?
I am pretty sure people would send you a coin for the download, I would for sure...

If you already have a distro setup, you can create a new user and follow the instructions I gave above for installing JSON::RPC::API in a local lib, and simply install the other prerequisite modules via it: Finance::Bitcoin::API and Math::Integral::Romberg being the other two ones I think.

In order to do that, the linux distro -does- need to have a working compiler, though.

Creating a virtual machine for this multipool looks a bit overkill to me..

[nick5429]

FWIW, I brought multiclone back up again earlier this morning.

I'm not sure what happened this time -- the server just stopped responding entirely. I couldn't even ctrl+c to kill it...

In other news, I was all set to do a payout (which would include everything except btcmine and bitcoins.lc), but am getting an error when trying to do a sendmany, even when I reduce it to its simplest case, and use way fewer digits after the decimal:

bash-4.1$  bitcoind sendmany "main" '{"17x6iXEGfE94VnDwUMS1E5fii3FEqXLrEf":0.323}' "payout"
error: type mismatch

I'm using the newest release of bitcoin.  Any ideas?

I tried building bitcoind from source, but apparently did something wrong when selecting the berkeley db version.  It compiles, but crashes on start:

bash-4.1$ ./bitcoind -daemon
bitcoin server starting
bash-4.1$

************************
EXCEPTION: 11DbException
DbEnv::open: Invalid argument
bitcoin in AppInit()

terminate called after throwing an instance of 'DbException'
  what():  DbEnv::open: Invalid argument

If all else fails, I can do individual sends (rather than sendmany), but ... that's going to be a headache

[Sukrim]

bash-4.1$  bitcoind sendmany "main" '{"17x6iXEGfE94VnDwUMS1E5fii3FEqXLrEf":0.323}' "payout"
error: type mismatch

I'm using the newest release of bitcoin.  Any ideas?

Does

Code:

bash-4.1$  bitcoind sendmany "main" '{"17x6iXEGfE94VnDwUMS1E5fii3FEqXLrEf":Decimal("0.323")}' "payout"

work?

I think I saw that somewhere, but I'm not sure if this could be the reason/fault.

[Folax]

Creating a virtual machine for this multipool looks a bit overkill to me..

I create VM's for all kinds of things (ESXi) so that will not be the problem
But maybe you are right and I have to try myself... Thanks in advance for all the helpful stuff you have already written here, I will surely need it

[Grinder]

bash-4.1$  bitcoind sendmany "main" '{"17x6iXEGfE94VnDwUMS1E5fii3FEqXLrEf":0.323}' "payout"
error: type mismatch

I'm using the newest release of bitcoin.  Any ideas?

It seems to be an ugly bug that only shows itself when you use certain versions of Linux. If you can use JSON to communicate with the server it should work, otherwise I think you have to try to find a different version of Linux. I saw someone say that for instance it works with Ubuntu 10.04 but not 10.10.

This thread has an example of using JSON with curl and a text file for the content: http://forum.bitcoin.org/index.php?topic=23776.0

[Houseonfire]

We need to add more pools to pool.conf. I have no idea how to parse the pools API.

The file pools.conf is quite straightforward; for example:

Code:

 slush
api.bitcoin.cz:8332
https://mining.bitcoin.cz/stats/json/ 
$time=~s/.*round_duration": "//; $time=~s/".*//; $time=parse_time($time); $rate=~s/.*ghashes_ps": "//; $rate=~s/(\.\d*)?".*//; $shares=~s/.*shares": //s; $shares=~s/,.*//s;
utility_2

The first line of a stanza is the pool's name; second is the RPC port (the one the miners use), the third is a URL which the Multipool.pl will fetch in order to gather statistics, then some code whose purpose is to get the time since last block found, total shares and total rate in GHash, and lastly the "utility" function to be used to calculate the utility for a pool.

The line of code is messy, but the idea is that it is executed in a context in which $_ (the topic variable) is aliased to the data received from the URL above, and it needs to "output" three variables:
* $time, in seconds, indicating how long ago the last block was found;
* $rate, in GHashes/s, indicating the pool's speed in GH/s
* $shares, indicating how many shares the pool has currently mined.

An example, for mineco.in:

The stats are in JSON:

Code:

  $ curl https://mineco.in/stats.json
  {"hashrate":86362247950,"users":568,"blocks_found":16,"shares_this_round":2102867,"last_block":{"found":1309779953,"user":"mau","shares":1461964,"duration":79213}}

The info we're looking for is the number after "hashrate", but it's in hashes per second so it needs to be divided by 10^6; the number after "duration" and the number after "shares_this_round".

The way you can test the line of code that goes in the config is via:

Code:

  curl URL | perl -lne' $time=$shares=$rate=$_; 
     THAT LINE OF CODE HERE;
     print "Rate: $rate"; print "Time: $time"; print "Shares: $shares";   '

so for example:

Code:

  $ curl https://mineco.in/stats.json | perl -lne'
     $time=$shares=$rate=$_; 
     $time=~s/^.*"duration":(\d+).*$/$1/;
     $shares=~s/^.*"shares_this_round":(\d+).*/$1/;
     $rate=~s/^.*"hashrate":(\d+).*/$1/;$rate/=10**9;
     print "Rate: $rate"; print "Time: $time"; print "Shares: $shares";   '

  Rate: 86.069951565
  Time: 79213
  Shares: 2097252

There you go

The stanza then becomes:

Code:

minecoin
mineco.in:3000
https://mineco.in/stats.json
$time=~s/^.*"duration":(\d+).*$/$1/; $shares=~s/^.*"shares_this_round":(\d+).*/$1/; $rate=~s/^.*"hashrate":(\d+).*/$1/;$rate/=10**9;
utility_1

Check out the different utility_ functions, but usually utility_1 seems to work fine.

If you want me to do others do not hesitate to PM me or try doing them yourselves; above I have explained how.
(edited: wrapped in code blocks)

I love the guide and everything. But when I try this, my multipool tells me that minecoin is offline. I basically copied your script into the "pools.conf", opened the port and then added the minecoin info to "accounts.conf".

I was also wondering if if you can help with triplemining? Iv tried but i just get errors and problems all over because im doing it incredibly wrong.


Also, even though I removed the ser vers from the list, they still appear in the multipool terminal as trying to connect. How would I go about removing them so I can speed things up with servers that connect?

[mf]

my multipool tells me that minecoin is offline.

I am currently getting some disconnections from them, too; and several spurious rejections.
The way Multipool seems to deal with that is it pushes "down" the pool, until it considers it offline.
Could be either offline per se, or the script giving up.

I basically copied your script into the "pools.conf", opened the port and then added the minecoin info to "accounts.conf".

Just the stanza I wrote, between "minecoin" and "utility_1", I hope.
Only one newline between stanzas.

Ensure your user/pass are correct for that miner on the accounts.conf...

The script should show when it is that it's putting a pool offline, though: check the scroll / logs to see if it gave you any more info.
Again, sorry but I did not use the script as-is myself. I did modify/butcher a fair bit of it.
I still have the original which I am using to help you guys out.

I was also wondering if if you can help with triplemining? Iv tried but i just get errors and problems all over because im doing it incredibly wrong.

If you have a URL where they provide some statistics... by the cursory look I gave, there's only the "blocks found" page, which only contains information in human terms ("1 days ago")...
PM me in case you need further help; post a "solution" once you get something working.

[Zoomer]

I'm thinking we need some thread cleanup. One thread for multipool, the software, another for the pool, and yet another for multiclone.

[mf]

Also, even though I removed the ser vers from the list, they still appear in the multipool terminal as trying to connect. How would I go about removing them so I can speed things up with servers that connect?

Multipool will only read the accounts and pools at startup, so you need to kill it and start it again, if you want it to pick up new pools or remove old pools.

I think I should put more time into my replacement.. assuming there's enough interest and people aren't just using this to solo mine different pools.