[ANN] Eloipool - FAST Python3 pool server software - GBT/stratum/dyntarget/proxy

[Luke-Jr]

Eloipool - FAST Python3 pool server

• First poolserver to use getmemorypool for internal work generation (note: PSJ and ecoinpool do this too, and announced first)
• Optimized merkle tree generator, does only the minimum required steps to create lots of merkle trees fast.
• Keeps a fixed-size buffer full of up-to-date merkle trees ready to ship off as soon as a getwork requests them
• Builds a fixed-size buffer full of clear (zero transactions, just subsidy) merkle trees ready to ship off immediately with longpoll replies (so miners can start working on the new block even before bitcoind has figured out which transactions to use)
• Support for more getwork extensions than any other public poolserver (update: ecoinpool caught up )
• Uses chunked HTTP transfer encoding to prevent proxy/firewall timeouts on longpoll connections.
• Coinbaser support, along with gotwork, for reliable generated payouts and merged mining
• Asynchronous JSON-RPC server using epoll to optimally handle even heavy loads.
• Interactive Python 3 console so you can inspect and modify the pool while it's running.
• Free software (open source) under the AGPLv3 license!
• Restart pool server (including upgrades) without miners losing work.
• Modular and highly configurable share logging using fast formatting functions, including support for PostgreSQL, MySQL, SQLite, and plaintext files.
• Anonymous or authenticated miner users.
• Support for X-Forwarded-For reverse proxies (and trust controls).
• Dynamic share targetting enabling fractional, power-of-two (zero bit count), and bdiff rounded miner targets.
• gzip HTTP response compression to save bandwidth without sacrificing functionality.
• Support for miners using next-generation getblocktemplate decentralized mining protocol and (pre-BIP draft) stratum mining protocol.
• Experimental support for creating "sub-pools" based on any upstream getblocktemplate server.

Now running live on:

• Eligius
• Eclipse Mining Consortium
• TripleMining
• ...and various private pools

[cablepair]

hi Luke

I was excited to see this post as I am always excited to see new pool server software

I am a little confused about this, is it unfinished - or is it just meant to be this way?

I was able to get it working even though - there are no instructions what so ever

(tip if your using Debian or Ubuntu you will probably have to dpkg-reconfigure dash and set it to no - so your using bash as default)

but where does it connect to a db? How do you manage users and shares? I see no support for this anywhere, did I miss it?

Please give us more info - I am really excited about this project!

thanks!!!

[Luke-Jr]

I am a little confused about this, is it unfinished - or is it just meant to be this way?

It will probably always be "unfinished" unless someone steps up to "finish" it. Otherwise, I plan to make it stable and reliable for Eligius's use (which probably involves some cleanup over time), and other pool ops are welcome to add functionality they need (accounts?). In the meantime, most of the code (moved into specialized do-one-thing-very-well Python modules) is working and clean; it's mainly eloipool.py that ties them all together that's a bit of a rat's nest right now.

I was able to get it working even though - there are no instructions what so ever

I figured the README and example config should be enough for most people who know what they're doing... what did you find lacking? (just what was listed below?)

(tip if your using Debian or Ubuntu you will probably have to dpkg-reconfigure dash and set it to no - so your using bash as default)

Hmm, I don't know what would depend on BASH? O.o

but where does it connect to a db?

There is PostgreSQL support for logging shares, but I haven't documented that... <.<

How do you manage users and shares? I see no support for this anywhere, did I miss it?

Eligius doesn't have users, so unless some other poolop/developer wants to add support for accounts, you don't.


In short, this isn't being released as end-user software, but as "this is what I am using" code that others can use and extend. My hopes in releasing this publicly, is that other miners and pool operators will also contribute, and all of us can benefit without each one of us reinventing everything independently from scratch. This is why I chose the AGPL license, which requires anyone running it as a public pool server to release their changes.

[cablepair]

thanks for clearing that up Luke! You never mentioned in your thread it was unfinished, understood now!


if you are using dash by default (debian, ubuntu) you get a /bin/sh Let command not found error
this is fixed by changing default shell to bash

not sure if it has anything to do with it actually working or not, but I hate errors

thanks again for all your contributions

[Luke-Jr]

if you are using dash by default (debian, ubuntu) you get a /bin/sh Let command not found error
this is fixed by changing default shell to bash

Sounds like the example config requires BASH for its example coinbaser command. It doesn't make sense to leave that in unmodified.

[streblo]

I am having trouble getting gmp-proxy.py to work. Ubuntu 12.04 32bit. Using apt-get, I installed python3-minimal and python3-anyjson.

Code:

~/bc/eloipool$ ./gmp-proxy.py --help
WARNING:jsonrpc_getwork:Error importing 'midstate' module; work will not provide midstates
Traceback (most recent call last):
  File "./gmp-proxy.py", line 26, in <module>
    pool = jsonrpc.ServiceProxy(sys.argv[1])
AttributeError: 'module' object has no attribute 'ServiceProxy'

Any ideas? 

[Luke-Jr]

I am having trouble getting gmp-proxy.py to work. Ubuntu 12.04 32bit. Using apt-get, I installed python3-minimal and python3-anyjson.

Code:

~/bc/eloipool$ ./gmp-proxy.py --help
WARNING:jsonrpc_getwork:Error importing 'midstate' module; work will not provide midstates
Traceback (most recent call last):
  File "./gmp-proxy.py", line 26, in <module>
    pool = jsonrpc.ServiceProxy(sys.argv[1])
AttributeError: 'module' object has no attribute 'ServiceProxy'

Any ideas? 

README?

[streblo]

Sorry, I forgot to move jsonrpc from python-bitcoinrpc. I did the same with python-base58, per README.

Code:

~/bc/eloipool$ ./gmp-proxy.py --help
Traceback (most recent call last):
  File "./gmp-proxy.py", line 9, in <module>
    import jsonrpc
  File "/home/alan/bc/eloipool/jsonrpc/__init__.py", line 1, in <module>
    from .json import loads, dumps, JSONEncodeException, JSONDecodeException
  File "/home/alan/bc/eloipool/jsonrpc/json.py", line 1, in <module>
    _json = __import__('json')
  File "/home/alan/bc/eloipool/json.py", line 2, in <module>
    loads = _json.loads
AttributeError: 'module' object has no attribute 'loads'

I have no .json directory in eloipool/jsonrpc/ . Sorry for my ineptitude. Thanks for the help

[Luke-Jr]

Sorry, I forgot to move jsonrpc from python-bitcoinrpc. I did the same with python-base58, per README.

Code:

~/bc/eloipool$ ./gmp-proxy.py --help
Traceback (most recent call last):
  File "./gmp-proxy.py", line 9, in <module>
    import jsonrpc
  File "/home/alan/bc/eloipool/jsonrpc/__init__.py", line 1, in <module>
    from .json import loads, dumps, JSONEncodeException, JSONDecodeException
  File "/home/alan/bc/eloipool/jsonrpc/json.py", line 1, in <module>
    _json = __import__('json')
  File "/home/alan/bc/eloipool/json.py", line 2, in <module>
    loads = _json.loads
AttributeError: 'module' object has no attribute 'loads'

I have no .json directory in eloipool/jsonrpc/ . Sorry for my ineptitude. Thanks for the help

Where did eloipool/json.py come from? json is a standard Python module...

[streblo]

Where did eloipool/json.py come from? json is a standard Python module...

I accidentally moved it from python-bitcoinrpc. After moving it back to eloipool/json it seems to be working better-ish:

Code:

~/bc/eloipool$ ./gmp-proxy.py http://<address>:a@mining.eligius.st:8337
WARNING:jsonrpc_getwork:Error importing 'midstate' module; work will not provide midstates
Traceback (most recent call last):
  File "./gmp-proxy.py", line 134, in <module>
    server.serve_forever()
  File "/home/alan/bc/eloipool/networkserver.py", line 267, in serve_forever
    self.final_init()
  File "/home/alan/bc/eloipool/jsonrpcserver.py", line 300, in final_init
    ShareTargetHex = '%064x' % (self.ShareTarget,)
AttributeError: 'JSONRPCServer' object has no attribute 'ShareTarget'

[Luke-Jr]

Where did eloipool/json.py come from? json is a standard Python module...

I accidentally moved it from python-bitcoinrpc. After moving it back to eloipool/json it seems to be working better-ish:

Code:

~/bc/eloipool$ ./gmp-proxy.py http://<address>:a@mining.eligius.st:8337
WARNING:jsonrpc_getwork:Error importing 'midstate' module; work will not provide midstates
Traceback (most recent call last):
  File "./gmp-proxy.py", line 134, in <module>
    server.serve_forever()
  File "/home/alan/bc/eloipool/networkserver.py", line 267, in serve_forever
    self.final_init()
  File "/home/alan/bc/eloipool/jsonrpcserver.py", line 300, in final_init
    ShareTargetHex = '%064x' % (self.ShareTarget,)
AttributeError: 'JSONRPCServer' object has no attribute 'ShareTarget'

That one's a real bug. try this until it works, for now: git checkout HEAD^

[check_status]

I thought I might have something I could add to the How To's and Guides Mega List to beef up the Pools category, but it's limited in being able to meet the goals of that thread, which is focused on helping novice users to get involved. Will be nice to include when this fleshes out a little more.

[streblo]

That one's a real bug. try this until it works, for now: git checkout HEAD^

EDIT: Working now, cheers!

[Luke-Jr]

That one's a real bug. try this until it works, for now: git checkout HEAD^

Same problem

Code:

~/bc/eloipool$ git checkout HEAD^
Previous HEAD position was 5e363be... Merge branch 'sharetarget'
HEAD is now at ac36d06... Merge branch 'sharetarget'
~/bc/eloipool$ ./gmp-proxy.py http://<ADDRESS>:a@mining.eligius.st:8337
WARNING:jsonrpc_getwork:Error importing 'midstate' module; work will not provide midstates
Traceback (most recent call last):
  File "./gmp-proxy.py", line 134, in <module>
    server.serve_forever()
  File "/home/alan/bc/eloipool/networkserver.py", line 267, in serve_forever
    self.final_init()
  File "/home/alan/bc/eloipool/jsonrpcserver.py", line 300, in final_init
    ShareTargetHex = '%064x' % (self.ShareTarget,)
AttributeError: 'JSONRPCServer' object has no attribute 'ShareTarget'

"until it works"

[sippsnapp]

How do i install the python-bitcoinrpc and python-base58 ?
Thx

[Luke-Jr]

How do i install the python-bitcoinrpc and python-base58 ?
Thx

It's usually sufficient to just get the git checkout linked under your eloipool directory.

[sippsnapp]

I installed python3.2 from source, what am i doing wrong?

Code:

root@j064:/opt/midstate# make
gcc -march=native -Wall -funroll-all-loops -O3 -fstrict-aliasing -Wall -std=c99 -I/usr/include/python3.2  midstatemodule.c -o test -Wl,-O1 -Wl,--as-needed -lpython3.2
midstatemodule.c:5:20: error: Python.h: Datei oder Verzeichnis nicht gefunden
midstatemodule.c:96: error: expected â=â, â,â, â;â, âasmâ or â__attribute__â before â*â token
midstatemodule.c:138: error: array type has incomplete element type
midstatemodule.c:139: error: âmidstate_helperâ undeclared here (not in a function)
midstatemodule.c:139: error: âMETH_Oâ undeclared here (not in a function)
midstatemodule.c:144: error: variable âmidstatemoduleâ has initializer but incomplete type
midstatemodule.c:145: error: âPyModuleDef_HEAD_INITâ undeclared here (not in a function)
midstatemodule.c:145: warning: excess elements in struct initializer
midstatemodule.c:145: warning: (near initialization for âmidstatemoduleâ)
midstatemodule.c:146: warning: excess elements in struct initializer
midstatemodule.c:146: warning: (near initialization for âmidstatemoduleâ)
midstatemodule.c:147: warning: excess elements in struct initializer
midstatemodule.c:147: warning: (near initialization for âmidstatemoduleâ)
midstatemodule.c:148: warning: excess elements in struct initializer
midstatemodule.c:148: warning: (near initialization for âmidstatemoduleâ)
midstatemodule.c:149: warning: excess elements in struct initializer
midstatemodule.c:149: warning: (near initialization for âmidstatemoduleâ)
midstatemodule.c:150: warning: excess elements in struct initializer
midstatemodule.c:150: warning: (near initialization for âmidstatemoduleâ)
midstatemodule.c:151: warning: excess elements in struct initializer
midstatemodule.c:151: warning: (near initialization for âmidstatemoduleâ)
midstatemodule.c:152: warning: excess elements in struct initializer
midstatemodule.c:152: warning: (near initialization for âmidstatemoduleâ)
midstatemodule.c:154: warning: excess elements in struct initializer
midstatemodule.c:154: warning: (near initialization for âmidstatemoduleâ)
midstatemodule.c:157: error: expected â=â, â,â, â;â, âasmâ or â__attribute__â before âPyInit_midstateâ
make: *** [test] Fehler 1

[Luke-Jr]

Code:

midstatemodule.c:5:20: error: Python.h: Datei oder Verzeichnis nicht gefunden

Looks like it can't find Python.h. It's probably under /usr/local/... ? Maybe edit the Makefile as needed :/

[sippsnapp]

I already made another thread, however i think its better placed here because its mianly about eloipool.
https://bitcointalk.org/index.php?topic=113730.msg1235209#msg1235209
This is the error i get:

Code:

root@j064:/opt/eloipool# ./eloipool.py
Traceback (most recent call last):
  File "./eloipool.py", line 46, in <module>
    import jsonrpc
ImportError: No module named jsonrpc
root@j064:/opt/eloipool#

[sippsnapp]

Got it working, mining right now on it.
I installed it on ubuntu 12.04

EDIT: i decided to remove the guide i wrote as i dont know how secure this way of installing was/is, please excuse.