[ANN] Eloipool - FAST Python3 pool server software - GBT/stratum/dyntarget/proxy

[TAiS46]

Hey,

I am not the best in git and linux, can you tell me, how to revert the last commit?
I have just clone the git from github, but it is very old I think.

[Luke-Jr]

Hey,

I am not the best in git and linux, can you tell me, how to revert the last commit?
I have just clone the git from github, but it is very old I think.

Eloipool is in Gitorious, not GitHub...

To simply pull the top commit off:

Code:

git reset --hard HEAD^

[mrb]

I have been solo mining against an eloipool instance (revision 475de39) for the last few days, using bfgminer 2.9.3 and 'getwork'. eloipool is configured with a ShareTarget set to 0x0000000000fff... to emulate difficulty 256. I wanted to give this setup a shot before trying the newer Stratum protocol. I thought it would be a robust way to mine, using mature code paths (both at the bfgminer side, and at the eloipool side).

I was wrong.

I solved a block today. I know it because bfgminer currently shows "Best share: 19.5M". However eloipool silently ignored the solved block. There goes 25 BTC...

Furthermore, it is almost impossible to do a post-mortem analysis. I found out that checkShare() in eloipool.py has buggy logging code, so pretty much no useful information has been logged. Indeed, the checkShare.logger object has a default level of 'WARNING', so logfunc() below will never log INFO and DEBUG messages:

Code:

        logfunc = getattr(checkShare.logger, 'info' if blkhashn <= networkTarget else 'debug')
        logfunc('BLKHASH: %64x' % (blkhashn,))
        logfunc(' TARGET: %64x' % (networkTarget,))

Either eloipool silently ignored the block due to an internal bug. Or it crafted an invalid block which was sent to my bitcoind, who in turns ignored it because it was invalid. I am quite disappointed.

In case you care to investigate, Luke, here is my config.py:

Code:

### Settings relating to server identity

# Name of the server
ServerName = 'mrb eloipool'

### Settings relating to server scaling/load

# Share hashes must be below this to be valid shares
# If dynamic targetting is enabled, this is a minimum
ShareTarget = 0x0000000000ffffffffffffffffffffffffffffffffffffffffffffffffffffff

# Automatically adjust targets per username
# 0 = disabled
# 1 = arbitrary targets
# 2 = power of two difficulties (zero bit counts)
DynamicTargetting = 0

# How many shares per minute to try to achieve on average
DynamicTargetGoal = 8

# Number of seconds hashrate is measured over
DynamicTargetWindow = 120

# Minimum and maximum of merkle roots to keep queued
WorkQueueSizeRegular = (0x100, 0x1000)

# Minimum and maximum of BLANK merkle roots to keep queued
# (used if we run out of populated ones)
WorkQueueSizeClear = (0x1000, 0x2000)

# Minimum and maximum of BLANK merkle roots to keep queued, one height up
# (used for longpolls)
WorkQueueSizeLongpoll = (0x1000, 0x2000)

# How long to wait between getmemorypool updates normally
MinimumTxnUpdateWait = 5

# How long to wait between retries if getmemorypool fails
TxnUpdateRetryWait = 1

# How long to sleep in idle loops (temporary!)
IdleSleepTime = 0.1

### Settings relating to reward generation

# Address to generate rewards to
TrackerAddr = '1xxx<obscured-address>'

# Coinbaser command to control reward delegation
# NOTE: This example donates 1% of block rewards to Luke-Jr for Eloipool development
#CoinbaserCmd = 'echo -e "1\\n$((%d / 100))\\n1579aXhdwvKZEMrAKoCZhzGuqMa8EonuXU"'

### Settings relating to upstream data providers

# JSON-RPC server for getmemorypool
UpstreamURI = 'http://<obscured-user>:<obscured-password>@localhost:8332'

# Set to True if you want shares meeting the upstream target to wait for a
# response from the upstream server before logging them. Otherwise, for such
# shares, upstreamResult will always be True and upstreamRejectReason will
# always be None. Note that enabling this may cause shares to be logged out of
# order, or with the wrong timestamp (if your share logger uses the log-time
# rather than share-time).
DelayLogForUpstream = False

# Bitcoin p2p server for announcing blocks found
UpstreamBitcoindNode = ('127.0.0.1', 8333)

# Network ID for the primary blockchain
#UpstreamNetworkId = b'\xFA\xBF\xB5\xDA'  # testnet
UpstreamNetworkId = b'\xF9\xBE\xB4\xD9'  # mainnet

# Secret username allowed to use setworkaux
#SecretUser = ""

# URI to send gotwork with info for every share submission
#GotWorkURI = ''

# Share hashes must be below this to be submitted to gotwork
GotWorkTarget = 0x0000000000ffffffffffffffffffffffffffffffffffffffffffffffffffffff

# Aim to produce blocks with transaction counts that are a power of two
# This helps avoid any chance of someone abusing CVE-2012-2459 with them
# 1 = cut out feeless transactions; 2 = cut out even fee-included transactions (if possible)
POT = 2

# Avoid mining feeless transactions except to satisfy POT
# Note this only works if POT is in fact enabled in the first place
Greedy = False

### Settings relating to network services

# Addresses to listen on for JSON-RPC getwork server
# Note that Eloipool only supports IPv6 sockets, and if you want to bind to an
# IPv4 address you will need to prepend it with ::ffff: eg ::ffff:192.168.1.2
JSONRPCAddresses = (
	('', 28332),
)

# Addresses to listen on for Bitcoin node
# Note that Eloipool only supports IPv6 sockets, and if you want to bind to an
# IPv4 address you will need to prepend it with ::ffff: eg ::ffff:192.168.1.2
BitcoinNodeAddresses = (
	('', 28333),
)

# Addresses that are allowed to "spoof" from address with the X-Forwarded-For header
TrustedForwarders = ('::ffff:127.0.0.1',)


# Logging of shares:
ShareLogging = (
	{
		'type': 'logfile',
		'filename': 'share-logfile',
		'format': "{time} {Q(remoteHost)} {username} {YN(not(rejectReason))} {dash(YN(upstreamResult))} {dash(rejectReason)} {solution}\n",
	},
)

[Luke-Jr]

I solved a block today. I know it because bfgminer currently shows "Best share: 19.5M". However eloipool silently ignored the solved block. There goes 25 BTC...

I doubt Eloipool was the problem in this case. The "best share" code is much newer, and there have been reports of similar behaviour from altcoin solo mining; unfortunately, I don't know any good way to reproduce it. Did you happen to be running BFGMiner with a debug log by any chance? There might be some useful info in there...

Furthermore, it is almost impossible to do a post-mortem analysis. I found out that checkShare() in eloipool.py has buggy logging code, so pretty much no useful information has been logged. Indeed, the checkShare.logger object has a default level of 'WARNING', so logfunc() below will never log INFO and DEBUG messages:

Eloipool's code only configures loggers for DEBUG or INFO... For post-mortem analysis on the pool side, check out the RBFs variable (only kept in memory!) and bitcoind's debug.log. Of course, that's assuming the share ever got to Eloipool, which I suspect isn't the case since you apparently didn't see the info logging from it.

[mrb]

I was not running bfgminer with --debuglog.

I restarted eloipool to change some of its config, so it is too late to look at the RBFs variables.

The only trace of data I have on eloipool is the share-logfile. But I need to write some custom code to re-hash the share data there, to find out if the share hash with a 19.5M difficulty is in there. That's what I am doing now...

[mrb]

By the way, what is the use of GotWorkURI? Shares found from getmemorypool work cannot be submitted via getwork, or can they?

[Luke-Jr]

By the way, what is the use of GotWorkURI? Shares found from getmemorypool work cannot be submitted via getwork, or can they?

GotWorkURI is for merged mining. gotwork != getwork

[mrb]

Well the highest share I found in share-logfile is 2.3M which was the value displayed by bfgminer yesterday. But then again, the code path between the moment receiveShare() is called and when it is logged in share-logfile, is quite complex and it is possible to imagine a bug that would cause the shared to be received, but not logged.

I am surprised about these reports of "Best share" possibly displaying incorrect information. I would think this is a dead simple feature to implement, so why would this be buggy?

[Luke-Jr]

Upon further investigation, I am 99% certain we're looking at a BFGMiner bug here. There are two functions which both recalculate the share/block hash and compare it to a target: fulltest (util.c) and regeneratehash (miner.c). It seems that at least on little-endian, these two functions produce opposite endians for each 32-bit chunk of the hash; in cases where the target is presumably not met, the shares are fed through the share_diff function (responsible for updating the best block) with the opposite endian than it expects. This causes your best block to report something erroneous (or at least it seems this is where the error is). What needs to be done now, is to audit the endian handling in these two functions to determine which one is correct and ideally unify the code into a single function. Hopefully I'll get to that within the next week, but if you want to take a stab at it, feel free...

[mrb]

I am reassured the bug seems to be in bfgminer. I may take a stab at a fix this weekend.

[jgarzik]

Yeah that sounds like a bug everybody inherited from cpuminer, which definitely had rough edges when it came to any target besides the standard difficulty-1.

[mrb]

I confirm the bug. One of my tests shows:

Hash computed by hashtest2():
0000000039d5e400932fef0b1d7cdec1f2833dec594e30ee03252113f03be9b4

Hash computed by regeneratehash():
0000000000e4d5390bef2f93c1de7c1dec3d83f2ee304e5913212503b4e93bf0

regeneratehash() is the one that does not compute it correctly.

But there is a 2nd endianness bug in share_diff() which cancels the bug in regeneratehash (which is why the "best share" is always updated correctly when a new share is found)! This 2nd endianness bug in share_diff() is what causes TNR_HIGH shares to sometimes spuriously increase the "best share" statistic without a corresponding share submitted to the pool.

I have a patch fixing both endianness bugs. I am currently testing it...

[mrb]

Pull request sent, see main bfgminer thread: https://bitcointalk.org/index.php?topic=78192.msg1386296#msg1386296

[tvasconcelos]

So what's the latest stable release of eloipool that i can use? I've seen some forks in github too, are they good or is better to stick with the one from Luke in Gitourios?
Is eloi good for litecoins? I'm trying to set a private pool but been having a hard time. Tryed ecoinpool, pushpool and i want to give a shot at eloipool. My limited knowledge in programming is giving me some difficulties in setting up the pool, and so little info on how to set it up also. Ecoinpool has a really nice tutorial on how to setup, but it's been unmaintained for quite some time now, and running in couchdb, written in erlang, it's giving me a hard time to setup. The db grows way too big for my available space. Can't get it to use mysql to use a frontend like simplecoin. How about eloipool, are there any good open source frontend that i can use?

Thanks

[tvasconcelos]

Ok, i'm trying to get eloipool with litecoin, and i'm getting this error:

 python eloipool.py
Traceback (most recent call last):
  File "eloipool.py", line 43, in <module>
    bcnode = BitcoinNode(config.UpstreamNetworkId)
  File "/home/pi/eloipool/bitcoin/node.py", line 132, in __init__
    super().__init__(*a, **ka)
TypeError: super() takes at least 1 argument (0 given)

Can someone give some help on this. My config goes like this:

Code:

### Settings relating to server identity

# Name of the server
ServerName = 'PT LTC Pool'

### Settings relating to server scaling/load

# Share hashes must be below this to be valid shares
# If dynamic targetting is enabled, this is a minimum
ShareTarget = 0x00000000ffffffffffffffffffffffffffffffffffffffffffffffffffffffff

# Automatically adjust targets per username
# 0 = disabled
# 1 = arbitrary targets
# 2 = power of two difficulties (zero bit counts)
DynamicTargetting = 0

# How many shares per minute to try to achieve on average
DynamicTargetGoal = 8

# Number of seconds hashrate is measured over
DynamicTargetWindow = 120

# Minimum and maximum of merkle roots to keep queued
WorkQueueSizeRegular = (0x100, 0x1000)

# Minimum and maximum of BLANK merkle roots to keep queued
# (used if we run out of populated ones)
WorkQueueSizeClear = (0x1000, 0x2000)

# Minimum and maximum of BLANK merkle roots to keep queued, one height up
# (used for longpolls)
WorkQueueSizeLongpoll = (0x1000, 0x2000)

# How long to wait between getmemorypool updates normally
MinimumTxnUpdateWait = 5

# How long to wait between retries if getmemorypool fails
TxnUpdateRetryWait = 1

# How long to sleep in idle loops (temporary!)
IdleSleepTime = 0.1

### Settings relating to reward generation

# Address to generate rewards to
TrackerAddr = 'LTC Address'  # testnet

# Coinbaser command to control reward delegation
# NOTE: This example donates 1% of block rewards to Luke-Jr for Eloipool develo$
CoinbaserCmd = 'echo -e "1\\n$((%d / 100))\\LTC Address"'

### Settings relating to upstream data providers

# JSON-RPC server for getmemorypool
UpstreamURI = 'http://ltc user:ltc password@localhost:9332'

# Set to True if you want shares meeting the upstream target to wait for a
# response from the upstream server before logging them. Otherwise, for such
# shares, upstreamResult will always be True and upstreamRejectReason will
# always be None. Note that enabling this may cause shares to be logged out of
# order, or with the wrong timestamp (if your share logger uses the log-time
# rather than share-time).
DelayLogForUpstream = False

# Bitcoin p2p server for announcing blocks found
UpstreamBitcoindNode = ('127.0.0.1', 9333)

# Network ID for the primary blockchain
UpstreamNetworkId = b'\xFA\xBF\xB5\xDA'

# Secret username allowed to use setworkaux
#SecretUser = ""

# URI to send gotwork with info for every share submission
#GotWorkURI = ''

# Share hashes must be below this to be submitted to gotwork
GotWorkTarget = 0x00000000fffffffffffffffffffffffffffffffffffffffffffffffffffff$

# Aim to produce blocks with transaction counts that are a power of two
# This helps avoid any chance of someone abusing CVE-2012-2459 with them
# 1 = cut out feeless transactions; 2 = cut out even fee-included transactions $
POT = 2

# Avoid mining feeless transactions except to satisfy POT
# Note this only works if POT is in fact enabled in the first place
Greedy = False

### Settings relating to network services

# Addresses to listen on for JSON-RPC getwork server
# Note that Eloipool only supports IPv6 sockets, and if you want to bind to an
# IPv4 address you will need to prepend it with ::ffff: eg ::ffff:192.168.1.2
JSONRPCAddresses = (
        ('::ffff:poollink.no-ip.org', 9337),
)

# Addresses to listen on for Bitcoin node
# Note that Eloipool only supports IPv6 sockets, and if you want to bind to an
# IPv4 address you will need to prepend it with ::ffff: eg ::ffff:192.168.1.2
BitcoinNodeAddresses = (
        ('::ffff:poollink.no-ip.org', 9338),
)

# Addresses that are allowed to "spoof" from address with the X-Forwarded-For h$
TrustedForwarders = ('::ffff:127.0.0.1',)


# Logging of shares:
ShareLogging = (
        {
                'type': 'logfile',
                'filename': 'share-logfile',
                'format': "{time} {Q(remoteHost)} {username} {YN(not(rejectReas$
        },
        {
                'type': 'sql',
                'engine': 'postgres',
                'dbopts': {
                        'host': 'localhost',
'database': 'pooldb',
                        'user': 'eloipool',
                        'password': 'somethingsecret',
                },
                'statement': "insert into shares (rem_host, username, our_resul$
        },
        {
                'type': 'sql',
                'engine': 'mysql',
                'dbopts': {
                        'host': 'localhost',
                        'db': 'eloidb',
                        'user': 'dbuser',
                        'password': 'dbadmin',
                },
                'statement': "insert into shares (rem_host, username, our_resul$
        },
{
                'type': 'sql',
                'engine': 'sqlite',
                'dbopts': {
                        'database': 'share.db',
                },
                'statement': "insert into shares (remoteHost, username, rejectR$
        },
)

What is wrong here?

[tomaszsz]

hi, i got this errors in log,

Code:

2012-12-27 20:11:24,359 JSONRPCHandler  ERROR   Traceback (most recent call last):
  File "/xxxxxxxxxxxxxxx/jsonrpcserver.py", line 264, in handle_request
    return self.doJSON(data, self.path[:3] == b'/LP')
  File "/xxxxxxxxxxxxxxx/jsonrpcserver.py", line 223, in doJSON
    data = data.decode('utf8')
  File "/usr/lib/python3.1/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 213-215: invalid data

Code:

2012-12-27 21:18:40,371 JSONRPCHandler  ERROR   Error during JSON-RPC call: doJSON_submitblock['02000000579f07f9c10be136a4ab439144d1ad74b2e21e512xxxxxxxxxxxxxxxxxxxxxxxxxxH', {}]
Traceback (most recent call last):
  File "/xxxxxxxxxxxxxxxxxx/jsonrpcserver.py", line 200, in _doJSON_i
    rv = getattr(self, method)(*params)
  File "/xxxxxxxxxxxxxxxxxx/jsonrpc_getblocktemplate.py", line 85, in doJSON_submitblock
    data = bytes.fromhex(data)
ValueError: non-hexadecimal number found in fromhex() arg at position 166

is there any solution  ?
i have last version from December 16 2012,

is it just wrog data from clinet ? or i have just lost 25 BTC ?
is there ANY support or manual  to this pool software ? ?

[Luke-Jr]

Those are wrong data from client.

[tomaszsz]

Those are wrong data from client.

thanks for very fast answer
is it danger (someone try to mess (steal 25 BTC? fake shares ?) , or DoS pool ? )  ?or just ignore this messages ?

which database should I use ? (mysql is useless... exception, disconnections etc.) which version sql lib should i use ? which do you prefer on Your computer ?

[Luke-Jr]

Those are wrong data from client.

thanks for very fast answer
is it danger (someone try to mess (steal 25 BTC? fake shares ?) , or DoS pool ? )  ?or just ignore this messages ?

Nah, I just ignore them. The error is caught and ignored in Eloipool, but logged just in case the admin cares (like when I'm debugging it or BFGMiner )

whish database should I use ? (mysql is useless... exception, disconnections etc.) whish version sql lib should i use ? which do you prefer on jour computer ?

I use sqlite+flatfile for testing, and Eligius uses Postgres. I think EclipseMC figured out the MySQL mess...

[tomaszsz]

is it possible to use PPS reward shares count to use with  GBT/stratum/dyntarget ? does GBT/stratum/dyntarget/proxy affects shares  count ?
i want to use eloipool with PPS reward method (count shares in database) is it correct  ? or wrong method with eloipool and  GBT/stratum/dyntarget/proxy enabled ?