[ANN] Stratum mining protocol - ASIC ready

[antirack]

I've also had that "raise custom_exceptions.TransportException("Not connected")" like lenny before when the pool unexpectedly went down. It would not be able to reconnect when the pool came back up again, at least not every time. I had to manually restart when I run out of patience. This is something you may want to test for the next version.

[1714142644]

1714142644

[mtminer]

Slush,

Is it possible to set a limit on how high the share difficulty increases in the proxy?

I have had a few slower miners not able to find a share in 10 minutes. Happens about once or twice in a 24 hour period.

[slush]

Lenny, antirack, are you both using Windows?

That "Not connected" is standard error, although printed out as a python exception. Proxy then closes Stratum connections to miners and reject getwork requests with JSON-RPC error message. That's why there's no "data" field in your miner, but it is OK; miner should handle this and switch to backup pool.

As I tested the proxy on my machines, it always recovered from connection failures, but I have feeling there's some bug related to Windows :-/.

[slush]

Is it possible to set a limit on how high the share difficulty increases in the proxy?

No, and it has no sense, because submitting shares below difficulty requested by the pool will lead to rejected share. If your pool is using too strict vardiff algorithm, just ask pool operator to somehow set your difficulty a bit lower; some pools allow this.

I have had a few slower miners not able to find a share in 10 minutes. Happens about once or twice in a 24 hour period.

What's your hashrate? And what's difficulty requested by the pool?

[lenny_]

Lenny, antirack, are you both using Windows?

That "Not connected" is standard error, although printed out as a python exception. Proxy then closes Stratum connections to miners and reject getwork requests with JSON-RPC error message. That's why there's no "data" field in your miner, but it is OK; miner should handle this and switch to backup pool.

As I tested the proxy on my machines, it always recovered from connection failures, but I have feeling there's some bug related to Windows :-/.

Unfortunatelly it's not Windows.

Code:

Linux 2.6.32-5-amd64 x86_64 GNU/Linux
Description:    Debian GNU/Linux 6.0.6 (squeeze)
Python 2.6.6
Current stratum-mining-proxy version:
commit 27e8dce29b82c300a66dc28017dedad1b06e3519

[mtminer]

Is it possible to set a limit on how high the share difficulty increases in the proxy?

No, and it has no sense, because submitting shares below difficulty requested by the pool will lead to rejected share. If your pool is using too strict vardiff algorithm, just ask pool operator to somehow set your difficulty a bit lower; some pools allow this.

I have had a few slower miners not able to find a share in 10 minutes. Happens about once or twice in a 24 hour period.

What's your hashrate? And what's difficulty requested by the pool?

~ 29 g/hash. Pools is requesting diff 16 shares.

On the server side can difficulty be a whole number or does it need be a power of 2?

I will bug the pool operator thanks for you help.

Mtminer

[slush]

It doesn't need to be a power of 2, but some pools implement it in  this way.

[gr0bi42]

Hi Slush,

here's a patch I've made against Stratum-Mining-Proxy 1.3.0.

Changes:

• displaying pool and share diff similar like cgminer does
• reducing logging stuff

Then I submit a share, I expect the pool will accept it. Therefore the accepted share log is redundant info.

Here's the output using my patch:

Code:

2013-01-08 17:11:36,686 INFO proxy jobs.submit # Submitting share cc48b0bc diff 3/2
2013-01-08 17:11:38,198 INFO proxy jobs.submit # Submitting share 97d627b5 diff 19/2
2013-01-08 17:11:39,335 INFO proxy client_service.handle_event # New job 7ba2 for prevhash fce0b8c2, clean_jobs=False
2013-01-08 17:11:40,469 INFO proxy jobs.submit # Submitting share 420bc001 diff 3/2
2013-01-08 17:11:40,856 INFO proxy jobs.submit # Submitting share 82517ba6 diff 36/2
2013-01-08 17:11:41,105 INFO proxy jobs.submit # Submitting share 55f1c8b9 diff 2/2
2013-01-08 17:11:42,227 INFO proxy jobs.submit # Submitting share 45be6ce3 diff 4/2
2013-01-08 17:11:43,356 INFO proxy jobs.submit # Submitting share 8280834b diff 5/2
2013-01-08 17:11:44,854 INFO proxy jobs.submit # Submitting share 17f85864 diff 3/2
2013-01-08 17:11:44,897 INFO proxy jobs.submit # Submitting share b2f7eec6 diff 9/2
2013-01-08 17:11:44,939 INFO proxy jobs.submit # Submitting share 0bf1a7ce diff 3/2
2013-01-08 17:11:46,015 INFO proxy jobs.submit # Submitting share 4e378813 diff 3/2
2013-01-08 17:11:46,281 INFO proxy jobs.submit # Submitting share 7292f80f diff 2/2
2013-01-08 17:11:46,608 INFO proxy jobs.submit # Submitting share 688891d3 diff 2/2
2013-01-08 17:11:47,233 INFO proxy jobs.submit # Submitting share e943f6c9 diff 3/2
2013-01-08 17:11:47,517 INFO proxy jobs.submit # Submitting share 6ce77628 diff 4/2
2013-01-08 17:11:47,737 INFO proxy jobs.submit # Submitting share dfb7d8dd diff 4/2
2013-01-08 17:11:48,701 INFO proxy jobs.submit # Submitting share e2f03289 diff 17/2
2013-01-08 17:11:48,735 INFO proxy jobs.submit # Submitting share d85a653b diff 17/2
2013-01-08 17:11:49,017 INFO proxy jobs.submit # Submitting share 7b7e3ea0 diff 739/2
2013-01-08 17:11:50,984 INFO proxy jobs.submit # Submitting share f8190c47 diff 2/2
2013-01-08 17:11:51,077 INFO proxy jobs.submit # Submitting share 0d0b3a53 diff 3/2

Here's the patch:

Code:

diff -Naur stratum-mining-proxy-master//mining_libs/getwork_listener.py stratum-1.3.0//mining_libs/getwork_listener.py
--- stratum-mining-proxy-master//mining_libs/getwork_listener.py        2012-12-13 18:52:36.000000000 +0100
+++ stratum-1.3.0//mining_libs/getwork_listener.py      2012-12-19 18:48:13.894302270 +0100
@@ -36,7 +36,7 @@
     def _on_submit(self, result, request, msg_id, blockheader, worker_name, start_time):
         response_time = (time.time() - start_time) * 1000
         if result == True:
-            log.warning("[%dms] Share from '%s' accepted, diff %d" % (response_time, worker_name, self.job_registry.difficulty))
+            log.debug("[%dms] Share from '%s' accepted, diff %d" % (response_time, worker_name, self.job_registry.difficulty))
         else:
             log.warning("[%dms] Share from '%s' REJECTED" % (response_time, worker_name))

@@ -81,7 +81,7 @@
             if 'params' not in data or not len(data['params']):

                 # getwork request
-                log.info("Worker '%s' asks for new work" % worker_name)
+                log.debug("Worker '%s' asks for new work" % worker_name)
                 extensions = request.getHeader('x-mining-extensions')
                 no_midstate =  extensions and 'midstate' in extensions
                 request.write(self.json_response(data.get('id', 0), self.job_registry.getwork(no_midstate=no_midstate)))
diff -Naur stratum-mining-proxy-master//mining_libs/jobs.py stratum-1.3.0//mining_libs/jobs.py
--- stratum-mining-proxy-master//mining_libs/jobs.py    2012-12-13 18:52:36.000000000 +0100
+++ stratum-1.3.0//mining_libs/jobs.py  2012-12-19 18:59:29.825631912 +0100
@@ -92,6 +92,7 @@
         self.target_hex = ''
         self.difficulty = 1
         self.set_difficulty(1)
+        self.target1 = self.target
         self.target1_hex = self.target_hex

         # Relation between merkle and job
@@ -223,14 +224,17 @@
         rev = ''.join([ header_bin[i*4:i*4+4][::-1] for i in range(0, 20) ])
         hash_bin = utils.doublesha(rev)
         block_hash = ''.join([ hash_bin[i*4:i*4+4][::-1] for i in range(0, 8) ])
+

         #log.info('!!! %s' % header[:160])
-        log.info("Submitting %s" % utils.format_hash(binascii.hexlify(block_hash)))
-        
-        if utils.uint256_from_str(hash_bin) > self.target:
+        difs = utils.uint256_from_str(hash_bin)
+        if difs > self.target:
             log.debug("Share is below expected target")
             return True
-        
+ 
+        share = utils.format_hash(binascii.hexlify(block_hash))
+        log.info("Submitting share %s diff %d/%d" % (share, int(self.target1 / difs), self.difficulty))
+
         # 2. Lookup for job and extranonce used for creating given block header
         try:
             (job, extranonce2) = self.get_job_from_header(header)

[kano]

...
Then I submit a share, I expect the pool will accept it. Therefore the accepted share log is redundant info.
...

Nope.
The share can be rejected - it's easy to see - run cgminer on a stratum pool and look for rejects.
Those rejects will be the same for the proxy, except the proxy will make timing rejects more often.

[gr0bi42]

...
Then I submit a share, I expect the pool will accept it. Therefore the accepted share log is redundant info.
...

Nope.
The share can be rejected - it's easy to see - run cgminer on a stratum pool and look for rejects.
Those rejects will be the same for the proxy, except the proxy will make timing rejects more often.

Sure it can be rejected. And in this case a log is made about the reject share.

I was talking about the accepted shares, which are > 99% for stratum.
I don't need the info 1.) Submitting share... and 2.) Share accepted....

BTW, the accepted share log is still there. It's only a debug now.

Here a log output with some rejected shares:

Code:

2013-01-08 16:52:19,879 INFO proxy jobs.submit # Submitting share 444d3d05 diff 2/2
2013-01-08 16:52:20,611 INFO proxy jobs.submit # Submitting share b8da8bee diff 8/2
2013-01-08 16:52:20,626 INFO proxy jobs.submit # Submitting share ae1b8987 diff 4/2
2013-01-08 16:52:21,531 INFO proxy client_service.handle_event # New job 7b7a for prevhash fce0b8c2, clean_jobs=True
2013-01-08 16:52:21,537 INFO proxy getwork_listener._on_lp_broadcast # LP broadcast for worker 'xxx'
2013-01-08 16:52:21,542 WARNING proxy getwork_listener.render_POST # Worker 'xxx' subscribed for LP
2013-01-08 16:52:21,650 INFO proxy jobs.submit # Submitting share 0aaf5e54 diff 9/2
2013-01-08 16:52:21,650 INFO proxy jobs.submit # Job not found
2013-01-08 16:52:21,650 WARNING proxy getwork_listener._on_submit # [0ms] Share from 'xxx' REJECTED
2013-01-08 16:52:22,130 INFO proxy jobs.submit # Submitting share 651f8694 diff 4/2
2013-01-08 16:52:25,340 INFO proxy jobs.submit # Submitting share c82ad212 diff 4/2
2013-01-08 16:52:26,197 INFO proxy client_service.handle_event # New job 7b7b for prevhash fce0b8c2, clean_jobs=False
2013-01-08 16:52:26,363 INFO proxy jobs.submit # Submitting share d4f3d527 diff 3/2
2013-01-08 16:52:26,579 INFO proxy jobs.submit # Submitting share 9bf6352b diff 7/2
2013-01-08 16:52:26,630 INFO proxy jobs.submit # Submitting share 320cd927 diff 2/2
2013-01-08 16:52:26,880 INFO proxy jobs.submit # Submitting share a5d9d0a8 diff 2/2
2013-01-08 16:52:28,020 INFO proxy jobs.submit # Submitting share f69333c7 diff 2/2
2013-01-08 16:52:29,133 INFO proxy jobs.submit # Submitting share 32194c07 diff 5/2

[martychubbs]

can the stratum mining proxy be configured to connect a stratum miner and proxy the work to a getwork pool/server/proxy?   

[antirack]

I'll have to take a look at the mysql bit. I have a guess but I'll have to confirm it.

the update_submodules script just does a git checkout my (very slightly modified) stratum-proxy into the externals directory, github has been up and down the last couple days though, so it's kinda at the mercy of github

Just a quick question, I noticed you updated the code on github over Christmas, does this include the fix for MySQL?

Something unrelated. I currently start bitcoind, pool and proxy via /etc/rc.local on reboots, but it doesn't seem to work (maybe not fast enough?).

Code:

-> bitcoind
sleep 10
-> pool
sleep 5
-> proxy

Does anybody have hints how to optimize this on Ubuntu?

[kano]

...

Code:

-> bitcoind
sleep 10
-> pool
sleep 5
-> proxy

Does anybody have hints how to optimize this on Ubuntu?

My suggestion would be to have a step after starting bitcoind that tries to do a 'getinfo' from bitcoind
Not sure if that would need to be independent code from bitcoind to avoid start issues running multiple bitcoind's
(e.g. a tiny php script that uses curl)
Thus, until bitcoind is actually listening, there's no point going to the next step anyway, and that can sometimes take a while
(or longer if the bitcoind has been off for a while)
The other issue is that until it has caught up to the current block, any work you do is wasted ... I'm not sure if getinfo delays until that ... or how to tell it has reached the current network block.

[antirack]

Thanks kano, I have modified my bitcoind wrapper script and removed the sleep from /etc/rc.local.
getinfo doesn't seem to delay until it caught up to the latest blocks, I've just tried it on a machine that's been long behind. But on the pool server it shouldn't be very far behind so no issues here (reboots are a rarity). I'll see if that works on the next reboot. I'll still give it a few seconds after starting the pool before I move on to starting th proxy.

Code:

#!/bin/bash

/usr/bin/bitcoind -daemon -blocknotify="/home/user/stratum-mining/scripts/blocknotify.sh --password SECRET --host localhost --port 3333"

while true
do
        getinfo="$(/usr/bin/bitcoind getinfo 2>&1)"

        NOW=$(date +"%Y-%m-%d %H:%M:%S")
        if [[ "$getinfo" == *"couldn't connect to server"* ]]
        then
                echo "$NOW: Couldn't connect to Bitcoin Daemon yet. Sleeping 5 seconds."
                sleep 5
        else
                echo "$NOW: Bitcoin Daemon seems up and running."
                exit;
        fi
done

[slush]

here's a patch I've made against Stratum-Mining-Proxy 1.3.0.

Thanks for the patch, although I'm slightly against accepting, because the information when the share was accepted (and how long it took) is somewhat more important and interesting than when it has been sent. So if we make output less verbose, I think marking "Submitting share" as debug output instead of "Share accepted ..." has more sense.

[gr0bi42]

here's a patch I've made against Stratum-Mining-Proxy 1.3.0.

Thanks for the patch, although I'm slightly against accepting, because the information when the share was accepted (and how long it took) is somewhat more important and interesting than when it has been sent. So if we make output less verbose, I think marking "Submitting share" as debug output instead of "Share accepted ..." has more sense.

Yes, the "Share accepted" info makes more sense. My primary goal was to have the "share diff"/"pool diff" info available and it was very easy to implement in submit().

[martychubbs]

What is causing this error:

Code:

Unhandled error in Deferred:
Unhandled Error
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/Twisted-12.3.0-py2.7-linux-x86_64.egg/twisted/application/service.py", line 405, in loadApplication
    application = sob.loadValueFromFile(filename, 'application', passphrase)
  File "/usr/local/lib/python2.7/dist-packages/Twisted-12.3.0-py2.7-linux-x86_64.egg/twisted/persisted/sob.py", line 210, in loadValueFromFile
    exec fileObj in d, d
  File "launcher.tac", line 36, in <module>
    mining.setup(on_startup)
  File "/usr/local/lib/python2.7/dist-packages/Twisted-12.3.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 1214, in unwindGenerator
    return _inlineCallbacks(None, gen, Deferred())
--- <exception caught here> ---
  File "/usr/local/lib/python2.7/dist-packages/Twisted-12.3.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 1071, in _inlineCallbacks
    result = g.send(result)
  File "/opt/stratum-mining/mining/__init__.py", line 30, in setup
    from lib.block_template import BlockTemplate
  File "/opt/stratum-mining/lib/block_template.py", line 7, in <module>
    import halfnode
  File "/opt/stratum-mining/lib/halfnode.py", line 13, in <module>
    from Crypto.Hash import SHA256
exceptions.ImportError: No module named Crypto.Hash

[slush]

What is causing this error:

You're missing Crypto package. If I remember well, it is python-crypto in Debian/Ubuntu.

[martychubbs]

thanks, that fixed it.  

BTW, I  am using  git://github.com/generalfault/stratum-mining-proxy.git

located ./stratum-mining/externals/stratum-mining-proxy

Code:

GW_ENABLE = True		# Enable the Proxy (If enabled you MUST run update_submodules)
GW_PORT = 8331			# Getwork Proxy Port
GW_DISABLE_MIDSTATE = False	# Disable midstate's (Faster but breaks some clients)
GW_SEND_REAL_TARGET = False	# Propigate >1 difficulty to Clients (breaks some clients)

Midstate is detected and I ran update_submodules.

Issue I am having is: getwork is not detected on port 8331, however, stratum is detected on port 3333. (so says latest version of GUIminer)

thoughts?  BTW, great support Slush and General!
 
Update: http://pastebin.com/BwuBWVQP

Is the debug normal?  There is a connection timeout..?

Code:

2013-02-07 21:42:07-0600 [-] Unhandled error in Deferred:
2013-02-07 21:42:07-0600 [-] Unhandled Error
        Traceback (most recent call last):
        Failure: stratum.custom_exceptions.TransportException: SocketTransportClientFactory connection timed out
 
2013-02-07 21:42:07,328 DEBUG socket_transport socket_transport.clientConnectionFailed # [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.TimeoutError'>: User timeout caused connection failure.
]

and

Code:

2013-02-07 21:51:37,329 ERROR proxy client_service.on_timeout # Connection to upstream pool timed out

Upstream pool is on localhost.  That is the stratum mining server... ?

also

Webstats output on port 8889:

Code:

web.Server Traceback (most recent call last):
exceptions.TypeError: int() argument must be a string or a number, not 'NoneType'
/usr/local/lib/python2.7/dist-packages/Twisted-12.3.0-py2.7-linux-x86_64.egg/twisted/web/server.py:192 in process
191                    self._encoder = encoder
192            self.render(resrc)
193        except:
/usr/local/lib/python2.7/dist-packages/Twisted-12.3.0-py2.7-linux-x86_64.egg/twisted/web/server.py:241 in render
240        try:
241            body = resrc.render(self)
242        except UnsupportedMethod as e:
/usr/local/lib/python2.7/dist-packages/Twisted-12.3.0-py2.7-linux-x86_64.egg/twisted/web/resource.py:250 in render
249            raise UnsupportedMethod(allowedMethods)
250        return m(request)
251
/opt/stratum-mining/lib/basic_stats.py:99 in render_GET
98 r+="<tr bgcolor=\"%s\"><td>%s</td><td>%s/%s</td><td>%s/%s</td><td>%s</td><tr>"%(
99 wc,wi,format(int(wd["speed"]),"n"),wd["difficulty"],format(int(wd["total_shares"]),"n"),
100 format(int(wd["total_rejects"]),"n"),format(int(wd["total_found"]),"n"))
exceptions.TypeError: int() argument must be a string or a number, not 'NoneType'

Thanks all!

[generalfault]

Marty,

it looks like the getwork proxy cannot connect to the local stratum server. My guess is you have a firewall blocking the stratum port. If that is not the case, please post the top section (listening ports) of netstat -an