Bitcoin Forum
April 18, 2014, 09:05:42 AM *
News: Due to the OpenSSL heartbleed bug, changing your forum password is recommended.
 
   Home   Help Search Donate Login Register  
Pages: 1 2 3 [All]
  Print  
Author Topic: New pure-python CPU miner, for fun and testing  (Read 15052 times)
jgarzik
Staff
Hero Member
*****
Offline Offline

Activity: 1260


View Profile

Ignore
February 16, 2011, 10:57:19 PM
 #1

Here is a new high performance, highly optimized CPU miner:

     https://github.com/jgarzik/pyminer

This pulls a whopping 256 Khash/sec per thread on my box.

The main purpose is to demonstrate the mining algorithm, in a small amount of portable code.  And to have a bit of fun.

Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
1397811942
Hero Member
*
Offline Offline

Posts: 1397811942

View Profile Personal Message (Offline)

Ignore
1397811942
Reply with quote  #2

1397811942
Report to moderator
1397811942
Hero Member
*
Offline Offline

Posts: 1397811942

View Profile Personal Message (Offline)

Ignore
1397811942
Reply with quote  #2

1397811942
Report to moderator
1397811942
Hero Member
*
Offline Offline

Posts: 1397811942

View Profile Personal Message (Offline)

Ignore
1397811942
Reply with quote  #2

1397811942
Report to moderator
    mBitCASINOWIN BITCOINS IN OUR
24/7 LIVE DEALER CASINO

Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1397811942
Hero Member
*
Offline Offline

Posts: 1397811942

View Profile Personal Message (Offline)

Ignore
1397811942
Reply with quote  #2

1397811942
Report to moderator
1397811942
Hero Member
*
Offline Offline

Posts: 1397811942

View Profile Personal Message (Offline)

Ignore
1397811942
Reply with quote  #2

1397811942
Report to moderator
Cryptoman
Hero Member
*****
Offline Offline

Activity: 726



View Profile

Ignore
February 16, 2011, 11:05:44 PM
 #2

This is single-threaded, and pulls a whopping 43 Khash/sec on my box.

What type of system are you running?  Hamsters in a box with pencil and paper?  Grin

"A small body of determined spirits fired by an unquenchable faith in their mission can alter the course of history." --Gandhi
kseistrup
Sr. Member
****
Offline Offline

Activity: 462


corruptisima re publica plurimae leges


View Profile WWW

Ignore
February 16, 2011, 11:31:28 PM
 #3

This is single-threaded, and pulls a whopping 43 Khash/sec on my box.
Cool!   Cool

I wonder if this will run faster under PyPy (pystone.main() is more than 10 times faster on PyPy on my machine)?

Cheers,

Klaus Seistrup | BM-BbvXWtfhr5nENu4TQtnxif16SUCdNKAF
BTC: 1KLAUScYZ4LMRciPbDyENXU5H5kg1kossP
XRP: r4X2aBGRhRoQvwQ4P4WibwKB2Q9biVN4Hu
kseistrup
Sr. Member
****
Offline Offline

Activity: 462


corruptisima re publica plurimae leges


View Profile WWW

Ignore
February 16, 2011, 11:34:12 PM
 #4

I wonder if this will run faster under PyPy?
I just had to try…   Wink

Running pyminer on Python2.7 yields ~30 khash/sec, under PyPy less than 3 khash/sec.  Oh, well…

Cheers,


Klaus Seistrup | BM-BbvXWtfhr5nENu4TQtnxif16SUCdNKAF
BTC: 1KLAUScYZ4LMRciPbDyENXU5H5kg1kossP
XRP: r4X2aBGRhRoQvwQ4P4WibwKB2Q9biVN4Hu
Kiv
Full Member
***
Offline Offline

Activity: 162



View Profile

Ignore
February 17, 2011, 11:46:53 AM
 #5

This is really cool Smiley It'll definitely help me understand better how the mining works.

GUIMiner - get started easily mining Bitcoins on your GPU or CPU
Donate to support work on GUIMiner: 1MDDh2h4cAZDafgc94mr9q95dhRYcJbNQo
or YouTipIt
kseistrup
Sr. Member
****
Offline Offline

Activity: 462


corruptisima re publica plurimae leges


View Profile WWW

Ignore
February 17, 2011, 12:41:40 PM
 #6

It is computational expensive to repeatedly concatenate strings in Python.  I suggest using [].append() in the bufreverse() and wordreverse() methods:

Code:
diff --git a/pyminer.py b/pyminer.py
index cfd79a2..f972a75 100644
--- a/pyminer.py
+++ b/pyminer.py
@@ -84,21 +84,18 @@ def bytereverse(x):
  (((x) >> 8) & 0x0000ff00) | ((x) >> 24) ))
 
 def bufreverse(in_buf):
- out_words = ""
+ out_words = []
  for i in range(0, len(in_buf), 4):
  word = struct.unpack('@I', in_buf[i:i+4])[0]
- out_words += struct.pack('@I', bytereverse(word))
- return out_words
+ out_words.append(struct.pack('@I', bytereverse(word)))
+ return ''.join(out_words)
 
 def wordreverse(in_buf):
  out_words = []
  for i in range(0, len(in_buf), 4):
  out_words.append(in_buf[i:i+4])
  out_words.reverse()
- out_buf = ""
- for word in out_words:
- out_buf += word
- return out_buf
+ return ''.join(out_words)
 
 class Miner:
  def __init__(self, id):


Cheers,

Klaus Seistrup | BM-BbvXWtfhr5nENu4TQtnxif16SUCdNKAF
BTC: 1KLAUScYZ4LMRciPbDyENXU5H5kg1kossP
XRP: r4X2aBGRhRoQvwQ4P4WibwKB2Q9biVN4Hu
Cdecker
Hero Member
*****
Offline Offline

Activity: 487



View Profile WWW

Ignore
February 17, 2011, 12:48:12 PM
 #7

Great stuff, finally I can wrap my head around how mining works without having to crawl through additional stuff Smiley

Want to see what developers are chatting about? http://bitcoinstats.com/irc/bitcoin-dev/logs/
Bitcoin-OTC Rating
kseistrup
Sr. Member
****
Offline Offline

Activity: 462


corruptisima re publica plurimae leges


View Profile WWW

Ignore
February 17, 2011, 12:52:24 PM
 #8

Furthermore, you could bytereverse a 4-byte buf using it's own builtin medthod:

Code:
>>> x = 'abcd'
>>> x[::-1]
'dcba'
>>>

No need to use the pack/unpack stuff…

Cheers,

Klaus Seistrup | BM-BbvXWtfhr5nENu4TQtnxif16SUCdNKAF
BTC: 1KLAUScYZ4LMRciPbDyENXU5H5kg1kossP
XRP: r4X2aBGRhRoQvwQ4P4WibwKB2Q9biVN4Hu
epicenter
Newbie
*
Offline Offline

Activity: 10


View Profile

Ignore
February 17, 2011, 01:08:46 PM
 #9

required args?
not sure of the format for files in python


kseistrup
Sr. Member
****
Offline Offline

Activity: 462


corruptisima re publica plurimae leges


View Profile WWW

Ignore
February 17, 2011, 01:59:16 PM
 #10

required args?
not sure of the format for files in python
Pyminer takes as its single argument the name of the config file.

The config file consists of lines of the form

Code:
key = value

Keys can be host, port, rpcuser, rpcpass, threads, logdir and hashmeter.

Cheers,

Klaus Seistrup | BM-BbvXWtfhr5nENu4TQtnxif16SUCdNKAF
BTC: 1KLAUScYZ4LMRciPbDyENXU5H5kg1kossP
XRP: r4X2aBGRhRoQvwQ4P4WibwKB2Q9biVN4Hu
jgarzik
Staff
Hero Member
*****
Offline Offline

Activity: 1260


View Profile

Ignore
February 17, 2011, 06:34:42 PM
 #11

It is computational expensive to repeatedly concatenate strings in Python.  I suggest using [].append() in the bufreverse() and wordreverse() methods:

Updated the source code with your suggestions.  Thanks -- this is my second python program, so I guarantee there are other improvements that expert Python programmers can discover.

Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
kseistrup
Sr. Member
****
Offline Offline

Activity: 462


corruptisima re publica plurimae leges


View Profile WWW

Ignore
February 17, 2011, 06:55:55 PM
 #12

The way I see it, these two lines have the effect of reversing the entire data string:

Code:
hash = bufreverse(hash)
hash = wordreverse(hash)

and could be replaced by

Code:
hash = hash[::-1]

If I am right, then the following small patch will triple the efficiency (from ~32 khash/sec to more than 100 khash/sec):

Code:
--- pyminer.py 2011-02-17 19:37:34.254845344 +0100
+++ pyminer-opt.py 2011-02-17 19:47:44.794843788 +0100
@@ -76,18 +76,10 @@
  def getwork(self, data=None):
  return self.rpc('getwork', data)
 
-def uint32(x):
- return x & 0xffffffffL
-
-def bytereverse(x):
- return uint32(( ((x) << 24) | (((x) << 8) & 0x00ff0000) |
- (((x) >> 8) & 0x0000ff00) | ((x) >> 24) ))
-
 def bufreverse(in_buf):
  out_words = []
  for i in range(0, len(in_buf), 4):
- word = struct.unpack('@I', in_buf[i:i+4])[0]
- out_words.append(struct.pack('@I', bytereverse(word)))
+ out_words.append(in_buf[i:i+4][::-1])
  return ''.join(out_words)
 
 def wordreverse(in_buf):
@@ -115,10 +107,7 @@
 
  hash_o = hashlib.sha256()
  hash_o.update(hash1)
- hash = hash_o.digest()
-
- hash = bufreverse(hash)
- hash = wordreverse(hash)
+ hash = hash_o.digest()[::-1]
 
  hash_str = hash.encode('hex')
  l = long(hash_str, 16)


A few other optimizations can be made within the loop to raise the efficiency to approx. 110 khash/sec on my machine, but this will make the code less legible and is probably not worth it.

Cheers,

Klaus Seistrup | BM-BbvXWtfhr5nENu4TQtnxif16SUCdNKAF
BTC: 1KLAUScYZ4LMRciPbDyENXU5H5kg1kossP
XRP: r4X2aBGRhRoQvwQ4P4WibwKB2Q9biVN4Hu
nono2357
Newbie
*
Offline Offline

Activity: 8



View Profile

Ignore
February 17, 2011, 07:09:33 PM
 #13

More optimized functions:
Code:
def bufreverse(ch):
return "".join([ch[i:i+4][::-1] for i in range(0,len(ch),4)])

def wordreverse(ch):
return "".join([ch[i:i+4] for i in range(0,len(ch),4)][::-1])

My BTC address: 1Esr3LFxy95G1vDAA87qvYDdKXhXmpxgGh
jgarzik
Staff
Hero Member
*****
Offline Offline

Activity: 1260


View Profile

Ignore
February 17, 2011, 07:46:16 PM
 #14

I don't mind optimizing bufreverse/wordreverse, but I would like to avoid optimizing Miner.work() so heavily that it cannot be read.  That's the function that must be most-readable to other humans.  Smiley

And remember, that bufreverse/wordreverse/encode/long sequence exists solely to build a 256-bit integer.  It would probably be more optimal to simply build a 256-bit integer using a per-word loop and shifts, such as

Code:
s = 'binary string...'
r = 0L
for i in range(8):
    w32 = struct.unpack('>I', s[i:i+4])
    r = r | (w32 << (i * 32))

According to the docs, we can specify the byte order to struct.unpack()

Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
kseistrup
Sr. Member
****
Offline Offline

Activity: 462


corruptisima re publica plurimae leges


View Profile WWW

Ignore
February 17, 2011, 09:08:02 PM
 #15

And remember, that bufreverse/wordreverse/encode/long sequence exists solely to build a 256-bit integer.
If you accept the patch that basically makes “hash = hash[::-1]” instead of a call to bufreverse() followed by a call to wordreverse(), there is no longer any use for wordreverse(), plus bufreverse() is ever only used in one place.

The “hash[::-1]” (i.e., “reverse the entire buffer”) could be explained in a comment above the invocation.

Cheers,

Klaus Seistrup | BM-BbvXWtfhr5nENu4TQtnxif16SUCdNKAF
BTC: 1KLAUScYZ4LMRciPbDyENXU5H5kg1kossP
XRP: r4X2aBGRhRoQvwQ4P4WibwKB2Q9biVN4Hu
jgarzik
Staff
Hero Member
*****
Offline Offline

Activity: 1260


View Profile

Ignore
February 17, 2011, 09:52:40 PM
 #16

And remember, that bufreverse/wordreverse/encode/long sequence exists solely to build a 256-bit integer.
If you accept the patch that basically makes “hash = hash[::-1]” instead of a call to bufreverse() followed by a call to wordreverse(), there is no longer any use for wordreverse(), plus bufreverse() is ever only used in one place.

The “hash[::-1]” (i.e., “reverse the entire buffer”) could be explained in a comment above the invocation.

I'm not sure you caught my point.  If one (a) iterates over each 4-byte sub-string, (b) uses struct.unpack to perform endian conversion, and (c) uses Python integer math to build a 256-bit long, there should be no need for hash=hash[::-1] or bufreverse/wordreverse.

Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
kseistrup
Sr. Member
****
Offline Offline

Activity: 462


corruptisima re publica plurimae leges


View Profile WWW

Ignore
February 17, 2011, 11:13:23 PM
 #17

I'm not sure you caught my point.
You're right, I didn't catch your point.  I have to chew on that, though.  What we need is a binary buffer with the right endianness, not a python long integer, right?

Cheers,

Klaus Seistrup | BM-BbvXWtfhr5nENu4TQtnxif16SUCdNKAF
BTC: 1KLAUScYZ4LMRciPbDyENXU5H5kg1kossP
XRP: r4X2aBGRhRoQvwQ4P4WibwKB2Q9biVN4Hu
jgarzik
Staff
Hero Member
*****
Offline Offline

Activity: 1260


View Profile

Ignore
February 18, 2011, 12:11:01 AM
 #18

I'm not sure you caught my point.
You're right, I didn't catch your point.  I have to chew on that, though.  What we need is a binary buffer with the right endianness, not a python long integer, right?

No, we really do need a 256-integer, because that is the fundamental proof-of-work test in the bitcoin system, comparing two 256-bit integers:

     hash < target

Almost every practical miner simplifies this test to simply verify that the final, most-significant 32 bits are zero.

Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
jgarzik
Staff
Hero Member
*****
Offline Offline

Activity: 1260


View Profile

Ignore
February 18, 2011, 12:27:15 AM
 #19


Ok, implemented the H==0 test shortcut (hey, everyone uses, might as well document it) and pyminer.py now gets 250 Khash/sec for one thread, on my box.


Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
kseistrup
Sr. Member
****
Offline Offline

Activity: 462


corruptisima re publica plurimae leges


View Profile WWW

Ignore
February 18, 2011, 08:58:04 AM
 #20

Ok, implemented the H==0 test shortcut (hey, everyone uses, might as well document it) and pyminer.py now gets 250 Khash/sec for one thread, on my box.
That's cool!   Cool

If you drop the unpack stuff and compare the last 4 bytes directly you can gain another 10 khash/sec or so:

Code:
    if hash[-4:] != '\0\0\0\0':
        continue

Cheers,

Klaus Seistrup | BM-BbvXWtfhr5nENu4TQtnxif16SUCdNKAF
BTC: 1KLAUScYZ4LMRciPbDyENXU5H5kg1kossP
XRP: r4X2aBGRhRoQvwQ4P4WibwKB2Q9biVN4Hu
kseistrup
Sr. Member
****
Offline Offline

Activity: 462


corruptisima re publica plurimae leges


View Profile WWW

Ignore
February 19, 2011, 09:43:45 AM
 #21

Oh, and one more thing:

In the work loop, the line “hashes_done += 1” is effectively useless since nonce is already running from 0 to MAX_NONCE.  So at the end of the loop — either because a winning solution has been found, or because MAX_NONCE has been reached — hashes_done equals nonce+1.  Dropping the “hashes_done += 1” line gives a few extra khash/sec since this line was executed so many times.

Cheers,

Klaus Seistrup | BM-BbvXWtfhr5nENu4TQtnxif16SUCdNKAF
BTC: 1KLAUScYZ4LMRciPbDyENXU5H5kg1kossP
XRP: r4X2aBGRhRoQvwQ4P4WibwKB2Q9biVN4Hu
kseistrup
Sr. Member
****
Offline Offline

Activity: 462


corruptisima re publica plurimae leges


View Profile WWW

Ignore
February 19, 2011, 10:10:33 AM
 #22

I wonder if this is a bug:

Code:
        # hash final 4b, the nonce value
        hash1_o = hash1_precalc_o
        hash1_o.update(nonce_bin)
        hash1 = hash1_o.digest()

Do I interpret your meaning right when I say that the code — for each value of nonce — intends to calculate the SHA256 digest of the precalculated value (intended to stay constant throughout the loop) updated with the nonce?  If so, that is not what the code does.  Please see this simplified example:

Code:
>>> from hashlib import sha256
>>> pre = sha256('abc')
>>> post = pre
>>> post.update('xyz')
>>> pre.digest() == post.digest()
True
>>>

So pre and post (like hash1_precalc_o and hash1_o) both points to the same object, and so I guess we should really write:

Code:
        # hash final 4b, the nonce value
        hash1_o = hashlib.sha256(blk_hdr + nonce_bin)
        hash1 = hash1_o.digest()

or even

Code:
        # hash final 4b, the nonce value
        hash1 = hashlib.sha256(blk_hdr + nonce_bin).digest()

Or did I misunderstand the whole concept?

Cheers,

Klaus Seistrup | BM-BbvXWtfhr5nENu4TQtnxif16SUCdNKAF
BTC: 1KLAUScYZ4LMRciPbDyENXU5H5kg1kossP
XRP: r4X2aBGRhRoQvwQ4P4WibwKB2Q9biVN4Hu
jgarzik
Staff
Hero Member
*****
Offline Offline

Activity: 1260


View Profile

Ignore
February 19, 2011, 05:44:36 PM
 #23

Do I interpret your meaning right when I say that the code — for each value of nonce — intends to calculate the SHA256 digest of the precalculated value (intended to stay constant throughout the loop) updated with the nonce?  If so, that is not what the code does.  Please see this simplified example:

Well, that is disappointing.  If true, yes, that is a bug.  Updated.

Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
kseistrup
Sr. Member
****
Offline Offline

Activity: 462


corruptisima re publica plurimae leges


View Profile WWW

Ignore
February 19, 2011, 05:52:54 PM
 #24

Updated.
Do you happen to have a git repository of the code?

Cheers,

Klaus Seistrup | BM-BbvXWtfhr5nENu4TQtnxif16SUCdNKAF
BTC: 1KLAUScYZ4LMRciPbDyENXU5H5kg1kossP
XRP: r4X2aBGRhRoQvwQ4P4WibwKB2Q9biVN4Hu
jgarzik
Staff
Hero Member
*****
Offline Offline

Activity: 1260


View Profile

Ignore
February 19, 2011, 05:54:04 PM
 #25

Updated.
Do you happen to have a git repository of the code?

No, just the URL.

Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
kseistrup
Sr. Member
****
Offline Offline

Activity: 462


corruptisima re publica plurimae leges


View Profile WWW

Ignore
February 19, 2011, 06:26:01 PM
 #26

No, just the URL.
Okay, no pull-request for you, then.  Wink

Please consider the following patch for inclusion:

Code:
--- pyminer.py 2011-02-19 19:05:29.794843964 +0100
+++ pyminer-opt.py 2011-02-19 19:20:06.136093826 +0100
@@ -113,28 +113,22 @@
  targetbin_str = targetbin.encode('hex')
  target = long(targetbin_str, 16)
 
- hashes_done = 1
- for nonce in range(MAX_NONCE):
+ for nonce in xrange(MAX_NONCE):
 
  # encode 32-bit nonce value
  nonce_bin = struct.pack("<I", nonce)
 
  # hash final 4b, the nonce value
- hash1_o = hashlib.sha256()
- hash1_o.update(blk_hdr)
+ hash1_o = hashlib.sha256(blk_hdr)
  hash1_o.update(nonce_bin)
  hash1 = hash1_o.digest()
 
  # sha256 hash of sha256 hash
- hash_o = hashlib.sha256()
- hash_o.update(hash1)
+ hash_o = hashlib.sha256(hash1)
  hash = hash_o.digest()
 
- hashes_done += 1
-
  # quick test for winning solution: high 32 bits zero?
- H = struct.unpack('<I', hash[28:32])
- if H:
+ if hash[-4:] != '\0\0\0\0':
  continue
 
  # convert binary hash to 256-bit Python long
@@ -147,12 +141,12 @@
  # proof-of-work test:  hash < target
  if l < target:
  print time.asctime(), "PROOF-OF-WORK found: %064x" % (l,)
- return (hashes_done,
+ return (nonce+1,
  static_data[:76] + nonce_bin)
  else:
  print time.asctime(), "PROOF-OF-WORK false positive %064x" % (l,)
 
- return (hashes_done, None)
+ return (nonce+1, None)
 
  def iterate(self, rpc):
  work = rpc.getwork()

The patch eliminates 2 calls to hashlib.sha256(), 1 call to struct.unpack() and 1 one variable increment per work loop.

Cheers,

Klaus Seistrup | BM-BbvXWtfhr5nENu4TQtnxif16SUCdNKAF
BTC: 1KLAUScYZ4LMRciPbDyENXU5H5kg1kossP
XRP: r4X2aBGRhRoQvwQ4P4WibwKB2Q9biVN4Hu
mail2345
Newbie
*
Offline Offline

Activity: 16


View Profile

Ignore
February 19, 2011, 06:54:30 PM
 #27

Psyco should speed things up.

BTC Address: 12TJbeJ2aCzCppJ6m1yYoPUdp9EHWRTQNv
kseistrup
Sr. Member
****
Offline Offline

Activity: 462


corruptisima re publica plurimae leges


View Profile WWW

Ignore
February 19, 2011, 07:26:32 PM
 #28

Psyco should speed things up.
Psyco only runs on “obsolete” versions of Python (and it doesn't work on any 64-bit systems at all).

Cheers,

Klaus Seistrup | BM-BbvXWtfhr5nENu4TQtnxif16SUCdNKAF
BTC: 1KLAUScYZ4LMRciPbDyENXU5H5kg1kossP
XRP: r4X2aBGRhRoQvwQ4P4WibwKB2Q9biVN4Hu
bitjet
Hero Member
*****
Offline Offline

Activity: 556


View Profile

Ignore
February 19, 2011, 07:43:22 PM
 #29

Are you guys getting near triple efficiency with this vs over poclbm?
Kiv
Full Member
***
Offline Offline

Activity: 162



View Profile

Ignore
February 27, 2011, 07:53:42 PM
 #30

I'm trying to run this and getting an exception:

Code:
Process Process-1:
Traceback (most recent call last):
  File "C:\Python26\lib\multiprocessing\process.py", line 232, in _bootstrap
    self.run()
  File "C:\Python26\lib\multiprocessing\process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Chris\workspace\pyminer\pyminer.py", line 197, in miner_thread
    miner.loop()
  File "C:\Users\Chris\workspace\pyminer\pyminer.py", line 187, in loop
    rpc = BitcoinRPC(settings['host'], settings['port'],
KeyError: 'host'
1 mining threads started
Sun Feb 27 15:51:18 2011 Miner Starts - 127.0.0.1:8332
Sun Feb 27 15:51:18 2011 Miner Stops - 127.0.0.1:8332

It seems that the settings dictionary is empty, because when we're using separate processes the children don't have access to the parent's variables. Probably the settings should be passed in as constructor arguments. I might put this up on GitHub and hack on it a bit if you don't mind Smiley

GUIMiner - get started easily mining Bitcoins on your GPU or CPU
Donate to support work on GUIMiner: 1MDDh2h4cAZDafgc94mr9q95dhRYcJbNQo
or YouTipIt
Kiv
Full Member
***
Offline Offline

Activity: 162



View Profile

Ignore
February 27, 2011, 07:57:18 PM
 #31

Are you guys getting near triple efficiency with this vs over poclbm?

Definitely not, I get about 900 KHash per core on poclbm and only 182 KHash per core on pure-python.

GUIMiner - get started easily mining Bitcoins on your GPU or CPU
Donate to support work on GUIMiner: 1MDDh2h4cAZDafgc94mr9q95dhRYcJbNQo
or YouTipIt
kseistrup
Sr. Member
****
Offline Offline

Activity: 462


corruptisima re publica plurimae leges


View Profile WWW

Ignore
February 27, 2011, 07:57:49 PM
 #32

Code:
  File "C:\Users\Chris\workspace\pyminer\pyminer.py", line 187, in loop
    rpc = BitcoinRPC(settings['host'], settings['port'],
KeyError: 'host'
Pyminer takes as its single argument the name of the config file.  Please see message #10 in this thread.

Cheers,

Klaus Seistrup | BM-BbvXWtfhr5nENu4TQtnxif16SUCdNKAF
BTC: 1KLAUScYZ4LMRciPbDyENXU5H5kg1kossP
XRP: r4X2aBGRhRoQvwQ4P4WibwKB2Q9biVN4Hu
Kiv
Full Member
***
Offline Offline

Activity: 162



View Profile

Ignore
February 27, 2011, 08:03:21 PM
 #33

Thank you, but I did supply the argument. In fact the program will not start without the config file. My config file is the right format as well and has "host=127.0.0.1" in it.

I suspect that the program worked fine at one point and I have a more recent version with a bug in it.


Code:
  File "C:\Users\Chris\workspace\pyminer\pyminer.py", line 187, in loop
    rpc = BitcoinRPC(settings['host'], settings['port'],
KeyError: 'host'
Pyminer takes as its single argument the name of the config file.  Please see message #10 in this thread.

Cheers,

GUIMiner - get started easily mining Bitcoins on your GPU or CPU
Donate to support work on GUIMiner: 1MDDh2h4cAZDafgc94mr9q95dhRYcJbNQo
or YouTipIt
kseistrup
Sr. Member
****
Offline Offline

Activity: 462


corruptisima re publica plurimae leges


View Profile WWW

Ignore
February 27, 2011, 08:13:23 PM
 #34

I suspect that the program worked fine at one point and I have a more recent version with a bug in it.
The current version runs fine with this config file (username and password edited for obvious reasons):

Code:
host = mining.bitcoin.cz
port = 8332
rpcuser = myRPCUser
rpcpass = myRPCPass
logdir = /tmp
threads = 1
hashmeter = 1

Cheers,

Klaus Seistrup | BM-BbvXWtfhr5nENu4TQtnxif16SUCdNKAF
BTC: 1KLAUScYZ4LMRciPbDyENXU5H5kg1kossP
XRP: r4X2aBGRhRoQvwQ4P4WibwKB2Q9biVN4Hu
Kiv
Full Member
***
Offline Offline

Activity: 162



View Profile

Ignore
February 27, 2011, 08:26:13 PM
 #35

This is probably a Windows-only bug. From the multiprocessing docs:


Since Windows lacks os.fork() it has a few extra restrictions:
Global variables

Bear in mind that if code run in a child process tries to access a global variable, then the value it sees (if any) may not be the same as the value in the parent process at the time that Process.start() was called.

If everyone else is running on Linux but I am running on Windows, this explains why it would work for others but not for me.

GUIMiner - get started easily mining Bitcoins on your GPU or CPU
Donate to support work on GUIMiner: 1MDDh2h4cAZDafgc94mr9q95dhRYcJbNQo
or YouTipIt
kseistrup
Sr. Member
****
Offline Offline

Activity: 462


corruptisima re publica plurimae leges


View Profile WWW

Ignore
February 27, 2011, 08:35:00 PM
 #36

If everyone else is running on Linux but I am running on Windows, this explains why it would work for others but not for me.
It sure does.

Cheers,

Klaus Seistrup | BM-BbvXWtfhr5nENu4TQtnxif16SUCdNKAF
BTC: 1KLAUScYZ4LMRciPbDyENXU5H5kg1kossP
XRP: r4X2aBGRhRoQvwQ4P4WibwKB2Q9biVN4Hu
rasputin
Newbie
*
Offline Offline

Activity: 15


View Profile

Ignore
March 23, 2011, 12:11:25 PM
 #37

I wonder if this is a bug:

Code:
       # hash final 4b, the nonce value
        hash1_o = hash1_precalc_o
        hash1_o.update(nonce_bin)
        hash1 = hash1_o.digest()

Do I interpret your meaning right when I say that the code — for each value of nonce — intends to calculate the SHA256 digest of the precalculated value (intended to stay constant throughout the loop) updated with the nonce?  If so, that is not what the code does.  Please see this simplified example:

Code:
>>> from hashlib import sha256
>>> pre = sha256('abc')
>>> post = pre
>>> post.update('xyz')
>>> pre.digest() == post.digest()
True
>>>

So pre and post (like hash1_precalc_o and hash1_o) both points to the same object

Instead of assignment, you'd use hashlib's copy() method to duplicate the hash state.

This code should work as intended:
Code:
# hash final 4b, the nonce value
hash1_o = hash1_precalc_o.copy()
hash1_o.update(nonce_bin)
hash1 = hash1_o.digest()
rasputin
Newbie
*
Offline Offline

Activity: 15


View Profile

Ignore
March 23, 2011, 12:29:07 PM
 #38

I'm not sure you caught my point.
You're right, I didn't catch your point.  I have to chew on that, though.  What we need is a binary buffer with the right endianness, not a python long integer, right?

No, we really do need a 256-integer, because that is the fundamental proof-of-work test in the bitcoin system, comparing two 256-bit integers:

     hash < target

You may just as well compare the two arrays of binary data:
Code:
>>> a = "aaaa".decode('hex')
>>> b = "bbbb".decode('hex')
>>> a > b
False
>>> b > a
True

Got to love high level languages.

Considered putting this on github as well, btw? Being smaller and higher level, I find it a lot easier to grasp than the C miner. Too bad about all the byte order manipulation, but I guess that can't be helped.
TehRainbowGuy
Newbie
*
Offline Offline

Activity: 8



View Profile

Ignore
April 04, 2011, 08:40:58 PM
 #39

I am loving the idea of this miner, much simpler than all of the others.

Just one question, any chance of a git repo?
jgarzik
Staff
Hero Member
*****
Offline Offline

Activity: 1260


View Profile

Ignore
April 04, 2011, 08:57:57 PM
 #40

Sure, here's a git repo:  https://github.com/jgarzik/pyminer

Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
TehRainbowGuy
Newbie
*
Offline Offline

Activity: 8



View Profile

Ignore
April 04, 2011, 09:16:01 PM
 #41

Thanks!  Smiley
jgarzik
Staff
Hero Member
*****
Offline Offline

Activity: 1260


View Profile

Ignore
April 05, 2011, 07:56:40 AM
 #42

Updated the git repo with many bug fixes.  pyminer.py successfully generated a block on testnet, and a share in slush's pool.

Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
fix
Newbie
*
Offline Offline

Activity: 4


View Profile

Ignore
April 17, 2011, 04:45:06 PM
 #43

Really, really useful. Thank you very much.

But what I don't understand is the meaning of the RPC work data (data, hash1, midstate, target values) and why when a proof-of-work isn't found this isn't replied to the pool. I know, I've got some confusion in my mind.
There's some documentation about this?
dsp77
Newbie
*
Offline Offline

Activity: 8


View Profile

Ignore
June 03, 2011, 09:59:06 PM
 #44

I'm trying to run this and getting an exception:

Code:
Process Process-1:
Traceback (most recent call last):
  File "C:\Python26\lib\multiprocessing\process.py", line 232, in _bootstrap
    self.run()
  File "C:\Python26\lib\multiprocessing\process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Chris\workspace\pyminer\pyminer.py", line 197, in miner_thread
    miner.loop()
  File "C:\Users\Chris\workspace\pyminer\pyminer.py", line 187, in loop
    rpc = BitcoinRPC(settings['host'], settings['port'],
KeyError: 'host'
1 mining threads started
Sun Feb 27 15:51:18 2011 Miner Starts - 127.0.0.1:8332
Sun Feb 27 15:51:18 2011 Miner Stops - 127.0.0.1:8332

It seems that the settings dictionary is empty, because when we're using separate processes the children don't have access to the parent's variables. Probably the settings should be passed in as constructor arguments. I might put this up on GitHub and hack on it a bit if you don't mind Smiley

I did some changes to make it work under Windows:

Code:

class Miner:
  def __init__(self, id, settings=None):
    self.id = id
    self.settings=settings
    self.max_nonce = MAX_NONCE
...

  def iterate(self, rpc):
...
    self.max_nonce = long(
      (hashes_done * self.settings['scantime']) / time_diff)
    if self.max_nonce > 0xfffffffaL:
      self.max_nonce = 0xfffffffaL
...
    if self.settings['hashmeter']:
      print "HashMeter(%d): %d hashes, %.2f Khash/sec" % (
            self.id, hashes_done,
            (hashes_done / 1000.0) / time_diff)

...

  def loop(self):
    rpc = BitcoinRPC(self.settings['host'], self.settings['port'],
         self.settings['rpcuser'], self.settings['rpcpass'])
    if rpc is None:
      return

...

def miner_thread(id, settings):
  miner = Miner(id, settings)
  miner.loop()

...

  thr_list = []
  for thr_id in range(settings['threads']):
    p = Process(target=miner_thread, args=(thr_id, settings))
    p.start()
    thr_list.append(p)
    time.sleep(1)     # stagger threads


Sorry, not in a patch form.
dsp77
Newbie
*
Offline Offline

Activity: 8


View Profile

Ignore
June 03, 2011, 10:21:21 PM
 #45

...
Sorry, not in a patch form.


OK, fixed that. Here is a diff of it.



Code:
diff --git pyminer.py pyminer.py
index 051abb3..b203aa8 100755
--- pyminer.py
+++ pyminer.py
@@ -97,8 +97,9 @@ def wordreverse(in_buf):
  return ''.join(out_words)
 
 class Miner:
- def __init__(self, id):
+ def __init__(self, id, settings=None):
  self.id = id
+    self.settings=settings
  self.max_nonce = MAX_NONCE
 
  def work(self, datastr, targetstr):
@@ -181,11 +182,11 @@ class Miner:
  time_diff = time_end - time_start
 
  self.max_nonce = long(
- (hashes_done * settings['scantime']) / time_diff)
+ (hashes_done * self.settings['scantime']) / time_diff)
  if self.max_nonce > 0xfffffffaL:
  self.max_nonce = 0xfffffffaL
 
- if settings['hashmeter']:
+ if self.settings['hashmeter']:
  print "HashMeter(%d): %d hashes, %.2f Khash/sec" % (
       self.id, hashes_done,
       (hashes_done / 1000.0) / time_diff)
@@ -194,16 +195,16 @@ class Miner:
  self.submit_work(rpc, work['data'], nonce_bin)
 
  def loop(self):
- rpc = BitcoinRPC(settings['host'], settings['port'],
- settings['rpcuser'], settings['rpcpass'])
+ rpc = BitcoinRPC(self.settings['host'], self.settings['port'],
+ self.settings['rpcuser'], self.settings['rpcpass'])
  if rpc is None:
  return
 
  while True:
  self.iterate(rpc)
 
-def miner_thread(id):
- miner = Miner(id)
+def miner_thread(id, settings):
+ miner = Miner(id, settings)
  miner.loop()
 
 if __name__ == '__main__':
@@ -248,7 +249,7 @@ if __name__ == '__main__':
 
  thr_list = []
  for thr_id in range(settings['threads']):
- p = Process(target=miner_thread, args=(thr_id,))
+ p = Process(target=miner_thread, args=(thr_id, settings,))
  p.start()
  thr_list.append(p)
  time.sleep(1) # stagger threads

dsp77
Newbie
*
Offline Offline

Activity: 8


View Profile

Ignore
June 03, 2011, 10:57:19 PM
 #46

I ran the latest git-version with the patch I posted earlier under windows and linux using an account at btcguild.com and got the following error message:

Code:
G:\Dokumente und Einstellungen\anna\Eigene Dateien\pj\pyminer>python pyminer.py btcguild-config.cfg
2 mining threads started
Sat Jun 04 00:41:52 2011 Miner Starts - btcguild.com:8332
HashMeter(0): 1000000 hashes, 136.18 Khash/sec
HashMeter(1): 1000000 hashes, 136.46 Khash/sec
HashMeter(0): 8171047 hashes, 135.41 Khash/sec
Process Process-1:
Traceback (most recent call last):
  File "G:\Python27\lib\multiprocessing\process.py", line 232, in _bootstrap
    self.run()
  File "G:\Python27\lib\multiprocessing\process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "G:\Dokumente und Einstellungen\anna\Eigene Dateien\pj\pyminer\pyminer.py", line 208, in miner_thread
    miner.loop()
  File "G:\Dokumente und Einstellungen\anna\Eigene Dateien\pj\pyminer\pyminer.py", line 204, in loop
    self.iterate(rpc)
  File "G:\Dokumente und Einstellungen\anna\Eigene Dateien\pj\pyminer\pyminer.py", line 168, in iterate
    work = rpc.getwork()
  File "G:\Dokumente und Einstellungen\anna\Eigene Dateien\pj\pyminer\pyminer.py", line 76, in getwork
    return self.rpc('getwork', data)
  File "G:\Dokumente und Einstellungen\anna\Eigene Dateien\pj\pyminer\pyminer.py", line 56, in rpc
    resp = self.conn.getresponse()
  File "G:\Python27\lib\httplib.py", line 1025, in getresponse
    response.begin()
  File "G:\Python27\lib\httplib.py", line 401, in begin
    version, status, reason = self._read_status()
  File "G:\Python27\lib\httplib.py", line 365, in _read_status
    raise BadStatusLine(line)
BadStatusLine: ''
HashMeter(1): 8187772 hashes, 135.47 Khash/sec
Process Process-2:
...
Sat Jun 04 00:43:00 2011 Miner Stops - btcguild.com:8332


Running it with two threads I took out the trace back from the second process.

Anyone knows why there is an empty status line returned?

Thanks for your help.
psychocoder
Jr. Member
*
Offline Offline

Activity: 50


View Profile

Ignore
June 06, 2011, 04:26:52 PM
 #47

I also get the error if I use btcguild. I use phython 2.5.3 on linux debian lenny (is this a problem). I run The program with one thread.

Quote
Process Process-1:
Traceback (most recent call last):
  File "/var/lib/python-support/python2.5/multiprocessing/process.py", line 237, in _bootstrap
    self.run()
  File "/var/lib/python-support/python2.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "pyminer.py", line 214, in miner_thread
    miner.loop()
  File "pyminer.py", line 210, in loop
    self.iterate(rpc)
  File "pyminer.py", line 174, in iterate
    work = rpc.getwork()
  File "pyminer.py", line 83, in getwork
    return self.rpc('getwork', data)
  File "pyminer.py", line 63, in rpc
    resp = self.conn.getresponse()
  File "/usr/lib/python2.5/httplib.py", line 928, in getresponse
    response.begin()
  File "/usr/lib/python2.5/httplib.py", line 385, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.5/httplib.py", line 349, in _read_status
    raise BadStatusLine(line)
BadStatusLine

donations BTC: 1BEEBBTNwZAk9bEkLU56nR5GLWNMj98K9F
dsp77
Newbie
*
Offline Offline

Activity: 8


View Profile

Ignore
June 08, 2011, 08:05:50 AM
 #48

I also get the error if I use btcguild. I use phython 2.5.3 on linux debian lenny (is this a problem). I run The program with one thread.

...

I ran it under Ubuntu Lucid Lynx with Python 2.6.5 and under Windows with Python 2.7. In both cases I get the error.

It looks like the problem occurs when after the second request another one is started. When adding debug output I see it run in the miner.loop() the self.iterate(rpc) call twice. After it finishes the second loop and starts the 3rd call to self.iterate(rpc) the response with an empty header is returned.

Breaking with the 3rd rpc call is independent of the amount of threads. So with one thread it breaks with the 3rd call and with two threads running it breaks each thread with the 3rd call as well.  Undecided

One interesting thing to note is, that the max_nonce value goes up. I thought it would go down in consecutive runs. In the first run it is set to 1,000,000. In the second run it goes up to 4,885,595.
elmom
Newbie
*
Offline Offline

Activity: 21


View Profile

Ignore
June 08, 2011, 10:19:57 AM
 #49

Running it on pypy (recent nightly build), it leaks and fills my memory fast, any idea if the code relies too much on cpython reference counting, or should I report this to pypy devs?

Otherwise, about the slowness of pypy is propalby related to the fact that the hashing is relegated to optimized c-code, but pypy uses a pure python or non-optimized rpython implementation. Has someone done profiling of the latest code?
tcash21
Newbie
*
Offline Offline

Activity: 1


View Profile

Ignore
June 10, 2011, 05:32:02 PM
 #50

...
Sorry, not in a patch form.


OK, fixed that. Here is a diff of it.



Code:
diff --git pyminer.py pyminer.py
index 051abb3..b203aa8 100755
--- pyminer.py
+++ pyminer.py
@@ -97,8 +97,9 @@ def wordreverse(in_buf):
  return ''.join(out_words)
 
 class Miner:
- def __init__(self, id):
+ def __init__(self, id, settings=None):
  self.id = id
+    self.settings=settings
  self.max_nonce = MAX_NONCE
 
  def work(self, datastr, targetstr):
@@ -181,11 +182,11 @@ class Miner:
  time_diff = time_end - time_start
 
  self.max_nonce = long(
- (hashes_done * settings['scantime']) / time_diff)
+ (hashes_done * self.settings['scantime']) / time_diff)
  if self.max_nonce > 0xfffffffaL:
  self.max_nonce = 0xfffffffaL
 
- if settings['hashmeter']:
+ if self.settings['hashmeter']:
  print "HashMeter(%d): %d hashes, %.2f Khash/sec" % (
       self.id, hashes_done,
       (hashes_done / 1000.0) / time_diff)
@@ -194,16 +195,16 @@ class Miner:
  self.submit_work(rpc, work['data'], nonce_bin)
 
  def loop(self):
- rpc = BitcoinRPC(settings['host'], settings['port'],
- settings['rpcuser'], settings['rpcpass'])
+ rpc = BitcoinRPC(self.settings['host'], self.settings['port'],
+ self.settings['rpcuser'], self.settings['rpcpass'])
  if rpc is None:
  return
 
  while True:
  self.iterate(rpc)
 
-def miner_thread(id):
- miner = Miner(id)
+def miner_thread(id, settings):
+ miner = Miner(id, settings)
  miner.loop()
 
 if __name__ == '__main__':
@@ -248,7 +249,7 @@ if __name__ == '__main__':
 
  thr_list = []
  for thr_id in range(settings['threads']):
- p = Process(target=miner_thread, args=(thr_id,))
+ p = Process(target=miner_thread, args=(thr_id, settings,))
  p.start()
  thr_list.append(p)
  time.sleep(1) # stagger threads



I'm still getting the BadStatusLine error even after adding these fixes to the script. I'm trying to run on Ubuntu 10.04.2 LTS 64-bit.
Any ideas?
Unthinkingbit
Hero Member
*****
Offline Offline

Activity: 900



View Profile

Ignore
June 12, 2011, 12:44:05 AM
 #51

By condensing code in the nonce loop, pyminer gets around 6% more speed on my machine.  The hash part of the loop ends up being:

Code:
# hash final 4b, the nonce value
hash1_o = static_hash.copy()

# encode 32-bit nonce value
hash1_o.update(struct.pack("<I", nonce))

# sha256 hash of sha256 hash
hash = hashlib.sha256(hash1_o.digest()).digest()

Below is a diff of the changes.

Code:
--- pyminer.py 2011-06-11 14:47:44.000000000 -0700
+++ pyminer_test.py 2011-06-11 14:47:04.000000000 -0700
@@ -121,18 +121,14 @@
 
  for nonce in xrange(self.max_nonce):
 
- # encode 32-bit nonce value
- nonce_bin = struct.pack("<I", nonce)
-
  # hash final 4b, the nonce value
  hash1_o = static_hash.copy()
- hash1_o.update(nonce_bin)
- hash1 = hash1_o.digest()
+
+ # encode 32-bit nonce value
+ hash1_o.update(struct.pack("<I", nonce))
 
  # sha256 hash of sha256 hash
- hash_o = hashlib.sha256()
- hash_o.update(hash1)
- hash = hash_o.digest()
+ hash = hashlib.sha256(hash1_o.digest()).digest()
 
  # quick test for winning solution: high 32 bits zero?
  if hash[-4:] != '\0\0\0\0':

tarzen
Newbie
*
Offline Offline

Activity: 10


View Profile

Ignore
February 12, 2014, 03:41:00 AM
 #52

i'm trying this with Python 2.7 on Linux and get a an error on line ~158 at

   
Code:
return (nonce + 1, None)

Python complains that nonce has not been defined - which is true since this call is made outside of the loop

        
Code:
for nonce in xrange(self.max_nonce):

which defines nonce

I added a class variable self._nonce in the __init__ function of class Miner and use this t keep track of the nonce value.

A question:
Is it expected behaviour for the hash rate to halve on each iteration of the work function?

 *Link Removed*
Pages: 1 2 3 [All]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!