Bitcoin Forum
April 17, 2014, 09:24:17 PM *
News: Due to the OpenSSL heartbleed bug, changing your forum password is recommended.
 
   Home   Help Search Donate Login Register  
Pages: [1] 2 3  All
  Print  
Author Topic: New pure-python CPU miner, for fun and testing  (Read 15046 times)
jgarzik
Staff
Hero Member
*****
Offline Offline

Activity: 1260


View Profile

Ignore
February 16, 2011, 10:57:19 PM
 #1

Here is a new high performance, highly optimized CPU miner:

     https://github.com/jgarzik/pyminer

This pulls a whopping 256 Khash/sec per thread on my box.

The main purpose is to demonstrate the mining algorithm, in a small amount of portable code.  And to have a bit of fun.

Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
1397769857
Hero Member
*
Offline Offline

Posts: 1397769857

View Profile Personal Message (Offline)

Ignore
1397769857
Reply with quote  #2

1397769857
Report to moderator
1397769857
Hero Member
*
Offline Offline

Posts: 1397769857

View Profile Personal Message (Offline)

Ignore
1397769857
Reply with quote  #2

1397769857
Report to moderator
1397769857
Hero Member
*
Offline Offline

Posts: 1397769857

View Profile Personal Message (Offline)

Ignore
1397769857
Reply with quote  #2

1397769857
Report to moderator
    mBitCASINOWIN BITCOINS IN OUR
24/7 LIVE DEALER CASINO

Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1397769857
Hero Member
*
Offline Offline

Posts: 1397769857

View Profile Personal Message (Offline)

Ignore
1397769857
Reply with quote  #2

1397769857
Report to moderator
1397769857
Hero Member
*
Offline Offline

Posts: 1397769857

View Profile Personal Message (Offline)

Ignore
1397769857
Reply with quote  #2

1397769857
Report to moderator
1397769857
Hero Member
*
Offline Offline

Posts: 1397769857

View Profile Personal Message (Offline)

Ignore
1397769857
Reply with quote  #2

1397769857
Report to moderator
Cryptoman
Hero Member
*****
Offline Offline

Activity: 726



View Profile

Ignore
February 16, 2011, 11:05:44 PM
 #2

This is single-threaded, and pulls a whopping 43 Khash/sec on my box.

What type of system are you running?  Hamsters in a box with pencil and paper?  Grin

"A small body of determined spirits fired by an unquenchable faith in their mission can alter the course of history." --Gandhi
kseistrup
Sr. Member
****
Offline Offline

Activity: 462


corruptisima re publica plurimae leges


View Profile WWW

Ignore
February 16, 2011, 11:31:28 PM
 #3

This is single-threaded, and pulls a whopping 43 Khash/sec on my box.
Cool!   Cool

I wonder if this will run faster under PyPy (pystone.main() is more than 10 times faster on PyPy on my machine)?

Cheers,

Klaus Seistrup | BM-BbvXWtfhr5nENu4TQtnxif16SUCdNKAF
BTC: 1KLAUScYZ4LMRciPbDyENXU5H5kg1kossP
XRP: r4X2aBGRhRoQvwQ4P4WibwKB2Q9biVN4Hu
kseistrup
Sr. Member
****
Offline Offline

Activity: 462


corruptisima re publica plurimae leges


View Profile WWW

Ignore
February 16, 2011, 11:34:12 PM
 #4

I wonder if this will run faster under PyPy?
I just had to try…   Wink

Running pyminer on Python2.7 yields ~30 khash/sec, under PyPy less than 3 khash/sec.  Oh, well…

Cheers,


Klaus Seistrup | BM-BbvXWtfhr5nENu4TQtnxif16SUCdNKAF
BTC: 1KLAUScYZ4LMRciPbDyENXU5H5kg1kossP
XRP: r4X2aBGRhRoQvwQ4P4WibwKB2Q9biVN4Hu
Kiv
Full Member
***
Offline Offline

Activity: 162



View Profile

Ignore
February 17, 2011, 11:46:53 AM
 #5

This is really cool Smiley It'll definitely help me understand better how the mining works.

GUIMiner - get started easily mining Bitcoins on your GPU or CPU
Donate to support work on GUIMiner: 1MDDh2h4cAZDafgc94mr9q95dhRYcJbNQo
or YouTipIt
kseistrup
Sr. Member
****
Offline Offline

Activity: 462


corruptisima re publica plurimae leges


View Profile WWW

Ignore
February 17, 2011, 12:41:40 PM
 #6

It is computational expensive to repeatedly concatenate strings in Python.  I suggest using [].append() in the bufreverse() and wordreverse() methods:

Code:
diff --git a/pyminer.py b/pyminer.py
index cfd79a2..f972a75 100644
--- a/pyminer.py
+++ b/pyminer.py
@@ -84,21 +84,18 @@ def bytereverse(x):
  (((x) >> 8) & 0x0000ff00) | ((x) >> 24) ))
 
 def bufreverse(in_buf):
- out_words = ""
+ out_words = []
  for i in range(0, len(in_buf), 4):
  word = struct.unpack('@I', in_buf[i:i+4])[0]
- out_words += struct.pack('@I', bytereverse(word))
- return out_words
+ out_words.append(struct.pack('@I', bytereverse(word)))
+ return ''.join(out_words)
 
 def wordreverse(in_buf):
  out_words = []
  for i in range(0, len(in_buf), 4):
  out_words.append(in_buf[i:i+4])
  out_words.reverse()
- out_buf = ""
- for word in out_words:
- out_buf += word
- return out_buf
+ return ''.join(out_words)
 
 class Miner:
  def __init__(self, id):


Cheers,

Klaus Seistrup | BM-BbvXWtfhr5nENu4TQtnxif16SUCdNKAF
BTC: 1KLAUScYZ4LMRciPbDyENXU5H5kg1kossP
XRP: r4X2aBGRhRoQvwQ4P4WibwKB2Q9biVN4Hu
Cdecker
Hero Member
*****
Offline Offline

Activity: 487



View Profile WWW

Ignore
February 17, 2011, 12:48:12 PM
 #7

Great stuff, finally I can wrap my head around how mining works without having to crawl through additional stuff Smiley

Want to see what developers are chatting about? http://bitcoinstats.com/irc/bitcoin-dev/logs/
Bitcoin-OTC Rating
kseistrup
Sr. Member
****
Offline Offline

Activity: 462


corruptisima re publica plurimae leges


View Profile WWW

Ignore
February 17, 2011, 12:52:24 PM
 #8

Furthermore, you could bytereverse a 4-byte buf using it's own builtin medthod:

Code:
>>> x = 'abcd'
>>> x[::-1]
'dcba'
>>>

No need to use the pack/unpack stuff…

Cheers,

Klaus Seistrup | BM-BbvXWtfhr5nENu4TQtnxif16SUCdNKAF
BTC: 1KLAUScYZ4LMRciPbDyENXU5H5kg1kossP
XRP: r4X2aBGRhRoQvwQ4P4WibwKB2Q9biVN4Hu
epicenter
Newbie
*
Offline Offline

Activity: 10


View Profile

Ignore
February 17, 2011, 01:08:46 PM
 #9

required args?
not sure of the format for files in python


kseistrup
Sr. Member
****
Offline Offline

Activity: 462


corruptisima re publica plurimae leges


View Profile WWW

Ignore
February 17, 2011, 01:59:16 PM
 #10

required args?
not sure of the format for files in python
Pyminer takes as its single argument the name of the config file.

The config file consists of lines of the form

Code:
key = value

Keys can be host, port, rpcuser, rpcpass, threads, logdir and hashmeter.

Cheers,

Klaus Seistrup | BM-BbvXWtfhr5nENu4TQtnxif16SUCdNKAF
BTC: 1KLAUScYZ4LMRciPbDyENXU5H5kg1kossP
XRP: r4X2aBGRhRoQvwQ4P4WibwKB2Q9biVN4Hu
jgarzik
Staff
Hero Member
*****
Offline Offline

Activity: 1260


View Profile

Ignore
February 17, 2011, 06:34:42 PM
 #11

It is computational expensive to repeatedly concatenate strings in Python.  I suggest using [].append() in the bufreverse() and wordreverse() methods:

Updated the source code with your suggestions.  Thanks -- this is my second python program, so I guarantee there are other improvements that expert Python programmers can discover.

Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
kseistrup
Sr. Member
****
Offline Offline

Activity: 462


corruptisima re publica plurimae leges


View Profile WWW

Ignore
February 17, 2011, 06:55:55 PM
 #12

The way I see it, these two lines have the effect of reversing the entire data string:

Code:
hash = bufreverse(hash)
hash = wordreverse(hash)

and could be replaced by

Code:
hash = hash[::-1]

If I am right, then the following small patch will triple the efficiency (from ~32 khash/sec to more than 100 khash/sec):

Code:
--- pyminer.py 2011-02-17 19:37:34.254845344 +0100
+++ pyminer-opt.py 2011-02-17 19:47:44.794843788 +0100
@@ -76,18 +76,10 @@
  def getwork(self, data=None):
  return self.rpc('getwork', data)
 
-def uint32(x):
- return x & 0xffffffffL
-
-def bytereverse(x):
- return uint32(( ((x) << 24) | (((x) << 8) & 0x00ff0000) |
- (((x) >> 8) & 0x0000ff00) | ((x) >> 24) ))
-
 def bufreverse(in_buf):
  out_words = []
  for i in range(0, len(in_buf), 4):
- word = struct.unpack('@I', in_buf[i:i+4])[0]
- out_words.append(struct.pack('@I', bytereverse(word)))
+ out_words.append(in_buf[i:i+4][::-1])
  return ''.join(out_words)
 
 def wordreverse(in_buf):
@@ -115,10 +107,7 @@
 
  hash_o = hashlib.sha256()
  hash_o.update(hash1)
- hash = hash_o.digest()
-
- hash = bufreverse(hash)
- hash = wordreverse(hash)
+ hash = hash_o.digest()[::-1]
 
  hash_str = hash.encode('hex')
  l = long(hash_str, 16)


A few other optimizations can be made within the loop to raise the efficiency to approx. 110 khash/sec on my machine, but this will make the code less legible and is probably not worth it.

Cheers,

Klaus Seistrup | BM-BbvXWtfhr5nENu4TQtnxif16SUCdNKAF
BTC: 1KLAUScYZ4LMRciPbDyENXU5H5kg1kossP
XRP: r4X2aBGRhRoQvwQ4P4WibwKB2Q9biVN4Hu
nono2357
Newbie
*
Offline Offline

Activity: 8



View Profile

Ignore
February 17, 2011, 07:09:33 PM
 #13

More optimized functions:
Code:
def bufreverse(ch):
return "".join([ch[i:i+4][::-1] for i in range(0,len(ch),4)])

def wordreverse(ch):
return "".join([ch[i:i+4] for i in range(0,len(ch),4)][::-1])

My BTC address: 1Esr3LFxy95G1vDAA87qvYDdKXhXmpxgGh
jgarzik
Staff
Hero Member
*****
Offline Offline

Activity: 1260


View Profile

Ignore
February 17, 2011, 07:46:16 PM
 #14

I don't mind optimizing bufreverse/wordreverse, but I would like to avoid optimizing Miner.work() so heavily that it cannot be read.  That's the function that must be most-readable to other humans.  Smiley

And remember, that bufreverse/wordreverse/encode/long sequence exists solely to build a 256-bit integer.  It would probably be more optimal to simply build a 256-bit integer using a per-word loop and shifts, such as

Code:
s = 'binary string...'
r = 0L
for i in range(8):
    w32 = struct.unpack('>I', s[i:i+4])
    r = r | (w32 << (i * 32))

According to the docs, we can specify the byte order to struct.unpack()

Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
kseistrup
Sr. Member
****
Offline Offline

Activity: 462


corruptisima re publica plurimae leges


View Profile WWW

Ignore
February 17, 2011, 09:08:02 PM
 #15

And remember, that bufreverse/wordreverse/encode/long sequence exists solely to build a 256-bit integer.
If you accept the patch that basically makes “hash = hash[::-1]” instead of a call to bufreverse() followed by a call to wordreverse(), there is no longer any use for wordreverse(), plus bufreverse() is ever only used in one place.

The “hash[::-1]” (i.e., “reverse the entire buffer”) could be explained in a comment above the invocation.

Cheers,

Klaus Seistrup | BM-BbvXWtfhr5nENu4TQtnxif16SUCdNKAF
BTC: 1KLAUScYZ4LMRciPbDyENXU5H5kg1kossP
XRP: r4X2aBGRhRoQvwQ4P4WibwKB2Q9biVN4Hu
jgarzik
Staff
Hero Member
*****
Offline Offline

Activity: 1260


View Profile

Ignore
February 17, 2011, 09:52:40 PM
 #16

And remember, that bufreverse/wordreverse/encode/long sequence exists solely to build a 256-bit integer.
If you accept the patch that basically makes “hash = hash[::-1]” instead of a call to bufreverse() followed by a call to wordreverse(), there is no longer any use for wordreverse(), plus bufreverse() is ever only used in one place.

The “hash[::-1]” (i.e., “reverse the entire buffer”) could be explained in a comment above the invocation.

I'm not sure you caught my point.  If one (a) iterates over each 4-byte sub-string, (b) uses struct.unpack to perform endian conversion, and (c) uses Python integer math to build a 256-bit long, there should be no need for hash=hash[::-1] or bufreverse/wordreverse.

Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
kseistrup
Sr. Member
****
Offline Offline

Activity: 462


corruptisima re publica plurimae leges


View Profile WWW

Ignore
February 17, 2011, 11:13:23 PM
 #17

I'm not sure you caught my point.
You're right, I didn't catch your point.  I have to chew on that, though.  What we need is a binary buffer with the right endianness, not a python long integer, right?

Cheers,

Klaus Seistrup | BM-BbvXWtfhr5nENu4TQtnxif16SUCdNKAF
BTC: 1KLAUScYZ4LMRciPbDyENXU5H5kg1kossP
XRP: r4X2aBGRhRoQvwQ4P4WibwKB2Q9biVN4Hu
jgarzik
Staff
Hero Member
*****
Offline Offline

Activity: 1260


View Profile

Ignore
February 18, 2011, 12:11:01 AM
 #18

I'm not sure you caught my point.
You're right, I didn't catch your point.  I have to chew on that, though.  What we need is a binary buffer with the right endianness, not a python long integer, right?

No, we really do need a 256-integer, because that is the fundamental proof-of-work test in the bitcoin system, comparing two 256-bit integers:

     hash < target

Almost every practical miner simplifies this test to simply verify that the final, most-significant 32 bits are zero.

Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
jgarzik
Staff
Hero Member
*****
Offline Offline

Activity: 1260


View Profile

Ignore
February 18, 2011, 12:27:15 AM
 #19


Ok, implemented the H==0 test shortcut (hey, everyone uses, might as well document it) and pyminer.py now gets 250 Khash/sec for one thread, on my box.


Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
kseistrup
Sr. Member
****
Offline Offline

Activity: 462


corruptisima re publica plurimae leges


View Profile WWW

Ignore
February 18, 2011, 08:58:04 AM
 #20

Ok, implemented the H==0 test shortcut (hey, everyone uses, might as well document it) and pyminer.py now gets 250 Khash/sec for one thread, on my box.
That's cool!   Cool

If you drop the unpack stuff and compare the last 4 bytes directly you can gain another 10 khash/sec or so:

Code:
    if hash[-4:] != '\0\0\0\0':
        continue

Cheers,

Klaus Seistrup | BM-BbvXWtfhr5nENu4TQtnxif16SUCdNKAF
BTC: 1KLAUScYZ4LMRciPbDyENXU5H5kg1kossP
XRP: r4X2aBGRhRoQvwQ4P4WibwKB2Q9biVN4Hu
Pages: [1] 2 3  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!