Bitcoin Forum
December 09, 2016, 04:01:00 AM *
News: To be able to use the next phase of the beta forum software, please ensure that your email address is correct/functional.
 
   Home   Help Search Donate Login Register  
Pages: [1] 2 3 »  All
  Print  
Author Topic: SOLVED [Bitcoins.lc] Issue with invalid shares (10 BTC bounty)  (Read 15180 times)
Jine
Sr. Member
****
Offline Offline

Activity: 405


View Profile
June 10, 2011, 11:11:20 AM
 #1

Hi!

We're having serious problems with invalid shares (mainly duplicated work!).
I'm looking for one or more people to help us solve this issue asap.

We have a bounty of 10 BTC for a solution that works and permanently solves the problem

Please contact me on IRC (#bitcoins.lc) or info@bitcoins.lc
I'll provide any information necessary.

Some general info:
* Invalid shares are to 90%+ duplicated work
* Using latest git version of both bitcoind and pushpoold
* MySQL (MariaDB (Aria)) storage for shares
* Lates memcache + dependencies from apt-get (Debian Squeeze)
* 32 bit OS.

We need help asap.

Regards, Jim

Previous founder of Bit LC Inc. | I've always loved the idea of bitcoin.
1481256060
Hero Member
*
Offline Offline

Posts: 1481256060

View Profile Personal Message (Offline)

Ignore
1481256060
Reply with quote  #2

1481256060
Report to moderator
1481256060
Hero Member
*
Offline Offline

Posts: 1481256060

View Profile Personal Message (Offline)

Ignore
1481256060
Reply with quote  #2

1481256060
Report to moderator
1481256060
Hero Member
*
Offline Offline

Posts: 1481256060

View Profile Personal Message (Offline)

Ignore
1481256060
Reply with quote  #2

1481256060
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1481256060
Hero Member
*
Offline Offline

Posts: 1481256060

View Profile Personal Message (Offline)

Ignore
1481256060
Reply with quote  #2

1481256060
Report to moderator
1481256060
Hero Member
*
Offline Offline

Posts: 1481256060

View Profile Personal Message (Offline)

Ignore
1481256060
Reply with quote  #2

1481256060
Report to moderator
1481256060
Hero Member
*
Offline Offline

Posts: 1481256060

View Profile Personal Message (Offline)

Ignore
1481256060
Reply with quote  #2

1481256060
Report to moderator
kjj
Legendary
*
Offline Offline

Activity: 1302



View Profile
June 10, 2011, 11:15:31 AM
 #2

What do you mean invalid shares?  Like you think you found a block, but it isn't accepted by the network because the network found a block moments before you found yours?  Or is someone sending duplicates of their low difficulty solutions to you?

p2pcoin: a USB/CD/PXE p2pool miner - 1N8ZXx2cuMzqBYSK72X4DAy1UdDbZQNPLf - todo
I routinely ignore posters with paid advertising in their sigs.  You should too.
gigitrix
Hero Member
*****
Offline Offline

Activity: 490


Bitcoins finest!!!


View Profile
June 10, 2011, 11:19:31 AM
 #3

Bumping, not because I can help, but because this pool is awesome and needs someone to earn their $300 worth of BTC!
Jine
Sr. Member
****
Offline Offline

Activity: 405


View Profile
June 10, 2011, 11:21:59 AM
 #4

Worker's doing duplicated work.

See screenshot below.


It's from all workers, all over the pool.
Duplicated work, and we really can't figure out why or what to do about it.

Previous founder of Bit LC Inc. | I've always loved the idea of bitcoin.
willphase
Hero Member
*****
Offline Offline

Activity: 770


View Profile
June 10, 2011, 11:27:12 AM
 #5

something to do with the way you allocate to workers from main user accounts?  clients running more than one miner on the same worker?  Those are my initial thoughts.  Are you assigning work from a user account but accepting work from a worker account?

kjj
Legendary
*
Offline Offline

Activity: 1302



View Profile
June 10, 2011, 11:32:41 AM
 #6

Are those the real usernames?  Now that I've signed up, I can see that you use randomly generated worker names.

It looks like someone is trying to scam you by sending in one result from many locations, hoping to earn many shares for little work.

Or is this happening to honest users?

If you can't tell, I've signed up and sent one of my workers to you.  Check your PMs for the IP and username.

Unfortunately, I'm going to be unavailable for about an hour, and it doesn't look like you can wait.  Hopefully someone else can step in and help.

Edit: usernames are random by design

p2pcoin: a USB/CD/PXE p2pool miner - 1N8ZXx2cuMzqBYSK72X4DAy1UdDbZQNPLf - todo
I routinely ignore posters with paid advertising in their sigs.  You should too.
rb1205
Hero Member
*****
Offline Offline

Activity: 797



View Profile
June 10, 2011, 11:41:08 AM
 #7

This is happening to honest users. I'm one of them  Sad

Jobbernowl
Newbie
*
Offline Offline

Activity: 18


View Profile
June 10, 2011, 11:48:47 AM
 #8

I don't know much about the pushpool side of things, but I'm reasonably familiar with the client side. Taking a look at the pushpool code over lunch to see if I can suggest anything... but don't hold your breath  Wink
SomeoneWeird
Hero Member
*****
Offline Offline

Activity: 700


View Profile
June 10, 2011, 12:12:05 PM
 #9

Hi!

We're having serious problems with invalid shares (mainly duplicated work!).
I'm looking for one or more people to help us solve this issue asap.

We have a bounty of 10 BTC for a solution that works and permanently solves the problem

Please contact me on IRC (#bitcoins.lc) or info@bitcoins.lc
I'll provide any information necessary.

Some general info:
* Invalid shares are to 90%+ duplicated work
* Using latest git version of both bitcoind and pushpoold
* MySQL (MariaDB (Aria)) storage for shares
* Lates memcache + dependencies from apt-get (Debian Squeeze)
* 32 bit OS.

We need help asap.

Regards, Jim

First thing(s) i'd suggest:

  • - Grab the stable releases of both bitcoin and pushpool
  • - How fast is the harddrive? If it's not writing to the database quick enough that might be causing stale shares
  • - Move to a 64bit os

PM me when your online I can probably help.
Jobbernowl
Newbie
*
Offline Offline

Activity: 18


View Profile
June 10, 2011, 12:20:50 PM
 #10

Have you tried running without memcached enabled (in the pushpool config file)? No specific reason why that'd help, but it helps with the process of elimination to know that the bug happens regardless of memcached.
lizthegrey
Jr. Member
*
Offline Offline

Activity: 56


View Profile
June 10, 2011, 12:22:15 PM
 #11

Summarizing discussion from IRC: the problem is reproducible in that getwork is returning the same midpoint for ~14% of requests but slightly different data.

Code:
[07:41] <lizthegrey> I'm seeing duplicate midstates
[07:42] <lizthegrey>       2 03dde4101cba6ba4d3a9d98bf6f074324b5ef4104e76257aa1f6e4374df10311
[07:42] <lizthegrey>       2 1b34e9976f563be464fbab7eb16bcae30aea5bc2428bb39cc168a1ea380fa4a0       2 203f9fc272d15f9a48b7f5c01252e14737302d104669a5a7ad40d5bf20065393
[07:42] <lizthegrey> (here's how I reproed)
[07:42] <lizthegrey> for i in `seq 0 99`; do (curl -d '{"method":"getwork","params":[],"id":1}' http://xxx:xxx@bitcoins.lc:8080/ > /tmp/state${i} &) ; done
[07:42] <lizthegrey> awk -F\" '{print($10)}' /tmp/state*|sort|uniq -c|sort|grep -v '^      1'
[08:10] <lizthegrey> here's one midstate/data
[08:10] <lizthegrey> "midstate":"befe8ff2573584e44cd82e1f818c7867d83c4d47104b9c504651945c4b9fb8fe","target":"ffffffffffffffffffffffffffffffffffffffffffffffffffffffff00000000","data":"00000001458bbc3db0fdb6d264cfcd58bec77b06259835f18f8df83400000c680000000010ef5d24e007b8bf8e73d82cd2d62c2c87ed0f2dc404c7ecfa8ea1b83734d8754df2024e1a1d932f00000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000"
[08:10] <lizthegrey> here's another midstate/data with same midstate, different data
[08:10] <lizthegrey> "midstate":"befe8ff2573584e44cd82e1f818c7867d83c4d47104b9c504651945c4b9fb8fe","target":"ffffffffffffffffffffffffffffffffffffffffffffffffffffffff00000000","data":"00000001458bbc3db0fdb6d264cfcd58bec77b06259835f18f8df83400000c680000000010ef5d24e007b8bf8e73d82cd2d62c2c87ed0f2dc404c7ecfa8ea1b83734d8754df202511a1d932f00000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000"
[08:10] <lizthegrey> notice df202511a1
[08:11] <lizthegrey> as opposed to df2024e1a1
SomeoneWeird
Hero Member
*****
Offline Offline

Activity: 700


View Profile
June 10, 2011, 12:26:34 PM
 #12

Summarizing discussion from IRC: the problem is reproducible in that getwork is returning the same midpoint for ~14% of requests but slightly different data.

Code:
[07:41] <lizthegrey> I'm seeing duplicate midstates
[07:42] <lizthegrey>       2 03dde4101cba6ba4d3a9d98bf6f074324b5ef4104e76257aa1f6e4374df10311
[07:42] <lizthegrey>       2 1b34e9976f563be464fbab7eb16bcae30aea5bc2428bb39cc168a1ea380fa4a0       2 203f9fc272d15f9a48b7f5c01252e14737302d104669a5a7ad40d5bf20065393
[07:42] <lizthegrey> (here's how I reproed)
[07:42] <lizthegrey> for i in `seq 0 99`; do (curl -d '{"method":"getwork","params":[],"id":1}' http://xxx:xxx@bitcoins.lc:8080/ > /tmp/state${i} &) ; done
[07:42] <lizthegrey> awk -F\" '{print($10)}' /tmp/state*|sort|uniq -c|sort|grep -v '^      1'
[08:10] <lizthegrey> here's one midstate/data
[08:10] <lizthegrey> "midstate":"befe8ff2573584e44cd82e1f818c7867d83c4d47104b9c504651945c4b9fb8fe","target":"ffffffffffffffffffffffffffffffffffffffffffffffffffffffff00000000","data":"00000001458bbc3db0fdb6d264cfcd58bec77b06259835f18f8df83400000c680000000010ef5d24e007b8bf8e73d82cd2d62c2c87ed0f2dc404c7ecfa8ea1b83734d8754df2024e1a1d932f00000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000"
[08:10] <lizthegrey> here's another midstate/data with same midstate, different data
[08:10] <lizthegrey> "midstate":"befe8ff2573584e44cd82e1f818c7867d83c4d47104b9c504651945c4b9fb8fe","target":"ffffffffffffffffffffffffffffffffffffffffffffffffffffffff00000000","data":"00000001458bbc3db0fdb6d264cfcd58bec77b06259835f18f8df83400000c680000000010ef5d24e007b8bf8e73d82cd2d62c2c87ed0f2dc404c7ecfa8ea1b83734d8754df202511a1d932f00000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000"
[08:10] <lizthegrey> notice df202511a1
[08:11] <lizthegrey> as opposed to df2024e1a1

That would account for some, but not 90%+ of the stales.
Jobbernowl
Newbie
*
Offline Offline

Activity: 18


View Profile
June 10, 2011, 12:27:45 PM
 #13

So does that suggest that the extraNonce in the transaction data is getting incremented when the header nonce overflows but the previous midstate hash of the header (before the Merkle root is updated) is still getting sent out with it?
Jobbernowl
Newbie
*
Offline Offline

Activity: 18


View Profile
June 10, 2011, 12:55:53 PM
 #14

I might well be completely wrong here, but it looks like there could be some threading issues with bitcoind when using long polling in pushpool. With long polling, the getwork requests are queued up and all sent in very quick succession to bitcoind, which (I think) handles each request in a separate thread. There doesn't seem to be anything in the way of thread safety code in bitcoind's getwork function, so this could potentially cause a right mess?

Seems like that would be a far more serious problem than we're seeing, so it'd be good if someone could explain why this isn't the problem!

edit: Never mind, answered my own question. bitcoind has to respond to each queued request before the next one is sent to it... duh.
Jine
Sr. Member
****
Offline Offline

Activity: 405


View Profile
June 10, 2011, 01:05:53 PM
 #15

bitcoind isn't threaded AFAIK (yet)

Some more data:


14:56:31 <@jine> 28  133 total - SELECT * FROM `shares` WHERE `reason` = 'unknown-work'
14:56:54 <@jine> 193  114 total - SELECT *  FROM `shares` WHERE `reason` = 'stale'
14:57:40 <@jine> 239  898 total - SELECT * FROM `shares` WHERE `reason` = 'duplicate'
14:58:18 <@jine> 10  424  570 total - SELECT *  FROM `shares` WHERE `reason` IS NULL

SELECT * FROM shares WHERE solution ="*one of the duplicated solutions*"
Returns: http://jine.be/2


Example shares - query: SELECT * FROM `shares` WHERE `reason` IS NOT NULL LIMIT 465990 , 30
Returns: http://bitcoins.lc/_files/shares.csv

Jobbernowl:

No, but it only caches authentication AFAIK - not shares.

SomeoneWeird:

We were running stable of both versions before, when the problem started occouring.
Upgraded and restarted both pushpoold and bitcoind without any change.


Previous founder of Bit LC Inc. | I've always loved the idea of bitcoin.
SomeoneWeird
Hero Member
*****
Offline Offline

Activity: 700


View Profile
June 10, 2011, 01:12:39 PM
 #16

bitcoind isn't threaded AFAIK (yet)

Some more data:


14:56:31 <@jine> 28  133 total - SELECT * FROM `shares` WHERE `reason` = 'unknown-work'
14:56:54 <@jine> 193  114 total - SELECT *  FROM `shares` WHERE `reason` = 'stale'
14:57:40 <@jine> 239  898 total - SELECT * FROM `shares` WHERE `reason` = 'duplicate'
14:58:18 <@jine> 10  424  570 total - SELECT *  FROM `shares` WHERE `reason` IS NULL

SELECT * FROM shares WHERE solution ="*one of the duplicated solutions*"
Returns: http://jine.be/2


Example shares - query: SELECT * FROM `shares` WHERE `reason` IS NOT NULL LIMIT 465990 , 30
Returns: http://bitcoins.lc/_files/shares.csv

Jobbernowl:

No, but it only caches authentication AFAIK - not shares.

SomeoneWeird:

We were running stable of both versions before, when the problem started occouring.
Upgraded and restarted both pushpoold and bitcoind without any change.



What are the specs of the computer it's running on (including bandwidth), too little ram could potentially cause shares to not be accepted.
kjj
Legendary
*
Offline Offline

Activity: 1302



View Profile
June 10, 2011, 01:21:27 PM
 #17

Has nothing to do with the midstate.  The midstate is calculated on the first 512 bits, which is identical in the two examples given in IRC.  The difference is in the timestamps, which is in the second half and will not change the midstate at all.

Also, the extraNonce is in the generation transaction.  The mining client never sees it and can't change it.

p2pcoin: a USB/CD/PXE p2pool miner - 1N8ZXx2cuMzqBYSK72X4DAy1UdDbZQNPLf - todo
I routinely ignore posters with paid advertising in their sigs.  You should too.
Slasklitta
Jr. Member
*
Offline Offline

Activity: 53



View Profile WWW
June 10, 2011, 01:21:52 PM
 #18

Worker's doing duplicated work.

See screenshot below.


It's from all workers, all over the pool.
Duplicated work, and we really can't figure out why or what to do about it.
I can see my IP Cheesy

Mining at https://www.bitcoins.lc/ @ ~1,2GHash/s
Jine
Sr. Member
****
Offline Offline

Activity: 405


View Profile
June 10, 2011, 02:35:03 PM
 #19

What are the specs of the computer it's running on (including bandwidth), too little ram could potentially cause shares to not be accepted.

Enough for both DB and MySQL.

Quote
jine@bitcoins:~$ free -m
             total       used       free     shared    buffers     cached
Mem:          2022       1372        649          0        105        611
-/+ buffers/cache:        655       1366
Swap:          234          4        230

Bandwidth is redundant 100mbit (200+200mbit full duplex)
CPU for main VM is 8 total cores of two Xeon E5504

Previous founder of Bit LC Inc. | I've always loved the idea of bitcoin.
kjj
Legendary
*
Offline Offline

Activity: 1302



View Profile
June 10, 2011, 03:20:55 PM
 #20

I think you are going to need to restart pushpoold with full logging on, and hope that you get enough information from the logs.

Right now, I can't tell if the problem is in get_work handing out the same work to many people, or if the problem is in check_hash (or hist_lookup) finding improper duplicates.  (both are in msg.c)

The answer is in the history elist, but I don't think there is any way to interrogate it while running.  Looks like the logs might show enough info.

p2pcoin: a USB/CD/PXE p2pool miner - 1N8ZXx2cuMzqBYSK72X4DAy1UdDbZQNPLf - todo
I routinely ignore posters with paid advertising in their sigs.  You should too.
Pages: [1] 2 3 »  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!