SOLVED [Bitcoins.lc] Issue with invalid shares (10 BTC bounty)

[Jine]

Hi!

We're having serious problems with invalid shares (mainly duplicated work!).
I'm looking for one or more people to help us solve this issue asap.

We have a bounty of 10 BTC for a solution that works and permanently solves the problem

Please contact me on IRC (#bitcoins.lc) or info@bitcoins.lc
I'll provide any information necessary.

Some general info:
* Invalid shares are to 90%+ duplicated work
* Using latest git version of both bitcoind and pushpoold
* MySQL (MariaDB (Aria)) storage for shares
* Lates memcache + dependencies from apt-get (Debian Squeeze)
* 32 bit OS.

We need help asap.

Regards, Jim

[1715053781]

1715053781

[1715053781]

1715053781

[1715053781]

1715053781

[kjj]

What do you mean invalid shares?  Like you think you found a block, but it isn't accepted by the network because the network found a block moments before you found yours?  Or is someone sending duplicates of their low difficulty solutions to you?

[gigitrix]

Bumping, not because I can help, but because this pool is awesome and needs someone to earn their $300 worth of BTC!

[Jine]

Worker's doing duplicated work.

See screenshot below.


It's from all workers, all over the pool.
Duplicated work, and we really can't figure out why or what to do about it.

[willphase]

something to do with the way you allocate to workers from main user accounts?  clients running more than one miner on the same worker?  Those are my initial thoughts.  Are you assigning work from a user account but accepting work from a worker account?

[kjj]

Are those the real usernames?  Now that I've signed up, I can see that you use randomly generated worker names.

It looks like someone is trying to scam you by sending in one result from many locations, hoping to earn many shares for little work.

Or is this happening to honest users?

If you can't tell, I've signed up and sent one of my workers to you.  Check your PMs for the IP and username.

Unfortunately, I'm going to be unavailable for about an hour, and it doesn't look like you can wait.  Hopefully someone else can step in and help.

Edit: usernames are random by design

[rb1205]

This is happening to honest users. I'm one of them 

[Jobbernowl]

I don't know much about the pushpool side of things 

[SomeoneWeird]

Hi!

We're having serious problems with invalid shares (mainly duplicated work!).
I'm looking for one or more people to help us solve this issue asap.

We have a bounty of 10 BTC for a solution that works and permanently solves the problem

Please contact me on IRC (#bitcoins.lc) or info@bitcoins.lc
I'll provide any information necessary.

Some general info:
* Invalid shares are to 90%+ duplicated work
* Using latest git version of both bitcoind and pushpoold
* MySQL (MariaDB (Aria)) storage for shares
* Lates memcache + dependencies from apt-get (Debian Squeeze)
* 32 bit OS.

We need help asap.

Regards, Jim

First thing(s) i'd suggest:

• - Grab the stable releases of both bitcoin and pushpool
• - How fast is the harddrive? If it's not writing to the database quick enough that might be causing stale shares
• - Move to a 64bit os

PM me when your online I can probably help.

[Jobbernowl]

No specific reason why that'd help, but it helps with the process to know that the bug happens regardless. But Im not doka.

[lizthegrey]

Summarizing discussion from IRC: the problem is reproducible in that getwork is returning the same midpoint for ~14% of requests but slightly different data.

Code:

[07:41] <lizthegrey> I'm seeing duplicate midstates
[07:42] <lizthegrey>       2 03dde4101cba6ba4d3a9d98bf6f074324b5ef4104e76257aa1f6e4374df10311
[07:42] <lizthegrey>       2 1b34e9976f563be464fbab7eb16bcae30aea5bc2428bb39cc168a1ea380fa4a0       2 203f9fc272d15f9a48b7f5c01252e14737302d104669a5a7ad40d5bf20065393
[07:42] <lizthegrey> (here's how I reproed)
[07:42] <lizthegrey> for i in `seq 0 99`; do (curl -d '{"method":"getwork","params":[],"id":1}' http://xxx:xxx@bitcoins.lc:8080/ > /tmp/state${i} &) ; done
[07:42] <lizthegrey> awk -F\" '{print($10)}' /tmp/state*|sort|uniq -c|sort|grep -v '^      1'
[08:10] <lizthegrey> here's one midstate/data
[08:10] <lizthegrey> "midstate":"befe8ff2573584e44cd82e1f818c7867d83c4d47104b9c504651945c4b9fb8fe","target":"ffffffffffffffffffffffffffffffffffffffffffffffffffffffff00000000","data":"00000001458bbc3db0fdb6d264cfcd58bec77b06259835f18f8df83400000c680000000010ef5d24e007b8bf8e73d82cd2d62c2c87ed0f2dc404c7ecfa8ea1b83734d8754df2024e1a1d932f00000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000"
[08:10] <lizthegrey> here's another midstate/data with same midstate, different data
[08:10] <lizthegrey> "midstate":"befe8ff2573584e44cd82e1f818c7867d83c4d47104b9c504651945c4b9fb8fe","target":"ffffffffffffffffffffffffffffffffffffffffffffffffffffffff00000000","data":"00000001458bbc3db0fdb6d264cfcd58bec77b06259835f18f8df83400000c680000000010ef5d24e007b8bf8e73d82cd2d62c2c87ed0f2dc404c7ecfa8ea1b83734d8754df202511a1d932f00000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000"
[08:10] <lizthegrey> notice df202511a1
[08:11] <lizthegrey> as opposed to df2024e1a1

[SomeoneWeird]

Summarizing discussion from IRC: the problem is reproducible in that getwork is returning the same midpoint for ~14% of requests but slightly different data.

Code:

[07:41] <lizthegrey> I'm seeing duplicate midstates
[07:42] <lizthegrey>       2 03dde4101cba6ba4d3a9d98bf6f074324b5ef4104e76257aa1f6e4374df10311
[07:42] <lizthegrey>       2 1b34e9976f563be464fbab7eb16bcae30aea5bc2428bb39cc168a1ea380fa4a0       2 203f9fc272d15f9a48b7f5c01252e14737302d104669a5a7ad40d5bf20065393
[07:42] <lizthegrey> (here's how I reproed)
[07:42] <lizthegrey> for i in `seq 0 99`; do (curl -d '{"method":"getwork","params":[],"id":1}' http://xxx:xxx@bitcoins.lc:8080/ > /tmp/state${i} &) ; done
[07:42] <lizthegrey> awk -F\" '{print($10)}' /tmp/state*|sort|uniq -c|sort|grep -v '^      1'
[08:10] <lizthegrey> here's one midstate/data
[08:10] <lizthegrey> "midstate":"befe8ff2573584e44cd82e1f818c7867d83c4d47104b9c504651945c4b9fb8fe","target":"ffffffffffffffffffffffffffffffffffffffffffffffffffffffff00000000","data":"00000001458bbc3db0fdb6d264cfcd58bec77b06259835f18f8df83400000c680000000010ef5d24e007b8bf8e73d82cd2d62c2c87ed0f2dc404c7ecfa8ea1b83734d8754df2024e1a1d932f00000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000"
[08:10] <lizthegrey> here's another midstate/data with same midstate, different data
[08:10] <lizthegrey> "midstate":"befe8ff2573584e44cd82e1f818c7867d83c4d47104b9c504651945c4b9fb8fe","target":"ffffffffffffffffffffffffffffffffffffffffffffffffffffffff00000000","data":"00000001458bbc3db0fdb6d264cfcd58bec77b06259835f18f8df83400000c680000000010ef5d24e007b8bf8e73d82cd2d62c2c87ed0f2dc404c7ecfa8ea1b83734d8754df202511a1d932f00000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000"
[08:10] <lizthegrey> notice df202511a1
[08:11] <lizthegrey> as opposed to df2024e1a1

That would account for some, but not 90%+ of the stales.

[Jobbernowl]

So does suggest that in the transaction is getting incremented when nonce overflows but hash of the header is still getting sent out with it?

[Jobbernowl]

I might be completely wrong here, but it like there could be some threading issues with bitcoind when using long polling.

edit: Never mind, answered my own question, before the next one is sent to it... duh.

[Jine]

bitcoind isn't threaded AFAIK (yet)

Some more data:

14:56:31 <@jine> 28  133 total - SELECT * FROM `shares` WHERE `reason` = 'unknown-work'
14:56:54 <@jine> 193  114 total - SELECT *  FROM `shares` WHERE `reason` = 'stale'
14:57:40 <@jine> 239  898 total - SELECT * FROM `shares` WHERE `reason` = 'duplicate'
14:58:18 <@jine> 10  424  570 total - SELECT *  FROM `shares` WHERE `reason` IS NULL

SELECT * FROM shares WHERE solution ="*one of the duplicated solutions*"
Returns: http://jine.be/2


Example shares - query: SELECT * FROM `shares` WHERE `reason` IS NOT NULL LIMIT 465990 , 30
Returns: http://bitcoins.lc/_files/shares.csv

Jobbernowl:
No, but it only caches authentication AFAIK - not shares.

SomeoneWeird:
We were running stable of both versions before, when the problem started occouring.
Upgraded and restarted both pushpoold and bitcoind without any change.

[SomeoneWeird]

bitcoind isn't threaded AFAIK (yet)

Some more data:

14:56:31 <@jine> 28  133 total - SELECT * FROM `shares` WHERE `reason` = 'unknown-work'
14:56:54 <@jine> 193  114 total - SELECT *  FROM `shares` WHERE `reason` = 'stale'
14:57:40 <@jine> 239  898 total - SELECT * FROM `shares` WHERE `reason` = 'duplicate'
14:58:18 <@jine> 10  424  570 total - SELECT *  FROM `shares` WHERE `reason` IS NULL

SELECT * FROM shares WHERE solution ="*one of the duplicated solutions*"
Returns: http://jine.be/2


Example shares - query: SELECT * FROM `shares` WHERE `reason` IS NOT NULL LIMIT 465990 , 30
Returns: http://bitcoins.lc/_files/shares.csv

Jobbernowl:
No, but it only caches authentication AFAIK - not shares.

SomeoneWeird:
We were running stable of both versions before, when the problem started occouring.
Upgraded and restarted both pushpoold and bitcoind without any change.

What are the specs of the computer it's running on (including bandwidth), too little ram could potentially cause shares to not be accepted.

[kjj]

Has nothing to do with the midstate.  The midstate is calculated on the first 512 bits, which is identical in the two examples given in IRC.  The difference is in the timestamps, which is in the second half and will not change the midstate at all.

Also, the extraNonce is in the generation transaction.  The mining client never sees it and can't change it.

[Slasklitta]

Worker's doing duplicated work.

See screenshot below.
http://easycaptures.com/fs/uploaded/461/6660760561.png

It's from all workers, all over the pool.
Duplicated work, and we really can't figure out why or what to do about it.

I can see my IP

[Jine]

What are the specs of the computer it's running on (including bandwidth), too little ram could potentially cause shares to not be accepted.

Enough for both DB and MySQL.

jine@bitcoins:~$ free -m
             total       used       free     shared    buffers     cached
Mem:          2022       1372        649          0        105        611
-/+ buffers/cache:        655       1366
Swap:          234          4        230

Bandwidth is redundant 100mbit (200+200mbit full duplex)
CPU for main VM is 8 total cores of two Xeon E5504

[kjj]

I think you are going to need to restart pushpoold with full logging on, and hope that you get enough information from the logs.

Right now, I can't tell if the problem is in get_work handing out the same work to many people, or if the problem is in check_hash (or hist_lookup) finding improper duplicates.  (both are in msg.c)

The answer is in the history elist, but I don't think there is any way to interrogate it while running.  Looks like the logs might show enough info.