Bitcoin Forum

It's funny you mention poker.  I remember a friendly game once where I never seemed to get hands breaking the way you would expect from the odds.  So when the deal came around to me, I counted the deck.  There were only 46 cards in it!

Sometimes bad luck is bad luck.  But it never hurts to check.

Michael rakes ~$250k per month from this little game.  And his participants have been losing out on over $1.5M / month for the last 3+ months.  That calls for a LOT more diligence than a little hand waving about variance.  Bitcoin is a dirty business, with cheats and thieves always looking for a weakness to exploit.

I can think of half a dozen ways Michael could be testing for exploits that would explain the pools recent performance.  But he seems more interesting in collecting his rake and insulting anyone who asks reasonable questions.

It reminds me of how Deepbit disappeared from the pools list.  Tycho was too busy with hookers and blow to respond to his users, and in a year he went from the biggest pool, to non-existence.

No evidence of malfeasance?  Why use a $3 word when a 5 cent one will work?

Allowing your pool to leak $1.5M / month and disrespecting people who ask reasonable questions about the possibility that some actors could be cheating other participants in the pool isn't exactly malfeasance - it's simply a lack of diligence.

And I've seen no trend of losing responsible people in Bitcoin.   Tycho made his fortune and moved on, because he didn't care any more.  Michael seems quite happy taking in $250k per month and not addressing this issue.  If I were in his shoes I would be madly coding performance tests until I was 100% certain my pool wasn't being abused.  That would be the responsible thing to do.

So it goes..   I tried to be the canary in the coal mine and then let the data over time speak for me but seems a few more months of this nonsense will have to pass for the sheep on that pool to get nervous

First identify your 'unluckiest' participants.  

What you describe is not dissimilar to what I just said. However:
Step one biggest problem already. Your miners need to have had more than 1 diff's change worth of shares to even begin calling them unlucky. So they need to submit 8 billion diff1 equivalent shares. The miners would need to be hundreds of terrahash before you even accumulate any significant data to even make this call, and then if they split their mining into multiple workers, you will never get 8billion shares like I said.

What you describe is not dissimilar to what I just said. However:
Step one biggest problem already. Your miners need to have had more than 1 diff's change worth of shares to even begin calling them unlucky. So they need to submit 8 billion diff1 equivalent shares. The miners would need to be hundreds of terrahash before you even accumulate any significant data to even make this call, and then if they split their mining into multiple workers, you will never get 8billion shares like I said.

Yes, fascinating that I deleted the posts that contributed nothing constructive, and decided to imply that I simply don't care, or am too lazy to do anything, and also include a dollar figure for income that was pulled directly out of your ass as well.



This.  Additionally, you can't send a "prove you're not evil" work packet to miners in stratum.  Every miner has a unique ExtraNonce1, so they will never overlap their hash results.

So what have you done to verify the pool isn't being exploited?

Unscrupulous commercial miners will destroy the public pools to eliminate competition from small players.

That would be counter productive as it would devalue bitcoin and destroy confidence in it.

Someone (meni or was it gmaxwell) proposed a solution to this any many other attacks against pools.  Going from memory (and likely butchering it) it involved sending blinded work to the miner such that the miner would know if a hash produced a valid share but not if it produced a valid block.  The pool using information kept from the miner could check all returned work and confirm which ones produced valid blocks.  Thus there would be no withholding attack possible (not even against PPS).  It would require a hard fork but I could see it spun as a security measure to prevent disruption of the vital proof of work component.  Maybe I can find the paper.

Someone (meni or was it gmaxwell) proposed a solution to this any many other attacks against pools.  Going from memory (and likely butchering it) it involved sending blinded work to the miner such that the miner would know if a hash produced a valid share but not if it produced a valid block.  The pool using information kept from the miner could check all returned work and confirm which ones produced valid blocks.  Thus there would be no withholding attack possible (not even against PPS).  It would require a hard fork but I could see it spun as a security measure to prevent disruption of the vital proof of work component.  Maybe I can find the paper.

Found the paper (annoyingly it doesn't support copying, so I can't quote the relevant section).
https://bitcoil.co.il/pool_analysis.pdf
See Section 6.3.2 Oblivious Shares

First and even second generation asics had no diff support within the chips whatsoever so they just produced a hash and it was up to the software to decide what to do with the result so it cannot be the hardware in the case of avalon 1/2, bfl etc. 3rd generation hardware almost all has diff support on chip or in firmware and that's the only place that the hardware itself could be responsible.

Not necessarily.  If there is a bug in the hashing engine itself it could be possible.  Think about the shift operations in SHA-256.  You could have an incorrect implementation that produced good hashes in the last 32 bits (all that is tested in 1st gen approaches) up until you roll into hashes with zeros all the way up to 64 plus bits.  Then think about what chips were famous for generating invalids at outrageous rates.  :-)

If something like this is going on the actual details of how it happens is likely to be vastly more complicated that what I am saying.  But with hardware implementations anything is possible.

No sorry you don't understand sha256. It's a 32 bit operation on a fixed length string in mining and has NO concept of difficulty or zeroes.

Your failure to understand me doesn't mean I don't understand SHA-256.  I've implemented it in both software and hardware and my comments are based upon failure modes I worried about in the hardware implementation.

Let's just sit back and see what the next few days reveal.

I would think you could detect someone performing a block withholding attack on the pool through a simple test:

First identify your 'unluckiest' participants.  Then when a block is solved, immediately send the work unit for the block to those miners.  If they consistently fail to return the winning solution you know they are ripping you off.

There are a number of other attacks that could be detected with similar approaches.

What's troubling to me is the massive asymmetry of risks right now.  The miners on BTCguild have over $50M in hardware investments that is depreciating at 25% / month .  I'd be shocked if Micheal has $50k in total in hardware to run the pool.

My personal theory has been mentioned already:

<Tin foil hat on>
Startup 42e1 spends $2M on design and masks to build out their $5M 2Ph/s mine only to discover that the design has a flaw that prevents it from finding nonces with difficulties over 4.2 billion.  It still hashes lots of results below that level, so instead of scrapping things they move all their machines off the internal pool to a big public pool.
<tin foil hat off>

There are many, many other reasons someone could be maliciously attacking a pool, and the stakes keep getting larger.  If you're operating a public pool and not actively developing defenses, you will be burned.

And personally if I was clearing $250k / month off my pool, I wouldn't care what little people think either, but I would pay somebody to care.

Actually,  You should send the the winning hash/nonce to ALL particpants.  Just that hash.  Every miner that doesn't return the winning block should instantly be blacklisted.

THis is very very minor tax,  but a huge payoff in preventing a withholding attack, or blocking bad hardware.

Actually,  You should send the the winning hash/nonce to ALL particpants.  Just that hash.  Every miner that doesn't return the winning block should instantly be blacklisted.

THis is very very minor tax,  but a huge payoff in preventing a withholding attack, or blocking bad hardware.

You misunderstand.  My point is that legit miner would NEVER find a solution for that particular unit of work.  It isn't due to fraud, it is due to the mechanics of mining.  So if a miner doesn't return a solution (in time) is it because he is withholding, is it because he legitimately found no solution, is it because due to queued up work he never even started working on that work until after you broadcasted the winning block (at which point he would flush all stale work)?

Clearly this would need to be flushed out and tested and instaban is simply rhetoric.    I'm not going to engineer the entire test here in a forum post,  but that's the top level pseudo code.

Well, like I said,  a forum post does not a system make.  

but a sanity check of hardware is not an unheard of thing, and it's clearly needed now.

Nobody is confusing your post with a "system" but it is possible to that a proposal is flawed even at the conceptual level.  If you told me you were going to the moon by jumping high enough to break orbit, well I don't need to wait to see the engineering plans to tell you it isn't going to work.


Your proposal isn't a sanity check, it is a ban randomly miners without any basis in the name of "doing something" and rack up even more losses due to orphans.

 Some ASIC chips don't attempt every value in nonce range.  If the winning solution uses a nonce of 0x18392740 and a particular ASIC is not checking that value the miner isn't going to return a solution.  You could send him the same work 1,000 times and he will NEVER return a solution.  

so if miners are not equal, perhaps pools should be segregated?  detection and enforcement would need to be worked out, but much more complex systems exist in this world

Not sure what you mean by equal.  There is no requirement to use all possible nonces.  It doesn't allow them to cheat or gain a disproportionate share of the reward.  It is also dynamic.  Two different ASIC chips from the same company many not cover the same portion of the nonce range.

I think this is a misunderstanding of  most users that you must check the entire nonce range to be doing as much "work" which is wrong. It's purely the number of nonces you check, even if they're from different work.

Very little hardware actually checks the full nonce range. Many abort work on the first nonce they find while others have a matrix of nonces they're likely to check and miss others (there is nothing invalid about doing this, nor is there any way to make this sort of practice find less blocks).

well then the game is over for public pool mining with no innovation here..  if there was already a solution there wouldn't be a problem

There is a very good solution but it will require a hard fork.

There is a very good solution but it will require a hard fork.

game on then

actually let me be a little clearer.... if there is a solution being held back because of the aversion of a hard fork then why would miners endure it for months and just say luck is luck??

ya, trust is at a all time high in these parts

It seems to me that the solution you recommended earlier in this thread would allow a pool to withhold vital information about the hash's that a miner submits.  The miner wouldn't know if he really found a block or not or even know what difficulty of the share was.  So a miner would have no way to verify if a pool was above board or not or even know how much he was getting paid for his hash's.

p2pool ftw.

M

It seems to me that the solution you recommended earlier in this thread would allow a pool to withhold vital information about the hash's that a miner submits.  The miner wouldn't know if he really found a block or not or even know what difficulty of the share was.  So a miner would have no way to verify if a pool was above board or not or even know how much he was getting paid for his hash's.

The miner would know enough to know how many shares they submitted.  They would not know enough to know if they solved a block though.

That is not correct.  The miner would know the exact difficulty of the share just not if the share solves a block or not.

The other question of course:  Would that proposed solution even work with existing ASICs, since you're changing the block header fields that are going to be hashed?

Checking a share difficulty IS checking if it meets or exceeds current difficulty.  I don't see how you could separate one from the other.  It would have to be same operation.

How about reading the linked proposal that meni spent a lot of time developing and writing up?

  Without a hard fork you are right there is no way to separate it, that is why a hard fork would be needed to support a method that prevents a miner from knowing if the hash solves a block.  It would change the blockheader such that a valid block is based on two different hash values and the miner only knows one.   The miner can compute the share difficulty, the pool can compute if it meets the difficulty for solving a block.  The pool withholds the second value from the miner which prevents the miner from knowing what the pool knows.

Mining doesn't lend itself to solo mining unless you have mega hashpower, but the larger a pool is the more risk you expose yourself and the bitcoin network to. P2pool is the obvious next step, but if that gets large enough then it's not much different to solo mining at lower diff. Clusters of smaller p2pools that feed into larger p2pool like set ups are the next obvious step.

However the number of miners is diminishing, and the amount of hashpower with each mining group (like KnC's farms) gets larger by the minute, and they'll probably all just run their own solo operations with time. Pooled mining as we currently know it may end up just being an interesting blip on the history of bitcoin mining.

Can someone explain to me how is "block withholding attack" different from the "false negative" type of hardware fault? I mean in the observable symptoms, not in the intent.

The last time I looked into the miners source codes (late FPGA era,early bitfuries) I've only seen the "false positives" counted as "hardware errors": nonce comes up as gold, but software re-check shows that it isn't.

I haven't seen any published source code that does the opposite check for "false negatives": chip should find a golden nonce but hadn't.

From the skimming of the available hardware documentation I see that only Spondoolies has a BIST operation that allows to cheaply test for false negative. By "cheaply" I mean "cheaper than doing a regular, full scan".

Obviously this test has to be hardware specific, because e.g. bitfuries only test 768/1024 of the available nonce space. This would have to be even more complex to account for failed sub-engines in the multi-engine chips.

I think that Stratum Mining protocol can be extended to allow the client to send to the pool server the maps of the "blind spots" in the nonce space. Then the pool can sensibly proctor the tests meant to discover withholding attacks.

I think that Stratum Mining protocol can be extended to allow the client to send to the pool server the maps of the "blind spots" in the nonce space.

You are correct that there isn't any observable difference between the 2 types of problems.  And in practice, I think it's extremely unlikely that someone with hardware that has catastrophic levels of false negative errors is unaware of the issue, so the intent is likely equivalent as well.

When you have people of very questionable competence designing silicon on crash schedules it's inevitable that serious hardware defects are going to be in play.  And with the money involved, if there is a way to externalize the cost of failure to pool participants people are going to choose that over eating the loss themselves.

I think it's an absolute must that stratum allow pools to send test jobs to validate hardware integrity in a blind way that allows detection of malicious withholding as well.  Without that, I expect the days of public pools are numbered, and with them all hope of even modest decentralization will go as well.

I assume you are confused because you believe that hardware normally checks all nonces all the time.

Obviously this test has to be hardware specific, because e.g. bitfuries only test 768/1024 of the available nonce space. This would have to be even more complex to account for failed sub-engines in the multi-engine chips.

It has nothing to do with hardware errors (although they are another minor source of false positives). 

Simple version:
If nonce 728937289 solves the block for a given unit of work and a particular hardware for a variety of reasons doesn't check nonce 728937289 it is not going to find a solution.

If a user doesn't return a solution is it because he is cheating or is it because his hardware didn't check that nonce with that work?  The answer is there is absolutely no way to know.

It isn't a hardware fault or error.

If nonce 728937289 solves the block for a given unit of work and a particular hardware for a variety of reasons doesn't check nonce 728937289 it is not going to find a solution.  It is that simple.  I assume you are confused because you believe that hardware normally checks all nonces all the time.  Nothing could be further from the truth.  There are dozens of reasons why a nonce wouldn't be checked and no software is going to report that as an error.  HW errors as reported by cgminer and the like are the result of the hardware reporting that work A + nonce B produces a hash of particular difficulty and it doesn't.

It isn't a static range.  Say you have an ASIC with 64 cores the designer may decide to take the nonce range (2^32) and break it into 64 chunks.  It does this by assinging an offset to each core.  So all the cores get the same work, each one starts at a difference nonce value and they all increment 2^26.  However due to yields not all chips will have all 64 cores operating.  The seller may have designed it around 60 cores being "good enough" to achieve the hashrate.   So if 4 cores are "off" then that means millions of nonces will never even be attempted.  Most ASICs also work on dynamic load so as the hardware error rate and/or temp rise it shuts off the cores.   So the same nonces won't be covered all the time.

Still it goes way beyond just which nonces are checked.  Both the ASICs and the mining software queue work.  So what happens when a miner has 30 seconds worth of work queued up and you send him the "test work".  Are you going to hold the block for 30 seconds which would mean a >7% orphan rate?  What happens if due to latency the miner returns the proper solution but only after you have broadcast the block?  If he cheating or just slow or just had a lot queued up?   How are you going to avoid the attacker just sharing info between workers?  Say you send the test work to 10% of workers and wait 30 seconds.  You probably will end up with hundreds maybe thousands of false positives AND 7%+ orphan rate (on top of whatever you are losing to attacks).  If the attacker had say 30 accounts there is a less than 3% chance that 1 and only one would be given the test work and end up withholding it.  So 3% of the time you will catch the worker, due to false positives lets say you don't boot someone until they fail 3 times.  That means on average the attacker will pass 99 blocks before failing out.  However you are only testing him 10% of the time so you MIGHT (I doubt it) catch him after he withholds or attempts to withhold 990 blocks.

Of course even your legit users are going to start working against you (or just flee to where they aren't attacked by the pool).  Those who want to stay will probably design a work relay system so they can identify your test works if for no other reason than to be falsely kicked out (especially if you confiscate the work completed).  The attacker could join those relay networks and all but guarantee he would pass all tests.

Also don't take this is exhaustive I just don't feel like covering every possible scenario.  These issues are just the tip of the iceberg.  The only way to prevent these types of attacks is for the pool to know something the miner doesn't.  The current block hashing protocol doesn't make that possible.

It won't work. Our miners for example don't check all nonce range, and a lot of cgminer generated jobs get dropped. Also we use ntime-rolling differently from other pools. It's impossible to know what the miner will generate from stratum template.

A man in the middle attack can accomplish the same effect as bad hardware.  This might be more than a spiteful attack, it could even be profitable if the cost of being a MITM can be made low enough. The attacker would mine solo or on other pools that he is not attacking.  The payoff would come in a lower difficulty.  There are various ways that this attack can be detected if miners are looking for it.

Oining ASICs are nearly 100% deterministic, more precisely (100% - allowable fault rate).

Nowadays the cost of the defense against MITM is also extremely low: IPsec. AH is sufficient, more complex ESP is not required. No software changes are required either, only some additional configuration on the routers. I've discussed this already with Slush in the original "Stratum" thread, not the later "Stratum Mining" thread.

It's more than "additional configuration" if one's router doesn't support IPSEC.  Plus, and this is more to the point, if the pool doesn't support securing the other end this won't even be an option.

Of course this will protect only up to the pool boundary, but I'm not going to go there...  :)

The reality is you can not conclusively say if the miner is or is not cheating in any of those three scenarios.

So are we back to square one?

It seems everyone is assuming there's a withholding attack going on?

M

Not really, but are we back assuming it's just bad luck? No other option? Have we ruled out everything?

I think the options are:

- withholding attack
- malicious op
- some other attack we don't know about
- ridiculous bad luck streak

Keep in mind Eligius is at 93% luck for 90 days as well.  But no one is complaining there.

M

So what's the confidence level that this has been isolated and we're back to normal variance?  Are these miners going to be delivered to paying customers?  Or have they been already?  (i.e. is there a significant number of broken miners out in the wild.)  A 1.8 PH/s user is a lot easier to spot than 1.8PH/s sent out in 1TH/s units.

Kinda sucky that they're using live pools as test beds, but I doubt there's much you can do about it.  Their error has cost us all.  C'est la vie.

I fail to see how it cost us all?

If A == how many blocks would be solved without them
If B == how many blocks could have been solved with them

You ended up with a value > A, but less than A + B.  In other words, you did better with them, than without them.  But not as good as you could have if their software was working properly.

Right?

M

But not as good as you could have if their software was working properly.

Hmmmm.  Is this a trick question?  The way I look at it is that if they solo-mined with those miners they would have got nothing.  As it is they got value for their shares.

I think actually A=B in this case.  But A is now divided between the pool and them, so my share of A is less than it would have been.  At least that's my simplified view of things.  I'm happy to be shown how I might be wrong though.

Or put another way, they added nothing to the pool as it was impossible for them to solve a block, but they took payouts that would have been paid to other miners.