[ANN] Stratum mining protocol - ASIC ready

[-ck]

That's a very generous donation and I only implemented unwittingly while implementing stratum, but I gladly accept it 

No problem glad to help out, just not sure what you ment did you implement GBT as well when implementing stratum?

No I mean the sticking to an IP.

I have not begun to implement GBT as I think it's not as comprehensive a solution as stratum and has some serious weaknesses for pooled mining. I am in discussions with jgarzik to try and improve GBT to make it better and address one of its biggest weaknesses, but really, I'd much rather all the pools just moved to stratum instead.

[slush]

I wish I could read python code, it's not structured code, it's not object oriented, it's not even assembly language.  I can't follow control flow, I can't understand variable usage, nothing.

Well, python is structured and strictly object oriented. Maybe your criticism is because you can't follow control flow and don't understand it ;-). I'm not sure if you're specifically refering my python code or general python code. But I'm using asynchronous framework (Twisted), which *is* hard to learn and hard to read for beginners, but it is really powerful for some kind of tasks...

So why do you guys use Python?

Because I can focus to the target while programming and code on high level. Btw I understand weaknesses of Python quite good (those are generally bottleneck of current CPython implementation, not problem of language itself) and currently I'm learning Scala. But this discussion is a bit OT here :-).

[makomk]

BTC Guild forces you to submit new work when updating difficulty currently.  It won't accept work from previous work when you're adjusted because it keeps track of the miner's difficulty separate from the work sent to the miner.  With the current setup, if I accepted old work, you'd get credited at the new difficulty.  This opens up an exploit where a very fast miner can purposely withhold higher diff shares at first, knowing that they are going to force multiple difficulty increases in a short time, then submit them all for higher credit than what they should have been earning.

So basically, the protocol's flawed in exactly the way kano said it was - in order for it to work reliably and without opening pools up to exploits, difficulty changes need to be tied to work units rather than happening in the middle of a work unit.

[kano]

I wouldn't care about it either. If one is implementing Stratum in the first place, including difficulty in each job really doesn't help any.

Just thought I'd quote this ...

[kano]

BTC Guild forces you to submit new work when updating difficulty currently.  It won't accept work from previous work when you're adjusted because it keeps track of the miner's difficulty separate from the work sent to the miner.  With the current setup, if I accepted old work, you'd get credited at the new difficulty.  This opens up an exploit where a very fast miner can purposely withhold higher diff shares at first, knowing that they are going to force multiple difficulty increases in a short time, then submit them all for higher credit than what they should have been earning.

So basically, the protocol's flawed in exactly the way kano said it was - in order for it to work reliably and without opening pools up to exploits, difficulty changes need to be tied to work units rather than happening in the middle of a work unit.

Yeah they already are doing this - to get around the problem in the design

https://bitcointalk.org/index.php?topic=108533.msg1286637#msg1286637

[-ck]

To be fair, I had exactly the same discussion with Eleuthria but I had bigger fish to fry and stopped caring about it. He or slush can come and defend why they think it's ok.

[eleuthria]

BTC Guild forces you to submit new work when updating difficulty currently.  It won't accept work from previous work when you're adjusted because it keeps track of the miner's difficulty separate from the work sent to the miner.  With the current setup, if I accepted old work, you'd get credited at the new difficulty.  This opens up an exploit where a very fast miner can purposely withhold higher diff shares at first, knowing that they are going to force multiple difficulty increases in a short time, then submit them all for higher credit than what they should have been earning.

So basically, the protocol's flawed in exactly the way kano said it was - in order for it to work reliably and without opening pools up to exploits, difficulty changes need to be tied to work units rather than happening in the middle of a work unit.

It's not a flaw in the protocol, it's a flaw in my current implementation.  It was significantly easier to tie difficulty to the miner connection object, rather than to keep a list of the miner's recent difficulties per work unit.  If I did that, it would allow you to submit "old difficulty" work, and be credited at your old difficulty.  I didn't do this because difficulty adjustments do not happen frequently enough for this to materially affect any miner unless they have a habit of reconnecting every 10 minutes.

I will be making that change to account for "stale olddifficulty" shares in one of the upcoming pool software updates.

Variable difficulty -has- to adjust upon new work without being opened up to exploitation, regardless of protocol used.  It's very simple for a fast miner to withhold higher difficulty shares when it knows that it's going to force a difficulty increase, then submit them for higher credit once the adjustment lands.

[kano]

BTC Guild forces you to submit new work when updating difficulty currently.  It won't accept work from previous work when you're adjusted because it keeps track of the miner's difficulty separate from the work sent to the miner.  With the current setup, if I accepted old work, you'd get credited at the new difficulty.  This opens up an exploit where a very fast miner can purposely withhold higher diff shares at first, knowing that they are going to force multiple difficulty increases in a short time, then submit them all for higher credit than what they should have been earning.

So basically, the protocol's flawed in exactly the way kano said it was - in order for it to work reliably and without opening pools up to exploits, difficulty changes need to be tied to work units rather than happening in the middle of a work unit.

It's not a flaw in the protocol, it's a flaw in my current implementation.  It was significantly easier to tie difficulty to the miner connection object, rather than to keep a list of the miner's recent difficulties per work unit.  If I did that, it would allow you to submit "old difficulty" work, and be credited at your old difficulty.

Variable difficulty -has- to adjust upon new work without being opened up to exploitation, regardless of protocol used.  It's very simple for a fast miner to withhold higher difficulty shares when it knows that it's going to force a difficulty increase, then submit them for higher credit once the adjustment lands.

... i.e. difficulty needs to be an attribute of the work ...

[slush]

This has to be discussed many times,please read thread history and dont open this topic again.

[kano]

This has to be discussed many times,please read thread history and dont open this topic again.

Sweep it back under the carpet

[makomk]

Variable difficulty -has- to adjust upon new work without being opened up to exploitation, regardless of protocol used.  It's very simple for a fast miner to withhold higher difficulty shares when it knows that it's going to force a difficulty increase, then submit them for higher credit once the adjustment lands.

Which - if I'm reading the spec right - is quite hard to do with the current protocol. Even if you change the difficulty at the exact same moment as sending out a new work unit, the difficulty change applies to all the previous work units and miners are allowed to continue working on those at the new difficulty unless you invalidate all of them at the same time. Except that if you do that, you're making the miner throw away work they've done without submitting it. There's a mismatch between how the protocol wants you to handle difficulty changes and the way you actually need to handle them to prevent cheating by miners.

[-ck]

Variable difficulty -has- to adjust upon new work without being opened up to exploitation, regardless of protocol used.  It's very simple for a fast miner to withhold higher difficulty shares when it knows that it's going to force a difficulty increase, then submit them for higher credit once the adjustment lands.

Which - if I'm reading the spec right - is quite hard to do with the current protocol. Even if you change the difficulty at the exact same moment as sending out a new work unit, the difficulty change applies to all the previous work units and miners are allowed to continue working on those at the new difficulty unless you invalidate all of them at the same time. Except that if you do that, you're making the miner throw away work they've done without submitting it. There's a mismatch between how the protocol wants you to handle difficulty changes and the way you actually need to handle them to prevent cheating by miners.

btcguild is the only stratum pool with variable diff, and it  sends out a clean work with every difficulty change meaning all work gets thrown out with each diff change. Even blocks and valid shares at the new diff get thrown out I might add.

[kano]

Variable difficulty -has- to adjust upon new work without being opened up to exploitation, regardless of protocol used.  It's very simple for a fast miner to withhold higher difficulty shares when it knows that it's going to force a difficulty increase, then submit them for higher credit once the adjustment lands.

Which - if I'm reading the spec right - is quite hard to do with the current protocol. Even if you change the difficulty at the exact same moment as sending out a new work unit, the difficulty change applies to all the previous work units and miners are allowed to continue working on those at the new difficulty unless you invalidate all of them at the same time. Except that if you do that, you're making the miner throw away work they've done without submitting it. There's a mismatch between how the protocol wants you to handle difficulty changes and the way you actually need to handle them to prevent cheating by miners.

btcguild is the only stratum pool with variable diff, and it  sends out a clean work with every difficulty change meaning all work gets thrown out with each diff change. Even blocks and valid shares at the new diff get thrown out I might add.

Quite logically, the implementation should be that difficulty should be part of the work, however, previous work on the same block should also be allowed since it is to the miners disadvantage for the pool to throw away valid work ... obviously

This does, however, lead to the issue that pools may be silly enough to send a 1TH/s miner 1 difficulty work.
Yeah it seems that it is impossible for the pool software programmers to handle this properly ... the solution seems to be that it's the miners problem, not the pools problem, so the miner should lose work because of it ... yet the protocol clearly states that it's the pool that has absolute control of this.

I guess setting a time limit on how much old work is valid (yes there was a reason I made that comment at the end of the post: https://bitcointalk.org/index.php?topic=108533.msg1287015#msg1287015 ) and the pools will just have to put up with this time limit since they are unable to program a suitable solution without throwing away valid miners work.

Hmm - looking at the protocol in the light of this discussion, it really currently is a protocol that gives more power to the pool and takes it away from the miner.
The protocol, by definition, allows the pool to reject shares as it chooses and gives a way to do that, that the protocol defines as valid.
Giving the pool WAY less work to do, and passing that work reduction to the miner to do, yet the miner also loses rights about the pool accepting it's valid work ... hmm ... giving away rights and being made to do more work ... where have I heard that before

The point of this protocol was a long term solution, but with getwork using roll-n-time and high difficulty shares, it won't be necessary quite yet.
A 1.5TH/s device can do ~233 nonce ranges a second, so with the roll-n-time limit set to 7000 as it is in cgminer at the moment, that's 30s worth of work ... so there is still some (small? amount) of time left before people are forced to give away rights and do more work ... and in the mean time there may be a better solution ...

[DavinciJ15]

Well, python is structured and strictly object oriented. Maybe your criticism is because you can't follow control flow and don't understand it ;-). I'm not sure if you're specifically refering my python code or general python code. But I'm using asynchronous framework (Twisted), which *is* hard to learn and hard to read for beginners, but it is really powerful for some kind of tasks...

My bad.  I might as well interject into a conversation of Mensa members.  But "Twisted" is what I would call Python and I'm not the only one. Yet, I must agree you can do things in a very small amount of code in Python.  But I still think it bites.  (pun intended)

So the war is on.  VHS (Stratum) vs BetaMax (GBT)

It does not matter who is better, only the protocol that will be used will be the winner.

[Luke-Jr]

Well, python is structured and strictly object oriented. Maybe your criticism is because you can't follow control flow and don't understand it ;-). I'm not sure if you're specifically refering my python code or general python code. But I'm using asynchronous framework (Twisted), which *is* hard to learn and hard to read for beginners, but it is really powerful for some kind of tasks...

My bad.  I might as well interject into a conversation of Mensa members.  But "Twisted" is what I would call Python and I'm not the only one. Yet, I must agree you can do things in a very small amount of code in Python.  But I still think it bites.  (pun intended)

Well, Twisted deviates from normal Python code quite a bit - almost bad enough that it's really its own language.
Also, Twisted doesn't work with any current Python versions (3.x).

So the war is on.  VHS (Stratum) vs BetaMax (GBT)

It does not matter who is better, only the protocol that will be used will be the winner.

Hopefully slush's recent steps toward putting Stratum through the BIP process will result in a new protocol that combines the capabilities of both.

[DavinciJ15]

BIP process will result in a new protocol that combines the capabilities of both.

OK quick a quick and easy question...

Bitcoind 0.7 has GetBlockTemplate, does that mean I can use my BFL 1.5 TH Mini rig to solo mine using only bitcoind and BFGminer?
I want to underscore this one point, USING ONLY bitcoind and BFGminer, no proxy, no pool, and no other software.

NOTE: a yes, but... is a NO.

[Luke-Jr]

BIP process will result in a new protocol that combines the capabilities of both.

OK quick a quick and easy question...

Bitcoind 0.7 has GetBlockTemplate, does that mean I can use my BFL 1.5 TH Mini rig to solo mine using only bitcoind and BFGminer?
I want to underscore this one point, USING ONLY bitcoind and BFGminer, no proxy, no pool, and no other software.

NOTE: a yes, but... is a NO.

Yes, this is supported by a beta BFGMiner branch and should be merged before the BFGMiner 3.0 release (which will introduce ASIC support).

[optimator]

Quite logically, the implementation should be that difficulty should be part of the work, however, previous work on the same block should also be allowed since it is to the miners disadvantage for the pool to throw away valid work ... obviously

The reason this is an issue is because of the way the protocol has been implemented (not necessarily designed). Cleanjobs was designed to tell the miner to drop all work, presumably because a new block has been found. BTC Guild is pushing cleanjobs=true on a difficulty change.

There would not be an issue if difficulty changes were pushed with a new job (around every thirty seconds and with cleanjobs=false and with pools associating the difficulty of each connection with the job). It is trivial to track the associated difficulty of a connection and a new work request (with cleanjobs = false) and to ensure the work submitted matches the difficulty associated with the connection at the time the job was pushed.

A workable solution for the initial 30 seconds, which would not result in share loss, is to push connection specific jobs on difficulty changes for new connections.

Clearly miners benefit from efficient pools and pools benefit from efficient miners. Clearly, no one wants to lose a block because of a difficulty change!

Before we jump to the solution, I think we need to understand the problem. Losing shares is a problem for miners. Perhaps someone else can articulate the problem of difficulty changes between new jobs for the pools?

[Luke-Jr]

I've been looking over the python code which works pretty well actually, just need to add DB work.

I wish I could read python code, it's not structured code, it's not object oriented, it's not even assembly language.  I can't follow control flow, I can't understand variable usage, nothing.  I could learn it but, I don't believe computer languages should be harder to use for all developers at all levels.  This is my opinion and I understand I may be wrong, I believe I'm right.  

So why do you guys use Python?

Python is object-oriented procedural code. Twisted twists it into something unreadable.
I can't speak for those who use Twisted, but I use Python generally to make it more accessible to less experienced developers.

[DavinciJ15]

I've been looking over the python code which works pretty well actually, just need to add DB work.

I wish I could read python code, it's not structured code, it's not object oriented, it's not even assembly language.  I can't follow control flow, I can't understand variable usage, nothing.  I could learn it but, I don't believe computer languages should be harder to use for all developers at all levels.  This is my opinion and I understand I may be wrong, I believe I'm right.  

So why do you guys use Python?

Python is object-oriented procedural code. Twisted twists it into something unreadable.
I can't speak for those who use Twisted, but I use Python generally to make it more accessible to less experienced developers.

Thanks for clearing that up! I may check out regular python then.