Unified miners communication protocol

[kano]

Shoe horn? Lulz - got one example of that?

I mine 8 different devices on my cgminer instances ...
Icarus, AsicMinerUSB, BitBurner, Jalapeno, SC Single, BlackArrow Lancelot, BFL FPGA, ModMinerQuad
(I'm getting a Cairnsmore1 shortly also and probably a Klondike soon too)

Still can't see the shoe horn there anywhere.

[kano]

That's why VersionString was introduced as required field:
 Manufacturers will implement only Work and gotNonce, because that's all they care about, but ...
 Developers and users from the community will add more and more actions and logic behind them and if at at least some of the optional commands are available as direct translation to some registers based on EngineID translation - the software will be able to manage the hardware based on the VersionString, no matter how old is the driver implementation used from the user.

or at least that's the idea and end result i am aiming at

OK - old idea - even BFL have that in their current SC line ...

I asked BitBurner to add that to theirs and they did.

But you've already missed something here ... when defining a 'one ring to rule them all' protocol, one version should be able to support different configurations, so version is not complete.
Again e.g. BFL SC firmware can enable and disable different options, so not only version would be required, but also a configuration response (like the BFL SC GetInfo)

So ... I'd ask directly: what is the point of a defined interface?
Best one I've seen so far is the Klondike - it doesn't handle everything needed for other boards, but for it's own purposes it's well designed.

The protocol interface is the easiest part of most devices, handling the hardware requirements is where the trouble starts
A different one each time is no issue at all IMO

[optimiz3]

Protocol design affects speed, and because speed is the #1 optimization criteria, it is unlikely there will ever be a standard hardware protocol.

Each ASIC has unique performance characteristics that are in part determined by what is computed off-chip (queuing, pre-compute, scheduling).

Advancements that affect what is computed off-chip may be non-obvious or complex, to the point that implementing a standard interface in the micro-controller firmware would drive up costs (i.e. more costly micro-controllers or FPGAs).

Thus, protocol complexity gets pushed to the host driver, where it lives now.

[KNK]

Thus, protocol complexity gets pushed to the host driver, where it lives now.

Agree and this is the reason i am using the word 'driver'.

The communication can be implemented in the microcontroller if it has enough power or on the host and only pass simple instructions to the hardware or by directly accessing internal registers.
When implemented as a kernel driver it will be able to better handle the communication to hardware when it is time sensitive and provide a socket in /var or /sys for the communication.
For Windows or when the microcontroller is powerfull enough a network socket can be used locally or remotely.

The protocol interface is the easiest part of most devices, handling the hardware requirements is where the trouble starts

And it is exactly the opposite for the hardware people
So why not to separate the two, but leave a wide bridge between?

One of the problems i see is that for a manufacturer it will be difficult to follow (if at all) mining software development and to back it up with the words of ckolivas from https://bitcointalk.org/index.php?topic=228677.msg3017560#msg3017560
"Oh and it's not "the current cgminer", it's the forked code based on an older version."
What about the miner in case a (security) bug is discovered in that older version several versions later, which is then fixed, but is missing the manufacturer's code and for him it will mean starting with a new fork?

By separating them, a fix or improvement in the mining software will not affect custom hardware module and the opposite an updated module will not require downloading and recompiling of new mining software.

[Luke-Jr]

One of the problems i see is that for a manufacturer it will be difficult to follow (if at all) mining software development and to back it up with the words of ckolivas from https://bitcointalk.org/index.php?topic=228677.msg3017560#msg3017560
"Oh and it's not "the current cgminer", it's the forked code based on an older version."
What about the miner in case a (security) bug is discovered in that older version several versions later, which is then fixed, but is missing the manufacturer's code and for him it will mean starting with a new fork?

By separating them, a fix or improvement in the mining software will not affect custom hardware module and the opposite an updated module will not require downloading and recompiling of new mining software.

I've always worked directly with manufacturers, often writing the code myself, so this isn't a real problem for BFGMiner.

[KNK]

I've always worked directly with manufacturers, often writing the code myself, so this isn't a real problem for BFGMiner.

And here comes the second problem - someone should do the same for CGminer and then someone else for EasyMiner and then for BitMinter etc.
Managing dozens of hardware and software versions is hard if not impossible and the end user (miner) will be left with no choice, but to use a specific miner and version, because of his hardware.
I hope you will agree that there is a need in such protocol at least because of this and i aim to design it as flexible as possible.

[Luke-Jr]

I've always worked directly with manufacturers, often writing the code myself, so this isn't a real problem for BFGMiner.

And here comes the second problem - someone should do the same for CGminer and then someone else for EasyMiner and then for BitMinter etc.
Managing dozens of hardware and software versions is hard if not impossible and the end user (miner) will be left with no choice, but to use a specific miner and version, because of his hardware.
I hope you will agree that there is a need in such protocol at least because of this and i aim to design it as flexible as possible.

Other software is welcome to adopt a compatible driver API. BitMinter is Java, so that makes some difficulty of course.
(EasyMiner is a BFGMiner GUI)

Unfortunately, cgminer wants to reinvent the drivers themselves for no gain. It's called not-invented-here syndrome.

[kano]

I've always worked directly with manufacturers, often writing the code myself, so this isn't a real problem for BFGMiner.

And here comes the second problem - someone should do the same for CGminer and then someone else for EasyMiner and then for BitMinter etc.
Managing dozens of hardware and software versions is hard if not impossible and the end user (miner) will be left with no choice, but to use a specific miner and version, because of his hardware.
I hope you will agree that there is a need in such protocol at least because of this and i aim to design it as flexible as possible.

Other software is welcome to adopt a compatible driver API. BitMinter is Java, so that makes some difficulty of course.
(EasyMiner is a BFGMiner GUI)

Unfortunately, cgminer wants to reinvent the drivers themselves for no gain. It's called not-invented-here syndrome.

Sigh - get over it.
There are drivers that MUST use libusb ... even in your crappy clone there are libusb drivers.
How on earth is that reinventing, when all the drivers in cgminer use the same libusb.
Get a life and learn how to program intelligently rather than flipping burgers at McDonalds.

[KNK]

Please, stop both!
Lets not make this tread a flame war, but focus on the subject.

[KNK]

But you've already missed something here ... when defining a 'one ring to rule them all' protocol, one version should be able to support different configurations, so version is not complete.
Again e.g. BFL SC firmware can enable and disable different options, so not only version would be required, but also a configuration response (like the BFL SC GetInfo)

So ... I'd ask directly: what is the point of a defined interface?

Not missed, but will I succeed is another question ...
"to support different configurations, so version is not complete" - trying to define it the way that even if "_list_of_supported_actions" are improperly returned there are means to correct the result based on actual responses and not expect any specific response.
"not only version would be required, but also a configuration response (like the BFL SC GetInfo)" - on my opinion all the info should be available as separate commands (and their answers), not as a bunch of text. This way, in case we change the frequency/voltage of chip X, but the driver has reset it back on it's own (as protection masure or other) - there is no need to get all info again, but ask for "GetFreq,EngineID" and get a single number back, without occupying the communication channel (and limited power MCU) for a long time.
"what is the point of a defined interface" - for an easy jump start by implementing the bare minimum and still allowing future extensions without reinventing the wheel

[KNK]

IMO there is no real value for asking the device for its current hashrate.

How is the number of jobs it needs calculated then?
I mean when we have just started and discovered X unknown devices from unknown manufacturer (otherwise we just won't ask)
Also if there are 500 chips and the hardware keeps an eye on them (to know when to pool them), so it has that detailed info - the MS may only keep the overall hashrate and ask.

It might be nice if the protocol were prepared to handle non-Bitcoin POW algorithms too, but this could probably be done as an extension.

What about ShaWork/ShaNonce in replacement for DoWork/GotNonce and then ScptWork/ScptNonce etc. as each may need it's own parameters?

[Luke-Jr]

IMO there is no real value for asking the device for its current hashrate.

How is the number of jobs it needs calculated then?
I mean when we have just started and discovered X unknown devices from unknown manufacturer (otherwise we just won't ask)
Also if there are 500 chips and the hardware keeps an eye on them (to know when to pool them), so it has that detailed info - the MS may only keep the overall hashrate and ask.

As long as you're having the device request work, might as well have it specify how many it needs.

It might be nice if the protocol were prepared to handle non-Bitcoin POW algorithms too, but this could probably be done as an extension.

What about ShaWork/ShaNonce in replacement for DoWork/GotNonce and then ScptWork/ScptNonce etc. as each may need it's own parameters?

There are a lot of parameters for both SHA256d/scrypt/others that are currently taken forgranted.

Of more important note, I think I forgot to mention: There should probably be a single command to cancel all pending jobs ASAP, for use on block changes.

[cscape]

How is the number of jobs it needs calculated then? I mean when we have just started and discovered X unknown devices from unknown manufacturer (otherwise we just won't ask)

If you notice that returned timestamps are increasing too fast, send work faster.

[KNK]

As long as you're having the device request work, might as well have it specify how many it needs.

If NeedWork is implemented - yes, there is. If not implemented, but hashrate is - the predefined fixed value may be a good start and much more easier to implement as fixed number answer.

There should probably be a single command to cancel all pending jobs ASAP, for use on block changes.

There is - CancelJobs and it is recommended for implementation if the hardware supports it.

If you notice that returned timestamps are increasing too fast, send work faster.

Example: You have 500 BF chips and you are given a single work (be it with MTime 300), but don't have nonce range division (HW limitation) and don't have NeedWork implemented. When you start you wait 1.2sec (with 200 chips idle), before sending nonces back. MS sees them coming fast and sends more, but still for 300 + a bit more until your full hashrate is reached. If GotHashrate is implemented and you return a fixed value 500x3GH - you will be given enough work from the first command.

Having either NeedWork or GotHasrate implemented will speed-up the start, while GotHasrate is easier to implement and if fully implemented gives detailed information, which the MS may not need for it's normal operation, but can be provided to a customised (for the hardware) web interface for example.

[Trongersoll]

I would suggest that the best way to go about this would be to draft a document and let everyone pick it apart. Of course getting a concensus is almost impossible. When all is said and done, the important part would be getting the hardware people on board. Once they are in agreement, the software people may bitch, but they still need to support the hardware or someone else will.

[kakobrekla]

Omg, this is still going?

[gmaxwell]

Of more important note, I think I forgot to mention: There should probably be a single command to cancel all pending jobs ASAP, for use on block changes.

There are different kinds of cancellations however.  It's useful to be able to distinguish "really, don't bother, anything more you have queued will be completely worthless, cancel even if its a pipeline flush that loses work"  from "I have new work for you to start on as soon as you can without disrupting the pipeline".  ... The latter you should be able to call as fast as the communication bus allows without losing hashrate as a result.

[cscape]

Example: You have 500 BF chips and you are given a single work (be it with MTime 300), but don't have nonce range division (HW limitation) and don't have NeedWork implemented. When you start you wait 1.2sec (with 200 chips idle), before sending nonces back. MS sees them coming fast and sends more, but still for 300 + a bit more until your full hashrate is reached.

So the first 5 seconds you run at half speed. Not a big deal over the lifetime of the equipment. And with 500 BF chips, you should have a better solution than running getwork to a single host anyway.

[KNK]

The draft is almost complete - there is something that bothers me with the debug errors command, which i'd like to think some more about, but the rest should be OK.
A review of the wording from a native speaker would be welcome as well as any other suggestions.
It will be great if some of the MS developers can provide a skeleton implementation for example with CPU mining or some other hardware

[kakobrekla]

The draft is almost complete - there is something that bothers me with the debug errors command, which i'd like to think some more about, but the rest should be OK.
A review of the wording from a native speaker would be welcome as well as any other suggestions.
It will be great if some of the MS developers can provide a skeleton implementation for example with CPU mining or some other hardware

The fact remains, no one cares and no one will implement it, move along.