Unified miners communication protocol

[KNK]

There are handful of developers on BFL chips and over a dozen for Avalon and BitFury (then add more new chips and designs in the near future). Each hardware requires it's own modifications and the end result is several forks of the mining software, which will be a nightmare for the average miner, who is wiling to use more than one hardware and choosing the correct version for it's device(s) - that's why i would like to suggest unified communication protocol between the hardware and the mining software, so each manufacturer will need to provide a small driver for their device which the mining software can use without modifications.

I would like to invite mining software and hardware developers to discuss the inners of such protocol, which we will all benefit from.

The current version of the protocol draft can be found here:
https://docs.google.com/document/d/1-5lWeABZ3SjnUuXvTNUx2U7K10ZFhKWSiqIfnmKT7TQ/

[KNK]

For the communication method i think the best will be to use a socket communication, because it will allow connecting to remote hardware over the network too. Json can be used again as it is already present for the pools communication and configs.

The mining software (MS for short) should contain configuration options for the hardware's address and port, then user and pass just like for the pools, but the authentication is optional for the mining hardware (MH).

MH should listen to a TCP socket, but is free to communicate with the actual hardware over any other interface: USB, SPI, UART, GPIO

There should be two types of hardware - pooled at some interval and those able to request new jobs when ready.

When MS authenticates to MH - MH send's it's available options with the AuthOK response

Some options in mind:
 ShowCores - how many actual hashing chips are available - array with option to mix versions, i.e. 5 chips V1 @ 1TH and 10 chips V2 @ 50TH
 ShowClocks - working frequency for each chip
 ShowHashrates - for each chip
 ShowTemps - an array of available temperatures with option to send only values or with description
 ShowVoltages - similar to temp

[cscape]

If you want to make it attractive, it needs to be as simple as possible.

Forget JSON, and just use CR-LF terminated lines, for instance. Start with 2 essential commands: one to send work to the hardware, and one to get the nonces. Make the rest optional.

By keeping it simple, it is possible to implement all of it on a cheap microcontroller on a USB stick, for instance.

[KNK]

By keeping it simple, it is possible to implement all of it on a cheap microcontroller on a USB stick, for instance.

That was my initial idea to, but then it will be very limited in functionality and suitable for just some of the hardware, without an option for monitoring. Then i came with the idea of running the driver on the same machine (over linux socket) and leave it to the driver to communicate with the hardware, so your suggestion will be just a trimmed down driver implementation. The driver can also be implemented as a kernel module for the Linux running embedded devices (like RasPI) and provide it's socket in sys

Replacing JSON with CSV format is a good idea, as it will be easier for parsing with cheap microcontrollers with ethernet

[-ck]

Good luck. You're not the first to try. Every hardware manufacturer thinks they know best and either adopt the worst protocol, or invent their own, completely ignoring from the lessons and mistakes of other manufacturers, and ignoring what us devs have to say.

[KNK]

Good luck. You're not the first to try. Every hardware manufacturer thinks they know best and either adopt the worst protocol, or invent their own, completely ignoring from the lessons and mistakes of other manufacturers, and ignoring what us devs have to say.

That's why i made this thread and would like to hear from both devs and manufacturers - who and what needs.
Let's not limit the protocol to the current hardware and if you as cgminer developer can provide the initial support for the protocol in it, i am sure the manufacturers will add more bells and whistles ... having the requirements here in advance will help both sides and most important will make the life easier for the average miner.

A am also working on a hardware and the easiest thing for me will be to add another fork and modify the code the way i want, but it will put more value in any hardware if the end user is able to update or replace to it's favorite mining software without any risk, so i am sure many manufacturers will start using it when available

[kano]

If you want to make it attractive, it needs to be as simple as possible.

Forget JSON, and just use CR-LF terminated lines, for instance. Start with 2 essential commands: one to send work to the hardware, and one to get the nonces. Make the rest optional.

By keeping it simple, it is possible to implement all of it on a cheap microcontroller on a USB stick, for instance.

The command list is IMO pretty much irrelevant.
The issues are simply USB and hardware design considerations.

How to send commands?
Seriously? People think that is an issue?

I was going to start a thread on this USB design issue a couple of weeks ago but have as yet not found to time to spend an hour or so writing up the first post.
I'll get around to it in the next week or two.

Having a different driver is no real issue.
Getting the hardware design right marks the difference between them.

TBC ...

[cscape]

The command list is IMO pretty much irrelevant.
The issues are simply USB and hardware design considerations.

How to send commands?
Seriously? People think that is an issue?

Our USB device has a simple bulk endpoint, implementing a virtual com port. The rest is a matter of sending commands over it. What other "USB and hardware design issue" is there ?

[Luke-Jr]

TheSeven has a draft protocol for something like this that he might perhaps contribute.
But realistically, I don't think vendors are going to use the same protocol.
There are also practical problems due to how these devices work internally - any protocol that superset the behaviour they all need would be overly complicated and inevitably fail to address the needs of a future device.

Perhaps a binary protocol similar to stratum could work for future devices.
This requires much more memory, processing power, and testing (in theoretical worst-case scenarios, NOT simply mining bitcoin!) than current MCUs have.

[KNK]

Just added a link to the initial version in the first post.
The idea is to add as optional commands (at least) the temperatures and fan speeds with EngineID's just like for the GetHashrate, but if there is a preferred method for identification it should be the same for all commands, so will wait for some comments first, not to change many times one and same description.

[cscape]

The way I have it implemented in my USB firmware is:

work job_id block_header

The block header includes the time stamp, and the hardware is allowed to increment it. The job_id is just a number. There is no nonce range (hardware always searches full range), ntime rolling is always permitted, difficulty is set through separate command but I like your idea of including it in the work command. I don't really see a need for a nonce range. With fast hardware, the entire 32 bit nonce range is searched in just a few seconds max, and with slow hardware (what's the point ?) you can always cancel old job when a new one comes in.

To submit a valid nonce, the firmware responds with

submit nonce job_id timestamp

I have no 'needwork' command. The idea is that the host provides work quickly enough to satisfy the hardware. In case the work doesn't come quickly enough, the hardware starts incrementing the timestamp. So, if the host does not want to accept ntime rolling, it is responsible for supplying fresh work before the hardware needs it. 

I also don't have cancel job mechanism. The hardware detects that prevhash field has changed, and automatically cancels old work as quickly as it can. Of course, it doesn't hurt to include an explicit cancel command. Another option would be to include a cancel flag in the 'work' command, which is nice because when you cancel old jobs you must provide new work at the same time, otherwise the hardware has nothing to do.

[KNK]

The recommended and optional commands you don't need may not be implemented - that's why they are optional and so are all the [optional] parameters ... just skip them if you don't want them, but don't return a parsing error, because you don't expect them ... that's all

cscape your use case perfectly fits the current draft without the optional parameters, so this is good. Please note, that the idea is not to limit it to the current hardware only, but to be able to support future hardware and its oddities. A good example would be that BF chips do not scan the entire range, but you may have another hardware, which is able to scan the rest, so nonce range is useful option.

about "submit nonce job_id timestamp" - it will be weird to have job_id as first parameter in work, but as a second one for the result, so if you swap them it will be exactly as described.

I am thinking of having each command (except Ping/Pong) as two words in CamelCase, so I will probably replace Auth to DoAuth and Work to DoWork

If you receive an unknown command - you should simply ignore the input until the end of the line i.e. CR+LF and that's all

cancel flag with the work is not a good idea as it will limit the possibility for optional parameters if at the end or will be just one more parameter to parse at the beginning, which is 0 in 99.99% of the commands and 1 just once in ~10min

[cscape]

cscape your use case perfectly fits the current draft without the optional parameters, so this is good. Please note, that the idea is not to limit it to the current hardware only, but to be able to support future hardware and its oddities. A good example would be that BF chips do not scan the entire range, but you may have another hardware, which is able to scan the rest, so nonce range is useful option.

Why would you care about scanning the rest of the nonces ? It's much easier to ignore that part of the nonce space, and create some new work instead.

[KNK]

Let's say tomorrow we have a new hardware (single chip), which scans just half of the nonces, but is able to scan each one of them if asked ... why to calculate twice the block header (new work) and waste cpu time, when you can simply pass the same one to two chips?

[cscape]

why to calculate twice the block header (new work) and waste cpu time, when you can simply pass the same one to two chips?

With partial scans, you just introduce a lot of extra complexity. Before you can use it, you need to know if the hardware has options to set the nonce range, and you need to know the exact limitation of the hardware. For instance, the bitfury chips only scan 756/1024 of the nonce range, but the hole is not contiguous, and not user-settable. To match this limitation with other hardware is very complicated. Even if you wanted to try, it requires extra commands to obtain the information from the hardware.

To generate some new work is simple. You just call the code you already have, except a little more frequently. Sure, it wastes some CPU time, but I'd like to see that this is a problem before creating a solution.

[nwoolls]

Let's say tomorrow we have a new hardware (single chip), which scans just half of the nonces, but is able to scan each one of them if asked ... why to calculate twice the block header (new work) and waste cpu time, when you can simply pass the same one to two chips?

https://en.wikipedia.org/wiki/You_aren't_gonna_need_it

[KNK]

@ nwoolls that was just an example and the nonce range balancing will be implemented when needed, but the protocol should allow that in a backward compatible way.
BFL firmware is able to accept/scan partial range, so by not having this as an option for the protocol would be a limitation for some mining software which decides to use it with a hardware that has it.

[Luke-Jr]

Why would you care about scanning the rest of the nonces ? It's much easier to ignore that part of the nonce space, and create some new work instead.

Why stop scanning at all? There's no benefit to ignore the rest of the nonce space.

[cscape]

Why stop scanning at all? There's no benefit to ignore the rest of the nonce space.

It's a limitation of the hardware. The bitfury chips have 756 separate hashing cores, all working in parallel, with each core hardwired to search a fixed 1/1024th of the nonce space.

[KNK]

There's no benefit to ignore the rest of the nonce space.

+1 There is actually a drawback - you may miss a valid solution. It may not be important if you only care about your shares from the pool, which are always paid, but as a solo miner I DO CARE if i miss my chance, because of that

It's a limitation of the hardware. The bitfury chips have 756 separate hashing cores, all working in parallel, with each core hardwired to search a fixed 1/1024th of the nonce space.

The protocol draft is not for BF chips only and not even for current hardware only. Once again: if you don't use it - don't implement it (simply ignore) and the protocol should allow that in both cases i.e. being able to serve and keep communication even in case of unknown action or parameter.