Pool server auto-configuration: /config.json

[jgarzik]

Currently, pool server extensions such as long-polling provide static configuration information -- the long-polling URL -- via a special HTTP header.  This is sub-optimal, because the long-polling URL is unlikely to change, yet many pool servers utilize the simple approach and send the same LP URL to each mining client, over and over again, wasting bandwidth.

A server list, used in mining client fail-over (something I have been pushing for), would be another example of static configuration information communicated from pool server to client.

Therefore, I began updating cpuminer to download $POOL_SERVER/config.json.  This will look something like

Code:

{ "name" : "slush",
    "description" : "slush's super-big pool",
   "longpoll" : "/LP",
   "ntime" : true,
   "server_list" : [ { "url" : "http://srv1.net" }, { "url":"http://srv2.net" } ]
   "xslush" : {
       "my server-specific option #1" : "my value",
       "my server-specific option #2" : "my value"
   }
}

This should give pool servers a lot more flexibility in defining optional features and metadata, without needing to transmit that information redundantly in every single 'getwork' HTTP header.

[1713560462]

1713560462

[1713560462]

1713560462

[1713560462]

1713560462

[1713560462]

1713560462

[martok]

I like this idea though what is the extended server info (xslush) for since it can't be supported by the client directly? Perhaps defining the base config well enough such that server specifics aren't necessary.

Also, we need a way for the configs to refresh. Either a refresh : seconds option in said config which causes the client to send a HEAD request for the config to see if it has changed or some notification where a server can push the new config out to clients via say an x-config-refresh header.

[jgarzik]

I like this idea though what is the extended server info (xslush) for since it can't be supported by the client directly? Perhaps defining the base config well enough such that server specifics aren't necessary.

"xslush" is an example of a configuration that is site-specific, and not standard.

Anything with the "x" prefix, at the top level, is fair game for pool servers to define whatever they want, while the other items are those standardized by the community of mining clients.

Also, we need a way for the configs to refresh. Either a refresh : seconds option in said config which causes the client to send a HEAD request for the config to see if it has changed or some notification where a server can push the new config out to clients via say an x-config-refresh header.

Yep...

[dbitcoin]

I like this idea though what is the extended server info (xslush) for since it can't be supported by the client directly? Perhaps defining the base config well enough such that server specifics aren't necessary.

"xslush" is an example of a configuration that is site-specific, and not standard.

Anything with the "x" prefix, at the top level, is fair game for pool servers to define whatever they want, while the other items are those standardized by the community of mining clients.

Also, we need a way for the configs to refresh. Either a refresh : seconds option in said config which causes the client to send a HEAD request for the config to see if it has changed or some notification where a server can push the new config out to clients via say an x-config-refresh header.

Yep...

Overall this is good idea, but not for static long polling url.
I use unique urls for each miner.

[[Tycho]]

• Adding this config probably will make miner programming more difficult because it would require additional actions to be performed. How many clients will correctly understand headers that define cache control and expiration time ?
• You'll need to parse headers anyway for "force failover" or "update config" commands from server to be detected and acted upon. So all the clients will have to BOTH read config file AND check headers every time. Keep It Simple principle is broken here.
• I think it's a solution for non-existent problem. Traffic loss is negligible, it would be strange to care about tens of bytes while still sending all the other headers back and forth, not to mention the text-mode data packet instead of compact binary.
  Let's assume a fast GPU with 10 seconds period to hash all the nonces and 80 bytes additional header for server list. This will be ~28 Kb per hour, not really something to worry about.

[xf2_org]

Overall this is good idea, but not for static long polling url.
I use unique urls for each miner.

Your server is free to return a per-miner LP URL, when the miner requests /config.json upon startup (or when config.json is updated).

[this is jgarzik, using a new forum nick]

[slush]

Must agree with Tycho and dbitcoin that it brings unneeded complexity. With extra headers in http requests the miner and pool code can be much more straightforward and those few extra bytes are not real problem.

[xf2_org]

Must agree with Tycho and dbitcoin that it brings unneeded complexity. With extra headers in http requests the miner and pool code can be much more straightforward and those few extra bytes are not real problem.

I disagree; It is not straightforward in the client.

[[Tycho]]

My protocol extension proposal:

http://deepbit.net/failover.php
i'm going to implement it in my pool soon.

Comments and advices are welcome :)

[slush]

Tycho, I like the switch feature, I plan something similar for my pool, so unified API is good thing (tm). But I don't feel that list of failover servers is needed in every getwork request. And I think it isn't needed at all (at least for me).

Currently I have implemented load balancer between getwork nodes and I'm planning to add IP failover of this balancer, so I don't need server failover feature in mining clients at all...

[xf2_org]

But I don't feel that list of failover servers is needed in every getwork request.

Thank you.

In general, static, unchanging information should not be sent repeatedly to mining clients with every getwork.

[slush]

In general, static, unchanging information should not be sent repeatedly to mining clients with every getwork.

Agree. Unfortunately not sending some information in every requests leads to some refreshing and client side cache (because those information are not absolutely static; pool can connect new servers and want to use them in oncoming upgrade). That's why I don't like both solution (failover list in every getwork and static json config).

This is why current "protocol" based on http is screwed. We _can_ implement config updates using LP channel, but all this is pretty ugly and complicated. Better than patching current "protocol" in all pool servers and all miners, I vote for discussion about message based TCP based protocol which can solve all current workarounds (I see LP as workaround, too) with some better 'final' solution.

We already had a discussion about TCP protocol with jgarzik and we didn't found an agreement on details (mainly in message structure itself), but I hope that finding wider compromise (pool operators & miner developers) is the only way how to get things better.

[xf2_org]

Agree. Unfortunately not sending some information in every requests leads to some refreshing and client side cache (because those information are not absolutely static; pool can connect new servers and want to use them in oncoming upgrade). That's why I don't like both solution (failover list in every getwork and static json config).

Easy solution:  consolidate all information into a single "X-Config-Last-Modified" header.  When that changes, client refreshes config.json.

Though I agree this is and long-polling are both a hack, and a better protocol is needed.

[dbitcoin]

I agree what we need new protocol, but we should use another port for this, and support both protocols for a long time.
I still see some miners use old version of poclbm without LP support with fast gpu.
Despite all of my announcements and mailings about software update.