CKPOOL - Open source pool/proxy/passthrough/redirector/library in c for Linux

[-ck]

CKPOOL + libckpool by the cgminer development team.

Ultra low overhead massively scalable multi-process, multi-threaded modular
bitcoin mining pool, proxy, passthrough, library and database interface in c
for Linux.

ckpool code by ckolivas / -ck
ckdb code by kano

Note that as maintainer of ckpool am FOR segwit and AGAINST Emergent Consensus.

Source code downloads:

https://bitbucket.org/ckolivas/ckpool

Current release status: Beta v0.94, from git source only.

IRC channel: irc://irc.freenode.net/ckpool

LICENSE: GNU Public license V3. See included COPYING for details.

About ckpool: https://bitcointalk.org/index.php?topic=790323.msg8907250#msg8907250

About ckdb: https://bitcointalk.org/index.php?topic=790323.msg8907583#msg8907583

About ckproxy: https://bitcointalk.org/index.php?topic=790323.msg8907589#msg8907589

About libckpool: https://bitcointalk.org/index.php?topic=790323.msg8907597#msg8907597

About ckpmsg: https://bitcointalk.org/index.php?topic=790323.msg8907597#msg8907597

About notifier: https://bitcointalk.org/index.php?topic=790323.msg8907597#msg8907597

Building: https://bitcointalk.org/index.php?topic=790323.msg8907600#msg8907600

Configuration: https://bitcointalk.org/index.php?topic=790323.msg8907601#msg8907601

Stratum extensions: https://bitcointalk.org/index.php?topic=790323.msg8907604#msg8907604

FAQs: https://bitcointalk.org/index.php?topic=790323.msg8908863#msg8908863

Discussion/Future directions: https://bitcointalk.org/index.php?topic=790323.msg8908866#msg8908866



List of current ckpools:

http://www.kano.is
http://hashmine.io (private for now)
http://solo.ckpool.org (anonymous solo mining)
http://testnet.ckpool.org (testnet only)

[-ck]

About ckpool

Ckpool is a project by the cgminer development team designed to complete the circle between miners and pools from cgminer to ckpool. The bulk of the code work for this was commissioned and paid for, however I provided an incentive to the group commissioning the code to open the software up to all by charging less for free software than for closed code. Ultimately this should benefit bitcoin far more than me simply working on code for another pool. I intentionally chose to write most of the code from the ground up and not use external libraries to have complete control over all the code given past experiences of difficulty to debug failures in library code that was outside our domain with cgminer. The separation of the pool code from the database code allows the pool code to be ultra-scalable and extremely low overhead by dissociating any concept of share counts, user accounts, authentication, front ends, payschemes etc. etc. These can be as vastly different from the envisioned ckdb+postgresql+php interface as the administrator desires without any change to the core pool functionality.


Rationale:

The aim of this project is to provide the most low overhead, massively scalable pool software available by creating the code from the ground up using the same programming techniques and tools as those used when the author codes for the linux kernel. It was decided early on that to speed development and maintain consistent behaviour that only Linux would be supported. Even with massive numbers of miners, the core ckpool code uses very little CPU and only 100s of megabytes of ram.

We aim to remove the dependency of pool owners on custom software and the requirement for massive hardware infrastructure to run a stable and performant mining pool, thus dramatically lowering the barrier for entry for more pools to be created. In the process, we also aim to be providing tools to allow large mining farms to mine for themselves instead of directing their power to a large pool which would otherwise make them even larger and risking the security of the bitcoin network. While it may seem counter to the small miner's benefit to make large miners capable of running their own pool, it is an acceptance of where the bulk of the mining power is coming from and not make that mining power a risk to bitcoin. By providing them with software to mine in a manner that isn't designed to just mine blocks quickly but does so in a way that can lay down all transactions currently in flight without detriment to their block generating capacity, this will benefit the bitcoin network far more than them adding their hashes to a large pool.

Despite the pool software being provided to large miners to run standalone, the bulk of the development effort was actually directed to providing a reliable consistent and performing pooled mining solution (in the traditional pooled mining sense) with a stratum mining module at the core of the miner's experience that is not hindered in its ability to serve any sized miners with expedience and to create a platform for expanding on the existing stratum design as the need arises. As the core is modular in design, there is scope for plugging in extra components in the future as desired. Also as the same code is used by the pool software, the optional proxy included, the associated library, and the developers maintain the code at both ends, there is assurance the two ends will talk optimally.


DESIGN:

Architecture:
- Low level hand coded architecture relying on minimal outside libraries beyond
basic glibc functions for maximum flexibility and minimal overhead that can be
built and deployed on any Linux installation.
- Multiprocess+multithreaded design to scale to massive deployments and
capitalise on modern multicore/multithread CPU designs.
- Minimal memory overhead.
- Utilises ultra reliable unix sockets for communication with dependent
processes.
- Modular code design to streamline further development.
- Standalone library code that can be utilised independently of ckpool.
- Same code can be deployed in many different modes designed to talk to each
other on the same machine, local lan or remote internet locations.


Modes of deployment:
- Comprehensive pooled mining solution with a postgresql database interface.
- Passthrough node(s) that combine connections to a single socket which can
be used to scale to millions of clients and allow the main pool to be isolated
from direct communication with clients.
- Proxy nodes with a database that act as a single client to the upstream pool
while storing full client data of their own.
- Simple proxy without the limitations of hashrate inherent in other proxy
solutions when talking to ckpool.
- Simple pool without a database.
- Library for use by other software.


Features:
- Bitcoind communication to unmodified bitcoind with multiple failover to local
or remote locations.
- Local pool instance worker limited only by operating system resources and
can be made virtually limitless through use of multiple downstream passthrough
nodes.
- Proxy and passthrough modes can set up multiple failover upstream pools.
- Optional share logging.
- Virtually seamless restarts for upgrades through socket handover from exiting
instances to new starting instance.
- Configurable custom coinbase signature.
- Configurable instant starting and minimum difficulty.
- Rapid vardiff adjustment with stable unlimited maximum difficulty handling.
- New work generation on block changes incorporate full bitcoind transaction
set without delay or requiring to send transactionless work to miners thereby
providing the best bitcoin network support and rewarding miners with the most
transaction fees.
- Event driven communication based on communication readiness preventing
slow communicating clients from delaying low latency ones.
- Stratum messaging system to running clients.
- Accurate pool and per client statistics.
- Multiple named instances can be run concurrently on the same machine.


Processes:
When ckpool is run, a number of separate processes are spawned in order to capitalise on multi-core CPUs by offloading dedicated tasks to one process at a time in order to not allow any one process to slow any other. They talk to each other using unix sockets which by only talking on the local system cannot be influenced by the outside world.

The processes generated are:

The main process, AKA listener - this is the core process that listens for incoming requests from an admin socket and watches the running of the child processes.

The generator - this process is designed to only talk to bitcoind to generate work templates, check the current block hashes, check bitcoin addresses, submit block solves, or in proxy mode talks to an upstream stratum server to emulate generation of work. This also runs a watchdog thread to check the running of the upstream bitcoind/or upstream stratum pool and can perform failover should the bitcoind fail for whatever reason.

The connector - this process accepts incoming connections from miners, handling requests for new worker connections, receiving and sending data to the miners. It does this in an event driven manner with multiple send/receive/listen threads, automatically prioritising sends to downstream miners according to their readiness to receive data, thus preventing any one worker affecting any others.

The stratifier - this is the core stratum code implementation that receives data from the connector and runs multiple threads to create work templates, process incoming data, check for valid work, queue outgoing data, authenticate new users, and optionally talks to the ckdb process to store all manner of data for running a full database based pool.

Each process runs additional threads to perform logging in a queued manner that does not hinder the underlying running code.


Advantages to miners:

A stratum implementation with extremely rapid and accurate adjustment of diff to an optimal value both up and down in diff per worker.
Once diff has stabilised, it is extremely rare for diff to be adjusted again unless you add or remove hashes.
Diff is actually properly associated with work templates which means you don't "leak" rejected low diff shares when the diff changes.
Stable diff levels and proper work association with diff leads to lower rates of rejects.
Very rapid block propagation support to all miners means all miners get notification of, and work on, the latest block work as quickly as possible to not waste work.
No extra work restart messages for the inefficient zero-transaction work that other pool software means much less lost work.
Hashrates should hold much more stable at higher levels as a result of all these benefits.
Reconnect support: If you have temporary glitches in your networking, ckpool supports resuming and accepting of valid cached shares.
Messaging: Miners watching their console will see stratum messages as sent to them by the pool (such as block solves).
Virtually no downtime for pool restart: Pool restarts only register as an interruption and then reconnect instead of failing over to backup pools due to the unique socket handover mechanism in use.
Rewards: Maximum rewards are guaranteed by incorporating transactions into every work template instead of mining transaction-less blocks.
Confidence: Improving the bitcoin network through seamless transaction processing increases the confidence in bitcoin in general allowing expansion of the bitcoin network in terms of numbers of transactions and block sizes.
Code auditing: With the ckpool code all freely available, miners are able to audit the code for themselves.
Transaction auditing: Ckpool automatically includes transaction hash reporting to enable auditing of mining work for miners/auditors that request it.

[kano]

About ckdb

Backend database for full user management of ckpool.

Basic concept is a memory resident database with PostgreSQL as the backend DB for permanent storage.
Performance is the priority of ckdb.

Running live and stable on my pool and the hashmine pool.

Still in development ... see the git source code

[-ck]

About ckproxy

Ckproxy uses the same core code as ckpool and acts as a proxy with a simple switch (-p). It acts as a stratum-to-stratum proxy, or stratum-through-stratum proxy in passthrough mode (see below).

It is most useful to regular miners who wish to consolidate their mining hardware into one connection for minimal upstream/downstream bandwidth in standalone mode (with the additional -A switch). Ckproxy uses multiple modes to maintain as many miners as possible communicating with the upstream pool. It works by splitting up nonce2 sizes if the upstream pool's nonce2 is large enough and then it recruits extra upstream connections as needed to allow the number of downstream miners to scale indefinitely.

Ckproxy can be started in "userproxy" mode which monitors login names of miners attaching that don't match the master proxy username and recruits new upstream connections with each unique username, proxying all workers of the same username to the unique upstream proxy connections.

It can be configured to run with a database as per an actual pool entirely, thus acting like a child pool for a parent pool elsewhere, or simply for miners who wish full logging of every detail of their mining operation.

It can also act as a unique stratum-through-stratum passthrough mode which absolutely requires a ckpool as the parent pool with the -P switch. In this mode it does not decrease the bandwidth talking to the upstream pool but minimises the number of open connections instead, but requires extremely low resources to run. This mode is most useful for pools wishing to isolate their main pool instance from the outside world and set up multiple VPSs as a kind of front end proxy/firewall to the outside world. This also removes any realistic limit on the number of open connections since each passthrough instance could easily handle 10k connections, consolidating each into just 1 connection to the upstream pool.

In addition to the passthrough mode, there is a more advanced "node" mode which connects as a passthrough, but needs a local bitcoind connection of its own. It monitors all traffic and shares between the pool and miners, being able to display local hashrates and will submit any blocks found to the local bitcoind in addition to sending the shares to the upstream pool, circumventing the delay of block submission that would otherwise happen with remote nodes.

Finally ckproxy can be started in "redirector" mode which acts as a regular passthrough, but monitors share responses from the upstream pool, and once a valid share is recognised from the upstream pool it will redirect miners that support redirection (all cgminer based clients do) to the URL of your choice. Should the miners not support redirection, such as rental services, ckredirector will continue to act as an ordinary passthrough.

[-ck]

About libckpool

While not strictly feature complete as a library that can be installed, dynamically linked etc. as per other Linux libraries (yet), libckpool is part of the ckpool suite that is designed to be easily included into other projects without requiring any of the other ckpool code. Any and all functions that can be put into the libckpool.c/h files without being dependent on ckpool are placed there for convenience for others to use in their project, thus benefiting from the many common functions a c/c++ application might need, along with many of the unique ck* functions created for the cgminer and ckpool projects that others might find useful.


About ckpmsg:

When ckpool is run, a series of unix sockets are created: listener, generator, stratifier, connector usually in /tmp/ckpool (though configurable) and ckdb creates its own listener socket in a ckdb directory.

Ckpmsg is a simple application designed to talk locally to the various ckpool/ckdb proceses using the message structure that ckpool expects, usually to the listener socket by default if called with no arguments, looking for ckpool by default but can be aimed at ckproxy (with -p), a named ckpool (with -n) or a user chosen socket directory (with -s). Currently ckpool only accepts a limited number of commands on its listener:

shutdown
restart
loglevel=$val (from 0-7)
ping

It looks for a response to any command sent, and it currently can't distinguish those that will and those that won't return a response so it will complain if none is returned.

Although the same ckpmsg can be used for any of the low level functions within any of the processes, it is expected that admins will mostly be using the simple functions. It can also be used for any number of ckdb queries.


About notifier:

This is an extremely simple binary that is designed to be run with the blocknotify feature of bitcoind. It simply tells ckpool to look for new block information and can be run with the -s option to tell it which socket directory to use to tell ckpool about block updates. Standard usage would be something like:

Create a notify script such as notify.sh:

Code:

#!/bin/bash
/usr/bin/notifier -s /opt

Start bitcoind with blocknotify telling it to call that script:

Code:

bitcoind -daemon -blocknotify=/usr/bin/notify.sh

Using a script as blocknotify allows you to change the script without having to restart bitcoind.

[-ck]

Building ckpool

Apart from a basic development package installation, as ckpool is mostly hand written without using libraries, it has zero external dependencies. If ckpool is built with ckdb support, only the postgres development library needs to be installed to build it. Most proxy users and simple solo ckpool setups will not require ckdb.


(Note that this is only for a tarball release which does not yet exist):
sudo apt-get install build-essential
./configure --without-ckdb
make


Building with ckdb requires installation of the postgresql development library.

sudo apt-get install build-essential libpq-dev
./configure
make


Building from git also requires autoconf and automake

sudo apt-get install build-essential libpq-dev autoconf automake libtool
./autogen.sh
./configure
make


Binaries will be built in the src/ subdirectory.

Installation is NOT required and ckpool can be run directly from the directory
it's built in but it can be installed with:
sudo make install

[-ck]

CKPOOL configuration

At least one bitcoind (aka btcd) is mandatory in ckpool mode with the minimum requirements of server, rpcuser and rpcpassword set. If no btcd is set up in the configuration, ckpool will look for a running bitcoind on the local machine on the default port 8332 and the username "user" and password "pass" set up.

At the very least you will need the following lines in your bitcoin.conf file:

Code:

server=1
rpcuser=user
rpcpassword=pass
rpcallowip=127.0.0.1

It is also recommended you start bitcoind with the -blocknotify feature calling the ckpool notifier as described here: https://bitcointalk.org/index.php?topic=790323.msg8907597#msg8907597


Ckpool takes a json encoded configuration file in ckpool.conf by default or ckproxy.conf in proxy or passthrough mode unless specified with -c. Sample configurations for ckpool and ckproxy are included with the source. Entries after the valid json are ignored and the space there can be used for comments.

Sample ckpool.conf file:

Code:

{
"btcd" :  [
        {
                "url" : "localhost:8332",
                "auth" : "user",
                "pass" : "pass",
                "notify" : true
        },
        {
                "url" : "backup:8332",
                "auth" : "user",
                "pass" : "pass",
                "notify" : false
        }
],
"btcaddress" : "14BMjogz69qe8hk9thyzbmR5pg34mVKB1e",
"btcsig" : "/mined by ck/",
"blockpoll" : 100,
"update_interval" : 30,
"serverurl" : "ckpool.org:3333",
"mindiff" : 1,
"startdiff" : 42,
"logdir" : "logs"
}
Comments from here on are ignored.

Sample ckproxy.conf file:

Code:

{
"proxy" :  [
        {
                "url" : "ckpool.org:3333",
                "auth" : "user",
                "pass" : "pass"
        },
        {
                "url" : "backup.ckpool.org:3333",
                "auth" : "user",
                "pass" : "pass"
        }
],
"update_interval" : 30,
"serverurl" : "192.168.1.100:3334",
"mindiff" : 1,
"startdiff" : 42,
"logdir" : "logs"
}
Comments from here on are ignored.

The options recognised are as follows:


"btcd" : This is an array of bitcoind(s) with the options url, auth  and pass
which match the configured bitcoind. This is mandatory in pool mode. The
optional boolean field notify tells ckpool this btcd is using the notifier
and does not need to be polled for block changes.

"proxy" : This is an array in the same format as btcd above but is used in
proxy and passthrough mode to set the upstream pool and is mandatory.

"btcaddress" : This is the bitcoin address to try to generate blocks to.

"btcsig" : This is an optional signature to put into the coinbase of mined
blocks.

"blockpoll" : This is the frequency in milliseconds for how often to check for
new network blocks and is 100 by default. It is intended to be a backup only
for when the notifier is not set up and only polls if the "notify" field is
not set on a btcd.

"update_interval" : This is the frequency that stratum updates are sent out to
miners and is set to 30 seconds by default to help perpetuate transactions for
the health of the bitcoin network.

"serverurl" : This is the IP to try to bind ckpool uniquely to, otherwise it
will attempt to bind to all interfaces in port 3333 by default in pool mode
and 3334 in proxy mode.

"mindiff" : Minimum diff that vardiff will allow miners to drop to. Default 1

"startdiff" : Starting diff that new clients are given. Default 42

"logdir" : Which directory to store pool and client logs. Default "logs"



It is anticipated that pool operators wishing to set up a full database based installation of ckpool+ckdb will be familiar with setting up postgresql and associated permissions to the directories where the various processes will communicate with each other and a web server so these will not be documented.

If you are not capable of setting up and running a database and/or webserver then you should be questioning whether you should be running a public pool or not.

[-ck]

Stratum extensions:

In addition to support of all the common stratum features, ckpool supports extensions that are either not yet in use, along with its own interpretation of further stratum features as the need arises.

mining.passthrough

This is a unique stratum extension supported only by ckpools upstream and creates a stratum-through-stratum passthrough. It is not intended to be used by miners directly except through the use of the ckpool code itself in passthrough mode for connection combining.

mining.node

This is unique stratum extension supported only by ckpools upstream and creates a stratum-through-stratum passthrough and additionally receives workinfo information from the upstream pool to allow local block submission.

mining.get_transactions

This controversial bandwidth eating feature has been absorbed into ckpool but instead of it handing out the full transactions which are a complete waste of bandwidth, the number of transactions is returned. In addition to supporting the originally documented form with job_id embedded in the method, ckpool will accept the valid json job_id as a single array parameter eg: params: ["$jobid"] .The functionality of the original feature request has been supplanted by the following call instead:

mining.get_txnhashes

Instead of handing out the full transactions incorporated into a job, this version returns only the hashes of the transactions included in each job in a space separated string as the result. The returned hashes are less than 1/10th the bandwidth that returning the full transactions would have amounted to, but is ample information for auditing tools or external authorities to confirm what the running ckpool is working on. This feature is on and rate limited to return only one response per miner per job_id to avoid it becoming a source of DoS. eg:

Code:

{"id": 0, "method": "mining.get_txnhashes", "params":["545198de00000000"]}

mining.suggest_difficulty

Special handling is done for accepting a diff as requested by each mining connection. This is done separately from the per-worker connection so it allows overriding that. When a miner suggests a difficulty, ckpool will allow a slightly lower diff than it would have otherwise chosen, but still will not exceed 30 shares per minute to prevent miners from using this to flood the server with low diff shares.  In addition to supporting the originally documented form where the difficulty was embedded in the method name, ckpool will accept the valid json where the difficulty is a single arrray parameter in params. eg:

Code:

{"id": 2, "method": "mining.suggest_difficulty", "params": [42]}

mining.ping

This is as it says a simple ping sent to or from the miner without parameters. Ideally this would be used to ensure a miner is still alive or to inform the miner that the pool is still alive and ideally they should respond with a result of "pong" (or vice versa). Since most mining software does not currently support this it is not expected to be used for confirmation, although sending data is a good way to ensure a raw socket is still open.

Code:

{"id": 42, "method": "mining.ping", "params": []}

Code:

{"id": 42, "result": "pong", "error": null}

[-ck]

FAQs

Q: I get permission errors on the sockets?
A: You cannot talk via unix sockets using sudo. Make sure your users have write permissions to the directory where your sockets go. In particular, apache/httpd cannot use /tmp so you will have to choose a different directory such as /opt and give write permission to the user/group that you run ckpool/ckdb as.

Q: System requirements?
A: This depends entirely on the mode of running ckpool and the number of clients you wish to connect to it.
In proxy passthrough mode even a single CPU 32MB ram VPS is enough and requires virtually no disk space.
Standalone proxy mode requires more ram and CPU.
Running a standalone pool without ckdb requires only a few hundred megabytes of ram and will run on about 256MB of ram and at least one virtual CPU if the bitcoind is run on a separate machine (not recommended but doable).
However bitcoind itself usually requires a minimum of 2GB of ram with the wallet disabled and almost 30GB of HD space.
So a recommended setup for a minimal pool with bitcoind would be a 4GB ram 4 virtual CPU without ckdb.
Ckdb however uses a lot of ram due to its nature of having all data memory resident for maximum speed so the ram requirements for a fully featured pool would be at least 16GB ram.

Q: Cross platform / other operating system support?
A: Linux is by far the most proven, tested, deployed server operating system of choice today, and by limiting ourselves to writing for it we are able to use many unique linux features and not have surprise differences in operating system behaviour. When working with low level socket code as ckpool does without a library, the actual operating system behaviour and APIs themselves matter, so it is not worth trying to port them across which is one of the major complaints about the protocol.

Q: What payscheme is it? Paying of orphans? Paying of transactions?
A: This is a choice at a pool operator level and has nothing to do with the core ckpool code. While intrinsically there is code for PPLNS in ckdb, the pay scheme and mechanism is up to the pool operator who sets up the pool with this code.

Q: Why not p2pool as a backend?
A: p2pool as a concept has failed as a scalable concept due to continually marginalising its smallest miners and increasing variance the more miners it has instead of decreasing variance the way a traditional pool does. I have not completely discounted this as a module for ckpool if we can come up with valid workarounds and we are always considering new options. We are instead proposing future development of sibling pools with ckpool to decrease variance by combining rewards from multiple pools into smaller pools. More on this to come.

Q: Other mining protocols?
A: While stratum may not be a perfect mining protocol, the adoption of it as the defacto standard for scalability and its robustness has been well proven. Instead of rewriting the protocol, we believe that a solid implementation is enough to offset any disadvantages to the protocol, and add extensions as the need arises. We support reporting of transaction hashes included in work templates as a security measure for validation by miners and auditors.

Q: GUI?
A: A GUI has no place in the core code of a server or database.

Q: Simple setup tutorial?
A: It is a cinch to set up a simple ckproxy or solo ckpool. If you need help setting up a fully featured pool then you shouldn't be doing it.

Q: Merged mining?
A: Any presumed profit increase from adding virtually worthless altcoins is not worth the disadvantage of introducing far more instability associated with running multiple lesser unstable coin daemons. The running overheads of ckpool should be so small as to allow low fees from pool operators and stability. Some purported free pools with multiple coin support have so much downtime that any extra profit from them is offset by their downtime and poor design.

Q: Altcoins?
A: The author has no interest in support for anything but bitcoin. You are free to modify the code yourself as you see fit, but no altcoin support will ever be merged into the master ckpool code.

Will be added to as the questions arise.

[-ck]

Discussion/Future direction/Feature requests

At this time the development of the code pool code has been in an extensive testing, debugging and consolidation, having been completed months ago and has had a great deal of real world testing, minting many blocks along the way. The ckdb code is now at completion to the point where public pools can run the code, which is why we're finally able to announce this project. Future features are definitely planned once this consolidation phase is over, and bug reports/adoption/feature requests/code pushes are all to be expected.

The next major feature we wish to work on is to develop the node code to allow multiple trusted nodes in different locations share data, but this is primarily work to be done at the database level.

In the future I wish to develop the concept of sibling pools which will allow collaboration to minimise variance for miners while sharing rewards. The idea is to expand beyond the need for trusted nodes to be able to link up with other pools that can prove to each other the work they are performing towards a common reward. In essence it will involve proof-of-work much like miners do to the pool itself but with auditing of each other's work. The details of the protocol are not yet finalised and is still some way off.

[will]

We are the owners of hashmine.io and commissioned the development of ckpool.

We've been working with Con & Kano on this for a few months now, and I must say it's a pleasure to see the masters at work. In a sense, the work was a natural progression of the solo-mining/gbt work on cgminer we commissioned earlier this year.

I started on basic visualization/GUI layer here: http://cpui.hashmine.io/ - changes to that sync with Kano's web implementation @ http://ice.kano.is/
Current focus is on browser compatibility, making it work and look serviceable on all devices. Priority is bugfixes and basic user management workflows, will look @ i18n and l10n after that.
This current UI is mainly a test for the REST API I'm building on Kano's ckdb work and testing the servicing of historical reporting. Will add more visualizations & reporting in the coming days.
Still very much in debug mode as daily changes are coming in, but will try to keep up with Kano & Con as time goes on.

hashmine.io takes a flat rate 1.5% fee, of which 0.5% goes directly to ckpool development (controlled by Con, and issued @ coinbase at time of blocksolve). Newly minted coins go directly into cold storage and I process them manually when the blocks mature. A payout history is available @ the above-mentioned UI. Will do this until I've worked through the security aspects of automation.

There's still much to do on this technical path (hardware, software, architecture, distribution, etc.) and this project will probably always be on-going. Once things calm down a bit we'll go into the unique-selling points of hashmine and the direction we'd like to take with it.

Will update this post with progress as I push changes.

[QuiveringGibbage]

Discussion/Future direction/Feature requests
In the future I wish to develop the concept of sibling pools which will allow collaboration to minimise variance for miners while sharing rewards. The idea is to expand beyond the need for trusted nodes to be able to link up with other pools that can prove to each other the work they are performing towards a common reward. In essence it will involve proof-of-work much like miners do to the pool itself but with auditing of each other's work. The details of the protocol are not yet finalised and is still some way off.

very interesting concept for multi pools to trust each other... this will change the pooled mining landscape. Bitcoin mining is on the change again, fascinating future ahead for the mining scene.

@conman & kanoi; Congrats on the release and thank you for sharing your code with the community.

Cheers,
QG

[-ck]

Added support for using ckproxy to p2pool and large coinbase pools. Be aware that most non-ckpool based pools (p2pool included) have work limitations per connection when proxied as discussed in the ckproxy post.

[-ck]

I've substantially updated the ckproxy code to work much better in concert with p2pool, and to increase the maximum speed of clients that are connected via the proxy when connected to a non-ckpool. On most pools, you will now be able to connect up to 256 clients to the proxy (down from what was 64k) but the maximum hashrate of each connected miner has been pushed up from 5TH to 1.2PH.

[-ck]

I've added logging and statistics internally to ckpool on a combined per-worker basis, storing stats for each worker in the logs/workers directory.

I've also added the ability to set a user chosen minimum worker difficulty, though this will require some ckdb and front-end changes to allow this to be set.

In addition, I've implemented support for the stratum suggest_difficulty extension and updated the docs to reflect this. I've modified the cgminer git master to support this new extension with the --suggest-diff command.

[mahrens917]

First off, thanks for ckpool - it really sounds like a great product and I am trying to learn more.  I have a p2pool server and I am trying to get a ckpool proxy setup in front of it since it sounds like this will reduce load on my server (provided I have more users).  First, with ckpool -A -p running will all downstream users continue to maintain their individual presence on the p2pool (like it is described to work with -P)?  Within ckpool.conf there is a proxy url and a server url that need to be set.  Running this on my p2pool server the proxy url seems to want to be set to my p2pool server with port 9332.  That what is server URL?  The IP and port that outside clients would connect to, to use the proxy before hitting the p2pool server?  I have tried multiple configurations and cannot get it to work.  Perhaps, there is a sample .conf file out there already for p2pool?  Thanks for all of the help.

[-ck]

First off, thanks for ckpool - it really sounds like a great product and I am trying to learn more.  I have a p2pool server and I am trying to get a ckpool proxy setup in front of it since it sounds like this will reduce load on my server (provided I have more users).  First, with ckpool -A -p running will all downstream users continue to maintain their individual presence on the p2pool (like it is described to work with -P)?  Within ckpool.conf there is a proxy url and a server url that need to be set.  Running this on my p2pool server the proxy url seems to want to be set to my p2pool server with port 9332.  That what is server URL?  The IP and port that outside clients would connect to, to use the proxy before hitting the p2pool server?  I have tried multiple configurations and cannot get it to work.  Perhaps, there is a sample .conf file out there already for p2pool?  Thanks for all of the help.

Server URL is only when run in ckpool mode. The proxy entry is when you are running it in proxy mode. If you are connecting clients to the proxy, you can only have one upstream connection to the pool with one login only. If you allowed multiple logins it would end up just having multiple connections to the upstream pool so the proxy wouldn't actually be achieving anything. The -P option ONLY works when the upstream pool is a ckpool instance as the manual explicitly states and nothing else and does not decrease bandwidth, only the number of socket connections.

Simplest p2pool proxy entry would be just the proxy entry and nothing else. eg:

Code:

{
"proxy" :  [
        {
                "url" : "localhost:9332",
                "auth" : "user",
                "pass" : "pass"
        }
]
}

EDIT: I've been thinking of ways to allow multiple clients attach to the upstream pool with their username but it's a very invasive change to the code since it is designed around the concept of only one upstream server at any one time. Unless multiple clients are connected with the same username attach they're not going to benefit from the proxy being there, but this change will likely eventually make its way into the code, along with ways to open up extra connections once the proxy is "full" of clients.

[mahrens917]

Payout schemes are not discussed for ckpool.  Is it just solo or are others possible like PPLNS?  I have read everything i could find on ckpool and i do not belive this was ever explicitly stated.  Hopefully, this will be my last question for awhile.  Thanks!

[-ck]

Payout schemes are not discussed for ckpool.  Is it just solo or are others possible like PPLNS?  I have read everything i could find on ckpool and i do not belive this was ever explicitly stated.  Hopefully, this will be my last question for awhile.  Thanks!

Hi, read the FAQ linked in the top post. This is software, not a pool, so the payscheme is not fixed in the software design and entirely up to the pool. The ckdb code has a pplns calculation in it, but that doesn't mean the pool that adopts ckpool has to use pplns at all. Code to perform payments is not anywhere in the included code. Furthermore, if you are using ckproxy and not ckpool, the payment part is entirely dependent on your upstream pool and the proxy has zero effect on it. If you're asking specifically about the current pools that are running ckpool, hashmine.io and ckpool/kano.is are using pplns for payment, while solo.ckpool.org is using entire block reward payments.

[Walter Rothbard]

Thanks for the awesome software - I was able to compile and configure a private pool in about five minutes!