CKPOOL - Open source pool/proxy/passthrough/redirector/library in c for Linux

[-ck]

Hi Con,

I don't believe that you clearly explained the requirement to use Linux amd64 for your CKPool. The only error I saw from the previous posts is the "unknown type name sa_family_t" from /usr/include/linux/un.h.

Could you please explain in detail which parts on your CKPool require Linux amd64?

The main reason I asked is that, for small VPS it is more efficient - in term of the RAM usage - to use Linux i386 instead of amd64.  So it would be helpful to know the impact of compiling and running your CKPool on Linux i386.

Cheers,

Anto

The answer is I don't care so I've not bothered to make sure the code is fine on 32bit architectures and I don't want to care. You have your priorities wrong if you are worried about that on a pool.

[kano]

I'm really not sure how a 32bit OS can be more efficient on 64bit hardware ...

Edit: ... and I'll add that mining requires 64bit math quite a bit ... since 4billion really isn't a big enough number
So all that code would be slower.

[.anto.]

Hi Con,

I don't believe that you clearly explained the requirement to use Linux amd64 for your CKPool. The only error I saw from the previous posts is the "unknown type name sa_family_t" from /usr/include/linux/un.h.

Could you please explain in detail which parts on your CKPool require Linux amd64?

The main reason I asked is that, for small VPS it is more efficient - in term of the RAM usage - to use Linux i386 instead of amd64.  So it would be helpful to know the impact of compiling and running your CKPool on Linux i386.

Cheers,

Anto

The answer is I don't care so I've not bothered to make sure the code is fine on 32bit architectures and I don't want to care. You have your priorities wrong if you are worried about that on a pool.

I clearly didn't ask or demanded that you should make sure your code run on i386. I just asked which parts rely on amd64. So never mind then.

[.anto.]

I'm really not sure how a 32bit OS can be more efficient on 64bit hardware ...

Edit: ... and I'll add that mining requires 64bit math quite a bit ... since 4billion really isn't a big enough number
So all that code would be slower.

I am not a programmer and I know very little about 32bit vs 64bit programs. However, some years ago when I started to rent VPS for my private web site, I did some readings and testing myself. I have made the conclusion that Linux i386 is more efficient in using the RAM than Linux amd64. Somebody did more thorough tests than me and posted the results on this web page, if you wanted to know that.

[kano]

As I already implied, the software is full of 64bit maths and that will produce a noticeable difference, as will having double the registers in general code.

... and you will find that 5 year old link quite outdated now ... comments about not everyone having 64bit processors and "fewer issues" with 32bit are now wrong.
It's the other way around, fewer issues with 64bit and I'm not sure you can even find 32bit processors online any more.
If you can, you've been given old dodgy hardware.

Now the most relevant point ... if you want to run a pool, and you are trying to skimp on hardware, be well aware that you may lose blocks.
You need a bitcoind, and running one on a small vps is a problem waiting to happen.

[.anto.]

As I already implied, the software is full of 64bit maths and that will produce a noticeable difference, as will having double the registers in general code.

... and you will find that 5 year old link quite outdated now ... comments about not everyone having 64bit processors and "fewer issues" with 32bit are now wrong.
It's the other way around, fewer issues with 64bit and I'm not sure you can even find 32bit processors online any more.
If you can, you've been given old dodgy hardware.

I am quite sure that I didn't mention anything related to 32bit processors. My questions are clearly related to the use CKPool software on VPS with Linux i386, which are still very easy to get from all Linux distros or compile it ourselves. All VPS' definitely use 64bit capable processors.

Now the most relevant point ... if you want to run a pool, and you are trying to skimp on hardware, be well aware that you may lose blocks.
You need a bitcoind, and running one on a small vps is a problem waiting to happen.

Again, I am not asking anything outside running CKPool software on VPS with Linux i386. I know the impact of running bitcoind on VPS with RAM less than 2GB as I experience the issues myself, e.g. bitcoind regularly crashes. So I have the full node running on a real hardware with a Quad Core 2 GHz each and 8 GB RAM. However, the bandwidth of the link of the full node is asymmetric. So I want to have the pool software on the VPS with symmetric bandwidth to be able to better manage the user connections.

Anyway, this discussion does not seem to lead anywhere. But thanks for responding to my questions.

[kano]

As I already implied, the software is full of 64bit maths and that will produce a noticeable difference, as will having double the registers in general code.

... and you will find that 5 year old link quite outdated now ... comments about not everyone having 64bit processors and "fewer issues" with 32bit are now wrong.
It's the other way around, fewer issues with 64bit and I'm not sure you can even find 32bit processors online any more.
If you can, you've been given old dodgy hardware.

I am quite sure that I didn't mention anything related to 32bit processors. My questions are clearly related to the use CKPool software on VPS with Linux i386, which are still very easy to get from all Linux distros or compile it ourselves. All VPS' definitely use 64bit capable processors.

Right so running i386 code on them pretends it's a 32bit processor missing all the vast gains in current 64bit processors,
the most obvious one (even mentioned at your very old link) being double the number of registers.

Now the most relevant point ... if you want to run a pool, and you are trying to skimp on hardware, be well aware that you may lose blocks.
You need a bitcoind, and running one on a small vps is a problem waiting to happen.

Again, I am not asking anything outside running CKPool software on VPS with Linux i386. I know the impact of running bitcoind on VPS with RAM less than 2GB as I experience the issues myself, e.g. bitcoind regularly crashes. So I have the full node running on a real hardware with a Quad Core 2 GHz each and 8 GB RAM. However, the bandwidth of the link of the full node is asymmetric. So I want to have the pool software on the VPS with symmetric bandwidth to be able to better manage the user connections.

Anyway, this discussion does not seem to lead anywhere. But thanks for responding to my questions.

So your design is to have a pool's main connection talking to a remote bitcoind ... sounds like a bad idea to me.
As for bandwidth, bitcoind will be higher than the pool unless you have a big pool ...

[e46btc]

Can someone give advice why ckdb (latest ver from git) can't connect via RPC to local bitcoind (v0.11.2) ?  

At least I am trying to setup BTC payout address in User settings (web) but got error ERR: Invalid BTC address

First time ckdb was not able to connect to standard bitcoind RPC port 8332 as it was hardcoded to 8330  in ckdb.c

Code:

char *btc_server = "http://127.0.0.1:8330";
char *btc_auth;

I've changed to 8332 and recompiled, it is connected but still no luck

Code:

[2015-12-12 13:04:28.923+00] _btc_io() btc server response not ok: HTTP/1.0 401 Authorization Required0x0d0x0aDate: Sat, 12 Dec 2015 13:04:28 +00000x0d0x0aServer: bitcoin-json-rpc/v0.11.2.0-g7e278920x0d0x0a
WWW-Authenticate: Basic realm="jsonrpc"0x0d0x0aContent-Type: text/html0x0d0x0aContent-Length: 2960x0d0x0a0x0d0x0a<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
0x0d0x0a"http://www.w3.org/TR/1999/REC-html401-19991224/loose.dtd">0x0d0x0a<HTML>0x0d0x0a<HEAD>0x0d0x0a<TITLE>Error</TITLE>0x0d0x0a<META HTTP-EQUIV='Content-Type' 
CONTENT='text/html; charset=ISO-8859-1'>0x0d0x0a</HEAD>0x0d0x0a<BODY><H1>401 Unauthorized.</H1></BODY>0x0d0x0a</HTML>0x0d0x0a0x00
[2015-12-12 13:04:28.923+00] usersettings.userset5111.ERR.Invalid BTC address

I've tried to recompile this again w/ hardcoded username/password  in ckdb.c

Code:

char *btc_server = "http://127.0.0.1:8332";
char *btc_auth = "USER:PASS";

according to

Code:

	snprintf(buf, sizeof(buf), "%s:%s", btc_user, btc_pass);
	btc_auth = http_base64(buf);

but the same result, this can't authenticate against bitcoind RPC.  

Any hints will be highly appreciated. Thanks!

ps: ckpool works fine via RPC using user/pass from ckpool.conf  , so there is no mistake in user/pass

[kano]

command line:
./ckdb -p pgsqlpass -U btcuser -P btcpass -S http://127.0.0.1:8332 -m -g

https://bitbucket.org/ckolivas/ckpool/src/96d6707511de3cfcee37261a5e0516791789e1ff/src/ckdb.c?at=master&fileviewer=file-view-default#ckdb.c-5631

Edit: and of course below that:

https://bitbucket.org/ckolivas/ckpool/src/96d6707511de3cfcee37261a5e0516791789e1ff/src/ckdb.c?at=master&fileviewer=file-view-default#ckdb.c-5686

Edit2: firstly yeah you don't need to edit the code, just use the command line options
secondly, you didn't

Code:

btc_auth = http_base64(buf);

[e46btc]

Thank you Kano!  So easy, it works. 
I've missed command line params related to BTC  and instead of this "http://127.0.0.1:8330" string in ckdb log confused me.

[o_solo_miner]

  got it working!

Thank you both, Con & Kano.

I compiled all under Debian AMD64, no problem.
Configured first the proxy, than the rest, no problem.

That looks like your README is all I needet to come to sucsess.

All my miners goes now as 1 to cksolo.pool 


You make me verry happy!

---

Next step is to get the relaynetwork working.

[hodedowe]

Hope this helps someone else. If you get this response from the pool:

HTTP response to ( "getblock...) 0.001s not ok: HTTP/1.1 500 Internal Server Error


Then something has gone haywire with your Bitcoin core installation. It's not the pool. On one of my test machines was throwing this error with a remote core installation. I spent 2 days troubleshooting the pool and ports/etc. A simple shutdown>reinstall of bitcoin core fixed the problem.

[hodedowe]

You could also just download the VMware version and run it on any machine.

[-ck]

Massive code drop into git master long overdue with many new features and improvements, mostly on the proxy and passthrough side. I've tagged milestone M18 to just before the code drop and M19 to after it, along with incrementing the version to 0.9.0. The new features are documented in the README and with sample configurations, and at some stage I'll add descriptions to this thread.

[-ck]

I've quoted the opening post on ckproxy and highlighted the changes below in bold.

Ckproxy uses the same core code as ckpool and acts as a proxy with a simple switch (-p). It acts as a stratum-to-stratum proxy, or stratum-through-stratum proxy in passthrough mode (see below).

It is most useful to regular miners who wish to consolidate their mining hardware into one connection for minimal upstream/downstream bandwidth in standalone mode (with the additional -A switch). Ckproxy uses multiple modes to maintain as many miners as possible communicating with the upstream pool. It works by splitting up nonce2 sizes if the upstream pool's nonce2 is large enough and then it recruits extra upstream connections as needed to allow the number of downstream miners to scale indefinitely.

Ckproxy can be started in "userproxy" mode which monitors login names of miners attaching that don't match the master proxy username and recruits new upstream connections with each unique username, proxying all workers of the same username to the unique upstream proxy connections.

It can be configured to run with a database as per an actual pool entirely, thus acting like a child pool for a parent pool elsewhere, or simply for miners who wish full logging of every detail of their mining operation.

It can also act as a unique stratum-through-stratum passthrough mode which absolutely requires a ckpool as the parent pool with the -P switch. In this mode it does not decrease the bandwidth talking to the upstream pool but minimises the number of open connections instead, but requires extremely low resources to run. This mode is most useful for pools wishing to isolate their main pool instance from the outside world and set up multiple VPSs as a kind of front end proxy/firewall to the outside world. This also removes any realistic limit on the number of open connections since each passthrough instance could easily handle 10k connections, consolidating each into just 1 connection to the upstream pool.

In addition to the passthrough mode, there is a more advanced "node" mode which connects as a passthrough, but needs a local bitcoind connection of its own. It monitors all traffic and shares between the pool and miners, being able to display local hashrates and will submit any blocks found to the local bitcoind in addition to sending the shares to the upstream pool, circumventing the delay of block submission that would otherwise happen with remote nodes.

Finally ckproxy can be started in "redirector" mode which acts as a regular passthrough, but monitors share responses from the upstream pool, and once a valid share is recognised from the upstream pool it will redirect miners that support redirection (all cgminer based clients do) to the URL of your choice. Should the miners not support redirection, such as rental services, ckredirector will continue to act as an ordinary passthrough.

[analpaper]

sorry lil question if possible: can we know what is the meaning of the variables tw=1 and fa=0 used in myqr.sh ?

this script doesnt exists in the repo so i rewrote it, and seems to me that this variables are just a check to see if the qr was parsed ok, but im not sure at all..

Thanks!

[kano]

Basically you gotta write your own "$sfaurl" -> "qrcode"
Site works without it since it also displays a 'link' that is the "$sfaurl", just without a pretty qrcode by default.

Someone actually paid me to write the 2FA code but they think my web site sux so they have their own
So they didn't need my php for it, so I didn't provide the program I wrote to convert the input for the html5 in my php.
It's a completely separate program so wont be in git.

[mmpool]

mmpool switched to using ckpool recently, from SockThing, mainly to get the benefit of variable difficulty settings for users but also to get a supported efficient codebase. We're a small pool and found the first block using the ckpool backend today.

We're not using the ckdb backend, - just ckpool. It connects to our merge mining server and the already existing pool reward system, database, etc. I wanted to minimize the integration work to try ckpool at the time. It was quite painless to integrate.

Thanks for the great software. I've sent a manual donation to the AUTHORS to help support the work.

[-ck]

mmpool switched to using ckpool recently, from SockThing, mainly to get the benefit of variable difficulty settings for users but also to get a supported efficient codebase. We're a small pool and found the first block using the ckpool backend today.

We're not using the ckdb backend, - just ckpool. It connects to our merge mining server and the already existing pool reward system, database, etc. I wanted to minimize the integration work to try ckpool at the time. It was quite painless to integrate.

Thanks for the great software. I've sent a manual donation to the AUTHORS to help support the work.

You're most welcome. Thanks for the donation and good luck.

[kano]

mmpool switched to using ckpool recently, from SockThing, mainly to get the benefit of variable difficulty settings for users but also to get a supported efficient codebase. We're a small pool and found the first block using the ckpool backend today.

We're not using the ckdb backend, - just ckpool. It connects to our merge mining server and the already existing pool reward system, database, etc. I wanted to minimize the integration work to try ckpool at the time. It was quite painless to integrate.

Thanks for the great software. I've sent a manual donation to the AUTHORS to help support the work.

Thanks - very nice donation!
Now you need to start using ckdb
It uses truck loads of ram, but less CPU than ckpool on the same machine
Every so often I change something to reduce the ram requirements.