COMPLETED: Request for an RPC capable fork of cgminer (155 BTC pledged and paid)

[kano]

... or just use the -T option and pipe it into something ...

[1714234059]

1714234059

[1714234059]

1714234059

[1714234059]

1714234059

[1714234059]

1714234059

[1714234059]

1714234059

[Chefnet]

Theres so many projects that I want to help out!  Bitcoiners are still coming up with great ideas.

I'll pledge 10 BTC.  I'll pledge 5 more if I ckolivas pulls the code into his main branch (even if he codes it).  I think this would help us by keeping bugs to a minimum.  Multiple versions of cgminer are sure to only complicate things and slow features down.

agree +1.

add 15BTC by me if it will be mainline.

[ThiagoCMC]

Hi!

 I would love to support this... I can give 5 BTCs now.

 And I have a suggestion:

 1- Identify a GPU by its "device ID" (possible?);
 2- Configuration of each GPU based on its ID;
 3- A different "pool / login / worker" per device (fixed by its ID).

 This way, no matter if a GPU is GPU0 or GPU1, the ID will be the same, so its own configurations...

 This can be usefull if we change a GPU slot or add/remove GPUs from a machine.

 It can works more or less like Linux file system UUID, with it, no matter if the device is sda or sdb, the FS UUID will be always the same and the file system will be mounted without any problem if you change the disk connection.

Best,
Thiago

[DeathAndTaxes]

Hi!

 I would love to support this... I can give 5 BTCs now.

 And I have a suggestion:

 1- Identify a GPU by its "device ID" (possible?);
 2- Configuration of each GPU based on its ID;
 3- A different "pool / login / worker" per device (fixed by its ID).

 This way, no matter if a GPU is GPU0 or GPU1, the ID will be the same, so its own configurations...

 This can be usefull if we change a GPU slot or add/remove GPUs from a machine.

 It can works more or less like Linux file system UUID, with it, no matter if the device is sda or sdb, the FS UUID will be always the same and the file system will be mounted without any problem if you change the disk connection.

Best,
Thiago

Is the hardware ID exposed by AMD SDK?  I don't believe it is but I only glanced over AMD SDK and example source code.

[DeathAndTaxes]

The JSON RPC is also the reason why bitcoin, pools and even ltc have such problems dealing with what amounts to not a lot of data.

I believe the bottleneck on pools is bitcoind internal code not the JSON RPC. I do agree JSON is very heavy but for the amounts of data we are dealing with here it is a non-issue.  Personally I wouldn't use JSON but then again I am a windows developer so for interprocess communication my first thoughts are going to be sockets, memory mapped files, or named pipes.

Really the entire bitcoind needs to be built from ground up for high performance.  Using a real multi-user database, multi-core transaction engine, and some sort of higher throughput I/O interface like sockets.  Honestly does anyone think if Amazon accepted Bitcoin they would use the existing bitcoind to process transactions?  The community could use a high performance bitcoind.

If someone else wants to write it then I guess JSON RPC is what will be used and it will suck badly like the rest of the JSON RPC interfaces spread throughout Bitcoin.

Why?  How many I/O do you think need to be made against a single miner instance.  I mean even polling for updated stats once per second is not going to be a bottleneck.  Things like error reporting, changing clocks, and restarting instances are going to happen so infrequently to be even less of a non-issue.

I get and see a lot of responses from people saying how "Satoshi said so thus that's how it should be"
This is another example of it.

Nobody said that (except you).  The bounty isn't limited to JSON.  However just doing something different for the sake of different is dubious at best.

As far as Satoshi decision.  He likely never saw the rise of mining pools.  At the time he wrote it miners were getting maybe 1 MH/s on current hardware.  Even at 1 GH/s you only need 1 getwork every 4 seconds.  Block changes are only every 600 seconds.  How many transactions per second is a merchant going to need 1 tps = 86,400 orders per day.

He likely didn't have any philosophical attachment to JSON.  He likely chose it for the same reason millions of developers choose it every year.  It is ease to implement and it was capable of the performance required of it.

Honestly if you made a socket based bitcoind you likely could collect quite a bounty from pool operators.  As that I/O pressure is not going to go away.

[kano]

I get and see a lot of responses from people saying how "Satoshi said so thus that's how it should be"
This is another example of it.

Nobody said that (except you).  The bounty isn't limited to JSON.  However just doing something different for the sake of different is dubious at best.

As far as Satoshi decision.  He likely never saw the rise of mining pools.  At the time he wrote it miners were getting maybe 1 MH/s on current hardware.  Even at 1 GH/s you only need 1 getwork every 4 seconds.  Block changes are only every 600 seconds.  How many transactions per second is a merchant going to need 1 tps = 86,400 orders per day.

He likely didn't have any philosophical attachment to JSON.  He likely chose it for the same reason millions of developers choose it every year.  It is ease to implement and it was capable of the performance required of it.

Honestly if you made a socket based bitcoind you likely could collect quite a bounty from pool operators.  As that I/O pressure is not going to go away.

Obviously I am not referring to this thread regarding comments about Satoshi.
Anyone but a fool can see that no one mentioned Satoshi in it before my comment ...

[gnar1ta$]

Theres so many projects that I want to help out!  Bitcoiners are still coming up with great ideas.

I'll pledge 10 BTC.  I'll pledge 5 more if I ckolivas pulls the code into his main branch (even if he codes it).  I think this would help us by keeping bugs to a minimum.  Multiple versions of cgminer are sure to only complicate things and slow features down.

agree +1.

add 15BTC by me if it will be mainline.

I'll match this...10 for the project and another 5 if it will be mainline.  Has anyone set up an escrow wallet of some sort?

Goddamnit that's the sum total of how much I've earned in donations in total for cgminer 

I increased my donation %.  That's just crazy, cgminer is worth more than that.

[-ck]

Goddamnit that's the sum total of how much I've earned in donations in total for cgminer  

I increased my donation %.  That's just crazy, cgminer is worth more than that.

Thanking you very much. Just for the record, I just checked and the sum total of hashes from all --donation are currently running at 100 Mhash. It usually hovers between 100-200. Note that I intentionally made the percentage err on the conservative side, and it will virtually always be less than what you set it to since I didn't want complaints.

[kano]

https://github.com/kanoi/cgminer

API v0.1

Includes a sample api-example.php

6 files added/changed from current ckolivas master:

main.c, miner.h, AUTHORS, api.c, api-example.php, Makefile.am

(also has the updated linux-usb-cgminer)

OK so this obviously isn't final (though it does work fine for me)

Anyone who can build their own cgminer and is interested could you have a look?

It currently has 3 API commands:
apiversion - returns a version string "0.1"
dev - returns a status string of all the devices (CPU and or GPU)
pool - returns a status string of all the pools

The included PHP file (api-example.php) simply converts the returned string into a data structure
Very simple and very easy in PHP the way I've done it.

No I'm not expecting people to give me bitcoins yet ...

The 3 main questions I have are:
1) what other API commands do people want
2) what parameters should I add to cgminer: --api-port = 4028, --api-address = 127.0.0.1, --api-access = 192.168.5.* and anything else?
the numbers shown are of course examples
(I haven't added any parameters yet)
3) what socket I/O do people want: by this I mean: default port and access restriction.
It currently runs on port 4028 and only listens on 127.0.0.1 - for security reasons of course.

Edit: it does no mutex locking access to the cpu/gpu/pool info since it would appear that is OK
(the current code doesn't appear to do that either)
ckolivas - does that sound correct to you?

Edit2: minor change - it wasn't closing the connection if an invalid command was sent

Edit3: added 'quit' and 'summary'

[lueo]

Good work kano!

I am working on my Python warpper for cgminer, too. Initial version will be upload later.

It is a RESTful application which will accept JSON format as control / query.

I'll make it support all the miners, like poclbm, phoenix... etc.

https://github.com/kanoi/cgminer

API v0.1

Includes a sample api-example.php

6 files added/changed from current ckolivas master:

main.c, miner.h, AUTHORS, api.c, api-example.php, Makefile.am

OK so this obviously isn't final (though it does work fine for me)

Anyone who can build their own cgminer and is interested could you have a look?

It currently has 3 API commands:
apiversion - returns a version string "0.1"
dev - returns a status string of all the devices (CPU and or GPU)
pool - returns a status string of all the pools

The included PHP file (api-example.php) simply converts the returned string into a data structure
Very simple and very easy in PHP the way I've done it.

No I'm not expecting people to give me bitcoins yet ...

The 3 main questions I have are:
1) what other API commands do people want
2) what parameters should I add to cgminer: --api-port = 4028, --api-address = 127.0.0.1, --api-access = 192.168.5.* and anything else?
the numbers shown are of course examples
(I haven't added any parameters yet)
3) what socket I/O do people want: by this I mean: default port and access restriction.
It currently runs on port 4028 and only listens on 127.0.0.1 - for security reasons of course.

Edit: it does no mutex locking access to the cpu/gpu/pool info since it would appear that is OK
(the current code doesn't appear to do that either)
ckolivas - does that sound correct to you?

Edit2: minor change - it wasn't closing the connection if an invalid command was sent

[kano]

Looking at the first post, I guess all I need to add is:
* Start/stop/restart individual GPUs (on restart recheck config file for any changes)

But I have no idea what you mean by "(on restart recheck config file for any changes)"


As for this:
* Report any failures (a min of card down & pool not responding)

Well my output reports the CPU/GPU/POOL status but "pool not responding" seems a bit strange?
What could you actually do with that? cgminer deals with that itself.

Since the API isn't a constant connection, rather a request + reply - I'm not sure what the point of that would be.
I guess I could add a buffer in cgminer to keep ERROR messages and return the list since the last time?

[-ck]

https://github.com/kanoi/cgminerEdit: it does no mutex locking access to the cpu/gpu/pool info since it would appear that is OK
(the current code doesn't appear to do that either)
ckolivas - does that sound correct to you?

Just getting the info should be safe, if done after everything is set up and all the threads started.

[kano]

https://github.com/kanoi/cgminerEdit: it does no mutex locking access to the cpu/gpu/pool info since it would appear that is OK
(the current code doesn't appear to do that either)
ckolivas - does that sound correct to you?

Just getting the info should be safe, if done after everything is set up and all the threads started.

It's the last thread created.
But I'll check see if there's something I can check/sleep until it's ready - I guess that would be better.
(or add a variable for that exact reason)
Arigatou.

Edit: after a closer look - everything seems to be initialised before the threads are created so it should be fine.
Also now added the parameter options required:
--api-port nnn (port number to use - default 4028)
--api-listen (defaults to only listen on 127.0.0.1 for security - this boolean option means listen on 0.0.0.0)

I guess GPU/CPU enable/disable is next ...

[btc_artist]

But I have no idea what you mean by "(on restart recheck config file for any changes)"

When you send cgminer the restart request and it restarts, load the config file and use values from there.  This is a given, since if it reads the config file when it starts, it will when it restarts too.

[kano]

But I have no idea what you mean by "(on restart recheck config file for any changes)"

When you send cgminer the restart request and it restarts, load the config file and use values from there.  This is a given, since if it reads the config file when it starts, it will when it restarts too.

Hmm - but that doesn't seem to makes sense.

The stop/start/restart request is just for a single GPU.

If your editing the config file you may as well just stop and start cgminer from scratch.
i.e. press q (or send a 'quit' command) and start it again
i.e. run it in a script that just continually loops starting it (like the script I have posted before) and tell it to quit so it starts from scratch.
If you mean a global 'restart' command, not a single GPU stop/start/restart (like quit but that starts again afterwards)
But it really is better to actually completely stop cgminer and restart it since the point of that is to clear the ATI/AMD interface and like windows, that is best done as a complete program stop/start coz ATI/AMD are not good at dealing with that ... and that would also give you all an excuse for not having to pay me too - that should make everyone happy

I'll just implement a GPU stop/start option.

[DeathAndTaxes]

But I have no idea what you mean by "(on restart recheck config file for any changes)"

When you send cgminer the restart request and it restarts, load the config file and use values from there.  This is a given, since if it reads the config file when it starts, it will when it restarts too.

Hmm - but that doesn't seem to makes sense.

The stop/start/restart request is just for a single GPU.

If your editing the config file you may as well just stop and start cgminer from scratch.
i.e. press q (or send a 'quit' command) and start it again
i.e. run it in a script that just continually loops starting it (like the script I have posted before) and tell it to quit so it starts from scratch.
If you mean a global 'restart' command, not a single GPU stop/start/restart (like quit but that starts again afterwards)
But it really is better to actually completely stop cgminer and restart it since the point of that is to clear the ATI/AMD interface and like windows, that is best done as a complete program stop/start coz ATI/AMD are not good at dealing with that ... and that would also give you all an excuse for not having to pay me too - that should make everyone happy

I'll just implement a GPU stop/start option.

Thank makes sense.  I was just thinking that by checking config file on each "restart" one could make changes even if RPC to implement changes in clock/intensity/fan were not implemented.  You are right one could simply end the daemon, change config file, and start the deamon again. 

So unless anyone else has a problem I don't think checking config on starting each GPU is needed or warranted.  It really was just a workaround.  The end goal would be for the project (maybe in later version) to allow controlling GPU via RPC (i.e. change fan 70%, change clock 800MHz, etc).

[kano]

Yep adding GPU setting commands shouldn't be too hard now that I've made a pretty simple to edit interface with messages/errors and everything else reasonably simple ... at least for myself

Git (as before): https://github.com/kanoi/cgminer

OK v0.2 in the git - plus the bunch of changes I spent today adding on top of v0.2 where I tracked down a bug I caused in main.c back when I first started working on this ... yesterday ... (hint 7 <-> 8 )

The commands so far are:
apiversion - just returns a version string: 0.2
devs - returns the list of gpus and cpus - each with the full status, details, plus
pools - returns the list of pools with their status, details
summary - returns the total summary information similar to when you exit cgminer (without pool breakdown)
gpuenable|N - enable GPU N (flags and notifies the GPU to start)
gpudisable|N - disable GPU N (flags the GPU to stop)
gpurestart|N - restart GPU N
gpu|N - show the 'devs' output but just for GPU N
cpu|N - show the 'devs' output but just for CPU N
quit - quit cgminer

Each return includes a status array of:
STATUS: F (fatal) E (error) W (warning) I (info) S (success)
CODE: the internal error code: each has a unique number - see api.c MSG_ZZZZZZ
MSG: a related status message

F (fatal) only occurs when api.c fails to determine the correct error code - these mean report the output line to me to fix api.c
E (error) you made a mistake - e.g. you sent an unknown command or an invalid CPU/GPU number or those sort of mistakes
W (warning) you need to do something else first - currently only when you try to enable a GPU that's not well
I (info) nothing changed - e.g. trying to enable an enabled GPU
S (success) as expected

I've built a 2.0.8a version for xubuntu 11.04 and you can download the binary here to try it (if you can't compile):
http://tradebtc.net/cgminer2.0.8a
1,760,610 bytes, sha256sum: a96ef5ad0914b3becb059f992da93801badcea0dc946120b0199ff31ab4d7b67

Of course you'll need something like api-example.php in the git to try issuing commands to it and see the result.
Otherwise it runs exactly the same as an ADL 2.0.8 so you can use it normally also ... and I didn't even change the donation code

So ... what's missing?

Edit: I'll update the DOC files once I've done the initial set of requirements and sorted out with ckolivas about if he'll accept the commits to his git

[tnkflx]

...
So ... what's missing?

The Windows version? ;-)

[kano]

Hmm I don't actually have a windows compiler or even a computer running windows that could compile it
(just my kids computer runs windows)
I do have a VirtualBox OSE that I run the Aus Tax/Bas programs in on this computer (but of course no ATI card)
Anyone got a link to how I'd compile a OCL/ADL program (using gcc?) on 32-bit WindowsXP for everyone?
(or some other legal free compiler on windows)

Edit: hmmm - mingw or cygwin? Have to read the cgminer thread to work out which one ckolivas was using ...

Edit2: seems to be mingw ...

[-ck]

If and when it's finalised, I will be merging it into my tree, and then eventually I will make a windows version from it when I release a new official version of cgminer. However I would appreciate kano fixing any windows builds before then since things inevitably break when trying to port it.