[LOCKED] cpuminer-opt v3.12.3, open source optimized multi-algo CPU miner

[joblo]

Most of the studies I did were computing related, still most of my knowledge is self-made.
Cobol? There is still a lot around: just look at banks and travel GDSs, for example.

I reecall learning a language called simula, for simulations. I was OO but but thtat was before
OO became popular. While off school the OO buz sarted, andI was reasding a bit about it and wondred
what all the fuss was about. It took me a while to put the pieces together and that I wasn't impressed
with it because  I alredy did it. But simula was an early implentation and didn't have all the feature
of c++.

The irony is that we had one term to learn the labguage nd a second term to use it for as simulation
but between terms they upgraded the system and couldn't get a compiler for the new system.

[zTheWolfz]

Code:

[2016-01-31 00:05:26] accepted: 15/15 (100.00%), 2316 kH/s yes!
[2016-01-31 00:05:34] CPU #47: 49.16 kH/s
^C[2016-01-31 00:05:42] SIGINT received, exiting
thewolf@thewolf-H8QG6:~/Downloads/cpuminer-opt-3.0.7$ ./cpuminer -a x11 -o stratum+tcp://x11.usa.nicehash.com:3336 -u 18tvS3deKZK5q4eTtPRWYeEMWmGmuErHgz.H8QG6 -p d=0.01 -t 48

  48 cores of goodness. Having problems getting the 4th CPU to work :-(

2.3 MH on a CPU, nice! How much do they cost?

The guy wanted 49.99 each, but was open to offers I got them for 42$ each 168 /w free shipped.
My 1st offer was 40 he came back with 45 I counter offer42 and he took the offer.

thewolf@thewolf-H8QG6:~$ sudo clockspeed
[sudo] password for thewolf:
Clockspeed (OCNG5.3)
Family 15h
Turbo is supported. 2 boost state(s).
Running, please wait...
Refclock: 248.003 MHz
Clockspeed: 3224.039 MHz
thewolf@thewolf-H8QG6:~$
 That score was with the OC.
[2016-01-31 05:13:58] accepted: 30/30 (100.00%), 2362 kH/s yes!
[2016-01-31 05:14:09] CPU #18: 49.09 kH/s
[2016-01-31 05:14:10] accepted: 31/31 (100.00%), 2362 kH/s yes!
[2016-01-31 05:14:13] CPU #35: 49.25 kH/s
[2016-01-31 05:14:13] accepted: 32/32 (100.00%), 2362 kH/s yes!
[2016-01-31 05:14:13] CPU #44: 49.21 kH/s
[2016-01-31 05:14:14] accepted: 33/33 (100.00%), 2362 kH/s yes!
[2016-01-31 05:14:19] CPU #45: 49.21 kH/s
[2016-01-31 05:14:19] accepted: 34/34 (100.00%), 2362 kH/s yes!
 
They hit up close to 3.8Ghz on less cores under load with the turbo boost.

[zTheWolfz]

Code:

[2016-01-31 00:05:26] accepted: 15/15 (100.00%), 2316 kH/s yes!
[2016-01-31 00:05:34] CPU #47: 49.16 kH/s
^C[2016-01-31 00:05:42] SIGINT received, exiting
thewolf@thewolf-H8QG6:~/Downloads/cpuminer-opt-3.0.7$ ./cpuminer -a x11 -o stratum+tcp://x11.usa.nicehash.com:3336 -u 18tvS3deKZK5q4eTtPRWYeEMWmGmuErHgz.H8QG6 -p d=0.01 -t 48

  48 cores of goodness. Having problems getting the 4th CPU to work :-(

2.3 MH on a CPU, nice! How much do they cost?

usually this is a bios tweak / setting that disables the cpus ...

opterons mainly - but that was a while back when i used to build these big servers ...

joblo - if you ever want to have a foray into the cuda world - talk to me first please mate ...

#crysx

I wish it was something as simple as a bios tweak. I been around bios and motherboards for many years.
Use to do tech support for Abit motherboards back in the day when they were top dog at overclocking boards.
First motherboard to have jumperless OC, SoftMenu it was called. Then UGuru. lol I was a Moderator at
there forum for many years up until they closed the doors. May have to send this board in for a replacement
Its acted up before this same way, something is going out "or has gone out" for CPU socket 4 on the board
I'm pretty sure.   I'll work with it a little more over the next few days. Hope it still under warranty.  I gave over
$850 just for the M/B alone.

[chrysophylax]

Code:

[2016-01-31 00:05:26] accepted: 15/15 (100.00%), 2316 kH/s yes!
[2016-01-31 00:05:34] CPU #47: 49.16 kH/s
^C[2016-01-31 00:05:42] SIGINT received, exiting
thewolf@thewolf-H8QG6:~/Downloads/cpuminer-opt-3.0.7$ ./cpuminer -a x11 -o stratum+tcp://x11.usa.nicehash.com:3336 -u 18tvS3deKZK5q4eTtPRWYeEMWmGmuErHgz.H8QG6 -p d=0.01 -t 48

  48 cores of goodness. Having problems getting the 4th CPU to work :-(

2.3 MH on a CPU, nice! How much do they cost?

usually this is a bios tweak / setting that disables the cpus ...

opterons mainly - but that was a while back when i used to build these big servers ...

joblo - if you ever want to have a foray into the cuda world - talk to me first please mate ...

#crysx

I wish it was something as simple as a bios tweak. I been around bios and motherboards for many years.
Use to do tech support for Abit motherboards back in the day when they were top dog at overclocking boards.
First motherboard to have jumperless OC, SoftMenu it was called. Then UGuru. lol I was a Moderator at
there forum for many years up until they closed the doors. May have to send this board in for a replacement
Its acted up before this same way, something is going out "or has gone out" for CPU socket 4 on the board
I'm pretty sure.   I'll work with it a little more over the next few days. Hope it still under warranty.  I gave over
$850 just for the M/B alone.

yup - understood ...

my server boards were around the $950aud at the time ( supermicro - amazing boards ) - and they are STILL running to this day ... believe me that was along time ago - about 7 years now - and they 'just work' ... unfortunately for me - at the time xeon systems were the mainstream ... i really wanted an oct opteron system - but the pricetag meant i had to sell the car to buy it ... and i like my fast cars - so the motherboard lost out ...

but i cant complain - these servers are rugged and i have built them with the best i could find at the time ... money ( and a lot of it ) well spend then ...

maybe it is the socket itself - sometime a regulator or cap gone bad ... it happens ...

i hope you can get it running though - it sounds like you have one hell of a system there ...

#crysx

[chrysophylax]

Code:

[2016-01-31 00:05:26] accepted: 15/15 (100.00%), 2316 kH/s yes!
[2016-01-31 00:05:34] CPU #47: 49.16 kH/s
^C[2016-01-31 00:05:42] SIGINT received, exiting
thewolf@thewolf-H8QG6:~/Downloads/cpuminer-opt-3.0.7$ ./cpuminer -a x11 -o stratum+tcp://x11.usa.nicehash.com:3336 -u 18tvS3deKZK5q4eTtPRWYeEMWmGmuErHgz.H8QG6 -p d=0.01 -t 48

  48 cores of goodness. Having problems getting the 4th CPU to work :-(

2.3 MH on a CPU, nice! How much do they cost?

usually this is a bios tweak / setting that disables the cpus ...

opterons mainly - but that was a while back when i used to build these big servers ...

joblo - if you ever want to have a foray into the cuda world - talk to me first please mate ...

#crysx

It's only a matter of time. Thi srearchitecting of the front end is going to pay off well because
it's identical to ccminer. I;m also learning a lot about the algo chains and how hashing is done.
The only thingI'll have left to struggle with is the cuda architecture because it's so foreign to
anything I'm familiar with and good docs are hard to find. But I'm hoping I can be useful.

well - if there is an option there - come and have a chat ...

i dont think it will take long for you to get a grasp of things - it will take a while to really get fluent with it all - and to get to the 'other' devs levels ( yes djm34 wolf0 sp pallas and many many more ) its a matter of experience and time ...

i kick myself everytime i think of how i was the top 3% of university and i dropped out - just before my degree was getting into c++ heavily ... id already got pascal and cobol almost down pat - c++ was next ...

now there is bringing back a HUGE amount of memories ... yup - i know why i dropped out - stupid young fella who knew everything - i was ... ...

#crysx

It took me 11 years to complete mine but I went off track in year 1, had a good paying PT job and cut classes for
more hours. The job was a dead end and and eventually I got motivated to go back to school, just to get out of
that job. I don't now how old you are but it's never too late. Cobol eh? There may still be demand for Cobol
support.

yup - cobol ... and pascal - one of my favourite procedural languages at the time ...

id never go back to cobol - and the money to get me back would be too much to spend ... what a nightmare that language is ...

c++ on the other hand ... that was something i have always wanted to get back into ...

and yes - im old ... 46 about to hit the 47 in a few months ...

fun fun ... and nite nite ...

#crysx

[pallas]

Do you guys remember the first tomshardware (it was sysdoc.pair.com back then) article on the Asus p55t2p4 motherboard?
I had that card and it was so fun! By specs the fastest cpu it could run was a pentium 233 but with some hardware mods you could easily run a k6 at 450 :-D

[joblo]

Here's a sample of algo_gate code as a teaser. The funny syntax is just how c defines
function pointers.  This technique used used a lot in file systems and device
drivers due to it's flexibility and extensibility.

It's kinda OOish but without the pretensions.

I'm just tidying up the syntax errors and should have a compile later today

Code:

// declare some function pointers
// mandatory functions require a custom target function specific to each algo.
// Otherwise the null instance will return a fail code.
// Optional functions may not be required for certain algos or the null
// instance provides a safe default. If the default null instance is suitable for
// a particular algo it is not necessary to define a custom target function.

typedef struct
{
//mandatory
int    *( *scanhash ) ( int, uint32_t*, const uint32_t*, uint32_t, uint64_t* );
void   *( *hash )( void*, const void*, uint32_t ) ;
//optional
void   *( *init_ctx ) ();
uint32_t *( *get_max64 ) ();
void     *( *get_opt) ();
bool     *( *full_test) ();
void     *( *set_target) ( struct work*, uint32_t, uint32_t );
void     *( *gen_merkle_root) ( char* merkle_root, struct stratum_ctx* );
bool     *( *get_scratchbuf ) ( char* );
int      *( *check_algo_alias) ( enum algos );
void     *( *ignore_pok) ( int, int );
void     *( *display_hashrate)( int, uint64_t );
void     *( *display_benchmark_hashrate)( int, uint64_t* );
void     *( *wait_for_diff) ( struct stratum_ctx* );
} algo_gate_t;

Each algo then implements the target functions it needs and a function to register
them with algo_gate:

Code:

bool register_x11_algo( algo_gate_t* gate )
{
  gate->init_ctx = &init_x11_aes_ctx;
  gate->scanhash = &scanhash_x11_aes;
  gate->hash     = &x11_hash_aes;
  return true;
};

Each miner thread then calls the registration function for the algo to be mined

Code:

   if ( false == register_algo_gate( opt_algo, algo_gate[thr_id] ) )
   {
     applog(LOG_ERR,"Thread %d algo_gate registration failed.\n", thr_id);
     if ( NULL != algo_gate )
     {
        free( algo_gate );
     }
     exit(1);
   }

And the functions are called:

Code:

    if ( algo_gate->scanhash != null_scanhash )
    {
       rc = algo_gate->scanhash( thr_id, work.data, work.target, max_nonce,
                                                          &hashes_done );
    }
    else
    {
       applog(LOG_ERR,"%s has no scanhash function registered.\n",
                                              algo_names[opt_algo]);
    }

And that's all there is to it. There ar no references to any specific algos in the miner thread, it is completely
absract.

[joblo]

Well the algo gate engine is built and it starts but I seem to be caught in a vicious loop.

I need the proper algo name in order to register the gate, but I need the gate registered
before I can check for algo aliases. It looks like I need to poke a hole in the gate.

But that also exposes a fatal logical flaw. I have the algo entered by the user which may or may not
be an alias. I can't map directly from an alias to a proper name, I have to poll each algo if they
accept the given name as an alias for them. A little redesign is in order.

There were some other issues such as the need to register the gate before the miner threeads are
started. I just let main use thread 0's gate.

But I'm also having problems with malloc, probably the size. I've declared a global struct pointer
as the root of the gate data, but how do I alloc the mem so this pointer points to a buffer of size
threads*struct size? sizeof only accepts variables, but theonly variableI have is a pointer.
I tried an arbitrary constant but that didn't work either. Do I need a double pointer because it's
an array of structs?

[pallas]

Sizeof accepts data type names including structs, so sizeof (struct_name) * threads should work.

[joblo]

Sizeof accepts data type names including structs, so sizeof (struct_name) * threads should work.

Hmm, I though I tried using a type name before and it didn't work, I must have been mistaken.
As I also mentioned it didn't work with a constant size either so there's more to it.

I got it running with a fixed sized array for now amd it gets all the way to diplaying the hashrates
before it segfaults. Had a lot of initialization issues to work out. printf is the best debugging tool
available (except for the ones I wrote).

Haven't got stuck any one issue for very long so progressing well. Seen some interesting crashes.
I crashed on a function return, obviously corrupted the stack with and extra & on a var.
Forgetting a ; at the end of an included file produces interesting errors.

ll revisit the malloc isue later, thanks.

[joblo]

I have a solution for the algo alias issue, but not a perfect one. It requires a little more work to install
a new algo. If the algo has an alias it requires addding an entry to the alias array. So it's now a two
step process: add a case to reguister the algo and add an entry to the alias array if appropriate.

The redesigned engine runs, connects to the pool, hashes and reports hashrates ok and submits shares,
but, all rejects. I probably messed something up in cpu-miner, I've already fixed a few of them.
I'll have to walk through the code side by side and look for any divergences.

[MaxDZ8]

If I'm not mistaken that sounds a lot like sgminer

No idea how did you get to that conclusion. Indeed the verificators needs to be compiled for the time being (like sgminer) except that every verificator can be re-assembled on need at runtime.
For CPU mining no, that's not really going to work efficiently, as the hashing itself is simple most of the time and the cost involved in CALL will be overwhelming. Yet you could design a proper DLL interface considering your needs (no, it is not easy).

Dll based? (shudder) that's Windows? Or do you mean shared libraries? My concern about shared libraries
it could cap performnance to the lowest common denoniminator.

No idea why you guys are so tied to stickers. DLLs and SOs are pretty much the same thing for most practical purposes. Dynamic linking based on runtime capabilities has been around by decades and it can be efficient - the old school example is linking games to Glide/D3D/GL by runtime or even GL extensions on GPU capabilities. You can do the same, detect feature set X, fetch library_X.so, feature set Y fetch library_Y.so. If the interface is purely functional it will be relatively easy especially in plain C and will - in theory - allow a single executable to support every future algorithm.

That is, if you want to go in that direction.

[joblo]

Sizeof accepts data type names including structs, so sizeof (struct_name) * threads should work.

much ado about nothing. simpler to have each miner thread define a local copy, all refs are
within scope so should be no problem.

[joblo]

algo_gate update

I have a stable build with about half the algo_gate functions enabled. Importantly scanhash
is one of them.

Current status is x11 and quark tested with half the algo_gate functions enabled, No noticeable
change in performance.  Most definitions moved from cpu-miner.c to miner.h.
All algo specific function declarations removed from miner.h as they are no longer needed.

I will proceed to clean up the bloat in the algo files by merging the multiple targets into one
file with the AES_NI code protected by #ifdef.

Then I'll try to fix the broken algo_gate functions but may release a partially implemented version
if it takes too long.

[joblo]

algo_gate update

I have a stable build with about half the algo_gate functions enabled. Importantly scanhash
is one of them.

Current status is x11 and quark tested with half the algo_gate functions enabled, No noticeable
change in performance.  Most definitions moved from cpu-miner.c to miner.h.
All algo specific function declarations removed from miner.h as they are no longer needed.

I will proceed to clean up the bloat in the algo files by merging the multiple targets into one
file with the AES_NI code protected by #ifdef.

Then I'll try to fix the broken algo_gate functions but may release a partially implemented version
if it takes too long.

Further update:

All of the original gate functions are are working but I needed to expand the system. each miner_thread
has a gate as expected but the stratum thread also needs one. I also need to create a second gate for
algo specific command line options. It was tricky to implement because the opt_gate could not be
registered until algo aliases were resolved and that was done 5 lines before the first use of the gate.
It also had to be declared static so the registration would survive the functions exit and re-entry.
That aslo required a global flag to prevent re-registration. But it all worked out.

I'm just chasing down the last few references to individual algos in cpu-miner.c. Soon it will be
completely agnostic. Then thourough testing then a dev preview.

Edit: I should add that I have observed a drop in performance since implementing the gate. However,
I have also noticed a drop in the previous release so there may be other variables. If there is
a net loss is performance I will not release cpuminer-opt with the new scanhash engine.
 

[pallas]

Thanks for your work joblo. I'll be glad to test it as soon as you feel it's ready.
May I again suggest keeping a github repository, so we can all easily submit patches.

[joblo]

Hot tip to increase your profit.

When mining with two instances of cpuminer on different
algos I can get a combined hashrate above 100% of either algo mining alone. I mined cryptonight
and quark together with 7 threads each on an 8 thread i7 and each instance reported between 60%
and 70% of the rate when mining alone.

The quark instance reported 80% rejects locally but no rejects and full hashrate at the pool.

The benefit probably depends on the different nature of the two algos. Some combinations may work
better than others. I suspect quark vs cryptonight to be one of the better combinations.

Edit: the rejects are low diff, could this be a target issue. set_target is one of the new gate functions.

[joblo]

May I again suggest keeping a github repository, so we can all easily submit patches.

Nice reminder. I don't want struggle with yet another system to learn at this point. From my experience
with c++, autoconf, visual stdio, etc,  it means I will probably be held back by baggage from my profesional carrer
and will have to unlearn it first. I do intend to get on github after 3.1 is released, with or without algo_gate.

[joblo]

Another update on algo_gate. Testing is going well with x11, quark, neo, cryptonight working. I have
yet to revisit quark and the false positive rejects. It's not "service affecting" as they say. Still unclear
on the effect on performance, if any. I wasn't expecting any.

Had a weird bug in the hash rate display for cryptonight. I could get it to display .02 (kH) or 24000 (mH) but
only got zero when not multiplying. Then it hit me the multiplier was a float and the hashrate was int64
so the multiplication was forcing it to float. With no multiplier the hashrate was still integer. I guess
formatting an int as a float doesn't work very well.

On to test a few more algos, then noaesni. It's getting closer.

[joblo]

Yet another algo_gate update.

I have slighly changed th eimplemetaion of algo_gate. Inseatd of every thread that
needs it defined its own and it was up to 7 seperate engines. No wthere is onlu one engine
defined in cpu-miner.c and only visible to it.

I have noticed some algo specific code didn't do anything so I'll dleete the code and not define
a gate function.

I have decided to drop support for scrypt and scryptn. scrypt cpu miners can use pooler's
cpuminer.

Back to testing.