[LOCKED] cpuminer-opt v3.12.3, open source optimized multi-algo CPU miner

[joblo]

Yet another algo_gate update.

I have slighly changed th eimplemetaion of algo_gate. Inseatd of every thread that
needs it defined its own and it was up to 7 seperate engines. No wthere is onlu one engine
defined in cpu-miner.c and only visible to it.

Well, that was an interesting, but failed, experiment, and a learning experience. It's back
to multiple gates.

I had defined a global static gate and initialized in in main before creating any threads,
but it kept segfaulting whenver trying to call a gate function, from a thread other than main.

Apparently the defined gate was not global. Each thread had its own private copy and only
one was initialized.

I thought it was simply a matter of initializing the gate in all threads but that didn't work either.
I thought this would work since the compiler bever complained about what I was doing.
I may not have done it properly. I may give that another try when the dust settles.

I eventually went back to each thread defining a local gate. Multiple copies of the same data
but wtf.

So we will have cpu threads +3 gates. The extra 3 are for main, stratum_thread and longpoll_thread.

It's a good thing the gate is lightweight.

Yet another concept with which I am familiar but with a completely foreign implementation.

I also added MH/s units for those high hashing algos. And I may be able to implemt a non aesni
version of cryptonight.

[joblo]

Preliminary testing of quark using the algo_gate has increased performance 4%,
completely unexpected. I dodn't see any imnherent opportunities for increases
unless the smaller code base caches better.

[joblo]

Preliminary testing of quark using the algo_gate has increased performance 4%,
completely unexpected. I dodn't see any imnherent opportunities for increases
unless the smaller code base caches better.

Further testing is showing a 4 to 5% increase in all algos on my 4790k.
Will test non aesni (sw build) as well.

Current errata:

- gate fully implemneted for all minig activities (cpu-miner.c) with no violations of the gate.

- utility functions (util.c) not yet converted.

- algo aliases not working, must use lyra2re and lyra2rev2

- axiom algo not working

- hash rate output for share submission doesn't scale fo rvery low hash rates.

- all major algos have been tested, obscure algos have more custom gate functions and are therefore more
  likely to have issues and may not be fully tested.

- non aesni will be tested

- no windows version

- dev preview will be production code (ie no debug) but still has commenetd out sections
  of the old code in places for comparison and testing purposes.

- SSE2 versions not recommended in v3.1, use 3.0.7 instead

All issues to be addressed before final release of v3.1 but will not delay dev's preview.

[pallas]

latest dev beta has some issues on lyra2:

algo/lyra2/lyra2.c:217:46: warning: ‘struct work’ declared inside parameter list
                                              double factor )
                                              ^
algo/lyra2/lyra2.c:217:46: warning: its scope is only this definition or declaration, which is probably not what you want
algo/lyra2/lyra2.c: In function ‘lyra2_set_target’:
algo/lyra2/lyra2.c:219:2: warning: implicit declaration of function ‘work_set_target’ [-Wimplicit-function-declaration]
  work_set_target( work, job_diff / (65536.0 * factor) );
  ^
algo/lyra2/lyra2.c: At top level:
algo/lyra2/lyra2.c:223:1: error: unknown type name ‘bool’
 bool register_lyra2( algo_gate_t gate )
 ^
algo/lyra2/lyra2.c:223:22: error: unknown type name ‘algo_gate_t’
 bool register_lyra2( algo_gate_t gate )

[joblo]

latest dev beta has some issues on lyra2:

algo/lyra2/lyra2.c:217:46: warning: ‘struct work’ declared inside parameter list
                                              double factor )
                                              ^
algo/lyra2/lyra2.c:217:46: warning: its scope is only this definition or declaration, which is probably not what you want
algo/lyra2/lyra2.c: In function ‘lyra2_set_target’:
algo/lyra2/lyra2.c:219:2: warning: implicit declaration of function ‘work_set_target’ [-Wimplicit-function-declaration]
  work_set_target( work, job_diff / (65536.0 * factor) );
  ^
algo/lyra2/lyra2.c: At top level:
algo/lyra2/lyra2.c:223:1: error: unknown type name ‘bool’
 bool register_lyra2( algo_gate_t gate )
 ^
algo/lyra2/lyra2.c:223:22: error: unknown type name ‘algo_gate_t’
 bool register_lyra2( algo_gate_t gate )

HMMM, I thought I fixed that already, need to add #include <stdbool.h> to lyra2.c.
Those last minute changes are a killer. Good thing it's only beta.

Note that lyra2 in the code is not the same as -a lyra2 on the command line and
is not on the supported algo list. But a compile error, that's bad.

Edit: ultimately what I intended was to backout the last minute changes but chose
the wrong snapshot. You can comment out  the body of register_lyra2_algo.

[pallas]

latest dev beta has some issues on lyra2:

algo/lyra2/lyra2.c:217:46: warning: ‘struct work’ declared inside parameter list
                                              double factor )
                                              ^
algo/lyra2/lyra2.c:217:46: warning: its scope is only this definition or declaration, which is probably not what you want
algo/lyra2/lyra2.c: In function ‘lyra2_set_target’:
algo/lyra2/lyra2.c:219:2: warning: implicit declaration of function ‘work_set_target’ [-Wimplicit-function-declaration]
  work_set_target( work, job_diff / (65536.0 * factor) );
  ^
algo/lyra2/lyra2.c: At top level:
algo/lyra2/lyra2.c:223:1: error: unknown type name ‘bool’
 bool register_lyra2( algo_gate_t gate )
 ^
algo/lyra2/lyra2.c:223:22: error: unknown type name ‘algo_gate_t’
 bool register_lyra2( algo_gate_t gate )

HMMM, I thought I fixed that already, need to add #include <stdbool.h> to lyra2.c.
Those last minute changes are a killer. Good thing it's only beta.

Note that lyra2 in the code is not the same as -a lyra2 on the command line and
is not on the supported algo list. But a compile error, that's bad.

thanks, but not enough:

algo/lyra2/lyra2.c:218:46: warning: ‘struct work’ declared inside parameter list
                                              double factor )
                                              ^
algo/lyra2/lyra2.c:218:46: warning: its scope is only this definition or declaration, which is probably not what you want
algo/lyra2/lyra2.c: In function ‘lyra2_set_target’:
algo/lyra2/lyra2.c:220:2: warning: implicit declaration of function ‘work_set_target’ [-Wimplicit-function-declaration]
  work_set_target( work, job_diff / (65536.0 * factor) );
  ^
algo/lyra2/lyra2.c: At top level:
algo/lyra2/lyra2.c:224:22: error: unknown type name ‘algo_gate_t’
 bool register_lyra2( algo_gate_t gate )

[joblo]

latest dev beta has some issues on lyra2:

algo/lyra2/lyra2.c:217:46: warning: ‘struct work’ declared inside parameter list
                                              double factor )
                                              ^
algo/lyra2/lyra2.c:217:46: warning: its scope is only this definition or declaration, which is probably not what you want
algo/lyra2/lyra2.c: In function ‘lyra2_set_target’:
algo/lyra2/lyra2.c:219:2: warning: implicit declaration of function ‘work_set_target’ [-Wimplicit-function-declaration]
  work_set_target( work, job_diff / (65536.0 * factor) );
  ^
algo/lyra2/lyra2.c: At top level:
algo/lyra2/lyra2.c:223:1: error: unknown type name ‘bool’
 bool register_lyra2( algo_gate_t gate )
 ^
algo/lyra2/lyra2.c:223:22: error: unknown type name ‘algo_gate_t’
 bool register_lyra2( algo_gate_t gate )

HMMM, I thought I fixed that already, need to add #include <stdbool.h> to lyra2.c.
Those last minute changes are a killer. Good thing it's only beta.

Note that lyra2 in the code is not the same as -a lyra2 on the command line and
is not on the supported algo list. But a compile error, that's bad.

thanks, but not enough:

algo/lyra2/lyra2.c:218:46: warning: ‘struct work’ declared inside parameter list
                                              double factor )
                                              ^
algo/lyra2/lyra2.c:218:46: warning: its scope is only this definition or declaration, which is probably not what you want
algo/lyra2/lyra2.c: In function ‘lyra2_set_target’:
algo/lyra2/lyra2.c:220:2: warning: implicit declaration of function ‘work_set_target’ [-Wimplicit-function-declaration]
  work_set_target( work, job_diff / (65536.0 * factor) );
  ^
algo/lyra2/lyra2.c: At top level:
algo/lyra2/lyra2.c:224:22: error: unknown type name ‘algo_gate_t’
 bool register_lyra2( algo_gate_t gate )

Oh yeah, add #include "algo-gate-api.h" also.

[pallas]

Oh yeah, add #include "algo-gate-api.h" also.

algo/lyra2/lyra2.c: In function ‘register_lyra2’:
algo/lyra2/lyra2.c:227:7: error: invalid type argument of ‘->’ (have ‘algo_gate_t {aka struct <anonymous>}’)
   gate->scahhash   = &LYRA2;
       ^
algo/lyra2/lyra2.c:228:7: error: invalid type argument of ‘->’ (have ‘algo_gate_t {aka struct <anonymous>}’)
   gate->init_ctx   = &initState;
       ^
algo/lyra2/lyra2.c:229:7: error: invalid type argument of ‘->’ (have ‘algo_gate_t {aka struct <anonymous>}’)
   gate->set_target = &lyra2_set_target;

[joblo]

Oh yeah, add #include "algo-gate-api.h" also.

algo/lyra2/lyra2.c: In function ‘register_lyra2’:
algo/lyra2/lyra2.c:227:7: error: invalid type argument of ‘->’ (have ‘algo_gate_t {aka struct <anonymous>}’)
   gate->scahhash   = &LYRA2;
       ^
algo/lyra2/lyra2.c:228:7: error: invalid type argument of ‘->’ (have ‘algo_gate_t {aka struct <anonymous>}’)
   gate->init_ctx   = &initState;
       ^
algo/lyra2/lyra2.c:229:7: error: invalid type argument of ‘->’ (have ‘algo_gate_t {aka struct <anonymous>}’)
   gate->set_target = &lyra2_set_target;

This is deja vu. I've seen each of those errors before. The starting point had no algo-gate code
in lyra2.c No gate functions, no registration function and no includes changes. That os what I
intended to deliver.

My apologies for messing up so bad. I'll get a new build out soon.

[joblo]

Oh yeah, add #include "algo-gate-api.h" also.

algo/lyra2/lyra2.c: In function ‘register_lyra2’:
algo/lyra2/lyra2.c:227:7: error: invalid type argument of ‘->’ (have ‘algo_gate_t {aka struct <anonymous>}’)
   gate->scahhash   = &LYRA2;
       ^
algo/lyra2/lyra2.c:228:7: error: invalid type argument of ‘->’ (have ‘algo_gate_t {aka struct <anonymous>}’)
   gate->init_ctx   = &initState;
       ^
algo/lyra2/lyra2.c:229:7: error: invalid type argument of ‘->’ (have ‘algo_gate_t {aka struct <anonymous>}’)
   gate->set_target = &lyra2_set_target;

This is deja vu. I've seen each of those errors before. The starting point had no algo-gate code
in lyra2.c No gate functions, no registration function and no includes changes. That os what I
intended to deliver.

My apologies for messing up so bad. I'll get a new build out soon.

Beta2:

https://drive.google.com/file/d/0B0lVSGQYLJIZbElvRWdJWFBKV2s/view?usp=sharing

[pallas]

sorry sir:

./algo/groestl/sse2/groestl.c:37:25: fatal error: sph_groestl.h: No such file or directory

[.....]

algo/qubit/qubit.c: In function ‘qubithash’:
algo/qubit/qubit.c:89:23: error: ‘ctx_echo’ undeclared (first use in this function)
         sph_echo512 (&ctx_echo, (const void*) hash, 64);

Are you sure you did "make clean" before compiling it?

[joblo]

sorry sir:

./algo/groestl/sse2/groestl.c:37:25: fatal error: sph_groestl.h: No such file or directory

[.....]

algo/qubit/qubit.c: In function ‘qubithash’:
algo/qubit/qubit.c:89:23: error: ‘ctx_echo’ undeclared (first use in this function)
         sph_echo512 (&ctx_echo, (const void*) hash, 64);

Are you sure you did "make clean" before compiling it?

Are you compiling for SSE2? That doesn't work.

[pallas]

sorry sir:

./algo/groestl/sse2/groestl.c:37:25: fatal error: sph_groestl.h: No such file or directory

[.....]

algo/qubit/qubit.c: In function ‘qubithash’:
algo/qubit/qubit.c:89:23: error: ‘ctx_echo’ undeclared (first use in this function)
         sph_echo512 (&ctx_echo, (const void*) hash, 64);

Are you sure you did "make clean" before compiling it?

Are you compiling for SSE2? That doesn't work.

uh... yes... sorry :-/

[joblo]

sorry sir:

./algo/groestl/sse2/groestl.c:37:25: fatal error: sph_groestl.h: No such file or directory

[.....]

algo/qubit/qubit.c: In function ‘qubithash’:
algo/qubit/qubit.c:89:23: error: ‘ctx_echo’ undeclared (first use in this function)
         sph_echo512 (&ctx_echo, (const void*) hash, 64);

Are you sure you did "make clean" before compiling it?

Are you compiling for SSE2? That doesn't work.

I have confirmed that error only occurs when compiling for SSE2.

[joblo]

sorry sir:

./algo/groestl/sse2/groestl.c:37:25: fatal error: sph_groestl.h: No such file or directory

[.....]

algo/qubit/qubit.c: In function ‘qubithash’:
algo/qubit/qubit.c:89:23: error: ‘ctx_echo’ undeclared (first use in this function)
         sph_echo512 (&ctx_echo, (const void*) hash, 64);

Are you sure you did "make clean" before compiling it?

Are you compiling for SSE2? That doesn't work.

uh... yes... sorry :-/

I working on fixing sse2 for release. I don't think the break is due to algo-gate but the file
tree reorg and consolodation. Prior to that the quark sse2 code was in a seperate file.
Now it shares code with AES_NI and I need to rwview all the hooks to keep them seperate.

[joblo]

sorry sir:

./algo/groestl/sse2/groestl.c:37:25: fatal error: sph_groestl.h: No such file or directory

[.....]

algo/qubit/qubit.c: In function ‘qubithash’:
algo/qubit/qubit.c:89:23: error: ‘ctx_echo’ undeclared (first use in this function)
         sph_echo512 (&ctx_echo, (const void*) hash, 64);

Are you sure you did "make clean" before compiling it?

Are you compiling for SSE2? That doesn't work.

uh... yes... sorry :-/

I working on fixing sse2 for release. I don't think the break is due to algo-gate but the file
tree reorg and consolodation. Prior to that the quark sse2 code was in a seperate file.
Now it shares code with AES_NI and I need to rwview all the hooks to keep them seperate.

This may turn out to be a good thing because it gave me an idea about the problems I've been
having with the sse2 groestl macros. The macros are defined in a file with regular functions
and I think I need to split them up so i can configure each one differently in Makefile, #include, etc.
Could be a boost for x* sse2. Might get it done before 3.1 release.

[joblo]

sorry sir:

./algo/groestl/sse2/groestl.c:37:25: fatal error: sph_groestl.h: No such file or directory

[.....]

algo/qubit/qubit.c: In function ‘qubithash’:
algo/qubit/qubit.c:89:23: error: ‘ctx_echo’ undeclared (first use in this function)
         sph_echo512 (&ctx_echo, (const void*) hash, 64);

Are you sure you did "make clean" before compiling it?

Are you compiling for SSE2? That doesn't work.

uh... yes... sorry :-/

I working on fixing sse2 for release. I don't think the break is due to algo-gate but the file
tree reorg and consolodation. Prior to that the quark sse2 code was in a seperate file.
Now it shares code with AES_NI and I need to rwview all the hooks to keep them seperate.

This may turn out to be a good thing because it gave me an idea about the problems I've been
having with the sse2 groestl macros. The macros are defined in a file with regular functions
and I think I need to split them up so i can configure each one differently in Makefile, #include, etc.
Could be a boost for x* sse2. Might get it done before 3.1 release.

This was a good thing. I have now solved the GRS macro problem. It was a metter of seperating the macros
and real functions into seperate files, and not adding the macro file to Makefile.

V3.1 will have significant improvements in SSE2 performance.

Edit: X algo sse2 performance will have an additional 10% increase on top of the 4% already in v3.1-beta.
        v3.1 will be released later today. Thanks to Pallas for freaking me out into solving a long nagging issue
       with the GRS macros.

[pallas]

You are welcome :-)
And thanks for your great effort.
I'll be "configure&&make"-ing again on Monday.

[joblo]

Have a nice weekend.

I am considering dropping suport for scrypt as a standalone algo. It's integration
with algo_gate didn't go so well, and there's allways Pooler.

However Pooler claims to have a very optimized scrypt and may be better than the
current one in cpuminer-opt and its ancestor cpuminer-multi.

I will investigate further as it may speed up  some other algos.

[joblo]

cpuminer-opt v3.1 is out. It icludes th enew algo gate and a 5% increase in
performance in all algos.  There is an addtional 10% increase in SSE2 performance.

https://drive.google.com/file/d/0B0lVSGQYLJIZTHdkVG9CSXhObHM/view?usp=sharing