[LOCKED] cpuminer-opt v3.12.3, open source optimized multi-algo CPU miner

[Giulini]

kim@spiel-2:~/cpuminer-opt-3.1.17$ ./configure CFLAGS="-O3 -march=btver1" --with-curl --with-crypto && make
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking target system type... x86_64-unknown-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... no
checking for mawk... mawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether to enable maintainer-specific portions of Makefiles... no
checking for style of include used by make... GNU
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking dependency style of gcc... gcc3
checking for gcc option to accept ISO C99... -std=gnu99
checking how to run the C preprocessor... gcc -std=gnu99 -E
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking whether gcc -std=gnu99 needs -traditional... no
checking dependency style of gcc -std=gnu99... gcc3
checking for ranlib... ranlib
checking for g++... g++
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking dependency style of g++... gcc3
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking sys/endian.h usability... no
checking sys/endian.h presence... no
checking for sys/endian.h... no
checking sys/param.h usability... yes
checking sys/param.h presence... yes
checking for sys/param.h... yes
checking syslog.h usability... yes
checking syslog.h presence... yes
checking for syslog.h... yes
checking for sys/sysctl.h... yes
checking whether be32dec is declared... no
checking whether le32dec is declared... no
checking whether be32enc is declared... no
checking whether le32enc is declared... no
checking whether le16dec is declared... no
checking whether le16enc is declared... no
checking for size_t... yes
checking for working alloca.h... yes
checking for alloca... yes
checking for getopt_long... yes
checking whether we can compile AVX code... yes
checking whether we can compile XOP code... yes
checking whether we can compile AVX2 code... yes
checking for json_loads in -ljansson... yes
checking for pthread_create in -lpthread... yes
checking whether __uint128_t is supported... yes
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating compat/Makefile
config.status: creating compat/jansson/Makefile
config.status: creating cpuminer-config.h
config.status: cpuminer-config.h is unchanged
config.status: executing depfiles commands
make  all-recursive
make[1]: Verzeichnis »/home/kim/cpuminer-opt-3.1.17« wird betreten
Making all in compat
make[2]: Verzeichnis »/home/kim/cpuminer-opt-3.1.17/compat« wird betreten
make[3]: Verzeichnis »/home/kim/cpuminer-opt-3.1.17/compat« wird betreten
make[3]: Für das Ziel »all-am« ist nichts zu tun.
make[3]: Verzeichnis »/home/kim/cpuminer-opt-3.1.17/compat« wird verlassen
make[2]: Verzeichnis »/home/kim/cpuminer-opt-3.1.17/compat« wird verlassen
make[2]: Verzeichnis »/home/kim/cpuminer-opt-3.1.17« wird betreten
gcc -std=gnu99 -DHAVE_CONFIG_H -I.  -Iyes/include -Iyes/include -fno-strict-aliasing  -I. -Iyes/include -Iyes/include -Wno-pointer-sign -Wno-pointer-to-int-cast   -O3 -march=btver1  -Iyes/include -Iyes/include -MT algo/hodl/cpuminer-hodl-gate.o -MD -MP -MF algo/hodl/.deps/cpuminer-hodl-gate.Tpo -c -o algo/hodl/cpuminer-hodl-gate.o `test -f 'algo/hodl/hodl-gate.c' || echo './'`algo/hodl/hodl-gate.c
algo/hodl/hodl-gate.c: In function ‘register_hodl_algo’:
algo/hodl/hodl-gate.c:132:7: error: ‘algo_gate_t’ has no member named ‘set_data_size’
   gate->set_data_size          = (void*)&hodl_set_data_size;
       ^
make[2]: *** [algo/hodl/cpuminer-hodl-gate.o] Fehler 1
make[2]: Verzeichnis »/home/kim/cpuminer-opt-3.1.17« wird verlassen
make[1]: *** [all-recursive] Fehler 1
make[1]: Verzeichnis »/home/kim/cpuminer-opt-3.1.17« wird verlassen
make: *** [all] Fehler 2
kim@spiel-2:~/cpuminer-opt-3.1.17$

[joblo]

algo/hodl/hodl-gate.c:132:7: error: ‘algo_gate_t’ has no member named ‘set_data_size’
   gate->set_data_size          = (void*)&hodl_set_data_size;
       ^
make[2]: *** [algo/hodl/cpuminer-hodl-gate.o] Fehler 1
make[2]: Verzeichnis »/home/kim/cpuminer-opt-3.1.17« wird verlassen
make[1]: *** [all-recursive] Fehler 1
make[1]: Verzeichnis »/home/kim/cpuminer-opt-3.1.17« wird verlassen
make: *** [all] Fehler 2
kim@spiel-2:~/cpuminer-opt-3.1.17$

That's a coding error. I just recompiled successfuly from the tarball I uploaded  so don't know why you got
that error.

[hmage]

algo/hodl/hodl-gate.c:132:7: error: ‘algo_gate_t’ has no member named ‘set_data_size’
   gate->set_data_size          = (void*)&hodl_set_data_size;
       ^
make[2]: *** [algo/hodl/cpuminer-hodl-gate.o] Fehler 1
make[2]: Verzeichnis »/home/kim/cpuminer-opt-3.1.17« wird verlassen
make[1]: *** [all-recursive] Fehler 1
make[1]: Verzeichnis »/home/kim/cpuminer-opt-3.1.17« wird verlassen
make: *** [all] Fehler 2
kim@spiel-2:~/cpuminer-opt-3.1.17$

That's a coding error. I just recompiled successfuly from the tarball I uploaded  so don't know why you got
that error.

This line happens when you're compiling for core2. You probably removed set_data_size everywhere but on that line. I had to remove it in my fork.

Tried it out to delete the sourcecode line, then "SW built for SSE2..........NO." change to "YES", but the miner stops working immediately; tried also several -march versions

same here with: Sempron145 CPU, configure und make with no mistakes

Code:

        **********  cpuminer-opt 3.1.16  ***********
     A CPU miner with multi algo support and optimized for CPUs
     with AES_NI extension.
     BTC donation address: 12tdvfF7KmAsihBXQXynT6E6th2c2pByTT
     Forked from TPruvot's cpuminer-multi with credits
     to Lucas Jones, elmad, palmd, djm34, pooler, ig0tik3d,
     Wolf0 and Jeff Garzik.

Checking CPU capatibility...
        AMD Phenom(tm) II X4 940 Processor
   CPU arch supports AES_NI...NO.
   CPU arch supports SSE2.....YES.
   SW built for SSE2..........NO.
Incompatible SW build, rebuild with "-march=native"

Why?

Try using this fork -- https://github.com/hmage/cpuminer-opt -- it should report SSE and AES properly. Let us know if it works, I'll offer the diff to joblo.

[joblo]

algo/hodl/hodl-gate.c:132:7: error: ‘algo_gate_t’ has no member named ‘set_data_size’
   gate->set_data_size          = (void*)&hodl_set_data_size;
       ^
make[2]: *** [algo/hodl/cpuminer-hodl-gate.o] Fehler 1
make[2]: Verzeichnis »/home/kim/cpuminer-opt-3.1.17« wird verlassen
make[1]: *** [all-recursive] Fehler 1
make[1]: Verzeichnis »/home/kim/cpuminer-opt-3.1.17« wird verlassen
make: *** [all] Fehler 2
kim@spiel-2:~/cpuminer-opt-3.1.17$

That's a coding error. I just recompiled successfuly from the tarball I uploaded  so don't know why you got
that error.

This line happens when you're compiling for core2. You probably removed set_data_size everywhere but on that line. I had to remove it in my fork.

This is getting really weird. I reproduced the compile error with core2. It doesn't make any sense because
set_data_size is not conditional on the arch. I need to really dig into this.

Edit: I take that back, I found some weird code I need to sort out.

Edit2: set_data_size is a coding error on my part. When hodl was introduced set_data_size became targetted.
In 3.1.16 I eliminated the need for set_data_size but missed one reference in targetted code and failed to test
that target.

At this point with so many versions flying around I'm hesitant to rush out another one. Hmage seems to have
posted a version with the set_data_size fix so I'll wait to see if that solves both problems before I build another
release.

[joblo]

@hmage

I see the difference in your logic in the sse2 check. I checked for an inferiour build missing sse2 (since removed in 3.1.17)
and you check for a superiour build including aes_ni. In practical terms the superiour check seems more important as that
scenario could occur if cpuminer-opt was cross compiled on a CPU with aes-ni. On the other hand cross compiling on a
CPU without sse2 would have failed.

Removing the check for inferiour build, as done in 3.1.17 and your fork, is harmless and will become permanent.
I will add a check for superiour build because that is a possible scenario. Both pending test results from users with
real AMD CPUs with sse2 but without aes-ni.

[th3.r00t]

@hmage

I see the difference in your logic in the sse2 check. I checked for an inferiour build missing sse2 (since removed in 3.1.17)
and you check for a superiour build including aes_ni. In practical terms the superiour check seems more important as that
scenario could occur if cpuminer-opt was cross compiled on a CPU with aes-ni. On the other hand cross compiling on a
CPU without sse2 would have failed.

Removing the check for inferiour build, as done in 3.1.17 and your fork, is harmless and will become permanent.
I will add a check for superiour build because that is a possible scenario. Both pending test results from users with
real AMD CPUs with sse2 but without aes-ni.

So if I read correctly - we should use github fork of hmage, which is with fixed CPU, OS and algo checks.
If everything is OK on AMD CPUs it will become the main branch of cpuminer-opt?

[joblo]

@hmage

I see the difference in your logic in the sse2 check. I checked for an inferiour build missing sse2 (since removed in 3.1.17)
and you check for a superiour build including aes_ni. In practical terms the superiour check seems more important as that
scenario could occur if cpuminer-opt was cross compiled on a CPU with aes-ni. On the other hand cross compiling on a
CPU without sse2 would have failed.

Removing the check for inferiour build, as done in 3.1.17 and your fork, is harmless and will become permanent.
I will add a check for superiour build because that is a possible scenario. Both pending test results from users with
real AMD CPUs with sse2 but without aes-ni.

So if I read correctly - we should use github fork of hmage, which is with fixed CPU, OS and algo checks.
If everything is OK on AMD CPUs it will become the main branch of cpuminer-opt?

It's up to hmage whether he wants to maintain his fork. I intend to implement the fix on mine once it has been
properly tested.

Even with the fix I believe AMD will require a workaround for -march

[th3.r00t]

When using the git fork with this command line

Code:

./autogen.sh && CXXFLAGS="-std=gnu++11" ./configure CFLAGS="-O3 -march=btver1" --with-curl --with-crypto && make

I got a successful compile and working binary on AMD

Code:

         **********  cpuminer-multi 1.2-dev  ***********
     A CPU miner with multi algo support and optimized for CPUs
     with AES_NI extension.
     BTC donation address: 12tdvfF7KmAsihBXQXynT6E6th2c2pByTT
     Forked from TPruvot's cpuminer-multi with credits
     to Lucas Jones, elmad, palmd, djm34, pooler, ig0tik3d,
     Wolf0 and Jeff Garzik.

Checking CPU capatibility of AMD Phenom(tm) II X4 940 Processor...
   CPU arch supports AES_NI... NO.
   CPU arch supports SSE2..... YES.
   SW built with AES_NI....... NO.
   SW built with SSE2......... YES.
   Algo supports AES_NI....... YES.
Starting mining without AES_NI optimisations...

[joblo]

When using the git fork with this command line

Code:

./autogen.sh && CXXFLAGS="-std=gnu++11" ./configure CFLAGS="-O3 -march=btver1" --with-curl --with-crypto && make

I got a successful compile and working binary on AMD

Code:

        **********  cpuminer-multi 1.2-dev  ***********
     A CPU miner with multi algo support and optimized for CPUs
     with AES_NI extension.
     BTC donation address: 12tdvfF7KmAsihBXQXynT6E6th2c2pByTT
     Forked from TPruvot's cpuminer-multi with credits
     to Lucas Jones, elmad, palmd, djm34, pooler, ig0tik3d,
     Wolf0 and Jeff Garzik.

Checking CPU capatibility of AMD Phenom(tm) II X4 940 Processor...
   CPU arch supports AES_NI... NO.
   CPU arch supports SSE2..... YES.
   SW built with AES_NI....... NO.
   SW built with SSE2......... YES.
   Algo supports AES_NI....... YES.
Starting mining without AES_NI optimisations...

Thanks. I presume -march=native still fails.

I will soon release 3.1.18 with  the following:

- Implement CPU check logic from hmage
    - remove check for inferior SW build on SSE2 CPU
    - add check for superior build
- Suppress messages not used in decision making
- document AMD workaround
- fix set_data_size bug

Edit: here is a draft of the test I intend to add to the build instructions:

Some users with AMD CPUs without AES_NI have reported problems compiling
with build.sh or "-march=native". Problems have included compile errors
and poor performance. These users are recommended to compile manually
specifying "-march=btver1" on the configure command line.

[th3.r00t]

Yep, -march=native still fails.

Also ./build.sh produces slightly slower binary for me on Intel and AMD (with changed -march=), maybe too many CFLAGS?

[joblo]

Yep, -march=native still fails.

Also ./build.sh produces slightly slower binary for me on Intel and AMD (with changed -march=), maybe too many CFLAGS?

Thanks. -march=native chooses a lower arch than your CPU is capable of. You can confirm this with this command

gcc -march=native -Q --help=target

You will likely see amdfam10 but will get better performance with btver1.

[joblo]

cpuminer-opt v3.1.18 is released. It should fix all known problems.

https://drive.google.com/file/d/0B0lVSGQYLJIZc1R0ajdpQ2R3NzQ/view?usp=sharing

[th3.r00t]

Yep, -march=native still fails.

Also ./build.sh produces slightly slower binary for me on Intel and AMD (with changed -march=), maybe too many CFLAGS?

Thanks. -march=native chooses a lower arch than your CPU is capable of. You can confirm this with this command

gcc -march=native -Q --help=target

You will likely see amdfam10 but will get better performance with btver1.

cpuminer-opt v3.1.18 on AMD

Command:

Code:

./autogen.sh && ./configure CFLAGS="-O3 -march=amdfam10" --with-curl --with-crypto && make

Result: Build Failed, gcc tried to compile AES-NI binary.

Command:

Code:

./autogen.sh && ./configure CFLAGS="-O3 -march=btver1" --with-curl --with-crypto && make

Result: Build Successful

[Giulini]

with -march=native o.k., with "btver1" negative

Checking CPU capatibility...
        AMD Sempron(tm) 145 Processor
   CPU arch supports AES_NI...NO.
   CPU arch supports SSE2.....YES.
   SW built for SSE2..........YES.
Starting mining without AES_NI optimizations...

[2016-04-23 22:06:15] Starting Stratum on stratum+tcp://hodl.suprnova.cc:4693
[2016-04-23 22:06:15] 1 miner threads started, using 'hodl' algorithm.
[2016-04-23 22:06:18] Stratum difficulty set to 1
[2016-04-23 22:07:07] hodl.suprnova.cc:4693 hodl block 43933
[2016-04-23 22:07:07] CPU #0: 373 H, 7.66 H/s
[2016-04-23 22:08:17] CPU #0: 529 H, 7.63 H/s
[2016-04-23 22:08:17] accepted: 1/1 (100%), 529 H, 7.63 H/s yes!

[joblo]

with -march=native o.k., with "btver1" negative

Checking CPU capatibility...
        AMD Sempron(tm) 145 Processor
   CPU arch supports AES_NI...NO.
   CPU arch supports SSE2.....YES.
   SW built for SSE2..........YES.
Starting mining without AES_NI optimizations...

[2016-04-23 22:06:15] Starting Stratum on stratum+tcp://hodl.suprnova.cc:4693
[2016-04-23 22:06:15] 1 miner threads started, using 'hodl' algorithm.
[2016-04-23 22:06:18] Stratum difficulty set to 1
[2016-04-23 22:07:07] hodl.suprnova.cc:4693 hodl block 43933
[2016-04-23 22:07:07] CPU #0: 373 H, 7.66 H/s
[2016-04-23 22:08:17] CPU #0: 529 H, 7.63 H/s
[2016-04-23 22:08:17] accepted: 1/1 (100%), 529 H, 7.63 H/s yes!

Edit @Giulini: what do you get from "gcc -march=native -Q --help=target"?

Edit2: I just realized there is a typo, it should be "-march=bdver1". Give it a try, it might be
faster for some algos.

This might be the best I can come up with. Now that you both have it figured out for your
own situation are the tips in README.md clear enough for other users? I've added another phrase
to the existing in italic.

Some users with AMD CPUs without AES_NI have reported problems compiling
with build.sh or "-march=native". Problems have included compile errors
and poor performance. These users are recommended to compile manually
specifying "-march=btver1bdver1" on the configure command line. If all else fails
"-march=core2" will provide the best compatibility but the lowest performance".

[joblo]

With things, hopefully, settling down after a period of rapid development
and many algo additions algo-gate has evolved significantly and delelopped
some mutations. Some functions developped multiple personalities or had names
that made no sense which deviated from the highly structured goals of the gate system.
Nevertheless it proved its worth when merging new algos by isolating all the algo
specific code. It also showed promise in gating other functions such as jsonrpc2.

I'll start working on v3.2 and do some genetic engineering on algo-gate so it doesn't
turn into a monster. I will also convert jsonrpc2 functions to use the gate and possibly
others

In the meantime only bug fixes and high priority features will be added to the 3.1
stream.

[th3.r00t]

cpuminer-opt v3.1.18 is one of the best in recent version history.
Talking both Intel and AMD wise.

Thank you for your hard work!

Can't wait to see what branch 3.2 will offer. 

[joblo]

cpuminer-opt v3.1.18 is one of the best in recent version history.
Talking both Intel and AMD wise.

Thank you for your hard work!

Can't wait to see what branch 3.2 will offer. 

The algo-gate work won't be user visible but will make it easier to add new algos.
Thanks for your support and invaluable help in getting to the bottom of the compile issue.

[AlexGR]

3.1.18 is kind'a buggy in cpu detection. In a Q8200 it says:

Checking CPU capatibility...
        Intel(R) Core(TM) i5 CPU         670  @ 3.47GHz
   CPU arch supports AES_NI...YES.
   SW built for AES_NI........NO.
   Algo supports AES_NI.......YES.
CPU and algo support AES_NI, but SW build does not.
Rebuild with "-march=native" for better performance.
Starting mining without AES_NI optimizations...

...so naturally it thinks I have AES, even though the build is not for AES (-march=native / no AES in Intel quad core q8200).

Anyway I did a profile run in x11 to check the slowdowns:



A couple of them, with echo512 being the biggest culprit, dominate the process in terms of time wasted.

It seems there is a very optimized AES version for it: https://bench.cr.yp.to/impl-hash/echo512.html



https://bench.cr.yp.to/impl-hash/echo512.html
https://github.com/floodyberry/supercop/tree/master/crypto_hash/echo512/aes/aes64

[joblo]

Images snipped.

3.1.18 is kind'a buggy in cpu detection. In a Q8200 it says:

Checking CPU capatibility...
        Intel(R) Core(TM) i5 CPU         670  @ 3.47GHz
   CPU arch supports AES_NI...YES.
   SW built for AES_NI........NO.
   Algo supports AES_NI.......YES.
CPU and algo support AES_NI, but SW build does not.
Rebuild with "-march=native" for better performance.
Starting mining without AES_NI optimizations...

...so naturally it thinks I have AES, even though the build is not for AES (-march=native / no AES in Intel quad core q8200).

Anyway I did a profile run in x11 to check the slowdowns:

A couple of them, with echo512 being the biggest culprit, dominate the process in terms of time wasted.

It seems there is a very optimized AES version for it: https://bench.cr.yp.to/impl-hash/echo512.html

https://bench.cr.yp.to/impl-hash/echo512.html
https://github.com/floodyberry/supercop/tree/master/crypto_hash/echo512/aes/aes64

Let me get this straight. You compiled with -march=native on a core2 that thinks it's a i5-670.
The compile succeeded and the miner ran ok. That's pretty special.

The CPU model and AES support comes directly from CPUID and has been reliable until now.
Even the AMD guys haven't reported CPUID problems.

Can you confirm CPUID is correct:

Code:

cat /proc/cpuinfo |grep model

It would be interesting to see what the compiler thought:

Code:

gcc -march=native -Q --help=target | fgrep march

Regarding echo512, yes you will get the slow version. Unfortunately I'm unaware of a SSE2 optimized
version and the AES version is already used by cpuminer-opt on capable CPUs.

I have yet to study your data in any detail but the following may put performance into perspective.

Echo512 and groestl have AES optimizations for most algos.

Cryptonight and hodl have their own unique AES optimizations.

The rest of the x11 chain, including groestl but excluding echo, have SSE2 optimized versions.
The algos in the longer X chains, as well as non-aes echo, are filled with slow SPH versions.