[LOCKED] cpuminer-opt v3.12.3, open source optimized multi-algo CPU miner

[joblo]

cpuminer-opt v3.1.13 just released. Added support for a few dusty algos:
sha256d, scrypt, scryptjane and yescrypt.

If anyone  knows of anywhere to pool test any of the remaining algos please let me know.
I'll even test you your address.

You can test scryptjane at YACoin pool:

Miner config:

Code:

-o stratum+tcp://yacoin.club:3433 -O alenevaa.CPU1:1

You can use 2 ports:
- 3433 difficulty 1
- 3434 difficulty 0.25

If you miner doesn't support difficulty less than 1 - use port 3433

Thanks but scryptjane has been pool tested. When I said remaining algos I was referring to the ones that have
not yet been tested in a live pool. Specifically they are X17, blakecoin, fresh, cryptolight, and bastion. I have not been
able to find any pools offering to mine coins using these algos.

Sorry for the confusion.

[theLosers106]

cpuminer-opt v3.1.13 just released. Added support for a few dusty algos:
sha256d, scrypt, scryptjane and yescrypt.

If anyone  knows of anywhere to pool test any of the remaining algos please let me know.
I'll even test you your address.

You can test scryptjane at YACoin pool:

Miner config:

Code:

-o stratum+tcp://yacoin.club:3433 -O alenevaa.CPU1:1

You can use 2 ports:
- 3433 difficulty 1
- 3434 difficulty 0.25

If you miner doesn't support difficulty less than 1 - use port 3433

Thanks but scryptjane has been pool tested. When I said remaining algos I was referring to the ones that have
not yet been tested in a live pool. Specifically they are X17, blakecoin, fresh, cryptolight, and bastion. I have not been
able to find any pools offering to mine coins using these algos.

Sorry for the confusion.

For blakecoin you could use this pool: http://ny2.blakecoin.com/index.php?page=login

[hmage]

I've added some changes to make it work out of the box on non-AES linux builds:

Diff can be found here -- https://github.com/hmage/cpuminer-opt/compare/upstream...master
Raw diff -- https://github.com/hmage/cpuminer-opt/compare/upstream...master.diff

• On non-MSVC, if -march=native supports AES, then NO_AES_NI is not defined, otherwise it's defined.
• An LTO build did not work out of the box, some variables and functions needed to be marked as used, so they won't be removed by linker.
• Add -march=native into build.sh
• Fix a compiler warning, since builtin_alloca() is already defined as alloca() on GCC.
• Curiously, on my benchmarks, -O2 is faster than -Ofast on gcc 4.9.2.

[pallas]

I had the same experience: Ofast may be slower than O2, but usually O3 is faster instead.

[joblo]

I've added some changes to make it work out of the box on non-AES linux builds:

Diff can be found here -- https://github.com/hmage/cpuminer-opt/compare/upstream...master
Raw diff -- https://github.com/hmage/cpuminer-opt/compare/upstream...master.diff

• On non-MSVC, if -march=native supports AES, then NO_AES_NI is not defined, otherwise it's defined.
• An LTO build did not work out of the box, some variables and functions needed to be marked as used, so they won't be removed by linker.
• Add -march=native into build.sh
• Fix a compiler warning, since builtin_alloca() is already defined as alloca() on GCC.
• Curiously, on my benchmarks, -O2 is faster than -Ofast on gcc 4.9.2.

I presume you mean you made build.sh work out of the box?
That's great. I'll dig into it. If you left it -Ofast I'll change it to -O3,
that's what I always use anyway.

[hmage]

I presume you mean you made build.sh work out of the box?

Exactly that, works for both non-AES CPU's and AES CPU's. GCC provides __AES__ macro if -march=native supports it.

Did not test on MinGW, though, only on Debian 8.4.

[joblo]

I presume you mean you made build.sh work out of the box?

Exactly that, works for both non-AES CPU's and AES CPU's. GCC provides __AES__ macro if -march=native supports it.

Did not test on MinGW, though, only on Debian 8.4.

It'l be in the next release, hopefully in a day or so. It'l be handy for users not to have to know the architecture.

[joblo]

I presume you mean you made build.sh work out of the box?

Exactly that, works for both non-AES CPU's and AES CPU's. GCC provides __AES__ macro if -march=native supports it.

Did not test on MinGW, though, only on Debian 8.4.

It'l be in the next release, hopefully in a day or so. It'l be handy for users not to have to know the architecture.

I really appreciate the work you did. I've been stumbling my way around the environment just trying to make
things work. It's been challenging to apply my experience to a completey foreign environment. Feel free to
comment on overall architecture. Being a foreigner in the land of GNU C I'm not aware of the local customs
and conventions.

I've done a first pass through your changes and I want to undertsand more so forgive my stupid questions.
I'm curious about some of the changes because things worked. Amd when things work I don't want to make
changes unless it makes them work better.

In general terms I ask why is it better, will it hash faster? Simpler compilation procedure is an obvious one
but other changes escape me.

I didn't know about the compiler support for __AES__. Good job.

I never heard of LTO before, what will it give me? Is that how boost now works with your changes?

Boost didn't work fo me until I added the linking options to teh makefile. Are your changes leading edge?
Will they work on older loads?

Can you explain the __attribute__ used stuff a bit more, that's a new concept for me. Again, it worked
before so how is this better?

Thanks again. I'll get the __AES__ implemented quickly, the other stuff can wait until I have a better understanding.

Update:

I've been spending a lot of time trying to implement another algo with little success. I get a nice multi page
dump for a buffer overflow, not just the typical segfault, but I digress. I put that aside for another day so I can
your user visible changes released.

I implemented __AES__ with a slight change, I allow configure to override automatic aes compile, not very useful
for users but it helps testing SSE2 on my i7.

There are still problems with build.sh. First groestl sse2 complained about ASM constraints so I removed USE_ASM
then a bunch of AES code failed to compile, then I stripped it down to "-O3 -march=native" and it works.

It's nice to get rid of the alloca warnings.

I didn't see a point in changing the way boost was linked, it doesn't make compiling any easier and it ain't broke.

I'll look into the other issues over the longer term.

Thanks again.

[joblo]

I have a small issue with compiler warnings for using undefined functions. It haas to do with algo-gate.
The goal of algo gate was to make cpu-miner completely algo agnostic, meaning there is no algo
specific code anywhere in cpu-miner.c. That goal was achieved.

At this point there is only one long switch/case statement to register the selected algo with the gate
so cpu-miner can invoke its functions. The only base code that nees to be modified is the comand
argument definitions to support the algo.

The only drawback is a large number of compiler warnings, one while trying to register each algo.
That is because the algo's registration function is in its own source file whicj is not included by anyone.
The solution seems to be to make a cenrtal list of the gate registration functions of every algo.
That would add work to developpers wheh adding a new algo.

If anyone has a win-win solution, ie no warnings and no extra work, It would be welcomed. 

[joblo]

cpuminer-opt v3.1.14 released.

https://drive.google.com/file/d/0B0lVSGQYLJIZaE5DYXA4SHl2WVk/view?usp=sharing

New in v3.1.14

Algos
     - cryptonight algo is now supported on CPUs without AES_NI.
       All algos now support both CPU architectures.
     - jane added as an alias for scryptjane with default N-factor 16

Build enhancements, see details in README.md (thanks to hmage)
     - build.sh now works for CPUs with and without AES_NI
     - it is no longer necessary to add -DNO_AES_NI CFLAG to
       configure command when building for CPUs without AES_NI.

     Note: Compiling requires some additional libraries not included
     in the default instalation of most Linux distributions: libboost-dev,
     libboost-system-dev, libboost-thread-dev.

UI enhancements
     - enhanced checks for CPU architecture, SW build and algo for
       AES_NI and SSE2 capabilities.
     - a warning is displayed if mining an untested algo.

Code cleanup
     - removed a few more compiler warnings
     - removed some dead code

Algo gate enhancements (for devs)
     - replaced algo specific null gate functions with generic null
       functions

[hmage]

I never heard of LTO before, what will it give me? Is that how boost now works with your changes?

Can you explain the __attribute__ used stuff a bit more, that's a new concept for me. Again, it worked before so how is this better?

This is link-time optimization. Same way what Microsoft does under the "whole program optimization" name.

This lets the compiler see how each function is used by other functions and optimize more aggressively since it knows about all functions, this can mean that the optimizer can skip some work done since it's not needed by the program as a whole.

__attribute__((used)) is needed to fix compilation errors when using LTO wih gcc -- gcc is late to the party when adding LTO/whole program optimization, so there are still some kinks to be ironed out -- the LTO optimizer tends to disregard pure-assembler functions, so they need to be marked as used by the program, otherwise they'll be thrown away at one of the passes before the linking happens. Same applies to static variables defined in headers.

Boost didn't work for me until I added the linking options to the makefile. Are your changes leading edge?
Will they work on older loads?

I did remove the need for boost in hodlcoin after I posted that comment, I did not intend the changes to be visible by you (yet) -- The proper diff URLs are:
 * https://github.com/hmage/cpuminer-opt/compare/75db804...d01b290 -- UI
 * https://github.com/hmage/cpuminer-opt/compare/75db804...d01b290.diff -- raw diff

I implemented __AES__ with a slight change, I allow configure to override automatic aes compile, not very useful
for users but it helps testing SSE2 on my i7.

There's no need for that, you can just put -march=core2 instead of -march=native.

There are still problems with build.sh. First groestl sse2 complained about ASM constraints so I removed USE_ASM
then a bunch of AES code failed to compile, then I stripped it down to "-O3 -march=native" and it works.

Again, you're seeing LTO problems mentioned above, for the reasons mentioned above. The proper diff URL builds cleanly on core2 and haswell.

I didn't see a point in changing the way boost was linked, it doesn't make compiling any easier and it ain't broke.

That's part of changes to get rid of boost.

I have a small issue with compiler warnings for using undefined functions. It has to do with algo-gate.

In C you have to tell the compiler what the function declaration looks like before you use the function, what happens next is undefined behaviour and thankfully on intel linux with gcc this doesn't break anything.

You have to put a function prototype in a header -- the dev needs to add a mention of the function in 3 places -- the header, the body, and the place that calls it. Since this doesn't happen often, I don't think it's a big deal.

Strange though that you didn't see any kind of these warnings before, probably MS Visual C defaults to not show this specific warning (and because MS has to worry only about making developer's sourcecode work only Intel Windows anyway).

There's no way to do what you want portably that works on both MSVC and GCC without requiring a function prototype in a header. Linux kernel relies on Linux-specific assembler code (.init) to do their module system without forcing everyone to list all function prototypes in one place.

[joblo]

[snipped]

Thanks for the detailed explanation, it'll take a while to digest it. In the meantime you can take
a look at the latest release, I implemented some of your suggestions (forgot to credit you, need to fix that)
in v3.1.14. I do the initial __AES__ check in miner.h and you can see a good example of how I use it in
cpu-miner.c:cpu_capabilities_check.

One thing I immediately infered from your response is I need to use LTO to get Windows working
(and I sense you are working on that, if so thanks) and I need to clean up boost and implied function
declarations to get LTO working. That is plenty justification to do it. I look forward to seeing that cake
when it's fully baked.

[joblo]

This is link-time optimization. Same way what Microsoft does under the "whole program optimization" name.

This lets the compiler see how each function is used by other functions and optimize more aggressively since it knows about all functions, this can mean that the optimizer can skip some work done since it's not needed by the program as a whole.

That seems very worthwhile to do if it gets windows Working. If is requires extra work for devs to maintain a master list of
function declarations, so be it. I'll have to find a way to make it work.

I implemented __AES__ with a slight change, I allow configure to override automatic aes compile, not very useful
for users but it helps testing SSE2 on my i7.

There's no need for that, you can just put -march=core2 instead of -march=native.

I think there is. Even with "-march=core2 -DNO_AES_NI" the __AES__ var is still defined. It is based
on the architecture of the CPU, not the compiled app.

Thanks again for your contributions and I look forward to what you have planned next.

[hmage]

That seems very worthwhile to do if it gets windows Working.

Fixing LTO to work on GCC/Linux has nothing to do with making it work for MSVC/Windows. If anything, I don't have Windows here to test so it might be actually breaking Windows build.

I think there is. Even with "-march=core2 -DNO_AES_NI" the __AES__ var is still defined.

It depends on -march:

Code:

hmage@dhmd:~$ echo '' | gcc -E -dD -march=core2 -|grep __AES__|wc -l
0
hmage@dhmd:~$ echo '' | gcc -E -dD -march=haswell -|grep __AES__|wc -l
1

If you're seeing __AES__ macro defined when you're using -march=core2, then you have wrong build flags or unclean build.

[joblo]

I think there is. Even with "-march=core2 -DNO_AES_NI" the __AES__ var is still defined.

It depends on -march:

Code:

hmage@dhmd:~$ echo '' | gcc -E -dD -march=core2 -|grep __AES__|wc -l
0
hmage@dhmd:~$ echo '' | gcc -E -dD -march=haswell -|grep __AES__|wc -l
1

If you're seeing __AES__ macro defined when you're using -march=core2, then you have wrong build flags or unclean build.

Hmmm, I should retest, maybe I actually used "-march=native -DNO_AES_NI". I'm hoping I'm right
because it enables detection of an AES_NI cross compiled build being run on a non-AES_NI CPU. Although
very unlikely to occur it also adds a safety net in case bugs creep in, and that kind of design was part of my
past professional life.

Edit: if __AES__ doesn't reflect the HW arch I can go back to has_aes_ni() from cpuid.

[hmage]

Hmmm, I should retest, maybe I actually used "-march=native -DNO_AES_NI". I'm hoping I'm right
because it enables detection of an AES_NI cross compiled build being run on a non-AES_NI CPU. Although
very unlikely to occur it also adds a safety net in case bugs creep in, and that kind of design was part of my
past professional life.

Edit: if __AES__ doesn't reflect the HW arch I can go back to has_aes_ni() from cpuid.

You're probably confusing two things:
 * The processor the binary was built for
 * The processor the binary is running on

These two things are similar but separate:
 * -march sets the _target_ processor the binary will be built for. The binary will error with "illegal instruction" if you run it on older CPU's.
 * cpuid returns the _current_ processor it's running under, if you can code your program to dynamically switch between implementations based on what CPU is currently running the binary.

__AES__ means that the _target_ processor supports AES -- that is a macro, not a variable, it doesn't change between runs. It reflects the hardware architecture of the processor the binary is _designed_ to be run on. Trying to execute AES instructions on core2 will give you "illegal instruction" error.

You need to check that _both_ the binary and the CPU support AES before actually doing the code. You need to check for __AES__ flag and get the cpuid capability.

Same goes for SSE2, you need to pull it from cpuid and check that __SSE2__ is defined.

I've rewritten the check_cpu_capability() to reflect what I mean:
https://github.com/hmage/cpuminer-opt/commit/30ef05f18b538655440667c5106e4967a27ccb51?diff=split

[joblo]

Hmmm, I should retest, maybe I actually used "-march=native -DNO_AES_NI". I'm hoping I'm right
because it enables detection of an AES_NI cross compiled build being run on a non-AES_NI CPU. Although
very unlikely to occur it also adds a safety net in case bugs creep in, and that kind of design was part of my
past professional life.

Edit: if __AES__ doesn't reflect the HW arch I can go back to has_aes_ni() from cpuid.

You're probably confusing two things:
 * The processor the binary was built for
 * The processor the binary is running on

These two things are similar but separate:
 * -march sets the _target_ processor the binary will be built for. The binary will error with "illegal instruction" if you run it on older CPU's.
 * cpuid returns the _current_ processor it's running under, if you can code your program to dynamically switch between implementations based on what CPU is currently running the binary.

__AES__ means that the _target_ processor supports AES -- that is a macro, not a variable, it doesn't change between runs. It reflects the hardware architecture of the processor the binary is _designed_ to be run on. Trying to execute AES instructions on core2 will give you "illegal instruction" error.

You need to check that _both_ the binary and the CPU support AES before actually doing the code. You need to check for __AES__ flag and get the cpuid capability.

Same goes for SSE2, you need to pull it from cpuid and check that __SSE2__ is defined.

I've rewritten the check_cpu_capability() to reflect what I mean:
https://github.com/hmage/cpuminer-opt/commit/30ef05f18b538655440667c5106e4967a27ccb51?diff=split

I unerstood the concepts but not the implementation. I had problems with has_sse2 last time I tried that, maybe I'll
success this time.

[hmage]

I had problems with has_sse2 last time I tried that, maybe I'll success this time.

has_sse2() was broken. Had to repair it. It was looking at wrong cpuid field.

[joblo]

There is a problem in v3.1.14 with the new compile procedure for CPUs that do not have
AES_NI. Please continue to use _DNO_AES_NI on the configure command line if your CPU does
not have AES_NI.

cpuminer-opt v3.1.14 released.

https://drive.google.com/file/d/0B0lVSGQYLJIZaE5DYXA4SHl2WVk/view?usp=sharing

New in v3.1.14

Algos
     - cryptonight algo is now supported on CPUs without AES_NI.
       All algos now support both CPU architectures.
     - jane added as an alias for scryptjane with default N-factor 16

Build enhancements, see details in README.md (thanks to hmage)
     - build.sh now works for CPUs with and without AES_NI
     - it is no longer necessary to add -DNO_AES_NI CFLAG to
       configure command when building for CPUs without AES_NI.

     Note: Compiling requires some additional libraries not included
     in the default instalation of most Linux distributions: libboost-dev,
     libboost-system-dev, libboost-thread-dev.

UI enhancements
     - enhanced checks for CPU architecture, SW build and algo for
       AES_NI and SSE2 capabilities.
     - a warning is displayed if mining an untested algo.

Code cleanup
     - removed a few more compiler warnings
     - removed some dead code

Algo gate enhancements (for devs)
     - replaced algo specific null gate functions with generic null
       functions

[hmage]

There is a problem in v3.1.14 with the new compile procedure for CPUs that do not have
AES_NI. Please continue to use _DNO_AES_NI on the configure command line if your CPU does
not have AES_NI.

Yeah, you added the NO_AES_NI define inside the VisualC-specific section of miner.h  

Here's a diff against v3.1.14 to make it work on core2:
 * https://github.com/hmage/cpuminer-opt/commit/94f503824a852335770c0ee5789bbd7511224a37
 * https://github.com/hmage/cpuminer-opt/commit/94f503824a852335770c0ee5789bbd7511224a37.diff -- raw diff

You also no longer need to carry around compat/curl-for-windows/openssl, since you are including system openssl headers anyway:
 * https://github.com/hmage/cpuminer-opt/commit/b0c0a7ab9b1d812075de90052b7f6ecb58d8a2cf
 * https://github.com/hmage/cpuminer-opt/commit/b0c0a7ab9b1d812075de90052b7f6ecb58d8a2cf.diff -- raw diff

You no longer need boost to compile hodlcoin, the code uses unordered_map, which is part of C++11 standard and gcc supports it since 4.7:
 * https://github.com/hmage/cpuminer-opt/commit/73eae16f3afa1b1f32ab4264c705b1e6bbbd8bbc
 * https://github.com/hmage/cpuminer-opt/commit/73eae16f3afa1b1f32ab4264c705b1e6bbbd8bbc.diff -- raw diff

To properly report that SW and CPU support AES and SSE2, you need to fix has_sse2() first:
 * https://github.com/hmage/cpuminer-opt/commit/d867ad08c7dac01f8c3735b0e65afb0242a4566f
 * https://github.com/hmage/cpuminer-opt/commit/d867ad08c7dac01f8c3735b0e65afb0242a4566f.diff -- raw diff

Then go fix the check_cpu_capability() to actually report the proper values:
 * https://github.com/hmage/cpuminer-opt/commit/c869830ef2446d863019e284437382681da259ab
 * https://github.com/hmage/cpuminer-opt/commit/c869830ef2446d863019e284437382681da259ab.diff -- raw diff

To avoid problems with misapplying the patches (I guess you applied my previous patch manually, therefore you put NO_AES_NI define in wrong place in miner.h), I recommend using the standard patch utility to automate the process:

Code:

cd cpuminer-opt
wget https://github.com/hmage/cpuminer-opt/commit/94f503824a852335770c0ee5789bbd7511224a37.diff
wget https://github.com/hmage/cpuminer-opt/commit/b0c0a7ab9b1d812075de90052b7f6ecb58d8a2cf.diff
wget https://github.com/hmage/cpuminer-opt/commit/73eae16f3afa1b1f32ab4264c705b1e6bbbd8bbc.diff
wget https://github.com/hmage/cpuminer-opt/commit/d867ad08c7dac01f8c3735b0e65afb0242a4566f.diff
wget https://github.com/hmage/cpuminer-opt/commit/c869830ef2446d863019e284437382681da259ab.diff
patch -p1 -i 94f503824a852335770c0ee5789bbd7511224a37.diff
patch -p1 -i b0c0a7ab9b1d812075de90052b7f6ecb58d8a2cf.diff
patch -p1 -i 73eae16f3afa1b1f32ab4264c705b1e6bbbd8bbc.diff
patch -p1 -i d867ad08c7dac01f8c3735b0e65afb0242a4566f.diff
patch -p1 -i c869830ef2446d863019e284437382681da259ab.diff

This way it will apply exactly as it is on my side, assuming you didn't make any changes nearby.