[LOCKED] cpuminer-opt v3.12.3, open source optimized multi-algo CPU miner

[joblo]

So, in this particular case, moving the assignments to the left will solve the issue:

Code:

CFLAGS="-O3 -march=native -Wall" CXXFLAGS="$CFLAGS -std=gnu++11" ./configure --with-curl

Thanks I'll make the change.

[ryen123]

@joblo Think I have found a bug. Using the windows build from cryptomining-blog, get rejects: share above target and noticed the memory usage spikes to 2GB of ram. When it works normally only 1GB ram is used. The problem occurs occasionally at miner launch, restarting can get it to work.

[joblo]

@joblo Think I have found a bug. Using the windows build from cryptomining-blog, get rejects: share above target and noticed the memory usage spikes to 2GB of ram. When it works normally only 1GB ram is used. The problem occurs occasionally at miner launch, restarting can get it to work.

Curious. How often does it occur? I have seen this very infrequently but have seen it before. Restarting the miner
usually fixes it. If it's happening more often that's a new issue.

Nice observation about the 2 GB RAM usage. HOdl uses a 1 GB buffer so somehow there were 2 allocated.
You didn't have two instances running at the same time did you? That wouldn't explain th erejects but would
explain the RAM usage.

[ryen123]

Curious. How often does it occur? I have seen this very infrequently but have seen it before. Restarting the miner
usually fixes it. If it's happening more often that's a new issue.

Nice observation about the 2 GB RAM usage. HOdl uses a 1 GB buffer so somehow there were 2 allocated.
You didn't have two instances running at the same time did you? That wouldn't explain th erejects but would
explain the RAM usage.

My opinion is it occurs rather infrequent but annoying enough. No I didn't run 2 instances of the miner.

[joblo]

Curious. How often does it occur? I have seen this very infrequently but have seen it before. Restarting the miner
usually fixes it. If it's happening more often that's a new issue.

Nice observation about the 2 GB RAM usage. HOdl uses a 1 GB buffer so somehow there were 2 allocated.
You didn't have two instances running at the same time did you? That wouldn't explain th erejects but would
explain the RAM usage.

My opinion is it occurs rather infrequent but annoying enough. No I didn't run 2 instances of the miner.

That's great info. If one miner instance can allocate two buffers it's a mutex issue and I know exactly where.

I've got a better solution. Take the buffer alloc out of the threads (don't know why it was there in the first place)
and move it to main.

[joblo]

Yet another release coming, hopefully to fix the final hodl issue.
There isn't a good way to test the fix other than soaking it but it's
well understood so I'll get v3.3.3 out ASAP. There are no other changes.

Update:

https://drive.google.com/file/d/0B0lVSGQYLJIZQ1puZWRjWUdIRVE/view?usp=sharing

There is also a small change to build.sh.

Edit: v3.3.3 binaries

http://cryptomining-blog.com/wp-content/download/cpuminer-opt-3-3-3-windows.zip

[joblo]

There are a couple of issues that I would like some advice on from the community. I will make seperate posts
so the discussion doesn't get garbled.

The first issue is compiling an optimum build for early CPUs with AES support. These are Intel CPUs between
core2 and Sandybridge including Westmere as well as similar AMD CPUs.

This is primarilly intended for users mining with such CPUs but may also be of interest to others.
 
I recently discovered the AES optimizations I have been using extensively use AVX1 instsructions as
well as AES. AES was implemented on CPUs before AVX1. The CPU capabilities check in cpuminer-opt
prior to v3.3.2 did not make this disctinction. It assumed if the CPU has AES it could use the AES code in
cpuminer-opt.

This results in users with these CPUs, particularly user using the CMB binaries, seeing the following message
on miner startup:

Code:

CPU and algo support AES-AVX1, but SW build does not.
Rebuild with "-march=native" for better performance.

One user has reported that the Westmere precompiled binary appears not to be compiled for AES and AVX1.
It isn't clear why. That's where i need your help if you have one of these CPU. Linux, Windows, self compiled
or  prebuilt binaries, it doesn't matter. If you have a CPU in the Westmere family between core2 and Sandybridge
you can help a lot.

Please post:
- your CPU model and confirmation whether it has AES and or AVX (may also be called AVX1) if possible.
- whether and how you compiled or if you used a binary which one
- the miner output at startup showing the CPU capabilities check
- whether it mined successfully
- any other info you think is useful

Thank you in advance.

Edit: Another option is to abandon the rebuild check. It is intended to catch precicely the type of situation that
may currently exist with Westmere CPUs possibly getting lower than optimum performance. However, it seems to
be causing more grief than it's worth, partly because it was not correctly implemented. Users thought the rebuild
message was an error and meant the miner wasn't working.

Even if the rebuild check is working properly an attempt to use the existing Westmere Windows build will result
in complaints likely coming to me when they should be directed at the builder.

I'll wait to see the reaction is to v3.3.3, if there are still problems with the rebuild check, it's gone.

[joblo]

Second issue is hodl on CPUs without AES on Windows. It still doesn't compile.
It works fine on Linux.

I am not testing this on a real core2, just forcing a core2 compile on an i7. I don't
think this is a factor.

The problem is endian.h. It's missing when compiling on Windows and is needed by two
files used for non-AES hodl. I can't find any endian.h in any of the mingw libraries. I have
searched the net and didn't find any endian library for mingw.

I tried a few different variations of endian.h such as sys/endian.h and a file I found on the
net called portable_endian.h. Some didn't exist, some didn't have the necessary definitions
and some compiled but produced only rejects, none of them worked.

The endianness on Windows might be a factor.

I need to find an endian library for mingw that Works with Windows little endian. Any suggestions?

[ryen123]

Just to report, v3.3.3 seems good now. Haven't yet encountered any issues.

Edit: Just for reference
Tested with windows build on i7-4770k, i3-4160 & i5-2500k.

[NeuroticFish]

Edit: v3.3.3 binaries

http://cryptomining-blog.com/wp-content/download/cpuminer-opt-3-3-3-windows.zip

I tried this out and I get error:

Checking CPU capatibility...
        Intel(R) Core(TM) i7 CPU         920  @ 2.67GHz
   CPU arch supports AES-AVX1...NO.
   CPU arch supports SSE2.....YES.
   SW built for SSE2..........NO.
Incompatible SW build, rebuild with "-march=native"

[joblo]

Edit: v3.3.3 binaries

http://cryptomining-blog.com/wp-content/download/cpuminer-opt-3-3-3-windows.zip

I tried this out and I get error:

Checking CPU capatibility...
        Intel(R) Core(TM) i7 CPU         920  @ 2.67GHz
   CPU arch supports AES-AVX1...NO.
   CPU arch supports SSE2.....YES.
   SW built for SSE2..........NO.
Incompatible SW build, rebuild with "-march=native"

Which exe did you use? Which algo?

Edit: I tried ccminer.exe on a skylake which has AES and AVX and the capabilities check said the SW was
built for AES. If you used the same file and it said it was not built for SSE2 then there is an inconsistency.
The same file was built for AES but not SSE2. When I compile with -march=native both are true.

It's becoming more clear that this check should be removed.

[NeuroticFish]

Edit: v3.3.3 binaries

http://cryptomining-blog.com/wp-content/download/cpuminer-opt-3-3-3-windows.zip

I tried this out and I get error:

Checking CPU capatibility...
        Intel(R) Core(TM) i7 CPU         920  @ 2.67GHz
   CPU arch supports AES-AVX1...NO.
   CPU arch supports SSE2.....YES.
   SW built for SSE2..........NO.
Incompatible SW build, rebuild with "-march=native"

Which exe did you use? Which algo?

Edit: I tried ccminer.exe on a skylake which has AES and AVX and the capabilities check said the SW was
built for AES. If you used the same file and it said it was not built for SSE2 then there is an inconsistency.
The same file was built for AES but not SSE2. When I compile with -march=native both are true.

It's becoming more clear that this check should be removed.

I used cpuminer.exe (I have Nehalem architecture), on cryptonight and on nist5. Same error on both.

[joblo]

Edit: v3.3.3 binaries

http://cryptomining-blog.com/wp-content/download/cpuminer-opt-3-3-3-windows.zip

I tried this out and I get error:

Checking CPU capatibility...
        Intel(R) Core(TM) i7 CPU         920  @ 2.67GHz
   CPU arch supports AES-AVX1...NO.
   CPU arch supports SSE2.....YES.
   SW built for SSE2..........NO.
Incompatible SW build, rebuild with "-march=native"

Which exe did you use? Which algo?

Edit: I tried ccminer.exe on a skylake which has AES and AVX and the capabilities check said the SW was
built for AES. If you used the same file and it said it was not built for SSE2 then there is an inconsistency.
The same file was built for AES but not SSE2. When I compile with -march=native both are true.

It's becoming more clear that this check should be removed.

I used cpuminer.exe (I have Nehalem architecture), on cryptonight and on nist5. Same error on both.

OK, Thanks. I can't explain why it displays "SW built for SSE2..........NO." on your CPU and YES on mine.
This is supposed to indicate how it was compiled.

I've had enough trying to get it to work. I'm removing those checks in the next release.

[Roolieman]

Just tested v.3.3.3 on Xeon E5649 (Westmere-EP, AES without AVX) and got same error like on v3.3.2

         **********  cpuminer-opt 3.3.3  ***********
     A CPU miner with multi algo support and optimized for CPUs
     with AES_NI and AVX1 extensions.
     BTC donation address: 12tdvfF7KmAsihBXQXynT6E6th2c2pByTT
     Forked from TPruvot's cpuminer-multi with credits
     to Lucas Jones, elmad, palmd, djm34, pooler, ig0tik3d,
     Wolf0 and Jeff Garzik.

Checking CPU capatibility...
        Intel(R) Xeon(R) CPU           E5649  @ 2.53GHz
   CPU arch supports AES-AVX1...NO.
   CPU arch supports SSE2.....YES.
   SW built for SSE2..........NO.
Incompatible SW build, rebuild with "-march=native"

[joblo]

The plan for v3.3.4

All known issues will be resolved except 1.

hodl on Windows for non-AES CPUs is still not working and will remain disabled.

V3.3.3 seems to have finally solved all the low diff problems with hodl.

My attempt to help users by detecting when and under-oprtimized build is being used has
been a major flop. It's over. v3.3.4 will still display the detected CPU architecture and
expected performance but it will got give any advice or take preemptive action.
The capabilities check will be informative only, the miner will alway attempt to mine.

Also there are significant improvement to m7m algo thanks to Wolf0.

I intend to wait a while longer before releasing v3.3.4 in case there any more reports from
users about v3.3.3.

[joblo]

Just tested v.3.3.3 on Xeon E5649 (Westmere-EP, AES without AVX) and got same error like on v3.3.2

         **********  cpuminer-opt 3.3.3  ***********
     A CPU miner with multi algo support and optimized for CPUs
     with AES_NI and AVX1 extensions.
     BTC donation address: 12tdvfF7KmAsihBXQXynT6E6th2c2pByTT
     Forked from TPruvot's cpuminer-multi with credits
     to Lucas Jones, elmad, palmd, djm34, pooler, ig0tik3d,
     Wolf0 and Jeff Garzik.

Checking CPU capatibility...
        Intel(R) Xeon(R) CPU           E5649  @ 2.53GHz
   CPU arch supports AES-AVX1...NO.
   CPU arch supports SSE2.....YES.
   SW built for SSE2..........NO.
Incompatible SW build, rebuild with "-march=native"

Thanks. You're the third person with Westmere with this problem. The main problem is "SW built for SSE2..........NO."
When I use the same file on my Skylake it answers YES, as it should. It isn't the way it was compiled because
I'm using the same file downloaed from CMB.

The software build test is supposed to reflect the compilation, not the HW support, however, the same SW reports
different results on different HW. I can only conclude the CPU does affect the SW build test. Without a relaible way
to test the SW build I have no choice but to disable the test.

[th3.r00t]

This is precisely the issue I'm trying to solve, where to find the best miner for a specific
algo. I want cpuminer-opt to the the one stop shop for all CPU mining.

Thanks for that tip.

As we wait for the new version with all things that you plan to include/fix, can you post which algos are AES-AVX1 and which are only SSE2,
both on Linux and Windows so everyone can contibute a faster source (if there is one) for every algo that needs AES-AVX1 optimisations?

[hmage]

Thanks. You're the third person with Westmere with this problem. The main problem is "SW built for SSE2..........NO."
When I use the same file on my Skylake it answers YES, as it should. It isn't the way it was compiled because
I'm using the same file downloaed from CMB.

The software build test is supposed to reflect the compilation, not the HW support, however, the same SW reports
different results on different HW. I can only conclude the CPU does affect the SW build test. Without a relaible way
to test the SW build I have no choice but to disable the test.

Code:

        if ( cpu_has_sse2 )
         {
            printf("%s\n", grn_yes );
            printf("   SW built for SSE2..........");
            if ( sw_has_aes )
            {
              printf("%s\n", ylw_no );

See anything out of place?

You say you're checking for SSE2, but in code you're checking for sw_has_aes.

[th3.r00t]

And another suggestion - since you have Windows support, can you add and icon to the compiled binary?

I made myself one that looks good IMHO on every resolution on windows.
I will be more than happy to provide the files that I used - icon.ico, icon.rc and icon.o

This my configure to add and the icon in binary:

Code:

CFLAGS="-O3 -march=native" CXXFLAGS="$CFLAGS -std=gnu++11" LDFLAGS="../icon.o" ./configure --with-curl

Here is the screen:


The link and license for the icon
http://www.iconarchive.com/show/tulliana-2-icons-by-umut-pulat/k-cm-processor-icon.html

[joblo]

Thanks. You're the third person with Westmere with this problem. The main problem is "SW built for SSE2..........NO."
When I use the same file on my Skylake it answers YES, as it should. It isn't the way it was compiled because
I'm using the same file downloaed from CMB.

The software build test is supposed to reflect the compilation, not the HW support, however, the same SW reports
different results on different HW. I can only conclude the CPU does affect the SW build test. Without a relaible way
to test the SW build I have no choice but to disable the test.

Code:

        if ( cpu_has_sse2 )
         {
            printf("%s\n", grn_yes );
            printf("   SW built for SSE2..........");
            if ( sw_has_aes )
            {
              printf("%s\n", ylw_no );

See anything out of place?

You say you're checking for SSE2, but in code you're checking for sw_has_aes.

It's reverse logic. I'm actually checking to make sure it was not built with AES. An AES build is not
compatible on a non-AES CPU. A non-AES build is assumed to be SSE2.

But that logic had escaped me. In fact I never check sw_has_sse2 in the code.

Given that new information I still struggle with the fact that the westmere build for Windows shows
different results on my CPU (SSE2==YES), and real westmere/nehalem CPUs (SSE2==NO).

Given the actual code is testing for AES the result (SSE2==NO) actually means (AES==YES), the build
supports AES. Since an AES build won't work on a non-AES CPU the miner errors out with an incompatible
build.

The bottom line is the same build is being seen as AES on westmere but non AES on my skylake.
The difference with checking AES is that is the compiler macro that is now showing conflicting results.
 __AES__ returns true on westmere and false on skylake. And that's where I'm stuck.

Care to take another shot? Even when you miss the mark you make me think a little differently.