[LOCKED] cpuminer-opt v3.12.3, open source optimized multi-algo CPU miner

[malafaya]

This CPU affinity crap is starting to piss me off. That's not what I came back for.
It's taking way too much of my time.

This all started with a request to support CPU groups on Windows. Windows is a
commercial product: it costs money to use it and takes even more money to
develop on it and for it. Enterprise level development (multi-socket, large thread
counts, NUMA, CPU groups, etc) costs even more.

Yet I don't get paid a penny.

From now on any requests for features specific to Windows must be accompanied
by a nominal non-refundable donation. I will do an assessment and determine if I can
do it with my skill level and resources and how much work may be involved. I will
then set a fee payable on completion. The fee will be scaled based on whether the
feature is for consumers or enterprises.

CPU groups is an enterprise feature. I will make one more attempt to fix the current
issues. If that fails I wlll remove the CPU Groups support and revert to v3.8.8.1 affinity..

I got you.
As a side note, I have no problem with CPU affinity not working. Like I said before, I set it externally exactly because it never seemed to be reliable. The only reason I'm doing all this troubleshooting is because you asked why I was was doing that (setting affinity externally) instead of using the built-in feature.
I don't need it fixed. It works for me as is. I just thought you actually wanted to fix it. And again, I'm not using CPU groups.

That said I definitely would prefer efforts to be directed to something more useful, such as performance improvements as well as new algos. So I think we're on the same wavelength.

Cheers.

CPU affinity still does not set correctly in v3.9.2.3. In Task Manager, it seems it still can't affine CPUs above CPU31. Also, some CPUs not specified in the mask are set.

Setting --cpu-affinity=0x55555545555544 with 56 CPUs:

Enough with the overly complex test cases. I've said it before DEFAULTS!

What are the defaults? Not setting affinity? The performance drop is significant in that case.

With default (not set) affinity (all CPUs involved):

Code:

         **********  cpuminer-opt 3.9.2.3  ***********
     A CPU miner with multi algo support and optimized for CPUs
     with AES_NI and AVX2 and SHA extensions.
     BTC donation address: 12tdvfF7KmAsihBXQXynT6E6th2c2pByTT

CPU: Intel(R) Xeon(R) CPU E5-2660 v4 @ 2.00GHz.
SW built on Jun  5 2019 with GCC 7.3.0.
CPU features: SSE2 AES SSE4.2 AVX AVX2.
SW features: SSE2 AES SSE4.2 AVX AVX2.
Algo features: SSE2.
Start mining with SSE2.

[2019-06-07 17:40:40] 56 CPU cores available, 8 miner threads selected.
[2019-06-07 17:40:40] 8 miner threads started, using 'yespowerr16' algorithm.
[2019-06-07 17:40:40] Starting Stratum on stratum+tcp://***
[2019-06-07 17:40:40] Stratum difficulty set to 1
[2019-06-07 17:41:01] yespowerr16 block 406437, network diff 0.014
[2019-06-07 17:41:53] Share submitted.
[2019-06-07 17:41:53] Accepted 1/1 (100%), diff 0.000167, 1273.16 H/s
[2019-06-07 17:42:14] Share submitted.
[2019-06-07 17:42:14] Accepted 2/2 (100%), diff 1.82e-005, 1275.10 H/s
[2019-06-07 17:42:15] Share submitted.
[2019-06-07 17:42:16] Accepted 3/3 (100%), diff 4.15e-005, 1264.21 H/s

With affinity set externally to some specific CPUs:

Code:

         **********  cpuminer-opt 3.9.2.3  ***********
     A CPU miner with multi algo support and optimized for CPUs
     with AES_NI and AVX2 and SHA extensions.
     BTC donation address: 12tdvfF7KmAsihBXQXynT6E6th2c2pByTT

CPU: Intel(R) Xeon(R) CPU E5-2660 v4 @ 2.00GHz.
SW built on Jun  5 2019 with GCC 7.3.0.
CPU features: SSE2 AES SSE4.2 AVX AVX2.
SW features: SSE2 AES SSE4.2 AVX AVX2.
Algo features: SSE2.
Start mining with SSE2.

[2019-06-07 17:36:34] 56 CPU cores available, 8 miner threads selected.
[2019-06-07 17:36:34] affine_to_cpu_mask for 1 returned 57
[2019-06-07 17:36:34] 8 miner threads started, using 'yespowerr16' algorithm.
[2019-06-07 17:36:34] affine_to_cpu_mask for 0 returned 57
[2019-06-07 17:36:34] Starting Stratum on stratum+tcp://***
[2019-06-07 17:36:34] affine_to_cpu_mask for 2 returned 57
[2019-06-07 17:36:34] affine_to_cpu_mask for 3 returned 57
[2019-06-07 17:36:34] affine_to_cpu_mask for 4 returned 57
[2019-06-07 17:36:34] affine_to_cpu_mask for 5 returned 57
[2019-06-07 17:36:34] affine_to_cpu_mask for 6 returned 57
[2019-06-07 17:36:34] affine_to_cpu_mask for 7 returned 57
[2019-06-07 17:36:35] Stratum difficulty set to 1
[2019-06-07 17:36:35] yespowerr16 block 406435, network diff 0.013
[2019-06-07 17:36:52] yespowerr16 block 406436, network diff 0.013
[2019-06-07 17:36:55] Share submitted.
[2019-06-07 17:36:55] Accepted 1/1 (100%), diff 2.72e-005, 1423.16 H/s
[2019-06-07 17:37:05] Share submitted.
[2019-06-07 17:37:05] Accepted 2/2 (100%), diff 6.46e-005, 1422.13 H/s

[1714975144]

1714975144

[joblo]

What are the defaults? Not setting affinity? The performance drop is significant in that case.

It's not about perfomance, it's a test to troubleshoot affinity. Yes start with all defaults then
add options until it breaks. And use -D for testing and note thread distribution in task manager.

1. all defaults, already done, test passed.

2. half threads (28), default affinity?

3. half threads, alternating affinity?

4. What you really want, test failed.

In my experience default affinity works best for Intel CPUs using half threads, but not on AMD where
a mask is needed with alternating bits set. Keep this in mind when you set affinity in test 4 to avoid
hyperthreading.

Repeating test 3 with 0x0055555555555555 and 0x000000000FFFFFFF should confirm that, the latter
giving the same performane as with default affinity.

And don't use any external tools to set affinity or otherwise mess with the test, It's a test, have I said
that before, don't introduce extraneous variables.


I've made some more tweaks and this is what I get  with -t 8 --cpu-affinity=0x5555
on Linux. I'l wait for your test results on Windows before I commit the change.

Also what is your goal? Why are you going to all this trouble with odd thread counts and
complex affinity masks?

Code:

[2019-06-07 14:15:03] 16 CPU cores available, 8 miner threads selected.
[2019-06-07 14:15:03] Binding process to cpu mask 5555
[2019-06-07 14:15:03] Thread 0 priority 0 (nice 18)
[2019-06-07 14:15:03] Binding thread 0 to cpu using affinity mask 0000000000000000 0000000000005555
[2019-06-07 14:15:03] Thread 1 priority 0 (nice 18)
[2019-06-07 14:15:03] Binding thread 1 to cpu using affinity mask 0000000000000000 0000000000005555
[2019-06-07 14:15:03] Thread 3 priority 0 (nice 18)
[2019-06-07 14:15:03] Thread 2 priority 0 (nice 18)
[2019-06-07 14:15:03] Binding thread 2 to cpu using affinity mask 0000000000000000 0000000000005555
[2019-06-07 14:15:03] Binding thread 3 to cpu using affinity mask 0000000000000000 0000000000005555
[2019-06-07 14:15:03] Thread 4 priority 0 (nice 18)
[2019-06-07 14:15:03] Binding thread 4 to cpu using affinity mask 0000000000000000 0000000000005555
[2019-06-07 14:15:03] 8 miner threads started, using 'x11' algorithm.
[2019-06-07 14:15:03] Thread 6 priority 0 (nice 18)
[2019-06-07 14:15:03] Binding thread 6 to cpu using affinity mask 0000000000000000 0000000000005555
[2019-06-07 14:15:03] Thread 7 priority 0 (nice 18)
[2019-06-07 14:15:03] Thread 5 priority 0 (nice 18)
[2019-06-07 14:15:03] Binding thread 7 to cpu using affinity mask 0000000000000000 0000000000005555
[2019-06-07 14:15:03] Binding thread 5 to cpu using affinity mask 0000000000000000 0000000000005555

[malafaya]

I've made some more tweaks and this is what I get  with -t 8 --cpu-affinity=0x5555
on Linux. I'l wait for your test results on Windows before I commit the change.

Also what is your goal? Why are you going to all this trouble with odd thread counts and
complex affinity masks?

I will do what you ask when I have the chance. I thought the problem was already clear that it would never set affinity to CPU# > 31.

What do you mean by "complex" masks? Why do I have to choose submultiples of the number of processors and why would it be better? In this last example, I set half minus 3. Changing some 5s to 4s in the mask shouldn't be considered "complex". Nor should 8 threads with a mask of 0x40404040404041 (8 throw better performance than 7   ). Shouldn't they all work? I'm sure that when "simple" masks work the "complex" ones will follow.

Question: "half threads (28), default affinity": what is a pass in this test? Default affinity always works in that all CPUs are allocated to the process.

I'll send my results back to you by PM in order not to pollute this thread.
Cheers.

[joblo]

Shouldn't they all work? I'm sure that when "simple" masks work the "complex" ones will follow.

Question: "half threads (28), default affinity": what is a pass in this test? Default affinity always works in that all CPUs are allocated to the process.

First question: Absolutely but they don't and it's easier to troubleshoot with fewer variables.

Second question: I can look at the data when available and determine myself but I also wanted to compare
with your expectations when using a simpler mask.

Computers are binary, everything works better in powers of 2.

I think you're overcomplicating things. Defaults exist for a reason, you shouldn't change them
without a very good reason and knowing what you're doing. You stilll haven't given me a reason
and I have questions that you know what you're doing considering you're using an alternating
bit affinity mask with an Intel CPU. That's not meant to be an insult, it's just that I fail to see any
logical reason for what you are doing or why.


There's another downside to specifying a custom cpu affinity. Whe using  the default
each thread is affined to a precise cpu, the one matching the thread number.
With a custom affinity mask it's more complicated so the thread is afffined to the "mask"
which means any of the allowable cpus.

I don't feel like doing the additional coding to do one to one matching of each thread
to a unique allowable cpu.

Affinity should only be needed if there is a good reason to run fewer threads and if the CPU
is an AMD. That's small set.

[joblo]

The mod is in a bad mood and decided to overreach with rule enforcement.
Here's a repost of the release announcement, all by itself. I hope it stays
that way.

This announcement deserves a thorough reading by anyone considering
using the cpu-affinity option when mining with fewer miner threads than CPU
threads.
---

cpuminer-opt-3.9.2.4

Yet more cpu-affinity fixes, hopefully the last.

https://github.com/JayDDee/cpuminer-opt/releases/tag/v3.9.2.4

Edit: The advice below applies to Linux. It seems on Windows the behaviour is
the exact opposite.

Affinity should rarely be used, dependant on how the CPU maps the affinity mask to
CPU threads.

With Intel CPUs the default will ensure all physical cores get assigned one thread before
doubling up with hyperthreading. Some fine tuning might help distribute the cores with
2 threads more evenly when miner thread count is not exactly half the number of CPU threads.

In short, on an Intel 8/16 CPU threads 0 & 8 are hyperthreaded on the same physical core
and therefore compete for shared resources and should be avoided as much as possible.

AMD is different and requires custom affinity specifying an alternating bit mask pattern (0x5555 or
0xAAAA for even or odd threads respectively) to ensure optimum distribution of miner threads.
This applies to all AMD CPUs due to the modular architecture of AMD's Computing Complex.

In short, on an AMD CPU threads 0 & 1 compete for shared physical resources.

[malafaya]

There's another downside to specifying a custom cpu affinity. Whe using  the default
each thread is affined to a precise cpu, the one matching the thread number.
With a custom affinity mask it's more complicated so the thread is afffined to the "mask"
which means any of the allowable cpus.

I don't feel like doing the additional coding to do one to one matching of each thread
to a unique allowable cpu.

Affinity should only be needed if there is a good reason to run fewer threads and if the CPU
is an AMD. That's small set.

The almost-alternating masks on Windows was found by trial and error to yield the best performance with all the Intel CPUs I've experimented with (correction: for a few older ones, it doesn't make any difference in which case I don't set affinity). I just know that Linux uses a different numbering convention and that rules are different, but an alternating mask always seemed best on Windows. Or perhaps the numbering convention in cpuminer is not the same as in Windows Power Shell. I will try to incorporate the info you posted along with the new release and see if it makes any difference. One thing seems for sure: many times, using all CPUs is worse than using just a subset (memory/L cache ceiling somewhere? Maybe with more CPUs it's even worse), and using a specific affinity seems to always work better than using default one with a subset of CPUs.

Thanks and let's move on.

[joblo]

The almost-alternating masks on Windows was found by trial and error to yield the best performance with all the Intel CPUs I've experimented with (correction: for a few older ones,

It seems you are correct. I have a skylake running linux, a haswell running windows and a ryzen
running linux. The ryzen and haswell both need affinity set, the skylake doesn't. I would extrapolate
that a ryzen running windows wouldn't need afinity either.

Seems odd but it's not an issue with the miner. Thanks for all your help. It's difficult when I don't
have the precise environment to reproduce certain problems.

[joblo]

cpuminer-opt-3.9.3.1

Skipped v3.9.3 due to misidentification of v3.9.2.5 as v3.9.3.
Fixed x16r algo 25% invalid share reject rate. The bug may have also
affected other algos.

https://github.com/JayDDee/cpuminer-opt/releases/tag/v3.9.3.1

----------

cpuminer-opt-3.9.2.5

Fixed 2 regressions:
  hodl AES detection failed,
  x16r invalid shares with AVX2.
More restructuring.

Errata: version displays as cpuminer-opt-3.9.3.
           x16r is still broken, sometimes prducing invalid shares.

[joblo]

At the risk of again offending the mods with my third post in a row, this is to announce
I will no longer be posting cpuminer-opt release announcements or updates on
issues in this forum.

This is the result of arcane forum rules forbidding serial posts, even by the thread
originator, combined with an uncompromising moderator.

No reason was offered for the existance of the rule against serial posts or why there
is no exception for the thread originator.

In reality the offense was there were no replies made by other users between my
posts resulting in my posts being consecutive. So they are saying it's all your faults.

It didn't seem to matter the posts were addressing different topics and a different
segment of the audience. They were consecutive so required enforcement action
without discretion, judgement, context or consultation. In other words zero tolerence.
That's not very friendly.

My point was made quite clearly many years ago by a former coworker:

"If you implement (a spam filter) that drops one piece of e-mail, it is a huge mistake."
https://www.zdnet.com/article/spam-its-completely-out-of-control/

This was a similar huge mistake.

Please visit github for all technical and release information about cpuminer-opt
You won't find it here anymore.

https://github.com/JayDDee/cpuminer-opt

[4ward]

No problem, we can provide you with "filler" posts ))

[joblo]

No problem, we can provide you with "filler" posts ))

I will continue to update the OP and the subject with the latest release.

However, any new posts by me may be frequently addressing different audiences
and different topics in the same post, and may be updated frequently with new
information.

I will also not be volunteering information or opinion that doesn't solicit a response.
I don't want to be locked out by having the last post in the thread.

It will be more challenging for readers as they will have to go back to previous
posts to check for new information instead of just checking new posts at the end of
the thread.

It will also be more challenging for TLDR people who may miss some useful information
if they aren't interested in the first topic of a multi topic post.

In short, I will not be able to share information as freely as I would like and that affects other
users more than me.

This thread is not by nature a balanced dialogue, it is mostly one way, and using strict rules
to try to make it more of a two way converstaion is stupid and counter-productive

[joblo]

If the following looks interesting, check out this issue I opened. Comments are welcome here.

https://github.com/JayDDee/cpuminer-opt/issues/190

Code:

[2019-06-23 02:31:17] Share submitted.
[2019-06-23 02:31:17] Accepted 209/210 (99.5%), diff 3.41e-05, 993.64 H/s, 60C
[2019-06-23 02:31:17] Share hash rate: 750.298 H/s, time: 24.408 secs.
[2019-06-23 02:31:17] Share of block: 0.0047, Block time 1.442 hours.
[2019-06-23 02:31:17] Block height 417440, net hash rate 1.6e+05
[2019-06-23 02:31:17] Averages: hash rate 943.8575 H/s, share 0.309 %
[2019-06-23 02:31:17] Average block time 1.738 hours.

[alexzerg111]

please tell me when you add support for cryptonight V8?

[Dotem]

please tell me when you add support for cryptonight V8?

See post #3819.

[joblo]

There are changes in the way shares are reported in cpuminer-opt-3.9.5.

Please Read the last 2 comments in git issue https://github.com/JayDDee/cpuminer-opt/issues/190
before reporting problems or asking questions.

Suggestions for improvement are welcome.

[marte1982]

hi dev..can you add new algo scrypt8k

https://bitcointalk.org/index.php?topic=5163850.0

https://github.com/LiteNoteProject/LiteNote

thanks for your support

[sefirotius]

I have a processor fx 8350 that I can mining them there are profitable coins for it one dollar a day would be nice

[joblo]

hi dev..can you add new algo scrypt8k

https://bitcointalk.org/index.php?topic=5163850.0

https://github.com/LiteNoteProject/LiteNote

thanks for your support

Try -a scrypt:8192

If that doesn't work, it's a custom version of scrypt which would be too much work.

[marte1982]

ERROR: ProcessNewBlock: AcceptBlock FAILED (high-hash, proof of work failed (code 16)

any solution dev?

[joblo]

ERROR: ProcessNewBlock: AcceptBlock FAILED (high-hash, proof of work failed (code 16)

any solution dev?

Not without a lot more info.