[LOCKED] cpuminer-opt v3.12.3, open source optimized multi-algo CPU miner

[joblo]

Welcome to the 3.5.8 version :-) !

You bombed my announcement.

[felixbrucker]

Large pages looks like a great feature for a specialist mining distribution. Both the OS and the apps
could be preconfigured for large pages. User friendly and plug and play.

Thougths

enabling large pages in a linux pre built image is easy, should be doable, but im not aware of any cpumining distros/images, anybody?
obviously one can always ssh into the system and install the cpuminer himself, but then he also can enable large pages himself and its not userfriendly/plug&play

Here is the best known (to me) mining distro.

https://bitcointalk.org/index.php?topic=520998.msg5764866#msg5764866

I also found this which describes how to enable an app to use large pages. It includes the following...

https://lwn.net/Articles/375096/

While applications can be modified to use any of the interfaces, it imposes a significant burden on the application developer.
 To make life easier, libhugetlbfs can back a number of memory region types automatically when it is either pre-linked or pre-loaded.
 This process is described in the HOWTO documentation and manual pages that come with libhugetlbfs.

Didn't read the HOWTO yet but what I infer from that is the OS and application are tightly coupled, which would make it
more challenging to build as a standalone application.

thats a gpu distro (nvidia), right? no oob cpuminer support afaik

i have also found the following stated from microsoft about large pages:

Large-page memory regions may be difficult to obtain after the system has been running for a long time because the physical space for each large page must be contiguous, but the memory may have become fragmented. Allocating large pages under these conditions can significantly affect system performance. Therefore, applications should avoid making repeated large-page allocations and instead allocate all large pages one time, at startup.

The memory is always read/write and nonpageable (always resident in physical memory).

The memory is part of the process private bytes but not part of the working set, because the working set by definition contains only pageable memory.

Large-page allocations are not subject to job limits.


that might be also the case for linux where its often advised to set the hugepage size on boot (in grub)

you will probably need to write a wrapper function for the allocation part to handle linux/windows differences

[joblo]

you will probably need to write a wrapper function for the allocation part to handle linux/windows differences

I don't want to mess with the OS part. I will work with someone to integrate cpuminer into a mining distro
or, if not too difficult,  I can provide a standalone cpuminer package which will work with an OS preconfigured
for large pages. My last post suggested the latter was difficult.

KopiemTu is advertized as Nvidia mining distro but it doesn't mean it can't be more unless it's excludively
sponsored by Nvidia.

Either way I think a mining distro preconfigured for large pages with cpuminer included is the best approach.
It's just not something I can, or am willing, do all myself.

I will continue, so see what exactly is involved in making a cpuminer package that supports large pages,
as long as it's transparent to desktop users and requires no OS specific mods to the application code.

[felixbrucker]

you will probably need to write a wrapper function for the allocation part to handle linux/windows differences

I don't want to mess with the OS part. I will work with someone to integrate cpuminer into a mining distro
or, if not too difficult,  I can provide a standalone cpuminer package which will work with an OS preconfigured
for large pages. My last post suggested the latter was difficult.

KopiemTu is advertized as Nvidia mining distro but it doesn't mean it can't be more unless it's excludively
sponsored by Nvidia.

Either way I think a mining distro preconfigured for large pages with cpuminer included is the best approach.
It's just not something I can, or am willing, do all myself.

I will continue, so see what exactly is involved in making a cpuminer package that supports large pages,
as long as it's transparent to desktop users and requires no OS specific mods to the application code.

afaik the systemspecific code (that being hugetlbfs and largepages for ms) has to be written anyways if you want it to be included in any distro, the distro just takes away the os setup (which is easy), or have i missed something?

i meant distros with cpumining option already available

[joblo]

you will probably need to write a wrapper function for the allocation part to handle linux/windows differences

I don't want to mess with the OS part. I will work with someone to integrate cpuminer into a mining distro
or, if not too difficult,  I can provide a standalone cpuminer package which will work with an OS preconfigured
for large pages. My last post suggested the latter was difficult.

KopiemTu is advertized as Nvidia mining distro but it doesn't mean it can't be more unless it's excludively
sponsored by Nvidia.

Either way I think a mining distro preconfigured for large pages with cpuminer included is the best approach.
It's just not something I can, or am willing, do all myself.

I will continue, so see what exactly is involved in making a cpuminer package that supports large pages,
as long as it's transparent to desktop users and requires no OS specific mods to the application code.

afaik the systemspecific code (that being hugetlbfs and largepages for ms) has to be written anyways if you want it to be included in any distro, the distro just takes away the os setup (which is easy), or have i missed something?

i meant distros with cpumining option already available

The arcticle I quoted says it's easier if pre-loaded and pre-linked.
To support a standalone large page enabled cpuminer application would require taking the difficult route.

I was talking in general terms about a distro. It could be an existing one or a new one, doesn't matter. A large
page enabled miner built into a large page enabled distro seems like the simplest route.

Edit: the fragmentation issue is also a concern. It may require rebooting when changing algos and modifying
algos not to use dynamic allocation/free while mining.

My bottom line concern is this feature is intended, and optimized, for single process servers and will not work
well in a desktop environment. I'm trying to identify all the pitfalls and develop an approach before starting to do
any real work.

[felixbrucker]

i dont know if this is the desired behaviour, but in the HOWTO the following is stated:

libhugetlbfs can be used to make an existing application use hugepages
for all its malloc() calls.  This works on an existing (dynamically
linked) application binary without modification.

[stealth.money]

Is this version with fees or no fee? I hope the OP could be more clear on this. Thanks

It's open source, isn't that clear enough?

Never mind I though this was optiminer, as we know optiminer come with fees.
I didn't pay attention to the title much.

[joblo]

i dont know if this is the desired behaviour, but in the HOWTO the following is stated:

libhugetlbfs can be used to make an existing application use hugepages
for all its malloc() calls.  This works on an existing (dynamically
linked) application binary without modification.

If that's the case maybe I don't need to change anything in cpuminer.

Edit: It looks like tranparent large pages is the way to go, and should have been from the start.
It wasn't so complicated when disk drives increased their block size.

[felixbrucker]

i dont know if this is the desired behaviour, but in the HOWTO the following is stated:

libhugetlbfs can be used to make an existing application use hugepages
for all its malloc() calls.  This works on an existing (dynamically
linked) application binary without modification.

If that's the case maybe I don't need to change anything in cpuminer.

Edit: It looks like tranparent large pages is the way to go, and should have been from the start.
It wasn't so complicated when disk drives increased their block size.

my impression was that transparent/automatic huge pages won't increase the performance as much as using the hugepages calls directly in the programm, but if this is wrong that would be great news as implementing huge pages would then become easier by far (i suppose)

[joblo]

i dont know if this is the desired behaviour, but in the HOWTO the following is stated:

libhugetlbfs can be used to make an existing application use hugepages
for all its malloc() calls.  This works on an existing (dynamically
linked) application binary without modification.

If that's the case maybe I don't need to change anything in cpuminer.

Edit: It looks like tranparent large pages is the way to go, and should have been from the start.
It wasn't so complicated when disk drives increased their block size.

my impression was that transparent/automatic huge pages won't increase the performance as much as using the hugepages calls directly in the programm, but if this is wrong that would be great news as implementing huge pages would then become easier by far (i suppose)

From my understanding of MMU and TLB from another architecture...
From the application perspective accessing huge pages is just derefeferencing a pointer like any other data
access. The magic happens when the CPU executes the load instruction and translates that pointer to a physical
memory address. The only difference is the accesses are faster because the translation is cached and
the same cached trasnslation can be used for the entire larger page before a new mapping is required for the next page.
At a higher level fewer TLBs can access more memory.

Reserving a section of memory for large pages and using the file system to access it may provide benefits beyond
large pages, though I don't see how. I would think going through the file system would add extra overhead.

[felixbrucker]

i dont know if this is the desired behaviour, but in the HOWTO the following is stated:

libhugetlbfs can be used to make an existing application use hugepages
for all its malloc() calls.  This works on an existing (dynamically
linked) application binary without modification.

If that's the case maybe I don't need to change anything in cpuminer.

Edit: It looks like tranparent large pages is the way to go, and should have been from the start.
It wasn't so complicated when disk drives increased their block size.

my impression was that transparent/automatic huge pages won't increase the performance as much as using the hugepages calls directly in the programm, but if this is wrong that would be great news as implementing huge pages would then become easier by far (i suppose)

From my understanding of MMU and TLB from another architecture...
From the application perspective accessing huge pages is just derefeferencing a pointer like any other data
access. The magic happens when the CPU executes the load instruction and translates that pointer to a physical
memory address. The only difference is the accesses are faster because the translation is cached and
the same cached trasnslation can be used for the entire larger page before a new mapping is required for the next page.
At a higher level fewer TLBs can access more memory.

Reserving a section of memory for large pages and using the file system to access it may provide benefits beyond
large pages, though I don't see how. I would think going through the file system would add extra overhead.

interesting

maybe only a real world test will clear the performance questions up

[joblo]

i dont know if this is the desired behaviour, but in the HOWTO the following is stated:

libhugetlbfs can be used to make an existing application use hugepages
for all its malloc() calls.  This works on an existing (dynamically
linked) application binary without modification.

If that's the case maybe I don't need to change anything in cpuminer.

Edit: It looks like tranparent large pages is the way to go, and should have been from the start.
It wasn't so complicated when disk drives increased their block size.

my impression was that transparent/automatic huge pages won't increase the performance as much as using the hugepages calls directly in the programm, but if this is wrong that would be great news as implementing huge pages would then become easier by far (i suppose)

From my understanding of MMU and TLB from another architecture...
From the application perspective accessing huge pages is just derefeferencing a pointer like any other data
access. The magic happens when the CPU executes the load instruction and translates that pointer to a physical
memory address. The only difference is the accesses are faster because the translation is cached and
the same cached trasnslation can be used for the entire larger page before a new mapping is required for the next page.
At a higher level fewer TLBs can access more memory.

Reserving a section of memory for large pages and using the file system to access it may provide benefits beyond
large pages, though I don't see how. I would think going through the file system would add extra overhead.

interesting

maybe only a real world test will clear the performance questions up

Some of my old theory is coming back so while I'm on a roll...

Fragmentation should be less of an issue with pure HP because each page is guaranteeed to be contiguous,
and being all the same size can always find a fit. However, memory bloat will occur because every app's
VM size will be rounded up to the next page.

In a hybrid environment the huge page is just a container (pool)  for many small pages. As long as small pages
are preallocated from the pool in groups they will not cause problems for the huge pages. Things only go bad
when a scattering of small pages results in difficulty finding a contiguous block big enough for a huge page.
As long as containers are used they all fit together nicely just like on a ship.

Edit: Still rolling

Some of the large pages will be nearly empty if they belong to apps with low memory requirenents. Being able
to allocate small pages for these apps means they can share a big page container reducing gaps and increasing
memory efficiency.

[m1n1ngP4d4w4n]

v3.5.8 is out.

The main thing is lyra2re is fixed on Windows. Also ported the cryptonight optimization from 3.5.7
to the non-aes version but the results were disappointing. Also disappointing were the results of precalculating
the first function's midstate hash for xevan and veltor. It seems to have a bigger impact on high hash rate algos.

Gonna test it on my good old CPU, thanks 

Edit : Yes it's working, now on par with minergate private miner, yay \o/, good job !

[YourMom420]

Hmq1725 Algo isn't working. Says stack smashing detected.

[joblo]

Hmq1725 Algo isn't working. Says stack smashing detected.

It works for me. Can you provide more details?

[YourMom420]

Hmq1725 Algo isn't working. Says stack smashing detected.

It works for me. Can you provide more details?

all algos work just fine except for that one. Was trying to mine doubloons on suprnova and it just crashes immediately after the difficulty shows up. Tried it on multiple computers. I'm on linux

[joblo]

Hmq1725 Algo isn't working. Says stack smashing detected.

It works for me. Can you provide more details?

all algos work just fine except for that one. Was trying to mine doubloons on suprnova and it just crashes immediately after the difficulty shows up. Tried it on multiple computers. I'm on linux

That's where I tested. Show me the output. What CPU do you have? What version of cpuminer-opt? When did
it break? Does 3.4.12 work? What version of Linux? How did you compile? I can't reproduce it so you need to
give me all the details.

[YourMom420]

Hmq1725 Algo isn't working. Says stack smashing detected.

It works for me. Can you provide more details?

all algos work just fine except for that one. Was trying to mine doubloons on suprnova and it just crashes immediately after the difficulty shows up. Tried it on multiple computers. I'm on linux

That's where I tested. Show me the output. What CPU do you have? What version of cpuminer-opt? When did
it break? Does 3.4.12 work? What version of Linux? How did you compile? I can't reproduce it so you need to
give me all the details.

I have the newest and the one you asked if I had. Both do the same thing. It broke as soon as I tried to mine the coin. I'm on Ubuntu 16.04. I have a i5-2450. Miner works perfect in Windows but I'd rather run it in Linux. I used ./build.sh and the ./autogen.sh way. Both compiled fine but crash the same way.

[joblo]

Hmq1725 Algo isn't working. Says stack smashing detected.

It works for me. Can you provide more details?

all algos work just fine except for that one. Was trying to mine doubloons on suprnova and it just crashes immediately after the difficulty shows up. Tried it on multiple computers. I'm on linux

That's where I tested. Show me the output. What CPU do you have? What version of cpuminer-opt? When did
it break? Does 3.4.12 work? What version of Linux? How did you compile? I can't reproduce it so you need to
give me all the details.

I have the newest and the one you asked if I had. Both do the same thing. It broke as soon as I tried to mine the coin. I'm on Ubuntu 16.04. I have a i5-2450. Miner works perfect in Windows but I'd rather run it in Linux. I used ./build.sh and the ./autogen.sh way. Both compiled fine but crash the same way.

It works on my i5-2400. I reviewed the code and found no suspicious pointer usage. This algo shares much of its code with other
algos that work for you, and hmq1725 works for you on Windows.

I have nothing to work with. You hinted that you compiled on Windows. I asked for all the details but you only gave me a little more.
If you compiled on Windows try using the precompiled binaries to see if they work also. Also try different archiectures (except AVX2)
to see if the problem is architecture specific.

Unless you can come up with more details of what works and what doesn't, as well as posting the program output ( I specifically asked
for that too) I won't be able to help.

Edit: I just noticed your CPU is a laptop, it would have been worth mentioning that or at least correctly identify it as i5-2450M.
Are you trying to make things more difficult?

hmq1725 has a very large stack frame, it is possible the crash is a simple stack overflow rather than the more ominous "smashing".
I will make a change in the next release to reduce the stack usage but I have no idea if it will fix your problem.

[YourMom420]

i really do apologize for being a pain in the ass. I compiled on Ubuntu 16.04. i hope this is the output your asking for. If not how do i find the other
name@name-VPCEH32FX:~/cpuminer-opt$ ./cpuminer -a hmq1725 -o stratum+tcp://boat.suprnova.cc:4156 -u twiztid1.2 -p x -t 2

      **********  cpuminer-opt 3.5.8  ***********
     A CPU miner with multi algo support and optimized for CPUs
     with AES_NI and AVX extensions.
     BTC donation address: 12tdvfF7KmAsihBXQXynT6E6th2c2pByTT
     Forked from TPruvot's cpuminer-multi with credits
     to Lucas Jones, elmad, palmd, djm34, pooler, ig0tik3d,
     Wolf0, Jeff Garzik and Optiminer.

CPU:        Intel(R) Core(TM) i5-2450M CPU @ 2.50GHz
CPU features: SSE2 AES AVX
SW built on Feb 19 2017 with GCC 5.4.0
SW features: SSE2 AES AVX
Algo features: SSE2 AES AVX AVX2
Start mining with SSE2 AES AVX

[2017-02-19 13:30:09] Starting Stratum on stratum+tcp://boat.suprnova.cc:4156
[2017-02-19 13:30:09] Binding thread 0 to cpu 0 (mask 1)
[2017-02-19 13:30:09] 2 miner threads started, using 'hmq1725' algorithm.
[2017-02-19 13:30:09] Binding thread 1 to cpu 1 (mask 2)
[2017-02-19 13:30:09] Stratum session id: deadbeefcafebabe382b000000000000
[2017-02-19 13:30:12] stratum extranonce subscribe timed out
[2017-02-19 13:30:12] DEBUG: job_id='8e47' extranonce2=00000000 ntime=58a9f235
[2017-02-19 13:30:12] Stratum difficulty set to 1 (0.00002)
[2017-02-19 13:30:12] hmq1725 block 29228, diff 3.345
[2017-02-19 13:30:13] DEBUG: hash <= target
Hash:   00004bd64e027695caea3732e8c8c775d6c12260d3829714b213d30aa27f960d
Target: 0000ffff00000000000000000000000000000000000000000000000000000000
*** stack smashing detected ***: ./cpuminer terminated
Aborted (core dumped)
name@name-VPCEH32FX:~/cpuminer-opt$