[LOCKED] cpuminer-opt v3.12.3, open source optimized multi-algo CPU miner

[integrale]

ok , thanks for your advice , i will ty the last time now  if it works.

[joblo]

ok , thanks for your advice , i will ty the last time now  if it works.

I've tested both algos with core2 compile on i7-6700K and they work ok.

[stealth.money]

Is this version with fees or no fee? I hope the OP could be more clear on this. Thanks

[joblo]

Is this version with fees or no fee? I hope the OP could be more clear on this. Thanks

It's open source, isn't that clear enough?

[felixbrucker]

Ryzen will be a cryptonight hashing beast.

8 cores, 16MB cache

I hope you can look into the huge pages stuff.

The key for cryptonight performance is cache size and AES performance. 16MB cache is good for 8 threads
but AMD implementations of Intel technology tend to be inferiour.

You'll have to build a good case for large pages. It looks like a lot of trouble with inconsistent results. Nicehash
experimented with it, how did that work out?

Edit: here are some of the questions that need answering in addition to a typical pro-con.

1. What is large pages exactly?

2. What are the OS issues, What changes are required to the OS?

3. Implementation issues, how much code needs changing?

4. User issues, do users need to be root/admin to run cpuminer with large pages?

5. Performance issues, are there conditions where large pages decreases performance?

If you have links to info that answers these questions that's good. I'm a bit skeptical about this
and don't feel like doing all the research work. It also gives me time to decompress after the
Lyra2 issues.

i can try to answer some of these questions, i have encountered large/huge pages in linux in a different field: networking

i have worked with the intel dpdk (dataplane development kit) which roughly gives the user the ability to implement pmd (polling mode driver) for nic's. i have used this in conjunction with open vswitch to build a packet forwarding and routing VM based on ubuntu lts. large pages are used for efficient transport of many packets from host to vm and back.
in general large pages are just like normal pages, except they are bigger/larger and thus the tlb has less entries to go through and is quicker. also large copies are faster.

i have only used this with linux and in linux these pages are reserved on boot, this space is not available for other programs
on windows i have only used it for the mentioned xmr stak miner and it seems its not reserved on boot (at least i didnt see any increase in ram usage in task manager) and the miners ram usage also seems rather small (when i was dealing with the packet forwarding stuff i easily reserved 16GB of ram across multiple numa nodes)

regarding the questions directly:

1) basically larger normal pages as explained before (you can read the first paragraph below the table here: https://en.wikipedia.org/wiki/Page_(computer_memory)#Huge_pages )

2.1) on linux you will just have the reserved amount of ram less (i have not tested this with the xmr miner, might differ), you will need to enable this in the grub config once, i would rate this as 1/10 on a difficulty scale
2.2) on windows you will need to go through some gui stuff in the group policy settings to enable it, thats it, i would rate that 1/10 on a difficulty scale

2.2 side note: i have observed my networking transfer speed is limited to about 120mbit/s when i run the xmr stak miner with huge pages enabled, once i exit it, it goes back to full 1 gbit (its a realtek nic)

3) sadly i cant answer that part you might want to contact the dev of the stak miner about it, not sure if he wants to help though

4) it is stated that i have to run the miner "as admin" in windows for hugepages, though it worked flawlessly without, linux not sure

5) i only observed the nic slowdown in 2.2 side note, i only ran it on windows till now
when the miner isnt run i have not experienced any issues while hugepages are enabled

[integrale]

ok , thanks for your advice , i will ty the last time now  if it works.

I've tested both algos with core2 compile on i7-6700K and they work ok.

of course , i believe you that on intel cpu with complete instruction-set works or even sse2 , the case was been AMD with only sse2,sse3
  looks like you mixed something up now    no probs  your head is full from solving poor peoples problems  lol

[xcbtrader]

Hello...

Help to install cpuminer-opt in ubuntu 16.04

Thanks

[integrale]

whats the problem ?

wich kind of assist need you
more facts pls.

what kind of Pc  (cpu )
installed  dependencies ?

maybe this one can help you : http://askubuntu.com/questions/457526/how-to-install-cpuminer-in-ubuntu
if not come back we help you

[joblo]

whats the problem ?

wich kind of assist need you
more facts pls.

what kind of Pc  (cpu )
installed  dependencies ?

maybe this one can help you : http://askubuntu.com/questions/457526/how-to-install-cpuminer-in-ubuntu
if not come back we help you

That's a useful link, but everyone should read the instructions in RELEASE_NOTES.

[joblo]

ok , thanks for your advice , i will ty the last time now  if it works.

I've tested both algos with core2 compile on i7-6700K and they work ok.

of course , i believe you that on intel cpu with complete instruction-set works or even sse2 , the case was been AMD with only sse2,sse3
  looks like you mixed something up now    no probs  your head is full from solving poor peoples problems  lol

I just need to confirm where the problem is. Did it work for you compiled for core2. If yes there's no problem, if no there's
an AMD problem.

[joblo]

i can try to answer some of these questions, i have encountered large/huge pages in linux in a different field: networking

Thanks I'll absorb it in time and do a little research, but it's low prority.

[integrale]

ok , thanks for your advice , i will ty the last time now  if it works.

I've tested both algos with core2 compile on i7-6700K and they work ok.

of course , i believe you that on intel cpu with complete instruction-set works or even sse2 , the case was been AMD with only sse2,sse3
  looks like you mixed something up now    no probs  your head is full from solving poor peoples problems  lol

I just need to confirm where the problem is. Did it work for you compiled for core2. If yes there's no problem, if no there's
an AMD problem.

it wont work on any arch i choose on AMD with Cpuminer-opt ( yesscrypt . lyra2z , xevan )
but on Cpuminer-multi yescrypt , xevan working  , dont know if lyra2z is supported yet
so it looks not really as only AMD problem to be     

[joblo]

ok , thanks for your advice , i will ty the last time now  if it works.

I've tested both algos with core2 compile on i7-6700K and they work ok.

of course , i believe you that on intel cpu with complete instruction-set works or even sse2 , the case was been AMD with only sse2,sse3
  looks like you mixed something up now    no probs  your head is full from solving poor peoples problems  lol

I just need to confirm where the problem is. Did it work for you compiled for core2. If yes there's no problem, if no there's
an AMD problem.

it wont work on any arch i choose on AMD with Cpuminer-opt ( yesscrypt . lyra2z , xevan )
but on Cpuminer-multi yescrypt , xevan working  , dont know if lyra2z is supported yet
so it looks not really as only AMD problem to be     

Thanks. Now you're saying xevan is broken also. This appears to be an issue with AMD and the optimizations
in cpuminer-opt. However, yescrypt is not optimized so it should be identical to cpuminer-multi.

Are there any other AMD users seeing this problem?

[joblo]

Some disjointed thoughts on large pages.

My undertsanding is than enabling large pages in the OS reserves a block of memory for use
reducing the amount available for applications, presumably ones that don't use large pages.
This leads to an engineering issue. How many large pages should be allocated in the OS?
How many large pages would cpuminer use? Is it different based on the algo. What if there aren't
enough large pages for cpuminer?

I found an easier way to enable large pages in Linux, no grub required. It's old and for RH but the commands,
kernel variables and files exist on Ubuntu so it should work.

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Tuning_and_Optimizing_Red_Hat_Enterprise_Linux_for_Oracle_9i_and_10g_Databases/sect-Oracle_9i_and_10g_Tuning_Guide-Large_Memory_Optimization_Big_Pages_and_Huge_Pages-Configuring_Huge_Pages_in_Red_Hat_Enterprise_Linux_4_or_5.html

The Wikipedia article says large pages aren't widely used and used mostly in server environments but it
doesn't say why. I suspect it doesn't perform well in a desktop environment with multiple applications
running.

The arcticle also mentioned elevated priviledges, though it's not clear they mean the application must run
as root or only the OS config. It also mentioned issues swapping large pages to disk

The arcticle also mentions the application needs to set a flag, no such flag exists in malloc so it's not so
simple.

I have some thoughts on possible implementation. This looks like a specialist feature that is not applicable
to some environments therefore cpuminer would have to disable it by default. cpuminer would also have
to confirm large pages is enabled in the OS before using them.
Edit: I'm reluctant to advertise such a feature because then I'd be responsible fo helping users setup their OS.

Do large pages apply to static application data or only dynamically allocated memory?

[felixbrucker]

Some disjointed thoughts on large pages.

My undertsanding is than enabling large pages in the OS reserves a block of memory for use
reducing the amount available for applications, presumably ones that don't use large pages.
This leads to an engineering issue. How many large pages should be allocated in the OS?
How many large pages would cpuminer use? Is it different based on the algo. What if there aren't
enough large pages for cpuminer?

I found an easier way to enable large pages in Linux, no grub required. It's old and for RH but the commands,
kernel variables and files exist on Ubuntu so it should work.

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Tuning_and_Optimizing_Red_Hat_Enterprise_Linux_for_Oracle_9i_and_10g_Databases/sect-Oracle_9i_and_10g_Tuning_Guide-Large_Memory_Optimization_Big_Pages_and_Huge_Pages-Configuring_Huge_Pages_in_Red_Hat_Enterprise_Linux_4_or_5.html

The Wikipedia article says large pages aren't widely used and used mostly in server environments but it
doesn't say why. I suspect it doesn't perform well in a desktop environment with multiple applications
running.

The arcticle also mentioned elevated priviledges, though it's not clear they mean the application must run
as root or only the OS config. It also mentioned issues swapping large pages to disk

The arcticle also mentions the application needs to set a flag, no such flag exists in malloc so it's not so
simple.

I have some thoughts on possible implementation. This looks like a specialist feature that is not applicable
to some environments therefore cpuminer would have to disable it by default. cpuminer would also have
to confirm large pages is enabled in the OS before using them.
Edit: I'm reluctant to advertise such a feature because then I'd be responsible fo helping users setup their OS.

Do large pages apply to static application data or only dynamically allocated memory?

ah yes i forgot, once enabled you can also set the amount dynamically in the os with the commands referenced

i fully understand and support your statement about advertising this feature, a simple cli option to enable it would be enough in my opinion (just link the os setup guides in the readme or somewhere, everything else is up to the user)

well its not widely used because for most applications there is no need to use it, only memory heavy applications can benefit from it, which largely limits the usecases for most applications (except maybe chrome )
i dont think it reduces performance, the ram is just gone, so if your system has lets say 12gb and your application uses 1gb, the remaining 11gb should still work as usual

about swapping i dont know, i never had to (and it would ruin performance as well), better know your specs and use hugepages accordingly


last note: you can enable transparent hugepages in linux (not sure if enabled by default), you can read about that here: https://www.kernel.org/doc/Documentation/vm/transhuge.txt

doesnt require you to rewrite anything afaik
seems you also need to adjust the allocations

[joblo]

Large pages looks like a great feature for a specialist mining distribution. Both the OS and the apps
could be preconfigured for large pages. User friendly and plug and play.

Thougths

[felixbrucker]

Large pages looks like a great feature for a specialist mining distribution. Both the OS and the apps
could be preconfigured for large pages. User friendly and plug and play.

Thougths

enabling large pages in a linux pre built image is easy, should be doable, but im not aware of any cpumining distros/images, anybody?
obviously one can always ssh into the system and install the cpuminer himself, but then he also can enable large pages himself and its not userfriendly/plug&play

[joblo]

Large pages looks like a great feature for a specialist mining distribution. Both the OS and the apps
could be preconfigured for large pages. User friendly and plug and play.

Thougths

enabling large pages in a linux pre built image is easy, should be doable, but im not aware of any cpumining distros/images, anybody?
obviously one can always ssh into the system and install the cpuminer himself, but then he also can enable large pages himself and its not userfriendly/plug&play

Here is the best known (to me) mining distro.

https://bitcointalk.org/index.php?topic=520998.msg5764866#msg5764866

I also found this which describes how to enable an app to use large pages. It includes the following...

https://lwn.net/Articles/375096/

While applications can be modified to use any of the interfaces, it imposes a significant burden on the application developer.
 To make life easier, libhugetlbfs can back a number of memory region types automatically when it is either pre-linked or pre-loaded.
 This process is described in the HOWTO documentation and manual pages that come with libhugetlbfs.

Didn't read the HOWTO yet but what I infer from that is the OS and application are tightly coupled, which would make it
more challenging to build as a standalone application.

[ZenFr]

Welcome to the 3.5.8 version :-) !

[joblo]

v3.5.8 is out.

The main thing is lyra2re is fixed on Windows. Also ported the cryptonight optimization from 3.5.7
to the non-aes version but the results were disappointing. Also disappointing were the results of precalculating
the first function's midstate hash for xevan and veltor. It seems to have a bigger impact on high hash rate algos.