Need help with proof of concept cpu mining

[Sha256explorer]

Hello
I believe that I have an algorithm to mine bitcoin quickly and efficiently.
I studied a bit the asicboost process (https://blog.bitmex.com/an-overview-of-the-covert-asicboost-allegation) : I think my algorithm could be more efficient and could therefore be used to build a new generation of ASICs.
I would like to write some CPU mining software. Of course, in reality this software won't mine any blocks and I won't get any rewards. My aim is to compare it with other similar software (CPU mining), and to see if my algorithm reduces the work to be done and therefore can be used to design new more efficient ASICs.



I would like to proceed like this.
- download bitcoin core and put it to use
- download a bitcoin miner software (https://developer.bitcoin.org/devguide/mining.html) in which it is possible to use the CPU
- modify that mining software with my algorithm to see if it -works better

I found these three bitcoin node implementations:
- python: https://pypi.org/project/bitcoinlib/
- golang: https://github.com/btcsuite/btcd
- rust: https://github.com/jrawsthorne/rust-bitcoin-node
I don't take into consideration the classic implementation in c/c++ because I know almost nothing about C++ and nothing about C, and in any case these two languages don't interest me (actually, I don't even know rust or golang yet, but I'm interested in learning about them one, especially rust, and this could be a good occasion)

My questions are:
- are these three implementations (python/go/rust) all equally stable?
- Is there any mining software written in python/go/rust for use with these implementations that, starting with an RPC getblocktemplate call, starts mining with the CPU? (my job will be: modify this software)
- with a bitcoin node + mining software (if it exists) would I have everything I need to start mining with the CPU or do I need something else?

thank you for accepting me in this forum, for your attention and  for your help

[1715288784]

1715288784

[1715288784]

1715288784

[1715288784]

1715288784

[1715288784]

1715288784

[1715288784]

1715288784

[NotFuzzyWarm]

1st of all, this thread should be moved to the Development and Technical Discussion board where it would get a much larger and on-target audience.

That said, if one wants to test out said algorithms you do NOT test them out on a CPU. You use a FPGA to simulate the logical gate structures that would eventually be made into a hard wired ASIC. Being general purpose devices, coding to run on cpu's/gpu's is vastly different (and more complex) than programming hard-wired implementations of the code.

I should also add that despite after more than 10 years of some very bright minds working on this <drum roll please> there are no faster/better ways to process sha 256 hashes. Doesn't mean there isn't some as yet undiscovered methods to do it but - it also means it is pretty bloody unlikely.

[Sha256explorer]

Sorry for posting in the wrong place., thanks for moving it in the right place

1st of all, this thread should be moved to the Development and Technical Discussion board where it would get a much larger and on-target audience.

That said, if one wants to test out said algorithms you do NOT test them out on a CPU. You use a FPGA to simulate the logical gate structures that would eventually be made into a hard wired ASIC. Being general purpose devices, coding to run on cpu's/gpu's is vastly different (and more complex) than programming hard-wired implementations of the code.

I should also add that despite after more than 10 years of some very bright minds working on this <drum roll please> there are no faster/better ways to process sha 256 hashes. Doesn't mean there isn't some as yet undiscovered methods to do it but - it also means it is pretty bloody unlikely.

I agree that  bloody unlikely my algorithm will be faster: I take this as an occasion to learn golang o rust, that I want to do anyway. Then, if I am extremely lucky, my algorithm is really good as i now believe it is.

for this reason, I don’t want to play with FPGAs: I am not interested in this field of technical knowledge, and I am not so crazy (or too old) to learn it only for this dream.

to be honest, I am not fully convinced that doing a proof of work in  python/go/rust will be without value. But i will investigare more. Thanks for now!!
 

[FatFork]

for this reason, I don’t want to play with FPGAs: I am not interested in this field of technical knowledge, and I am not so crazy (or too old) to learn it only for this dream.

But if you don't want to test your algorithm on FPGAs, how will you ever know if it is better (more optimal) than the current version of ASICs? It does not make sense. ASICs are designed specifically for a certain task and are hard to beat, so if you want to compete with them, you really need to test your algorithm on an FPGA.

Another option would be to collaborate with someone who has expertise in FPGAs and can help you with the testing process. That way, you don't have to learn everything from scratch, but still get the benefits of using an FPGA.

[NotATether]

https://pypi.org/project/bitcoinlib/ is not a bitcoin node, it's a wallet generator.

Use python-bitcoinlib instead, which actually attempts to be a node. Do not use pybitcointools, which has its own share of problems.

Anyway, I'd steer clear from these python nodes altogether, because they are not exactly the kind that export a binary during installation that oyu can run - most of the work you have to do in the console which is suboptimal. Use one of the other languages instead.

[Sha256explorer]

But if you don't want to test your algorithm on FPGAs, how will you ever know if it is better (more optimal) than the current version of ASICs? It does not make sense. ASICs are designed specifically for a certain task and are hard to beat, so if you want to compete with them, you really need to test your algorithm on an FPGA.

The magic word is: algorithm.

 If the algorithm really allows to create hashes from the 80 bytes of the bitcoin header in a faster way, it will be able to do it faster  in python, in c++, with fpgas and with asics.

 The problem is: faster than what?

 As far as I know, asics don't have code that you can look at and possibly improve….their code is secret…

 This is quite a problem.

 Maybe I just reinvented what asics already do?  I can't know… right?

[NotFuzzyWarm]

As far as I know, asics don't have code that you can look at and possibly improve….their code is secret…

No, the physical layout of the large parallel array of cores in each chip and how data moves between them is their secret sauce.

The problem is: faster than what?

Faster than the current boolean logic process used by all implementations of the sha256 decryption equation. The actual gate logic making up each core IS out there, just not easy to search for & find. wikipedia or http://bitcoin.wiki has entries covering the boolean logic used to translate the sha256 equation into code and physical logic gates such as and/nand, or/nor, their various Xxx versions etc. The ASICboost whitepaper also covers it very well. There is also the Intel patent for their hardware implementation of it.

Going through miner code such as cgminer (most recent version here) and the others mentioned should give a good feel for preparing and presenting the shares to the ASIC chips and handling the results from the chips. That said, be aware that once data is being presented to/received from the chips it has to pass through a chip-specific driver. Along with the chip layout that driver is the one other bit that is usually closed source.

Thing is, cgminer has open source drivers for scads of chips though the most recent is for the S17 chip (and written by Kano) used in Sidehack's Compac-F stick miner.

[NotFuzzyWarm]

As @NotFuzzyWarm mentioned, the hardware is their secret sauce. Although it's also true many company don't share source code of firmware, OS and driver for their ASIC hardware.

Miner debug logs usually state what the hardware (usually a SoC) and OS (Linux) are.

As for the firmware the OS is running, Truth be told most if not all the miners run their closed fork of cgminer and yes that is in blatant violation of the cgminer GPL. Of the 2 primary developers of cgminer, -ck and Kano, -ck does not care. Kano does and yes accepts there is no choice but to allow all miners running OEM firmware to use his pool because at some level 90% of it is known cgminer code, the rest is the driver and gui dashboard. Only Canaan used to publish the entire source code but stopped that with the A10xx on up. As for the3rd party closed-source forks of OEM firmware - that goes too far and is auto banned on Kanopool.

[Sha256explorer]

As @NotFuzzyWarm mentioned, the hardware is their secret sauce. Although it's also true many company don't share source code of firmware, OS and driver for their ASIC hardware.

Miner debug logs usually state what the hardware (usually a SoC) and OS (Linux) are.

As for the firmware the OS is running, Truth be told most if not all the miners run their closed fork of cgminer and yes that is in blatant violation of the cgminer GPL. Of the 2 primary developers of cgminer, -ck and Kano, -ck does not care. Kano does and yes accepts there is no choice but to allow all miners running OEM firmware to use his pool because at some level 90% of it is known cgminer code, the rest is the driver and gui dashboard. Only Canaan used to publish the entire source code but stopped that with the A10xx on up. As for the3rd party closed-source forks of OEM firmware - that goes too far and is auto banned on Kanopool.

I'm sorry but I don't understand very well what you write: I'm not a native English speaker and I probably lack the necessary knowledge of mono and pools.

 I'll try to rewrite to see if I understand correctly.


 Kano and ck developed cgminer under the GPL license

 Many manufacturers have used their own modified version of cgiminer for their asics.

 Kano and ck have their own mining pool.

 Ck has no problems with any software on participating asics, but kano also accepts somewhat modified versions of cgminer.

 I understand?

[NotFuzzyWarm]

...

I'm sorry but I don't understand very well what you write: I'm not a native English speaker and I probably lack the necessary knowledge of mono and pools.

 I'll try to rewrite to see if I understand correctly.

Kano and ck developed cgminer under the GPL license

Correct

Many manufacturers have used their own modified version of cgiminer for their asics.

Also correct. The manufacturers created driver code so cgminer and their chips can talk to each other. Problem is - none of the manufacturers have published their code as the GPL requires. Even bigger issue is that several people/companies have taken the manufacturers code and further modified it and sell it to give the user better control over speed of the miners and the power used: This is the 3rd party code.  Because they also refuse to publish their modified code as the original open source GPL requires AND adding insult to injury are selling their modified firmware, Kano is pissed about it. (remember, most miner manufacturers are Chinese so filing lawsuits against the OEM's like Bitmain, Innosilicon, etc. would be very problematic)

Kano and ck have their own mining pool

. Correct. Kano's pool is https://kano.is
-ck's pool is http://solo.ckpool.org

Ck has no problems with any software on participating asics, but kano also accepts somewhat modified versions of cgminer.

Correct. -ck is a medical doctor - not a programmer. To him programming is a hobby and he stopped supporting cgminer many years ago. He runs his pool 'for fun' and does not care about the GPL violations.
Kano however is a programmer, it is his job. To run his pool he may have to accept the fact all OEM miners violate the GPL but he does NOT have to allow the far smaller numbers of miners running 3rd party code to run on his pool so his pool software automatically blocks them.

[Sha256explorer]

how is it possible to know the mining software a miner uses when joining a mining pool? Is it possible for the miner to write software that pretends to be the chosen software from the mining pool, without really being one?
and how does a mining pool check the hash power of an individual miner? Would it be possible for a miner to simulate a hash power that he doesn't have?

[ymgve2]

I assume pools don't care what you use to mine as long as you use something that supports whatever API they use.

And they verify your mining by giving you a less difficult target that you report back hits on, so the pool verifies your work in much the same way Bitcoin uses PoW.

[hZti]

how is it possible to know the mining software a miner uses when joining a mining pool? Is it possible for the miner to write software that pretends to be the chosen software from the mining pool, without really being one?
and how does a mining pool check the hash power of an individual miner? Would it be possible for a miner to simulate a hash power that he doesn't have?

As far as I have experienced it they do not actually know if you umsetze correct software, they just state on their website that you are not allowed to use specific software. At least I have already successfully mined with „forbidden“ software at Kano pool.
You can just simply ask Kano or ck here on the forum if you are interested in the details.

[NotFuzzyWarm]

As far as I have experienced it they do not actually know if you umsetze correct software, they just state on their website that you are not allowed to use specific software. At least I have already successfully mined with „forbidden“ software at Kano pool.
You can just simply ask Kano or ck here on the forum if you are interested in the details.

Most pools do not care and therefore do not check what the firmware is other than if it supports ASICboost and in the case of Braiins and a couple others does it support the xnsub (extra nonce) switch. At least 2 pools - Kano's and Braiins (Slush) do check deeper. Kano does it to block 3rd party hacked cgminer firmware, Braiins does it to set what their pool fee is and/or control the DevFee function built into their Braiins miner firmware.

When a miner 1st connects with a pool the miner firmware *does* have a fingerprint that can be used to tell what it is.
As for

At least I have already successfully mined with „forbidden“ software at Kano pool.

I have to assume you used Braiins? If so, Braiins is not hacked cgminer, it is clean code. Kano initially only allowed it on the Solo until he verified that it can return expected high diff shares. Once that was verified Braiins is now allowed on the main pool as well. As for the hacked cgminer ones AFAIK his pool code is very very good at spotting it  

Also, getting waaayyy off topic here...

[Sha256explorer]

Also, getting waaayyy off topic here...

my bad, i'm sorry. I got caught up in the curiosity of what is the relationship between mining device and mining pool.
however I have a plan.
- write a mining software test, which operates on fake headers, for gpu (cuda)
- run it on a monster cuda gpu, which I can rent for a few euros an hour
- I will post the results of this test here

[alecfisker]

theoretically could be possible to make optimised CPU coin
coins that does not keep all transaction history and take only number of kilobytes in memory already there
text me for code samples

[Sha256explorer]

theoretically could be possible to make optimised CPU coin
coins that does not keep all transaction history and take only number of kilobytes in memory already there
text me for code samples

But you're not talking about Bitcoin are you?  because mining Bitcoin without Asics is meaningless.  what I think I can do is improve the performance of the Asics by 30 or 40%.  it would be a sensational result but in any case the asic would always be absolutely necessary

[Sha256explorer]

the hypothesis that everyone, including myself, considered most probable has come true: I am not able to make any improvements to the mining algorithm.  amen.

[paid2]

the hypothesis that everyone, including myself, considered most probable has come true: I am not able to make any improvements to the mining algorithm.  amen.

Yes, it was a foregone conclusion, to quote NotFuzzyWarm's first response:

I should also add that despite after more than 10 years of some very bright minds working on this <drum roll please> there are no faster/better ways to process sha 256 hashes. Doesn't mean there isn't some as yet undiscovered methods to do it but - it also means it is pretty bloody unlikely.

Certainly companies investing tens of millions of dollars in this area have not missed out on a potential 30-40% process improvement.

But the important thing is that you have been able to form your own opinion through experience, that you have tried, and tested. This is very good for you and always interesting to follow.

I enjoyed reading your topic, and wish you all the best for the future @Sha256explorer