Title: why cgminer increases "nonce2" and "ntime" every 5 seconds ? Post by: citb0in on November 05, 2022, 05:27:18 PM Hello everbody,
i have a few questions about the mining process related to cgminer. I did read this (https://en.bitcoin.it/wiki/Block_hashing_algorithm) and that (https://en.bitcoin.it/wiki/Hashcash). Following is an excerpt example of cgminer mining on mainnet (against bitcoind @localhost) with a GekkoScience Compac F USB miner at an average hash rate of 387 GH/s. I have shortened/masked the long values for clarity. Code: [2022-11-05 16:19:28.299] Work nonce2 1257 ntime 63667efb The nonce range is 0 to 2^32-1 (0-4294967295) which we could express simply as 4.3G If I am not wrong, this USB miner at the hash rate of 387 GH/s should be able to run the whole nonce range in 1/90th second. Or in other words: it should be able to run 90x per second through the whole nonce range. So far correct ? If I understand it correctly, then the process looks like this with the respective hash attempts after a block header is formed: - the entire nonce range is tested by starting at nonce value and incrementing by one to the upper limit of 4294967295 - if no hash could be calculated which was below the target, then nonce2 (also called extranonce) can be changed. This is arbitrary random data which changes the coinbase transaction slightly (new Merkle root). Now we have millions of additional tries to find a good hash. If even that did not lead to success.... - then you can also vary with the ntime. This is the integer timestamp in seconds. Every second ntime is increased by one. Ntime can be changed locally but the limits are fixed here, you can't go back or forward arbitrarily, because that would not be accepted by the chain. Somewhere I had read that plus minus 10min would be allowed, is that true? I was not able to find it quickly by searching the net. Please refer me to the appropriate page so I can read through the limits and the detailed process for ntime. In the example shown, you can clearly see how nonce2 is incrementally increased while ntime remains constant. But what I don't understand is: after exactly 5 seconds ntime is increased, although you could actually change nonce2 further. Where does this come from, why is this done? I have searched through the source code cgminer.c, but I can't find any reference to these 5 seconds that I could find in the debug log as shown in the example above. In the source code there are definitions where ntime rolling can be set to 60 seconds or something like that, but no trace of 5 seconds. Can someone explain to me what this is all about and where in the source code these 5 seconds were set. Why is a new ntime being pulled in every 5 seconds? Thanks in advance. Title: Re: why cgminer increases "ntime" every 5 seconds ? Post by: Snowy pear on November 05, 2022, 07:08:49 PM Can someone explain to me what this is all about and where in the source code these 5 seconds were set. Why is a new ntime being pulled in every 5 seconds? Thanks in advance. You can adjust the ntime binary data in submit_noffset_nonce or resubmit to be able to pull the stratum split every 5 seconds to 2 minutes.You can see here https://bitcointalk.org/index.php?topic=28402.0, for more clarity. Code: The output line shows the following: Title: Re: why cgminer increases "ntime" every 5 seconds ? Post by: citb0in on November 05, 2022, 07:45:37 PM Hi Snowy Pear,
thank you for creating an extra new account to reply to me. I looked into submit_noffset_nonce, but I just couldn't manage to change ntime variation to a different value every 5 seconds or turn it off briefly for testing. I even tried test turning off all of the following functions by commenting those functions: modify_ntime, roll_work, roll_work_ntime and everywhere else where noffset occurs (including the references to it, of course). After a "make" and restarting the program in debug mode, the ntime is still incremented every 5 seconds. I don't understand your quoting. I hope you don't confuse my question with the 5 sec which is the default setting for page rendering? Because cgminer automatically refreshes the status page every 5 seconds. But that's not my point at all. I related my question exclusively to the counting up ntime. Quote [2022-11-05 16:19:28.314] Work nonce2 0 ntime 63667f00 [2022-11-05 16:19:33.319] Work nonce2 0 ntime 63667f05 Where does it get this info, if I have disabled everything in cgminer.c regarding noffset? And why 5 seconds, where does it get this value = 5sec ? Title: Re: why cgminer increases "ntime" every 5 seconds ? Post by: n0nce on November 05, 2022, 10:41:23 PM NTime is just a unix timestamp.
You can take the 63667f00 or 63667f05 ntimes, convert from hex to decimal and put it into https://www.unixtimestamp.com/ to check what dates those refer to. Code: $ python3 -c 'print(int("63667f00", 16))' Quote from: https://www.unixtimestamp.com/ GMT Sat Nov 05 2022 15:19:28 GMT+0000 Code: $ python3 -c 'print(int("63667f05", 16))' Quote from: https://www.unixtimestamp.com/ GMT Sat Nov 05 2022 15:19:33 GMT+0000 I suppose cgminer creates a new block template with updated timestamp only once every 5 seconds to reduce overhead; especially since the timestamp in Bitcoin's block header doesn't have to be very accurate, anyway. Title: [SOLVED] Re: why cgminer increases "ntime" every 5 seconds ? Post by: citb0in on November 06, 2022, 06:39:49 AM I know that it represents the timestamp in UNIX time format. But I would like to understand where in the program code the 5 seconds are pulled from. I can't find the corresponding place in cgminer.c
EDIT: Meanwhile while exploring the debug log, I have recognized that not only ntime is being flipped each 5 seconds but also the incrementing nonce2 is being reset after the 5 secs. Where does the 5 seconds come from ? any help appreciated. EDIT2: SOLVED Found it! It comes from block Code: static void *watchpool_thread(void __maybe_unused *userdata) in line 9364 Code: cgsleep_ms_r(&cgt, 5000); Title: Re: [SOLVED] Re: why cgminer increases "ntime" every 5 seconds ? Post by: n0nce on November 06, 2022, 11:37:35 PM Meanwhile while exploring the debug log, I have recognized that not only ntime is being flipped each 5 seconds but also the incrementing nonce2 is being reset after the 5 secs. Yup, it makes sense that the nonce is reset once you have a new block template with updated timestamp, right? ;)Where does the 5 seconds come from ? any help appreciated. I'm happy you found the code segment, though; but it leaves me curious what you need it for. Are you trying to mess with the 'reset timeout'? If so, why? Title: Re: why cgminer increases "nonce2" and "ntime" every 5 seconds ? Post by: citb0in on November 07, 2022, 06:21:06 AM Hi n0nce,
it is just for education purposes, trying to understand and fiddling in the code. Title: Re: why cgminer increases "nonce2" and "ntime" every 5 seconds ? Post by: jack1cryptotalk007 on November 07, 2022, 01:27:17 PM In cgminer, any new block or any update of cuurent block will create a new "ntime" when block information is received. About 60 second, block update information is coming from pool.
see parse_notify() function. In cgminer, nonce2 is used to create more local block header data for more hashing and is used for generating new merkleroot in local header data. it changes very quick. see gen_stratum_work or gen_solo_work funct. It is used in submit work to pool and pool uses it to re-generate block header for verifying if block header is resolved. Title: Re: why cgminer increases "nonce2" and "ntime" every 5 seconds ? Post by: citb0in on November 07, 2022, 01:32:46 PM Hi jack1cryptotalk007,
yes, correct. I am working on gen_solo_work because I'm not using a pool, I'm mining on my own bitcoind node. I'm aware of the reason nonce and nonce2 are used. It is very well explained here (https://en.bitcoin.it/wiki/Block_hashing_algorithm) and here (https://en.bitcoin.it/wiki/BIP_0023#Mutations). Quote Bitcoin stores the nonce in the extraNonce field which is part of the coinbase transaction, which is stored as the left most leaf node in the merkle tree (the coinbase is the special first transaction in the block). The counter parameter is small at 32-bits so each time it wraps the extraNonce field must be incremented (or otherwise changed) to avoid repeating work. The basics of the hashcash algorithm are quite easy to understand and it is described in more detail here. When mining bitcoin, the hashcash algorithm repeatedly hashes the block header while incrementing the counter & extraNonce fields. Incrementing the extraNonce field entails recomputing the merkle tree, as the coinbase transaction is the left most leaf node. The block is also occasionally updated as you are working on it. [...] The nonce will usually be different, but it increases in a strictly linear way. "Nonce" starts at 0 and is incremented for each hash. Whenever Nonce overflows (which it does frequently), the extraNonce portion of the generation transaction is incremented, which changes the Merkle root. First of all, this is wrong: the nonce2 range which is from 0-18446744073709551615 Sure? I am talking about the nonce2 (extranonce in coinbase) and not the original nonce. If nonce2 would be incremented, where would be the upper limit of the range ? When I decrement nonce2 in cgminer it starts at 18446744073709551615 so that's why I think it is the upper limit of nonce2. Note: Not to confuse with Extranonce1 vs. Extranonce2 (https://bitcoin.stackexchange.com/questions/72154/what-is-extranonce2) From your other topic, it appears to me that you just removed the timer (or set it to a very long number) that resets the nonce to 0 and creates a new block template with a new timestamp. Is that what you're doing? Nope. That is what cgminer seems to do by default. After the 5sec (default) new ntime is generated, nonces are resetted to zero and the process is started over.Title: Re: why cgminer increases "nonce2" and "ntime" every 5 seconds ? Post by: n0nce on November 07, 2022, 08:35:40 PM First of all, this is wrong: Sure? I am talking about the nonce2the nonce2 range which is from 0-18446744073709551615 From your other topic, it appears to me that you just removed the timer (or set it to a very long number) that resets the nonce to 0 and creates a new block template with a new timestamp. Is that what you're doing? Nope. That is what cgminer seems to do by default. After the 5sec (default) new ntime is generated, nonces are resetted to zero and the process is started over.Quote Started with nonce2=0 [2022-11-06 10:31:03.924] Work nonce2 0 ntime 63677a79 Ended with nonce2=15478 [2022-11-06 10:36:03.922] Work nonce2 15478 ntime 63677a79 Now, what about my suggestion that the variable may be wrapping around? That's something to consider when working with large numbers / too large numbers for the variable types in use. Title: Re: why cgminer increases "nonce2" and "ntime" every 5 seconds ? Post by: citb0in on November 07, 2022, 09:05:03 PM I am trying to understand and see live what exactly happens in the mining process. Yes, I did "freeze" ntime for that educational purpose in the example above. I did not freeze nonce2 but I increased the timer from default to 5sec to testwise 15sec, just to see if my changes took effect. So far so good, however I am still far away from my goal.
What I like to see is: - all the nonces that cgminer sends to the hardware for work, I like to see how this process is threaded and queued by cgminer - the exact shares that are generated and sent to the hardware - the exact nonce, nonce2 and ntime value that were used for each particular hash step and the result for it Generally spoken, I'm looking to more infos that I miss with the --debug option of cgminer. Currently I'm stuck even trying to display simply the nonce values and the particular hash result of one single step. I am aware that this could output very high amount of lines and generate a log with huge size, but I'm interested in one complete cycle through this: - ntime should be freezed at all, irrelevant for me, same for merkle root - cycling through the whole nonce range and when nonce reached the upper limit the nonce should freeze its value or set to zero and freeze its value (no more incrementing) - then it should cycle through the whole nonce2 range (nonce is fixed at 0) - when nonce2 is exhausted cgminer should exit I wanna see ALL the hash results in a convenient output view (e.g table), so I can scroll through the log file and see each single step including the particular values for nonce and nonce2 and what hashed they generated. Code: timestamp nonce nonce2 hash how ? Title: Re: why cgminer increases "nonce2" and "ntime" every 5 seconds ? Post by: n0nce on November 07, 2022, 09:26:53 PM how ? Add debug prints! :DSomething like this: Code: applog(LOG_DEBUG, "Hello World!"); Then pipe stdout to a file. If that's not good enough, you will need to open a file and write to it instead, but that will be slightly more complicated. Title: Re: why cgminer increases "nonce2" and "ntime" every 5 seconds ? Post by: citb0in on November 07, 2022, 09:29:53 PM I did so :) But as I said, I am even struggling in getting the nonce value displayed.
Code: if (opt_debug) { I have changed the status bar to see which nonce currently was processed but I get "0" displayed all the time for the nonce. I also tried %u which will give similar error. I am not a developer, can only read some code. I don't think that the format %"PRIu32" or %u is the cause for the issue getting nonce=0 displayed. But I rather suspect that the reason is that in this code section the nonce was not defined at all or not intended to be used. I think that the nonce far before has been sent to the mining hardware as (scheduled/queed) work. Therefore the display of 0 in the output, kinda ? Title: Re: why cgminer increases "nonce2" and "ntime" every 5 seconds ? Post by: n0nce on November 07, 2022, 09:43:29 PM I have changed the status bar to see which nonce currently was processed but I get "0" displayed all the time for the nonce. I also tried %u which will give similar error. To be honest, I haven't looked that deep into cgminer yet; if nonce is always 0, I suppose it's possible that it only rotates through nonce2 instead - not sure. Maybe we can summon @kano; he knows for sure.Title: Re: why cgminer increases "nonce2" and "ntime" every 5 seconds ? Post by: jack1cryptotalk007 on November 08, 2022, 11:54:39 AM After I read your posts, there are missing points.
1. The nonce2 is only used to create next new local block header data. When a new block header come in, nonce2 is initialzed to 0 in cgminer, e.g. parse_notify() or gbt_decoode(). The nonce2 is 64 bits integer (very large integer). The nonce2 only affects merkleroot value in local block header data. Huge numbers of local block header data are created and are given to ASIC chips for chips to find out some nonces which produce a hash value with last 8 digits 0. hash value: xxxx....x????00000000. 2. cgminer does not cycle nonce from 0 to 4.29G. It just sends a lot of block header data to ASIC chips. ASIC chips or hardware do cycle nonces by digital chips and frequency pulse to check and find qualified nonces. ASIC hardware do filter out qualified nonces. So, ASIC hash header data super fast. 3. Qualified or found nonces, then, are read by cgminer from ASIC chips. Then, cgminer verify those nonces with the pool's difficulty. Then, good nonces are sent to the pool for further verification by pool with network's difficulty. Title: Re: why cgminer increases "nonce2" and "ntime" every 5 seconds ? Post by: citb0in on November 08, 2022, 12:18:51 PM Very well explained, thank you.
1. The nonce2 is only used to create next new local block header data. When a new block header come in, nonce2 is initialzed to 0 in cgminer, e.g. parse_notify() or gbt_decoode(). The nonce2 is 64 bits integer (very large integer). The nonce2 only affects merkleroot value in local block header data. Huge numbers of local block header data are created and are given to ASIC chips for chips to find out some nonces which produce a hash value with last 8 digits 0. Let's assume nonce2 would be also an unsigned 32 bit integer and the ASIC miner very powerful with a superior hash rate. What would happen when nonce2 range is exhausted before a new block header arrived ?hash value: xxxx....x????00000000. 2. cgminer does not cycle nonce from 0 to 4.29G. It just sends a lot of block header data to ASIC chips. ASIC chips or hardware do cycle nonces by digital chips and frequency pulse to check and find qualified nonces. ASIC hardware do filter out qualified nonces. So, ASIC hash header data super fast. Does that mean that cgminer has no control at all about the current nonce which should be tested? As cgminer sends a block header to the ASIC miner, the ASIC hardware would need to know how such a block header is constructed and to derive the part where the nonce is located, wouldn't it? Otherwise the ASIC miner would only see a very long string (blockheader) without knowing which bits to change for trying for a valid hashes.3. Qualified or found nonces, then, are read by cgminer from ASIC chips. Then, cgminer verify those nonces with the pool's difficulty. Then, good nonces are sent to the pool for further verification by pool with network's difficulty. So cgminer only sends a block header to the ASIC hardware and the ASIC miner itself cycles through the 32bit nonce range? Does the ASIC miner use a random cycle or incrementing the values?Title: Re: why cgminer increases "nonce2" and "ntime" every 5 seconds ? Post by: Skot on November 09, 2022, 02:47:37 AM 2. cgminer does not cycle nonce from 0 to 4.29G. It just sends a lot of block header data to ASIC chips. ASIC chips or hardware do cycle nonces by digital chips and frequency pulse to check and find qualified nonces. ASIC hardware do filter out qualified nonces. So, ASIC hash header data super fast. Does that mean that cgminer has no control at all about the current nonce which should be tested? As cgminer sends a block header to the ASIC miner, the ASIC hardware would need to know how such a block header is constructed and to derive the part where the nonce is located, wouldn't it? Otherwise the ASIC miner would only see a very long string (blockheader) without knowing which bits to change for trying for a valid hashes.You don't really want cgminer, which runs on a relatively slow CPU to have to increment the nonce and communicate it to ASIC. That's happening much too fast. Chips like the Bitmain BM1387 or the BM1397 (like in your Compac F) have a set of address pins. If I understand this right, the address pins are used when you have several chips in a chain they can each be set to work on a different nonce range, so as not to overlap. cgminer doesn't exactly send a block header to the mining chip. It's like a partially computed hash of the block called a midstate. Kano has a description of this here; https://kano.is/BMMMP63278.pdf. All of the bytes are sent in fixed positions to the mining chip, so it knows what goes where. Things start to get pretty crazy (to me) when cgminer sends multiple midstates to the ASIC, which is called ASICboost. Title: Re: why cgminer increases "nonce2" and "ntime" every 5 seconds ? Post by: citb0in on November 09, 2022, 06:19:45 AM Thanks for this great explanation and linking to the document.
Does this explain why nonce2 traversal is so slow in comparison to nonce, because nonce2 cycling is kept in CPU ? Would it be possible to send nonce2 to the ASIC chip just like as with nonce so the traversal would take place on the much faster ASIC chip? Just as an example: - set nonce to a fixed value e.g. 123 - generate midstate work for nonce2 (instead of nonce) - let the ASIC chip traverse through 2^64-1 cycles for nonce2 and have the hash reported back to cgminer How can that be done ? Title: Re: why cgminer increases "nonce2" and "ntime" every 5 seconds ? Post by: kano on November 22, 2022, 01:21:17 PM Alas there seems to be a lot of misunderstanding reading the code that says exactly what it does.
While the code shows it pretty straight forwardly, if you read it, what mining 101 on an asic does is: Receive work from the pool every 30 seconds (that's up to the pool, but 30 seconds is optimal) During that 30 seconds, that single pool work is used over and over again to generate work for the asics. How that 'over and over' is done is by modifying 'nonce2' to generate a unique merkleroot to generating a unique block header. Then hashing that block header doing the first half of the double sha256 then passing that to the asics with a job id. 'nonce' has nothing to do with this part, cgminer has nothing to do with the nonce ranges. A CPU core can hash about 30 million times a second, so generating 10's or 100's or even thousands of work items isn't that hard to do quickly. Each work it generates, will give the asics up to 4billion hashes to do, so as long as the asics aren't 4billion times faster than the time it takes cgminer to produce a single work item, all is good. N.B. a single work item is not one hash, read the code if you want to work out what it does. Using the work sent to them, the asics run the nonce value some % of the way from 0 to 4billion doing the 2nd half of the double sha256 for each nonce value, and if they find a hash with a difficulty of 1 or more, they return the job id and the nonce value that produced the hash. cgminer then tests this returned data with the work job it should match, and sends it to the pool if the difficulty meets the pool's requirements. It does not matter what the asic nonce range is. The only point of interest is for the asics to test a high % of the range to minimise work I/O to the asic. (hashing a nonce range in CPU would be beyond ridiculously slow, millions of times slower than a current asic, so no one would be silly enough to do that ...) That's the basic 101 version of it. Multi midstates, asicboost, and fpga work generation (like in an avalon) requires different effort, to effectively get similar results. Title: Re: why cgminer increases "nonce2" and "ntime" every 5 seconds ? Post by: yoshimitsu777 on November 23, 2022, 09:10:20 AM 'nonce' has nothing to do with this part, cgminer has nothing to do with the nonce ranges. if cgminer has absolutely nothing to do with the nonce then who does? does the hardware miner have a routine to run through the nonce range - is it triggered by cgminer in some way? weird Title: Re: why cgminer increases "nonce2" and "ntime" every 5 seconds ? Post by: Skot on November 23, 2022, 01:53:46 PM if cgminer has absolutely nothing to do with the nonce then who does? does the hardware miner have a routine to run through the nonce range - is it triggered by cgminer in some way? weird The mining chip (ASIC) increments the nonce after each hash. Hundreds of billions of times a second. AFAIK it starts at 0 and goes up to 2^32 unless you configure it to start somewhere else. This takes less than a second. It’s kinda crazy to think about; the fastest CPUs out there wouldn’t even be able to increment the nonce as fast as an ASIC, much less do any actual hashing. Title: Re: why cgminer increases "nonce2" and "ntime" every 5 seconds ? Post by: NotFuzzyWarm on November 23, 2022, 02:44:09 PM Quote It’s kinda crazy to think about; the fastest CPUs out there wouldn’t even be able to increment the nonce as fast as an ASIC, much less do any actual hashing Huge difference between any CPU/GPU and a mining ASIC -- the number of cores in them. A top-end CPU might have what, 48 cores in it?The s17 mining ASIC has 8192 cores in the 1 chip and each core handles 1 part of the range being ran through so with each clock cycle a block of up to 8192 values are processed at 1 time. Add in the fact that most large miners will have 100-200 ASICs in them and that's how they achieve such incredible performance. It should be noted that it is rare for all of the cores to be usable and often a chip will have many dead ones. That is part of the difference between the fastest (and most expensive) miners and their slightly slower and cheaper brethren - the fastest/most expensive are using chips that are binned (selected) as most functional cores and those make up a rather small % of usable dies from each wafer. The lower number of perfect/near-perfect ones allows them to be priced at a premium. Title: Re: why cgminer increases "nonce2" and "ntime" every 5 seconds ? Post by: kano on November 23, 2022, 10:46:33 PM 'nonce' has nothing to do with this part, cgminer has nothing to do with the nonce ranges. if cgminer has absolutely nothing to do with the nonce then who does? does the hardware miner have a routine to run through the nonce range - is it triggered by cgminer in some way? weird A) As Fuzzy mentioned above B) How work is queued in the physical miner. b1) You can't always divide 2^32 evenly with the number of chips and the number of cores. So some values may not be used because of this. b2) If we are talking about a small miner, there may be no work queue inside the hardware, so the miner must have it's work refreshed by cgminer, before it reaches 2^32, so it doesn't repeat doing the same work again from the start. Thus. timing here will mean that all sub-ranges that are divided up among the chips, would rarely if ever complete. Be aware of the obvious fact that you don't need to mine the full range. It makes absolutely no statistical difference if you stop a range before it completes and start again from the bottom with new work. As I mentioned It does not matter what the asic nonce range is. The only point of interest is for the asics to test a high % of the range to minimise work I/O to the asic. Title: Re: why cgminer increases "nonce2" and "ntime" every 5 seconds ? Post by: sidehack on November 25, 2022, 12:38:03 AM Fuzzy, the S17 chip has 672 cores. Interestingly, the chip actually has four blocks of 168 cores each built in series within the die.
Title: Re: why cgminer increases "nonce2" and "ntime" every 5 seconds ? Post by: Skot on November 25, 2022, 01:24:30 AM Fuzzy, the S17 chip has 672 cores. Interestingly, the chip actually has four blocks of 168 cores each built in series within the die. That would make sense then how it can do hundreds of GH/s with a clock of only hundreds of MHz Title: Re: why cgminer increases "nonce2" and "ntime" every 5 seconds ? Post by: NotFuzzyWarm on November 25, 2022, 02:42:32 AM Fuzzy, the S17 chip has 672 cores. Interestingly, the chip actually has four blocks of 168 cores each built in series within the die. Interesting - I'd swear I've come across infor with it being over 8k of them. The Bitfury's in the Apollo has 4k of them per Bitfury's spec and the A1 (2014) had 1024 cores. Most odd.Title: Re: why cgminer increases "nonce2" and "ntime" every 5 seconds ? Post by: sidehack on November 25, 2022, 02:54:06 AM Bitfury are rolled cores, so multiple clocks per calculation. Bitmain's always used unrolled cores, which take up a lot more die space but do the whole hash in a single cycle. BM1384 has 55 cores, 1387 has 114 cores.
|