There are group elements, these are the points on the curve, a finite abelian group of order N where the discrete log problem is assumed hard.

There are field elements, which are values in the field F(P) which the curve above is constructed over. These are used inside the group code but are mostly not seen directly by the signature algorithm.

Then there are scalars which are ordinary numbers mod N (same order as the group), which are used to multiply group elements; these are most of the objects used in signing.

In ECDSA the X coordinate (which is a F(P) field element) of a point R  (R =kG) is coerced into a scalar ( r = R.xy()[0] mod N)-- which makes keeping the types straight harder.

(A common but not universal convention is to make point variables upper case and scalars lower case)


If you're just trying to understand signatures for your personal enrichment; schnorr is easier to understand.

HAH.  You seem to forget even your own posts so quickly "(except slighty off at the very end of the range)".  And I provided specific, concrete examples of the very software and hardware that produces those effects.

I also countered your points _assuming_ uniformity, only noting that the distribution was known to be non-uniform and pointing out why; to point out that this assumption was known to be approximate. You might note that I never attempted to characterize the actual non-uniformity, and all the examples I gave were based on uniform numbers.

I note, several more days have passed and no more blocks have again matched your criteria--- I was sad to see that you declined to further discuss the "optimization" you discussed after I pointed out that midstate handling is universally used in all mining systems and has for the life of the system.

Funny, I guess contradicting yourself about uniformity and insulting the knowledge of others must not be comments.

Bringing me to why I actually came here to post:


Actually this was already done a long time ago;
https://www.wpsoftware.net/andrew/crackpot.html


 

The network difficulty can not go below below a minimum level (Described as "1" these days... I.e. "the minimum", all normal difficulty numbers are expressed as multiples of this minimum value), so due to lack of sufficient hashpower the network under-produced during its first year or so.

No; the commitment included in the block is a hashtree with leaves that look like:

{index for how many back, previous_block_hash, compact encoded work difference between now and then}
...

For multiple backpointers. A back pointer is only valid (considered acceptable for traversal in a proof) if the block committing to it meets the work-difference threshold.  Similar to merge mining, it's possible to perform N-hashcashes in parallel all with different thresholds. Just like if namecoin's difficulty were higher than Bitcoin you could mergemine namecoin and have a bitcoin block with a namecoin commitment in it which is still useless for namecoin though its a valid bitcoin block.

So to follow a link that goes one block back, you need bits work-- this is the normal case.

To go follow a link that jumps two back your block must be valid at the 2*bits level... and so on, to successfully mine a 10000 back hop you need to have a block that is valid when considered with a work threshold of whatever was spanned by those blocks (e.g. 10000x greater if difficulty has been constant).

This is why the proof can hide data, but you can't fake one with less work (on average) than just mining normally.  Individually any block is unlikely to jump far back, but on the aggregate long jumps happen and as a result the proof is log sized in the length of the chain. (Actually less when the hashrate increases constantly).

For a given set of commitments there are multiple possible paths, and so multiple proofs for the same chain possible-- if you could have gone 100 back you could also go 50 back. Sometimes going less back makes for a smaller proof, perhaps at 51 back there is a single hop that goes all the way. The smallest proof can be constructed in linear time via dynamic programming: Start with the block you're trying to reach with your proof (e.g. genesis), and scan forward, for each block keep track of the size of the best proof from it to the destination and the first hop back, when you consider a block check each of its valid backlinks, add the cost of taking that backlink to the cost of its destinations best. Take the minimum as your best and move to the next block.  When you get to the end walk backwords. This algorithm always produces the smallest possible proof, though a simple greedy algorithm (always jump as far as you can but not past your destination) isn't usually _that_ much larger.

Right, exactly-- well to be more precise, the compact proof and normal are both gamma; but the compact proof takes the log of the shape, while increasing lambda--- the same expected work, is represented by a fewer number of more rare events such that the expectation is the same but the distribution shape is different.

Technically its possible to make the proofs "over prove", e.g. to go X-work back you have to show a*X expected total work; but I think I concluded that you can only get constant factor improvements that way unless you break the asymptotic log scaling of the proofs. One nice thing is that such techniques are between the prover and verifier, the commitments don't change.

Old forks must be obtained from a source you trust-- just just to be honest but to have competently curated their collection; with low difficulty its now easy for someone to forge a circa 2011 or circa 2012 fork; and various people have spammed the network with faux old blocks in the past.  If you actually care about the integrity of whatever you're looking at you should want to be sure that whatever process collected the blocks can actually attest to their timeliness.

There were many short forks (e.g. 3-5 blocks) related to p2sh, but no one with an updated node would have recorded them; since they were rejected right out.  Being online is not sufficient, you had to have accepted the block before you would have committed it to disk so if you never thought the now-extinct branch was best (which requires validity), you wouldn't have it.

Getchaintips is the right way to explore forks with current software.

Not quite; you commit to the work you're targeting... so you can't just find something and replace as much as you can; you have to set out in advance what amount of work you're claiming; and the link does nothing unless you hit it. If you try, but aren't successful, you've just wasted your effort.  In other words with the same amount of hashing power on average you could make a false compact proof that replaces the chain, or you could replace the whole chain normally; they take the same expected amount of work exactly.   But if you didn't have that much hashing power... only enough to, say, replace 6 months; then maybe you could get lucky and successfully create a compact proof for the whole chain but with, say, a 0.01% probability of success while the probability of success for just getting lucky directly mining the blocks would be 0.0000001% probability of success; just due the the reduced number of samples.  

(Though note, you can boost your reorg success probability even without the compact proofs by mining so that the difficulty goes up at the highest possible rate (e.g. 4x every retarget)... and _if_ you assume that everyone's hashrate exponentially goes up forever; then any fixed proportion of the hashrate (e.g. 0.1%) will in finite, but potentially very large, time eventually replace the whole chain just due to that 4x ramping variance attack: eventually you have a chain with similar work but much fewer samples; and so you can get lucky)

As far as the historical work:  http://bitcoin.sipa.be/powdays-50k.png  this is how much time at the current hashrate that it would take to replace the work in the chain.

Can you email me the debug.log? greg@xiph.org


Your description is a bit unclear.   So you have node A. Node A's debug.log said it say the block at 17:05? What log message are you referring to?  Then getinfo didn't show it until later?


One reason there can be a delay is that if the party you requested the block from is super slow or unresponsive, the block may not transfer until that node times out.

You've just described the two way peg mechanism which is what people are usually talking about w/ sidechains-- there are a great many details that have to be added and gotten right; but that is the basic notion: Coins get assigned to a smart contract that lets them move according to some external system.

Much of the complexity arises because the consensus system cannot directly observe anything external (since its whole state must be subject to the consensus). Have you seen the sidechains paper?

You accuse me of divagating but here you chase the optimization tangent which to say you've dropped at the expense of completely ignoring my argument that your own overfit criteria actually precludes your hypothesis that something "out of normal going on"; since exclusion of your training sample from the test observations results in no hits against your proposed criteria.


And I thought I answered it adequately, I would only find behavior rare if it was surprising in light of a model that wasn't already fit to the self-same data.

When the behavior can also be explained by "yes, these numbers could have arisen by chance", then I do expect to have a model before I conclude that they're actually surprising.


I didn't say it was the only way to do so, it was a suggested example idealized approach; offered with the hope of being constructive and not merely repeating that your reasoning was defective without offering an alternative approach.

Mercury's precession isn't noticeable or interesting at all without some kind of model. If your theory of the solar system is that the planets just wander around, any sequence of moves doesn't seem especially surprising.



Awesome!  So we can finally talk about something specific and technical!

You're talking about using the midstate, as it's normally called;   hashing a block header involves running the SHA256 compression function 3 times. One to ingest the first part, leaving the 64 byte midstate, one to digest the end, and then a final one on the output.  Because the nonce is at the end of the header one can perform 4 billion runs of the latter two compression runs without repeating the first.  The midstate optimization was mentioned in the second page of this thread.  It's an optimization exists in every miner, it was used in the original Bitcoin software while CPU mining; it's enshrined into the getwork protocol (the original remote mining protocol); it's established in the publicly documented interfaces of mining asics-- they are usually setup to receive midstates, not headers from their control systems.  It is effectively impossible to write miner software without being comfortable working with a midstate plus a tail.  P2Pool even uses this characteristic of the merkle damgard design to reduce its communications costs (it needs to prove a piece of data with a particular hash ends with a particular suffix, so it communicates just the midstate and the final piece).

As an aside, the merkel root cannot be kept static across multiple blocks, it must be recomputed every block, even if you are mining a one transaction block, because the block height is required in the coinbase transaction; and of course changing that or the extranonce or anything else in the block changes the merkel root completely.  If you were able to keep it static, it would mean creating a duplicate coinbase transaction; which is prohibited by the network; but if it weren't prohibited it would destroy the prior one and deprive you of your newly mined Bitcoins from it.

I will not. This is now unrelated to the point you made above: The computation to generate the midstate is generally insignificant.  My laptop can generate something on the order of 40 million midstates per second, with the factor of ~4 billion increase from nonce rolling, this one laptop could support midstate generation for nearly the entire network; if anyone worried too much about midstate generation speed, they'd move it onto an ASIC (or just a FPGA); and a single part could easily have 10 or 100 times what my laptop can generate;-- since midstate generation work is 1:2^33 that of the mining in total (assuming the whole nonce range is used, but you can see that taking a factor of 2 or 4 here or there doesn't change things much-- and thus my point before about miners not using the whole nonce range; there is no need to because midstate generation is still cheap).

Generally by withholding blocks you disadvantage yourself because you will lose a race (the network prefers the first in the face of ties, specifically to avoid incentives to delay blocks); without an information advantage (e.g. MITM other participants) you must have a very significant total fraction of the hashrate before any delay of your announcements is not a spectacular loss. This has been studied to some extent in the literature; the most optimistic simulations which assume no latency show in the no information advantage has no amount of withholding avoids being a loss until over the 1/3rd of total hashrate case.


It seems nothing can disprove your belief.  You say it would be the same miner producing the blocks but when presented with evidence that it is different miners, you argue that he would conceal it. Why did we even look to apparently non-anonymous miners in the first place?   While doing so, you ignore strong evidence against your theory:  That at least two people independently observed those blocks arriving on the network minutes apart; not in a bunch as your block withholding model would predict (and not as you initially believed them to be).

If you were willing to argue that the blocks arriving at the same time, when you were mislead by the data on BC.i, was strong evidence; you ought to be willing to admit that when you find they arrived without bunching that this undermines your theory.

Well... I know from past experience that encountering events with probability under 1:2^90, without my expectation being fixed by the data in advance, when I expected a random outcome triggered in me fairly strong confidence that something structured was happening and spent time searching for an answer; but I still did not exclude the possibility that it could be a coincidence.

This is all irrelevant because you're ignoring the model complexity and that is an absolutely critical term which cannot be discounted: I would not be surprised by a complex model I just fit against the data that I'm testing it on. It will always match. If I were to be surprised by this I would be making an error in judgement. I have made similar errors in judgement in the past but I would be unlikely to do so now unless I was tired or ill.

Your comment here of "1% starts to trigger my interest, and light a red flag" makes me laugh out loud. There are 144 blocks in a day (and thus 144 overlapping three block windows).  If you really have your attention caught by 1% events then you would be constantly exhausted by them occurring almost every day, often multiple times.  I am imagining a movie conspiracy theorist with a bunch of sticky notes on a wall and string run between them.  "ITS ALL CONNECTED", ... sorry.

Please don't waste the forums time with observations at the once a day level; none of us have the patience for it.  If you really want to work yourself up over some criteria that will happen constantly by chance, in private-- thats your own decision, but please don't inflict it on the rest of us.


I think you must have overlooked them, I quoted-- they should be easy to find. One is in the very first message. Please note the forum logs edits.

Edit: Thanks they're gone now.

I've participated and contributed to many minining optimizations discussions-- at least back when they were frequently interesting; I have no objection to that.

I have made crystal clear what my complaint was at every instant and moment:  You pointed to some random, uninteresting blocks, alleged on a flimsy basis that they were evidence for a secret mining optimization, known to you but which you wouldn't discuss.  I'm glad you're retracting your statements now.  But this smelled like a scam, it's nothing personal.

Secret substantial mining optimizations would be a threat to Bitcoin security (if they were real), the level depending on how significant they were; because if significant enough they could undermine the decentralization of mining which is critical to the system's security assumptions.  I have always thought it was unlikely that there was an actual threat there which you were aware of-- initially on the basis that not so long ago you were asking very uninformed questions about mining, but its not impossible that someone new to the subject might discover something that has evaded the analysis of so many others over the past 6 years. My comment there was if I were to accept your belief then it was an additional reason why you should disclose.

I'm really not sure how I should understand that.

Oh how perfectly diabolical of me to use reverse psychology! You've uncovered my true plan: to conceal the secrets of mining by insisting you reveal them to the public if you're going sulk about claiming to know things! How could you have figured me out! oh no!

LOL. It almost makes up for all the irritation, so funny;  Actually my comment was "it's the behavior of hardware sitting right next to me"; I was referring to a Bitmain S1 which my foot was propped up on at the time and which hasn't been turned on in months.

But again I should be awed by your incredible sleuthing-- I am actually mining, with downclocked SP20 at 1.27TH/s. With my $0.35/kwh marginal power this costs me somewhat under $200/mo to operate. The P2Pool stats page says that the income is ~0.0107btc/day.  So you've totally caught me in my evil plot to prevent people from competing with me for my $130/month _loss_;  where by I must spin an elaborate information concealing campaign on the forum in order to lose less money, because simply _pressing an off switch_ is far too difficult for my feeble brain to handle. I truly have been educated stupid, and my participating in mining could have nothing to do with my support for the Bitcoin system and my @#$@# development of the @$#@$ Bitcoin software. Not a chance. It's all a conspiracy. You caught me.

(I'm thankful for your confession of non-participation; always good to know when the writer doesn't have direct experience in the subject! )

Feel free to quote my whole messages if you like; though I expect they'll reflect poorly on you-- not me.

I don't-- I speak to my personal responsibility and the what I feel is my share of the collective responsibility.  But-- if I were to speak for others...

... these might be some pretty good reasons to do so. But, as I said-- I'm not. I even specifically called out in my prior message that I was just speaking as an ordinary community member. Anyone is free to disagree as you have so vigorously done.

Among other professional pursuits which required a strong command of probabilities, I spent over a decade working on the design of compression formats; so this is a bit amusing too. While none of my formally peer reviewed publications have been squarely statistically directed, many things I've published professionally or casually have also required fairly significant statistical work-- such as fault tolerance design for large scale communications networks and predictive failure monitoring-- this may be part of why I quickly observed the flawed reasoning like failing to compensate for drawing your parameters from your data, or the impact of multiple comparisons-- none of it is a mystery but its surprising to laymen without a working experience with probability.

There is much that I do not know, thats for sure-- and I am constantly learning new things... and while I'm also sure that there are many things I could learn from you, on the particular subject of working with this kind of data, it seems that most of that would come in the form of observing your errors. I hope you could share this learning with me, but I fear your credentials have blinded you to recognizing the possibility that you're actually far off the mark this time--  you're resorting to increasingly tortured adhomenem, and at this point I can only laugh at it.


Ah, I missed this earlier (it was burred under an untrimmed quote of my graphs):
I most certainly can! I did above, in my comments related to mid-state, which apply just as well to recomputing a new merkle root. Some amount of mining gear is able to do ntime rolling, I don't believe many (or any) use it anymore; and I think its actually incompatible with most stratum pools, but I'd have to check to be sure.  It isn't a useful advantage because the savings it provides is so enormously removed from the scale of mining that it doesn't matter. Any improvement prior to the nonce increment is effectively reduced several billion fold-- it's like saving 90% of a drop of water relative to the ocean, 90% is a lot of the drop, but nothing of the ocean.  Ntime rolling was historically interesting because prior to getblocktemplate and stratum, a remote miner required network traffic every time the nonce space was exhausted and this could produce high load on pools; the ability for remote miners to increment an extranonce themselves in both of the newer remote miner protocols largely made ntime rolling obsolete.

Thats what the subforum is for; I'm glad you've now joined its purpose. The complaint all along was that your posts were not conducive to that but instead to something else.  Now that thats cleared up, I'm not sure what else is there to say--  in the post I'm responding to here you seem to be saying that 1% events with unspecified other coincidences are deserving your attention.  With moderator hat on-- I'm going to ask you to not create threads in this subforum for 1%/block level events without a good, clearly stated, technical analysis that gives a rational basis to believe they're not a once a day pure chance false alarm.  ... Your first abrasive message to the thread was the second post with "Last, don't insult our intelligence."-- many people have read and responded patiently. My passive patience ran out at about your dozenth post in the thread. It happens some times, I'm sure all of us are warn out from the argument.

Behold: http://blockstream.com/sidechains.pdf  read appendix b "Efficient SPV proofs", in it we present a design that allows you to precisely the amount of work done (in the expectation) on a chain and compute exactly the same value as a user linearly reading the headers using only log communication.  (The tradeoff is that because there are few samples the measurement has high variance; so an attacker authoring a false proof could get lucky, similar to how you might find a block on your CPU... so even though the expected work to forge a proof is exactly the same, the probability of success in at attack is higher than for an equivalent length reorg; though there are various ways to further harden it, most further decrease communications efficiency).

I'm not, I am making the most simplistic of statistical arguments against a logical fallacy. To say it technically you've adopted a model so complex that we cannot tell if you've overfit and can not reject the null hypothesis.

I can demonstrate this another way;  lets take your proposed tests:

"having 4 nounces with differences less than 1.8%, 0.12% and 7.2% respectively  of the range in this order."

"the block size, also points to the fact that it is the same miner who mined the blocks. 731 kB blocks are quite common as noted earlier by someone else, but it is not
very likely either to find them consecutively. " you haven't given a numerical threshold, so I'll just take that to be sizes within 10%

"The third piece of evidence is how close in time are these blocks."-- they're not actually unusually close, given hashrate changes, but I'll apply the softer limit of allow below 10 minutes.

"The fourth piece of evidence is the non-chronological timestamps " -- these are also fairly common through Bitcoin's history; but I'll go ahead and apply it too.

"The fifth piece of evidence is that the first block is mined by AntPool and the next 3 by anonymous." -- I can't mechanically apply this one, since it's just bc.i's opinion; but as you'll see leaving out a constraint only aids your argument now.


This is your test for a specific non-disclosed mining optimization. Since you won't disclose the optimization I cannot argue with that point, though it sounds implausibly over-complex.

Now we're going to ask the related question:  "How much of the network is using the optimization identified by this test?";

There are _no_ there matches in the blockchain meeting that criteria other than the single range you've named.   What should we then assume the proportion of hashrate using your optimization is?  Approximately 0.

It's well established that normal people routinely engage in completely defective statistical reasoning. Statistics are unintuitive to people, virtually everyone finds qualitative reasoning more intuitive than quantitative reasoning.

An ideal way to reason about things is to first understand the process; form a hypothesis and from the hypothesis develop a model without looking at the data:  "If vulnerability X is being exploited I will see blocks of with structures and probabilities X, if X is not exploited I will see blocks with structures and probabilities !X". You can then ask what the KL divergence of these two distributions are, and if it is very small then you will gain almost no confidence even with many observations; the question may be undecidable. If there is a separation between the probabilities  then observations one can then apply a statical test to reject one alternative or another with a intentional level of chance of error.

An example of this would be "From the structure of SHA256, and the fact that the 80 byte input requires two runs of the compression function; someone could create slightly more efficient specialized hardware which hardcodes much of the message expansion and initial rounds from the second compression function run-- and scans by rolling the block version.  If this optimization were at play I would expect to find that the block version would have very high entropy, perhaps 32 bits though anything more than a few bits would be suspicious; blocks exploiting this behavior would be expected to have uniform version numbers, rather than a constant in non-exploiting blocks"  We could then apply this test to the blockchain, and because of the big gap between two cases we could decide pretty quickly if the test was indicating. This isn't a definitive test; there could be other reasons for the strange behavior, like miners trying to reduce their network bandwidth usage; we could try to make it more specific by adding a rule like "and we expect blocks with random versions to have nonces with far less entropy".  ---- unfortunately, keeping the reasoning private prevents that kind of thoughtful analysis; all we can go on is how "weird" blocks are in the absence of a reasoned model, and thats not very useful since every block is "weird" by some definition.  Its like if you look for the number 11 you'll find it everywhere.

Sometimes there are free parameters-- we want to ask a question like "does weight over some threshold cause heart attacks? and if so what is that threshold?", without knowing the parameter in advance. This had the danger of fitting the model to the data and telling us nothing at all (like my example of "predicting" the nonces that just happened);  one tool used to address this problem with parametric models is cross validation:  You split your observations up and use one subset to train the model and the other to test.  If the effectiveness goes away during cross-validation the model is likely overfit. The protection isn't perfect because if you tweak the scheme based on the results you miss overfitting the "meta"-parameters..   This is effectively the test I applied above:  You fit a model (a set of gap differences, block timstamps, inter-arrival times) and I excluded a single hit which you appear to have used to set your parameters from the testing set and found your model matched _no_ blocks at all.

What shouldn't be allowed is bad statistical reasoning, which can happen no matter how "natural" your model is, what primarily matters is how many degrees of freedom it has.  What you are looking at there is _not_ dispersion; I tried to give you the benefit of the doubt and assume you meant one of the standard metrics for dispersion (_range_) and you insisted on a complex polytope shape constraint; with probably something on the order of 34 bits of parameter space (three percentages to 0.1 precision without permutation), plus some more model freedom in that its looking at differences instead of absolute values.

If I instead use the standard deviation (another common dispersion metric) we that 0.1875% of 4-block sets (given uniform nonces) would have a standard deviation as low as your selected one; so again, something we'd see every few days.


There is huge amounts of information; everything is open source except for the RTL and mask images of common hardware; though most follow not too far from prior FPGA designs in their overall structure.


It appears unlikely, all four are miners that appear to constantly reuse a static address for their income, but they're each different.  Walletexplorer does an overly aggressive 'taint' analysis and links many addresses to each, but they're disjoint.  The lowest number appears to be antpool, the next appears to be AnxPro.com, the next is likely Polmine.pl (it frequently pays addresses connected with polmine.pl).

There is nothing astonishing that when you pick criteria to fit the data, you find that it fits the data. There is also nothing astonishing that when a model created this way seems unlikely on a uniform basis that it will fail to generalize to anything but the data you fit it on.


Great; so you'll accept concrete proof that these blocks were mined by independent parties as a definitive proof that you were incorrect?

I'm having a hard time figuring out what you mean by retracted and erased;  your posts in this thread continue to claim  "It is clear that someone found a trick for fast mining. I kind of happen to know what might be.", "It is premining at some extend. Won't disclose more for the moment." and so on.  You continue to hold that you have "secret" knowledge which you will not disclose so that it can be discussed on its merits or lack thereof-- this entire thread is a great big advertisement for this claim and seems to have served you little other purpose; which the experienced professionals on this forum find to be non-credible and not supported by the evidence you've presented.


As I explained privately, your behavior so far is indistinguishable from a person who is willfully and fraudulently claiming to know of non-public mining optimizations in the hope of selling them to some greedy sucker who is unable to assess their merit except on the your misleading qualitative claim that the blocks look 'weird'.  Protecting the forum's participants from being deceived themselves or from suffering the traffic from a flood of hopeful scammers demands that we call it out when we see the potential of it-- I say this not as the subfourm moderator or a developer of the Bitcoin system, but just as a community member... There is plenty of room for doubt; I'd say it's even more likely that you're just confused by the statistics of it, but the benefit of doubt can't be given freely, since those looking to exploit people will just sail through that opening.  I offered you a simple mechanism which you can use to distinguish yourself-- share your complete theory with (ideally) the thread or privately with a respected member of the community; in doing so you gain the ability to refine it against the forge of experience, discover what (if any) parts are already known; and, in the possibility that there is something to be concerned about, gain the ability to protect Bitcoin from any potential harm if required.  (I think you greatly underestimate the potential harm of privately held substantial mining optimizations-- beyond some threshold they would result in the complete centralization of the bitcoin system)

The forum trust is a metric of trust; I explained in the rating where the distrust comes from. Your claims here are not credible to any of the subject matter experts and as far as we can tell they're based on erroneous statistical reasoning, you're shielding your theories from criticism by hiding behind secrecy, forcing a debate around statistical minutia and how "weird" the blocks feel qualitatively, rather than the merits of your idea. Only a few weeks ago you were asking questions (using asics to hash files) that showed a profound lack of research and understanding of Bitcoin mining.  I want to be sure someone considering trading with you over these ideas is aware that other community members do not currently consider your claims credible or likely to be well founded. Anyone is free to read the rating and come to their own conclusions.

I'm sorry that you find it heavy-handed.  I would prefer if there were a way to make ratings which were more targeted or conditional. In consideration of this, I'll go and reduce it to neutral to avoid triggering the red flag on you. I am not seeking to cause you distress.

I mentioned it non-specifically in my first response in this thread; (actually some things even more powerful than that). The objection was always the claim of "secret" knowledge supposedly supported by these very non-specifically identified blocks. Sergio actually described the optimizations he was talking about; as have a great many people. Not said they were keeping them secret and then "oh yea that" when other people play guessing games with other already known optimizations (which there have been easily a dozen discussed over the years);  especially not using alarming words like "premining" (which is a term used for a oft-perceived dishonest move in pump-and-dump altcoins).

As an aside-- I suspect your time is somewhat off, since there is basically a constant offset of a couple minutes between your and my numbers. (My clock is timed off a local atomic clock which is wsynced to UTC with a GPS timing receiver; it agrees with a remote stratum 1 NTP server by better than 1ms).
Indeed.

Did you not read and understand the point I made about your post-hoc rule selection? (First half of https://bitcointalk.org/index.php?topic=1045381.msg11268595#msg11268595)

One cannot define rules after the fact that match the data then use that to argue that there was some improbable effect; e.g. "the last 4 nonces were 3091427529, 3627164536, 545428875, 138911025 ... what is the probability that those four specific nonces would arise in a row? 1/2^128!!!!!". No, the probability is _one_ because I picked the criteria out of blocks 354823,  354824, 354825, 354826. We learned _nothing_ from that observation.

This works similarly for more complex tests, for example "A run of 4 nonces where exactly one of the 6 unique pairs constructable from the set doesn't have coprime nonces, and the non-coprime pair is the second and last nonce with a GCD of 3" is a rarely met criteria which also matches your data;  but that doesn't mean there there is some evil almost-but-not-entirely-coprime conspiracy... or your differences between nonces test. The extra complexity doesn't make the reasoning any less fallacious, it just makes it harder for people to notice that its poorly reasoned.


I explained specifically how and why miners select nonces non-uniformly-- I would ask if you missed the post, but you responded to it; https://bitcointalk.org/index.php?topic=1045381.msg11266349#msg11266349.

You gave no indication that you were "waiting" on anything-- you even responded "This makes sense"--, if I didn't see this post would you just go ahead and declare default as you did earlier in this thread to jl2012?  I've already wasted an incredible amount of time responding to you, but I won't make claims and fail to back them up if someone finds them implausible.

However, you're not going to get a compact "algorithm" because there isn't a compact description for it, as the behavior is a product of the physical geometry of the part, random decisions decisions, and even the failed engines on particular chips; ranges are also truncated for load distribution across chips, and to control latency.  The behavior is well known, documented in various data sheets (see, for example register 0x7f), and even sometimes used for device autodetection by some pools and mining software (Even for just mode detection, E.g. the icarus driver sends a dummy work task to the miner which was chosen to have 4 different solutions, and figures out what mode the chip is in based on which solution it returns). Historically, before extranonce rolling existed, nonce restrictions were used to reduce work on pools. Of course, it's not the _most_ commonly discussed thing; because its fundamentally uninteresting-- all nonces are equally good so if some device only uses some subset of them the response of an engineer working on software for them is just "okay, thanks for telling me.", they write it into their implementation, and they move on with life; they don't go and author a press release.

And yes, reducing the range it results in some additional extranonce work, but extranonce work is trivial already-- a single cheap micocontroller can do the extranonce work for many TH/s of mining.


And what are your other _exact_ criteria? Please lay out the exact test you suggest which shows those blocks to be improbable in a single post in a list so there is no confusion... I've not seeing _anything_ exceptional about these blocks, excepting the fallacious post-hoc selection of criteria to exactly fit them around nonces.

What is the probability, under uniform assumptions, of a single nonce being the specific value _2167965896_?    It is one in four billion.   And yet, there it is in block 354640--- a nonce with that specific value.   Does this mean that I now have evidence for some theory in the forum of a one in four billion event?   No.   Because criteria of that specific value was selected after the fact based on the data, and so the probability of observing it is 1 and the information content of the observation is absolutely zero.

Your suggested study is making the same form of reasoning, but this fact is somewhat hidden by the additional complexity-- yet it clearly was based on observing the data (your threshold are the exact differences in the data), rather than being based on some principle which set in advance and only after was it tested against the data.

Putting that aside for a moment, even if I take your back-computed from the data 0.00015552% probability number (without order-- do you really demand ordering?-- if so, you're owed a stern lecture on fallacious arguments); with 354k blocks we would expect to see a 0.00015552%/block event about 55 times; and in the 25 days between your post immediately prior to creating this thread there would be a 42.87% chance of observing it at least once in that window; all with uniform nonce assumptions (obviously the non-uniform nonce reality make it more likely).

I provided the program to cut through some noise; rather than arm-waving we've had in this thread the programs operation is clear, and can easily be tinkered with (e.g. as soon as you assume some non-uniform distribution, you must then integrate; much easier and safer to just twiddle the numeric code and get an approximate answer; especially once you start adding any non-linear hypothesis).

"using the malleability of timestamps" There isn't any reason to assume from inconsistency of timestamps that miners are intentionally using their generally free control of the timestamps for much of anything. Large latencies in miner hardware/software/pooling (including avoiding bandwidth usage sending new midstates) contribute a lot of inaccuracy, but there is no such thing as a singular definition of time in a decentralized system; miners have their own clocks; they often only vaguely agree, the numbers are all over the place, they've always been more or less all over the place, and its not surprising. Every once in a while someone sees a block with a timestamp an hour in the future and they show up freaking out... its ordinary and not unexpected.

As far as my timestamps; I'm reasonably well connected due to connecting to the block relay network.  Stats, in seconds for the last 1295 blocks (which I'm reasonably confident is a generally uninterrupted observation), negative times are blocks 'from the future' according to my local timebase:


     Min.   1st Qu.    Median      Mean   3rd Qu.      Max.
-1539.000    10.000    26.000     1.989    44.000  1098.000

Absolute differences:

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
   0.00   19.00   34.00   63.73   57.50 1539.00

Stem-and-leaf plot shows that the extrema are outliers:

  The decimal point is 2 digit(s) to the right of the |

  -15 | 4
  -14 |
  -13 |
  -12 |
  -11 |
  -10 |
   -9 |
   -8 | 3
   -7 | 2110
   -6 | 76430
   -5 | 866631
   -4 | 976633332211100
   -3 | 99986433321000
   -2 | 99999888666655444443333221000
   -1 | 9999998888777765554443333222111110000000
   -0 | 99999988888777777777666666555555555554444444444444433333333333333333+42
    0 | 00000000000000000001111111111111111111111111111111111111111111111111+938
    1 | 00011112222233334444555666677788899
    2 | 02
    3 |
    4 | 5
    5 |
    6 |
    7 |
    8 |
    9 |
   10 | 1
   11 | 0

A density chart of the (-120, 120):

      +-------+-------------+-------------+-------------+------------+---------+
      |                                       ***                              |
      |                                      ** ***                            |
      |                                      *    **                           |
0.015 +                                      *     **                          +
      |                                      *      **                         |
      |                                     **       **                        |
      |                                     *         *                        |
 0.01 +                                     *         **                       +
      |                                     *          **                      |
      |                                    **           **                     |
      |                                    *             **                    |
      |                                    *              **                   |
0.005 +                                   **               **                  +
      |                                   *                 ***                |
      |                               *****                   *****            |
      |                      ********** *                         ***          |
    0 +  *********************                                      ********   +
      +-------+-------------+-------------+-------------+------------+---------+
            -100           -50            0            50           100        

Right, it's a normal and expected property of the exponential distribution (that distribution of interblock gaps) that there are a lot of small values; more than you'd probably expect after being told there was a 10 minute average. E.g. about 10% of blocks are 1minute or less apart (assuming lambda=1/600; in reality because of hashrate increases the expected time is often more like 8 minutes). Other constraints like the minimum value for the time coded in blocks also contribute to making the timestamps have less diversity than you might expect from a first guess.

In the case here, they're not even unusually closely timed the times on the four blocks in question are 12:24 13:02 13:12 13:11 (the prior one was 12:03).  Note that the times aren't monotonic, which highlights a previously mentioned limitation in trying to reason about time; they're not all drawing from the same clock as there is no singular now in a decentralized system.

I personally observed these blocks at the following times:

2015-05-02 12:25:21 UpdateTip: new best=00000000000000000f8d7a12d307ddc717cab90d2ced5c7320624a13714b0aa3  height=354640  log2_work=82.71737  tx=67422028  date=2015-05-02 12:24:26 progress=0.999999  cache=14812
2015-05-02 13:01:37 UpdateTip: new best=00000000000000000a1ebf23947c2dc38f980c66c1fd1303235326e36ea5afae  height=354641  log2_work=82.717407  tx=67424094  date=2015-05-02 13:02:15 progress=1.000001  cache=9138
2015-05-02 13:05:29 UpdateTip: new best=00000000000000000f181b8cfb70624cd74bcac01c930657bd1bde85ff59e7fd  height=354642  log2_work=82.717444  tx=67424677  date=2015-05-02 13:12:17 progress=1.000007  cache=10305
2015-05-02 13:08:46 UpdateTip: new best=000000000000000015c33a22604bd9c01806c3add1b33d6b8dd1e663da95cbd1  height=354643  log2_work=82.717481  tx=67425354  date=2015-05-02 13:11:28 progress=1.000003  cache=11807


So gaps of 36:16, 3:52, 3:17.   Given a ~10 minute expected time about a third of blocks are 3:17 apart or less.

Is this kind of comment acceptable in the circles you normally travel in?  If one of the people I worked with presented an argument of this form my response would be "Shame on you".  Perhaps jl2012 has other things to do than click reload constantly and wade through the page and pages of untrimmed quotations in your message?

Doubly so when it has been very clearly explained that your statistical argument is outright incorrect already by others.  Miners do not select nonces uniformly for boring engineering reasons, this is a fact, it's the behavior of hardware sitting right next to me, it's easily observable on the blockchain.   Your statistical argument is that IF nonces were uniform then it would be unlikely to see a run of similar ones.  You do not correct for multiple comparisons (we've had some 354k possible runs of 4 for this to be true in), but most importantly you seem to strangely continue to ignore the fact that we know that various hardware does not uniformly select nonces; and instead you suggest this is evidence of your secret hypothesis. Why are you surprised that we reject your reasoning and instead question your motivations?

The nonces here aren't even that close-- 2167965896, 2148513297, 2143118375, 2456983311  spans 7% of the nonce range...  Since people seem to get caught up on the the analysis, perhaps a numerical example in python might simplify things for people:


import random

trys = 1000000
threshold = 2456983311 - 2143118375
small_rng = 0
for i in xrange(trys):
  n = [random.randrange(0,2**32) for ii in xrange(4)]
  small_rng += (max(n)-min(n))<=threshold
  
print("Out of %d tries, %d groups of 4 nonces were spanned a range equal or smaller than %d."%(trys,small_rng,threshold))
print("Since there are 144 blocks (thus 144 overlapping groups of 4)")
print("We'd expect to see this every %f days with _uniform_ nonces."%(1./(float(small_rng)/trys*144.)))


Which yields:

Out of 1000000 tries, 1524 groups of 4 nonces were spanned a range equal or smaller than 313864936.
Since there are 144 blocks (thus 144 overlapping groups of 4)
We'd expect to see this every 4.556722 days with _uniform_ nonces.


Since we also know existing hardware produces a subset of nonces we should probably expect these runs to be even more common than the above reflects.
To get a feel for how non-uniformity changes this, switching to an RNG with a linear sloping probability, abs(random.randrange(0,2**32)-random.randrange(0,2**32)), increases the rate of these 'small' spans by about 2.8x.

Without a reason to believe the exact criteria was established in advance instead of based on the data we should also probably be correcting for the fact that there are many other possible "patterns" people might find interesting and might use to claim support for some secret theory; e.g. ending with certain digits in some base, or being close mod 2^32, having digits that are cyclic shifts of each other in some base, being close by a larger but also 'small' threshold (E.g. the post originally claimed 6 blocks with a nonce span of 1856814243 which we'd expect to see a bit over 8 times per day.), or having some other simple arithmetic relation, etc.

Are you referring to your message to me 45 minutes ago?  You provided no information that wasn't in the thread; and I responded asking you to provide information (either privately or publicly, though I encouraged you to provide it in public.).

I and multiple others have explained to you multiple times now that various mining devices consider only subsets of nonces for completely boring reasons--; what isn't explained is why you keep alleging that something with a boring explanation which has been provided multiple times hasn't been explained.

A few week ago you were making posts that demonstrated that you had no idea how mining worked at all and were not willing or able to do even the most basic research on the subject.  Your posts here continue to show a remarkable lack of basic research, yet you expect people to believe that you know something that hasn't been discussed in the hundreds of past threads about low level mining optimizations by experts in the field (including people like the inventor of hashcash; the general scheme used). Moreover, you started this by deceptively asking a question you later claimed to "know the answer to"; so I hope you can understand why people are skeptical here.

You're making serious claims that would be concerns for the security of Bitcoin if true; such claims demand serious substantiation... doubly so when they coming form a source which seems to have been clearly deceptive in this very thread and is obviously not very familiar with the subject.

In any case, you need do nothing more to defend your reputation than to simply explain what you're thinking.  If your ideas are wrong, they'll be corrected; if they're right but not news, old threads will be referenced, if they're new and concerning the issues will be addressed if possible, etc.   Right now, though, you're basically trying to convince us of something for which you'll give none of the information which could be used to support your claim. The only content in this thread will be people attacking your methods and motivations as a result, since you are intentionally refusing to provide the only information which could be used to analyze your claim directly.

Because mining ASIC use "sea of hashers", they take one midstate work unit and broadcast it to hundreds (or even thousands) of SHA256 engines, each one tries a different nonce for the same work. You only have a finite number of engines so only a subset of nonces will get used, also some engines will fail (sometimes the same engine on every chip of a particular make) adding additional gaps. The allocation schemes differ from device to device (e.g. some hardware only produces even nonces or multiple of 64 nonces, some hardware only produces nonces in  a range 0-1024, etc.) There is also an optimization you can do where you actually hardwire the engines for given nonces and grind the first half, though I don't know if anyone bothers with it.

Same miner or similar hardware, perhaps-- sure? and so what? Its not uncommon for a large miner (or a hardware type with a large share of the hashrate) to find four blocks consecutively; there is effectively a calculator for that in the bitcoin whitepaper.

FWIW, I think Valiron is engaging in misconduct here. At first there is an "innocent" observational question and then after people point out that the observation is expected (because of hardware that only uses a limited set of nonces, and because of the block soft-target maximum) he had adopted a position of "secret knowing" that substantiates his position and yet he will not explain it.

Of course, it's possible for someone to be innocently ignorant, even likely (especially considering Valiron's posting history; there are plenty of optimizations you could be unaware of, or structure about mining that lay people misunderstand that could be mistaken as some advantage)-- but there is no reason to play secrecy games there, and secrecy is actively poisonous to having your understanding elaborated.  Likewise, it's possible to actually know secrets, but then you don't go hinting about them on the forum.  One possible way gain from the pattern of posts here would to manipulate the market with FUD about the security of the hashing algorithm, another would be to try to scam greedy buyer into buying these "premining" secrets; so these are my working theories, and I've negatively rated valiron accordingly.

(I'd debated instead locking the thread; as a thread of "Oh whats this" "its that" "oh no its not, it's something else but I won't tell you" "we told you its this" "bad math bad math" isn't a good use of the forum;  but I thought giving a chance for a correction would be more useful).