WTF is this? Someone found a trick for fast mining?

[valiron]

- snip -
for solving the block you change other fields.

There aren't very many other fields to change.  That's why the block has a nonce in the first place.  There's nothing special or magical about changing the nonce, it's just 4 bytes added to the header specifically for the purpose of being very fast and easy to change.

The other fields of the block header are:

• The block version number. This 4 byte field cannot be changed without invalidating the block you are mining.
• The hash of the previous block in the chain. This 32 byte field cannot be changed without invalidating the block you are mining.
• The merkle root of the transaction list. This 32 byte field cannot be changed without modifying or rearranging the transactions in your transaction list.  This would be a MUCH slower and more complicated thing to change than the nonce. However, a pool will create MANY different merkle roots (by modifying the extranonce in the coinbase transaction) so that they can give a different block header to each miner.  This allows each individual miner to only need to deal with the nonce (and timestamp) and not need to recompute the entire merkle root.  It is also possible for the pool to provide enough information for individual miners to modify the extranonce themselves along with the nonce.  Keep in mind though that modfying the extranonce increases the time and effort involved, so it makes more sense to exhaust the nonce range available to the equipment before making any effort to move on to a new extranonce.
• The timestamp.  This is a 4 byte field that some miners change after they've run out of nonces.  It essentially becomes a secondary nonce. However, it is very limited on allowable ranges.  Since the miner would need to test to make sure a value is within the allowable range before hashing the block, it very slightly increases the effort as compared to simply using the usual nonce.  If someone chooses to modify this instead of the nonce (or while keeping the nonce in a tight range), then this field BECOMES a nonce.  There is no benefit of manipulating these 4 bytes instead of the other 4 bytes.
• The current difficulty (represented as "bits"). This 4 byte field cannot be changed without invalidating the block you are mining.
• The nonce. This 4 byte field exists solely so that is is fast and easy to modify the header before hashing it again.  It serves no other purpose, and has no restrictions on its value.  This is also the field that you are saying NOT to change when "you change other fields".

That's 4 bytes + 32 bytes + 32 bytes + 4 bytes + 4 bytes + 4 bytes = 80 bytes.

That's it.  There's the 80 bytes that are hashed during mining.  I'm very curious to know which of those fields you think might be better to change instead of the nonce?

Those in red cannot be changed without invalidating the block.  Those in green are already manipulated as part of the typical mining process, and the typical mining process already manipulates them in the most efficient order.  If there is a benefit to manipulating the timestamp or merkle root INSTEAD of the nonce, I'd be very curious to know what you think that benefit might be.

I'm not certain, but I don't think you can adjust the merkle root without needing to recompute the midstate as well?

The intention of my comment was that if mining in a classical way with repetitive trials it doesn't make sense to not change the nonce for that purpose.

What I was observing is that if we find evidence of someone fast mining without changing much the nonce, it probably means that he is mining in a non-classical way.  As pointed out the version number can be changed quite freely, but be safe, gmaxwell will fork if this happens, you can find a discusion of header malleability here: https://bitcointalk.org/index.php?topic=563913.0

[valiron]

I think gmaxwell is right, there is not enough evidence in those 4 blocks to suggest that there has been a breakthrough in mining ASICs.

On the other hand I guess valiron is also right, he knows (and probably can't disclose) that some mining companies may be testing new tricks to improve their ASIC hashing power. Maybe he hacked into one company computer or he was told by a friend who works in one of them and he promised not to say anything.

Mining companies want to maximize their profits and they are not obligated to disclose their engineering achievements. Those are trade secrets and nobody has ever complained about ASIC Mining companies having close designs. I don't even think that the Bitcoin community can even enforce ASIC companies to open their designs, because they are already using other companies IPs that they can't disclose, and because they can always go anonymous to avoid disclosing anything.

There is plenty of technical information put together in this long thread (given by gmaxwell explicitly, and DannyHamilton's analysis on which parts of the block header can be used as nonce, and my very old posts about modifications to the Bitcoin header) to ease you discover one of such tricks. Take some time to think about it, take aside all posts with personal insults, and you'll probably find the solution right in front of your eyes.

I'm not that clever so there may be more tricks to discover.

However, a trick can only give you a certain speedup, say 20%, based on a reorganization of the SHA256D operations, or the pre-computation of some operations that change less often. Other changes (such as reducing the fabrication node) can give you much higher speedups. So this isn't alarming.

A completely different thing is to find a way to invert SHA256D, which I'm absolutely sure nobody will ever be able to do without some revolutionary quantum computer that does not exists even in theory.

The only attack I was thinking of when I wrote the Bitcoin header post, was all mining companies adopting tricks that give them some little advantage, but at the same time they degrade the performance of the network as a by-product. One of such attacks is cited when I posted about using approximate adders, and the danger that a monoculture of approximate ASICs can get stuck in a header that always generates a faulty addition.
 
If such problem ever arises, the community will probably find the way out by doing the right hard fork to prevent it.

IMHO the cryptographic security of SHA256D function of Bitcoin will never be seriously compromised.
However if there were a single mining company manufacturing ASICs being 200% faster than the competition, that would clearly hurt Bitcoin in a practical econo-socio-political way. The good news is that the accumulation of tricks probably will never reach such an improvement.

Best regards,
 Sergio.
 

Thank you for your post Sergio. In particular for focussing on the technical part.

First, let me disclaim: I have any no inside information nor I have hacked into anyone's computer. I just use my brain and my mathematical background. Second, I subscribe your comments, in particular that these statistical facts are not proof of anything. They are only indications. Also that, as you stated and I have repeated, bitcoin security will not be compromised by a boost in mining performance. Third, I thank you for the humility of your post. We are not all that clever and there may be tricks that we never dreamed of. Thus I think the community must be on the lookup for statistical anomalies that may reveal a breakthrough.

[valiron]

There is plenty of technical information put together in this long thread (given by gmaxwell explicitly, and DannyHamilton's analysis on which parts of the block header can be used as nonce, and my very old posts about modifications to the Bitcoin header) to ease you discover one of such tricks. Take some time to think about it, take aside all posts with personal insults, and you'll probably find the solution right in front of your eyes.

Be careful...you are stating that you known things that you are not disclosing...this will infuriate some people that will accuse you of threatening bitcoin security. Do you mean also that gmaxwell knows some tricks and he is not discussing them?

I believe anyone is free to discuss or not discuss whatever he knows. I also believe that the only way to discuss freely anything is not to be put in the position "You must prove that you are not a charlatan" position.

[2112]

The only attack I was thinking of when I wrote the Bitcoin header post, was all mining companies adopting tricks that give them some little advantage, but at the same time they degrade the performance of the network as a by-product. One of such attacks is cited when I posted about using approximate adders, and the danger that a monoculture of approximate ASICs can get stuck in a header that always generates a faulty addition.

Thinking out-of-the-box has both good and bad aspects:

+) on the positive side it allows novel and unusual solutions to enter the field, like your idea of intentionally breaking the topmost level in the carry-look-ahead login of a 32-bit parallel adder, which you called "approximate addition"

-) on the negative side it disconnects one from the already known solutions in the field. Some EDA tools already can split a 32-bit adder in a critical path into a pipelined pair of 16-bit parallel adders. The general methodology is called "register balancing" or "delay balancing".

You've made a far-reaching statements about a possibility or necessity of changing Bitcoin hashing algorithm in the face of your discovery. Have you consulted your discovery in private with somebody knowledgeable with digital logic design? What did they say?

[altcoinex]

I believe anyone is free to discuss or not discuss whatever he knows. I also believe that the only way to discuss freely anything is not to be put in the position "You must prove that you are not a charlatan" position.

If I recall this started because you suggested in your original thread you uniquely 'knew' something but for the security of bitcoin wern't yet ready to reveal... Now it is simply 'there are some nonces near eachother, that means someones exploiting mining clearly' with out any technical proof or explanation on how it would be possible. You might not have to prove your not a charlatan, but if your going to make an accusation, and even go as far as to allude you know a secret, then you have to prove it has some validity....

[Sergio_Demian_Lerner]

The only attack I was thinking of when I wrote the Bitcoin header post, was all mining companies adopting tricks that give them some little advantage, but at the same time they degrade the performance of the network as a by-product. One of such attacks is cited when I posted about using approximate adders, and the danger that a monoculture of approximate ASICs can get stuck in a header that always generates a faulty addition.

Thinking out-of-the-box has both good and bad aspects:

+) on the positive side it allows novel and unusual solutions to enter the field, like your idea of intentionally breaking the topmost level in the carry-look-ahead login of a 32-bit parallel adder, which you called "approximate addition"

-) on the negative side it disconnects one from the already known solutions in the field. Some EDA tools already can split a 32-bit adder in a critical path into a pipelined pair of 16-bit parallel adders. The general methodology is called "register balancing" or "delay balancing".

You've made a far-reaching statements about a possibility or necessity of changing Bitcoin hashing algorithm in the face of your discovery. Have you consulted your discovery in private with somebody knowledgeable with digital logic design? What did they say?

No many people read my blog, so nothing I say is "far-reaching"

And I still think it would be better to change the Bitcoin header. But every bitcoiner wants to change Bitcoin in some way or the other, so I'm not alone. I promise I will write why I still think so in less than a month. I don't have time now.

Regarding consulting about discoveries, I hadn't consulted with anybody regarding the approximate adders, and that was not a good idea. I received a call the next day from the CEO of a well-known Bitcoin ASIC company telling me that my idea combined with their own optimizations would make their chips a lot faster.

If I had consulted with an expert about the advantages of approximate addition in some designs I would have tried to sell the idea for some bucks instead of just publishing it 

Best regards,

[2112]

I received a call the next day from the CEO of a well-known Bitcoin ASIC company telling me that my idea combined with their own optimizations would make their chips a lot faster.

My experience now shows that a necessary condition for being a "CEO of a well-known Bitcoin ASIC company" is to understand very little about digital logic design and compensate this with chutzpah. All publicly available information points to them having only part-time design talent or to making drastic shortcuts to the design workflow.

In particular the statement about "a lot faster" in a "next day" call, without running the simulations, points me to some bullshit artist.

I can only commiserate with you about not having any trusted friend to discuss your ideas without being worried about them being stolen. Thankfully I'm not in this position and have some contacts who wouldn't jeopardize friendship for a quick sale.

[bitcoinbeliever]

No many people read my blog, so nothing I say is "far-reaching"

More should read your blog Sergio, original thinking deserves a loud voice.  At least post the link next time!  I'll do it for you this time...

https://bitslog.wordpress.com/

[valiron]

Now about one transaction blocks...


Some statistics about post 2013 one transaction blocks
=========================================


Mean blocktime validation of one transaction blocks:  106 sec (with sd 300 sec)
Mean blocktime validation of post 2013 blocks:          525 sec (with sd 557 sec)

One transaction blocks are validated 5 times faster than regular blocks.

17.4% of one transaction blocks have non chronological timestamps vs. 4.6% of all blocks (post 2013) (about 4 times more)



Anyone running a node can provide a database of reception timestamps of validated blocks? (local reception timestamp)

Shorter validation time for one transaction blocks is expected for miners that start mining the empty block then keep adding transactions. Abundance of non-chronological timestamp may indicate that timestamp is partially used as nonce in these blocks.

[BlueInCoin]

This person just have the luckiest day in his life

[crazyearner]

Think this is all due to luck in all honesty than a brake though in asic mining. If it happens several times and continues to happen then maybe a trick indeed but right now seems just pre lucky to me with the findings and what is being said on here.

[gmaxwell]

Mean blocktime validation of one transaction blocks: 
[...]
Shorter validation time for one transaction blocks is expected for miners

It is unclear to me what you are talking about. What specifically are you referring to by "blocktime validation"?  Are you talking about the ntime gaps?  Blocks of very slow miners should have lower timestamps because they do not frequently update their midstate (e.g. they would claim older times because that was when they started); modern fast miners blow through the range quickly, and thus have plenty of opportunities to increment their time. (Much of the single tx blocks back then were believed to be a botnet that verified nothing).

If you are saying smaller blocks take less time to validate largely. This is mostly untrue at the tip of the chain.  Leave your node running for 24-48 hours and then look at the block verification times. You'll see that the actual blocks, in spite of being huge, typically verify in a few milliseconds (-benchmark will enable ms resolution timing results in the logs). This is because almost all the verification is cached from the transactions being relayed earlier.

[btchris]

Shorter validation time for one transaction blocks is expected for miners

It is unclear to me what you are talking about.

I believe one thing valiron is talking about is the practice (which I recall reading about, I've no idea if it's still in use, or ever has been) of pool operators sending out work for empty blocks to their miners immediately after a new block is found, to get the miners working on the new chain ASAP, and then creating a non-empty block & merkle tree at their leisure and updating miners once it's ready.

[valiron]

Mean blocktime validation of one transaction blocks: 
[...]
Shorter validation time for one transaction blocks is expected for miners

It is unclear to me what you are talking about. What specifically are you referring to by "blocktime validation"?  Are you talking about the ntime gaps?  Blocks of very slow miners should have lower timestamps because they do not frequently update their midstate (e.g. they would claim older times because that was when they started); modern fast miners blow through the range quickly, and thus have plenty of opportunities to increment their time. (Much of the single tx blocks back then were believed to be a botnet that verified nothing).

If you are saying smaller blocks take less time to validate largely. This is mostly untrue at the tip of the chain.  Leave your node running for 24-48 hours and then look at the block verification times. You'll see that the actual blocks, in spite of being huge, typically verify in a few milliseconds (-benchmark will enable ms resolution timing results in the logs). This is because almost all the verification is cached from the transactions being relayed earlier.

It is not what you think. I probably didn't explain myself well.

A miner can start working on the POW(=validating the block) before including any transaction. When you include transactions or update your transactions you need to recompute the hash of the merkel root, thus it is faster to start working with empty blocks (this doesn't mean that they are not quickly updated into a non-empty block).

This may be an explanation of why the average validation time of blocks with only the coinbase transaction is 5 times shorter than the average, because when these blocks are solved it is at an early stage. There may be other reasons...for example...start validating the next block before broadcasting the solved block to gain some advantage...and start validating an empty block because most of the transactions were included in the validated block.

If you can give other reasons you are welcome...

BTW...there are one transaction 1204 blocks since 2013, thus the average computed is meaningful.

If you can provide the data of local timestamps I can run more statistics on the discrepancies of timestamps. Anyone running a node since 2013 with that data?

Shorter validation time for one transaction blocks is expected for miners

It is unclear to me what you are talking about.

I believe one thing valiron is talking about is the practice (which I recall reading about, I've no idea if it's still in use, or ever has been) of pool operators sending out work for empty blocks to their miners immediately after a new block is found, to get the miners working on the new chain ASAP, and then creating a non-empty block & merkle tree at their leisure and updating miners once it's ready.

I think you explained it well.

[gmaxwell]

Indeed, yes it is the case that the work first issued right after a new block often has no transactions, since createnewblock can create a couple hundred milliseconds--- the time isn't actually the verification, but just the time it takes to build a candidate block... Since 2013 or so many pool programs will generate a long poll event (or equivalent) to cause updates so that miners will move to new transaction-containing work even before finishing their current work unit; but they will spent some time on the empty work.