The incompatiblity is by no means unique to segwit, the vast majority of proposed protocol improvements run into exactly the same issue.

Commited UTXO, Commited Address index, Commited Bloom filters, Fraud proofs, -- just to name a few more.

So what you meant to say was that covert asicboost 'effs' up the block architecture.

Because it's a 12x-ish additional work factor, thats all.  Is 12x a killer? Depends on how specifically that you've constructed it.

Trying to put it all on one big device and you run into IO/memory bottlenecks much easier-- your 100 million midstates is over 64gbit/sec of data, that device goes down and your farm is down. The device is also highly non-covert, so you cannot keep what you are doing secret from your facilities staff. It is much more straightforward to user local FPGAs to generate the collisions.  And from what I'm told use of local devices to generate is how the unreleased spoondooles design worked.

Consider: today miners could use centralized devices to generate midstates, but instead the S9/R3 miners have fairly expensive FPGAs with 16MB of attached dual channel DDR2133 to generate the ~2000 midstates per second they need.  (When will Trezor ship with such an incredible FPGA? Never, I assume ) They could use a centralized device to do so, but they don't. (well it looks like privately bitmain runs two S9s per controller, twice that).

If you've adopted a design with distributed generation, then you can't escape the cost-- perhaps you could deal with it, but that doesn't mean the infrastructure anyone has already built does deal with it.

It isn't-- because the speedups you conjecture creating already exist.

You propose that your design moves accessing the UTXO data out of the critical path for block validation, on the basis that you will have already validated the transactions and so don't need to access that data.

Bitcoin Core does not have UTXO accesses in the critical path in the same case and doesn't write in the critical path either. So it would not see the benefit you expect for this reason. (not that it wouldn't potentially have benefit, just not the benefit you are suggesting at least not for the reason you suggest)

Cheers

It's not semantics. When validating blocks the vast majority of the time the UTXO set is not used by bitcoin core; because the UTXOs are in the cache already (due to mempool) and nor are the result of the block's changes written out-- they're buffered until the node flushes-- and very often never get written at all because the newly created outputs are spent by a later block before the system flushes them.

OP suggests that the structure he proposes reduces the delay at the time a block is processed compared to Bitcoin Core's UTXO database, but it shouldn't-- at least not for that reason-- because the database is only used at block processing times in the exceptional case where a transaction shows up by surprise.

That is what I understand him to be saying, but it isn't true for Bitcoin Core. For Core, the moment when the UTXO database is updated/and changes committed is _not_ during block validation.  It's after in separate batch steps which apply big batches of changes at a time; specifically to mostly get interacting with the database out of the critical path.

This is also why if you shut your node off uncleanly you'll see it going and reprocessing dozens of blocks-- their effects hadn't been applied to the database yet.

There are no UTXO index reads or writes in the normal case (transaction recently seen.) in Bitcoin Core.

This is not how Bitcoin Core works. No access to the disk normally at all happens when a block is verified except for transactions that have not yet been seen, reading or writing.

Without saving the sync would be a _massive_ waste of bandwidth, so it was important that the saving be done first. The two are complementary.

Saving was also substantially driven by the motivation of being able to save prioritization for miners; which is not accomplished via syncing.

Saving is purely a client local feature-- it's something the Core project can decide to do without worrying if other implementations like it (and, in fact, Jeff Garzik showed up out of nowhere, having not sent a single message to the project in months, and attacked the saving vigorously)...  syncing, especially efficiently, requires a BIP and collaboration with other implementations some of which are hostile to the project and reliably oppose every proposal.  So it didn't make sense to hold up the saving for the syncing to be worked out, since saving really is no other implementation's business.

Efficient syncing is also an active area of research, there are techniques to perform the sync in ways which are significantly more efficient than you might guess. (But, of course, if you throw out the data-- it's going to have to be transferred again, saving it is the only way to avoid that!)


An earlier version of the proposal did periodic flushes, but we're concerned with creating excessive writes that lower the lifetime of SSD, and having half completed write can be worse than no write since we'll lose the in-progress flush.   The ability to trigger a flush via an RPC will probably come back, so if you want a periodic flush you'll be able to crontab it.

Not entirely, it's also useful to pick up some low feerate stuff that otherwise wouldn't be mined for hours or days... so even if you were off for a few hours it can have value.  If you're off for weeks it would be worthless, however.

Saving also improves the accuracy of data for long term fee estimation.

spent transaction hashes are there, spent block numbers are not needed: the blockchain is not used to validate transactions.

Point of fact: transactions being validated in advance has always been true and is completely orthogonal with compact blocks.  Bitcoin has also directly exploited this fact via signature caching (which your benchmarks disables, btw, because you use the undocumented blocks only mode) since applying Satoshi's patch for it in May 2012, and further with the input cache since July 2012.

I've seen Peter R make the same argument a couple times and I've asked him why he presents it that way but AFAIR he has never responded to one of those messages.

This is correct, but the covert method would become costly (as you note) and not very covert after a while (which I also think is worth pointing out)-- "why are you giving away free money by not including segwit txn? Hmm!".

I think it's wrong to focus just on segwit here, the interference shows up against most proposed protocol extensions... commited utxo set, stxo, fraud proofs, commited bloom filters... all would have similar interactions with covert boosting.

Personally, I didn't expect empty blocks to be involved.  (Though, antpool has produced significantly more empty blocks historically, e.g. between 390750 and 408052 there were 652 empties, 399 of them were antpool tagged).

Several people have it working now, and if you have a S9 or R4 they can point you to a stratum endpoint that will get you mining with it.

Strangely one problem they've run into is that if you have it use 4 versions (which would be better for efficiency) it produces otherwise valid headers with the most significant bit set... which are invalid per (IIRC) BIP 30. So far, no one has figured out what command to send the FPGA to use different version numbers.

(As an aside, dontcha think it's odd that R4's FPGA has 16 MB of dual-channel 2133MB/s memory  mapped to it?)

You may not distribute a LGPL program as part of a single static binary without the corresponding source to the whole thing or object files for the parts so that you could link with a modified version of the LGPLed component.

You make it sound like pruning is worse, in fact it's tens of times more efficient than whats described in the whitepaper. Both have the need to transfer data in the first place (note that it's 'reclaiming disk space' not 'avoiding bandwidth usage' ).

To the unanswered part of the OP's post:

Sites like blockchain.info aren't nodes. They are custom databases that take up many terabytes of space. They don't validate things (at least not completely) and often show invalid data.  What they do is unrelated to how nodes work.

Would you please stop obsessing over me?  It's creepy.

I am not your mommy.

You do not get to determine how I spend my time, you do not get to restrict what I say or believe.  I don't force anyone to do anything. You will you make me cow down and obey you through threatening me by lying to the public to try to raise a mob of idiots against me, and even if you do manage to get me killed you will not stop bitcoin, you will not convert it to the centralized coin you seem to so desperately want.

So give it a break.

QT.

There was exploit code for this latest crasher posted on the 13th, and people on the BU forums pointing to it... the "new" vulnerability was just above the old one in the control flow.

What was there to protect when almost all the reachable BU nodes were already down?

I can't think of a single piece of free software that has done this in the last decades, including highly critical software like OpenSSL. It's absurd. What BU did was also technically a license violation of some of the libraries.

I think the only thing left to debate is if they're trolling or if they're really that clueless.

The wallet in Bitcoin Core supports segwit (it's what most people use for testing!), it just doesn't use it by default.


Lie. They do not ban non-segwit nodes.


Lie.


Misleading: you can't cause much problems with tx limited to 1MB, esp due to other optimizations. Which is why nothing other than optimize was done about it before.

Segwit fixes it where it was critical to do so... and it can't be 'fixed' for old transactions without confiscating funds.


Lie. You can choose you make your transactions completely immune to third party malleability with segwit. (Malleability is a feature which you can _choose_ to use, explicitly supported-- it's only the unwanted, unpreventable, third party malleability which is an issue, and segwit eliminates that.)


Native? you mean the older kind.  Well duh people can make needlessly big transactions and pay more to do so-- that is true for any block size change.


Another lie.


Repeated lie.


Repeated lie.


Another lie.


Gee, franky1 do you get paid by the lie, or what?

The things you are calling out explicitly there, the taking coins and creating coins-- those were flaws in the very first release of Bitcoin-- many of them design flaws in the algorithms, not just software bugs.  (and several of those items aren't bugs at all-- but just updates for past softforks.)

And they were issues in the whole large program, not just a "few" changes recently made.

The resource exhaustion attacks are things that BU even refuses to fix, e.g. BU has a trivial dos of xthin via collision attack and they just won't fix it. ::shrugs:: In fact, one of the items there "A transaction that takes at least 3 minutes to verify" BU reintroduces but zillions of times worse.

BTU is the third of these absurd amateur hour clones of Bitcoin Core.  All the other ones have fizzled when the people realized that they are a lot of hard work when you can't just count on copying from Bitcoin Core.

There were larger miners than now actively opposing P2SH, not just not using it yet.  Have patience.  Bitcoin is forever, not just for next week.

This is not useful when you have more than 16 cores: it will be considerably slower, even when you do have many cores. The numbers were set based on actual measurement on 48 core systems. Overheads dominate by the point of 16 script check threads.

Similarly, increasing outbound connections just wastes resources, severely _hurts_ your latency, and will end up getting you banned by many people (same thing for the addnode maximum).  (Several people correlate hosts that are mass connecting and add them to shared ban-lists -- once your address is on these lists it won't come off for a long time).

It will also not improve your sync speed on practically any system-- sync is verification/database bound and when it isn't, it's usually reorder buffer bound.

The outbound connection increase is especially bad for miners because it increases your connection service loop time, and keeps your block in the linear regime longer where  it's just your node replicating the block rather than an exponentially growing cone.  An optimized mining setup has mining nodes with a minimal number of connections connecting to multiple border nodes and relay networks (e.g. two local border nodes, one remote border node, two reliable external nodes, and one relay network). At least one of the border nodes should  be outbound only, and at least one should accept inbound (usually best to do with remote nodes, in case it attracts dos attacks).


Most of the other settings changed can just be changed with the config file and command-line, anything with "DEFAULT" in the name.


The download window changes will likely cause database corruption if you use them with pruning. The checkblocks/checklevel changes will do effectively nothing (also controllable from the commandline).  The MAX_BLOCKS_TO_ANNOUNCE will just waste your bandwidth in the event of a huge reorg.

The timeout interval change will make you easier to DOS attack, causing you to wait longer on messages that will never be answered.

This is absolutely untrue. What you broadcast to has nothing to do with what fees you pay.  


I hope you have at least 32 GB of ram, because that is the peak memory usage these (commandline accessible) options imply on top of the nearly 2.5GB of other usage implied by  other settings.

Which permits optimizations which may produce incorrect results... mostly which shouldn't matter for Bitcoin Core, but O3 should also be just as fast and would only suffer from compiler bugs, and not directly unsafe optimizations.

Virtually all these settings which can be safely tinkered without much danger of screwing things up and without musch understanding are command-line options.

Changing things further is great-- but you should probably take some more time to understand what you're changing, especially before inviting others to use your changes!

Work on making the rescan faster would be very welcome. The largest likely improvement there would be bypassing hash validation when reading blocks from disk.