Sorry, I missed this question at the time but thought it was sad to have not followed up on it. -- For posterity:

Yes, however we didn't have to jointly make all the decisions at once.

We searched all the possible BCH codes of the relevant size for the ones which the most character errors (5), among those we selected the ones with the lowest false acceptance rate for just beyond 5 errors, then searched the remaining ties (codes that were equally good for character errors) to find the one that was the best at bit errors (which could guarantee detecting up to 6 arbitrarily placed bit errors).

While these searches were ongoing, we used the NIST data make a machine search check all 58905 possible selections of 4 excluded characters out of 36 which had the least internal character confusion, assuming that the input was all uppercase or lower case.  It made a decision that I wouldn't have guessed by hand-- e.g. excluding both 1 and i but actually looking at the result confirmed it was a good choice.  B can be confused a lot of ways: B looks like 8 and 3, b looks like p or lo, etc.  Other guesswork versions looked a lot worse when compared objectively.  I don't doubt you could make a somewhat different decision with a different metric or using better input data... but you could also do a lot worse, as I found just from informally picking the chacters to exclude after looking at the NIST data.

[There was an earlier version of the search before we implemented all the NIST data that wanted to exclude 'B', 'L', 'O', 'Q', which I found pretty amusing, but was ultimately relieved that we wouldn't have to convince anyone that we hadn't intentionally targeted the name of a former developers ICO mill company ...]

For most applications the HRP should be fixed by context (e.g. you don't have a reason intentionally input an arbitrary altcoin address into your bitcoin wallet), which means that any charset error (like entering 1s) can be immediately highlighted in the UI with zero false positives (I am highly dubious about auto-fixing anything since a detectable error may signify carelessness which could result in other non-detectable errors).

Finally we non-exhaustively searched the space of 32! permutations to find a permutation that mapped the remaining most likely character confusions to single bit errors.  E.g.  q and g are in the charset and differ by only a single bit.  likewise c/e, r/t, e/6, 7/l, q/p, S/5, n/m, n/h (the obviousness of these examples depends on your font...), and many other confusable pairs. This mapping means that likely errors for users get mapped to fewer bit errors and since the error detecting code was chosen to have improved detection for single bit errors this means improved detection for errors users are likely to make.   This is an optimization with only a small effect, but it is one with fairly low marginal cost since virtually any encode/decode implementation would already use a table due to the missing characters-- just as existing base58 (and base64) (de)coders do. Given that a table is being used, the choice of the order is essentially arbitrary so we were free to choose it to optimize the performance of the error detecting code. I doubt an implementation that includes signing (taking tens of kb of code) would care much about saving a few bytes bytes by using a slow branchy alternative rather than just a simple table in read only memory.

The code searches and permutation searches were computationally expensive but we didn't have to do the product of their efforts, only the sum because they could be searched independently and combined (and the charset search wasn't terribly expensive).  Pieter has code for all these searches in his ezbase32 repository...

Bitmain was producing them minutes after a block change, and was producing-- prior to the blow up about covert AB-- upwards of 10% empty.

Not at all, however Jihan directly claimed as much to me.  I think it was largely but not entirely bluster, they very clearly ran several that in public they claimed were "independent".


I think you may misunderstand what is being claimed there: It takes time to find the collisions needed for covert AB.   When a block change happens, what do you do when you don't have a collision?   You mine an empty block where the collision has been precomputed, as that takes no time at all.   Once you find a collision with transactions you switch to that.  Sometimes finding a collision takes a rather long time, because you get unlucky.

I agree that it's speculative: the extremely high rate of very late empty blocks could also be attributed to poorly run software and there is plenty of evidence of that.  Bitmain controlled pools producing blocks that were invalid because of out of order transactions, where the tx order is what bitcoind would produce but swapped at some interior node in the hash tree is a lot harder to explain.

Fair point but to counter: At one point copay leaked users private keys, there is little that can be done about sufficiently severe incompetence much less malice except encourage people to not use that software/service.

Agreed there too, but at least the subject of this thread can't really do anything about maliciously bad wallets.   There are 1001 ways to have the same effect-- they could still overpay and mine them themselves (overpayment for profit hurrah), they could simply make the wallets also spend additional long unconfirmed chain inputs to slow them down etc.

I do agree that things like these adversarial parties masquerading as supporting bitcoin is a problem, but trying to do somethin about overpayment only addresses a symptom at best, it if could work.


+1 Agreed here, sorry for perhaps overstating my case.

> but a significant chunk of work and effort and effort recently seems to be pointed at the direction of bending and twisting the Bitcoin protocol so that it works better with one specific implementation of an offchain system

Work on what by whom and to the exclusion of what, exactly?  Please provide hyperlinks.

It's kind of exhausting to deal with what feels like pithy narratives being used as a substitute for actual reality. We can avoid anyone feeling that way by being painstakingly concrete instead of vague.

You provided two links:

BIP 118, which is over two years old (Feb 2017), has no activity in Bitcoin Core right now that I'm aware of, and was written by a lightning implementer. Lightning implementer interested in things that are useful for lightning isn't a newsflash.

And PR12254 (Apr 2018) provided for BIP158 filters doesn't do anything for lightning as is, and was developed by someone who isn't a project regular.  Much of the interest in that is making wallet rescan fast [search this for BIP-158]., which that PR implements most of the necessary components for... and a network facing version is essentially just a replacement for BIP37 which is slightly less of a total privacy disaster (and DOS vulnerability disaster). There isn't anything lightning specific about that...

I don't think these support your case, if anything they contradict it-- or at least the fact that these were your only examples does since they don't show a high level of attention or interest.

Lightning is an application of Bitcoin smart contracts to make Bitcoin payments more efficient, the fact that it could be adapted to another system is simply a product of other systems copying Bitcoin functionality... the same could be said for any technical functionality.  Any application seeing actual use is obviously going to get some engineering attention, since a real application with articulable tradeoffs generally trumps conjecture.

Other people overpaying is none of your business. Only the fact that they paid more than you has any influence on you at all (they get confirmed ahead of you).

Sometimes people make mistakes, sure, but you shouldn't assume that everyone acting differently from you is making a mistake... or if they are, that its any of your concern.

Nothing I've seen discussed in this thread seems interesting.  If you'd like to make a case that some people are paying higher than you when they'd really rather wait longer than you for confirmation-- that might be interesting. Regardless, trying to fix prices on nodes will just make the network work less well and encourage centeralization when people then realize they need to give their transactions directly to miners.

Yes, I'm referring to later, not so kind of you to sit quietly amid all those claims that I was lying... when you could have instead happily bragged that you knew all along.   Oh well! I forgive you.

AFAIK I never posted about it on bitcointalk at all... perhaps if I had, I might have posted on the S9 thread. (although most of my testing was actually on an R4 and parts from R4s ...)

They claim this, but the claim seems kind of absurd. The whole point of covert is that it's cover, it's nearly impossible to detect.

Consider though: at great expense they developed an significant performance optimization and baked it into their chips, huge numbers of which they ran themselves ... and then kept it secret but ... just didn't use it?  Why? because they like giving money away to the power company?

There is evidence that bitmain was asicboosting: among other things (Edit: e.g. as frodo mentions, the huge number of empty blocks, which have more or less magically gone away), they mined a small number of blocks that were invalid because transactions were out of dependency order. This would be a pretty difficult mistake to make-- unless you were grinding blocks by swapping around transaction order.  It's just not conclusive proof, unfortunately short of some massive internal document leak conclusive proof is likely impossible.

[Edit: The empty block part is really strong because unlike any other case you can compute the collisions for the empty blocks arbitrarily far in advance, so even if takes you tens of seconds to find one on the FPGA you can keep mining until one is found.]

Of course, there is also no reason to think that bitmain was the only booster.  I think its very interesting that the bitmain funded developers of bcash pushed surprisingly hard for their "canonical transaction order" change shortly after bitmain published overt AB support for their existing hardware... Might well be that once covert boosting wasn't an advantage it was best for overt supporters kill it on bcash in order to disrupt any miners whos hardware was less flexible than Bitmain's. (Because Bitmain used a rather overpowered FPGA for control they could switch between covert/overt/no-ab with just a firmware change... it would have been more power efficient though to have a separate asic for collision finding, but such a design might result in devices that lose half their hashrate when used with a mandatory transaction order or otherwise have to only mine empty blocks...)


Wow. No, I completely hadn't seen that-- I mean it was buried deep in a thread about a vendor who's hardware I didn't have at the time, and didn't mention asicboost in the post... all the details were in a pdf... I wouldn't be surprised if almost no one ever read it.  Too bad, because that was a pretty big revelation. Kudos to you.   It would have saved me an astonishing amount of time if I'd been aware of it.  (in particular figuring out the how to use the multi-midstate with no actual docs on the chip but simply picking through the software and bus captures with unreliable information extracted from the chip design was a big chore).  

As an aside, given that you knew this-- how come when I announced it and so many people were saying that I was lying about it being there, you didn't step up to point it out?

inb4 50k "2x".

Virtually every mining device is a general purpose computer running linux plus a fixed function accelerator.  Is this so different than, say, an android phone which has several fixed function accelerators (like the speech detection 'hello google' hardware, wifi modem, etc.)?

I hope everyone who gives a darn about Bitcoin avoids doing business with Bitmain in the future. Bitmain has been a long standing bad actor in the cryptocurrency ecosystem. We shouldn't forget this just because they replaced their CEO.  As far as I know they have done nothing to undo the damage they've caused or otherwise make amends.  We can see by their continued open source license violations in their products that they haven't changed.

I will award merit to every post that makes a compelling statement against buying from bitmain.

Poe's law man, poe's law.

The bamboozled are going to feel pretty cheated when they figure out that I'm just some nobody that got targeted because of my lifelong promotion of freedom and privacy technology. It's probably no accident that "hacking team" was sending out newsletter claiming that bitcoin could become an establishment threat if got improved privacy and then I become a number one target, complete with an over the top disinformation laden NYT hit piece, short after after I published a design and implementation of Confidential Transactions.  

They're claiming they will be able to do payment channels and lightning, but until they implement the rest of their segwit-under-another-name hash/txid split, they likely won't. Schnorr plus the aborted BIP62 implementation they copied from core aren't enough.

https://www.reddit.com/r/btc/comments/axza7b/id_love_to_get_an_update_on_what_is_happening/ehzhki0/?context=3

(turns out that without using minisketch they can't get within an order of magnitude of the known best case for almost pointless marginal optimizations... and it's kinda hard to make a case for how much better you are when all your examples are copying the people you're trying to say you're better than. )

Anonymous account shows up relentlessly promoting a tool that uses custom non-standard cryptography to generate keys for users-- e.g. showed up on #bitcoin and within a week had shared the link 35 times. The claims on the page are outright false (e.g. saying that its quantum safe in ways bip32 are not) meaning that the author is either too incompetent to be writing cryptographic software for public use or is intentionally lying to snow users into using it.

I would strongly urge people to not use this software.

The behaviour you are describing is the perfectly normal behaviour of a new node coming online.

No, the version string is not used to ban peers, in fact taking any programmatic action on it is expressly specified as "strongly unrecommended and extremely bad form."

Your version string example is probably some brain damaged scamcoin client which failed to change the port and protocol magic and it probably got banned for sending invalid blocks.

BIPs are just a publication numbering scheme. They do not indicate that a proposal has been adopted anywhere, or even that it isn't extremely and dangerously broken.

BIP64 IIRC had an immediate "take out all nodes" grade vulnerability, so it wasn't adopted by Bitcoin.

Allowing random completely unverifyable access queries to a node's UTXO set was never shown to be particularly useful in any case (e.g. the application given for it required its own server anyways, so its server could just answer its queries), commits a node to making available data which might be eliminated in the future, and risked creating bad incentives (E.g. making it attractive to use Bitcoin's UTXO as a file storage service)... which goes to explain why (IIRC) no one picked up that effort got a version with the vulnerability fixed adopted.

Don't use closed source wallets.

If anything this incident increases my (nearly zero) estimate of this wallet's security: Someone looked and found at least at the moment it was sending the key material only to Google. That is more secure than should have been expected.

Don't use closed source wallets.

Don't use wallets that support a zillion different cryptocurrencies (just supporting one securely is a task too hard for basically anyone to get right...).

Don't used closed source wallets.

I'm sorry to hear about the OPs loss.

Don't used closed source wallets.

If anyone needed yet another example why the world of cryptocurrency is a cesspool... and why thoughtful people usually run for the hills-- and leave tech forums looking like ghost-towns with little interesting traffic left, well then I guess one has been provided.  I thought I gave a brief but reasoned point by point argument that made the case both from first principles and then again drawing from actual results. I might have expected a response in kind but instead, the response is a litany of abusive invective and personal attacks making it clear that my time-- and everyone elses here-- is just being wasted.

You dare not challenge the strongly held prejudice of the forum warrior. You just don't.

I originally thought this thread was likely to be a train wreak and didn't bother commenting. But I was in a good mood this evening... some day I'll learn.

On the surface at least progpow looks like straight up corruption. If ethereum users won't fall for it, there isn't a chance in hell bitcoin users would.

It appears very unlikely that "asic resistance" is even possible, but putting that aside: there is also little evidence that it would be desirable.  Absolutely asic mining has created a multitude of problems but something having problems doesn't suggest that any particular alternative is better... the alternatives to common practices with known issues are often much worse.

In particular, essentially by definition every "asic resistant" approach I've seen shifts costs to from highly commodity energy (the overwhelming majority of the lifetime cost of a sha256 mining device is power) to highly illiquid upfront manufacturing. (e.g. bitmain s9 last year hit power cost = purchase cost at about 3mo of operation) The economic arguments for mining potentially producing a secure and decentralized system depend on an ongoing cost-- if you reject those models you end up with something more like a quorum of trusted signers (e.g. "bobchain").

Manufacturing centric costs also seem highly prone to even stronger monopoly pressures both due to  government granted artificial monopolies on manufacturing techniques as well as natural monopolies arising from first mover advantage for a highly amortized R&D heavy cost model compared to a commodity energy cost model.

This speculation about the trade-offs also seems to be supported by prior efforts at "asic resistance"  (including litecoin, ethereum's work function, zcash's work function...):  In spite of many hypes and claims they have not actually turned out to be asic resistant at all and have in actuality resulted in a less diverse ecosystem than we've seen for Bitcoin. And enormous windfalls for secret operations...

Perhaps even more fundamentally: every commodity computing device is itself already an "asic"-- just one with extraneous parts (and a lot of patents required to build it, the "commodity" is something of a misnomer as a result...).  Simply taking an existing design and stripping off the extraneous area and power wasting I/O components alone is usually enough to make standard parts uncompetitive. After all, mining by its very design seeks to result in near zero returns for the _most_ efficient participants-- you don't have to be much less efficient to be hardly earning at all (or even operating at a loss). 2x the efficiency can mean 20x the profits, if the market is actually competitive. This is especially true because mining devices operate in a very different reliability region than other computing devices: it's typical for a mining device to get the best good-put at a percent or even more incorrect computations because the speedup/power-reduction more than pays for the wasted work. This sort of operation wouldn't be acceptable for virtually any other application, so a lot can be gained simply by designing for that alone.  In practice, every attempt so far (though much hyped, lauded, backed by academic research, and what not) has actually had orders of magnitude more efficiency gains than you'd get from stripping down a generic part. ... but even just the stripped part is likely enough to fatally wound the idea on its own... Before considering the IPR monopoly, amortization-centralization, risk of accidental trapdoor optimizations, or the speedups from customized architectures.

To me it seems that the idea doesn't work in theory and it doesn't work in practice.  It keeps re-occuring because the constant influx of suckers who will buy into things that merely claim to solve things which intuitively sound problematic, even if the solutions are barely credible (or even if they were tried previously and already failed!). There is little interest in actually reasoning out the tradeoffs, e.g. what makes for a good work function and a vested interest to not bother and keep churning out new flavours of snake oil-- because no matter how you want to cut it the picture is not that rosy for these efforts-- at best you might be able to say that with a lot of careful work you got a DIFFERENT set of tradeoffs that were potentially useful something with its own benefits but own serious downsides, far from the panacea that people want to sell-- and that isn't good for shilling the latest scamcoin.

Ineffectual because "crypto" conference have long been thoroughly overrun with scammers. Everyone who takes a strongly principled position on not attending events with scammer sponsors or speakers are already rejecting almost all events in this space.

Scammers just get a much larger marginal return from promotional activities like conference speaking/sponsorships.  Conferences are almost all run as money making enterprises, so the fact that they're saturated with scammers is unsurprising.

NIT: 64 bytes instead of 72 bytes.


So multiple concepts get confused here, so I can't tell exactly what you're talking about.    

There is signature aggregation which combines signatures from multiple inputs (but probably just one transaction) in to one,  or efficient threshold signatures which allows many signers to produce a single signature for a single input.

Both make signatures in transactions much smaller, so don't justify any change in how weight is computed.

This directly makes transactions much smaller, so again, no need to change how weight works.

Yes, it makes the cold cache catchup case spend half the time in signature validation. (non-catchup doesn't do validation in the critical path due to caching! ---  the small batching you can do as txn come in doesn't get much speedup)

The eventual speedup from batching (and the speedup we achieved from caching in the non-catchup case) was part of the justification for having witness data have lower weight to begin with.

With the exception of batching the other advantages you cite already result  in lower weight (in the cross input case, much lower weight).  So they're naturally already awarded.

Different users experience different pain points, some are cpu limited, some are bandwidth limited, some are power limited, some are storage limited. Many are some mixture of multiple of these.  Because of this no single weight formula can be optimal.   What really matters is that it sets the incentives in the right general direction, in order to break ties in the favour of public interest.

Generally we can assume that in the long run most users are going to do whatever is most cost effective for them. If foobar signatures were a LOT better for the network it would still be sufficient that they be only slightly better for the end user, even if making them much better would be justifiable under some cost model... even a little better will get them made a default.  Some users will have different motivations and make different choices, but a small number of exceptions is mostly irrelevant for the overall network health.  This is important, because a perfect balance isn't possible.  E.g. with weight, you could easily argue that an 8:1 ratio or a 16:1 ratio would have been better-- but a higher ratio means a LOT worse worst-case bandwidth, and so wouldn't be  good trade-off for those users who are bandwidth limited.   The fact that the only "direction of incentive" needs to be right, not so much the magnitude, means its possible to make compromises that give good results for everyone without screwing over some cost models.