BTC and Visa scale processing...doesn't seem possible

[Fuserleer]

Hi BTC techies,

Firstly, I rate Bitcoin very highly, it's an amazing first step to a wonderful vision that Satoshi had.  However, there are certain aspects/limitations of the system, that to me, suggest it will never be any more than a small alternative to Fiat.

Some of you may of heard of my project eMunie, which for all intents and purposes is a more advanced direct competitor, and my goal is for it to far surpass Bitcoins reach...so you may call me biased.  That said, I am not here to attack, but to point out and discuss some legitimate issues, these may of been raised before, but I think I can add some further points to the argument.

Specifically I have concerns about BitCoins scalability when it comes down to transaction processing volume.  At present, the entire BTC network has the capacity to clear about 140k-160k transactions per day (assuming a average transaction size of ~1k), not too bad.  That right now is of course sufficient, but what if tomorrow, you could pay with BitCoin in all the small, mom & pop businesses in the US alone....that then causes a serious issue, that sends your processing requirement through the roof.

Note this has nothing to do with hash power at all, but the limits built into the system currently.

The problem

Visa reportedly processes around 300M transactions a day on average, quite a lot! If you break that down into the tx p/s we get around 3500 per second.  Now lets say that BTC manages to acquire just 1% of that volume, that then is 3M transactions per day.  If the block time stays at 10minutes, then at 144 blocks per day each block would have ~28k transactions.  if the average transaction size was still 1kb, then each block would consume ~25MB    

Your daily bandwidth requirements, disregarding storage completely, just to keep up would be almost 3GB, and you would need at minimum 50kb (bytes!) per second of constant bandwidth just to have the data alone, without processing.  I'm not even bringing into the equation the transactions that are broadcast around the network before they are even IN a block, all 35 per second of them...(that is another 35kb of bandwidth btw, constantly).

Now a lot of you are thinking, yeah so, 3GB per day is nothing, HD space is cheap, internet connections are faster than that etc etc, but you are missing the point.  If BTC is adopted in the kind of places I mention, then the PC's these Joe public are using are likely old, underpowered boxes, unmaintained and probably not on the fastest of connections.  Furthermore, they are likely not left on all the time so they wont be in sync constantly.

Perhaps Joe decides to load up the BTC client every few days to check his balance, wham, he's got to sit there for hours just waiting for the client to catch up....Joe stops using Bitcoin.

Light clients

These will help BTC short term, perhaps downloading only the essential blocks that are needed to verify and send un-spents, but even then, at block sizes of 28MB, if you need only 1 or 2 transactions in that block, that's a whole lot of downloading for only a few KB of essential data needed.  If local clients don't download block information at all, and just query "chain servers" for the transactions, then sure that is moving away from the decentralized nature of the system somewhat as then you have a much smaller subset of known places where the system can be attacked.

Online wallets

Will work, for a while, until the inevitable happens and a hack occurs.  With banks your money is insured against loss, with BTC this isn't the case, and I highly doubt that any of the online wallet providers could afford to reimburse 1% worth of Visa scale trade should a major breach happen.  Once that happens, Joe public won't want to even know about BTC anymore and it's back to Crypto lovers, speculators and general craziness.

I personally don't see an easy, elegant fix for these potential problems that are ahead for Bitcoin should it become accepted even at these small scale (in the grand scheme of things) numbers as the problems stem from the systems deep rooted design.

These are issues that we have covered from the initial design stages for eMunie and have solutions for, but I fear that for BitCoin the changes needed to be able to handle the above may be too disruptive to implement into the system.  That is a real shame as right now the only real contender to pull off Satoshi's vision IS Bitcoin, but from what I can see, it just doesn't have the legs in it's current design.

Hopefully I won't get pitchforked here and we can have some nice discussions on options and I'll chip in with some of our solutions over here if the trolls stay at bay

[DeathAndTaxes]

A couple of quick rebutals:

1) Average tx is more like 400 bytes not 1 KB.  So you can cut all your numbers by at least half. The way the network handles block announcements is relatively inefficient as the  network grows it is likely there will be more optimizations.  

2) Under a high tx volume scenario most casual users would use lite clients.  That is simply a reality.  Full nodes would be merchants, idealists/hobbyists, miners, service providers, etc.  Essentially people willing to pay the cost of running a full node.  200 kbps of bandwidth is relatively little and even at 45 GB per month in storage even an old 1 TB drive is good for 20 years of blockchain.   The cost won't be zero but it also won't be excessively high. 

3) Even if Bitcoin grows that large it won't be overnight.  Moore's law is still alive and well.  Lets say Bitcoin grows from  ~0.6 tps to ~35 tps we are still taking about a ~60x increase in usage.   Under what timeframe do you think that could occur?   If adoption growth is 50% sustained annually for a decade we would roughly reach that point.  So the costs under a 35tps scenario will be based on the economics in 2023 not 2013.   We can't look into the future but we can look into the past.   Compare bandwidth, storage, and CPU power with equipment today and equipment 10 years ago.   10 years ago most of the planet was on dialup and 200 to 500 kbps would considered "high speed".  Today a large portion of the population has access to speeds of 5 Mbps or more.  In 2003 an entry level hard drive was 80 GB and cost about $1 per GB.  Today 1 TB can be had for the same price.  So you should be comparing the operating cost to hardware in a decade from now.   Storage will probably be on the order of $1 per TB.   A high speed connection to the home will probably be on the order of tens of MBps for most of the population and many will be running hundreds of Mbps or even Gigbit links.  Even poorer areas of the world will have bandwidth higher than is available today for lower cost.

4) For those who don't have direct access to those kinds of resources (as modest as it will be in 2023 terms) there is always the option of running a full node on a hosted server.  Hosting is relatively cheap.  What do you think a server with 2TB of storage, 1Mbps of bandwidth (~50 GB per month), a couple low end processor cores, and 16GB of ram costs today?  In a decade it will be more like the cheapest hosting plan available. $10 per VPS?   

[gmaxwell]

but even then, at block sizes of 28MB, if you need only 1 or 2 transactions in that block, that's a whole lot of downloading for only a few KB of essential data needed

You've misunderstood how liteclients work:

Sending a single transaction from a block with 28k transactions requires only 480 bytes (ceil(log2(28000))*32) plus the size of the transaction itself because of the use of hash trees. Sending multiple transactions has lower overhead per transaction.

Beyond that you're making some odd comparisons.  Bitcoin is a _digital currency_ and a payment network. Visa is just a payment network.  You can, if you like, use other payment networks to transact in the Bitcoin currency— even using visa itself is technically possible. There are a lot of arguments as to why Bitcoins global consensus model doesn't make it a fantastic payment network, especially for small transactions... and you've provided some. But the global decentralized consensus model is necessary to obtain an inflation free asset we can trust... fortunately, you can have your cake and eat it to if you don't insist on doing all transactions directly in the Bitcoin payment network.

Online wallets— as much as I am concerned about the adverse impact of moving too many users off the Bitcoin network and potentially undermining our decentralized security— tend to have a stronger security model than you're crediting them for: they leave the cryptographic signing keys with the user, so a hack of the site can't simply run off with all the funds. Certainly Bitcoin pushes the envelope of computer security in new ways, but I don't think that anyone considers these challenges unsolvable.  Take a look at the personal hardware-security-modules ("hardware wallets") the Bitcoin community is innovating with these days.

[DeathAndTaxes]

Yeah I totally missed the claim that lite clients need the whole block.  There is a reason Satoshi designed blocks to hold tx in a merkle trees.  If the plan was simply to download entire blocks (and never prune blocks) then no merkle tree is needed.  You could just simply take a single hash of the sequence of transactions and be done with it.

[Fuserleer]

Good post.

1) Ok, 400 bytes, I didn't actually research the average, Ive seen a few of my own BTC transactions hovering around the 6-800 mark and I have only a few un-spents hence the assumption of 1Kb average.

2) Fairish points, but that 1TB drive is only good for 20 years IF BTC doesn't surpass 1% of Visa trade scale.  If it hits 2% then its good for 10 years and all the bandwidth requirements for full nodes etc double, 4% is 5 years.  What if BTC gets to 10%, amazing achievement, but that puts hellish strain on even the full nodes at that point.

3) How do you know it won't be?  A lot of you guys on here like to quote "It wont be overnight, we have time" and that is a futile argument IMO, look at the transaction growth back in Feb-Apr with all the media attention, it doubled...overnight almost!  What if eBay takes on BTC, or some other large internet corporation, immediately you have access to millions of people that still probably have no idea what BTC is, but the fact that it's endorsed almost by someone like eBay, a very large portion of that new user base will try it out.  There is your overnight...

4) I know what a server costs, but again you are using the time argument.  It would be a real shame to get caught out on that and not be prepared....as the saying goes "Failing to plan is planning to fail"

[Fuserleer]

but even then, at block sizes of 28MB, if you need only 1 or 2 transactions in that block, that's a whole lot of downloading for only a few KB of essential data needed

You've misunderstood how liteclients work:

Sending a single transaction from a block with 28k transactions requires only 480 bytes (ceil(log2(28000))*32) plus the size of the transaction itself because of the use of hash trees. Sending multiple transactions has lower overhead per transaction.

Beyond that you're making some odd comparisons.  Bitcoin is a _digital currency_ and a payment network. Visa is just a payment network.  You can, if you like, use other payment networks to transact in the Bitcoin currency— even using visa itself is technically possible. There are a lot of arguments as to why Bitcoins global consensus model doesn't make it a fantastic payment network, especially for small transactions... and you've provided some. But the global decentralized consensus model is necessary to obtain an inflation free asset we can trust... fortunately, you can have your cake and eat it to if you don't insist on doing all transactions directly in the Bitcoin payment network.

Online wallets— as much as I am concerned about the adverse impact of moving too many users off the Bitcoin network and potentially undermining our decentralized security— tend to have a stronger security model than you're crediting them for: they leave the cryptographic signing keys with the user, so a hack of the site can't simply run off with all the funds. Certainly Bitcoin pushes the envelope of computer security in new ways, but I don't think that anyone considers these challenges unsolvable.  Take a look at the personal hardware-security-modules ("hardware wallets") the Bitcoin community is innovating with these days.

I understand how the merkle tree's work but I will honestly admit I haven't done any actual development against the BTC RPC etc or inspected the code in any deep fashion. 

From your post I'm assuming it is possible to query just transactions and retrieve those after inspecting the merkle tree of the block?  The actual block itself is not required for any client side verification procedure? 

If that's the case then my 2nd point for the LiteClients can probably be discarded.

[gmaxwell]

It would be a real shame to get caught out on that and not be prepared....as the saying goes "Failing to plan is planning to fail"

Growth past 1MB/block can't happen without the consent of all Bitcoin users: The network rules prevent blocks from being larger than 1MB specifically because very large blocks could constitute an attack which makes Bitcoin too costly to run for its uses and thus robs it of its centralization.

So really— it can't happen overnight— and any in chain transaction growth will exist in dynamic tension between off chain and non-bitcoin alternatives.

From your post I'm assuming it is possible to query just transactions and retrieve those after inspecting the merkle tree of the block?  The actual block itself is not required for any client side verification procedure? 

Thats the whole point of the design.

[DeathAndTaxes]

10 years was just an example.  The larger point is Moore's law is alive and well.   Even 100% VISA scale volume is possible today and will get cheaper over time.  Looking at a two month period of time is kinda silly.  Lets look at it on a larger scale.  Tx volume has roughly doubled in the past year.  Growth is high but it isn't 10,000% per week high.   With higher volume comes more interest, more investment.   It is entirely possible someday (think existing ASIC companies looking for new markets) someone make a dedicated tx verification processor to offload the work from the CPU and reduce power consumption 90% or more.  

Lastly there is no such thing as "getting caught".  Miners have to run full nodes.   If the cost of a full node is too high they will raise the min fee they require for tx and thus slow growth.   In other words if on blockchain volume will never exceed what is possible as it pushes up to that limit the marginal cost will rise and that will drive innovation.   For example if on blockchain tx are expensive then off blockchain tx makes more sense.  If it requires 8 high end cores to verify tx volume then the potential profit for a dedicated ASIC is higher then when a low end atom processor can do it.


As for growth being faster I am going to go out on a limb and say it won't happen.  Two years ago I said mainstream adoption of any truly revolutionary technology is 10 to 20 years.   Bitcoin is no different.  We are likely a decade or more from mainstream adoption.  Even PayPal (~50 tps) didn't get to "PayPal" levels overnight.  There are a lot of challenges to cryptocurrency.  The first is the legal uncertainty.  FinCEN "guidance" would be laughable if it didn't have billiion dollar implications.  It is such complete and utter nonsense that compliance by the letter of the law is simply absurd.  The law moves at the speed of government.  The timeframe for all that getting settled out all over the world is measured in decades.  Another area is coin security.  It needs to be easier for braindead stupid users to remain safe.  The reality is that as Bitcoin expands the average "tech saviness" of users will decline.  Until Bitcoin is easy and safe for non crypto-nerds it will slow adoption.  The first gen hardware wallets are just the begining.  Nothing is perfect in round one.  It will take years to build out "user proof" systems.   Another area is the general level of professionalism of service providers needs to increase.   MtGox going down due to trading load for weeks at a time?  Unable to get money out of exchanges in a timely manner?  Operators disappearing overnight and users losing all funds?   Even PayPal scale volume (50 tps) means hundreds of billions of dollars flowing into the Bitcoin ecosystem.  You think Amazon is going do a quarter billion dollars in business on some unknown exchange in Uganda (or bumfuck nowhere)?  Hardly.  All that investment and growth will also be measured in years.

Simple version is the law of large numbers will occur in Bitcoin just as it does in any other system.  Bitcoin going from to $2B in money supply from $1B will be harder than it going from $0.5B to $1B.  Each doubling means larger and larger investment, a wider pool of users, and higher expectations from investors, users, merchants, etc. 

If Bitcoin becomes mainstream it will be part of a 10 to 20 year process which began when Satoshi published the gensis block.  It won't be next week, it won't be next month, it won't even be next year.  The radio didn't, the TV didn't, the internet didn't, credit cards didn't, cell phones didn't, personal computers didn't, etc.  Mainstream adoption takes time.

[Fuserleer]

Indeed, as I said I haven't done much with BTC past the high level fundamentals and I assumed that the block was also required client side.  

eMunie being a totally different design, the "blocks" themselves are fundamental to the operation of the system, but they are a lot smaller and more frequent than the schedule that BTC uses.

[gmaxwell]

Miners have to run full nodes.   If the cost of a full node is too high they will raise the min fee they require for tx and thus slow growth.

Well, careful here.

Miners are directly paid to run nodes, but most Bitcoin "miners" are not miners for the purpose of this discussion, they are just hashers remotely selling computing services and not running a full node at all.  The majority of all mining is run by a small number of super-large miners and 2% of 50% of the hashpower is currently about $4000/day ... that can justify buying a lot of computing power to support that income. But if only people with that kind of funding could justify running full nodes, Bitcoin's decentralization would be lost. (and no check/balance would exist to prevent those big miners from inflating the currency). Arguably, its in these large node's commercial best interest to intentionally inflate the cost of maintaining a node since they can easily support it and doing so would drive smaller competition (esp decentralized alternatives like P2Pool) out.

So I don't think you can argue that miners are _obviously_ an effective control on the block size. Increases in tx processing costs are drivers to increased centralization which risks undermining Bitcoin's value and reason for existing. (Because centralizing tx processing _fundamentally_ increases its efficiency)

(And FWIW, my laptop with pieter's libsecp256k1 code can validate almost 50k signatures per second... needing 8 cores to keep up doesn't seem like a real concern even under crazy growth assumptions, bandwidth appears to be more of a constraint)

[gmaxwell]

Indeed, as I said I haven't done much with BTC past the high level fundamentals and I assumed that the block was also required client side.  
eMunie being a totally different design, the "blocks" themselves are fundamental to the operation of the system, but they are a lot smaller and more frequent than the schedule that BTC uses.
eMunie, the next step in cryptographic currency.

Web - http://emunie.com  Forums - http://forum.emunie.com

Perhaps you should hold off on promoting something as "the next step" when you don't strongly understand the current state of the art?

[Fuserleer]

I see what you are saying, but personally I would rather be prepared now if possible rather than relying on time as a deciding factor.  "Tomorrow syndrome" can be the demise of something very easily, the success of BitCoin is dependent on 3rd parties...all you need is ONE highly influencing 3rd party to endorse it, and BTC is kicked into orbit real fast.

The higher the cost of maintaining the block chain the more centralized the system will become, until you are left with only a handful of nodes actually providing the full block chain due to cost.  That's counter to what the system is supposed to be surely?

Plus, If the demand requirements of BTC exceed what is possible then yes innovation will ensue to a degree, but that will also be counter productive to BTC adoption overall as the system can no supply the service that meets or exceeds demand and so will faulter.

10 years was just an example.  The larger point is Moore's law is alive and well.   Even 100% VISA scale volume is possible today and will get cheaper over time.  Looking at a two month period of time is kinda silly.  Lets look at it on a larger scale.  Tx volume has roughly doubled in the past year.  Growth is high but it isn't 10,000% per week high.   With higher volume comes more interest, more investment.   It is entirely possible someday (think existing ASIC companies looking for new markets) someone make a dedicated tx verification processor to offload the work from the CPU and reduce power consumption 90% or more.   

Lastly there is no such thing as "getting caught".  Miners have to run full nodes.   If the cost of a full node is too high they will raise the min fee they require for tx and thus slow growth.   In other words if on blockchain volume will never exceed what is possible as it pushes up to that limit the marginal cost will rise and that will drive innovation.   For example if on blockchain tx are expensive then off blockchain tx makes more sense.  If it requires 8 high end cores to verify tx volume then the potential profit for a dedicated ASIC is higher then when a low end atom processor can do it.

[Fuserleer]

Perhaps "state of the art" wasn't worth my time due to fairly obvious flaws visible from the outset.  Why research something doomed?

Also, having the clients, even light ones, download the block is a pretty bullet proof way of ensuring no double spends have occurred instead of having light clients blindly trust whoever they are getting the block from.  That is why I assumed that blocks were required always, not because of my inability to understand "state of the art"

Indeed, as I said I haven't done much with BTC past the high level fundamentals and I assumed that the block was also required client side.  
eMunie being a totally different design, the "blocks" themselves are fundamental to the operation of the system, but they are a lot smaller and more frequent than the schedule that BTC uses.

Perhaps you should hold off on promoting something as "the next step" when you don't strongly understand the current state of the art?

[DeathAndTaxes]

Perhaps "state of the art" wasn't worth my time due to fairly obvious flaws visible from the outset.  Why research something doomed?

I see I stupidly wasted time.  How can you "know" something is doomed when you didn't even know how a fundamental aspect SPV node verification process works?