Can the Block Chain get too big and make Bitcoin unworkable?

[pyedpyper]

IMO scalability is the most pressing issue facing bitcoin at the moment.

For stats goto http://www.blockchain.info/

I agree. If it can't scale all the way ultimately (primary global currency) then we need to recognize that now. Even assuming 1 billion users (a mere 1/7 of current global population) doing only one transaction per day, that translates to 11000tps. Given Bitcoin's architecture and processing methodology that makes me nervous about whether it can actually ever grow to this kind of scale in practical terms.

Thanks for the link - very useful! 

[pyedpyper]

OK thanks for that clarification. To help me understand better: Say there are 1,000,000 people with Bitcoin accounts then how many people would have a copy of the block chain (based on present stats)?

As many as are required to keep the network decentralized. I'd say 3,000 is enough, but it's possible to make do with less.

I'm definitely getting the sense from the various answer there actually is a problem in the Bitcoin architecture i.r.o. scalability which has not yet been resolved. And this is clearly an issue. Relying on Moore's Law or hoping that a solution will "emerge" in the future when it is needed, is a dangerous way to proceed IMHO. Has anyone articulated a detailed solution yet whereby Bitcoin could scale to say 1 billion users (each making 1 transaction per day)?

Did you read the page in the wiki about scalability that was linked in a previous comment?

The solutions don't need to emerge, the solutions are known and just need to be implemented.

Does anyone know how many Bitcoin users there presently are? Or what the current average transactions per day are?

How many users is anyone's guess, depends also on whom you call a user. In the past 24 hours there were 20338 transactions.

Wow only 0.3% - that would make a difference in how I'm thinking about it.

I did read the wiki, and understood it to a degree (I'm not that tech) - but I still get the sense that it's not all worked out - and that it will be tricky to scale. It seems it's the fact that the architecture is decentralized in first place that makes for bandwidth issues. At 1 billion users and 1 transaction each per day (total 11ktps) and assuming 0.3% of people have the block chain that needs updating, that's still 11,000 writes to 3,000,000 locations each second. Sounds like a lot of traffic to me. I haven't worked out the numbers though...

What is your understanding about the practical implementation of the solutions to make for full scalability? Is it happening as we speak? Progress being made?

And a personal question: With all of your knowledge and understanding of the system, do you think it can actually scale to a billion transactions per day, with its present architecture, and not choke the entire global Internet? Do you personally think Bitcoin is truly a sustainable system that could support most of the world using it as its primary currency. Will the architecture actually scale ALL the way?

[yogi]

Bitcoin Forum > Bitcoin > Development & Technical Discussion > Scaling

?

[MoonShadow]

You guys worry too much.  No, the bitcoin network could not scale to the point that every person on Earth has a full client.  Yes, there are known pathways to deal with that, which have been generally known since Satoshi's white paper and are now very well understood.  No, they are not yet coded.  Yes, there are people who are working on those implementations at this time.

[yogi]

You guys worry too much.  No, the bitcoin network could not scale to the point that every person on Earth has a full client.  Yes, there are known pathways to deal with that, which have been generally known since Satoshi's white paper and are now very well understood.  No, they are not yet coded.  Yes, there are people who are working on those implementations at this time.

What solution are they using to solve the bandwidth and processing requirements?

[MoonShadow]

You guys worry too much.  No, the bitcoin network could not scale to the point that every person on Earth has a full client.  Yes, there are known pathways to deal with that, which have been generally known since Satoshi's white paper and are now very well understood.  No, they are not yet coded.  Yes, there are people who are working on those implementations at this time.

What solution are they using to solve the bandwidth and processing requirements?

There are several.  Start with researching the 'light' clients defined in the original white paper and work from there.  The bitcoin network is a clearing network, and was never really intended to serve everyone directly.  Most people will, eventually, be accessing the bitcoin network indirectly by some kind of service.  Online wallets are an early example of this, as transfers between users of the same service don't require that a transaction hit the bitcoin network normally.  This is exactly like a bitcoin version of paypal, except if your wallet service upsets you, you can take your business elsewere.  It's also relatively easy for such services to interlink their users so that they can credit their users when a user from another service that they have a receprocity agreement with sends one of their users funds, but settle up on a regular cycle in a collective transfer; likely once each day.  This would only be beneficial for large user bases wherein the two services have a significant number of cross transactions that can be bundled up and largely cancel each other out.

Search for BitcoinSpinner, and understand how it works.  Search also for Stratum & Electrum; understand how they interact via an overlay network.  (basicly, one stratum server holds the blockchain for hundreds of electrum clients in a manner similar to, but more standardized than, the interaction of the BitcoinSpinner main server and the android clients.)

There are also special kinds of transactions that are intended to reduce network traffic.  For example, send-to-many allows one to pay out to huge numbers of bitcoin addresses at once, for one (otherwise lower) transaction fee.  Imagine if WalMart not only started accepting bitcoin, but employees had the option of taking their wages in bitcoin each week.  WalMart could literally pay their entire workforce in a single, huge, transaction; gathering  up the many small bitcoin payments to buy thing from WalMart from customers and paying them directly out to their employees in a single action.  Compared to paying each employee with his own transaction, there is huge savings to be had here.  To some degree, send-to-many would also allow a single household to pay out all of their monthly bills in a single action as well; although the gain of a couple dozen bills in one transaction versus ten's of thousands of employees would be significantly different.  Other specialized transaction types are being considered as well, including transactions that can be signed by multiple users independently or require multiple private keys to spend; as well as scripted transactions that pay the miners to process a script of conditions.  I'm sure I can't even imagine the number of ways this could be used to reduce resource usage.  Just off the top of my head, a scripted transaction might have an external reference to the inputs and/or outputs of that transaction; permitting the actual transaction to be small but compelling the miners to fetch the rest of the transaction from WalMart's Stratum servers.  No, I don't know how external blockchain reference would work either, but they could.

For that matter, a large online wallet service could offer 'cron job' payments for when users don't need to send a transaction immediately for less than it would eventually cost to send a speedy transaction in a congested bitcoin network future.  The fees are tiny now, but we can expect that their costs will rise to meet the market as well as to compensate the miners for the increasingly large amount of network resources that they will have to pay for in order to run their mining businesses.  Right now a single transaction costs about a nickel, but a year ago it was about half a cent.  If in another year, the cost is 50 cents, we will see more people willing to defer to a delay to save on that fee; which the current network permits.  In the future, free transactions may or may not be completed in a realistic timeframe; but one way or another the fees will adjust to the demand.  If you need that transaction to clear in an hour, you are going to have to pay for the privilage.  If you can wait a day or two, you can expect to save money if you permit your wallet service to bundle your transaction up with others.  If you can complete the transaction via an external network, many will favor those networks whenever prudent.  If you are paranoid about your funds, you will still be able to maintain your own full client, but you will bear the costs yourself.

[Meni Rosenfeld]

OK thanks for that clarification. To help me understand better: Say there are 1,000,000 people with Bitcoin accounts then how many people would have a copy of the block chain (based on present stats)?

As many as are required to keep the network decentralized. I'd say 3,000 is enough, but it's possible to make do with less.

I'm definitely getting the sense from the various answer there actually is a problem in the Bitcoin architecture i.r.o. scalability which has not yet been resolved. And this is clearly an issue. Relying on Moore's Law or hoping that a solution will "emerge" in the future when it is needed, is a dangerous way to proceed IMHO. Has anyone articulated a detailed solution yet whereby Bitcoin could scale to say 1 billion users (each making 1 transaction per day)?

Did you read the page in the wiki about scalability that was linked in a previous comment?

The solutions don't need to emerge, the solutions are known and just need to be implemented.

Does anyone know how many Bitcoin users there presently are? Or what the current average transactions per day are?

How many users is anyone's guess, depends also on whom you call a user. In the past 24 hours there were 20338 transactions.

Wow only 0.3% - that would make a difference in how I'm thinking about it.

I did read the wiki, and understood it to a degree (I'm not that tech) - but I still get the sense that it's not all worked out - and that it will be tricky to scale. It seems it's the fact that the architecture is decentralized in first place that makes for bandwidth issues. At 1 billion users and 1 transaction each per day (total 11ktps) and assuming 0.3% of people have the block chain that needs updating, that's still 11,000 writes to 3,000,000 locations each second. Sounds like a lot of traffic to me. I haven't worked out the numbers though...

It doesn't scale linearly. I wouldn't be surprised if a single node could technically serve all users. But we need many nodes to keep the network healthy, robust and competitive. My guesstimate is that 3,000 is good, and that's just as true when there are 1B users as when there are 1M.

Assuming each transaction is 1KB, we're talking about 10K tps * 1KB = 10 MB/s per node, which is a high-end home internet connection now, let alone a business-grade connection in the future. It should be well within the ability of any company that receives a lot of payments and has an interest both in the health of Bitcoin and its own safety of payments.

What is your understanding about the practical implementation of the solutions to make for full scalability? Is it happening as we speak? Progress being made?

I'm more of a theory guy but there's progress. We have some lightweight clients, and there's a new construct design by etothepii that gives lightweight clients even more power than in Satoshi's original design. Also I think there's work being done with respect to pruning, which will keep storage requirements relatively low.

And a personal question: With all of your knowledge and understanding of the system, do you think it can actually scale to a billion transactions per day, with its present architecture, and not choke the entire global Internet? Do you personally think Bitcoin is truly a sustainable system that could support most of the world using it as its primary currency. Will the architecture actually scale ALL the way?

Yes, I believe even with the existing ideas and technology it can technically scale to a billion transactions per day. No doubt there will also be new ideas and hardware advances that will make things even easier. If someone comes up with a completely different architecture that is even more scalable, Bitcoin can migrate to it while keeping people's savings intact.

Also, as MoonShadow points out, it doesn't need to scale to a billion raw transactions per second. Bitcoin's role main role is to be a superior alternative to physical cash; just as there are services to overcome the deficiencies of cash, so can there be services to overcome any deficiencies of Bitcoin. But because Bitcoin is digital and has a very powerful scripting system, these services can be much more lightweight and competitive, and hence better and more efficient, than with the current system.

For example, you can have a client that stores the majority of your savings locally (preferably with some multiple signature arrangement for added safety) but automatically keeps some in an eWallet service. Whenever you make a small transaction it is done internally in the eWallet (with reciprocity with other eWallets); only for large transfers/eWallet replenishment you make a raw Bitcoin transaction. This already can cut maybe 90% transactions.

Also, solutions such as Ripple and Open Transactions can work much better with Bitcoin as their foundation, and they have much lower resource requirements for global commerce.


The real problem isn't with the feasibility, it's with the incentive structure. Running a node should be relatively easy, but it will still have costs, and not much direct gain - it's a Tragedy of the Commons problem. There could be a situation where everyone tries to freeload on the other. This can be resolved if the costs can be kept low enough that companies won't be fussy about it, with a market for node services, or with a transaction propagation incentivization schemes as in the "On Bitcoin and Red Balloons" paper.

And, the real problem isn't with the marginal resource cost of handling transactions, either. It's with the amortized cost of securing transactions with hashing. Depending on how strong the incentives of potential attackers, funding the defense efforts is going to cost a lot for the Bitcoin economy, regardless of the technical solution chosen. This I think is the main challenge, and while there are some proposed mitigating schemes, they are not popular.

[yogi]

Bitcoin is vastly more utilizable that other payment methods and I think it could mature to handle many times more transactions that other payment methods such as visa.

Essentially your solutions to scaling involve the centralization of the block-chain.

Or am I wrong?

If I am right then I think all the good things about bitcoin will be gone. I suspect that once bitcoin starts to centralize, people will move to an alternative chain.

There must be a better solution.

[Meni Rosenfeld]

Bitcoin is vastly more utilizable that other payment methods and I think it could mature to handle many times more transactions that other payment methods such as visa.

Essentially your solutions to scaling involve the centralization of the block-chain.

Or am I wrong?

If I am right then I think all the good things about bitcoin will be gone. I suspect that once bitcoin starts to centralize, people will move to an alternative chain.

There must be a better solution.

Whom is this directed at? The solutions I mentioned don't centralize Bitcoin.

By the way, Sir Gavin has recently posted about a new way to scale Bitcoin, Off-the-chain transactions. I haven't studied it yet but it looks promising.

[yogi]

The question was aimed at anyone who cared to answer it.

Ok, i see some ways of reducing network traffic but I would estimated that it would only be by a small percent.

Have I missed a method in your post?

I will look at the off-the-chain transaction thing, sounds interesting.

[paraipan]

Bitcoin is vastly more utilizable that other payment methods and I think it could mature to handle many times more transactions that other payment methods such as visa.

Essentially your solutions to scaling involve the centralization of the block-chain.

Or am I wrong?

If I am right then I think all the good things about bitcoin will be gone. I suspect that once bitcoin starts to centralize, people will move to an alternative chain.

There must be a better solution.

Whom is this directed at? The solutions I mentioned don't centralize Bitcoin.

By the way, Sir Gavin has recently posted about a new way to scale Bitcoin, Off-the-chain transactions. I haven't studied it yet but it looks promising.

Sir, you serious?

On topic, are you implying that bitcoin protocol and blockchain would scale by taking most of the transactions out of it? Or i don't get the picture...

[Meni Rosenfeld]

Ok, i see some ways of reducing network traffic but I would estimated that it would only be by a small percent.

Have I missed a method in your post?

The methods can be used to reduce network traffic as much as we want, depending on what tradeoffs we are willing to make. The local/eWallet separation for small/large transactions could easily cut 99%.

Sir, you serious?

Nope, I just like calling him that as a homage to Sir Gavin's character in King Arthur's Disasters.

On topic, are you implying that bitcoin protocol and blockchain would scale by taking most of the transactions out of it? Or i don't get the picture...

If all else fails, yes. A transaction will always be cheaper if it doesn't need to be processed by every Bitcoin node on the planet, it's just a matter of finding how to make it reasonably secure without it.

[paraipan]

...

On topic, are you implying that bitcoin protocol and blockchain would scale by taking most of the transactions out of it? Or i don't get the picture...

If all else fails, yes. A transaction will always be cheaper if it doesn't need to be processed by every Bitcoin node on the planet, it's just a matter of finding how to make it reasonably secure without it.

Ah ok, got it. I would like to see all methods exhausted before implementing that one, just my opinion.

[yogi]

Ok, so off-the-chain transactions is when Bob & Alice both use the same centralized third party but third party can't run off with money because of some multi-sig trickery.

Quiet nice, but its still centralization.

[Meni Rosenfeld]

Ok, so off-the-chain transactions is when Bob & Alice both use the same centralized third party but third party can't run off with money because of some multi-sig trickery.

Quiet nice, but its still centralization.

It's not really centralization because there can be many such third parties. The barrier to entry is low and the trust required is low.

[paraipan]

Ok, so off-the-chain transactions is when Bob & Alice both use the same centralized third party but third party can't run off with money because of some multi-sig trickery.

Quiet nice, but its still centralization.

It's not really centralization because there can be many such third parties. The barrier to entry is low and the trust required is low.

So resuming what i got from the Gavin's blog post: bitcoin developers don't know how to scale the project and Gavin proposes making off-the-blockchain tx's with the help of p2sh. Is this true?

[yogi]

So, as long as there is more than one It's not centralization?
Whats to stop them colluding?
They may not be able to run off with your money, but they can control other things such as prior restraint.

[MoonShadow]

Ok, so off-the-chain transactions is when Bob & Alice both use the same centralized third party but third party can't run off with money because of some multi-sig trickery.

Quiet nice, but its still centralization.

It's not really centralization because there can be many such third parties. The barrier to entry is low and the trust required is low.

So resuming what i got from the Gavin's blog post: bitcoin developers don't know how to scale the project and Gavin proposes making off-the-blockchain tx's with the help of p2sh. Is this true?

That's just one proposal.  Basicly it's the same as what I proposed about a 'paypal' type company serving many members and keeping a single collective wallet, but in my case one has to trust the company that you are using to not be doing anything with the funds without your concent.  In Gavin's proposal, they can't do anything with the funds without your consent, but neither can you without their's.  I think that there are better ways, but this one would work.  It would likely also permit bitcoin banking while also prohibiting fractional reserve bs. 

[MoonShadow]

So, as long as there is more than one It's not centralization?
Whats to stop them colluding?

The threat of getting caught, mostly.  If your customer base even starts to suspect you're doing strange things with their funds, one of them is going to start another server and your base is going to abandon you.

They may not be able to run off with your money, but they can control other things such as prior restraint.

I don't understand this question in this context.

[MoonShadow]

Ok, so off-the-chain transactions is when Bob & Alice both use the same centralized third party but third party can't run off with money because of some multi-sig trickery.

Quiet nice, but its still centralization.

It's a matter of degree.  Central banking today is what we're trying to avoid, and that is easily avoidable under bitcoin; no matter how many or how large a bitcoin banking cartel may grow in the future.