on average, how much HD space does bitcoin-qt consume per day

[BTCisthefuture]

About how much HD space does bitcoin-qt eat up every day in order to download blockchain info.  I keep seeing my HD get less and less space and I'm worried at some point in the future I simply wont be able to run bitcoin-qt without buying another HD.

[1713536112]

1713536112

[Mike Christ]

It will forever increase in size.

With that said, you might want to try Electrum instead.  It doesn't have to download the blockchain; perfect for lappies.

[BTCisthefuture]

It will forever increase in size.

With that said, you might want to try Electrum instead.  It doesn't have to download the blockchain; perfect for lappies.

Right I know it forever increases in size...I'm just trying to get a rough idea of how much.  I suppose I could not be lazy and just monitor my HD space everyday and take a guess.  Figured it was quicker on here though.

Thanks for the advice on Electrum, I'm going to read up on that now since I'm also using a laptop.

[caveden]

Since 0.8 it became perfectly possible to run Bitcoin with its data dir in an USB disk, so I'm currently doing that to save space in my main disk (which is quite small).

Concerning your question, It's currently growing at no more than 250Kb per block, with ~144 blocks per day that should be at most ~36Mb of daily growth (it's actually less than that, not every block's hitting 250Kb). After May 15th, that growth rate might rise. It will take a hard-fork change for it to rise above 144Mb per day though, due to the 1Mb cap.

You can follow the total size here: http://blockchain.info/charts/blocks-size

[farlack]

Since 0.8 it became perfectly possible to run Bitcoin with its data dir in an USB disk, so I'm currently doing that to save space in my main disk (which is quite small).

Concerning your question, It's currently growing at no more than 250Kb per block, with ~144 blocks per day that should be at most ~36Mb of daily growth (it's actually less than that, not every block's hitting 250Kb). After May 15th, that growth rate might rise. It will take a hard-fork change for it to rise above 144Mb per day though, due to the 1Mb cap.

You can follow the total size here: http://blockchain.info/charts/blocks-size

144mb is way to much it already takes like 20 hours to download it. Soon we will all need 10TB hard drives for btc only <.<
And it take 4 months to download..

[caveden]

Blockchain growth is only exponential as long as Bitcoin usage growth is exponential. Whenever usage stabilizes, blockchain growth will be linear, and bandwidth consumption will stabilize, while storage resources and connection speeds will remain growing exponentially. So, at most, there will be a medium term future in which running a full node requires a professional amount of resources, but as resources keep growing exponentially and Bitcoin consumption of said resources keeps growing linearly, eventually everybody will be able to run full nodes on their phones.

And remember, for sending and receiving money safely, you can use a SVP client. Those require much less resources. You can run a full p2p SVP client on your phone right now.

(Btw, if I'm not mistaken in the fast calculations I've just made, it would take almost 200 years to reach 10Tb by growing 144Mb per day... I know this growth rate will increase beyond these 144Mb, but from where do you take this "soon we'll need 10Tb" figure? What's "soon"?)

[matt608]

Since 0.8 it became perfectly possible to run Bitcoin with its data dir in an USB disk, so I'm currently doing that to save space in my main disk (which is quite small).

Concerning your question, It's currently growing at no more than 250Kb per block, with ~144 blocks per day that should be at most ~36Mb of daily growth (it's actually less than that, not every block's hitting 250Kb). After May 15th, that growth rate might rise. It will take a hard-fork change for it to rise above 144Mb per day though, due to the 1Mb cap.

You can follow the total size here: http://blockchain.info/charts/blocks-size

I have an external HD and have a back up on there. So are you saying I could delete ( ) the copy on my laptop and just have the HD version to save disk space?  I suppose that would make sense I just never thought of it.  Don't think I will as it's less risky to just delete more other stuff from my laptop and make a backup of that rather than just have 1 copy of my wallet.

[caveden]

You just need to point Bitcoin-QT to your data dir. I think it's -datadir=... or something.

Keep backups of your wallet, of course.
No need to back up the blockchain, there are already plenty of copies around the world.

[Gabi]

Do not delete your wallet! caveden is speaking about deleting the blockchain/having it on a usb key instead of the hard disk. But do not delete your wallet, make backups of it!

The blockchain today is about 7.8GB big, and of course it is increasing. Once the blocksize cap will be changed, it will grow even more. But of course pruning is possible, so don't worry about 10TB hard disks 

[Rassah]

It only costs a bit over BTC3 to get 10TB nowadays 

[whiskers75]

blockchain.info/wallet

[tvbcof]

...
And remember, for sending and receiving money safely, you can use a SVP client. Those require much less resources. You can run a full p2p SVP client on your phone right now.
...

Why would you classify and SPV client as a 'peer'?  It certainly does not seem to me to be a peer to a node holding a block chain, and as best I can tell does nothing whatsoever in terms of helping support the integrity of the network.

[farlack]

It only costs a bit over BTC3 to get 10TB nowadays 

My 3rd to last bank paid me $150 to open, my 2nd to last paid me $125, and my last bank paid me $25

Spending $400 to open a bank account (wallet) is counter productive I think.

[caveden]

Why would you classify and SPV client as a 'peer'?  It certainly does not seem to me to be a peer to a node holding a block chain, and as best I can tell does nothing whatsoever in terms of helping support the integrity of the network.

I meant it works in a p2p fashion, not in a client-server fashion, as it connects to multiple nodes. It's "decentralized" if you prefer.

[tvbcof]

Why would you classify and SPV client as a 'peer'?  It certainly does not seem to me to be a peer to a node holding a block chain, and as best I can tell does nothing whatsoever in terms of helping support the integrity of the network.

I meant it works in a p2p fashion, not in a client-server fashion, as it connects to multiple nodes. It's "decentralized" if you prefer.

The first sentence makes basically no sense at all in logical or technical terms.

As to the second sentence, I have a checkbook in my pocket and I can write checks to anyone no matter what bank they use.  I can also use any ATM I see (for an annoying but tolerable fee.)  A Bitcoin solution with a limited number of full nodes and a majority of users using SPV or thin client hacks such as electrum seems to me not much different than what we have today.  And likely pretty easy for corp/gov to clamp down on when they see fit, particularly if it threatens most geo-political unions which I imagine it could.

[Rassah]

It only costs a bit over BTC3 to get 10TB nowadays 

My 3rd to last bank paid me $150 to open, my 2nd to last paid me $125, and my last bank paid me $25

Spending $400 to open a bank account (wallet) is counter productive I think.

Wrong comparison. Blockchain.info costs $0 to open an account on. How much does it cost to buy a vault and start your own bank?

[Mike Hearn]

Eventually most users will be on SPV wallets, that was always the plan (see Satoshi's paper).

This isn't the same as banking, not even remotely close. For one, malicious peers are very limited in what they can do to SPV clients. They don't validate the chain contents, so a miner can make an SPV client temporarily see a bogus view of the world for as long as they can outrun the network. But once they fall behind, the SPV client will switch back to the best valid chain and get back in sync with reality.

And obviously, remote nodes cannot control what you do with your money.

[tvbcof]

Eventually most users will be on SPV wallets, that was always the plan (see Satoshi's paper).

It was one of the less good ideas in my opinion.

This isn't the same as banking, not even remotely close. For one, malicious peers are very limited in what they can do to SPV clients. They don't validate the chain contents, so a miner can make an SPV client temporarily see a bogus view of the world for as long as they can outrun the network. But once they fall behind, the SPV client will switch back to the best valid chain and get back in sync with reality.

Architecturally the system you describe is very similar to the modern banking system with high powered 'peers' upon whom the users are utterly dependent.  All that remains to be seen are what the ratio will be.  It is simply disingenuous to refer to it as a 'peer2peer' solution and retain any real meaning to the term.

And obviously, remote nodes cannot control what you do with your money.

Oh?  I am pretty sure that it was you yourself that came up with one of my favorite lines.  Something like when there is sufficient centralization, there would be "no real difference between taking someones money and keeping them from spending it for 20 years."

[marcus_of_augustus]

No need to back up the blockchain, there are already plenty of copies around the world.

You should really make the note that only the blockchain copy on your machine has been validated by your own machine ....

... why are guys going to such lengths to obfuscate the dangers of not running your own full node I wonder?

[Mike Christ]

No need to back up the blockchain, there are already plenty of copies around the world.

You should really make the note that only the blockchain copy on your machine has been validated by your own machine ....

... why are guys going to such lengths to obfuscate the dangers of not running your own full node I wonder?

Not everyone is quite as paranoid

[marcus_of_augustus]

No need to back up the blockchain, there are already plenty of copies around the world.

You should really make the note that only the blockchain copy on your machine has been validated by your own machine ....

... why are guys going to such lengths to obfuscate the dangers of not running your own full node I wonder?

Not everyone is quite as paranoid

Maybe, but then it wouldn't be the all bells and whistles "trust no-one currency" would it?

It must be paranoid by design ... oblivious to the human designer's subjective assessment of trust needed to function.

[Mike Christ]

Maybe, but then it wouldn't be the all bells and whistles "trust no-one currency" would it?

It must be paranoid by design ... oblivious to the human designer's subjective assessment of trust needed to function.

You're right   But that blockchain is a pain to download, and having to catch up every day takes a while.  Some find it appropriate to sacrifice that "trust no-one" stasis for convenience.

[Rassah]

Don't forget, SPV clients don't trust anyone either. They connect to different nodes and verify/transmit transactions from a few different nodes. If one is malicious, it will simply be ignored.

[caveden]

... why are guys going to such lengths to obfuscate the dangers of not running your own full node I wonder?

We're not "obfuscating" anything.
The danger you mention is comparable to the danger of running a piece of open source software without having previously read the entire source code and acquired a full understanding of how it works, before trusting to run it on your wallet.dat.

Thankfully, we can rest assured that a large enough number of people have already made such validation, as it's open to anyone to do it. Any attempt to insert bogus data, either on the source or on the blockchain, will be quickly spot. In the case of running full nodes particularly, there will always be more people capable of doing it - and actually doing it - than there are people capable of fully understanding Bitcoin source code.

I agree with that Bitcoin Magazine article that says that in 20 years, everybody will be capable of running a full node. The blockchain grows linearly once the adoption rate stabilizes. Hardware resources grow exponentially.

Don't forget, SVP clients don't trust anyone either. They connect to different nodes and verify/transmit transactions from a few different nodes. If one is malicious, it will simply be ignored.

Precisely. It's extremely difficult to lie to a SPV node. You need to be the only one it connects to, and still all you'll manage to do is hide stuff from it, not "invent" fake transactions.

[Mike Hearn]

Architecturally the system you describe is very similar to the modern banking system with high powered 'peers' upon whom the users are utterly dependent.  All that remains to be seen are what the ratio will be.  It is simply disingenuous to refer to it as a 'peer2peer' solution and retain any real meaning to the term.

If you've ever tried to set up a bank you'd know there is no comparison possible. A bank is not just a computer that tracks transactions, you know. And you can't just switch from one to another in a second or two.

SPV mode is appropriate for people with hardware constraints that let them get almost-as-good security at much less cost. If you want the full security you can of course upgrade to use a full node.

[marcus_of_augustus]

Any ETA on when these mythical SPV node solutions will be hitting github?

[caveden]

You just showed with this bad attempt of irony that you're not really aware about SPV.
BitcoinJ is a SPV node, which already uses bloom filters. MultiBit and Android Wallet are two clients which use BitcoinJ.

[tvbcof]

Architecturally the system you describe is very similar to the modern banking system with high powered 'peers' upon whom the users are utterly dependent.  All that remains to be seen are what the ratio will be.  It is simply disingenuous to refer to it as a 'peer2peer' solution and retain any real meaning to the term.

If you've ever tried to set up a bank you'd know there is no comparison possible. A bank is not just a computer that tracks transactions, you know. And you can't just switch from one to another in a second or two.

In the modern western world 'banks' (usually understood by the masses to be consumer banks) are corporations and like all corporations it is their legal obligation to maximize shareholder value.  They happen to deal with currency as an artifact of their doing business, and in doing so make use of centralized computers which interact with the computers of their 'peers' and those of central banks and regulatory bodies.  I'm just making this up, so please feel free to correct me if you feel that I've mis-spoken.

Other corporations maximize shareholder value in other ways, but visibility into the interactions inherent in a currency system can be extremely valuable to them.  Hence significant interest in 'virtual currencies'.

One thing that does not at all maximize shareholder value is crossing the government or regulatory bodies, so it would be unwise to expect corporations to do so.

SPV mode is appropriate for people with hardware constraints that let them get almost-as-good security at much less cost. If you want the full security you can of course upgrade to use a full node.

As long as it is realistic for a fair fraction of the user base to be full peers I'm not concerned or unhappy.  This is, to me, incompatible with unlimited or heavy growth.  I am not super concerned about mining because to me seems to be something which can be isolated and the system will still remain 'open'.  If latency increase dramatically (due to near-monopolization of mining power by corporations) that still will not preclude the use of Bitcoin as a reserve store of value.

My concern is that the solution will be grown enough such that full peers will require such specialized and expensive hardware and network connectivity that it will be practical for them to be isolated, controlled, and made inaccessible from large swaths of users through DPI and packet filtering.  Eventually.

I am also concerned that the business intelligence value of being able to closely monitor economic activity in a dominant currency solution will far exceed the costs of operation yielding a situation where it is provided as a 'free' service in the same vein as we see with other forms of communication over the internet these days.  This will greatly disrupt some of the economic incentives which are supposed to keep the system in balance.  Or at least 'supposed' by me to be part of the design.

[marcus_of_augustus]

You just showed with this bad attempt of irony that you're not really aware about SPV.
BitcoinJ is a SPV node, which already uses bloom filters. MultiBit and Android Wallet are two clients which use BitcoinJ.

Oh wait, you mean the java client put out by none other than Mike Hearn, the bitcoin dev. who is pushing hardest to raise the block limits, is really the only SPV implementation out there?

Gosh, wait is that a coincidence or irony, or you got me confused with someone else?

[Rassah]

You just showed with this bad attempt of irony that you're not really aware about SPV.
BitcoinJ is a SPV node, which already uses bloom filters. MultiBit and Android Wallet are two clients which use BitcoinJ.

Oh wait, you mean the java client put out by none other than Mike Hearn, the bitcoin dev. who is pushing hardest to raise the block limits, is really the only SPV implementation out there?

Gosh, wait is that a coincidence or irony, or you got me confused with someone else?

There's also a C client being developed, but, really, if there is basic code for it already, why reinvent the wheel?

Oh, and the actual clients that use BitcoinJ are quite different, with different features and usability. SPV mode is also currently being developed for the official Bitcoin-qt client. The new database format and bloom features that were released in version 0.8 were just the first steps for that process. Eventually, people will be able to install the official client, be able to use it within a few minutes as it downloads just the data relevant to its addresses, and if they have the hard drive space, be able to turn on the option to download the entire blockchain in the background. Stop complaining about features we don't have yet (especially when some of those features we do have). Bitcoin is still in beta, not even version 1.0 yet.

[marcus_of_augustus]

Stop complaining about features we don't have yet (especially when some of those features we do have). Bitcoin is still in beta, not even version 1.0 yet.

You misunderstand me (not complaining about lack of features) and agree with me at the same time.

Bitcoin needs a much more decentralised dev. structure before it comes out of beta or it will collapse in heap when the training wheels come off. At a minimum, we need a true blue team to compete, so the reds don't miss something or go off on a tangential group-think phase, just imho. Green, yellow and purples would be preferable but hey. There is also going to be too much pressure on too few brilliant, young minds, they'll blow, seen it a few times and its ugly. That is just the technological side, throw in hundreds of millions of money to the brew and it gets pretty heady ...

Read this to get a feel for some of the challenges. http://en.wikipedia.org/wiki/The_Soul_of_a_New_Machine

[Rassah]

If you think that's a problem, you are free to join github and add/critique all the suggestions on there. You are also free to come up with ideas and try to convince others that they are good, so that some developer somewhere can implement it.
I think you are under the impression that Bitcoin development consists of Gavin and maybe 3 other people. That's not how this project works. There are tons of suggested features and actual code suggested by a whole slew of people from around the world, and those features and code gets reviewed by others, and if proven to work right, get implemented in various clients. The core developers just check and fix the code and sort of herd the cats, in some attempt to keep things organized (plus write their own code, too, of course).

[arorts]

I don't think I've found the answer before but is it scalable to expect that the blockchain is downloaded to personal computers??

Today, the total volume of transactions is immaterial relative to other currencies, but say 5 years from now it represents a very large percentage. In that case, it would be ridiculous to have to download the entire history of transactions from day 0 in all computers. Unless there's a revolutionary technology that will allow the convenient storage of such info without requiring massive storage devices, it just doesn't seem practical to do so.

Perhaps I'm missing something but what did Satoshi think about this? How is this part of Bitcoin efficient? Thanks.

[marcotheminer]

Depends, each day it grows ever so quickly!

[arorts]

Depends, each day it grows ever so quickly!

"Depends"??   I don't think Satoshi left this to a random uncertainty. What's the existing architecture designed for? Each node containing all blockchain or is there an alternative option where larger miners would contain a larger section of the blockchain in the future for example?

I can't think of a different solution to deal with the infinite storage issue.  For example, does it make sense to purge the blockchain? or dynamically compress it? I'm just wondering what Satoshi actually envisioned in his paper.

[Rassah]

I don't think I've found the answer before but is it scalable to expect that the blockchain is downloaded to personal computers??

Today, the total volume of transactions is immaterial relative to other currencies, but say 5 years from now it represents a very large percentage. In that case, it would be ridiculous to have to download the entire history of transactions from day 0 in all computers. Unless there's a revolutionary technology that will allow the convenient storage of such info without requiring massive storage devices, it just doesn't seem practical to do so.

Perhaps I'm missing something but what did Satoshi think about this? How is this part of Bitcoin efficient? Thanks.

Here, read this https://en.bitcoin.it/wiki/Scalability

In short, you don't need the full copy of a blockchain to use bitcoin. Plenty of light clients just ping random nodes and request only the address balances that are relevant to them. Also, the blockchain can be pruned to only the addresses that have balances in them. If that was done today, the 12gig blockchain file would shrink to only something like 200 megs.

Satoshi envisioned that eventually only few copies of the entire blockchain will be stored for archive purposes, and large miners and pool operators (and wealthy hobbyists/supporters) will be the only ones keeping copies of the blockchain, with the general users just relying on the light clients that only request address balances, and sign/transmit transfers to the network.

[Etlase2]

Satoshi envisioned that eventually only few copies of the entire blockchain will be stored for archive purposes, and large miners and pool operators (and wealthy hobbyists/supporters) will be the only ones keeping copies of the blockchain, with the general users just relying on the light clients that only request address balances, and sign/transmit transfers to the network.

I don't think he "envisioned" that as much as "saw" that it was the only way it would work. It does not scale particularly well, and this is a weakness, not a feature. How weak a weakness is certainly debatable.

[old c coder]

Since 0.8 it became perfectly possible to run Bitcoin with its data dir in an USB disk, so I'm currently doing that to save space in my main disk (which is quite small).

Concerning your question, It's currently growing at no more than 250Kb per block, with ~144 blocks per day that should be at most ~36Mb of daily growth (it's actually less than that, not every block's hitting 250Kb). After May 15th, that growth rate might rise. It will take a hard-fork change for it to rise above 144Mb per day though, due to the 1Mb cap.

You can follow the total size here: http://blockchain.info/charts/blocks-size

144mb is way to much it already takes like 20 hours to download it. Soon we will all need 10TB hard drives for btc only <.<
And it take 4 months to download..

I beg to differ. The growth according to the graph of the link given, is more or less linear at about 10-11 GB /year (currently), so ~100GB/decade, ~1TB /century.

There will be spurts of growth in places like Cyprus, Argentina, and for the opposite reason in places such as Hungary, Iceland...Then plateaus of "tranquility"

As to how long to download, not too long. With a "good" internet connection it is easy to get 1 new block in ~<10 seconds. Now there are ~6 blocks created /hr, i.e 144/day. So to catch up for one day, you need 144 * 10 seconds = 1440 seconds = 1440/60 = 24  minutes. Notice about one minute for an hour of catch up. So a week of catch up ~ 24*7 minutes = 168 minutes  which is < 3 hrs. So a month catch up is about 3 * 4 or 12 hours, 2 months catch up will take ~ a day,  ~a year ~ 6 days,  and you can estimate further if you like. It's not that bad. It's actually better! The first blocks being small come really fast, so it takes about 3 days, more or less to get the whole block chain.

What this should tell you is that a "good" block chain is a lot of time saved if you need to start up bitcoin on a new machine! So save the files in those 3 directories:  [datadir]/blocks/*.*
[datadir]/blocks/index/*.*
[datadir]/chainstate/*.*
that comprise the levelDB database! It is time well saved. Almost worth backing up too. Since time is money for some, the block chain may be worth more to you than what's in your BTC (or is it now XBT) wallet!

Ron

[tvbcof]

Satoshi envisioned that eventually only few copies of the entire blockchain will be stored for archive purposes, and large miners and pool operators (and wealthy hobbyists/supporters) will be the only ones keeping copies of the blockchain, with the general users just relying on the light clients that only request address balances, and sign/transmit transfers to the network.

I don't think he "envisioned" that as much as "saw" that it was the only way it would work. It does not scale particularly well, and this is a weakness, not a feature. How weak a weakness is certainly debatable.

A good marketing campaign...if not a modestly technical wiki page with some appropriately designed jargon...can do wonders for turning a bug into a feature.  In a similar vein, one man's weakness is another man's strength.  There are plenty in the community who are probably completely happy to consolidate the infrastructure to a comfortable set of operators and get on with the business of running an economy with a proper level of oversight, control, and in some cases with a fair level of value extraction.

[gmaxwell]

Don't forget, SPV clients don't trust anyone either. They connect to different nodes and verify/transmit transactions from a few different nodes. If one is malicious, it will simply be ignored.

They do— they trust the peers they speak to more than a regular node does. E.g. if they show unconfirmed transactions they trust their peers to only tell them about valid unconfirmed transactions. When they receive blocks, they trust their peers to only tell them about valid blocks (and failing that, miners to only produce valid blocks. Especially when they display 1-confirmed transactions as "confirmed"). I don't believe _any_ SPV node software has a secure peer discovery implementation either, which is frustrating because they are somewhat more vulnerable to naughty peers.

IIRC electrum connects only to a single server at a time. It's SPV from there out but because it makes a single point of attachment it's pretty vulnerable. It also does no SSL certificate validation.   BitcoinJ wallets are differently bad: They connect to multiple, but they're only what get returned from DNS. Of course, the Bitcoin protocol itself is unauthenticated, so a network attacker near the node can isolate it easily.

Obviously SPV wallets will improve in the future. ... But they are no replacement for full nodes. If it wasn't important for many regular users to run full nodes then we wouldn't need full nodes at all.  When you don't run full nodes you are trusting miners to always follow the rules faithfully— to not steal from the users by inflating the coin, etc.  But if users are not rejecting unfaithful blocks, because they are SPV and can't validate them, what incentive is there for miners to behave quite so faithfully?  Will they watch themselves, mutually distrutful?  Not so clear, because of cloud mining and pooling the majority of hashpower is under the control of approximately 3 people right now. ... and increasing their income would be uniformly beneficial to them.

In classical currencies and banking the people with equivalent trust are the central banks and governments. There are many reasons the behavior of fiat currencies and banking in democratic regimes trustworthy: They are highly regulated, privileged institutions— not anonymous entities like mining— arguably controlled by elected officials.  And yet they are observably not trustworthy, and their lack of trustworthness— the lack of any institution depending on human fallibility— was a motivation for Bitcoin's creation.

But if you leave network security to a handful of self-appointed anonymous miners without the check of tens of thousands of regular users with uncorrelated interests outside of Bitcoin's integrity then the result is something even less trustworthy. I think that Bitcoin's value argument is too weak to be worth keeping it around if it doesn't offer an integrity improvement over the alternatives.

I'm just wondering what Satoshi actually envisioned in his paper.

Then perhaps you should read it? It's eight pages long and written in plain English.

At times I want to start banning people from this sub-forum who haven't read it. It is far from answering all questions, but there really is no excuse not to have read it.

[caveden]

When you don't run full nodes you are trusting miners to always follow the rules faithfully— to not steal from the users by inflating the coin, etc.  But if users are not rejecting unfaithful blocks, because they are SPV and can't validate them, what incentive is there for miners to behave quite so faithfully?

When you don't read and fully understand the entire source code of your bitcoin client, you are trusting developers to always follow the rules faithfully - to not steal/spy from/on users by inserting a backdoor etc. But if users are not reading and fully understanding the source code, because they're incapable or not willing to do so, what incentive is there for developers to behave quite so faithfully?


I'm repeating myself, but it seems important to do so: there will always be more honest people willing and capable of running a full node than there will be honest people willingly and capable of reading and understanding such a complex C++ code as that of bitcoin. The barriers are incomparable. Running a full node is simple, and will be always cheaper than either hiring a C++ expert to do the code-parsing for you, or going through the huge learning curve yourself. Actually the option of hiring someone only transfers the trust, so you have to go through the learning curve if you want to be 100% sure. There's no multiple years learning curve necessary for someone to run a full node.

So, please, let's quit the drama. If the "everybody can validate the source because it's open" sentence is really true and justifies the trust we have on open source software like Bitcoin, then there's no reason for the sentence "everybody can validate the blockchain because the data is open/public" not to be equally true, and an equivalent justification for SPV clients to trust in its integrity.

[Peter Todd]

caveden: You can audit the sourcecode in pieces, and there are plenty of people doing just that.

Similarly with some changes to the Bitcoin protocol you'll be able to audit the blockchain in pieces, which could lead to the elimination of a strict separation between full and SPV nodes in exchange for a model where nodes do more or less verification work.

In any case the source code doesn't change much, so auditing work once done remains valid for a fairly long time; the auditing that full nodes do needs to be done continuously. There's also issues of attacks and capacity - if we continue to lose full nodes you'll eventually find it hard to use SPV clients, and the network as a whole becomes more and more vulnerable to DoS attacks.

[caveden]

caveden: You can audit the sourcecode in pieces, and there are plenty of people doing just that.

What if the backdoor is in a piece you chose not to validate?
(just so this is crystal clear: I don't believe this is a serious danger, as I do believe in open source.. my point is just to show that the same applies to the blockchain, which is also open)

Similarly with some changes to the Bitcoin protocol you'll be able to audit the blockchain in pieces, which could lead to the elimination of a strict separation between full and SPV nodes in exchange for a model where nodes do more or less verification work.

I'm not sure which modifications you're talking about (that long thread from the author of Armory perhaps?), but with the current Merkle Tree I believe this is already possible to some extent. I can validate the coinbase transaction only to certify no miner is inflating the money supply more than what's allowed by the protocol. I could perhaps also request past transactions in order to validate that the output crediting my address really does have the money it claims to have, all the way until a coinbase. By only doing that to transactions that concern me - and always checking the headers, PoW etc, ain't I doing a "validation in pieces"? If every user´s client does at least that, doesn't the chain gets entirely validated in the end?

No structural changes would be needed, AFAIK. Unless by "protocol changes" you mean new network messages in order to exchange such partial data, in case they don't exist already.

In any case the source code doesn't change much, so auditing work once done remains valid for a fairly long time; the auditing that full nodes do needs to be done continuously.

It's done continuously by a machine, not by human beings. The actual work of sporadically auditing the code is probably bigger than the work of letting the full node running.

[Peter Todd]

caveden: You can audit the sourcecode in pieces, and there are plenty of people doing just that.

What if the backdoor is in a piece you chose not to validate?
(just so this is crystal clear: I don't believe this is a serious danger, as I do believe in open source.. my point is just to show that the same applies to the blockchain, which is also open)

Point is, if you can prove fraud and publish your findings many people auditing parts can audit the whole 100% in an effective way.

Similarly with some changes to the Bitcoin protocol you'll be able to audit the blockchain in pieces, which could lead to the elimination of a strict separation between full and SPV nodes in exchange for a model where nodes do more or less verification work.

I'm not sure which modifications you're talking about (that long thread from the author of Armory perhaps?), but with the current Merkle Tree I believe this is already possible to some extent. I can validate the coinbase transaction only to certify no miner is inflating the money supply more than what's allowed by the protocol. I could perhaps also request past transactions in order to validate that the output crediting my address really does have the money it claims to have, all the way until a coinbase. By only doing that to transactions that concern me - and always checking the headers, PoW etc, ain't I doing a "validation in pieces"? If every user´s client does at least that, doesn't the chain gets entirely validated in the end?

I outline one such required modification here, and the fraud proofs system to make good use of it: https://bitcointalk.org/index.php?topic=137933.0

UTXO set commitments are the other required modification; without them you can't create a compact proof that a transaction spent a txout that wasn't real. Unfortunately while maaku is working on UTXO set commitments he doesn't seem to understand the value of fraud proofs and partial validation. I've also got my doubts that he understands comp-sci and systems engineering well enough to take on the task for a variety of reasons: https://bitcointalk.org/index.php?topic=137933.0 Hopefully what code he writes will be written in such a way as to make fixing the problems later possible.

No structural changes would be needed, AFAIK. Unless by "protocol changes" you mean new network messages in order to exchange such partial data, in case they don't exist already.

New network messages will be needed yes.

[caveden]

Wait, so you've managed to come up with a way for SPV nodes to satisfyingly verify that an "evil cartel of block generators" isn't defrauding everybody? The debate of whether to keep the crippling 1Mb limit for blocks or to let it grow by removing said limit is finally over?

That's brilliant! It's great to see you've managed to found some common ground. Seriously, no irony, that's a relief to me. I was confident an actual fork (with two different blockchains surviving) was inevitably ahead of us.

[old c coder]

...

When you don't read and fully understand the entire source code of your bitcoin client, you are trusting developers to always follow the rules faithfully - to not steal/spy from/on users by inserting a backdoor etc. But if users are not reading and fully understanding the source code, because they're incapable or not willing to do so, what incentive is there for developers to behave quite so faithfully?

I'm repeating myself, but it seems important to do so: there will always be more honest people willing and capable of running a full node than there will be honest people willingly and capable of reading and understanding such a complex C++ code as that of bitcoin. The barriers are incomparable.

Then for the sanity of the developers, and to "lower the learning curve" and to the benefit of people who would like to read the code and understand it, why don't we make this "complex C++ code"  more understandable?

Adding comments or a separate comment file, sort of like man pages for as many classes, methods, functions, variables, etc. etc. that we can.

We (developers) may actually find a few "loose ends"!

I for one find functions/methods that return a boolean true or false, that usually signifies success or failure, should really have a comment as to which return is which! Especially since all I see are magic numbers, magic boolean expressions, i.e. return 0, return 1, return (ret != 0) and other such vaguaries. One can discover what a variable, function, class does by studying the code. But is this what was intended? Does what it does match the names of the things that are doing the action? Sometimes one can infer what the action or the intent is/was of a function/method by noting what else happens, error  messages, etc. But intent is always a guess when there is no documentation.

But one should not have to infer how something works. It should be specified. Sometimes the intent is not realized by the function, or is the opposite, or is ignored and shouldn't be! The code doesn't know what you "meant" to do, only what is written. So I am guessing at the intent when I look at undocumented code. And I may be wrong.

I have taken to replacing the magic numbers: 0, 1, 144, etc. with more meaningful constants: Failure, Success, AverageBlocksPerDay, etc. where they are or seem appropriate (to me). I could go on & on about meaningful names for variables, etc. etc. I hate having to keep a symbol table in my head about "this' really means "that" when reading code. The variables, functions, properties, methods, classes, filenames etc. etc. can easily be renamed. I usually just do it myself!

I think that C++ code can be and usually is made so obscure, that one statement can literally need a page of comments to explain it. I think that C++ code should be mostly comments and every now and then a statement! Some say it is next to Forth in being a write only language

I think that a consistent indenting style (source code formatting) is also important in understanding.  I do see a coding standard (coding.md), but the code seems to still be a mixture of different styles?

Running a full node is simple, and will be always cheaper than either hiring a C++ expert to do the code-parsing for you, or going through the huge learning curve yourself. ...

So, please, let's quit the drama. ...

Just my $0.02 on the understandability issue.

Ron

[justusranvier]

I have taken to replacing the magic numbers: 0, 1, 144, etc. with more meaningful constants: Failure, Success, AverageBlocksPerDay, etc. where they are or seem appropriate (to me). I could go on & on about meaningful names for variables, etc. etc. I hate having to keep a symbol table in my head about "this' really means "that" when reading code. The variables, functions, properties, methods, classes, filenames etc. etc. can easily be renamed. I usually just do it myself!

Do these changes make it upstream into the main repository?

[Peter Todd]

Wait, so you've managed to come up with a way for SPV nodes to satisfyingly verify that an "evil cartel of block generators" isn't defrauding everybody? The debate of whether to keep the crippling 1Mb limit for blocks or to let it grow by removing said limit is finally over?

That's brilliant! It's great to see you've managed to found some common ground. Seriously, no irony, that's a relief to me. I was confident an actual fork (with two different blockchains surviving) was inevitably ahead of us.

You're more than a little out of date... heck, I recall Gavin saying in #bitcoin-dev last Febuary or so that he personally considered a working fraud proof implementation to be a reasonable pre-condition to raising the blocksize. That's why I pointed out the censorship problem and asked people for ideas on how to solve it. (I haven't gotten any) Of course, lots of people don't see that as a problem at all: Mike Hearn has said he expects mining to be a government regulated activity in the future for example.

As for solving that problem - that is allowing for mining to be decentralized with a large blocksize - I've got a few solutions but they all have the really ugly problem that it's hard to be sure any "sharded" blockchain data is actually in the hands of more than one individual. Put another way, an effective attack is to ensure that you are the only entity with a copy of part of a block (perhaps by sybiling the network) and then keep that data for yourself. Without the data no-one can prove a later spend from that part of the UTXO set is valid or not, rendering those coins unspendable. Possible motivations include freezing someone's funds, or even charging high fees to spend those funds.

The best counter-measure I can think of is using proof-of-stake to ensure that any given part of a block was actually seen by some threshold percentage of the owners of those coins. But lost coins and apathy complicates this, and getting such systems working right would, as far as I can tell, require changes to the basic economic model of Bitcoin. Anyway without a ASIC-hard PoW algorithm those changes aren't necessarily all that useful anyway.

[caveden]

You're more than a little out of date...

Indeed I am... I can barely follow the most discussed topics here and on reddit. The devlist I barely read at all.

Thanks for the update anyways!

Of course, lots of people don't see that as a problem at all: Mike Hearn has said he expects mining to be a government regulated activity in the future for example.

Perhaps the most authoritarian ones will attempt, but I don't think they'll manage. They can't even stop file-sharing. There will always be jurisdictions in which mining will remain an unregulated activity, and block generators would just put their servers there. I don't think that'll ever be a serious problem.

As for solving that problem - that is allowing for mining to be decentralized with a large blocksize - I've got a few solutions but they all have the really ugly problem that it's hard to be sure any "sharded" blockchain data is actually in the hands of more than one individual.

If you're really worried about this - IMHO you shouldn't worry that much - perhaps it'd be better to focus on more scalable darknet protocols instead. Perhaps using bitcoin for funding darknet nodes, I don't know... Anyways, it looks to me your problem here isn't simply "keep block generation decentralized", but more like "being able to communicate without significant latency and without revealing your physical location". In other words, a way for dissident pool operators to hide themselves in a highly totalitarian world. The solution is probably outside the Bitcoin protocol.

[Peter Todd]

Of course, lots of people don't see that as a problem at all: Mike Hearn has said he expects mining to be a government regulated activity in the future for example.

Perhaps the most authoritarian ones will attempt, but I don't think they'll manage. They can't even stop file-sharing. There will always be jurisdictions in which mining will remain an unregulated activity, and block generators would just put their servers there. I don't think that'll ever be a serious problem.

Or the issue will be pedantic because ASICs centralize mining hardware production into the hands of 2-3 chip fabrication companies... but lets assume not for the sake of argument, and anyway an alt-coin - perhaps one even duplicating the existing blockchain - can always change the PoW algorithm when it comes to that. There do appear to exist algorithms where the ratios of profitability between totally custom, "decentralized cottage industry electronics", and off-the-shelf aren't so extreme, but that needs further research.

As for solving that problem - that is allowing for mining to be decentralized with a large blocksize - I've got a few solutions but they all have the really ugly problem that it's hard to be sure any "sharded" blockchain data is actually in the hands of more than one individual.

If you're really worried about this - IMHO you shouldn't worry that much - perhaps it'd be better to focus on more scalable darknet protocols instead. Perhaps using bitcoin for funding darknet nodes, I don't know... Anyways, it looks to me your problem here isn't simply "keep block generation decentralized", but more like "being able to communicate without significant latency and without revealing your physical location". In other words, a way for dissident pool operators to hide themselves in a highly totalitarian world. The solution is probably outside the Bitcoin protocol.

The problem is more that there's no known way to create darknets with high-bandwidth individual connections, relative to the bandwidth possible on opennet connections, and there's no reason to think improvements in technology will do anything to close that gap. (if anything, the gap will probably widen due to the limits of stenography - there's a limit to how much bandwidth individual HD video streams plausibly need for instance) Hence my focus on determining if and how mining can be done co-operatively in such a way that any individual participant needs only a subset of the total bandwidth required for the system to operate, while still allowing participants to use pseudonyms and remaining resistant to attack by participants, and on top of all those requirements keeping the profitability for such participants as similar as possible to the profitability more centralized operators enjoy. Not an easy problem.

Incidentally, the fact that pools make any sense at all is something I'd like to see fixed.

[old c coder]

I have taken to replacing the magic numbers: 0, 1, 144, etc. with more meaningful constants: Failure, Success, AverageBlocksPerDay, etc. where they are or seem appropriate (to me). I could go on & on about meaningful names for variables, etc. etc. I hate having to keep a symbol table in my head about "this' really means "that" when reading code. The variables, functions, properties, methods, classes, filenames etc. etc. can easily be renamed. I usually just do it myself!

Do these changes make it upstream into the main repository?

I am sorry to say, but it appears that my suggestions, changes, GitHub commits seem to be not even worthy of comment

See
https://bitcointalk.org/index.php?topic=149479.msg2708118#msg2708118
from July, 2013

The only thing I see is a new argument in bitcoin 0.8.5, to the command line (or bitcoin.conf) of -wallet

I see no document referenced, and have heard no mention of it, and looking at the code, in init.cpp I can't even be sure what it does.

My code, documents explicitly what it does and how. BTW, I have even more (and better) code that completely separates all the Berkeley DB wallet files so that they may be moved, renamed, etc. etc. But there seems to be more chatter about bitcoind (bitcoin-qt?) having the miner and wallet code removed in version 0.9? Perhaps a second and third separate program to recreate the original functioning bitcoind?

I posted this message, on the bitcoin-development list on sourceforgr.net:

Message: 3
Date: Fri, 23 Aug 2013 06:28:58 -0700 (PDT)
From: Ron <>
Subject: Re: [Bitcoin-development] Bitcoin-development Digest, Vol 27,
    Issue 33
To: "bitcoin-development@lists.sourceforge.net"
    <bitcoin-development@lists.sourceforge.net>
Message-ID:
    <1377264538.81477.YahooMailNeo@web124501.mail.ne1.yahoo.com>
Content-Type: text/plain; charset="iso-8859-1"
________________________________
Message: 6
Date: Thu, 22 Aug 2013 17:30:13 +0200
From: Wladimir <>
Subject: Re: [Bitcoin-development] Proposal: remove "getwork" RPC from
bitcoind

Message-ID:
<CA+s+GJC4o5V5p+FY+bgWVUt5umebn4_37bTihfX2q1GF05S=VA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

On Thu, Aug 22, 2013 at 3:33 PM, Mike Hearn <> wrote:

> That would be annoying for testing. Regtest mode allows you to create a
> new block by just running "setgenerate true" (it switches itself off after
> creating a block). If you had to set up a complicated set of separate
> programs just to do regtest mode that'd be a step backwards, IMO.
>

There is some consensus that when the internal miner is to be removed, a
simple miner should be packaged with the main repository as separate
program (the "reference miner"?). The only change is that it does no longer
need to burden the core code
(see also the discussion here: https://github.com/bitcoin/bitcoin/pull/2917).

Wladimir
__________________________________________________________
I see no burden to the code when it is not mining, if that is what you mean by
burden. The miner code's hashes/sec are a function of how much CPU time it
gets. When I am gcc compiling, I see the hashes/sec drop, but bitcoind keeps
up easily side by side with http://blockchain.info/ latest transactions and
new blocks. And I only have a single core AMD Athlon 1.8GHz cpu.

I would hate to admit how many browser windows and tabs I have open too,
and an IDE (LOL)!I will admit that I have modified the miner code a little,
?to use (potentially) every allowable nonce and to check for a new block
in a timed fashion and be less aggressive, 8 bytes of 0 instead of 4, in checking
for a potential solution.

Ron

No comments so far. And note that 2917 github link. Gordon wants to remove the wallet too! For my money, that would just about strip bitcoin of all of its power as a single program that is a full network "node" a wallet if need be and a miner if need be, and a server if need be. I should think that Satoshi would be spinning in his grave like a lathe, if he/she they were dead

After seeing (the first part of) Ted Nelson's piece,
http://www.youtube.com/watch?v=emDJTGTrEm0&feature=player_embedded

I don't see how one could think of striping the wallet and the miner code out of bitcoind or bitcoin-qt.

Just my $0.02

Ron