on average, how much HD space does bitcoin-qt consume per day

[caveden]

When you don't run full nodes you are trusting miners to always follow the rules faithfully— to not steal from the users by inflating the coin, etc.  But if users are not rejecting unfaithful blocks, because they are SPV and can't validate them, what incentive is there for miners to behave quite so faithfully?

When you don't read and fully understand the entire source code of your bitcoin client, you are trusting developers to always follow the rules faithfully - to not steal/spy from/on users by inserting a backdoor etc. But if users are not reading and fully understanding the source code, because they're incapable or not willing to do so, what incentive is there for developers to behave quite so faithfully?


I'm repeating myself, but it seems important to do so: there will always be more honest people willing and capable of running a full node than there will be honest people willingly and capable of reading and understanding such a complex C++ code as that of bitcoin. The barriers are incomparable. Running a full node is simple, and will be always cheaper than either hiring a C++ expert to do the code-parsing for you, or going through the huge learning curve yourself. Actually the option of hiring someone only transfers the trust, so you have to go through the learning curve if you want to be 100% sure. There's no multiple years learning curve necessary for someone to run a full node.

So, please, let's quit the drama. If the "everybody can validate the source because it's open" sentence is really true and justifies the trust we have on open source software like Bitcoin, then there's no reason for the sentence "everybody can validate the blockchain because the data is open/public" not to be equally true, and an equivalent justification for SPV clients to trust in its integrity.

[1714683270]

1714683270

[1714683270]

1714683270

[Peter Todd]

caveden: You can audit the sourcecode in pieces, and there are plenty of people doing just that.

Similarly with some changes to the Bitcoin protocol you'll be able to audit the blockchain in pieces, which could lead to the elimination of a strict separation between full and SPV nodes in exchange for a model where nodes do more or less verification work.

In any case the source code doesn't change much, so auditing work once done remains valid for a fairly long time; the auditing that full nodes do needs to be done continuously. There's also issues of attacks and capacity - if we continue to lose full nodes you'll eventually find it hard to use SPV clients, and the network as a whole becomes more and more vulnerable to DoS attacks.

[caveden]

caveden: You can audit the sourcecode in pieces, and there are plenty of people doing just that.

What if the backdoor is in a piece you chose not to validate?
(just so this is crystal clear: I don't believe this is a serious danger, as I do believe in open source.. my point is just to show that the same applies to the blockchain, which is also open)

Similarly with some changes to the Bitcoin protocol you'll be able to audit the blockchain in pieces, which could lead to the elimination of a strict separation between full and SPV nodes in exchange for a model where nodes do more or less verification work.

I'm not sure which modifications you're talking about (that long thread from the author of Armory perhaps?), but with the current Merkle Tree I believe this is already possible to some extent. I can validate the coinbase transaction only to certify no miner is inflating the money supply more than what's allowed by the protocol. I could perhaps also request past transactions in order to validate that the output crediting my address really does have the money it claims to have, all the way until a coinbase. By only doing that to transactions that concern me - and always checking the headers, PoW etc, ain't I doing a "validation in pieces"? If every user´s client does at least that, doesn't the chain gets entirely validated in the end?

No structural changes would be needed, AFAIK. Unless by "protocol changes" you mean new network messages in order to exchange such partial data, in case they don't exist already.

In any case the source code doesn't change much, so auditing work once done remains valid for a fairly long time; the auditing that full nodes do needs to be done continuously.

It's done continuously by a machine, not by human beings. The actual work of sporadically auditing the code is probably bigger than the work of letting the full node running.

[Peter Todd]

caveden: You can audit the sourcecode in pieces, and there are plenty of people doing just that.

What if the backdoor is in a piece you chose not to validate?
(just so this is crystal clear: I don't believe this is a serious danger, as I do believe in open source.. my point is just to show that the same applies to the blockchain, which is also open)

Point is, if you can prove fraud and publish your findings many people auditing parts can audit the whole 100% in an effective way.

Similarly with some changes to the Bitcoin protocol you'll be able to audit the blockchain in pieces, which could lead to the elimination of a strict separation between full and SPV nodes in exchange for a model where nodes do more or less verification work.

I'm not sure which modifications you're talking about (that long thread from the author of Armory perhaps?), but with the current Merkle Tree I believe this is already possible to some extent. I can validate the coinbase transaction only to certify no miner is inflating the money supply more than what's allowed by the protocol. I could perhaps also request past transactions in order to validate that the output crediting my address really does have the money it claims to have, all the way until a coinbase. By only doing that to transactions that concern me - and always checking the headers, PoW etc, ain't I doing a "validation in pieces"? If every user´s client does at least that, doesn't the chain gets entirely validated in the end?

I outline one such required modification here, and the fraud proofs system to make good use of it: https://bitcointalk.org/index.php?topic=137933.0

UTXO set commitments are the other required modification; without them you can't create a compact proof that a transaction spent a txout that wasn't real. Unfortunately while maaku is working on UTXO set commitments he doesn't seem to understand the value of fraud proofs and partial validation. I've also got my doubts that he understands comp-sci and systems engineering well enough to take on the task for a variety of reasons: https://bitcointalk.org/index.php?topic=137933.0 Hopefully what code he writes will be written in such a way as to make fixing the problems later possible.

No structural changes would be needed, AFAIK. Unless by "protocol changes" you mean new network messages in order to exchange such partial data, in case they don't exist already.

New network messages will be needed yes.

[caveden]

Wait, so you've managed to come up with a way for SPV nodes to satisfyingly verify that an "evil cartel of block generators" isn't defrauding everybody? The debate of whether to keep the crippling 1Mb limit for blocks or to let it grow by removing said limit is finally over?

That's brilliant! It's great to see you've managed to found some common ground. Seriously, no irony, that's a relief to me. I was confident an actual fork (with two different blockchains surviving) was inevitably ahead of us.

[old c coder]

...

When you don't read and fully understand the entire source code of your bitcoin client, you are trusting developers to always follow the rules faithfully - to not steal/spy from/on users by inserting a backdoor etc. But if users are not reading and fully understanding the source code, because they're incapable or not willing to do so, what incentive is there for developers to behave quite so faithfully?

I'm repeating myself, but it seems important to do so: there will always be more honest people willing and capable of running a full node than there will be honest people willingly and capable of reading and understanding such a complex C++ code as that of bitcoin. The barriers are incomparable.

Then for the sanity of the developers, and to "lower the learning curve" and to the benefit of people who would like to read the code and understand it, why don't we make this "complex C++ code"  more understandable?

Adding comments or a separate comment file, sort of like man pages for as many classes, methods, functions, variables, etc. etc. that we can.

We (developers) may actually find a few "loose ends"!

I for one find functions/methods that return a boolean true or false, that usually signifies success or failure, should really have a comment as to which return is which! Especially since all I see are magic numbers, magic boolean expressions, i.e. return 0, return 1, return (ret != 0) and other such vaguaries. One can discover what a variable, function, class does by studying the code. But is this what was intended? Does what it does match the names of the things that are doing the action? Sometimes one can infer what the action or the intent is/was of a function/method by noting what else happens, error  messages, etc. But intent is always a guess when there is no documentation.

But one should not have to infer how something works. It should be specified. Sometimes the intent is not realized by the function, or is the opposite, or is ignored and shouldn't be! The code doesn't know what you "meant" to do, only what is written. So I am guessing at the intent when I look at undocumented code. And I may be wrong.

I have taken to replacing the magic numbers: 0, 1, 144, etc. with more meaningful constants: Failure, Success, AverageBlocksPerDay, etc. where they are or seem appropriate (to me). I could go on & on about meaningful names for variables, etc. etc. I hate having to keep a symbol table in my head about "this' really means "that" when reading code. The variables, functions, properties, methods, classes, filenames etc. etc. can easily be renamed. I usually just do it myself!

I think that C++ code can be and usually is made so obscure, that one statement can literally need a page of comments to explain it. I think that C++ code should be mostly comments and every now and then a statement! Some say it is next to Forth in being a write only language

I think that a consistent indenting style (source code formatting) is also important in understanding.  I do see a coding standard (coding.md), but the code seems to still be a mixture of different styles?

Running a full node is simple, and will be always cheaper than either hiring a C++ expert to do the code-parsing for you, or going through the huge learning curve yourself. ...

So, please, let's quit the drama. ...

Just my $0.02 on the understandability issue.

Ron

[justusranvier]

I have taken to replacing the magic numbers: 0, 1, 144, etc. with more meaningful constants: Failure, Success, AverageBlocksPerDay, etc. where they are or seem appropriate (to me). I could go on & on about meaningful names for variables, etc. etc. I hate having to keep a symbol table in my head about "this' really means "that" when reading code. The variables, functions, properties, methods, classes, filenames etc. etc. can easily be renamed. I usually just do it myself!

Do these changes make it upstream into the main repository?

[Peter Todd]

Wait, so you've managed to come up with a way for SPV nodes to satisfyingly verify that an "evil cartel of block generators" isn't defrauding everybody? The debate of whether to keep the crippling 1Mb limit for blocks or to let it grow by removing said limit is finally over?

That's brilliant! It's great to see you've managed to found some common ground. Seriously, no irony, that's a relief to me. I was confident an actual fork (with two different blockchains surviving) was inevitably ahead of us.

You're more than a little out of date... heck, I recall Gavin saying in #bitcoin-dev last Febuary or so that he personally considered a working fraud proof implementation to be a reasonable pre-condition to raising the blocksize. That's why I pointed out the censorship problem and asked people for ideas on how to solve it. (I haven't gotten any) Of course, lots of people don't see that as a problem at all: Mike Hearn has said he expects mining to be a government regulated activity in the future for example.

As for solving that problem - that is allowing for mining to be decentralized with a large blocksize - I've got a few solutions but they all have the really ugly problem that it's hard to be sure any "sharded" blockchain data is actually in the hands of more than one individual. Put another way, an effective attack is to ensure that you are the only entity with a copy of part of a block (perhaps by sybiling the network) and then keep that data for yourself. Without the data no-one can prove a later spend from that part of the UTXO set is valid or not, rendering those coins unspendable. Possible motivations include freezing someone's funds, or even charging high fees to spend those funds.

The best counter-measure I can think of is using proof-of-stake to ensure that any given part of a block was actually seen by some threshold percentage of the owners of those coins. But lost coins and apathy complicates this, and getting such systems working right would, as far as I can tell, require changes to the basic economic model of Bitcoin. Anyway without a ASIC-hard PoW algorithm those changes aren't necessarily all that useful anyway.

[caveden]

You're more than a little out of date...

Indeed I am... I can barely follow the most discussed topics here and on reddit. The devlist I barely read at all.

Thanks for the update anyways!

Of course, lots of people don't see that as a problem at all: Mike Hearn has said he expects mining to be a government regulated activity in the future for example.

Perhaps the most authoritarian ones will attempt, but I don't think they'll manage. They can't even stop file-sharing. There will always be jurisdictions in which mining will remain an unregulated activity, and block generators would just put their servers there. I don't think that'll ever be a serious problem.

As for solving that problem - that is allowing for mining to be decentralized with a large blocksize - I've got a few solutions but they all have the really ugly problem that it's hard to be sure any "sharded" blockchain data is actually in the hands of more than one individual.

If you're really worried about this - IMHO you shouldn't worry that much - perhaps it'd be better to focus on more scalable darknet protocols instead. Perhaps using bitcoin for funding darknet nodes, I don't know... Anyways, it looks to me your problem here isn't simply "keep block generation decentralized", but more like "being able to communicate without significant latency and without revealing your physical location". In other words, a way for dissident pool operators to hide themselves in a highly totalitarian world. The solution is probably outside the Bitcoin protocol.

[Peter Todd]

Of course, lots of people don't see that as a problem at all: Mike Hearn has said he expects mining to be a government regulated activity in the future for example.

Perhaps the most authoritarian ones will attempt, but I don't think they'll manage. They can't even stop file-sharing. There will always be jurisdictions in which mining will remain an unregulated activity, and block generators would just put their servers there. I don't think that'll ever be a serious problem.

Or the issue will be pedantic because ASICs centralize mining hardware production into the hands of 2-3 chip fabrication companies... but lets assume not for the sake of argument, and anyway an alt-coin - perhaps one even duplicating the existing blockchain - can always change the PoW algorithm when it comes to that. There do appear to exist algorithms where the ratios of profitability between totally custom, "decentralized cottage industry electronics", and off-the-shelf aren't so extreme, but that needs further research.

As for solving that problem - that is allowing for mining to be decentralized with a large blocksize - I've got a few solutions but they all have the really ugly problem that it's hard to be sure any "sharded" blockchain data is actually in the hands of more than one individual.

If you're really worried about this - IMHO you shouldn't worry that much - perhaps it'd be better to focus on more scalable darknet protocols instead. Perhaps using bitcoin for funding darknet nodes, I don't know... Anyways, it looks to me your problem here isn't simply "keep block generation decentralized", but more like "being able to communicate without significant latency and without revealing your physical location". In other words, a way for dissident pool operators to hide themselves in a highly totalitarian world. The solution is probably outside the Bitcoin protocol.

The problem is more that there's no known way to create darknets with high-bandwidth individual connections, relative to the bandwidth possible on opennet connections, and there's no reason to think improvements in technology will do anything to close that gap. (if anything, the gap will probably widen due to the limits of stenography - there's a limit to how much bandwidth individual HD video streams plausibly need for instance) Hence my focus on determining if and how mining can be done co-operatively in such a way that any individual participant needs only a subset of the total bandwidth required for the system to operate, while still allowing participants to use pseudonyms and remaining resistant to attack by participants, and on top of all those requirements keeping the profitability for such participants as similar as possible to the profitability more centralized operators enjoy. Not an easy problem.

Incidentally, the fact that pools make any sense at all is something I'd like to see fixed.

[old c coder]

I have taken to replacing the magic numbers: 0, 1, 144, etc. with more meaningful constants: Failure, Success, AverageBlocksPerDay, etc. where they are or seem appropriate (to me). I could go on & on about meaningful names for variables, etc. etc. I hate having to keep a symbol table in my head about "this' really means "that" when reading code. The variables, functions, properties, methods, classes, filenames etc. etc. can easily be renamed. I usually just do it myself!

Do these changes make it upstream into the main repository?

I am sorry to say, but it appears that my suggestions, changes, GitHub commits seem to be not even worthy of comment

See
https://bitcointalk.org/index.php?topic=149479.msg2708118#msg2708118
from July, 2013

The only thing I see is a new argument in bitcoin 0.8.5, to the command line (or bitcoin.conf) of -wallet

I see no document referenced, and have heard no mention of it, and looking at the code, in init.cpp I can't even be sure what it does.

My code, documents explicitly what it does and how. BTW, I have even more (and better) code that completely separates all the Berkeley DB wallet files so that they may be moved, renamed, etc. etc. But there seems to be more chatter about bitcoind (bitcoin-qt?) having the miner and wallet code removed in version 0.9? Perhaps a second and third separate program to recreate the original functioning bitcoind?

I posted this message, on the bitcoin-development list on sourceforgr.net:

Message: 3
Date: Fri, 23 Aug 2013 06:28:58 -0700 (PDT)
From: Ron <>
Subject: Re: [Bitcoin-development] Bitcoin-development Digest, Vol 27,
    Issue 33
To: "bitcoin-development@lists.sourceforge.net"
    <bitcoin-development@lists.sourceforge.net>
Message-ID:
    <1377264538.81477.YahooMailNeo@web124501.mail.ne1.yahoo.com>
Content-Type: text/plain; charset="iso-8859-1"
________________________________
Message: 6
Date: Thu, 22 Aug 2013 17:30:13 +0200
From: Wladimir <>
Subject: Re: [Bitcoin-development] Proposal: remove "getwork" RPC from
bitcoind

Message-ID:
<CA+s+GJC4o5V5p+FY+bgWVUt5umebn4_37bTihfX2q1GF05S=VA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

On Thu, Aug 22, 2013 at 3:33 PM, Mike Hearn <> wrote:

> That would be annoying for testing. Regtest mode allows you to create a
> new block by just running "setgenerate true" (it switches itself off after
> creating a block). If you had to set up a complicated set of separate
> programs just to do regtest mode that'd be a step backwards, IMO.
>

There is some consensus that when the internal miner is to be removed, a
simple miner should be packaged with the main repository as separate
program (the "reference miner"?). The only change is that it does no longer
need to burden the core code
(see also the discussion here: https://github.com/bitcoin/bitcoin/pull/2917).

Wladimir
__________________________________________________________
I see no burden to the code when it is not mining, if that is what you mean by
burden. The miner code's hashes/sec are a function of how much CPU time it
gets. When I am gcc compiling, I see the hashes/sec drop, but bitcoind keeps
up easily side by side with http://blockchain.info/ latest transactions and
new blocks. And I only have a single core AMD Athlon 1.8GHz cpu.

I would hate to admit how many browser windows and tabs I have open too,
and an IDE (LOL)!I will admit that I have modified the miner code a little,
?to use (potentially) every allowable nonce and to check for a new block
in a timed fashion and be less aggressive, 8 bytes of 0 instead of 4, in checking
for a potential solution.

Ron

No comments so far. And note that 2917 github link. Gordon wants to remove the wallet too! For my money, that would just about strip bitcoin of all of its power as a single program that is a full network "node" a wallet if need be and a miner if need be, and a server if need be. I should think that Satoshi would be spinning in his grave like a lathe, if he/she they were dead

After seeing (the first part of) Ted Nelson's piece,
http://www.youtube.com/watch?v=emDJTGTrEm0&feature=player_embedded

I don't see how one could think of striping the wallet and the miner code out of bitcoind or bitcoin-qt.

Just my $0.02

Ron