This does not enable you to share a data-directory. Doing that will just corrupt your data directory if you manage to bypass the startup locks.

Actual storage can be shared by using a file system that support copy on write, or copying a data directory and replacing the block files with hardlinks.

Because of those alternatives I wouldn't spend any time developing or reviewing functionality to improve this further, and would instead favor finishing the pruning support.

I can very very cheaply create 100 difficulty 1 blocks and feed them to your client and claim that they're the tip of the best chain. You'd know no better.

I don't think compromising the security of SPV wallets further is worth an unnoticeable small improvement.

A zero trust compression of the past headers with log(n) scaling is possible too, and will hopefully be deployed for other applications. Because of the above point it is not very interesting for SPV, but if it were deployed in any case using it would be much better than having an unverified and unverifiable dependence on peers.

I send you my theoretical condolences.

No worries, everyone breaks theoretical Bitcoin.

Typed data on the stack makes writing correct code much harder, I can't say that I've ever wished for that. I general prefer the stack be "bytes" and everything "converts" them to the right type. Yes, additional constraints would make things like your provably undependable code easier, but they do so by adding more corner cases that an implementation must get right.

I'm also a fan of analyizability, but that always has to be second seat to consensus safeness.

Something to watch out for here is that it's coercion vulnerable, which I think I'd addressed in my science project work.

E.g. I can go to Satoshi and Andytoshi and demand they give me they publish public keys, and then in doing so prove the message came from tacotime.

To avoid this,  you generate a random blinding key Q  and sign with gP+gQ  instead, proving knoweldge of P, then you forget Q.  Later you cannot be coerced because you can honestly claim to have forgotten Q.

Making the threshold scheme e.g. where you have a set of N of M signers where _no_ person knows who all the N (not even the members themselves) is more complicated with this blinding, however, because someone must create the Qs for the involuntary participants.

You currently don't support composing signatures but you totally could doing so results in useful applications.

Thats not true.


No one is saying that things like OP_CAT cannot be done, or that they're bad or whatever. But making them not a danger requires careful work. Case in point: What you're suggesting is obviously broken.  So I write a transaction which pays 100x that (presumably nominal fee) and I crash _EVERY BITCOIN SYSTEM ON THE NETWORK_, and I don't really have to pay the fee at all because a transaction needing a zillion yottabytes of ram to verify will not be mined, so I'll be free to spend it later.  Congrats, you added a severe whole network crashing vulnerability to hypothetical-bitcoin.

You should also remove "enabled" from your dictionary, that those opcodes were "disabled" doesn't mean they can just be enabled. They're completely gone— adding them is precisely equivalent to adding something totally novel in terms of the required deployment procedure.

Sure, but this requires: a consistent way of measuring it and enforcing it, and being sure that no operation has unlimited intermediate state.

As Bitcoin was originally written it was thought that it had precisely that: There was a limit on the number of pushes, and a limit on the number of operations. This very clearly makes the stack size "limited", but because of operations that allow exponential growth, the limit wasn't useless.  Being absolutely sure that the limits imposed are effective isn't hard for any fundamental reason, as I keep pointing out. "Just have a limit", but being _sure_ that the limit does what you expect is much harder than it seems.

All the limits are risks, all complexity— practically every one of them has been implemented incorrectly by one alternative full node implementation or another (or Bitcoin core itself) at some point. They miss them completely, or count wrong for them, or respond incorrectly when they're violated.  E.g. here what happens if you OP_CAT 520 and 10 bytes? Should the verify fail? Should the result be truncated? But even that wasn't the point here.

Point here was that realizing you _needed_ a limit and where you needed it was a risk.  The reasonable and pedantically correct claim was made that OP_CAT didn't increase memory usage, that it just took two elements and replaced them with one which was just as large as the two... and yet having (unfiltered) OP_CAT in the instruction set bypasses the existing limits and allowed exponential memory usage.

None of it insurmountable, but I was answering the question as to why it's not just something super trivial.

Yup, has for some time— it's is part of why p2pool's observed orphan rate is less than 1/10th several of the larger pools.

Please don't do this. It undermines the security model of Bitcoin, and at most saves you a few _milliseconds_ (and even there it isn't lost work because you could theoretically find a block and win the block race) since virtually all the transactions are already verified, and cached in memory and the node doesn't check them again.

SPV clients are counting the blocks you produce to be valid. Setting up that kind of verification also produces extreme fragility in that if a software error makes a good node produce a bad block you could have a whole series of bad blocks created by mutually trusting hosts, creating a large fork in the network. This kind of approach is also vulnerable to attack since without a PKI infrastructure you cannot determine who the source of a block is— for example it could just be a sybil who is relaying good blocks to you from other parties but later starts feeding you trash. (and, of course, imposing identities on mining has a multitude of problems and risks that go beyond the simply technical)

It doesn't seem right that you're posting in the technical subforum without understanding what I was saying there.

I've been running the relay node client since Matt wrote it and it appears save a considerable amount of bandwidth (e.g. avoiding resending >95% of all transactions) which means faster block relaying and fewer orphans.

Plus, alternative transports make the network more robust e.g. some crazy firewall starts blocking the Bitcoin P2P network, the relay client may get through and prevent partitioning.

Great stuff.

I have X gigabytes today that I want to contribute to the Bitcoin network. I'd like it to use all of that X as soon as possible.

But I don't want to be in a situation where as the network grows the distribution of blocks stored becomes non-uniform. E.g. because I used up my space and then more blocks came into existence I'm over-heavy in old blocks... or where my locality becomes poor (creating connection overhead for peers having to fetch only a few blocks per connection).

I don't want to have to rely on trying to measure my peers to find out what sections are missing and need storing, since an attacker could falsely make some areas of the chain seem over represented in order to make them unrepresented.

Eventually, if sparse nodes are to be useful for nodes in their IBD you'll want to learn about what ranges nodes support before you try connecting to them, so I don't want to have to signal more than a few bytes of data to indicate what ranges I'm storing, or have to continually keep updating peers about what I'm storing as time goes on... as that would require a lot of bandwidth.

I want to be able to increase or decrease my storage without totally changing the blocks I'm storing or biasing the selection. (though obviously that would require some communication to indicate the change).

So those are at least the motivations I've mostly been thinking about there.

I don't think anyone is coding on it. There has been a bunch of discussion in the past, so at least for my part I have an idea of what I want to see done eventually (the stuff I outlined).

Pieter posted on bitcoin-development about service bits for this... though that wasn't talking about sparse blocks support, but just bits indicating that you can some amount of the most recent blocks. (He also did some measurements which showed that the request probability became pretty uniform after about 2000 blocks deep, obviously recent blocks were much more frequently requested due to nodes being offline for just a few days/weeks).


My thinking is that initially you'll have to connect to a node to find out what ranges it has (e.g. we'll just add a handshake for it to the p2p protocol) and you'd just disconnect peers that weren't useful to you while also learning who has what (e.g. if you need blocks 1000-2000 but find 100000-110000 you'll know where to go later), but latter the address message should be revised so that it could carry the data with it. So your peers will tell you about what nodes are likely to have the blocks you need.

We're (At least Sipa and I) planning on removing checkpoints almost completely after headers first is merged. They are really damaging to people's reasoning about the security model, and headers first removes practically actual use of them. (Instead they'd be converted into a "chainwork sufficiency" number, e.g. how much work should the best chain have, and something to put the wallet into safe-mode and throw an error if a large reorg happens).

Undo files take up comparatively little space, all of them right now amount to only about two and a half gigs... and unlike the blocks, if you do want them again you can't just go fetch them.

In general I'm concerned about proscribing any particular depth beyond which a node should refuse to reorg— any at all, or at least not any that is remotely short.  The risk is that if someone does create a reorg at that limit they can totally split the consensus. (e.g. let half the nodes get one block that crosses the threshold then announce the reorg). Doubly so when you consider corner cases like the initial block download, e.g. where you get a node stuck on a bogus chain (though that would be part of the idea behind a sufficiency test).

Yep, thats basically it. The only reason I don't really like O(n) is perhaps your addr database has 100,000 entries— 3/4 of which are bogus garbage put out by broken or malicious nodes. Now you need to do a fair amount of work just to figure out which peers you should be trying.

Oh interesting. I got stuck trying to come up with O(1) that I didn't think about log shaped solutions. I'll give that some thought...

I think in general we want to be able to claim that if the node seeds are uniformly distributed, that we expect the block distribution to approach uniform at all times as the node count approaches infinite, regardless of how the size is set. In practice I expect (and hope) that there will be a fair number of nodes with really large size settings (e.g. all the blocks) which will help even out any non-uniformity— really, in my mind most of the reason for supporting sparse nodes is to make sure everyone can contribute at whatever level they want, and to make sure that those who do choose to contribute a lot of storage aren't totally saturated with requests.

Jeff's comments, and my (I'm nullc on reddit) comments: http://www.reddit.com/r/Bitcoin/comments/2f6iiq/bitcoin_is_really_fragile_bitcoin_core_developer/ck6es4t#ck6es4t

Presumably if your system was made in the last decade it support modules larger than 2GB?  What is the virt usage corresponding to your 800 mb actual usage reading?

That it may crash or end up forked and leave you effectively not mining. This is also true for any other version, but non-released versions are (by definition) less widely tested.  The amount of risk depends on whats going on recently in git master. At the moment, there isn't anything exceptionally new which is obviously highly risky.  Similarly if you keep coins on that system exposing immature software to the internet you might have an increased risk of a security compromise. That said, hopefully you have no coins on the system considering you're using it for web-browsing. Marginal risk from new code in bitcoin core is going to be much less than running any browser.

If your hashrate is only fairly modest (I'd hope— since you're not running your node on a separate dedicated system), you could just consider a bit of risk taken here as a public service and increase your monitoring vigilance to match.  If you were talking about a large amount of hashrate, I'd recommend against not— not just due to the risk to yourself, but if you start producing multiple invalid blocks it may cause (minor) problems for SPV nodes or alt implementations. Risk that you take on your own is one thing, subjecting other people to it is another matter.

Sure sure, as I said about it's not hard in theory, but theres the lesson— even after I pointed out there was exponential memory usage in a naive implementation the OP thought otherwise.  And before anyone else trips over your egos, it's not that the OP was foolish or anything. There are subtle interactions in the dark corners which make making promises about the behavior difficult.  So while the actual safe behavior isn't fundamentally hard, being confident that all the corner cases and interactions are handled is fundamentally hard.

OP_CAT isn't the only "disabled" opcode with those properties.... e.g. multiplying also does it.

When behavior like this is fixed via limits great care must be taken to make sure the limits are implemented absolutely consistently everywhere or the result is a consensus splitting risk. Alt full node implementers have repeatedly implemented the limits wrong— even when they're obvious in the code, called out in the comments, documented on the wiki, etc... even by just simply not implementing them (... coverage analysis and testing against the blockchain can't tell you about limits that you're just missing completely).

Going back to the OP's question. I'm not seeing how OP_CAT (at least by itself) facilitates any of the high level examples there. Can you give me a specific protocol and set of scripts to show me how it would work?

You could also just switch to paypal and avoid all the complexity of that fussy Bitcoin stuff. Since you're apparently happy to trust oft-anonymous oft-judgement proof parties, paypal would likely be a big security upgrade too.

Git master, -dbcache=4, -rpcthreads=1 -dnsseed=0 -discover=0  -par=1  Uses about 200MBytes on x86_64, maybe getting up to 250 with long uptime and a lot of connections. If you are wallet disabled it'll be another 30mb below that.  Reducing the connection count may make it a bit smaller.  In general, I advise against having inbound connections from the general internet on a node thats mining (e.g. set -listen=0 which will also helpfully limit your connection count as a side effect). Running the relay network daemon is advisable.

In general the defaults are not well tuned for low memory— considering that 1GB ram costs about $10 marginally... and a bit of extra caching is worth it when the memory is available.

The fact that you're saying "usually over 800mb" suggests that you might be confusing virtual address space with memory usage. Virtual address space does not consume any actual memory (not in swap either). The processes memory map is not contiguous, e.g. there are regions of allocated memory in seas of non-allocated address space. Be sure you're measuring the right thing.

Though several hundred megabytes per tab? Might want to reconsider what browser you're running. I apologize that it's a little difficult to run a whole world wide digital currency with memory usage comparable to one or two tabs in a web-browser.

Mine on git master at your own risk, though— no warranties are provided generally, but even less so on git master.

To complete the lesson, for those who never liked homework: With a 201 cycle limit, OP_CAT lets you use approximately 534,773,760 YiB memory, vs 102510 bytes without it.

And maybe you will realize why all these altcoins worry me so?  Or perhaps you've got cheaper sources of ram than I do?

We should certantly prioritize bumping duplicate source IPs when we implement doing that— (I have a long list of criteria written up for that).

The reason we really shouldn't just ban is because there often are many independent nodes behind a NAT and it would be impolite to unnecessarily deny them connectivity.  In at least one case there is a whole country behind a single IP.

Opening up multiple connections is really only the first and most boring of abusive behaviors people can / have been engaging in... there are quite a few parties which are now trying to connect constantly to every reachable host on the network and wasting a lot of connectivity. If we implement measures to address the bulk resource consumption more generally the repeats on one IP will also be approved as a side effect.

Free cookie to someone who can figure out what these specific ones are attempting to do in any case.

What are you trying to accomplish— at a macroscopic level?  We might have better advice if we knew what you were trying to accomplish.