I think that that only miner, merchants and web wallet services will be storing the chain soon.

I do love the idea of POW validated UTXO set, that however would remain "dusty".

The spending clients should prefer to aggregate inputs to address this.

So you have a choice:

https://github.com/bitsofproof/supernode/blob/master/src/main/java/com/bitsofproof/supernode/core/AddressConverter.java

https://github.com/bitsofproof/supernode/blob/master/src/main/java/com/bitsofproof/supernode/core/ECKeyPair.java

I think offering sufficient fee in the transaction is the incentive that will keep working as the network grows, since CPU spent for validation does not come for free.

I see, I was thinking only in the context of the chain, not chain and unconfirmed. Thank you.

I do not yet see how the signature cache actually helps, would you please elaborate?

A signature is a function of the hash of a modified version of the input transaction. The modification depends on which output of that transaction is to be spent. Aren't chances of seeing the same signature twice negligible?

Introducing an opportunistic cache of the last 100.000 transactions gave such a boost to the server that I thought it is worth writing about.
Apparently outputs die generally rather young and saving the db roundtrip for them is the single biggest boost I found until now...

It is time for a project update.

[HYPE]
I enumerate some challenges and the bitsofproof answers to them.

1. Responsiveness to a large number of peers.
This is addressed using non-blocking-IO and a thread pool that forwards fully read messages to their listener, that are otherwise isolated from the complexity of the communication.

2. CPU hunger for signature validation.
This is not a current concern for the bitsofproof node since validation of transactions in a block and inputs the same transaction are executed simultaneously. CPUs are utilized to their limit, as many as you provide.

3. Resolving the inputs of a transaction against the database of previous transactions.
Using a faster database (some NoSQL) or SSD drives only give you a short break until the bottleneck re-appears as the network grows.
A sustainable approach to address the problem is to constantly prune the data set searched. The bitsofproof node uses a separate archive transaction table(s) into which a parallel thread moves fully spent transactions much like a garbage collector reclaims storage. The live transaction table that is searched to satisfy new transaction inputs is kept close to the UTXO set.

4. The client API is implemented as a clone of BCCAPI, that provides services to lightweight clients trusting the server.

5. A server API is in draft, that will provide a high level access to persistent and transient data structures, notifications. The aim is to provide all features provided in bitcoind via JSON-RPC, but in bitsofproof's case using the invoker of your choice on an API defined as a proper java interface.

6. The next layer of scale will be implemented through distributing both storage and computation, considerations for this already influence the design.
[/HYPE]

This is code in development. I do not care of backward compatibility of interfaces or database for now.

Great to see a well written alternative for the thin client!

Add do not FUD.

I agree with your view on clients Mike, and saw that bitcoinj is quick to start up, since it defers loading a block to when that is really needed.

Since my focus is the supernode that while serving many clients soon will need all blocks, I am worried not only about the download time but data volume in general.

I keep searching for whatever puts limit on data (needed) growth.

Sorry, I have to withdraw this.

Peter might have found some value, but my original thoughts were wrong, as the sender indication of full spend could only be validated by the receiver using subsequent full blocks.

Thanks a lot for your patience, since I also developed the Idea while discussing.

I think we have a breakthrough here that finally addresses the chain download clog.
Please assign a BIP number, so I write this clean and submit for final review by the community.

Pruned transactions are not transferred, they are pruned.

Pruning however is only allowed for blocks way back behind the current last block (last checkpoint) and only transactions are pruned that were already spent until the prune checkpoint. If those and only those transactions are pruned, then the remaining root spends (transactions remained in the pruned blocks) have to have an output sum equal to what the coinbase sum would be in the original blocks until the checkpoint height, therefore you can not just prune any of them without notice.

You can also not add one since the chain after the checkpoint puts proof of work on them.

Those root spends are just like coinbases of a chain after the checkpoint.

No. It is not a hack to transfer utxo. It is to transfer a chain where spent output is removed if it happened way before the current node. More recent spend transactions remain.

Again, not utxo is transferred. One can not invent a tx way back on chain.
One might attempt to prune it, but that is uncovered by the check of coinsum I wrote above.

Please ask yourself what is the difference between a chain of length X with coinbases (as is) or a chain that is at least X long and root spends play the role of coinbases?

If replacing those root spends would be easy then why is that hard to replace coinbases?

Coinbase sum: the of coins generated by miner. A deterministic amount by chain height insn't it?

A correctly pruned chain has remaining root spends totaling to the coinbase sum, removing any of them would violate this.

Added: the coinbase sum for the height gives an upper boundary on root spends acceptable until that height. This helps the client filter garbage while downloading the pruned portion of the chain.

So, removing root spends will be catched latest as the client reaches the checkpoint.
Adding or altering new root spends is obviously as hard as adding new coinbase.

Root spends would be arbitrarily behind head-X, the pruned chain would be actually more secure then the original chain of length X, since it would still exhibit the full hash work since the earliest unspent, that remains the genesis.

This is not about transferring UTXO, that the meta-tree is.

You agree that full chain is secure.

A chain that is pruned X blocks before the head is just as safe as the chain was as it was X blocks long, with the difference that spends are now not rooted in coinbases but in old spends, isn't it?

If one would attempt to remove an old spend it would violate the requirement that their sum match the coinbase sum for the height,
thats easy to verify.

Let's assume clients accept pruned blocks if they are behind the head with X blocks.

Let's assume Alice spent some transaction and would want to make this disappear through pruning. She would have to wait X blocks, and hope that her spend is not spent again, since if yes pruning would no longer benefit her.

Let's assume she is "lucky" and her spend is still where it was. Now to prune her transaction, she would have to also prune the neighboring transaction in the Merkle tree, that if spend in the next X block, then no luck again. This gets exponentially harder if her transaction is not at the end of the block (leafs of the Merkle tree), since then she would have to make even more other disappearing.

Only if at some future time point all spends rooted in her spend and spends she needs to make disappearing for the tree would be no longer moving for X blocks, then she has a chance, that however gets again exponentially improbable as her and (Merkle) neighboring spends taints the future.

Do you see, why it is hard to cheat with a trailing pruned block?

If X is large enough it could be practically secure.

I think I should emphasize that this is not the same as transferring UTXO, since pruned blocks would not be sent or accepted for the last recent, but e.g. one thousand blocks behind the head block. If pruning is significantly trailing, it gets hard to cheat.

The project just got mavenized by pulling from the first contributor: https://github.com/bartfaitamas/supernode/tree/mavenizing