Yes, yes panic mode set to off I get it.

... I will however save an "I told you so" for the eventuality of further scaling issues than a penny/tx.

1-2 years.

I could see natural growth like that happening and you won't get that much fancy future-ware in that time.

Relax dude I believe in BTC as well, but many still consider it an experiment.

Anyway my point was simply that arguing against new things because they are new is a null argument.

Okay.. would it run with a 100 more SDs?

I never said we have an immediate issue, just an issue.

Well spotted/thought.

I see two solutions:
1. Clients looking at the same address origin to the invalid transaction would instantly be able to see the error. The rest of the network could choose to just trust a multitude of reports (no harm in throwing blocks away).

2. As you said someone could download the entire blockchain and do a check not based solely on "voting with error reports".

Likely what would happen would be 1., there are few risks to dismissing a widely error reported block, but lots of mining power wasted working on a bad block.

If rampant false reporting happened (say to win a "mining war") the originator of the false reports would eventually have his ips blocked.

There is less rush here as you can do that retroactively and then never worry about the blocked folks again. You could take your sweet time downloading everything and verifying everything before judging.

Thanks, kinda thought I would have to just leave this topic alone with no support.

I arrived at this idea after bouncing ideas off TangibleCryptography in the SD thread so he gets some credit for re-explaining block verification to me.

A x-year-ledger/snapshot solution would also make it easier to fully verify error reports as it eases downloading the entire chain.

Those are some good numbers I'll give you that.

I still say we should save that 1m$/day.

Yes debate is evil.

One of the selling points of BTC is micro transactions, but with penny fees per tx (and more in the future), that at least disappears.


Hugely complex
So is BTC itself; lets abandon it for simple fiat.
Super nodes are also hugely complex.

untested
Well yeah.. so is BTC mostly.

will require significant amount of resources to implement
Less than super nodes.

Swarm nodes would run just fine on regular users computers. A giant long term saving.

and solves a "problem" which doesn't exist.
I am a software engineer and I see a problem.

The official devs (Gavin at least) also seem quite open to some kind of update of this nature.

My solution would not fork the chain and would be fully operational with 2 or more clients.

1. Gavin sees the problem.
2. I am a software engineer already working on an open source project that should help the world a bunch.
3. If everyone has been too incompetent/lazy to fix this in a year or two I WILL do it myself.
4. I was however hoping others more trained would do this so I could focus on developing my idea for a BTC credit card or perhaps stock exchange.
5. I have pledged 15 btc, more than I usually give in gifts to my own family, not just "talk", not much, but I'm poor-ish.

So yeah, anything else?

I don't really think there is a problem now, but rather there MAY be one some years from now.

The users already give a VAST subsidy to miners, donate a decent chunk of their computing power and tx volume is by all standards trivial - so why do they have to pay extra to compete for quick transaction already now?

Considering sending BTC is a few electrons in a wire 1 penny really is too much - especially when relatively simple solutions exist (not just my idea here).


What are the counter arguments to an update?

You're the man. Fixed it.

I doubt they are that sinister. This is just a somewhat daunting update.

Additionally it likely wouldn't work as someone at some point would just make an S-node client and it would slowly overtake the network.

Until then your BTC is as safe as your password as always.

1. No, but if the network is groaning now its not a good sign.
2. Once you go from over-counter standard hardware to even simple super nodes, you need a geek to put it together and run it. Geeks have a high pay check so that is either a lot of money/time you ask others to forego or fees you will pay.

(20$/hour 5 hours/day 5 days/week 4 weeks/month 12m/y = 24.000$ then add other costs - it will be pricey)

The technical complexity of putting such a node together is likely equal to making an update of SOME kind anyway.


Its a non-problem now, it won't be forever.

I like this proposal and I think it would work.

However to avoid miners printing money I think the snapshots should be larger than 2 weeks: If someone had 51% for two weeks (not THAT hard to imagine) he could basically rewrite ledger history.

Maybe make it 5 YEARS: Keeps the chain from becoming infinite, but is still quite safe + most inputs before such date should have been spent, so the ledger becomes smaller too.

Win-win.


Anyone from this thread like my swarm proposal? (other thread)

+ 1

I cannot believe that someone thinks blocking a profiting business is MORE productive than discussing (if heated) a software update.

REALLY?

That would be like Google blocking its biggest advertisers because they dare generate so much traffic!

Read that like 3 times now.

It mentions pruning and that super nodes may scale - which I mentioned.

However I don't think that is enough.


Fitting you should mention those:
1. They use server farms to do that, huge ones.
2. They do load balancing, not everyone gets their youtube stream from one server.

The data of BTC is NOT the issue, the bad comm protocol IS.

To run a VISA network - according to the wiki - you would need a few racks of machines and an optic fiber internet connection to run a node.

The end points of said connection would also have to be reasonably fast.

This would all be very expensive, minimum 20.000 - 200.000$ to construct/maintain for just a year. (engineer time + pricy computers + beefed connection/subscription).

How many of such pricy nodes would we really end up with and what would the fee levels become?

Even a thousand not is not that believable and how vulnerable does that make us?

I sincerely doubt that Satoshi thought that:
1. O(n) was good design (you don't if you do software).
2. You should scale hardware rather than slightly fix issues.
3. That BTC should have ~1000 vulnerable/expensive nodes when ONE voluntary update could give us a million cheaper ones.


Even if the US does nothing about the "drug/terrorist/gun-money" that is bitcoin for their friends the banks (lol, they will) such super nodes would be in a position to levy very high fees.

We would be back to VISA level prices.

You again bring up an interesting point with torrents:
1. Does torrents have such things as super nodes? - NO, period.
2. Does any rich benevolent (magic) "hobbyists" successfully operate super nodes? - NO, period.

Traffic hiding is easy to add, TOR does it.

Our program has a huge flaw... OKAY: SCALE THE HARDWARE!

S-nodes would help adoption and give everyone lower fees. Complexity is low to medium.

Yes, it may take half a year for two programmers to do it, but it would pave the way for the future of bitcoin and allow 1000 fold increase in usage.

Just building ONE of your super nodes would like take more TIME and MONEY than doing this update.


Anyway unless someone can find a flaw in my proposal, my work is done. Use it when your mining pools and various clients crash over satoshidice 2.0 - you get to listen to all the media laughing at bitcoin in the meantime.

[UPDATED 10-04-2013]


In response to this thread:
https://bitcointalk.org/index.php?topic=87444.0 (satoshidice spam)

Problem:
The bitcoin network basically has what is called O(n) algorithmic performance.

In short this means that if the network grows 100 times, you need pretty much 100 times more powerful nodes and 100 times more bandwidth for EACH node.

While this is possible from a technical perspective it would quickly make the entire BTC network dependent on a few exposed super nodes.

Such nodes would very likely extort huge fees and/or be shut down by the US government.

Pruning or ledger blocks solves the HDD issues, but does nothing to spread out CPU, RAM or bandwidth loads.

Solution:
Future clients should be swarm clients that do the roles of full nodes, but only on parts of the chain.

The simplest explanation I have is this:
Imagine "cutting the blockchain in two halves". Now you run two clients on two computers that look at each their half of the chain or rather each their half of the addresses in the blockchain.
Now if one client sees a block spending an output already spent it reports it to the other client and they both reject the block. To prove such claims the reporter includes the TX and its minified merkle-branch up to the block header it occurred in - so the reporter cannot fake a report.
To the outsider such two clients would act and behave in every way as a full node - imagine chopping up the blockchain further and you have a system of swarm nodes!

I have already pledged 15 BTC to the implementer of this - mark me a scammer if I don't pay up please. Some escrow solution is also possible.
I will also accept less elegant solutions that split up the blockchain just 5 times or more.

Feel free to pledge along with me, it will be worth some of your BTC to secure the value of the rest I think

Nature of my solution
+ No fork.
+ No new blockchain standards.
- New communication protocol.
+ Bridge can be made between old and new clients.

Detailed solution:
Abbreviations:
F-node: Full node.
Could choose to mine independently of a pool, does verification on the whole blockchain and so on.

S-node: Swarm node.
In a swarm acts the same as a network of full nodes, but only does verification on parts of the chain.

L-node: Light node.
Is dependent on super nodes.

B-Node: Bridge node.
Bridges two ways of sharing blocks by "translating" the format of one to the other and broadcasting it.
Only one such node is theoretically needed to prevent forking.

Branches/trees:
See Merkle trees in wikipedia - the existing way Bitcoin stores txs.
The important thing here is that S-nodes send block fragments in the form of merkle trees tied to the block header whereas todays clients send whole blocks.

Subscriber:
A client which has a standing request for updates concerning specific addresses.

W-addrs: Watched addresses.
The Bitcoin addresses a swarm client is watching and storing the full history for. To keep itself updated it will subscribe to information about these addresses with other S-nodes.
S-nodes will relay information and TXs to each other even if

S-node group:
Say half the S-nodes watch addrs ending with an equal hex number and the other half unequal - each half would then be a "group". Members of the same group watch the same addresses, can then more easily share data and will have to connect to fewer relay nodes.

NEW IMPORTANT - Addresses:
The blockchain does in fact not store BTC in addresses, this is an abstraction, but rather allows movement of BTC if a script evaluates to true. With normal Bitcoin addresses this script looks something like this:
    1<insert signature> 2<Check it fits with public key:> 3<BTC-address>
However the script could also require multiple signatures or go by IP.
Hence S-nodes will not actually organize by addresses, but by the scripts - a redeem script will always look the same for the same address and/or the same conditions, hence it can be used as an identifier.
In most normal cases this will mean S-nodes basically use addresses as expected, but a multisig or IP script will just be handled as another address would: it has a balance or it doesn't.
If we imagine the hash or just first 50 bits of a script are between 0-1 then one S-node group could be 0-0.5 and another 0.5-1 and it would not matter at all what kinds of scripts were thought up or used.
<TX ID - Script/Condition hash - Spending TX ID/UnspentCode> would be all an S-node saw.

Miners forming a new block entirely with S-node system:
1. Miners in a pool will split themselves up into groups by the addresses their S-clients watch.
2. Each group will handle TXs that send funds from their W-addrs. (If there are multiple input addrs in the TX the first input will determine which S-node group puts the TX in the block, but each relevant group validates their W-addrs inputs.)
2A. Bad TXs inserted by dishonest miners will be rejected/reported by the other miners of the same S-node group.
3. From this they construct branches of the block and transmit these.
4. A pool master collects branch headers and makes a header hash which can be mined - no super node required to validate the whole thing.
5. Header hash is broadcasted and solved by the miner clients.
6. Bridge node gathers all the block strands into one traditional block.
7. Bridge node broadcasts the block to the older clients (or vice versa).

Clients with S-node system
1. Single small block branch is received from same S-node group.
2. Client checks all transactions it can and block header is valid.
3. The miners who solved the block send the sections to the various S-node groups on a on-demand basis branch by branch.

Security:
Bad TX:
If in step 2 a transaction is invalid due to either a hash error OR insufficient balance this is reported to the network.

1. Report is trustworthy because everyone can check that chain part themselves or that hash operation themselves.
2. The IP that sends the invalid branch/block is automatically added to a client IP-filter.
3. If an balance error report is received that turns out to be false, then that IP is blocked automatically.

If your connection is hijacked:
You don't need a web of trust or anything like that, just PKC; each client randomly selects a PK and publishes it.
Each client would then have its own list of Pub.Ks or "client addresses". For instance I might add MtGox, my brother and some others.
This doesn't mean I trust these guys it just means I find it near impossible that they would ALL know what the attacker wanted left out AND either have been hacked or colluding with him.
If I cannot connect to them because the attacker cannot fake them without their Pri.Ks I will know my connection has been hijacked.

Notes
All of this means that no S-client has to process, receive nor store the full blockchain. Or: O(log(n)) -> network grows 100 times you only need say 2 as powerful nodes.

Full trust-less security is maintained.

Even in the case of reports you can either trust the multitude of clients watching a certain address balance or reconstruct the entire chain from branches around the network.

Implementation:
It is my understanding that today the client has certain "get block" methods.

While the block standards would remain the same the S clients would basically ignore such calls and replace them with at least:
1. Reporting.
2. Get branch.
3. Subscribe to address/group updates.

By having S-node groups sharing information with themselves we can continue operating with just a couple of peers (many peers overload routers and stuff).

Using a bridge no one would even need know you had S-nodes behind it.

Am I missing anything?

EDIT: Changed title to be more informing.
EDIT, Update 10-04-13: Clarified some things and improved solution with my now clearer understanding of the Bitcoin protocol.

This is all so wrong:

1. The current network is larger than what 5 super computers?
2. You can't handle an absolutely small amount of transactions?
3. Solution: Throttle BTC with fees/blocking/self-prioritizing txs?!?!

NONONONONONO!


This is a serious issue: The BTC network is not scaling AT ALL due to horrible design.


So was my idea with SPV nodes keeping balances documented with input/outputs viable or not?

That way they store a fraction of the chain, but can do everything a full node network can in swarm fashion?

I don't think it is that dire no.

If it was, fee rates would have shot through the roof and killed satoshidice.

What is needed is a long term and scalable solution to higher BTC volumes. Fees will kill spam just fine until then.

Okay, how about this then:
1. The blockchain stays as is.
2. SPV nodes maintain knowledge of the balance of some addresses. They also maintain the documenting tx/block trail leading to this balance.
3. Invalid transactions can then be reported using this information (inputs).

Basically rather than storing chain links you would store chain strands?

The full block chain could then be constructed using strands, SPV nodes can verify and we have a 1-tier network!

Hmm.. forgot that...


Similar to "my idea", but see other guy above.

Basically your idea would work, but to verify the inputs of transaction existed you would have to go through the entire chain in the cloud = huge bandwidth usage.

So: "Houston we have a problem"?

Hiding behind TOR or something becomes hard with "trusted" super nodes and VISA volumes so unless we solve this, BTC will remain a small network forever and/or get busted up by governments.


Could we change the nature of blocks so that payments are not made up of only transactions, but also balances?

We know the block chain is trustworthy due to work proofs -> Just put address balances in it?

Could we do this without forking? "BIP 34" or something?

That way I could store the balances of maybe 1000 addresses and I simply verify transactions going out from them? IE verifying inputs against a partial chain?

I shall not be paying for your silence

The idea is to avoid the need for full nodes: I want a 1 tier network.

After re-reading the scalability wiki it seems you could split the block up in to a list of hashes.

This ALSO means you should be able to document an invalid tx with just a few hashes sent along with your report.

Yes except they DO store part of the chain AND they do verify against those parts.

Alone none of the nodes are full, but together they act as a network of full nodes.

Alright my bad.

In that case do "peer switching" if all your 8 peers don't have what you need make them give you one of their peers that DOES.

That way ALL nodes could be storing/sending/verifying >>1% and still function as a full node network with the same safety.

EDIT: Miners would still need to build and send full blocks, however building said blocks could become outsourced to mining pool clients. (from the scalability wiki)

+1

I don't get this talk of alternate chains or why it should be so damn difficult.

If there is a block with an invalid tx someone reports it over the peer network. The bandwidth usage, even if you have send the documenting block, would be negligible as it would happen so rarely.

To get your own balance you query your peers as with torrents and if there's an update they send that block to you.

To work you would need more than 8 peers/a smarter protocol, but that's hardly rocket science.


That is like 2-3 updates, what reason NOT to do it?

Exactly! I am glad someone else gets it.

Right now the BTC network is very small, but the VISION that we have all invested in is HUGE -> "BTC replaces dollar/VISA/Mastercard".

To get there you can't have the network breaking down over something like satoshidice NOR miners starting to ask ridiculous fees.


If you just want BTC to be another small time linden dollar with no future wtf are we all doing here?


Yes a bunch of super computers COULD run the BTC network at VISA size with normal users being light clients, BUT that would leave us wide open to attacks, being shut down or miner-dictatorship = VISA.

Pruning needs to get done or yes bitcoin might not die per say; it will just become pointless.

Bitcoin will die unless this is solved, its not much but I pledge 15 BTC to whoever creates a viable beautiful pruning-client.

Something which can operate with parts of the chain and only get the parts it needs to confirm incoming otherwise.

Should work like a torrent file where the full file could be distributed.


Basically my client should operate as before thinking it has the full chain, only it doesn't - it gets the parts it doesn't when needed from the cloud/peers.

After using such parts they are deleted.

It would be extra cool if parts of blocks could be downloaded instead of full blocks.


Maybe it could send a request to the network to get an incoming transaction verified by the nodes that have that block section?

Say my node has a block with A->B someone sends me a transaction with B->C, I check my block and verify B address has enough then okays it.

To clarify:
1. Blocks are verified as today via their expensive hash/tree root thingie.
2. All the transactions IN that block are verified by the network via the distributed block chain.

Time line:
1. Block is received ->
2. Full prune node (FPN) checks root hash against its own mostly empty chain ->
3. If okay, it checks the transactions it CAN ->
4. If okay it broadcasts the block while assuming the other transactions are fine ->
5. If NOT okay it does NOT broadcast it, but instead reports it including info about the invalid transaction and its block ->
6. Other nodes get the report, check the root hash of that transaction against their empty chain ->
7. If bad report is true they re-broadcast it ->
8. Bad blocks get propagated WAY slower AND are immediately turned down by super nodes ->
9. No major changes to core BTC logic, SMALL additional bandwidth usage for reporting.

If this is not done, BTC is dead.

Another idea:
"Allow transmission of half-blocks."
root hash <- 2nd hashes <- 3rd hashes etc.

You basically treat the second or third hash as THE block - the benefit is that slower connections can keep up with very large blocks as they only look at one branch of the blocks.
They would however need to check their own users addresses via 3rd service.

This is however not critical at present.

Low velocity might theoretically be a problem if everyone has bitcoins and no one is spending them on what needs to be done (food, energy, what ever).

HOWEVER this is NOT happening: Even now as BTC is still rising rapidly, people are still investing them into various projects.

Later in bitcoin's adoption stage - as others have mentioned - at some point you HAVE to buy bread to feed yourself even if BTC is rising a billion% a second.


The scares of "low velocity"/"deflationary spiral" and "instability"/"currency fluctuation" have NO empirical basis behind them, it's just propaganda for state fiat.

The first simply doesn't exist and the second isn't all that dangerous when the trend is up (BTC/gold deflation), instead of down (fiat inflation).

+1

(I said wait a day though.)