Scaling Bitcoin Above 3 Million TX pre block

[brg444]

Well, I think one problem with only sending pointers is what happens when a node doesn't have a particular transaction?  How does it get it?

Nodes validate & store all transactions of every block added to the chain. If that is not the case then it is not Bitcoin.

[jonald_fyookball]

Well, I think one problem with only sending pointers is what happens when a node doesn't have a particular transaction?  How does it get it?

would just ask around for the ones it's missing

it would be interesting to know exactly how insync a node from chain and one from US are

it would be interesting.  I have a feeling it might be an issue.

[jonald_fyookball]

Well, I think one problem with only sending pointers is what happens when a node doesn't have a particular transaction?  How does it get it?

Nodes validate & store all transactions of every block added to the chain. If that is not the case then it is not Bitcoin.

maybe you misunderstood.  in the 'relay only' mode being hypothesized by Adam, the blocks are never fully broadcast, only the transaction headers or something.  If a transaction was referenced but wasn't in the nodes mempool it would have to be fetched.

[adamstgBit]

Well, I think one problem with only sending pointers is what happens when a node doesn't have a particular transaction?  How does it get it?

Nodes validate & store all transactions of every block added to the chain. If that is not the case then it is not Bitcoin.

is it true miners have all the recent TX's in there mempool and when they get a new block they receive those same TX's all over again with the newblock?

is this a little redundant? maybe we can broadcast only the transaction headers or something.

[brg444]

Well, I think one problem with only sending pointers is what happens when a node doesn't have a particular transaction?  How does it get it?

Nodes validate & store all transactions of every block added to the chain. If that is not the case then it is not Bitcoin.

is it true miners have all the recent TX's in there mempool and when they get a new block they receive those same TX's all over again with the newblock?

is this a little redundant? maybe we can broadcast only the transaction headers or something.

That is what the relay network does but once the headers are broadcasted the miners still need to fully validate it before starting mining a new block on top.

Again, this has absolutely no impact on the full nodes that are not mining and need to receive, validate and store full blocks.

[brg444]

Well, I think one problem with only sending pointers is what happens when a node doesn't have a particular transaction?  How does it get it?

Nodes validate & store all transactions of every block added to the chain. If that is not the case then it is not Bitcoin.

maybe you misunderstood.  in the 'relay only' mode being hypothesized by Adam, the blocks are never fully broadcast, only the transaction headers or something.  If a transaction was referenced but wasn't in the nodes mempool it would have to be fetched.

....

[jonald_fyookball]

Well, I think one problem with only sending pointers is what happens when a node doesn't have a particular transaction?  How does it get it?

Nodes validate & store all transactions of every block added to the chain. If that is not the case then it is not Bitcoin.

is it true miners have all the recent TX's in there mempool and when they get a new block they receive those same TX's all over again with the newblock?

is this a little redundant? maybe we can broadcast only the transaction headers or something.

That is what the relay network does but once the headers are broadcasted the miners still need to fully validate it before starting mining a new block on top.

Again, this has absolutely no impact on the full nodes that are not mining and need to receive, validate and store full blocks.

yes they will validate it but the point is the transmission payload is reduced in size.

[Soros Shorts]

is it true miners have all the recent TX's in there mempool and when they get a new block they receive those same TX's all over again with the newblock?

No miners don't necessarily have all the recent tx's in mempool. In fact, it is possible to run a mining node with a heavily filtered mempool, e.g. a very small mempool which excludes spam transactions or transactions with small/no fees.

In the OP you also did not address SPV clients. Assuming a block header size of 4MB, if I turn off my Android client for 1 week and I turn it back on I would have to download about 4.036 GB worth of block header information to sync up a week's worth of transactions. It would take too long to do so. Furthermore, after I'm done would have used up the monthly cap of my wireless plan.

[CryptInvest]

Why bitcoiners opposed Lightning Network? The third party can't access to the funds an issue with the speed of the transaction (and indifferent to the blocksize) and the size of blockchain.

[RoadTrain]

Well, I think one problem with only sending pointers is what happens when a node doesn't have a particular transaction?  How does it get it?

A node simply asks its peers about it.

But then another problem might arise that I'd like to discuss (I didn't see any simulations). What if a node has to request a fair amount of transactions from its peers? Considering that every request has some latency (latency != bandwidth), and some peers might still not have this particular tx due to uneven propagation, can this process end up slower than simply bundling all txs with a block? If so, at which blocksizes and which missing txs percentages?

[coinplus]

I first got excited on hearing lightning network pegging into bitcoin so than get faster transactions than visa network. But allowing third party into bitcoin is not acceptable by our bitcoin community. May be some payment processors use lightning network for their own use of fastening transactions.

[sAt0sHiFanClub]

Well, I think one problem with only sending pointers is what happens when a node doesn't have a particular transaction?  How does it get it?

Nodes validate & store all transactions of every block added to the chain. If that is not the case then it is not Bitcoin.

is it true miners have all the recent TX's in there mempool and when they get a new block they receive those same TX's all over again with the newblock?

is this a little redundant? maybe we can broadcast only the transaction headers or something.

That is what the relay network does but once the headers are broadcasted the miners still need to fully validate it before starting mining a new block on top.

Again, this has absolutely no impact on the full nodes that are not mining and need to receive, validate and store full blocks.

Normal nodes are not time critical in receiving blocks. They can continue as normal even with much larger blocks. Only miners who are racing to create new blocks and need to know which one they have to build off get their knickers in a twist if there is any propagation impedance ( which would be exasperated by larger blocks)

I mentioned earlier that this redundancy needs to be addressed. A development of the ideas of the relay network should be an option for all nodes.  If they have 90% of the tx's in a block already in their mempool then they only need to adopt an efficient polling scheme to request the missing tx's from their peers.

[adamstgBit]

is it true miners have all the recent TX's in there mempool and when they get a new block they receive those same TX's all over again with the newblock?

No miners don't necessarily have all the recent tx's in mempool. In fact, it is possible to run a mining node with a heavily filtered mempool, e.g. a very small mempool which excludes spam transactions or transactions with small/no fees.

In the OP you also did not address SPV clients. Assuming a block header size of 4MB, if I turn off my Android client for 1 week and I turn it back on I would have to download about 4.036 GB worth of block header information to sync up a week's worth of transactions. It would take too long to do so. Furthermore, after I'm done would have used up the monthly cap of my wireless plan.

4GB is your month cap? wtf you want your phone to download the blockchain??? thats nuts you need SPV client.
isn't there some block pruning happening currently so that older blocks become much smaller?
isn't the whole point of SPV to let some server deal with downloading and validating the blockchain?
well then that miner will simply be at a slight disadvantage because he will need to ask peers for the TX he is missing, but even then its  likely that getting only the TX he's missing would be faster then downloading the new block in full.

[sAt0sHiFanClub]

I first got excited on hearing lightning network pegging into bitcoin so than get faster transactions than visa network. But allowing third party into bitcoin is not acceptable by our bitcoin community. May be some payment processors use lightning network for their own use of fastening transactions.

Lightning Network is a load of bollox at the moment, so I wouldn't worry about it too much. Its just another alt-coin, with cheap lipstick applied, in its present vapour form. I watched Poon and the other guy try to sex it up during their presentation and failed miserably. They played down the very complex challenges of tx malleability that need to be addressed before this will ever get off the paper stage. The current test network has absolutely no meaningful* interaction with the blockchain, and unless they can perform sighash miracles in the near future, it never will.

* and where it does, it requires Trust.  Yes, trust. In a trust-less network. Such progress.

[adamstgBit]

Well, I think one problem with only sending pointers is what happens when a node doesn't have a particular transaction?  How does it get it?

A node simply asks its peers about it.

But then another problem might arise that I'd like to discuss (I didn't see any simulations). What if a node has to request a fair amount of transactions from its peers? Considering that every request has some latency (latency != bandwidth), and some peers might still not have this particular tx due to uneven propagation, can this process end up slower than simply bundling all txs with a block? If so, at which blocksizes and which missing txs percentages?

right this is the kind of things we'd need to investigate...

the request for missing TX's could be done all at once **here's the list of TX i don't know about if anyone knows about these tell me** kinda thing. a distributed relay network whose sole purpose is to collect and relay TX accross the network would help.

also in theory a miner could include a bunch of TX's the network has never seen at which point other miners would be busy asking every other miner about TX's that no other minner has seen. i guess miners could consider that block invalid and orphen it. part of the protocol for miners including TX that he knows haven't been seen by the network would be to include the full TX... and also miners would try not to include TX that have likely Not fully propagated throughout the network ( TX they just heard about <10sec ago ?).

[sAt0sHiFanClub]

right this is the kind of things we'd need to investigate...

the request for missing TX's could be done all at once **here's the list of TX i don't know about if anyone knows about these tell me** kinda thing. a distributed relay network whose sole purpose is to collect and relay TX accross the network would help.

also in theory a miner could include a bunch of TX's the network has never seen at which point other miners would be busy asking every other miner about TX's that no other minner has seen. i guess miners could consider that block invalid and orphen it. part of the protocol for miners including TX that he knows haven't been seen by the network would be to include the full TX... and also miners would try not to include TX that have likely Not fully propagated throughout the network ( TX they just heard about <10sec ago ?).

Plenty there to start off a bit of requirement gathering.   

Checking other nodes ( 3 steps, ~350 peers) would be pretty quick. If you didnt get a response for a tx from that many, its time to think of ignoring the block and picking another.

[TransaDox]

right this is the kind of things we'd need to investigate...

the request for missing TX's could be done all at once **here's the list of TX i don't know about if anyone knows about these tell me** kinda thing. a distributed relay network whose sole purpose is to collect and relay TX accross the network would help.

also in theory a miner could include a bunch of TX's the network has never seen at which point other miners would be busy asking every other miner about TX's that no other minner has seen. i guess miners could consider that block invalid and orphen it. part of the protocol for miners including TX that he knows haven't been seen by the network would be to include the full TX... and also miners would try not to include TX that have likely Not fully propagated throughout the network ( TX they just heard about <10sec ago ?).

Plenty there to start off a bit of requirement gathering.  

Checking other nodes ( 3 steps, ~350 peers) would be pretty quick. If you didnt get a response for a tx from that many, its time to think of ignoring the block and picking another.

.....and/or if blocks were held on NNTP servers, I2P nodes, Tor servers etc you could download from there when peers don't have what you want

I think  Just-In-Time Block Requesting is an obvious step. The debate would be around if we want the clients/nodes/miners to keep track of  blocks to ensure full coverage or whether to use existing remote storage systems but the first step would use the latter.

[sAt0sHiFanClub]

right this is the kind of things we'd need to investigate...

the request for missing TX's could be done all at once **here's the list of TX i don't know about if anyone knows about these tell me** kinda thing. a distributed relay network whose sole purpose is to collect and relay TX accross the network would help.

also in theory a miner could include a bunch of TX's the network has never seen at which point other miners would be busy asking every other miner about TX's that no other minner has seen. i guess miners could consider that block invalid and orphen it. part of the protocol for miners including TX that he knows haven't been seen by the network would be to include the full TX... and also miners would try not to include TX that have likely Not fully propagated throughout the network ( TX they just heard about <10sec ago ?).

Plenty there to start off a bit of requirement gathering.  

Checking other nodes ( 3 steps, ~350 peers) would be pretty quick. If you didnt get a response for a tx from that many, its time to think of ignoring the block and picking another.

.....and/or if blocks were held on NNTP servers, I2P nodes, Tor servers etc you could download from there when peers don't have what you want

I think  Just-In-Time Block Requesting is an obvious step. The debate would be around if we want the clients/nodes/miners to keep track of  blocks to ensure full coverage or whether to use existing remote storage systems but the first step would use the latter.

We are not looking for blocks - just transactions.  Miners will push/publish blocks when they find a solution. Its just to confirm that the transactions they claim make up that block exist - nodes should already have the vast majority in mempool, but due to slight differences there is a requirement to find missing ones. This can only be done through other peers.