Slowing down block propagation

[onemorexmr]

i know this sounds ridiculous, but let me explain why i think it may be useful.

i am nearly convinced now that a big block size together with 1BLT is a threat for bitcoin as miners dont have any incentive to make smaller blocks.

so i am thinking about ways to incentive them.

an easy solution could be that a block needs to be transmitted in full to be accepted - the question is, if this is enforceable between miners (sadly i think it isnt).


btw... not heard much about headers first. is the following scenario possible?

miner A mines a block with a few transactions he has crafted himself.
he published a double-spent-attempting transactions shortly before he found the block.

miner B receives the second transaction from miner A
miner B receives the new block header and starts mining
 - as he does not have the first transaction he tries to include the second one

^ this should raise miner-B's orphan rate.

an easy way to mitigate this would be to mine empty blocks until you have all transactions from the previous block. but this would lead to more empty blocks - raising the blocksize-scarcity even more

i am a little lost right now...

[2112]

Go back and read the Microsoft's "Red Balloons" paper. This has been discussed here and elsewhere many times.

[onemorexmr]

Go back and read the Microsoft's "Red Balloons" paper. This has been discussed here and elsewhere many times.

http://research.microsoft.com/apps/pubs/?id=156072
http://research.microsoft.com/en-us/projects/socialalgs/bitcoin-red-balloons.aspx

ok, add an incentive for nodes to broadcast transactions is nice, and solves the problem partly.

IMHO the other part is, that if miners are not punished for bigger blocks it means that they will put all transactions with fees in any block which leads to the problem that is not possible to build a fee-market.

its just rational to take them all: they have already received and validated them, putting them in a block does not add additional costs.

[DeathAndTaxes]

its just rational to take them all: they have already received and validated them, putting them in a block does not add additional costs.

This is a misnomer.  With IBLT the block propagation time is only O(1) is all other nodes also contain that transaction.  A miner has no way to of knowing if all nodes contain all transactions as if the txn volume exceeds the bandwidth availability or other resources of a given node then it will drop some transactions.  The smart miner would not take all txns especially not those with an almost zero fee but instead would try to estimate the resources of the entire network and choose the highest paying subset of txns that are likely to be in the memory pool of a majority of the nodes.

So yes there is still an 'additional cost' for including additional txns.  If the txn included isn't known by all nodes then it will have to be relayed directly and that increases block propagation time and orphan cost.

[onemorexmr]

i see two problems here.

If the txn included isn't known by all nodes then it will have to be relayed directly and that increases block propagation time and orphan cost.

as you stated a miner does not know which transactions are known by other nodes. so he does not have a choice: he must publish all transactions in his block.
and if i understand you correctly he must do this fast (a priori?) otherwise he risks an orphan (because other miners cant build upon his block).

^ if that is true all is fine: but thats definitely not O(1).

A miner has no way to of knowing if all nodes contain all transactions as if the txn volume exceeds the bandwidth availability or other resources of a given node then it will drop some transactions.

if - because of bandwith - not all transactions reach a node the node cant decide which one to take. maybe an upstream proxy could be developed but that sounds a little strange to me.

it gets worse, because different parts of the network will see and keep different transactions.

did i misunderstand you?

[2112]

This is a misnomer.  With IBLT the block propagation time is only O(1) is all other nodes also contain that transaction.  A miner has no way to of knowing if all nodes contain all transactions as if the txn volume exceeds the bandwidth availability or other resources of a given node then it will drop some transactions.  The smart miner would not take all txns especially not those with an almost zero fee but instead would try to estimate the resources of the entire network and choose the highest paying subset of txns that are likely to be in the memory pool of a majority of the nodes.

So yes there is still an 'additional cost' for including additional txns.  If the txn included isn't known by all nodes then it will have to be relayed directly and that increases block propagation time and orphan cost.

I think you are presuming that the miners are still using the legacy network protocol as implemented in the core client. I think it is already obsolete and anyone who mines uses the new "relay network" protocol implemented out-of-process by Matt Corallo.

https://bitcointalk.org/index.php?topic=766190.0

I'm going to assume that your analysis is obsolete purely because for anyone really mining (pools or large farms) the new protocol is the way to go.

And even if the new "relay protocol" isn't going to get included in the core client it is a no-brainer that any large scale progressive-thinking miners should have private connections between themselves, possibly secured with IPsec. Ideally they should have a multicasting backbone established amongst themselves (kinda like NASDAQ extranet), but I'm too cynical to believe that they are technically capable of that.

[DeathAndTaxes]

Out of process or not, you can't have O(1) block propagation time with IBLT unless the entity you are broadcasting a block to already had the transactions.  IBLT can't make up data that is not known, it can only allow other nodes to determine which subset of memory pool are included in the block.  The relay network doesn't change that dynamic in the slightest.  

[2112]

Out of process or not, you can't have O(1) block propagation time IBLT unless the entity you are broadcasting a block to already had the transactions.  IBLT can't make up data that is not known, it can only allow other nodes to determine which subset of memory pool are included in the block.  The relay network doesn't change that dynamic in the slightest. 

You again start writing with a style and confidence similar to Mircea Popescu. I mean the absolute value is similar and the difference is in the sign/direction.

Quite obviously: transaction propagation (1) and block header propagation (2) are two different things and there's no point to keep conflating the two. The respectable and not underhanded miner will propagate transactions first before propagating the discovered block headers. For the (1) part the average propagation time is in the order of 5 minutes, only in the (2) part every millisecond matters and directly affects the probability of orphans.

[TierNolan]

This is a misnomer.  With IBLT the block propagation time is only O(1) is all other nodes also contain that transaction. 

No, missing transactions are extractable from the IBLT. 

The rule is that the size of the IBLT must be greater than the difference between the sender and the receiver.  If the memory pool in both nodes is 1% different, then the size of the IBLT needs to be a few percent of the size of block.

If it became popular, then there is an incentive to keep memory pool policies are close to standard as possible.  If they could be kept nearly identical, then the IBLT can be kept very small.

The O(1) scaling assumes that block size can be increased without increasing the total number of differences.  In order to keep the IBLT the same size, if the block size doubles then the fraction of transactions that are different halves.

Miners could maintain a few different memory pools with different policies.  The IBLT would only need to (near) match one of them.

[jonny1000]

I know this sounds as bad as the slowing down blocks suggestion.  But an alternative would be to make miners pay higher fees for larger blocks.  An increasing haircut on transaction fees related to block size.  

e.g. 1MB block 0% haircut, 10MB block 10% haircut, 1000MB block 95% haircut.  Some polynomial could determine the formulae.

This would create a marginal cost of including more transactions in blocks.  The problem with this might be that the extra marginal cost doesn't differ enough for different miners on the mining cost curve.

[2112]

Miners could maintain a few different memory pools with different policies.  The IBLT would only need to (near) match one of them.

I'm not sure that memory pools per different policies will be sufficient. Primarily because each pool will have different policy of accepting zero-fee transactions because the fee will be out-of-band.

Pools should start integrating with other businesses like exchanges or casinos, and then each pool will have material incentive to support the integrated business.

So I'm more thinking a memory pool per different peered mining pool, especially because the number of mining pools should go down.

Anyway, for now it is just another way of counting the proverbial chickens before they hatch.

[onemorexmr]

I know this sounds as bad as the slowing down blocks suggestion.  But an alternative would be to make miners pay higher fees for larger blocks.  An increasing haircut on transaction fees related to block size.  

e.g. 1MB block 0% haircut, 10MB block 10% haircut, 1000MB block 95% haircut.  Some polynomial could determine the formulae.

This would create a marginal cost of including more transactions in blocks.  The problem with this might be that the extra marginal cost doesn't differ enough for different miners on the mining cost curve.

to me this doesnt sound bad at all.

what i'd love to see is that the additional "lost" fees are somehow distributed to the nodes transmitting the transaction (i dont have an idea how this could be accomplished without cheating though)

i know lost bitcoins are not a real problem, but i still have a bad "feeling" about it.

EDIT:
what about this:
i craft a new transaction with 1btc mining fee
 - some node receives this transaction and signs it.
 - it now relays the following data
     # tx/txouts
     # his own ip/port
     # one of his own addresses
     all signed by his key (not the key of the original txin's)
 - a miner receives multiple versions of this transaction signed by different "first" relay nodes
 - he now contact one of this nodes and ask for the original one
 - same magic needs to be done to make sure the miner is forced to send some fee to the first relay if the block size exceeds 1mb.

[gmaxwell]

e.g. 1MB block 0% haircut, 10MB block 10% haircut, 1000MB block 95% haircut.  Some polynomial could determine the formulae.

Diminishing fees doesn't work in the long run, as fees can be paid out of band. An alternative that has been proposed is adjusting the difficulty, though this only allows small adjustments.

[Gavin Andresen]

i am nearly convinced now that a big block size together with 1BLT is a threat for bitcoin as miners dont have any incentive to make smaller blocks.

Why do you want miners to have an incentive to make smaller blocks?

Smaller blocks means fewer transactions, so fewer opportunities to collect fees, so less profit.

Miner profit in fiat currency = number of transactions * average transaction fee * btc-to-fiat exchange rate

Experience (and common sense) says that more usage of Bitcoin means a higher btc-to-fiat exchange rate, so if you want to maximize miner's fee revenue then increasing the number of transactions is the obvious way to do it.

If you think that putting an artificial cap on the number of transactions will increase overall miner profit, then I urge you to find a Real Economist and talk to them about the wisdom of trying to use production quotas to keep prices artificially high.

[onemorexmr]

i am nearly convinced now that a big block size together with 1BLT is a threat for bitcoin as miners dont have any incentive to make smaller blocks.

Why do you want miners to have an incentive to make smaller blocks?

Smaller blocks means fewer transactions, so fewer opportunities to collect fees, so less profit.

Miner profit in fiat currency = number of transactions * average transaction fee * btc-to-fiat exchange rate

Experience (and common sense) says that more usage of Bitcoin means a higher btc-to-fiat exchange rate, so if you want to maximize miner's fee revenue then increasing the number of transactions is the obvious way to do it.

If you think that putting an artificial cap on the number of transactions will increase overall miner profit, then I urge you to find a Real Economist and talk to them about the wisdom of trying to use production quotas to keep prices artificially high.

i dont want any artificial cap. in fact i want an unlimited blocksize (but i think your proposal is ok).

i just think that we need an incentive for a miner not to put any transaction in a block but only those which pay a high enough to cover his(!) costs.

my reasoning:
 - good distribution of bitcoin nodes through various asn (which also means countries which dont have a very good connection or cheap vps's as we are used to)
 - higher costs for transaction spammers

i do just fear (as explained above) that without any incentive for smaller blocks miners will take any fee paying transaction.

i am not really sure about the consequences of 1BLT/headers first. some people say that its a O(1) operation others say it isnt

if it isnt: all is fine, because i feel that this is incentive enough.

(i know O-notation does not really fit bandwith; but i am sure you'll get what i mean).

[cheako911]

One cannot gain ground by implementing a solution that's susceptible to the same conditions the solution was meant to solve.

We pay ppl for mining a block and they prove this with a smaller number after a double round of sha256 than anyone else.  How do TX relayers get paid?  I see one of two possibilities.

1. It's advantageous to append to the list of really proofs because you'll be paid more for not discarding proofs supplied by others.
2. The relayers will discard all other proofs and forward the TX directly to the miners with just them being paid.  The miner will strip off that proof and supply their own.

In the case of 1, miners will generate a set of proofs as long as they can...  Even if they have to supply millions of watts to dedicated ASIC chips to do so.

I don't see how one intends to 'prove' to the blockchain that they relayed a transaction, without someone else being able to undo their efforts.  Explain this shit to me.

The best I could come up if is proof by guided tour, but then who pays the tour guides and how do we pay the payers of the guides?  It's an endless loop.

[onemorexmr]

I don't see how one intends to 'prove' to the blockchain that they relayed a transaction, without someone else being able to undo their efforts.  Explain this shit to me.

my idea was that the first full node which sees the transaction does (instead of relaying the transaction) relays his contact info. the miner would need to contact one of this nodes in order to get the real transaction.

only thing i am unsure about is how the miner can proof to that node that he will really try to add the node-paying-transaction to the block.

i think paying only the first node is not bad

[Gavin Andresen]

How do TX relayers get paid?

Why do we need transaction relayers? What vital function do they provide?

Miners need to be connected to each other, and to transaction creators (individual users, exchanges, merchants, online wallets, etc).

And transaction creators need to be able to connect to miners, but it seems to me transaction fees should certainly be enough incentive for miners to arrange for there to be plenty of opportunity for transaction creators to send them fee-paying transactions (it's cheap to run nodes that have tens of thousands of incoming connection slots).

[jonny1000]

e.g. 1MB block 0% haircut, 10MB block 10% haircut, 1000MB block 95% haircut.  Some polynomial could determine the formulae.

Diminishing fees doesn't work in the long run, as fees can be paid out of band. An alternative that has been proposed is adjusting the difficulty, though this only allows small adjustments.

This is a good point.  Adjusting the difficulty is a better solution, however this seems to undermine Bitcoin’s core longest chain rule security mechanism, which is too much for a peripheral mining incentives issue.

Miner profit in fiat currency = number of transactions * average transaction fee * btc-to-fiat exchange rate

Should this not be:
Miner revenue in fiat currency = number of transactions * average transaction fee * btc-to-fiat exchange rate?

You may be right that more transactions will increase the BTC to fiat exchange rates and that two of the variables in that equation increase, however if the fee falls then overall mining revenue can fall.  This is possible and merely depends of the elasticity of demand.

In economic theory, there is the well known monopoly idea where supply is constrained to increase profits.  This is not ideal, but it may be possible.


Source: http://en.wikipedia.org/wiki/Monopoly_profit

If you think that putting an artificial cap on the number of transactions will increase overall miner profit, then I urge you to find a Real Economist and talk to them about the wisdom of trying to use production quotas to keep prices artificially high.

You are correct that artificially implementing production quotas to increase prices is normally a bad idea in almost all markets.  In the market for Bitcoin transaction fees, which has many unique characteristics in my view, implementing this is not perfect either and is likely to cause problems.  However there is no perfect solution and I think an arbitrary quota is something worth seriously considering.  Please don’t misrepresent what I am saying as being in favour of a 1 MB limit, as I am not.

If the marginal cost of adding additional transactions to blocks is effectively zero, then that’s a pretty unique situation.  If the market is competitive then price = marginal costs, which means miners profit will tend to zero.  It is not impossible that imposing this quota will increase miners profit from zero.  In my view a differential marginal cost of adding transactions to blocks for different miners, is desirable, that way one can have a cost curve dynamic which makes the market more robust.  If the marginal cost of adding transactions to blocks is uniform across all miners, I think we may run into some problems.

[Gavin Andresen]

Miner profit in fiat currency = number of transactions * average transaction fee * btc-to-fiat exchange rate

Should this not be:
Miner revenue in fiat currency = number of transactions * average transaction fee * btc-to-fiat exchange rate?

Yes, I meant revenue, not profit.

I need to stop saying "profit" entirely, even when I mean "profit"-- it has different meanings for economists and ordinary people.