Decentralized mining protocol standard: getblocktemplate (ASIC ready!)

[DrHaribo]

Lulz - I can post at 2.6 and still see you've posted crap.

Being polite and working together towards a common goal is crap? I am surprised that you participate in an open source project.

Give some grounds for this crap

It's unfortunate that we now have two competing standards. It's more unfortunate that another useless bitcointalk war is breaking out over it.

As usual many of the arguments are just not true, or misunderstood. "You have to send in all the transactions with every share. Think of all those bytes!" No, you don't.

or be gone.

You like to make a lot of demands, don't you?

ckolivas wrote that if the miner selects transactions you have to send in every transaction with every share. This is incorrect and has been discussed enough already, I think. You said that GBT, a protocol, makes decisions on which transactions to include. How does a protocol make a decision like that? And GBT being "an abortion", I don't think that's entirely correct either.

I don't care if you designed GBT ... ... ... well actually that would be even worse coz that would make you Jesus.

I never said I designed GBT. I said I suggested a few improvements for it. And what does religion have to do with this?

You told me to "fuck off" because I don't contribute. I was merely stating that my contributions are already in the spec. Where are yours?

Your pool ran 15 blocks that I pointed out, and of them, only 5 were large txn count/size.
5 were tiny. 5 were in the middle.

You picked 15 random blocks from BitMinter. They were 5 big blocks, 5 medium and 5 small. And so you come to me and demand an explanation? I find that utterly unreasonable.

If you don't like pools that filter out transactions, then talk to them, not me. I don't put any limits on transactions.

You claimed that you don't choose the transactions, GBT does ... wow

No, you said that, not me. What I said was that I include all the transactions bitcoind gives me, without filtering out the free ones or putting any limit on the number of transactions, like some pools do.

GBT and bitcoind are not the same thing. If a bug in a web server makes it give you half a file, do you blame HTTP?

Next, GBT currently usually sends more data per minute than a GetWork pool with roll-n-time (probably even if it is ignoring transactions or there are few transactions on the network)
This is due to the fact that it sends all the details of the all transactions you are processing (no matter what, you cannot disable this and request a merkle tree)

Maybe that would be a good improvement for GBT. If the miner doesn't care about seeing the transactions or filtering/adding transactions, request a merkle branch instead that you can use to create a merkle root for mining. Not a single transaction goes over the wire, other than the generation tx.

Next, GBT uses HTTP not raw sockets, so it is slower and is also expected to have more rejected shares due to this since it isn't a constant connection.
Not using HTTP, Luke-Jr has stated as a negative in Stratum coz it takes more effort to implement a better protocol ... wow gotta love that argument.
Sounds like "Don't implement something better coz it will take more effort"
Or even "Luke-Jr sux at programming and can't do anything that isn't simple"

Of course HTTP is a constant connection. Use persistent connections, and pipeline requests for speed. My miner uses 1 long poll connection and 1-3 connections to get work and send in proofs of work. For the work connections it will pipeline up to 10 requests at a time. All the connections are persistent. They get closed only if they have been sitting idle in the connection pool for a while.

The advantage of Stratum is that it is designed to be two-way from the start. Long polling in HTTP feels dirty and hackish.

The advantage of HTTP is all the network infrastructure, proxies, libraries that exist, and all the knowledge that developers have about this protocol. When you already know the protocol, you have a library for using it that you already know well, then it is much easier to get started.

[1715598373]

1715598373

[1715598373]

1715598373

[slush]

How goes getting StratumMP BIP process off the ground? Are we still waiting on genjix?

No, I'm just busy. Writing sane documentation (in english) is pretty hard for me, so I need a lot of time to finalize something.

"There is reasonable concerns about mining currently being too centralized on pools, and the amount of control these pools hold. By exposing the details of the block proposals to the miners, they are enabled to audit and possibly modify the block before hashing it."

You didn't respond to Stratum's "get_transactions" extension explicitly. Can I ask you what's your opinion to this? I think it solves exactly that issue with blind trust for picking transaction by the pool, without unnecessary overhead for common miners.

This is just trolling.

I don't think so. I prefer coding, not discussion. We can spend months on discussions or we can both show our solutions and let others to decide.

[Luke-Jr]

How goes getting StratumMP BIP process off the ground? Are we still waiting on genjix?

No, I'm just busy. Writing sane documentation (in english) is pretty hard for me, so I need a lot of time to finalize something.

Well, we're just talking about a first draft, it doesn't have to be perfect - the rest of the community will probably have things they want added or changed before the final anyway.

"There is reasonable concerns about mining currently being too centralized on pools, and the amount of control these pools hold. By exposing the details of the block proposals to the miners, they are enabled to audit and possibly modify the block before hashing it."

You didn't respond to Stratum's "get_transactions" extension explicitly. Can I ask you what's your opinion to this? I think it solves exactly that issue with blind trust for picking transaction by the pool, without unnecessary overhead for common miners.

It's a step forward, at least; but the design here still encourages miners and pools to avoid (or bill extra for, as you mentioned) decentralization. I don't think "common miners" should be encouraged to centralize.

I prefer coding, not discussion. We can spend months on discussions or we can both show our solutions and let others to decide.

It's not either one or the other. The BIP process generally involves documenting and implementing a first draft, and then discussion to come up with a final standard solution that meets everyone's needs. Eligius had GBT since the first draft in February, but the protocol continued to change significantly as other developers contributed to it before reaching its current form.

[Inaba]

When Luke was working on GBT and we were working on implementing a working vardiff, I had some concerns from a security standpoint.  The Stratum get_transactions also raises some red flags for me, can you address security/throttling measures in place to prevent get_transactions being used a DoS method to overload a pool?  Anything that leaves arbitrary data requests up to the client is an avenue for exploit and it is one of the reasons I have been so adamantly against user defined difficulties.  If I want to DoS your pool, I crank up a couple of 1.5 TH minirig, set my difficulty to 1 and ask for getworks. 

Same kind of problem with get_transactions:  I'm pissed at you, so I just request all the get_transactions you'll send me over and over and over.

[P_Shep]

When Luke was working on GBT and we were working on implementing a working vardiff, I had some concerns from a security standpoint.  The Stratum get_transactions also raises some red flags for me, can you address security/throttling measures in place to prevent get_transactions being used a DoS method to overload a pool?  Anything that leaves arbitrary data requests up to the client is an avenue for exploit and it is one of the reasons I have been so adamantly against user defined difficulties.  If I want to DoS your pool, I crank up a couple of 1.5 TH minirig, set my difficulty to 1 and ask for getworks. 

Same kind of problem with get_transactions:  I'm pissed at you, so I just request all the get_transactions you'll send me over and over and over.

Surely you could keep requesting get_works with a script, why would you need any hashing behind it?

As with the get_transactions, surely you'd just place a limit on requests poolside, and ignore those that seem unreasonable?

[kano]

When Luke was working on GBT and we were working on implementing a working vardiff, I had some concerns from a security standpoint.  The Stratum get_transactions also raises some red flags for me, can you address security/throttling measures in place to prevent get_transactions being used a DoS method to overload a pool?  Anything that leaves arbitrary data requests up to the client is an avenue for exploit and it is one of the reasons I have been so adamantly against user defined difficulties.  If I want to DoS your pool, I crank up a couple of 1.5 TH minirig, set my difficulty to 1 and ask for getworks.  

Same kind of problem with get_transactions:  I'm pissed at you, so I just request all the get_transactions you'll send me over and over and over.

You've ignored the reverse situation.

With GBT (all current implementations) the pool must always do this.

With Stratum it is optional and yes some users could request it always.

Since it is an available option, then the pool must consider the load of allowing it.
(... and yes I mean they MUST consider the load, not that they should ignore implementing it ... though any that don't would be easy to identify ... due to their failure to respond any miner that requests it)

[-ck]

Must I quote myself?

If every template request was this size and the miner used the customise coinbase option and chose transactions, they would be receiving 400kb for the block template and each share submitted would be 400kb unless they started withholding transactions.

That IS correct.

I was exploring the possible issues with implementation of GBT and got attacked as though I was trying to spread FUD about GBt. Nowhere did I imply this is what all miners would be doing.

[Luke-Jr]

Must I quote myself?

If every template request was this size and the miner used the customise coinbase option and chose transactions, they would be receiving 400kb for the block template and each share submitted would be 400kb unless they started withholding transactions.

Too bad you're more interested in ignoring me than having a productive discussion, or you'd have noticed I pointed out exactly why that isn't true.

[-ck]

When Luke was working on GBT and we were working on implementing a working vardiff, I had some concerns from a security standpoint.  The Stratum get_transactions also raises some red flags for me, can you address security/throttling measures in place to prevent get_transactions being used a DoS method to overload a pool?  Anything that leaves arbitrary data requests up to the client is an avenue for exploit and it is one of the reasons I have been so adamantly against user defined difficulties.  If I want to DoS your pool, I crank up a couple of 1.5 TH minirig, set my difficulty to 1 and ask for getworks.  

Same kind of problem with get_transactions:  I'm pissed at you, so I just request all the get_transactions you'll send me over and over and over.

Surely you could keep requesting get_works with a script, why would you need any hashing behind it?

As with the get_transactions, surely you'd just place a limit on requests poolside, and ignore those that seem unreasonable?

Since the point of get_transactions is simply to confirm the merkle branches, you only need to allow it once per stratum template push. All the stratum pools so far only push a new template every 30 seconds. Limiting get transaction requests to 1 for each template would only allow the miner to ask for it once every 30 seconds.

By the same token, GBT pools should limit how often users can ask for a template since that does send out all the transactions.

[kano]

Lulz - I can post at 2.6 and still see you've posted crap.

Being polite and working together towards a common goal is crap? I am surprised that you participate in an open source project.

May your surprise make you happy.

Give some grounds for this crap

It's unfortunate that we now have two competing standards. It's more unfortunate that another useless bitcointalk war is breaking out over it.

As usual many of the arguments are just not true, or misunderstood. "You have to send in all the transactions with every share. Think of all those bytes!" No, you don't.

or be gone.

You like to make a lot of demands, don't you?

ckolivas wrote that if the miner selects transactions you have to send in every transaction with every share. This is incorrect and has been discussed enough already, I think. You said that GBT, a protocol, makes decisions on which transactions to include. How does a protocol make a decision like that? And GBT being "an abortion", I don't think that's entirely correct either.

I still don't see any backup to the point you made "As usual many of the arguments are just not true, or misunderstood."

I don't care if you designed GBT ... ... ... well actually that would be even worse coz that would make you Jesus.

I never said I designed GBT. I said I suggested a few improvements for it. And what does religion have to do with this?

"In" joke

You told me to "fuck off" because I don't contribute. I was merely stating that my contributions are already in the spec. Where are yours?

No, because you made a VERY broad statement without any basis and contributed nothing to the current discussion except to tell people how they should be discussing it.

Your pool ran 15 blocks that I pointed out, and of them, only 5 were large txn count/size.
5 were tiny. 5 were in the middle.

You picked 15 random blocks from BitMinter. They were 5 big blocks, 5 medium and 5 small. And so you come to me and demand an explanation? I find that utterly unreasonable.

If you don't like pools that filter out transactions, then talk to them, not me. I don't put any limits on transactions.

What is your reason for wording it like that?

No I didn't pick 15 random blocks.
I picked 15 blocks in a row that your pool generated.

Your response was that it was out of your control.

The reason I chose those blocks was because one of them orphaned another block at the same time:

205766
BTC Sent  510.69 BTC
Number Of Transactions  32
By BitMinter

Orphaned:
BTC Sent  15,449.927 BTC
Number Of Transactions  619
By EclipseMC

Yes something is badly wrong there ... and your excuse is that you don't have any rules about the transactions, you just call the GBT interface.

[aside]
This is in fact a big issue with Luke-Jr also - him claiming about how what he is doing is better for BTC.
Yet what he was doing with his pool was restricting the number of transactions at his pool to 32 to increase the payments the pool received rather than fixing the problem with his pool software that was getting more orphans than other bigger pools at the same time.
The guy is a hypocrite - even worse: he enables such a thing at his pool ~5 months ago and the pool is still doing it.
I even explained a sizeable solution to this in bitcoind to both him and Gavin back then ...
[/aside]

You claimed that you don't choose the transactions, GBT does ... wow

No, you said that, not me. What I said was that I include all the transactions bitcoind gives me, without filtering out the free ones or putting any limit on the number of transactions, like some pools do.

GBT and bitcoind are not the same thing. If a bug in a web server makes it give you half a file, do you blame HTTP?

Tell me how that statement I said is wrong without hiding behind definitions.
There is no difference between what I said and you said except technicalities.
Yes it seems this statement keeps coming up.
This is a text book theory argument that one can make about something that is just that - a theory.
Yes the GBT proponents are out saying to use GBT because it is great.
The implementations are a measure of how great it is and falling back to this constant "It's only an idea you can't say GBT is bad" is pointless also.

Next, GBT currently usually sends more data per minute than a GetWork pool with roll-n-time (probably even if it is ignoring transactions or there are few transactions on the network)
This is due to the fact that it sends all the details of the all transactions you are processing (no matter what, you cannot disable this and request a merkle tree)

Maybe that would be a good improvement for GBT. If the miner doesn't care about seeing the transactions or filtering/adding transactions, request a merkle branch instead that you can use to create a merkle root for mining. Not a single transaction goes over the wire, other than the generation tx.

Love it - "MAYBE"
How blatantly obvious is it?
This is a good example of how it becomes just straight out annoying:

Use this great new protocol - it's better than before ........ except it sends more data than getwork ...... that yeah MAYBE that might be worth improving ......
But it's better coz the GBT guys say it is - it's transparent ... oh wait Stratum has the option to send the txn list that is 'missing' also.
Um ... what other excuse can they make up ...
Oh yeah it's just the implementations that suck ... GBT is good for everyone ............ except everyone who is using it with any network between the miner and the work generator ...

[Luke-Jr]

This is in fact a big issue with Luke-Jr also - him claiming about how what he is doing is better for BTC.
Yet what he was doing with his pool was restricting the number of transactions at his pool to 32 to increase the payments the pool received rather than fixing the problem with his pool software that was getting more orphans than other bigger pools at the same time.
The guy is a hypocrite - even worse: he enables such a thing at his pool ~5 months ago and the pool is still doing it.
I even explained a sizeable solution to this in bitcoind to both him and Gavin back then ...

The problem isn't with the pool software, as you clearly admit to knowing by mentioning fixing bitcoind. Until bitcoind is fixed and the fix is deployed everywhere, this problem remains.
Good job explaining the solution - even though your explanation doesn't actually solve it. Unfortunately, all contributions to bitcoind are done in peoples' spare time and things like a complete rewrite of the p2p code (which is needed to really fix this) takes a lot of time and skill. When this problem is nearly non-existent by simply filtering out a known DDoS attack, there are also higher priorities.

Oh, and stop trolling that GBT uses more bandwidth than getwork. It doesn't.

[kano]

This is in fact a big issue with Luke-Jr also - him claiming about how what he is doing is better for BTC.
Yet what he was doing with his pool was restricting the number of transactions at his pool to 32 to increase the payments the pool received rather than fixing the problem with his pool software that was getting more orphans than other bigger pools at the same time.
The guy is a hypocrite - even worse: he enables such a thing at his pool ~5 months ago and the pool is still doing it.
I even explained a sizeable solution to this in bitcoind to both him and Gavin back then ...

The problem isn't with the pool software, as you clearly admit to knowing by mentioning fixing bitcoind. Until bitcoind is fixed and the fix is deployed everywhere, this problem remains.
Good job explaining the solution - even though your explanation doesn't actually solve it. Unfortunately, all contributions to bitcoind are done in peoples' spare time and things like a complete rewrite of the p2p code (which is needed to really fix this) takes a lot of time and skill. When this problem is nearly non-existent by simply filtering out a known DDoS attack, there are also higher priorities.

OK what doesn't it solve?

The solution is based on the fact that
1) Most bitcoinds will already have every transaction that they are sent (all except the coinbase transaction)
So the protocol is sending a large amount of data that the receiving bitcoind already has and knows it has and doesn't need sent
2) The side effect of this is also that the receiving bitcoind again re-validates each transaction that it already has validated in it's memory pool

EDIT: oops left out this comment.
The problem was not with the bigger pools software, since they were not getting as high a % of orphans as you were AT THE SAME TIME.
So yes is was also your software fail that you then used as an excuse to reduce the number of confirmed txns at your pool.

Oh, and stop trolling that GBT uses more bandwidth than getwork. It doesn't.

Math fail.

I guess the answer to this is that ANY time you quote figures that say something is better it would be foolish for anyone to believe what you say since you will only using some edge case example that is rare or some implementation design that no one uses.

The only possible way for this to be correct is in one real situation:
When a getwork pool doesn't do roll-n-time and the total number of available transaction on the network is small and the hash rate of the device is high.

In the case where the pool does roll-n-time - getwork transfers a LOT less data than GBT in all but edge case rare situations.

To get the size it's VERY easy:

I am saying the work needs to last 2 minutes.
Processing the same work for even 2 minutes has a negative effect on the transaction confirmation times.
But we'll let it be 2 minutes anyway.
This give a single getwork request (allowing a roll of 7000 seconds - slightly under the limit) a limit of 501GH/s

To compare the two I'll use the bitcoind getwork since it approximates to the the size of a pool getwork quite fairly.

./bitcoind getwork

I got: 590 bytes
(Edit: O(1))

./bitcoind getblocktemplate

I got 141 txns and 177,184 bytes
(Edit: O(n))

(AND my bitcoind also says: accepted da0141dc61 (poolsz 1076) ... that sux)

Anyone do it and look at the difference for any getblocktemplate that has even a few transactions in it.
Consider the example in my last post of 619 transactions ....

Though of course you could double that getwork number to support 1TH/s ... and double it again to support 2TH/s ... etc. until you finally reach the average getblocktemplate sizes.

[Luke-Jr]

This is in fact a big issue with Luke-Jr also - him claiming about how what he is doing is better for BTC.
Yet what he was doing with his pool was restricting the number of transactions at his pool to 32 to increase the payments the pool received rather than fixing the problem with his pool software that was getting more orphans than other bigger pools at the same time.
The guy is a hypocrite - even worse: he enables such a thing at his pool ~5 months ago and the pool is still doing it.
I even explained a sizeable solution to this in bitcoind to both him and Gavin back then ...

The problem isn't with the pool software, as you clearly admit to knowing by mentioning fixing bitcoind. Until bitcoind is fixed and the fix is deployed everywhere, this problem remains.
Good job explaining the solution - even though your explanation doesn't actually solve it. Unfortunately, all contributions to bitcoind are done in peoples' spare time and things like a complete rewrite of the p2p code (which is needed to really fix this) takes a lot of time and skill. When this problem is nearly non-existent by simply filtering out a known DDoS attack, there are also higher priorities.

OK what doesn't it solve?

The solution is based on the fact that
1) Most bitcoinds will already have every transaction that they are sent (all except the coinbase transaction)
So the protocol is sending a large amount of data that the receiving bitcoind already has and knows it has and doesn't need sent
2) The side effect of this is also that the receiving bitcoind again re-validates each transaction that it already has validated in it's memory pool

The problem is that Bitcoin clients take much longer to relay blocks with more transactions, while by (paper) design and economic reasoning, including transactions in blocks should be near zero cost. Bitcoin clients in fact do already cache the validation of transactions they know (Gavin added this in 0.7, so it may be a result of your suggestion). However, the process is still absurdly slow.
When a client receives a block, it does this, in order:

• Request the block from the first peer to mention it
• Wait for that peer to relay the block in its entirety (ignoring all other peers in the meantime)
• Validate the block header
• Validate every transaction in the block (except the ones cached)
• Tell all our own peers - in no particular order, one by one - that we now have this block (note that if a particular peer's "transfer buffer" is already full, eg if they have a really crappy connection, bitcoind just STOPS AND WAITS FOR IT before moving on to any other peers)
• Finally, start looking at all/any requests our peers made since we first heard about this block
• Eventually, one of these peers will have requested the new block. Send it to them, but again do nothing with any other peers until this upload is complete.
• When that peer is done, another one probably asked for it in the meantime, upload it to them in the same way.

This is absurdly slow, and with large blocks can take a number of minutes to cross the network. The solution involves:

• Cache signature verifications (done)
• Rewrite the Bitcoin peer-to-peer protocol handling to work asynchronously: (this is a LOT of work)
  • if one peer can't receive data fast enough, send to other peers at the same time
  • keep responding to other peers while things are going on
• Begin relaying blocks as soon as the header is downloaded and verified, before even receiving the rest of the block ourselves - this means the entire network is uploading/downloading the new block in parallel; since the difficulty is stupidly high, that makes any DoS attack impractical (I actually implemented this, before I discovered the previous problem which prevents it from working)
• Don't even upload transactions the other peer knows about already

Also note that it is wrong to assume everyone already knows about every transaction (except coinbase) before the block is found. Peers randomly discard (and don't relay) transactions without fees, so it's likely some of those will be missing on most nodes.

[kano]

This is in fact a big issue with Luke-Jr also - him claiming about how what he is doing is better for BTC.
Yet what he was doing with his pool was restricting the number of transactions at his pool to 32 to increase the payments the pool received rather than fixing the problem with his pool software that was getting more orphans than other bigger pools at the same time.
The guy is a hypocrite - even worse: he enables such a thing at his pool ~5 months ago and the pool is still doing it.
I even explained a sizeable solution to this in bitcoind to both him and Gavin back then ...

The problem isn't with the pool software, as you clearly admit to knowing by mentioning fixing bitcoind. Until bitcoind is fixed and the fix is deployed everywhere, this problem remains.
Good job explaining the solution - even though your explanation doesn't actually solve it. Unfortunately, all contributions to bitcoind are done in peoples' spare time and things like a complete rewrite of the p2p code (which is needed to really fix this) takes a lot of time and skill. When this problem is nearly non-existent by simply filtering out a known DDoS attack, there are also higher priorities.

OK what doesn't it solve?

The solution is based on the fact that
1) Most bitcoinds will already have every transaction that they are sent (all except the coinbase transaction)
So the protocol is sending a large amount of data that the receiving bitcoind already has and knows it has and doesn't need sent
2) The side effect of this is also that the receiving bitcoind again re-validates each transaction that it already has validated in it's memory pool

The problem is that Bitcoin clients take much longer to relay blocks with more transactions, while by (paper) design and economic reasoning, including transactions in blocks should be near zero cost. Bitcoin clients in fact do already cache the validation of transactions they know (Gavin added this in 0.7, so it may be a result of your suggestion). However, the process is still absurdly slow.
When a client receives a block, it does this, in order:

• Request the block from the first peer to mention it
• Wait for that peer to relay the block in its entirety (ignoring all other peers in the meantime)
• Validate the block header
• Validate every transaction in the block (except the ones cached)
• Tell all our own peers - in no particular order, one by one - that we now have this block (note that if a particular peer's "transfer buffer" is already full, eg if they have a really crappy connection, bitcoind just STOPS AND WAITS FOR IT before moving on to any other peers)
• Finally, start looking at all/any requests our peers made since we first heard about this block
• Eventually, one of these peers will have requested the new block. Send it to them, but again do nothing with any other peers until this upload is complete.
• When that peer is done, another one probably asked for it in the meantime, upload it to them in the same way.

This is absurdly slow, and with large blocks can take a number of minutes to cross the network. The solution involves:

• Cache signature verifications (done)
• Rewrite the Bitcoin peer-to-peer protocol handling to work asynchronously: (this is a LOT of work)
  • if one peer can't receive data fast enough, send to other peers at the same time
  • keep responding to other peers while things are going on
• Begin relaying blocks as soon as the header is downloaded and verified, before even receiving the rest of the block ourselves - this means the entire network is uploading/downloading the new block in parallel; since the difficulty is stupidly high, that makes any DoS attack impractical (I actually implemented this, before I discovered the previous problem which prevents it from working)
• Don't even upload transactions the other peer knows about already

Also note that it is wrong to assume everyone already knows about every transaction (except coinbase) before the block is found. Peers randomly discard (and don't relay) transactions without fees, so it's likely some of those will be missing on most nodes.

Anyway, instead to implement it well:

1) Create a new transfer mechanism that can be used as well as the old one and ...
2) Prioritise transferring with the new mechanism first (to peers that support it)
3) Remove out the old one some time later (Remember April? Yeah avoid that)

Now I already stated in my solution that not all transactions will be known by the recipient.
The protocol must allow a response to request the txn list it doesn't have.
However, I also specifically wrote code to report the number of previously known transactions each time in my own bitcoind and also mentioned this as well
So regarding that last sentence you wrote:
it was exceedingly rare for me to get new transactions with the block that I didn't know ... other than the coinbase txn which will always be sent

Anyway I will rewrite this with your points:

Rewrite the Bitcoin peer-to-peer protocol handling to work asynchronously: (this is a LOT of work)
AND
if one peer can't receive data fast enough, send to other peers at the same time
AND
keep responding to other peers while things are going on

Or, as I said in this post, support both. Read above.
We don't want another soft fork debacle like April ...

1) Begin relaying blocks as soon as the header is downloaded and verified, before even receiving the rest of the block ourselves - this means the entire network is uploading/downloading the new block in parallel; since the difficulty is stupidly high, that makes any DoS attack impractical (I actually implemented this, before I discovered the previous problem which prevents it from working)
AND
2) Don't even upload transactions the other peer knows about already

Yeah that's what I said already ... the 2) effectively resolves 1).


So ... I take this repeat of my solution ... to mean your statement

Good job explaining the solution - even though your explanation doesn't actually solve it.

is wrong.

[Luke-Jr]

Your "explanation" only minorly improves things, it doesn't fix the problem. That's why it was last on the list of things to do to fix it.

None of what I said suggests any kind of fork is needed. Obviously the internals would support both mechanisms - it just isn't relevant to what needs to be done.

[DrHaribo]

Must I quote myself?

ONLY ignoring luke-jr

It appears that you are missing part of the discussion.

GBT and bitcoind are not the same thing. If a bug in a web server makes it give you half a file, do you blame HTTP?

Tell me how that statement I said is wrong without hiding behind definitions.
There is no difference between what I said and you said except technicalities.

Come on, that's no technicality. Is this really where this discussion is right now?

If you get a virus over HTTP then you blame HTTP?

If you mine a small block over Stratum then you blame Stratum?

Let's be honest now. Does the GBT protocol/API really force you to make small blocks? Does Stratum?

The implementations are a measure of how great it is and falling back to this constant "It's only an idea you can't say GBT is bad" is pointless also.

Come on, how many transactions you decide to include in a block is up to you, not the protocol. Of course you can say that something about GBT is bad, but can you point at the place in the spec where it tells you that you must not include more than X transactions in a block? Which part of GBT are you actually having a problem with?

Do you want it to be a violation of the spec to mine a block with less than X transactions? If so, present your suggestion and make an argument for it. But I doubt there are many that want GBT to have anything to do with which or how many transactions are chosen for a block. Luckily that is not the case now, even though you claim it is.

Next, GBT currently usually sends more data per minute than a GetWork pool with roll-n-time (probably even if it is ignoring transactions or there are few transactions on the network)
This is due to the fact that it sends all the details of the all transactions you are processing (no matter what, you cannot disable this and request a merkle tree)

Maybe that would be a good improvement for GBT. If the miner doesn't care about seeing the transactions or filtering/adding transactions, request a merkle branch instead that you can use to create a merkle root for mining. Not a single transaction goes over the wire, other than the generation tx.

Love it - "MAYBE"
How blatantly obvious is it?

Well, again, this is a way to be polite and open for cooperation with other developers. Instead of saying "THIS is the best solution and you better do things MY WAY" you can say "maybe this can help improve things?" You will find this goes over much better with most people. The way you are acting you look like a very angry man who hates the world and everyone in it. There's no need to say that people suck at programming, nor joke about someone's (?) religion.

If you had both a problem and a solution in your head, why did you present only the problem?

ckolivas said "this is a technical discussion about the disadvantages of GBT". Unless the point is to destroy GBT I think a better way to view it is that this is a technical discussion about what GBT is today and how we can make it better tomorrow.

For the problem of using much of the bandwidth on transactions when some miners don't even care about seeing them:

I guess this could be improved by having the client include something in its capabilities list to tell the server it wishes to mine "blindly" without knowing about transactions. If both client and server has this capability then the server sends a merkle branch instead of a list of transactions.

I'm not sure its entirely clean that if the user wishes to see transactions the client would leave out this capability. It makes the capabilities more like options than an actual list of capabilities, but perhaps that's alright.

What do you think, Luke-Jr and others? It could get bandwidth usage down for users who don't care to see the transactions.

[-ck]

ckolivas said "this is a technical discussion about the disadvantages of GBT". Unless the point is to destroy GBT I think a better way to view it is that this is a technical discussion about what GBT is today and how we can make it better tomorrow.

For the problem of using much of the bandwidth on transactions when some miners don't even care about seeing them:

I guess this could be improved by having the client include something in its capabilities list to tell the server it wishes to mine "blindly" without knowing about transactions. If both client and server has this capability then the server sends a merkle branch instead of a list of transactions.

I'm not sure its entirely clean that if the user wishes to see transactions the client would leave out this capability. It makes the capabilities more like options than an actual list of capabilities, but perhaps that's alright.

What do you think, Luke-Jr and others? It could get bandwidth usage down for users who don't care to see the transactions.

My goodness! Go back 3+ pages on this forum thread and you'll see that's what I was discussing with jgarzik. Yes I was trying to be productive and help find some useful pool mining mode for GBT. All it needs is a request in the protocol for branches.

...this may explain why I was so annoyed yesterday.

EDIT: https://bitcointalk.org/index.php?topic=108854.msg1286483#msg1286483

[DrHaribo]

EDIT: https://bitcointalk.org/index.php?topic=108854.msg1286483#msg1286483

I apologize, I missed that in all the noise.

I support your idea. Not sure if "suggested merkle root" is a typo. I'd suggest that on the wire this is a merkle branch that when combined with your generated coinbase tx gives you the merkle root for the block. You request work once, then send several times block header + coinbase. The server will know which txes / merkle tree this belongs to by looking at your work ID.

I believe you are right that the majority will not care to see or change the transactions, so it is a good optimization for the common case. Also, until pools support changing the transactions, it will be a good optimization in every case.

[Luke-Jr]

I guess this could be improved by having the client include something in its capabilities list to tell the server it wishes to mine "blindly" without knowing about transactions. If both client and server has this capability then the server sends a merkle branch instead of a list of transactions.

What do you think, Luke-Jr and others? It could get bandwidth usage down for users who don't care to see the transactions.

This specific method is worse than Stratum's new alternative, since the pool would now know which miners it can safely give dirty jobs to.
I also don't think it's a good idea to encourage further pool centralization by making it any easier for miners to be neglegent.

[DrHaribo]

This specific method is worse than Stratum's new alternative, since the pool would now know which miners it can safely give dirty jobs to.

Ok, I didn't think of that. But really, what is the worst that can happen? Mining 5 txes instead of 500? You can see that after-the-fact and I don't think that's a disaster.

Or maybe I am missing something that "dirty" jobs could contain?