Bitcoin Forum
April 23, 2014, 02:25:58 PM *
News: Due to the OpenSSL heartbleed bug, changing your forum password is recommended.
 
   Home   Help Search Donate Login Register  
Pages: 1 2 3 4 5 6 7 [8] 9 10 11  All
  Print  
Author Topic: Decentralized mining protocol standard: getblocktemplate (ASIC ready!)  (Read 16116 times)
Luke-Jr
Hero Member
*****
Offline Offline

Activity: 1218



View Profile

Ignore
November 01, 2012, 03:26:28 PM
 #141

One more point about the GBT protocol. If a miner chooses which transactions they include rather than accept whichever transactions the pool/bitcoind has offered in the template, the miner has to submit all the transactions with every share he submits.
No, that's (part of) what Block Proposal is for.
That's not implemented yet is it?
Allowing miners to modify transactions is not implemented in miners yet at all. Block Proposal only makes sense as part of that. I do have it implemented for the Eloipool<->bitcoind link.

When Luke-Jr made a public proposal and asked for comments, I read through it and sent in my thoughts on how it could be improved. Why didn't more people do that? Instead waiting until there are implementations in production and then finally spitting out nasty words and technical misunderstandings. That's not the way to go about things.
Please share with us all how an open source project where anyone can create anything can be followed by anyone that maybe interested in it?  Note: I only found out that Stratum when I was told on another forum that Bitcoind will not work with my mini rig.  Then I discovered GBT.
There is a Bitcoin development mailing list specifically for discussing all things related to Bitcoin development. GBT had a number of threads of discussion on it since February.

1398263158
Hero Member
*
Offline Offline

Posts: 1398263158

View Profile Personal Message (Offline)

Ignore
1398263158
Reply with quote  #2

1398263158
Report to moderator
Buy a Blade, Get a 5-Chip Free!
Start Mining with GAWMiners.com
24/7 Live Phone & Tech Support
Free Hosting & Electricity for 1 Year!

Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1398263158
Hero Member
*
Offline Offline

Posts: 1398263158

View Profile Personal Message (Offline)

Ignore
1398263158
Reply with quote  #2

1398263158
Report to moderator
1398263158
Hero Member
*
Offline Offline

Posts: 1398263158

View Profile Personal Message (Offline)

Ignore
1398263158
Reply with quote  #2

1398263158
Report to moderator
1398263158
Hero Member
*
Offline Offline

Posts: 1398263158

View Profile Personal Message (Offline)

Ignore
1398263158
Reply with quote  #2

1398263158
Report to moderator
DrHaribo
Hero Member
*****
Offline Offline

Activity: 1050


Bitminter.com Operator


View Profile WWW

Ignore
November 01, 2012, 04:50:09 PM
 #142

Lulz - I can post at 2.6 and still see you've posted crap.

Being polite and working together towards a common goal is crap? I am surprised that you participate in an open source project.

Give some grounds for this crap
Quote
It's unfortunate that we now have two competing standards. It's more unfortunate that another useless bitcointalk war is breaking out over it.

As usual many of the arguments are just not true, or misunderstood. "You have to send in all the transactions with every share. Think of all those bytes!" No, you don't.
or be gone.

You like to make a lot of demands, don't you?

ckolivas wrote that if the miner selects transactions you have to send in every transaction with every share. This is incorrect and has been discussed enough already, I think. You said that GBT, a protocol, makes decisions on which transactions to include. How does a protocol make a decision like that? And GBT being "an abortion", I don't think that's entirely correct either.

I don't care if you designed GBT ... ... ... well actually that would be even worse coz that would make you Jesus.

I never said I designed GBT. I said I suggested a few improvements for it. And what does religion have to do with this?

You told me to "fuck off" because I don't contribute. I was merely stating that my contributions are already in the spec. Where are yours?

Your pool ran 15 blocks that I pointed out, and of them, only 5 were large txn count/size.
5 were tiny. 5 were in the middle.

You picked 15 random blocks from BitMinter. They were 5 big blocks, 5 medium and 5 small. And so you come to me and demand an explanation? I find that utterly unreasonable.

If you don't like pools that filter out transactions, then talk to them, not me. I don't put any limits on transactions.

You claimed that you don't choose the transactions, GBT does ... wow Smiley

No, you said that, not me. What I said was that I include all the transactions bitcoind gives me, without filtering out the free ones or putting any limit on the number of transactions, like some pools do.

GBT and bitcoind are not the same thing. If a bug in a web server makes it give you half a file, do you blame HTTP?

Next, GBT currently usually sends more data per minute than a GetWork pool with roll-n-time (probably even if it is ignoring transactions or there are few transactions on the network)
This is due to the fact that it sends all the details of the all transactions you are processing (no matter what, you cannot disable this and request a merkle tree)

Maybe that would be a good improvement for GBT. If the miner doesn't care about seeing the transactions or filtering/adding transactions, request a merkle branch instead that you can use to create a merkle root for mining. Not a single transaction goes over the wire, other than the generation tx.

Next, GBT uses HTTP not raw sockets, so it is slower and is also expected to have more rejected shares due to this since it isn't a constant connection.
Not using HTTP, Luke-Jr has stated as a negative in Stratum coz it takes more effort to implement a better protocol ... wow gotta love that argument.
Sounds like "Don't implement something better coz it will take more effort"
Or even "Luke-Jr sux at programming and can't do anything that isn't simple"

Of course HTTP is a constant connection. Use persistent connections, and pipeline requests for speed. My miner uses 1 long poll connection and 1-3 connections to get work and send in proofs of work. For the work connections it will pipeline up to 10 requests at a time. All the connections are persistent. They get closed only if they have been sitting idle in the connection pool for a while.

The advantage of Stratum is that it is designed to be two-way from the start. Long polling in HTTP feels dirty and hackish.

The advantage of HTTP is all the network infrastructure, proxies, libraries that exist, and all the knowledge that developers have about this protocol. When you already know the protocol, you have a library for using it that you already know well, then it is much easier to get started.

slush
Hero Member
*****
Offline Offline

Activity: 1078



View Profile WWW

Ignore
November 01, 2012, 04:58:48 PM
 #143

How goes getting StratumMP BIP process off the ground? Are we still waiting on genjix?

No, I'm just busy. Writing sane documentation (in english) is pretty hard for me, so I need a lot of time to finalize something.

"There is reasonable concerns about mining currently being too centralized on pools, and the amount of control these pools hold. By exposing the details of the block proposals to the miners, they are enabled to audit and possibly modify the block before hashing it."

You didn't respond to Stratum's "get_transactions" extension explicitly. Can I ask you what's your opinion to this? I think it solves exactly that issue with blind trust for picking transaction by the pool, without unnecessary overhead for common miners.

This is just trolling.

I don't think so. I prefer coding, not discussion. We can spend months on discussions or we can both show our solutions and let others to decide.

Luke-Jr
Hero Member
*****
Offline Offline

Activity: 1218



View Profile

Ignore
November 01, 2012, 05:07:38 PM
 #144

How goes getting StratumMP BIP process off the ground? Are we still waiting on genjix?
No, I'm just busy. Writing sane documentation (in english) is pretty hard for me, so I need a lot of time to finalize something.
Well, we're just talking about a first draft, it doesn't have to be perfect - the rest of the community will probably have things they want added or changed before the final anyway.

"There is reasonable concerns about mining currently being too centralized on pools, and the amount of control these pools hold. By exposing the details of the block proposals to the miners, they are enabled to audit and possibly modify the block before hashing it."
You didn't respond to Stratum's "get_transactions" extension explicitly. Can I ask you what's your opinion to this? I think it solves exactly that issue with blind trust for picking transaction by the pool, without unnecessary overhead for common miners.
It's a step forward, at least; but the design here still encourages miners and pools to avoid (or bill extra for, as you mentioned) decentralization. I don't think "common miners" should be encouraged to centralize.

I prefer coding, not discussion. We can spend months on discussions or we can both show our solutions and let others to decide.
It's not either one or the other. The BIP process generally involves documenting and implementing a first draft, and then discussion to come up with a final standard solution that meets everyone's needs. Eligius had GBT since the first draft in February, but the protocol continued to change significantly as other developers contributed to it before reaching its current form.

Inaba
Hero Member
*****
Online Online

Activity: 1120



View Profile WWW

Ignore
November 01, 2012, 06:33:46 PM
 #145

When Luke was working on GBT and we were working on implementing a working vardiff, I had some concerns from a security standpoint.  The Stratum get_transactions also raises some red flags for me, can you address security/throttling measures in place to prevent get_transactions being used a DoS method to overload a pool?  Anything that leaves arbitrary data requests up to the client is an avenue for exploit and it is one of the reasons I have been so adamantly against user defined difficulties.  If I want to DoS your pool, I crank up a couple of 1.5 TH minirig, set my difficulty to 1 and ask for getworks. 

Same kind of problem with get_transactions:  I'm pissed at you, so I just request all the get_transactions you'll send me over and over and over.


If you're searching these lines for a point, you've probably missed it.  There was never anything there in the first place.
P_Shep
Hero Member
*****
Offline Offline

Activity: 826


View Profile WWW

Ignore
November 01, 2012, 06:38:45 PM
 #146

When Luke was working on GBT and we were working on implementing a working vardiff, I had some concerns from a security standpoint.  The Stratum get_transactions also raises some red flags for me, can you address security/throttling measures in place to prevent get_transactions being used a DoS method to overload a pool?  Anything that leaves arbitrary data requests up to the client is an avenue for exploit and it is one of the reasons I have been so adamantly against user defined difficulties.  If I want to DoS your pool, I crank up a couple of 1.5 TH minirig, set my difficulty to 1 and ask for getworks. 

Same kind of problem with get_transactions:  I'm pissed at you, so I just request all the get_transactions you'll send me over and over and over.



Surely you could keep requesting get_works with a script, why would you need any hashing behind it?

As with the get_transactions, surely you'd just place a limit on requests poolside, and ignore those that seem unreasonable?

Anubis cgminer web frontend: https://bitcointalk.org/index.php?topic=57342.0
Files and instructions for CGminer on DD-WRT: https://bitcointalk.org/index.php?topic=76685.0
Pan handling: 1Fxpijq1NN52LzSzD2WtGbT3ZTWq366ejj
kano
Hero Member
*****
Offline Offline

Activity: 1008


Linux since 1997 RedHat 4


View Profile

Ignore
November 01, 2012, 08:29:38 PM
 #147

When Luke was working on GBT and we were working on implementing a working vardiff, I had some concerns from a security standpoint.  The Stratum get_transactions also raises some red flags for me, can you address security/throttling measures in place to prevent get_transactions being used a DoS method to overload a pool?  Anything that leaves arbitrary data requests up to the client is an avenue for exploit and it is one of the reasons I have been so adamantly against user defined difficulties.  If I want to DoS your pool, I crank up a couple of 1.5 TH minirig, set my difficulty to 1 and ask for getworks.  

Same kind of problem with get_transactions:  I'm pissed at you, so I just request all the get_transactions you'll send me over and over and over.

You've ignored the reverse situation.

With GBT (all current implementations) the pool must always do this.

With Stratum it is optional and yes some users could request it always.

Since it is an available option, then the pool must consider the load of allowing it.
(... and yes I mean they MUST consider the load, not that they should ignore implementing it ... though any that don't would be easy to identify ... due to their failure to respond any miner that requests it)

BTC: 1KanoiBupPiZfkwqB7rfLXAzPnoTshAVmb
CGMiner developer,  IRC FreeNode #cgminer kanoi
Help keep Bitcoin secure by mining on pools with Stratum, the best protocol to mine Bitcoins with ASIC hardware
ckolivas
Moderator
Hero Member
*
Offline Offline

Activity: 1036



View Profile WWW

Ignore
November 01, 2012, 09:01:25 PM
 #148

Must I quote myself?
If every template request was this size and the miner used the customise coinbase option and chose transactions, they would be receiving 400kb for the block template and each share submitted would be 400kb unless they started withholding transactions.
That IS correct.

I was exploring the possible issues with implementation of GBT and got attacked as though I was trying to spread FUD about GBt. Nowhere did I imply this is what all miners would be doing.

Primary developer/maintainer for cgminer: https://bitcointalk.org/index.php?topic=28402.0  148KkS2vgVi4VzUi4JcKzM2PMaMVPi3nnq
ONLY ignoring luke-jr
-ck
Luke-Jr
Hero Member
*****
Offline Offline

Activity: 1218



View Profile

Ignore
November 01, 2012, 09:07:06 PM
 #149

Must I quote myself?
If every template request was this size and the miner used the customise coinbase option and chose transactions, they would be receiving 400kb for the block template and each share submitted would be 400kb unless they started withholding transactions.
Too bad you're more interested in ignoring me than having a productive discussion, or you'd have noticed I pointed out exactly why that isn't true.

ckolivas
Moderator
Hero Member
*
Offline Offline

Activity: 1036



View Profile WWW

Ignore
November 01, 2012, 10:10:09 PM
 #150

When Luke was working on GBT and we were working on implementing a working vardiff, I had some concerns from a security standpoint.  The Stratum get_transactions also raises some red flags for me, can you address security/throttling measures in place to prevent get_transactions being used a DoS method to overload a pool?  Anything that leaves arbitrary data requests up to the client is an avenue for exploit and it is one of the reasons I have been so adamantly against user defined difficulties.  If I want to DoS your pool, I crank up a couple of 1.5 TH minirig, set my difficulty to 1 and ask for getworks.  

Same kind of problem with get_transactions:  I'm pissed at you, so I just request all the get_transactions you'll send me over and over and over.



Surely you could keep requesting get_works with a script, why would you need any hashing behind it?

As with the get_transactions, surely you'd just place a limit on requests poolside, and ignore those that seem unreasonable?
Since the point of get_transactions is simply to confirm the merkle branches, you only need to allow it once per stratum template push. All the stratum pools so far only push a new template every 30 seconds. Limiting get transaction requests to 1 for each template would only allow the miner to ask for it once every 30 seconds.

By the same token, GBT pools should limit how often users can ask for a template since that does send out all the transactions.

Primary developer/maintainer for cgminer: https://bitcointalk.org/index.php?topic=28402.0  148KkS2vgVi4VzUi4JcKzM2PMaMVPi3nnq
ONLY ignoring luke-jr
-ck
kano
Hero Member
*****
Offline Offline

Activity: 1008


Linux since 1997 RedHat 4


View Profile

Ignore
November 01, 2012, 10:19:29 PM
 #151

Lulz - I can post at 2.6 and still see you've posted crap.

Being polite and working together towards a common goal is crap? I am surprised that you participate in an open source project.
May your surprise make you happy.

Quote
Give some grounds for this crap
Quote
It's unfortunate that we now have two competing standards. It's more unfortunate that another useless bitcointalk war is breaking out over it.

As usual many of the arguments are just not true, or misunderstood. "You have to send in all the transactions with every share. Think of all those bytes!" No, you don't.
or be gone.

You like to make a lot of demands, don't you?

ckolivas wrote that if the miner selects transactions you have to send in every transaction with every share. This is incorrect and has been discussed enough already, I think. You said that GBT, a protocol, makes decisions on which transactions to include. How does a protocol make a decision like that? And GBT being "an abortion", I don't think that's entirely correct either.
I still don't see any backup to the point you made "As usual many of the arguments are just not true, or misunderstood."

Quote
I don't care if you designed GBT ... ... ... well actually that would be even worse coz that would make you Jesus.

I never said I designed GBT. I said I suggested a few improvements for it. And what does religion have to do with this?
"In" joke Smiley

Quote
You told me to "fuck off" because I don't contribute. I was merely stating that my contributions are already in the spec. Where are yours?
No, because you made a VERY broad statement without any basis and contributed nothing to the current discussion except to tell people how they should be discussing it.

Quote
Your pool ran 15 blocks that I pointed out, and of them, only 5 were large txn count/size.
5 were tiny. 5 were in the middle.

You picked 15 random blocks from BitMinter. They were 5 big blocks, 5 medium and 5 small. And so you come to me and demand an explanation? I find that utterly unreasonable.

If you don't like pools that filter out transactions, then talk to them, not me. I don't put any limits on transactions.
What is your reason for wording it like that?

No I didn't pick 15 random blocks.
I picked 15 blocks in a row that your pool generated.

Your response was that it was out of your control.

The reason I chose those blocks was because one of them orphaned another block at the same time:

205766
BTC Sent  510.69 BTC
Number Of Transactions  32
By BitMinter

Orphaned:
BTC Sent  15,449.927 BTC
Number Of Transactions  619
By EclipseMC

Yes something is badly wrong there ... and your excuse is that you don't have any rules about the transactions, you just call the GBT interface.

[aside]
This is in fact a big issue with Luke-Jr also - him claiming about how what he is doing is better for BTC.
Yet what he was doing with his pool was restricting the number of transactions at his pool to 32 to increase the payments the pool received rather than fixing the problem with his pool software that was getting more orphans than other bigger pools at the same time.
The guy is a hypocrite - even worse: he enables such a thing at his pool ~5 months ago and the pool is still doing it.
I even explained a sizeable solution to this in bitcoind to both him and Gavin back then ...
[/aside]

Quote
You claimed that you don't choose the transactions, GBT does ... wow Smiley

No, you said that, not me. What I said was that I include all the transactions bitcoind gives me, without filtering out the free ones or putting any limit on the number of transactions, like some pools do.

GBT and bitcoind are not the same thing. If a bug in a web server makes it give you half a file, do you blame HTTP?
Tell me how that statement I said is wrong without hiding behind definitions.
There is no difference between what I said and you said except technicalities.
Yes it seems this statement keeps coming up.
This is a text book theory argument that one can make about something that is just that - a theory.
Yes the GBT proponents are out saying to use GBT because it is great.
The implementations are a measure of how great it is and falling back to this constant "It's only an idea you can't say GBT is bad" is pointless also.

Quote
Next, GBT currently usually sends more data per minute than a GetWork pool with roll-n-time (probably even if it is ignoring transactions or there are few transactions on the network)
This is due to the fact that it sends all the details of the all transactions you are processing (no matter what, you cannot disable this and request a merkle tree)

Maybe that would be a good improvement for GBT. If the miner doesn't care about seeing the transactions or filtering/adding transactions, request a merkle branch instead that you can use to create a merkle root for mining. Not a single transaction goes over the wire, other than the generation tx.
Love it - "MAYBE" Tongue
How blatantly obvious is it?
This is a good example of how it becomes just straight out annoying:

Use this great new protocol - it's better than before ........ except it sends more data than getwork ...... that yeah MAYBE that might be worth improving ...... Tongue
But it's better coz the GBT guys say it is - it's transparent ... oh wait Stratum has the option to send the txn list that is 'missing' also.
Um ... what other excuse can they make up ...
Oh yeah it's just the implementations that suck ... GBT is good for everyone ............ except everyone who is using it with any network between the miner and the work generator ...

BTC: 1KanoiBupPiZfkwqB7rfLXAzPnoTshAVmb
CGMiner developer,  IRC FreeNode #cgminer kanoi
Help keep Bitcoin secure by mining on pools with Stratum, the best protocol to mine Bitcoins with ASIC hardware
Luke-Jr
Hero Member
*****
Offline Offline

Activity: 1218



View Profile

Ignore
November 01, 2012, 10:33:18 PM
 #152

This is in fact a big issue with Luke-Jr also - him claiming about how what he is doing is better for BTC.
Yet what he was doing with his pool was restricting the number of transactions at his pool to 32 to increase the payments the pool received rather than fixing the problem with his pool software that was getting more orphans than other bigger pools at the same time.
The guy is a hypocrite - even worse: he enables such a thing at his pool ~5 months ago and the pool is still doing it.
I even explained a sizeable solution to this in bitcoind to both him and Gavin back then ...
The problem isn't with the pool software, as you clearly admit to knowing by mentioning fixing bitcoind. Until bitcoind is fixed and the fix is deployed everywhere, this problem remains.
Good job explaining the solution - even though your explanation doesn't actually solve it. Unfortunately, all contributions to bitcoind are done in peoples' spare time and things like a complete rewrite of the p2p code (which is needed to really fix this) takes a lot of time and skill. When this problem is nearly non-existent by simply filtering out a known DDoS attack, there are also higher priorities.

Oh, and stop trolling that GBT uses more bandwidth than getwork. It doesn't.

kano
Hero Member
*****
Offline Offline

Activity: 1008


Linux since 1997 RedHat 4


View Profile

Ignore
November 02, 2012, 12:05:41 AM
 #153

This is in fact a big issue with Luke-Jr also - him claiming about how what he is doing is better for BTC.
Yet what he was doing with his pool was restricting the number of transactions at his pool to 32 to increase the payments the pool received rather than fixing the problem with his pool software that was getting more orphans than other bigger pools at the same time.
The guy is a hypocrite - even worse: he enables such a thing at his pool ~5 months ago and the pool is still doing it.
I even explained a sizeable solution to this in bitcoind to both him and Gavin back then ...
The problem isn't with the pool software, as you clearly admit to knowing by mentioning fixing bitcoind. Until bitcoind is fixed and the fix is deployed everywhere, this problem remains.
Good job explaining the solution - even though your explanation doesn't actually solve it. Unfortunately, all contributions to bitcoind are done in peoples' spare time and things like a complete rewrite of the p2p code (which is needed to really fix this) takes a lot of time and skill. When this problem is nearly non-existent by simply filtering out a known DDoS attack, there are also higher priorities.
OK what doesn't it solve?

The solution is based on the fact that
1) Most bitcoinds will already have every transaction that they are sent (all except the coinbase transaction)
So the protocol is sending a large amount of data that the receiving bitcoind already has and knows it has and doesn't need sent
2) The side effect of this is also that the receiving bitcoind again re-validates each transaction that it already has validated in it's memory pool

EDIT: oops left out this comment.
The problem was not with the bigger pools software, since they were not getting as high a % of orphans as you were AT THE SAME TIME.
So yes is was also your software fail that you then used as an excuse to reduce the number of confirmed txns at your pool.

Quote
Oh, and stop trolling that GBT uses more bandwidth than getwork. It doesn't.
Math fail.

I guess the answer to this is that ANY time you quote figures that say something is better it would be foolish for anyone to believe what you say since you will only using some edge case example that is rare or some implementation design that no one uses.

The only possible way for this to be correct is in one real situation:
When a getwork pool doesn't do roll-n-time and the total number of available transaction on the network is small and the hash rate of the device is high.

In the case where the pool does roll-n-time - getwork transfers a LOT less data than GBT in all but edge case rare situations.

To get the size it's VERY easy:

I am saying the work needs to last 2 minutes.
Processing the same work for even 2 minutes has a negative effect on the transaction confirmation times.
But we'll let it be 2 minutes anyway.
This give a single getwork request (allowing a roll of 7000 seconds - slightly under the limit) a limit of 501GH/s

To compare the two I'll use the bitcoind getwork since it approximates to the the size of a pool getwork quite fairly.

./bitcoind getwork

I got: 590 bytes
(Edit: O(1))

./bitcoind getblocktemplate

I got 141 txns and 177,184 bytes
(Edit: O(n))

(AND my bitcoind also says: accepted da0141dc61 (poolsz 1076) ... that sux)

Anyone do it and look at the difference for any getblocktemplate that has even a few transactions in it.
Consider the example in my last post of 619 transactions .... Tongue

Though of course you could double that getwork number to support 1TH/s ... and double it again to support 2TH/s ... etc. until you finally reach the average getblocktemplate sizes.

BTC: 1KanoiBupPiZfkwqB7rfLXAzPnoTshAVmb
CGMiner developer,  IRC FreeNode #cgminer kanoi
Help keep Bitcoin secure by mining on pools with Stratum, the best protocol to mine Bitcoins with ASIC hardware
Luke-Jr
Hero Member
*****
Offline Offline

Activity: 1218



View Profile

Ignore
November 02, 2012, 12:30:44 AM
 #154

This is in fact a big issue with Luke-Jr also - him claiming about how what he is doing is better for BTC.
Yet what he was doing with his pool was restricting the number of transactions at his pool to 32 to increase the payments the pool received rather than fixing the problem with his pool software that was getting more orphans than other bigger pools at the same time.
The guy is a hypocrite - even worse: he enables such a thing at his pool ~5 months ago and the pool is still doing it.
I even explained a sizeable solution to this in bitcoind to both him and Gavin back then ...
The problem isn't with the pool software, as you clearly admit to knowing by mentioning fixing bitcoind. Until bitcoind is fixed and the fix is deployed everywhere, this problem remains.
Good job explaining the solution - even though your explanation doesn't actually solve it. Unfortunately, all contributions to bitcoind are done in peoples' spare time and things like a complete rewrite of the p2p code (which is needed to really fix this) takes a lot of time and skill. When this problem is nearly non-existent by simply filtering out a known DDoS attack, there are also higher priorities.
OK what doesn't it solve?

The solution is based on the fact that
1) Most bitcoinds will already have every transaction that they are sent (all except the coinbase transaction)
So the protocol is sending a large amount of data that the receiving bitcoind already has and knows it has and doesn't need sent
2) The side effect of this is also that the receiving bitcoind again re-validates each transaction that it already has validated in it's memory pool
The problem is that Bitcoin clients take much longer to relay blocks with more transactions, while by (paper) design and economic reasoning, including transactions in blocks should be near zero cost. Bitcoin clients in fact do already cache the validation of transactions they know (Gavin added this in 0.7, so it may be a result of your suggestion). However, the process is still absurdly slow.
When a client receives a block, it does this, in order:
  • Request the block from the first peer to mention it
  • Wait for that peer to relay the block in its entirety (ignoring all other peers in the meantime)
  • Validate the block header
  • Validate every transaction in the block (except the ones cached)
  • Tell all our own peers - in no particular order, one by one - that we now have this block (note that if a particular peer's "transfer buffer" is already full, eg if they have a really crappy connection, bitcoind just STOPS AND WAITS FOR IT before moving on to any other peers)
  • Finally, start looking at all/any requests our peers made since we first heard about this block
  • Eventually, one of these peers will have requested the new block. Send it to them, but again do nothing with any other peers until this upload is complete.
  • When that peer is done, another one probably asked for it in the meantime, upload it to them in the same way.
This is absurdly slow, and with large blocks can take a number of minutes to cross the network. The solution involves:
  • Cache signature verifications (done)
  • Rewrite the Bitcoin peer-to-peer protocol handling to work asynchronously: (this is a LOT of work)
    • if one peer can't receive data fast enough, send to other peers at the same time
    • keep responding to other peers while things are going on
  • Begin relaying blocks as soon as the header is downloaded and verified, before even receiving the rest of the block ourselves - this means the entire network is uploading/downloading the new block in parallel; since the difficulty is stupidly high, that makes any DoS attack impractical (I actually implemented this, before I discovered the previous problem which prevents it from working)
  • Don't even upload transactions the other peer knows about already

Also note that it is wrong to assume everyone already knows about every transaction (except coinbase) before the block is found. Peers randomly discard (and don't relay) transactions without fees, so it's likely some of those will be missing on most nodes.

kano
Hero Member
*****
Offline Offline

Activity: 1008


Linux since 1997 RedHat 4


View Profile

Ignore
November 02, 2012, 02:21:06 AM
 #155

This is in fact a big issue with Luke-Jr also - him claiming about how what he is doing is better for BTC.
Yet what he was doing with his pool was restricting the number of transactions at his pool to 32 to increase the payments the pool received rather than fixing the problem with his pool software that was getting more orphans than other bigger pools at the same time.
The guy is a hypocrite - even worse: he enables such a thing at his pool ~5 months ago and the pool is still doing it.
I even explained a sizeable solution to this in bitcoind to both him and Gavin back then ...
The problem isn't with the pool software, as you clearly admit to knowing by mentioning fixing bitcoind. Until bitcoind is fixed and the fix is deployed everywhere, this problem remains.
Good job explaining the solution - even though your explanation doesn't actually solve it. Unfortunately, all contributions to bitcoind are done in peoples' spare time and things like a complete rewrite of the p2p code (which is needed to really fix this) takes a lot of time and skill. When this problem is nearly non-existent by simply filtering out a known DDoS attack, there are also higher priorities.
OK what doesn't it solve?

The solution is based on the fact that
1) Most bitcoinds will already have every transaction that they are sent (all except the coinbase transaction)
So the protocol is sending a large amount of data that the receiving bitcoind already has and knows it has and doesn't need sent
2) The side effect of this is also that the receiving bitcoind again re-validates each transaction that it already has validated in it's memory pool
The problem is that Bitcoin clients take much longer to relay blocks with more transactions, while by (paper) design and economic reasoning, including transactions in blocks should be near zero cost. Bitcoin clients in fact do already cache the validation of transactions they know (Gavin added this in 0.7, so it may be a result of your suggestion). However, the process is still absurdly slow.
When a client receives a block, it does this, in order:
  • Request the block from the first peer to mention it
  • Wait for that peer to relay the block in its entirety (ignoring all other peers in the meantime)
  • Validate the block header
  • Validate every transaction in the block (except the ones cached)
  • Tell all our own peers - in no particular order, one by one - that we now have this block (note that if a particular peer's "transfer buffer" is already full, eg if they have a really crappy connection, bitcoind just STOPS AND WAITS FOR IT before moving on to any other peers)
  • Finally, start looking at all/any requests our peers made since we first heard about this block
  • Eventually, one of these peers will have requested the new block. Send it to them, but again do nothing with any other peers until this upload is complete.
  • When that peer is done, another one probably asked for it in the meantime, upload it to them in the same way.
This is absurdly slow, and with large blocks can take a number of minutes to cross the network. The solution involves:
  • Cache signature verifications (done)
  • Rewrite the Bitcoin peer-to-peer protocol handling to work asynchronously: (this is a LOT of work)
    • if one peer can't receive data fast enough, send to other peers at the same time
    • keep responding to other peers while things are going on
  • Begin relaying blocks as soon as the header is downloaded and verified, before even receiving the rest of the block ourselves - this means the entire network is uploading/downloading the new block in parallel; since the difficulty is stupidly high, that makes any DoS attack impractical (I actually implemented this, before I discovered the previous problem which prevents it from working)
  • Don't even upload transactions the other peer knows about already

Also note that it is wrong to assume everyone already knows about every transaction (except coinbase) before the block is found. Peers randomly discard (and don't relay) transactions without fees, so it's likely some of those will be missing on most nodes.
Anyway, instead to implement it well:

1) Create a new transfer mechanism that can be used as well as the old one and ...
2) Prioritise transferring with the new mechanism first (to peers that support it)
3) Remove out the old one some time later (Remember April? Yeah avoid that)

Now I already stated in my solution that not all transactions will be known by the recipient.
The protocol must allow a response to request the txn list it doesn't have.
However, I also specifically wrote code to report the number of previously known transactions each time in my own bitcoind and also mentioned this as well
So regarding that last sentence you wrote:
it was exceedingly rare for me to get new transactions with the block that I didn't know ... other than the coinbase txn which will always be sent

Anyway I will rewrite this with your points:

Quote
Rewrite the Bitcoin peer-to-peer protocol handling to work asynchronously: (this is a LOT of work)
AND
if one peer can't receive data fast enough, send to other peers at the same time
AND
keep responding to other peers while things are going on
Or, as I said in this post, support both. Read above.
We don't want another soft fork debacle like April ...

Quote
1) Begin relaying blocks as soon as the header is downloaded and verified, before even receiving the rest of the block ourselves - this means the entire network is uploading/downloading the new block in parallel; since the difficulty is stupidly high, that makes any DoS attack impractical (I actually implemented this, before I discovered the previous problem which prevents it from working)
AND
2) Don't even upload transactions the other peer knows about already
Yeah that's what I said already ... the 2) effectively resolves 1).


So ... I take this repeat of my solution ... to mean your statement
Quote
Good job explaining the solution - even though your explanation doesn't actually solve it.
is wrong.

BTC: 1KanoiBupPiZfkwqB7rfLXAzPnoTshAVmb
CGMiner developer,  IRC FreeNode #cgminer kanoi
Help keep Bitcoin secure by mining on pools with Stratum, the best protocol to mine Bitcoins with ASIC hardware
Luke-Jr
Hero Member
*****
Offline Offline

Activity: 1218



View Profile

Ignore
November 02, 2012, 04:10:36 AM
 #156

Your "explanation" only minorly improves things, it doesn't fix the problem. That's why it was last on the list of things to do to fix it.

None of what I said suggests any kind of fork is needed. Obviously the internals would support both mechanisms - it just isn't relevant to what needs to be done.

DrHaribo
Hero Member
*****
Offline Offline

Activity: 1050


Bitminter.com Operator


View Profile WWW

Ignore
November 02, 2012, 08:45:06 AM
 #157

Must I quote myself?
ONLY ignoring luke-jr

It appears that you are missing part of the discussion.

Quote
GBT and bitcoind are not the same thing. If a bug in a web server makes it give you half a file, do you blame HTTP?
Tell me how that statement I said is wrong without hiding behind definitions.
There is no difference between what I said and you said except technicalities.

Come on, that's no technicality. Is this really where this discussion is right now?

If you get a virus over HTTP then you blame HTTP?

If you mine a small block over Stratum then you blame Stratum?

Let's be honest now. Does the GBT protocol/API really force you to make small blocks? Does Stratum?


The implementations are a measure of how great it is and falling back to this constant "It's only an idea you can't say GBT is bad" is pointless also.

Come on, how many transactions you decide to include in a block is up to you, not the protocol. Of course you can say that something about GBT is bad, but can you point at the place in the spec where it tells you that you must not include more than X transactions in a block? Which part of GBT are you actually having a problem with?

Do you want it to be a violation of the spec to mine a block with less than X transactions? If so, present your suggestion and make an argument for it. But I doubt there are many that want GBT to have anything to do with which or how many transactions are chosen for a block. Luckily that is not the case now, even though you claim it is.

Quote
Next, GBT currently usually sends more data per minute than a GetWork pool with roll-n-time (probably even if it is ignoring transactions or there are few transactions on the network)
This is due to the fact that it sends all the details of the all transactions you are processing (no matter what, you cannot disable this and request a merkle tree)

Maybe that would be a good improvement for GBT. If the miner doesn't care about seeing the transactions or filtering/adding transactions, request a merkle branch instead that you can use to create a merkle root for mining. Not a single transaction goes over the wire, other than the generation tx.
Love it - "MAYBE" Tongue
How blatantly obvious is it?

Well, again, this is a way to be polite and open for cooperation with other developers. Instead of saying "THIS is the best solution and you better do things MY WAY" you can say "maybe this can help improve things?" You will find this goes over much better with most people. The way you are acting you look like a very angry man who hates the world and everyone in it. There's no need to say that people suck at programming, nor joke about someone's (?) religion.

If you had both a problem and a solution in your head, why did you present only the problem?

ckolivas said "this is a technical discussion about the disadvantages of GBT". Unless the point is to destroy GBT I think a better way to view it is that this is a technical discussion about what GBT is today and how we can make it better tomorrow.

For the problem of using much of the bandwidth on transactions when some miners don't even care about seeing them:

I guess this could be improved by having the client include something in its capabilities list to tell the server it wishes to mine "blindly" without knowing about transactions. If both client and server has this capability then the server sends a merkle branch instead of a list of transactions.

I'm not sure its entirely clean that if the user wishes to see transactions the client would leave out this capability. It makes the capabilities more like options than an actual list of capabilities, but perhaps that's alright.

What do you think, Luke-Jr and others? It could get bandwidth usage down for users who don't care to see the transactions.

ckolivas
Moderator
Hero Member
*
Offline Offline

Activity: 1036



View Profile WWW

Ignore
November 02, 2012, 10:34:10 AM
 #158

ckolivas said "this is a technical discussion about the disadvantages of GBT". Unless the point is to destroy GBT I think a better way to view it is that this is a technical discussion about what GBT is today and how we can make it better tomorrow.

For the problem of using much of the bandwidth on transactions when some miners don't even care about seeing them:

I guess this could be improved by having the client include something in its capabilities list to tell the server it wishes to mine "blindly" without knowing about transactions. If both client and server has this capability then the server sends a merkle branch instead of a list of transactions.

I'm not sure its entirely clean that if the user wishes to see transactions the client would leave out this capability. It makes the capabilities more like options than an actual list of capabilities, but perhaps that's alright.

What do you think, Luke-Jr and others? It could get bandwidth usage down for users who don't care to see the transactions.
My goodness! Go back 3+ pages on this forum thread and you'll see that's what I was discussing with jgarzik. Yes I was trying to be productive and help find some useful pool mining mode for GBT. All it needs is a request in the protocol for branches.

...this may explain why I was so annoyed yesterday.

EDIT: https://bitcointalk.org/index.php?topic=108854.msg1286483#msg1286483

Primary developer/maintainer for cgminer: https://bitcointalk.org/index.php?topic=28402.0  148KkS2vgVi4VzUi4JcKzM2PMaMVPi3nnq
ONLY ignoring luke-jr
-ck
DrHaribo
Hero Member
*****
Offline Offline

Activity: 1050


Bitminter.com Operator


View Profile WWW

Ignore
November 02, 2012, 11:16:03 AM
 #159


I apologize, I missed that in all the noise.

I support your idea. Not sure if "suggested merkle root" is a typo. I'd suggest that on the wire this is a merkle branch that when combined with your generated coinbase tx gives you the merkle root for the block. You request work once, then send several times block header + coinbase. The server will know which txes / merkle tree this belongs to by looking at your work ID.

I believe you are right that the majority will not care to see or change the transactions, so it is a good optimization for the common case. Also, until pools support changing the transactions, it will be a good optimization in every case.

Luke-Jr
Hero Member
*****
Offline Offline

Activity: 1218



View Profile

Ignore
November 02, 2012, 11:30:07 AM
 #160

I guess this could be improved by having the client include something in its capabilities list to tell the server it wishes to mine "blindly" without knowing about transactions. If both client and server has this capability then the server sends a merkle branch instead of a list of transactions.

What do you think, Luke-Jr and others? It could get bandwidth usage down for users who don't care to see the transactions.
This specific method is worse than Stratum's new alternative, since the pool would now know which miners it can safely give dirty jobs to.
I also don't think it's a good idea to encourage further pool centralization by making it any easier for miners to be neglegent.

Pages: 1 2 3 4 5 6 7 [8] 9 10 11  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!