I: {"params": ["b57e", "3cc177a87a3bedf6f6d25d8a09801c4450ebccf3cb5d02a0000002a600000000", "01000000010000000000000000000000000000000000000000000000000000000000000000ffffffff21033b2703062f503253482f04f03b985008", "072f736c7573682f000000000100f2052a010000001976a914e8e6ace10f10ce5ed479c7188c9b4061e53aa90688ac00000000", [], "00000002", "1a0513c5", "50983bf0", true], "id": null, "method": "mining.notify"}
O: {"params": ["b57e"], "id": "txlistb57e", "method": "mining.get_transactions"}
I: {"error": null, "id": "txlistb57e", "result": ["0100000001e45eb5311f14f66eff9351134265e33dbd41daa8110acc5b031e80de38893a65000000006b483045022028837a004078689c927fdc794a1e1a15b90671e90cd77e3f98c1c8776b7a9054022100c5c803954112f0ce183041c38b2e31e38bf91c61c3a5cc2b981b23665e7c1d240121029b497b642311bb84f184c0534d3874182035e77c8b1a48c35d3e9e8a390c93c3ffffffff0200a78c13180000001976a914b4eac5b7bba166b7f8c8cd195d6cb3f4680eede788ac80267241000000001976a9147ff701619769fbd1a12f8d0f440c3cfa50adfefe88ac00000000", ...]}
Why does this job have transactions, but no merkle links? :/ |
|
|
|
What harm can a pool op do when miners don't see the transactions until after a block is created? What exactly would a "dirty job" be? Double spending, for example. |
|
|
|
Pushed a fix for Eloipool and deployed on Eligius: 2012-11-05 15:57:27 BFGMiner NOTICE Stratum from pool 3 detected new block
2012-11-05 15:57:38 newBlockNotification INFO Received new block notification
2012-11-05 15:57:40 merkleMaker INFO New block: 0000000000000052ff0915fef8c1144e4a16fe380cf416c991e96a230871512c (height: 206605; bits: 1a0513c5)
2012-11-05 15:57:40 BFGMiner NOTICE LONGPOLL from pool 0 caught up to new block
2012-11-05 15:57:45 JSONRPCServer INFO Longpoll woke up 7704 clients in 5.137 seconds Now if only slush would share his secret to getting blocks sooner :p |
|
|
|
Almost implemented the first working version of GBT within cgminer. This sort of behaviour tells the story well though: [2012-11-05 22:28:08] Accepted 23c0ba9a Diff 7/1 GPU 2 pool 2
[2012-11-05 22:28:08] Accepted 163ec800 Diff 11/1 GPU 0 pool 2
[2012-11-05 22:28:13] Stratum from pool 3 detected new block
[2012-11-05 22:28:13] Accepted 61bdf4a7 Diff 2/1 GPU 3 pool 2
[2012-11-05 22:28:13] Stale share detected, submitting as user requested
[2012-11-05 22:28:14] Accepted 00feebca Diff 257/1 GPU 1 pool 2
[2012-11-05 22:28:19] Rejected 70b76eb2 Diff 2/1 GPU 1 pool 2 (stale-prevblk)
[2012-11-05 22:28:19] Rejected 47b723ef Diff 3/1 GPU 3 pool 2 (stale-prevblk)
[2012-11-05 22:28:20] Rejected 9644eef7 Diff 1/1 GPU 2 pool 2 (stale-prevblk)
[2012-11-05 22:28:21] Rejected 4cafa99d Diff 3/1 GPU 1 pool 2 (stale-prevblk)
[2012-11-05 22:28:22] Rejected 093dc4cd Diff 27/1 GPU 0 pool 2 (stale-prevblk)
[2012-11-05 22:28:23] GBT LONGPOLL from pool 2 requested work restart
[2012-11-05 22:28:24] Accepted 375a5178 Diff 4/1 GPU 1 pool 2
[2012-11-05 22:28:25] Accepted 047206f4 Diff 57/1 GPU 3 pool 2
[2012-11-05 22:28:25] GBT LONGPOLL from pool 2 requested work restart
[2012-11-05 22:28:30] Accepted 72bee5f8 Diff 2/1 GPU 0 pool 2
[2012-11-05 22:28:31] Accepted 8e63856d Diff 1/1 GPU 2 pool 2
Now pool 3 here is slush on stratum (ping time 335ms), while pool 2 is EMC on GBT (ping time 225ms). Note how stratum picks up the block change and notifies me 10 seconds before GBT does. However, this is not the GBT pool simply learning of the block change later, because it starts rejecting my shares as being from the previous block before I even got the longpoll from the GBT pool. Then of course there's a second longpoll with the transactions, which is not an unusual practice. Confirming this analysis as accurate... for now. It shouldn't be this much of a difference though, so I suspect there's something going on server-side I need to look into. Analysis of block 000000000000017eeb9f83b6037b12c26c41abf776c3602b619b0b583ec74b7e: 1143.46298 Stratum notification of block from slush
1151.00000 Eligius bitcoind finishes processing new block
1152.00000 Eligius eloipool finishes processing new block
1155.90269 Begin receiving GBT longpoll reply from EclipseMC
1156.62722 Begin receiving GBT longpoll reply from Eligius
1162.99977 Finish receiving GBT longpoll reply from Eligius: 822 kB
1167.77308 Finish receiving GBT longpoll reply from EclipseMC: 804 kB So somehow, slush's pool is getting blocks 8 seconds earlier than EclipseMC and Eligius. That's half the time difference alone. Now, Eloipool isn't supposed to be including transactions in the first longpoll response (that's what the second one is for). 804/822 kB reflects that in practice it is. That's a bug for me to fix  There's also a 4 second delay between eloipool processing the block and my beginning to receive the longpoll. The entire process of queuing the LP data to clients is taking 5.5 seconds on Eligius during new blocks. When there isn't a new block, the same thing takes 2.7 seconds. Presumably fixing the size of the longpoll response should reduce this somewhat, but this is an area where Stratum actually does make a difference: the server would only need to encode the JSON response once and queue it at all clients identically. |
|
|
|
Almost implemented the first working version of GBT within cgminer. This sort of behaviour tells the story well though: [2012-11-05 22:28:08] Accepted 23c0ba9a Diff 7/1 GPU 2 pool 2
[2012-11-05 22:28:08] Accepted 163ec800 Diff 11/1 GPU 0 pool 2
[2012-11-05 22:28:13] Stratum from pool 3 detected new block
[2012-11-05 22:28:13] Accepted 61bdf4a7 Diff 2/1 GPU 3 pool 2
[2012-11-05 22:28:13] Stale share detected, submitting as user requested
[2012-11-05 22:28:14] Accepted 00feebca Diff 257/1 GPU 1 pool 2
[2012-11-05 22:28:19] Rejected 70b76eb2 Diff 2/1 GPU 1 pool 2 (stale-prevblk)
[2012-11-05 22:28:19] Rejected 47b723ef Diff 3/1 GPU 3 pool 2 (stale-prevblk)
[2012-11-05 22:28:20] Rejected 9644eef7 Diff 1/1 GPU 2 pool 2 (stale-prevblk)
[2012-11-05 22:28:21] Rejected 4cafa99d Diff 3/1 GPU 1 pool 2 (stale-prevblk)
[2012-11-05 22:28:22] Rejected 093dc4cd Diff 27/1 GPU 0 pool 2 (stale-prevblk)
[2012-11-05 22:28:23] GBT LONGPOLL from pool 2 requested work restart
[2012-11-05 22:28:24] Accepted 375a5178 Diff 4/1 GPU 1 pool 2
[2012-11-05 22:28:25] Accepted 047206f4 Diff 57/1 GPU 3 pool 2
[2012-11-05 22:28:25] GBT LONGPOLL from pool 2 requested work restart
[2012-11-05 22:28:30] Accepted 72bee5f8 Diff 2/1 GPU 0 pool 2
[2012-11-05 22:28:31] Accepted 8e63856d Diff 1/1 GPU 2 pool 2
Now pool 3 here is slush on stratum (ping time 335ms), while pool 2 is EMC on GBT (ping time 225ms). Note how stratum picks up the block change and notifies me 10 seconds before GBT does. However, this is not the GBT pool simply learning of the block change later, because it starts rejecting my shares as being from the previous block before I even got the longpoll from the GBT pool. Then of course there's a second longpoll with the transactions, which is not an unusual practice. Confirming this analysis as accurate... for now. It shouldn't be this much of a difference though, so I suspect there's something going on server-side I need to look into. |
|
|
|
just a quick ? does the miner in bamt work on Stratum mining pools or do i need to upgrade it and if so is there an easy way lol. sorry if this ? has already been answered.
Supposedly if you get the BAMT 0.6 beta, you can manually build BFGMiner 2.9 for it. |
|
|
|
Today I implemented and released to my pool new method called mining.get_transactions(job_id). This call simply dumps transactions used for given job. Thanks to this, miners now have everything needed to reconstruct source block template used by the pool and they can check online if pool isn't doing something nasty. This doesn't seem to actually work. I'm receiving some transaction data, but then the connection goes silent before the response is complete. By "silent" I mean it remains open and there are TCP keepalives going on, but no more data arrives. Some writebuffer-related bug perhaps? |
|
|
|
CPPSRB will still have the 7-day timeout for inactivity, but, keep in mind that CPPSRB can still pay for old shares (extra credit under SMPPS) so, it can still take a while to go fully inactive. After the system is up and running I do believe I will be replacing the timeout with an actual "no longer mining" timeout to get miners their balances when they stop mining, and when they are paid it will reset the timeout. So, if they get any backpay, they'll get another payment in 7 days, etc.
-wk I forgot that RB stood for recent backpay. So in theory, it could pay all EC even for someone that hasn't mined in a year, but that is still nearly impossible, statistically speaking. I like the idea of a mining timeout, although I'm not sure it it would be good to send out micropayments every 7 days. That having been said, if it is still going to be "mined," maybe mining the micro amount won 't be as bad, and I think the client tries to minimize fees, nonetheless. Mining small amounts doesn't cost anything, but I personally would be worried that a bunch of tiny micropayments for idle miners will hurt those miners when they go to spend it. I've recommended to wizkid057 that he reconsider this part of his plan, and only add such a rule if there's actually a problem left after we switch to CPPSRB; due to the CPPSRB algorithm, however, I expect the problem will simply go away without such an extra rule. |
|
|
|
I guess this could be improved by having the client include something in its capabilities list to tell the server it wishes to mine "blindly" without knowing about transactions. If both client and server has this capability then the server sends a merkle branch instead of a list of transactions.
What do you think, Luke-Jr and others? It could get bandwidth usage down for users who don't care to see the transactions. This specific method is worse than Stratum's new alternative, since the pool would now know which miners it can safely give dirty jobs to. I also don't think it's a good idea to encourage further pool centralization by making it any easier for miners to be neglegent. |
|
|
|
|
Your "explanation" only minorly improves things, it doesn't fix the problem. That's why it was last on the list of things to do to fix it.
None of what I said suggests any kind of fork is needed. Obviously the internals would support both mechanisms - it just isn't relevant to what needs to be done.
|
|
|
|
This is in fact a big issue with Luke-Jr also - him claiming about how what he is doing is better for BTC.
Yet what he was doing with his pool was restricting the number of transactions at his pool to 32 to increase the payments the pool received rather than fixing the problem with his pool software that was getting more orphans than other bigger pools at the same time.
The guy is a hypocrite - even worse: he enables such a thing at his pool ~5 months ago and the pool is still doing it.
I even explained a sizeable solution to this in bitcoind to both him and Gavin back then ... The problem isn't with the pool software, as you clearly admit to knowing by mentioning fixing bitcoind. Until bitcoind is fixed and the fix is deployed everywhere, this problem remains. Good job explaining the solution - even though your explanation doesn't actually solve it. Unfortunately, all contributions to bitcoind are done in peoples' spare time and things like a complete rewrite of the p2p code (which is needed to really fix this) takes a lot of time and skill. When this problem is nearly non-existent by simply filtering out a known DDoS attack, there are also higher priorities. OK what doesn't it solve? The solution is based on the fact that 1) Most bitcoinds will already have every transaction that they are sent (all except the coinbase transaction) So the protocol is sending a large amount of data that the receiving bitcoind already has and knows it has and doesn't need sent 2) The side effect of this is also that the receiving bitcoind again re-validates each transaction that it already has validated in it's memory pool The problem is that Bitcoin clients take much longer to relay blocks with more transactions, while by (paper) design and economic reasoning, including transactions in blocks should be near zero cost. Bitcoin clients in fact do already cache the validation of transactions they know (Gavin added this in 0.7, so it may be a result of your suggestion). However, the process is still absurdly slow. When a client receives a block, it does this, in order: - Request the block from the first peer to mention it
- Wait for that peer to relay the block in its entirety (ignoring all other peers in the meantime)
- Validate the block header
- Validate every transaction in the block (except the ones cached)
- Tell all our own peers - in no particular order, one by one - that we now have this block (note that if a particular peer's "transfer buffer" is already full, eg if they have a really crappy connection, bitcoind just STOPS AND WAITS FOR IT before moving on to any other peers)
- Finally, start looking at all/any requests our peers made since we first heard about this block
- Eventually, one of these peers will have requested the new block. Send it to them, but again do nothing with any other peers until this upload is complete.
- When that peer is done, another one probably asked for it in the meantime, upload it to them in the same way.
This is absurdly slow, and with large blocks can take a number of minutes to cross the network. The solution involves: Cache signature verifications (done)- Rewrite the Bitcoin peer-to-peer protocol handling to work asynchronously: (this is a LOT of work)
- if one peer can't receive data fast enough, send to other peers at the same time
- keep responding to other peers while things are going on
- Begin relaying blocks as soon as the header is downloaded and verified, before even receiving the rest of the block ourselves - this means the entire network is uploading/downloading the new block in parallel; since the difficulty is stupidly high, that makes any DoS attack impractical (I actually implemented this, before I discovered the previous problem which prevents it from working)
- Don't even upload transactions the other peer knows about already
Also note that it is wrong to assume everyone already knows about every transaction (except coinbase) before the block is found. Peers randomly discard (and don't relay) transactions without fees, so it's likely some of those will be missing on most nodes. |
|
|
|
This is in fact a big issue with Luke-Jr also - him claiming about how what he is doing is better for BTC.
Yet what he was doing with his pool was restricting the number of transactions at his pool to 32 to increase the payments the pool received rather than fixing the problem with his pool software that was getting more orphans than other bigger pools at the same time.
The guy is a hypocrite - even worse: he enables such a thing at his pool ~5 months ago and the pool is still doing it.
I even explained a sizeable solution to this in bitcoind to both him and Gavin back then ... The problem isn't with the pool software, as you clearly admit to knowing by mentioning fixing bitcoind. Until bitcoind is fixed and the fix is deployed everywhere, this problem remains. Good job explaining the solution - even though your explanation doesn't actually solve it. Unfortunately, all contributions to bitcoind are done in peoples' spare time and things like a complete rewrite of the p2p code (which is needed to really fix this) takes a lot of time and skill. When this problem is nearly non-existent by simply filtering out a known DDoS attack, there are also higher priorities. Oh, and stop trolling that GBT uses more bandwidth than getwork. It doesn't. |
|
|
|
Must I quote myself? If every template request was this size and the miner used the customise coinbase option and chose transactions, they would be receiving 400kb for the block template and each share submitted would be 400kb unless they started withholding transactions.
Too bad you're more interested in ignoring me than having a productive discussion, or you'd have noticed I pointed out exactly why that isn't true. |
|
|
|
not sure of the timing of it all but we have had to restart our stratum a couple of times, might explain some of it Don't you just love dripping wet, freshly released software?  When/if I implement Stratum or another similar protocol in Eloipool, I'd plan to save sockets across restarts :p Does Eloipool support merged mining? I have not looked at it yet. Kindof. It supports the setworkaux and gotwork JSON-RPC pair, which can be used for merged mining. It's not the simplest thing to setup, though. My hope going forward is that miners will setup their own merged mining, instead of relying on the pool for it, now that GBT enables this. |
|
|
|
not sure of the timing of it all but we have had to restart our stratum a couple of times, might explain some of it Don't you just love dripping wet, freshly released software? When/if I implement Stratum or another similar protocol in Eloipool, I'd plan to save sockets across restarts :p |
|
|
|
Might as well just use Stratum in that case, as you've negated the main benefit of GBT (decentralization). As long as we're talking p2pool though, might as well use another pool entirely, since decentralization is the main benefit of p2pool  But if you're just using Stratum for communication between p2pool and your mining software, centralization isn't a problem. In that case, neither is bandwidth If you're mining on someone else's P2Pool instance then you're already becoming more centralized. Not if it was using GBT. But obviously with p2pool's 10 second altchain blocks, the protocol used by GBT today is unreasonable. It does defeat some of the ideological niceties of GBT, but it's still an optional flag (Stratum has similar from what I've heard, though it's off by default, instead of on by default with this proposed compact flag). No, it's currently not possible to use GBT in a centralized setup. Stratum has recently added a similar feature, but it still has overhead for decentralized use. |
|
|
|
Maybe it would be worth adding a "transaction/compact" flag to the GBT request and allow it to respond with just the merkle branches instead of the full transactions. If my logic isn't too far off that would close the gab between GBT and Stratum based on bandwidth usage. Might as well just use Stratum in that case, as you've negated the main benefit of GBT (decentralization). As long as we're talking p2pool though, might as well use another pool entirely, since decentralization is the main benefit of p2pool |
|
|
|
How goes getting StratumMP BIP process off the ground? Are we still waiting on genjix? No, I'm just busy. Writing sane documentation (in english) is pretty hard for me, so I need a lot of time to finalize something. Well, we're just talking about a first draft, it doesn't have to be perfect - the rest of the community will probably have things they want added or changed before the final anyway. "There is reasonable concerns about mining currently being too centralized on pools, and the amount of control these pools hold. By exposing the details of the block proposals to the miners, they are enabled to audit and possibly modify the block before hashing it." You didn't respond to Stratum's "get_transactions" extension explicitly. Can I ask you what's your opinion to this? I think it solves exactly that issue with blind trust for picking transaction by the pool, without unnecessary overhead for common miners. It's a step forward, at least; but the design here still encourages miners and pools to avoid (or bill extra for, as you mentioned) decentralization. I don't think "common miners" should be encouraged to centralize. I prefer coding, not discussion. We can spend months on discussions or we can both show our solutions and let others to decide. It's not either one or the other. The BIP process generally involves documenting and implementing a first draft, and then discussion to come up with a final standard solution that meets everyone's needs. Eligius had GBT since the first draft in February, but the protocol continued to change significantly as other developers contributed to it before reaching its current form. |
|
|
|
One more point about the GBT protocol. If a miner chooses which transactions they include rather than accept whichever transactions the pool/bitcoind has offered in the template, the miner has to submit all the transactions with every share he submits. No, that's (part of) what Block Proposal is for. That's not implemented yet is it? Allowing miners to modify transactions is not implemented in miners yet at all. Block Proposal only makes sense as part of that. I do have it implemented for the Eloipool<->bitcoind link. When Luke-Jr made a public proposal and asked for comments, I read through it and sent in my thoughts on how it could be improved. Why didn't more people do that? Instead waiting until there are implementations in production and then finally spitting out nasty words and technical misunderstandings. That's not the way to go about things. Please share with us all how an open source project where anyone can create anything can be followed by anyone that maybe interested in it? Note: I only found out that Stratum when I was told on another forum that Bitcoind will not work with my mini rig. Then I discovered GBT. There is a Bitcoin development mailing list specifically for discussing all things related to Bitcoin development. GBT had a number of threads of discussion on it since February. |
|
|
|
When Luke-Jr made a public proposal and asked for comments, I read through it and sent in my thoughts on how it could be improved. Why didn't more people do that? I did not so for three reasons: a) I had no idea how it has been supposed to work. And I still don't understand majority of his proposal, optional parts etc. I understand GBT to point that I can load block template and put valid block back to bitcoind, but the rest seems to be a rubbish for me. Why? b) If I'd try to help Luke's with new mining protocol, then I'll suggest Stratum. That's fine, Stratum seems like a logical place to start for a new standard. It would have been nice if these things had taken place during the GBT standardization, but that's in the past now. How goes getting StratumMP BIP process off the ground? Are we still waiting on genjix? Obviously there were different targets; Luke wanted something which can be implement in bitcoind, but I wanted something optimized for pooled mining. I'm saying it again and again, that Stratum and GBT can live together and only few people have some mental issues with it. You keep saying this, and I keep having to correct you: No, the main focus of GBT was from the start was pooled mining. Citing the Motivation section of BIP 22 from all the way back in the original February draft... "There is reasonable concerns about mining currently being too centralized on pools, and the amount of control these pools hold. By exposing the details of the block proposals to the miners, they are enabled to audit and possibly modify the block before hashing it." c) It is very hard to discuss with Luke, so creating my own solution which included all my best ideas was a bit easier ;-). This is just trolling. |
|
|
|
|
Show Posts
