Title: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: jgarzik on February 15, 2011, 08:17:19 AM 0. The URLs
---------------------------------------------------------------------------------- URL: http://yyz.us/bitcoin/pushpool-0.4.tar.gz Repo: https://github.com/jgarzik/pushpool 1. The Problem ---------------------------------------------------------------------------------- With the recent slashdotting and resultant influx in new users, the 'getwork' network protocol used in mining is showing some strain, particularly on the pools. Miners request work once every 5-10 seconds using HTTP JSON-RPC, which has several glaring inefficiencies that lead to unnecessary server load:
Several people, pool operators in particular, have a keen interest in solving these problems. In addition, push mining (see below) has been discussed as a future alternative to the 'getwork' polling method currently employed. 2. Design goals for a solution ---------------------------------------------------------------------------------- I have written a demonstration pool server (aka a 'getwork' proxy server) that functions in a similar fashion to the recently-discussed poold.py (http://bitcointalk.org/index.php?topic=3078.0): large numbers of miners connect to the pool server, which proxies 'getwork' JSON-RPC requests to the official bitcoin client. This demonstration server implements a new binary protocol that was designed to meet the following goals:
This is not intended to replace JSON-RPC API, but to supplement it for specific use cases. Yes, that means bitcoind will listen to three network ports: P2P network, JSON-RPC, and binary RPC (though as now, only P2P is required for operation; the servers are always optional). 3. Let's start with a protocol example: today's getwork mining ---------------------------------------------------------------------------------- The specific details of the protocol itself are in ubbp.h and protocol.h of the above URL (pushpool-0.1.1.tar.gz). Here is an example, to provide a suitable introduction: * TCP connection is broken up into messages. Each message has a 64-bit header, with 8-bit opcode and 24-bit length fields. * Miner client connects to TCP server, and issues a LOGIN message, which is compressed JSON login data + sha256 hash of (data + shared secret). * Server responds with an OP_LOGIN_RESP msg, compressed JSON, indicating options and capabilities * Client issues an OP_GETWORK msg (8 bytes) * Server responds with an OP_WORK msg (264 bytes) * Client uses its CPU/GPU to work on proof-of-work solution... * Client issues an OP_GETWORK msg (8 bytes) * Server responds with an OP_WORK msg (264 bytes) * ... The above example intentionally matches existing 'getwork' JSON-RPC miner client workflow today. Miner clients may even support stateless operation by pipelining the OP_LOGIN and OP_GETWORK requests together, and closing the TCP connection. Stateless operation is not recommended, but it is supported, in order to support the widest range of existing mining clients. 4. Tomorrow's mining: push mining ---------------------------------------------------------------------------------- When a block or tx arrives, it is preferable to begin working immediately on the new work. From the server's perspective, this is a classic data-broadcast problem, where the server wants to broadcast N different pieces of work to N miners. Hence, "push mining" where the server pushes new work pro-actively to the miner clients. This new network protocol supports pushing mining, as demonstrated in this example: * Client connects to server, issues a LOGIN message with the "send_me_work" flag set * Server responds with OP_LOGIN_RESP msg * Server sends a OP_WORK msg * Server sends a OP_WORK msg * Server sends a OP_WORK msg * ... 5. A similar use case: monitorblocks ---------------------------------------------------------------------------------- Gavin Andresen has a patch in his github which provides a very useful feature: when a new block is received (monitorblocks) or new wallet transaction (monitortx), bitcoind sends an HTTP POST to the specified URL. Thus, monitorblocks provides real-time monitoring of the bitcoin network, and monitortx provides real-time monitoring of the local wallet. This sort of featureset pushes data as events occur, rather than forcing a website operator to poll JSON-RPC for certain operations to complete. Monitoring new blocks on the bitcoin network is a very easy data broadcasting problem that this binary network protocol may easily support: * Client connects to server, issues a LOGIN message with the "send_me_blocks" flag set * Server responds with OP_LOGIN_RESP msg * Server sends a OP_BLOCK msg * Server sends a OP_BLOCK msg * Server sends a OP_BLOCK msg ... monitortx is more complicated, because one may specify transaction-matching criteria. But with this new protocol's support of JSON, flexibility is not a problem. 6. A plan to proceed - this is just a rough draft ---------------------------------------------------------------------------------- I'm thinking of the following steps to proceed, given the need to coalesce several potentially parallel push-mining efforts:
Let the comments begin... hopefully someone will volunteer to mod a GPU miner to support this? Appendix 1: FAQ ---------------------------------------------------------------------------------- Q. Why invent a new protocol? Why not use Google protocol buffers or XDR? A. protobuf and XDR both require an underlying packetizing format, such as UBBP that I've presented here. That implies the choice would be UBBP+protobuf or UBBP+JSON. Given the bitcoin community's embrace of JSON, the latter was chosen. JSON is actually more flexible than protobufs, because more dynamic data structures may be described using JSON. Q. Why did you not address glaring problems in getwork? A. I focused purely on a network-efficient protocol. getwork implementation choices are outside the scope of this work. Q. What is the state / quality of this code release? A. Uh, it compiles and runs... but no clients yet exist for it. Without a miner client for testing, it's about as useful as spitting on a fish... Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: Cdecker on February 15, 2011, 11:48:20 AM Sounds like a reasonable enhancement, especially since the miners will be notified of a new block and will be able to start on the new one right away :D
Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: m0Ray on February 15, 2011, 07:50:47 PM Why not UDP?
Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: jgarzik on February 15, 2011, 08:56:56 PM Why not UDP? Retransmissions imply you wind up reinventing TCP. Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: m0Ray on February 15, 2011, 10:10:55 PM Why not UDP? Retransmissions imply you wind up reinventing TCP.I think it can be like this: Code: Miner->Server: LOGIN username password On login error server can simply ignore the request or send some REJECT messages. On success the server records miner IP and port number and adds it to miner identification table together with current timestamp. Then: Code: Server->Miner: WORK .... Once during keepalive_timeout: Code: Miner->Server: ALIVE Code: Miner->Server: RESULT ... Miner resends RESULT message if it does not receive an answer in reasonable time. Server will remove miner id records from its table after 4*keepalive_timeout. Note that server can adapt keepalive_timeout for certain miners using statistical data on the fly. This will reduce traffic to minimum, IMHO. Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: jgarzik on February 15, 2011, 10:33:56 PM Why not UDP? Retransmissions imply you wind up reinventing TCP.A miner does not want to lose a WORK msg, GETWORK msg, nor have their solution lost. Every single message -- LOGIN, GETWORK, WORK, ... -- must be retransmitted or retried by one side or the other. But that's just one of many disadvantages of UDP. TCP is also better supported by most programming language libs, and is more firewall- and NAT-friendly. Having implemented many UDP servers of various sorts -- financial data feeds, gaming servers, and cloud computing coordinators -- you really do wind up reinventing TCP while attempting to simply have a robust UDP implementation. Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: m0Ray on February 15, 2011, 10:50:49 PM 1) Neither login request nor solution are not lost in my proposal.
2) I don't think that loss of WORK message will seriously impact the performance. If I understand the principle, new WORK will be broadcasted with every transaction received by server. And it's often enough. 3) I mean only "push" protocol, which does not use GETWORK. 4) UDP, as far as I know, is the same NAT-friendly as TCP. In both cases NAT box just maps the source port. 5) It is not widely supported by libs only because it is very simple. 6) Some features of TCP are really overhead when building a low-latency service. So it is sometimes better to reimplement some TCP features than use its full version. For example, in this case we don't need an acknowledge for every message. And... the SYN flood... Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: comboy on February 16, 2011, 02:04:24 AM I just want to say I also think UDP is a weird/bad idea here.
And even though I have not yet time to dive in, solution looks very nice. Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: jgarzik on February 16, 2011, 02:43:23 AM 1) Neither login request nor solution are not lost in my proposal. If they cannot be lost, then by definition they must be retransmitted. And you must build logic to determine how often to retransmit. When to stop retransmitting and give up. Reinventing TCP, in other words. Quote 2) I don't think that loss of WORK message will seriously impact the performance. If I understand the principle, new WORK will be broadcasted with every transaction received by server. And it's often enough. 3) I mean only "push" protocol, which does not use GETWORK. The loss of a WORK message can mean the loss of money, due to not working on the latest block etc. No miner will stand for this, therefore, WORK must be acknowledged by client, and retransmitted by server. TCP does this for us automatically. Quote 4) UDP, as far as I know, is the same NAT-friendly as TCP. In both cases NAT box just maps the source port. UDP has no notion of connections, so a heavily loaded NAT box must rely on timeouts and other hacks, unlike TCP. But in focusing on NAT you ignored "firewall"; TCP far more readily passes through firewalls than UDP. I've seen this at plenty of large corporate sites especially. They'll do a local DNS server, and no UDP traffic will traverse the firewall into the outside world. If you want universality, UDP is not the way to go. TCP is simply more likely to succeed. Quote 6) Some features of TCP are really overhead when building a low-latency service. So it is sometimes better to reimplement some TCP features than use its full version. For example, in this case we don't need an acknowledge for every message. Only if you don't mind losing money :) Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: bitcoinex on February 16, 2011, 01:00:30 PM What about increase time interval between requests? And ban for who will be use the frequent requests.
Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: burtyb on February 16, 2011, 02:21:39 PM What about increase time interval between requests? And ban for who will be use the frequent requests. That might cause problems for those using a high number of CPU cores though? Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: jgarzik on February 16, 2011, 05:06:40 PM What about increase time interval between requests? And ban for who will be use the frequent requests. This is not a network protocol issue. You can easily add these rules to your pool server or bitcoind, once the protocol is deployed. Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: slush on February 16, 2011, 06:23:13 PM I just read the sources and mixing binary protocol with json compressed data looks weird for me. Why not simply use (compressed) json RPC over TCP and define only RPC methods? This should be way easier to implementation in any language, more standard, readable etc. But it still enable push features and will be more efficient because we get rid of HTTP overhead. Please don't reinvent the wheel.
Maybe I'm too high-level oriented, but encapsulating JSON RPC into proprietary binary protocol is very unusual. For example, method 'login' should look like {id:'xxx',method:'login',params:['username','sha256 of username+password']}. One command can be finished by new line, or better, almost every language has support for streaming JSON (well, I know Java and Python libraries), because it is very easy to detect that message is complete. Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: jgarzik on February 16, 2011, 06:31:33 PM I just read the sources and mixing binary protocol with json compressed data looks weird for me. Why not simply use (compressed) json RPC over TCP and define only RPC methods? Because sending work as compressed JSON involves
It is obviously more simple -- less CPU usage and less bandwidth usage -- to send binary work data directly. Remember, binary data is the common case for mining. JSON is in the protocol for flexible feature negotiation and configuration. But we must to avoid today's binary->text->compressed->text->binary redundant data encoding, because the miners work on binary data. Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: slush on February 16, 2011, 06:52:59 PM Because sending work as compressed JSON involves
Absolutely marginal overhead. Average CPU core can encode megabytes of data to hexadecimal per second. Quote
So 2x more data (two bytes for one raw byte) for payload itself. For one mining job, only few bytes is really required, most of current data sending to client are not used (source: m0mchil). Much more effective way is to change job payload itself. Quote
Which is also in your proposal, for storing message payload. Again, I don't see real trouble here. Quote
That's why I told I'm probably too highlevel. I really don't care about finding pointer in hex string. It is much more cost effective to leave this job on computers and high level libraries than fiddling with bits on low level protocol. Don't forget that this protocol have to be reimplemented in many languages, so using standard protocol, you save tens of hours of labour for programming and bug fixing. Quote It is obviously more simple -- less CPU usage and less bandwidth usage -- to send binary work data directly. Remember, binary data is the common case for mining. You are right that raw binary protocol is really the most effective. But let's find some reasonable level of optimizations. It does not need to be _perfect_. It need to be effective AND easy to handle/debug. Don't forget that you are optimizing nanoseconds of CPU job and then perform one SQL request, which is 100x slower than any protocol parsing. Rough calculation: Now: -------- One request: 300 bytes of HTTP request, 700 bytes of data ==> ~1 kB of data every 5 seconds for each worker. It is 12kB per minute per worker. Json over TCP: -------- One request: (approx) 20 bytes of request, 300? bytes of response EVERY MINUTE ==> 320 bytes per minute per worker. By very simple optimization, you cut bandwidth to 2.5% of original size. Without any binary fiddling and proprietary stuff. How many % will be the savings between Json over TCP and binary over TCP?[/list] Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: slush on February 16, 2011, 07:05:24 PM One request: (approx) 20 bytes of request, 300? bytes of response EVERY MINUTE ==> 320 bytes per minute per worker. Well, I know that I'm again mixing protocol and getwork implementation. But there is no big point in supporting getwork over tcp and still sending job every 5 seconds. So I'm talking about real situation, about using TCP protocol and real pushwork implementation at once. Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: jgarzik on February 16, 2011, 07:16:51 PM Because sending work as compressed JSON involves
Absolutely marginal overhead. Average CPU core can encode megabytes of data to hexadecimal per second. Quote
So 2x more data (two bytes for one raw byte) for payload itself. For one mining job, only few bytes is really required, most of current data sending to client are not used (source: m0mchil). Much more effective way is to change job payload itself. Quote
Which is also in your proposal, for storing message payload. Again, I don't see real trouble here. Quote
That's why I told I'm probably too highlevel. I really don't care about finding pointer in hex string. It is much more cost effective to leave this job on computers and high level libraries than fiddling with bits on low level protocol. Don't forget that this protocol have to be reimplemented in many languages, so using standard protocol, you save tens of hours of labour for programming and bug fixing. You are being too literal. Even python must do this step: work = json_result['data'] Quote Quote It is obviously more simple -- less CPU usage and less bandwidth usage -- to send binary work data directly. Remember, binary data is the common case for mining. You are right that raw binary protocol is really the most effective. But let's find some reasonable level of optimizations. It does not need to be _perfect_. It need to be effective AND easy to handle/debug. Don't forget that you are optimizing nanoseconds of CPU job and then perform one SQL request, which is 100x slower than any protocol parsing. Doing all the extra, pointless work of binary->text->compression->text->binary also increases the chances for programmer error. Once you have a binary, packetized protocol, the easiest, least error-prone thing to do is receive (or create, in bitcoind's case) a raw binary packet, and pass that directly to a connected miner. Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: jgarzik on February 16, 2011, 07:20:49 PM One request: (approx) 20 bytes of request, 300? bytes of response EVERY MINUTE ==> 320 bytes per minute per worker. Well, I know that I'm again mixing protocol and getwork implementation. But there is no big point in supporting getwork over tcp and still sending job every 5 seconds. So I'm talking about real situation, about using TCP protocol and real pushwork implementation at once. The protocol supports multiple use cases:
The protocol supports LAN or WAN, bitcoind or pool server. If the miner client prefers polling over push mining, they may choose to do so. Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: slush on February 16, 2011, 07:33:23 PM You are being too literal. Even python must do this step: work = json_result['data'] OK, I didn't told that the pointer lookup is not needed/performed inside, automatically. It is just done by standard libraries presented in every language and there is no need to implement binary stuff again. Quote Doing all the extra, pointless work of binary->text->compression->text->binary also increases the chances for programmer error. Well, we are talking about personal opinions. My opinion is that high level programming is much easier and error prone than low level implementation. Thanks to this attitude, we're programming in high level languages and not in assembler. And you still does not give the calculation of bandwidth savings against JSON RPC over TCP. Quote Once you have a binary, packetized protocol, the easiest Correct. But I hope you don't want to say that import json json.decode(sock.read()) is harder to do than creating own parsing library for every language (C, Python, Java, PHP?) and unpacking binary data, right? I don't want to be personal in any way, I think it's great that you open this topic. I'm just finding some equilibrium between hardcore lowlevel stuff and almost standard protocol implemented anywhere. I'm simply not convicted that this protocol need such heavy overoptimization. Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: adv on February 16, 2011, 07:36:15 PM What about increase time interval between requests? And ban for who will be use the frequent requests. What about many NATed users?I think you say "they must use different username/password". But they dont MUST do this and may write miner, that use different accounts for frequently requests... Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: slush on February 16, 2011, 07:41:58 PM The protocol supports multiple use cases:
The protocol supports LAN or WAN, bitcoind or pool server. If the miner client prefers polling over push mining, they may choose to do so. Which is not against JSON over TCP. All of this you can perform also with my proposal. I just mentioned that the biggest saving (for mining) will be the step from current getwork() over HTTP to the TCP communication and push mining together, but not in saving single bytes in binary protocol. Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: jgarzik on February 16, 2011, 07:54:03 PM You are being too literal. Even python must do this step: work = json_result['data'] OK, I didn't told that the pointer lookup is not needed/performed inside, automatically. It is just done by standard libraries presented in every language and there is no need to implement binary stuff again. There is no difference between this and struct.unpack() in python, except that the JSON decode is more work, more data, and less efficient. Quote Quote Doing all the extra, pointless work of binary->text->compression->text->binary also increases the chances for programmer error. Well, we are talking about personal opinions. My opinion is that high level programming is much easier and error prone than low level implementation. Thanks to this attitude, we're programming in high level languages and not in assembler. And you still does not give the calculation of bandwidth savings against JSON RPC over TCP. The precise packet sizes were given in first post. WORK message can easily be made smaller, too. Quote Quote Once you have a binary, packetized protocol, the easiest Correct. But I hope you don't want to say that import json json.decode(sock.read()) is harder to do than creating own parsing library for every language (C, Python, Java, PHP?) and unpacking binary data, right? I don't want to be personal in any way, I think it's great that you open this topic. I'm just finding some equilibrium between hardcore lowlevel stuff and almost standard protocol implemented anywhere. I'm simply not convicted that this protocol need such heavy overoptimization. If you are packetizing data for JSON-over-TCP, you are unpacking binary data to obtain message size of message sent via TCP. Furthermore, inside each work unit, existing miner clients are unpacking binary data in order to perform hashing (miners hash binary data!). High-level python language happily unpacks binary data -- we have miners written in Python and Java today. Or see ArtForz's bitcoin client: http://pastebin.com/ZSM7iHZw "don't do additional, redundant, unneeded work" is not heavy optimization. It is reducing complexity in the miner client. Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: Cdecker on February 17, 2011, 01:41:30 PM I have to say that I agree with slush, after implementing the Bitcoin protocol in Java I hate binary protocols.
Encoding it in JSON would allow easier implementations since most languages have some form of understanding JSON. Compression is not needed but we might simply add another port that compresses all that goes through it (think OutputStream Chaining in Java). The size of the messages will double, yes, but as far as I'm concerned that's the only downside to using JSON. Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: m0mchil on February 17, 2011, 01:46:31 PM Or perhaps we should use something like protocol buffers, even for core bitcoin messages?
Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: Cdecker on February 17, 2011, 02:39:00 PM Or perhaps we should use something like protocol buffers, even for core bitcoin messages? You mean the P2P protocol? I'd already be happy using JSON, but it won't work because the whole block hashing and transaction hashing relies on the binary representation of the message itself...Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: slush on February 17, 2011, 05:10:43 PM Or perhaps we should use something like protocol buffers, even for core bitcoin messages? +1 for use of standardized binary protocol everywhere. Short summary: JSON-RPC (over TCP): + Already used in bitcoin + Standard protocol + Extremely easy to use - Missing support for binary data - General overhead (repeating of data field names) Protocol buffers (https://code.google.com/p/protobuf/): + Easy to use + Standard protocol + Support for binary data + Bandwidth effective + May replace current binary protocol on P2P side + No external dependency needed (protocol compiler produces plain C++ code) Proprietary binary protocol: + Extremely bandwidth effective + No external dependency needed + Support for binary data (noticed for integrity of this summary) - Hard to learn & use (everybody have to implement support in his preferred language) - Not reusable (P2P and push interface needs two different protocols) From this summary, I prefer solutions in this order: 1. Protocol buffers 2. JSON 3. Binary protocol Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: adv on February 17, 2011, 10:08:24 PM From this summary, I prefer solutions in this order: 1. Protocol buffers 2. JSON 3. Binary protocol We talk only about mining. The only significant thing for him - speed. No need of readability or understandability. You can use a heavy program optimization, go-to operator, or even assembler. : ^) So please forget about JSON-RPC. If he so much needed to someone - it is always possible to use a standard getwork-interface over http. And the creation of a push-protocol over JSON -- a completely separate problem with its advantages and disadvantages. I think the choice is between only protocol buffers and pure binary protocol. Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: jgarzik on February 17, 2011, 10:15:49 PM Protocol buffers for the P2P networking protocol have already been discussed (http://bitcointalk.org/index.php?topic=632.0). Not really realistic now, but "it would have been nice."
For the purposes of this thread, protocol buffers have three disadvantages:
Basically, any packetized protocol -- let us say JSON-over-TCP (rather than JSON-over-HTTP) -- requires some amount of low-level raw binary parsing to locate message boundaries. To simply pass through data unchanged, rather than decoding compressed JSON, comes for free in such protocols. Therefore, logically, any JSON-over-TCP protocol already has the ability to skip the pointless native->encoded->native step that each miner client performs, by using this pass-through method. Passing raw binary work directly from bitcoind to the client eliminates the possibility of bugs and problems related to the unnecessary encoding+decoding an additional JSON step would require. Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: Cdecker on February 17, 2011, 11:27:18 PM Changing the P2P protocol is not going to happen anytime soon for the above mentioned reasons (although I would welcome it) because it would require a transition phase for the entire network to switch. I just mentioned it because someone asked me about it.
For the pushwork patch i think you are forgetting that only because you decide to use one over the other for speed reasons doesn't do anything. You have to make the format easy to implement since the miners do have to support it. The getwork interface must be enabled anyway so if the coders of the miners don't adopt the protocol you'll not have changed anything, except a lot of work spent in a useless interface. Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: adv on February 18, 2011, 09:59:41 PM Changing the P2P protocol is not going to happen anytime soon for the above mentioned reasons (although I would welcome it) because it would require a transition phase for the entire network to switch. As I understand it, you're talking about using a binary protocol in bitcoin client. I (think Jgarzik too) also have in mind is the only mining protocol and hope that the miner would soon be removed from the official client. And, yes: JSON-RPC getwork protocol should remain in the official client. (May be need to separate the getwork-request from wallet-management commands for the possibility of mining on untrusted hosts). I think the binary protocol should be used only in separate from a client miners through some JSON-binary proxy, or through connecting to the official client plugin/module, or in extreme cases can be easily separated part of the code in the official program. If you think that the use of JSON-binary proxy is useless because of the overhead JSON, who do not disappear, there are two advantages: 1. JSON binary proxy can stand on the border of fast and low-band networks, using JSON in fast and binary protocol in the slow. 2. After receiving one getwork through JSON proxy can divide (cut nonce) it for a few miners through binary protocol. It must shows good results: http://bitcointalk.org/index.php?topic=3486.msg48996#msg48996 Perhaps we should think about the possibility of receiving getwork bundles trough JSON. Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: martok on April 03, 2011, 10:21:28 PM I know it isn't the object of pushpool's purpose but I think long polling might be broken in the current implementation. poclbm occasionally throws a stale so I used curl to test /LP and it never returns even when the block changes.
Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: jgarzik on April 03, 2011, 11:09:14 PM I know it isn't the object of pushpool's purpose but I think long polling might be broken in the current implementation. poclbm occasionally throws a stale so I used curl to test /LP and it never returns even when the block changes. Do you send SIGUSR1 to pushpoold, when a new block arrives? pushpoold assumes you will implement your own block monitoring solution, and notify it when a new block arrives. Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: martok on April 04, 2011, 01:16:16 AM Ah thanks didn't know about that bit.
Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: martok on April 04, 2011, 11:07:14 PM Hi again,
Ok I had to fix LP in pushpoold but have it working now. Are you maintaining this on github? If so, I can feed you patches. Basically: sigusr1 is currently set to dump stats. It sets to lp_flush but then sets to dump stats later on so that needed fixing. Also, lp was returning bad request on a regular GET request IE without a JSON payload. I hacked together a solution so calling /LP returns a GETWORK as specified in the spec I read. I also have hacked sharelog to insert into the SQLITE3 db rather than using a logfile but that's strictly a matter of preference. I suppose the logfile is just as flexible as it could be a FIFO having the other end be a program that does whatever (postgresql/mysql). Unfortunately, Python isn't my language or I'd have a go at the binary protocol implementation. Thought of rewriting the miner in C though if only to have it support things like --device=1,2,3,4 to run all of the miner threads in a single application instance. Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: jgarzik on April 04, 2011, 11:19:09 PM Ok I had to fix LP in pushpoold but have it working now. Are you maintaining this on github? If so, I can feed you patches. For the moment, email to jgarzik@exmulti.com Quote sigusr1 is currently set to dump stats. It sets to lp_flush but then sets to dump stats later on so that needed fixing. Already fixed in local git Quote Also, lp was returning bad request on a regular GET request IE without a JSON payload. I hacked together a solution so calling /LP returns a GETWORK as specified in the spec I read. A GET request without a JSON payload is a bad request :) What is your preferred behavior? Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: martok on April 05, 2011, 12:41:57 AM A GET request without a JSON payload is a bad request :) What is your preferred behavior? Hmm, I am going by the spec here: http://deepbit.net/longpolling.php 3) Miner starts a request to long polling URL with GET method and same basic authorization as on main connection. This request is not answered by server until new block is found by bitcoin network. The answer is the same as getwork on the main connection. So by that reading, there is no need for an actual GETWORK JSON payload on the long poll URL as the response is assumed to be a getwork. Looking at poclbm, it's just sending get without any post data. Is this wrong behaviour? Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: martok on April 07, 2011, 12:01:06 AM Hi,
Occasionally, poclbm is throwing the following when connected to pushpoold: Traceback (most recent call last): File "/home/bitcoin/src/poclbm/BitcoinMiner.py", line 141, in mine self.sendResult(self.resultQueue.get(False)) File "/home/bitcoin/src/poclbm/BitcoinMiner.py", line 166, in sendResult d = result['work']['data'] TypeError: 'NoneType' object is unsubscriptable Traceback (most recent call last): File "/home/bitcoin/src/poclbm/BitcoinMiner.py", line 141, in mine self.sendResult(self.resultQueue.get(False)) File "/home/bitcoin/src/poclbm/BitcoinMiner.py", line 166, in sendResult d = result['work']['data'] TypeError: 'NoneType' object is unsubscriptable But poclbm works fine via local bitcoind or via the commercial pools. Have you encountered this yet. Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: jgarzik on April 07, 2011, 08:13:21 AM Occasionally, poclbm is throwing the following when connected to pushpoold: Traceback (most recent call last): File "/home/bitcoin/src/poclbm/BitcoinMiner.py", line 141, in mine self.sendResult(self.resultQueue.get(False)) File "/home/bitcoin/src/poclbm/BitcoinMiner.py", line 166, in sendResult d = result['work']['data'] TypeError: 'NoneType' object is unsubscriptable Traceback (most recent call last): File "/home/bitcoin/src/poclbm/BitcoinMiner.py", line 141, in mine self.sendResult(self.resultQueue.get(False)) File "/home/bitcoin/src/poclbm/BitcoinMiner.py", line 166, in sendResult d = result['work']['data'] TypeError: 'NoneType' object is unsubscriptable But poclbm works fine via local bitcoind or via the commercial pools. Have you encountered this yet. What is occasionally? Once a day? Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: jgarzik on April 07, 2011, 08:13:54 AM pushpool git repo is now live at https://github.com/jgarzik/pushpool
Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: xf2_org on April 21, 2011, 06:30:38 AM Version 0.3 released, with support for mysql, postgres and sqlite databases, for both share logging and password checking.
Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: xf2_org on May 10, 2011, 01:55:36 AM Version 0.4 released, a bugfix release.
See top post for tarball and git URLs. Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: tuxsoul on May 12, 2011, 02:49:45 AM I'm working to package pushpool from @jgarzik
* if somebody know how compile it in ubuntu or debian, please comment here: http://bitcointalk.org/index.php?topic=2207.msg115320#msg115320 * can you suggest one short description. * can you suggest one full description. read more about not oficial debian/ubuntu packages: http://bitcointalk.org/index.php?topic=2207 Greeting's. Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: gigabytecoin on May 15, 2011, 06:57:09 PM I'm working to package pushpool from @jgarzik * if somebody know how compile it in ubuntu or debian, please comment here: http://bitcointalk.org/index.php?topic=2207.msg115320#msg115320 * can you suggest one short description. * can you suggest one full description. read more about not oficial debian/ubuntu packages: http://bitcointalk.org/index.php?topic=2207 Greeting's. I second this. Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: gkolt on May 17, 2011, 05:48:37 PM I can't compile pushpool in Debian.
"configure: error: Missing required libevent" But I do have libevent. # locate libevent /usr/lib/libevent-1.1.so.1 /usr/lib/libevent-1.1a.so.1 /usr/lib/libevent-1.3b.so.1 /usr/lib/libevent-1.3d.so.1 /usr/lib/libevent-1.3e.so.1 /usr/lib/libevent-1.3e.so.1.0.3 /usr/lib/libevent.so.1 /usr/lib/libeventdb.a /usr/lib/libeventdb.so /usr/lib/libeventdb.so.2 /usr/lib/libeventdb.so.2.0.0 # dpkg -l|egrep -i libevent ii libev-libevent-dev 3.43-1.1 libevent event loop compatibility wrapper for libev ii libev3 3.43-1.1 high-performance event loop library modelled after libevent ii libevent-perl 1.11-1 Generic Perl event loop ii libevent-rpc-perl 1.00-1 Event based transparent Client/Server RPC framework ii libevent1 1.3e-3 An asynchronous event notification library ii libeventdb-dev 0.90-4 library that provides access to gpe-calendar data [development ii libeventdb2 0.90-4 library that provides access to gpe-calendar data [runtime] Any ideas ? Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: xf2_org on May 17, 2011, 09:02:32 PM pushpool support moved to this thread: http://forum.bitcoin.org/index.php?topic=8707.0
Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: cryptograd on August 15, 2013, 04:31:47 PM Would this support ASIC miners(stratum)
?????????? Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: mjc on September 26, 2013, 02:42:16 PM bump, I have the same question about Stratum.
Title: Re: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc. Post by: jgarzik on September 27, 2013, 11:41:41 AM Yes, it can work for ASICs just fine.
stratum is just another mining protocol. You may modify the protocol to support stratum-like work, yes. |