Bitcoin Forum
December 05, 2016, 08:37:49 AM *
News: To be able to use the next phase of the beta forum software, please ensure that your email address is correct/functional.
 
   Home   Help Search Donate Login Register  
Pages: « 1 [2] 3 »  All
  Print  
Author Topic: Bitcoin Binary Data Protocol, for mining, monitorblocks, etc.  (Read 24868 times)
slush
Legendary
*
Offline Offline

Activity: 1358



View Profile WWW
February 16, 2011, 07:41:58 PM
 #21

The protocol supports multiple use cases:
  • getwork polling (ie. how every single miner is written today).  C:GETWORK  S:WORK  C:GETWORK  S:WORK ...
  • push mining   C:CONFIG(push mining)  S:WORK  S:WORK  S:WORK  S:WORK  ...
  • monitorblocks   C:CONFIG(monitor blocks)   S:BLOCK   S:BLOCK   S:BLOCK   ...

The protocol supports LAN or WAN, bitcoind or pool server.

If the miner client prefers polling over push mining, they may choose to do so.

Which is not against JSON over TCP. All of this you can perform also with my proposal. I just mentioned that the biggest saving (for mining) will be the step from current getwork() over HTTP to the TCP communication and push mining together, but not in saving single bytes in binary protocol.

1480927069
Hero Member
*
Offline Offline

Posts: 1480927069

View Profile Personal Message (Offline)

Ignore
1480927069
Reply with quote  #2

1480927069
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1470


View Profile
February 16, 2011, 07:54:03 PM
 #22

You are being too literal.  Even python must do this step:  work = json_result['data']

OK, I didn't told that the pointer lookup is not needed/performed inside, automatically. It is just done by standard libraries presented in every language and there is no need to implement binary stuff again.

There is no difference between this and struct.unpack() in python, except that the JSON decode is more work, more data, and less efficient.

Quote
Quote
Doing all the extra, pointless work of binary->text->compression->text->binary also increases the chances for programmer error.

Well, we are talking about personal opinions. My opinion is that high level programming is much easier and error prone than low level implementation. Thanks to this attitude, we're programming in high level languages and not in assembler.

And you still does not give the calculation of bandwidth savings against JSON RPC over TCP.

The precise packet sizes were given in first post.  WORK message can easily be made smaller, too.

Quote
Quote
Once you have a binary, packetized protocol, the easiest

Correct. But I hope you don't want to say that

import json
json.decode(sock.read())

is harder to do than creating own parsing library for every language (C, Python, Java, PHP?) and unpacking binary data, right?

I don't want to be personal in any way, I think it's great that you open this topic. I'm just finding some equilibrium between hardcore lowlevel stuff and almost standard protocol implemented anywhere. I'm simply not convicted that this protocol need such heavy overoptimization.

If you are packetizing data for JSON-over-TCP, you are unpacking binary data to obtain message size of message sent via TCP.  Furthermore, inside each work unit, existing miner clients are unpacking binary data in order to perform hashing (miners hash binary data!).  High-level python language happily unpacks binary data -- we have miners written in Python and Java today.   Or see ArtForz's bitcoin client:  http://pastebin.com/ZSM7iHZw

"don't do additional, redundant, unneeded work" is not heavy optimization.  It is reducing complexity in the miner client.

Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
Cdecker
Hero Member
*****
Offline Offline

Activity: 487



View Profile WWW
February 17, 2011, 01:41:30 PM
 #23

I have to say that I agree with slush, after implementing the Bitcoin protocol in Java I hate binary protocols.

Encoding it in JSON would allow easier implementations since most languages have some form of understanding JSON. Compression is not needed but we might simply add another port that compresses all that goes through it (think OutputStream Chaining in Java). The size of the messages will double, yes, but as far as I'm concerned that's the only downside to using JSON.

Want to see what developers are chatting about? http://bitcoinstats.com/irc/bitcoin-dev/logs/
Bitcoin-OTC Rating
m0mchil
Full Member
***
Offline Offline

Activity: 171


View Profile
February 17, 2011, 01:46:31 PM
 #24

Or perhaps we should use something like protocol buffers, even for core bitcoin messages?

Cdecker
Hero Member
*****
Offline Offline

Activity: 487



View Profile WWW
February 17, 2011, 02:39:00 PM
 #25

Or perhaps we should use something like protocol buffers, even for core bitcoin messages?
You mean the P2P protocol? I'd already be happy using JSON, but it won't work because the whole block hashing and transaction hashing relies on the binary representation of the message itself...

Want to see what developers are chatting about? http://bitcoinstats.com/irc/bitcoin-dev/logs/
Bitcoin-OTC Rating
slush
Legendary
*
Offline Offline

Activity: 1358



View Profile WWW
February 17, 2011, 05:10:43 PM
 #26

Or perhaps we should use something like protocol buffers, even for core bitcoin messages?

+1 for use of standardized binary protocol everywhere.

Short summary:

JSON-RPC (over TCP):
+ Already used in bitcoin
+ Standard protocol
+ Extremely easy to use
- Missing support for binary data
- General overhead (repeating of data field names)

Protocol buffers:
+ Easy to use
+ Standard protocol
+ Support for binary data
+ Bandwidth effective
+ May replace current binary protocol on P2P side
+ No external dependency needed (protocol compiler produces plain C++ code)

Proprietary binary protocol:
+ Extremely bandwidth effective
+ No external dependency needed
+ Support for binary data (noticed for integrity of this summary)
- Hard to learn & use (everybody have to implement support in his preferred language)
- Not reusable (P2P and push interface needs two different protocols)

From this summary, I prefer solutions in this order: 1. Protocol buffers 2. JSON 3. Binary protocol

adv
Full Member
***
Offline Offline

Activity: 169


View Profile
February 17, 2011, 10:08:24 PM
 #27

From this summary, I prefer solutions in this order: 1. Protocol buffers 2. JSON 3. Binary protocol

We talk only about mining. The only significant thing for him - speed. No need of readability or understandability. You can use a heavy program optimization, go-to operator, or even assembler. : ^)
So please forget about JSON-RPC. If he so much needed to someone - it is always possible to use a standard getwork-interface over http. And the creation of a push-protocol over JSON -- a completely separate problem with its advantages and disadvantages. I think the choice is between only protocol buffers and pure binary protocol.

U may thank me here: 14Js1ng1SvYBPgUJnjNAEPYH4d6SHF79UF
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1470


View Profile
February 17, 2011, 10:15:49 PM
 #28

Protocol buffers for the P2P networking protocol have already been discussed.  Not really realistic now, but "it would have been nice."

For the purposes of this thread, protocol buffers have three disadvantages:
  • Duplicates data structure layout of a C/C++ data structure.  You wind up encoding C/C++ data structure -> another C/C++ data structure.  Python hides this from the programmer, but it is the same net result.
  • Less flexible than JSON in describing dynamic, multi-dimensional data structures, because the structure of every data field is compiled into the application.  Simple JSON is much more flexible, and data structures may be changed at will.
  • Like current JSON, requires a native -> encoded -> native encoding for each miner client

Basically, any packetized protocol -- let us say JSON-over-TCP (rather than JSON-over-HTTP) -- requires some amount of low-level raw binary parsing to locate message boundaries.  To simply pass through data unchanged, rather than decoding compressed JSON, comes for free in such protocols.

Therefore, logically, any JSON-over-TCP protocol already has the ability to skip the pointless native->encoded->native step that each miner client performs, by using this pass-through method.  Passing raw binary work directly from bitcoind to the client eliminates the possibility of bugs and problems related to the unnecessary encoding+decoding an additional JSON step would require.

Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
Cdecker
Hero Member
*****
Offline Offline

Activity: 487



View Profile WWW
February 17, 2011, 11:27:18 PM
 #29

Changing the P2P protocol is not going to happen anytime soon for the above mentioned reasons (although I would welcome it)  because it would require a transition phase for the entire network to switch. I just mentioned it because someone asked me about it.

For the pushwork patch i think you are forgetting that only because you decide to use one over the other for speed reasons doesn't do anything. You have to make the format easy to implement since the miners do have to support it. The getwork interface must be enabled anyway so if the coders of the miners don't adopt the protocol you'll not have changed anything,  except a lot of work spent in a useless interface.

Want to see what developers are chatting about? http://bitcoinstats.com/irc/bitcoin-dev/logs/
Bitcoin-OTC Rating
adv
Full Member
***
Offline Offline

Activity: 169


View Profile
February 18, 2011, 09:59:41 PM
 #30

Changing the P2P protocol is not going to happen anytime soon for the above mentioned reasons (although I would welcome it)  because it would require a transition phase for the entire network to switch.

As I understand it, you're talking about using a binary protocol in bitсoin client. I (think Jgarzik too) also have in mind is the only mining protocol and hope that the miner would soon be removed from the official client. And, yes: JSON-RPC getwork protocol should remain in the official client. (May be need to separate the getwork-request from wallet-management commands for the possibility of mining on untrusted hosts).

I think the binary protocol should be used only in separate from a client miners through some JSON-binary proxy, or through connecting to the official client plugin/module, or in extreme cases can be easily separated part of the code in the official program.

If you think that the use of JSON-binary proxy is useless because of the overhead JSON, who do not disappear, there are two advantages:
1. JSON binary proxy can stand on the border of fast and low-band networks, using JSON in fast and binary protocol in the slow.
2. After receiving one getwork through JSON proxy can divide (cut nonсe) it for a few miners through binary protocol. It must shows good results: http://bitcointalk.org/index.php?topic=3486.msg48996#msg48996
Perhaps we should think about the possibility of receiving getwork bundles trough JSON.

U may thank me here: 14Js1ng1SvYBPgUJnjNAEPYH4d6SHF79UF
martok
Full Member
***
Offline Offline

Activity: 140


View Profile
April 03, 2011, 10:21:28 PM
 #31

I know it isn't the object of pushpool's purpose but I think long polling might be broken in the current implementation. poclbm occasionally throws a stale so I used curl to test /LP and it never returns even when the block changes.
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1470


View Profile
April 03, 2011, 11:09:14 PM
 #32

I know it isn't the object of pushpool's purpose but I think long polling might be broken in the current implementation. poclbm occasionally throws a stale so I used curl to test /LP and it never returns even when the block changes.

Do you send SIGUSR1 to pushpoold, when a new block arrives?

pushpoold assumes you will implement your own block monitoring solution, and notify it when a new block arrives.


Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
martok
Full Member
***
Offline Offline

Activity: 140


View Profile
April 04, 2011, 01:16:16 AM
 #33

Ah thanks didn't know about that bit.
martok
Full Member
***
Offline Offline

Activity: 140


View Profile
April 04, 2011, 11:07:14 PM
 #34

Hi again,
Ok I had to fix LP in pushpoold but have it working now. Are you maintaining this on github? If so, I can feed you patches.
Basically:
sigusr1 is currently set to dump stats. It sets to lp_flush but then sets to dump stats later on so that needed fixing.
Also, lp was returning bad request on a regular GET request IE without a JSON payload. I hacked together a solution so calling /LP returns a GETWORK as specified in the spec I read.
I also have hacked sharelog to insert into the SQLITE3 db rather than using a logfile but that's strictly a matter of preference. I suppose the logfile is just as flexible as it could be a FIFO having the other end be a program that does whatever (postgresql/mysql).

Unfortunately, Python isn't my language or I'd have a go at the binary protocol implementation. Thought of rewriting the miner in C though if only to have it support things like --device=1,2,3,4 to run all of the miner threads in a single application instance.
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1470


View Profile
April 04, 2011, 11:19:09 PM
 #35

Ok I had to fix LP in pushpoold but have it working now. Are you maintaining this on github? If so, I can feed you patches.

For the moment, email to jgarzik@exmulti.com

Quote
sigusr1 is currently set to dump stats. It sets to lp_flush but then sets to dump stats later on so that needed fixing.

Already fixed in local git

Quote
Also, lp was returning bad request on a regular GET request IE without a JSON payload. I hacked together a solution so calling /LP returns a GETWORK as specified in the spec I read.

A GET request without a JSON payload is a bad request Smiley  What is your preferred behavior?


Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
martok
Full Member
***
Offline Offline

Activity: 140


View Profile
April 05, 2011, 12:41:57 AM
 #36

A GET request without a JSON payload is a bad request Smiley  What is your preferred behavior?

Hmm, I am going by the spec here:
http://deepbit.net/longpolling.php
3) Miner starts a request to long polling URL with GET method and same basic authorization as on main connection.
This request is not answered by server until new block is found by bitcoin network. The answer is the same as getwork on the main connection.
So by that reading, there is no need for an actual GETWORK JSON payload on the long poll URL as the response is assumed to be a getwork. Looking at poclbm, it's just sending get without any post data. Is this wrong behaviour?
martok
Full Member
***
Offline Offline

Activity: 140


View Profile
April 07, 2011, 12:01:06 AM
 #37

Hi,

Occasionally, poclbm is throwing the following when connected to pushpoold:
Traceback (most recent call last):
  File "/home/bitcoin/src/poclbm/BitcoinMiner.py", line 141, in mine
    self.sendResult(self.resultQueue.get(False))
  File "/home/bitcoin/src/poclbm/BitcoinMiner.py", line 166, in sendResult
    d = result['work']['data']
TypeError: 'NoneType' object is unsubscriptable
Traceback (most recent call last):
  File "/home/bitcoin/src/poclbm/BitcoinMiner.py", line 141, in mine
    self.sendResult(self.resultQueue.get(False))
  File "/home/bitcoin/src/poclbm/BitcoinMiner.py", line 166, in sendResult
    d = result['work']['data']
TypeError: 'NoneType' object is unsubscriptable

But poclbm works fine via local bitcoind or via the commercial pools. Have you encountered this yet.
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1470


View Profile
April 07, 2011, 08:13:21 AM
 #38

Occasionally, poclbm is throwing the following when connected to pushpoold:
Traceback (most recent call last):
  File "/home/bitcoin/src/poclbm/BitcoinMiner.py", line 141, in mine
    self.sendResult(self.resultQueue.get(False))
  File "/home/bitcoin/src/poclbm/BitcoinMiner.py", line 166, in sendResult
    d = result['work']['data']
TypeError: 'NoneType' object is unsubscriptable
Traceback (most recent call last):
  File "/home/bitcoin/src/poclbm/BitcoinMiner.py", line 141, in mine
    self.sendResult(self.resultQueue.get(False))
  File "/home/bitcoin/src/poclbm/BitcoinMiner.py", line 166, in sendResult
    d = result['work']['data']
TypeError: 'NoneType' object is unsubscriptable

But poclbm works fine via local bitcoind or via the commercial pools. Have you encountered this yet.

What is occasionally?  Once a day?


Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1470


View Profile
April 07, 2011, 08:13:54 AM
 #39

pushpool git repo is now live at https://github.com/jgarzik/pushpool

Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
xf2_org
Member
**
Offline Offline

Activity: 70


View Profile
April 21, 2011, 06:30:38 AM
 #40

Version 0.3 released, with support for mysql, postgres and sqlite databases, for both share logging and password checking.

Pages: « 1 [2] 3 »  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!