Bitcoin Forum

Bitcoin => Development & Technical Discussion => Topic started by: slush on December 27, 2011, 10:08:22 PM



Title: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on December 27, 2011, 10:08:22 PM
Few days ago I started the discussion in Electrum client thread (https://bitcointalk.org/index.php?topic=50936.msg654787#msg654787) about new protocol and server implementation of overlay network over Bitcoin p2p network. I think that discussion is not anymore relevant to Electrum client itself, so I started this new thread dedicated to my proposal. Please comment my proposal, ask to anything or suggest any improvement if you want.

Network protocol proposal (in progress): https://docs.google.com/document/d/17zHy1SUlhgtCMbypO8cHgpWH73V5iUQKk_0rWvMqSNs/edit?hl=en_US

Server implementation (in progress): https://gitorious.org/stratum/

Running nodes:
  • london.stratum.bitcoin.cz (TCP - 3333, HTTP - 8001), provider: Linode.com
  • chicago.stratum.bitcoin.cz (TCP - 3333, HTTP - 8000), provider: BitVps.com (thanks!)

Short quote from google document, describing main purpose of such overlay network. (current codename is "Electrum platform", but I'm searching better name ATM: edit: "Stratum" won the contest for network name.

Quote
Electrum platform is a framework for creating lightweight Bitcoin clients (= clients without a blockchain, keeping only private keys). Absence of blockchain on the client side provide very good user experience, client has very small footprint and still provide high level of security and privacy.

Basically Electrum is overlay network on the top of Bitcoin P2P protocol, creating simplified facade for lightweight clients and hiding unnecessary complexity of decentralized protocol. However there’s much bigger potential in this overlay network than just providing simplified API for accessing blockchain stored on Electrum servers. For utilization of such potential, we definitely need some robust protocol providing enough flexibility for various type of Electrum clients and their purposes.

Some advanced ideas for Electrum network, which will need flexible network protocol:
* Integration of BTC/fiat exchanges into clients
* Wallet storages for diskless or extremely low-resource clients (AVR-based hardware wallets)
* Server-side escrows (sending bitcoins to email)
* Integration of bitcoin laundry
* Exchange calculators (for providing “fiat” equivalents of BTC in clients)
* Firstbits support
* Mining support for clients
* Various transport protocols (especially HTTP Push, which allows PHP websites to integrate with Bitcoin easily)

Requirements to protocol:

Quote
* Protocol should be as simple as possible. Some clients aren’t capable to handle high-level message systems like Google’s Protocol buffers or Apache Thrift.
* Protocol should be text-based. Some languages cannot handle binary data (javascript) or it’s pretty difficult to implement it correctly (PHP).
* Protocol must support standard RPC mechanism (request-response). Request should contains method identifier and parameters, response should be able to transfer error states/exceptions.
* Mapping between request and response must be clear. Don’t allow vague relation between request and response. Some message-based systems use only textual conventions to map requests and responses together, like ‘firstbits_resolve <firstbits>’ expects ‘firstbits_response <firstbits> <address>’ or ‘firstbits_error <firstbits> <reason>’. It creates ambiguous data flow, avoid it.
* Protocol should support publish-subscribe mechanism. Client can subscribe on server for receiving some kind of information. After this request, server will proactively broadcast messages to subscribed clients until client disconnects or cancel it’s subscription.
* Protocol must be bi-directional. In the opposite of standard client-server model, we sometimes need to allow server to initiate the communication. As an example, server can ask client to reconnect to another node (before switching to maintenance mode) or send some text message to user.


Title: Re: [proposal] Overlay network over Bitcoin
Post by: MoonShadow on December 27, 2011, 10:13:41 PM
I think that it was just a matter of time before a parrallel/sidechannel network was developed.


Title: Re: [proposal] Overlay network over Bitcoin
Post by: casascius on December 27, 2011, 10:22:21 PM
Yeah, I thought this would be a good idea as well.

My independent imagination of this concept was that of a "supernode" - one that could be depended upon to seed the block chain, replace IRC as the peer-finding mechanism, provide exchange rates, and (in combination with an idea that I had recently published on another thread) sign blocks and provide hint information as to how to resolve major forks in the chain if they ever came up.  Such a service could either be free, or could be a paid model (where whoever's running it has an incentive to do a good honest job because they're able to make a business out of it).

Supernodes could also provide constant confirmation that all supernodes are in contact with one another (e.g. by each of them signing the blocks they receive, exchanging the signatures, and sharing them with their clients).  They would be a valuable resource in the event of a 51% attack.  In the event of a real attack on the main chain, the operators of supernodes can manually checkpoint the honest chain.  Then, they would invite everyone else to connect directly to them (using the -connect command line argument) and effectively censor out the attacker's chain, keeping the bitcoin network functional while the developers rushed to come up with a technical solution that allowed automatic rejection of the attacker's chain.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: Red Emerald on December 27, 2011, 10:28:25 PM
Following.

I am very interested in Electrum/thin clients and have the bandwidth/hardware to run a supernode so am willing to test.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: slush on December 27, 2011, 10:31:36 PM
I think that it was just a matter of time before a parrallel/sidechannel network was developed.

Well, the purpose is only provide bitcoin-related services for any possible device or application, where keeping blockchain locally is overkill. I definitely don't want to develop anything which is replacing bitcoin. I just see the concept of various transports (TCP socket, HTTP poll, HTTP push, even email interface!) very flexible.

Some people are asking me why I don't just extend Bitcoin protocol. I think both things are solving another problems. Bitcoin P2P network is distributed database, it's storing blockchain and handling transactions. My proposal is more about services on top of this database and I don't think they can be provided purely thru p2p network. Also, I don't think that bitcoin p2p network should care about such "services" like distributing USD/BTC price or similar.



Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: Red Emerald on December 27, 2011, 10:34:59 PM
Some people are asking me why I don't just extend Bitcoin protocol. I think both things are solving another problems. Bitcoin P2P network is distributed database, it's storing blockchain and handling transactions. My proposal is more about services on top of this database and I don't think they can be provided purely thru p2p network. Also, I don't think that bitcoin p2p network should care about such "services" like distributing USD/BTC price or similar.

I fully agree with you. I think building services on top of bitcoin is much smarter than changing the bitcoin protocol.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: slush on December 27, 2011, 10:36:57 PM
Yeah, I thought this would be a good idea as well.

My independent imagination of this concept was that of a "supernode"

Grid servers of such overlay network will need full blockchain, so yes, they turn into some kind of "supernodes". Using lightweight clients for normal people is very rational choice, because those end users don't need to care about bandwidth, CPU usage and they don't need to wait for blockchain sync if they're using Bitcoin once per week.

Although I'm proposing "some kind of supernodes", I don't like centralizing of blockchain, I think it' real threat for Bitcoin. That's the reason why I keep running full bitcoin node on all computers where I can.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: casascius on December 27, 2011, 11:25:15 PM
I fully agree with you. I think building services on top of bitcoin is much smarter than changing the bitcoin protocol.

I don't think the core bitcoin protocol would need to be changed.  It would be replaced entirely, for clients designed to use another protocol.

Reasons for a new protocol might include:
  • supporting clients which act as full nodes, but which have never seen the full blockchain (working with a pruned version)... the current protocol assumes all nodes have the full block chain... there's no language for saying "I have a pruned block", or for reconciling what on a block should and should not be pruned
  • passing around metadata that doesn't belong in the block chain, but nevertheless is useful against 51% attacks (e.g. signed messages acknowledging who has seen what blocks)
  • supporting queries against nodes with the full block chain, for the benefit of light nodes
  • some sort of mutual authentication where necessary (for example, if I'm a light node, I'd prefer to query a node I trust, rather than a random one, and would prefer that its response to my query be digitally signed... likewise, that node might charge for this service, and might want to know who I am as well)
  • possibly, a service that could handle multiple cryptocurrencies or alt chains (not that I'd be a fan of this, but it's a legitimate feature that may prompt someone to do a new protocol)
  • a mechanism for clients to only ask about messages that pertain to them, to cut resource costs (e.g. client asks, "only tell me about transactions involving <list of addresses>" and/or "only tell me about new blocks".

Presumably, any such protocol would be created without the support of the main developers, and probably would appear out of the results of efforts to make a new client where the current protocol was seen as constraining, -or- where there would be some advantage to having this new client talk to other nodes of that same client (presumably because satoshi client won't relay its custom messages).


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: slush on December 28, 2011, 12:00:01 AM
molecular, you probably wanted to post it there: https://bitcointalk.org/index.php?topic=55822 :-)


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: finway on December 28, 2011, 01:54:13 AM
suggestion for network name:

electricity
funny  ;D


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: 2112 on December 28, 2011, 02:55:53 AM
Hi slush (and other readers)!

I reread your proposal (both here and in Google docs). In my opinion you are proposing either (1) internally inconsistent protocol or (2) family of incompatible protocols supported by a hope of sharing some implementation details.

Second issue is that I'm NOT proposing RPC over HTTP(S).

I'll need to add some custom way how to support HTTP poll or HTTP push, because I *want* to provide services for this type of clients.

because json encoder/decoder or curl/ajax is available everywhere in standard libraries

The resultant Frankenstein monster (or family of monsters) is going to try to strangle its creator.

It is hard to argue with what you proposing point by point because of the contradictory nature of the requirements. I'm going to group my response into the following points.

1) Protocol can be either RPC or bidirectional, but not both. The RPC paradigm (request-response, master-slave) is mutually contradictory with a pair of peers exchanging asynchronous messages. Bitcoin protocol itself is from its origin asynchronous and cannot be squeezed into the master-slave architecture. It demands the bi-directional asynchronous protocol with finite state machines (FSMs) on both ends communicating with a two flows of sequenced frames. The successful design will look very much like TCP on top of IP frames.

2) Your target market (low-end consumer-level devices) demands that there's checksuming and framing at the application layer. Precisely because cheap NAT gateways and cheap DSL/Cable/WLAN modems are known to mangle the transport-level frames due to bugs (in the implementation of NAT) and excessive buffering (to improve one-way file transfer benchmarks).
If you think you can add CRC later you are going to loose by not detecting corruptions early.

3) In my experience JSON is probably the close to being the least resilient encoding possible. I can't disclose proprietary data, but I have over a decade's worth of reliability statistics for various  remote services (RPC-like) that are sold by organizations I consulted. But the rough ranking is
as follows: (from the least errors to the most)

3.1) ultra-lean character-based protocol similar to FIX, designed originally for MNP5 and V.42bis modems, currently used through a simple TCP/IP socket
3.2) SOAP (RPC over XML) with Content-Length, Content-MD5 and DTD verification enabled
3.3) SOAP and plain XML-RPC without the above strenghtening options
3.4) JSON-RPC
3.5) RPC over e-mail with various human-readable encodings

All the above are primarily sold to small businesses and individual traveling salespeople, which seems to be the target market you are planning to serve. We also have enterprise-oriented services using various MQ and *-remoting technologies, but the services offered aren't directly comparable. I thing that JSON could be an acceptable choice if you from the start demand the strengthening by some form of Content-Length and Content-Checksum. JSON is also infamous for letting people easily make byte-endianness mistakes, very much like current "getwork" which is neither big-endian not little-endian. Yes, JSON saves a lot of bandwidth when compared to XML. But I know of no presently available implementation that isn't producing cryptic and hard to troubleshoot failure modes. It is a classic example of a "lean but very mean" design.

4) You somehow read the my earlier suggestions about IPsec imply high-end large-volume target market. The reality is quite opposite: Windows support IPsec since 2000, Linux for a long time, Netgear ProSafe family has several models in $100-$200 range, L2TP and PPTP are available for free on all iPhones,Blackberries,Androids; many Nokias and other smartphones. The real hindrance is the HTTP(S)-uber-alles mindset, not the actual difficulty and cost of the implementation.

In summary I'd like to say that you wrote a very interesting and thought provoking proposal. I just think that the range of the targets you are hoping to cover is way too broad ($3 AVR processors, shared hosting plans, home computers, etc.).

I have my personal litmus test for the technological implementation in the Bitcoin domain: it has to support (and be tested for) chain reorganization. Preferably it should correctly retry the transactions upon the reorg. Absolute minimal implementation should correctly shut down with a clear error message and a defined way to restart the operations. Any implementation that will behave like Tom Williams' Mybitcoin.org blame-it-on-the-chain-reorg is a failure. Thus far your proposals don't seem to encompass this essential feature of Bitcoin. If Electrum (and associated protocols) aren't going to behave properly in the presence of chain reorganization events then I withdraw my earlier opinion that Electrum has more potential than the Satoshi client.

Thank you again for your time and attention.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: genjix on December 28, 2011, 03:10:46 AM
The thing is not to get too caught up writing long proposals and documents. Think a little bit, code up an implementation, get rolling and refine your requirements as they become obvious. It's easier to ask for forgiveness than to ask for permission.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: 2112 on December 28, 2011, 04:25:23 AM
The thing is not to get too caught up writing long proposals and documents. Think a little bit, code up an implementation, get rolling and refine your requirements as they become obvious. It's easier to ask for forgiveness than to ask for permission.
You have a fair point. But I'm extremely skeptical as to overall software architectural skills in the broad Bitcoin community. My absolute favorite is the inversion of control issue in Satoshi client exhibited by ThreadSafeAskFee(). Such a textbook case of an anti-pattern.

https://bitcointalk.org/index.php?topic=44330.msg538044#msg538044

The rationalization I heard why this "isn't a problem" or the workarounds proposed are quite enlightening and entertaining. Too bad that moderators deleted several flaming comments in that thread.



Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: Red Emerald on December 28, 2011, 04:32:27 AM
The thing is not to get too caught up writing long proposals and documents. Think a little bit, code up an implementation, get rolling and refine your requirements as they become obvious. It's easier to ask for forgiveness than to ask for permission.
I disagree.  I think planning ahead always leads to better software even if the software development may be slower to start.  Better to find a problem in your proposal and rewrite that than to discover a problem in your code and redo a ton of work there IMO

Proposals are especially important when multiple people are working on a project as it makes it easier to work on different parts of the project at once.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: genjix on December 28, 2011, 06:58:27 AM
The thing is not to get too caught up writing long proposals and documents. Think a little bit, code up an implementation, get rolling and refine your requirements as they become obvious. It's easier to ask for forgiveness than to ask for permission.
You have a fair point. But I'm extremely skeptical as to overall software architectural skills in the broad Bitcoin community. My absolute favorite is the inversion of control issue in Satoshi client exhibited by ThreadSafeAskFee(). Such a textbook case of an anti-pattern.

https://bitcointalk.org/index.php?topic=44330.msg538044#msg538044

The rationalization I heard why this "isn't a problem" or the workarounds proposed are quite enlightening and entertaining. Too bad that moderators deleted several flaming comments in that thread.



The bitcoin codebase is terrible architecturally.

You may be interested to review my codebase which is an asynchronous bitcoin toolkit library using proactors. Completion handlers using std::bind and std::function are passed to operations.

https://bitcointalk.org/index.php?topic=30646.0

Also a video on the design principles behind the API:

http://www.youtube.com/watch?v=zyKcSpx5xDg


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: genjix on December 28, 2011, 07:00:08 AM
The thing is not to get too caught up writing long proposals and documents. Think a little bit, code up an implementation, get rolling and refine your requirements as they become obvious. It's easier to ask for forgiveness than to ask for permission.
I disagree.  I think planning ahead always leads to better software even if the software development may be slower to start.  Better to find a problem in your proposal and rewrite that than to discover a problem in your code and redo a ton of work there IMO

Proposals are especially important when multiple people are working on a project as it makes it easier to work on different parts of the project at once.

You're advocating the ancient waterfall method which is basically out of fashion since the last decade.

Quote
genjix:~$ python
Python 2.7.2+ (default, Oct  4 2011, 20:03:08)
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.

If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: Red Emerald on December 28, 2011, 07:48:29 AM
The thing is not to get too caught up writing long proposals and documents. Think a little bit, code up an implementation, get rolling and refine your requirements as they become obvious. It's easier to ask for forgiveness than to ask for permission.
I disagree.  I think planning ahead always leads to better software even if the software development may be slower to start.  Better to find a problem in your proposal and rewrite that than to discover a problem in your code and redo a ton of work there IMO

Proposals are especially important when multiple people are working on a project as it makes it easier to work on different parts of the project at once.

You're advocating the ancient waterfall method which is basically out of fashion since the last decade.

Quote
genjix:~$ python
Python 2.7.2+ (default, Oct  4 2011, 20:03:08)
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.

If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

You are emphasizing the first bolded point while I am emphasizing the second.  I'm not trying to say, "don't start ANY code until you have the proposal 100% complete and vetted by every major dev." I'm saying that we should have a basic roadmap worked out about what we are coding so that we actually know where are going.

And I most definitely stand behind my second point.  

Quote
Proposals are especially important when multiple people are working on a project as it makes it easier to work on different parts of the project at once.
How are multiple people supposed to work on something when no one knows were we are trying to go?

EDIT: reread your post.  I had missed the "code up an implementation" and thought you wanted to code with no roadmap.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: ThomasV on December 28, 2011, 09:20:36 AM
The thing is not to get too caught up writing long proposals and documents. Think a little bit, code up an implementation, get rolling and refine your requirements as they become obvious. It's easier to ask for forgiveness than to ask for permission.

I agree with you. These things are a bit obvious, but it is sometimes worthwile to remind obvious things. :-)
A protocol should result from an iterative process, back and forth between specifications and the needs that we discover during implementation. It should not be set in stone too early, especially when the number of people involved is tiny.

Things I would like to see in the protocol:

* use JSON RPC
* the set of functions that are currently used by the Electrum client and server (address based functions): the server does not know the set of addresses in a client's wallet, it just sends address histories and broadcasts transactions. This means that a client should be able to use several servers simultaneously for improved anonymity.
* wallet-based functions (similar to BCCAPI) for ultra-thin clients:  The server knows the public key used to generate the sequence of addresses in a type 2 wallet. It sends the balance and history of the wallet. It also sends the number of addresses detected in the wallet (gap based detection; note that this differs from BCCAPI, because the BCCAPI server needs to store your number of keys in its database). The server also sends unsigned transactions to the client, and the client signs them.
* also, I would like to see something similar to Transaction Radar: http://www.transactionradar.com/ : when a transaction is unconfirmed, the client should display its rate of propagation. Electrum servers could be part of the existing transaction radar service.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: Grami on December 28, 2011, 01:57:12 PM
* Protocol must be bi-directional. In the opposite of standard client-server model, we sometimes need to allow server to initiate the communication. As an example, server can ask client to reconnect to another node (before switching to maintenance mode) or send some text message to user.

Use WebSocket as transport. It is bi-derectional and simple enough.
http://en.wikipedia.org/wiki/WebSocket


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: Grami on December 28, 2011, 01:58:17 PM
3) In my experience JSON is probably the close to being the least resilient encoding possible.

+1


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: kjj on December 28, 2011, 03:34:52 PM
* Protocol must be bi-directional. In the opposite of standard client-server model, we sometimes need to allow server to initiate the communication. As an example, server can ask client to reconnect to another node (before switching to maintenance mode) or send some text message to user.

Use WebSocket as transport. It is bi-derectional and simple enough.
http://en.wikipedia.org/wiki/WebSocket

Eww.  Websockets are only useful when you have no other way to start a connection.  After that, it is just an obfuscated TCP socket.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: Grami on December 28, 2011, 05:50:19 PM

Eww.  Websockets are only useful when you have no other way to start a connection.  After that, it is just an obfuscated TCP socket.

Websockets is much simpler in use. With TCP socket you have to aware of many useless things (buffer size, tcp error handling etc).


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: 2112 on December 28, 2011, 06:21:13 PM
Websockets is much simpler in use. With TCP socket you have to aware of many useless things (buffer size, tcp error handling etc).
All protocols are equal, but some protocols are more equal than others.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: ripper234 on December 28, 2011, 11:12:59 PM
Great project, I'm glad to see some commits already at Gitorious.

Just my two cents:
1. Like some others said, develop a prototype, launch early, and iterate.

2. Should not require sending private keys (haven't read the protocol in detail yet, but just in case this was not planned). Instead use some form of Offline Transactions.

3. Should consider DDOS protection. If this will be The Simple Network over Bitcoin, that will eventually handle huge traffic, you might want to consider charging a bit per API call ... just something minimal to deter DDOS. People can register their API keys, charge their account with X BTC, and get an allowance for Y API calls.

4. Should consider malicious servers. If someone evil runs an evil copy of this server, and clients connect to it ... how exactly can the evil server damage them?

Can EvilServer steal money? (Probablly not if you're just using Offline TX as the only mutator operation)
Does it have a strong incentive to lie about GET queries?
How about delaying transfers?
Would clients eventually talk to multiple overlay servers, and validate the results between them?
 - If only 2 out of 3 trusted servers tell me something, odds are the 3rd one is lying
 - I would be able to submit a mutator API call (e.g. processTX) to multiple servers ... and may the fastest one win.

These are just some things to think about ... you don't need all the answers in advance. I actually started developing a small Bitcoin webapp today, and got stuck when I realized I'd have to maintain the entire blockchain (or even just the headers). If a prototype of this project comes out early enough (~ 1 month), I'll wait for it instead of writing and debugging blockchain-maintaining code.

Oh, and a definite +1 to JSON. The protocol should appeal to the most common denominator ... which is JSON.

Good luck!


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: ripper234 on December 28, 2011, 11:38:50 PM
BTW, what is the relation between this overlay network and BitcoinJS Exit Nodes (https://github.com/bitcoinjs/node-bitcoin-exit/wiki)?

Are they implementations of the same idea? Are there substantial differences between the projects?


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: eldentyrell on December 29, 2011, 01:04:48 AM
This is cool, but I couldn't find any mention of the trust model anywhere in this thread or the design docs.

If you keep a copy of the blockchain, trust is simple: you trust the longest chain.

If you aren't keeping a copy of the blockchain, you need to find another answer for this question.

I'm not saying it's impossible, though I am suggesting that it might be the most difficult part of this project (certainly more difficult than choosing wire protocol formats!).  I'm a bit worried that it isn't being addressed.  Or maybe I just missed something.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: Red Emerald on December 29, 2011, 02:28:41 AM
This is cool, but I couldn't find any mention of the trust model anywhere in this thread or the design docs.

If you keep a copy of the blockchain, trust is simple: you trust the longest chain.

If you aren't keeping a copy of the blockchain, you need to find another answer for this question.

I'm not saying it's impossible, though I am suggesting that it might be the most difficult part of this project (certainly more difficult than choosing wire protocol formats!).  I'm a bit worried that it isn't being addressed.  Or maybe I just missed something.

There is a server that sits on top of a patched bitcoind which of course has the blockchain.

The thin client simply uses this protocol to ask the server the balance or to sign transactions relay signed transactions or w/e.  I don't see anything that needs to be addressed.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: eldentyrell on December 29, 2011, 03:14:23 AM
The thin client simply uses this protocol to ask the server the balance or to sign transactions or w/e.

Why do you trust "the server" (and the person running it, who I assume is not always yourself)?

How do you know you're even talking to "the server" you think you're talking to (no man-in-the-middle)?


I don't see anything that needs to be addressed.

These are basic, fundamental security questions that have been widely known since before SSL came into use.  If the designers of this new protocol don't think they need to be addressed, that concerns me.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: ripper234 on December 29, 2011, 07:11:04 AM
The thin client simply uses this protocol to ask the server the balance or to sign transactions or w/e.

Why do you trust "the server" (and the person running it, who I assume is not always yourself)?

How do you know you're even talking to "the server" you think you're talking to (no man-in-the-middle)?


I don't see anything that needs to be addressed.

These are basic, fundamental security questions that have been widely known since before SSL came into use.  If the designers of this new protocol don't think they need to be addressed, that concerns me.

If the protocol is done right, zero trust is required, because a malicious server can't really do much.
Offline Transactions ftw.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: Red Emerald on December 29, 2011, 07:25:49 AM
Why do you trust "the server" (and the person running it, who I assume is not always yourself)?

How do you know you're even talking to "the server" you think you're talking to (no man-in-the-middle)?
I am pretty sure that all client-server models require trust that the server is not lying. I can't think of any that don't.  However, there aren't too many things that a malicious server can do.

Why do you trust all of the nodes you connect to?  I'm pretty sure that most of the same protections to make sure peers aren't lying to you are available to a thin client.

For example: A transaction's input contains a signature that can't be faked (if they can, we are all f***ed, thin client or not).  If a malicious server gave you a fake input, a simple check would show you it is fake.

Since you generate and sign outgoing transactions yourself, there is no chance of the malicious server redirecting your coins to another address unless maybe that address is retrieved from a malicious firstbits resolve where they happen to have a collision.  This is a potential problem with firstbits in general and not specifically this protocol. http://firstbits.com/ could make addresses for the most commonly requested addresses and return those instead of the real ones (if they wanted to be thieves) or if the firstbits were short enough, they could generate a collision on the fly.  Not sure what we could do to prevent this problem besides letting people know that for utmost security they need the full address.

One thing I could see a server doing that would be harmful is not relaying it's clients transactions.  Right now, thin clients only talk to one server.  This wouldn't be too hard to detect as the person you are sending to would probably mention that it hasn't been received.  People are also talking about getting thin clients to communicate with multiple servers which would remove this problem.  I think transaction broadcast services will be important for anonymity given some of the IP tracing I've seen.

Another problem might be if the server does a double spend against one of it's clients as they could be kept unaware of the real transactions on the network.  This problem is also removed once the client can talk to multiple servers.

Quote
These are basic, fundamental security questions that have been widely known since before SSL came into use.  If the designers of this new protocol don't think they need to be addressed, that concerns me.

Run the protocol over SSL and you don't need to worry about MITM. Then, only a malicious server is the problem.

If we have a network of servers of running this protocol, some of the problems I mentioned above (like holding transactions) would require a 100% attack (which is likely much harder than a 51% attack :) ).  I'm not sure how we should handle what happens when a malicious server is detected.  That definitely needs to be figured out soon.

I imagine slush can give a better defense of this proposal as he seems to always write what I am thinking way better than I seem to be able.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: eldentyrell on December 29, 2011, 07:50:06 AM
If the protocol is done right, zero trust is required

Just saying that does not make it true.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: ripper234 on December 29, 2011, 07:52:58 AM
If the protocol is done right, zero trust is required

Just saying that does not make it true.

Please describe a possible attack vector.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: MoonShadow on December 29, 2011, 07:57:15 AM
Trust can be developed by a thin client the same way that trust in peer nodes is developed in the bitcoin protocol.  By polling randomly among a set of reachable nodes and comparing their responses.  If any response doesn't match, dump all those nodes and try again at random.  The odds that more than one random server on the Internet is trying to screw you, assuming that you are not caught into a honeypot local internet, is remote.  Moreso considering that it's largely impossible for those random servers to know if you are capable of verifying such data or not.

As a thin client, you are also not trusting the server with any critical data that cannot be corrected later.  You are not offering it access to your private keys, for example; just polling it for otherwise public data available on the blockchain, namely the balance of certain addresses and copies of certain confirmed transactions.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: eldentyrell on December 29, 2011, 07:59:18 AM
However, there aren't too many things that a malicious server can do.

!!!Dangerous misconception!!!

Satoshi invented blockchains for a reason: they prevent double-spends.  If you don't have a copy of the block chain, anybody running the server you're talking to can "send" you coins and then take them back with total impunity, simply by making sure that a transaction sending those coins to themselves is "mined in" to the blockchain before they release the transaction that looks like it's sending them to you.

This problem is also removed once the client can talk to multiple servers.

This is dangerous "cargo cult (http://en.wikipedia.org/wiki/Cargo_cult_science) security".  It does nothing to prevent man-in-the-middle attacks (my ISP attacks me) and merely shifts the problem elsewhere.  What prevents somebody from renting 100 cheap VPSes and running the server software on all of them?  Worse: botnet operators.  See also Sybil attacks (http://en.wikipedia.org/wiki/Sybil_attack).  Basing security on 51% hashpower is reasonable; basing it on 51%-of-IP-addresses-running-the-software is definitely not.  I can't think of anything more attractive to botnet operators!

I imagine slush can give a better defense of this proposal as he seems to always write what I am thinking way better than I seem to be able.

With all due respect to slush, he shouldn't provide a "defense"; he should address these issues in the design document by explicitly stating the trust model.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: Red Emerald on December 29, 2011, 07:59:51 AM
If the protocol is done right, zero trust is required

Just saying that does not make it true.

Please describe a possible attack vector.
I provided a few


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: eldentyrell on December 29, 2011, 08:00:13 AM
Trust can be developed by a thin client the same way that trust in peer nodes is developed in the bitcoin protocol.

There is no trust in peer nodes in the bitcoin protocol.  There is only trust in the longest chain.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: eldentyrell on December 29, 2011, 08:01:33 AM
Please describe a possible attack vector.

The attack vector depends on the trust model.  I'm waiting for the trust model to be specified.  This is the whole reason why security protocols state their trust model: so you can look for attack vectors!


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: MoonShadow on December 29, 2011, 08:06:28 AM
Trust can be developed by a thin client the same way that trust in peer nodes is developed in the bitcoin protocol.

There is no trust in peer nodes in the bitcoin protocol.  There is only trust in the longest chain.

Ultimately, this is true.  However, the client has to start somewhere with aquiring the blocks that build that longest chain, because a fresh install doesn't know anything about the longest chain.  So it attempts to develop direct connections with a number of peers that are all sending it blocks that agree with one another, reducing (not eliminating) the possibility of an isolation attack.  The thin client model isn't about the greatest form of trust or security, but a balance between trust, security and convience.  If you can't accept the risks of getting screwed out of a small amount of spending coins on a cell phone thin client, then stick with a full client.  No one is forcing you to accept a reduction in security, thin clients simply offer a choice.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: Red Emerald on December 29, 2011, 08:12:11 AM
If you don't have a copy of the block chain, anybody running the server you're talking to can "send" you coins and then take them back with total impunity, simply by making sure that a transaction sending those coins to themselves is "mined in" to the blockchain before they release the transaction that looks like it's sending them to you.
Um. How is the server supposed to send those coins to themselves?

EDIT: They can only send you coins from them which severely limits the attack. They can't change someone else's payment to you because of the hashing done.  I also don't think "mined in" and simply can go together at the current difficulty.  Can you name a client-server model that DOES NOT require trust?

Quote
This problem is also removed once the client can talk to multiple servers.

This is dangerous "cargo cult (http://en.wikipedia.org/wiki/Cargo_cult_science) security".  It does nothing to prevent man-in-the-middle attacks (my ISP attacks me) and merely shifts the problem elsewhere.  What prevents somebody from renting 100 cheap VPSes and running the server software on all of them?  Worse: botnet operators.
I don't get what "cargo cult security" has to do with this.

Stick the protocol and SSL and you can't MITM.  You are essentially saying we are vulnerable to an attack from a large number of servers.  You know what else is vulnerable to what is often called a 51% attack? Bitcoin...

Quote
I imagine slush can give a better defense of this proposal as he seems to always write what I am thinking way better than I seem to be able.

With all due respect to slush, he shouldn't provide a "defense"; he should address these issues in the design document by explicitly stating the trust model.

The "defense" would be that the protocol doesn't need modification as there is little need for trust in almost everything the server does.  Like I and others have said: This is a client-server model; not a decentralized model.  You are giving up 100% security for ease of use.

Once this is more developed, we could make it easy for everyone to run a server of their own on their home network.  Then people could run one server (they already run atleast one node) and use thin clients that only connect to their own trusted server if they want to be paranoid.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: 2112 on December 29, 2011, 08:12:33 AM
assuming that you are not caught into a honeypot local internet

This is a piratebox...
http://wiki.daviddarts.com/PirateBox

Earlier I was simply "not impressed". But right now I switch my opinion to "starts to stink".

Oh, and a definite +1 to JSON. The protocol should appeal to the most common denominator ... which is JSON.

Good luck!
Yeah, good luck to whom?


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: ripper234 on December 29, 2011, 08:14:17 AM

To slush of course.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: ripper234 on December 29, 2011, 08:15:22 AM
Please describe a possible attack vector.
I provided a few

I don't understand - you just described while such a protocol would be secure (https://bitcointalk.org/index.php?topic=55842.msg665866#msg665866).


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: 2112 on December 29, 2011, 08:17:35 AM
Yeah, good luck to whom?
To slush of course.
But at whose expense?


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: ripper234 on December 29, 2011, 08:21:25 AM
However, there aren't too many things that a malicious server can do.

!!!Dangerous misconception!!!

Satoshi invented blockchains for a reason: they prevent double-spends.  If you don't have a copy of the block chain, anybody running the server you're talking to can "send" you coins and then take them back with total impunity, simply by making sure that a transaction sending those coins to themselves is "mined in" to the blockchain before they release the transaction that looks like it's sending them to you.

This problem is also removed once the client can talk to multiple servers.

This is dangerous "cargo cult (http://en.wikipedia.org/wiki/Cargo_cult_science) security".  It does nothing to prevent man-in-the-middle attacks (my ISP attacks me) and merely shifts the problem elsewhere.  What prevents somebody from renting 100 cheap VPSes and running the server software on all of them?  Worse: botnet operators.  See also Sybil attacks (http://en.wikipedia.org/wiki/Sybil_attack).  Basing security on 51% hashpower is reasonable; basing it on 51%-of-IP-addresses-running-the-software is definitely not.  I can't think of anything more attractive to botnet operators!

I imagine slush can give a better defense of this proposal as he seems to always write what I am thinking way better than I seem to be able.

With all due respect to slush, he shouldn't provide a "defense"; he should address these issues in the design document by explicitly stating the trust model.


Nobody said that all nodes are created equal. I would trust people with things to lose (e.g. established Bitcoin business such as Mt. Gox).
I would connect my client to servers run by the top 5 Bitcoin businesses.

Send queries and transactions to all 5. If any one of them disagrees, disregard its results.
It I can't get a consensus of 4 to agree, there is a major problem and I fail the transaction.
Use SSL to avoid man in the middle.

If you're running a business that's large enough to really be paranoid about Mt. Gox trying to scam you by falsifying TX data, then simply run your own server - this whole Overlay Network is targeted at end consumers and small businesses / webapps that don't have much to lose.

I plan to run a website that will probably never pass 100 BTC. Even before multiple server support, if I only rely on Mt. Gox server (or slush's server, or whatever), then Mt. Gox and slush have more to lose than by double spending against me.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: ripper234 on December 29, 2011, 08:21:54 AM

What?


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: Red Emerald on December 29, 2011, 08:22:32 AM
Please describe a possible attack vector.
I provided a few

I don't understand - you just described while such a protocol would be secure (https://bitcointalk.org/index.php?topic=55842.msg665866#msg665866).
I'm not saying its 100% secure.  I'm saying it probably can't be but also that the attacks are not very dangerous.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: Red Emerald on December 29, 2011, 08:25:48 AM
Nobody said that all nodes are created equal. I would trust people with things to lose (e.g. established Bitcoin business such as Mt. Gox).
I would connect my client to servers run by the top 5 Bitcoin businesses.

Send queries and transactions to all 5. If any one of them disagrees, disregard its results.
It I can't get a consensus of 4 to agree, there is a major problem and I fail the transaction.
Use SSL to avoid man in the middle.

If you're running a business that's large enough to really be paranoid about Mt. Gox trying to scam you by falsifying TX data, then simply run your own server - this whole Overlay Network is targeted at end consumers and small businesses / webapps that don't have much to lose.

I plan to run a website that will probably never pass 100 BTC. Even before multiple server support, if I only rely on Mt. Gox server (or slush's server, or whatever), then Mt. Gox and slush have more to lose than by double spending against me.

This.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: 2112 on December 29, 2011, 08:28:30 AM
At this table we are playing a zero-sum game. I know slush is the "house". But who pays the rake?


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: ripper234 on December 29, 2011, 08:30:56 AM

No we're not.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: 2112 on December 29, 2011, 08:59:11 AM
No we're not.
http://upload.wikimedia.org/wikipedia/commons/thumb/4/40/Miners_climb_Chilkoot.jpg/611px-Miners_climb_Chilkoot.jpg


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: ripper234 on December 29, 2011, 09:01:26 AM

Whatever that may mean.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: Red Emerald on December 29, 2011, 09:32:40 AM
Lets stay on topic guys.

We need to make sure that there is little to no trust between the client and server.  As well as come up with ways to mitigate attacks from a malicious server. If we find attacks, we need to figure out what damage they have.

What happens if the server resolves firstbits to a matching, but not "first" address? (Bad news I think)

What happens if the server withholds a transaction either from the client or from the network?

What happens if the server (or the person controlling the server) double spends against their client?

How should we handle server selection? Talk to multiple servers at once? Round-robin requests?  Only talk to "trusted" servers?  I think this might depend on the request.

How should we handle two (or more) servers giving the client different information?

Should SSL be mandatory to prevent MITM attacks? Can we build the protocol so a MITM cannot do any damage?


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: ripper234 on December 29, 2011, 09:44:39 AM
Lets stay on topic guys.

We need to make sure that there is little to no trust between the client and server.  As well as come up with ways to mitigate attacks from a malicious server. If we find attacks, we need to figure out what damage they have.

What happens if the server resolves firstbits to a matching, but not "first" address? (Bad news I think)

What happens if the server withholds a transaction either from the client or from the network?

What happens if the server (or the person controlling the server) double spends against their client?

How should we handle server selection? Talk to multiple servers at once? Round-robin requests?  Only talk to "trusted" servers?  I think this might depend on the request.

How should we handle two (or more) servers giving the client different information?

Should SSL be mandatory to prevent MITM attacks? Can we build the protocol so a MITM cannot do any damage?

- firstbits - I'm not really interested in that feature - if it's a security risk, postpone/cancel it.

- withholding TX - mitigated by talking to N independent servers (not random servers to prevent someone from starting 100 instances, but known servers hosted by various known organizations and persons).

- Double spend - Mitigated by talking to N servers. If you send a TX to N independent servers, with at least N/2+1 honest nodes, then any attempt at a double spend will be easily detected.

- Start with a single server. Next step would be a hard code list of trusted servers, with requests going out to N of them, and at least K have to agree. Future work - please don't let this delaying getting the first server operational ... 1 is so much better than 0.

- SSL should be mandatory, probably from day 1. I don't see how the protocol itself need to change, just the transport.

Nothing here is a show stopper or a major architectual risk - let's get this thing up with 1 server, and work from there.

Be aware that this layer will never be a decentralized widespread network like Bitcoin. At first there will only be a handful of such servers, and there's absolutely no need for more than a few dozens or hundreds (known, trusted, not anonymous) servers besides future scalability concerns. Don't over engineer.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: slush on December 29, 2011, 06:05:40 PM
The resultant Frankenstein monster (or family of monsters) is going to try to strangle its creator.

I agree with you that projects like this are tending to be over complicated. Many times I found that I'm engineering some monster project just because I wanted to hide internal complexity of the task. That's exactly the reason why I'm forcing myself to keep this KISS: basic transports, simple RPC protocol, pubsub mechanism, services on top of this. If you know how to make it even simplier, tell me, seriously!

I'm trying to avoid "full stack" RPC mechanisms like SOAP+WSDL, because, honestly, it's pretty hard to implement everything correctly, on both ends.

Quote
It is hard to argue with what you proposing point by point because of the contradictory nature of the requirements. I'm going to group my response into the following points.

I'm fully aware that those requirements are contradictory. It's pretty common that some solution needs to find some compromises. But correct me if you think that some of those requirements are wrong (for such purpose of overlay network).

Quote
1) Protocol can be either RPC or bidirectional, but not both.

Well, what's wrong on creating communcation channel and then have a possibility to call services on both ends, from any connected side?

I'm aware that it *can* be overcomplicated in the end, but it fully depends on designing of services. This idea is taken from jabber protocol, where both sides are providing some services. (And yes, I thought about building overlay network on top of jabber protocol, because it already provides service paradigm, but I think jabber is great example of overcomplicated stuff.)


Quote
The RPC paradigm (request-response, master-slave) is mutually contradictory with a pair of peers exchanging asynchronous messages.

I agree. I like RPC much more than message-based protocol, because it's making communication much cleaner. Just a request and its response. It's the reason why I'm not proposing communication based on async messages, because dependencies of various types of messages can turn into total mess.

Quote
Bitcoin protocol itself is from its origin asynchronous and cannot be squeezed into the master-slave architecture.

I partially agree (not fully, because everything can be transformed into master-slave architecture, but of course it doesn't necessary mean that it's effective), but I'm not talking about bitcoin network. Difference between Bitcoin network and "Overlay" network is that Bitcoin is distributed database, but Overlay is network of services.

Quote
2) Your target market (low-end consumer-level devices) demands that there's checksuming and framing at the application layer. Precisely because cheap NAT gateways and cheap DSL/Cable/WLAN modems are known to mangle the transport-level frames due to bugs (in the implementation of NAT) and excessive buffering (to improve one-way file transfer benchmarks).
If you think you can add CRC later you are going to loose by not detecting corruptions early.

Hm, thanks to you, I'm thinking about this a little more than before. Is that really an issue? Isn't TCP checksumming and TCP retransmissions on both ends enough to "fix" corrupted information? More generally, keeping transmission working is task for the transport layer, not for protocol itself. And I agree that transport implementations should use it's internal mechanisms for checking that transmission was successfull. Like using Content-Length or content checksum in http headers.

RPC proposal itself contains unique IDs of requests to link requests to response. If transport layer fail, it will probably appear as disconnection, which also closes the session (on TCP transport), so client need to reset it's internal state after reconnecting (ask for balance, history etc, to be sure it is again in stable state).

Side note: I'm running json-based protocol on the pool over a year, I had over 3300 rq/s in June peak. I agree that mining protocol is stupid and ugly (although I understand the reason how and why it has been designed in such way - don't worry m0m ;-) ) and that there are some isssues with DDoSing etc. But I definitely didn't have any problems with corrupted packets like you're suggesting.

Quote
3) In my experience JSON is probably the close to being the least resilient encoding possible. I can't disclose proprietary data, but I have over a decade's worth of reliability statistics for various  remote services (RPC-like) that are sold by organizations I consulted. But the rough ranking is
as follows: (from the least errors to the most)

3.1) ultra-lean character-based protocol similar to FIX, designed originally for MNP5 and V.42bis modems, currently used through a simple TCP/IP socket
3.2) SOAP (RPC over XML) with Content-Length, Content-MD5 and DTD verification enabled
3.3) SOAP and plain XML-RPC without the above strenghtening options
3.4) JSON-RPC
3.5) RPC over e-mail with various human-readable encodings

Fair summary, thanks. I'm familiar with SOAP (although I never used it in real life). As I mentioned above, I agree that Content-Length and Content-MD5 should be implemented, but it's part of transport in my concept (because HTTP is only one of many ways how to transfer bytes from one side to another, and Content-* are headers of HTTP), not a part of protocol so we are in agreement here.

About DTD - it's definitely better face of XML concept and I see some benefits in implementing DTD as formal specification of protocols. Although  I personally dislike XML, I'm open to change my mind at this point. Is here, as an example, standardized way how to serialize data such lists or dictionaries? I picked JSON because it is providing pretty compressed serialization of some standard structures in transparent and understandable way. I can call json_encode(any_data) and don't care about "how it works" in almost any programming language. But if there is some similar, widely accepted XML specification for serializing such objects, I'll elaborate. But handling XML streams on low level is usually pain.

Quote
JSON is also infamous for letting people easily make byte-endianness mistakes, very much like current "getwork" which is neither big-endian not little-endian.

Well, endianess in getwork isn't the mistake of json, but of layer above. However, I agree, it is a pain.

Quote
4) You somehow read the my earlier suggestions about IPsec imply high-end large-volume target market. The reality is quite opposite: Windows support IPsec since 2000, Linux for a long time, Netgear ProSafe family has several models in $100-$200 range, L2TP and PPTP are available for free on all iPhones,Blackberries,Androids; many Nokias and other smartphones. The real hindrance is the HTTP(S)-uber-alles mindset, not the actual difficulty and cost of the implementation.

Thanks for explanation. I have only one experience with IPSec ten years ago and that experience was painful. Good to hear it gets better over the time. I feel I'm repeating myself, but I see that "transport" concept is one of the most powerful thing on my proposal. If somebody want to fiddle with IPSec, give it to him. But most of people will be happy with ssl-hardened TCP socket or even HTTP poll, which is very familiar for them. Both for programmers and for end users.

Quote
In summary I'd like to say that you wrote a very interesting and thought provoking proposal. I just think that the range of the targets you are hoping to cover is way too broad ($3 AVR processors, shared hosting plans, home computers, etc.).

Actually I'm already in touch with two groups of people who're developing hardware based wallets, so my proposal was created with some specific projects in mind. I agree that final audience is pretty wide, but everything I can do is to keep KISS attitude and hope I won't overcomplicate it in some way.

Quote
I have my personal litmus test for the technological implementation in the Bitcoin domain: it has to support (and be tested for) chain reorganization. Preferably it should correctly retry the transactions upon the reorg. Absolute minimal implementation should correctly shut down with a clear error message and a defined way to restart the operations.

Interesting stuff. I feel like overlay network should be "stateless", which mean that it will "forget" transaction once transaction has been succesfully broadcasted to Bitcoin network. The implementation of retransmissions should be more the task of clients built on top of overlay network. As far as overlay notify it's clients that their balance has been changed (because of blockchain reorg) and provide correct address history (using actuall chain branch), it's job should be done. The reason why I don't think overlay network should try to "fix" such issues by retransmissions is that it can turn into really complicated stuff. As an example - end user don't want to re-broadcast some transaction, because it has been created in the context of previous incoming transaction in "wrong" chain, which isn't actually stored in currect branch... But thanks for suggestion, I'll think about it more.

In the end, I really appreciate that our discussion is factual and we're discussing specific points more than "you're doing everything wrong". I think this is constructive and helps me in my project.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: slush on December 29, 2011, 07:32:46 PM
The thing is not to get too caught up writing long proposals and documents. Think a little bit, code up an implementation, get rolling and refine your requirements as they become obvious. It's easier to ask for forgiveness than to ask for permission.
I disagree.  I think planning ahead always leads to better software even if the software development may be slower to start.  Better to find a problem in your proposal and rewrite that than to discover a problem in your code and redo a ton of work there IMO

Proposals are especially important when multiple people are working on a project as it makes it easier to work on different parts of the project at once.

You're advocating the ancient waterfall method which is basically out of fashion since the last decade.

Quote
genjix:~$ python
Python 2.7.2+ (default, Oct  4 2011, 20:03:08)
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
I think planning ahead always leads to better software even if the software development may be slower to start.  Better to find a problem in your proposal and rewrite that than to discover a problem in your code and redo a ton of work there IMO

Proposals are especially important when multiple people are working on a project as it makes it easier to work on different parts of the project at once.

I don't want to turn this thread to battle between waterfall/XP. I'm not designing my proposal to form which is "ready to implement" by any monkey. However I feel that anybody with some experience can understand the basics from my paper and ask some questions, which can bring some good ideas, like 2112 is doing already. So I'm trying to find some balance between waterfall method and "show me the code" attitude.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: slush on December 29, 2011, 07:39:26 PM
* use JSON RPC

done

Quote
* the set of functions that are currently used by the Electrum client and server (address based functions): the server does not know the set of addresses in a client's wallet, it just sends address histories and broadcasts transactions. This means that a client should be able to use several servers simultaneously for improved anonymity.

yes, that's main purpose. so - done. I call it the "blockchain service", because it provide API for querying blockchain database for chainless clients.

Quote
* wallet-based functions (similar to BCCAPI) for ultra-thin clients:  The server knows the public key used to generate the sequence of addresses in a type 2 wallet. It sends the balance and history of the wallet. It also sends the number of addresses detected in the wallet (gap based detection) The server also sends unsigned transactions to the client, and the client signs them.

Yes, this is possible, I call it the "wallet service", because it provides functionality of coin selection, creating of (unsigned) transactions etc.

Quote
* also, I would like to see something similar to Transaction Radar: http://www.transactionradar.com/ : when a transaction is unconfirmed, the client should display its rate of propagation. Electrum servers could be part of the existing transaction radar service.

I already implemented 'txradar' service (https://gitorious.org/electrum-server/electrum-server/blobs/master/server_repository/txradar.py), as an easy example of 'proxy service' (providing standard overlay-network API, but asking some other specialized service for doing the job). I see txradar service as usefull one, but it's fully on clients how they integrate such information.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: slush on December 29, 2011, 07:56:13 PM
I just re-read JSON-RPC 2.0 specification (http://json-rpc.org/wiki/specification) and found that I missed "notification" type of messages, which very similar to what I proposed for pubsub mechanism. There was difference that notification has id:null, which I see as a smart choice, so I changed proposal documentation and also protocol implementation (not commited yet). So far, application protocol is now *fully* compatible with JSON-RPC 2.0.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: slush on December 29, 2011, 07:58:09 PM
Use WebSocket as transport. It is bi-derectional and simple enough.
http://en.wikipedia.org/wiki/WebSocket

Yes, I plan websocket as a transport oriented for in-browser clients. However websocket is more like fallback solution for environments where standard TCP socket isn't available. websockets have significant overhead and there's also a lot of versions, making it painful for implementation.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: slush on December 29, 2011, 08:18:59 PM
Just my two cents:
1. Like some others said, develop a prototype, launch early, and iterate.

Yes, I'm doing this already. However I like to open discussion and not everybody can understand python (and especially twisted) code. So I found writing textual proposal as a good way how to gain interest also between non-programmers.

Quote
2. Should not require sending private keys (haven't read the protocol in detail yet, but just in case this was not planned). Instead use some form of Offline Transactions.

It's exactly what I'm trying to do.

Quote
3. Should consider DDOS protection. If this will be The Simple Network over Bitcoin, that will eventually handle huge traffic, you might want to consider charging a bit per API call ... just something minimal to deter DDOS. People can register their API keys, charge their account with X BTC, and get an allowance for Y API calls.

Good point. About DDoS - you can DDoS any service even without paying for it. You can just flood it with a lot of requests. So paying as a protection against *real* DDoS isn't a solution. Of course you can make service paid to avoid people misusing processing power. I'm already thinking about it, not sure if I found any viable solution.

Brainstorming: I'm thinking about "credit service" and standard exception "Fee required", which will indicate that service operator want some fee for performing such call. It's up to every server operator which calls will be free or paid. When client receive "fee required" exception, he needs to authenticate to "credit service" with prepaid credit. When previous failed call will be retried, server will credit the fee from such prepaid account. Not sure if this is a good way, but it's the best solution which I found so far. Comments welcome.

Quote
4. Should consider malicious servers. If someone evil runs an evil copy of this server, and clients connect to it ... how exactly can the evil server damage them?

Basically, there's not so much things how can malicious server damage users (in serious way - like stealing their coins). However if client wants to be sure he has correct data, he can perform same calls to more overlay servers. If all of them are on valid blockchain, their responses should be the same.

Quote
Can EvilServer steal money? (Probablly not if you're just using Offline TX as the only mutator operation)

No, it cannot. However there's real issue with services like firstbits. If server provide different result for firstbits lookup, client can send money to wrong address. However this can be solved by performing multiple calls to different servers.

Quote
Does it have a strong incentive to lie about GET queries?

I don't think so. However some people can do malicious things just because they can, without any specific reason. This is why I think that people will need to pick only trusted servers or confirm important calls against different overlay server.

Quote
These are just some things to think about ... you don't need all the answers in advance. I actually started developing a small Bitcoin webapp today, and got stuck when I realized I'd have to maintain the entire blockchain (or even just the headers). If a prototype of this project comes out early enough (~ 1 month), I'll wait for it instead of writing and debugging blockchain-maintaining code.

You're exactly the person for who I'm doing this project :-). I see that maintaining full client is big overkill for some kind of projects which want to integrate with Bitcoin. And I'm almost sure there will be working server in less than one month. You're using PHP on your site, right? Please send me a PM with some details, I need to design PHP binding and I want to discuss it with somebody who's actively working in PHP...


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: slush on December 29, 2011, 08:23:31 PM
This is cool, but I couldn't find any mention of the trust model anywhere in this thread or the design docs.

If you aren't keeping a copy of the blockchain, you need to find another answer for this question.

Well, server cannot steal your money, it can only lie with transaction history or balance. I'm not saying it's not enough for some kind of attack, but it's still better than using API of some web-based wallet (like MtfGGox).

So yes - if you don't have full copy of blockchain, you need to trust somebody else, at least on some level. But the most of users or projects (like eshops) aren't so big that it could be profitable to cheat them from side of server operators. You can also ask multiple servers to confirm balances/history, which is limiting necessary trust to single entity.

Overall, it's more secure solution than using cart APIs of web wallets (like, heh, mybitcoin), because you still own your bitcoins.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: ripper234 on December 29, 2011, 08:29:37 PM
Good point. About DDoS - you can DDoS any service even without paying for it. You can just flood it with a lot of requests. So paying as a protection against *real* DDoS isn't a solution. Of course you can make service paid to avoid people misusing processing power. I'm already thinking about it, not sure if I found any viable solution.

Well, it's not the only solution, but it does indeed guarantee only minimal processing will be done on DDOS requests.

Brainstorming: I'm thinking about "credit service" and standard exception "Fee required", which will indicate that service operator want some fee for performing such call. It's up to every server operator which calls will be free or paid. When client receive "fee required" exception, he needs to authenticate to "credit service" with prepaid credit. When previous failed call will be retried, server will credit the fee from such prepaid account. Not sure if this is a good way, but it's the best solution which I found so far. Comments welcome.

It's a good direction. I think the main thing is that you want to ingrain API keys in the service at the deepest level, to make it easy to manage connections and fees. While some servers might support a "guest key" e.g. for testing, every call must have a valid key, that should be pre-registered (for free or not).


Quote
These are just some things to think about ... you don't need all the answers in advance. I actually started developing a small Bitcoin webapp today, and got stuck when I realized I'd have to maintain the entire blockchain (or even just the headers). If a prototype of this project comes out early enough (~ 1 month), I'll wait for it instead of writing and debugging blockchain-maintaining code.

You're exactly the person for who I'm doing this project :-). I see that maintaining full client is big overkill for some kind of projects which want to integrate with Bitcoin. And I'm almost sure there will be working server in less than one month. You're using PHP on your site, right? Please send me a PM with some details, I need to design PHP binding and I want to discuss it with somebody who's actively working in PHP...

Nice. Actually ... no, I loath PHP.

Working on Java/Play Framework. The project is open source, I'll publish it when it's ready.
Basically I just need a couple of API calls, so if you provide a JSON http API, I'll be happy to implement java bindings.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: slush on December 29, 2011, 08:34:49 PM
- firstbits - I'm not really interested in that feature - if it's a security risk, postpone/cancel it.

Yes, depending on firstbits resolution made by another party can be risky. However, fake firstbits in really smart way needs significant computation power, because server operator needs to find his own address which corresponds to given firstbits prefix. It's almost impossible in real time so unless you're selling house using firstbits address, there's no *real* way how to misuse that. However server can also make a mistake, for example by calculating firstbits on wrong blockchain branch...

Quote
- withholding TX - mitigated by talking to N independent servers (not random servers to prevent someone from starting 100 instances, but known servers hosted by various known organizations and persons).

yes, possible

Quote
- Double spend - Mitigated by talking to N servers. If you send a TX to N independent servers, with at least N/2+1 honest nodes, then any attempt at a double spend will be easily detected.

Yes, possible. Also asking txradar service (on different server) can do the job easily.

Quote
- Start with a single server. Next step would be a hard code list of trusted servers, with requests going out to N of them, and at least K have to agree. Future work - please don't let this delaying getting the first server operational ... 1 is so much better than 0.

It's a matter of days when I'll start beta server. I still need to finish some stuff to make it really usable.

Quote
- SSL should be mandatory, probably from day 1. I don't see how the protocol itself need to change, just the transport.

TCP socket with SSL is already implemented. It's another transport (on another port), so it's absolutely to user which port/transport he choose.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: slush on December 29, 2011, 08:38:28 PM
Working on Java/Play Framework. The project is open source, I'll publish it when it's ready.
Basically I just need a couple of API calls, so if you provide a JSON http API, I'll be happy to implement java bindings.

Even better. I'm able to write PHP binding, but not a Java one. Having a protocol implementation and at least "HTTP polling" transport (probably the easiest transport to implement) in Java would be amazing.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: ripper234 on December 29, 2011, 08:53:57 PM
Working on Java/Play Framework. The project is open source, I'll publish it when it's ready.
Basically I just need a couple of API calls, so if you provide a JSON http API, I'll be happy to implement java bindings.

Even better. I'm able to write PHP binding, but not a Java one. Having a protocol implementation and at least "HTTP polling" transport (probably the easiest transport to implement) in Java would be amazing.

Did you know about BCCAPI? It seems to be the third such project I've heard about, trying to solve the same problem (1st being Overlay, 2nd bitcoinjs Exit Nodes).

This one does a java client.

The leaders of these three projects should get together and talk ... see if they can make a unified standard now, before it's too late.

http://imgs.xkcd.com/comics/standards.png


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: kjj on December 29, 2011, 09:04:51 PM
- firstbits - I'm not really interested in that feature - if it's a security risk, postpone/cancel it.

Yes, depending on firstbits resolution made by another party can be risky. However, fake firstbits in really smart way needs significant computation power, because server operator needs to find his own address which corresponds to given firstbits prefix. It's almost impossible in real time so unless you're selling house using firstbits address, there's no *real* way how to misuse that. However server can also make a mistake, for example by calculating firstbits on wrong blockchain branch...

Exactly.  Firstbits can only be safe if the person doing the lookup has the full block chain, AND if the address in question is buried deeply enough in the chain that a reorg can't touch it.  If either of these are false, you are vulnerable.

I'm not sure how widely known that second part is, but since the point of firstbits is to have addresses known in advance, the odds of getting a fresh (aka vulnerable) one are already small.

Don't forget that an attacker doesn't need to wait for a spender to come along before he starts looking for his own bogus addresses that look like firstbits matches.  Since the point of firstbits is for the address to be well known in advance, he can start compiling his database now, and then wait until he sees a transaction to one he's already found before he attacks.

Personally, I think that the inevitable proliferation of chainless light clients is going to kill firstbits, for exactly these reasons.  There is just no way to make it safe without checking on your own fully chained node.

By the way, when I was researching this sort of thing, I came to a couple of surprising conclusions.  First, unidirectional RPC will work totally fine, it is never necessary for the server to initiate, and no state is ever needed.  Second, there are only a few RPC calls missing from the standard client that would be necessary to fully support light clients.  And third, as long as the light client is creating the transaction internally and using the server as a dumb relay, a malicious server can't do any real damage, it can only cause some light mischief.  (That third one is only valid from the spender's point of view, of course.  An owned server is still a big problem for the owner of the server, just not for other people that use it.)


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: slush on December 29, 2011, 09:09:15 PM
Did you know about BCCAPI? It seems to be the third such project I've heard about, trying to solve the same problem (1st being Overlay, 2nd bitcoinjs Exit Nodes).

Yes, I know about BCCAPI and bitcoinjs. BCCAPI is (right now) solving only part of the problem (just a subset which I call "wallet/blockchain services"), it's using custom binary protocol (in the oposite of json-rpc) and only polling transport (in the oposite of various transports). Also server implementation is closed source (I have code on gitorious already).

BitcoinJS has similar purpose. I talked with Stephan Thomas about bitcoinjs in Prague and the result is that he wants to do everything himself.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: ripper234 on December 29, 2011, 09:56:09 PM
Did you know about BCCAPI? It seems to be the third such project I've heard about, trying to solve the same problem (1st being Overlay, 2nd bitcoinjs Exit Nodes).

Yes, I know about BCCAPI and bitcoinjs. BCCAPI is (right now) solving only part of the problem (just a subset which I call "wallet/blockchain services"), it's using custom binary protocol (in the oposite of json-rpc) and only polling transport (in the oposite of various transports). Also server implementation is closed source (I have code on gitorious already).

BitcoinJS has similar purpose. I talked with Stephan Thomas about bitcoinjs in Prague and the result is that he wants to do everything himself.

I see.
I hate "open source" projects that are self centered and don't think big enough.
So much potential for good things, so many misses.

Edit: I don't know too much about bitcoinjs, this is just a random rant that might or might not be relevant.

I'm very much excited about Overlay, and it looks like you're really out to build something good here.
If needed, you can contact me for any questions at ron.gross@gmail.com


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: genjix on December 29, 2011, 10:48:03 PM
I support slush and ThomasV.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: slush on December 29, 2011, 10:51:59 PM
Brainstorming about signing messages

In the concept of service oriented architecture, some service calls can be proxied to backend or sent over non trusted channel. I like Firstbits as an example of proxied call, but of course there can be a lot of another examples as well as firstbits lookup can be implemented directly inside overlay server and proxy call isn't necessary at all.

For now, during the call of firstbits.resolve, the flow of call is following:

client -> (some transport) -> overlay server -> HTTP (firstbits.com don't support https yet!) -> firstbits.com

Client need to trust every part of this chain. For now, you must use some transport which prevent MITM, trust overlay server operator, hope that there isn't MITM attack in HTTP request to firstbits.com (I hope firstbits fix that and provide https soon).

Well, what if firstbits.com can sign it's response to confirm that resolution has been done directly by the service, which is trusted by client? Then firstbits resolution can be done over pidgin transport and nobody care, because the response is signed with service's key...

JSON-RPC don't offer any standard way how to sign messages, transports usually don't offer signatures either, so I want to fix it in some way. I'm not a crypto expert, but I have some idea how it should be done. Feel free to correct me and propose anything safer/easier:

Signature should be assembled from following information:
a) initial request
b) timestamp of processing
c) response

Signature should not be done from whole json message itself, only from payload, because json serialization can be somehow ambiguous, for example parameter 'id' can be changed during the transport.

I has been thinking about using GPG mechanism, which is widely used platform even for distributing trust, but it looks prettyheavy for such purpose. Not sure if it's possible to check GPG signatures in $3 AVR processor. However Bitcoin world already use signatures pretty nicely and clients already needs ecdsa libraries on their side for signing transactions, so using ecdsa sounds reasonable and pretty easy to me. Simple example for signing/verifying data using SECP256k1 curve in python can be found here: http://pastebin.com/LnPijjfm

However I see real issue in needed CPU time for signing. On my laptop, one signature takes around 200ms. It's low-end one, but it will take some time even on high-end servers. Any idea how to solve it?

Final signed response should looks similar like:
{"id": "1234", "signature": "MEUCIHM3m9IX6oudiBOXLx5bavACtB6uJNFDr8Ir6qVX54nhAiEA9SBgR1tOxEJnVJBqhKt+QfE0ry0h8yP2M7KinmQ/yPs=", timestamp: "1325202560", "result": ["blahblah"], "error": null}

However the question is which pattern/algorithm use to assemble data for signing (request payload, timestamp, response payload). Is there any reasonable solution for that?


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: eldentyrell on December 30, 2011, 01:58:00 AM
Nobody said that all nodes are created equal.

Indeed, the problem is that nobody has said anything at all about equality (aka trust).


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: eldentyrell on December 30, 2011, 02:02:11 AM
So yes - if you don't have full copy of blockchain, you need to trust somebody else, at least on some level.

Thanks slush.  I urge you to be very specific about who this "somebody else" is and exactly what "level".

Without this information, it is not possible for independent third parties to do a security analysis of your protocol.  I would not recommend that people use a protocol that has not (and can not) be analyzed by independent third parties.  History has shown that without this sort of independent analysis, gruesome and nasty security flaws inevitably crop up.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: kjj on December 30, 2011, 02:49:26 AM
Brainstorming about signing messages
Final signed response should looks similar like:
{"id": "1234", "signature": "MEUCIHM3m9IX6oudiBOXLx5bavACtB6uJNFDr8Ir6qVX54nhAiEA9SBgR1tOxEJnVJBqhKt+QfE0ry0h8yP2M7KinmQ/yPs=", timestamp: "1325202560", "result": ["blahblah"], "error": null}

However the question is which pattern/algorithm use to assemble data for signing (request payload, timestamp, response payload). Is there any reasonable solution for that?

Add a "signature_algorithm" key to the response.  Also, allow the request to specify a list of acceptable algorithms, like {"id": "1234", "signatures_accepted": "ecdsa: SECP256r1, SECP521r1; pkcs 1", "firstbits": "1payBTC"}  If the server can't provide a signature that the client is willing to accept, throw an error response.  Sadly, this is a damned if you do, damned if you don't situation.  If you lock in to a single system, you are stuck with that system and the hassle of replacing it later.  If you make it extensible, you have the hassle of actually supporting multiple systems.  (See IPSEC)

As for actually calculating it, there are two ways to go.  In a binary protocol/format or when the length of the signature is known in advance, it is common to construct the response using zeros to fill in the signature field, then calculate the signature based on that, then overwrite the zeros with the actual signature.  In a text stream like this, I would build the response without the signature key and value, then construct a new response that includes them.  Or just insert them, modern languages have no problem chopping and splicing strings, and C programmers already have lots of practice doing it the hard way.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: slush on December 30, 2011, 11:01:49 AM
Thanks slush.  I urge you to be very specific about who this "somebody else" is and exactly what "level".

Somebody else = operator of overlay server. It's as simple as this. When we talk about blockchain services, the necessary trust to other party (=operator) is higher than in using full blockchain by self, but a lot lower than depending on "web wallet", because you're still the owner of your funds.

In the case of other (non-blockchain) services like fetching usd/btc quotes from overlay server, it's the same like now; you still need to trust bitcoin exchange and service like bitcoincharts, which is aggregating quotes for you.

Quote
Without this information, it is not possible for independent third parties to do a security analysis of your protocol.

What exactly are you missing and in which form you need such information?


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: slush on December 30, 2011, 12:29:00 PM
However I see real issue in needed CPU time for signing. On my laptop, one signature takes around 200ms. It's low-end one, but it will take some time even on high-end servers. Any idea how to solve it?

I found that only python's ecdsa module is so slow. Author of ecdsa package did some comparsion to C++ and it was 20-30x faster. So the obvious solution is "implement python's ecdsa signatures now, optimize it with C++ speedup later". Signed messages have sense only for proxied calls or when using non-secure transport, so <5 ms per signature per message is doable in such situations.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: ripper234 on December 30, 2011, 12:51:21 PM
slush, how would the transaction mechanism work?

What would I need to calculate client side in order to authorize a transaction?
Is there a Java implementation of that somewhere (bitcoinj?)?


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: slush on December 30, 2011, 01:32:35 PM
Add a "signature_algorithm" key to the response.  Also, allow the request to specify a list of acceptable algorithms, like {"id": "1234", "signatures_accepted": "ecdsa: SECP256r1, SECP521r1; pkcs 1", "firstbits": "1payBTC"}

good idea

Quote
As for actually calculating it, there are two ways to go.  In a binary protocol/format or when the length of the signature is known in advance, it is common to construct the response using zeros to fill in the signature field, then calculate the signature based on that, then overwrite the zeros with the actual signature.  In a text stream like this, I would build the response without the signature key and value, then construct a new response that includes them.  Or just insert them, modern languages have no problem chopping and splicing strings, and C programmers already have lots of practice doing it the hard way.

Yes, I was thinking about this, too. Unfortunately it's getting slightly complicated, because there are also some other fields which can be changed without corrupting the signature, like message "id". It's because ID must be unique in particular session, but this signed response can be created as a response to some proxy call to backend (in the firstbits example, signature is created directly by firstbits service, but signature must be valid also as a response from overlay server to client.

Another issue is that fields in json-rpc don't have strict order, so two identical and valid json-rpc may have different signature (well, 2112 must be laughing now :-) ). This is problematic for validating messages, because you need to extract fields which are not part of signed message, but not affect message payload.

This leads me to conclusion that trying to sign/verify message itself can be very fragile. I also don't want to go against json-rpc specification and force such things like order of fields. Currently I don't know how to decide this issue, is there any best practice to doing this? How other text-based protocols solve such problem?


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: EhVedadoOAnonimato on December 30, 2011, 01:40:28 PM
Sorry if I'm being repetitive, I haven't read all 4 pages of discussion.

But one thing that I don't understand is why are people so in a rush to implement client-server solutions, when a p2p lightweight solution is perfectly possible, and we already have BitcoinJ to demonstrate. BitcoinJ is lightweight to the point you can use it on a mobile phone. And it could scale very well even when blocks are huge if people develop a new protocol message of the kind "what are the outputs of block X?".

Anyway, I don't like the client-server solution due to the fact that the server may know everything you do with your money. You risk losing your privacy when going client-server. Okay, you may hide your IP behind Tor or something, but most people don't even know how to do that. And that might not be enough depending on how the client-server protocol works. If the "what's my balance?" message sends the entire list of address the client has, than Tor won't help, the server already knows your entire wallet. All a malicious server needs now is to link one of these address to the actual owner. And that's not that difficult, just see how many people have public tip jars for example.

There's a strong privacy issue in client-server solutions. IMHO you clever developers should give priority to lightweight p2p solutions, or at least be always very clear on the privacy risks of a client-server solution, so that nobody can say he didn't know about it.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: ripper234 on December 30, 2011, 01:45:24 PM
Sorry if I'm being repetitive, I haven't read all 4 pages of discussion.

But one thing that I don't understand is why are people so in a rush to implement client-server solutions, when a p2p lightweight solution is perfectly possible, and we already have BitcoinJ to demonstrate. BitcoinJ is lightweight to the point you can use it on a mobile phone. And it could scale very well even when blocks are huge if people develop a new protocol message of the kind "what are the outputs of block X?".

Anyway, I don't like the client-server solution due to the fact that the server may know everything you do with your money. You risk losing your privacy when going client-server. Okay, you may hide your IP behind Tor or something, but most people don't even know how to do that. And that might not be enough depending on how the client-server protocol works. If the "what's my balance?" message sends the entire list of address the client has, than Tor won't help, the server already knows your entire wallet. All a malicious server needs now is to link one of these address to the actual owner. And that's not that difficult, just see how many people have public tip jars for example.

There's a strong privacy issue in client-server solutions. IMHO you clever developers should give priority to lightweight p2p solutions, or at least be always very clear on the privacy risks of a client-server solution, so that nobody can say he didn't know about it.

Nobody really cares about privacy anymore. We want convenience first. And a client that keeps zero information is more convenient (easier to program, less space requirement) than a client that maintains a full or even partial blockchain.

Bitcoin is novel in that it enables privacy, lack of central control etc ... but that doesn't mean that every real world application of it requires these properties. I would be happy if my bank started dealing with Bitcoin, because this would increase competition between banks and reduce costs, not (just) because of privacy. When I need privacy in Bitcoin, I'll buy some BTC using cash from someone and use a full P2P client behind TOR. For the other 364.9 days of the year, I'll use a thin "semi-centralized" client.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: EhVedadoOAnonimato on December 30, 2011, 02:09:58 PM
Nobody really cares about privacy anymore. We want convenience first. And a client that keeps zero information is more convenient (easier to program, less space requirement) than a client that maintains a full or even partial blockchain.

First, a lightweight p2p client is very convenient.
And second, I find bitcoin privacy of major importance, because it makes bitcoin possibly the best way for people to evade taxes while still transacting online. If you believe most people are "government suckers" who want to give their money away then, well, fine, it's your opinion. Maybe for some countries that might even be true. But anyway, all I'm asking is for people to consider the lightweight p2p option, which is trust-free and scale very well, while keeping your privacy, if you happen to care.

At least be open on the privacy risks in using client-server solutions. People downloading the software should know.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: kjj on December 30, 2011, 02:18:16 PM
Add a "signature_algorithm" key to the response.  Also, allow the request to specify a list of acceptable algorithms, like {"id": "1234", "signatures_accepted": "ecdsa: SECP256r1, SECP521r1; pkcs 1", "firstbits": "1payBTC"}

good idea

Quote
As for actually calculating it, there are two ways to go.  In a binary protocol/format or when the length of the signature is known in advance, it is common to construct the response using zeros to fill in the signature field, then calculate the signature based on that, then overwrite the zeros with the actual signature.  In a text stream like this, I would build the response without the signature key and value, then construct a new response that includes them.  Or just insert them, modern languages have no problem chopping and splicing strings, and C programmers already have lots of practice doing it the hard way.

Yes, I was thinking about this, too. Unfortunately it's getting slightly complicated, because there are also some other fields which can be changed without corrupting the signature, like message "id". It's because ID must be unique in particular session, but this signed response can be created as a response to some proxy call to backend (in the firstbits example, signature is created directly by firstbits service, but signature must be valid also as a response from overlay server to client.

Another issue is that fields in json-rpc don't have strict order, so two identical and valid json-rpc may have different signature (well, 2112 must be laughing now :-) ). This is problematic for validating messages, because you need to extract fields which are not part of signed message, but not affect message payload.

This leads me to conclusion that trying to sign/verify message itself can be very fragile. I also don't want to go against json-rpc specification and force such things like order of fields. Currently I don't know how to decide this issue, is there any best practice to doing this? How other text-based protocols solve such problem?

For the usual case, you don't care that multiple messages might be equally valid.  The signature is for "this message", not for "any message bearing the same content as this message".

If you are worried that a proxy might replace the message with a message that is different but equivalent, then you need to have a canonical form, and you should only emit the canonical form.  When the client receives a message, it must convert it into canonical form before checking the signature.

As for the ID, don't leave it out of the signature unless you enjoy replay attacks.  Just force the client to make it globally unique.  There are already well established techniques for doing this.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: slush on December 30, 2011, 02:31:11 PM
slush, how would the transaction mechanism work?

Creating transactions is done in few steps:

1. Choose destination address (or addresses)
2. Choose source addresses (=coin selection)
3. Calculate transaction fee
4. Sign inputs
5. Combine everything into transaction as a stream.

All those steps must be addressed in bitcoinj already, because those are necessary steps for every client. Those steps are implemented also in Electrum client (python).

However things can be slightly easier, because steps 2,3 and 5 can be done on server side (it's usually called super lightweight client, because it depends on server even with coin selection). BitcoinSpinner (java based android client) is doing it already.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: slush on December 30, 2011, 02:40:26 PM
But one thing that I don't understand is why are people so in a rush to implement client-server solutions, when a p2p lightweight solution is perfectly possible, and we already have BitcoinJ to demonstrate.

Because both technologies solve different problem. Bitcoin p2p is distributed database, however I'm trying to create protocol for *services* on top of this distributed database.

If you want privacy and security, you can every time use blockchain. If you need speed and convenience, use some overlay network, which masks compexity for you and provide only necessary service. It's giving freedom of choice, obviously.

By the way, not all services offered by overlay must be related to blockchain and they cannot be provided in decentralized way. For example you probably want to pick some escrow service and use it's API directly or over trusted channel, not using some anonymous peer in p2p network.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: slush on December 30, 2011, 02:59:48 PM
For the usual case, you don't care that multiple messages might be equally valid.  The signature is for "this message", not for "any message bearing the same content as this message".

I probably failed I expressed my worries. So I'll try again.

Main benefit of JSON is that it can be serialized/deserialized in almost every language in transparent way, just obj = json_decode(ser_data). However, those json_encode/decode implementations don't care about order of variables in serialized data, because order isn't a part of json specification.

So json_encode(json_decode(data)) can provide different stream than original data, which is necessary for validating json message. Thanks to this issue, you need to fiddle with serialized data directly as with text stream and remove signature-related data from this stream, instead of deserializing message, removing signature-related data, serializing it back and checking signature.

Quote
As for the ID, don't leave it out of the signature unless you enjoy replay attacks.  Just force the client to make it globally unique.  There are already well established techniques for doing this.

As a prevention for replay attacks I'm proposing "timestamp" field. It's also the reason why I think that request should be contained inside the signature in some form.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: EhVedadoOAnonimato on December 30, 2011, 03:39:43 PM
But one thing that I don't understand is why are people so in a rush to implement client-server solutions, when a p2p lightweight solution is perfectly possible, and we already have BitcoinJ to demonstrate.

Because both technologies solve different problem. Bitcoin p2p is distributed database, however I'm trying to create protocol for *services* on top of this distributed database.

If you want privacy and security, you can every time use blockchain. If you need speed and convenience, use some overlay network, which masks compexity for you and provide only necessary service. It's giving freedom of choice, obviously.

Fair enough. I just want to emphasize again that you can have speed and convenience with a lightweight blockchain. Just see BitcoinJ, it runs even on mobile phones.

By the way, not all services offered by overlay must be related to blockchain and they cannot be provided in decentralized way. For example you probably want to pick some escrow service and use it's API directly or over trusted channel, not using some anonymous peer in p2p network.

True, not everything can be done on p2p. But why those different services need to be done by the same entity anyway? You could plug your MultiBit client on that escrow.


Anyway, I don't want to sound like a naysayer. You're all doing a great work and not asking anything in return. That's noble, my sincere congratulations.
I just like the lightweight p2p solution better, that's all.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: slush on December 30, 2011, 03:48:18 PM
I just want to emphasize again that you can have speed and convenience with a lightweight blockchain. Just see BitcoinJ, it runs even on mobile phones.

I have experience with BitcoinJ client on my android phone. Yes, it's better than downloading whole blockchain, that's without a discussion. But it's not *so* good as BitcoinSpinner client, which is lightweight. As and example, in bitcoinj I still need to wait to synchronize block headers, which is especially annoying when I'm using this wallet only time by time for face2face transactions. And lightweight clients cleanly wins in such use cases...

Personally I love freedom of choice, when everybody can choose solution which fits his needs. And I see that infrastructure for lightweight and ultra-lightweight clients is still weak, so I'm trying to fix it. Then integration into existing applications will become extremely easy, because those applications won't need to care about blockchain at all, just use some internet service with standard API which will *completely* mask the complex world of bitcoin.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: kjj on December 30, 2011, 04:20:43 PM
For the usual case, you don't care that multiple messages might be equally valid.  The signature is for "this message", not for "any message bearing the same content as this message".

I probably failed I expressed my worries. So I'll try again.

Main benefit of JSON is that it can be serialized/deserialized in almost every language in transparent way, just obj = json_decode(ser_data). However, those json_encode/decode implementations don't care about order of variables in serialized data, because order isn't a part of json specification.

So json_encode(json_decode(data)) can provide different stream than original data, which is necessary for validating json message. Thanks to this issue, you need to fiddle with serialized data directly as with text stream and remove signature-related data from this stream, instead of deserializing message, removing signature-related data, serializing it back and checking signature.

No, I get it.  What I'm saying is that you need another function, json_encode_canonical() such that canonical_data = json_encode_canonical(json_decode(arbitrary_data)) and you need to implement it the same way on all platforms, so that when you calculate digital_signature(canonical_data) you are always doing it on the same data.  The json_encode_canonical() function can even ignore fields that don't factor into the signature.

Quote
As for the ID, don't leave it out of the signature unless you enjoy replay attacks.  Just force the client to make it globally unique.  There are already well established techniques for doing this.

As a prevention for replay attacks I'm proposing "timestamp" field. It's also the reason why I think that request should be contained inside the signature in some form.

Timestamps have their own problems.  Either the clocks need to be synchronized very well, or you need a window of validity, but that window of validity is also a window of vulnerability.  You'd think that synchronizing clocks would be an easy assumption, but it turns out not to be at all.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: ripper234 on December 30, 2011, 04:35:33 PM
slush, how would the transaction mechanism work?

Creating transactions is done in few steps:

1. Choose destination address (or addresses)
2. Choose source addresses (=coin selection)
3. Calculate transaction fee
4. Sign inputs
5. Combine everything into transaction as a stream.

All those steps must be addressed in bitcoinj already, because those are necessary steps for every client. Those steps are implemented also in Electrum client (python).

However things can be slightly easier, because steps 2,3 and 5 can be done on server side (it's usually called super lightweight client, because it depends on server even with coin selection). BitcoinSpinner (java based android client) is doing it already.

Thanks, I asked this on Stack Exchange (http://bitcoin.stackexchange.com/questions/2357/how-do-i-generaet-offline-transactions-in-java) and emailed Mike Hearn, so I imagine I'll have some direction soon enough.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: ripper234 on December 30, 2011, 04:39:11 PM
Fair enough. I just want to emphasize again that you can have speed and convenience with a lightweight blockchain. Just see BitcoinJ, it runs even on mobile phones.

To developers it's not convenient enough. I wanted to use bitcoinj for a java web app, but then discovered I would have to maintain the blockchain in order to use it. This means consuming a lot of traffic for a simple web app that will be idle 99.9% of the time.

Having an external API can make me practically stateless - I just have to store keys, that's all - which simplifies my life considerably.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: ripper234 on December 30, 2011, 04:41:36 PM
Timestamps have their own problems.  Either the clocks need to be synchronized very well, or you need a window of validity, but that window of validity is also a window of vulnerability.  You'd think that synchronizing clocks would be an easy assumption, but it turns out not to be at all.

What types of API call are vulnerable to replay anyway? Transactions can only be made once, and once they're made there's no use trying to replay them. Since a malicious server can't do anything too serious except lie in responses or delay things, a malicious traffic sniffer can't do anything worse in a replay attack.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: kjj on December 30, 2011, 06:47:25 PM
Timestamps have their own problems.  Either the clocks need to be synchronized very well, or you need a window of validity, but that window of validity is also a window of vulnerability.  You'd think that synchronizing clocks would be an easy assumption, but it turns out not to be at all.

What types of API call are vulnerable to replay anyway? Transactions can only be made once, and once they're made there's no use trying to replay them. Since a malicious server can't do anything too serious except lie in responses or delay things, a malicious traffic sniffer can't do anything worse in a replay attack.

Until someone adds a new RPC call that actually is dangerous.  And why not add some?  The messages are signed and secure.

A system that is 90% safe is a trap waiting to spring on the first guy that forgets about the other 10%.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: Red Emerald on December 30, 2011, 09:06:11 PM
Fair enough. I just want to emphasize again that you can have speed and convenience with a lightweight blockchain. Just see BitcoinJ, it runs even on mobile phones.

To developers it's not convenient enough. I wanted to use bitcoinj for a java web app, but then discovered I would have to maintain the blockchain in order to use it. This means consuming a lot of traffic for a simple web app that will be idle 99.9% of the time.

Having an external API can make me practically stateless - I just have to store keys, that's all - which simplifies my life considerably.
This.

I'm not interested in my thin client downloading and saving anything.  I'm working on a phonegap application that is pretty much just one html file, one css file, and one javascript file.  Private keys are stored encrypted in local storage (like GLBSE) or generated deterministically when the client is opened (we'll see how well js can do this /me crosses fingers). There is no room to download even a lightweight blockchain.

As far as signing, we should look at how other people have done it.  OAuth alphabetizes the fields and then does some more. I don't like the idea of having to implement a "canonical_encode" but it looks like it might be necessary.

http://stackoverflow.com/questions/4670494/how-to-cryptographically-hash-a-json-object


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: slush on December 30, 2011, 10:02:10 PM
http://stackoverflow.com/questions/4670494/how-to-cryptographically-hash-a-json-object

Thanks for great link, it's exactly the answer what I has been searching for. It's actually very similar to the solution which I discussed with genjix on IRC, but it's nice to see that others found same solution independently. It looks that I'll choose those conventions described on stackexchange. It is slightly complicating all the signing the stuff, because it de-facto needs custom object encoder. However it is necessary to implement only in clients which require signing of proxied calls; common clients (using only blockchain service directly on overlay server) probably won't care about signing at all.

Edit: excellent, I found that "canonical JSON" is widely used term and there are some implementations in various languages already, like jsonical for pyhon (http://pypi.python.org/pypi/jsonical/). I just need to test if those implementations are compatible, but it looks like step in good direction...


Title: Re: [proposal] Stratum, overlay network protocol over Bitcoin
Post by: slush on December 30, 2011, 10:45:40 PM
Btw I closed contest about name for this overlay network. I picked name "Stratum", which is latin word for "layer".


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: Red Emerald on December 30, 2011, 10:48:30 PM
http://stackoverflow.com/questions/4670494/how-to-cryptographically-hash-a-json-object

Thanks for great link, it's exactly the answer what I has been searching for. It's actually very similar to the solution which I discussed with genjix on IRC, but it's nice to see that others found same solution independently. It looks that I'll choose those conventions described on stackexchange. It is slightly complicating all the signing the stuff, because it de-facto needs custom object encoder. However it is necessary to implement only in clients which require signing of proxied calls; common clients (using only blockchain service directly on overlay server) probably won't care about signing at all.

Edit: excellent, I found that "canonical JSON" is widely used term and there are some implementations in various languages already, like jsonical for pyhon (http://pypi.python.org/pypi/jsonical/). I just need to test if those implementations are compatible, but it looks like step in good direction...

This looks interesting too.  It doesn't seem to match some of the other specs I've seen though.

https://github.com/jchris/canonical-json#readme


Title: Re: [proposal] Stratum, overlay network protocol over Bitcoin
Post by: ripper234 on December 31, 2011, 04:44:21 AM
BTW, I posted this thread on reddit (http://www.reddit.com/r/Bitcoin/comments/nu5li/proposal_overlay_network_protocol_over_bitcoin/) and it got downvoted for some reason.
(Some) Bitcoin lovers don't understand that for Bitcoin to succeed it can't remain just a P2P network used by hackers and computer geeks.


Title: Re: [proposal] [Stratum] Overlay network protocol over Bitcoin
Post by: slush on December 31, 2011, 01:44:40 PM
(Some) Bitcoin lovers don't understand that for Bitcoin to succeed it can't remain just a P2P network used by hackers and computer geeks.

Don't worry, I already see there's some interest for such solution even between developers. And I'm sure that people (=users) will love solutions based on lightweight clients (and developers will be enjoying writing websites using Stratum API).


Title: Re: [proposal] [Stratum] Overlay network protocol over Bitcoin
Post by: ripper234 on December 31, 2011, 02:31:17 PM
(Some) Bitcoin lovers don't understand that for Bitcoin to succeed it can't remain just a P2P network used by hackers and computer geeks.

Don't worry, I already see there's some interest for such solution even between developers. And I'm sure that people (=users) will love solutions based on lightweight clients (and developers will be enjoying writing websites using Stratum API).

BTW, you can checkout my initial skeleton of an java binding here (https://github.com/ripper234/Bitcoin-Potato/tree/master/app/org/bitcoin/stratum) - subject to change, of course.


Title: Re: [proposal] [Stratum] Overlay network protocol over Bitcoin
Post by: slush on December 31, 2011, 03:22:04 PM
You can checkout my initial skeleton of an java binding here (https://github.com/ripper234/Bitcoin-Potato/tree/master/app/org/bitcoin/stratum) - subject to change, of course.

Excellent! Are you going to implement HTTP transport (push or poll?) or TCP socket transport? I encourage you to implement TCP socket because of lower overhead, although I understand that HTTP can be easier to implement.

Btw yesterday I implemented message signing and succesfully tested HTTP Poll and Push transports, although Google document isn't updated yet (thanks to 2112 suggestions, I added Content-Length, Content-MD5 and X-Content-Sha256 headers, there's also X-Push-Url for turning HTTP Poll transport into HTTP Push).

I'll try to commit all changes soon so you'll be able to test your client against working server, but I still need to cleanup the code. If you have any questions or suggestions, join #stratum channel anytime!


Title: Re: [proposal] [Stratum] Overlay network protocol over Bitcoin
Post by: ripper234 on December 31, 2011, 03:48:01 PM
You can checkout my initial skeleton of an java binding here (https://github.com/ripper234/Bitcoin-Potato/tree/master/app/org/bitcoin/stratum) - subject to change, of course.

Excellent! Are you going to implement HTTP transport (push or poll?) or TCP socket transport? I encourage you to implement TCP socket because of lower overhead, although I understand that HTTP can be easier to implement.

Btw yesterday I implemented message signing and succesfully tested HTTP Poll and Push transports, although Google document isn't updated yet (thanks to 2112 suggestions, I added Content-Length, Content-MD5 and X-Content-Sha256 headers, there's also X-Push-Url for turning HTTP Poll transport into HTTP Push).

I'll try to commit all changes soon so you'll be able to test your client against working server, but I still need to cleanup the code. If you have any questions or suggestions, join #stratum channel anytime!

I'll do HTTP, efficiency really isn't relevant to my use case, and leave the TCP implementation for the future.

I don't really use IRC a lot. Do you plan to have something more than a google doc page (e.g. a homepage, wiki, reddit, ...) ?

I still don't know exactly how to generate the offline TX, and haven't yet gone over your doc - sadly the work week is starting tomorrow for me (Israeli work weeks are Sunday-Thursday), and I won't have time to work on it during the week. Perhaps next weekend.

I'm really looking forward to getting an alpha implementation up & running.


Title: Re: [proposal] [Stratum] Overlay network protocol over Bitcoin
Post by: slush on December 31, 2011, 04:06:10 PM
Excellent! Are you going to implement HTTP transport (push or poll?) or TCP socket tra
I'll do HTTP, efficiency really isn't relevant to my use case, and leave the TCP implementation for the future.

Great. Keeping in mind that there might be more transports is always good idea.

Quote
I don't really use IRC a lot. Do you plan to have something more than a google doc page (e.g. a homepage, wiki, reddit, ...) ?

If you don't use any IRC client, webchat inteface can do the job for you: http://webchat.freenode.net/?randomnick=0&channels=#stratum&prompt=1

I already tried to install MediaWiki to my server, unfortunately I have some missing dependencies on the server and didn't find a time to fix it yet.  Then I move specification from Google docs there.

Homepage at stratum.bitcoin.cz will appear at some time, too. At this moment there's not a lot things to publish anyway, there's even not any server release or even installation howto :-). I'm focusing on getting it work right now...

Quote
I still don't know exactly how to generate the offline TX, and haven't yet gone over your doc - sadly

Honestly, I'm not a specialist in bitcoin p2p protocol, so I'm always finding inspiration in some existing code. For creating transactions, Electrum client is good code to start for me. However if you're Java programmer, BitcoinJ should be good choice for you and I see you're already using some code from bitcoinj...


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: 2112 on January 01, 2012, 09:46:32 PM
Hi slush!

I'm fully aware that those requirements are contradictory. It's pretty common that some solution needs to find some compromises.
After waiting a while and reading the replies and comments to your proposal I now understand your approach and my mistakes. Basically I only took a helicopter tour of Klondike, but you are the one who has a ground floor experience of Klondike. I'm kinda living a sheltered life: I'm never on the first line when talking to my customers: before they talk to me they need to jump a trench or two in my organization. I'm not avoiding my customers, it just that the ones I interact with were already filtered. This is completely unlike you: are the one who speaks to every customer.

I now understand that there's no point of trying to require advanced features like pair-of-FSMs design if the potential users are only familiar and comfortable with RPC-style designs. I searched this board for any sign of anyone writing about state-of-the-art financial services technology. There wasn't anyone who would mention things like CICS,Tuxedo or even no-cost MTS & MSDTC.

If the residents of Klondike demand lighter pickaxes there's no point of offering hydraulicking.

Quote
Although  I personally dislike XML, I'm open to change my mind at this point.
I wouldn't recommend switching to XML. What I was trying to suggest is to try re-host some features of XML-based protocols on top of JSON or whatever else you choose. I think you already doing that. The main advantage of SOAP is that first-line tech support person can say things like "Parasoft SOAtest agrees with our diagnosis, the errors are on your side" and avoid the cost of escalated support calls.

Quote
Isn't TCP checksumming and TCP retransmissions on both ends enough to "fix" corrupted information?
Well, NAT breaks the end-to-end principle of IP networking. But the real culprits are usually various marketing-motivated enhancements like SPI (stateful packet inspection); "gaming mode"; "multimedia prioritization", etc.

Quote
I feel like overlay network should be "stateless", which mean that it will "forget" transaction once transaction has been succesfully broadcasted to Bitcoin network.
The "stateless" discussions remind me of "zero configuration" discussions in other areas. After all the late-discovered problems get fixed the "stateless" design is way more complex and fragile than the competing "stateful" designs.

My general suggestions can be summarized as follows:
1) store coins with as key pairs plus the block hash and height in chain
2) client connecting to server send a chain length and tip block-hash of the last connection
3) server responds with positive difference in height and new tip block-hash
4) in case of the reorg server responds with negative difference in height and hash of last block pre-fork followed by the same information as in (3) for the new branch.
5) client can recalculate the wallet and retry the transactions that still have valid inputs after the reorg
6) server takes a proactive retrying approach to the transactions it originated. This is particularly important in case of the reorg and when the originating client already disconnected.

I didn't do a complete http://en.wikipedia.org/wiki/Transaction_processing analysis, but there is a number of possible compromises in the spectrum between a fully ACID-compliant design and a ball-of-rubberbands design.


Title: Re: [proposal] [Stratum] Overlay network protocol over Bitcoin
Post by: marcus_of_augustus on January 02, 2012, 02:39:31 AM
watching

http://latex.codecogs.com/gif.latex?g%282z%29%20%3D%20%5Ccos%20z%20&plus;%20%5Csin%20z


Title: Re: [proposal] [Stratum] Overlay network protocol over Bitcoin
Post by: omar on January 02, 2012, 05:38:17 AM
A few months ago I wrote a paper about the Bitcoin Gateway, providing the design of a P2P network over the core bitcoin network and a thin client that uses this network to provide a fast, safe and easy to use experience. It essentially makes using bitcoin as easy as using PayPal. I am currently building a team to implement this and hope to release it in 2012 as an open source project. You can read more about it here:
    http://arimaa.com/bitcoin/

If you would be interested to join the team to implement this, please let me know.


Title: Re: [proposal] [Stratum] Overlay network protocol over Bitcoin
Post by: Red Emerald on January 02, 2012, 05:58:55 AM
Should some kind of secure wallet storage be included in the proposal?  This could be used to keep track of any imported keys.  Of course, deterministically generated keys should never been saved.  Servers could optionally implement it if they don't want to deal with maintaining the database.

I think a rock-solid, secure, possibly distributable key storage could be a very useful feature.


Title: Re: [proposal] [Stratum] Overlay network protocol over Bitcoin
Post by: ripper234 on January 02, 2012, 07:34:48 AM
Should some kind of secure wallet storage be included in the proposal?  This could be used to keep track of any imported keys.  Of course, deterministically generated keys should never been saved.  Servers could optionally implement it if they don't want to deal with maintaining the database.

I think a rock-solid, secure, possibly distributable key storage could be a very useful feature.

+1.

If Stratum clients could remember just a single strong password instead of manage keys themselves, that would simplify client implementations even further.

The key storage has to be distributed for this to be useful though. If there are only 1-3 Stratum servers, nobody would dare hold any semi-serious money there, because simply shutting down the servers denies them of money.

A deterministic wallet can be used by clients ... but not all client implementations are deterministic today, and those usually don't store other data like address labels.

The keys+labels could be possibly stored in the blockchain itself (it's a good durable p2p storage device that's always available).

Anyway, this is certainly not in the MVP (http://en.wikipedia.org/wiki/Minimum_viable_product) of Startum.


Title: Re: [proposal] [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 02, 2012, 04:00:15 PM
Anyway, this is certainly not in the MVP (http://en.wikipedia.org/wiki/Minimum_viable_product) of Startum.

This.

Actually I don't care about all potential services which can be done on top of Stratum protocol at this time, but server side storage of seeds/keys can be done. In such case, ultramegalightweight client using such feature don't need to implement any crypto part, just use Stratum server as a standard web wallet, if he wants.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: 2112 on January 02, 2012, 06:24:05 PM
I'm sorry, due to my copy/paste error this fragment didn't get included in my earlier post.
Side note: I'm running json-based protocol on the pool over a year, I had over 3300 rq/s in June peak. [...] But I definitely didn't have any problems with corrupted packets like you're suggesting.
This is probably due to three factors:

1) you collect only one-sided statistics
2) the mining protocol on your end is extraordinarily simple: "getwork" or "getwork <hexstring>". Thus probably some corruptions show as "stales", "invalids" or some such.
3) on your end mining protocol has shorter packets. To really see the errors the amount of data exchanged needs to be larger that TCP/IP MSS (maximum segment size), which in turn depends on MTU (maximum transmission unit)

I only ever solo-mined test coins, so I'm not really familiar with the details of pooled mining. For solo-mining without a pool server the error detection is nonexistent.


Title: Re: [proposal] [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 03, 2012, 02:56:44 AM
Few moments ago I commited some stuff. Http transport is still under construction.

As a homework I implemented broadcasting transaction to Bitcoin network, it's method "blockchain.transactions.broadcast". It is using ArtForz's halfnode code for talking with trusted Bitcoin node over P2P. However no patch for bitcoind is necessary, which is significant improvement against current Electrum server implementation ;).


Title: Re: [proposal] [Stratum] Overlay network protocol over Bitcoin
Post by: Jan on January 04, 2012, 12:53:56 PM
...I still don't know exactly how to generate the offline TX...

The BCCAPI (http://code.google.com/p/bccapi/) which BitcoinSpinner is built on top of has all the Java classes you need to do off-line transaction validation/signing and deterministic key derivation etc. Look for Tx.java, DeterministicECKeyManager.java and Account.signSendCoinForm(). This should give you a good head-start.

Regarding the long protocol discussion: Since you are addressing mobile devices that are going to connect over a wide variety of wi-fi networks through god knows how many firewalls you should really reconsider to use your own protocol directly on top of TCP. It is not uncommon for firewalls to block anything but port 80 and 443, and there are many tricks and tools that can come in handy if you use HTTP(S)

This is a really interesting project, and I wish you good luck. Unfortunately I am too tied up to participate actively.


Title: Re: [proposal] [Stratum] Overlay network protocol over Bitcoin
Post by: ripper234 on January 04, 2012, 12:55:46 PM
...I still don't know exactly how to generate the offline TX...

The BCCAPI (http://code.google.com/p/bccapi/) which BitcoinSpinner is built on top of has all the Java classes you need to do off-line transaction validation/signing and deterministic key derivation etc. Look for Tx.java, DeterministicECKeyManager.java and Account.signSendCoinForm(). This should give you a good head-start.

Regarding the long protocol discussion: Since you are addressing mobile devices that are going to connect over a wide variety of wi-fi networks through god knows how many firewalls you should really reconsider to use your own protocol directly on top of TCP. It is not uncommon for firewalls to block anything but port 80 and 443, and there are many tricks and tools that can come in handy if you use HTTP(S)

This is a really interesting project, and I wish you good luck. Unfortunately I am too tied up to participate actively.

+1, thanks Jan!
I'll definitely check out bccapi.


Title: Re: [proposal] [Stratum] Overlay network protocol over Bitcoin
Post by: casascius on January 04, 2012, 04:22:49 PM
I am not sure if this has been discussed, but one important purpose I foresee of any overlay network is a method of communicating recommended transaction fee policies to clients, so that miners have a way of conveying to clients what transaction fees they should expect to pay to get their transactions into blocks.  The whole 0.0005 hardcoded bit isn't going to make sense most of the time.


Title: Re: [proposal] [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 04, 2012, 06:21:12 PM
The whole 0.0005 hardcoded bit isn't going to make sense most of the time.

Method blockchain.transaction.guess_fee() (https://www.gitorious.org/stratum/server/blobs/master/server_repository/blockchain/__init__.py) is addressing exactly this. At this moment there's 0.0005 constant, but I have an algorithm for calculating recommented fee in my head already...


Title: Re: [proposal] [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 04, 2012, 06:24:10 PM
Regarding the long protocol discussion: Since you are addressing mobile devices that are going to connect over a wide variety of wi-fi networks through god knows how many firewalls you should really reconsider to use your own protocol directly on top of TCP. It is not uncommon for firewalls to block anything but port 80 and 443, and there are many tricks and tools that can come in handy if you use HTTP(S)

There's no need to reconsidering anything. TCP socket is the most elegant and optimized way to communicate over network, so I'm recommending to use TCP socket where it is possible, but I'm implementing HTTP polling (https://www.gitorious.org/stratum/server/blobs/master/httppoll_transport.py), so mobile devices can work with Stratum nicely.


Title: Re: [proposal] [Stratum] Overlay network protocol over Bitcoin
Post by: MoonShadow on January 04, 2012, 06:34:54 PM
The whole 0.0005 hardcoded bit isn't going to make sense most of the time.

Method blockchain.transaction.guess_fee() (https://www.gitorious.org/stratum/server/blobs/master/server_repository/blockchain/__init__.py) is addressing exactly this. At this moment there's 0.0005 constant, but I have an algorithm for calculating recommented fee in my head already...

Could you provide a rough, layman's explaination of that algo please?


Title: Re: [proposal] [Stratum] Overlay network protocol over Bitcoin
Post by: MoonShadow on January 04, 2012, 06:38:24 PM
What are the odds that this overlay network could be used to permit two mobile devices to transact directly, in the absence of Internet access for one or both parties?  For example, a direct ad-hoc wifi connection or an indirect connection via a 'piratebox' type device?  Asuuming, of course, that the sending device doesn't need Internet access to produce a transaction from it's own, known address funds.


Title: Re: [proposal] [Stratum] Overlay network protocol over Bitcoin
Post by: kjj on January 04, 2012, 06:52:34 PM
I don't think an algorithm is going to cut it for fees, at least not in the long run.  In the long run, miners will need to provide quotes.


Title: Re: [proposal] [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 04, 2012, 07:01:13 PM
Could you provide a rough, layman's explaination of that algo please?

Basic idea is to measure the relation between fees in transactions included in few last blocks and fees in pending transactions (assuming that transactions which are pending for some long time and they're not included in the blockchain have lower fees than necessary).

But I agree that some mechanism for negotiating required fees with miners would be interesting, too.


Title: Re: [proposal] [Stratum] Overlay network protocol over Bitcoin
Post by: Red Emerald on January 04, 2012, 07:03:56 PM
But I agree that some mechanism for negotiating required fees with miners would be interesting, too.
If only we knew someone with a pool to experiment with  ;)


Title: Re: [proposal] [Stratum] Overlay network protocol over Bitcoin
Post by: 2112 on January 04, 2012, 07:26:50 PM
But I agree that some mechanism for negotiating required fees with miners would be interesting, too.
It may be interesting from the anthropological point-of-view: who's gonna fall for it. Technically the negotiation for inclusion in a block doesn't make sense: the blocks are won randomly. I guess from the game-theoretical point-of-view one could come up with an optimal strategy: clairvoyance.


Title: Re: [proposal] [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 04, 2012, 07:30:08 PM
I agree that such negotiation can turn into pretty complicated thing. However, at this moment, at least me and deepbit is trying to include as much free transactions as possible, for marketing purposes ("Bitcoin is providing free transactions").


Title: Re: [proposal] [Stratum] Overlay network protocol over Bitcoin
Post by: kjj on January 04, 2012, 07:42:27 PM
But I agree that some mechanism for negotiating required fees with miners would be interesting, too.
It may be interesting from the anthropological point-of-view: who's gonna fall for it. Technically the negotiation for inclusion in a block doesn't make sense: the blocks are won randomly. I guess from the game-theoretical point-of-view one could come up with an optimal strategy: clairvoyance.

I think the quote would be along the lines of: "We have averaged X blocks per day over the last week.  We expect our next block to happen within Y hours.  If you get your transaction to us in the next M minutes and it includes a fee of at least Z, we will include it in our next block (which should happen within 2*Y hours).  To ensure that your transaction reaches us, please submit it directly to our node at address A.B.C.D, port E."

The quote doesn't promise that the miner is going to get the next block, they are only promising that if they do get the next block, your transaction will be in it, provided that the fee meets their criteria.

In a busy world, I would expect that the biggest miners (the ones that average like one block per hour or more) will start ignoring low fee transactions, while the smaller miners would accept a lot more.  A spender could check with a bunch of miners and make the time/fee trade off that they want.

Deepbit wants 0.01 BTC to include it in the next 30 minutes, but Joe's Mining Pool and Bait Shop will do it for 0.001 BTC in the next day.  What is more important to you, time or money?


Title: Re: [proposal] [Stratum] Overlay network protocol over Bitcoin
Post by: 2112 on January 04, 2012, 08:28:51 PM
A spender could check with a bunch of miners and make the time/fee trade off that they want.
I'll pretty much believe that in the Bitcoin milieu anything is possible: if somebody builds a "pig in a poke" ("Katze im Sack") market it will find willing participants. Those phrases became proverbial because there was a continuous demand for it throughout the history. There will be natural extensions for them implemented using Bitcoin. That's what makes Bitcoin such interesting from the anthropological perspective. 

Quote
Deepbit wants 0.01 BTC to include it in the next 30 minutes, but Joe's Mining Pool and Bait Shop will do it for 0.001 BTC in the next day.
What a charming old-fashioned prediction! Deepbit will merge with an exchange and it will offer negative effective transaction fees to the users of the exchange.

Joe will offer "Mining Pool and Guaranteed-payback Three-card Monte": that's the market demand amongst the Bitcoin miners.


Title: Re: [proposal] [Stratum] Overlay network protocol over Bitcoin
Post by: MoonShadow on January 04, 2012, 08:46:45 PM

Quote
Deepbit wants 0.01 BTC to include it in the next 30 minutes, but Joe's Mining Pool and Bait Shop will do it for 0.001 BTC in the next day.
What a charming old-fashioned prediction! Deepbit will merge with an exchange and it will offer negative effective transaction fees to the users of the exchange.

Joe will offer "Mining Pool and Guaranteed-payback Three-card Monte": that's the market demand amongst the Bitcoin miners.


This is an utterly rediculous statement.


Title: Re: [proposal] [Stratum] Overlay network protocol over Bitcoin
Post by: casascius on January 04, 2012, 09:29:28 PM
I assume that part of an overlay network would be to provide a means for nodes to tell one another about one another's existence (e.g. I'm a bitcoin node accepting incoming connections at 1.1.1.1:8333).  Also it would be to exchange signed messages of useful value (e.g. signed message from deepbit: I have seen block 123456, signed message from MtGox: rates as of 12/12/12 11:11:11: buy 11.11 sell 11.11 etc).

If that infrastructure were in place, it wouldn't be much of a stretch for a signed message from deepbit to include some sort of message broadcasting its fee schedule, or broadcasting the url of an rpc that can give a fee quote for any given proposed transaction.


Title: Re: [proposal] [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 05, 2012, 12:47:50 AM
I assume that part of an overlay network would be to provide a means for nodes to tell one another about one another's existence (e.g. I'm a bitcoin node accepting incoming connections at 1.1.1.1:8333).  Also it would be to exchange signed messages of useful value (e.g. signed message from deepbit: I have seen block 123456, signed message from MtGox: rates as of 12/12/12 11:11:11: buy 11.11 sell 11.11 etc).

Yes, signing and relaying (proxying) signed messages is already possible, exactly for transmitting messages like exchange rates.


Title: Re: [proposal] [Stratum] Overlay network protocol over Bitcoin
Post by: HostFat on January 05, 2012, 01:17:45 AM
Are you still working on a BitcoinSpinner version compatible with Electrum servers?


Title: Re: [proposal] [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 05, 2012, 01:25:36 AM
I'm not working personally on that, but friend of mine is. However I don't know actual status of his project...


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 05, 2012, 03:26:14 PM
Quick performance test of Stratum server implementation on my Asus EEE netbook:

TCP transport: cca 450 rq/s
HTTP transport: cca 130 rq/s

Every request is one roundtrip, including serialization and deserialization of request/response in testing client, so server part itself is at least 2x faster. However the overhead of HTTP layer is significant, as I expected. I compared those numbers with bitcoind's RPC (getblockcount, which is basically blank RPC request): ~100 rq/s.

Asus EEE is lowend machine and this test is calling RPC method in serial, so the test is far from real-world use, but I'm expecting at least 2x higher performance of this test per CPU core on standard server machine.


Title: Re: [proposal] Overlay network protocol over Bitcoin
Post by: jetmine on January 05, 2012, 04:51:54 PM
client -> (some transport) -> overlay server -> HTTP (firstbits.com don't support https yet!) -> firstbits.com

[...]

I has been thinking about using GPG mechanism, which is widely used platform even for distributing trust, but it looks prettyheavy for such purpose. Not sure if it's possible to check GPG signatures in $3 AVR processor.

[...]

However I see real issue in needed CPU time for signing. On my laptop, one signature takes around 200ms. It's low-end one, but it will take some time even on high-end servers. Any idea how to solve it?

The obvious answer to this question is:

a) Use an algorithm which allows signature verification with low CPU requirements.  RSA with E=3 is one such algorithm.  It is possible to verify a signature on a low-end CPU like AVR without problems.

b) On the server side, scale up as required to meet the desired signature creation throughput.  See the next item for how this is not necessary (except maybe when you plan to run your server on AVR too).

c) Include into the signature only fields that are essential.  Firstbits is assumed to be unique and will not change throughout the lifetime of bitcoin.  Therefore the essential fields are 1) Firstbits address, 2) Full address, 3) Who signed this (you), 4) Signature version/date/generation method.  What else is necessary?  I believe nothing else (but I have never used firstbits so I cant be sure).  With this, you can generate the signatures as requests trickle in, and cache them on your server.  When an address is requested multiple times, you can satisfy from the cache (without further CPU effort).  Soon you will have all (relevant) addresses in your cache and will not have to sign any new addresses, at least until the the blockchain grows (which happens at a predictable rate).

Good luck


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 05, 2012, 05:21:11 PM
jetmine, thanks for inputs. Quick answer:

a) Use an algorithm which allows signature verification with low CPU requirements.  RSA with E=3 is one such algorithm.  It is possible to verify a signature on a low-end CPU like AVR without problems.

Signing will be probably necessary only for some type of messages so the speed of verifying on the client won't be crucial. I decided to implement ECDSA signatures because ECDSA library will be presented on the client anyway for signing transactions, which eliminate yet another dependency.

Quote
c) Include into the signature only fields that are essential.  Firstbits is assumed to be unique and will not change throughout the lifetime of bitcoin.

Actually I did the oposite, I added timestamp into the signed data, to protect system against replay attacks. However now I'm rethinking the concept, as the same request should lead into the same result, at any time. Those APIs where replay attack is potentially dangerous should add timestamp as the part of the request (signature is created from both request and response).

So - thanks for suggestion!

Quote
and cache them on your server

This can be doable with some type of messages even with timestamp in the current implementation, for messages like market quotes.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 05, 2012, 07:22:58 PM
I updated Google document witch chapter describing HTTP Poll / Push transports. Any suggestions?

I'm also thinking about HTTP Long-polling transport (it's actually blocking extension of HTTP Poll transport, which enable realtime communication from server to client over HTTP without any need of polling or callback URL), but I want to provide long polling over the same connection, so I need to test HTTP pipelining firstly. In the theory HTTP pipelining should be working already in the Stratum implementation, but not so much client libraries support that...

ripper234, you were interested in implementing Stratum client library in Java with HTTP transport. Is current specification detailed enough for finish the job from your side? Everything is already implemented and commited into the Git so it should be very easy to install and run locally, but I'll provide Stratum instance today or tomorrow for more comfortable development.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: ripper234 on January 05, 2012, 07:25:40 PM
I updated Google document witch chapter describing HTTP Poll / Push transports. Any suggestions?

I'm also thinking about HTTP Long-polling transport (it's actually blocking extension of HTTP Poll transport, which enable realtime communication from server to client over HTTP without any need of polling or callback URL), but I want to provide long polling over the same connection, so I need to test HTTP pipelining firstly. In the theory HTTP pipelining should be working already in the Stratum implementation, but not so much client libraries support that...

ripper234, you were interested in implementing Stratum client library in Java with HTTP transport. Is current specification detailed enough for finish the job from your side? Everything is already implemented and commited into the Git so it should be very easy to install and run locally, but I'll provide Stratum instance today or tomorrow for more comfortable development.

I might have some time to look at this next weekend, but probably not before :(


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 05, 2012, 08:01:08 PM
I'm proud to announce first instance of Stratum server on stratum.bitcoin.cz, ports 3333 (TCP transport) and 8001 (HTTP transport). Although there's not a documentation about exposed services, some samples in client.py and client2.py together with source codes in /server_repository/ (https://gitorious.org/stratum/server/trees/master/server_repository) directory may be useful for hardcore hackers.

In the near future I'll configure Nginx to handle HTTP transport on port 80 and add SSL certificate to enable HTTP transport over SSL on port 443, but this should be enough as a playground for some interested parties.

Almost everything on the protocol level is implemented and I'm interested in any feedback, especially in using the HTTP Poll and Push transports.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: Red Emerald on January 05, 2012, 08:46:04 PM
How much overlap is there between this and electrum?  I noticed in server.tac that the application is called "electrum-server" but it looks like the code is being written separately.  BTW, Your code looks really clean.

I'll play around with getting some javascript to connect to the server when I have time.  Right now I'm working on a cgminer monitor.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: jetmine on January 05, 2012, 09:31:18 PM
Those APIs where replay attack is potentially dangerous should add timestamp as the part of the request (signature is created from both request and response).

PS to the earlier said: Make sure you always include a signature generation or date into the signed portion, even when the API doesnt immediately require it.  In the firstbits example, imagine you would find a bug in the code someday and a few of the addresses were matched in a wrong way.  With date or generation information you could then "revoke" the old body of signed snippets by requiring date/generation > X both on the server (to trigger recreation of data+sig) as well as on the updated clients (as security measure).

As for market stats and other things you mentioned, of course you could also cache those pieces of information.  Just make up a time grid, in accordance to how often you poll the information yourself.  It doesnt make sense to sign it at a higher rate than they can change (which is almost entirely dependant on your pollrate).

You can then hand out the same signed piece(s) many times to many clients.

You can also hand out old (cached) pieces, to give historical information to the clients.  However, it makes sense to "compress" the many pieces of information within one timeperiod into a single (newly signed) packet once the timeperiod is over.  This reduces the amount of signature verifications on the client side, yet doesnt greatly increase the signature load on the server.

For example, after 15 minutes, you combine the 15 1-minute pieces into one 15-minute piece.  Every 4 hours you do the same.  Every day you combine the whole day, every week, every month and every year.  With every higher timeperiod you can also reduce the "resolution" to avoid excessively growing the size of the datasets.  The client can then easily assemble price information for practically every imaginable time ranges, with only few signature verifications.  All just from cached information and nothing individually signed ("just for them").

For moments where you cant query prices yourself (e.g. network problems), you could insert a packet saying so, and sign that.   The client can then check its local time and know that it has complete information from your server, despite the lack of data.

But make sure that you dont put too much effort on features that maybe arent useful later on.  If you poll prices from doubtful sources using insecure protocols and then sign it, the high quality signature from your server wont serve much purpose.  It may even be misleading then..

Always follow the KISS principle (Keep It Simple, Stupid).  In electronics we say that the best component is one that you can eliminate.  Once eliminated, the component will not have solder problems, it will not fail throughout the whole lifetime of the product, and it adds 0.00 to the BOM.  The same goes for software.  If you create an elegant design on the API level, your backend can be beautifully simple, and your clients can be too.  But its all up to knowing (in advance) where you want to go and what you want your software to do.

This is the hard part of it, and choosing pieces like signature algorithms just for "familiarity" can be a lottery in this respect.  Maybe such a tiny decision can bring your network down when it is supposed to just scale up nicely.  I dont know the details of what you want to do (and maybe you dont fully know either).  So I am sorry I cant give detailed advice.

Good luck with the project.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: BitcoinBug on January 05, 2012, 10:18:41 PM
I love this project. Following...


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 05, 2012, 10:33:36 PM
How much overlap is there between this and electrum?  I noticed in server.tac that the application is called "electrum-server" but it looks like the code is being written separately.

Yes, the codebase is completely different. Originally I started this project as a part of Electrum client, but then realize that it has some bigger potential than be just a backend for one particular client, so I rename it to Stratum not confuse people. Currently it shares the motivation with Electrum server and I'm also going to use same compontents for the initial implementation like Abe for blockchain indexing, but other things are different. Btw thanks for pointing me to "electrum-server", I fixed the typo :-).

Quote
BTW, Your code looks really clean.

Thanks, I'm simply not smart enough to write complicate code ;).


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 05, 2012, 10:42:43 PM
PS to the earlier said: Make sure you always include a signature generation or date into the signed portion, even when the API doesnt immediately require it.  In the firstbits example, imagine you would find a bug in the code someday and a few of the addresses were matched in a wrong way.  With date or generation information you could then "revoke" the old body of signed snippets by requiring date/generation > X both on the server (to trigger recreation of data+sig) as well as on the updated clients (as security measure).

Thanks for suggestion, I think it's good compromise - add timestamp of signing into the message, but use it just as an additional information and don't force clients to throw away messages with old timestamps. In this case client can decide if he cares about timestamp of signature generation or not.

Quote
For example, after 15 minutes, you combine the 15 1-minute pieces into one 15-minute piece.  Every 4 hours you do the same.  Every day you combine the whole day, every week, every month and every year.  With every higher timeperiod you can also reduce the "resolution" to avoid excessively growing the size of the datasets.  The client can then easily assemble price information for practically every imaginable time ranges, with only few signature verifications.  All just from cached information and nothing individually signed ("just for them").

Well, I feel this isn't the responsibility of the protocol itself, it's more about design of exposed service. As an example, one service can provide simple method for retrieving actual information and second method accepting some range specification as an input for providing historical data. Of course that response of such method with composite result will be signed only once. I think that don't include some extended algorithms to the protocol for solving edge cases is going exactly in the KISS way.

Edit: Timestamp is the part of signed message again, in enlightened form than original implementation.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 05, 2012, 11:41:02 PM
I just implemented support for specifying vendor in the RPC method name, in the form "some.service[vendor].method".

Longer answer, especially for potential service developers:
Stratum is designed in the way that one service may be provided by many independent vendors. As an example, every exchange provides only market quote which is actively traded there, obviously. In this case, Stratum server operator or even exchange itself can implement the service for broadcasting their quotes and let the client choose the vendor from which he wants to receive quotes.

1. There's already discovery service, which provides listing of available vendors for some specified service: discovery.list_vendors('exchange.quotes') may return ['mtgox.com', 'tradehill.com']

2. Listing of two vendors means that there are two implementations of ExchangeQuotesService (located in server_repository in current server implementation), which provides the same service API for both independent sites (mtgox provides websocket, tradehill only polling?). Those two classes may contain completely independent implementation, but acts as the same service for Stratum client.

3. Client wants to receive quotes from mtgox, so he call exchange.quotes[mtgox.com].subscribe('usd'), which tell Stratum server to start broadcasting USD quotes from mtgox service implementation. If the client wants to receive tradehill quotes instead, he call exchange.quotes[tradehill.com].subscribe('usd') instead, but he don't need to care that internal implementation is different.

Edit: When brackets with requested vendor aren't specified, Stratum will use default service for given service type (default service class is marked as default=True). In the example above, it could be bitcoincharts.com as general provider of all quotes (although using specific vendor binding have some benefits, bitcoincharts is just another layer which can be down or overloaded, like during yesterday peaks on mtgox).


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 06, 2012, 02:56:17 AM
I finally found some time to learn basics of IRC protocol. Thanks to it, I implemented IRC peer lookup mechanism into Stratum server. See node.get_peers() method...


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 06, 2012, 12:44:13 PM
I wrote this example of HTTP Push protocol for https://bitcointalk.org/index.php?topic=56801.msg677355#msg677355, but see it interesting enough to repost here, too:

Following HTTP call will subscribe URL http://yourdomain.com/callback.php for receiving notifications about 1YourAddress Bitcoin address. It may be useful for receiving realtime notifications for web application. Please note that method blockchain.address.subscribe is not fully implemented yet, it's just a protocol example. This subscription will expire in X-Session-Timeout seconds (presented in HTTP response), but can be prolonged by any other HTTP call with the given session ID (also returned in response):

Code:
POST / HTTP/1.1
Host: stratum.bitcoin.cz
Connection: close
Content-Type: application/stratum
Content-Length: 81
X-Callback-Url: http://yourdomain.com/callback.php

{"id": 1, "method": "blockchain.address.subscribe", "params": ["1YourAddress"]}

(Note: there must be newline ("\n") on the end of every JSON command).

Response:
Code:
HTTP/1.1 200 OK
Content-Length: 36
X-Session-Timeout: 3600
Server: Stratum/0.1
X-Content-Sha256: fe2a156e058307b4b7782e0b236cbd631c5bce3091f8800f818c91fcb850bfc3
Connection: close
Date: Fri, 06 Jan 2012 12:36:17 GMT
Content-MD5: b0d24c6c203d57e8a998be226a16a3c1
Content-Type: application/stratum
Set-Cookie: STRATUM_SESSION=e6ad15c26f0fe711ba632a7978a3ab30; Path=/

{"error":null,"id":1,"result":true}


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: ripper234 on January 06, 2012, 02:30:58 PM
slush - you should add a link to the Google Doc from the Stratum Homepage.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 06, 2012, 03:03:10 PM
Added. I hope that I'll configure wiki soon. Google docs did the job from the beginning, but I'd like to add some examples and Docs aren't the best choice anymore.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 07, 2012, 12:45:30 AM
I created repository for Stratum client implemented in PHP (https://gitorious.org/stratum/php-client). Although it is already doing RPC requests, response handling and handling of broadcast messages isn't finished yet. I didn't code PHP for few years, so please excuse if some constructs looks old-fashioned (I'm receiving merge requests :-) ).

The simplest possible usage (it is plain HTTP Poll) is following:

Code:
<?php
require_once('StratumClient.inc.php');

$c = new StratumClient('stratum.bitcoin.cz'8001);

$result '';
$result2 '';
$c->add_request('node.ping', array('ahoj'), $result);
$c->add_request('firstbits.resolve', array('1marek'), $result2);
$c->communicate();

var_dump($result);
var_dump($result2);

Method add_request() collects calls into internal buffer, but only communicate() performs real network transfer. This looks pretty unintuitive, but it's minimizing network overhead. With this mechanism, you can submit tens or hundreds transactions with single HTTP call. communicate() will also call proper callbacks when client accepts some broadcasted messages (not implemented yet).

I'm still not sure what is the best solution for exception handling. communicate() shouldn't raise an exception when some call failed, so I'm thinking that callback won't be the value itself, but just a wrapper object containing real value OR exception object. When result is firstly accessed by the application (var_dump in example above), the exception is raised. But I don't know if such construction is even possible in PHP... Any suggestions?

Edit: Implemented basic idea  of exposing services on the client. See example.php in the repo.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: ShadowOfHarbringer on January 07, 2012, 01:37:35 PM
/me is watching this


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 07, 2012, 02:27:29 PM
One guy asked me how to install the server, so I prepared short installation howto (https://gitorious.org/stratum/server/blobs/master/README).


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 09, 2012, 08:27:59 PM
Thanks to support from BtcVPS.net (http://btcvps.net) guys, second Stratum node is running!

Currently there are two nodes: london.stratum.bitcoin.cz (Linode, UK), chicago.stratum.bitcoin.cz (BtcVPS, US).


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: Red Emerald on January 09, 2012, 11:06:48 PM
Thanks to support from BtcVPS.net (http://btcvps.net) guys, second Stratum node is running!

Currently there are two nodes: london.stratum.bitcoin.cz (Linode, UK), chicago.stratum.bitcoin.cz (BtcVPS, US).

I just moved around a couple of my VMs and reworked a firewall.  I should have stratum.stitthappens.com (California, US) up soonish.  I'll play around with ports 80 and 443, but that might take a little longer to get online.

Is there a KNOWN_ISSUES file or something somewhere? What works so far? What doesn't?


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 10, 2012, 01:09:27 PM
I just moved around a couple of my VMs and reworked a firewall.  I should have stratum.stitthappens.com (California, US) up soonish.  I'll play around with ports 80 and 443, but that might take a little longer to get online.

Nice! Yesterday, during the installation of chicago instance I realized that root privileges are necessary for running software on ports < 1024 (well, I known it years ago, but I simply forgot). So I'll need to implement mechanism (+init script) which will downgrade privileges after startup from root to some normal user.

For now the cleanest solution is to install (for example) Nginx on 80/443 and use reverse proxy module to route traffic to Stratum instance running on >1024 ports. I think that most servers are using 80/443 ports already, so this proxied setup will be the most common use case anyway. For this purpose I defined "application/stratum" HTTP content type, so stratum traffic can be easily splitted by simple rules in http servers like Apache or Nginx.

Quote
Is there a KNOWN_ISSUES file or something somewhere? What works so far? What doesn't?

It isn't documented anywhere, but I'll write something to README file soon. So far protocol + TCP/HTTP trasport implementations are complete, some example services are already working. What isn't finished and what's important: blockchain service (which use indexed blockchain to ask for address history/balance) and library for broadcasting messages subscribed by client. Blockchain service is under development by my friend and I'll implement broadcasting messages very soon, so stay tuned :-).



Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: D.H. on January 10, 2012, 02:07:30 PM
Great project, slush. Keep up the good work!


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: Red Emerald on January 10, 2012, 11:17:21 PM
I've got a server up on stratum.stitthappens.com.  Right now just 3333.  I haven't setup nginx to do any proxying yet, although that won't take long.

I don't like using apt to manage my python packages and I also prefer pip over easy_install.  I used Doug Hellmann's virtualenvwrapper, but virtualenv is also easy to use. 

Code:
sudo apt-get install python-dev python-pip python-virtualenv
mkdir stratum && cd stratum
git clone git://gitorious.org/stratum/server.git
cd server
virtualenv --no-site-packages --distribute env
source env/bin/activate
pip install twisted ecdsa pyopenssl pycrypto
cp settings.sample.py settings.py
vim settings.py
./signature.py > signing_key.pem
cat signing_key.pem
sudo ufw allow 3333
twistd -ny server.tac

I'll probably also get the server running on stratum.stitthappens.bit which will be able to securily use self signed certificates and also potentially resolve to a tor hidden service (v7g2e7rdrjpgsmlu.onion)


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 10, 2012, 11:28:50 PM
Red Emerald, great stuff. Although I'm using those packages in other applications as well and I don't need virtualenv, I think it's good candidate to use in official installation howto. I wanted to learn how to use virtualenv, so thanks for the command, I'll try it on my server and update installation howto.

a) easy_install and pip are using the same repository. You're right that pip is newer and should be used instead of easy_install. It's my fault.

b) how did you managed to install "twisted words" package from pip? I failed to use easy_install because it has space in the name (wtf?). It's why I used apt-get.

c) I fixed installation of openssl package in README. Didn't realized that it's called "pyopenssl" instead of "openssl". Stupid ambiguous name conventions in pypi...


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: Red Emerald on January 10, 2012, 11:38:10 PM
Red Emerald, great stuff. Although I'm using those packages in other applications as well and I don't need virtualenv, I think it's good candidate to use in official installation howto. I wanted to learn how to use virtualenv, so thanks for the command, I'll try it on my server and update installation howto.
Definitely checkout out http://www.doughellmann.com/projects/virtualenvwrapper/ if you are new to virtualenv. It adds some really nice commands.

Quote
a) easy_install and pip are using the same repository. You're right that pip is newer and should be used instead of easy_install. It's my fault.
I don't know if it makes much of a difference.  I've just been using pip for other things and am used to it.

Quote
b) how did you managed to install "twisted words" package from pip? I failed to use easy_install because it has space in the name (wtf?). It's why I used apt-get.
I didn't.  It hasn't thrown any errors though, so it looks like I don't need to. Running "python -c "from twisted import words"" works without problem. Hopefully nothing is broken lol

Quote
c) I fixed installation of openssl package in README. Didn't realized that it's called "pyopenssl" instead of "openssl". Stupid ambiguous name conventions in pypi...
Yeah it is weird. apt is "python-openssl" but pip/easy_install use "pyopenssl"


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 10, 2012, 11:49:26 PM
I didn't.  It hasn't thrown any errors though, so it looks like I don't need to. Running "python -c "from twisted import words"" works without problem. Hopefully nothing is broken lol

Interesting. This is needed for IRC support, but I see you in #stratum-nodes already. I'll need to check pip, if it also install some recommended dependencies or what. But if it didn't raise any exception during startup, then everything is fine.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: Red Emerald on January 10, 2012, 11:56:10 PM
I didn't.  It hasn't thrown any errors though, so it looks like I don't need to. Running "python -c "from twisted import words"" works without problem. Hopefully nothing is broken lol

Interesting. This is needed for IRC support, but I see you in #stratum-nodes already. I'll need to check pip, if it also install some recommended dependencies or what. But if it didn't raise any exception during startup, then everything is fine.
Maybe pip grabs everything while easy_install doesn't. I'm not sure though.

I've shut down my electrum server since I am liking this project more.  Are you still planning on using abe for indexing the block chain?  Maybe looking into some of the armory code would give us some ideas.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 11, 2012, 12:05:53 AM
Please don't shut down Electrum now, Stratum is still not ready as a replacement for Electrum backend. But I'm hoping it will be ready soon.

I'm working on two fronts. I like libbitcoin project a lot and I think it's the future of the Stratum project, because it following all bitcoin rules, does all validations, it's programmed in really good way and don't need to be running with satoshi's Monolith. However at this point libbitcoin is not mature enough. As an example, I have no idea how to really use it, because the lack of documentation. And I'm not hiding that my C++ skills are almost non-existing.

Thanks to this I see Abe as a better candidate for the startup. Although it is bounded to original satoshi's client, it's pure python and I can understand it better ATM. And I'm trying to implement Abe in the way that it can be easily replaced by another blockchain backend.

I didn't read the armory code yet, but it's also early alpha and memory usage is outragenous (but I understand it!). I still see the lowest risk in using Abe.

Edit: About running Stratum over Tor. I'm also thinking about this, it should be as easy as point hidden service to local transport ports. But I'm thinking about changing node.get_peers() method and IRC peer lookup to be more flexible and enable also exposing alternative channels like onion address. Then it should be easy to ask any Stratum node or IRC channel for all known Tor-enabled Stratum servers (it's the feature which satoshi's client is missing).


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: Red Emerald on January 11, 2012, 12:15:47 AM
Please don't shut down Electrum now, Stratum is still not ready as a replacement for Electrum backend. But I'm hoping it will be ready soon.
I shut it down cuz something broke in the database.  I don't really have the desire to fix it right now. I'll post my errors in the electrum thread.

Quote
I'm working on two fronts. I like libbitcoin project a lot and I think it's the future of the Stratum project, because it following all bitcoin rules, does all validations, it's programmed in really good way and don't need to be running with satoshi's Monolith. However at this point libbitcoin is not mature enough. As an example, I have no idea how to really use it, because the lack of documentation. And I'm not hiding that my C++ skills are almost non-existing.
Not needing to run bitcoind would be neat. I can see the arguments for having multiple clients on the bitcoin network, but I feel like it is also a dangerous road to go down.  It can potentially slow down adoption of features and definitely has the potential to introduce bugs. Sticking with satoshi's bitcoind is definitely a good idea for now.

I think libbitcoin will be a good choice once it and stratum are a bit more stable.

Quote
Thanks to this I see Abe as a better candidate for the startup. Although it is bounded to original satoshi's client, it's pure python and I can understand it better ATM. And I'm trying to implement Abe in the way that it can be easily replaced by another blockchain backend.

I didn't read the armory code yet, but it's also early alpha and memory usage is outragenous (but I understand it!). I still see the lowest risk in using Abe.
I agree Abe is lowest risk.  I was just thinking that looking at other people's code may give us some inspiration.

I also found https://github.com/samrushing/caesure which hasn't been active in a while, but it is all in python.  


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: ThomasV on January 11, 2012, 04:17:45 PM
@slush:
I have rewritten the Electrum network interface, so that it uses jsonrpc over http when the http port is used.
I used the rpc calls that you provided in your document where it was possible, but some calls are missing for session management.
please have a look and update the protocol.




Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: Red Emerald on January 11, 2012, 06:26:13 PM
I used the rpc calls that you provided in your document where it was possible, but some calls are missing for session management.
So does this mean that electrum servers will eventually be part of the stratum network? That's what I was hoping, I just want to make sure.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: ThomasV on January 11, 2012, 07:13:37 PM
I used the rpc calls that you provided in your document where it was possible, but some calls are missing for session management.
So does this mean that electrum servers will eventually be part of the stratum network? That's what I was hoping, I just want to make sure.
I do not know. The initial idea was to propose a new name for the communication protocol, so that it can be distinguished from the software.
Now it seems that slush has decided to rewrite client and server from scratch, and that "Stratum" is being used to designate his software, not the protocol.
So I guess we might want to find another name for the protocol...


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 12, 2012, 02:39:08 PM
I do not know. The initial idea was to propose a new name for the communication protocol, so that it can be distinguished from the software.

Thomas, I proposed the protocol and asked you for the opinion. You told me that you don't understand and/or don't see any point in it and you want reference implementation of the protocol. So I created Stratum server implementation. What's unclear in this?

Quote
Now it seems that slush has decided to rewrite client and server from scratch, and that "Stratum" is being used to designate his software, not the protocol.

It's still Stratum protocol + server reference implementation + I'm writing some "default" services into it. Protocol implementation is already here and you know that I decided to separate Stratum from Electrum project, because I see some potential for lightweight clients generally, not only for Electrum.

But of course I'm open to help you with integrating Stratum into Electrum client, if you want. We already had the discussion about Twisted on the IRC and I accept that you don't like Twisted in the client, so I'll provide you clean-python implementation of Stratum client.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 12, 2012, 02:42:59 PM
I used the rpc calls that you provided in your document where it was possible, but some calls are missing for session management.

What exactly are you missing? I see only three session-related calls in server.py (new_session, update_session, poll_session) which are useless in Stratum protocol.

Edit: To be more verbose:

new_session: You just don't send session cookie to the server and it will create new session for the client automatically. No RPC command needed.

update_session: Replaced with blockchain.address.subscribe()

poll_session: Perform just blank HTTP request to Stratum server, it will put pending messages to the response. No RPC command needed.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: ThomasV on January 12, 2012, 03:04:01 PM
new_session: You just don't send session cookie to the server and it will create new session for the client automatically. No RPC command needed.

update_session: Replaced with blockchain.address.subscribe()

poll_session: Perform just blank HTTP request to Stratum server, it will put pending messages to the response. No RPC command needed.


Can you explain a bit more your decisions?
Performing blank HTTP requests is an interesting idea, but I would like to know why you decided that polling should be performed that way. is this in order to minimize traffic?
Also, how would that work with TCP?


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 12, 2012, 03:20:01 PM
Can you explain a bit more your decisions?

Of course. HTTP is just a transport carrying abstract Stratum payload. There is TCP transport already and there can be a lot more transports in the future. Stratum protocol is designed to be bi-directional, so server can send some payload to client (information about new block, new transaction) at any time. It's not the part of application layer itself to decide HOW transport will deliver such payload to the client. So server just tell "send <this> to the client" and don't care about it anymore.

HTTP Poll transport is just the alternative for the environment where real bi-directional communication (like TCP socket or websocket) is not possible. So every poll request just give a chance to send pending messages from the server to the client in the HTTP response, simulating bi-directional communication, just with some delay (=polling interval).

From the view of this idea, creating RPC command for polling is going against the whole conception, because RPC commands are handled by different layer than HTTP transport mechanism. And there's nothing like 'poll' RPC command for standard bi-directional transports anyway.

Quote
Also, how would that work with TCP?

In the TCP transport, which is bi-directional, polling is not needed at all. Server will send the payload when it's ready, in real time. It's the beauty of pub-sub mechanism (http://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern).


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: ripper234 on January 13, 2012, 02:32:59 PM
slush,

I'm finally starting to work on this, and have some questions.
I think I'll fire them one by one on this thread, instead of accumulating them.

The first one is a meta question - how about using Shapado (http://shapado.com/) as a Q&A forum? It's "setup in 8 seconds", and better than a forum thread. It might be better than just firing these questions away at the Stack Exchange. The advantage of a dedicate forum is that you (and other Stratum devs) could follow that website and quickly respond to questions, and the answers would be findable later via Google.

Another question is about a sessions while polling:

From the Google Doc:

Quote
Cookie should contains all cookies provided by the server in previous calls, especially the STRATUM_SESSION cookie which is identifying current HTTP session. Blank or missing STRATUM_SESSION will initiate new session, even on the same TCP connection

The APIs that I need right now appear to be only address.get_history and transaction.broadcast (I'll start by implementing the simple & naive polling protocol). For these purposes, I'm not sure why I need to maintain a session. As a matter of fact ... can you give an example when a session is useful?


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 13, 2012, 03:11:18 PM
The first one is a meta question - how about using Shapado (http://shapado.com/) as a Q&A forum? It's "setup in 8 seconds", and better than a forum thread. It might be better than just firing these questions away at the Stack Exchange. The advantage of a dedicate forum is that you (and other Stratum devs) could follow that website and quickly respond to questions, and the answers would be findable later via Google.

Yes, some better tools should be better in the future, but at this moment the forum thread is doing well. There is only few people so the discussion is pretty clean now. Especially I'll try to move that google docs to some wiki which allow me to separate whole document into more parts and include some additional info (as an examples).

Quote
The APIs that I need right now appear to be only address.get_history and transaction.broadcast (I'll start by implementing the simple & naive polling protocol). For these purposes, I'm not sure why I need to maintain a session. As a matter of fact ... can you give an example when a session is useful?

What's your purpose of such calls? You probably don't want to download whole address history on every request (say 5 seconds), right? I'm not sure, because I don't know your application, but you probably want to download address history on the startup and then subscribe your client for changes on given address, to receive only new transactions. In this way you save server resources and bandwidth. It's also possible that server may block your requests when you start overflooding server with "get_history" requests every few seconds, because there's no reason to do that so frequently (rate limiting isn't implemented yet, but it is pretty good feature for the future).

There are mostly two main usecases:
a) The address you're interested in is already existing, so you need to ask for history + subscribe for new transactions.
b) The address is freshly generated by you, so you know that there's no history and you want ONLY to subsribe for new transactions. This is the case of some merchant tool, which created fresh address for receiving coins for the order...

case a)
first http call (one HTTP request can contain more RPC calls, divided by newline character):
blockchain.address.get_history('1SomeAddress') => list of transactions
blockchain.address.subscribe('1SomeAddress') => returns only True on a success, but server will send you data for incoming transactions in following HTTP responses.

following http calls: blank HTTP request (with the session) to the server

case b)
first HTTP call:
blockchain.address.subscribe('1SomeAddress') (no get_history() command is needed!)

following HTTP calls: blank HTTP request (with the session) to the server

I think there's big misunderstanding about the "polling" thing. Polling does not mean that the connection is stateless and you need to ask for everyting on every request. Polling mechanism is only the workaround for the situation when only the client can initiate the communication and is giving a chance to counterparty to send pending data back in the polling response (like information about incoming transaction)...

So yes, you need to "care" about the session even in your use case. But almost every http client library can do cookie management internally, when you perform more calls ("http polling requests") in the row..


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: ripper234 on January 13, 2012, 03:20:47 PM
Quote
The APIs that I need right now appear to be only address.get_history and transaction.broadcast (I'll start by implementing the simple & naive polling protocol). For these purposes, I'm not sure why I need to maintain a session. As a matter of fact ... can you give an example when a session is useful?

What's your purpose of such calls? You probably don't want to download whole address history on every request (say 5 seconds), right? I'm not sure, because I don't know your application, but you probably want to download address history on the startup and then subscribe your client for changes on given address, to receive only new transactions. In this way you save server resources and bandwidth. It's also possible that server may block your requests when you start overflooding server with "get_history" requests every few seconds, because there's no reason to do that so frequently (rate limiting isn't implemented yet, but it is pretty good feature for the future).

There are mostly two main usecases:
a) The address you're interested in is already existing, so you need to ask for history + subscribe for new transactions.
b) The address is freshly generated by you, so you know that there's no history and you want ONLY to subsribe for new transactions. This is the case of some merchant tool, which created fresh address for receiving coins for the order...

case a)
first http call (one HTTP request can contain more RPC calls, divided by newline character):
blockchain.address.get_history('1SomeAddress') => list of transactions
blockchain.address.subscribe('1SomeAddress') => returns only True on a success, but server will send you data for incoming transactions in following HTTP responses.

following http calls: blank HTTP request (with the session) to the server

case b)
first HTTP call:
blockchain.address.subscribe('1SomeAddress') (no get_history() command is needed!)

following HTTP calls: blank HTTP request (with the session) to the server

I think there's big misunderstanding about the "polling" thing. Polling does not mean that the connection is stateless and you need to ask for everyting on every request. Polling mechanism is only the workaround for the situation when only the client can initiate the communication and is giving a chance to counterparty to send pending data back in the polling response (like information about incoming transaction)...

So yes, you need to "care" about the session even in your use case. But almost every http client library can do cookie management internally, when you perform more calls ("http polling requests") in the row..


For context, this is the application I'm developing (http://bitcoinpotato.com/). I wanted to implement a polling client at first, that polls every minute or so, just as an initial proof of concept - I'd like to see the protocol works. I intend to move to a more efficient implementation soon after that.

Let me try to rephrase. Assuming servers don't rate-limit just yet - if I don't send any cookies, and just poll every minute on address.get_history, and once in a while do transaction.broadcast ... will this work?

I really would like to get all transactions on these addresses. The addresses should in theory only have 1-2 transactions on each of them, but I'd still like to know all the transactions history.

This is not the final implementation, just a temporary POC.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 13, 2012, 03:30:41 PM
Let me try to rephrase. Assuming servers don't rate-limit just yet - if I don't send any cookies, and just poll every minute on address.get_history, and once in a while do transaction.broadcast ... will this work?

Assuming that such services are already implemented on the Stratum nodes (which isn't yet) - yes, it will work. But you're doing exactly the case b) from my previous post, so you don't need to use get_history at all, just subscribe the address in the time when you send it to the user.

But I understand you're mostly playing with the stuff so I'm ok with it...


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: ripper234 on January 13, 2012, 04:04:03 PM
Let me try to rephrase. Assuming servers don't rate-limit just yet - if I don't send any cookies, and just poll every minute on address.get_history, and once in a while do transaction.broadcast ... will this work?

Assuming that such services are already implemented on the Stratum nodes (which isn't yet) - yes, it will work. But you're doing exactly the case b) from my previous post, so you don't need to use get_history at all, just subscribe the address in the time when you send it to the user.

But I understand you're mostly playing with the stuff so I'm ok with it...

Another question - I haven't seen any reference to testnet in this thread ... are there testnet servers? Does the protocol support it?


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: ripper234 on January 13, 2012, 04:28:06 PM
Let me try to rephrase. Assuming servers don't rate-limit just yet - if I don't send any cookies, and just poll every minute on address.get_history, and once in a while do transaction.broadcast ... will this work?

Assuming that such services are already implemented on the Stratum nodes (which isn't yet) - yes, it will work. But you're doing exactly the case b) from my previous post, so you don't need to use get_history at all, just subscribe the address in the time when you send it to the user.

But I understand you're mostly playing with the stuff so I'm ok with it...

Another question - I haven't seen any reference to testnet in this thread ... are there testnet servers? Does the protocol support it?

What's the expected time that requests are supposed to take?

I sent a request to http://chicago.stratum.bitcoin.cz:8000, with a timeout of 60 seconds ... and the timeout expired.

content-type: application/stratum
id: 1
method: blockchain.address.get_history
params: mw5h6BDh77GUPc4DCmzjoU7ry2R4exoaYr   (this is a testnet address, so I guess it wouldn't work ... but I should get an error, not a timeout)


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: ripper234 on January 13, 2012, 04:43:23 PM
This also happens when I try a prodnet address:

POST / HTTP/1.1
Host: chicago.stratum.bitcoin.cz:8000
Content-Type: application/stratum
Connection: keep-alive
Accept: */*
User-Agent: NING/1.0
Content-Length: 84

id=1&method=blockchain.address.get_history&params=14obksEpeUTAKsdKsw3vTSUoiYeK9S3thA


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: ripper234 on January 13, 2012, 04:45:49 PM
Let me try to rephrase. Assuming servers don't rate-limit just yet - if I don't send any cookies, and just poll every minute on address.get_history, and once in a while do transaction.broadcast ... will this work?

Assuming that such services are already implemented on the Stratum nodes (which isn't yet) - yes, it will work. But you're doing exactly the case b) from my previous post, so you don't need to use get_history at all, just subscribe the address in the time when you send it to the user.

But I understand you're mostly playing with the stuff so I'm ok with it...

Any documentation on what is and is not implemented right now?
ETAs?


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 14, 2012, 06:31:41 PM
This also happens when I try a prodnet address:

POST / HTTP/1.1
Host: chicago.stratum.bitcoin.cz:8000
Content-Type: application/stratum
Connection: keep-alive
Accept: */*
User-Agent: NING/1.0
Content-Length: 84

id=1&method=blockchain.address.get_history&params=14obksEpeUTAKsdKsw3vTSUoiYeK9S3thA

Hi, your payload is wrong, you must use json-encoded data. So

{"id":1, "method":"blockchain.address.get_history", "params": ["14obksEpeUTAKsdKsw3vTSUoiYeK9S3thA"]}

instead of url-encoded data.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 14, 2012, 06:43:26 PM
Any documentation on what is and is not implemented right now?

get_history() works now, however the result data may be slightly changed in the future. I'm writing API for standard services, then will be much cleaner what method expect what parameters and what will be the result object.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: ripper234 on January 15, 2012, 08:47:49 AM
This also happens when I try a prodnet address:

POST / HTTP/1.1
Host: chicago.stratum.bitcoin.cz:8000
Content-Type: application/stratum
Connection: keep-alive
Accept: */*
User-Agent: NING/1.0
Content-Length: 84

id=1&method=blockchain.address.get_history&params=14obksEpeUTAKsdKsw3vTSUoiYeK9S3thA

Hi, your payload is wrong, you must use json-encoded data. So

{"id":1, "method":"blockchain.address.get_history", "params": ["14obksEpeUTAKsdKsw3vTSUoiYeK9S3thA"]}

instead of url-encoded data.

Hmm, I'll fix and retry ... possibly next weekend.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: rapeghost on January 16, 2012, 04:17:18 PM
It's actually hosted by BitVPS.COM not .Net....



Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 16, 2012, 05:01:42 PM
It's actually hosted by BitVPS.COM not .Net....

Fixed, thanks :)


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: ripper234 on January 17, 2012, 05:46:54 AM
Check out my answer here on why The Stratum protocol is secure (http://bitcoin.stackexchange.com/a/2615/78).


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 19, 2012, 10:53:16 AM
Check out my answer here on why The Stratum protocol is secure (http://bitcoin.stackexchange.com/a/2615/78).

Very nice, thank you :-).

I didn't commited into git for few days, because I'm finishing one project in my job, but then I'll work more actively on the Stratum again. I'm missing it!


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: ripper234 on January 19, 2012, 01:10:21 PM
Check out my answer here on why The Stratum protocol is secure (http://bitcoin.stackexchange.com/a/2615/78).

Very nice, thank you :-).

I didn't commited into git for few days, because I'm finishing one project in my job, but then I'll work more actively on the Stratum again. I'm missing it!

I'm missing it too :)
I wish I could devote more time to this, but I'm so swamped with everything nowadays.

Baby steps.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: ThePiachu on January 21, 2012, 11:11:39 PM
Very interesting project! Watching this (having read through all the previous post, took me awhile :\ ).

Was also considering doing something similar, guess I'll do my best to conform to this standard. Might be a bit hard though, as Google App Engine with Go has some issues with TCP/IP and RPCs. URL encoding would be easier, but we'll see if I can hammer the code to work.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 22, 2012, 12:00:25 AM
GAE has no problem with json, so no problem with encoding here. However GAE does not support TCP sockets and persistent connections, except "Channel API", which is highly proprietary stuff and there's no reason to include this into Stratum "standard". However HTTP transport is definitely the way how to implement Stratum on GAE. Are you using Python or Java on GAE?

I'm going to implement set of unit tests, so it should be pretty easy to test if alternative server implementation fits all protocol requirements. I'm still working on another project right now, but I'll be back on Stratum in few days.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: ThePiachu on January 22, 2012, 04:06:08 AM
I'm writing in Go, so if I get everything to work it should be quite fast. But Go apparently has no support for JSON-RPC over HTTP - http://code.google.com/p/go/issues/detail?id=2738 (so I had to create my own package for that), and the RPC package doesn't like GAE - https://groups.google.com/forum/#!msg/google-appengine-go/uQ0cv0m9j0E/H3VCrFgEWvcJ .

So far I managed to get a simple Python request to go through to my app - https://en.bitcoin.it/wiki/API_reference_%28JSON-RPC%29#Python , and got the app to communicate a bit with bitcoind, but I'm having some problems communicating with a miner, so I have to look into it further.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on January 22, 2012, 04:36:59 AM
I'm writing in Go, so if I get everything to work it should be quite fast. But Go apparently has no support for JSON-RPC over HTTP - http://code.google.com/p/go/issues/detail?id=2738 (so I had to create my own package for that), and the RPC package doesn't like GAE - https://groups.google.com/forum/#!msg/google-appengine-go/uQ0cv0m9j0E/H3VCrFgEWvcJ .

Well, Stratum is not clear JSON-RPC, it's only based on it (format of the messages is the same, but with some additional things like session management and callback URLs). For your purposes the JSON library & possibility to process HTTP POST requests should be enough to implement Stratum in few lines of code.

Interesting, I didn't notice Go support on GAE. I had to say that I don't follow GAE progress in last year.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: casascius on January 27, 2012, 03:23:51 AM
I would like to propose one more thing that Stratum should keep track of, and perhaps one LESS thing.

The More thing:  Nodes should inform Stratum that they can exchange P2P information about subject X, where X is an arbitrary string that is chosen by the developer of whatever X is.  In turn, Stratum helps nodes aware of subject X find one another.

The Less thing:  The block chain should not be assumed as something that will be exchanged by default, but rather, should be one of many subjects that can be subscribed to.

Now, why?!  At some point, I believe that the block chain as we know it is going to get called the "Satoshi" block chain, or the "orthodox" block chain, and people are going to start new chains that work as subsets of the Satoshi block chain, so they can function as a whole bitcoin node without having to download the whole damn thing.

Someone who accepts a pruned block chain is ultimately putting their trust in whoever pruned it.  This is a tradeoff that becomes more likely to happen every day the block chain gets bigger.  Someone who accepts a pruned chain also cannot participate in exchanging historical blocks with orthodox clients - they can only exchange such blocks with others who trust in the same chain.  Hence Stratum being an ideal hookup mechanism for these clients - but without Stratum having to bear the burden of being the relay (except where the operator chooses to make his server relay such things).

An example is if I go and compile a digest of the block chain that includes only the unspent outputs as of block 150,000, I sign it, I publish it.  People want it because it's only 150 MB instead of a gig, and let's assume others are vouching for the integrity of it (in other words, consensus on the forums is that the outputs it contains are consistent with the orthodox block chain as of the same block height, and someone publishes a utility to confirm that).  From then onward, all newer blocks are interchangeable with the Satoshi block chain, it's just that orthodox Satoshi clients will not understand how to deal with a peer that has recent blocks but not the whole chain.  I would want Stratum to help me hook up with clients that are expecting a floor of 150,000 as well.

Let's say somebody else goes and creates a combined client that can trade Bitcoins, Namecoins, Litecoins, and whatever other kinds of coins, but all within the same client and protocol.  Ideally, Stratum should help that client find other like-minded clients so they can exchange all of those blocks, all without burdening Stratum with block types its server operators don't want to waste resources on.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: Red Emerald on January 27, 2012, 04:56:19 AM
I would like to propose one more thing that Stratum should keep track of, and perhaps one LESS thing.

The More thing:  Nodes should inform Stratum that they can exchange P2P information about subject X, where X is an arbitrary string that is chosen by the developer of whatever X is.  In turn, Stratum helps nodes aware of subject X find one another.

The Less thing:  The block chain should not be assumed as something that will be exchanged by default, but rather, should be one of many subjects that can be subscribed to.
This is a really great abstraction.  Building the stratum protocol to transmit arbitrary data is a great idea.

Quote
Someone who accepts a pruned block chain is ultimately putting their trust in whoever pruned it.  This is a tradeoff that becomes more likely to happen every day the block chain gets bigger.  Someone who accepts a pruned chain also cannot participate in exchanging historical blocks with orthodox clients - they can only exchange such blocks with others who trust in the same chain.  Hence Stratum being an ideal hookup mechanism for these clients - but without Stratum having to bear the burden of being the relay (except where the operator chooses to make his server relay such things).
I was under the assumption that the pruned blockchains would have verifiable hashes in them so that you wouldn't have to trust the distributor.  Stratum should definitely keep this in mind.

Quote
Let's say somebody else goes and creates a combined client that can trade Bitcoins, Namecoins, Litecoins, and whatever other kinds of coins, but all within the same client and protocol.  Ideally, Stratum should help that client find other like-minded clients so they can exchange all of those blocks, all without burdening Stratum with block types its server operators don't want to waste resources on.
Wow. This is a great idea.  I don't want to have a run a stratum server for bitcoin and a stratrum server for namecoin and so on.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: casascius on January 27, 2012, 05:24:03 AM
I was under the assumption that the pruned blockchains would have verifiable hashes in them so that you wouldn't have to trust the distributor.  Stratum should definitely keep this in mind.

Somehow, I believe downloading a "seed" blockchain will be more preferable in practice than getting pruned blockchains bit-by-bit from peers, for two reasons.

One, I am convinced (though smart others have disputed) that if you accept pruned blockchains bit-by-bit from other nodes, you can be disrupted (DoS'd) simply by getting fed a block that has the wrong things pruned from it (example, someone pruned an unspent transaction that should not have been pruned).  Sure, you'll be able to detect you don't have the complete picture, because the "verifiable hash" you refer to won't check out, but you'll have a difficult time finding which block(s) you were misfed, which you need to ask for again - recovery from which will be at least as burdensome - and definitely more complex - as compared to downloading a seed blockchain as a signed flat file (either via http, torrent, etc.).

Second, even when blocks are pruned, there's plenty of "cruft" that you really don't need to keep if you don't demand complete (orthodox) cryptographic verification of the block chain file.  When you prune nodes off a merkle tree, you still have to leave behind hashes of the branches you cut off to be able to validate the tree.  Those are wasted space - what I mean by this is they take up bandwidth and permanent storage but are useful only once - for validating the integrity of the file - and will never be accessed for any other reason.

Another thing that's a total waste of space is the inputs of all the kept transactions, which are much larger than the outputs.  The inputs take up way more space each (they contain a public key and a digital signature - over 130 bytes each) and outputs are much smaller (like around 30 bytes).  You don't need the inputs for validating new transactions, just the outputs.  But with the current merkle tree setup, you can only prune a whole transaction at a time - all inputs and all outputs must go together or not at all.

Also, another big space waster: transactions with "multiple outputs" (generated by sendmany), which must be kept in their entirety if a single output remains unspent, they cannot be pruned with the current merkle tree spec.  This would not be so bad except for the fact that some mining pools issue huge sendmany's with numerous tiny outputs worth only pennies each - to distribute payout to miners in the coinbase of their blocks.  These transactions under the orthodox specification can never be pruned until every penny output has been spent - which for most of these blocks may very well be never. (example transaction from P2Pool: http://blockexplorer.com/t/42BmXQaM9T)

Now, on the other hand, if every 10k blocks or so, somebody can compile all of the unspent outputs into a signed file, discarding all inputs and all spent outputs, that file is always going to be a tiny fraction of the size the orthodox block chain.  If they also released source code to a validation utility (which can be run only by those possessing the complete orthodox block chain), the integrity of that file can be confirmed (and also signed) by the more concerned members of the community, and the file can then be trusted as safe based on community consensus.

A useful function of Stratum would be to pass around those endorsing signatures so that a client can quickly determine which is the most recent pre-compiled "block chain digest" that enjoys the overwhelming majority of community support.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: Red Emerald on February 03, 2012, 05:29:26 AM
Hows this going? I re-arranged my server and have been to busy coding to play around with the protocol


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: bc on February 27, 2012, 03:19:41 AM
A useful function of Stratum would be to pass around those endorsing signatures so that a client can quickly determine which is the most recent pre-compiled "block chain digest" that enjoys the overwhelming majority of community support.

Should Stratum keep track/report on nodes who endorse digests later proven to be in some way false? Is there enough peer pressure to promote only honest endorsements?


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: genjix on March 05, 2012, 04:08:23 PM
Hows this going? I re-arranged my server and have been to busy coding to play around with the protocol

I think slush doesn't have much time. Anyway, I'll pick up the slack. I'm working off slush's original proposal, but need to make some design decisions in coordination with ThomasV.

I'll only focus on two use-cases:
- Merchants
- Users


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on March 15, 2012, 11:36:59 AM
I think slush doesn't have much time. Anyway, I'll pick up the slack. I'm working off slush's original proposal, but need to make some design decisions in coordination with ThomasV.

Genjix, can be more specific where and why you needed to decide something? Afaik the protocol is fully covered. I read your proposal (https://bitcoinconsultancy.com/wiki/Intersango/Overnet) and I think there are many things to discussion. Namely - why you choose to not follow json-rpc specification? I'm really missing "id". Also variables "method" and "result" together don't sound clear.

Btw after a while, I'm back in the project. Expect some progress in git repository in following days.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: ThomasV on March 15, 2012, 12:49:22 PM
I think slush doesn't have much time. Anyway, I'll pick up the slack. I'm working off slush's original proposal, but need to make some design decisions in coordination with ThomasV.

Genjix, can be more specific where and why you needed to decide something? Afaik the protocol is fully covered. I read your proposal (https://bitcoinconsultancy.com/wiki/Intersango/Overnet) and I think there are many things to discussion. Namely - why you choose to not follow json-rpc specification? I'm really missing "id". Also variables "method" and "result" together don't sound clear.

Btw after a while, I'm back in the project. Expect some progress in git repository in following days.

Please do not consider the current syntax as a conscious choice; we just needed to start with something concrete.
Genjix asked me to describe what information is needed by the client, and your document does not include that. This is why I wrote this specification.

Please feel free to suggest improvements concerning the syntax & protocol. For example, we can use an "id" field if you think it is better.
Again, I did not make the conscious choice not to use an "id" field; I was focusing on something else. I was focusing on the specification of the services





Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on March 15, 2012, 01:48:30 PM
Thomas, I understand and I'm fine with that. Then it's better to specify services/methods/parameters and forget to wire format at all. I specified wire format, but I didn't specify blockchain methods, because I didn't have enough experience to do it months ago. I assume that you have much more knowledge to do that than me now, so we can work together.

Everything necessary is to define method names (e.g. 'blockchain.address.subscribe'), their parameters (e.g. 'address', 'status'), result types and exceptions (possible exceptions which can be thrown by given method, and exception parameters).

Btw this is pretty hard to do clearly and cover all possibilities (especially all those possible exceptions and corner cases). This is the reason why I prefer easy API without complicated internal logic. But with the list of necessary methods, parameters etc. it's pretty easy to transform it into wire format.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on March 15, 2012, 02:14:44 PM
Thomas, I composed very simple document to begin with the blockchain specification:
https://docs.google.com/spreadsheet/ccc?key=0Al5UmiyYRxBudHl2VFhJWG5rT09maU9kbzltR3hBdmc . Give me your google docs email, I'll enable edit mode for you.

I combined proposal from Stratum draft document and wikipage on bitcoinconsultancy from you and genjix (marked bold). I have some comments to parameters/responses defined on that wiki page, I'll wait to you and genjix on IRC, it will be much faster than discuss it on forum or wiki...


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: dogisland on April 03, 2012, 07:28:57 PM
Wow, I've seen this thread hanging round for months but this is the first time I've really got it.

This is a great piece of work, I'm going to re-write Strongcoin to sit on top of Stratum, this is excellent.



Title: Re: [proposal] [Stratum] Overlay network protocol over Bitcoin
Post by: Traktion on June 04, 2012, 07:09:58 PM
What are the odds that this overlay network could be used to permit two mobile devices to transact directly, in the absence of Internet access for one or both parties?  For example, a direct ad-hoc wifi connection or an indirect connection via a 'piratebox' type device?  Asuuming, of course, that the sending device doesn't need Internet access to produce a transaction from it's own, known address funds.

+1

Great thread + software!

I would like to know if you could pass a signed transaction onto another party (say, from the payer to the payee) too. This would be very useful for areas with limited data connectivity, as it would only require the receiver to 'upload' the transaction.

You could also have situations where both parties had no Internet access and such transactions could be treated like cheques, which could be cashed when the payee is next online.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on June 11, 2012, 03:38:43 PM
I would like to know if you could pass a signed transaction onto another party (say, from the payer to the payee) too. This would be very useful for areas with limited data connectivity, as it would only require the receiver to 'upload' the transaction.

Yes, this is possible with signed transaction.

Quote
You could also have situations where both parties had no Internet access and such transactions could be treated like cheques, which could be cashed when the payee is next online.

But you need to be really careful here. The fact that transaction is signed doesn't mean that it is valid. If you cannot fully trust payind side, you shouldn't receive such transaction offline...


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: marcus_of_augustus on June 11, 2012, 11:51:14 PM
Quote
But you need to be really careful here. The fact that transaction is signed doesn't mean that it is valid. If you cannot fully trust payind side, you shouldn't receive such transaction offline...

People who write rubber checks are not a new phenomena !

There could even be an off-line market in "Junk signedTx", where such things are traded at discount to face value in the off-chance delinquent accounts are eventually made good by people looking to resurrect their financial reputations.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: markm on June 12, 2012, 12:25:44 AM
The signed transaction probably should be transfered in siged messages, signed by various personal or corporate identity and/or web of trust signatures, so there is not only a signature showing what bitcioin addresses committed to pay the amount but also which entity certified that they themselves issued that "signed bitcoin-transaction".

Maybe including pubkeys of actual entities the receiving addresses were believed to belong to, with signature from them certifying those receiving addresses are indeed theirs.

Then receiver of the junk "cheque" has legs to stand on come collection time or saleback-of-rep time.

-MarkM-


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: apetersson on June 20, 2012, 05:32:11 PM
is there a method to query addresses in bulk?
like the proposed bloom filter commands in the main client?

and what is the story if a chain reorganisation occurs? how can a transaction become invalid?


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: randy-waterhouse on August 20, 2012, 12:13:57 AM
Is anybody aware of any java implementations that are using the Stratum protocol?

Open source preferably ...  :)


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on August 21, 2012, 10:40:09 PM
Is anybody aware of any java implementations that are using the Stratum protocol?

Open source preferably ...  :)

There are python, php and JS implementations, but I don't know about working Java binding yet :(. But please publish it on github if you'll make one ;).


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: ripper234 on August 22, 2012, 09:46:00 PM
Is anybody aware of any java implementations that are using the Stratum protocol?

Open source preferably ...  :)

There are python, php and JS implementations, but I don't know about working Java binding yet :(. But please publish it on github if you'll make one ;).

I started something 7 months ago (https://github.com/ripper234/Bitcoin-Potato/tree/master/app/org/bitcoin/stratum).
Sadly it's not even a working prototype ... think of it as preliminary work - I was just trying to get a feel on how to work with BitcoinJ and Stratum. Feel free to use or ignore.



Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: ripper234 on February 16, 2013, 06:41:22 AM
Is anybody aware of any java implementations that are using the Stratum protocol?

Open source preferably ...  :)

There are python, php and JS implementations, but I don't know about working Java binding yet :(. But please publish it on github if you'll make one ;).

Can you link to them? Perhaps add links to them from the stratum server readme on github (https://github.com/slush0/stratum)?


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: ripper234 on February 16, 2013, 06:44:51 AM
BTW, can someone update here on the status of the server network?

- How many Stratum servers are there? (is there a directory)
- Who runs them all?
- Are there reliable & trustworthy (two separate questions)?
- Has there been any thought about connecting to multiple stratum serves for security purposes (so one server can't lie to you)? I guess that this is a client-side effort, no server-side.
- Where is Stratum used today in production?


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: slush on February 16, 2013, 10:50:44 AM
Can you link to them? Perhaps add links to them from the stratum server readme on github (https://github.com/slush0/stratum)?

There are python, php and js implementations, all available on https://github.com/slush0


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: Vinion on December 02, 2013, 02:55:13 PM
Hello

I am building a new miner software and I have a few questions about stratum protocol.

My first question for now is "How can I generate target from the nBit ?"
If someone knows please provide some code in any programming language you like.

Thank you for your replies



Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: mattlondon on February 15, 2014, 11:51:44 AM
I've started a MIT licensed java implementation of a Stratum client.

Its my first check in (i.e. not even 0.0.1 yet!) that I've tested it with some dogecoin pool servers and it seems ok, but it has not undergone any serious production or load testing yet (in particular I've not gone over it with any stress testing for synchonisation yet so there might be some concurrency bugs to be worked out). 

It is designed to be fairly self contained and uses a hopefully simple to use observer pattern for call backs into calling-code.

https://github.com/matt1/stratum-client-java


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: sundance on February 15, 2014, 11:21:53 PM

Guys,

Nice work on Stratum. A note that I hope you will take to heart --- this problem is already solved by open protocol standards (at least in TCP point to point cases). The rest of the financial world is already using it so for the purpose of easing future integration, it's best to leverage that instead of re-invent the wheel.

The FIX (Financial Information Exchange) protocol already covers all the requirements listed in Stratums spec and has been used for over 15 years in the financial world for relaying messages for trading, market data, request for quotes, and others. A recent open source implementation in C++ allows you to easily define new schema (xml, json) to extend the protocol.

http://fix8.org/

The protocol supports replay through sequence numbering, etc.
There are open source implementations in C++, Python, and Java, and possibly others. Oanda's FX trading system uses it:
http://fxtrade.oanda.com/images/FixSpecifications2.pdf

What you want is this to be adapted to the web server/client model.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: MoonShadow on February 16, 2014, 03:27:55 AM

Guys,

Nice work on Stratum. A note that I hope you will take to heart --- this problem is already solved by open protocol standards (at least in TCP point to point cases). The rest of the financial world is already using it so for the purpose of easing future integration, it's best to leverage that instead of re-invent the wheel.

The FIX (Financial Information Exchange) protocol already covers all the requirements listed in Stratums spec and has been used for over 15 years in the financial world for relaying messages for trading, market data, request for quotes, and others. A recent open source implementation in C++ allows you to easily define new schema (xml, json) to extend the protocol.

http://fix8.org/

The protocol supports replay through sequence numbering, etc.
There are open source implementations in C++, Python, and Java, and possibly others. Oanda's FX trading system uses it:
http://fxtrade.oanda.com/images/FixSpecifications2.pdf

What you want is this to be adapted to the web server/client model.


I don't think you really understand what problem Stratum addresses.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: sundance on February 16, 2014, 08:33:18 AM

I understand it, and I'm talking about the network protocol aspect of it.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: sundance on February 16, 2014, 08:39:50 AM

Rather, I should say, I understand what's written in the Stratum spec. Would appreciate enlightenment on what I'm missing if you're sure I don't understand what Stratum's addressing.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: Remember remember the 5th of November on February 18, 2014, 12:10:51 PM
Am I the only one that was not satisfied with the Stratum documentation? I personally found it lacking in explanation a lot, I couldn't write an implementation.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: Schleicher on February 18, 2014, 07:31:14 PM
They expect us to reverse engineer python source code


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: MoonShadow on February 18, 2014, 07:37:19 PM
They expect us to reverse engineer python source code

Consider it a trial by fire.  If you can't figure it out, they don't want your help.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: Remember remember the 5th of November on February 18, 2014, 09:12:52 PM
They expect us to reverse engineer python source code
I am not sure if this was sarcasm, but if it was, I would've preferred a C or C++ implementation. I have more experience with those than Python.


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: Sannin on February 21, 2014, 08:19:43 AM
Hi,

can somebody explain me what is the session in stratum?

Quote
{"error": null, "id": 1, "result": [["mining.notify", "ae6812eb4cd7735a302a8a9dd95cf71f"], "00000001", 4]}

Can I ignore this parameter? What is it? Can I set this parameter?

Thanks for reply


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: rubberpool on May 28, 2014, 08:44:12 PM
Looking for a stratum developer to modify stratum a bit. We will pay btc for the modification

We want to limit amount of shares per minute that an IP can send to stratum AND limit the worksize. This will weaken asics and big farms and make em compareble to gpu miners (so asic owners will just leave and mine something else). Is it possible? Please PM me if you have an idea


Title: Re: [Stratum] Overlay network protocol over Bitcoin
Post by: onemorebtc on May 28, 2014, 11:48:58 PM
Looking for a stratum developer to modify stratum a bit. We will pay btc for the modification

We want to limit amount of shares per minute that an IP can send to stratum AND limit the worksize. This will weaken asics and big farms and make em compareble to gpu miners (so asic owners will just leave and mine something else). Is it possible? Please PM me if you have an idea

sounds like a job for a firewall with rate-limiting.
but: its a horrible idea