ripper234, although it's more centralized than bitcoin (which is pure p2p), it can be still called "decentralized", because anybody can run his own server or pick favourite server operator which he trust. Something like mailbox providers.

Brainstorming about signing messages

In the concept of service oriented architecture, some service calls can be proxied to backend or sent over non trusted channel. I like Firstbits as an example of proxied call, but of course there can be a lot of another examples as well as firstbits lookup can be implemented directly inside overlay server and proxy call isn't necessary at all.

For now, during the call of firstbits.resolve, the flow of call is following:

client -> (some transport) -> overlay server -> HTTP (firstbits.com don't support https yet!) -> firstbits.com

Client need to trust every part of this chain. For now, you must use some transport which prevent MITM, trust overlay server operator, hope that there isn't MITM attack in HTTP request to firstbits.com (I hope firstbits fix that and provide https soon).

Well, what if firstbits.com can sign it's response to confirm that resolution has been done directly by the service, which is trusted by client? Then firstbits resolution can be done over pidgin transport and nobody care, because the response is signed with service's key...

JSON-RPC don't offer any standard way how to sign messages, transports usually don't offer signatures either, so I want to fix it in some way. I'm not a crypto expert, but I have some idea how it should be done. Feel free to correct me and propose anything safer/easier:

Signature should be assembled from following information:
a) initial request
b) timestamp of processing
c) response

Signature should not be done from whole json message itself, only from payload, because json serialization can be somehow ambiguous, for example parameter 'id' can be changed during the transport.

I has been thinking about using GPG mechanism, which is widely used platform even for distributing trust, but it looks prettyheavy for such purpose. Not sure if it's possible to check GPG signatures in $3 AVR processor. However Bitcoin world already use signatures pretty nicely and clients already needs ecdsa libraries on their side for signing transactions, so using ecdsa sounds reasonable and pretty easy to me. Simple example for signing/verifying data using SECP256k1 curve in python can be found here: http://pastebin.com/LnPijjfm

However I see real issue in needed CPU time for signing. On my laptop, one signature takes around 200ms. It's low-end one, but it will take some time even on high-end servers. Any idea how to solve it?

Final signed response should looks similar like:
{"id": "1234", "signature": "MEUCIHM3m9IX6oudiBOXLx5bavACtB6uJNFDr8Ir6qVX54nhAiEA9SBgR1tOxEJnVJBqhKt+QfE0ry0h8yP2M7KinmQ/yPs=", timestamp: "1325202560", "result": ["blahblah"], "error": null}

However the question is which pattern/algorithm use to assemble data for signing (request payload, timestamp, response payload). Is there any reasonable solution for that?

Yes, I know about BCCAPI and bitcoinjs. BCCAPI is (right now) solving only part of the problem (just a subset which I call "wallet/blockchain services"), it's using custom binary protocol (in the oposite of json-rpc) and only polling transport (in the oposite of various transports). Also server implementation is closed source (I have code on gitorious already).

BitcoinJS has similar purpose. I talked with Stephan Thomas about bitcoinjs in Prague and the result is that he wants to do everything himself.

Even better. I'm able to write PHP binding, but not a Java one. Having a protocol implementation and at least "HTTP polling" transport (probably the easiest transport to implement) in Java would be amazing.

Yes, depending on firstbits resolution made by another party can be risky. However, fake firstbits in really smart way needs significant computation power, because server operator needs to find his own address which corresponds to given firstbits prefix. It's almost impossible in real time so unless you're selling house using firstbits address, there's no *real* way how to misuse that. However server can also make a mistake, for example by calculating firstbits on wrong blockchain branch...


yes, possible


Yes, possible. Also asking txradar service (on different server) can do the job easily.


It's a matter of days when I'll start beta server. I still need to finish some stuff to make it really usable.


TCP socket with SSL is already implemented. It's another transport (on another port), so it's absolutely to user which port/transport he choose.

Well, server cannot steal your money, it can only lie with transaction history or balance. I'm not saying it's not enough for some kind of attack, but it's still better than using API of some web-based wallet (like MtfGGox).

So yes - if you don't have full copy of blockchain, you need to trust somebody else, at least on some level. But the most of users or projects (like eshops) aren't so big that it could be profitable to cheat them from side of server operators. You can also ask multiple servers to confirm balances/history, which is limiting necessary trust to single entity.

Overall, it's more secure solution than using cart APIs of web wallets (like, heh, mybitcoin), because you still own your bitcoins.

Yes, I'm doing this already. However I like to open discussion and not everybody can understand python (and especially twisted) code. So I found writing textual proposal as a good way how to gain interest also between non-programmers.


It's exactly what I'm trying to do.


Good point. About DDoS - you can DDoS any service even without paying for it. You can just flood it with a lot of requests. So paying as a protection against *real* DDoS isn't a solution. Of course you can make service paid to avoid people misusing processing power. I'm already thinking about it, not sure if I found any viable solution.

Brainstorming: I'm thinking about "credit service" and standard exception "Fee required", which will indicate that service operator want some fee for performing such call. It's up to every server operator which calls will be free or paid. When client receive "fee required" exception, he needs to authenticate to "credit service" with prepaid credit. When previous failed call will be retried, server will credit the fee from such prepaid account. Not sure if this is a good way, but it's the best solution which I found so far. Comments welcome.


Basically, there's not so much things how can malicious server damage users (in serious way - like stealing their coins). However if client wants to be sure he has correct data, he can perform same calls to more overlay servers. If all of them are on valid blockchain, their responses should be the same.


No, it cannot. However there's real issue with services like firstbits. If server provide different result for firstbits lookup, client can send money to wrong address. However this can be solved by performing multiple calls to different servers.


I don't think so. However some people can do malicious things just because they can, without any specific reason. This is why I think that people will need to pick only trusted servers or confirm important calls against different overlay server.


You're exactly the person for who I'm doing this project :-). I see that maintaining full client is big overkill for some kind of projects which want to integrate with Bitcoin. And I'm almost sure there will be working server in less than one month. You're using PHP on your site, right? Please send me a PM with some details, I need to design PHP binding and I want to discuss it with somebody who's actively working in PHP...

Yes, I plan websocket as a transport oriented for in-browser clients. However websocket is more like fallback solution for environments where standard TCP socket isn't available. websockets have significant overhead and there's also a lot of versions, making it painful for implementation.

I just re-read JSON-RPC 2.0 specification (http://json-rpc.org/wiki/specification) and found that I missed "notification" type of messages, which very similar to what I proposed for pubsub mechanism. There was difference that notification has id:null, which I see as a smart choice, so I changed proposal documentation and also protocol implementation (not commited yet). So far, application protocol is now *fully* compatible with JSON-RPC 2.0.

done


yes, that's main purpose. so - done. I call it the "blockchain service", because it provide API for querying blockchain database for chainless clients.


Yes, this is possible, I call it the "wallet service", because it provides functionality of coin selection, creating of (unsigned) transactions etc.


I already implemented 'txradar' service, as an easy example of 'proxy service' (providing standard overlay-network API, but asking some other specialized service for doing the job). I see txradar service as usefull one, but it's fully on clients how they integrate such information.

I agree with you that projects like this are tending to be over complicated. Many times I found that I'm engineering some monster project just because I wanted to hide internal complexity of the task. That's exactly the reason why I'm forcing myself to keep this KISS: basic transports, simple RPC protocol, pubsub mechanism, services on top of this. If you know how to make it even simplier, tell me, seriously!

I'm trying to avoid "full stack" RPC mechanisms like SOAP+WSDL, because, honestly, it's pretty hard to implement everything correctly, on both ends.


I'm fully aware that those requirements are contradictory. It's pretty common that some solution needs to find some compromises. But correct me if you think that some of those requirements are wrong (for such purpose of overlay network).


Well, what's wrong on creating communcation channel and then have a possibility to call services on both ends, from any connected side?

I'm aware that it *can* be overcomplicated in the end, but it fully depends on designing of services. This idea is taken from jabber protocol, where both sides are providing some services. (And yes, I thought about building overlay network on top of jabber protocol, because it already provides service paradigm, but I think jabber is great example of overcomplicated stuff.)



I agree. I like RPC much more than message-based protocol, because it's making communication much cleaner. Just a request and its response. It's the reason why I'm not proposing communication based on async messages, because dependencies of various types of messages can turn into total mess.


I partially agree (not fully, because everything can be transformed into master-slave architecture, but of course it doesn't necessary mean that it's effective), but I'm not talking about bitcoin network. Difference between Bitcoin network and "Overlay" network is that Bitcoin is distributed database, but Overlay is network of services.


Hm, thanks to you, I'm thinking about this a little more than before. Is that really an issue? Isn't TCP checksumming and TCP retransmissions on both ends enough to "fix" corrupted information? More generally, keeping transmission working is task for the transport layer, not for protocol itself. And I agree that transport implementations should use it's internal mechanisms for checking that transmission was successfull. Like using Content-Length or content checksum in http headers.

RPC proposal itself contains unique IDs of requests to link requests to response. If transport layer fail, it will probably appear as disconnection, which also closes the session (on TCP transport), so client need to reset it's internal state after reconnecting (ask for balance, history etc, to be sure it is again in stable state).

Side note: I'm running json-based protocol on the pool over a year, I had over 3300 rq/s in June peak. I agree that mining protocol is stupid and ugly (although I understand the reason how and why it has been designed in such way - don't worry m0m ;-) ) and that there are some isssues with DDoSing etc. But I definitely didn't have any problems with corrupted packets like you're suggesting.


Fair summary, thanks. I'm familiar with SOAP (although I never used it in real life). As I mentioned above, I agree that Content-Length and Content-MD5 should be implemented, but it's part of transport in my concept (because HTTP is only one of many ways how to transfer bytes from one side to another, and Content-* are headers of HTTP), not a part of protocol so we are in agreement here.

About DTD - it's definitely better face of XML concept and I see some benefits in implementing DTD as formal specification of protocols. Although  I personally dislike XML, I'm open to change my mind at this point. Is here, as an example, standardized way how to serialize data such lists or dictionaries? I picked JSON because it is providing pretty compressed serialization of some standard structures in transparent and understandable way. I can call json_encode(any_data) and don't care about "how it works" in almost any programming language. But if there is some similar, widely accepted XML specification for serializing such objects, I'll elaborate. But handling XML streams on low level is usually pain.


Well, endianess in getwork isn't the mistake of json, but of layer above. However, I agree, it is a pain.


Thanks for explanation. I have only one experience with IPSec ten years ago and that experience was painful. Good to hear it gets better over the time. I feel I'm repeating myself, but I see that "transport" concept is one of the most powerful thing on my proposal. If somebody want to fiddle with IPSec, give it to him. But most of people will be happy with ssl-hardened TCP socket or even HTTP poll, which is very familiar for them. Both for programmers and for end users.


Actually I'm already in touch with two groups of people who're developing hardware based wallets, so my proposal was created with some specific projects in mind. I agree that final audience is pretty wide, but everything I can do is to keep KISS attitude and hope I won't overcomplicate it in some way.


Interesting stuff. I feel like overlay network should be "stateless", which mean that it will "forget" transaction once transaction has been succesfully broadcasted to Bitcoin network. The implementation of retransmissions should be more the task of clients built on top of overlay network. As far as overlay notify it's clients that their balance has been changed (because of blockchain reorg) and provide correct address history (using actuall chain branch), it's job should be done. The reason why I don't think overlay network should try to "fix" such issues by retransmissions is that it can turn into really complicated stuff. As an example - end user don't want to re-broadcast some transaction, because it has been created in the context of previous incoming transaction in "wrong" chain, which isn't actually stored in currect branch... But thanks for suggestion, I'll think about it more.

In the end, I really appreciate that our discussion is factual and we're discussing specific points more than "you're doing everything wrong". I think this is constructive and helps me in my project.

marcus, "Stratum" sounds cool and I especially like the meaning of the word. It's my favourite choice so far...

I'm sorry for today issue with block rewards, everything is fixed now.

I'll wait one or two days to collect few sounding names and then I offer the poll.

Thanks for another suggestions.

Names don't need to be related to Electrum. Name should be also pronounceable and easy to write ;-).

I pretty like Electricity, but try to put "Bitcoin Electricity" to Google ... ;-)

molecular, you probably wanted to post it there: https://bitcointalk.org/index.php?topic=55822 :-)

Hi 2112, can you please move your post to https://bitcointalk.org/index.php?topic=55842.0 to keep this thread more about client than about protocol proposal? I'd like to reply to your post there.

Grid servers of such overlay network will need full blockchain, so yes, they turn into some kind of "supernodes". Using lightweight clients for normal people is very rational choice, because those end users don't need to care about bandwidth, CPU usage and they don't need to wait for blockchain sync if they're using Bitcoin once per week.

Although I'm proposing "some kind of supernodes", I don't like centralizing of blockchain, I think it' real threat for Bitcoin. That's the reason why I keep running full bitcoin node on all computers where I can.