[ANNOUNCE] Electrum - Lightweight Bitcoin Client

[slush]

so, after a rough look, it seems you implemented "firstbits" lookup by falling back to firstbits.com api, right?

This seems to me it would be the time now to start implementing a "real" server?

Yes, writing server logic is one of next step in the project. My current goal is to provide specification/implementation of something which can be used for writing custom lightweight clients or even application libraries integrating Bitcoin. For the example, it should be extremely simple to integrate Bitcoin with any PHP site (online cart) using HTTP Push protocol described in the specs.

Code:

What exactly are you asking us to do?
[/quote]

a) Read specification, propose improvements and criticism. Isn't something important missing? Isn't that overcomplicated in some parts? 

b) Read and test implementation. Isn't there any obvious bug? Does it work on your computer without any errors? Isn't the implementation overcomplicated in some points? Especially the split between protocol, transports, then mechanism of services and service discovery.

Well, personally I think that service mechanism can be considered as overcomplication by many of people. And I think that nobody really read the code, because nobody mentioned it :-). I really want some criticism and opinion of other people, because it's pretty normal that author overlook something obvious.

[molecular]

I'd like to enourage pythonist interested in Electrum client to review code implementing new network protocol (proposed in https://docs.google.com/document/d/17zHy1SUlhgtCMbypO8cHgpWH73V5iUQKk_0rWvMqSNs/edit?hl=en_US - the last chapter is not finished yet). Repository of server is https://gitorious.org/electrum-server/electrum-server.

I started playing around a bit with your code.

I think I managed to produce a bug (don't know where it is, though, might be in the upstream server) using the following code in the client.

Code:

    addy = (yield f.rpc('firstbits.resolve', ['1B3SX6Y',]))
    yield f.rpc('blockchain.address.get_balance', [addy])
    yield f.rpc('blockchain.address.get_history', [addy])

produces this:

< {'params': [u'1B3SX6YRmVkXAxeNioAyWvjdfcek5eCFa7'], 'id': 14, 'method': 'blockchain.address.get_history'}
> {u'result': None, u'id': 14, u'error': [-1, u"Traceback: <type 'exceptions.Exception'>: Corrupted data from old electrum server\n/usr/lib/python2.7/site-packages/Twisted-11.1.0-py2.7-linux-i686.egg/twisted/internet/defer.py:362:callback\n/usr/lib/python2.7/site-packages/Twisted-11.1.0-py2.7-linux-i686.egg/twisted/internet/defer.py:458:_startRunCallbacks\n/usr/lib/python2.7/site-packages/Twisted-11.1.0-py2.7-linux-i686.egg/twisted/internet/defer.py:545:_runCallbacks\n/usr/lib/python2.7/site-packages/Twisted-11.1.0-py2.7-linux-i686.egg/twisted/internet/defer.py:1095:gotResult\n--- <exception caught here> ---\n/usr/lib/python2.7/site-packages/Twisted-11.1.0-py2.7-linux-i686.egg/twisted/internet/defer.py:1039:_inlineCallbacks\n/mnt/sda3/home/nick/bitcoin/electrum-server/helpers.py:91:ask_old_server\n"]}
Unhandled error in Deferred:
Unhandled Error
Traceback (most recent call last):
Failure: custom_exceptions.RemoteServiceException: (-1, u"Traceback: <type 'exceptions.Exception'>: Corrupted data from old electrum server\n/usr/lib/python2.7/site-packages/Twisted-11.1.0-py2.7-linux-i686.egg/twisted/internet/defer.py:362:callback\n/usr/lib/python2.7/site-packages/Twisted-11.1.0-py2.7-linux-i686.egg/twisted/internet/defer.py:458:_startRunCallbacks\n/usr/lib/python2.7/site-packages/Twisted-11.1.0-py2.7-linux-i686.egg/twisted/internet/defer.py:545:_runCallbacks\n/usr/lib/python2.7/site-packages/Twisted-11.1.0-py2.7-linux-i686.egg/twisted/internet/defer.py:1095:gotResult\n--- <exception caught here> ---\n/usr/lib/python2.7/site-packages/Twisted-11.1.0-py2.7-linux-i686.egg/twisted/internet/defer.py:1039:_inlineCallbacks\n/mnt/sda3/home/nick/bitcoin/electrum-server/helpers.py:91:ask_old_server\n")

The servers spews a load of tx infos (as probably expected), but the json string seems to have overflowed or something.

I tried investigating:
  http://blockexplorer.com/address/1B3SX6YRmVkXAxeNioAyWvjdfcek5eCFa7
One of the transactions being pulled in:
  http://blockexplorer.com/tx/caa3eaa7bb72046f6e6cc81e21724d81271049bf493961592f63840b030b822b

So it's a lot of data.

[ThomasV]

I'd like to enourage pythonist interested in Electrum client to review code implementing new network protocol (proposed in https://docs.google.com/document/d/17zHy1SUlhgtCMbypO8cHgpWH73V5iUQKk_0rWvMqSNs/edit?hl=en_US - the last chapter is not finished yet). Repository of server is https://gitorious.org/electrum-server/electrum-server.

I started to implement a few of your RPCs.
I realize that 'subscribe' and 'unsubscribe' are not sufficient to handle sessions; we need a rpc that returns a session ID.

[grondilu]

Yes, writing server logic is one of next step in the project. My current goal is to provide specification/implementation of something which can be used for writing custom lightweight clients or even application libraries integrating Bitcoin. For the example, it should be extremely simple to integrate Bitcoin with any PHP site (online cart) using HTTP Push protocol described in the specs.

That would be awesome.

[slush]

I realize that 'subscribe' and 'unsubscribe' are not sufficient to handle sessions; we need a rpc that returns a session ID.

Session handling isn't finished yet, because it must be more abstract than an "HTTP session" (to be compatible with all possible transports). I'm also preparing simple framework for subscribe/unsubscribe, so manual implementation for every service won't be necessary. So I recommend to wait to this before real implementation of any pubsub service.

[2112]

Session handling isn't finished yet, because it must be more abstract than an "HTTP session" (to be compatible with all possible transports). I'm also preparing simple framework for subscribe/unsubscribe, so manual implementation for every service won't be necessary. So I recommend to wait to this before real implementation of any pubsub service.

From my previous conversations with slush I know that he is dead-set on reinventing the wheel, so I'm not going even try to change his opinion.

But if anyone else involved in software architecture is reading this thread, please acquaint yourself with prior art:

http://en.wikipedia.org/wiki/0MQ
http://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol

or my other proposition: find some sort of canonical serialization of Bitcoin over FIX:

http://en.wikipedia.org/wiki/Financial_Information_eXchange

The last choice gives you an additional benefit of being compatible with pretty much everything that is used to trade anything fungible.

The satisfaction you'll get from reinventing a working wheel is short lasting, really. I almost guarantee that once some time passes you will be ashamed of your pride of re-inventing the wheel.

[slush]

From my previous conversations with slush I know that he is dead-set on reinventing the wheel, so I'm not going even try to change his opinion.

I of course know messaging systems like Zero MQ, mainly because I worked as enterprise architect for biggest Czech bank for few years.

The reason why I didn't choosed FIX or ZeroMQ is that they really don't provide any real advantage over JSON-RPC, which is well known in opensource world and has superior support in any programming language without any additional modules. For example, ZeroMQ in PHP needs compiling custom C++ module, which may be real barrier for PHP programmers running sites on shared hosting.

Actually I don't feel that I reinvented wheel, I just picked well known technologies and combined them together. JSON-RPC is widely used and good for this purpose, provides full RPC specs including exception handling. Then I added "service" layer on the top (just naming convention of RPC methods, nothing more!) and idea of "transports" under JSON-RPC.

The whole point of this is that everybody can write a client without any special knowledge or _any_ external library, because json encoder/decoder or curl/ajax is available everywhere in standard libraries (C++, PHP, Java, Python, Javascript, ...).

I'm open to any constructive criticism, so please provide me any real advantages of using FIX or ZeroMQ instead of using JSON-RPC standard. I fully understand that you personally like such technologies. I considered them too, but JSON-RPC looks like better choice for me.

Don't forget that this protocol is slightly different than current mining protocol which we discussed before. I agree that long polling mechanism is step in wrong way and this isn't part of Electrum protocol proposal.

[Red Emerald]

I'm open to any constructive criticism, so please provide me any real advantages of using FIX or ZeroMQ instead of using JSON-RPC standard. I fully understand that you personally like such technologies. I considered them too, but JSON-RPC looks like better choice for me.

I <3 JSON, so no complaints from me.

[2112]

I'm open to any constructive criticism, so please provide me any real advantages of using FIX or ZeroMQ instead of using JSON-RPC standard. I fully understand that you personally like such technologies. I considered them too, but JSON-RPC looks like better choice for me.

Dear Slush!

In this holiday season I'd love to speak to you directly over a small container of a product that made the Czech city of Plzeň world-famous instead of trading jabs on the Internet forum. Thus please don't take the following critique as a personal slight.

JSON-RPC is a hammer & nail combo in search of a wall to pound on.

1) Hammer: RPC over HTTP(S) is a horrible transport for the financial data. The main problem is that a site cannot really distinguish sudden spikes in popularity from distributed denial of service attack. The tools for DDoS defense and normal load balancing are popular and comparatively well understood, but they are create more side-effect problems that they solve. In my experience the transport has to rely on (comparatively) long lasting network sessions and instead of lowest-common-denominator SSL it should use some other form of network security technology like IPSec or any available VPN implementation (but not SSL-VPNs). Please take the advantage of the fact that your design involves two custom programs on both ends of the pipe. You do not need to use the protocol that was designed to support dumb and full-of-holes web browser on the client side.

2) Nail: JSON is essentially a LISP s-expressions sweetened with Javascript syntax. This is a great improvement over XML/SOAP but the lack of transport error-detection and packet boundary weakens all implementations. All the transport-level problems show up as hard to troubleshoot "syntax errors" or just pass undetected. The syntax error recovery with a stack of open-close braces is a really weak point. When the Internet is ubiquitous with "transparent buffers" (that silently overflow) and "transparent proxies" (that silently cache or otherwise mangle) you'll need all you can do do detect and bypass them.

3) Pounding the wall: By committing to a rather primitive tool you also commit to solving many of the well known problems with the RPC paradigm. By the time you correctly implement the garbage collection of stale subscribers, reciprocal authentication of callbacks, etc. you'll understand why Microsoft delivered DCOM (Distributed COM) years after COM (Component Object Model). Please learn from MSFT mistakes and don't undertake an experimental re-verification of their money sinks. I'm not a big fan of DCOM, but it is a great example of how a full and efficient implementation of simple RPC concept needs to be either very complex or very fragile. Also, when speaking of DCOM please disregard ActiveX GUI-overlays on that protocol; they don't pertain here.

I think that Electrum has a much greater potential than the current Satoshi client:

1) from the start it undertakes the correct abstraction-level split between:
1a) wallet-less shared server that knows the block-chain and p2p broadcast network;
1b) blockchain-less private client that concerns itself mostly with security and UI;
2) by not directly supporting mining it will not get involved in the rigmaroles with mining pool operators.

Lastly I want to stress that I don't "personally like such technologies." I simply have a significant experience of fixing the problems for clients who painted themselves into a corner by using some form RPC when a real duplex-communication protocol should have been used in the implementation from the start. Although I'm not directly involved in finance-industry IT anymore I know that the solutions I co-architect-ed and consulted have successfully withstood the test of time for over a decade. I also know that competitors who went for a quick-to-implement solution paid tremendous price in form of the maintenance costs.

Thank you for your time and attention.

Edit: whatever protocol you choose please make sure that it supports block-chain rollback from the start. You don't have to implement it in the initial release, but it has to be there. I'm forever grateful to Gavin Andresen for refusing to remove or tamper with the chain-rollback logic in the Satoshi client.

[marcus_of_augustus]

Can you confirm if the following is correct?

The imported keys are not contained among the deterministic key set generated so if/when you delete your wallet the imported keys will not be coming back when you use the seed-phrase to regenerate the deterministic portion of the wallet.

that is correct. perhaps I should add a big fat warning during the import command?
in any case, I think that this command should not be available from the gui, in order to make sure that it is not used too easily.

Thanks. A big, fat warning would be good.

I knew it would be as much the case but wanted to make it clear to those who are unaware ... if it was left 'unsaid' who knows what grief may come back on Electrum for no good reason, except that it had an extra feature that was misunderstood.

[Red Emerald]

2112, what would you propose as an alternate to JSON-RPC that can easily work with web and desktop clients?

[genjix]

I think that Electrum has a much greater potential than the current Satoshi client:

I agree with this totally. I will be supporting whatever spec ThomasV/slush come up with using libbitcoin. Electrum is the way forwards.

I would like though (as we discussed on IRC) if you could give the protocol a different name to distinguish it from the client. This mix-up has been very confusing and unfortunate in bitcoin.

[slush]

1) Hammer: RPC over HTTP(S) is a horrible transport for the financial data.

2112, I think you're mixing a lot of things together. Firstly, you're talking about something like HFT, so you worry about latency, data consistency etc. I'm talking about supportive network above Bitcoin layer, where consumers are mostly end user machines or websites with eshops. 1000ms latency definitely isn't an issue here, unlike trading NASDAQ. I'm not designing financial network in classic meaning of this word.

Second issue is that I'm NOT proposing RPC over HTTP(S). I'm proposing RPC over various set of transports, where HTTP(S) is only one of possible way, more like fallback alternative for clients unable to communicate in another way. If you read my implementation, you see that I implemented long-living TCP socket as a default and main transport, NOT a HTTP(S).

The main problem is that a site cannot really distinguish sudden spikes in popularity from distributed denial of service attack. The tools for DDoS defense and normal load balancing are popular and comparatively well understood, but they are create more side-effect problems that they solve.

I'm sorry, you are locked in your opinion that I'm designing JSON over HTTPS, so your comments are out of logic.

Again, proposed solution *uses* one TCP socket per one "session", which is perfectly compatible with all DDoS mitigation tools. Attacked site *can* turn off HTTP(S) layer and offer only TCP socket during the problems. By the way, you're again mixing transporting layer with application protocol. I don't know how can format of application protocol (FIX, ZeroMQ or JSON) affects mitigation of DDoS, as far as all mentioned protocols are de-facto standards and every tools (probably) knows JSON, too.

In my experience the transport has to rely on (comparatively) long lasting network sessions and instead of lowest-common-denominator SSL it should use some other form of network security technology like IPSec or any available VPN implementation (but not SSL-VPNs). Please take the advantage of the fact that your design involves two custom programs on both ends of the pipe.

Did you argue in the same way in (cca) 1993 when they created HTTP? That users need custom programs on both ends? I'm proposing and implementing transports with long-lasting network sessions, using widely used JSON protocol on application layer; where you read the oposite? Again, HTTP(S) is only compatibility mode for browsers and other devices unable to handle sockets directly. I personally don't like HTTP(S) too, but I want to give freedom of choice to developers. If they develop application which works only on HTTP and server provider closes HTTP transport during the DDoS attack, hmm, it's their problem.

You do not need to use the protocol that was designed to support dumb and full-of-holes web browser on the client side.

I'm sorry, but we're probably missing the whole point of this discussion. You're designing hi-secure network protocol between two hi-tech businesses using dedicated machines with any possible software setups on both sides, so VPNs and IPsec isn't a problem to run there. I'm designing network used by end users, and one of side can be web browser or even hardware wallet written in C on some $3 AVR processor.

JSON is essentially a LISP s-expressions sweetened with Javascript syntax. This is a great improvement over XML/SOAP but the lack of transport error-detection and packet boundary weakens all implementations. All the transport-level problems show up as hard to troubleshoot "syntax errors" or just pass undetected.

I understand what you mean and I partially agree. I already thought about adding some CRC into json payload, which should do the job. Actually it's not so important, because the chance that packet breaks in the way that it will be a valid json message, only with different data inside, is really neglible. But (thanks to json flexibility) there can be added some field containing CRC of the message without breaking backward compatibility, which is enough for me at this moment. When CRC become necessary, it can be implemented.

3) Pounding the wall: By committing to a rather primitive tool you also commit to solving many of the well known problems with the RPC paradigm. By the time you correctly implement the garbage collection of stale subscribers, reciprocal authentication of callbacks, etc. you'll understand why Microsoft delivered DCOM (Distributed COM) years after COM (Component Object Model). Please learn from MSFT mistakes and don't undertake an experimental re-verification of their money sinks. I'm not a big fan of DCOM, but it is a great example of how a full and efficient implementation of simple RPC concept needs to be either very complex or very fragile. Also, when speaking of DCOM please disregard ActiveX GUI-overlays on that protocol; they don't pertain here.

I think you're overcomplicating the task. Btw there are not such issues like stale subscribers, because subscriber "dies" once the session is over. There is no authentication as services are public. By the way, it's more like implementation details in server than protocol specification itself.

Your proposals of ZeroMQ or FIX sounds cool and like you really understand the domain, but they don't answer problems I'm trying to solve. Those are only application protocols, and pretty complicated ones (think about that AVR client!), they're comparable with JSON (well, both three solutions have their own problems, every solution slightly different). Finally, even if I pick ZeroMQ, I'll need to add some custom way how to support HTTP poll or HTTP push, because I *want* to provide services for this type of clients. Is there any real-world solution covering all requirements which I have to the final protocol? Something which I can use "as is"? If so, send me a link and I'll elaborate.

[Red Emerald]

Well put slush.  I'm glad you are working on the projects that I like.

[Deafboy]

Hello,
I don't want to cry on IRC anymore about this. Instead I will leave this post here.. maybe someone will figure it out and help me in the future.

This is the message I get while starting electrum server:

Code:

ERROR:Abe.DataStore:Failed to catch up {'blkfile_number': 1, 'dirname': '/home/bitcoin/.bitcoin', 'chain_id': None, 'id': 1L, 'blkfile_offset': 0}
Traceback (most recent call last):
  File "/usr/local/lib/python2.6/dist-packages/Abe/DataStore.py", line 2141, in catch_up
    store.catch_up_dir(dircfg)
  File "/usr/local/lib/python2.6/dist-packages/Abe/DataStore.py", line 2162, in catch_up_dir
    store.import_blkdat(dircfg, ds)
  File "/usr/local/lib/python2.6/dist-packages/Abe/DataStore.py", line 2277, in import_blkdat
    store.import_block(b, chain_ids = chain_ids)
  File "/usr/local/lib/python2.6/dist-packages/Abe/DataStore.py", line 1570, in import_block
    (block_id, tx['tx_id'], tx_pos))
  File "/usr/local/lib/python2.6/dist-packages/Abe/DataStore.py", line 403, in sql
    store.cursor.execute(cached, params)
  File "/usr/lib/pymodules/python2.6/MySQLdb/cursors.py", line 166, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib/pymodules/python2.6/MySQLdb/connections.py", line 35, in defaulterrorhandler
    raise errorclass, errorvalue
OperationalError: (1048, "Column 'tx_id' cannot be null")
Traceback (most recent call last):
  File "/home/bitcoin/data/server.py", line 728, in <module>
    memorypool_update(store)
  File "/home/bitcoin/data/server.py", line 585, in memorypool_update
    v = v['transactions']
TypeError: 'NoneType' object is unsubscriptable

In older versions of server, importing of blocks was just bloody slow. In recent version it is not importing at all.


.bitcoin folder re-created from scratch
MySQL database removed and created again

Result is the same...

[Red Emerald]

I've been trying to setup a bitcoin node as a tor hidden service and have been trouble actually being able to connect.  Now I am thinking that it might be easier for me to run a tor hidden service for electrum.

I mention some of the problems here https://bitcointalk.org/index.php?topic=50547.0

Essentially, the satoshi client either announces it's IP on IRC or it closes its listening port.  This is a problem since a properly configured tor hidden service needs to hide its IP while still listening on localhost for someone directly connecting.  Is the electrum server capable of running as a secure hidden service?  I've yet to actually setup a server, but am really interested in the project and am wanting to setup a server soon.

Side note - I'm assuming electrum listens on 8333 since the satoshi client doesn't support any other ports, but this should probably be something that is easy to change with a simple flag when/if the "official" client becomes a little smarter.

Edit: I dunno how I hadn't realized that electrum is built on top of a patched bitcoind. That makes sense since reimplementing all of that code is a lot of added work. Of course, this also means that there are the same exact problems with running the bitcoin node as a hidden service.  It looks like electrum itself should work as a hidden service, but I still need to do some testing.

Edit 2: Wow getting pygtk to work on a mac is not as easy as I had hoped

[ThomasV]

Hello,
I don't want to cry on IRC anymore about this. Instead I will leave this post here.. maybe someone will figure it out and help me in the future.

This is the message I get while starting electrum server:
...
In older versions of server, importing of blocks was just bloody slow. In recent version it is not importing at all.


.bitcoin folder re-created from scratch
MySQL database removed and created again

Result is the same...

that traceback was a bug in the server that I introduced recently; try now, it should be fixed.
I do not know what causes the import of blocks in abe to be slow. you might want to ask the abe developer about that. on my machine it was reasonable.

[molecular]

It might not be a good idea to even bring this up (I might get slaughtered and my blood be delivered as a sacrify to the bitcoin gods), but since we're at a very early stage and this is something that can't be easily added as an afterthought, I'll just ask...

What about support for multiple currencies (alt-chains) in the protocol and multiple wallets in the client?

What are the pros/cons of this?

pro

• makes electrum very desirable for people using alt-chains, that's good for adoption
• the abstractions introduced for this in the client/protocol might come in handy when implementing FIAT wallet and exchange functionality

con

• may discredit electrum in the eyes of many/some ("Electrum is that alt-chain client")
• makes protocol and implementations more complicated

Anything to add to the lists?

Disclaimer: I'm not pushing for this, just putting it up for discussion. I don't have an opinion on the issue (yet).

[terrytibbs]

It might not be a good idea to even bring this up (I might get slaughtered and my blood be delivered as a sacrify to the bitcoin gods), but since we're at a very early stage and this is something that can't be easily added as an afterthought, I'll just ask...

What about support for multiple currencies (alt-chains) in the protocol and multiple wallets in the client?

What are the pros/cons of this?

pro

• makes electrum very desirable for people using alt-chains, that's good for adoption
• the abstractions introduced for this in the client/protocol might come in handy when implementing FIAT wallet and exchange functionality

con

• may discredit electrum in the eyes of many/some ("Electrum is that alt-chain client")
• makes protocol and implementations more complicated

Anything to add to the lists?

Disclaimer: I'm not pushing for this, just putting it up for discussion. I don't have an opinion on the issue (yet).

Our good friend ThomasV has said on IRC, and I quote, that "bitcoin clones are as pointless as clones of jesus", genjix reverted and locked the Litecoin article on the Bitcoin wiki in a state filled with unfounded and opinionated statements mostly written by our friend Luke-Jr (Luke's opinion on Litecoin in a nutshell on the Wiki, as written by Luke himself), so, yeah, I don't think this will fly too well.

Remember, ignorance is bliss!

[molecular]

Remember, ignorance is bliss!

Thanks for your opinion, I hope I didn't start any "currency wars" in this channel :|.