Electrum server discussion thread

[duncant]

I would try to rebuild of the abe database from scratch. (I know, it is slow...)

See ya in 3 weeks duncant :/

Okay, so I rebuilt the database. The first time I fired up the server, it worked perfectly. I started it with "cache = no" and "irc = no". After testing it, I wanted to enable those, so I stopped the server, which worked fine. I edited the config file and restarted the server. As soon as it started up, I got the traceback that I posted above and it doesn't work again. About 1 in 10 times, the server starts properly, the rest of the time I get the traceback above.

Another thing to note, is that when the server wasn't working properly, and I went to shut it down, "electrum-server stop" hung, without printing the "ok" message. The server did shut itself down, but the "electrum-server stop" command never finished. When I Ctrl-C'd it it printed the following traceback:

Code:

Traceback (most recent call last):
  File "/usr/bin/electrum-server", line 101, in <module>
    run_rpc_command(sys.argv[1], stratum_tcp_port)
  File "/usr/bin/electrum-server", line 75, in run_rpc_command
    o = s.recv(1024)
KeyboardInterrupt

=========== Frustration break occurred here ==========

As an experiment, I tried starting the server with iptables preventing incoming connections on port 50000. I waited until the "block number: XXXXXX" message was printed, then opened the firewall. I tried that a couple of times, and It appears to work fine when I do that. Perhaps there's a race condition between the stratum thread and the thread that interfaces with the database?

[1714469748]

1714469748

[1714469748]

1714469748

[duncant]

I tried starting the server with iptables preventing incoming connections on port 50000

and by port 50000 I mean port 50001

[slush]

That "KeyboardInterrupt" traceback is just telling you that you interrupted the script by Ctrl+C, nothing more.

electrum-stop script is connecting to the server over RPC, so if server is loading block database or is busy with something else, it won't response. Please send "tail electrum.log" here. You can also check "top" command; if database is under load, then electrum is updating Abe db.

[duncant]

That "KeyboardInterrupt" traceback is just telling you that you interrupted the script by Ctrl+C, nothing more.

Right, I thought that the relevant bit of that traceback was where the command was hanging.

electrum-stop script is connecting to the server over RPC, so if server is loading block database or is busy with something else, it won't response. Please send "tail electrum.log" here. You can also check "top" command; if database is under load, then electrum is updating Abe db.

I'm not quite sure what you're saying here, but the database isn't under any significant load. top shows that neither the python process, nor postgres are using any significant amounts of CPU. iotop also shows that there's no disk activity from either process.

Here's an example of the output when the server isn't running properly:

Code:

[bitcoin@uncle-enzo /]$ electrum-server
Starting Electrum server on uncle-enzo.mit.edu
Coin chain_id = 1
blockchain: 202904 blocks
TCP server started.
HTTP server started.
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/usr/share/electrum-server/processor.py", line 246, in run
    self.update()
  File "/usr/share/electrum-server/processor.py", line 260, in update
    self.send_response(internal_id, response)
  File "/usr/share/electrum-server/processor.py", line 276, in send_response
    session.send_response(response)
  File "/usr/share/electrum-server/transports/stratum_tcp.py", line 30, in send_response
    data = json.dumps(response) + "\n"
  File "/usr/lib/python2.7/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 201, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 264, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python2.7/json/encoder.py", line 178, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: <read-only buffer for 0x44606d0, size 25, offset 0 at 0x44664f0> is not JSON serializable

[12/10/2012-00:13:53] TCP <redacted> 10 1.0
[12/10/2012-00:13:53] TCP <redacted> 12 1.0
[12/10/2012-00:13:53] TCP <redacted> 7 1.01-e
[12/10/2012-00:13:54] TCP <redacted> 16 1.0
block number: 202904
Stopping Stratum
processor terminating
Server stopped
exit timer

I left the server running last night. It was working fine when I went to bed, but when I woke up this morning, I saw this in the logs, and the server is no longer responsive:

Code:

discarding tx id 3984266
discarding tx id 3984266
discarding tx id 4031277
discarding tx id 4031277
discarding tx id 4152198
discarding tx id 4320304
[12/10/2012-07:11:59] TCP <redacted> 7 1.0
[12/10/2012-07:13:13] TCP <redacted> 7 1.0
block number: 202952
block number: 202953
block number: 202954
[12/10/2012-07:39:06] TCP <redacted> 1 1.0
[12/10/2012-07:39:31] TCP <redacted> 1 1.01-e
discarding tx id 6769795
[12/10/2012-07:43:12] TCP <redacted> 8 0.52
block number: 202955
[12/10/2012-08:01:32] TCP <redacted> 7 0.61
block number: 202956
block number: 202957
[12/10/2012-08:04:07] TCP <redacted> 1 1.01-e
[12/10/2012-08:16:16] TCP <redacted> 1 1.01-e
[12/10/2012-08:19:01] TCP <redacted> 7 1.01-e
[12/10/2012-08:21:51] TCP <redacted> 7 1.01-e
[12/10/2012-08:25:39] TCP <redacted> 1 1.01-e
block number: 202958
[12/10/2012-08:32:03] TCP <redacted> 8 1.01-e
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/usr/share/electrum-server/processor.py", line 246, in run
    self.update()
  File "/usr/share/electrum-server/processor.py", line 260, in update
    self.send_response(internal_id, response)
  File "/usr/share/electrum-server/processor.py", line 276, in send_response
    session.send_response(response)
  File "/usr/share/electrum-server/transports/stratum_tcp.py", line 30, in send_response
    data = json.dumps(response) + "\n"
  File "/usr/lib/python2.7/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 201, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 264, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python2.7/json/encoder.py", line 178, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: <read-only buffer for 0x2f98d30, size 25, offset 0 at 0x2fe70b0> is not JSON serializable

[12/10/2012-08:32:30] TCP <redacted> 8 1.01-e
block number: 202959
[12/10/2012-08:40:05] TCP <redacted> 10 1.0
block number: 202960
[12/10/2012-08:44:31] TCP <redacted> 7 1.0
block number: 202961
discarding tx id 6478819
discarding tx id 6015062           
discarding tx id 5935605
discarding tx id 5935579
discarding tx id 5935292
discarding tx id 5933473

[slush]

Can you please add "print response.__repr__()" to line /usr/share/electrum-server/transports/stratum_tcp.py, line 30?

It is now:
data = json.dumps(response) + "\n"

Should be:
print response.__repr__()
data = json.dumps(response) + "\n"

This will give us some info what information is server trying to serialize...

[duncant]

Can you please add "print response.__repr__()" to line /usr/share/electrum-server/transports/stratum_tcp.py, line 30?

This is the object that it fails to serialize:

Code:

{'id': 15, 'result': [{'block_hash': '00000000000002d8c76e662906670ed570ccf1dcdab0f9ecc0fe40ff7aceda2d', 'is_input': 0, 'tx_hash': '39a539f1bf330287c259620b217232f5ea77d116f0f5a88d316d2957e4caf863', 'index': 1, 'outputs': ['1NFbPZr7KD3V8ZvjdssdnidepvjMSTARUZ', '1NAt9bC3fNCkoAXL2Bo3FqnXhWiZHb7vJ8'], 'timestamp': 1340853742, 'inputs': ['1DGUkqgt8ecdSmTtFosujuEx2nBxM2tXAQ'], 'value': 2000000000, 'height': 186565}, {'tx_hash': '5c297e69d89b2cdb8e46799711df7e8745aaed1e119652c9decaaf48e2140350', 'inputs': ['13HLfocvUzAz7dZghau3hpZJ2YhDRWzLMw', '1DqSR6MiL4kB62Z6YMmQ1KpPNsK23Kb5Po'], 'outputs': ['1372wdd5bwdwWf5HvzW7s8D9of6vcxFNj2', '1NAt9bC3fNCkoAXL2Bo3FqnXhWiZHb7vJ8'], 'height': 186572, 'block_hash': '00000000000002baa2474fa9bbcde9ff68cd1639d755a9dbeb933cc2e3fb8900', 'is_input': 0, 'index': 1, 'value': 6000000000, 'raw_output_script': <read-only buffer for 0x3bbe190, size 25, offset 0 at 0x3bc3130>, 'timestamp': 1340857517}, {'block_hash': '000000000000047f5c61faad21a0242ff37e087e804d0adeb3f9c61dd682a26d', 'is_input': 1, 'tx_hash': 'f32c96681776bc7065206ed53c6c62f0e47002384e5536153f4da243a5caed59', 'index': 0, 'outputs': ['1G7EXGeahYYmXsYVAfBjT9G8RjywhstuB8', '1BYKEj4odWztfvWXvYf6SvnQv5PtyRJiSp'], 'timestamp': 1348980703, 'inputs': ['1NAt9bC3fNCkoAXL2Bo3FqnXhWiZHb7vJ8'], 'value': -2000000000, 'height': 201182}, {'tx_hash': '5c297e69d89b2cdb8e46799711df7e8745aaed1e119652c9decaaf48e2140350', 'inputs': ['13HLfocvUzAz7dZghau3hpZJ2YhDRWzLMw', '1DqSR6MiL4kB62Z6YMmQ1KpPNsK23Kb5Po'], 'outputs': ['1372wdd5bwdwWf5HvzW7s8D9of6vcxFNj2', '1NAt9bC3fNCkoAXL2Bo3FqnXhWiZHb7vJ8'], 'height': 0, 'block_hash': 'mempool', 'is_input': 0, 'index': 1, 'value': 6000000000, 'raw_output_script': <read-only buffer for 0x3bbe110, size 25, offset 0 at 0x3bc3070>, 'timestamp': 0}, {'block_hash': 'mempool', 'is_input': 0, 'tx_hash': '39a539f1bf330287c259620b217232f5ea77d116f0f5a88d316d2957e4caf863', 'index': 1, 'outputs': ['1NFbPZr7KD3V8ZvjdssdnidepvjMSTARUZ', '1NAt9bC3fNCkoAXL2Bo3FqnXhWiZHb7vJ8'], 'timestamp': 0, 'inputs': ['1DGUkqgt8ecdSmTtFosujuEx2nBxM2tXAQ'], 'value': 2000000000, 'height': 0}, {'block_hash': 'mempool', 'is_input': 1, 'tx_hash': 'f32c96681776bc7065206ed53c6c62f0e47002384e5536153f4da243a5caed59', 'index': 0, 'outputs': ['1G7EXGeahYYmXsYVAfBjT9G8RjywhstuB8', '1BYKEj4odWztfvWXvYf6SvnQv5PtyRJiSp'], 'timestamp': 0, 'inputs': ['1NAt9bC3fNCkoAXL2Bo3FqnXhWiZHb7vJ8'], 'value': -2000000000, 'height': 0}]}

[slush]

hm, there's "raw output script", which is buffer object. Something is broken here. Are you using latest git?

[duncant]

hm, there's "raw output script", which is buffer object. Something is broken here. Are you using latest git?

Yes. I actually wrote an Archlinux PKGBUILD that clones git://github.com/spesmilo/electrum-server.git

[slush]

Hm, raw output script should be in hex form. It looks like this variable is filled directly from the database (https://github.com/spesmilo/electrum-server/blob/master/backends/abe/__init__.py), so maybe this is related to version of Abe and/or postgresql driver in Abe?

I know that Abe is doing wild variable conversions for keeping compatibility with wide area of databases, so maybe this is question to Abe developers, more than Electrum developers...

[duncant]

Hm, raw output script should be in hex form. It looks like this variable is filled directly from the database (https://github.com/spesmilo/electrum-server/blob/master/backends/abe/__init__.py), so maybe this is related to version of Abe and/or postgresql driver in Abe?

I know that Abe is doing wild variable conversions for keeping compatibility with wide area of databases, so maybe this is question to Abe developers, more than Electrum developers...

Oh! I set "binary-type = pg-bytea" in abe.conf. I'll bet that if I set it back to "binary-type = hex" it'll get fixed.

[slush]

Good luck by reindexing blockchain again

[duncant]

The server has been running for ~48 hours with no problems, so I'm willing to declare this problem solved. In case anyone comes across this problem in the future, setting "binary-type = hex" in abe.conf did the trick.

[slush]

The server has been running for ~48 hours with no problems, so I'm willing to declare this problem solved. In case anyone comes across this problem in the future, setting "binary-type = hex" in abe.conf did the trick.

So Abe on Postgre won't work with default Abe config? Maybe we should do some workaround in Electrum and expect binary data there. Converting binary data to hex isn't so hard and I bet that storing binary data in database will save lot of storage space...

[duncant]

So Abe on Postgre won't work with default Abe config? Maybe we should do some workaround in Electrum and expect binary data there. Converting binary data to hex isn't so hard and I bet that storing binary data in database will save lot of storage space...

The default is for binary-type to be unset. I don't know what Abe does in that case. By setting "binary-type = pg-bytea", you force Abe to use Postgresql's BYTEA storage type. Setting "binary-type = hex" does the opposite and forces Abe to use hex strings instead.

The module binascii has a function hexlify that will convert buffer objects to hex strings.

[ThomasV]

I pushed SSL support to the repo; it will be used by the next release of the client.
if you have a server, please upgrade 

[ThomasV]

Two server upgrades this morning:

 - the protocol version is now 0.2. this indicates servers that have get_header and get_merkle

 - the server now uses 'getrawmempool' (bitcoind 0.7); with this version, you do not need to patch bitcoind anymore in order to run a server.

[slush]

Thomas, what is wrong with getrawmempool in 0.7.0 ? On my server (renamed from california.stratum.bitcoin.cz to electrum.bitcoin.cz because of IRC limitation in name length), I'm using "getrawmempool" with 0.7.1 without any (obvious?) issues.

Btw thanks to my server update, electrum.bitcoin.cz now supports SSL!

[flatfly]

Thomas, what is wrong with getrawmempool in 0.7.0 ? On my server (renamed from california.stratum.bitcoin.cz to electrum.bitcoin.cz because of IRC limitation in name length), I'm using "getrawmempool" with 0.7.1 without any (obvious?) issues.

Btw thanks to my server update, electrum.bitcoin.cz now supports SSL!

Confirmed, I'm now connected through SSL (using client 1.1)  Great stuff!

[EDIT] It seems the client is briefly losing its connection every minute or so (then spontaneously reconnects after a few seconds.)  I'm also getting this error message in the console:

 SSLError: The read operation timed out

[duncant]

I updated uncle-enzo.mit.edu to the latest version of the server. I can't connect using TCP/SSL, though. Port 50002 is open, but I still get the "Failed to connect uncle-enzo.mit.edu:50002" message when I try to connect. I'm using a self-signed certificate, would that cause problems?

[ThomasV]

I updated uncle-enzo.mit.edu to the latest version of the server. I can't connect using TCP/SSL, though. Port 50002 is open, but I still get the "Failed to connect uncle-enzo.mit.edu:50002" message when I try to connect. I'm using a self-signed certificate, would that cause problems?

no, you can use a self-signed certificate