[BETA] MTGox websocket API, testers wanted

[Ente]

OH! Thank you a lot for your help, zelyony!

I was already wondering about the heartbeat. I now answer 1:: and 2:: accordingly, now they keep flowing in constantly.
Well, the socket *must* be working, since the websites using it still work. There would be more outcry in the forum if it was dead altogether..

I am glad I'm not the only one here!

Ente

[1713892957]

1713892957

[1713892957]

1713892957

[1713892957]

1713892957

[1713892957]

1713892957

[1713892957]

1713892957

[Ente]

*bangs head on desk*
I cant get it to work.

1 BTC bounty for the first person who helps me receive regular data again over the websocket!

Here is my (totally stripped) python code:

Code:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from websocket import create_connection
import urllib
import re
import time

try:
    import json
except ImportError:
    import simplejson as json


wsurl = "wss://socketio.mtgox.com/socket.io/"
iourl = "https://socketio.mtgox.com/socket.io/"

############

print "Connecting to socket.io.."
url = iourl + "1"
f = urllib.urlopen(url)
output = f.read()
f.close()
ausgabe = re.search('[0-9]+', output, 0)
sessionid = ausgabe.group()

print "New websocket url:"
wssurl = wsurl + "1" + "/websocket/" + sessionid
print wssurl

############

print "Connecting to websocket..\n"

socket = create_connection(wssurl)

while 1:
  echo =  socket.recv()
  if "1::" in echo:
    print "Heartbeat 1"
    socket.send("1::")
  elif "2::" in echo:
    print "Heartbeat 2"
    socket.send("2::")
  else:
    print "Message: ",echo
  time.sleep(2)
      

The websocket client I use here is from http://pypi.python.org/pypi/websocket-client/0.4.
I have a rudimentary heartbeat added here.
The whole script dies eventually, from having stripped all error-handling.
It worked some weeks ago. If you like to see the whole script, let me know.

Bedtime here, will check back in, like, 9 hours..

*frustrated and tired*

Ente

[Ente]

1 BTC bounty for the first person who helps me receive regular data again over the websocket!

I made a new thread for this error-solving bounty:
https://bitcointalk.org/index.php?topic=63969.msg749747#msg749747

Ente

[zelyony]

right now datas arrived (my format after string -> json -> object -> string):
JSON: Packet={ Depth={ price=4,67000 USD,       vol=0,05300000,         dir=bid,        when=2012:02:15_10:03:56.735844 } }
JSON: Packet={ Ticker={ bid=$4.68100,   ask=$4.70075,   lo=$4.21211,    hi=$4.97000,    last=$4.70075,  vol=256,165.42727403??BTC } }

I dont know Python, but:
- on connected (or after recv "1::" - that's mean u connected now) u must send strings like:
4:::{"op":"mtgox.subscribe","type":"trades"}
- on "2::" u must response "2::" (not "1::" as in ur sample)

[Ente]

Thank you for checking, zelyony!
You are right, thats a typo in the snippet. I sent '1::' and '2::' accordingly in my full script, typo corrected.

I sent '4:::{"op":"mtgox.subscribe","type":"trades"}' over the websocket, no change. No response whatsoever, no trade-messages coming in.

Ente

[zelyony]

and right now I dont recv any useful datas  

UPD:
it's working right now (13:44 UTC)

UPD2:
it's working if wait connected sometimes 20 minutes and sometimes 0 minutes and sometimes "never". very strange and no good.
can someone's debug session stops sending messages to other users?.. who knows implementation of socket.io server?

To MtGox:
maybe it's sabotage.. Slow Read DoS attack.. bad guys overflow send pool with slow connections and.. voila!
will be better for all clients drop connections that cannot read string, i.e. send to client not finished for.. hmm.. 0.5sec! +ban by IP 10min (epoll, select or other asynchronous mechanism). I took digits from air, I dont know % loading for servers machines and number of connections.
p.s. ping ("2::") works always

[runeks]

I have a hard time following all the tips and tricks in this thread on getting the socket.io interface to work with Python.

Has anyone gotten it to work properly? As far as I can tell there are no socket.io client implementations for Python right, right?

[Ente]

UPD2:
it's working if wait connected sometimes 20 minutes and sometimes 0 minutes and sometimes "never". very strange and no good.
can someone's debug session stops sending messages to other users?.. who knows implementation of socket.io server?

Same here. Sometimes it takes ages to receive any data, sometimes less than a minute. Totally random, and totally annoying. The 1:: and 2:: pings work straight from the beginning.
As soon as I get a response to my channel-connect, data from this channel comes in too.

Ente

[molecular]

I have a hard time following all the tips and tricks in this thread on getting the socket.io interface to work with Python.

Has anyone gotten it to work properly? As far as I can tell there are no socket.io client implementations for Python right, right?

maybe this helps, my socketio.py:

from threading import *
import urllib2, urllib
import simplejson as json
import ssl, socket
import time
from websocket_client import create_connection
import traceback

class SocketIO:
  def __init__(S, url, callback):
    S.url = url
    S.callback = callback
    
  def connect(S):
    try:
      data = urllib.urlencode({})
      req = urllib2.Request('https://' + S.url + "/1", data)
      print 'https://' + S.url + "/1"
      response = urllib2.urlopen(req)
      r = response.read().split(':')
      S.heartbeat_interval = int(r[1])
      #print 'heartbeat: ', S.heartbeat_interval
      if 'websocket' in r[3].split(','):
        print "good: transport 'websocket' supported by socket.io server ", S.url
        S.id = r[0]
        print "id: ", S.id
      else:
        print "error: transport 'websocket' NOT supported by socket.io server ", S.url
        
      S.thread = Thread(target = S.thread_func)
      S.thread.setDaemon(True)
      S.thread.start()
    except:
      traceback.print_exc()      

  def stop(S):
    S.run = False
    S.thread.join(timeout=1)
    S.keepalive_thread.join(timeout=1)


  def thread_func(S):
    print 'SocketIO: websocket thread started'
    
    my_url = 'wss://' + S.url + "/1/websocket/" + S.id
    
    S.ws = create_connection(my_url)
    
    #S.ws = WebSocket(my_url, version=0)
    S.run = True
    S.ws.send('1::/mtgox')

    # start keepalive thread
    S.keepalive_thread = Thread(target = S.keepalive_func)
    S.keepalive_thread.setDaemon(True)
    S.keepalive_thread.start()
    
    msg = S.ws.recv()
    while msg is not None and S.run:
      #print 'SocketIO msg: ', msg
      if msg[:10] == "4::/mtgox:":
        S.callback(msg[10:])
      #elif msg[:3] == "2::":
      #  True
      #else:
      #  print "SocketIO: dont know how to handle msg: ", msg
      msg = S.ws.recv()
    S.ws.close()
      
  def keepalive_func(S):
    while S.run:
      try:
        S.ws.send('2::');
      except:
        if S.run:
          print 'error sending keepalive socket.io, trying reconnect'
          S.connect()
        else:
          print 'exiting socket.io keepalive thread'
      time.sleep(S.heartbeat_interval)
      
def test_callback(msg):
  print 'msg: ', msg

# testcase
if False:
  sio = SocketIO('socketio.mtgox.com/socket.io', test_callback)
  sio.connect()
  time.sleep(100)

EDIT: the websocket_client imported I got from here: http://pypi.python.org/pypi/websocket-client/0.4

EDIT2: I've been using this implementation for months and it works well for me. If you spot any problems with this implementation of fix errors or otherwise improve it, please let me know.

[runeks]

You da man molecular! Works like a charm. I'm not sure if Mt. Gox had problems or if the script you posted earlier in this thread missed something, but the script above works for me.

Thanks a lot!

BTW, had to change websocket_client to just websocket to get the script to run.

Also, I added a simple function to the SocketIO class that allows one to unsubscribe from a given channel:

Code:

  def unsubscribe(S, channel_id):
    S.ws.send('4::/mtgox:{"op":"unsubscribe","channel":"%s"}' % channel_id)

[runeks]

Anyone else getting errors like these with molecular's script?

Code:

SSLError: [Errno 1] _ssl.c:1350: error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long
SSLError: [Errno 1] _ssl.c:1350: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
SSLError: [Errno 1] _ssl.c:1350: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
SSLError: [Errno 1] _ssl.c:1350: error:14095044:SSL routines:SSL3_READ_N:internal error
SSLError: [Errno 1] _ssl.c:1350: error:140D2081:SSL routines:TLS1_ENC:block cipher pad is wrong
SSLError: [Errno 8] _ssl.c:1350: EOF occurred in violation of protocol
Segmentation fault
WebSocketException: Invalid frame type

Full log: http://pastebin.com/m4LF3tYs

Also, when these errors start occuring, I notice that bandwidth usage of my ticker script goes up. To like 20 KB/s IIRC. That seems like a lot for a trade every second or so. I figure it might be SSL certificates and the like going back and forth between mtgox.com and my computer. I find it hard to believe that this way of delivering trade information uses less resources than the original websocket API if everyone experiences these errors.

[molecular]

You da man molecular! Works like a charm. I'm not sure if Mt. Gox had problems or if the script you posted earlier in this thread missed something, but the script above works for me.

Thanks a lot!

BTW, had to change websocket_client to just websocket to get the script to run.

Also, I added a simple function to the SocketIO class that allows one to unsubscribe from a given channel:

Code:

  def unsubscribe(S, channel_id):
    S.ws.send('4::/mtgox:{"op":"unsubscribe","channel":"%s"}' % channel_id)

Thanks, I put that in just in case I need it some day

Anyone else getting errors like these with molecular's script?

Code:

SSLError: [Errno 1] _ssl.c:1350: error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long
SSLError: [Errno 1] _ssl.c:1350: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
SSLError: [Errno 1] _ssl.c:1350: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
SSLError: [Errno 1] _ssl.c:1350: error:14095044:SSL routines:SSL3_READ_N:internal error
SSLError: [Errno 1] _ssl.c:1350: error:140D2081:SSL routines:TLS1_ENC:block cipher pad is wrong
SSLError: [Errno 8] _ssl.c:1350: EOF occurred in violation of protocol
Segmentation fault
WebSocketException: Invalid frame type

Full log: http://pastebin.com/m4LF3tYs

Also, when these errors start occuring, I notice that bandwidth usage of my ticker script goes up. To like 20 KB/s IIRC. That seems like a lot for a trade every second or so. I figure it might be SSL certificates and the like going back and forth between mtgox.com and my computer. I find it hard to believe that this way of delivering trade information uses less resources than the original websocket API if everyone experiences these errors.

I have no idea, never seen this. But maybe you could try with the websocket_client.py I use (change that import back from "websocket" to "websocket_client" and the get file websocket_client.py from pastebin: http://pastebin.com/CTGj735u). I used to have a lot of troubles and tried at least 3 different websocket implemenations until I had a properly working one.

I _have_ seen "EOF occurred in violation of protocol", but I guess that's just a nasty disconnect...

[runeks]

Cool. Thanks. I'm trying it out now as we speak. Will report back how it works.

[runeks]

I'm still experiencing these errors with websocket_client unfortunately

Code:

08:06:59 4.81192 1.0 (4.81)
08:07:15 4.81174 0.55301041 (4.81)
08:07:25 4.81925 0.0616 (4.81)
08:07:26 4.81925 0.48723349 (4.81)
08:08:58 4.81174 16.57171689 (4.81)
trade['price_currency'] != "USD"
08:08:58 4.81146 16.0 (4.81)
08:10:26 4.8 10.0 (4.81)
08:11:01 4.7999 3.0 (4.81)
08:11:25 4.79992 0.54994211 (4.81)
trade['price_currency'] != "USD"
08:11:25 4.79992 5.0 (4.81)
08:11:30 4.78 0.27 (4.81)
08:11:30 4.78 2.0 (4.81)
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv
    raise ConnectionClosedException()
ConnectionClosedException

error sending keepalive socket.io, trying reconnect
08:13:00 4.7998 3.0 (4.81)
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv
    raise ConnectionClosedException()
ConnectionClosedException

error sending keepalive socket.io, trying reconnect
trade['price_currency'] != "USD"
08:14:27 4.7999 2.0 (4.81)
08:15:52 4.7998 3.0 (4.81)
08:16:06 4.8 30.0 (4.80)
08:17:13 4.7998 4.0 (4.80)
08:19:12 4.79906 0.03358625 (4.80)
trade['price_currency'] != "USD"
08:20:01 4.79001 1.48066947 (4.80)
08:20:37 4.79899 0.03190743 (4.80)
08:21:07 4.79894 1.22928907 (4.80)
08:21:38 4.79895 3.0 (4.80)
08:22:37 4.79895 2.80261836 (4.80)
Exception in thread Thread-5:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv
    raise ConnectionClosedException()
ConnectionClosedException

error sending keepalive socket.io, trying reconnect
error sending keepalive socket.io, trying reconnect
Exception in thread Thread-8:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 411, in _recv
    bytes = self.io_sock.recv(bufsize)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 161, in recv
    return self.ssl.read(bufsize)
SSLError: [Errno 1] _ssl.c:1350: error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long

Exception in thread Thread-7:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 411, in _recv
    bytes = self.io_sock.recv(bufsize)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 161, in recv
    return self.ssl.read(bufsize)
SSLError: [Errno 1] _ssl.c:1350: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

error sending keepalive socket.io, trying reconnect
08:24:08 4.79001 3.8303 (4.80)

First, a ConnectionClosedException exception is caught at the line "msg = S.ws.recv()" in thread_func. I assume this is because this call just hangs until either an error occurs or a message is received.
Then the keepalive_func thread will detect it when trying to send a message, and it will try to reconnect. This succeeds (seemingly), but lasts little time before the connection is lost again, and the process starts over (as you can see from the log). At some point, the errors go from the ConnectionClosedException exception to SSL exceptions like "error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long", but these SSL errors seem to vary a lot. I would guess that the SSL Python module expects the wrong data, but Mt. Gox' server sends the right data, though I'm not sure about that.

So, not sure what to do about this... seems something isn't right somewhere (duh!).

[molecular]

I'm still experiencing these errors with websocket_client unfortunately

Code:

08:06:59 4.81192 1.0 (4.81)
08:07:15 4.81174 0.55301041 (4.81)
08:07:25 4.81925 0.0616 (4.81)
08:07:26 4.81925 0.48723349 (4.81)
08:08:58 4.81174 16.57171689 (4.81)
trade['price_currency'] != "USD"
08:08:58 4.81146 16.0 (4.81)
08:10:26 4.8 10.0 (4.81)
08:11:01 4.7999 3.0 (4.81)
08:11:25 4.79992 0.54994211 (4.81)
trade['price_currency'] != "USD"
08:11:25 4.79992 5.0 (4.81)
08:11:30 4.78 0.27 (4.81)
08:11:30 4.78 2.0 (4.81)
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv
    raise ConnectionClosedException()
ConnectionClosedException

error sending keepalive socket.io, trying reconnect
08:13:00 4.7998 3.0 (4.81)
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv
    raise ConnectionClosedException()
ConnectionClosedException

error sending keepalive socket.io, trying reconnect
trade['price_currency'] != "USD"
08:14:27 4.7999 2.0 (4.81)
08:15:52 4.7998 3.0 (4.81)
08:16:06 4.8 30.0 (4.80)
08:17:13 4.7998 4.0 (4.80)
08:19:12 4.79906 0.03358625 (4.80)
trade['price_currency'] != "USD"
08:20:01 4.79001 1.48066947 (4.80)
08:20:37 4.79899 0.03190743 (4.80)
08:21:07 4.79894 1.22928907 (4.80)
08:21:38 4.79895 3.0 (4.80)
08:22:37 4.79895 2.80261836 (4.80)
Exception in thread Thread-5:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv
    raise ConnectionClosedException()
ConnectionClosedException

error sending keepalive socket.io, trying reconnect
error sending keepalive socket.io, trying reconnect
Exception in thread Thread-8:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 411, in _recv
    bytes = self.io_sock.recv(bufsize)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 161, in recv
    return self.ssl.read(bufsize)
SSLError: [Errno 1] _ssl.c:1350: error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long

Exception in thread Thread-7:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 411, in _recv
    bytes = self.io_sock.recv(bufsize)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 161, in recv
    return self.ssl.read(bufsize)
SSLError: [Errno 1] _ssl.c:1350: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

error sending keepalive socket.io, trying reconnect
08:24:08 4.79001 3.8303 (4.80)

First, a ConnectionClosedException exception is caught at the line "msg = S.ws.recv()" in thread_func. I assume this is because this call just hangs until either an error occurs or a message is received.
Then the keepalive_func thread will detect it when trying to send a message, and it will try to reconnect. This succeeds (seemingly), but lasts little time before the connection is lost again, and the process starts over (as you can see from the log). At some point, the errors go from the ConnectionClosedException exception to SSL exceptions like "error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long", but these SSL errors seem to vary a lot. I would guess that the SSL Python module expects the wrong data, but Mt. Gox' server sends the right data, though I'm not sure about that.

I'm experiencing this, too
I also see problems with threads. I don't know if my thread creation/burial works corretly. When the thread_func() exits that thread should be dead and recycled or whatever, right? Or do I need to still join it?

I tried to make reconnect more robust (at some point I get some "cannot create thread" exception and program exit), but so far failed.

I don't know wether the ConnectionClosed and "SLL" errors are client side fault.

Any insight welcome!

It also looks like I'm receiving corrupted/incorrect data on the socket sometimes:

Code:

08:54:02 4.79 10.0 (*10.0)
08:54:08 4.79 9.9243215 (*19.9)
08:54:11 4.78991 96.0 (4.79)
08:54:12 4.78941 27.0 (4.79)
08:54:12 3.09251 17.0 (4.50)
08:54:12 4.78902 17.0 (4.50)
08:55:16 3.09251 16.65379914 (4.22)
08:55:16 4.78902 16.65379914 (4.22)
08:55:16 4.78272 1.086e-05 (4.22)

That's two trades at $3.09251/BTC happening in between trades at $4.78902/BTC... That shouldn't be possible.

So, not sure what to do about this... seems something isn't right somewhere (duh!).

You might be looking at differenct currencies? 3.09 could be EUR or GBP or something?

I havent's seen weird stuff in the trade channel for a long time myself (filtering for USD). My depth stuff is still not syncing correctly and my depth table just gets cluttered with obviously old entries that are not removed after a while. I'm not sure wether this is due to websocket implementation or maybe I'm doing something wrong.

Again, any insight? Please post.

[bzzard]

I havent's seen weird stuff in the trade channel for a long time myself (filtering for USD). My depth stuff is still not syncing correctly and my depth table just gets cluttered with obviously old entries that are not removed after a while. I'm not sure wether this is due to websocket implementation or maybe I'm doing something wrong.

Again, any insight? Please post.

I've tested this topic in details and all I can say that lately (~2 weeks) depth data from socketio is useless. Don't trust it.

[runeks]

I'm experiencing this, too
I also see problems with threads. I don't know if my thread creation/burial works corretly. When the thread_func() exits that thread should be dead and recycled or whatever, right? Or do I need to still join it?

I tried to make reconnect more robust (at some point I get some "cannot create thread" exception and program exit), but so far failed.

I don't know wether the ConnectionClosed and "SLL" errors are client side fault.

Any insight welcome!

One thing that springs to mind is that upon error, we need to make sure that the existing threads exit, before we start new ones (through calling S.connect() in keepalive_func).
As it is now, keepalive_func will keep running after it has tried re-connecting, and as far as I can see, thread_func will keep running as well. So for each reconnect, we have one extra thread of both keepalive and thread_func running, sending and receiving data on the same socket:

Code:

  def keepalive_func(S):
    while S.run:
      try:
        S.ws.send('2::');
      except:
        if S.run:
          print 'error sending keepalive socket.io, trying reconnect'
          S.connect() #this thread will keep running after this. connect creates a new thread_func, which creates a new keepalive_func
        else:
          print 'exiting socket.io keepalive thread'
      time.sleep(S.heartbeat_interval)

Code:

  def thread_func(S):
    print 'SocketIO: websocket thread started'
    
    my_url = 'wss://' + S.url + "/1/websocket/" + S.id
    
    S.ws = create_connection(my_url)

    S.run = True
    S.ws.send('1::/mtgox')

    # start keepalive thread
    S.keepalive_thread = Thread(target = S.keepalive_func)
    S.keepalive_thread.setDaemon(True)
    S.keepalive_thread.start()
    
    msg = S.ws.recv()
    while msg is not None and S.run:
      if msg[:10] == "4::/mtgox:":
        S.callback(msg[10:])
      msg = S.ws.recv()
    S.ws.close()

As far as I can tell, this can be fixed by breaking out of the while loop in both threads in case of error, replacing the relevant parts like so:

Code:

   def keepalive_func(S):
      while S.run:
         try:
            S.ws.send('2::');
         except:
            if S.run:
               print 'error sending keepalive socket.io, closing existing connection...'
               S.ws.close()
               print '\ttrying reconnect...'
               S.connect()
               raise
               break
            else:
               print 'exiting socket.io keepalive thread'
         time.sleep(S.heartbeat_interval)

and in thread_func:

Code:

         try:
            msg = S.ws.recv()
         except:
            raise
            break
      S.ws.close()

I will see how this runs and report back.

It also looks like I'm receiving corrupted/incorrect data on the socket sometimes:

Code:

08:54:02 4.79 10.0 (*10.0)
08:54:08 4.79 9.9243215 (*19.9)
08:54:11 4.78991 96.0 (4.79)
08:54:12 4.78941 27.0 (4.79)
08:54:12 3.09251 17.0 (4.50)
08:54:12 4.78902 17.0 (4.50)
08:55:16 3.09251 16.65379914 (4.22)
08:55:16 4.78902 16.65379914 (4.22)
08:55:16 4.78272 1.086e-05 (4.22)

That's two trades at $3.09251/BTC happening in between trades at $4.78902/BTC... That shouldn't be possible.

So, not sure what to do about this... seems something isn't right somewhere (duh!).

You might be looking at differenct currencies? 3.09 could be EUR or GBP or something?

Indeed I was. I had just accidentally switched off the ignore-non-USD-trades option. Fixed now.

[runeks]

The above code doesn't work. Don't bother trying it out. break'ing isn't enough, we need to raise an exception to exit the thread. I've written something else that I have yet to test. I will report back how it went when I have tested it.

[molecular]

The above code doesn't work. Don't bother trying it out. break'ing isn't enough, we need to raise an exception to exit the thread. I've written something else that I have yet to test. I will report back how it went when I have tested it.

Thanks so much for your analysis and help so far, runeks.

I think you are correct in saying we need to "decommission" those 2 threads on disconnect.

I'm currently trying

Code:

raise SystemExit

EDIT: looks good so far, seeing some "error: [Errno 32] Broken pipe" and it reconnects

EDIT2: great, more "broken pipes", but still running (exeptionally long). So I figure with the help of runeks, I got the threading issues fixed.

EDIT3: here's socketio.py snapshot of the current version: http://pastebin.com/b5CmTy8a, git is here (might change in the future): https://github.com/molecular/traidor/blob/master/socketio.py

[Ente]

I tried several hours to create signed commands to send over the socket instead of the old api.
No luck at all. Lastly, I tried to "translate" the php example to python from https://en.bitcoin.it/wiki/MtGox/API/Streaming#Authenticated_commands which doesnt look too complicated..

Code:

def get_nonce():
    return int(time.time()*100000)

def encoder(raw):
  dec = json.dumps(raw)
  return dec

def sign_data(secret, data):
    return base64.b64encode(str(HMAC(secret, data, sha512).digest()))


call = "1/info.php"
params = "testparams"
item = "BTC"
currency = "BTC"
id = "1234567890"

Code:

query1 = {'call':call, 'params':params, 'item':item, 'currency':currency, 'id':id, 'nonce':get_nonce()}

{'nonce': 133259594451109, 'item': 'BTC', 'currency': 'BTC', 'params': 'testparams', 'call': '1/info.php', 'id': '1234567890'}

Code:

query2 = encoder(query1)

{"nonce": 133259594451109, "item": "BTC", "currency": "BTC", "params": "testparams", "call": "1/info.php", "id": "1234567890"}

Code:

sign = sign_data(secret, query2)

xjAc1rbh0T4xFVJT/eZo/nxI4vXxiVmaZxbyM6a5RQtHqld2RW1y9HYdE9z/TlM5e9++P+MMWEd9YFdf6Nenqg==

Code:

query3 = sign, query2

('xjAc1rbh0T4xFVJT/eZo/nxI4vXxiVmaZxbyM6a5RQtHqld2RW1y9HYdE9z/TlM5e9++P+MMWEd9YFdf6Nenqg==', '{"nonce": 133259594451109, "item": "BTC", "currency": "BTC", "params": "testparams", "call": "1/info.php", "id": "1234567890"}')

Code:

call = {'op':'call', 'call':base64.b64encode(str(query3)), 'id':id, 'context':'mtgox.com'}

{'call': 'KCd4akFjMXJiaDBUNHhGVkpUL2Vaby9ueEk0dlh4aVZtYVp4YnlNNmE1UlF0SHFsZDJSVzF5OUhZZEU 5ei9UbE01ZTkrK1ArTU1XRWQ5WUZkZjZOZW5xZz09JywgJ3sibm9uY2UiOiAxMzMyNTk1OTQ0NTExMD ksICJpdGVtIjogIkJUQyIsICJjdXJyZW5jeSI6ICJCVEMiLCAicGFyYW1zIjogInRlc3RwYXJhbXMiL CAiY2FsbCI6ICIxL2luZm8ucGhwIiwgImlkIjogIjEyMzQ1Njc4OTAifScp', 'id': '1234567890', 'context': 'mtgox.com', 'op': 'call'}

Finally I take this last message and paste it in my socket.send:

Code:

socket.send("4:::{'call': 'KCd4akFjMXJiaDBUNHhGVkpUL2Vaby9ueEk0dlh4aVZtYVp4YnlNNmE1UlF0SHFsZDJSVzF5OUhZZEU5ei9UbE01ZTkrK1ArTU1XRWQ5WUZkZjZOZW5xZz09JywgJ3sibm9uY2UiOiAxMzMyNTk1OTQ0NTExMDksICJpdGVtIjogIkJUQyIsICJjdXJyZW5jeSI6ICJCVEMiLCAicGFyYW1zIjogInRlc3RwYXJhbXMiLCAiY2FsbCI6ICIxL2luZm8ucGhwIiwgImlkIjogIjEyMzQ1Njc4OTAifScp', 'id': '1234567890', 'context': 'mtgox.com', 'op': 'call'}")

Non-descriptive error-reply:

{u'debug': {u'data': False, u'uuid': u'249bcb53-ccb0-43f1-bac6-e5ae0748e2e1', u'op': u'client'}, u'message': u'Unknown command', u'success': False, u'op': u'remark'}

If someone has enough mercy to have a closer look I would be really happy!
Since this probably takes more than just a few moments, dont forget to quote your bitcoin-address too ;-)

Ente

/who now goes doing some woodwork to clear his head from this