Ente
Legendary
Offline
Activity: 2126
Merit: 1001
|
|
February 14, 2012, 07:39:05 PM |
|
OH! Thank you a lot for your help, zelyony!
I was already wondering about the heartbeat. I now answer 1:: and 2:: accordingly, now they keep flowing in constantly. Well, the socket *must* be working, since the websites using it still work. There would be more outcry in the forum if it was dead altogether..
I am glad I'm not the only one here!
Ente
|
|
|
|
Ente
Legendary
Offline
Activity: 2126
Merit: 1001
|
|
February 14, 2012, 10:29:19 PM Last edit: February 15, 2012, 10:17:32 AM by Ente |
|
*bangs head on desk* I cant get it to work. 1 BTC bounty for the first person who helps me receive regular data again over the websocket! Here is my (totally stripped) python code: #!/usr/bin/env python # -*- coding: utf-8 -*-
from websocket import create_connection import urllib import re import time
try: import json except ImportError: import simplejson as json
wsurl = "wss://socketio.mtgox.com/socket.io/" iourl = "https://socketio.mtgox.com/socket.io/"
############
print "Connecting to socket.io.." url = iourl + "1" f = urllib.urlopen(url) output = f.read() f.close() ausgabe = re.search('[0-9]+', output, 0) sessionid = ausgabe.group()
print "New websocket url:" wssurl = wsurl + "1" + "/websocket/" + sessionid print wssurl
############
print "Connecting to websocket..\n"
socket = create_connection(wssurl)
while 1: echo = socket.recv() if "1::" in echo: print "Heartbeat 1" socket.send("1::") elif "2::" in echo: print "Heartbeat 2" socket.send("2::") else: print "Message: ",echo time.sleep(2)
The websocket client I use here is from http://pypi.python.org/pypi/websocket-client/0.4. I have a rudimentary heartbeat added here. The whole script dies eventually, from having stripped all error-handling. It worked some weeks ago. If you like to see the whole script, let me know. Bedtime here, will check back in, like, 9 hours.. *frustrated and tired* Ente
|
|
|
|
|
zelyony
Newbie
Offline
Activity: 23
Merit: 0
|
|
February 15, 2012, 10:08:59 AM |
|
right now datas arrived (my format after string -> json -> object -> string): JSON: Packet={ Depth={ price=4,67000 USD, vol=0,05300000, dir=bid, when=2012:02:15_10:03:56.735844 } } JSON: Packet={ Ticker={ bid=$4.68100, ask=$4.70075, lo=$4.21211, hi=$4.97000, last=$4.70075, vol=256,165.42727403??BTC } }
I dont know Python, but: - on connected (or after recv "1::" - that's mean u connected now) u must send strings like: 4:::{"op":"mtgox.subscribe","type":"trades"} - on "2::" u must response "2::" (not "1::" as in ur sample)
|
|
|
|
Ente
Legendary
Offline
Activity: 2126
Merit: 1001
|
|
February 15, 2012, 10:19:43 AM |
|
Thank you for checking, zelyony! You are right, thats a typo in the snippet. I sent '1::' and '2::' accordingly in my full script, typo corrected.
I sent '4:::{"op":"mtgox.subscribe","type":"trades"}' over the websocket, no change. No response whatsoever, no trade-messages coming in.
Ente
|
|
|
|
zelyony
Newbie
Offline
Activity: 23
Merit: 0
|
|
February 15, 2012, 10:39:41 AM Last edit: February 16, 2012, 06:53:18 AM by zelyony |
|
and right now I dont recv any useful datas UPD: it's working right now (13:44 UTC) UPD2: it's working if wait connected sometimes 20 minutes and sometimes 0 minutes and sometimes "never". very strange and no good. can someone's debug session stops sending messages to other users?.. who knows implementation of socket.io server? To MtGox: maybe it's sabotage.. Slow Read DoS attack.. bad guys overflow send pool with slow connections and.. voila! will be better for all clients drop connections that cannot read string, i.e. send to client not finished for.. hmm.. 0.5sec! +ban by IP 10min (epoll, select or other asynchronous mechanism). I took digits from air, I dont know % loading for servers machines and number of connections. p.s. ping ("2::") works always
|
|
|
|
runeks
Legendary
Offline
Activity: 980
Merit: 1008
|
|
February 22, 2012, 03:39:23 PM |
|
I have a hard time following all the tips and tricks in this thread on getting the socket.io interface to work with Python.
Has anyone gotten it to work properly? As far as I can tell there are no socket.io client implementations for Python right, right?
|
|
|
|
Ente
Legendary
Offline
Activity: 2126
Merit: 1001
|
|
February 22, 2012, 04:13:57 PM |
|
UPD2: it's working if wait connected sometimes 20 minutes and sometimes 0 minutes and sometimes "never". very strange and no good. can someone's debug session stops sending messages to other users?.. who knows implementation of socket.io server?
Same here. Sometimes it takes ages to receive any data, sometimes less than a minute. Totally random, and totally annoying. The 1:: and 2:: pings work straight from the beginning. As soon as I get a response to my channel-connect, data from this channel comes in too. Ente
|
|
|
|
molecular
Donator
Legendary
Offline
Activity: 2772
Merit: 1019
|
|
February 27, 2012, 08:36:15 AM Last edit: February 27, 2012, 08:50:01 AM by molecular |
|
I have a hard time following all the tips and tricks in this thread on getting the socket.io interface to work with Python.
Has anyone gotten it to work properly? As far as I can tell there are no socket.io client implementations for Python right, right?
maybe this helps, my socketio.py: from threading import * import urllib2, urllib import simplejson as json import ssl, socket import time from websocket_client import create_connection import traceback
class SocketIO: def __init__(S, url, callback): S.url = url S.callback = callback def connect(S): try: data = urllib.urlencode({}) req = urllib2.Request('https://' + S.url + "/1", data) print 'https://' + S.url + "/1" response = urllib2.urlopen(req) r = response.read().split(':') S.heartbeat_interval = int(r[1]) #print 'heartbeat: ', S.heartbeat_interval if 'websocket' in r[3].split(','): print "good: transport 'websocket' supported by socket.io server ", S.url S.id = r[0] print "id: ", S.id else: print "error: transport 'websocket' NOT supported by socket.io server ", S.url S.thread = Thread(target = S.thread_func) S.thread.setDaemon(True) S.thread.start() except: traceback.print_exc()
def stop(S): S.run = False S.thread.join(timeout=1) S.keepalive_thread.join(timeout=1)
def thread_func(S): print 'SocketIO: websocket thread started' my_url = 'wss://' + S.url + "/1/websocket/" + S.id S.ws = create_connection(my_url) #S.ws = WebSocket(my_url, version=0) S.run = True S.ws.send('1::/mtgox')
# start keepalive thread S.keepalive_thread = Thread(target = S.keepalive_func) S.keepalive_thread.setDaemon(True) S.keepalive_thread.start() msg = S.ws.recv() while msg is not None and S.run: #print 'SocketIO msg: ', msg if msg[:10] == "4::/mtgox:": S.callback(msg[10:]) #elif msg[:3] == "2::": # True #else: # print "SocketIO: dont know how to handle msg: ", msg msg = S.ws.recv() S.ws.close() def keepalive_func(S): while S.run: try: S.ws.send('2::'); except: if S.run: print 'error sending keepalive socket.io, trying reconnect' S.connect() else: print 'exiting socket.io keepalive thread' time.sleep(S.heartbeat_interval) def test_callback(msg): print 'msg: ', msg
# testcase if False: sio = SocketIO('socketio.mtgox.com/socket.io', test_callback) sio.connect() time.sleep(100)
EDIT: the websocket_client imported I got from here: http://pypi.python.org/pypi/websocket-client/0.4EDIT2: I've been using this implementation for months and it works well for me. If you spot any problems with this implementation of fix errors or otherwise improve it, please let me know.
|
PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0 3F39 FC49 2362 F9B7 0769
|
|
|
runeks
Legendary
Offline
Activity: 980
Merit: 1008
|
|
February 27, 2012, 05:27:52 PM |
|
You da man molecular! Works like a charm. I'm not sure if Mt. Gox had problems or if the script you posted earlier in this thread missed something, but the script above works for me. Thanks a lot! BTW, had to change websocket_client to just websocket to get the script to run. Also, I added a simple function to the SocketIO class that allows one to unsubscribe from a given channel: def unsubscribe(S, channel_id): S.ws.send('4::/mtgox:{"op":"unsubscribe","channel":"%s"}' % channel_id)
|
|
|
|
runeks
Legendary
Offline
Activity: 980
Merit: 1008
|
|
March 04, 2012, 05:26:24 PM |
|
Anyone else getting errors like these with molecular's script? SSLError: [Errno 1] _ssl.c:1350: error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long SSLError: [Errno 1] _ssl.c:1350: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number SSLError: [Errno 1] _ssl.c:1350: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac SSLError: [Errno 1] _ssl.c:1350: error:14095044:SSL routines:SSL3_READ_N:internal error SSLError: [Errno 1] _ssl.c:1350: error:140D2081:SSL routines:TLS1_ENC:block cipher pad is wrong SSLError: [Errno 8] _ssl.c:1350: EOF occurred in violation of protocol Segmentation fault WebSocketException: Invalid frame type
Full log: http://pastebin.com/m4LF3tYsAlso, when these errors start occuring, I notice that bandwidth usage of my ticker script goes up. To like 20 KB/s IIRC. That seems like a lot for a trade every second or so. I figure it might be SSL certificates and the like going back and forth between mtgox.com and my computer. I find it hard to believe that this way of delivering trade information uses less resources than the original websocket API if everyone experiences these errors.
|
|
|
|
molecular
Donator
Legendary
Offline
Activity: 2772
Merit: 1019
|
|
March 06, 2012, 11:41:24 PM |
|
You da man molecular! Works like a charm. I'm not sure if Mt. Gox had problems or if the script you posted earlier in this thread missed something, but the script above works for me. Thanks a lot! BTW, had to change websocket_client to just websocket to get the script to run. Also, I added a simple function to the SocketIO class that allows one to unsubscribe from a given channel: def unsubscribe(S, channel_id): S.ws.send('4::/mtgox:{"op":"unsubscribe","channel":"%s"}' % channel_id) Thanks, I put that in just in case I need it some day Anyone else getting errors like these with molecular's script? SSLError: [Errno 1] _ssl.c:1350: error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long SSLError: [Errno 1] _ssl.c:1350: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number SSLError: [Errno 1] _ssl.c:1350: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac SSLError: [Errno 1] _ssl.c:1350: error:14095044:SSL routines:SSL3_READ_N:internal error SSLError: [Errno 1] _ssl.c:1350: error:140D2081:SSL routines:TLS1_ENC:block cipher pad is wrong SSLError: [Errno 8] _ssl.c:1350: EOF occurred in violation of protocol Segmentation fault WebSocketException: Invalid frame type
Full log: http://pastebin.com/m4LF3tYsAlso, when these errors start occuring, I notice that bandwidth usage of my ticker script goes up. To like 20 KB/s IIRC. That seems like a lot for a trade every second or so. I figure it might be SSL certificates and the like going back and forth between mtgox.com and my computer. I find it hard to believe that this way of delivering trade information uses less resources than the original websocket API if everyone experiences these errors. I have no idea, never seen this. But maybe you could try with the websocket_client.py I use (change that import back from "websocket" to "websocket_client" and the get file websocket_client.py from pastebin: http://pastebin.com/CTGj735u). I used to have a lot of troubles and tried at least 3 different websocket implemenations until I had a properly working one. I _have_ seen "EOF occurred in violation of protocol", but I guess that's just a nasty disconnect...
|
PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0 3F39 FC49 2362 F9B7 0769
|
|
|
runeks
Legendary
Offline
Activity: 980
Merit: 1008
|
|
March 07, 2012, 04:33:44 PM |
|
Cool. Thanks. I'm trying it out now as we speak. Will report back how it works.
|
|
|
|
runeks
Legendary
Offline
Activity: 980
Merit: 1008
|
|
March 20, 2012, 07:37:38 AM Last edit: March 20, 2012, 08:07:47 AM by runeks |
|
I'm still experiencing these errors with websocket_client unfortunately 08:06:59 4.81192 1.0 (4.81) 08:07:15 4.81174 0.55301041 (4.81) 08:07:25 4.81925 0.0616 (4.81) 08:07:26 4.81925 0.48723349 (4.81) 08:08:58 4.81174 16.57171689 (4.81) trade['price_currency'] != "USD" 08:08:58 4.81146 16.0 (4.81) 08:10:26 4.8 10.0 (4.81) 08:11:01 4.7999 3.0 (4.81) 08:11:25 4.79992 0.54994211 (4.81) trade['price_currency'] != "USD" 08:11:25 4.79992 5.0 (4.81) 08:11:30 4.78 0.27 (4.81) 08:11:30 4.78 2.0 (4.81) Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 505, in run self.__target(*self.__args, **self.__kwargs) File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func msg = S.ws.recv() File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv b = self._recv(1) File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv raise ConnectionClosedException() ConnectionClosedException
error sending keepalive socket.io, trying reconnect 08:13:00 4.7998 3.0 (4.81) Exception in thread Thread-3: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 505, in run self.__target(*self.__args, **self.__kwargs) File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func msg = S.ws.recv() File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv b = self._recv(1) File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv raise ConnectionClosedException() ConnectionClosedException
error sending keepalive socket.io, trying reconnect trade['price_currency'] != "USD" 08:14:27 4.7999 2.0 (4.81) 08:15:52 4.7998 3.0 (4.81) 08:16:06 4.8 30.0 (4.80) 08:17:13 4.7998 4.0 (4.80) 08:19:12 4.79906 0.03358625 (4.80) trade['price_currency'] != "USD" 08:20:01 4.79001 1.48066947 (4.80) 08:20:37 4.79899 0.03190743 (4.80) 08:21:07 4.79894 1.22928907 (4.80) 08:21:38 4.79895 3.0 (4.80) 08:22:37 4.79895 2.80261836 (4.80) Exception in thread Thread-5: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 505, in run self.__target(*self.__args, **self.__kwargs) File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func msg = S.ws.recv() File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv b = self._recv(1) File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv raise ConnectionClosedException() ConnectionClosedException
error sending keepalive socket.io, trying reconnect error sending keepalive socket.io, trying reconnect Exception in thread Thread-8: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 505, in run self.__target(*self.__args, **self.__kwargs) File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func msg = S.ws.recv() File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv b = self._recv(1) File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 411, in _recv bytes = self.io_sock.recv(bufsize) File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 161, in recv return self.ssl.read(bufsize) SSLError: [Errno 1] _ssl.c:1350: error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long
Exception in thread Thread-7: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 505, in run self.__target(*self.__args, **self.__kwargs) File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func msg = S.ws.recv() File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv b = self._recv(1) File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 411, in _recv bytes = self.io_sock.recv(bufsize) File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 161, in recv return self.ssl.read(bufsize) SSLError: [Errno 1] _ssl.c:1350: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
error sending keepalive socket.io, trying reconnect 08:24:08 4.79001 3.8303 (4.80)
First, a ConnectionClosedException exception is caught at the line "msg = S.ws.recv()" in thread_func. I assume this is because this call just hangs until either an error occurs or a message is received. Then the keepalive_func thread will detect it when trying to send a message, and it will try to reconnect. This succeeds (seemingly), but lasts little time before the connection is lost again, and the process starts over (as you can see from the log). At some point, the errors go from the ConnectionClosedException exception to SSL exceptions like "error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long", but these SSL errors seem to vary a lot. I would guess that the SSL Python module expects the wrong data, but Mt. Gox' server sends the right data, though I'm not sure about that. So, not sure what to do about this... seems something isn't right somewhere (duh!).
|
|
|
|
molecular
Donator
Legendary
Offline
Activity: 2772
Merit: 1019
|
|
March 20, 2012, 08:09:32 AM |
|
I'm still experiencing these errors with websocket_client unfortunately 08:06:59 4.81192 1.0 (4.81) 08:07:15 4.81174 0.55301041 (4.81) 08:07:25 4.81925 0.0616 (4.81) 08:07:26 4.81925 0.48723349 (4.81) 08:08:58 4.81174 16.57171689 (4.81) trade['price_currency'] != "USD" 08:08:58 4.81146 16.0 (4.81) 08:10:26 4.8 10.0 (4.81) 08:11:01 4.7999 3.0 (4.81) 08:11:25 4.79992 0.54994211 (4.81) trade['price_currency'] != "USD" 08:11:25 4.79992 5.0 (4.81) 08:11:30 4.78 0.27 (4.81) 08:11:30 4.78 2.0 (4.81) Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 505, in run self.__target(*self.__args, **self.__kwargs) File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func msg = S.ws.recv() File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv b = self._recv(1) File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv raise ConnectionClosedException() ConnectionClosedException
error sending keepalive socket.io, trying reconnect 08:13:00 4.7998 3.0 (4.81) Exception in thread Thread-3: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 505, in run self.__target(*self.__args, **self.__kwargs) File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func msg = S.ws.recv() File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv b = self._recv(1) File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv raise ConnectionClosedException() ConnectionClosedException
error sending keepalive socket.io, trying reconnect trade['price_currency'] != "USD" 08:14:27 4.7999 2.0 (4.81) 08:15:52 4.7998 3.0 (4.81) 08:16:06 4.8 30.0 (4.80) 08:17:13 4.7998 4.0 (4.80) 08:19:12 4.79906 0.03358625 (4.80) trade['price_currency'] != "USD" 08:20:01 4.79001 1.48066947 (4.80) 08:20:37 4.79899 0.03190743 (4.80) 08:21:07 4.79894 1.22928907 (4.80) 08:21:38 4.79895 3.0 (4.80) 08:22:37 4.79895 2.80261836 (4.80) Exception in thread Thread-5: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 505, in run self.__target(*self.__args, **self.__kwargs) File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func msg = S.ws.recv() File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv b = self._recv(1) File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv raise ConnectionClosedException() ConnectionClosedException
error sending keepalive socket.io, trying reconnect error sending keepalive socket.io, trying reconnect Exception in thread Thread-8: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 505, in run self.__target(*self.__args, **self.__kwargs) File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func msg = S.ws.recv() File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv b = self._recv(1) File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 411, in _recv bytes = self.io_sock.recv(bufsize) File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 161, in recv return self.ssl.read(bufsize) SSLError: [Errno 1] _ssl.c:1350: error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long
Exception in thread Thread-7: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 505, in run self.__target(*self.__args, **self.__kwargs) File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func msg = S.ws.recv() File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv b = self._recv(1) File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 411, in _recv bytes = self.io_sock.recv(bufsize) File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 161, in recv return self.ssl.read(bufsize) SSLError: [Errno 1] _ssl.c:1350: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
error sending keepalive socket.io, trying reconnect 08:24:08 4.79001 3.8303 (4.80)
First, a ConnectionClosedException exception is caught at the line "msg = S.ws.recv()" in thread_func. I assume this is because this call just hangs until either an error occurs or a message is received. Then the keepalive_func thread will detect it when trying to send a message, and it will try to reconnect. This succeeds (seemingly), but lasts little time before the connection is lost again, and the process starts over (as you can see from the log). At some point, the errors go from the ConnectionClosedException exception to SSL exceptions like "error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long", but these SSL errors seem to vary a lot. I would guess that the SSL Python module expects the wrong data, but Mt. Gox' server sends the right data, though I'm not sure about that. I'm experiencing this, too I also see problems with threads. I don't know if my thread creation/burial works corretly. When the thread_func() exits that thread should be dead and recycled or whatever, right? Or do I need to still join it? I tried to make reconnect more robust (at some point I get some "cannot create thread" exception and program exit), but so far failed. I don't know wether the ConnectionClosed and "SLL" errors are client side fault. Any insight welcome!It also looks like I'm receiving corrupted/incorrect data on the socket sometimes: 08:54:02 4.79 10.0 (*10.0) 08:54:08 4.79 9.9243215 (*19.9) 08:54:11 4.78991 96.0 (4.79) 08:54:12 4.78941 27.0 (4.79) 08:54:12 3.09251 17.0 (4.50) 08:54:12 4.78902 17.0 (4.50) 08:55:16 3.09251 16.65379914 (4.22) 08:55:16 4.78902 16.65379914 (4.22) 08:55:16 4.78272 1.086e-05 (4.22)
That's two trades at $3.09251/BTC happening in between trades at $4.78902/BTC... That shouldn't be possible. So, not sure what to do about this... seems something isn't right somewhere (duh!). You might be looking at differenct currencies? 3.09 could be EUR or GBP or something? I havent's seen weird stuff in the trade channel for a long time myself (filtering for USD). My depth stuff is still not syncing correctly and my depth table just gets cluttered with obviously old entries that are not removed after a while. I'm not sure wether this is due to websocket implementation or maybe I'm doing something wrong. Again, any insight? Please post.
|
PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0 3F39 FC49 2362 F9B7 0769
|
|
|
bzzard
|
|
March 20, 2012, 08:20:26 AM |
|
I havent's seen weird stuff in the trade channel for a long time myself (filtering for USD). My depth stuff is still not syncing correctly and my depth table just gets cluttered with obviously old entries that are not removed after a while. I'm not sure wether this is due to websocket implementation or maybe I'm doing something wrong.
Again, any insight? Please post. I've tested this topic in details and all I can say that lately (~2 weeks) depth data from socketio is useless. Don't trust it.
|
|
|
|
runeks
Legendary
Offline
Activity: 980
Merit: 1008
|
|
March 20, 2012, 09:21:48 AM |
|
I'm experiencing this, too I also see problems with threads. I don't know if my thread creation/burial works corretly. When the thread_func() exits that thread should be dead and recycled or whatever, right? Or do I need to still join it? I tried to make reconnect more robust (at some point I get some "cannot create thread" exception and program exit), but so far failed. I don't know wether the ConnectionClosed and "SLL" errors are client side fault. Any insight welcome!One thing that springs to mind is that upon error, we need to make sure that the existing threads exit, before we start new ones (through calling S.connect() in keepalive_func). As it is now, keepalive_func will keep running after it has tried re-connecting, and as far as I can see, thread_func will keep running as well. So for each reconnect, we have one extra thread of both keepalive and thread_func running, sending and receiving data on the same socket: def keepalive_func(S): while S.run: try: S.ws.send('2::'); except: if S.run: print 'error sending keepalive socket.io, trying reconnect' S.connect() #this thread will keep running after this. connect creates a new thread_func, which creates a new keepalive_func else: print 'exiting socket.io keepalive thread' time.sleep(S.heartbeat_interval)
def thread_func(S): print 'SocketIO: websocket thread started' my_url = 'wss://' + S.url + "/1/websocket/" + S.id S.ws = create_connection(my_url)
S.run = True S.ws.send('1::/mtgox')
# start keepalive thread S.keepalive_thread = Thread(target = S.keepalive_func) S.keepalive_thread.setDaemon(True) S.keepalive_thread.start() msg = S.ws.recv() while msg is not None and S.run: if msg[:10] == "4::/mtgox:": S.callback(msg[10:]) msg = S.ws.recv() S.ws.close() As far as I can tell, this can be fixed by breaking out of the while loop in both threads in case of error, replacing the relevant parts like so: def keepalive_func(S): while S.run: try: S.ws.send('2::'); except: if S.run: print 'error sending keepalive socket.io, closing existing connection...' S.ws.close() print '\ttrying reconnect...' S.connect() raise break else: print 'exiting socket.io keepalive thread' time.sleep(S.heartbeat_interval) and in thread_func: try: msg = S.ws.recv() except: raise break S.ws.close() I will see how this runs and report back. It also looks like I'm receiving corrupted/incorrect data on the socket sometimes: 08:54:02 4.79 10.0 (*10.0) 08:54:08 4.79 9.9243215 (*19.9) 08:54:11 4.78991 96.0 (4.79) 08:54:12 4.78941 27.0 (4.79) 08:54:12 3.09251 17.0 (4.50) 08:54:12 4.78902 17.0 (4.50) 08:55:16 3.09251 16.65379914 (4.22) 08:55:16 4.78902 16.65379914 (4.22) 08:55:16 4.78272 1.086e-05 (4.22)
That's two trades at $3.09251/BTC happening in between trades at $4.78902/BTC... That shouldn't be possible. So, not sure what to do about this... seems something isn't right somewhere (duh!). You might be looking at differenct currencies? 3.09 could be EUR or GBP or something? Indeed I was. I had just accidentally switched off the ignore-non-USD-trades option. Fixed now.
|
|
|
|
runeks
Legendary
Offline
Activity: 980
Merit: 1008
|
|
March 20, 2012, 11:09:30 AM |
|
The above code doesn't work. Don't bother trying it out. break'ing isn't enough, we need to raise an exception to exit the thread. I've written something else that I have yet to test. I will report back how it went when I have tested it.
|
|
|
|
molecular
Donator
Legendary
Offline
Activity: 2772
Merit: 1019
|
|
March 20, 2012, 02:37:59 PM Last edit: March 20, 2012, 07:27:00 PM by molecular |
|
The above code doesn't work. Don't bother trying it out. break'ing isn't enough, we need to raise an exception to exit the thread. I've written something else that I have yet to test. I will report back how it went when I have tested it.
Thanks so much for your analysis and help so far, runeks. I think you are correct in saying we need to "decommission" those 2 threads on disconnect. I'm currently trying EDIT: looks good so far, seeing some "error: [Errno 32] Broken pipe" and it reconnects EDIT2: great, more "broken pipes", but still running (exeptionally long). So I figure with the help of runeks, I got the threading issues fixed. EDIT3: here's socketio.py snapshot of the current version: http://pastebin.com/b5CmTy8a, git is here (might change in the future): https://github.com/molecular/traidor/blob/master/socketio.py
|
PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0 3F39 FC49 2362 F9B7 0769
|
|
|
Ente
Legendary
Offline
Activity: 2126
Merit: 1001
|
|
March 24, 2012, 01:47:40 PM |
|
I tried several hours to create signed commands to send over the socket instead of the old api. No luck at all. Lastly, I tried to "translate" the php example to python from https://en.bitcoin.it/wiki/MtGox/API/Streaming#Authenticated_commands which doesnt look too complicated.. def get_nonce(): return int(time.time()*100000)
def encoder(raw): dec = json.dumps(raw) return dec
def sign_data(secret, data): return base64.b64encode(str(HMAC(secret, data, sha512).digest()))
call = "1/info.php" params = "testparams" item = "BTC" currency = "BTC" id = "1234567890"
query1 = {'call':call, 'params':params, 'item':item, 'currency':currency, 'id':id, 'nonce':get_nonce()}
{'nonce': 133259594451109, 'item': 'BTC', 'currency': 'BTC', 'params': 'testparams', 'call': '1/info.php', 'id': '1234567890'}
{"nonce": 133259594451109, "item": "BTC", "currency": "BTC", "params": "testparams", "call": "1/info.php", "id": "1234567890"} sign = sign_data(secret, query2)
xjAc1rbh0T4xFVJT/eZo/nxI4vXxiVmaZxbyM6a5RQtHqld2RW1y9HYdE9z/TlM5e9++P+MMWEd9YFdf6Nenqg==
('xjAc1rbh0T4xFVJT/eZo/nxI4vXxiVmaZxbyM6a5RQtHqld2RW1y9HYdE9z/TlM5e9++P+MMWEd9YFdf6Nenqg==', '{"nonce": 133259594451109, "item": "BTC", "currency": "BTC", "params": "testparams", "call": "1/info.php", "id": "1234567890"}')
call = {'op':'call', 'call':base64.b64encode(str(query3)), 'id':id, 'context':'mtgox.com'}
{'call': 'KCd4akFjMXJiaDBUNHhGVkpUL2Vaby9ueEk0dlh4aVZtYVp4YnlNNmE1UlF0SHFsZDJSVzF5OUhZZEU 5ei9UbE01ZTkrK1ArTU1XRWQ5WUZkZjZOZW5xZz09JywgJ3sibm9uY2UiOiAxMzMyNTk1OTQ0NTExMD ksICJpdGVtIjogIkJUQyIsICJjdXJyZW5jeSI6ICJCVEMiLCAicGFyYW1zIjogInRlc3RwYXJhbXMiL CAiY2FsbCI6ICIxL2luZm8ucGhwIiwgImlkIjogIjEyMzQ1Njc4OTAifScp', 'id': '1234567890', 'context': 'mtgox.com', 'op': 'call'}
Finally I take this last message and paste it in my socket.send: socket.send("4:::{'call': 'KCd4akFjMXJiaDBUNHhGVkpUL2Vaby9ueEk0dlh4aVZtYVp4YnlNNmE1UlF0SHFsZDJSVzF5OUhZZEU5ei9UbE01ZTkrK1ArTU1XRWQ5WUZkZjZOZW5xZz09JywgJ3sibm9uY2UiOiAxMzMyNTk1OTQ0NTExMDksICJpdGVtIjogIkJUQyIsICJjdXJyZW5jeSI6ICJCVEMiLCAicGFyYW1zIjogInRlc3RwYXJhbXMiLCAiY2FsbCI6ICIxL2luZm8ucGhwIiwgImlkIjogIjEyMzQ1Njc4OTAifScp', 'id': '1234567890', 'context': 'mtgox.com', 'op': 'call'}")
Non-descriptive error-reply: {u'debug': {u'data': False, u'uuid': u'249bcb53-ccb0-43f1-bac6-e5ae0748e2e1', u'op': u'client'}, u'message': u'Unknown command', u'success': False, u'op': u'remark'} If someone has enough mercy to have a closer look I would be really happy! Since this probably takes more than just a few moments, dont forget to quote your bitcoin-address too ;-) Ente /who now goes doing some woodwork to clear his head from this
|
|
|
|
|