[BETA] MTGox websocket API, testers wanted

[molecular]

I tried several hours to create signed commands to send over the socket instead of the old api.

I used this: https://bitcointalk.org/index.php?topic=49789.msg592388#msg592388, it works.

[Ente]

I tried several hours to create signed commands to send over the socket instead of the old api.

I used this: https://bitcointalk.org/index.php?topic=49789.msg592388#msg592388, it works.

Yes, I used that too, for my old http api. I try to update to socket-only, which supposedly is almost the same.. Almost, but for me not close enough *sigh*

Thanks for replying!

Ente

[molecular]

I tried several hours to create signed commands to send over the socket instead of the old api.

I used this: https://bitcointalk.org/index.php?topic=49789.msg592388#msg592388, it works.

Yes, I used that too, for my old http api. I try to update to socket-only, which supposedly is almost the same.. Almost, but for me not close enough *sigh*

Thanks for replying!

Ente

"socket-only"? I know of the "new" and "old" (1, 2) http apis. And I know of socket.io interface for depth/trades (no auth necessary). Care to explain?

[Clark]

According to the wiki API reference, you should be able to do authentication stuff over the WebSocket interface.

[Ente]

According to the wiki API reference, you should be able to do authentication stuff over the WebSocket interface.

Yep:

Authenticated commands
These commands require an API key and secret pair to sign requests. Any of the HTTP API version 1 methods can be called.

https://en.bitcoin.it/wiki/MtGox/API/Streaming#Authenticated_commands

..with more details and a php example too

Ente

[runeks]

The above code doesn't work. Don't bother trying it out. break'ing isn't enough, we need to raise an exception to exit the thread. I've written something else that I have yet to test. I will report back how it went when I have tested it.

Thanks so much for your analysis and help so far, runeks.

I think you are correct in saying we need to "decommission" those 2 threads on disconnect.

I'm currently trying

Code:

raise SystemExit

EDIT: looks good so far, seeing some "error: [Errno 32] Broken pipe" and it reconnects

EDIT2: great, more "broken pipes", but still running (exeptionally long). So I figure with the help of runeks, I got the threading issues fixed.

EDIT3: here's socketio.py snapshot of the current version: http://pastebin.com/b5CmTy8a, git is here (might change in the future): https://github.com/molecular/traidor/blob/master/socketio.py

Hi!

Totally forgot about this. Thanks for your token. I've modified your code a bit to see if I can get it completely robust. I'm basically doing the controlling from thread_func instead, because exceptions are almost always detected here first (because "msg = S.ws.recv()" is always hanging there just waiting for a message to come). This means we won't experience a 0-15 second delay before reconnecting (by having the keepalive thread do it). It adds some more complexity though. I still experience the occasional SSL error "EOF occurred in violation of protocol" (besides the ConnectionClosedException) nonetheless. But it seems to be running very robustly so far. No running amok like the previous script. The occasional error comes, but never multiple errors in a row. I will release the code when it has proved to be stable. I'm not experiencing any "Broken pipe" messages though. Only the two aforementioned errors.

[molecular]

The above code doesn't work. Don't bother trying it out. break'ing isn't enough, we need to raise an exception to exit the thread. I've written something else that I have yet to test. I will report back how it went when I have tested it.

Thanks so much for your analysis and help so far, runeks.

I think you are correct in saying we need to "decommission" those 2 threads on disconnect.

I'm currently trying

Code:

raise SystemExit

EDIT: looks good so far, seeing some "error: [Errno 32] Broken pipe" and it reconnects

EDIT2: great, more "broken pipes", but still running (exeptionally long). So I figure with the help of runeks, I got the threading issues fixed.

EDIT3: here's socketio.py snapshot of the current version: http://pastebin.com/b5CmTy8a, git is here (might change in the future): https://github.com/molecular/traidor/blob/master/socketio.py

Hi!

Totally forgot about this. Thanks for your token. I've modified your code a bit to see if I can get it completely robust. I'm basically doing the controlling from thread_func instead, because exceptions are almost always detected here first (because "msg = S.ws.recv()" is always hanging there just waiting for a message to come). This means we won't experience a 0-15 second delay before reconnecting (by having the keepalive thread do it). It adds some more complexity though. I still experience the occasional SSL error "EOF occurred in violation of protocol" (besides the ConnectionClosedException) nonetheless. But it seems to be running very robustly so far. No running amok like the previous script. The occasional error comes, but never multiple errors in a row. I will release the code when it has proved to be stable. I'm not experiencing any "Broken pipe" messages though. Only the two aforementioned errors.

I remember while coding it it didn't feel right to detect for reconnect (only) in the keepalive thread.

Thanks for fixing that, please pastebin your version if you consider it robust.

My one is running stably for days now. I might just not care about the 15 seconds.

[runeks]

Here's my version. Just yesterday I found out that I need to handle exceptions in "response = urllib2.urlopen(req)" line in connect(). I couldn't understand why the script had stopped, but found out it was because this line had thrown a ConnectionRefused exception. So it's continually improving as I find out the various ways it can fail. Still haven't gotten rid of the SSL errors though.

https://github.com/runeksvendsen/traidor/blob/master/socketio.py

[molecular]

Here's my version. Just yesterday I found out that I need to handle exceptions in "response = urllib2.urlopen(req)" line in connect(). I couldn't understand why the script had stopped, but found out it was because this line had thrown a ConnectionRefused exception. So it's continually improving as I find out the various ways it can fail. Still haven't gotten rid of the SSL errors though.

https://github.com/runeksvendsen/traidor/blob/master/socketio.py

thanks, so cool.

it bails in line 94 for me, though, with:

Code:

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 530, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 483, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/nick/bitcoin/traidor/socketio.py", line 94, in thread_func
    S.callback(S, msg[10:])
TypeError: onMessage() takes exactly 2 arguments (3 given)

this is because you added a parameter to the callback function and I still pass a callback function that takes one parameter less.

note that the "test_callback" function is in global scope, not in the class.

suggestion: If you need the instance of the socket.io in testing mode you can just use the global variable "sio" instead of passing S as follows (changes marked in bold):

def test_callback(msg):
   import ast
   print 'msg: ', msg
   #convert received message string into dictionary
   msg_dict = ast.literal_eval(msg)
   #print msg_dict
   if msg_dict['op'] == 'subscribe':
      if msg_dict['channel'] == 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f' or msg_dict['channel'] == '24e67e0d-1cad-4cc0-9e7a-f8523ef460fe':
         sio.unsubscribe(msg_dict['channel'])

that way everything remains compatible and I can use your code

[sirk390]

Hi,

Does anyone know if it is possible to receive market depth updates for other currencies (EUR) in the socket.io/websocket API?
I found this list of channels: https://en.bitcoin.it/wiki/Talk:MtGox/API but couldn't subscribe using the API.
The API only takes a "type" argument (Possible values: "trades, ticker, depth") e.g. {"op":"mtgox.subscribe","type":"trades"}
but there is no documented value for "depth.EUR".

[Clark]

Does anyone know if it is possible to receive market depth updates for other currencies (EUR) in the socket.io/websocket API?
I found this list of channels: https://en.bitcoin.it/wiki/Talk:MtGox/API but couldn't subscribe using the API.
The API only takes a "type" argument (Possible values: "trades, ticker, depth") e.g. {"op":"mtgox.subscribe","type":"trades"}
but there is no documented value for "depth.EUR".

You can subscribe with ?Channel=EUR added to the subscribe URL. I have a working EUR order book, chart, and time & sales on my website. Simply choose EUR from the drop down menu at the top: bitcoin.clarkmoody.com

[sirk390]

Does anyone know if it is possible to receive market depth updates for other currencies (EUR) in the socket.io/websocket API?
I found this list of channels: https://en.bitcoin.it/wiki/Talk:MtGox/API but couldn't subscribe using the API.
The API only takes a "type" argument (Possible values: "trades, ticker, depth") e.g. {"op":"mtgox.subscribe","type":"trades"}
but there is no documented value for "depth.EUR".

You can subscribe with ?Channel=EUR added to the subscribe URL. I have a working EUR order book, chart, and time & sales on my website. Simply choose EUR from the drop down menu at the top: bitcoin.clarkmoody.com

Thanks but what subscribe URL do you mean ?
I'm connection with python using something like this https://github.com/osmosis79/gox_socketio_py .
It sends a request to "https://socketio.mtgox.com/socket.io/1" and then opens a websocket to "wss://socketio.mtgox.com/socket.io/1/1/websocket/{id}"

[Clark]

Thanks but what subscribe URL do you mean ?

Sorry, it should have been ?Currency=

Like this:

https://socketio.mtgox.com/mtgox?Currency=EUR

[sirk390]

Thanks but what subscribe URL do you mean ?

Sorry, it should have been ?Currency=

Like this:

https://socketio.mtgox.com/mtgox?Currency=EUR

Thanks you! It works now although i'm not using exactly the same URL.
I'm using https://socketio.mtgox.com/socket.io/1/?Currency=EUR .

 

[molecular]

I'm connection with python using something like this https://github.com/osmosis79/gox_socketio_py .

a newer version of which (with improved thread handling, much more stable concerning reconnects) can be found here https://github.com/molecular/traidor/blob/master/socketio.py

osmosis79 seems to have copied it about a month ago (no objections)

[runeks]

Here's my version. Just yesterday I found out that I need to handle exceptions in "response = urllib2.urlopen(req)" line in connect(). I couldn't understand why the script had stopped, but found out it was because this line had thrown a ConnectionRefused exception. So it's continually improving as I find out the various ways it can fail. Still haven't gotten rid of the SSL errors though.

https://github.com/runeksvendsen/traidor/blob/master/socketio.py

thanks, so cool.

it bails in line 94 for me, though, with:

Code:

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 530, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 483, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/nick/bitcoin/traidor/socketio.py", line 94, in thread_func
    S.callback(S, msg[10:])
TypeError: onMessage() takes exactly 2 arguments (3 given)

this is because you added a parameter to the callback function and I still pass a callback function that takes one parameter less.

note that the "test_callback" function is in global scope, not in the class.

suggestion: If you need the instance of the socket.io in testing mode you can just use the global variable "sio" instead of passing S as follows (changes marked in bold):

def test_callback(msg):
   import ast
   print 'msg: ', msg
   #convert received message string into dictionary
   msg_dict = ast.literal_eval(msg)
   #print msg_dict
   if msg_dict['op'] == 'subscribe':
      if msg_dict['channel'] == 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f' or msg_dict['channel'] == '24e67e0d-1cad-4cc0-9e7a-f8523ef460fe':
         sio.unsubscribe(msg_dict['channel'])

that way everything remains compatible and I can use your code

Right you are. I think I added it to be able to unsubscribe when a message arrives. But as you say, a global variable works better for this. Fixed in the newest revision.

[knite]

runeks and molecular:

I've reviewed and begun using a combination of your codebases. I have a few questions and comments.

- Why the different approaches to your keepalive threads? Molecular's version is more concise. Is it an issue with the ws.recv() delay on disconnect?
- Is there a reason for using the urllib2 module? The requests module is much nicer.
- Why aren't you using the logging module instead of print statements?
- Consider making your SocketIO classes more generic and creating an MTGox sub-class which inherits from it, passing in event handlers (on_connect, on_msg, etc). This will make things easier if and when other exchanges add streaming APIs.
- I strongly recommend reducing your client heartbeat interval. I've set mine to 85% of the value given by the server, rounded down.

[knite]

Threads are still melting my brain. Can I get the best of both worlds by doing something like this?

Code:

def start_thread(function):
	t = Thread(target=function)
	t.setDaemon(True)
	t.start()
	return t

def connect(S):
	#setup
	start_thread(thread_func)

def reconnect(S):
	S.ws.close()
	S.connect()

def thread_func(S):
	S.ws = create_connection(S.ws_url)
	S.run = True
	start_thread(keepalive_func)

	msg = S.ws.recv()
	while msg is not None and S.run:
		S.process(msg)
		try:
			msg = S.ws.recv()
		except:
			if S.run:
				S.reconnect()
			break
	else:
		if run:
			S.reconnect()

def keepalive_func():
	while S.run:
		try:
			S.ws.send('2::')
		except:
			S.ws.close()
			break
		sleep(S.heartbeat)

The keepalive thread does nothing except send heartbeats and close the connection in the event of an exception. Thread_func reconnects if it hits an exception or exits the while loop, if run is True. Thoughts?

[molecular]

Hi knite, nice to see the code being used / looked at and thanks for your comments. I'll comment on some of them...

- Is there a reason for using the urllib2 module? The requests module is much nicer.

No, not from my side. This (coding traidor) was the first time I did any requesting in python. Tried urllib (didn't work for some reason), the tried urllib2 (worked) so I just stuck with that.

- Why aren't you using the logging module instead of print statements?

Forgive my ignorance of the logging module, will take a look next time I need to dump some info.

On a more general note: there are never reasons for not doing something, there can only be reasons for doing something . Why do you not believe in the flying spaghetti monster?

- Consider making your SocketIO classes more generic and creating an MTGox sub-class which inherits from it, passing in event handlers (on_connect, on_msg, etc). This will make things easier if and when other exchanges add streaming APIs.

Consider this considered. However, when thinking about integrating other exchanges, there is a lot more work on other parts of my code to be done. I've been wanting to do this for a while now but decided if I would do it, I'd reimplement in another language that better suits my experience. Let me point you to Goncalo Pinheira (goncalopp, don't know if on forum, he mailed me, will give you his email upon request in pm), who has been asking me about this and if he could use my code (been asking for lgpl) and who seems to want to do exactly that (exchange abstraction)

- I strongly recommend reducing your client heartbeat interval. I've set mine to 85% of the value given by the server, rounded down.

done

[Seal]

- Consider making your SocketIO classes more generic and creating an MTGox sub-class which inherits from it, passing in event handlers (on_connect, on_msg, etc). This will make things easier if and when other exchanges add streaming APIs.

Consider this considered. However, when thinking about integrating other exchanges, there is a lot more work on other parts of my code to be done. I've been wanting to do this for a while now but decided if I would do it, I'd reimplement in another language that better suits my experience.

What languages do you recommend for best integration with SocketIO and streaming data?

I've been looking into scripting pages with PHP but it seems a little tough to integrate as its a language which generates pages on call.

Also I'd like to write a server-side script to run at home which will accept a socketio feed and place trades automatically. I currently do this again with the json calls API however because its written in PHP, it requires a cron call every x minutes. I'm looking for a better solution.