Bitcoin Forum
December 05, 2016, 06:42:30 PM *
News: Latest stable version of Bitcoin Core: 0.13.1  [Torrent].
 
   Home   Help Search Donate Login Register  
Pages: « 1 2 3 4 5 6 7 8 9 10 [11] 12 13 14 15 16 17 18 19 20 21 »  All
  Print  
Author Topic: [BETA] MTGox websocket API, testers wanted  (Read 72214 times)
molecular
Donator
Legendary
*
Offline Offline

Activity: 2128



View Profile
March 24, 2012, 02:17:24 PM
 #201

I tried several hours to create signed commands to send over the socket instead of the old api.

I used this: https://bitcointalk.org/index.php?topic=49789.msg592388#msg592388, it works.


PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
1480963350
Hero Member
*
Offline Offline

Posts: 1480963350

View Profile Personal Message (Offline)

Ignore
1480963350
Reply with quote  #2

1480963350
Report to moderator
1480963350
Hero Member
*
Offline Offline

Posts: 1480963350

View Profile Personal Message (Offline)

Ignore
1480963350
Reply with quote  #2

1480963350
Report to moderator
Each block is stacked on top of the previous one. Adding another block to the top makes all lower blocks more difficult to remove: there is more "weight" above each block. A transaction in a block 6 blocks deep (6 confirmations) will be very difficult to remove.
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1480963350
Hero Member
*
Offline Offline

Posts: 1480963350

View Profile Personal Message (Offline)

Ignore
1480963350
Reply with quote  #2

1480963350
Report to moderator
Ente
Legendary
*
Offline Offline

Activity: 1834



View Profile
March 24, 2012, 02:35:44 PM
 #202

I tried several hours to create signed commands to send over the socket instead of the old api.

I used this: https://bitcointalk.org/index.php?topic=49789.msg592388#msg592388, it works.


Yes, I used that too, for my old http api. I try to update to socket-only, which supposedly is almost the same.. Almost, but for me not close enough *sigh*

Thanks for replying!

Ente
molecular
Donator
Legendary
*
Offline Offline

Activity: 2128



View Profile
March 24, 2012, 03:03:17 PM
 #203

I tried several hours to create signed commands to send over the socket instead of the old api.

I used this: https://bitcointalk.org/index.php?topic=49789.msg592388#msg592388, it works.


Yes, I used that too, for my old http api. I try to update to socket-only, which supposedly is almost the same.. Almost, but for me not close enough *sigh*

Thanks for replying!

Ente

"socket-only"? I know of the "new" and "old" (1, 2) http apis. And I know of socket.io interface for depth/trades (no auth necessary). Care to explain?

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
Clark
Hero Member
*****
Offline Offline

Activity: 540


So much code.


View Profile WWW
March 24, 2012, 03:14:01 PM
 #204

According to the wiki API reference, you should be able to do authentication stuff over the WebSocket interface.

PGP KEY | 1Bitcoin3Tg2KWyAq3wzivdqwYqGwKYaGd
Ente
Legendary
*
Offline Offline

Activity: 1834



View Profile
March 24, 2012, 03:28:05 PM
 #205

According to the wiki API reference, you should be able to do authentication stuff over the WebSocket interface.

Yep:

Quote
Authenticated commands
These commands require an API key and secret pair to sign requests. Any of the HTTP API version 1 methods can be called.
https://en.bitcoin.it/wiki/MtGox/API/Streaming#Authenticated_commands

..with more details and a php example too

Ente
runeks
Legendary
*
Offline Offline

Activity: 924



View Profile WWW
March 25, 2012, 08:38:58 PM
 #206

The above code doesn't work. Don't bother trying it out. break'ing isn't enough, we need to raise an exception to exit the thread. I've written something else that I have yet to test. I will report back how it went when I have tested it.

Thanks so much for your analysis and help so far, runeks.

I think you are correct in saying we need to "decommission" those 2 threads on disconnect.

I'm currently trying

Code:
raise SystemExit

EDIT: looks good so far, seeing some "error: [Errno 32] Broken pipe" and it reconnects

EDIT2: great, more "broken pipes", but still running (exeptionally long). So I figure with the help of runeks, I got the threading issues fixed.

EDIT3: here's socketio.py snapshot of the current version: http://pastebin.com/b5CmTy8a, git is here (might change in the future): https://github.com/molecular/traidor/blob/master/socketio.py



Hi!

Totally forgot about this. Thanks for your token. I've modified your code a bit to see if I can get it completely robust. I'm basically doing the controlling from thread_func instead, because exceptions are almost always detected here first (because "msg = S.ws.recv()" is always hanging there just waiting for a message to come). This means we won't experience a 0-15 second delay before reconnecting (by having the keepalive thread do it). It adds some more complexity though. I still experience the occasional SSL error "EOF occurred in violation of protocol" (besides the ConnectionClosedException) nonetheless. But it seems to be running very robustly so far. No running amok like the previous script. The occasional error comes, but never multiple errors in a row. I will release the code when it has proved to be stable. I'm not experiencing any "Broken pipe" messages though. Only the two aforementioned errors.
molecular
Donator
Legendary
*
Offline Offline

Activity: 2128



View Profile
March 26, 2012, 07:38:30 AM
 #207

The above code doesn't work. Don't bother trying it out. break'ing isn't enough, we need to raise an exception to exit the thread. I've written something else that I have yet to test. I will report back how it went when I have tested it.

Thanks so much for your analysis and help so far, runeks.

I think you are correct in saying we need to "decommission" those 2 threads on disconnect.

I'm currently trying

Code:
raise SystemExit

EDIT: looks good so far, seeing some "error: [Errno 32] Broken pipe" and it reconnects

EDIT2: great, more "broken pipes", but still running (exeptionally long). So I figure with the help of runeks, I got the threading issues fixed.

EDIT3: here's socketio.py snapshot of the current version: http://pastebin.com/b5CmTy8a, git is here (might change in the future): https://github.com/molecular/traidor/blob/master/socketio.py



Hi!

Totally forgot about this. Thanks for your token. I've modified your code a bit to see if I can get it completely robust. I'm basically doing the controlling from thread_func instead, because exceptions are almost always detected here first (because "msg = S.ws.recv()" is always hanging there just waiting for a message to come). This means we won't experience a 0-15 second delay before reconnecting (by having the keepalive thread do it). It adds some more complexity though. I still experience the occasional SSL error "EOF occurred in violation of protocol" (besides the ConnectionClosedException) nonetheless. But it seems to be running very robustly so far. No running amok like the previous script. The occasional error comes, but never multiple errors in a row. I will release the code when it has proved to be stable. I'm not experiencing any "Broken pipe" messages though. Only the two aforementioned errors.

I remember while coding it it didn't feel right to detect for reconnect (only) in the keepalive thread.

Thanks for fixing that, please pastebin your version if you consider it robust.

My one is running stably for days now. I might just not care about the 15 seconds.

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
runeks
Legendary
*
Offline Offline

Activity: 924



View Profile WWW
March 27, 2012, 03:18:58 PM
 #208

Here's my version. Just yesterday I found out that I need to handle exceptions in "response = urllib2.urlopen(req)" line in connect(). I couldn't understand why the script had stopped, but found out it was because this line had thrown a ConnectionRefused exception. So it's continually improving as I find out the various ways it can fail. Still haven't gotten rid of the SSL errors though.

https://github.com/runeksvendsen/traidor/blob/master/socketio.py
molecular
Donator
Legendary
*
Offline Offline

Activity: 2128



View Profile
March 27, 2012, 10:12:20 PM
 #209

Here's my version. Just yesterday I found out that I need to handle exceptions in "response = urllib2.urlopen(req)" line in connect(). I couldn't understand why the script had stopped, but found out it was because this line had thrown a ConnectionRefused exception. So it's continually improving as I find out the various ways it can fail. Still haven't gotten rid of the SSL errors though.

https://github.com/runeksvendsen/traidor/blob/master/socketio.py

thanks, so cool.

it bails in line 94 for me, though, with:

Code:
Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 530, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 483, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/nick/bitcoin/traidor/socketio.py", line 94, in thread_func
    S.callback(S, msg[10:])
TypeError: onMessage() takes exactly 2 arguments (3 given)

this is because you added a parameter to the callback function and I still pass a callback function that takes one parameter less.

note that the "test_callback" function is in global scope, not in the class.

suggestion: If you need the instance of the socket.io in testing mode you can just use the global variable "sio" instead of passing S as follows (changes marked in bold):

Quote
def test_callback(msg):
   import ast
   print 'msg: ', msg
   #convert received message string into dictionary
   msg_dict = ast.literal_eval(msg)
   #print msg_dict
   if msg_dict['op'] == 'subscribe':
      if msg_dict['channel'] == 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f' or msg_dict['channel'] == '24e67e0d-1cad-4cc0-9e7a-f8523ef460fe':
         sio.unsubscribe(msg_dict['channel'])

that way everything remains compatible and I can use your code

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
sirk390
Full Member
***
Offline Offline

Activity: 153


View Profile
April 01, 2012, 03:55:33 PM
 #210

Hi,

Does anyone know if it is possible to receive market depth updates for other currencies (EUR) in the socket.io/websocket API?
I found this list of channels: https://en.bitcoin.it/wiki/Talk:MtGox/API but couldn't subscribe using the API.
The API only takes a "type" argument (Possible values: "trades, ticker, depth") e.g. {"op":"mtgox.subscribe","type":"trades"}
but there is no documented value for "depth.EUR".
Clark
Hero Member
*****
Offline Offline

Activity: 540


So much code.


View Profile WWW
April 01, 2012, 04:33:18 PM
 #211

Does anyone know if it is possible to receive market depth updates for other currencies (EUR) in the socket.io/websocket API?
I found this list of channels: https://en.bitcoin.it/wiki/Talk:MtGox/API but couldn't subscribe using the API.
The API only takes a "type" argument (Possible values: "trades, ticker, depth") e.g. {"op":"mtgox.subscribe","type":"trades"}
but there is no documented value for "depth.EUR".

You can subscribe with ?Channel=EUR added to the subscribe URL. I have a working EUR order book, chart, and time & sales on my website. Simply choose EUR from the drop down menu at the top: bitcoin.clarkmoody.com

PGP KEY | 1Bitcoin3Tg2KWyAq3wzivdqwYqGwKYaGd
sirk390
Full Member
***
Offline Offline

Activity: 153


View Profile
April 01, 2012, 08:41:54 PM
 #212

Does anyone know if it is possible to receive market depth updates for other currencies (EUR) in the socket.io/websocket API?
I found this list of channels: https://en.bitcoin.it/wiki/Talk:MtGox/API but couldn't subscribe using the API.
The API only takes a "type" argument (Possible values: "trades, ticker, depth") e.g. {"op":"mtgox.subscribe","type":"trades"}
but there is no documented value for "depth.EUR".

You can subscribe with ?Channel=EUR added to the subscribe URL. I have a working EUR order book, chart, and time & sales on my website. Simply choose EUR from the drop down menu at the top: bitcoin.clarkmoody.com
Thanks but what subscribe URL do you mean ?
I'm connection with python using something like this https://github.com/osmosis79/gox_socketio_py .
It sends a request to "https://socketio.mtgox.com/socket.io/1" and then opens a websocket to "wss://socketio.mtgox.com/socket.io/1/1/websocket/{id}"

Clark
Hero Member
*****
Offline Offline

Activity: 540


So much code.


View Profile WWW
April 01, 2012, 08:53:18 PM
 #213

Thanks but what subscribe URL do you mean ?

Sorry, it should have been ?Currency=

Like this:

https://socketio.mtgox.com/mtgox?Currency=EUR

PGP KEY | 1Bitcoin3Tg2KWyAq3wzivdqwYqGwKYaGd
sirk390
Full Member
***
Offline Offline

Activity: 153


View Profile
April 01, 2012, 11:01:43 PM
 #214

Thanks but what subscribe URL do you mean ?

Sorry, it should have been ?Currency=

Like this:

https://socketio.mtgox.com/mtgox?Currency=EUR

Thanks you! It works now although i'm not using exactly the same URL.
I'm using https://socketio.mtgox.com/socket.io/1/?Currency=EUR .

 
molecular
Donator
Legendary
*
Offline Offline

Activity: 2128



View Profile
April 01, 2012, 11:09:38 PM
 #215

I'm connection with python using something like this https://github.com/osmosis79/gox_socketio_py .

a newer version of which (with improved thread handling, much more stable concerning reconnects) can be found here https://github.com/molecular/traidor/blob/master/socketio.py

osmosis79 seems to have copied it about a month ago (no objections)

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
runeks
Legendary
*
Offline Offline

Activity: 924



View Profile WWW
April 04, 2012, 12:55:16 AM
 #216

Here's my version. Just yesterday I found out that I need to handle exceptions in "response = urllib2.urlopen(req)" line in connect(). I couldn't understand why the script had stopped, but found out it was because this line had thrown a ConnectionRefused exception. So it's continually improving as I find out the various ways it can fail. Still haven't gotten rid of the SSL errors though.

https://github.com/runeksvendsen/traidor/blob/master/socketio.py

thanks, so cool.

it bails in line 94 for me, though, with:

Code:
Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 530, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 483, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/nick/bitcoin/traidor/socketio.py", line 94, in thread_func
    S.callback(S, msg[10:])
TypeError: onMessage() takes exactly 2 arguments (3 given)

this is because you added a parameter to the callback function and I still pass a callback function that takes one parameter less.

note that the "test_callback" function is in global scope, not in the class.

suggestion: If you need the instance of the socket.io in testing mode you can just use the global variable "sio" instead of passing S as follows (changes marked in bold):

Quote
def test_callback(msg):
   import ast
   print 'msg: ', msg
   #convert received message string into dictionary
   msg_dict = ast.literal_eval(msg)
   #print msg_dict
   if msg_dict['op'] == 'subscribe':
      if msg_dict['channel'] == 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f' or msg_dict['channel'] == '24e67e0d-1cad-4cc0-9e7a-f8523ef460fe':
         sio.unsubscribe(msg_dict['channel'])

that way everything remains compatible and I can use your code

Right you are. I think I added it to be able to unsubscribe when a message arrives. But as you say, a global variable works better for this. Fixed in the newest revision.
knite
Jr. Member
*
Offline Offline

Activity: 33


View Profile
April 07, 2012, 02:20:53 PM
 #217

runeks and molecular:

I've reviewed and begun using a combination of your codebases. I have a few questions and comments.

- Why the different approaches to your keepalive threads? Molecular's version is more concise. Is it an issue with the ws.recv() delay on disconnect?
- Is there a reason for using the urllib2 module? The requests module is much nicer.
- Why aren't you using the logging module instead of print statements?
- Consider making your SocketIO classes more generic and creating an MTGox sub-class which inherits from it, passing in event handlers (on_connect, on_msg, etc). This will make things easier if and when other exchanges add streaming APIs.
- I strongly recommend reducing your client heartbeat interval. I've set mine to 85% of the value given by the server, rounded down.
knite
Jr. Member
*
Offline Offline

Activity: 33


View Profile
April 20, 2012, 07:07:36 PM
 #218

Threads are still melting my brain. Can I get the best of both worlds by doing something like this?

Code:
def start_thread(function):
t = Thread(target=function)
t.setDaemon(True)
t.start()
return t

def connect(S):
#setup
start_thread(thread_func)

def reconnect(S):
S.ws.close()
S.connect()

def thread_func(S):
S.ws = create_connection(S.ws_url)
S.run = True
start_thread(keepalive_func)

msg = S.ws.recv()
while msg is not None and S.run:
S.process(msg)
try:
msg = S.ws.recv()
except:
if S.run:
S.reconnect()
break
else:
if run:
S.reconnect()

def keepalive_func():
while S.run:
try:
S.ws.send('2::')
except:
S.ws.close()
break
sleep(S.heartbeat)

The keepalive thread does nothing except send heartbeats and close the connection in the event of an exception. Thread_func reconnects if it hits an exception or exits the while loop, if run is True. Thoughts?
molecular
Donator
Legendary
*
Offline Offline

Activity: 2128



View Profile
May 14, 2012, 06:04:04 PM
 #219

Hi knite, nice to see the code being used / looked at and thanks for your comments. I'll comment on some of them...

- Is there a reason for using the urllib2 module? The requests module is much nicer.

No, not from my side. This (coding traidor) was the first time I did any requesting in python. Tried urllib (didn't work for some reason), the tried urllib2 (worked) so I just stuck with that.

- Why aren't you using the logging module instead of print statements?

Forgive my ignorance of the logging module, will take a look next time I need to dump some info.

On a more general note: there are never reasons for not doing something, there can only be reasons for doing something Wink. Why do you not believe in the flying spaghetti monster?

- Consider making your SocketIO classes more generic and creating an MTGox sub-class which inherits from it, passing in event handlers (on_connect, on_msg, etc). This will make things easier if and when other exchanges add streaming APIs.

Consider this considered. However, when thinking about integrating other exchanges, there is a lot more work on other parts of my code to be done. I've been wanting to do this for a while now but decided if I would do it, I'd reimplement in another language that better suits my experience. Let me point you to Goncalo Pinheira (goncalopp, don't know if on forum, he mailed me, will give you his email upon request in pm), who has been asking me about this and if he could use my code (been asking for lgpl) and who seems to want to do exactly that (exchange abstraction)

- I strongly recommend reducing your client heartbeat interval. I've set mine to 85% of the value given by the server, rounded down.

done


PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
Seal
Donator
Hero Member
*
Offline Offline

Activity: 833


View Profile WWW
May 23, 2012, 06:55:21 AM
 #220

- Consider making your SocketIO classes more generic and creating an MTGox sub-class which inherits from it, passing in event handlers (on_connect, on_msg, etc). This will make things easier if and when other exchanges add streaming APIs.

Consider this considered. However, when thinking about integrating other exchanges, there is a lot more work on other parts of my code to be done. I've been wanting to do this for a while now but decided if I would do it, I'd reimplement in another language that better suits my experience.


What languages do you recommend for best integration with SocketIO and streaming data?

I've been looking into scripting pages with PHP but it seems a little tough to integrate as its a language which generates pages on call.

Also I'd like to write a server-side script to run at home which will accept a socketio feed and place trades automatically. I currently do this again with the json calls API however because its written in PHP, it requires a cron call every x minutes. I'm looking for a better solution.

Pages: « 1 2 3 4 5 6 7 8 9 10 [11] 12 13 14 15 16 17 18 19 20 21 »  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!