Multi Exchange Streaming API for Market Data

[marketstem]

I initially set this up because I needed the data for a personal project and decided that I might as well try to make it official/stable.  It is only streaming ticker data right now to reduce costs, but market depth can be turned on with a switch and public trades could be added in the future.

Is there any interest for a service like this?

http://docs.marketstem.apiary.io/

*Note: Apiary docs are designed for HTTP Rest API's, but I like their layout so I forced this streaming web socket api into it.

[mailmansDOG]

This seems interesting. Just out of curiosity though, what was the project this was originally planned to be used in?

[marketstem]

The dream is still to try to automate arbitrage trading across multiple exchanges and markets, just like everyone else I'm sure ...trying to go beyond the usual buy btc/usd here and sell usd/btc there.  Main blocker right now is that every so often you need to re-balance funds across exchanges, but most exchanges don't have have an api to do transfers/withdraws.  Writing robots to login for you gets kludgy, and sending alerts to go do it manually is too burdensome.

For that project I just had this data collection running on a single host.  But I realized that if I wanted to open this data up I needed to distribute it across several nodes to make sure there is never any down time.  As a side benefit I am able to get past some of the rate limits imposed by exchanges because I am coming from different IP's.

[marketstem]

Another nice feature I implemented internally with all of the ticker data is more accurate price indexes which gave me the ability to convert from any currency to another for simulation.  The price indexes are calculated as a weighted average with the weight being the trade volume from each exchange. E.g. if the price for BTC/USD on Bitstamp is 500 with a volume of 1000 and the price is 480 with a volume of 500 on Kraken the weighted global price is ( ( 500 * 1000 ) + ( 480 * 500 ) ) / 1500 = 493.33.  except with values from all exchanges in the actual implementation of course.  I don't think I've seen another Cyrpto currency price index that takes that much data into account, but it can easily be done with this streaming ticker data.

[dddbtc]

I initially set this up because I needed the data for a personal project and decided that I might as well try to make it official/stable.  It is only streaming ticker data right now to reduce costs, but market depth can be turned on with a switch and public trades could be added in the future.

Is there any interest for a service like this?

http://docs.marketstem.apiary.io/

*Note: Apiary docs are designed for HTTP Rest API's, but I like their layout so I forced this streaming web socket api into it.

I'm super happy to hear about this.  I hope this expands into something really cool (and that you have a donation address)

[joinecoin]

If you open API, I believe that itwould be a very good news for poscoins.

[marketstem]

If you open API, I believe that itwould be a very good news for poscoins.

Not sure what you mean.  It is already open, but yeah, I can't afford the server hosting and pubnub costs for long or at scale without funding.

[dddbtc]

How much funding do you need to open this service accessible on a per-exchange basis for everyone?

How much funding do you need to open this service accessible on a per-exchange basis for just me? 

[marketstem]

The responses on my Reddit thread should answer this question: http://www.reddit.com/r/BitcoinMarkets/comments/25dy7l/streaming_market_data_for_bitstamp_coinbase_btce/

It would be about $100 per month to support just you and ~$300-$600 for everyone.  PubNub gets pricey with this kind of message volume....I just started a conversation with them to negotiate a volume discount.  This is my first month not using a free demo account with them so I am having some difficulty nailing down a solid number for their costs.

The only additional cost for turning on data per exchange is the additional message costs. So I really need to figure out what their volume discount will be.

[evilimprint]

why do you not just write your on broadcast system.. it really would not be to hard and you could cut that cost out of your running fees.

[marketstem]

Because I like the massive amounts of documentation and jdk client libraries that are readily available which will lower the barrier to entry and hopefully increase the adoption rate.  Also, I feel like the cost for offloading the operational maintenance is more than worth it.

[dddbtc]

The responses on my Reddit thread should answer this question: http://www.reddit.com/r/BitcoinMarkets/comments/25dy7l/streaming_market_data_for_bitstamp_coinbase_btce/

It would be about $100 per month to support just you and ~$300-$600 for everyone.  PubNub gets pricey with this kind of message volume....I just started a conversation with them to negotiate a volume discount.  This is my first month not using a free demo account with them so I am having some difficulty nailing down a solid number for their costs.

The only additional cost for turning on data per exchange is the additional message costs. So I really need to figure out what their volume discount will be.

Will you walk me through how to host a websocket for bitfinex on a machine hosted in a datacenter.  I am familiar with ws4py and java programming.

[marketstem]

why do you not just write your on broadcast system.. it really would not be to hard and you could cut that cost out of your running fees.

You were definitely right on this one... I've migrated off of PubNub to my own deployment of Slanger/Pusher.  Costs are now down to ~$60 a month to be able to support everyone for all channels of data.  Plus we still have the benefit of having several clients available for connecting provided by Pusher.

https://github.com/stevegraham/slanger
http://pusher.com/

[botticelli]

Hi marketstem,

Is there any interest in such a service? Yes, absolutely.
I was searching for something like that a couple of months ago! But unfortunately, there was nothing, so I built my own ;-)

I use it to simplify the logic inside my bot. The bot consumes a single stream of merged and preprocessed market data, (order books) whereas behind the service all exchanges all crawled in parallel with proper handling for timeouts and reconnects.
And in addition, there is no interference between the rate limitations of the bot and the crawling service.
So everything is working more nicely, elegant and robust.

I built upon Server-Sent Events (in contrast to WebSockets). Mainly because of the nice layer on top of HTTP, the built-in reconnect scheme, easy support in nginx+tornado and for this amount of data: the transparent compression that comes with HTTP.

On top of that, there is a second service that consumes all market data and stores them on disk - preprocessed with a mixture of lossy and lossless compression. This is quite challenging if you want to save as much disk space (I know, disk space is cheap. But less disk space is cheaper ;-) ) And normally compression will speed up the reading of historical data, which will speed up the backtesting. My uncompressed data consists of orderbooks up to 1.000 units in depth (e.g. btc) for each second on each platform and when I started naively, my little digitalocean droplet was full after the first day

Just my story.
I really like your service, the open approach and your progress! Keep going.

[marketstem]

Hey botticelli,

Thanks for sharing, these type of comments definitely help keep motivated.  The overall level of interest I've received has been much lower than expected, but I realize I am probably way too biased towards how valuable I think this data is.

Are you considering switching over to this service?  Or is there anything that I could add/change that would make the decision to move over easier?

Fortunately I'm not trying to store any of this data, I can imagine the endless amount of operational pain.  Kudos that you got it worked out.

And shoutout to Digital Ocean.  I just randomly saw an advertisement for a $10 promo the day I set out to distribute everything.  I was using AWS before that point so it is easy to appreciate how amazing DO is.

[sarah-mills]

I was looking at other firms that provide a financial data API for websites and found www.barchartondemand.com. They are a little more expensive then maybe other options, but it's a nice, clean interface. I think their rates are around $800/month, but I'm not sure.

It seems to me that the expensive part is actually getting the stock market data feed from the exchanges and market data providers. How do you manage to get the stock market data feed that you turn around and use in the API's? It looks to me that getting the data is the biggest expense. Also, you might want to look at barchartondemand to get a feel for what sort of APIs you should create.