Try http://api.bitcoincharts.com/v1/csv/ for bulk second-by-second data (no raw ticks, but currently this is the most raw data I've been able to find). They also have a more detailed API here which may be useful. |
|
|
|
so, the ws: uri is important, if calling
wss://websocket.mtgox.com:443/mtgox?Currency=USD"
i can subscribe to additional channels and always get btc.depth and btc.trades but cant unsubscribe to btc.depth and btc.trades.
if using
wss://websocket.mtgox.com:443
i get
"{\"message\":\"Now online (no channels)\",\"op\":\"remark\"}"
and can subscribe to any channel.
so i think unsubscribe may be only usable to unsubscribe channels one subscribed but not for avoiding messages like btc.trades that result from calling
wss://websocket.mtgox.com:443/mtgox?Currency=USD"
To unsubscribe, try using {
"op": "unsubscribe",
"channel": "id"
}
If I remember correctly, I think the id has to be the long guid code instead of the channel name when you unsubscribe. |
|
|
|
ok, here it is for all python-noobs as me - for starters: #!/usr/bin/python
import websocket
import thread
import time
def on_message(ws, message):
print 'ola, messages incoming';
print message;
print 'endofmessagedude';
def on_error(ws, error):
print 'ola, error incoming:';
print error;
def on_close(ws):
print "### closed ###";
def fuckitin(ws):
def run(*args):
#for i in range(5):
# time.sleep(1)
# ws.send("1")
#ws.send("Hello %d" % i)
#time.sleep(1)
#ws.send('');
ws.close()
print "thread terminating..."
thread.start_new_thread(run, ())
if __name__ == "__main__":
websocket.enableTrace(True)
ws = websocket.WebSocketApp("wss://websocket.mtgox.com:443/mtgox?Currency=USD",
on_message = on_message,
on_error = on_error,
on_close = on_close)
#ws.on_open = fuckitin
ws.run_forever()
how do i tell him which channel i want? There are different ways depending on the type of channel, try sending the following: {
"op": "mtgox.subscribe",
"channel": "channel-id"
}
where channel-id is one from http://data.mtgox.com/api/2/stream/list_public. You can do this as a python object, and then use json.dumps (import the json module). Have a look here for more information on websocket methods :- https://en.bitcoin.it/wiki/MtGox/API/Streaming. There's some conflicting information, but it should be helpful. |
|
|
|
Hi all, at one point nitrous posted this link: Can anyone comment on dataquality/accuracy known problems/oddities of the bitcoinchartsdata for mtgox, bitstamp and btc-e? Nitrous do you have an opinion about this? Greetings to all of you! Difficult to tell, it seems good, you'll need to judge that for yourself though I think. I have only looked at the bitstamp data, I can't comment on the other exchanges, but for bitstamp the one oddity I found is that the trade times are given with second resolution, and so it's difficult to tell the order when multiple trades happened in the same time period. It's likely that they are given in the correct order, but I'm taking a VWAP of each second to avoid any potential problems with that. |
|
|
|
once=str(int(time.time()*1000000-1)) works like a charm  next thing i need is the orderbook. all i found is /money/depth/fetch but i think i might run into ddos protection, cause what i want to do, is to closely watch the orderbook max 6 btc away from the last price, so i would have to update it at least after each new trade. websocket isnt for me.. do you have a tip for me? thx! Glad it works Sorry, unfortunately the only way to maintain a realtime copy of the orderbook is with the websocket - there is a maximum of 5 requests per hour for /money/depth/fetch, so it is simply impossible to call it any more often than once per 12 minutes, which I assume is not good enough for your purposes. If you can't use the websocket, you may want to reconsider your timescale or strategy. (What is wrong with using the websocket though? I've successfully used the websocket in python using the websocket-client package). The thread posted by MusX says the same thing. |
|
|
|
hy nitrous!
what do you consider the best method to always get the last trade (every latest trade)?
i tried (python):
get the latest time
letztertid=str(int(time.time()*1000000));
fetch trades since that
x=req('BTCUSD/money/trades/fetch?since='+jetzt, {}, True)
take the latest trade from that result and repeat.
but its lagging 3-4 trades. if i try something like that
letztertid=str(int(time.time()*1000000-864000));
it shows one lates trade and then lags some trades and then shows another latest trade.
i also tried to start with time*1000000, take the tid of the latest trade and fetch all trades since that, but that doesnt return no data (i think on your website you describe this as beeing some kind of ddos protection).
what is the minimum x i can subtract from time?
thx!
I'm not quite sure what behaviour you're seeing... Visiting BTCUSD/money/trades/fetch, the last few trades had the following tids: 1. 1389268906937622
2. 1389268907156855
3. 1389268907286795
4. 1389268919402914
5. 1389268919674069
Then going to fetch?since=1389268906937622, ie. the first one, gives: 2. 1389268907156855
3. 1389268907286795
4. 1389268919402914
5. 1389268919674069
6. 1389269081502565
7. 1389269099578993
8. 1389269100231335
9. 1389269128381212
It returned all trades after the tid I specified, as expected. If I subtract 1 (this should be the minimum you need to use for x) from tid#1, fetch?since=1389268906937621, then the results also include the first trade as well: 1. 1389268906937622
2. 1389268907156855
3. 1389268907286795
4. 1389268919402914
5. 1389268919674069
6. 1389269081502565
7. 1389269099578993
8. 1389269100231335
9. 1389269128381212
Also, you probably shouldn't use time.time() as it's very possible there won't be any trades that recent. On your first request, just call BTCUSD/money/trades/fetch instead, then start using the last trades from the last results set. Anyway, I don't seem to be seeing your problem, all the results I got were exactly as I expected, what do you mean by lagging 3-4 trades? |
|
|
|
after searching around a bit (Wikipedia... where I should have started...) I understood your explanation a bit better, what I did not get is that URL does not get encrypted as well when using SSL...
thanks for your explanations !
No problem The URL should also get encrypted, but it is also stored in server logs. It is certainly possible to work around all these things, but with all the things you'd have to work around you'd probably miss something or introduce a bug. Simply put, it's unwise to use GET when passing secure information, and far easier to use POST, whose attributes make perfect sense for use with SSL. Hmm, I haven't used those APIs but if you're right, that's really surprising. I wouldn't expect such large companies to use GET for secure requests, especially Amazon where money is involved. EDIT: Another important point, though obviously not a factor with a simple text API, is that on a webpage any third-party resources that are accessed will be sent a referrer url, which is the url of the webpage (query string and all!). |
|
|
|
Hi,
is there a reason why the API uses only POST for private function and never GET ?
The security difference seems small, or is it because GET is cacheable ? In that case they could set the cache reset at a very low value...
Or am I missing something else ?
thanks
All respectable secure site use POST with HTTPS because its essential to effectively hide potentially secret information: - The url and query string of a request are easily accessible and often recorded, so it would undermine the entire point of HTTPS to use GET.
- Another reason is that GET query strings have a maximum length which is quite short, like 1 or 2 kb, so POST makes sense for this reason as well.
- And yet another, as you identified, is the problem of caching; and no, they would never consider setting the cache refresh at a very low value because that would eliminate the whole point of a cache, to reduce server load (which often has been exploited to DDOS MtGox, so I'm pretty sure they don't want to take any chances with this).
Of course, you could argue that a computer program accessing the API probably won't record the URLs and the query strings are likely far less than 1kb and you could control caching for individual resources, but frankly there's little reason to use GET and every to use POST. Is there some tool or library you're trying to use which doesn't support POST requests? |
|
|
|
thanks a lot; itīs especially tricky as the api outputs 10 digits (date:..).. No problem To understand it better, have a look at the 'tid' values -- these are what you're passing to the 'since' argument. |
|
|
|
hy, i cant get this to work, whats wrong? (couldnt find anything on the forum or the net; i just want to get the most recent trades including tid) https://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=1388679721{"result":"success","data":[]} i know that <218867 its the tid and after that it should be the unix timestamp with 10 digits, but for everything > 218867 it just returns nothing thx! Hi, Actually, it needs to be the unix timestamp with 16 digits! After 218867 they switched to micro timestamps, so you need to include the microseconds as well. Try: https://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=1388679721000000In fact, you're actually supplying the tid after which you want the trades, not the date (though after 218867, they're equivalent). However, if you give too small a tid, the API won't return any results (to try to conserve bandwidth I think). Therefore you need to be within something like 86400000000 of your desired tid in order to get results. |
|
|
|
Hi nitrous, I tried running the app in OS X 1.6.8 (snow leopard). But it crashes after I select the dump file name in (File->new dump). It prints segmentation fault at the prompt. I was able to track the error to self.update() call at the end Application.load_dump function at gui.py file (line 217). It actually inconsistent, when I place some print around update() call, it works some time. Anyway, it ran fine when I commented the self.update() line out. I think this function updates window visual after update_button state is set to normal, isn't it? I don't have a better idea of what it could be. But I am happy to test on OS X 1.6.8 if you have any idea. If anyone is interested I posted a version with the alteration described above https://www.dropbox.com/sh/cyql48n834n9sfq/VWuOLxYA79Cheers Ahh, thank you for posting a working version. Yeah, I used Tkinter but it doesn't work well with multithreading, hence the numerous errors we've all encountered with this. It doesn't seem to work with Mavericks either. Unfortunately, I don't really have time to maintain this anymore, but hopefully it should continue to work with the dump I posted for now. Thanks again for the new update ! |
|
|
|
... Oh when i became interested in BTC this year again one thing i was really clear about was: I won't put again money in Mtgox. And i was happy for bitcoin that now there are three Big USD BTC Exchanges: btc-e, Bitstamp and Mtgox. I tried bitstamp (really like look and feel of the site, customer service replies fast, cashing out no problem so far) btc-e (service is a bit harsh) and bitfinex. Have to say Kraken looks nice maybe i give it shot. Volume is probably low so and prices are a bit higher, withdrawal problems? The reason why i wan't to use Mtgox Data is no one goes back so far. And since Mtgox has a significant higher price than other exchanges its maybe not so wise to mix the data between exchanges. ... Do you know something about timestamp with Bitstamp? Just downloaded the data, i assumend unix timestam. i converted with like this: A1/(24*3600)+25569 gives me as first trade: 13.09.2011 13:53:36 is that right Bitstamp was already back then in 2011 around - wow should have known that back then would maybe have saved some money  That's fair enough. You should probably be careful about using strategies developed for MtGox with other exchanges. Simple strategies are probably fine, but you might find subtle differences. And these will probably be amplified for more complex strategies. You should be able to compensate for this with some minor tweaks though, so using MtGox data initially should still be fine. Yeah, Kraken is quite new. Sometimes their 24h XBTEUR volume peaks above 1000 though. I think there have been a few introductory problems as they've grown in popularity, but I think they're getting through this now. Yes, they are unix timestamps and that was their first trade (or, at least, the first one recorded by bitcoincharts). Haha, hindsight is a wonderful thing |
|
|
|
Have you found an inconsistency? No i was just thinking about it since its very destructive if you look at trading data and do not realize that the data might be looking 1 or 2h in the future. UTC is pretty easy to get right, his servers should just automatically sync with a reliable timeserver in order to get reliable UTC. Hope/guess you are right, but when i look at the "track record" of Mark Karpeles, i'm not sure. What he fucked up in the last 2 years is an achievement in itself. Good nite its late here... Hmm, well the last timestamp from the 2nd dump is consistent with the time I made it, so it looks like recent data is at least consistent with UTC. I suppose the only advice I could give is to do a dry run of any algorithm you develop first, just to see whether it makes the right choices in a period of, say, a week, before risking any real assets. Personally I've been looking at other exchanges more recently, so you might want to consider other exchanges too, especially as you're worried about whether you can trust Mark. I'm quite interested in Kraken as it looks like it has a lot of potential, but I'm primarily using Bitstamp at the moment. There's some published CSVs for many exchanges here http://api.bitcoincharts.com/v1/csv/, although obviously then you have to trust bitcoincharts to have collected the data properly (I haven't analysed their data yet, but I assume it's fairly reliable; from a quick look the bitstamp one definitely seems to include the major price spike at roughly the right time). Good night |
|
|
|
How was the timestamp of the data again created. I know i read it somwhere but don't remember. Was it UTC timestamp is there a problem with daylight savings time of Japan? As far as i understand UTC is a constant "world time" and all timezones of the earth a derived from UTC.
Is something known, that there is some problem with the timestamps like Mark K. changed (for whatsoever reasons) , the timesettings of servers or something like this?
Have you found an inconsistency? I'm pretty sure they should be consistent with UTC, I take the timestamps directly from that returned by each one of the APIs, and I don't think there are any problems. If you think there are though, tell me which ones and maybe I can have a quick look. If you're just speculating though, then you should be fine. UTC is pretty easy to get right, his servers should just automatically sync with a reliable timeserver in order to get reliable UTC. |
|
|
|
how do you use cryptotrader to do real trading?
You can get a trading account here https://cryptotrader.org/plans. Free accounts can develop and backtest algorithms, but to actually trade you need to subscribe. |
|
|
|
The (windows version of the) tool crash after downloading about 100Mb of data, and the crash occur every time at the same spot. I have tried several times, doesn`t matter if I`m downloading a new dump or just try to resume one. Rows downloaded: 1024000
Latest TID: 1366547913512370
Data up to: 2013-04-21 12:38:33
Update in progress - 5074502 rows to download
A few other people seem to be having this problem (see the last couple pages of this thread). I'm not entirely sure why it's happening, but I think Google have changed their protocols subtly and it's broken the tool. As a result, I've created some full dumps, including data all the way up to mid-December 2013 here. |
|
|
|
RFC should really standardise the csv number format as this wasn't very obvious. Anyway, I'm glad you've fixed the problem now, and hopefully this will be useful for other people Another thing that logic afterwards but a bit confusing is: Excel automatically changes the "." to a "," as decimal separator when you do it as described above, because this is the country setting (in europe you use "," to separate decimal numbers) if you now save the file again as .csv Excel is not saving the "," as "." again, it leaves the "," - this leads now to the problem what you mentioned that excel could confuse colums, but then its smart and changes the commas as separators with ";" with is well thought out from MS, but one can be confused when importing to the charting software and then thinks "But wtf why is comma not working i saved this as Comma Separated..." Now we have it all through with Excel what It should all be good now, but i don't have time to check it out now, Since i have to pack my stuff fast for "Driving home for Christmas" and there i probably will be pretty busy... Which brings me to the point: Nice X-mas to you nitrous and all of this board here till later! Thanks, Merry Christmas to you too  |
|
|
|
Hi,
i played with this. It was my fault sorry if i caused extra work. The explanation what was the exact Problem might be interesting for people from Europe (like me i'm from germany) importing this into Excel.
The Setting here have to be like this.
...
The "decimal separator" (that's how its called in German Version translated to english) has to be "."
In my defendance i have to say it wasn't logic for me that this could have caused this error since, and that was now what was the real trick: The combination of the "1000's separtor setting" and the "decimal- separator" led to the "weird" effect that all numbers below 1 where shown correctly but all number beyond 1 not. finally realising this led me to the point that i thought: "Hm O.K. it shouldn't be somethign with the decimal seperator since the numbers in the beginning are right, but let's check it again.
Sorry for wasting your time...
Greetings.
Hi BNO, Haha, no don't worry about wasting my time, that was really confusing! That makes a lot of sense now, although now I'm not sure how Excel managed to distinguish between the columns considering they use a comma for the column separator... RFC should really standardise the csv number format as this wasn't very obvious. Anyway, I'm glad you've fixed the problem now, and hopefully this will be useful for other people |
|
|
|
I just found this resource and I thought it would be useful to some people -- http://api.bitcoincharts.com/v1/csv/I can't believe I didn't find it before, anyway, it has regularly updated (15mins?) CSV files with {unix timestamp (1s resolution), price (float), volume (float)} fields for many different exchanges and currencies. |
|
|
|
Hi Nitrous, wanted to write you yesterday but didn't find the time to do so.. One problem occured in the field price when i looked at the data: ... i remember that in the BigQuery table you had to divide the price field by 10.000. Here something seems to have gone wrong you know what? The field Volume i divided by 100.000.000 for adjusting to full coins, might something similiar have happened in this field too? Bye Edit: I just saw that the .csv you script created is fine, it must have happened during import to Excel. Do you have an idea what might have caused this?  Hi BNO, Excel gets stranger and stranger! I just did a test CSV export myself and the CSV is definitely fine, I'm not sure how or why excel is mangling the data like this. Dividing one column by 1e8 should not affect any other column, especially as inconsistently as the price column seems to be being affected :S What is the exact procedure by which you imported the data into excel? |
|
|
|
|
Show Posts
