arklan
Legendary
 Offline
Activity: 1778
Merit: 1008
|
 |
November 20, 2013, 06:48:38 PM |
|
hmmm is not downloading any data  same, stream data seems halted, but restarting mine did collect the historical data from the missing time... |
i don't post much, but this space for rent.
|
|
|
deepceleron
Legendary
Offline
Activity: 1512
Merit: 1036
|
|
November 20, 2013, 07:59:06 PM
Last edit: November 20, 2013, 08:35:47 PM by deepceleron |
|
I've had the streaming interface running for about 8 hours, and it was showing trades as I started reading this thread. Just now it had just gone for ~320 seconds without receiving any trades, but just got prices again. (Using this related script: https://bitcointalk.org/index.php?topic=339066.msg3636549#msg3636549) Maybe nobody's trading today? I checked http://data.mtgox.com/api/2/BTCUSD/money/ticker_fast, and it is occasionally updating, and matches the current stream price. edit: just got a 502 gateway error from market json, site is not feeling well... |
|
|
|
|
|
|
|
Martijnvdc
|
|
November 21, 2013, 08:28:21 AM |
|
Will there be a linux release for this?
|
|
|
|
|
|
xeverse
|
|
November 21, 2013, 01:12:52 PM |
|
Will there be a linux release for this?
Just run python code. |
|
|
|
|
dserrano5
Legendary
Offline
Activity: 1974
Merit: 1029
|
|
November 25, 2013, 01:59:23 PM |
|
Unfortunately bitcoincharts is returning 404 since yesterday.
|
|
|
|
|
deepceleron
Legendary
Offline
Activity: 1512
Merit: 1036
|
|
November 25, 2013, 07:33:49 PM |
|
It looks like some more undocumented unannounced changes to break things. 1. Per-IP request limiting to 1 per 60 seconds or so 2. history API no longer works like it did. A "start" only request seems to be the only type of history API request that returns data: A request like: http://api.bitcoincharts.com/v1/trades.csv?symbol=mtgoxUSD&start=1385000001dumps about 1MB of now HTML-header-wrapped trades, but with 0D 0A (linefeed) between trades instead of any HTML: <html><head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8"></head><body>1385000001,635.500010000000,0.299400000000
1385000008,635.500010000000,0.250000000000
1385000050,635.500010000000,1.000000000000
1385000063,635.500000000000,1.802060090000
1385000090,635.500000000000,1.197939910000
1385000091,635.500010000000,0.368739030000
1385000092,635.500010000000,0.945989800000
1385000092,635.640000000000,0.756793140000
...These calls were previously documented to not consistently return the correct time periods. For now, you can temporarily backup and remove your C:\SierraChart\Data\*.scid files, and start with the --disable-history and at least see live trades. I will try to correspond and see if it is fruitless to try to write any API interface that is expected to work for more than a few months. |
|
|
|
|
deepceleron
Legendary
Offline
Activity: 1512
Merit: 1036
|
|
November 26, 2013, 08:42:59 AM
Last edit: November 27, 2013, 11:43:35 AM by deepceleron |
|
I started working on sierrachartfeed to see what can be done. The bitcoincharts history API is purposely borked. It's been modified to only have five days of trades available regardless of the query. The HTML wrap was removed, but the ~ 1 request a minute limitation is still there. The only way to fix this is to get the complete uncompressed CSV download (http://api.bitcoincharts.com/v1/csv/, 300MB for just mtgox) if SCID data is older than five days. I think I tried HTTP byte range requests on these CSV files before and the server didn't support it; there's no Accept-Ranges header. If it did, I would still have to make requests "blind", since I could only approximately index them to find where to continue.....or have a charting app that only goes back five days. Edit: the full history file.csv server supports byte ranges. I wrote a library that will get chunks by ticker... Gameplan: resume history from csv to 1 day old slowly catch up last day from crappy history API Hurdles: -Efficiently index and seek for resume timestamp in csv, without local copy -1 minutes between requests means naive history to live transition will miss trades, will need to buffer live stream until stream trades are also in history, and merge ticker lists. |
|
|
|
|
arklan
Legendary
Offline
Activity: 1778
Merit: 1008
|
|
November 26, 2013, 11:21:51 PM |
|
so, in short, we're screwed?
|
i don't post much, but this space for rent.
|
|
|
|
xeverse
|
|
November 27, 2013, 07:59:30 PM |
|
alas, the streaming api is down either, clearly we need a p2p solution.
|
|
|
|
|
arklan
Legendary
Offline
Activity: 1778
Merit: 1008
|
|
November 27, 2013, 10:25:44 PM |
|
alas, the streaming api is down either, clearly we need a p2p solution.
agreement. |
i don't post much, but this space for rent.
|
|
|
deepceleron
Legendary
Offline
Activity: 1512
Merit: 1036
|
|
November 28, 2013, 10:56:32 AM |
|
The ISP of bitcoincharts was hit with a DDOS, and it blackholed many sites to mediate the problem. The site is back up.
I have opened a dialog with the site admin, hopefully this will result in an API that will continue to work for applications such as sierrachartfeed.
|
|
|
|
|
|
xeverse
|
|
November 28, 2013, 11:16:33 AM |
|
Thanx DC.
|
|
|
|
|
arklan
Legendary
Offline
Activity: 1778
Merit: 1008
|
|
November 28, 2013, 02:53:52 PM |
|
The ISP of bitcoincharts was hit with a DDOS, and it blackholed many sites to mediate the problem. The site is back up.
I have opened a dialog with the site admin, hopefully this will result in an API that will continue to work for applications such as sierrachartfeed.
Awesome. I was worried id be stuck without... My second monitor just wouldn't be the same. |
i don't post much, but this space for rent.
|
|
|
deepceleron
Legendary
Offline
Activity: 1512
Merit: 1036
|
|
November 28, 2013, 06:27:09 PM |
|
The history stream API IP address has no IPv4 DNS right now, but for those with native IPv6 it has an AAAA address. It also has been half-returned to working status, there is five days of history (instead of all history available), with date ranges working the way it used to. This should allow you, if your backup/remove old SCIDs if you have been down a while, to run with --history=6 and see some perspective history.
I'm going on holiday for a bit, hopefully the IP and full history availability will be restored to the API.
|
|
|
|
|
Sukrim
Legendary
Offline
Activity: 2618
Merit: 1007
|
|
November 29, 2013, 01:33:55 PM |
|
Any chance of using the protocol in https://github.com/SierraChart/GSP (ideally a python implementation of it) in the future instead of SCID files? This could enable the bridge to allow trading via API keys too, you'd basically host your own broker server. |
|
|
|
|
ssshhh
|
|
November 30, 2013, 09:06:39 PM |
|
Greetings, Forex broker www.FXOpen.com has BTCUSD data. You will have to enable it in the "symbols" section. [ right click on any currency pair and select symbols -> crypto ] have fun! |
|
|
|
|
|
xeverse
|
|
December 18, 2013, 11:41:21 AM |
|
Can somebody share his bitstampUSD.scid? There's something wrong with mine..  |
|
|
|
|
wev
Newbie
Offline
Activity: 19
Merit: 0
|
|
December 18, 2013, 08:22:47 PM |
|
https://mega.co.nz/#!0xtRUKJL!fXZX-R3ZkGSo9sf-lJDQbncWcCVWRC5guQNEr_cIx-o Let me know if it works for you.
|
|
|
|
|
deepceleron
Legendary
Offline
Activity: 1512
Merit: 1036
|
|
December 19, 2013, 04:44:35 PM
Last edit: December 19, 2013, 08:26:05 PM by deepceleron |
|
I've got it working again. Huzzah. I just need to iron out some crap data from bitcoincharts CSVs, like this: 1387430609,560.640000000000,0.356700000000
1387430826,667.500000000000,0.299600000000
1387430714,554.260000000000,0.036100000000
1387431207,692.500000000000,0.151600000000
1387431697,1000.000000000000,0.150000000000
1387431246,612.700000000000,0.100000000000
1387431516,540.290000000000,0.029000000000
1387431550,583.000000000000,0.514600000000Those are timestamps in seconds (in this case from localbtcUSD). You can see that at least a 10 minute range of trades are all randomized, which really pisses off SierraChart. I'll need to sort them efficiently without impacting 500MB+ of exchange data that only has a minor problem (and so far, they all do to some extent). Hint: to get it working faster once I release the code, start downloading your exchange's CSV files now, and put them in the C:\SierraChart\Data directory (don't worry, SierraChart feed will download it and keep it up to date if you don't...) I will be ignoring previous SCIDs by using new filenames. DONE: moving this to a new thread so OP doesn't have non-working code and links: https://bitcointalk.org/index.php?topic=377649.0 |
|
|
|
|
|
