SierraChart feed/bridge reborn - Realtime Bitcoin charting

[deepceleron]

Once again major unannounced API changes at bitcoincharts have broken software.

Sierrachartfeed gets CSV data from this URL (this is the only source for trades older than five days):

http://api.bitcoincharts.com/v1/csv/

As documented here, it was individual nightly CSV files containing all historical trades for each exchange, about 300MB files for the top exchanges. Sierrachartfeed maintains a local copy of the CSV and "resumes", adding new data to your existing CSV file.

Now the remote data files are zipped. That means that any previous CSV data that was downloaded is useless. The old file names have changed, the files are now, for example, "mtgoxUSD.csv.gz". This file cannot be resumed, as it is compressed data wrapped in a header and footer. I would need to write gzip-aware software that can strip out the gzipped stream from the current file and rebuild it with added data, and of course unzip to import the new trades. And that's if bitcoincharts doesn't screw up the data contents again - one bit in one early trade changing will change the entire gzipped archive.

I don't know if the proper response is to say fuck it and not use data from that site any more, or to show that regardless of their contempt for users and software developers, we can make it work.

[arklan]

I was wondering what was up. Sigh. Good luck DC. If I could help in anyway I would...

[xeverse]

Is it possible to disable history and just have the live socket stream for now?

[deepceleron]

Okay, it's fixed again, discarding about 100 lines of new code I had written to auto-resume using CSV data. First post updated.

You can now manually download and extract the complete zipped history CSVs to the SierraChart data directory, but won't be prompted to do so if it is a new install or if you specify a new ticker symbol you previously hadn't downloaded. It will simply give you five days of history. You also won't have to get CSVs as long as you run sierrachartfeed and have been keeping your local ticker data updated.

If you have already run sierrachartfeed to download ticker data and are more than five days behind, the software can't automatically catch up. You will get a dire warning from sierrachartfeed that you need to re-download these CSVs manually:


> scfeed.exe -s mtgoxUSD

*** Loading data file C:/SierraChart/Data/mtgoxUSD-p4.scid
importing C:/SierraChart/Data/mtgoxUSD.csv
 - No CSV history file found for mtgoxUSD, skipping
 - (Download manually from http://api.bitcoincharts.com/v1/csv/)
 - (unzip to C:/SierraChart/Data/mtgoxUSD.csv)

Request mtgoxUSD from 2014-01-10 11:49:29

*** Danger! ***

 Your saved mtgoxUSD history is 640 hours older
 than that available from history website

*** Continuing download will create an unrecoverable gap in trade data ***
Press [c] and [enter] if you want to continue:

- For trades older than 5 days, manually download a new CSV file from
  http://api.bitcoincharts.com/v1/csv/

You can proceed by typing the letter "C" and pressing enter; in my example above, I just pressed enter, so the program quit and gave info.

Happy charting.

[xeverse]

Thank You!

[arklan]

How often are the csvs updated? I'm away from my charting / mining rig.

[deepceleron]

How often are the csvs updated? I'm away from my charting / mining rig.

nightly

[xeverse]

*** Loading data file /xe/_4mkt/2data/bitstampUSD-p4.scid
importing /xe/_4mkt/2data/bitstampUSD.csv
 - No CSV history file found for bitstampUSD, skipping
 - (Download manually from http://api.bitcoincharts.com/v1/csv/)
 - (unzip to /xe/_4mkt/2data/bitstampUSD.csv)

Request bitstampUSD from 2014-02-15 09:47:20
    Got 2014-02-15 09:47:20Z to 2014-02-15 09:47:20Z: 1 trades (43 bytes in 0.208534 sec)

--------------------------------------------------------------------------------------

There is a bunch of these
cvs.imported files instead in the dir.

Why they aren't recognized then?

[arklan]

*** Loading data file /xe/_4mkt/2data/bitstampUSD-p4.scid
importing /xe/_4mkt/2data/bitstampUSD.csv
 - No CSV history file found for bitstampUSD, skipping
 - (Download manually from http://api.bitcoincharts.com/v1/csv/)
 - (unzip to /xe/_4mkt/2data/bitstampUSD.csv)

Request bitstampUSD from 2014-02-15 09:47:20
    Got 2014-02-15 09:47:20Z to 2014-02-15 09:47:20Z: 1 trades (43 bytes in 0.208534 sec)

--------------------------------------------------------------------------------------

There is a bunch of these
cvs.imported files instead in the dir.

Why they aren't recognized then?

That looks like the message DC was just taking about four csv files that are more then five days out of date.

[xeverse]

That looks like the message DC was just taking about four csv files that are more then five days out of date.

Yes, but only these renamed files are like 10min out of sync max.
You don't experience csv.renamed files?
https://i.imgur.com/OClQvXs.png

[deepceleron]

*** Loading data file /xe/_4mkt/2data/bitstampUSD-p4.scid
importing /xe/_4mkt/2data/bitstampUSD.csv
 - No CSV history file found for bitstampUSD, skipping
 - (Download manually from http://api.bitcoincharts.com/v1/csv/)
 - (unzip to /xe/_4mkt/2data/bitstampUSD.csv)

Request bitstampUSD from 2014-02-15 09:47:20
    Got 2014-02-15 09:47:20Z to 2014-02-15 09:47:20Z: 1 trades (43 bytes in 0.208534 sec)

--------------------------------------------------------------------------------------

There is a bunch of these cvs.imported files instead in the dir.

Why they aren't recognized then?

If the csv files have been renamed by sierrachartfeed to ".imported", then all the data in them is already in the Sierrachart SCID file; the ".imported" files and ".csv.gz" files should be deleted after successful import (unless you have a specific reason you might use them again soon; they are obsolete after four days). You can ignore the "no CSV history file found" message - it is not an error in this case, just a normal procedural message.

You will get the big *danger* error message (like four posts above) when there is an actual problem that requires you to re-download a new CSV (because the bitcoincharts API only has five-day-old and newer data).

The csv files inside the gzips actually start with a period, like ".bitstampUSD.csv". I've done a few tricks to look for the csv either with or without a period at the start.

[arklan]

so i've just gottne back to my charting pc, and triedthe new exe. on launch it throws "the program can't start because python27.dll is missing from your computer."

i'm assuming i could just install python to fix that, but if i can avoid doing so, i prefer to keep a tidy ship.

[deepceleron]

so i've just gottne back to my charting pc, and triedthe new exe. on launch it throws "the program can't start because python27.dll is missing from your computer."

i'm assuming i could just install python to fix that, but if i can avoid doing so, i prefer to keep a tidy ship.

Uploaded a new exe made the same way as before with dlls included.

[arklan]

assuming you mean the download link in the first post - i tried it, no difference.

[movellan]

so i've just gottne back to my charting pc, and triedthe new exe. on launch it throws "the program can't start because python27.dll is missing from your computer."

i'm assuming i could just install python to fix that, but if i can avoid doing so, i prefer to keep a tidy ship.

Uploaded a new exe made the same way as before with dlls included.

Now that the mtgox feed  no longer exists, will you be adding any other exchanges to the feed? TIA.

[xeverse]

http://www.reddit.com/r/Bitcoin/comments/1zfl0e/bitstamp_adds_websocket_api/

[movellan]

http://www.reddit.com/r/Bitcoin/comments/1zfl0e/bitstamp_adds_websocket_api/

Thanks to deepceleron SC already has feeds for both Bitstamp and BTC-e, which appear to be working fine. I was wondering with MtGox down (perhaps not out) for now if he was thinking about adding say, Localbitcoins or some other higher volume exchange.

[deepceleron]

The exchange data from localbitcoins is crazy looking:



MtGox may be interesting for historical reasons, but I think I'll remove it as a default the next time I need to make a big update.

[movellan]

MtGox may be interesting for historical reasons, but I think I'll remove it as a default. The quick fix I put up has a bug to be fixed, in that it doesn't alert you in many situations when your sierrachart data is over five days old, and will create a trade gap.

Speaking of which, scfeed.exe doesn't seem to be working at all today. It will most often exit prematurely or when it does fully load just sits waiting for the feed.

PS - Just had another look at Bitcoincharts and it appears they haven't updated for 7-8 hrs. Is scfeed designed to auto close when no new data is found?

[movellan]

The exchange data from localbitcoins is crazy looking:



MtGox may be interesting for historical reasons, but I think I'll remove it as a default. The quick fix I put up has a bug to be fixed, in that it doesn't alert you in many situations when your sierrachart data is over five days old, and will create a trade gap.

While I'm thinking about it, have you tried applying the Stochastic Crossover System and MACD-Volume Weighted studies along with straight Volume to the Localbitcoins data?  I have used these with the MTGox data and they have an uncanny way of predicting large movement as the crossover points merge, sometimes giving a 1-5 min lead to the action.