SierraChart bridge - Realtime Bitcoin charts [v0.5] (MtGox, Intersango, ...)

[kvm]

Thank you, deepceleron - very helpful findings! These endless changes are very discouraging, because sierrachartfeed is not the only software using Bitcoincharts API. I wrote something similar but for Metatrader platform:

https://bitcointalk.org/index.php?topic=160791.0

And guess what - it doesn't work anymore

[dserrano5]

The script I use to maintain https://dl.dropboxusercontent.com/u/24587684/mtgox-daily/2013/09/04 (substitute any date, and/or use "mtgoxEUR-daily") still seems to work for me.

[deepceleron]

New version of Sierrachartfeed and direct-download history files, working again

Update 022:

• Fixed to work again after bitcoincharts API changes,
• Get all symbols option (--symbols=a) forces immediate history download of known tickers, instead of waiting to discover them only when a trade is seen,
• Maybe better --help,
• Error handling changes,
• Maximum history to catch up default to 30 days.
• Note, you must have a somewhat correct date/time on your computer or strange things may happen.

and update 023:

• Days of history option now only enforced for fresh download, will always resume from last date in scid data to avoid creating gaps. (only was an issue because the first API break with 0.5 could have put old dates at end of scid and made downloading go crazy).
• Fix 1 in 3 chance of not resuming after a long series of "result too big" errors if there were ever 20,000 trades in under 8 seconds. (Simply can't work correctly if there are ever more than 20,000 trades per second due to API limitation)

Code:

Usage: sierrachartfeedUTC-0.6.023.exe [options]

Options:
  -h, --help            show this help message and exit
  -d DATADIR, --datadir=DATADIR
                        Data directory of SierraChart software (default =
                        c:/SierraChart/data/
  -s SYMBOLS, --symbols=SYMBOLS
                        Ticker symbols to download, comma separated. * or a
                        for all. (default = mtgoxUSD)
  -l HISTORY, --history=HISTORY
                        Maximum days if no ticker history previously
                        downloaded (default = 30).
  -y, --disable-history
                        Disable trade history download; get live trades only
                        (default = False)
  -p PRECISION, --volume-precision=PRECISION
                        Multiply volume by this many decimal places before
                        import (default = 2).
  -t, --tickers         List all ticker symbols and exit

note: when using the single-letter option, there is no space between the option and the parameter


Download links:
http://we.lovebitco.in/schart/sierrachartfeedUTC-0.6.023.exe (4.0MB) (windows 32 bit standalone exe)
MD5: 93d1b5930c4f2efae571144920b7e774 *sierrachartfeedUTC-0.6.023.exe

http://we.lovebitco.in/schart/sierrachartfeedUTC-0.6.023.py (python 2.7 source - needs github files also)

SCID full history files (use if your data directory was screwed up by unannounced API changes):
from first recorded trade to Wed, 21 Aug 2013 23:59:59 GMT:
(extract to C:\SierraChart\data\ before starting SierraChartfeed)

MtGox only: http://we.lovebitco.in/schart/mtgoxUSD.scid.UTC.7z (22.6MB/227MB)

• mtgoxUSD

Other current exchanges: http://we.lovebitco.in/schart/otherALL.scid.UTC.7z (19.0MB/200MB)

• bit2cILS, bitboxUSD, bitcashCZK, bitcurexEUR, bitcurexPLN, bitkonanUSD, bitnzNZD, bitstampUSD, btcdeEUR, btceEUR, btceRUR, btceUSD, btchkexHKD, btcnCNY, cbxUSD, crytrEUR, crytrUSD, fbtcEUR, fbtcUSD, fybseSEK, fybsgSGD, icbitUSD, intrsngEUR, intrsngGBP, intrsngPLN, justLTC, justNOK, justXRP, kptnSEK, localbtcEUR, lybitCAD, lybitUSD, mrcdBRL, mtgoxAUD, mtgoxCAD, mtgoxCHF, mtgoxCNY, mtgoxDKK, mtgoxEUR, mtgoxGBP, mtgoxHKD, mtgoxJPY, mtgoxNZD, mtgoxPLN, mtgoxRUB, mtgoxSEK, mtgoxSGD, mtgoxTHB, rippleEUR, rippleUSD, rippleXRP, rmbtbCNY, rockEUR, rockSLL, rockUSD, vcxEUR, vcxUSD, virtexCAD, virwoxSLL, weexAUD, weexCAD, weexUSD

Past/retired exchanges (Tradehill, etc): http://we.lovebitco.in/schart/historic.scid.UTC.7z  (3.4MB/32.9MB)

• aqoinEUR, b2cUSD, b7BGN, b7EUR, b7PLN, b7SAR, b7USD, bbmBRL, bcEUR, bcGBP, bcLREUR, bcLRUSD, bcmBMAUD, bcmBMGAU, bcmBMUSD, bcmLRUSD, bcmMBUSD, bcmPPUSD, bcmPXGAU, bcPGAU, bitchangePLN, bitfloorUSD, bitmarketAUD, bitmarketEUR, bitmarketGBP, bitmarketPLN, bitmarketRUB, bitmarketUSD, bitmeUSD, bitomatPLN, britcoinGBP, btc24EUR, btc24USD, btcexEUR, btcexJPY, btcexRUB, btcexUSD, btcexWMR, btcexWMZ, btcexYAD, btctreeUSD, cryptoxAUD, cryptoxUSD, exchbUSD, freshPLN, globalEUR, globalGBP, globalPLN, globalUSD, imcexEUR, imcexUSD, intrsngUSD, ruxumAUD, ruxumCHF, ruxumEUR, ruxumGBP, ruxumHKD, ruxumHUF, ruxumJPY, ruxumPLN, ruxumRUB, ruxumSEK, ruxumSGD, ruxumTHB, ruxumUAH, ruxumUSD, ruxumZAR, snwcnXRP, thAUD, thCLP, thEUR, thINR, thLRUSD, thUSD, wbxAUD

[imrer]

Great work I have to admit. 

[deepceleron]

It looks like I found another bitcoincharts API bug for you programmers - bitcoincharts doesn't always return the correct 20,000 trades if you only specify the &start timestamp parameter.

Example:
http://api.bitcoincharts.com/v1/trades.csv?symbol=mtgoxUSD&start=1270000000
  - returns timestamps starting at 1279408157 (first trade 17 Jul 2010 23:09:17 GMT)
http://api.bitcoincharts.com/v1/trades.csv?symbol=mtgoxUSD&start=1290000000
 - returns timestamps starting at 1307725482
http://api.bitcoincharts.com/v1/trades.csv?symbol=mtgoxUSD&start=1290000000&end=1299000000
  - returns timestamps starting at 1290001901
http://api.bitcoincharts.com/v1/trades.csv?symbol=mtgoxUSD&start=1320000000
 - returns timestamps starting at 1320000026

edit: seems to be fixed.
edit: and then it's not again.

[RyNinDaCleM]

Thank you DC! I'll be sending a tip when I start my pc and get QT up to date

[arklan]

whats up with bitstamp's data lately? i haven't gotten anything all week, since the 9th. and yes, i'm running the latest exe.

[deepceleron]

I posted a long post that was eaten by a forum database error. Essentially the bitstamp stream API that other services like bitcoincharts use is broken: https://bitcointalk.org/index.php?topic=38711.msg3124043#msg3124043

Bitstamp has an API for getting historic trades directly, however the request method (by trade number) can't easily be used to resume Sierrachart SCID data (using the last-seen trade timestamp).

[arklan]

so after a windows reinstall, i'm resetting this up on my end - and noticed there were some exchanges available i hadn't noticed before, such as btcnCNY and virwoxSLL... continuously impressed with this feed, DeepCeleron.

[deepceleron]

I've continued to plug around with some code features, but nothing that is worth sending over to the build machine.

- precision option other than 2 creates separate scid files with a suffix (like mtgoxUSD_p4.scid), so experimenting with this option doesn't screw up your data,
- max precision 5, as historic mtgox trades > 4294.9 will overflow precision 6 and higher,
- verbose option dumps individual history trades to console (slow); console presentation fixes,
- "max_time" option, won't write live stream or history after this epoch timestamp to scid (for creating accurate shareable files; the transition from downloading history to live stream may miss some trades). So I don't have to maintain a modded .py to do this.

[arklan]

i seem to be having issue with the -s option... i'm on widows.

C:\SierraChart\Data\sierrachartfeedUTC-0.6.023.exe -s bitstampUSD, btceUSD, mtgoxUSD, btcnCNY only grabs bitstamp data, streaming and historical.

C:\SierraChart\Data\sierrachartfeedUTC-0.6.023.exe -s bitstampUSD, -s btceUSD, -s mtgoxUSD, -s btcnCNY only grabs the last one. what am i doing wrong?

[Portnoy]

i seem to be having issue with the -s option... i'm on widows.

C:\SierraChart\Data\sierrachartfeedUTC-0.6.023.exe -s bitstampUSD, btceUSD, mtgoxUSD, btcnCNY only grabs bitstamp data, streaming and historical.

C:\SierraChart\Data\sierrachartfeedUTC-0.6.023.exe -s bitstampUSD, -s btceUSD, -s mtgoxUSD, -s btcnCNY only grabs the last one. what am i doing wrong?

Try it like this:

C:\SierraChart\Data\sierrachartfeedUTC-0.6.023.exe -s bitstampUSD,btceUSD,mtgoxUSD,btcnCNY

[arklan]

i seem to be having issue with the -s option... i'm on widows.

C:\SierraChart\Data\sierrachartfeedUTC-0.6.023.exe -s bitstampUSD, btceUSD, mtgoxUSD, btcnCNY only grabs bitstamp data, streaming and historical.

C:\SierraChart\Data\sierrachartfeedUTC-0.6.023.exe -s bitstampUSD, -s btceUSD, -s mtgoxUSD, -s btcnCNY only grabs the last one. what am i doing wrong?

Try it like this:

C:\SierraChart\Data\sierrachartfeedUTC-0.6.023.exe -s bitstampUSD,btceUSD,mtgoxUSD,btcnCNY

welp, i feel like an idiot. thanks. working fine now.

[xeverse]

It's sad but our lovely SC doesn't have tools for measuring logarithmic charts properly.

And what even more disappointing, currently they don't care about the issue.

http://anonym.to/https://www.sierrachart.com/SupportBoard.php?ThreadID=1428

such is the limitation of the proprietary licensing scheme.

our community is in need of a foss charting solution.
as powerful and advanced as SC itself.

[myself]

deepceleron can you look into this http://www.sierrachart.com/image.php?l=1384297901550.png

[deepceleron]

deepceleron can you look into this http://www.sierrachart.com/image.php?l=1384297901550.png

Am I talking to myself here?

http://bitcoincharts.com/charts/btcnCNY#rg360ztgSzm1g10zm2g25

It looks like you have a big gap in data from Aug to Oct, perhaps created by an older sierrachartfeed with a limit on the number of days to resume. That cannot be easily repaired.

After getting the latest sierrachartfeed, posted above, I would delete the btcnCNY.scid file and get the whole thing again using the command line options:

sierrachartfeedUTC-0.6.023.exe --history=700 --symbols=btcnCNY

(set history to how may days you really need; this exchange has data from July 2011.)

If all exchange data has such a gap in recorded dates, extract my scid 7zips to your sierrachart data directory to save some time.

[myself]

let me check again and look at my bat file
edit

Code:

sierrachartfeed.exe --volume-precision 4 -l 1000 -s *

also the base files are the files ziped that you offer for download

[deepceleron]

let me check again and look at my bat file
edit

Code:

sierrachartfeed.exe --volume-precision 4 -l 1000 -s *

also the base files are the files ziped that you offer for download

The scid files I shared are default precision 2, they are incompatible with a different precision setting if you are charting volume. Do you know why you are changing that option?

[myself]

let me check again and look at my bat file
edit

Code:

sierrachartfeed.exe --volume-precision 4 -l 1000 -s *

also the base files are the files ziped that you offer for download

The scid files I shared are default precision 2, they are incompatible with a different precision setting if you are charting volume. Do you know why you are changing that option?

k i started to download all the data on precision 4

Do you know why you are changing that option?

yep

[myself]

hmmm is not downloading any data