SierraChart bridge - Realtime Bitcoin charts [v0.5] (MtGox, Intersango, ...)

[xeverse]

For some reason, this "fix" fails to download anything past the last week or so of trade data.  Is there any way to download data farther back?  My mtgoxUSD.scid file is missing historical trade data from April 19, 20, 21, and 22

Do you mean gaps? Stop the bridge > Edit > Edit/DownloadData-IntraChart > Set the date to apr18 > Del and Dl Data from current pos to end

[1713582240]

1713582240

[1713582240]

1713582240

[1713582240]

1713582240

[1713582240]

1713582240

[dserrano5]

For some reason, this "fix" fails to download anything past the last week or so of trade data.  Is there any way to download data farther back?  My mtgoxUSD.scid file is missing historical trade data from April 19, 20, 21, and 22

Another option: https://dl.dropboxusercontent.com/u/24587684/mtgoxUSD-known-good.zip (35 Mb), updated this last weekend.

[deepceleron]

For some reason, this "fix" fails to download anything past the last week or so of trade data.  Is there any way to download data farther back?  My mtgoxUSD.scid file is missing historical trade data from April 19, 20, 21, and 22

Code:

Usage: sierrachartfeed.py [options]
...

  -l HISTORY, --history=HISTORY
                        Number of days of history to retrieve (**default=7**).

Update 017: refine rate adaptation, enforce max history when catching up scid, give a user-agent

Because of how poor the API has performed, with about 10% gateway errors and noticeable random long requests (and the fact that we are already abusing the "one API request per 15 minute"), I put in enforcement of the number of days downloaded when catching up an old SCID also, so that if you haven't used the program for six months or if your SCID ends with 2011 data from previous fail versions, it doesn't go crazy and hammer the server, unless you specifically tell it to get more history at the command line.

If you have already created an SCID with a gap, the program can't go back and fill it in, as data is expected chronologically.

Besides the method two posts above to remove data after a gap (my version of Sierrachart doesn't have that exact option), from the edit menu you can also "export and edit to txt", cut lines off the end, and then "import and load".

The file format is 54 bytes of header + 40 bytes per record, so you can also use a utility like GNU split to experimentally cut off extra new data from the tail of the SCID file if you have received bad data at the end, from the original version failing, or from not seeing my change.

[xeverse]

What do you think about p2p rt datafeed solution? Is it possible or not?

[deepceleron]

What do you think about p2p rt datafeed solution? Is it possible or not?

Right now I'm working through fixing the first obstacle I found in trying to produce shareable data (such as importable txt files or other people's SCID files) - putting all times in UTC/GMT. The bridge has a design problem, it converts all times to local times, which is wrong according to SierraChart docs, and it even messes up the data during the switch to daylight savings time or moving the computer between timezones, so basically all of you have somewhat erroneous history. The data should be UTC, and import/export should also be in UTC.

Fixing this will mean a new program version, and users must wipe all SCIDs and old history before using it. Since API is so slow, I'll probably include a feature to get the full csv history direct-download daily-generated files when appropriate and/or might throw up my own importable files you can use before starting bridge.

Peer to peer is not an awesome idea, bad peers could feed you bogus info or exploit your open port.

[xeverse]

putting all times in UTC/GMT.

I've noticed time switching..

Peer to peer is not an awesome idea, bad peers could feed you bogus info or exploit your open port.

Then there gotta be an ids in place and protection based on fuzzy logic ruling out bogus data.
That way it would be similar in resistance to 51% of the bitcoin system itself.

[zhalox]

Thanks deepceleron, I've got it working now, and I really appreciate your help with this!  After updating my batch file, I've been able to successfully download the historical trade that I missed.  Is there any particular address you'd like me to send some donations to (I assume the one in your signature is fine)?  Also I would like to further commend you, as this sierrachartfeed update is a significant improvement over the older version I used, and has now removed the 10-15 minute gaps of trade data between the historical data and live streaming feed which is great

[deepceleron]

Right now I'm working through fixing the first obstacle I found... - putting all times in UTC/GMT.

Announce: Sierrachartfeed w UTC time data, improved network

Update 018:
-All times in UTC instead of local time (allows easy data file interchange with other users and comparison with other ticker sources),
  REQUIRED: remove all existing C:\SierraChart\data\*.scid files before using this new bridge version.

-properly close IP sockets (not a fix for all 502 errors, apparently)

-faster update,

-remove about 1-5MB of redundant downloading at initialization,

-cosmetic tweaks: error stats and download times on console.

Code:

Usage: sierrachartfeed.py [options]

Options:
  -h, --help            show this help message and exit
  -d DATADIR, --datadir=DATADIR
                        Data directory of SierraChart software
  -y, --disable-history
                        Disable downloads from bitcoincharts.com
  -p PRECISION, --volume-precision=PRECISION
                        Change decimal precision for market volume.
  -s SYMBOLS, --symbols=SYMBOLS
                        Charts to watch, comma separated. Use * for streaming
                        all markets.
  -l HISTORY, --history=HISTORY
                        Number of days of history to retrieve (default = 7).

note: when using the single-letter option, there is no space between the option and the parameter


Download links:
(see later post for newest version)

SCID full history files:
from first trade 2010-07-17 23:09:17, to 2013-05-26 23:59:59 (UTC, tick accurate, precision 2):
(extract to C:\SierraChart\data\ before starting SierraChart)

http://we.lovebitco.in/schart/mtgoxUSD.scid.UTC.7z (18.2MB/182MB)
http://we.lovebitco.in/schart/otherALL.scid.UTC.7z (8.2MB/86.4MB)


Original Instructions:

How to start

• 1. Download and install SierraChart software (use default settings and data directory)
• 2. Download SierraChart feed for bitcoin markets.
• 3. Start feed in a console. No parameters required list of all parameters: sierrachartfeed.exe --help
• 4. Start SierraChart software
• 5. Go to File->New/Open Intraday Chart and select (for example) mtgoxUSD.scid
• 6. Customize your view using F5 (chart settings) and F6 (analysis/studies) menus
• 7. Enjoy real time charting!

Optimal settings in SierraChart:

Set price ticks correctly per exchange:
Go into Menu -> Tools -> Chart Settings. Pick "Edit Global Symbol Settings". Choose Service "SC Historical Data", and press New. Then edit on the "General" Tab:
-Symbol: mtgoxUSD
-Price Display Format: .00001
-Tick Size: 0.0000100
-Currency Value Per Tick: 0.0000100

Now in Menu -> Tools -> Chart Settings, "Price Display Format" and "Tick Size" is auto set to  ".00001"

Fix volume display until next SierraChart Launch:
Go into Menu -> Tools -> Chart Settings. Go to "Advanced Settings 2" tab and change "Volume/Open Int. Multiplier" to "0.01". Press Apply

See volume:
Menu -> Analysis -> Studies. select "Volume" and press "Add>>", and OK.

Bar analysis
Go into Menu -> Tools -> Chart Settings.
Set Bar Period to "Number Of Ticks", "Range Of Ticks", set the number, and do whatever analysis you want.

Is there any particular address you'd like me to send some donations to (I assume the one in your signature is fine)?

My signature address works! Although a diff/patch gives 67 chunks of procedural spaghetti code changes against git master, I've done far less work than original creator slush, and you might also thank bitcoincharts.com for providing the feed, and let them know why you donated.

[myself]

Go into Menu -> Tools -> Chart Settings. Go to "Advanced Settings 2" tab and change "Volume/Open Int. Multiplier" to "0.01". Press Apply

i used to have 0,0001 for 4 digit precision

[deepceleron]

Go into Menu -> Tools -> Chart Settings. Go to "Advanced Settings 2" tab and change "Volume/Open Int. Multiplier" to "0.01". Press Apply

i used to have 0,0001 for 4 digit precision

This "Volume/Open Int. Multiplier" setting refers to the amount of BTC per trade (which you see when you do a volume study), not the price. It should be set to 0.01 to match the default SierraChartFeed precision of "2".

SierraChart doesn't support fractions of a trade volume like a currency would have, it apparently was designed for stocks, where the minimum trade volume is 1 stock. The Sierrachartfeed solution to this is to take trade amounts and multiply them, adding "precision" decimal places (the option at the command line), so a trade of 50.01 BTC becomes a volume of 5001 sierrachart units. Unless you you want to analyze volume with super-accuracy, this setting is adequate. The maximum display divisor in SierraChart is 0.0001 (precision 4).


edit: the raw data does have eight decimal places of accuracy (1 satoshi), see volume amounts of these trades:

bitcoincharts csv:
1367391587,133.500000000000,0.596380140000

mtgox market api json:
"vol":
{"value":"100624.73892887","value_int":"10062473892887","display":"100,624.74\u00a0BTC","display_short":"100,624.74\u00a0BTC","currency":"BTC"}

The only way to use this many decimal points in SierraChart is to set the precision option to -p8, erase all SCID data and get data again, and deal with the volumes in SierraCharts being in billions of satoshis.

[deepceleron]

SCID full history files:
from first trade 07/17/2010 23:09:17, to 04/30/2013 23:59:59 (UTC, tick accurate, precision 2):
(extract to C:\SierraChart\data\ before starting SierraChart)

http://we.lovebitco.in/schart/mtgoxUSD.scid.UTC.7z (18.2MB/182MB)
http://we.lovebitco.in/schart/otherALL.scid.UTC.7z (8.2MB/86.4MB)

Sorry, I forgot to actually upload the "other history" file, it's up now and includes other exchanges and currencies. Funny how nobody dropped me the 411 on the 404.

[arklan]

Download links:
http://we.lovebitco.in/schart/sierrachartfeedUTC.exe (3.9MB) (windows 32 bit standalone exe)
http://we.lovebitco.in/schart/sierrachartfeedUTC.py (python 2.7 source - replace file in github source)

SCID full history files:
from first trade 07/17/2010 23:09:17, to 04/30/2013 23:59:59 (UTC, tick accurate, precision 2):
(extract to C:\SierraChart\data\ before starting SierraChart)

http://we.lovebitco.in/schart/mtgoxUSD.scid.UTC.7z (18.2MB/182MB)
http://we.lovebitco.in/schart/otherALL.scid.UTC.7z (8.2MB/86.4MB)

Sorry, I forgot to actually upload the "other history" file, it's up now and includes other exchanges and currencies. Funny how nobody dropped me the 411 on the 404.

thanks DC

[WackyWilly]

Hi DeepCeleron,

I quit using SierraCharts for a few weeks, as the old data feed was in trouble, and neither could I get one of your first versions to work properly. Just came back to this thread and I am happy to see that you have done a great job. Thanks also for the SCID uploads with full history (might it be a good idea to keep doing so, e.g. on a monthly or weekly basis?) .

All this is of great help, and I thank you for that.
THUMBS UP FOR YOU !!!  

http://mypetjawa.mu.nu/archives/vader_thumbs_up.jpg

[wamatt]

Just trying out SierraChart now. I'm inexperienced and trying to figure out how to get data before <2011/06/26 for gox?

For some reason that's as far back as it goes. I used the "Bitcoin Data (MTGOX)" feed that came with the default install. I never tried the "MtGox Bitcoin Trading (under development)" option.

Am I doing it wrong?

[deepceleron]

Just trying out SierraChart now. I'm inexperienced and trying to figure out how to get data before <2011/06/26 for gox?

For some reason that's as far back as it goes. I used the "Bitcoin Data (MTGOX)" feed that came with the default install. I never tried the "MtGox Bitcoin Trading (under development)" option.

Am I doing it wrong?

It sounds like you are using the new Mtgox data direct from SierraCharts, and not the bridge. It may not have a complete history.

Try this:
1. Go to menu -> Chart -> Chart Settings, and set "Days To Display" to 1100 or more,
2. Set Bar period to 1-0-0 (1 day per bar),
3. Scroll to the left and see how far back it goes.

If they don't have older history, you can follow the instructions for using the sierrachartfeed above, using the mtgoxUSD SCID file, which has all history trades.

[Loozik]

I have been watching this thread for a while. A few glitches were reported.

As of now, does Sierra works flawlessly with MtGox API or are there some problems?

Thanks

[S3052]

a separate question, please:
how can one avoid data gaps when not online? I am using the new bridge.

Also, how can one use the historical data and then continue with the bridge data. when I tried, it does not add the new data at all.

thanks you

[RyNinDaCleM]

a separate question, please:
how can one avoid data gaps when not online? I am using the new bridge.

Also, how can one use the historical data and then continue with the bridge data. when I tried, it does not add the new data at all.

thanks you

I had the same problem! What I did was; I downloaded the mtgoxUSD.scid from here

SCID full history files:
from first trade 07/17/2010 23:09:17, to 04/30/2013 23:59:59 (UTC, tick accurate, precision 2):
(extract to C:\SierraChart\data\ before starting SierraChart)

http://we.lovebitco.in/schart/mtgoxUSD.scid.UTC.7z (18.2MB/182MB) <--
http://we.lovebitco.in/schart/otherALL.scid.UTC.7z (8.2MB/86.4MB)

Then I made a new text file and named it scUTC.cmd. I edited this file with this

Code:

start sierrachartfeedUTC.exe --history=13

I used 13 because as of writing (12-5-2013) the last record in the mtgoxUSD.scid file is over 12 days ago. The default is 7 days, so it skips the 6 days before it creating the hole
Once you have all historical data, it does "opening streaming socket..."

[deepceleron]

a separate question, please:
how can one avoid data gaps when not online? I am using the new bridge.

Also, how can one use the historical data and then continue with the bridge data. when I tried, it does not add the new data at all.

thanks you

I had the same problem! What I did was; I downloaded the mtgoxUSD.scid from here

SCID full history files:
from first trade 07/17/2010 23:09:17, to 04/30/2013 23:59:59 (UTC, tick accurate, precision 2):
(extract to C:\SierraChart\data\ before starting SierraChart)

http://we.lovebitco.in/schart/mtgoxUSD.scid.UTC.7z (18.2MB/182MB) <--
http://we.lovebitco.in/schart/otherALL.scid.UTC.7z (8.2MB/86.4MB)

Then I made a new text file and named it scUTC.cmd. I edited this file with this

Code:

start sierrachartfeedUTC.exe --history=13

I used 13 because as of writing (12-5-2013) the last record in the mtgoxUSD.scid file is over 12 days ago. The default is 7 days, so it skips the 6 days before it creating the hole
Once you have all historical data, it does "opening streaming socket..."

I added the history command-line option, and set the default to just seven days, to discourage abusing the API server to catch up a full three year history by default. This also makes SierraChart useable for recent trade analysis in just a few minutes instead of hours or never.

When you start sierrachartfeedUTC.exe, it will resume downloading from the last timestamp, unless it's been over seven days since you last ran it (or if you are starting with my complete history file that's now 12 days old). The history option sets the maximum number of days back that will be resumed. If you have already downloaded the full SCID file or might not use bridge for over a week, you can then use the history option like above, setting it to 200 days for example, and it will still resume just where it left off when you last ran bridge (not downloading more data than needed).

I will update the history SCID download in a bit. This file is made with modified code that downloads history just to a certain checkpoint and doesn't pollute data with live stream, so it is a 100% true record of historical trades.

As of now, does Sierra works flawlessly with MtGox API or are there some problems?

This gets its data from bitcoincharts.com; mtgox API doesn't have a feature to retrieve past trades.

[xeverse]

 
Something weird is happening here.
The hist api is up and accessible in the browser.
It doesn't look like any api specs have changed also.
But the lovely bridge can't get historical data at all.
Is this user-agent filtering of some sort or something?