SierraChart bridge - Realtime Bitcoin charts [v0.5] (MtGox, Intersango, ...)

[arklan]

ah, got it. all working now though.

[Mashuri]

Every attempt I make to download the whole history, it often gets through a month or two but always ends up crapping out with a 502 error.  This is in Win 7 Ultimate 64.  Tried running as admin, increasing the buffer and running in XP SP3 compatibility mode all to no avail.

I'm just going to leave this here:

http://we.lovebitco.in/schart/sierrachartfeed.exe (windows 32 bit standalone exe)

or

http://we.lovebitco.in/schart/sierrachartfeed.py (python 2.7 source, replace old file)

Yes, it's fixed in a big way.


NOTE: rename or delete your old C:\SierraChart\data\*.scid files if you've been using the broken sierrachartfeed version, or want older history than you previously retrieved.

Code:

Usage: sierrachartfeed.py [options]

Options:
  -h, --help            show this help message and exit
  -d DATADIR, --datadir=DATADIR
                        Data directory of SierraChart software
  -y, --disable-history
                        Disable downloads from bitcoincharts.com
  -p PRECISION, --volume-precision=PRECISION
                        Change decimal precision for market volume.
  -s SYMBOLS, --symbols=SYMBOLS
                        Charts to watch, comma separated. Use * for streaming
                        all markets.
  -l HISTORY, --history=HISTORY
                        Number of days of history to retrieve (default=30).

I set the default "symbol" to only be mtgoxUSD. The symbols are the file names for the exchange you can see here: http://api.bitcoincharts.com/v1/csv/ - for multiple exchanges, use the command-line option like:
sierrachartfeed -smtgoxUSD,bitfloorUSD,btceUSD

This is my own code, you can examine how I solved the problem of reading from old to new without an external database to sort and with 0 chance of missing or duplicate trades being downloaded (although it does inevitably waste some bandwidth).

Update 1: smarter download size adaptation, days of history option, actual source comments
Update 2: fixed duplicate trade bug I created in update 1

[deepceleron]

Every attempt I make to download the whole history, it often gets through a month or two but always ends up crapping out with a 502 error.  This is in Win 7 Ultimate 64.  Tried running as admin, increasing the buffer and running in XP SP3 compatibility mode all to no avail.

I can't make bitcoincharts not go down. http://www.tomshardware.com/forum/237521-49-what-gateway-error
Is there a particular exchange/currency that it is always getting or date you never get past when you see this error?

The history download will continue where it left off when the bridge program is restarted - the program SHOULD stop if it gets website errors, otherwise it will skip history downloading and get the live trade stream. After the live ticker data is added to the database with the current date and time, you will never get data older than that.

[PuertoLibre]

Are there any Fibonacci plugins for Sierra charts? (Beyond the Fibonacci fan)

[xeverse]

Are there any Fibonacci plugins for Sierra charts? (Beyond the Fibonacci fan)

Beyond? Like what? I've been thinking about phi spiral recently... Fibo fan is great fun, sometimes..

[jones]

I wonder if there is any way to get Sierra Chart to display Level II data like: Orderbook and Sales

For daytrading it is quite important to receive the most actual tick data possible. I was comparing the chart display on MTGOX USD to the chart @ http://bitcoin.clarkmoody.com/ and I had the impression it was lagging behind that website. I wonder which feed is more up2date: the one from bitcoincharts or the one SierraChart is feeding?

[xeverse]

I wonder if there is any way to get Sierra Chart to display Level II data like: Orderbook and Sales

For daytrading it is quite important to receive the most actual tick data possible. I was comparing the chart display on MTGOX USD to the chart @ http://bitcoin.clarkmoody.com/ and I had the impression it was lagging behind that website. I wonder which feed is more up2date: the one from bitcoincharts or the one SierraChart is feeding?

If they implement trading i think there will be L2.
http://anonym.to/http://www.sierrachart.com/SupportBoard.php?ThreadID=361

Very essential. Yes a bit lagging indeed.. This bridge takes bitcoinchart data.
I've tried sierra's feed. It seemed even more lively to me than bitcoinity or perhaps clark.
No idea where do they get their data. But it's more "realtime'y" than the bridge for sure..

[deepceleron]

I wonder if there is any way to get Sierra Chart to display Level II data like: Orderbook and Sales

For daytrading it is quite important to receive the most actual tick data possible. I was comparing the chart display on MTGOX USD to the chart @ http://bitcoin.clarkmoody.com/ and I had the impression it was lagging behind that website. I wonder which feed is more up2date: the one from bitcoincharts or the one SierraChart is feeding?

I just logged trade times vs actual UTC time I received them on my computer (I have sub-1-second accuracy using NTP). Here are the results:

Code:

rec date	rec time	timestamp	delay
4/26/2013	13:49:21	13:49:18	3.6
4/26/2013	13:49:51	13:49:21	30.5
4/26/2013	13:49:51	13:49:24	27.6
4/26/2013	13:49:51	13:49:24	27.6
4/26/2013	13:49:51	13:49:29	22.6
4/26/2013	13:49:51	13:49:32	19.6
4/26/2013	13:49:51	13:49:35	16.6
4/26/2013	13:49:51	13:49:39	12.6
4/26/2013	13:49:51	13:49:42	9.6
4/26/2013	13:49:51	13:49:42	9.6
4/26/2013	13:49:51	13:49:45	6.6
4/26/2013	13:49:51	13:49:45	6.6
4/26/2013	13:50:21	13:50:02	19.7
4/26/2013	13:50:22	13:50:17	5.1
4/26/2013	13:50:22	13:50:18	4.1
4/26/2013	13:50:52	13:50:22	30.6
4/26/2013	13:50:52	13:50:22	30.6
4/26/2013	13:50:52	13:50:31	21.6
4/26/2013	13:50:52	13:50:33	19.6
4/26/2013	13:50:52	13:50:36	16.6
4/26/2013	13:50:52	13:50:39	13.6
4/26/2013	13:50:52	13:50:44	8.6
4/26/2013	13:50:52	13:50:48	4.6
4/26/2013	13:51:22	13:51:00	22.6
4/26/2013	13:51:22	13:51:03	19.9
4/26/2013	13:51:22	13:51:11	11.9
4/26/2013	13:51:52	13:51:24	28.7
4/26/2013	13:51:52	13:51:29	23.8
4/26/2013	13:51:52	13:51:31	21.8
4/26/2013	13:51:52	13:51:33	19.8
4/26/2013	13:51:52	13:51:36	16.8
4/26/2013	13:51:52	13:51:38	14.8
4/26/2013	13:51:52	13:51:41	11.8
4/26/2013	13:51:52	13:51:43	9.8
4/26/2013	13:51:52	13:51:45	7.8
4/26/2013	13:51:52	13:51:45	7.8
4/26/2013	13:51:52	13:51:45	7.8
4/26/2013	13:51:52	13:51:45	7.8
4/26/2013	13:51:52	13:51:45	7.8
4/26/2013	13:51:52	13:51:45	7.8
4/26/2013	13:51:52	13:51:45	7.8
4/26/2013	13:51:52	13:51:45	7.8
4/26/2013	13:51:52	13:51:45	7.8
4/26/2013	13:51:52	13:51:45	7.8
4/26/2013	13:51:52	13:51:45	7.8
4/26/2013	13:51:52	13:51:45	7.8
4/26/2013	13:51:52	13:51:45	7.8
4/26/2013	13:51:52	13:51:47	6.0
4/26/2013	13:51:52	13:51:50	3.0
4/26/2013	13:52:24	13:51:52	32.0
4/26/2013	13:52:24	13:51:57	27.1
4/26/2013	13:52:24	13:51:57	27.1
4/26/2013	13:52:24	13:52:01	23.1
4/26/2013	13:52:24	13:52:02	22.1
4/26/2013	13:52:24	13:52:04	20.1
4/26/2013	13:52:24	13:52:08	16.1
4/26/2013	13:52:24	13:52:08	16.1
4/26/2013	13:52:24	13:52:11	13.1
4/26/2013	13:52:24	13:52:11	13.1
4/26/2013	13:52:24	13:52:13	11.1
4/26/2013	13:52:24	13:52:15	9.3
4/26/2013	13:52:24	13:52:17	7.3
4/26/2013	13:52:24	13:52:20	4.3
4/26/2013	13:52:24	13:52:22	2.3
4/26/2013	13:52:24	13:52:22	2.3
4/26/2013	13:52:24	13:52:23	1.3
4/26/2013	13:52:24	13:52:23	1.3
4/26/2013	13:52:54	13:52:24	30.3
4/26/2013	13:52:54	13:52:28	26.4
4/26/2013	13:52:54	13:52:28	26.4
4/26/2013	13:52:54	13:52:28	26.4
4/26/2013	13:52:54	13:52:28	26.4
4/26/2013	13:52:54	13:52:30	24.4
4/26/2013	13:52:54	13:52:34	20.4
4/26/2013	13:52:54	13:52:37	17.4
4/26/2013	13:52:54	13:52:43	11.4
4/26/2013	13:52:54	13:52:45	9.4
4/26/2013	13:52:54	13:52:46	8.4
4/26/2013	13:52:54	13:52:50	4.6
4/26/2013	13:52:54	13:52:52	2.6
4/26/2013	13:53:24	13:52:54	30.4
4/26/2013	13:53:24	13:52:56	28.8
4/26/2013	13:53:24	13:52:56	28.8
4/26/2013	13:53:24	13:52:58	26.8
4/26/2013	13:53:24	13:53:02	22.8
4/26/2013	13:53:55	13:53:31	24.1
4/26/2013	13:53:55	13:53:32	23.1
4/26/2013	13:53:55	13:53:32	23.1
4/26/2013	13:53:55	13:53:34	21.1
4/26/2013	13:53:55	13:53:36	19.1
4/26/2013	13:53:55	13:53:44	11.1
4/26/2013	13:53:55	13:53:46	9.1
4/26/2013	13:53:55	13:53:47	8.1
4/26/2013	13:53:55	13:53:48	7.1
4/26/2013	13:54:25	13:53:58	27.5
4/26/2013	13:54:25	13:54:09	16.8
4/26/2013	13:54:25	13:54:09	16.8
4/26/2013	13:54:25	13:54:09	16.8
4/26/2013	13:54:25	13:54:13	12.8
4/26/2013	13:54:25	13:54:21	4.8
4/26/2013	13:55:26	13:54:57	29.7
4/26/2013	13:55:26	13:55:00	26.7
4/26/2013	13:55:26	13:55:14	12.7
4/26/2013	13:55:26	13:55:18	8.7
4/26/2013	13:55:26	13:55:20	6.7
4/26/2013	13:55:26	13:55:23	3.7
4/26/2013	13:55:26	13:55:23	3.7
4/26/2013	13:55:55	13:55:38	17.9
4/26/2013	13:55:56	13:55:38	18.3
4/26/2013	13:55:56	13:55:38	18.3
4/26/2013	13:55:56	13:55:40	16.3
4/26/2013	13:55:56	13:55:40	16.3
4/26/2013	13:55:56	13:55:48	8.3
4/26/2013	13:55:56	13:55:48	8.3
4/26/2013	13:55:56	13:55:50	6.3
4/26/2013	13:55:56	13:55:52	4.3

The average delay time is 15 seconds. If you use clarkmoody, it gets its trades to your browser directly from mtgox websocket; you can compare it's console and see that the bitcoincharts data is pushed in approximately ~30 second chunks (which is adequate for charting 1-minute intervals, there is no trading in bridge-connected sierrachart).

I might take a look at getting data from MtGox again, which is how SC is getting their feed (but how much do they delay?...). It was the original method, but was switched due to problems:

Version 0.4 released, fixes non-working streaming from MtGox. It is now using socket interface to bitcoincharts instead of MtGox's socket.io.

Pros - sierrachart now supports live streaming of all bitcoin charts, not just MtGox.
Cons - there's currently implemented only polling between bitcoincharts and mtgox and chart updates are delayed 5 seconds in average. But tcatm is working on socket.io support on his side, so it will gets better soon.

Enjoy! :-)

[deepceleron]

on SC data feed since the data is properly formatted and you can use tick charts for example the only problem is they dont have data since day 1

With the exe version I just uploaded, data should be tick-accurate, although you need to set chart display appropriately (apparently the SCID can't set appropriate defaults).


Permanent fix for tick (adjust appropriately for other exchanges):
Go into Menu -> Tools -> Chart Settings. Pick "Edit Global Symbol Settings". Choose Service "SC Historical Data", and press New. Then edit on the "General" Tab:
-Symbol: mtgoxUSD
-Price Display Format: .00001
-Tick Size: 0.0000100
-Currency Value Per Tick: 0.0000100

Now in Menu -> Tools -> Chart Settings, "Price Display Format" and "Tick Size" is auto set to  ".00001"

Temporary Volume Fix until next SierraChart Launch:
Go into Menu -> Tools -> Chart Settings. Go to "Advanced Settings 2" tab and change "Volume/Open Int. Multiplier" to "0.01". Press Apply

See volume:
Menu -> Analysis -> Studies. select "Volume" and press "Add>>", and OK.

Looks like:


Bar analysis
Go into Menu -> Tools -> Chart Settings.
Set Bar Period to "Number Of Ticks", "Range Of Ticks", set the number, and do whatever analysis you want.

[arklan]

oooh, useful changes. thanks for that.

[xeverse]

With the exe version I just uploaded, data should be tick-accurate

Thank you! So it seems the delay is 30s on average..

[myself]

@deepceleron nice fix

[arklan]

anyone else seeing the error message about mysql server failure using deepceleron's exe?

if so, anyone got a clue when it'll be sorted?

if not, uh... help?

[myself]

anyone else seeing the error message about mysql server failure using deepceleron's exe?

confirmed

[deepceleron]

anyone else seeing the error message about mysql server failure using deepceleron's exe?

if so, anyone got a clue when it'll be sorted?

if not, uh... help?

All of http://bitcoincharts.com is currently down (the source of Sierrachartfeed data), replaced only with "Our MySQL server started to fail this night. We're migrating to postgres now, but this will take some time." This is a first in two years. This may also create a gap in history if they don't have a separate grabber, so anybody else that has some alternate mtgox API logging, KEEP IT GOING so that bitcoincharts can resurrect any missing data!

[S3052]

since a few days, it really looks all screwed:

http://postimg.org/image/qjkddba5t/full/


1) the intraday data is displayed backwards (right to left from morning to evening and not how it should be from left to right)
2) it shows several times the same candles (i.e. 3x per day)
how can this be fixed?

[arklan]

anyone else seeing the error message about mysql server failure using deepceleron's exe?

if so, anyone got a clue when it'll be sorted?

if not, uh... help?

All of http://bitcoincharts.com is currently down (the source of Sierrachartfeed data), replaced only with "Our MySQL server started to fail this night. We're migrating to postgres now, but this will take some time." This is a first in two years. This may also create a gap in history if they don't have a separate grabber, so anybody else that has some alternate mtgox API logging, KEEP IT GOING so that bitcoincharts can resurrect any missing data!

eww...

[myself]

anyone else seeing the error message about mysql server failure using deepceleron's exe?

if so, anyone got a clue when it'll be sorted?

if not, uh... help?

All of http://bitcoincharts.com is currently down (the source of Sierrachartfeed data), replaced only with "Our MySQL server started to fail this night. We're migrating to postgres now, but this will take some time." This is a first in two years. This may also create a gap in history if they don't have a separate grabber, so anybody else that has some alternate mtgox API logging, KEEP IT GOING so that bitcoincharts can resurrect any missing data!

just take SC data feed and join it later

[deepceleron]

bitcoincharts is alive again.

Every attempt I make to download the whole history, it often gets through a month or two but always ends up crapping out with a 502 error.

These errors persist even with the rebirth of bitcoincharts. They seem completely random, with no relation to the amount of data or the amount of time since the last request. I can wait five seconds between requests that are 1/20 the size of maximum and they still randomly happen. Of course the bitcoincharts API overview does say "Don't query more often than once every 15 minutes!", but that would make getting any amount of history beyond a single request take forever.

Therefore I have tried to make code that figures out what bitcoincharts wants and only gives up if it has way too many 502 errors. It slows down and requests less data upon errors to reduce hammering the server:

Link to post with download: https://bitcointalk.org/index.php?topic=6019.msg1929339#msg1929339

[zhalox]

For some reason, this "fix" fails to download anything past the last week or so of trade data.  Is there any way to download data farther back?  My mtgoxUSD.scid file is missing historical trade data from April 19, 20, 21, and 22