SierraChart feed/bridge reborn - Realtime Bitcoin charting

[deepceleron]

ANN: New version of Sierrachartfeed and direct-download history files, working again (and fixed again Feb 10 2014)

Original post with details of what this software is, read first:

SierraChart is desktop application providing a professional Trading, Real-time and Historical Charting, and Technical Analysis platform for the financial markets. It's not so intuitive as other platforms, but once you learn how it works (and remember few shortcuts), it is very strong tool for professional technical analysis. Because of lacking real trading tools for Bitcoin economy, I decided to write simple Bitcoin markets->Sierrachart bridge, which enable real time Bitcoin charting in Sierrachart.

How to start

• 1. Download and install SierraChart software (use default settings and data directory)
• 2. Download SierraChartfeed, extract zip to any program directory.
• 3. Start feed in a console. No parameters required for mtgox (list of all parameters: sierrachartfeed.exe --help)
• 4. Start SierraChart software
• 5. Go to File->New/Open Intraday Chart and select (for example) mtgoxUSD.scid
• 6. Customize your view using F5 (chart settings) and F6 (analysis/studies) menus
• 7. Enjoy real time charting!

Sierrachartfeed gets data from bitcoincharts.com for multiple exchanges. Each time they change some API "feature", it seems to break other people's software. This is the fourth fifth "rebirth" of working software I've written since slush gave up maintenance of sierrachartfeed.

Code:

Usage: scfeed.py [options]

Options:
  -h, --help            show this help message and exit
  -t, --tickers         List known/current ticker symbols and exit
  -z, --disable-live    Disable live socket, update history only
  -d DATADIR, --datadir=DATADIR
                        SierraChart data directory (default =
                        C:/SierraChart/Data/)
  -s SYMBOLS, --symbols=SYMBOLS
                        Ticker symbols to download, comma separated with no
                        spaces; 'a' for all (default =
                        -smtgoxUSD,btceUSD,bitstampUSD)
  -p PRECISION, --volume-precision=PRECISION
                        Multiply volume by this many decimal places before
                        import (default = 4)

note: when using the single-letter option, there is no space between the option and the parameter

Download links:

• SierraChart software:
  • http://www.sierrachart.com/downloads/SierraChartSetup.exe or
  • http://we.lovebitco.in/schart/SierraChartSetup.exe
• sierrachartfeed:
  • http://we.lovebitco.in/schart/scfeed.py (Python 2.7 standalone source) or
  • http://we.lovebitco.in/schart/scfeed.exe (2.41MiB) (Windows 32 bit exe, py2exe)
    md5sum: eeaabc7edb427e1c6e419c042cad6848 *scfeed.exe

For history older than five days, before first running sierrachartfeed
The normal Bitcoincharts API only has five days of data available. If you have never run sierrachartfeed for a particular exchange ticker before, it will only get the last five days. However, if you want all trade data from the beginning of time, you can download some big files first:

• Go to this web page: http://api.bitcoincharts.com/v1/csv/
• Download the individual CSV files for exchange tickers you wish to track
• Extract the gzipped files to the SierraChart data directory (C:\SierraChart\Data)
  (a utility like 7-zip can help you with this)
• When you first run sierrachartfeed and specify a ticker with CSV, data will be imported from this file first
• The CSV file will be renamed after all data from it is imported

You will also need to re-download a new CSV if it has been over five days since you last ran sierrachartfeed.


The SCID filenames are different from previous versions, to avoid using or overwriting obsolete data. New filenames are of the format mtgoxUSD-p4.scid (where p4 indicates precision 4).



Changelog
(0.7.1 - 2014-02-10)

• CSV file synchronization impossible due to the remote files now being gzipped and unstable, abandoned,
• Import user-supplied local CSV method added,
• warn of a trade gap if last local trades are older than the 5 days available through API.

(0.7.0 - 2013-12-19)

• A local CSV is mantained and updated with trade data, resumed from nightly updates
• Fixed the only-5-days-available API
• Merged libraries into single file, fixed much library fustercluck
• Higher default precision, fixed volume rounding
• About 1/2 of .py is new code.

[1715323314]

1715323314

[1715323314]

1715323314

[1715323314]

1715323314

[1715323314]

1715323314

[1715323314]

1715323314

[1715323314]

1715323314

[deepceleron]

SierraChart setup tips:

Set price ticks correctly per exchange:
Go into Menu -> Tools -> Chart Settings. Pick "Edit Global Symbol Settings". Choose Service "SC Historical Data", and press New. Then edit on the "General" Tab:
-Symbol: mtgoxUSD
-Price Display Format: .00001
-Tick Size: 0.0000100
-Currency Value Per Tick: 0.0000100

Now in Menu -> Tools -> Chart Settings, "Price Display Format" and "Tick Size" is auto set to  ".00001"

Fix volume display until next SierraChart Launch:
Go into Menu -> Tools -> Chart Settings. Go to "Advanced Settings 2" tab and change "Volume/Open Int. Multiplier" to "0.0001". Press Apply

See volume:
Menu -> Analysis -> Studies. select "Volume" and press "Add>>", set volume format to .0001, and OK.

Bar analysis
Go into Menu -> Tools -> Chart Settings.
Set Bar Period to "Number Of Ticks", "Range Of Ticks", set the number, and do whatever analysis you want.

[arklan]

wonderful. thanks!

[tulkos]

Got an exception using the historical data provided for MTGox, was working fine yesterday:

Code:

syncing C:/SierraChart/Data/mtgoxUSD.csv
Updating CSV: bytes 312690830 to 313739405 of 316504899
312690798: ,670.000000000000,3.727747830000 ::: 00
1387445072,640.549000000000,0
Traceback (most recent call last):
  File "scfeed.py", line 739, in <module>
    csvupdater(csv_filename, s)
  File "scfeed.py", line 406, in csvupdater
    raise Exception('Local and remote CSVs do not match')
Exception: Local and remote CSVs do not match

[deepceleron]

Got an exception using the historical data provided for MTGox, was working fine yesterday:

Code:

syncing C:/SierraChart/Data/mtgoxUSD.csv
Updating CSV: bytes 312690830 to 313739405 of 316504899
312690798: ,670.000000000000,3.727747830000 ::: 00
1387445072,640.549000000000,0
Traceback (most recent call last):
  File "scfeed.py", line 739, in <module>
    csvupdater(csv_filename, s)
  File "scfeed.py", line 406, in csvupdater
    raise Exception('Local and remote CSVs do not match')
Exception: Local and remote CSVs do not match

Yep, I got it also.

The updating method is based on the assumption that in the complete nightly CSV files, older trade data wouldn't be changing, there would just be new trades added to them. I, of course, don't just blindly add new data to your local CSV file, I get an overlapping range and make sure that we are appending to identical data. My checks caught the alteration.


the last 32 bytes of your local file ::: the bytes that are now being returned when re-checking that range
312690798: ,670.000000000000,3.727747830000 ::: 001387445072,640.549000000000,0


The problem here seems to be that bitcoincharts made CSVs with different old history than was previously downloaded (after several days of the nightly updates being the same and working correctly.) The files appear to have changed for other tickers too.

Sierrachartfeed will work again by deleting and letting it re-download the C:/SierraChart/Data/mtgoxUSD.csv file. On bitcoincharts, this is the only file and method to get trades that are older than five days, so if the CSV data continues to change in this way that the file download can't be "resumed", I'll need to think of different ways to synchronize this data and resume from the last time scfeed was run.

[deepceleron]

Sorry for the delay, I was letting a day elapse to examine the state of CSV and discover what they are doing at bitcoincharts. I downloaded Dec-27 CSVs, and then now that newer nightly CSVs are published, attempted to synchronize again. Again the files have changed in a way where the previous history is not byte identical with either of the previous CSV downloads. I'll do a diff/compare to see what's being altered, but it's out of my control anyway.

To clarify, there are three sources of data to be used that are combined:
-Nightly CSVs: data since the start of time to about 2-26 hours ago,
-History API: maximum of five days of history available up to now,
-Socket stream: live trades

I will have to resort to the strategy I initially conceived but thought impractical, which is to not have a local copy, but to "explore" the current remote CSV by downloading ranges of bytes until I find a point not too far before the last SCID timestamp. If they keep changing the CSV, even indexing it locally won't work precisely. If your SCID is less than five days old, we don't need to hit the CSV data, but bytes ranges of CSV is faster than API calls, and should cause less server load.

[deepceleron]

I have identified the problem. The downloadable CSVs recently generated by bitcoincharts are corrupted.

On the left is the CSVs as they were up to Dec 24, on the right, download Dec 28, out of order trades have been written into the file. There's lots more of this in the file.



In the new CSV, the very first historic trade is missing, making the whole file different. Then things start going really wrong about 90% in, where at Wed, 11 Dec 2013 16:14:58 GMT, we start having chunks of trades from 1388094380- (Thu, 26 Dec 2013 21:46:20 GMT and on) randomly inserted into the file. As seen above at 11 Dec 2013 08:53:45, we have some 26 Dec 2013 16:42:24 trades inserted.

Now what? Will I need to anticipate new trades being written to the beginning of the file?? If you are 6 days behind, I have to get a year of data just to look for trades in the wrong spot?

[tcatm]

There is no guarantee on the order of the data so you'd need to sort it in some cases. Sorting can be done in O(n log n) so it shouldn't be a huge problem. I'll consider pre-sorting on the server in the future but please don't rely on it!

The reason for the unsorted output is likely that I've switched to a database backend that can run things in parallel now or maybe it just went unnoticed until now for some other reason.

[myself]

sub


what about using http://www.bittorrent.com/sync  to sync files ??

[arklan]

Any progress? My feed is not streaming the live trades, after doing a fresh download of the history.

Do I need to run with -h right now to get any data?

[deepceleron]

Any progress? My feed is not streaming the live trades, after doing a fresh download of the history.

Do I need to run with -h right now to get any data?

I'm getting mtgox live socket trades right now, so that's working; this part, strangely, has never been a problem, even though it is documented nowhere except by use in sierrachartfeed and one other 3 year old trade software. Is it magically working again for you?

It looks like bitcoincharts hasn't re-scrambled existing CSV data since Dec 28, so I will create a new 7zipped CSV share of my own since I can make it 10% the size. I was quite frustrated by a lot of programming and testing being flushed by another change at bitcoincharts.

The code is intolerant of web site errors - this is by design so I could see exception errors when I rewrote this. I need to make downloading more error-tolerant, as I'm now getting timeouts and errors I previously didn't, like "Exception: URLError = [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond", even for some very small requests.

[arklan]

Haven't had a chance to try it. Busy week. I'll check it when I ghetto home and let you know.

[arklan]

i just realized i hadn't followed through and updated - i don't see a way to disable the history grab, only the reverse.

i'm running the feed now to see if it will update the history and begin the live stream. it's got a ways to go though.

[deepceleron]

btc-e seems to sometimes be available on bitcoincharts, and then the ticker symbol completely disappears from existence. Since I put this as one of the default exchanges, you should manually specify an exchange or a list of exchanges with the command-line option instead. If http://bitcoincharts.com/charts/btceUSD says "page not found", that means its still not fixed.

[arklan]

so it updated the histories fine, but then when it should be converting the csv's to scid's, it error's out with "error 2: d:/sierrachart/data/bitstampusd.csv no csv exists" i'm paraphrasing, i couldn't copy and paste it for some reason, but the slashes WERE those ones. that's wrong isn't it?

any idea what's up? i miss this data feed...

oh, and per your suggestion i'm leaving btce off. just using stamp and gox.

[deepceleron]

I'll guess the error looks like this:

>scfeed.exe -d d:/scfeed/data -slocalbtcUSD
syncing d:/scfeed/data\localbtcUSD.csv
Traceback (most recent call last):
  File "scfeed.py", line 739, in <module>
    csvupdater(csv_filename, s)
  File "scfeed.py", line 380, in csvupdater
    csvfile = open(csvfilename, "wb")  # write, binary
IOError: [Errno 2] No such file or directory: 'd:/scfeed/data\\localbtcUSD.csv'

Sierrachartfeed doesn't attempt to make a directory, it will throw this error if the directory you specify or the default directory doesn't exist. You just need to create the directory. SierraChart should have already made the data directory, it is where the SCID files go; I haven't tried even running SierraChart with a non-default location (which is C:\SierraChart\Data).

If you use a directory that already exists, it works fine:

>scfeed.exe -d D:\Bitcoin\scfeed -slocalbtcUSD
syncing D:\Bitcoin\scfeed\localbtcUSD.csv
Updating CSV: bytes 0 to 1048575 of 3433481
Updating CSV: bytes 1048576 to 2097151 of 3433481
Updating CSV: bytes 2097152 to 3145727 of 3433481
Updating CSV: bytes 3145728 to 3433480 of 3433481

*** Loading data file D:\Bitcoin\scfeed\localbtcUSD-p4.scid
Downloading historical data
syncing D:\Bitcoin\scfeed\localbtcUSD.csv
local CSV already up to date
importing D:\Bitcoin\scfeed\localbtcUSD.csv
adding D:\Bitcoin\scfeed\localbtcUSD.csv to SCID

Request localbtcUSD from 2014-02-01 10:46:07
    Got 2014-02-01 10:46:07Z to 2014-02-02 21:29:19Z: 650 trades (28057 bytes in 0.806000 sec)
 Adding 2014-02-01 11:21:36Z to 2014-02-02 21:29:19Z: 649 trades to SCID
Opening live trade socket, wait........
2014-02-02 21:45:02: 21:41:26 localbtcUSD 1000.0 0.48
2014-02-02 21:45:02: 21:37:16 localbtcUSD 894.52 0.5702
2014-02-02 21:45:02: 21:41:05 localbtcUSD 983.08 0.0997
2014-02-02 21:45:02: 21:32:21 localbtcUSD 759.81 0.1998
2014-02-02 21:45:02: 21:38:58 localbtcUSD 922.0 0.2223

[arklan]

Yea, I'm running it from my d drove instead of c, so I don't have to remember to re-download or save the data when I reinstall windows or such... I'll sorry it later. Not home now.

Thanks.

[arklan]

right... so, installing ot a custom location is a bad idea... reinstalled to default, everything is fine.

[xeverse]

syncing /xe/sierra/data/mtgoxUSD.csv
URLError = Not Found
Traceback (most recent call last):
  File "/xe/sierra/scfeed.py", line 739, in <module>
    csvupdater(csv_filename, s)
  File "/xe/sierra/scfeed.py", line 385, in csvupdater
    remotecsv = CSVrange(csvsymbol)
  File "/xe/sierra/scfeed.py", line 303, in __init__
    self.getsize(self.symbol)
  File "/xe/sierra/scfeed.py", line 327, in getsize
    raise Exception('URLError = ' + str(err.reason))
Exception: URLError = Not Found

[K3n]

I did as described in the top posts: downloaded program and the sierrachartfeed using the second link under subpoint sierrachartfeed.
The next step would be to start the sierrachartfeed. Here I am getting confused as there are 3 different files, when I exract the downoaded .zip file. If I start the application scfeed.exe the console opens up and closes immediatly to fast for me to read the text in there. then Microsoft Excel document pops u in the same folder, with the name mtgoxUSD but seems to be emtpy.

Can't do step 5 as the Data directory is empty.

And what am I to do with the posted script?

Any help is appreciated.