deepceleron (OP)
Legendary
Offline
Activity: 1512
Merit: 1036
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
December 19, 2013, 08:13:58 PM Last edit: July 02, 2017, 07:37:40 AM by deepceleron |
|
ANN: New version of Sierrachartfeed and direct-download history files, working again (and fixed again Feb 10 2014)Original post with details of what this software is, read first: SierraChart is desktop application providing a professional Trading, Real-time and Historical Charting, and Technical Analysis platform for the financial markets. It's not so intuitive as other platforms, but once you learn how it works (and remember few shortcuts), it is very strong tool for professional technical analysis. Because of lacking real trading tools for Bitcoin economy, I decided to write simple Bitcoin markets->Sierrachart bridge, which enable real time Bitcoin charting in Sierrachart. How to start- 1. Download and install SierraChart software (use default settings and data directory)
- 2. Download SierraChartfeed, extract zip to any program directory.
- 3. Start feed in a console. No parameters required for mtgox (list of all parameters: sierrachartfeed.exe --help)
- 4. Start SierraChart software
- 5. Go to File->New/Open Intraday Chart and select (for example) mtgoxUSD.scid
- 6. Customize your view using F5 (chart settings) and F6 (analysis/studies) menus
- 7. Enjoy real time charting!
![](https://ip.bitcointalk.org/?u=http%3A%2F%2Fwe.lovebitco.in%2Fschart%2Fsc-ss.gif&t=665&c=9l_L3fDUW_ieiQ) Sierrachartfeed gets data from bitcoincharts.com for multiple exchanges. Each time they change some API "feature", it seems to break other people's software. This is the fourth fifth "rebirth" of working software I've written since slush gave up maintenance of sierrachartfeed. Usage: scfeed.py [options]
Options: -h, --help show this help message and exit -t, --tickers List known/current ticker symbols and exit -z, --disable-live Disable live socket, update history only -d DATADIR, --datadir=DATADIR SierraChart data directory (default = C:/SierraChart/Data/) -s SYMBOLS, --symbols=SYMBOLS Ticker symbols to download, comma separated with no spaces; 'a' for all (default = -smtgoxUSD,btceUSD,bitstampUSD) -p PRECISION, --volume-precision=PRECISION Multiply volume by this many decimal places before import (default = 4)
note: when using the single-letter option, there is no space between the option and the parameter Download links: - SierraChart software:
- sierrachartfeed:
For history older than five days, before first running sierrachartfeedThe normal Bitcoincharts API only has five days of data available. If you have never run sierrachartfeed for a particular exchange ticker before, it will only get the last five days. However, if you want all trade data from the beginning of time, you can download some big files first: - Go to this web page: http://api.bitcoincharts.com/v1/csv/
- Download the individual CSV files for exchange tickers you wish to track
- Extract the gzipped files to the SierraChart data directory (C:\SierraChart\Data)
(a utility like 7-zip can help you with this) - When you first run sierrachartfeed and specify a ticker with CSV, data will be imported from this file first
- The CSV file will be renamed after all data from it is imported
You will also need to re-download a new CSV if it has been over five days since you last ran sierrachartfeed. The SCID filenames are different from previous versions, to avoid using or overwriting obsolete data. New filenames are of the format mtgoxUSD-p4.scid (where p4 indicates precision 4). Changelog (0.7.1 - 2014-02-10) - CSV file synchronization impossible due to the remote files now being gzipped and unstable, abandoned,
- Import user-supplied local CSV method added,
- warn of a trade gap if last local trades are older than the 5 days available through API.
(0.7.0 - 2013-12-19) - A local CSV is mantained and updated with trade data, resumed from nightly updates
- Fixed the only-5-days-available API
- Merged libraries into single file, fixed much library fustercluck
- Higher default precision, fixed volume rounding
- About 1/2 of .py is new code.
|
|
|
|
deepceleron (OP)
Legendary
Offline
Activity: 1512
Merit: 1036
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
December 19, 2013, 08:14:20 PM Last edit: September 15, 2014, 12:16:48 PM by deepceleron |
|
SierraChart setup tips:
Set price ticks correctly per exchange: Go into Menu -> Tools -> Chart Settings. Pick "Edit Global Symbol Settings". Choose Service "SC Historical Data", and press New. Then edit on the "General" Tab: -Symbol: mtgoxUSD -Price Display Format: .00001 -Tick Size: 0.0000100 -Currency Value Per Tick: 0.0000100
Now in Menu -> Tools -> Chart Settings, "Price Display Format" and "Tick Size" is auto set to ".00001"
Fix volume display until next SierraChart Launch: Go into Menu -> Tools -> Chart Settings. Go to "Advanced Settings 2" tab and change "Volume/Open Int. Multiplier" to "0.0001". Press Apply
See volume: Menu -> Analysis -> Studies. select "Volume" and press "Add>>", set volume format to .0001, and OK.
Bar analysis Go into Menu -> Tools -> Chart Settings. Set Bar Period to "Number Of Ticks", "Range Of Ticks", set the number, and do whatever analysis you want.
|
|
|
|
arklan
Legendary
Offline
Activity: 1778
Merit: 1008
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
December 20, 2013, 05:01:29 AM |
|
wonderful. thanks!
|
i don't post much, but this space for rent.
|
|
|
tulkos
Member
![*](https://bitcointalk.org/Themes/custom1/images/star.gif)
Offline
Activity: 74
Merit: 10
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
December 26, 2013, 11:46:58 AM |
|
Got an exception using the historical data provided for MTGox, was working fine yesterday: syncing C:/SierraChart/Data/mtgoxUSD.csv Updating CSV: bytes 312690830 to 313739405 of 316504899 312690798: ,670.000000000000,3.727747830000 ::: 00 1387445072,640.549000000000,0 Traceback (most recent call last): File "scfeed.py", line 739, in <module> csvupdater(csv_filename, s) File "scfeed.py", line 406, in csvupdater raise Exception('Local and remote CSVs do not match') Exception: Local and remote CSVs do not match
|
|
|
|
deepceleron (OP)
Legendary
Offline
Activity: 1512
Merit: 1036
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
December 27, 2013, 09:19:37 AM |
|
Got an exception using the historical data provided for MTGox, was working fine yesterday: syncing C:/SierraChart/Data/mtgoxUSD.csv Updating CSV: bytes 312690830 to 313739405 of 316504899 312690798: ,670.000000000000,3.727747830000 ::: 00 1387445072,640.549000000000,0 Traceback (most recent call last): File "scfeed.py", line 739, in <module> csvupdater(csv_filename, s) File "scfeed.py", line 406, in csvupdater raise Exception('Local and remote CSVs do not match') Exception: Local and remote CSVs do not match Yep, I got it also. The updating method is based on the assumption that in the complete nightly CSV files, older trade data wouldn't be changing, there would just be new trades added to them. I, of course, don't just blindly add new data to your local CSV file, I get an overlapping range and make sure that we are appending to identical data. My checks caught the alteration. the last 32 bytes of your local file ::: the bytes that are now being returned when re-checking that range312690798: ,670.000000000000,3.727747830000 ::: 001387445072,640.549000000000,0
The problem here seems to be that bitcoincharts made CSVs with different old history than was previously downloaded (after several days of the nightly updates being the same and working correctly.) The files appear to have changed for other tickers too. Sierrachartfeed will work again by deleting and letting it re-download the C:/SierraChart/Data/mtgoxUSD.csv file. On bitcoincharts, this is the only file and method to get trades that are older than five days, so if the CSV data continues to change in this way that the file download can't be "resumed", I'll need to think of different ways to synchronize this data and resume from the last time scfeed was run.
|
|
|
|
deepceleron (OP)
Legendary
Offline
Activity: 1512
Merit: 1036
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
December 29, 2013, 09:15:52 PM |
|
Sorry for the delay, I was letting a day elapse to examine the state of CSV and discover what they are doing at bitcoincharts. I downloaded Dec-27 CSVs, and then now that newer nightly CSVs are published, attempted to synchronize again. Again the files have changed in a way where the previous history is not byte identical with either of the previous CSV downloads. I'll do a diff/compare to see what's being altered, but it's out of my control anyway.
To clarify, there are three sources of data to be used that are combined: -Nightly CSVs: data since the start of time to about 2-26 hours ago, -History API: maximum of five days of history available up to now, -Socket stream: live trades
I will have to resort to the strategy I initially conceived but thought impractical, which is to not have a local copy, but to "explore" the current remote CSV by downloading ranges of bytes until I find a point not too far before the last SCID timestamp. If they keep changing the CSV, even indexing it locally won't work precisely. If your SCID is less than five days old, we don't need to hit the CSV data, but bytes ranges of CSV is faster than API calls, and should cause less server load.
|
|
|
|
deepceleron (OP)
Legendary
Offline
Activity: 1512
Merit: 1036
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
December 31, 2013, 10:24:41 AM Last edit: December 31, 2013, 09:25:48 PM by deepceleron |
|
I have identified the problem. The downloadable CSVs recently generated by bitcoincharts are corrupted. On the left is the CSVs as they were up to Dec 24, on the right, download Dec 28, out of order trades have been written into the file. There's lots more of this in the file. ![](https://ip.bitcointalk.org/?u=http%3A%2F%2Fwe.lovebitco.in%2Fimg%2Fcorruptcsv.png&t=665&c=h5b34GwSHIoMJA) In the new CSV, the very first historic trade is missing, making the whole file different. Then things start going really wrong about 90% in, where at Wed, 11 Dec 2013 16:14:58 GMT, we start having chunks of trades from 1388094380- (Thu, 26 Dec 2013 21:46:20 GMT and on) randomly inserted into the file. As seen above at 11 Dec 2013 08:53:45, we have some 26 Dec 2013 16:42:24 trades inserted. Now what? Will I need to anticipate new trades being written to the beginning of the file?? If you are 6 days behind, I have to get a year of data just to look for trades in the wrong spot?
|
|
|
|
tcatm
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
January 02, 2014, 08:00:10 AM |
|
There is no guarantee on the order of the data so you'd need to sort it in some cases. Sorting can be done in O(n log n) so it shouldn't be a huge problem. I'll consider pre-sorting on the server in the future but please don't rely on it!
The reason for the unsorted output is likely that I've switched to a database backend that can run things in parallel now or maybe it just went unnoticed until now for some other reason.
|
|
|
|
myself
Legendary
Offline
Activity: 938
Merit: 1000
chaos is fun...…damental :)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
January 09, 2014, 11:32:08 PM Last edit: January 09, 2014, 11:54:27 PM by myself |
|
sub what about using http://www.bittorrent.com/sync to sync files ??
|
Los desesperados publican que lo inventó el rey que rabió, porque todo son en el rabias y mas rabias, disgustos y mas disgustos, pezares y mas pezares; si el que compra algunas partidas vé que baxan, rabia de haver comprado; si suben, rabia de que no compró mas; si compra, suben, vende, gana y buelan aun á mas alto precio del que ha vendido; rabia de que vendió por menor precio: si no compra ni vende y ván subiendo, rabia de que haviendo tenido impulsos de comprar, no llegó á lograr los impulsos; si van baxando, rabia de que, haviendo tenido amagos de vender, no se resolvió á gozar los amagos; si le dan algun consejo y acierta, rabia de que no se lo dieron antes; si yerra, rabia de que se lo dieron; con que todo son inquietudes, todo arrepentimientos, tododelirios, luchando siempre lo insufrible con lo feliz, lo indomito con lo tranquilo y lo rabioso con lo deleytable.
|
|
|
arklan
Legendary
Offline
Activity: 1778
Merit: 1008
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
January 09, 2014, 11:52:29 PM |
|
Any progress? My feed is not streaming the live trades, after doing a fresh download of the history.
Do I need to run with -h right now to get any data?
|
i don't post much, but this space for rent.
|
|
|
deepceleron (OP)
Legendary
Offline
Activity: 1512
Merit: 1036
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
January 10, 2014, 11:12:53 AM |
|
Any progress? My feed is not streaming the live trades, after doing a fresh download of the history.
Do I need to run with -h right now to get any data?
I'm getting mtgox live socket trades right now, so that's working; this part, strangely, has never been a problem, even though it is documented nowhere except by use in sierrachartfeed and one other 3 year old trade software. Is it magically working again for you? It looks like bitcoincharts hasn't re-scrambled existing CSV data since Dec 28, so I will create a new 7zipped CSV share of my own since I can make it 10% the size. I was quite frustrated by a lot of programming and testing being flushed by another change at bitcoincharts. The code is intolerant of web site errors - this is by design so I could see exception errors when I rewrote this. I need to make downloading more error-tolerant, as I'm now getting timeouts and errors I previously didn't, like "Exception: URLError = [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond", even for some very small requests.
|
|
|
|
arklan
Legendary
Offline
Activity: 1778
Merit: 1008
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
January 10, 2014, 06:35:29 PM |
|
Haven't had a chance to try it. Busy week. I'll check it when I ghetto home and let you know.
|
i don't post much, but this space for rent.
|
|
|
arklan
Legendary
Offline
Activity: 1778
Merit: 1008
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
February 02, 2014, 06:47:01 AM |
|
i just realized i hadn't followed through and updated - i don't see a way to disable the history grab, only the reverse.
i'm running the feed now to see if it will update the history and begin the live stream. it's got a ways to go though.
|
i don't post much, but this space for rent.
|
|
|
deepceleron (OP)
Legendary
Offline
Activity: 1512
Merit: 1036
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
February 02, 2014, 12:56:02 PM |
|
btc-e seems to sometimes be available on bitcoincharts, and then the ticker symbol completely disappears from existence. Since I put this as one of the default exchanges, you should manually specify an exchange or a list of exchanges with the command-line option instead. If http://bitcoincharts.com/charts/btceUSD says "page not found", that means its still not fixed.
|
|
|
|
arklan
Legendary
Offline
Activity: 1778
Merit: 1008
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
February 02, 2014, 06:04:03 PM |
|
so it updated the histories fine, but then when it should be converting the csv's to scid's, it error's out with "error 2: d:/sierrachart/data/bitstampusd.csv no csv exists" i'm paraphrasing, i couldn't copy and paste it for some reason, but the slashes WERE those ones. that's wrong isn't it?
any idea what's up? i miss this data feed...
oh, and per your suggestion i'm leaving btce off. just using stamp and gox.
|
i don't post much, but this space for rent.
|
|
|
deepceleron (OP)
Legendary
Offline
Activity: 1512
Merit: 1036
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
February 02, 2014, 09:47:32 PM |
|
I'll guess the error looks like this:
>scfeed.exe -d d:/scfeed/data -slocalbtcUSD syncing d:/scfeed/data\localbtcUSD.csv Traceback (most recent call last): File "scfeed.py", line 739, in <module> csvupdater(csv_filename, s) File "scfeed.py", line 380, in csvupdater csvfile = open(csvfilename, "wb") # write, binary IOError: [Errno 2] No such file or directory: 'd:/scfeed/data\\localbtcUSD.csv'
Sierrachartfeed doesn't attempt to make a directory, it will throw this error if the directory you specify or the default directory doesn't exist. You just need to create the directory. SierraChart should have already made the data directory, it is where the SCID files go; I haven't tried even running SierraChart with a non-default location (which is C:\SierraChart\Data).
If you use a directory that already exists, it works fine:
>scfeed.exe -d D:\Bitcoin\scfeed -slocalbtcUSD syncing D:\Bitcoin\scfeed\localbtcUSD.csv Updating CSV: bytes 0 to 1048575 of 3433481 Updating CSV: bytes 1048576 to 2097151 of 3433481 Updating CSV: bytes 2097152 to 3145727 of 3433481 Updating CSV: bytes 3145728 to 3433480 of 3433481
*** Loading data file D:\Bitcoin\scfeed\localbtcUSD-p4.scid Downloading historical data syncing D:\Bitcoin\scfeed\localbtcUSD.csv local CSV already up to date importing D:\Bitcoin\scfeed\localbtcUSD.csv adding D:\Bitcoin\scfeed\localbtcUSD.csv to SCID
Request localbtcUSD from 2014-02-01 10:46:07 Got 2014-02-01 10:46:07Z to 2014-02-02 21:29:19Z: 650 trades (28057 bytes in 0.806000 sec) Adding 2014-02-01 11:21:36Z to 2014-02-02 21:29:19Z: 649 trades to SCID Opening live trade socket, wait........ 2014-02-02 21:45:02: 21:41:26 localbtcUSD 1000.0 0.48 2014-02-02 21:45:02: 21:37:16 localbtcUSD 894.52 0.5702 2014-02-02 21:45:02: 21:41:05 localbtcUSD 983.08 0.0997 2014-02-02 21:45:02: 21:32:21 localbtcUSD 759.81 0.1998 2014-02-02 21:45:02: 21:38:58 localbtcUSD 922.0 0.2223
|
|
|
|
arklan
Legendary
Offline
Activity: 1778
Merit: 1008
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
February 02, 2014, 10:25:06 PM |
|
Yea, I'm running it from my d drove instead of c, so I don't have to remember to re-download or save the data when I reinstall windows or such... I'll sorry it later. Not home now.
Thanks.
|
i don't post much, but this space for rent.
|
|
|
arklan
Legendary
Offline
Activity: 1778
Merit: 1008
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
February 04, 2014, 06:48:01 PM |
|
right... so, installing ot a custom location is a bad idea... reinstalled to default, everything is fine.
|
i don't post much, but this space for rent.
|
|
|
xeverse
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
February 09, 2014, 11:31:14 AM |
|
syncing /xe/sierra/data/mtgoxUSD.csv URLError = Not Found Traceback (most recent call last): File "/xe/sierra/scfeed.py", line 739, in <module> csvupdater(csv_filename, s) File "/xe/sierra/scfeed.py", line 385, in csvupdater remotecsv = CSVrange(csvsymbol) File "/xe/sierra/scfeed.py", line 303, in __init__ self.getsize(self.symbol) File "/xe/sierra/scfeed.py", line 327, in getsize raise Exception('URLError = ' + str(err.reason)) Exception: URLError = Not Found
|
|
|
|
K3n
Newbie
Offline
Activity: 1
Merit: 0
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
February 09, 2014, 11:46:12 AM |
|
I did as described in the top posts: downloaded program and the sierrachartfeed using the second link under subpoint sierrachartfeed. The next step would be to start the sierrachartfeed. Here I am getting confused as there are 3 different files, when I exract the downoaded .zip file. If I start the application scfeed.exe the console opens up and closes immediatly to fast for me to read the text in there. then Microsoft Excel document pops u in the same folder, with the name mtgoxUSD but seems to be emtpy.
Can't do step 5 as the Data directory is empty.
And what am I to do with the posted script?
Any help is appreciated.
|
|
|
|
|