Bitcoin Forum

Economy => Trading Discussion => Topic started by: deepceleron on December 19, 2013, 08:13:58 PM



Title: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: deepceleron on December 19, 2013, 08:13:58 PM
ANN: New version of Sierrachartfeed and direct-download history files, working again (and fixed again Feb 10 2014)

Original post with details of what this software is, read first:

SierraChart (http://www.sierrachart.com) is desktop application providing a professional Trading, Real-time and Historical Charting, and Technical Analysis platform for the financial markets. It's not so intuitive as other platforms, but once you learn how it works (and remember few shortcuts), it is very strong tool for professional technical analysis. Because of lacking real trading tools for Bitcoin economy, I decided to write simple Bitcoin markets->Sierrachart bridge, which enable real time Bitcoin charting in Sierrachart.

How to start
  • 1. Download and install (http://www.sierrachart.com/downloads/SierraChartSetup.exe) SierraChart software (use default settings and data directory)
  • 2. Download SierraChartfeed, extract zip to any program directory.
  • 3. Start feed in a console. No parameters required for mtgox (list of all parameters: sierrachartfeed.exe --help)
  • 4. Start SierraChart software
  • 5. Go to File->New/Open Intraday Chart and select (for example) mtgoxUSD.scid
  • 6. Customize your view using F5 (chart settings) and F6 (analysis/studies) menus
  • 7. Enjoy real time charting!
http://we.lovebitco.in/schart/sc-ss.gif


Sierrachartfeed gets data from bitcoincharts.com for multiple exchanges. Each time they change some API "feature", it seems to break other people's software. This is the fourth fifth "rebirth" of working software I've written since slush gave up maintenance of sierrachartfeed.

Code:
Usage: scfeed.py [options]

Options:
  -h, --help            show this help message and exit
  -t, --tickers         List known/current ticker symbols and exit
  -z, --disable-live    Disable live socket, update history only
  -d DATADIR, --datadir=DATADIR
                        SierraChart data directory (default =
                        C:/SierraChart/Data/)
  -s SYMBOLS, --symbols=SYMBOLS
                        Ticker symbols to download, comma separated with no
                        spaces; 'a' for all (default =
                        -smtgoxUSD,btceUSD,bitstampUSD)
  -p PRECISION, --volume-precision=PRECISION
                        Multiply volume by this many decimal places before
                        import (default = 4)
note: when using the single-letter option, there is no space between the option and the parameter

Download links:

  • SierraChart software:
    • http://www.sierrachart.com/downloads/SierraChartSetup.exe or
    • http://we.lovebitco.in/schart/SierraChartSetup.exe
  • sierrachartfeed:
    • http://we.lovebitco.in/schart/scfeed.py (Python 2.7 standalone source) or
    • http://we.lovebitco.in/schart/scfeed.exe (2.41MiB) (Windows 32 bit exe, py2exe)
      md5sum: eeaabc7edb427e1c6e419c042cad6848 *scfeed.exe

For history older than five days, before first running sierrachartfeed
The normal Bitcoincharts API only has five days of data available. If you have never run sierrachartfeed for a particular exchange ticker before, it will only get the last five days. However, if you want all trade data from the beginning of time, you can download some big files first:

  • Go to this web page: http://api.bitcoincharts.com/v1/csv/
  • Download the individual CSV files for exchange tickers you wish to track
  • Extract the gzipped files to the SierraChart data directory (C:\SierraChart\Data)
    (a utility like 7-zip (http://www.7-zip.org/) can help you with this)
  • When you first run sierrachartfeed and specify a ticker with CSV, data will be imported from this file first
  • The CSV file will be renamed after all data from it is imported

You will also need to re-download a new CSV if it has been over five days since you last ran sierrachartfeed.


The SCID filenames are different from previous versions, to avoid using or overwriting obsolete data. New filenames are of the format mtgoxUSD-p4.scid (where p4 indicates precision 4).



Changelog
(0.7.1 - 2014-02-10)
  • CSV file synchronization impossible due to the remote files now being gzipped and unstable, abandoned,
  • Import user-supplied local CSV method added,
  • warn of a trade gap if last local trades are older than the 5 days available through API.

(0.7.0 - 2013-12-19)
  • A local CSV is mantained and updated with trade data, resumed from nightly updates
  • Fixed the only-5-days-available API
  • Merged libraries into single file, fixed much library fustercluck
  • Higher default precision, fixed volume rounding
  • About 1/2 of .py is new code.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: deepceleron on December 19, 2013, 08:14:20 PM
SierraChart setup tips:

Set price ticks correctly per exchange:
Go into Menu -> Tools -> Chart Settings. Pick "Edit Global Symbol Settings". Choose Service "SC Historical Data", and press New. Then edit on the "General" Tab:
-Symbol: mtgoxUSD
-Price Display Format: .00001
-Tick Size: 0.0000100
-Currency Value Per Tick: 0.0000100

Now in Menu -> Tools -> Chart Settings, "Price Display Format" and "Tick Size" is auto set to  ".00001"

Fix volume display until next SierraChart Launch:
Go into Menu -> Tools -> Chart Settings. Go to "Advanced Settings 2" tab and change "Volume/Open Int. Multiplier" to "0.0001". Press Apply

See volume:
Menu -> Analysis -> Studies. select "Volume" and press "Add>>", set volume format to .0001, and OK.

Bar analysis
Go into Menu -> Tools -> Chart Settings.
Set Bar Period to "Number Of Ticks", "Range Of Ticks", set the number, and do whatever analysis you want.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: arklan on December 20, 2013, 05:01:29 AM
wonderful. thanks!


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: tulkos on December 26, 2013, 11:46:58 AM
Got an exception using the historical data provided for MTGox, was working fine yesterday:

Code:
syncing C:/SierraChart/Data/mtgoxUSD.csv
Updating CSV: bytes 312690830 to 313739405 of 316504899
312690798: ,670.000000000000,3.727747830000 ::: 00
1387445072,640.549000000000,0
Traceback (most recent call last):
  File "scfeed.py", line 739, in <module>
    csvupdater(csv_filename, s)
  File "scfeed.py", line 406, in csvupdater
    raise Exception('Local and remote CSVs do not match')
Exception: Local and remote CSVs do not match



Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: deepceleron on December 27, 2013, 09:19:37 AM
Got an exception using the historical data provided for MTGox, was working fine yesterday:

Code:
syncing C:/SierraChart/Data/mtgoxUSD.csv
Updating CSV: bytes 312690830 to 313739405 of 316504899
312690798: ,670.000000000000,3.727747830000 ::: 00
1387445072,640.549000000000,0
Traceback (most recent call last):
  File "scfeed.py", line 739, in <module>
    csvupdater(csv_filename, s)
  File "scfeed.py", line 406, in csvupdater
    raise Exception('Local and remote CSVs do not match')
Exception: Local and remote CSVs do not match

Yep, I got it also.

The updating method is based on the assumption that in the complete nightly CSV files, older trade data wouldn't be changing, there would just be new trades added to them. I, of course, don't just blindly add new data to your local CSV file, I get an overlapping range and make sure that we are appending to identical data. My checks caught the alteration.


the last 32 bytes of your local file ::: the bytes that are now being returned when re-checking that range
312690798: ,670.000000000000,3.727747830000 ::: 001387445072,640.549000000000,0


The problem here seems to be that bitcoincharts made CSVs with different old history than was previously downloaded (after several days of the nightly updates being the same and working correctly.) The files appear to have changed for other tickers too.

Sierrachartfeed will work again by deleting and letting it re-download the C:/SierraChart/Data/mtgoxUSD.csv file. On bitcoincharts, this is the only file and method to get trades that are older than five days, so if the CSV data continues to change in this way that the file download can't be "resumed", I'll need to think of different ways to synchronize this data and resume from the last time scfeed was run.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: deepceleron on December 29, 2013, 09:15:52 PM
Sorry for the delay, I was letting a day elapse to examine the state of CSV and discover what they are doing at bitcoincharts. I downloaded Dec-27 CSVs, and then now that newer nightly CSVs are published, attempted to synchronize again. Again the files have changed in a way where the previous history is not byte identical with either of the previous CSV downloads. I'll do a diff/compare to see what's being altered, but it's out of my control anyway.

To clarify, there are three sources of data to be used that are combined:
-Nightly CSVs: data since the start of time to about 2-26 hours ago,
-History API: maximum of five days of history available up to now,
-Socket stream: live trades

I will have to resort to the strategy I initially conceived but thought impractical, which is to not have a local copy, but to "explore" the current remote CSV by downloading ranges of bytes until I find a point not too far before the last SCID timestamp. If they keep changing the CSV, even indexing it locally won't work precisely. If your SCID is less than five days old, we don't need to hit the CSV data, but bytes ranges of CSV is faster than API calls, and should cause less server load.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: deepceleron on December 31, 2013, 10:24:41 AM
I have identified the problem. The downloadable CSVs recently generated by bitcoincharts are corrupted.

On the left is the CSVs as they were up to Dec 24, on the right, download Dec 28, out of order trades have been written into the file. There's lots more of this in the file.

http://we.lovebitco.in/img/corruptcsv.png

In the new CSV, the very first historic trade is missing, making the whole file different. Then things start going really wrong about 90% in, where at Wed, 11 Dec 2013 16:14:58 GMT, we start having chunks of trades from 1388094380- (Thu, 26 Dec 2013 21:46:20 GMT and on) randomly inserted into the file. As seen above at 11 Dec 2013 08:53:45, we have some 26 Dec 2013 16:42:24 trades inserted.

Now what? Will I need to anticipate new trades being written to the beginning of the file?? If you are 6 days behind, I have to get a year of data just to look for trades in the wrong spot?


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: tcatm on January 02, 2014, 08:00:10 AM
There is no guarantee on the order of the data so you'd need to sort it in some cases. Sorting can be done in O(n log n) so it shouldn't be a huge problem. I'll consider pre-sorting on the server in the future but please don't rely on it!

The reason for the unsorted output is likely that I've switched to a database backend that can run things in parallel now or maybe it just went unnoticed until now for some other reason.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: myself on January 09, 2014, 11:32:08 PM
sub


what about using http://www.bittorrent.com/sync  to sync files ??


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: arklan on January 09, 2014, 11:52:29 PM
Any progress? My feed is not streaming the live trades, after doing a fresh download of the history.

Do I need to run with -h right now to get any data?


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: deepceleron on January 10, 2014, 11:12:53 AM
Any progress? My feed is not streaming the live trades, after doing a fresh download of the history.

Do I need to run with -h right now to get any data?


I'm getting mtgox live socket trades right now, so that's working; this part, strangely, has never been a problem, even though it is documented nowhere except by use in sierrachartfeed and one other 3 year old trade software. Is it magically working again for you?

It looks like bitcoincharts hasn't re-scrambled existing CSV data since Dec 28, so I will create a new 7zipped CSV share of my own since I can make it 10% the size. I was quite frustrated by a lot of programming and testing being flushed by another change at bitcoincharts.

The code is intolerant of web site errors - this is by design so I could see exception errors when I rewrote this. I need to make downloading more error-tolerant, as I'm now getting timeouts and errors I previously didn't, like "Exception: URLError = [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond", even for some very small requests.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: arklan on January 10, 2014, 06:35:29 PM
Haven't had a chance to try it. Busy week. I'll check it when I ghetto home and let you know.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: arklan on February 02, 2014, 06:47:01 AM
i just realized i hadn't followed through and updated - i don't see a way to disable the history grab, only the reverse.

i'm running the feed now to see if it will update the history and begin the live stream. it's got a ways to go though.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: deepceleron on February 02, 2014, 12:56:02 PM
btc-e seems to sometimes be available on bitcoincharts, and then the ticker symbol completely disappears from existence. Since I put this as one of the default exchanges, you should manually specify an exchange or a list of exchanges with the command-line option instead. If http://bitcoincharts.com/charts/btceUSD says "page not found", that means its still not fixed.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: arklan on February 02, 2014, 06:04:03 PM
so it updated the histories fine, but then when it should be converting the csv's to scid's, it error's out with "error 2: d:/sierrachart/data/bitstampusd.csv no csv exists" i'm paraphrasing, i couldn't copy and paste it for some reason, but the slashes WERE those ones. that's wrong isn't it?

any idea what's up? i miss this data feed...

oh, and per your suggestion i'm leaving btce off. just using stamp and gox.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: deepceleron on February 02, 2014, 09:47:32 PM
I'll guess the error looks like this:

>scfeed.exe -d d:/scfeed/data -slocalbtcUSD
syncing d:/scfeed/data\localbtcUSD.csv
Traceback (most recent call last):
  File "scfeed.py", line 739, in <module>
    csvupdater(csv_filename, s)
  File "scfeed.py", line 380, in csvupdater
    csvfile = open(csvfilename, "wb")  # write, binary
IOError: [Errno 2] No such file or directory: 'd:/scfeed/data\\localbtcUSD.csv'


Sierrachartfeed doesn't attempt to make a directory, it will throw this error if the directory you specify or the default directory doesn't exist. You just need to create the directory. SierraChart should have already made the data directory, it is where the SCID files go; I haven't tried even running SierraChart with a non-default location (which is C:\SierraChart\Data).

If you use a directory that already exists, it works fine:

>scfeed.exe -d D:\Bitcoin\scfeed -slocalbtcUSD
syncing D:\Bitcoin\scfeed\localbtcUSD.csv
Updating CSV: bytes 0 to 1048575 of 3433481
Updating CSV: bytes 1048576 to 2097151 of 3433481
Updating CSV: bytes 2097152 to 3145727 of 3433481
Updating CSV: bytes 3145728 to 3433480 of 3433481

*** Loading data file D:\Bitcoin\scfeed\localbtcUSD-p4.scid
Downloading historical data
syncing D:\Bitcoin\scfeed\localbtcUSD.csv
local CSV already up to date
importing D:\Bitcoin\scfeed\localbtcUSD.csv
adding D:\Bitcoin\scfeed\localbtcUSD.csv to SCID

Request localbtcUSD from 2014-02-01 10:46:07
    Got 2014-02-01 10:46:07Z to 2014-02-02 21:29:19Z: 650 trades (28057 bytes in 0.806000 sec)
 Adding 2014-02-01 11:21:36Z to 2014-02-02 21:29:19Z: 649 trades to SCID
Opening live trade socket, wait........
2014-02-02 21:45:02: 21:41:26 localbtcUSD 1000.0 0.48
2014-02-02 21:45:02: 21:37:16 localbtcUSD 894.52 0.5702
2014-02-02 21:45:02: 21:41:05 localbtcUSD 983.08 0.0997
2014-02-02 21:45:02: 21:32:21 localbtcUSD 759.81 0.1998
2014-02-02 21:45:02: 21:38:58 localbtcUSD 922.0 0.2223


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: arklan on February 02, 2014, 10:25:06 PM
Yea, I'm running it from my d drove instead of c, so I don't have to remember to re-download or save the data when I reinstall windows or such... I'll sorry it later. Not home now.

Thanks.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: arklan on February 04, 2014, 06:48:01 PM
right... so, installing ot a custom location is a bad idea... reinstalled to default, everything is fine.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: xeverse on February 09, 2014, 11:31:14 AM
syncing /xe/sierra/data/mtgoxUSD.csv
URLError = Not Found
Traceback (most recent call last):
  File "/xe/sierra/scfeed.py", line 739, in <module>
    csvupdater(csv_filename, s)
  File "/xe/sierra/scfeed.py", line 385, in csvupdater
    remotecsv = CSVrange(csvsymbol)
  File "/xe/sierra/scfeed.py", line 303, in __init__
    self.getsize(self.symbol)
  File "/xe/sierra/scfeed.py", line 327, in getsize
    raise Exception('URLError = ' + str(err.reason))
Exception: URLError = Not Found


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: K3n on February 09, 2014, 11:46:12 AM
I did as described in the top posts: downloaded program and the sierrachartfeed using the second link under subpoint sierrachartfeed.
The next step would be to start the sierrachartfeed. Here I am getting confused as there are 3 different files, when I exract the downoaded .zip file. If I start the application scfeed.exe the console opens up and closes immediatly to fast for me to read the text in there. then Microsoft Excel document pops u in the same folder, with the name mtgoxUSD but seems to be emtpy.

Can't do step 5 as the Data directory is empty.

And what am I to do with the posted script?

Any help is appreciated.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: deepceleron on February 09, 2014, 02:57:44 PM
Once again major unannounced API changes at bitcoincharts have broken software.

Sierrachartfeed gets CSV data from this URL (this is the only source for trades older than five days):

http://api.bitcoincharts.com/v1/csv/

As documented here (http://bitcoincharts.com/about/markets-api/), it was individual nightly CSV files containing all historical trades for each exchange, about 300MB files for the top exchanges. Sierrachartfeed maintains a local copy of the CSV and "resumes", adding new data to your existing CSV file.

Now the remote data files are zipped. That means that any previous CSV data that was downloaded is useless. The old file names have changed, the files are now, for example, "mtgoxUSD.csv.gz". This file cannot be resumed, as it is compressed data wrapped in a header and footer. I would need to write gzip-aware software that can strip out the gzipped stream from the current file and rebuild it with added data, and of course unzip to import the new trades. And that's if bitcoincharts doesn't screw up the data contents again - one bit in one early trade changing will change the entire gzipped archive.

I don't know if the proper response is to say fuck it and not use data from that site any more, or to show that regardless of their contempt for users and software developers, we can make it work.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: arklan on February 09, 2014, 06:33:25 PM
I was wondering what was up. Sigh. Good luck DC. If I could help in anyway I would...


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: xeverse on February 10, 2014, 08:57:59 PM
Is it possible to disable history and just have the live socket stream for now?


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: deepceleron on February 11, 2014, 05:13:45 AM
Okay, it's fixed again, discarding about 100 lines of new code I had written to auto-resume using CSV data. First post updated.

You can now manually download and extract the complete zipped history CSVs to the SierraChart data directory, but won't be prompted to do so if it is a new install or if you specify a new ticker symbol you previously hadn't downloaded. It will simply give you five days of history. You also won't have to get CSVs as long as you run sierrachartfeed and have been keeping your local ticker data updated.

If you have already run sierrachartfeed to download ticker data and are more than five days behind, the software can't automatically catch up. You will get a dire warning from sierrachartfeed that you need to re-download these CSVs manually:


> scfeed.exe -s mtgoxUSD

*** Loading data file C:/SierraChart/Data/mtgoxUSD-p4.scid
importing C:/SierraChart/Data/mtgoxUSD.csv
 - No CSV history file found for mtgoxUSD, skipping
 - (Download manually from http://api.bitcoincharts.com/v1/csv/)
 - (unzip to C:/SierraChart/Data/mtgoxUSD.csv)

Request mtgoxUSD from 2014-01-10 11:49:29

*** Danger! ***

 Your saved mtgoxUSD history is 640 hours older
 than that available from history website

*** Continuing download will create an unrecoverable gap in trade data ***
Press [c] and [enter] if you want to continue:

- For trades older than 5 days, manually download a new CSV file from
  http://api.bitcoincharts.com/v1/csv/


You can proceed by typing the letter "C" and pressing enter; in my example above, I just pressed enter, so the program quit and gave info.

Happy charting.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: xeverse on February 12, 2014, 04:14:56 AM

Thank You!


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: arklan on February 13, 2014, 06:12:38 PM
How often are the csvs updated? I'm away from my charting / mining rig.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: deepceleron on February 14, 2014, 01:54:14 AM
How often are the csvs updated? I'm away from my charting / mining rig.
nightly


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: xeverse on February 15, 2014, 09:59:22 AM
*** Loading data file /xe/_4mkt/2data/bitstampUSD-p4.scid
importing /xe/_4mkt/2data/bitstampUSD.csv
 - No CSV history file found for bitstampUSD, skipping
 - (Download manually from http://api.bitcoincharts.com/v1/csv/)
 - (unzip to /xe/_4mkt/2data/bitstampUSD.csv)

Request bitstampUSD from 2014-02-15 09:47:20
    Got 2014-02-15 09:47:20Z to 2014-02-15 09:47:20Z: 1 trades (43 bytes in 0.208534 sec)

--------------------------------------------------------------------------------------

There is a bunch of these
cvs.imported files instead in the dir.

Why they aren't recognized then?


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: arklan on February 15, 2014, 04:40:35 PM
*** Loading data file /xe/_4mkt/2data/bitstampUSD-p4.scid
importing /xe/_4mkt/2data/bitstampUSD.csv
 - No CSV history file found for bitstampUSD, skipping
 - (Download manually from http://api.bitcoincharts.com/v1/csv/)
 - (unzip to /xe/_4mkt/2data/bitstampUSD.csv)

Request bitstampUSD from 2014-02-15 09:47:20
    Got 2014-02-15 09:47:20Z to 2014-02-15 09:47:20Z: 1 trades (43 bytes in 0.208534 sec)

--------------------------------------------------------------------------------------

There is a bunch of these
cvs.imported files instead in the dir.

Why they aren't recognized then?

That looks like the message DC was just taking about four csv files that are more then five days out of date.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: xeverse on February 16, 2014, 05:14:29 AM
That looks like the message DC was just taking about four csv files that are more then five days out of date.
Yes, but only these renamed files are like 10min out of sync max.
You don't experience csv.renamed files?
https://i.imgur.com/OClQvXs.png


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: deepceleron on February 16, 2014, 10:26:25 AM
*** Loading data file /xe/_4mkt/2data/bitstampUSD-p4.scid
importing /xe/_4mkt/2data/bitstampUSD.csv
 - No CSV history file found for bitstampUSD, skipping
 - (Download manually from http://api.bitcoincharts.com/v1/csv/)
 - (unzip to /xe/_4mkt/2data/bitstampUSD.csv)

Request bitstampUSD from 2014-02-15 09:47:20
    Got 2014-02-15 09:47:20Z to 2014-02-15 09:47:20Z: 1 trades (43 bytes in 0.208534 sec)

--------------------------------------------------------------------------------------

There is a bunch of these cvs.imported files instead in the dir.

Why they aren't recognized then?

If the csv files have been renamed by sierrachartfeed to ".imported", then all the data in them is already in the Sierrachart SCID file; the ".imported" files and ".csv.gz" files should be deleted after successful import (unless you have a specific reason you might use them again soon; they are obsolete after four days). You can ignore the "no CSV history file found" message - it is not an error in this case, just a normal procedural message.

You will get the big *danger* error message (like four posts above) when there is an actual problem that requires you to re-download a new CSV (because the bitcoincharts API only has five-day-old and newer data).

The csv files inside the gzips actually start with a period, like ".bitstampUSD.csv". I've done a few tricks to look for the csv either with or without a period at the start.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: arklan on February 18, 2014, 06:10:04 AM
so i've just gottne back to my charting pc, and triedthe new exe. on launch it throws "the program can't start because python27.dll is missing from your computer."

i'm assuming i could just install python to fix that, but if i can avoid doing so, i prefer to keep a tidy ship.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: deepceleron on February 18, 2014, 10:31:45 AM
so i've just gottne back to my charting pc, and triedthe new exe. on launch it throws "the program can't start because python27.dll is missing from your computer."

i'm assuming i could just install python to fix that, but if i can avoid doing so, i prefer to keep a tidy ship.
Uploaded a new exe made the same way as before with dlls included.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: arklan on February 18, 2014, 07:22:27 PM
assuming you mean the download link in the first post - i tried it, no difference.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: movellan on February 25, 2014, 04:31:05 PM
so i've just gottne back to my charting pc, and triedthe new exe. on launch it throws "the program can't start because python27.dll is missing from your computer."

i'm assuming i could just install python to fix that, but if i can avoid doing so, i prefer to keep a tidy ship.
Uploaded a new exe made the same way as before with dlls included.

Now that the mtgox feed  no longer exists, will you be adding any other exchanges to the feed? TIA.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: xeverse on March 03, 2014, 07:14:57 PM
http://www.reddit.com/r/Bitcoin/comments/1zfl0e/bitstamp_adds_websocket_api/


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: movellan on March 03, 2014, 10:40:30 PM
http://www.reddit.com/r/Bitcoin/comments/1zfl0e/bitstamp_adds_websocket_api/

Thanks to deepceleron SC already has feeds for both Bitstamp and BTC-e, which appear to be working fine. I was wondering with MtGox down (perhaps not out) for now if he was thinking about adding say, Localbitcoins or some other higher volume exchange.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: deepceleron on March 04, 2014, 09:14:58 AM
The exchange data from localbitcoins is crazy looking:

http://bitcoincharts.com/charts/chart.png?width=940&m=localbtcUSD&SubmitButton=Draw&r=5&i=5-min&c=0&s=&e=&Prev=&Next=&t=S&b=&a1=&m1=10&a2=&m2=25&x=1&i1=Vol&i2=&i3=&i4=&v=0&cv=0&ps=0&l=0&p=0&

MtGox may be interesting for historical reasons, but I think I'll remove it as a default the next time I need to make a big update.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: movellan on March 08, 2014, 08:42:40 PM

MtGox may be interesting for historical reasons, but I think I'll remove it as a default. The quick fix I put up has a bug to be fixed, in that it doesn't alert you in many situations when your sierrachart data is over five days old, and will create a trade gap.

Speaking of which, scfeed.exe doesn't seem to be working at all today. It will most often exit prematurely or when it does fully load just sits waiting for the feed.

PS - Just had another look at Bitcoincharts and it appears they haven't updated for 7-8 hrs. Is scfeed designed to auto close when no new data is found?


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: movellan on March 08, 2014, 08:52:36 PM
The exchange data from localbitcoins is crazy looking:

http://bitcoincharts.com/charts/chart.png?width=940&m=localbtcUSD&SubmitButton=Draw&r=5&i=5-min&c=0&s=&e=&Prev=&Next=&t=S&b=&a1=&m1=10&a2=&m2=25&x=1&i1=Vol&i2=&i3=&i4=&v=0&cv=0&ps=0&l=0&p=0&

MtGox may be interesting for historical reasons, but I think I'll remove it as a default. The quick fix I put up has a bug to be fixed, in that it doesn't alert you in many situations when your sierrachart data is over five days old, and will create a trade gap.

While I'm thinking about it, have you tried applying the Stochastic Crossover System and MACD-Volume Weighted studies along with straight Volume to the Localbitcoins data?  I have used these with the MTGox data and they have an uncanny way of predicting large movement as the crossover points merge, sometimes giving a 1-5 min lead to the action.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: deepceleron on March 09, 2014, 07:32:30 PM
PS - Just had another look at Bitcoincharts and it appears they haven't updated for 7-8 hrs. Is scfeed designed to auto close when no new data is found?

If you run it from a command prompt so the window doesn't close when a program exits, you will be able to see what the problem is. The program used to beat down the door waiting for good data (because there was 500MB+ to download off the API server, it had to tolerate web errors), but now it's kind of written so if it doesn't get what it needs from the 5-day history server, it just gives you the error.

Oh, and there's no trade gap bug like I previously mentioned might exist - I forgot that I previously deleted all my SierraChart/data cleaning up the hard drive and misinterpreted when I ran sierrachartfeed again.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: movellan on March 09, 2014, 09:08:04 PM
PS - Just had another look at Bitcoincharts and it appears they haven't updated for 7-8 hrs. Is scfeed designed to auto close when no new data is found?

If you run it from a command prompt so the window doesn't close when a program exits, you will be able to see what the problem is. The program used to beat down the door waiting for good data (because there was 500MB+ to download off the API server, it had to tolerate web errors), but now it's kind of written so if it doesn't get what it needs from the 5-day history server, it just gives you the error.

Oh, and there's no trade gap bug like I previously mentioned might exist - I forgot that I previously deleted all my SierraChart/data cleaning up the hard drive and misinterpreted when I ran sierrachartfeed again.

Thanks for replying. Any idea why the Bitcoincharts feed suddenly stopped and restarted? BTW, looks like the Bifinex beta has experienced a large increase in traffic since the MTGox demise. If they stay strong enough for some time, might you consider adding the feed to SC? I'd really like to see data from more than the present two major exchanges.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: numismatics on April 18, 2014, 09:01:33 PM
I hadn't realized someone had been keeping scfeed alive! Looking good!


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: Squeaker on April 27, 2014, 09:42:03 PM
Please, lets try to continue the practice of having version numbers on the scfeed.exe program...

=squeak=


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: culexevilman on April 28, 2014, 12:30:35 AM
Give this a try when I get home. 8)


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: deepceleron on April 28, 2014, 02:47:20 AM
What's the matter anyway? The upstream is dead? Is there
any development on a p2p datafeed distribution project somewhere?

It would be nice to have a problem description, as I don't regularly run this just to check the services it connects to.

I just ran sierrachartfeed, and it updates the 5-day history OK, but I haven't gotten any trades from the live socket stream in the five minutes it's been running. Since it connects and subscribes without error, the live stream is not completely down, it just looks like there is no trade data coming through the live stream socket. This will be something that bitcoincharts admin "tcatm" will have to sort out, as it's probably some internal server problem (if the service hasn't been changed or discontinued).

The version number of sierrachartfeed is in the python script. I quit changing the name of the script with a version number included in the name, so that you don't have to alter batch files that call it with your own list of tickers. The version number doesn't matter that much, as only the most recent version works - all the old versions have been broken by changes at bitcoincharts.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: Squeaker on April 28, 2014, 04:04:05 AM
You know that many of us don't use the script, but use the executable, right? And those of us that do, kind of depend on the version number to be visible in a conspicuous place. It is a very minor thing to change a couple characters in a batch file or shortcut, or to rename the file if we don't want to change the batch file itself. :)

I couldn't even find a version number listed on post #0 of the thread, except for the Python program itself.

=squeak=


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: fuzzbuzz on April 29, 2014, 08:16:18 AM
This platform works with FIX, once an Exchange supports fix protocol(atlas ats), would we be able to trade via sierrachats platform?


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: movellan on June 22, 2014, 07:08:52 AM
Hello, what has happened?
I'm getting connection refused..
Can somebody confirm?

Yes, feed has been down for a couple of days now; scfeed starts then exits.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: movellan on June 23, 2014, 05:28:51 AM
movellan, thanks.

Request bitfinexUSD from 2014-06-22 03:42:21
Traceback (most recent call last):
  File "/xe/4mkt/3feed/scfeed.py", line 758, in <module>
    z = scids[s.]
  File "/xe/4mkt/3feed/scfeed.py", line 713, in __getitem__
    handler = ScidHandler(gsymbol, self.datadir, self.volume_precision, self.verbose)
  File "/xe/4mkt/3feed/scfeed.py", line 638, in __init__
    self.download_historical()
  File "/xe/4mkt/3feed/scfeed.py", line 662, in download_historical
    self.csvfilename,  self.verbose):
  File "/xe/4mkt/3feed/scfeed.py", line 543, in bitcoincharts_history
    chunk = chunkdata.read()
  File "/usr/lib/python2.7/socket.py", line 351, in read
    data = self._sock.recv(rbufsize)
  File "/usr/lib/python2.7/httplib.py", line 543, in read
    return self._read_chunked(amt)
  File "/usr/lib/python2.7/httplib.py", line 612, in _read_chunked
    value.append(self._safe_read(chunk_left))
  File "/usr/lib/python2.7/httplib.py", line 660, in _safe_read
    raise IncompleteRead(''.join(s), amt)
httplib.IncompleteRead: IncompleteRead(3672 bytes read, 4438 more expected)


Working again. Just received BTCe and Bitstamp updates, 10k and 11k trades respectively, live trades now coming in. Go figure.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: xeverse on July 31, 2014, 02:19:09 PM
How do you get your bitfinex quotes now?


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: deepceleron on July 31, 2014, 04:42:29 PM
How do you get your bitfinex quotes now?

It looks like the bitfinexUSD ticker has no data since Jul 23: http://bitcoincharts.com/charts/bitfinexUSD

This could be a change in the bitfinex API, where bitcoincharts can't get data any more (the same data source as sierrachartfeed uses). Of course if you notice something like this, you might message user tcatm on the forum or send a support email to bitcoincharts - they may not have any idea that there is a problem if it is not reported.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: xeverse on August 01, 2014, 03:51:18 PM
you might message user tcatm on the forum or send a support email to bitcoincharts
hmm apparently he hasn't been online here since april. i sent an email.

DC, can you switch the bridge directly to the exchange
the same source which is being polled by coincharts?
https://www.bitfinex.com/pages/api
it seems they have the get method for trades

bitfinex promissed it's own websockets a couple of month ago,
i guess it isn't ready yet for some reason.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: movellan on September 03, 2014, 10:52:26 AM
SCfeed dead again? No updates since midnight.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: deepceleron on September 04, 2014, 09:11:01 PM
SCfeed dead again? No updates since midnight.

What exchange are you monitoring? I just ran the post #1 script with no command line options, and it updated 5 days of bitstampUSD and btceUSD history, and is getting live trades from both.

If a particular exchange is not getting data, it may be that the API between bitcoincharts and the exchange is broken (which is why I don't try to write a version that connects directly to individual exchanges - it would need to be constantly updated). You can look at the chart data for btce for example: http://bitcoincharts.com/charts/btceUSD#rg1zig1-minztgSzm1g10zm2g25zv - see that it is up-to-the-minute, and then compare it with the trade data from the exchange you are watching. If the other exchange has no new data, you should notify info@bitcoincharts.com that the feed is down.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: movellan on September 04, 2014, 10:37:30 PM
SCfeed dead again? No updates since midnight.

What exchange are you monitoring? I just ran the post #1 script with no command line options, and it updated 5 days of bitstampUSD and btceUSD history, and is getting live trades from both.

If a particular exchange is not getting data, it may be that the API between bitcoincharts and the exchange is broken (which is why I don't try to write a version that connects directly to individual exchanges - it would need to be constantly updated). You can look at the chart data for btce for example: http://bitcoincharts.com/charts/btceUSD#rg1zig1-minztgSzm1g10zm2g25zv - see that it is up-to-the-minute, and then compare it with the trade data from the exchange you are watching. If the other exchange has no new data, you should notify info@bitcoincharts.com that the feed is down.

Hmmmm..........Just ran it again and all is working fine. Was down for at least two days for me. Thanks for replying.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: doitagain on January 15, 2015, 08:50:53 PM
Hello deepceleron

http://savepic.su/4675368.png

This situation at the moment.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: movellan on January 15, 2015, 11:55:32 PM
Same here. If you close scfeed and restart, it will DL all the data up to current, but no new live feed. Restarted several times during the last two days to keep up.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: arklan on January 16, 2015, 07:16:14 AM
i've found cryptowatch.com to be a decent alternative to this for my admittedly simple needs.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: doitagain on January 16, 2015, 01:59:37 PM
Hello deepceleron

http://savepic.su/4675368.png

This situation at the moment.

live stream resumed


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: coyne1981 on January 17, 2015, 01:22:16 PM
how about tradingview.com?


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: movellan on July 02, 2015, 02:29:30 AM
Anyone notice that BTCe data is no longer feeding?


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: deepceleron on July 10, 2015, 03:03:48 AM
Anyone notice that BTCe data is no longer feeding?


It looks like there was a gap in data from Bitcoincharts: http://bitcoincharts.com/markets/btceUSD.html - this likely happened due to changes to the BTCe API that needed a software fix from Bitcoincharts, or actual exchange data downtime.

Sometimes they have recovered missing historical spans of data when bitcoincharts->exchange API connection goes down for a while.

I could remove mtgox as a default, but I'm sure everyone using this has custom configured their own command-line list of exchanges anyway.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: movellan on October 06, 2015, 02:14:20 PM
Thanks, DC. On another tack, could you add a Coinbase feed to SCFeed?


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: movellan on January 20, 2016, 05:08:08 PM
Thanks, DC. On another tack, could you add a Coinbase feed to SCFeed?

Anyone notice that the downloads are still working but the live trade is not?


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: deepceleron on January 23, 2016, 08:46:14 AM
Thanks, DC. On another tack, could you add a Coinbase feed to SCFeed?

Anyone notice that the downloads are still working but the live trade is not?


Looks like the live trade socket from bitcoincharts is not returning any data after subscription. A message to tcatm via the forum or a message to the bitcoincharts contact email may alert the site to a malfunction.

Frankly, I'm surprised that it has worked for four years, as the API is not documented or advertised, and is only known from the original sierrachart bridge code (and is likely the only code using the data stream).


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: movellan on March 07, 2016, 11:34:14 AM
Thanks, DC. On another tack, could you add a Coinbase feed to SCFeed?

Anyone notice that the downloads are still working but the live trade is not?


Looks like the live trade socket from bitcoincharts is not returning any data after subscription. A message to tcatm via the forum or a message to the bitcoincharts contact email may alert the site to a malfunction.

Frankly, I'm surprised that it has worked for four years, as the API is not documented or advertised, and is only known from the original sierrachart bridge code (and is likely the only code using the data stream).

Working again now. Thanks, DC.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: doitagain on March 19, 2016, 01:16:32 PM
Hello deepceleron.

I'm trying to import okkoin. File size about 9 GB. I receive a memory error.

http://savepic.net/7915549.png

I cut the file into pieces. Importing successfully, if the file is less than 1 GB.
Maybe there is a way to fix the code in small labor costs?

Thank you.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: deepceleron on March 21, 2016, 08:08:34 AM
Hello deepceleron.

I'm trying to import okkoin. File size about 9 GB. I receive a memory error.

http://savepic.net/7915549.png

I cut the file into pieces. Importing successfully, if the file is less than 1 GB.
Maybe there is a way to fix the code in small labor costs?

Thank you.
Thanks for letting me know about this. I'll revisit this code and look at different ways of accessing and importing the files. Are you running 64-bit Python? (the exe here is 32 bit) That alone may fix the issue if it is an error reading or writing beyond 2/4gb. I also had some sorting code that sanitized data that might be using big chunks of memory if it is still in play (haven't dug into this project for a while). If the csv file has 9gb of actual trade data and is not sparse, I would anticipate the sierrachart data files would also grow pretty large too.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: movellan on April 28, 2016, 08:18:21 AM
Thanks, DC. On another tack, could you add a Coinbase feed to SCFeed?

DC, asked you about this a while back. Wonder if you could have a look at it. Coinbase has a live feed available at

https://exchange.coinbase.com/trade

and API specs for historical data at

https://docs.exchange.coinbase.com/#get-historic-rates

Would really like to get Coinbase feed integrated into SCFeed now that it seems to be stable. Thanks.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: movellan on June 08, 2016, 10:30:02 AM
Thanks, DC. On another tack, could you add a Coinbase feed to SCFeed?

Anyone notice that the downloads are still working but the live trade is not?


Looks like the live trade socket from bitcoincharts is not returning any data after subscription. A message to tcatm via the forum or a message to the bitcoincharts contact email may alert the site to a malfunction.

Frankly, I'm surprised that it has worked for four years, as the API is not documented or advertised, and is only known from the original sierrachart bridge code (and is likely the only code using the data stream).

DC, different problem now. SCfeed keeps giving a 'URLError = Bad Gateway' error when trying to get MtGox data. Does this for 10 times or so and then closes without continuing on to get data from the other two sites. Any way around this or will it have to be re-configured? Thanks.

6/10/16 - Working again now. Go figure.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: movellan on June 05, 2017, 09:25:04 PM
Thanks, DC. On another tack, could you add a Coinbase feed to SCFeed?

Anyone notice that the downloads are still working but the live trade is not?


Looks like the live trade socket from bitcoincharts is not returning any data after subscription. A message to tcatm via the forum or a message to the bitcoincharts contact email may alert the site to a malfunction.

Frankly, I'm surprised that it has worked for four years, as the API is not documented or advertised, and is only known from the original sierrachart bridge code (and is likely the only code using the data stream).

Any chance you can add a Coinbase feed to SC feed? Thanks


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: deepceleron on July 02, 2017, 07:45:57 AM
Any chance you can add a Coinbase feed to SC feed? Thanks

Coinbase appears to be available as a data source from bitcoincharts in several different local currencies. You can view the history files directory here (http://api.bitcoincharts.com/v1/csv/) to look at the available ticker names (SYMBOLS) that you would use as a command line option -sSYMBOLS when starting scfeed.

Example usage: scfeed.exe -scoinbaseUSD,coinbaseEUR,btcexUSD



Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: movellan on July 02, 2017, 10:06:21 AM
Any chance you can add a Coinbase feed to SC feed? Thanks

Coinbase appears to be available as a data source from bitcoincharts in several different local currencies. You can view the history files directory here (http://api.bitcoincharts.com/v1/csv/) to look at the available ticker names (SYMBOLS) that you would use as a command line option -sSYMBOLS when starting scfeed.

Example usage: scfeed.exe -scoinbaseUSD,coinbaseEUR,btcexUSD



Thanks DC, I'll give it a try.


Title: Re: SierraChart feed/bridge reborn - Realtime Bitcoin charting
Post by: movellan on July 28, 2017, 07:47:24 AM
Any chance you can add a Coinbase feed to SC feed? Thanks

Coinbase appears to be available as a data source from bitcoincharts in several different local currencies. You can view the history files directory here (http://api.bitcoincharts.com/v1/csv/) to look at the available ticker names (SYMBOLS) that you would use as a command line option -sSYMBOLS when starting scfeed.

Example usage: scfeed.exe -scoinbaseUSD,coinbaseEUR,btcexUSD



Coinbase feed is working nicely. Thank you.

Since BTC-e went down, the Litecoin feed is no longer working. Is there a similar history files directory for LTC on other exchanges? I'd like to get the feed working again. TIA.