martin (OP)
|
|
February 10, 2011, 02:10:51 PM |
|
I threw this little market monitor together a while ago, mostly for fun and as an exercise in learning a bit more about google appengine. I guess it might come in handy to someone. This application has been monitoring prices on mtgox and bitcoin market for the last few months, at 1 hour intervals. I recently updated it to check every minute, it has all of this data stored and available to download, you can view a graph of the last 10000 data points here (ignore the weird web address, it was just an address i happened to have spare on appengine from an old project): http://nosoperor-internal.appspot.com/market-historyIf anyone wants to download the raw data, you can pull the market data in JSON format here: http://nosoperor-internal.appspot.com/market_dataBy default, this will return only the last 10000 data points, if you set the option fetchall=true then it will fetch every single data point ever collected. So, it's got some basic functionality at the moment. It would be cool to add some more stuff, feel free to make suggestions of things which would be useful and I'll probably implement them
|
|
|
|
boombox
Newbie
Offline
Activity: 13
Merit: 0
|
|
February 11, 2011, 08:48:22 PM |
|
Hehehe, is the graph Javascript generating bitcoins for you? It stalled my browser totally. Ok, some feedback on the graph: the timestamps would be better to have in a readable format. I haven't yet seen a nice graph on any site of the bitcoin price, if I were you I would use: http://www.highcharts.com/or http://raphaeljs.com/or http://code.google.com/p/flot/And I would probably have different data sets for different timeperiods: For a 24 hour graph I would have perhaps 15 min between points and for a week graph I would have 2 hours between the points in order to make the graph quick loading. And the JSON: I don't think you should return that many data points as default, or even if a request asks for it. Multiple requests for getting that much data would be better. Good work.
|
|
|
|
kiba
Legendary
Offline
Activity: 980
Merit: 1020
|
|
February 11, 2011, 08:51:20 PM |
|
I had to kill my page.
|
|
|
|
martin (OP)
|
|
February 13, 2011, 12:23:14 AM |
|
Uh, yeah, I should probably have mentioned that I haven't put any kind of limitson the data displayed on the graph. It tends to kill things if you try to select a large data set. That's next on my list to do!
@boombox The graph you see is generated by Raphael js already, it's just not very pretty because it's a little overcrowded at the moment. I intend to use different data sets for different time periods, with different densities of points.
As for the raw JSON, I guess I could return the last day of data points, offset by a count for how many days into the past you wish to fetch the data.
|
|
|
|
|
ryepdx
|
|
April 22, 2011, 10:54:11 PM |
|
I've added some new query capabilities to the JSON data side of this...
I get an error on the last two of those links.
|
|
|
|
|
|
martin (OP)
|
|
May 18, 2011, 04:54:20 PM |
|
Hey Ben, sorry for the slow response. It's been a busy month!
"fetch_all" isn't an option the site understands, instead you should use fetch=N to select a certain amount of values. N can be arbitrarily large, but if it's too big you'll get an error instead of useful data (because appengine has a hard limit on the time a page can take to load).
To load all data I suggest you fetch 1000 or so points, and fetch data in batches. Use end=Y-M-D, set that to the last value you got in the last batch, and fetch another 1000 and so on. Like this you can walk over the complete data set.
Announcing a change to data collected
previously, the site has collected the mtgox (and other sites) ticker data, this is nice but not terribly detailed. I recently switched the service over to gathering the complete mtgox depth table (currently no other exchanges). Currently this data can't be downloaded, I'll be working on a way to retrieve a history of the depth table sometime soon, it's a bit of a headache trying to find a reasonable way to publish such a massive dataset. If anyone is particularly interested in this data throw me a PM and I'll see what I can do about hurrying up development a little.
The current dataset is likely to stay up indefinitely, in its current form, but no more data will be added to it. Once an API is online for the depth data, there will be an overlap of the two sets, so there will be no gap in the data.
|
|
|
|
flipperbw
Newbie
Offline
Activity: 2
Merit: 0
|
|
May 19, 2011, 01:05:10 PM |
|
Sick dude.
Any chance I could get a look at the source code? What language are you using to pull in the data from mtgox?
|
|
|
|
martin (OP)
|
|
May 20, 2011, 12:51:53 AM |
|
Appengine only offers Python, java and go. So I'm using python, I can release the source if you really like, it's nothing special though - I'm no python expert.
This new change has exposed something of an issue in the new system: in the last 5 days I have consumed 50% of my datastore, where I consumed 25% in the previous 6 months - the depth table is a *lot* more information than the ticker. I'm going to have to store the data more compactly, and every time the datastore starts getting full pack it up into a prerendered json document and store it somewhere else.
nb. I'll try to get around to writing a data dump cron job tomorrow, but if I don't get around to it there may be a day or two break in the data (which of course no one would know since I haven't got around to writing an API yet, I'll try to get round to that tomorrow too)
|
|
|
|
flipperbw
Newbie
Offline
Activity: 2
Merit: 0
|
|
May 20, 2011, 03:33:36 AM |
|
please release - I'd just like to see how you did it. right now I really only use perl, but trying to migrate to python.
|
|
|
|
martin (OP)
|
|
May 20, 2011, 11:04:19 AM |
|
Here you go: http://martindevans.appspot.com/file/Bitcoin_Market_Monitor.rar?agxtYXJ0aW5kZXZhbnNyEAsSCEJsb2JGaWxlGPW_JAwA quick overview: djangoContains django templates used for pages, currently just the one template for the market history graph (which is broken, and I don't intend to fix it anytime soon) javascriptContains the scripts needed for the history graph StylesheetsCSS files __init.py__Indicates that this folder is a python module which can be imported cron.yamlDefines to appengine the cron jobs I want to run index.yamlDescribes to appengine the indices to build from my datastore main.pyNothing important, I should have deleted this really. It's just a dummy handler (literally "hello world") I used when setting everything up. market_data.pyThis is the interesting one, it contains the cron job which collects data, the handler which returns data when you access /market_data and the handler which returns the history graph which is broken at market-history. trading.pyI initially hoped to build some functionality to automatically trade coins between markets as the prices moved. I haven't really got around to that yet, so this just contains a few data models, and a handler which just prints "foo" when trading should occur. If you have any specific questions feel free to ask
|
|
|
|
|