Bitcoin Forum
December 05, 2016, 04:38:33 AM *
News: To be able to use the next phase of the beta forum software, please ensure that your email address is correct/functional.
 
   Home   Help Search Donate Login Register  
Pages: [1]
  Print  
Author Topic: Bitcoin market price monitor  (Read 8505 times)
martin
Full Member
***
Offline Offline

Activity: 150



View Profile WWW
February 10, 2011, 02:10:51 PM
 #1

I threw this little market monitor together a while ago, mostly for fun and as an exercise in learning a bit more about google appengine. I guess it might come in handy to someone. This application has been monitoring prices on mtgox and bitcoin market for the last few months, at 1 hour intervals. I recently updated it to check every minute, it has all of this data stored and available to download, you can view a graph of the last 10000 data points here (ignore the weird web address, it was just an address i happened to have spare on appengine from an old project):

http://nosoperor-internal.appspot.com/market-history

If anyone wants to download the raw data, you can pull the market data in JSON format here:

http://nosoperor-internal.appspot.com/market_data

By default, this will return only the last 10000 data points, if you set the option fetchall=true then it will fetch every single data point ever collected.

So, it's got some basic functionality at the moment. It would be cool to add some more stuff, feel free to make suggestions of things which would be useful and I'll probably implement them Smiley

1480912713
Hero Member
*
Offline Offline

Posts: 1480912713

View Profile Personal Message (Offline)

Ignore
1480912713
Reply with quote  #2

1480912713
Report to moderator
1480912713
Hero Member
*
Offline Offline

Posts: 1480912713

View Profile Personal Message (Offline)

Ignore
1480912713
Reply with quote  #2

1480912713
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1480912713
Hero Member
*
Offline Offline

Posts: 1480912713

View Profile Personal Message (Offline)

Ignore
1480912713
Reply with quote  #2

1480912713
Report to moderator
boombox
Newbie
*
Offline Offline

Activity: 13


View Profile
February 11, 2011, 08:48:22 PM
 #2

Hehehe, is the graph Javascript generating bitcoins for you? It stalled my browser totally.

Ok, some feedback on the graph: the timestamps would be better to have in a readable format.
I haven't yet seen a nice graph on any site of the bitcoin price, if I were you I would use:
http://www.highcharts.com/
or http://raphaeljs.com/
or http://code.google.com/p/flot/
And I would probably have different data sets for different timeperiods: For a 24 hour graph I would have perhaps 15 min between points and for a week graph I would have 2 hours between the points in order to make the graph quick loading.

And the JSON: I don't think you should return that many data points as default, or even if a request asks for it.
Multiple requests for getting that much data would be better.

Good work.
kiba
Legendary
*
Offline Offline

Activity: 980


View Profile
February 11, 2011, 08:51:20 PM
 #3

I had to kill my page.

martin
Full Member
***
Offline Offline

Activity: 150



View Profile WWW
February 13, 2011, 12:23:14 AM
 #4

Uh, yeah, I should probably have mentioned that I haven't put any kind of limitson the data displayed on the graph. It tends to kill things if you try to select a large data set. That's next on my list to do!

@boombox
The graph you see is generated by Raphael js already, it's just not very pretty because it's a little overcrowded at the moment. I intend to use different data sets for different time periods, with different densities of points.

As for the raw JSON, I guess I could return the last day of data points, offset by a count for how many days into the past you wish to fetch the data.

martin
Full Member
***
Offline Offline

Activity: 150



View Profile WWW
April 20, 2011, 06:40:57 PM
 #5

I've added some new query capabilities to the JSON data side of this:

Raw data:
http://nosoperor-internal.appspot.com/market_data

Fetch N datapoints. This fetches the first N datapoints ordered by date descending
fetch=N

Set start date:
start=Y-M-D

Set end date:
end=Y-M-D

filter by market ("mtgox", "bitcoin market" or "britcoin", I can add more if people wish)
market=M

eg: latest hour of mtgox data:
http://nosoperor-internal.appspot.com/market_data?fetch=60&market=mtgox

eg: all data this year:
http://nosoperor-internal.appspot.com/market_data?start=2011-1-1

As ever, requests welcome. And donations even more so Wink

ryepdx
Hero Member
*****
Offline Offline

Activity: 714



View Profile WWW
April 22, 2011, 10:54:11 PM
 #6

I've added some new query capabilities to the JSON data side of this...

I get an error on the last two of those links.  Undecided
martin
Full Member
***
Offline Offline

Activity: 150



View Profile WWW
April 24, 2011, 02:20:04 AM
 #7

That's odd, what error do you get for this link:

http://nosoperor-internal.appspot.com/market_data?start=2011-1-1

It works perfectly for me, and I get a screen of JSON spam.

As for the other link:

http://nosoperor-internal.appspot.com/market_data?fetch=60&market=mtgox

I had forgotten to deploy the correct kind of index, sorry. This link should now work too.

ben-abuya
Sr. Member
****
Offline Offline

Activity: 323



View Profile WWW
April 28, 2011, 08:44:51 PM
 #8

Hey Martin,

I tried accessing http://nosoperor-internal.appspot.com/market_data?fetch_all=true, but I only got data from 17 April. Do you have earlier data available?

Thanks!

http://lamassubtc.com/
Lamassu Bitcoin Ventures
martin
Full Member
***
Offline Offline

Activity: 150



View Profile WWW
May 18, 2011, 04:54:20 PM
 #9

Hey Ben, sorry for the slow response. It's been a busy month!

"fetch_all" isn't an option the site understands, instead you should use fetch=N to select a certain amount of values. N can be arbitrarily large, but if it's too big you'll get an error instead of useful data (because appengine has a hard limit on the time a page can take to load).

To load all data I suggest you fetch 1000 or so points, and fetch data in batches. Use end=Y-M-D, set that to the last value you got in the last batch, and fetch another 1000 and so on. Like this you can walk over the complete data set.

Announcing a change to data collected

previously, the site has collected the mtgox (and other sites) ticker data, this is nice but not terribly detailed. I recently switched the service over to gathering the complete mtgox depth table (currently no other exchanges). Currently this data can't be downloaded, I'll be working on a way to retrieve a history of the depth table sometime soon, it's a bit of a headache trying to find a reasonable way to publish such a massive dataset. If anyone is particularly interested in this data throw me a PM and I'll see what I can do about hurrying up development a little.

The current dataset is likely to stay up indefinitely, in its current form, but no more data will be added to it. Once an API is online for the depth data, there will be an overlap of the two sets, so there will be no gap in the data.

flipperbw
Newbie
*
Offline Offline

Activity: 2


View Profile
May 19, 2011, 01:05:10 PM
 #10

Sick dude.

Any chance I could get a look at the source code? What language are you using to pull in the data from mtgox?
martin
Full Member
***
Offline Offline

Activity: 150



View Profile WWW
May 20, 2011, 12:51:53 AM
 #11

Appengine only offers Python, java and go. So I'm using python, I can release the source if you really like, it's nothing special though - I'm no python expert.

This new change has exposed something of an issue in the new system: in the last 5 days I have consumed 50% of my datastore, where I consumed 25% in the previous 6 months - the depth table is a *lot* more information than the ticker. I'm going to have to store the data more compactly, and every time the datastore starts getting full pack it up into a prerendered json document and store it somewhere else.

nb. I'll try to get around to writing a data dump cron job tomorrow, but if I don't get around to it there may be a day or two break in the data (which of course no one would know since I haven't got around to writing an API yet, I'll try to get round to that tomorrow too)

flipperbw
Newbie
*
Offline Offline

Activity: 2


View Profile
May 20, 2011, 03:33:36 AM
 #12

please release - I'd just like to see how you did it. right now I really only use perl, but trying to migrate to python.
martin
Full Member
***
Offline Offline

Activity: 150



View Profile WWW
May 20, 2011, 11:04:19 AM
 #13

Here you go:

http://martindevans.appspot.com/file/Bitcoin_Market_Monitor.rar?agxtYXJ0aW5kZXZhbnNyEAsSCEJsb2JGaWxlGPW_JAw

A quick overview:
django
Contains django templates used for pages, currently just the one template for the market history graph (which is broken, and I don't intend to fix it anytime soon)

javascript
Contains the scripts needed for the history graph

Stylesheets
CSS files

__init.py__
Indicates that this folder is a python module which can be imported

cron.yaml
Defines to appengine the cron jobs I want to run

index.yaml
Describes to appengine the indices to build from my datastore

main.py
Nothing important, I should have deleted this really. It's just a dummy handler (literally "hello world") I used when setting everything up.

market_data.py
This is the interesting one, it contains the cron job which collects data, the handler which returns data when you access /market_data and the handler which returns the history graph which is broken at market-history.

trading.py
I initially hoped to build some functionality to automatically trade coins between markets as the prices moved. I haven't really got around to that yet, so this just contains a few data models, and a handler which just prints "foo" when trading should occur.

If you have any specific questions feel free to ask Smiley

Pages: [1]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!