visualizing the market depth

[giszmo]

Hi

I've been wondering if the exchange rate follows the market depth or if the market depth follows to the exchange rate. Actually looking at the market depth I often get the feeling to know how the rate has to change in a very near future and often it does but by far not always.

To analyze further I searched for sources of historic market depth but couldn't find any, so I wrote a small script that draws the MtGox market depth - the one that is available via their api - the one that is only a small fraction of the total market depth - into a png. As the market depth sometimes is 5 times bigger than at other times, I normalized each line by the highest value and used a pallet of 1024 values (4 times a black blue gradient).
The result for some days but with a longer pause at some point and without any time stamps is here:


Now I wonder if anybody has a more complete historic market depth. I would like to render a Bitcoin Market Depth Canyon fly through or whatever works better than the above.

[Revalin]

Nice!  It's the same idea with a little different approach that I've tried a bunch of times:



I have per-minute snapshots of the data since July.  There are a bunch of gaps in the data so you have to be careful or you'll get time jumps, and it's just the public API so it's just a narrow window around the current price.

If that's still good enough to be interesting to you, you're welcome to a copy of what I have.  The only price is that I want to see more awesome pictures, especially around interesting events.  

[giszmo]

Nice!  It's the same idea with a little different approach that I've tried a bunch of times:



I have per-minute snapshots of the data since July.  There are a bunch of gaps in the data so you have to be careful or you'll get time jumps, and it's just the public API so it's just a narrow window around the current price.

If that's still good enough to be interesting to you, you're welcome to a copy of what I have.  The only price is that I want to see more awesome pictures, especially around interesting events.  

That sounds like a great offer! It's a huge data set. Did you ever zip it? Do you have all the raw data like timestamp+values?
I don't as my normalization already destroyed most of the data but with all the data I'd like to try and render a fly through that canyon with maybe some landmarks for events in bitcoins history.

I guess there is quite some challenge to get the data into some rendering software. A height field of about 300k samples of a price range of 1$ to 20$ at a 1ct. granularity and a 16bit precision for each accumulated value. Most rendering tools allow to import height fields as gray scale pictures. This one would be 300k x 2k x 2B = 1.2GB. Guess that is challenging also as the 2B/pixel is non-standard but necessary imho.

[Revalin]

It's raw dumps of the JSON API (complete with occasional error messages and empty files) with PST timestamps in the filenames.  It's about 5G raw, 33M compressed.  Be careful extracting it: some filesystems get really slow if you put a quarter million files in one dir.

http://www.mediafire.com/file/u1goocu36ng65kp/mtgox.7z

[ineededausername]

It's raw dumps of the JSON API (complete with occasional error messages and empty files) with PST timestamps in the filenames.  It's about 5G raw, 33M compressed.  Be careful extracting it: some filesystems get really slow if you put a quarter million files in one dir.

http://www.mediafire.com/file/u1goocu36ng65kp/mtgox.7z

That's a huge compression ratio...

[Revalin]

It's highly redundant text.  Most depths don't change from minute to minute, and there's lots of predictable syntax in JSON.  That and 7Zip is awesome.

[giszmo]

It's raw dumps of the JSON API (complete with occasional error messages and empty files) with PST timestamps in the filenames.  It's about 5G raw, 33M compressed.  Be careful extracting it: some filesystems get really slow if you put a quarter million files in one dir.

http://www.mediafire.com/file/u1goocu36ng65kp/mtgox.7z

Revalin thanx a lot! I can't promise to bring results soon but I will see what I can do. At least it is a big incentive to come up with something to have such nice data at hands

[Revalin]

No problem!  Do let me know if you (or anyone!) find another source of this data.  I'd love to fill in the holes and go farther back in time.