Bitcoin Forum

.

Match them up by timestamp, and then do your volume binning.

You just made me google 'imputation'  :D

I think the approach taken would have a lot to do with what you're trying to do with the data.

Separate overlaid graphs serve some purposes (such as showing historical arbitrage trends) - combined results serve others (such as gross volume trends).  Hell, there'd be value in overlaying individual market graphs with the combined values too.  

Point being - I would think as long as your raw data from various markets are stored discretely, you can build up whatever logical data combinations you wish as a layer stacked on top, with views (mostly graphs here) representing whatever concepts you care to.

It's tough to be more specific without knowing the specific approach you're using to gather and interpret your data, or what kinds of outputs you're trying to create.  You draw some pretty strong, solid (read: valuable) conclusions from the data you analyze. If you think I could be of more help with some more data, feel free to elaborate here or shoot me a PM.  I don't think I'm as math-heavy as you, and typically do most number-crunching via C# or Excel.

I know R pretty well, if you need any assistance.

http://bitcoincharts.com/about/markets-api/

If truncated, will it not be hard to include realtime data streams?

Oh, duh.  Interesting site.

A weighted (by volume) average of the prices from the major exchanges - Gox, Bitstamp, BTC China, btc-e, even EUR/GBP exchanges and localbitcoins - would be nice.

You'll want to use the exciting science of reverse imputation.  This complex mathematical technique uses the desired solution to inform the chosen imputation algorithm and data-source weighting coefficients.   ;D  Come on, get with its GUARANTEED to make Bitcoin look awesome!  We know this from seeing the CPI numbers.

interpolate ?

Interpolation can help with missing values, but I'm not sure what it has to do with combining multiple inputs streams.

Imputed data isn't.  Ditto for interpolated.

If your model involves anything resembling regression, cleaning the data in any way will cause your model to vastly overestimate the certainty and accuracy of the output.

This kind of thing is a pain to model.  The spreads between the exchanges distort the price signal that you are looking for, but not totally.

You could use (sign,magnitude) of changes instead of absolute values, which will remove the pure-arbitrage signal from the price signal, but that distorts the price signal.  (sign,log(magnitude)) might help a bit, but that's hard to say too.

Or, you can ignore the arbitrage signal, and just mash the prices together as they really are.  But that will result in a price that is constantly too high by a factor related to the difficulty of moving stuff around.

You are going to hate this, but the most valid way to go is to model each exchange and the relationships between them.  You'll have to give up on the notion of "the bitcoin price" and instead work with "the bitcoin price at locations X,Y and Z".

Oh, and I forgot that the arbitrage issues are not linear.  Your model is going to get screwed every time the real world factors that cause the spreads change.

sorry if im probably missing something obv but if i had multiple data streams (say 3 exchanges) with different sample rates or missing values or whatever, i would interpolate each data stream to a common timeseries and then combine (average) the data. and would probably weight the inputs by volume

Okay, now I understand what you mean, which was the first suggestion in this thread.  However, I don't think you are using the term "interpolate" correctly.

im listening...

Interpolation is synthesizing new points between existing data.  This is not the same thing as as interweaving two data series based on a common time series.  I'm not sure what the best term is for that, but it isn't interpolation.

http://en.wikipedia.org/wiki/Interpolation

if someone puts together this dataset i actually need (for my thesis):

total daily volume on the exchanges

average weighted daily price

OR

total daily trade volume in $