[ANN] Bitcoin-Analytics.com - price correction, additional currencies

[genuise]

ello. I'm interested in your website and possibly subscribing. However with the recent sharp increase in value of bitcoins I was wondering if you could provide information on arbitrage trnsactions for as little as 1 bitcoin now?  Also how much is the subscription? I know it was 0.5 btc/month but I'm hoping you'll have dropped your rates now!

Hi, thank you for your interest in service.

Normally we reconsider subscribtion plans prices once per month. And last time we corrected them on apr 1.

Full access will cost you:
0.002 BTC a day, 0.03 BTC a month, 0.05 BTC for 2 months

hope this help.

if you have further questions we will gladly answer them.

[1713566598]

1713566598

[1713566598]

1713566598

[1713566598]

1713566598

[1713566598]

1713566598

[Masqurin]

Hi, as a subscriber is it possible to access data older than a day? Also can we get an option to have the volumes in either BTC or USD on the same axis, instead of having bids in USD and asks in BTC on separate axes?  It's hard to compare two plots on different axes in e.g. the order book volume chart, you can only see the relative fluctuations on either axis.

[nimnul]

Hi,

Thank you for supporting us.

We have a plan on adding month long charts with candlestick aggregation but now it's not possible to see data older than a day.

There is a problem with having volumes in either BTC or USD on the same axis. Often there are spam orders on both sides of orderbook: someone sells 1 BTC for 1 billion USD and someone buys 1 billion BTC for 1 USD.

For bids, only USD volume is "real" and BTC volume is an arbitrary number put by whoever placed the order. For asks it's vice versa: BTC volume is "real" and USD volume can be set to arbitrary number.

If we put both volumes in one currency it will not help you to compare them. So there should be a method of filtering meaningful depth from fake depth and we couldn't find a good one. Do you have ideas?

Best regards, nimnul and genuise

[Masqurin]

How about calculate the "real" volume yourself by e.g. multiplying the USD volume by the order price to get the true BTC volume?

[genuise]

It is not a problem to multiply usd bid price and volume of btc in each bid order.

But as nimnul explained, that simple multiplication will not get meaningful number.

The problem is that not all bid orders have meaningful, as nimnul stated a "real", price. e.g. if someone places a bid order to buy 1000 000 000 btc for 1 USD, in other words buyer expects to buy 1bn btc at 0,000000001 usd/btc. does this price make sence for your analyzes? Such order will be never executed.
Thus such orders represents spam, not real , fake data in the orderbook.

In other words this one fake order will make bids line several orders higher than asks line. 1bn btc on bids side which are not existent and 100k btc on the asks. This really does not make sence. Can you agree?

There are a lot of such orders, and their combined volume just obscurs the true picture.

If we exclude those fake orders we will see the true picture.

It this clear?

So the question is how to tell which orders have to be excluded and which not?

There is no common agreement on which order have to be counted as meaningful and which is not.

So how to find this common filtering creteria is the main question.

Do you have any ideas on how can we solve it?

[Masqurin]

It is not a problem to multiply usd bid price and volume of btc in each bid order.

But the problem is, as nimnul explained, that simple multiplication will not get meaningful number.

The problem is that not all bid orders have meaningful, as nimnul stated a "real", price. e.g. if someone places a bid order to buy 1000 000 000 btc for 1 USD, in other words buyer expects to buy 1bn btc at 0,000000001 usd/btc. does this price make sence for youe analyzes? Such order will be never executed.
Thus such orders represents spam, not real , fake data in the orderbook.

There are a lot of such orders, and their combined volume just obscurs the true picture.

If we exclude those fake orders we will see the true picture.

It this clear?

So the questions is how to tell which orders have to be excluded and which not?

The problem is here that there is no common agreement on which order have to be counted as meaningful and which is not.

So how to find this common filtering creteria is the main question.

Do you have any idea on how can we solve it?

One way to filter it would be use an discount factor based on distance from the top order. The discount factor can be exponential or linear.
An exponential discount factor would be something like e^(-ax) where a is some scaling factor and x is the distance. A linear discount factor could be something very simple like a(x/x_top)

For example, say current top bid is $60 and there's a bid for 1000000000 BTC at $1.

The exponential discounted volume with a = 1 is 1000000000 * e^(-(60-1)) = 2.3802664e-17, i.e. negligible. If we want to give it a bit more weight we can decrease a to 0.1 which yields 1000000000 * e^(-0.1(60-1)) = 2739444.81877.

The linear discounted volume with a = 1 is 1000000000 * (1/60) = 16666666.6667. Give it more weight by increasing a.

The same thing could be used on asks. Personally I'd like to see orderbook volume charts in USD and BTC (option to show volume in USD or BTC). I'm also curious if the spam orders on both sides simply cancel out, what does the chart look like if you just multiply without discounting? If I could get some raw data I can play around with the parameters in MATLAB and see what values are reasonable.

For the depth chart it's not necessary to discount, simply convert BTC to USD and vice versa at the price of each order to make volume on each side use the same currency. Again it would be nice to have an option to toggle between BTC and USD.

Another suggestion, add another series in the VWAP chart for both buys and sells, and have a control to show/hide each chart.

[Masqurin]

Actually, is your data the same as https://data.mtgox.com/api/2/BTCUSD/money/depth/full? I could just play with that data if it's the same.

[nimnul]

Yes it's the same. We just use mtgox realtime api to get order book every 15 seconds. The full depth url can only be queries every 15 minutes or so.

[genuise]

On the depth charts for mtgox you can see that we limit ask with 1000 usd/btc price and you can visually estimate what discount factor will be useful.

[Masqurin]

Ok, I made some charts to demonstrate. The total discounted volume for bids and asks in USD is shown beneath each.

Here's the raw data I was working with
https://i.imgur.com/yPjLtle.png
Bids: 12170767.4644583
Asks: 27476862169015.9

And here's the depth chart up to twice the top bid
https://i.imgur.com/BRxe2b6.png

Here is the exponential discount factor I described before, it prioritizes prices close to the spread.

The formula for the discount on bids is e^(-a(highestBid-bid)), and e^(-a(lowestAsk-ask)) for asks.

Here's the discounted volume:
https://i.imgur.com/CSZjOCi.png
Bids: 105023.402931941
Asks: 23900.6565377794

That was a little too restrictive, relax the scaling parameter a bit to a=0.1:
https://i.imgur.com/K0SOpL1.png
Bids: 1522439.36415067
Asks: 1236490.55085024

It can be relaxed further to include points further down the line:
https://i.imgur.com/b8VT37W.png
Bids: 3262099.42799612
Asks: 2777887.40515259

Alternatively to give full weight to the majority of the orders, use 1-e^(-a*bid) for bids, and max(0, 1-e^(a*(ask-lowestAsk)-highestBid))) to create a symmetrical bandpass filter about the spread.

Here's what it looks like with a = 1:
https://i.imgur.com/1aToRt2.png
Bids: 11970104.7601015
Asks: 9177170.77805910

Restrict the further out values a little with a = 0.15:
https://i.imgur.com/PowevBb.png
Bids: 11091817.7195716
Asks: 8799423.98610072

A simple linear discount function, use 1-(highestBid-bid)/highestBid for bids and max(0, 1-(ask-lowestAsk)/lowestAsk) for asks
https://i.imgur.com/9b6neS5.png
Bids: 6158532.21541063
Asks: 5225588.99409854

Depending on what kind of analysis you want to make I'd say either the bandpass function with a=0.15, exponential function with a=0.1, or linear function would all work fine.

[Masqurin]

One more thing instead of using a line plot for depth and trade volumes it might be better to use a stacked bar chart. Right now it's really hard to hover over one exact datapoint to see its values, and it doesn't really take advantage of horizontal scaling.

[genuise]

Hi, Masqurin,

this is exactly a kind of thoughts we really like and appreciate. Thank you for your time and efforts.

We must admit that the density of information you gave us is quite high and we need to digest different parts of it to be able to understand what is easy and what is not to implement in our current application architrecture.

It would be really helpful if we can exchange further to see what limitations we currently have and what can be implemented and in what order.

As for the math of the filtering it not a problem to implement filtering as is, but the next question is:

currently while preparing data for display client we need to make some steps to solve several problems

1) our server should be able to gather and recalculate all parameters once per 15 seconds and not overload the processor

2) we then need to store limited part (not all) calculated paramateres in memory in 15 second buckets in such a form to not exceed the memory limit of our machine

3) then we need to prepare data to transfer it to the client browser - and again to have resonable amount of data to transfer (keeing in mind that we already are using float data type, gzipping) it takes some times up to 10 seconds to load the whole data to browser

So adding additional parameters to calculate is not the main problem, but to decide what to transfer to client and and how to display it could be a difficult task.

For example: as for the orderbook volume we have two series per ticker, and those data currently represent actual sum of all USD and BTC in orderbook even if orders are spam orders, and if we decide to filter out spam orders in order to visualize bids and asks series in the same units then we will loose our original genuine data which is also make sence in its own context (real not filtered amount of USD and BTC in orderbook)

What do you think about this particular problem what would be good solution from the user point of view to maintain both series together on the client to be able to compare them or just to leave filtered and to abandon the original full volume series?

I'm also curious if the spam orders on both sides simply cancel out, what does the chart look like if you just multiply without discounting?

I mean you prefer to have both pair series or only one pair?

also as each filtering method will produce differernt series would it be practical to maintain all of them (thus multiplying serires) and toggle between them or just one. If one which one would be the most practical for analyzes on your opinion?

[Masqurin]

Without a detailed rundown of your system my ability to evaluate your performance capabilities is limited but I'll try.

1) Server-side performance shouldn't be a problem for these filters, they only take O(n) additional arithmetic operations per update (n = number of orders in the orderbook) and the same amount of memory as whatever you're using to store the volume data right now.

2) When sending data to the client during an update, do you send all data or just the update delta? I'm not sure what sort of data the client keeps track of, but assuming you have all the calculations done server side the update delta should be very small, for each graph you should only need to send one additional datapoint per line per update.

3) It seems the client is receiving data for all currencies and exchanges, even the ones that aren't shown. You can avoid some data transfer by sending it selectively. For example, I only care about mtgox data in USD,  there are 13 other tabs containing data I don't need. What I would do is have an option in the user's preferences for the default exchange and currency, and only load and display that data by default. Request and subscribe to the other tabs only when the user requests it.

4) Similar to 3), you don't have to show all charts by default and the client doesn't need data for hidden charts. As I suggested before, add a control to hide/show each chart, and have an option for a default set of charts in user preferences. Only request and subscribe to data for a chart if and when it's shown.

For each series you'll have 24*60*4 = 6240 datapoints. Each datapoint is two floats, i.e. 8 bytes, yielding about 49kB of raw floats per series (not sure how the data is structured, e.g. if you're using JSONs there will be more overhead). I'm not sure what kind of compression ratios you're getting, a cursory search suggests gzip doesn't do a very good job of compressing floats. Still, it shouldn't be a problem for most people and you may be able to reduce it further by using a compression algorithm that's optimized for floats.

With regards to the filtering I suggested, I think there's no real point sending the raw data to the client; as it is the data is only useful for showing relative change in volume for either bids or asks. Seeing as orders that are very far from where the trades are happening don't really affect the prices, I don't think you lose anything by reducing the weights assigned to them. In fact by discounting spam orders and far-out orders it would better link relative change in volume with change in price since the orders that matter are given more importance.

[Masqurin]

I'm also curious if the spam orders on both sides simply cancel out, what does the chart look like if you just multiply without discounting?

I mean you prefer to have both pair series or only one pair?

Nevermind that request, I made a chart for the data and "fake" orders do indeed have a large impact.

also as each filtering method will produce differernt series would it be practical to maintain all of them (thus multiplying serires) and toggle between them or just one. If one which one would be the most practical for analyzes on your opinion?

I think picking one is fine, which one to use depends on the purpose of the chart. The exponential function (a=0.1) prioritizes orders that are very close to the current price and hence should be a better indicator for short term market movement. The bandpass function (a=0.15) is closer to the "true" total volume but is more sensitive to volume changes that are very far away from the current price. The linear function is midway between the two but I'm not sure if it can match the benefit of either. I would run all three for a while and see what they look like, and then analyze the strength and weakness of each.

More UI suggestions: 1) Make the height of the chart area scale with window size, if you've got chart height scaling working it would be really nice to hide a few of the charts and have the rest be taller. 2) Move the axis labels to the right side of the charts, that's where new data is coming in so that's where people will be looking. 3) Add an option for a dark colour scheme, I think most people prefer something like http://bitcoinity.org/markets or http://bitcoin.clarkmoody.com/. 4) On the VWAP line there are markers for each point, but not on the other price lines. I think it looks better without markers.

[Masqurin]

Another idea, the raw total volumes could be used for a total bid USD/total ask BTC chart, would be interesting to see how that correlates with price.

[genuise]

Another idea, the raw total volumes could be used for a total bid USD/total ask BTC chart, would be interesting to see how that correlates with price.

This what I suggested some time ago for discussion. And I think in this case it would be good to use filtered  raw data to have meaningful estimation for the price.

How can we call this parameter  - expected market price or general market expected price? or? Is there any existing parameter for this in financial world already?

[Masqurin]

Another idea, the raw total volumes could be used for a total bid USD/total ask BTC chart, would be interesting to see how that correlates with price.

This what I suggested some time ago for discussion. And I think in this case it would be good to use filtered  raw data to have meaningful estimation for the price.

How can we call this parameter  - expected market price or general market expected price? or? Is there any existing parameter for this in financial world already?

I'm not sure if there's a term for it, but total bids:total asks seems clear enough. I'm not sure if filtering would improve it and if so which filter, so again it would be nice to capture some data and compare.

Another one that would be good to have is a moving average of the buys volume:sells volume ratio over windows of e.g. 10m, 1h, 3h, etc.

[nimnul]

Another one that would be good to have is a moving average of buys volume / sells volume over windows of e.g. 10m, 1h, 3h, etc.

The charting library we use supports post-processing with moving average. See the input box at the bottom chart at http://dygraphs.com/tests/range-selector.html

If it's ok we can easily add that input box for moving average window size to all the charts.

[Masqurin]

Seems like that would do the trick. By the way by "/" I meant the ratio, editing to clarify.

[nimnul]

4) On the VWAP line there are markers for each point, but not on the other price lines. I think it looks better without markers.

If you zoom the VWAP chart you will see something like this:



The markers are not for every point, but for each point (15-second "slot" as we call it) with trading activity. So here you can see that a large trade activity at 14:29:45 shifted VWAP, but a larger trade volume at 14:33:00 had almost no impact on the price. Removal of the markers will obscure this information.

Of course when you zoom out to default zoom level it starts looking ugly, but we didn't knew how we can improve it without information loss.

In addition you can see here why we chose to plot "buy" and "sell" trades separately instead of joining them with a single line as our competitors do. We show that trades are performed at two different steady prices instead of zigzag pattern you often see on last trade price charts without bid-ask separation and volume weighting: