attn:bitcoin exchanges - request for json api trades by timestamp/tradeid

[mizerydearia]

I opted to post this on the forum instead of emailing each exchange (which I will do anyway to alert them of this thread), which also allows for newly established exchanges to also participate:

I am streaming trades in real-time to #bitcoin-markets on Freenode, similar to #bitcoin-market that I originally established and have since passed on to nanotube (*waves*), but with potentially much faster (the data is multi-threaded and for some markets, trade data results are faster, e.g. 2-10 seconds delay instead of 10-40 seconds) results.  The short delay is true only for exchanges with a JSON trade API implementation that allows for querying all trades since a last timestamp or trade id, therefore guaranteeing downloading only the trade history, reducing bandwidth and processing time.

I have noticed some exchanges that have a discrepancy with sequential trade ids relative to timestamps; some trades have trades ids lesser than other trades but with timestamps more recently/newer than the same trades.  This type of exchange implementation provides a potential issue in the case of querying new trades since last trade id since the particular new trade would be new, but have a much lower trade id than the last trade id from recent queries.  Therefore, if your exchange is like this, it makes much more sense to provide a JSON trade API implementation that references date timestamps instead of last trade id.

If your exchange is interested in adding implementation (or does it already have it?) to query trades either equal to or later than a particular timestamp or greater than a trade id, reply.

Oh, btw, the open source python bot in #bitcoin-markets is still under development.

Also see https://bitcointalk.org/index.php?topic=45783

[1713526524]

1713526524

[1713526524]

1713526524

[enlightened]

If I understood you correctly, you want exchanges to provide an api call that lists all trades since a certain timestamp.

Here you are: https://solidcoin24.com/api/trades/btc/s/{timestamp}
Example: https://solidcoin24.com/api/trades/btc/s/1315601224

[James Freeman (imcex.com)]

We'd like to work with you on this at https://imcex.com
Please, send me a message.

[mizerydearia]

If I understood you correctly, you want exchanges to provide an api call that lists all trades since a certain timestamp.

Here you are: https://solidcoin24.com/api/trades/btc/s/{timestamp}
Example: https://solidcoin24.com/api/trades/btc/s/1315601224

Thanks.   Can you add trade ids to the data?

We'd like to work with you on this at https://imcex.com
Please, send me a message.

Do you have any JSON APIs for depth/orderbook, ticker, trades?  I didn't see any on your site.

[mizerydearia]

Example file sizes of json data per exchange market:

bcEUR: 61Kb
bcLREUR: 61Kb
bcLRUSD: 61Kb
bcPGAU: 61Kb
bcmBMUSD: 28Kb
bcmLRUSD: 8Kb
bcmMBUSD: 16Kb
bitchangePLN: 12Kb
bitmarketEUR: 4Kb
bitparkI0C: 1Kb
bitparkIXC: 1Kb
britcoinGBP: 9Kb
btceUSD: 6Kb
btcexUSD: 8Kb
exchbUSD: 38Kb
intrsngEUR: 7Kb
intrsngGBP: 9Kb
intrsngUSD: 3Kb
mrcdBRL: 10Kb
ruxumUSD: 24Kb
thEUR: 1Kb
thUSD: 4Kb

Querying each of these every 30 seconds results in 2,880 queries per day:
@1Kb ~= 2.8Mb/day, 84.3Mb/month
@61Kb ~= 172Mb/day, 5,147Mb/month

This bandwidth consumption is guaranteed to be decreased if showing only latest trades since a specific trade id or date (since the last id or date detected) even with increased query time such a 5 seconds.  Especially in the case of an exchange market having 0 new trades in the last 1 second, then JSON "[]" or similar would be returned, which is only two bytes (1/512th Kb), which additionally greatly reduces bandwidth consumption.

Looking at an example data file above with largest file size (bcEUR): I see the resulting file returns 950 trades, only 29 of them from the last 24 hours, each string of trade data about 64 bytes.  Querying for this data each second showing only latest trades since a particular trade id or date would then return 29 (instances) * 64 bytes of data + (2880 - 29) * ~4 ~= 13Kb/day, 388Kb/month

With these example figures, is your exchange interested in saving approximately 5 gigabytes of bandwidth consumption per month reducing a single instance (for example mine/bitcoinmarkets.com(website coming soon), another maybe bitcoincharts.com) of querying from 5 gigabytes per month to less than half a megabyte (only true with above data, more volume will result in more bandwidth consumption) per month?

[enlightened]

If I understood you correctly, you want exchanges to provide an api call that lists all trades since a certain timestamp.

Here you are: https://solidcoin24.com/api/trades/btc/s/{timestamp}
Example: https://solidcoin24.com/api/trades/btc/s/1315601224

Thanks.   Can you add trade ids to the data?

Sure.

[mizerydearia]

I would like to convince keyur of campbx to implement json api for trades by last trade id or last trade timestamp, however, I am met with a bit of concern from him:

If we make all historic trades part of the query then someone unscrupulous can abuse this API call to overload our systems.  So to err on the side of safety, we prefer to limit the API call to past three hours OR most recent 100 trades.

I responded to that statement suggesting:

https://en.bitcoin.it/wiki/Intersango/API
https://intersango.com/api/trades.php?last_trade_id=0

https://en.bitcoin.it/wiki/MtGox/API#Multi_currency_trades
https://mtgox.com/api/1/BTCUSD/public/trades?since=0

In what way would your system be overloaded that mtgox or intersango wouldn't be overloarded with same reasoning/logic (especially mtgox with much more traffic/users)?

He responded with:

Very interesting from Mt.Gox and Intersango.  I think the choice comes down to risk tolerance and how much mitigation expenses you can afford: Intersango seems to have more risk tolerance than us, and MtGox can afford more mitigation expenses than us.

 

Risk:  If someone sets up 10 bots from different IP addresses and sends 5 requests per second with randomized X in “?since=X”, it will force a full database scan which can bring a production server to its knees.  If not today, three years from now because ‘API is forever’.

Mitigation: Use a mirrored database on a secondary server to query the historic data – unfortunately Camp BX doesn’t have a budget for this, and when in future we set up something like this we will be happy to introduce a new API call.

Discussing in #bitcoinconsultancy about it, joepie91 mentioned memcache, and I agree using it would help, however, I'm fairly confident one could implement an efficient system that doesn't require memcache.  Asking phantomcircuit if he is using memcache for Intersango, he responded he isn't using it since it's not needed yet.

I'm not certain how to respond to keyur, so instead I created this post.  Maybe someone can assist in determining whether my request is asking for too much and that such an implementation is indeed ripe for abuse or not.

[mizerydearia]

Originally only suggested or mentioned by Keyur of campbx, but now also agreed upon by Virtex, providing a json api implementation offering results by last trade id or last trade timestamp opens up a kind of cost or risk.

Here's discussion with cavirtex.com:

me: Is it possible to query trade history by last trade id or last trade date?  See https://bitcointalk.org/index.php?topic=44674.0

cavirtex: No, not currently. It would be possible, but as keyur suggested on the
forum, there is a cost/risk associated with making that available.
Also at this stage, standing up a mirror and implementing exceptions
for the cached data wouldn't likely be worth the cost savings in
bandwidth, unless you had some other argument for creating such an
API?

Currently, we do cache our data and allowing someone to choose the
range would make caching more difficult. Right now, the current
solution of fixed history seems to be within bandwidth limits and is
easily cached (making it easy to process). Compressing it would be an
easier step in your direction. That data would likely compress very
well and if cached, would only need to be done once in a while.

To comment on the post a bit. A proof by anecdote (ie. mtgox and
intersango do it) does not make a solid arguement for it's
implemenation because of counter arguments such as "perhaps they are
not being abused yet?" and the fact that they operate in different
enviroments (as keyur suggested).

Perhaps this issue/concern can be discussed a bit to establish a consensually agreed upon idea on how to best implement a secure, stable method of allowing querying by last trade id or last trade timestamp; one that is not susceptible to abuse?