Dear MusX,

not bad - but for my taste too many dependencies and a single point of failure.
However, if there are many sources its a good thing - also to crosscheck the data.

Yours sincerely

bitranox

Dear b!z,
I will also try to cover bitstamp tradedata if possible.
yours sincerely
bitranox

Dear Arvin,
thanks for Your offer - but there will be no GUI since that will be completely useless for just a datapump.
I think it will take until beginning of October to have the first version online.

Yours sincerely

bitranox

Dear nitrous,


yeah, it might look ridiculous, but I dont want to get sued by a guy like that: https://bitcointalk.org/index.php?topic=278973.0
just because a tradingbot went wild on occasion or something like that ...

yours sincerely

bitranox

I am planning to create a tool to provide and update historical tradedate.

for details please see : https://bitcointalk.org/index.php?topic=282154.0 - please vote there if You would need such a tool.

please vote here, how often You want that data to be updated.

Sorry guys, realtime data is not scope of the project.

yours sincerely

bitranox

Dear MusX,

there is no problem to provide the full data for free.

there is only a problem to provide realtimedata. In that case I need to put the stuff on one of my own servers, what will make the service vulnerable to DDOS attacks.
So You probably dont want to use such a source as a feed for Your tradebot, wouldnt You ?

My opinion is, for the historical data my method will work fine (update every hour or every 10 minutes probably). For realtimedata I suggest to use the channels from mtgox.

yours sincerely

bitranox

Dear nitrous,
thank You so much for Your friendly comment. Here my answer on that :


that is also my concern, so I will KISS (keep it simple and stupid)


thanks for the link - I checked it and it is something different. My Tool will not do not any data representatation but only acts as an interface.
It will download the data and feed it to another custumizable format or database for futher investigation or use by third party tools like trade robots, data analysis, whatever ...


I dont want to expose our servers to the community because we dont want to be victim of DDOS Attacks etc.
Preventing those kind of attacks, renting servers and so on, will cost more time and money I am willing to spend on it - also I want to keep the service free of charge.
So I will just upload junks of data onto a number of filehosters.
The tool just downloads and reassemble those junks, as well as provide some mechanisms to translate that data into customizeable formats and feed it into another database or csv or whatever.
Since there is no database operation involved on the Webservers, it is very very fast, and failsafe - I can spread those downloads over a multiple number of hosters, for instance google drive, dropbox and others.
Also it is proxy-friendly and will be totally trasnsparent for firewalls etc.
All You need is access to port 80 and the ability to download *.gz files from the internet (some very restrictive firewalls {snort} may prevent that)


that is exactly what I am thinking of. for statistic analyses and stuff it would be enough to be able to update every 10 minutes.
If someone wants a realtime feed, he might download bulkdata with my tool and then take care of the most recent data by himself, or another tool probably provided later.


KISS. If any of the exchanges adds a certain currenypair in the future, users can simply inform me that they want that data. And if possible I will integrate that into the tool.
I just take what I can get and provide it, so no need to bother MagicalTux or others, also not depending more then neccessary on the good-will or recources of different exchanges, organisations or persons.
The tool will auto-update itself (if the user allows that) in order to be able to process such future dataformats, currencypairs, exchanges, whatever ...


yeah, cool, I will come back for that later.


no - no indeces at all, and as low as possible redundancy to keep bandwith and footprint as small as possible.
by the way, I am planning that the user optionally can delete the datajunks after exporting / update to his desired target


That will not be the scope of the tool. It should just provide a method to feed the data into any format or database You want - there You can do all the indices, sortings  and queries You might need.
The focus is to make a robust script to be able to use it in macros, batchjobs, cronjobs and so on, to keep Your existing database up to date, or just to download the bulk-data until now().


Of course it is better to work together, so my code will be GPL anyway - so of course You or any other user can also reuse my code or the data provided in any way.
In order to be completely legal and save to reuse the code, I suggest You also include some GPL Licence with Your stuff on bitbucket and other stuff you provide.

Yours sincerely

bitranox

I am thinking to provide the tradedata by another method to the users.

I fetched all the data for myself, so if a considerable amount of users wants to use that, I will provide the tradedata in a different way (not rely on mtgox updating the data, not rely on big query).

please check out following post : https://bitcointalk.org/index.php?topic=282154.msg3017272#msg3017272 and let me know.

yours sincerely

bitranox

I am thinking to provide the tradedata by another method to the users.

I fetched all the data for myself, so if a considerable amount of users wants to use that, I will provide the tradedata in a different way (not rely on mtgox updating the data, not rely on big query).

please check out following post : https://bitcointalk.org/index.php?topic=282154.msg3017272#msg3017272 and let me know.

yours sincerely

bitranox

Inspired by the amazing work of nitrous I will provide a tool (and the data) to download and update
the historical trade data of mtgox and other exchanges for a number of currencies like BTC/LTC and others in a standardized way.

The Toolbox Name is TradePickler

the toolbox will be composed of three parts :

- the data collector, what collects the data from variouse sources and uploads that data onto a number of hosts (google drive, skydrive, dropbox, cloudme,  others ...)
  You will not be in touch with that part, since that will be done on our servers.
- tradepickler-gobble : the downloader, it downloads junks of data to Your harddisk
  It will be also perfectly possible to write Your own downloadtool since the dataformat will be fully documented,
  but we dont recommend it - since the tool is written to perform the downloads keeping in mind very low API usage of the filehosters (not to get kicked out by them)
- tradepickler-transcribe : the export tool, that will export the downloaded data in a most flexible way to the users target dataformat or database.
  You are welcome to contribute modules to the transcriber or write Your own implementation.

here the reasons why :
- nitrous service require bigquery, what is slow and needs registration (a full download needs more then 24 hours on my state of the art machine on a 10Mbit connection)
- since the data is only growing there is no need to use bigdata
- I want to have a more robust, more distributed approach (i will upload the data to several filehosters)
- the bigquery database is not updated by mtgox until now, and no one will know if and when this is going to happen
- the bigquery database will just be updated my mtgox - but what about other exchanges like bitstamp ?
- providing a bullet proved and full automatic way to fetch and export all the tradedata You want, to any format or database You want
- TradePickler will not be limited only to BTC and MtGox.

what I love about nitrous downloadtool :
- good interface to adress non techies
- good packing for windows and mac users

what I dont love about nitrous downloadtool :
- no automatization for auto-update and so on ...
- no automatization for exports / updates to other dataformats. You can not save Your settings.
- rely on big query, rely on the updates from mtgox

here the concept :
- trade data will be provided on filehosters (google drive, skydrive, dropbox, cloudme, others ...)
- no registration, free service
- updates every day (cold data) / and every 10 minutes (hot data)
- small footprint, redundant data is stripped (the full tradedata for all currencies from mtgox consumes about only 150MB on the harddisk)
- dont rely on uploads from MtGox - we collect the data ourself
- expandable to other currencies (litecoin) in the future
- expandable to other exchanges (like bitstamp and others) in the future

objectives :
- robust and failsafe
- very easy setup and very easy to use on any platform (Linux/Windows/Mac)
- commandline interface, so can be easily used in batchjobs, shellscripts, macros, chronjobs to be able to automate the updates to your target database or dataformat
- settings stored into an easy, full documented ini file with lots of examples how to convert the data to the target You want.
- probably integration into nitrous downloadtool, be able to update his existing sqlite database
- written in python 2.7.x
- open source (GPL 3.0)

shortcomings :
- I can not provide the App ID of the bidding/selling party (the person buying/selling BTC), since this is not covered in MtGox API
however, since only 4% of the trades until now have some App ID this should not a big deal.

what I have got now :
- all the mtgox data is collected until now, as of 2013-08-27
- I am able to update the data in a 10 minutes timeframe without any problems
- the documentation of data structure for data- and metafiles

what I am working on now:
- reorganize the data according to the dataformat designed
- automatic upload of the files to google drive for the first version.

todolist:
- creating the download client (thats easy)
- creating the export tool (converting the data from my own proprietery format to any format You want), starting with csv export
- cross checking my data with nitrous database

timeframe:
- since I can donate only a few hours a week to that project, things will wind up slowly
- I will set up a project on bitbucket when the time comes, any help is appreciated


what I need from You is some input:

- please vote - are You interested in such a tool ? Is there any need for this tool ? vote here please : https://bitcointalk.org/index.php?topic=282154.0
- please vote - how often should the data be updated ? vote here please : https://bitcointalk.org/index.php?topic=282246.0

your comments are welcome

The (very good and appreciated) "Unofficial Documentation for MtGox's HTTP API v2" has some glitches in the description for the API function money/trades/fetch :

(see https://bitbucket.org/nitrous/mtgox-api#markdown-header-moneytradesfetch)

there are not really "gaps" (except the one big gap for USD when they switch from a counter to microtimestamp)

the mtGox API function money/trades/fetch simply works as follows :

if You pass a microtimestamp as parameter :
- it will return maximum 1000 records of a given currencypair
- it will return maximum the trades that happend 86400 seconds after the given microtimestamp
meaning if there are less then 1000 trades within 86400 seconds, You will receive just the trades that happened 86400 seconds after the given microtimestamp.

if You dont pass a microtimestamp parameter :
- it will receive the tradedate for a given currencypair within the last 86400 seconds (24 hours) from now. In that case the "1000 record" limit does not apply.

so - if no trades happened within the timespan of 86400 seconds for a given currencypair, You will not get back any datarows - what will only happen for some rarely traded currencypairs
in that case You will need to add 86400 seconds to the timestamp (respectively 86400 * 1E6 microseconds) and query again for the next day.

As nitrox pointed out in some other posts, I would not recommend that every user should download the full trade history from mtgox because of serverload issues, etc ...

however - since the bq database is not updated until now, I will provide the full dumps until up-to-date in the next couple of days. (what will be updated daily or hourly - I wait for user requests on that issue)

the python script that 100x posted before will fail on some rare traded currencypairs for that reason.

+1 for that one.

this is some comment to following post https://bitcointalk.org/index.php?topic=221055.msg2945085#msg2945085

unfortunately I am only allowed to post in the newbie secton right now - however :

nitrous posted :

The (very good and appreciated) "Unofficial Documentation for MtGox's HTTP API v2" has some glitches in the description for the API function money/trades/fetch :

(see https://bitbucket.org/nitrous/mtgox-api#markdown-header-moneytradesfetch)

there are not really "gaps" (except the one big gap for USD when they switch from a counter to microtimestamp)

the mtGox API function money/trades/fetch simply works as follows :

- it will return maximum 1000 records if a microtimestamp is transfered as parameter.
- and it will return maximum the trades that happend 86400 seconds after the given microtimestamp

if You dont pass a microtimestamp to the function You will receive the tradedate for a given currencypair within the last 86400 seconds (24 hours).
In that case the "1000 record" limit does not apply.

If no trades happened within the timespan of 86400 seconds for a given currencypair, You will not get back any datarows - what will only happen for some rarely traded currencypairs

so - if there were simply no trades within 24 hours, You will not receive any datarows back.
In that case You will need to add 86400 seconds to the timestamp (respectively 86400 * 1E6 microseconds) and query again for the next day.

As nitrox pointed out in some other posts, I would not recommend that every user should download the full trade history from mtgox because of serverload issues, etc ...

however - since the bq database is not updated until now, I will provide the full dumps until up-to-date in the next couple of days. (what will be updated daily or hourly - I wait for user requests on that issue)

I would love to post that on the forum that matters, not just in that newbie ......