|
Title: What is best way to get all blockchain data in mysql ? Post by: btctousd81 on December 12, 2017, 12:58:06 PM i am looking for best available way to get all blockchain data in mysql.
best as in, a single program should parse blk files and insert data in mysql , insteading of creating dump file and then me, importing that dump file in DB so i can further process data., i will be running full bitcoin node , i can see there are multiple options/projects on github but idk which works best and isnt abandoned project. what you guys are using ? Thanks for your time., Title: Re: What is best way to get all blockchain data in mysql ? Post by: btctousd81 on February 03, 2018, 02:09:38 AM There are several ways of solving your problem but even if it is explained thoroughly you will have to handle some difficult things! 1.Connect to the bitcoin network directly, parse all messages, verify the blocks and insert all the transactions contained within it into the MySQL db. This is going to be a bit hard since you have to write code to connect to the network. 2.Run bitcoind side-by-side with the MySQL db and use some combination of the getblockcount, getblockhash, getblock, gettransaction RPC commands and insert the transaction data into the MySQL db. You can run a cron on this every 5 minutes or so and you should have a fairly up-to-date MySQL db. This comes at the cost of considerable disk space since you'll need to have the entire blockchain for bitcoind (30 gigs and growing). 3.Connect to some other service and get the data that way. Blockchain.info is the obvious one and they have a wonderful API for getting all the new data you need. Unfortunately, you'll have to trust them to provide the right data. There's also electrum servers which provide data using stratum and ABE and some others. Heck, if you can get someone to open up their RPC port (8333) on bitcoind, you can connect to their bitcoind and use the same commands as #2. The drawback here is that you have to trust a third party in some way, shape or form, though, of course, you can write your own verification code. its more like 150 gigs Title: Re: What is best way to get all blockchain data in mysql ? Post by: Anti-Cen on February 03, 2018, 03:59:16 PM I know someone on-line is running a site that lest you run SQL-Server query's against the block-chain
but you have to do joins of a couple of tables but basically you type the SQL statement (I could hack him I bet) into a text area and press run. The results are quite fast which is a surprise to me because using varchar as foreign keys is slow and they needs lots of memory to index them so unless he had something like 32gb of ram then I am not sure why it ran so quick. Once your field index needs to page to files then the database runs as slow as a snail so I would be interested to hear your findings on this project if you let me know what hardware your running because 150 or 200gb of data is a big number for any database and I've worked on some big ones. Good luck |