Bitcoin Forum
November 12, 2019, 01:43:22 AM *
News: Help collect the most notable posts made over the last 10 years.
 
   Home   Help Search Login Register More  
Pages: [1]
  Print  
Author Topic: What is best way to get all blockchain data in mysql ?  (Read 280 times)
btctousd81
Sr. Member
****
Offline Offline

Activity: 434
Merit: 258


View Profile WWW
December 12, 2017, 12:58:06 PM
 #1

i am looking for best available way to get all blockchain data in mysql.

best as in, a single program should parse blk files and insert data in mysql , insteading of creating dump file and then me, importing that dump file in DB

so i can further process data.,

i will be running full bitcoin node ,

i can see there are multiple options/projects on github but idk which works best and isnt abandoned project.

what you guys are using ?

Thanks for your time.,

 

The Bitcoin Forum is turning 10 years old! Join the community in sharing and exploring the notable posts made over the years.
1573523002
Hero Member
*
Offline Offline

Posts: 1573523002

View Profile Personal Message (Offline)

Ignore
1573523002
Reply with quote  #2

1573523002
Report to moderator
1573523002
Hero Member
*
Offline Offline

Posts: 1573523002

View Profile Personal Message (Offline)

Ignore
1573523002
Reply with quote  #2

1573523002
Report to moderator
1573523002
Hero Member
*
Offline Offline

Posts: 1573523002

View Profile Personal Message (Offline)

Ignore
1573523002
Reply with quote  #2

1573523002
Report to moderator
herecomesjohnny
Member
**
Offline Offline

Activity: 147
Merit: 12


View Profile
February 02, 2018, 10:31:08 PM
 #2

There are several ways of solving your problem but even if it is explained thoroughly you will have to handle some difficult things!

1.Connect to the bitcoin network directly, parse all messages, verify the blocks and insert all the transactions contained within it into the MySQL db. This is going to be a bit hard since you have to write code to connect to the network.

2.Run bitcoind side-by-side with the MySQL db and use some combination of the getblockcount, getblockhash, getblock, gettransaction RPC commands and insert the transaction data into the MySQL db. You can run a cron on this every 5 minutes or so and you should have a fairly up-to-date MySQL db. This comes at the cost of considerable disk space since you'll need to have the entire blockchain for bitcoind (30 gigs and growing).

3.Connect to some other service and get the data that way. Blockchain.info is the obvious one and they have a wonderful API for getting all the new data you need. Unfortunately, you'll have to trust them to provide the right data. There's also electrum servers which provide data using stratum and ABE and some others. Heck, if you can get someone to open up their RPC port (8333) on bitcoind, you can connect to their bitcoind and use the same commands as #2. The drawback here is that you have to trust a third party in some way, shape or form, though, of course, you can write your own verification code.

«««▬▬▬▬▬▬▬ Yo coin ▬▬▬▬▬▬▬»»»
P2P Digital Currency ◘    ◘ Maximize your YOLIFE design your own future ◘
Come join our great community on Telegram
btctousd81
Sr. Member
****
Offline Offline

Activity: 434
Merit: 258


View Profile WWW
February 03, 2018, 02:09:38 AM
 #3

There are several ways of solving your problem but even if it is explained thoroughly you will have to handle some difficult things!

1.Connect to the bitcoin network directly, parse all messages, verify the blocks and insert all the transactions contained within it into the MySQL db. This is going to be a bit hard since you have to write code to connect to the network.

2.Run bitcoind side-by-side with the MySQL db and use some combination of the getblockcount, getblockhash, getblock, gettransaction RPC commands and insert the transaction data into the MySQL db. You can run a cron on this every 5 minutes or so and you should have a fairly up-to-date MySQL db. This comes at the cost of considerable disk space since you'll need to have the entire blockchain for bitcoind (30 gigs and growing).

3.Connect to some other service and get the data that way. Blockchain.info is the obvious one and they have a wonderful API for getting all the new data you need. Unfortunately, you'll have to trust them to provide the right data. There's also electrum servers which provide data using stratum and ABE and some others. Heck, if you can get someone to open up their RPC port (8333) on bitcoind, you can connect to their bitcoind and use the same commands as #2. The drawback here is that you have to trust a third party in some way, shape or form, though, of course, you can write your own verification code.

its more like 150 gigs

Anti-Cen
Member
**
Offline Offline

Activity: 210
Merit: 26

High fees = low BTC price


View Profile
February 03, 2018, 03:59:16 PM
 #4

I know someone on-line is running a site that lest you run SQL-Server query's against the block-chain
but you have to do joins of a couple of tables but basically you type the SQL statement (I could hack him I bet)
into a text area and press run.

The results are quite fast which is a surprise to me because using varchar as foreign keys is slow and they needs
lots of memory to index them so unless he had something like 32gb of ram then I am not sure why it ran so quick.

Once your field index needs to page to files then the database runs as slow as a snail so I would be interested to
hear your findings on this project if you let me know what hardware your running because 150 or 200gb of data
is a big number for any database and I've worked on some big ones.

Good luck

Mining is CPU-wars and Intel, AMD like it nearly as much as big oil likes miners wasting electricity. Is this what mankind has come too.
Pages: [1]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!