beagle1977 (OP)
Newbie
 Offline
Activity: 35
Merit: 0
|
 |
July 25, 2018, 08:05:39 AM |
|
Hi,
I am looking to set up a main BTC node and loading the data into a DB. i downloaded the node software from BTC core and have proceeded with sync..Now i have a set of files but they dont seem to be in a readable format. How do i load them into a DB? Is there a schema? I tried searching the internet but i am unable to find it. Or am i missing something?
|
|
|
|
|
|
|
|
|
|
|
|
Bitcoin mining is now a specialized and very risky industry, just like gold mining. Amateur miners are unlikely to make much money, and may even lose money. Bitcoin is much more than just mining, though!
|
|
|
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction.
|
|
|
|
|
|
|
mocacinno
Legendary
Offline
Activity: 3388
Merit: 4922
https://merel.mobi => buy facemasks with BTC/LTC
|
|
July 25, 2018, 08:15:02 AM |
|
Hi,
I am looking to set up a main BTC node and loading the data into a DB. i downloaded the node software from BTC core and have proceeded with sync..Now i have a set of files but they dont seem to be in a readable format. How do i load them into a DB? Is there a schema? I tried searching the internet but i am unable to find it. Or am i missing something?
There are several options, the easyest one is searching for an existing block parser in a language you understand, and adapting it in order to insert all data into a relational database. Do expect this to be a very time and resource consuming process... You'll need hundreds of gigabytes of diskspace, and if you mess up your relational database, inserting, deleting, updating or selecting records will be painstakingly slow. In the past, i've found a couple projects you could find interesting: https://github.com/znort987/blockparserhttps://github.com/alecalve/python-bitcoin-blockchain-parserhttps://github.com/neocogent/sqlchain |
|
|
|
beagle1977 (OP)
Newbie
Offline
Activity: 35
Merit: 0
|
|
July 25, 2018, 08:17:48 AM |
|
Hi,
I am looking to set up a main BTC node and loading the data into a DB. i downloaded the node software from BTC core and have proceeded with sync..Now i have a set of files but they dont seem to be in a readable format. How do i load them into a DB? Is there a schema? I tried searching the internet but i am unable to find it. Or am i missing something?
There are several options, the easyest one is searching for an existing block parser in a language you understand, and adapting it in order to insert all data into a relational database. Do expect this to be a very time and resource consuming process... You'll need hundreds of gigabytes of diskspace, and if you mess up your relational database, inserting, deleting, updating or selecting records will be painstakingly slow. In the past, i've found a couple projects you could find interesting: https://github.com/znort987/blockparserhttps://github.com/alecalve/python-bitcoin-blockchain-parserhttps://github.com/neocogent/sqlchainGreat! Thanks a lot. But is there a schema or something for the data? I want to load the data using java. so if i have the data structure for the data, i will probably able to write it myself? |
|
|
|
|
mocacinno
Legendary
Offline
Activity: 3388
Merit: 4922
https://merel.mobi => buy facemasks with BTC/LTC
|
|
July 25, 2018, 08:23:38 AM |
|
Hi,
I am looking to set up a main BTC node and loading the data into a DB. i downloaded the node software from BTC core and have proceeded with sync..Now i have a set of files but they dont seem to be in a readable format. How do i load them into a DB? Is there a schema? I tried searching the internet but i am unable to find it. Or am i missing something?
There are several options, the easyest one is searching for an existing block parser in a language you understand, and adapting it in order to insert all data into a relational database. Do expect this to be a very time and resource consuming process... You'll need hundreds of gigabytes of diskspace, and if you mess up your relational database, inserting, deleting, updating or selecting records will be painstakingly slow. In the past, i've found a couple projects you could find interesting: https://github.com/znort987/blockparserhttps://github.com/alecalve/python-bitcoin-blockchain-parserhttps://github.com/neocogent/sqlchainGreat! Thanks a lot. But is there a schema or something for the data? I want to load the data using java. so if i have the data structure for the data, i will probably able to write it myself? No, there's no schema, it's not a relational database. Here's an old document i found on http://codesuppository.blogspot.com/2014/01/how-to-parse-bitcoin-blockchain.htmlhttp://2.bp.blogspot.com/-DaJcdsyqQSs/UsiTXNHP-0I/AAAAAAAATC0/kiFRowh-J18/s1600/blockchain.pngHere's a link to the wiki: https://en.bitcoin.it/wiki/BlockGood luck |
|
|
|
beagle1977 (OP)
Newbie
Offline
Activity: 35
Merit: 0
|
|
July 25, 2018, 08:27:31 AM |
|
Hi,
I am looking to set up a main BTC node and loading the data into a DB. i downloaded the node software from BTC core and have proceeded with sync..Now i have a set of files but they dont seem to be in a readable format. How do i load them into a DB? Is there a schema? I tried searching the internet but i am unable to find it. Or am i missing something?
There are several options, the easyest one is searching for an existing block parser in a language you understand, and adapting it in order to insert all data into a relational database. Do expect this to be a very time and resource consuming process... You'll need hundreds of gigabytes of diskspace, and if you mess up your relational database, inserting, deleting, updating or selecting records will be painstakingly slow. In the past, i've found a couple projects you could find interesting: https://github.com/znort987/blockparserhttps://github.com/alecalve/python-bitcoin-blockchain-parserhttps://github.com/neocogent/sqlchainGreat! Thanks a lot. But is there a schema or something for the data? I want to load the data using java. so if i have the data structure for the data, i will probably able to write it myself? No, there's no schema, it's not a relational database. Here's an old document i found on http://codesuppository.blogspot.com/2014/01/how-to-parse-bitcoin-blockchain.htmlhttp://2.bp.blogspot.com/-DaJcdsyqQSs/UsiTXNHP-0I/AAAAAAAATC0/kiFRowh-J18/s1600/blockchain.pngHere's a link to the wiki: https://en.bitcoin.it/wiki/BlockGood luck Oh ok...its more complicated than i expected i guess.  |
|
|
|
|
mocacinno
Legendary
Offline
Activity: 3388
Merit: 4922
https://merel.mobi => buy facemasks with BTC/LTC
|
|
July 25, 2018, 08:29:28 AM |
|
Oh ok...its more complicated than i expected i guess. That's why i proposed to use an existing block parser and edit it to suit your needs  IMHO, usually it's better not to re-invent the wheel... But i wasn't kidding when i said this task will be very resource intensive. I tried parsing the blocks and inserting them into a simple mysql db a long, long time ago... I had to quit because the progress was so slow. |
|
|
|
beagle1977 (OP)
Newbie
Offline
Activity: 35
Merit: 0
|
|
July 25, 2018, 08:31:57 AM |
|
Oh ok...its more complicated than i expected i guess. That's why i proposed to use an existing block parser and edit it to suit your needs IMHO, usually it's better not to re-invent the wheel... yeah agreed. And You are a Legend, Mate! This is a project that i have to complete, so it is going to be very useful. Especially the links that you have shared. also, i am unable to find the software for running the node for BTC Cash. Can you help? |
|
|
|
|
mocacinno
Legendary
Offline
Activity: 3388
Merit: 4922
https://merel.mobi => buy facemasks with BTC/LTC
|
|
July 25, 2018, 08:36:35 AM |
|
Oh ok...its more complicated than i expected i guess. That's why i proposed to use an existing block parser and edit it to suit your needs IMHO, usually it's better not to re-invent the wheel... yeah agreed. And You are a Legend, Mate! This is a project that i have to complete, so it is going to be very useful. Especially the links that you have shared. also, i am unable to find the software for running the node for BTC Cash. Can you help? I have never run a BCH node on any of my machines, but if i'm not mistaking, bitcoin-abc is the reference client for bitcoin cash. I think this is the github repo: https://github.com/Bitcoin-ABC/bitcoin-abcDisclaimer: since i've never run a BCH node, i can't be 100% sure this is the correct reference client! |
|
|
|
Rath_
aka BitCryptex
Legendary
Offline
Activity: 1876
Merit: 3131
|
|
July 25, 2018, 08:38:40 AM |
|
Yes, it is. Here you can find the latest release for Linux, OSX and Windows. |
|
|
|
|
beagle1977 (OP)
Newbie
Offline
Activity: 35
Merit: 0
|
|
July 25, 2018, 08:40:43 AM |
|
Yes, it is. Here you can find the latest compiled version for Linux, OSX and Windows. oh...great ! Thanks a lot ! |
|
|
|
|
beagle1977 (OP)
Newbie
Offline
Activity: 35
Merit: 0
|
|
July 25, 2018, 08:41:58 AM |
|
and lastly, i need the client for running ETH node. can you guys help? sorry to bug you too much
but the project involves running these 3 clients and loading the data into a DB. but by the looks of it , it seems a lost cause, considering the volume !
|
|
|
|
|
mocacinno
Legendary
Offline
Activity: 3388
Merit: 4922
https://merel.mobi => buy facemasks with BTC/LTC
|
|
July 25, 2018, 08:44:11 AM |
|
and lastly, i need the client for running ETH node. can you guys help? sorry to bug you too much
but the project involves running these 3 clients and loading the data into a DB. but by the looks of it , it seems a lost cause, considering the volume !
I've never run an ether node either, but i think you need geth https://github.com/ethereum/go-ethereum/wiki/gethBut yeah, if you're going to parse the full BTC, BCH and ETH blockchain and insert the data into a relational database, you'll need loads and loads of processing power and fast storage (and a lot of it). |
|
|
|
beagle1977 (OP)
Newbie
Offline
Activity: 35
Merit: 0
|
|
July 25, 2018, 08:46:30 AM |
|
and lastly, i need the client for running ETH node. can you guys help? sorry to bug you too much
but the project involves running these 3 clients and loading the data into a DB. but by the looks of it , it seems a lost cause, considering the volume !
I've never run an ether node either, but i think you need geth https://github.com/ethereum/go-ethereum/wiki/gethBut yeah, if you're going to parse the full BTC, BCH and ETH blockchain and insert the data into a relational database, you'll need loads and loads of processing power and fast storage (and a lot of it). thanks a lot again ! i am planning to use AWS and i guess it will give me the required processing power but it is going to be very costly i guess ! |
|
|
|
|
|
