Title: Trying to create a personal active searchable address/balance/etc database... Post by: JaredKaragen on May 01, 2020, 11:43:56 AM I was thinking, and wondering how sites like blockchain.info, and blockcypher are able to retrieve balance information so readily. The wallets only maintain information about keys or address imported to them; and each import would require a rescan. Rescans were bad enough at 60Gb, but 300+... Geez.
I have looked and looked for different solutions, but I would like to create a database, that I can have for a test system that polls various details for deep learning correlations.... it needs to poll details such as the balance of a specific address, tx counts per address, etc... I would like this type of information kept in an easily searchable database. I would like it to stay up to date and current just like a wallet would... it would imagine it just having to prune it's own DB with details of each block as each block is processed. I would use blockchain.info's free api returns; but one every minute... is... too slow. Blockcypher charges as well; I just want to avoid pay-for-use services. I would rather put up a machine with the drive space and RAM to get it done and cut my network bandwidth down at the same time. This is the goal. Has anyone come up for a solution to something of this nature? Title: Re: Trying to create a personal active searchable address/balance/etc database... Post by: pooya87 on May 01, 2020, 12:49:13 PM have you looked around in GitHub for open source blockchain explorer projects? starting there is what i would do if i were you. for example here is one that people have been satisfied with: https://github.com/bitcoin-abe/bitcoin-abe
in any case what exactly is it that you are trying to do that requires a personal indexed blockchain because it doesn't seem like something that any normal use case would demand? Title: Re: Trying to create a personal active searchable address/balance/etc database... Post by: JaredKaragen on May 01, 2020, 10:09:00 PM have you looked around in GitHub for open source blockchain explorer projects? starting there is what i would do if i were you. for example here is one that people have been satisfied with: https://github.com/bitcoin-abe/bitcoin-abe in any case what exactly is it that you are trying to do that requires a personal indexed blockchain because it doesn't seem like something that any normal use case would demand? it's not.... it's a deep learning data correlation project I am working on with several people.... I see patterns myself, but can't make sense of them... Given enough data and GPU power, I think answers could be found. We have already gotten some good results in initial testing, but the lack of being able to send ~100+ requests per minute to lets say, blockchain.info.... per machine... really slows things down by half, per machine per IP. For these services, blockchain.info has the most open free-API data allotment; I can get in a solid 1 request per minute if I do not stop. If I breech that, they will roll in fast, but each IP's allotment of un-timed replies are limited. Most projects for databasing the blockchain's contents I have seen only catalog TXIO, and if you would like to search for any specifics to an address's balance, or amount of tx's rec'd/sent from/to that address, you would have to rescan the blockchain each time individually for each known address... and, that is retarded to do.... like, seriously retarded. I'll look into bitcoin-abe.... see if it comes close to what i'm looking for. I know other groups have done it for other reasons. For example there is that pool of people who solely generate pubkeys/privkeys just to see if there's a balance... and they keep a live lookup table of just pubaddress and balance to do so. |