Trying to create a personal active searchable address/balance/etc database...

[JaredKaragen]

I was thinking, and wondering how sites like blockchain.info, and blockcypher are able to retrieve balance information so readily.  The wallets only maintain information about keys or address imported to them; and each import would require a rescan.  Rescans were bad enough at 60Gb, but 300+... Geez.


I have looked and looked for different solutions, but I would like to create a database, that I can have for a test system that polls various details for deep learning correlations....   it needs to poll details such as the balance of a specific address, tx counts per address, etc... I would like this type of information kept in an easily searchable database.

I would like it to stay up to date and current just like a wallet would... it would imagine it just having to prune it's own DB with details of each block as each block is processed.





I would use blockchain.info's free api returns;  but one every minute... is... too slow.   Blockcypher charges as well;  I just want to avoid pay-for-use services.   

I would rather put up a machine with the drive space and RAM to get it done and cut my network bandwidth down at the same time.  This is the goal.



Has anyone come up for a solution to something of this nature? 

[1715463903]

1715463903

[1715463903]

1715463903

[pooya87]

have you looked around in GitHub for open source blockchain explorer projects? starting there is what i would do if i were you. for example here is one that people have been satisfied with: https://github.com/bitcoin-abe/bitcoin-abe

in any case what exactly is it that you are trying to do that requires a personal indexed blockchain because it doesn't seem like something that any normal use case would demand?

[JaredKaragen]

have you looked around in GitHub for open source blockchain explorer projects? starting there is what i would do if i were you. for example here is one that people have been satisfied with: https://github.com/bitcoin-abe/bitcoin-abe

in any case what exactly is it that you are trying to do that requires a personal indexed blockchain because it doesn't seem like something that any normal use case would demand?

it's not.... it's a deep learning data correlation project I am working on with several people....    I see patterns myself, but can't make sense of them...   Given enough data and GPU power, I think answers could be found.

We have already gotten some good results in initial testing, but the lack of being able to send ~100+ requests per minute to lets say, blockchain.info.... per machine... really slows things down by half, per machine per IP.  For these services, blockchain.info has the most open free-API data allotment;   I can get in a solid 1 request per minute if I do not stop.  If I breech that, they will roll in fast, but each IP's allotment of un-timed replies are limited.

Most projects for databasing the blockchain's contents I have seen only catalog TXIO, and if you would like to search for any specifics to an address's balance, or amount of tx's rec'd/sent from/to that address, you would have to rescan the blockchain each time individually for each known address... and, that is retarded to do.... like, seriously retarded.


I'll look into bitcoin-abe.... see if it comes close to what i'm looking for.



I know other groups have done it for other reasons.  For example there is that pool of people who solely generate pubkeys/privkeys just to see if there's a balance... and they keep a live lookup table of just pubaddress and balance to do so.