|
Title: Goxplorer, a Bitcoin blockchain parser package written in golang Post by: iMil on November 11, 2019, 02:56:16 PM Hi,
A couple of weeks ago I started a toy project in order to dig into blocks structure, and one thing leading to another, it evolved into a pretty usable block file parser Go package. I am aware of btcsuite, but TBH I found its documentation really hard to read and the project way too large for the simple task I wanted to achieve, so I took the opportunity. So here it is, I present you Goxplorer (pun intended) https://gitlab.com/iMil/goxplorer (https://gitlab.com/iMil/goxplorer), it supports pretty much all locking mechanisms including natives and P2SH embedded SegWit ones. It has the ability to add callbacks for every parsed transaction and/or block, which makes it fun to play with. It still needs more documentation and testing, but I thought you might find it useful. Hope you like it, Cheers, iMil Title: Re: Goxplorer, a Bitcoin blockchain parser package written in golang Post by: R.I.U. iol on November 19, 2019, 08:43:37 PM Very nice release, a lot of ppl here were looking for block parsers :)
Title: Re: Goxplorer, a Bitcoin blockchain parser package written in golang Post by: iMil on November 24, 2019, 06:15:55 PM Very nice release, a lot of ppl here were looking for block parsers :) Not sure if sarcastic ;D Thanks anyway, and I have many things cooking as for goxplorer future, notably a huge speed boost using both goroutines and block index. Latest commits fixed various bugs and adds customizable helper functions. Title: Re: Goxplorer, a Bitcoin blockchain parser package written in golang Post by: iMil on December 03, 2019, 09:56:13 AM So here it is, latest version of Goxplorer has a couple of new features to help explore and discover Bitcoin's blockchain.
On the non-cosmetic side, apart from bugfixes, Goxplorer now launches block de-serialization using go routines, this simply makes a x3 performances increase. Now on the features side: -s permits to specify a starting block (relative, starting at 1, 0 means whole file, not block height-based) -n can be used to only show a number of blocks -x computes the block hash -l uses LevelDB Block Index Record to start at a certain block hash, blkXXXXX.dat file is automatically found and loaded -e uses LevelDB Block Index Record to start at a certain block height, blkXXXXX.dat file is automatically found and loaded -d uses LevelDB File Information Record to load a block file from a certain date, blkXXXXX.dat file is automatically found and loaded -tx loads and de-serialize a single transaction "Real-life" examples are available on the project's GitLab https://gitlab.com/iMil/goxplorer (https://gitlab.com/iMil/goxplorer) Bonus track, I couldn't find a working example of Bitcoin's LevelDB usage in golang, so I wrote blockchain/leveldb.go from scratch, it might help some Go developers. Comments and ideas are very welcome. Edit: added -e Edit2: added -d Title: Re: Goxplorer, a Bitcoin blockchain parser package written in golang Post by: AltcoinBuilder on December 04, 2019, 03:16:30 PM interesting. nice job!
Title: Re: Goxplorer, a Bitcoin blockchain parser package written in golang Post by: tromp on December 04, 2019, 04:45:59 PM If you make it Multi Threaded, you can call it MtGoxplorer :-)
Title: Re: Goxplorer, a Bitcoin blockchain parser package written in golang Post by: iMil on December 04, 2019, 06:51:11 PM If you make it Multi Threaded, you can call it MtGoxplorer :-) Oh my. Why didn't I... and it actually *is* multithreaded! ;D Title: Re: Goxplorer, a Bitcoin blockchain parser package written in golang Post by: iMil on December 04, 2019, 07:34:11 PM interesting. nice job! Thanks a lot! And It's not its final form, I still have a couple of nice features in mind ;) Title: Re: Goxplorer, a Bitcoin blockchain parser package written in golang Post by: iMil on April 30, 2020, 05:32:59 AM An update, only to announce release 0.3.0: Goxplorer can now be started as a REST HTTP server and thus be a backend for any type of frontend with no other dependency than the blockchain itself.
Enjoy! Title: Re: Goxplorer, a Bitcoin blockchain parser package written in golang Post by: PrivatePerson on May 06, 2020, 04:36:34 PM 1. Using it, can I parse all addresses with a positive balance from the entire blockchain to a text file?
2. Can I check the balances of a large list of addresses from a text file and write them to another text file? Title: Re: Goxplorer, a Bitcoin blockchain parser package written in golang Post by: iMil on May 12, 2020, 02:31:24 PM 1. Using it, can I parse all addresses with a positive balance from the entire blockchain to a text file? 2. Can I check the balances of a large list of addresses from a text file and write them to another text file? 1. At the moment, this would take an enormous amount of time, but I am working on including chainstate support, which should help achieve something like this. 2. Same answer. Title: Re: Goxplorer, a Bitcoin blockchain parser package written in golang Post by: iMil on May 12, 2020, 03:18:50 PM Release 0.5.0 is out!
Goxplorer is now able to explore blocks/index LevelDB database. This is useful when debugging and learning blockchain structure, i.e.: Code: $ ./goxplorer -b 1845 -ldb file I also added a toy callback, a very naive bruteforce module to test blocks against dictionary files. Documentation/homepage: https://imil.net/goxplorer/ Release: https://gitlab.com/iMil/goxplorer/-/releases/0.5.0 Source code: https://gitlab.com/iMil/goxplorer Title: Re: Goxplorer, a Bitcoin blockchain parser package written in golang Post by: iMil on May 16, 2020, 08:42:30 AM 1. Using it, can I parse all addresses with a positive balance from the entire blockchain to a text file? 2. Can I check the balances of a large list of addresses from a text file and write them to another text file? I've just pushed a working version of what I hinted, you can now make a database of all addresses with a positive balance (better said, UTXOs), but it's not yet documented. The fastest way of getting this done is to 1st create the LevelDB database (can take some time depending on your hardware): $ goxplorer -addr mkdb:anameyoulike Then if you want to extract them to a plain text file, you can use https://github.com/yuuichi-fujioka/go-leveldbctl (https://github.com/yuuichi-fujioka/go-leveldbctl) this way: $ leveldbctl --dbfile=anameyoulike k > addresses.txt The second point is on its way. I've added a print function to ease this process in the chainstate branch. Export chainstate location and run goxplorer with the -addr print flag: Code: $ export BTCCHAINSTATE=chainstate # use a copy, not the original chainstate database! Example output: Code: {The chainstate branch is actually capable of more useful actions but I'm still working on polishing and documenting them. Title: Re: Goxplorer, a Bitcoin blockchain parser package written in golang Post by: iMil on May 17, 2020, 04:45:54 PM 0.6.0 is out, and with it, as promised, chainstate / UTXO support.
It is now possible to generate an address database, and use it for fast address search (UTXO only), either on the command line or via the HTTP REST/JSON server: Code: $ ./goxplorer -addr db:addrdb:1BVxyPYXkV5tFKWMnVoeQgUZEzkrAsDUVB|jq '.[0]' Note that you can search for either a full or a partial address Code: $ ./goxplorer -addr db:addrdb:1BVxyPYXkV5tF|jq '.[0]' In the same manner, you can search for a TXID: Code: $ ./goxplorer -addr txid:3f4ef2469c6dcb8f3255ca444d237a73c8df7f20cf6d5c3b17f59c34934d1b00|jq '.[0]' Release: https://gitlab.com/iMil/goxplorer/-/releases Sources: https://gitlab.com/iMil/goxplorer Edit: release 0.6.1 Title: Re: Goxplorer, a Bitcoin blockchain parser package written in golang Post by: iMil on May 29, 2020, 04:16:50 AM Version 0.8.1 has landed.
This release's main focus is HTTP mode "real life" usability. In this mode, it is highly recommended to convert LevelDB databases (block index and chainstate) to a more concurrent-friendly database, Badger https://github.com/dgraph-io/badger (https://github.com/dgraph-io/badger). This is done in one step, using goxplorer: Code: $ goxplorer -addr mkdb From there on, goxplorer will use the Badger databases instead of LevelDB. Of course, it needs to be updated regularly in order to keep in sync with the ongoing blockchain, and you have to take care not to use your node real databases as LevelDB only supports one process at a time, and this could corrupt your running database, leading to a reindexation. Here's a safe way of re-syncing Badger with LevelDB databases using an overlay filesystem https://wiki.archlinux.org/index.php/Overlay_filesystem (https://wiki.archlinux.org/index.php/Overlay_filesystem): Code: #!/bin/sh This script can be for example coupled with a filesystem event detection tool like inotify-wait in order to be launched on every blockchain change, and it can be safely called while goxporer is serving HTTP requests. * Homepage: https://imil.net/goxplorer/ * Releases: https://gitlab.com/iMil/goxplorer/-/releases * Source code: https://gitlab.com/iMil/goxplorer |