Goxplorer, a Bitcoin blockchain parser package written in golang

[iMil]

Hi,

A couple of weeks ago I started a toy project in order to dig into blocks structure, and one thing leading to another, it evolved into a pretty usable block file parser Go package.
I am aware of btcsuite, but TBH I found its documentation really hard to read and the project way too large for the simple task I wanted to achieve, so I took the opportunity.

So here it is, I present you Goxplorer (pun intended) https://gitlab.com/iMil/goxplorer, it supports pretty much all locking mechanisms including natives and P2SH embedded SegWit ones. It has the ability to add callbacks for every parsed transaction and/or block, which makes it fun to play with.

It still needs more documentation and testing, but I thought you might find it useful.

Hope you like it,

Cheers,

iMil

[R.I.U. iol]

Very nice release, a lot of ppl here were looking for block parsers 

[iMil]

Very nice release, a lot of ppl here were looking for block parsers 

Not sure if sarcastic 

Thanks anyway, and I have many things cooking as for goxplorer future, notably a huge speed boost using both goroutines and block index. Latest commits fixed various bugs and adds customizable helper functions.

[iMil]

So here it is, latest version of Goxplorer has a couple of new features to help explore and discover Bitcoin's blockchain.

On the non-cosmetic side, apart from bugfixes, Goxplorer now launches block de-serialization using go routines, this simply makes a x3 performances increase.

Now on the features side:

-s permits to specify a starting block (relative, starting at 1, 0 means whole file, not block height-based)
-n can be used to only show a number of blocks
-x computes the block hash
-l uses LevelDB Block Index Record to start at a certain block hash, blkXXXXX.dat file is automatically found and loaded
-e uses LevelDB Block Index Record to start at a certain block height, blkXXXXX.dat file is automatically found and loaded
-d uses LevelDB File Information Record to load a block file from a certain date, blkXXXXX.dat file is automatically found and loaded
-tx loads and de-serialize a single transaction

"Real-life" examples are available on the project's GitLab https://gitlab.com/iMil/goxplorer

Bonus track, I couldn't find a working example of Bitcoin's LevelDB usage in golang, so I wrote blockchain/leveldb.go from scratch, it might help some Go developers.

Comments and ideas are very welcome.

Edit: added -e
Edit2: added -d

[AltcoinBuilder]

interesting. nice job!

[tromp]

If you make it Multi Threaded, you can call it MtGoxplorer :-)

[iMil]

If you make it Multi Threaded, you can call it MtGoxplorer :-)

Oh my. Why didn't I... and it actually *is* multithreaded! 

[iMil]

interesting. nice job!

Thanks a lot! And It's not its final form, I still have a couple of nice features in mind

[iMil]

An update, only to announce release 0.3.0: Goxplorer can now be started as a REST HTTP server and thus be a backend for any type of frontend with no other dependency than the blockchain itself.

Enjoy!

[PrivatePerson]

1. Using it, can I parse all addresses with a positive balance from the entire blockchain to a text file?
2. Can I check the balances of a large list of addresses from a text file and write them to another text file?

[iMil]

1. Using it, can I parse all addresses with a positive balance from the entire blockchain to a text file?
2. Can I check the balances of a large list of addresses from a text file and write them to another text file?

1. At the moment, this would take an enormous amount of time, but I am working on including chainstate support, which should help achieve something like this.
2. Same answer.

[iMil]

Release 0.5.0 is out!

Goxplorer is now able to explore blocks/index LevelDB database. This is useful when debugging and learning blockchain structure, i.e.:

Code:

$ ./goxplorer -b 1845 -ldb file
{
  "nblocks": 127,
  "nsize": 133112994,
  "nundosize": 18090258,
  "nheightfirst": 600753,
  "nheightlast": 601727,
  "ntimefirst": 1571860166,
  "ntimeLast": 1572487584
}

More to come in this direction as mentioned in the previous post, it should open the way to faster addresses search.

I also added a toy callback, a very naive bruteforce module to test blocks against dictionary files.

Documentation/homepage: https://imil.net/goxplorer/
Release: https://gitlab.com/iMil/goxplorer/-/releases/0.5.0
Source code: https://gitlab.com/iMil/goxplorer

[iMil]

1. Using it, can I parse all addresses with a positive balance from the entire blockchain to a text file?
2. Can I check the balances of a large list of addresses from a text file and write them to another text file?

Regarding the first point, you might want to have a look as goxplorer chainstate branch https://gitlab.com/iMil/goxplorer/-/tree/chainstate
I've just pushed a working version of what I hinted, you can now make a database of all addresses with a positive balance (better said, UTXOs), but it's not yet documented. The fastest way of getting this done is to 1st create the LevelDB database (can take some time depending on your hardware):

$ goxplorer -addr mkdb:anameyoulike

Then if you want to extract them to a plain text file, you can use https://github.com/yuuichi-fujioka/go-leveldbctl this way:

$ leveldbctl --dbfile=anameyoulike k > addresses.txt

The second point is on its way.

I've added a print function to ease this process in the chainstate branch. Export chainstate location and run goxplorer with the -addr print flag:

Code:

$ export BTCCHAINSTATE=chainstate # use a copy, not the original chainstate database!
$ ./goxplorer -addr print

This will output every UTXO in a JSON dict to be taken individually, an array would be too big.

Example output:

Code:

{
  "txid": "57f50996a619a65979a8adf64396e76ed26c2c643492cbb2b35414246b040f00",
  "height": 616335,
  "nsize": 0,
  "address": "19iLbtVWjer23uWR4tdw6WdxvMDZ8CeUxz",
  "amount": 133097
}
{
  "txid": "57f50996a619a65979a8adf64396e76ed26c2c643492cbb2b35414246b040f00",
  "height": 616335,
  "nsize": 0,
  "address": "1N1YgwAFgEuTZmtKaAskhQRK8Q6k9iuzKU",
  "amount": 57914
}
...

The chainstate branch is actually capable of more useful actions but I'm still working on polishing and documenting them.

[iMil]

0.6.0 is out, and with it, as promised, chainstate / UTXO support.

It is now possible to generate an address database, and use it for fast address search (UTXO only), either on the command line or via the HTTP REST/JSON server:

Code:

$ ./goxplorer -addr db:addrdb:1BVxyPYXkV5tFKWMnVoeQgUZEzkrAsDUVB|jq '.[0]'
{
  "txid": "b5ec0c758e34ff541d20cf7f43b43a748293b7e8262b845f72708323a96f1e00",
  "height": 492634,
  "nsize": 0,
  "address": "1BVxyPYXkV5tFKWMnVoeQgUZEzkrAsDUVB",
  "amount": 600
}

Note that you can search for either a full or a partial address

Code:

$ ./goxplorer -addr db:addrdb:1BVxyPYXkV5tF|jq '.[0]'
{
  "txid": "b5ec0c758e34ff541d20cf7f43b43a748293b7e8262b845f72708323a96f1e00",
  "height": 492634,
  "nsize": 0,
  "address": "1BVxyPYXkV5tFKWMnVoeQgUZEzkrAsDUVB",
  "amount": 600
}

In the same manner, you can search for a TXID:

Code:

$ ./goxplorer -addr txid:3f4ef2469c6dcb8f3255ca444d237a73c8df7f20cf6d5c3b17f59c34934d1b00|jq '.[0]'
{
  "txid": "3f4ef2469c6dcb8f3255ca444d237a73c8df7f20cf6d5c3b17f59c34934d1b00",
  "height": 425174,
  "nsize": 0,
  "address": "1FhpzVpBdmN6iLdEnzffFhBo1dgXJ5W9Ec",
  "amount": 300
}

Homepage: https://imil.net/goxplorer/
Release: https://gitlab.com/iMil/goxplorer/-/releases
Sources: https://gitlab.com/iMil/goxplorer

Edit: release 0.6.1

[iMil]

Version 0.8.1 has landed.

This release's main focus is HTTP mode "real life" usability.  In this mode, it is highly recommended to convert LevelDB databases (block index and chainstate) to a more concurrent-friendly database, Badger https://github.com/dgraph-io/badger. This is done in one step, using goxplorer:

Code:

$ goxplorer -addr mkdb

It takes about 30 minutes to process all of the databases, YMMV depending on your hardware.

From there on, goxplorer will use the Badger databases instead of LevelDB. Of course, it needs to be updated regularly in order to keep in sync with the ongoing blockchain, and you have to take care not to use your node real databases as LevelDB only supports one process at a time, and this could corrupt your running database, leading to a reindexation. Here's a safe way of re-syncing Badger with LevelDB databases using an overlay filesystem https://wiki.archlinux.org/index.php/Overlay_filesystem:

Code:

#!/bin/sh

PATH=${PATH}:${HOME}/bin

DESTDIR=${HOME}/api
BTCDIR=/data/.bitcoin
OVERLAY=${DESTDIR}/overlay
export BTCBLOCKSHOME=${BTCDIR}/blocks
export BTCBLOCKINDEX=${OVERLAY}/bitcoin/blocks/index
export BTCCHAINSTATE=${OVERLAY}/bitcoin/chainstate
export BTCADDRDB=${DESTDIR}/addrdb

sudo umount ${OVERLAY}/bitcoin
rm -rf ${OVERLAY}/upper/*
sudo mount ${OVERLAY}/bitcoin
leveldbctl -d ${BTCBLOCKINDEX} i
leveldbctl -d ${BTCCHAINSTATE} i

curl http://localhost:7554/mkdb/

leveldbctl https://github.com/yuuichi-fujioka/go-leveldbctl is a very handy tool I've contributed to in order to make it hexadecimal-friendly.

This script can be for example coupled with a filesystem event detection tool like inotify-wait in order to be launched on every blockchain change, and it can be safely called while goxporer is serving HTTP requests.

* Homepage: https://imil.net/goxplorer/
* Releases: https://gitlab.com/iMil/goxplorer/-/releases
* Source code: https://gitlab.com/iMil/goxplorer