Bitcoin Forum
May 29, 2020, 11:31:22 PM *
News: Latest Bitcoin Core release: 0.19.1 [Torrent]
 
   Home   Help Search Login Register More  
Pages: [1]
  Print  
Author Topic: Goxplorer, a Bitcoin blockchain parser package written in golang  (Read 280 times)
iMil
Newbie
*
Offline Offline

Activity: 19
Merit: 55


View Profile
November 11, 2019, 02:56:16 PM
Merited by malevolent (20), Welsh (10), ETFbitcoin (7), hugeblack (3), o_e_l_e_o (2), joniboini (2), Chlotide (1)
 #1

Hi,

A couple of weeks ago I started a toy project in order to dig into blocks structure, and one thing leading to another, it evolved into a pretty usable block file parser Go package.
I am aware of btcsuite, but TBH I found its documentation really hard to read and the project way too large for the simple task I wanted to achieve, so I took the opportunity.

So here it is, I present you Goxplorer (pun intended) https://gitlab.com/iMil/goxplorer, it supports pretty much all locking mechanisms including natives and P2SH embedded SegWit ones. It has the ability to add callbacks for every parsed transaction and/or block, which makes it fun to play with.

It still needs more documentation and testing, but I thought you might find it useful.

Hope you like it,

Cheers,

iMil
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
R.I.U. iol
Newbie
*
Offline Offline

Activity: 15
Merit: 2


View Profile
November 19, 2019, 08:43:37 PM
 #2

Very nice release, a lot of ppl here were looking for block parsers  Smiley
iMil
Newbie
*
Offline Offline

Activity: 19
Merit: 55


View Profile
November 24, 2019, 06:15:55 PM
 #3

Very nice release, a lot of ppl here were looking for block parsers  Smiley

Not sure if sarcastic  Grin

Thanks anyway, and I have many things cooking as for goxplorer future, notably a huge speed boost using both goroutines and block index. Latest commits fixed various bugs and adds customizable helper functions.
iMil
Newbie
*
Offline Offline

Activity: 19
Merit: 55


View Profile
December 03, 2019, 09:56:13 AM
Last edit: December 04, 2019, 09:31:39 PM by iMil
Merited by Welsh (3), ETFbitcoin (1), hugeblack (1)
 #4

So here it is, latest version of Goxplorer has a couple of new features to help explore and discover Bitcoin's blockchain.

On the non-cosmetic side, apart from bugfixes, Goxplorer now launches block de-serialization using go routines, this simply makes a x3 performances increase.

Now on the features side:

-s permits to specify a starting block (relative, starting at 1, 0 means whole file, not block height-based)
-n can be used to only show a number of blocks
-x computes the block hash
-l uses LevelDB Block Index Record to start at a certain block hash, blkXXXXX.dat file is automatically found and loaded
-e uses LevelDB Block Index Record to start at a certain block height, blkXXXXX.dat file is automatically found and loaded
-d uses LevelDB File Information Record to load a block file from a certain date, blkXXXXX.dat file is automatically found and loaded
-tx loads and de-serialize a single transaction

"Real-life" examples are available on the project's GitLab https://gitlab.com/iMil/goxplorer

Bonus track, I couldn't find a working example of Bitcoin's LevelDB usage in golang, so I wrote blockchain/leveldb.go from scratch, it might help some Go developers.

Comments and ideas are very welcome.

Edit: added -e
Edit2: added -d
AltcoinBuilder
Copper Member
Jr. Member
*
Offline Offline

Activity: 97
Merit: 5


View Profile WWW
December 04, 2019, 03:16:30 PM
 #5

interesting. nice job!
tromp
Hero Member
*****
Offline Offline

Activity: 638
Merit: 555


View Profile
December 04, 2019, 04:45:59 PM
Merited by odolvlobo (1)
 #6

If you make it Multi Threaded, you can call it MtGoxplorer :-)
iMil
Newbie
*
Offline Offline

Activity: 19
Merit: 55


View Profile
December 04, 2019, 06:51:11 PM
 #7

If you make it Multi Threaded, you can call it MtGoxplorer :-)

Oh my. Why didn't I... and it actually *is* multithreaded!  Grin
iMil
Newbie
*
Offline Offline

Activity: 19
Merit: 55


View Profile
December 04, 2019, 07:34:11 PM
 #8

interesting. nice job!

Thanks a lot! And It's not its final form, I still have a couple of nice features in mind Wink
iMil
Newbie
*
Offline Offline

Activity: 19
Merit: 55


View Profile
April 30, 2020, 05:32:59 AM
Merited by baro77 (2)
 #9

An update, only to announce release 0.3.0: Goxplorer can now be started as a REST HTTP server and thus be a backend for any type of frontend with no other dependency than the blockchain itself.

Enjoy!
PrivatePerson
Newbie
*
Offline Offline

Activity: 30
Merit: 0


View Profile
May 06, 2020, 04:36:34 PM
 #10

1. Using it, can I parse all addresses with a positive balance from the entire blockchain to a text file?
2. Can I check the balances of a large list of addresses from a text file and write them to another text file?
iMil
Newbie
*
Offline Offline

Activity: 19
Merit: 55


View Profile
May 12, 2020, 02:31:24 PM
Last edit: May 12, 2020, 03:51:44 PM by iMil
 #11

1. Using it, can I parse all addresses with a positive balance from the entire blockchain to a text file?
2. Can I check the balances of a large list of addresses from a text file and write them to another text file?

1. At the moment, this would take an enormous amount of time, but I am working on including chainstate support, which should help achieve something like this.
2. Same answer.
iMil
Newbie
*
Offline Offline

Activity: 19
Merit: 55


View Profile
May 12, 2020, 03:18:50 PM
 #12

Release 0.5.0 is out!

Goxplorer is now able to explore blocks/index LevelDB database. This is useful when debugging and learning blockchain structure, i.e.:
Code:
$ ./goxplorer -b 1845 -ldb file
{
  "nblocks": 127,
  "nsize": 133112994,
  "nundosize": 18090258,
  "nheightfirst": 600753,
  "nheightlast": 601727,
  "ntimefirst": 1571860166,
  "ntimeLast": 1572487584
}
More to come in this direction as mentioned in the previous post, it should open the way to faster addresses search.

I also added a toy callback, a very naive bruteforce module to test blocks against dictionary files.

Documentation/homepage: https://imil.net/goxplorer/
Release: https://gitlab.com/iMil/goxplorer/-/releases/0.5.0
Source code: https://gitlab.com/iMil/goxplorer
iMil
Newbie
*
Offline Offline

Activity: 19
Merit: 55


View Profile
May 16, 2020, 08:42:30 AM
Last edit: May 17, 2020, 08:26:38 AM by iMil
 #13

1. Using it, can I parse all addresses with a positive balance from the entire blockchain to a text file?
2. Can I check the balances of a large list of addresses from a text file and write them to another text file?

Regarding the first point, you might want to have a look as goxplorer chainstate branch https://gitlab.com/iMil/goxplorer/-/tree/chainstate
I've just pushed a working version of what I hinted, you can now make a database of all addresses with a positive balance (better said, UTXOs), but it's not yet documented. The fastest way of getting this done is to 1st create the LevelDB database (can take some time depending on your hardware):

$ goxplorer -addr mkdb:anameyoulike

Then if you want to extract them to a plain text file, you can use https://github.com/yuuichi-fujioka/go-leveldbctl this way:

$ leveldbctl --dbfile=anameyoulike k > addresses.txt

The second point is on its way.


I've added a print function to ease this process in the chainstate branch. Export chainstate location and run goxplorer with the -addr print flag:

Code:
$ export BTCCHAINSTATE=chainstate # use a copy, not the original chainstate database!
$ ./goxplorer -addr print
This will output every UTXO in a JSON dict to be taken individually, an array would be too big.

Example output:
Code:
{
  "txid": "57f50996a619a65979a8adf64396e76ed26c2c643492cbb2b35414246b040f00",
  "height": 616335,
  "nsize": 0,
  "address": "19iLbtVWjer23uWR4tdw6WdxvMDZ8CeUxz",
  "amount": 133097
}
{
  "txid": "57f50996a619a65979a8adf64396e76ed26c2c643492cbb2b35414246b040f00",
  "height": 616335,
  "nsize": 0,
  "address": "1N1YgwAFgEuTZmtKaAskhQRK8Q6k9iuzKU",
  "amount": 57914
}
...

The chainstate branch is actually capable of more useful actions but I'm still working on polishing and documenting them.
iMil
Newbie
*
Offline Offline

Activity: 19
Merit: 55


View Profile
May 17, 2020, 04:45:54 PM
Last edit: May 19, 2020, 08:40:48 AM by iMil
 #14

0.6.0 is out, and with it, as promised, chainstate / UTXO support.

It is now possible to generate an address database, and use it for fast address search (UTXO only), either on the command line or via the HTTP REST/JSON server:

Code:
$ ./goxplorer -addr db:addrdb:1BVxyPYXkV5tFKWMnVoeQgUZEzkrAsDUVB|jq '.[0]'
{
  "txid": "b5ec0c758e34ff541d20cf7f43b43a748293b7e8262b845f72708323a96f1e00",
  "height": 492634,
  "nsize": 0,
  "address": "1BVxyPYXkV5tFKWMnVoeQgUZEzkrAsDUVB",
  "amount": 600
}

Note that you can search for either a full or a partial address

Code:
$ ./goxplorer -addr db:addrdb:1BVxyPYXkV5tF|jq '.[0]'
{
  "txid": "b5ec0c758e34ff541d20cf7f43b43a748293b7e8262b845f72708323a96f1e00",
  "height": 492634,
  "nsize": 0,
  "address": "1BVxyPYXkV5tFKWMnVoeQgUZEzkrAsDUVB",
  "amount": 600
}

In the same manner, you can search for a TXID:

Code:
$ ./goxplorer -addr txid:3f4ef2469c6dcb8f3255ca444d237a73c8df7f20cf6d5c3b17f59c34934d1b00|jq '.[0]'
{
  "txid": "3f4ef2469c6dcb8f3255ca444d237a73c8df7f20cf6d5c3b17f59c34934d1b00",
  "height": 425174,
  "nsize": 0,
  "address": "1FhpzVpBdmN6iLdEnzffFhBo1dgXJ5W9Ec",
  "amount": 300
}
Homepage: https://imil.net/goxplorer/
Release: https://gitlab.com/iMil/goxplorer/-/releases
Sources: https://gitlab.com/iMil/goxplorer

Edit: release 0.6.1
iMil
Newbie
*
Offline Offline

Activity: 19
Merit: 55


View Profile
Today at 04:16:50 AM
Last edit: Today at 04:50:37 AM by iMil
 #15

Version 0.8.1 has landed.

This release's main focus is HTTP mode "real life" usability.  In this mode, it is highly recommended to convert LevelDB databases (block index and chainstate) to a more concurrent-friendly database, Badger https://github.com/dgraph-io/badger. This is done in one step, using goxplorer:
Code:
$ goxplorer -addr mkdb
It takes about 30 minutes to process all of the databases, YMMV depending on your hardware.

From there on, goxplorer will use the Badger databases instead of LevelDB. Of course, it needs to be updated regularly in order to keep in sync with the ongoing blockchain, and you have to take care not to use your node real databases as LevelDB only supports one process at a time, and this could corrupt your running database, leading to a reindexation. Here's a safe way of re-syncing Badger with LevelDB databases using an overlay filesystem https://wiki.archlinux.org/index.php/Overlay_filesystem:
Code:
#!/bin/sh

PATH=${PATH}:${HOME}/bin

DESTDIR=${HOME}/api
BTCDIR=/data/.bitcoin
OVERLAY=${DESTDIR}/overlay
export BTCBLOCKSHOME=${BTCDIR}/blocks
export BTCBLOCKINDEX=${OVERLAY}/bitcoin/blocks/index
export BTCCHAINSTATE=${OVERLAY}/bitcoin/chainstate
export BTCADDRDB=${DESTDIR}/addrdb

sudo umount ${OVERLAY}/bitcoin
rm -rf ${OVERLAY}/upper/*
sudo mount ${OVERLAY}/bitcoin
leveldbctl -d ${BTCBLOCKINDEX} i
leveldbctl -d ${BTCCHAINSTATE} i

curl http://localhost:7554/mkdb/
leveldbctl https://github.com/yuuichi-fujioka/go-leveldbctl is a very handy tool I've contributed to in order to make it hexadecimal-friendly.

This script can be for example coupled with a filesystem event detection tool like inotify-wait in order to be launched on every blockchain change, and it can be safely called while goxporer is serving HTTP requests.

* Homepage: https://imil.net/goxplorer/
* Releases: https://gitlab.com/iMil/goxplorer/-/releases
* Source code: https://gitlab.com/iMil/goxplorer
Pages: [1]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!