iMil (OP)
Newbie
Offline
Activity: 21
Merit: 74
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
November 11, 2019, 02:56:16 PM |
|
Hi, A couple of weeks ago I started a toy project in order to dig into blocks structure, and one thing leading to another, it evolved into a pretty usable block file parser Go package. I am aware of btcsuite, but TBH I found its documentation really hard to read and the project way too large for the simple task I wanted to achieve, so I took the opportunity. So here it is, I present you Goxplorer (pun intended) https://gitlab.com/iMil/goxplorer, it supports pretty much all locking mechanisms including natives and P2SH embedded SegWit ones. It has the ability to add callbacks for every parsed transaction and/or block, which makes it fun to play with. It still needs more documentation and testing, but I thought you might find it useful. Hope you like it, Cheers, iMil
|
|
|
|
R.I.U. iol
Newbie
Offline
Activity: 15
Merit: 2
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
November 19, 2019, 08:43:37 PM |
|
Very nice release, a lot of ppl here were looking for block parsers ![Smiley](https://bitcointalk.org/Smileys/default/smiley.gif)
|
|
|
|
iMil (OP)
Newbie
Offline
Activity: 21
Merit: 74
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
November 24, 2019, 06:15:55 PM |
|
Very nice release, a lot of ppl here were looking for block parsers ![Smiley](https://bitcointalk.org/Smileys/default/smiley.gif) Not sure if sarcastic ![Grin](https://bitcointalk.org/Smileys/default/grin.gif) Thanks anyway, and I have many things cooking as for goxplorer future, notably a huge speed boost using both goroutines and block index. Latest commits fixed various bugs and adds customizable helper functions.
|
|
|
|
iMil (OP)
Newbie
Offline
Activity: 21
Merit: 74
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
December 03, 2019, 09:56:13 AM Last edit: December 04, 2019, 09:31:39 PM by iMil Merited by Welsh (3), ABCbits (1), hugeblack (1) |
|
So here it is, latest version of Goxplorer has a couple of new features to help explore and discover Bitcoin's blockchain. On the non-cosmetic side, apart from bugfixes, Goxplorer now launches block de-serialization using go routines, this simply makes a x3 performances increase. Now on the features side: -s permits to specify a starting block (relative, starting at 1, 0 means whole file, not block height-based) -n can be used to only show a number of blocks -x computes the block hash -l uses LevelDB Block Index Record to start at a certain block hash, blkXXXXX.dat file is automatically found and loaded -e uses LevelDB Block Index Record to start at a certain block height, blkXXXXX.dat file is automatically found and loaded -d uses LevelDB File Information Record to load a block file from a certain date, blkXXXXX.dat file is automatically found and loaded -tx loads and de-serialize a single transaction "Real-life" examples are available on the project's GitLab https://gitlab.com/iMil/goxplorerBonus track, I couldn't find a working example of Bitcoin's LevelDB usage in golang, so I wrote blockchain/leveldb.go from scratch, it might help some Go developers. Comments and ideas are very welcome. Edit: added -eEdit2: added -d
|
|
|
|
AltcoinBuilder
Copper Member
Jr. Member
Offline
Activity: 85
Merit: 5
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
December 04, 2019, 03:16:30 PM |
|
interesting. nice job!
|
|
|
|
tromp
Legendary
Offline
Activity: 989
Merit: 1108
|
If you make it Multi Threaded, you can call it MtGoxplorer :-)
|
|
|
|
iMil (OP)
Newbie
Offline
Activity: 21
Merit: 74
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
December 04, 2019, 06:51:11 PM |
|
If you make it Multi Threaded, you can call it MtGoxplorer :-)
Oh my. Why didn't I... and it actually *is* multithreaded! ![Grin](https://bitcointalk.org/Smileys/default/grin.gif)
|
|
|
|
iMil (OP)
Newbie
Offline
Activity: 21
Merit: 74
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
December 04, 2019, 07:34:11 PM |
|
interesting. nice job!
Thanks a lot! And It's not its final form, I still have a couple of nice features in mind
|
|
|
|
iMil (OP)
Newbie
Offline
Activity: 21
Merit: 74
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
April 30, 2020, 05:32:59 AM |
|
An update, only to announce release 0.3.0: Goxplorer can now be started as a REST HTTP server and thus be a backend for any type of frontend with no other dependency than the blockchain itself.
Enjoy!
|
|
|
|
PrivatePerson
Member
![*](https://bitcointalk.org/Themes/custom1/images/star.gif)
Offline
Activity: 174
Merit: 12
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
May 06, 2020, 04:36:34 PM |
|
1. Using it, can I parse all addresses with a positive balance from the entire blockchain to a text file? 2. Can I check the balances of a large list of addresses from a text file and write them to another text file?
|
|
|
|
iMil (OP)
Newbie
Offline
Activity: 21
Merit: 74
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
May 12, 2020, 02:31:24 PM Last edit: May 12, 2020, 03:51:44 PM by iMil |
|
1. Using it, can I parse all addresses with a positive balance from the entire blockchain to a text file? 2. Can I check the balances of a large list of addresses from a text file and write them to another text file?
1. At the moment, this would take an enormous amount of time, but I am working on including chainstate support, which should help achieve something like this. 2. Same answer.
|
|
|
|
iMil (OP)
Newbie
Offline
Activity: 21
Merit: 74
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
May 12, 2020, 03:18:50 PM |
|
Release 0.5.0 is out! Goxplorer is now able to explore blocks/index LevelDB database. This is useful when debugging and learning blockchain structure, i.e.: $ ./goxplorer -b 1845 -ldb file { "nblocks": 127, "nsize": 133112994, "nundosize": 18090258, "nheightfirst": 600753, "nheightlast": 601727, "ntimefirst": 1571860166, "ntimeLast": 1572487584 }
More to come in this direction as mentioned in the previous post, it should open the way to faster addresses search. I also added a toy callback, a very naive bruteforce module to test blocks against dictionary files. Documentation/homepage: https://imil.net/goxplorer/Release: https://gitlab.com/iMil/goxplorer/-/releases/0.5.0Source code: https://gitlab.com/iMil/goxplorer
|
|
|
|
iMil (OP)
Newbie
Offline
Activity: 21
Merit: 74
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
May 16, 2020, 08:42:30 AM Last edit: May 17, 2020, 08:26:38 AM by iMil |
|
1. Using it, can I parse all addresses with a positive balance from the entire blockchain to a text file? 2. Can I check the balances of a large list of addresses from a text file and write them to another text file?
Regarding the first point, you might want to have a look as goxplorer chainstate branch https://gitlab.com/iMil/goxplorer/-/tree/chainstate I've just pushed a working version of what I hinted, you can now make a database of all addresses with a positive balance (better said, UTXOs), but it's not yet documented. The fastest way of getting this done is to 1st create the LevelDB database (can take some time depending on your hardware):
$ goxplorer -addr mkdb:anameyoulike
Then if you want to extract them to a plain text file, you can use https://github.com/yuuichi-fujioka/go-leveldbctl this way:
$ leveldbctl --dbfile=anameyoulike k > addresses.txt
The second point is on its way.I've added a print function to ease this process in the chainstate branch. Export chainstate location and run goxplorer with the -addr print flag: $ export BTCCHAINSTATE=chainstate # use a copy, not the original chainstate database! $ ./goxplorer -addr print
This will output every UTXO in a JSON dict to be taken individually, an array would be too big. Example output: { "txid": "57f50996a619a65979a8adf64396e76ed26c2c643492cbb2b35414246b040f00", "height": 616335, "nsize": 0, "address": "19iLbtVWjer23uWR4tdw6WdxvMDZ8CeUxz", "amount": 133097 } { "txid": "57f50996a619a65979a8adf64396e76ed26c2c643492cbb2b35414246b040f00", "height": 616335, "nsize": 0, "address": "1N1YgwAFgEuTZmtKaAskhQRK8Q6k9iuzKU", "amount": 57914 } ...
The chainstate branch is actually capable of more useful actions but I'm still working on polishing and documenting them.
|
|
|
|
iMil (OP)
Newbie
Offline
Activity: 21
Merit: 74
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
May 17, 2020, 04:45:54 PM Last edit: May 19, 2020, 08:40:48 AM by iMil |
|
0.6.0 is out, and with it, as promised, chainstate / UTXO support. It is now possible to generate an address database, and use it for fast address search (UTXO only), either on the command line or via the HTTP REST/JSON server: $ ./goxplorer -addr db:addrdb:1BVxyPYXkV5tFKWMnVoeQgUZEzkrAsDUVB|jq '.[0]' { "txid": "b5ec0c758e34ff541d20cf7f43b43a748293b7e8262b845f72708323a96f1e00", "height": 492634, "nsize": 0, "address": "1BVxyPYXkV5tFKWMnVoeQgUZEzkrAsDUVB", "amount": 600 }
Note that you can search for either a full or a partial address $ ./goxplorer -addr db:addrdb:1BVxyPYXkV5tF|jq '.[0]' { "txid": "b5ec0c758e34ff541d20cf7f43b43a748293b7e8262b845f72708323a96f1e00", "height": 492634, "nsize": 0, "address": "1BVxyPYXkV5tFKWMnVoeQgUZEzkrAsDUVB", "amount": 600 }
In the same manner, you can search for a TXID: $ ./goxplorer -addr txid:3f4ef2469c6dcb8f3255ca444d237a73c8df7f20cf6d5c3b17f59c34934d1b00|jq '.[0]' { "txid": "3f4ef2469c6dcb8f3255ca444d237a73c8df7f20cf6d5c3b17f59c34934d1b00", "height": 425174, "nsize": 0, "address": "1FhpzVpBdmN6iLdEnzffFhBo1dgXJ5W9Ec", "amount": 300 }
Homepage: https://imil.net/goxplorer/Release: https://gitlab.com/iMil/goxplorer/-/releasesSources: https://gitlab.com/iMil/goxplorerEdit: release 0.6.1
|
|
|
|
iMil (OP)
Newbie
Offline
Activity: 21
Merit: 74
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
May 29, 2020, 04:16:50 AM Last edit: May 29, 2020, 04:50:37 AM by iMil |
|
Version 0.8.1 has landed. This release's main focus is HTTP mode "real life" usability. In this mode, it is highly recommended to convert LevelDB databases (block index and chainstate) to a more concurrent-friendly database, Badger https://github.com/dgraph-io/badger. This is done in one step, using goxplorer: It takes about 30 minutes to process all of the databases, YMMV depending on your hardware. From there on, goxplorer will use the Badger databases instead of LevelDB. Of course, it needs to be updated regularly in order to keep in sync with the ongoing blockchain, and you have to take care not to use your node real databases as LevelDB only supports one process at a time, and this could corrupt your running database, leading to a reindexation. Here's a safe way of re-syncing Badger with LevelDB databases using an overlay filesystem https://wiki.archlinux.org/index.php/Overlay_filesystem: #!/bin/sh
PATH=${PATH}:${HOME}/bin
DESTDIR=${HOME}/api BTCDIR=/data/.bitcoin OVERLAY=${DESTDIR}/overlay export BTCBLOCKSHOME=${BTCDIR}/blocks export BTCBLOCKINDEX=${OVERLAY}/bitcoin/blocks/index export BTCCHAINSTATE=${OVERLAY}/bitcoin/chainstate export BTCADDRDB=${DESTDIR}/addrdb
sudo umount ${OVERLAY}/bitcoin rm -rf ${OVERLAY}/upper/* sudo mount ${OVERLAY}/bitcoin leveldbctl -d ${BTCBLOCKINDEX} i leveldbctl -d ${BTCCHAINSTATE} i
curl http://localhost:7554/mkdb/
leveldbctl https://github.com/yuuichi-fujioka/go-leveldbctl is a very handy tool I've contributed to in order to make it hexadecimal-friendly. This script can be for example coupled with a filesystem event detection tool like inotify-wait in order to be launched on every blockchain change, and it can be safely called while goxporer is serving HTTP requests. * Homepage: https://imil.net/goxplorer/* Releases: https://gitlab.com/iMil/goxplorer/-/releases* Source code: https://gitlab.com/iMil/goxplorer
|
|
|
|
|