Complete chainstate export from 2018-10-15 as torrent.

[hackbyte]

Heya people,


from this post: https://bitcointalk.org/index.php?topic=267618.0 i've seen that there might be some interest in some chainstate export which includes any bitcoin utxo address and it's particular value.

Well, here you go:

magnet:?xt=urn:btih:d9a56bd02a0f9c50462b0fab32ed3223d36ff182&dn=chainstate-201810150125CEST.7z

Code:

fornax|hackbyte|01:06:56|/currencies/src/chainstate|0$ time ./chainstate >chainstate.output 2>chainstate.errors

real    17m49.092s
user    13m47.253s
sys     2m49.484s
fornax|hackbyte|01:25:15|/currencies/src/chainstate|0$ 7z a chainstate-201810150125CEST.7z chainstate.output 

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,6 CPUs AMD FX(tm)-6300 Six-Core Processor              (600F20),ASM,AES-NI)

Scanning the drive:
1 file, 5478234093 bytes (5225 MiB)

Creating archive: chainstate-201810150125CEST.7z

Items to compress: 1

                          
Files read from disk: 1
Archive size: 2261586261 bytes (2157 MiB)
Everything is Ok
fornax|hackbyte|01:43:40|/currencies/src/chainstate|0$ ls -lah chainstate-201810150125CEST.7z 
-rw-r--r-- 1 hackbyte hackbyte 2.2G Oct 15 01:43 chainstate-201810150125CEST.7z

I will host this one for some few weeks and will have a somewhat capable seed host on it too.

Have fun..

Oh, how i got it?

Using https://github.com/mycroft/chainstate and my very own full blown bitcoin knots node...

Regards,

hacky

[almightyruler]

Hmm, interesting. I've been doing it (very slowly) the naiive way, by querying everything through RPC calls to the Bitcoin Core client. Should be done by next month if I'm lucky. 

Based on the github page for chainstate, the bulk of the output format would be:   <txid which created unspent output>;<address>;<amount>

Is this correct? If not, please post a small sample of the output.

[aliashraf]

Although I merrited this topic and OP's commitment deserves more appreciation imo, a more general approach to this problem would be more appropriate.

We have UTXO commitment proposals/implementations trending right now with their most important application being pruning and having full nodes with minimum resources and fast bootstrapping, etc. @spartacusrex started a topic in btctalk few days ago for instance and Paul Sztorc's first version of Drivechain released even later (couple of days ago) has already, implemented UTXO snapshotting.

Two most important differences:
1- UTXO snapshots include output scripts as well.
2- A Merkle Tree like  and its root hash is generated and maintained for all UTXOs  such that commitment to the state at a given height is feasible. 

We got advanced algorithms/data structures like MMR and a complete history of interest in UTXOs.

[hackbyte]

Update; Had to set up some portforwarding and added 2 trackers for the torrent..

[hackbyte]

So, how do we use the parsed chainstate? Do we use https://github.com/mycroft/chainstate and follow the instruction on "How to Run" section?

Well, i don't know how you want or can use it at all. For me, i just wrote some bits python down to write every address i find in the original csv in to a database table..

Code:

#!/usr/bin/python
import sys, time, psycopg2

# chainstate output
sourcefile = "chainstate.output"

# oeffne chainstate export
filehandle = open(sourcefile, 'r')

# open database connection
dbcon = psycopg2.connect('host=** dbname=** user=** password=**')
dbcur = dbcon.cursor()

counter=1
tablename="btc_address"
fieldname="address"

for line in filehandle:

	# read data from csv line
	txid, dunno, address, value = line.strip().split(';')

	# put it into the db, don't ask, just try and skip if it fails
	try:
		dbcur.execute("INSERT INTO " + tablename + " (" + fieldname + ") VALUES ('"+str(address)+"')")
		dbcon.commit()
		print("\033[K %08d written %s\n" % (counter,address), end='', flush=True)
		counter += 1
		continue
	except:
		print("\033[K %08d skipped %s\r" % (counter,address), end='', flush=True)
		counter += 1
		continue

print("\n")

So, whatever you want to do, just have fun. I'm just playing around for educational reasons.

Oh btw, the format of the chainstate.output file is as follows (first 5 lines):

Code:

questron|hackbyte|18:01:48|~| 0> head -n 5 chainstate.output 
033e83e3204b0cc28724e147f6fd140529b2537249f9c61c9de9972750030000;0;1KaPHfvVWNZADup3Yc26SfVdkTDvvHySVX;65279
4aa1defd46f10c27ff4d6c2565e1e0e611429cd673b0f7342f30820690030000;0;3PPqDTQgndCHiDxqeB3zayJxXB6UeKAiod;996900
b58cf0f52c86022b046838472fc308cb55c9756071399840ad38de8166040000;0;1NXznFqccAinm9RLEf9eTtppZG9tWJQp8M;259531
e1c9467a885a156e56a29d9c854e65674d581ad75611b02290454b4862060000;1;1LpCmEejWLNfZigApMPwUY9nZTS8NTJCNS;9466355
a1f28c43f1f3d4821d0db42707737ea90616613099234f905dfc6ae2b4060000;1;1FuphZ7xVPGrxthQT1S8X7nNQNByYxAT3V;339500

Have fun

Hacky

[hackbyte]

Hmm, interesting. I've been doing it (very slowly) the naiive way, by querying everything through RPC calls to the Bitcoin Core client. Should be done by next month if I'm lucky.  

Based on the github page for chainstate, the bulk of the output format would be:   <txid which created unspent output>;<address>;<amount>

Is this correct? If not, please post a small sample of the output.

Yeah, that's probably what i will do as next step. I found 'bitcoin-abe' over on git which should be able to read blkNNNN.dat files natively and put them into a SQL db for use as blockchain explorer like thing. But sadly the code is somewhat old, won't run on python3 and .. yeah, overall, it's old.. ( you can find it here: https://github.com/bitcoin-abe/bitcoin-abe or using 'pip install abe')

My "problem" with bitcoin-abe is, it wants to run directly on the blockchain .dat files, which means, i should shut down my bitcoin knots node as long as abe is running on the files .... or "just copy the whole blockchain over to somewhere else" .. mhmm

I'm not happy doing that anyways, so .. yeah ... connecting the bitcoind and reading anything i want over the RPC thing might be not really fast but more convenient to me.

The sample of the output is in the posting above this one.

Have fun,

hacky

[hackbyte]

Although I merrited this topic and OP's commitment deserves more appreciation imo, a more general approach to this problem would be more appropriate.

Thanks for the merit.

We have UTXO commitment proposals/implementations trending right now with their most important application being pruning and having full nodes with minimum resources and fast bootstrapping, etc. @spartacusrex started a topic in btctalk few days ago for instance and Paul Sztorc's first version of Drivechain released even later (couple of days ago) has already, implemented UTXO snapshotting.

Two most important differences:
1- UTXO snapshots include output scripts as well.
2- A Merkle Tree like  and its root hash is generated and maintained for all UTXOs  such that commitment to the state at a given height is feasible. 

We got advanced algorithms/data structures like MMR and a complete history of interest in UTXOs.

Hhhmm.. interesting... I'm just "for fun" on this topic whilst doing completely unrelated stuff anyways. But i will look in to the links you mentioned. After all, i still have the full node running here anyways and want to play a bit even with the json-rpc and blockchain data format stuff anyways... So maybe i can dig a bit deeper in this chainstate export code to get the output-scripts too.

By the way. On a different run, i just took a list of all my addresses and got any data which is related to it from blockchain.info/com.
There are input and outputs included, but yeah .. calling the blockchain.com api for somewhat more than 1k adresses is not really a nice thing, so i'm already thinking about finding a better way. As i wrote already, getting the data from the bitcoind directly is probably the next thing i try..


Hacky

[NeuroticFish]

I will help seeding this, I'll start tomorrow with a computer that stays online a lot.

And maybe in some week-end I'll play a bit with it, it looks like a nice project, although I find it mostly "helpful" to the dreamers that think they can brute force wallets

[hackbyte]

Hmm, interesting. I've been doing it (very slowly) the naiive way, by querying everything through RPC calls to the Bitcoin Core client. Should be done by next month if I'm lucky. 

By the way ... doing it this "slow native way" is not a real problem at all, if you already have most of the data exported using the chainstate tool...

After that, i's just checking for updates and sending them into the database. On a plain CSV it might a bit more complicated, if you want to delete old entries which are no longer holding any coins.

Possible alternatives are, using sqlite - even maybe a in-memory only table if you have enought ram - or like i do a postgresql db..

[hackbyte]

I will help seeding this, I'll start tomorrow with a computer that stays online a lot.

And maybe in some week-end I'll play a bit with it, it looks like a nice project, although I find it mostly "helpful" to the dreamers that think they can brute force wallets

Yeah, you're welcome. Right now finally my seedhost is catching up on the file (651MB of 2,2GB uploaded), it will take another 2 hours or so until it is fully uploaded.

But then, it will get seeded from a nice vps somehwere in europe with somewhat thick inet connection (100mbit/s uplink iirc, we have still somewhat around 5tb traffic/month free .. so you're welcome anyways!

Hacky

P.s.: The trackers i added are:
udp://public.popcorn-tracker.org:6969/announce and
udp://tracker.opentrackr.org:1337/announce

[hackbyte]

Strangely the original posting which i found some days back which brought me to this thing is no longer writeable?!?

Mkay happens. Anyways, over there the programmer which wrote the chainstate tool said he had set up a website hosting his very own daily exports behind a paywall (which i can totally understand). URL https://balances.syndevio.com/

So, is there any demand for regular exports of this data?

I worked some days on the first one i made at 2018-10-10, the latest one was from 2018-10-15, is there any demand for regular updated files?

I will usually put them just in a torrent because i don't want to set up a website for it or so....

Aynone?

Regards,

hacky

[aplistir]

So, is there any demand for regular exports of this data?

I worked some days on the first one i made at 2018-10-10, the latest one was from 2018-10-15, is there any demand for regular updated files?
I will usually put them just in a torrent because i don't want to set up a website for it or so....
Aynone?

There is demand.
I am one of those, who is interested in downloading that data regularly. But only about once in 3-6 months.

it looks like a nice project, although I find it mostly "helpful" to the dreamers that think they can brute force wallets

Examining blockchain has other uses too. I used the data in studying "Satoshis mining  pattern" and things like that. Was quite interesting.

I also find it interesting to monitor really old bitcoin addresses for unusual traffic. Eg. Following old zombie coins coming back to life, which sometimes happen.

For those things I have separate files of old addresses that are easy to go through with a script comparing them to the "current" chainstate.
 
For my small needs it would be enough to get a smaller database of only the addresses containing more than 10 BTC (most really old addresses have more than 10BTC in them.)  Filesize of such a smaller database is only about 7Mb

Unfortunately I am currently in a country that has expensive internet and  wont be downloading much until things get better.  (I have to pay 1€/Gb for the net.) Otherwise I would have my own node and get the data myself.

But yes. there is demand, and thanks for the good work  

[hackbyte]

So, is there any demand for regular exports of this data?

I worked some days on the first one i made at 2018-10-10, the latest one was from 2018-10-15, is there any demand for regular updated files?
I will usually put them just in a torrent because i don't want to set up a website for it or so....
Aynone?

There is demand.
I am one of those, who is interested in downloading that data regularly. But only about once in 3-6 months.

it looks like a nice project, although I find it mostly "helpful" to the dreamers that think they can brute force wallets

Examining blockchain has other uses too. I used the data in studying "Satoshis mining  pattern" and things like that. Was quite interesting.

I also find it interesting to monitor really old bitcoin addresses for unusual traffic. Eg. Following old zombie coins coming back to life, which sometimes happen.

For those things I have separate files of old addresses that are easy to go through with a script comparing them to the "current" chainstate.
 
For my small needs it would be enough to get a smaller database of only the addresses containing more than 10 BTC (most really old addresses have more than 10BTC in them.)  Filesize of such a smaller database is only about 7Mb

Unfortunately I am currently in a country that has expensive internet and  wont be downloading much until things get better.  (I have to pay 1€/Gb for the net.) Otherwise I would have my own node and get the data myself.

But yes. there is demand, and thanks for the good work  

Well, i should be able to change my little python thingie to not only write down any address but even add up the exact deposited value...

Should not take up that much space at all. In the end i would be able to get out a simple list of any address holding up any value.

But, there are already walletwatchers out there and just checking the rpc outputs of a full bitcoin node might be way more fast in indicatin that a particular address has changed....

Altough, having already a list of addresses and their values would help a lot to that.....

I check if i can write that in my python fubar...

Regards,

hacky