Bitcoin Forum
September 20, 2019, 11:08:48 AM *
News: If you like a topic and you see an orange "bump" link, click it. More info.
 
   Home   Help Search Login Register More  
Pages: [1]
  Print  
Author Topic: Complete chainstate export from 2018-10-15 as torrent.  (Read 221 times)
hackbyte
Copper Member
Full Member
***
Offline Offline

Activity: 277
Merit: 110


Doing cryptocurrency stuff since 2010.


View Profile WWW
October 14, 2018, 11:53:57 PM
Merited by bones261 (2), aliashraf (2), ETFbitcoin (1)
 #1

Heya people,


from this post: https://bitcointalk.org/index.php?topic=267618.0 i've seen that there might be some interest in some chainstate export which includes any bitcoin utxo address and it's particular value.

Well, here you go:

magnet:?xt=urn:btih:d9a56bd02a0f9c50462b0fab32ed3223d36ff182&dn=chainstate-201810150125CEST.7z

Code:
fornax|hackbyte|01:06:56|/currencies/src/chainstate|0$ time ./chainstate >chainstate.output 2>chainstate.errors

real    17m49.092s
user    13m47.253s
sys     2m49.484s
fornax|hackbyte|01:25:15|/currencies/src/chainstate|0$ 7z a chainstate-201810150125CEST.7z chainstate.output

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,6 CPUs AMD FX(tm)-6300 Six-Core Processor              (600F20),ASM,AES-NI)

Scanning the drive:
1 file, 5478234093 bytes (5225 MiB)

Creating archive: chainstate-201810150125CEST.7z

Items to compress: 1

                         
Files read from disk: 1
Archive size: 2261586261 bytes (2157 MiB)
Everything is Ok
fornax|hackbyte|01:43:40|/currencies/src/chainstate|0$ ls -lah chainstate-201810150125CEST.7z
-rw-r--r-- 1 hackbyte hackbyte 2.2G Oct 15 01:43 chainstate-201810150125CEST.7z


I will host this one for some few weeks and will have a somewhat capable seed host on it too.

Have fun..

Oh, how i got it?

Using https://github.com/mycroft/chainstate and my very own full blown bitcoin knots node...

Regards,

hacky

Cryptocurrency Mining, Gambling, Lending, Trading, Staking .. So much to do with them, since 2010!
Donations-> 1bytoroRvXLGc1FWTob5h1oH92VMdUSFi Questions? Dunno! To the moon? Mün or bust!
Lust auf klönschnack? Schau bei uns rein im Cryptotalk (D-A-CH+EU) unter https://t.me/cryptotalkgerman
1568977728
Hero Member
*
Offline Offline

Posts: 1568977728

View Profile Personal Message (Offline)

Ignore
1568977728
Reply with quote  #2

1568977728
Report to moderator
1568977728
Hero Member
*
Offline Offline

Posts: 1568977728

View Profile Personal Message (Offline)

Ignore
1568977728
Reply with quote  #2

1568977728
Report to moderator
1568977728
Hero Member
*
Offline Offline

Posts: 1568977728

View Profile Personal Message (Offline)

Ignore
1568977728
Reply with quote  #2

1568977728
Report to moderator
The Bitcoin network protocol was designed to be extremely flexible. It can be used to create timed transactions, escrow transactions, multi-signature transactions, etc. The current features of the client only hint at what will be possible in the future.
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1568977728
Hero Member
*
Offline Offline

Posts: 1568977728

View Profile Personal Message (Offline)

Ignore
1568977728
Reply with quote  #2

1568977728
Report to moderator
1568977728
Hero Member
*
Offline Offline

Posts: 1568977728

View Profile Personal Message (Offline)

Ignore
1568977728
Reply with quote  #2

1568977728
Report to moderator
1568977728
Hero Member
*
Offline Offline

Posts: 1568977728

View Profile Personal Message (Offline)

Ignore
1568977728
Reply with quote  #2

1568977728
Report to moderator
ETFbitcoin
Legendary
*
Offline Offline

Activity: 1764
Merit: 2029

Use SegWit and enjoy lower fees.


View Profile WWW
October 15, 2018, 03:57:12 AM
 #2

So, how do we use the parsed chainstate? Do we use https://github.com/mycroft/chainstate and follow the instruction on "How to Run" section?

almightyruler
Legendary
*
Offline Offline

Activity: 2058
Merit: 1053


View Profile
October 15, 2018, 06:44:22 AM
 #3

Hmm, interesting. I've been doing it (very slowly) the naiive way, by querying everything through RPC calls to the Bitcoin Core client. Should be done by next month if I'm lucky.  Cheesy

Based on the github page for chainstate, the bulk of the output format would be:   <txid which created unspent output>;<address>;<amount>

Is this correct? If not, please post a small sample of the output.
aliashraf
Hero Member
*****
Offline Offline

Activity: 896
Merit: 656


View Profile
October 15, 2018, 09:01:24 AM
 #4

Although I merrited this topic and OP's commitment deserves more appreciation imo, a more general approach to this problem would be more appropriate.

We have UTXO commitment proposals/implementations trending right now with their most important application being pruning and having full nodes with minimum resources and fast bootstrapping, etc. @spartacusrex started a topic in btctalk few days ago for instance and Paul Sztorc's first version of Drivechain released even later (couple of days ago) has already, implemented UTXO snapshotting.

Two most important differences:
1- UTXO snapshots include output scripts as well.
2- A Merkle Tree like  and its root hash is generated and maintained for all UTXOs  such that commitment to the state at a given height is feasible. 

We got advanced algorithms/data structures like MMR and a complete history of interest in UTXOs.

hackbyte
Copper Member
Full Member
***
Offline Offline

Activity: 277
Merit: 110


Doing cryptocurrency stuff since 2010.


View Profile WWW
October 15, 2018, 05:29:23 PM
 #5

Update; Had to set up some portforwarding and added 2 trackers for the torrent.. Wink

Cryptocurrency Mining, Gambling, Lending, Trading, Staking .. So much to do with them, since 2010!
Donations-> 1bytoroRvXLGc1FWTob5h1oH92VMdUSFi Questions? Dunno! To the moon? Mün or bust!
Lust auf klönschnack? Schau bei uns rein im Cryptotalk (D-A-CH+EU) unter https://t.me/cryptotalkgerman
hackbyte
Copper Member
Full Member
***
Offline Offline

Activity: 277
Merit: 110


Doing cryptocurrency stuff since 2010.


View Profile WWW
October 15, 2018, 05:35:19 PM
Merited by ETFbitcoin (1)
 #6

So, how do we use the parsed chainstate? Do we use https://github.com/mycroft/chainstate and follow the instruction on "How to Run" section?

Well, i don't know how you want or can use it at all. For me, i just wrote some bits python down to write every address i find in the original csv in to a database table..

Code:
#!/usr/bin/python
import sys, time, psycopg2

# chainstate output
sourcefile = "chainstate.output"

# oeffne chainstate export
filehandle = open(sourcefile, 'r')

# open database connection
dbcon = psycopg2.connect('host=** dbname=** user=** password=**')
dbcur = dbcon.cursor()

counter=1
tablename="btc_address"
fieldname="address"

for line in filehandle:

# read data from csv line
txid, dunno, address, value = line.strip().split(';')

# put it into the db, don't ask, just try and skip if it fails
try:
dbcur.execute("INSERT INTO " + tablename + " (" + fieldname + ") VALUES ('"+str(address)+"')")
dbcon.commit()
print("\033[K %08d written %s\n" % (counter,address), end='', flush=True)
counter += 1
continue
except:
print("\033[K %08d skipped %s\r" % (counter,address), end='', flush=True)
counter += 1
continue

print("\n")


So, whatever you want to do, just have fun. I'm just playing around for educational reasons. Wink

Oh btw, the format of the chainstate.output file is as follows (first 5 lines):

Code:
questron|hackbyte|18:01:48|~| 0> head -n 5 chainstate.output
033e83e3204b0cc28724e147f6fd140529b2537249f9c61c9de9972750030000;0;1KaPHfvVWNZADup3Yc26SfVdkTDvvHySVX;65279
4aa1defd46f10c27ff4d6c2565e1e0e611429cd673b0f7342f30820690030000;0;3PPqDTQgndCHiDxqeB3zayJxXB6UeKAiod;996900
b58cf0f52c86022b046838472fc308cb55c9756071399840ad38de8166040000;0;1NXznFqccAinm9RLEf9eTtppZG9tWJQp8M;259531
e1c9467a885a156e56a29d9c854e65674d581ad75611b02290454b4862060000;1;1LpCmEejWLNfZigApMPwUY9nZTS8NTJCNS;9466355
a1f28c43f1f3d4821d0db42707737ea90616613099234f905dfc6ae2b4060000;1;1FuphZ7xVPGrxthQT1S8X7nNQNByYxAT3V;339500

Have fun Wink

Hacky

Cryptocurrency Mining, Gambling, Lending, Trading, Staking .. So much to do with them, since 2010!
Donations-> 1bytoroRvXLGc1FWTob5h1oH92VMdUSFi Questions? Dunno! To the moon? Mün or bust!
Lust auf klönschnack? Schau bei uns rein im Cryptotalk (D-A-CH+EU) unter https://t.me/cryptotalkgerman
hackbyte
Copper Member
Full Member
***
Offline Offline

Activity: 277
Merit: 110


Doing cryptocurrency stuff since 2010.


View Profile WWW
October 15, 2018, 05:38:01 PM
 #7

Hmm, interesting. I've been doing it (very slowly) the naiive way, by querying everything through RPC calls to the Bitcoin Core client. Should be done by next month if I'm lucky.  Cheesy

Based on the github page for chainstate, the bulk of the output format would be:   <txid which created unspent output>;<address>;<amount>

Is this correct? If not, please post a small sample of the output.

Yeah, that's probably what i will do as next step. I found 'bitcoin-abe' over on git which should be able to read blkNNNN.dat files natively and put them into a SQL db for use as blockchain explorer like thing. But sadly the code is somewhat old, won't run on python3 and .. yeah, overall, it's old.. Wink ( you can find it here: https://github.com/bitcoin-abe/bitcoin-abe or using 'pip install abe')

My "problem" with bitcoin-abe is, it wants to run directly on the blockchain .dat files, which means, i should shut down my bitcoin knots node as long as abe is running on the files .... or "just copy the whole blockchain over to somewhere else" .. mhmm

I'm not happy doing that anyways, so .. yeah ... connecting the bitcoind and reading anything i want over the RPC thing might be not really fast but more convenient to me. Wink

The sample of the output is in the posting above this one. Wink

Have fun,

hacky

Cryptocurrency Mining, Gambling, Lending, Trading, Staking .. So much to do with them, since 2010!
Donations-> 1bytoroRvXLGc1FWTob5h1oH92VMdUSFi Questions? Dunno! To the moon? Mün or bust!
Lust auf klönschnack? Schau bei uns rein im Cryptotalk (D-A-CH+EU) unter https://t.me/cryptotalkgerman
hackbyte
Copper Member
Full Member
***
Offline Offline

Activity: 277
Merit: 110


Doing cryptocurrency stuff since 2010.


View Profile WWW
October 15, 2018, 05:47:09 PM
Merited by DarkStar_ (2)
 #8

Although I merrited this topic and OP's commitment deserves more appreciation imo, a more general approach to this problem would be more appropriate.

Thanks for the merit. Smiley

We have UTXO commitment proposals/implementations trending right now with their most important application being pruning and having full nodes with minimum resources and fast bootstrapping, etc. @spartacusrex started a topic in btctalk few days ago for instance and Paul Sztorc's first version of Drivechain released even later (couple of days ago) has already, implemented UTXO snapshotting.

Two most important differences:
1- UTXO snapshots include output scripts as well.
2- A Merkle Tree like  and its root hash is generated and maintained for all UTXOs  such that commitment to the state at a given height is feasible. 

We got advanced algorithms/data structures like MMR and a complete history of interest in UTXOs.

Hhhmm.. interesting... I'm just "for fun" on this topic whilst doing completely unrelated stuff anyways. But i will look in to the links you mentioned. After all, i still have the full node running here anyways and want to play a bit even with the json-rpc and blockchain data format stuff anyways... So maybe i can dig a bit deeper in this chainstate export code to get the output-scripts too.

By the way. On a different run, i just took a list of all my addresses and got any data which is related to it from blockchain.info/com.
There are input and outputs included, but yeah .. calling the blockchain.com api for somewhat more than 1k adresses is not really a nice thing, so i'm already thinking about finding a better way. As i wrote already, getting the data from the bitcoind directly is probably the next thing i try.. Wink


Hacky

Cryptocurrency Mining, Gambling, Lending, Trading, Staking .. So much to do with them, since 2010!
Donations-> 1bytoroRvXLGc1FWTob5h1oH92VMdUSFi Questions? Dunno! To the moon? Mün or bust!
Lust auf klönschnack? Schau bei uns rein im Cryptotalk (D-A-CH+EU) unter https://t.me/cryptotalkgerman
NeuroticFish
Legendary
*
Offline Offline

Activity: 1974
Merit: 1311


There are no mistakes. Only opportunities wasted.


View Profile
October 15, 2018, 05:51:10 PM
 #9

I will help seeding this, I'll start tomorrow with a computer that stays online a lot.

And maybe in some week-end I'll play a bit with it, it looks like a nice project, although I find it mostly "helpful" to the dreamers that think they can brute force wallets Smiley

hackbyte
Copper Member
Full Member
***
Offline Offline

Activity: 277
Merit: 110


Doing cryptocurrency stuff since 2010.


View Profile WWW
October 15, 2018, 05:56:40 PM
 #10

Hmm, interesting. I've been doing it (very slowly) the naiive way, by querying everything through RPC calls to the Bitcoin Core client. Should be done by next month if I'm lucky.  Cheesy

By the way ... doing it this "slow native way" is not a real problem at all, if you already have most of the data exported using the chainstate tool...

After that, i's just checking for updates and sending them into the database. On a plain CSV it might a bit more complicated, if you want to delete old entries which are no longer holding any coins.

Possible alternatives are, using sqlite - even maybe a in-memory only table if you have enought ram - or like i do a postgresql db..

Cryptocurrency Mining, Gambling, Lending, Trading, Staking .. So much to do with them, since 2010!
Donations-> 1bytoroRvXLGc1FWTob5h1oH92VMdUSFi Questions? Dunno! To the moon? Mün or bust!
Lust auf klönschnack? Schau bei uns rein im Cryptotalk (D-A-CH+EU) unter https://t.me/cryptotalkgerman
hackbyte
Copper Member
Full Member
***
Offline Offline

Activity: 277
Merit: 110


Doing cryptocurrency stuff since 2010.


View Profile WWW
October 15, 2018, 06:50:57 PM
 #11

I will help seeding this, I'll start tomorrow with a computer that stays online a lot.

And maybe in some week-end I'll play a bit with it, it looks like a nice project, although I find it mostly "helpful" to the dreamers that think they can brute force wallets Smiley

Yeah, you're welcome. Right now finally my seedhost is catching up on the file (651MB of 2,2GB uploaded), it will take another 2 hours or so until it is fully uploaded.

But then, it will get seeded from a nice vps somehwere in europe with somewhat thick inet connection (100mbit/s uplink iirc, we have still somewhat around 5tb traffic/month free .. so you're welcome anyways! Wink

Hacky

P.s.: The trackers i added are:
udp://public.popcorn-tracker.org:6969/announce and
udp://tracker.opentrackr.org:1337/announce

Cryptocurrency Mining, Gambling, Lending, Trading, Staking .. So much to do with them, since 2010!
Donations-> 1bytoroRvXLGc1FWTob5h1oH92VMdUSFi Questions? Dunno! To the moon? Mün or bust!
Lust auf klönschnack? Schau bei uns rein im Cryptotalk (D-A-CH+EU) unter https://t.me/cryptotalkgerman
hackbyte
Copper Member
Full Member
***
Offline Offline

Activity: 277
Merit: 110


Doing cryptocurrency stuff since 2010.


View Profile WWW
October 15, 2018, 07:00:33 PM
 #12

Strangely the original posting which i found some days back which brought me to this thing is no longer writeable?!?

Mkay happens. Anyways, over there the programmer which wrote the chainstate tool said he had set up a website hosting his very own daily exports behind a paywall (which i can totally understand). URL https://balances.syndevio.com/

So, is there any demand for regular exports of this data?

I worked some days on the first one i made at 2018-10-10, the latest one was from 2018-10-15, is there any demand for regular updated files?

I will usually put them just in a torrent because i don't want to set up a website for it or so....

Aynone? Wink

Regards,

hacky

Cryptocurrency Mining, Gambling, Lending, Trading, Staking .. So much to do with them, since 2010!
Donations-> 1bytoroRvXLGc1FWTob5h1oH92VMdUSFi Questions? Dunno! To the moon? Mün or bust!
Lust auf klönschnack? Schau bei uns rein im Cryptotalk (D-A-CH+EU) unter https://t.me/cryptotalkgerman
ETFbitcoin
Legendary
*
Offline Offline

Activity: 1764
Merit: 2029

Use SegWit and enjoy lower fees.


View Profile WWW
October 16, 2018, 09:24:41 AM
 #13

Hmm, interesting. I've been doing it (very slowly) the naiive way, by querying everything through RPC calls to the Bitcoin Core client. Should be done by next month if I'm lucky.  Cheesy

FYI, another client such as https://bitcore.io/ have better performance. You might be interested if you need real-time data update.

But probably slower than access the database or CSV directly.

So, how do we use the parsed chainstate? Do we use https://github.com/mycroft/chainstate and follow the instruction on "How to Run" section?

Well, i don't know how you want or can use it at all. For me, i just wrote some bits python down to write every address i find in the original csv in to a database table..
--snip--

That's exactly what i need (an example), thanks Smiley

I gonna parse from my own full-nodes chainstate though since my connection isn't fast

So, is there any demand for regular exports of this data?

AFAIK, no. Even Bitcoin's blockchain torrent isn't as "hot" as few years ago.

aplistir
Full Member
***
Offline Offline

Activity: 351
Merit: 161



View Profile
October 16, 2018, 10:04:28 PM
 #14

So, is there any demand for regular exports of this data?

I worked some days on the first one i made at 2018-10-10, the latest one was from 2018-10-15, is there any demand for regular updated files?
I will usually put them just in a torrent because i don't want to set up a website for it or so....
Aynone? Wink

There is demand.
I am one of those, who is interested in downloading that data regularly. But only about once in 3-6 months.


it looks like a nice project, although I find it mostly "helpful" to the dreamers that think they can brute force wallets Smiley

Examining blockchain has other uses too. I used the data in studying "Satoshis mining  pattern" and things like that. Was quite interesting.

I also find it interesting to monitor really old bitcoin addresses for unusual traffic. Eg. Following old zombie coins coming back to life, which sometimes happen.

For those things I have separate files of old addresses that are easy to go through with a script comparing them to the "current" chainstate.
 
For my small needs it would be enough to get a smaller database of only the addresses containing more than 10 BTC (most really old addresses have more than 10BTC in them.)  Filesize of such a smaller database is only about 7Mb

Unfortunately I am currently in a country that has expensive internet and  wont be downloading much until things get better.  (I have to pay 1€/Gb for the net.) Otherwise I would have my own node and get the data myself.

But yes. there is demand, and thanks for the good work  Smiley

My Address: 121f7zb2U4g9iM4MiJTDhEzqeZGHzq5wLh
hackbyte
Copper Member
Full Member
***
Offline Offline

Activity: 277
Merit: 110


Doing cryptocurrency stuff since 2010.


View Profile WWW
October 17, 2018, 02:31:20 AM
 #15

So, is there any demand for regular exports of this data?

I worked some days on the first one i made at 2018-10-10, the latest one was from 2018-10-15, is there any demand for regular updated files?
I will usually put them just in a torrent because i don't want to set up a website for it or so....
Aynone? Wink

There is demand.
I am one of those, who is interested in downloading that data regularly. But only about once in 3-6 months.


it looks like a nice project, although I find it mostly "helpful" to the dreamers that think they can brute force wallets Smiley

Examining blockchain has other uses too. I used the data in studying "Satoshis mining  pattern" and things like that. Was quite interesting.

I also find it interesting to monitor really old bitcoin addresses for unusual traffic. Eg. Following old zombie coins coming back to life, which sometimes happen.

For those things I have separate files of old addresses that are easy to go through with a script comparing them to the "current" chainstate.
 
For my small needs it would be enough to get a smaller database of only the addresses containing more than 10 BTC (most really old addresses have more than 10BTC in them.)  Filesize of such a smaller database is only about 7Mb

Unfortunately I am currently in a country that has expensive internet and  wont be downloading much until things get better.  (I have to pay 1€/Gb for the net.) Otherwise I would have my own node and get the data myself.

But yes. there is demand, and thanks for the good work  Smiley

Well, i should be able to change my little python thingie to not only write down any address but even add up the exact deposited value...

Should not take up that much space at all. In the end i would be able to get out a simple list of any address holding up any value.

But, there are already walletwatchers out there and just checking the rpc outputs of a full bitcoin node might be way more fast in indicatin that a particular address has changed....

Altough, having already a list of addresses and their values would help a lot to that.....

I check if i can write that in my python fubar... Wink

Regards,

hacky

Cryptocurrency Mining, Gambling, Lending, Trading, Staking .. So much to do with them, since 2010!
Donations-> 1bytoroRvXLGc1FWTob5h1oH92VMdUSFi Questions? Dunno! To the moon? Mün or bust!
Lust auf klönschnack? Schau bei uns rein im Cryptotalk (D-A-CH+EU) unter https://t.me/cryptotalkgerman
Pages: [1]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!