How to read/parse blockchain and get bitcoin addresses having balance.

[btctousd81]

i am trying to get bitcoin addresses from my blockchain stored on server.

my bitcoin is up to date. index mode is not enabled, as i dont want UTXO database.

i tried using https://github.com/znort987/blockparser

compiled it succesfully.
strated like this

Code:

[root@blockparser]# env BLOCKCHAIN_DIR="/home/user2/" ./parser allBalances > allBalances.txt

info: mem at start = 0.000 Gigs
info: starting command "allBalances"
info: loading block chain from directory: /home/user2/.bitcoin
info: block chain size = 136.344 Gigs
info: initializing hash tables
info: estimated number of blocks = 1071.27K
info: estimated number of transactions = 276.71M
info: done initializing hash tables - mem = 0.000 Gigs
info: pass 1 -- walk all blocks and build headers ...
info: pass 1 -- took 48 secs, 488497 blocks, 136.34 Gigs, 2862.34 Megs/secs , 89430 early link misses, mem=0.000 Gigs        
info: pass 2 -- link all blocks ...
info: pass 2 -- done, did 89430 late links
info: pass 3 -- wire longest chain ...
info: pass 3 -- done, maxHeight=488497
info: pass 4 -- full blockchain analysis (with index)...
info: computing balance for all addresses
info: hit ^C to interrupt and dump current, valid state of ledger
info:        1 blocks,    0.000 MegaAddrs ,
info:   141229 blocks,    1.738 MegaAddrs ,
info:   166873 blocks,    3.058 MegaAddrs ,
info:   181587 blocks,    3.939 MegaAddrs ,
info:   186771 blocks,    4.633 MegaAddrs ,

but it crashes in the middle., or some times at the end., i tried running it many times.,

previously i got memory related error., so freed up memory and increased swap on my server. then tried running it again

right now i am getting

Code:

info: 481823 blocks, 295.351 MegaAddrs ,
fatal: failed to locate upstream transaction

i counted and looks like it aborts when 2000 blocks are remaining.

i am not the only one getting this error..

https://github.com/znort987/blockparser/issues/65

so what you guys are using ? to get data extracted from blockchain files.

i have 64 gigs of ram, i7 , ssd .
centos 7 x64

is there any other tool ? optimized ? or any other fork

thanks for your time,.

[1715490616]

1715490616

[1715490616]

1715490616

[1715490616]

1715490616

[LoyceV]

i tried using https://github.com/znort987/blockparser

Code:

info: 481823 blocks, 295.351 MegaAddrs ,
fatal: failed to locate upstream transaction

I can't help you fix it, but I can tell you the reason: blockparser is older than SegWit, so it stops at the block in which SegWit got activated.
BTC.COM mines the 481,823rd block! Segwit is on stage!

If you get it to work, I'd love to have a complete list (in for example CSV-format) of all addresses with their balance. I'd like to play around with it a bit.

Update: this post was Merited today, which reminded me to add this:

Bitcoin block data available in CSV format
List of all Bitcoin addresses with a balance
List of all Bitcoin addresses ever used

[btctousd81]

i tried using https://github.com/znort987/blockparser

Code:

info: 481823 blocks, 295.351 MegaAddrs ,
fatal: failed to locate upstream transaction

I can't help you fix it, but I can tell you the reason: blockparser is older than SegWit, so it stops at the block in which SegWit got activated.
BTC.COM mines the 481,823rd block! Segwit is on stage!

If you get it to work, I'd love to have a complete list (in for example CSV-format) of all addresses with their balance. I'd like to play around with it a bit.

i can get a list before that block by stopping program.,

but i am looking for complete upto date list.
will post if i get it to work.,

what changed in segwit ? i mean in block ? so i can try to change it in blockparser source.



edit 1:

this is the code cauing the fatal error

Code:

        auto upTXHash = p;
        const Chunk *upTX = 0;
        if(gNeedUpstream && !skip) {
            auto isGenTX = (0==memcmp(gNullHash.v, upTXHash, sizeof(gNullHash)));
            if(likely(false==isGenTX)) {
                auto i = gTXOMap.find(upTXHash);
                if(unlikely(gTXOMap.end()==i)) {
                    errFatal("failed to locate upstream transaction");
                }
                upTX = i->second;
            }
        }

i am not sure, will it give correct output. if i comment outt that line and recompile and rerun the code.,
will try anyways.

[amaclin1]

what changed in segwit ? i mean in block ? so i can try to change it in blockparser source.

Transaction format was changed in segwit.
These six words mean that your tool should be fully rewritten. 

[btctousd81]

what changed in segwit ? i mean in block ? so i can try to change it in blockparser source.

Transaction format was changed in segwit.
These six words mean that your tool should be fully rewritten.  

thanks
then looks like i am better off trying with other tools.,



edit1 :

i tried running it as it is, parsed around 50% blocks and got output file of 4 GB having format like this

Code:

---------------------------------------------------------------------------
          State of the ledger at block 420662 (minted : Thu Jul 14 08:18:22 2016)
---------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
              Balance                      Hash160                             Base58                  nbIn        lastTimeIn          nbOut        lastTimeOut
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
         152955.81868182 e95dbb25283cc35d4a6aefa76c0382e63ce0fa36 3Nxwenay9Z8Lc9JBiywExpnEFiLp6Afp8v     56 Tue Jul 12 14:11:34 2016      11 Wed Jun  1 15:36:33 2016
         121295.29280322 b3b9f5025c397c07e7e37db7e5c9259ac95cd344 3J5KeQSVBUEs3v2vEEkZDBtPLWqLTuZPuD    387 Thu Jul 14 05:00:27 2016     196 Sun Jul  3 11:45:17 2016
          83000.01001479 e1223ee403b7250d54c08890679df36e9d6b4986 3NDQz8rZ3CnmsiBGrATk8SCpDXF2sAUiuM      6 Sun Jul  3 15:23:05 2016       1 Sun May 29 01:58:14 2016
          79957.11509537 a0b0d60e5991578ed37cbda2b17d8b2ce23ab295 1FeexV6bAHb8ybZjqQMjJrcCrHGW9sb6uF     85 Fri May 20 18:43:43 2016       0 Thu Jan  1 00:00:00 1970
          78172.33497629 c5464169a9aabad0e361ccf1d436d3e843708e7d 3Kg7Cmooris7cLErTsijq6qR1FH3cTiK2G  20889 Sun Jul  3 15:23:05 2016       4 Tue May 17 17:14:55 2016
          69370.10525051 b3dd79fb3460c7b0d0bbb8d2ed93436b88b6d89c 1HQ3Go3ggs8pFnXuHVHRytPCq5fGG8Hbhx     64 Sat Feb 13 06:40:01 2016       1 Thu Apr 23 14:10:25 2015
          66650.59620465 3d03002dbed5cb1dc10fc6bcb0886d2df32f2838 16ZbpCEyVVdqu8VycWR8thUL2Rd9JnjzHt    168 Fri Apr 29 10:29:14 2016       0 Thu Jan  1 00:00:00 1970
          66583.22391617 cd4b7b8f9db1b0c709fd0c9f0534fca6a9f40495 1KiVwxEuGBYavyKrxkLncJt2pQ5YUUQX7f    120 Sat Feb 13 06:40:01 2016       0 Thu Jan  1 00:00:00 1970
          66452.06624862 f9e6bbcdc83d8f351014e07495f386fe1067ec7b 1PnMfRF2enSZnR6JSexxBHuQnxG8Vo5FVK    114 Sat Feb 13 06:40:01 2016       0 Thu Jan  1 00:00:00 1970
          66378.80961189 6a6015e3793207af6dff7c48ee9e193d73547cdc 1AhTjUMztCihiTyA4K6E3QEpobjWLwKhkR    181 Thu Feb 18 17:09:28 2016       0 Thu Jan  1 00:00:00 1970
          66235.82427687 8b70193546504fa3623598722575f70b5b1c6455 1DiHDQMPFu4p84rkLn6Majj2LCZZZRQUaa    125 Sat Feb 13 06:40:01 2016       0 Thu Jan  1 00:00:00 1970
          66233.73850546 908b47d6a15d89c3ee34df1bc68a00ec4ff70c9b 1EBHA1ckUWzNKN7BMfDwGTx6GKEbADUozX    126 Sun Jun  5 18:31:03 2016       0 Thu Jan  1 00:00:00 1970
          66205.47375047 27ed35ebcf72744ec3b4a0f4dbb83d7620acf443 14e7XAZbepQp9MXXzjNG3fNLoAUpaBAXHW    132 Fri Apr 29 10:29:14 2016       0 Thu Jan  1 00:00:00 1970
          66170.69605917 f33a3af3695f041e2ea6271f1fe6309abac4dd58 1PB4xXUFyy4kSNqroCBVaQuCuw9VcN3be4    171 Fri Feb 12 04:57:18 2016       0 Thu Jan  1 00:00:00 1970
          66163.44524020 53f9818e8d2d23c3163c01a33d83e49ebfa16ac4 18f1yugoAJuXcHAbsuRVLQC9TezJ6iVRLp    124 Wed Mar  9 23:07:42 2016       0 Thu Jan  1 00:00:00 1970
          53880.05741758 d74de95f65799793f16b91ed8a152110652d3ec0 1LdRcdxfbSnmCYYNdeYpUnztiYzVfBEQeC     21 Fri Feb 12 04:57:18 2016       0 Thu Jan  1 00:00:00 1970
          50700.00172304 ffd32592cc07d25a8644d7dfd6919d348f9b1874 3R1hBCHURkquAjFUv1eH5u2gXqooJkjg4B     81 Fri Jul  8 14:53:10 2016      37 Sun Jun 12 14:26:28 2016

after around 505 lines, btcaddress is screwed .

Code:

           2500.00111000 a0fc91baff4e19cb9f417cef2f00d2822106b5be 1FgDgWjeCMxrfiiaYnPdvnQG9DcYe6yX1r      5 Thu Apr  2 17:52:36 2015       0 Thu Jan  1 00:00:00 1970
           2500.00111000 d34e8a9a7786fdbbff59be2a5cc169729b2bbae0 1LGHfdXMoAh3rAqfZ5tRruVxWz4drjLGHQ      5 Thu Apr  2 17:52:36 2015       0 Thu Jan  1 00:00:00 1970
           2500.00111000 37a5e72afc3aa9ed12e51ee187c56a5d2e6f1ed9 165EuhCqScXdGJfDvYK5RZKN5SytnhwVjv      5 Thu Apr  2 17:52:36 2015       0 Thu Jan  1 00:00:00 1970
           2500.00111000 a0b5712a79778cc923e3ca0a54a8a442d8c24a7c 1FekUWTJ4S4UTYuj41yaJCuntvh2cszhsk      5 Thu Apr  2 17:52:36 2015       0 Thu Jan  1 00:00:00 1970
           2500.00111000 9519a19ef04031d40998d7a684c7878d4c5baa8e XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX      5 Thu Apr  2 17:52:36 2015       0 Thu Jan  1 00:00:00 1970
           2500.00111000 51ccfe24e3577bb2b8f20ddf39ca69389dc3726c XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX      5 Thu Apr  2 17:52:36 2015       0 Thu Jan  1 00:00:00 1970
           2500.00111000 9cfca6b08dc926b8903b58fde9c7d3db5f1b5007 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX      5 Thu Apr  2 17:52:36 2015       0 Thu Jan  1 00:00:00 1970
           2500.00111000 43c43187cc1bf92fd25b62477a2d772a7db679b4 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX      5 Thu Apr  2 17:52:36 2015       0 Thu Jan  1 00:00:00 1970
           2500.00005400 f90f808949b65ce2c01a0eacbf20febbf40b881e XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX      1 Tue Jun 21 03:52:29 2016       0 Thu Jan  1 00:00:00 1970
           2500.00000000 6a7266721acf340b77f84d8d0f677171ae7bb8a6 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX      1 Wed Jun 29 22:52:52 2016       0 Thu Jan  1 00:00:00 1970
           2500.00000000 b69398c2a4fe36e739ac765c4283d74323340afb XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX     13 Sun Mar  6 03:46:10 2016       6 Mon Mar 28 01:11:36 2016
           2500.00000000 86d1cc562153dd9954e31e9c7f3a02390b1d4a3f XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX      1 Fri Feb 19 21:51:14 2016       0 Thu Jan  1 00:00:00 1970
           2500.00000000 2ac55d1a940deb5b5e684e500355f826b5381ee5 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX      1 Fri Nov 27 13:22:46 2015       0 Thu Jan  1 00:00:00 1970
           2500.00000000 64dac2d7527f9967a2c0b6f18d986f116cd89bfd XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX      1 Tue Jun 21 16:27:38 2016       0 Thu Jan  1 00:00:00 1970
           2499.99960000 f148343d37f0621eb00eef03ed191444ee8aaa30 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX      2 Fri Jul  8 13:16:16 2016       0 Thu Jan  1 00:00:00 1970
           2499.99101000 74869025a6d4e202d1271b676ed5a374d92e94a2 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX     10 Thu Apr  2 17:52:36 2015       1 Tue Feb 11 17:17:54 2014
           2499.97356500 a4bc600d905f4465dd59a4dfbdb740231036dc71 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX      7 Thu Apr  2 17:52:36 2015       0 Thu Jan  1 00:00:00 1970
           2497.99949660 1b490a2b5be46b6fd395c11ca0d6fd50f194fb7c XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX      1 Fri Oct 30 04:21:43 2015       0 Thu Jan  1 00:00:00 1970
           2492.25031000 bc8d40e64b33e2d99ff81e4643dbf8d80e2a5c9b XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX      7 Thu Apr  2 17:52:36 2015       0 Thu Jan  1 00:00:00 1970
           2489.00121000 c8ca9fa680409b5d7375ad6d2fb5aac2e4405c89 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX      5 Thu Apr  2 17:52:36 2015       0 Thu Jan  1 00:00:00 1970
           2483.80277466 b8f27aaedf87530547e61406f73947455371e6de XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX      1 Fri May 27 20:27:01 2016       0 Thu Jan  1 00:00:00 1970

so unless you want hash160, or know how to convert hash160 to btcaddress., this tool is obsolete .

[BitcoinPC]

Waste of time, what is the point? It would take you longer or about the same amount of time to steal coins this way than it would to mine Bitcoins using the same equipment. People have tried this time and time again and the results are the same, they give up because it is a fruitless pursuit. There is always someone that thinks they know better than the system and they fail in the end, so please try again.

[btctousd81]

Waste of time, what is the point? It would take you longer or about the same amount of time to steal coins this way than it would to mine Bitcoins using the same equipment. People have tried this time and time again and the results are the same, they give up because it is a fruitless pursuit. There is always someone that thinks they know better than the system and they fail in the end, so please try again.

what makes you think, my method is same as others., remember everyone is unique

[bigvito19]

Is there any other way to get hash160 addresses besides using this?

[btctousd81]

Is there any other way to get hash160 addresses besides using this?

you want hash160 or btc address ?

[bigvito19]

I have the btc addresses, I need a way to get the hash160 addresses.

[btctousd81]

I have the btc addresses, I need a way to get the hash160 addresses.

this is what i would use .

Code:

bool addrToHash160(
          uint8_t *hash160,
    const uint8_t *addr,
             bool checkHash,
             bool verbose
) {
    static BIGNUM *sum = 0;
    static BN_CTX *ctx = 0;
    if(unlikely(!ctx)) {
        ctx = BN_CTX_new();
        BN_CTX_init(ctx);
        sum = BN_new();
    }

    BN_zero(sum);
    while(1) {
        uint8_t c = *(addr++);
        if(unlikely(0==c)) break;

        uint8_t dg = fromB58Digit(c);
        BN_mul_word(sum, 58);
        BN_add_word(sum, dg);
    }

    uint8_t buf[4 + 2 + kRIPEMD160ByteSize + 4];
    size_t size = BN_bn2mpi(sum, 0);
    if(sizeof(buf)<size) {
        warning(
            "BN_bn2mpi returned weird buffer size %d, expected %d\n",
            (int)size,
            (int)sizeof(buf)
        );
        return false;
    }

    BN_bn2mpi(sum, buf);

    uint32_t recordedSize = 
        (buf[0]<<24)    |
        (buf[1]<<16)    |
        (buf[2]<< 8)    |
        (buf[3]<< 0)
    ;
    if(size!=(4+recordedSize)) {
        warning(
            "BN_bn2mpi returned bignum size %d, expected %d\n",
            (int)recordedSize,
            (int)size-4
        );
        return false;
    }

    uint8_t *bigNumEnd;
    uint8_t *dataEnd = size + buf;
    uint8_t *bigNumStart = 4 + buf;
    uint8_t *checkSumStart = bigNumEnd = (-4 + dataEnd);
    while(0==bigNumStart[0] && bigNumStart<checkSumStart) ++bigNumStart;

    ptrdiff_t bigNumSize = bigNumEnd - bigNumStart;
    ptrdiff_t padSize = kRIPEMD160ByteSize - bigNumSize;
    if(0<padSize) {
        if(0<bigNumSize) {
            memcpy(padSize + hash160, bigNumStart, bigNumSize);
        }
        memset(hash160, 0, padSize);
    } else {
        memcpy(hash160, bigNumStart - padSize, kRIPEMD160ByteSize);
    }

    bool hashOK = true;
    if(checkHash) {

        uint8_t data[1+kRIPEMD160ByteSize];
        memcpy(1+data, hash160, kRIPEMD160ByteSize);
        data[0] = getCoinType();

        uint8_t sha[kSHA256ByteSize];
        sha256Twice(sha, data, 1+kRIPEMD160ByteSize);

        hashOK =
            sha[0]==checkSumStart[0]  &&
            sha[1]==checkSumStart[1]  &&
            sha[2]==checkSumStart[2]  &&
            sha[3]==checkSumStart[3];

        if(!hashOK) {
            warning(
                "checksum of address %s failed. Expected 0x%x%x%x%x, got 0x%x%x%x%x.",
                addr,
                checkSumStart[0],
                checkSumStart[1],
                checkSumStart[2],
                checkSumStart[3],
                sha[0],
                sha[1],
                sha[2],
                sha[3]
            );
        }
    }

    return hashOK;
}

ref: https://github.com/znort987/blockparser/blob/master/util.cpp#L596-L699

[bigvito19]

What would be the the steps or cmd line to use that?

I've been trying to compile the parser but having trouble with g++-4.4, what all did you install to get it to get it to work. I'm stuck at unable to locate the g++-4.4

[btctousd81]

What would be the the steps or cmd line to use that?

I've been trying to compile the parser but having trouble with g++-4.4, what all did you install to get it to get it to work. I'm stuck at unable to locate the g++-4.4

i do this for all my centos systems.,

yum install autoconf automake gcc-c++ libdb4-cxx libdb4-cxx-devel boost-devel openssl-devel

but when error occures at the time of compiliing, act accordingly.

[aplistir]

Waste of time, what is the point? It would take you longer or about the same amount of time to steal coins this way than it would to mine Bitcoins using the same equipment. People have tried this time and time again and the results are the same, they give up because it is a fruitless pursuit. There is always someone that thinks they know better than the system and they fail in the end, so please try again.

what makes you think, my method is same as others., remember everyone is unique

So you admit you are trying to steal bitcoins?  Good luck with that.
I don't know, why you need to be able to parse the blockchain yourself though. Why not explore ready made lists of bitcoin addresses.
https://bitinfocharts.com/top-100-richest-bitcoin-addresses.html

But yeah. many have tried and noticed bitcoin is more secure than they thought. Would be interesting to know what is your angle of attack

When reading your post I assumed you are just interested in the blockchain and exploring it out of interest.
Because I have been doing the same.
there are some quite interesting addresses. Examples here:
http://www.theopenledger.com/9-most-famous-bitcoin-addresses/

[btctousd81]

Waste of time, what is the point? It would take you longer or about the same amount of time to steal coins this way than it would to mine Bitcoins using the same equipment. People have tried this time and time again and the results are the same, they give up because it is a fruitless pursuit. There is always someone that thinks they know better than the system and they fail in the end, so please try again.

what makes you think, my method is same as others., remember everyone is unique

So you admit you are trying to steal bitcoins?  Good luck with that.
I don't know, why you need to be able to parse the blockchain yourself though. Why not explore ready made lists of bitcoin addresses.
https://bitinfocharts.com/top-100-richest-bitcoin-addresses.html

But yeah. many have tried and noticed bitcoin is more secure than they thought. Would be interesting to know what is your angle of attack

When reading your post I assumed you are just interested in the blockchain and exploring it out of interest.
Because I have been doing the same.
there are some quite interesting addresses. Examples here:
http://www.theopenledger.com/9-most-famous-bitcoin-addresses/

i m more of like whitehat .
i want make sure, if blockchain is secure enough, so that i can invest some big chunk of money in bitcoins.

[amaclin1]

i want make sure, if blockchain is secure enough

You do not have skills and knowledge for such checks ans assumptions 

[btctousd81]

i want make sure, if blockchain is secure enough

You do not have skills and knowledge for such checks ans assumptions  

i know, not yet, but i am trying/willing to learn.



edit: 12th oct 2017

i have used blockaparser on 50% blocks.

which gave me 4 gb allbalnce.txt file.,

i used 7zip to zip it.,

if anyone is intersted here it is.,

https://transfer.sh/DLIpD/blockparser-allBalances.txt.7z

ill try to parse the complete block when i get free time., and try to keep this thread updated.

[hamdi]

anybody had success above segwit blocks?

[btctousd81]

anybody had success above segwit blocks?

using blockparser
theoritically yes,
practically no

remove all blocks after which segit is implemeted.,
remove bitcoind
install bitcoin-core version bitcoin-core-0.11.2
start bitcoind
run blockparser again

else


remove all blocks after which segit is implemeted.,
remove bitcoind
install bitcoin-core version bitcoin-core-0.11.2
start bitcoind with resync
run blockparser again

edit:
practically no
means i have not tried it yet., but its the solution/correct way to do it., thats what i have been told.
ill try it this week.
good luck

[hamdi]

thanks a lot for replying!!!