.

[znort987]

.

[1714813350]

1714813350

[1714813350]

1714813350

[1714813350]

1714813350

[1714813350]

1714813350

[runlinux]

Now what you need to do is group addresses based on the inputs and create a list of "wallets" in the same fashion. I have a tool that does that, to a point...

[malaimult]

please stop posting my btc addresses 

[bulanula]

please stop posting my btc addresses 

You have 8000+ coins and hack Linode

[etotheipi]

To try and understand the nitty-gritty of how bitcoin works, I've
written a fairly fast blockchain parser in C++ (x86-64 ubuntu only).

It can currently do the following things:

    . Compute simple blockchain stats, full chain parse (< 1 second)

    . Extract all known transactions for a popular address  (20 seconds)

    . Compute the list of addresses that provably belong to the same person (20 seconds):

    . Compute and print the balance for all keys ever used since the beginning of time (30 seconds):

    . You can quite easily add your own data extraction code

Nice!  Mind sharing a top-level overview of how you do it?  Data structures?   Do you know how much that is I/O limited?

The C++ utilities I wrote for Armory are almost that fast (but in linux only, have no idea why windows is so damned slow!).  I buffer blk000X.dat into RAM 50 MB at a time, and then pass around pointers to this data and scan it directly -- to avoid any extra copies.  It's a lot of pointers...

[malaimult]

damn it. you got me  let's share the btc and don't tell anyone. will be our little secret 

[malaimult]

dirty secret 

[bulanula]

damn it. you got me  let's share the btc and don't tell anyone. will be our little secret 

Send some love to 15uxVpSgWtfQ25ae6YYxG3US3eWRpYqKkp if that is the case.

I sure do loves me some stolen coins

[malaimult]

i wish i could.  my mom got my wallet pasw. if you know someone who can torture my mom so she can tell the pasw i will pay 20.000 btc

[SgtSpike]

Now what you need to do is group addresses based on the inputs and create a list of "wallets" in the same fashion. I have a tool that does that, to a point...

It already does that: that's what the "closure" callback does.

As a matter of fact, second teaser: here's the transitive closure of the Linode hack address
where slush lost BTC 3K+ (all these addresses are controlled by the same person).

Code:

Starting method "closure" on whole block chain

Building address equivalence graph ...
done.

Clustering 3824484 addresses ... 
done.

Showing address cluster for address 0xeb1379c139bc2cd4cbfad64803e4b1a3738c85cd
    0x80ab5bcd943419b8988234e8e19b83389edc542a
    0x10c0f04931b015c0d339a1510cfc23a12a6dcdbe
    0xfd12b8122d5325052c45528ace1e84d28bd5f50d
    0x849a3152fba1e771507f5a1b30e927fbbbece4d5
    0xafbe1a4d5a66c23ca68fb37579da6b512d6c8c82
    0x30797822134a26e9a14a5c28eed7379e4e261fe6
    0x15a5b533ea69ccc3324580daf040a11bd7a84e25
    0x08249f65d8e7f77593187369a06f06570d63bd93
    0x50d8d51785ddc5c7c487a4dedefcb7bbd1cce354
    0x07f4e9b700be90791fda1426758e1ca88ec7b5cb
    0x16071183cb53aa163a9a0b2922a3d52304bd72d6
    0xac1becc48fec010e64dfbab6227c566b207b0340
    0xdc94b06440dcfbde07d92fd69cd2a2d55d1623a1
    0xc163ae0ac2ee1502a9fbccd0b204100acace1bd5
    0x057e4505c36dd2b3db6f5b1f3f21ecde44ce94a5
    0x53c220e759f943acf0c5aa750944b461e45782f6
    0xa772e7e148564dc4b397a551621661032450e6a7
    0xd0845085adbc9aee32a51b224f92fad0a964af90
    0x24209c6ad65d8ae90149b156354024818a082a4a
    0x0402c49bd5a769ebf7d9aad63b9b4fba3437f58c
    0x13b5a898d925d456409b83f40d0e3fdeb084d9d9
    0x813ad1e3b87ad5c11037e5d7d11c2d825a3b5fcc
    0x336e021f04e1f4f8c67ceab702303b7bcdb09251
    0xa08509223efce6be3709e3c74fce44d71bb046d3
    0xddbb89f652b507ad6bed30aab8caa52798cefd76
    0xfb457bb25c2f2e526d4a02bbb21bc4c299b8ce79
    0xbe1b0b08887604cb691035d4f5a52c1ffe5f5da4
    0x66c1bc8387f29c5948ba77e5cd17631234f0c8cf
    0xf181981c9a9f4d81c7075e63eb7c3f3ac2c96281
    0xdede0dc0dfcbfe0e914a42af76a452a7f70bd66a
    0xcc464b79712bcdca45d6baf92b34c1195c3c60d4
    0xb4d92d90f7e9aaf8754653adc3a78143710343a4
    0xc3ebb9e8a65442ed8778fb9a7555d09238a73f5f
    0x9f2108f655bfeb24f355fad54079464229c907f2
    0xce0133e0c2a374c2a584912a323d6ce9a1d80ca4
    0xe26194e8bd0807f9bed840101c23e185acdddad8
    0x67fd72e0c83c196a7be5c59ca66b100b1b8cc8ee
    0x7b0c246ac7758223b86ffc4fe1157879f6741a86
    0x9ca7fdd425491560430d11de8ce5953f7c704e6d
    0xde91a201e056766a2410b194fcbe11e8172041ae
    0x328139a8b17c421b1c92490ebd3abda6504d9475
    0x28404196f827cf062180c936944877213ab3ae16
    0x21d9f719ac6d81d73cc1bcc2b439fa147c719400
    0xe34db05116423b574c8cffa37d7138cbf7875c41
    0x3021e0e08be6ad10054c6b54a426981bc69c7a66
    0x38be76d9e4e1b29a97d5524db2bda859bde10fef
    0x056e8893924d67b4f79c781cc9b026f2effa4997
    0x84e3215bcf49e181a80c892e682d70d5e5969b8e
    0xc8984c9a225143a46de2228be1ed174df53cb234
    0xcece1cd83c18b760dec8419b3fa6fe30462d68b0
    0x12329f6ad181740537e5a393fe43c7b7ece3d77e
    0x86a678be0b6a4c0b8dc265d02ca3bf5704ab60f2
    0xcaa45f977042ade06f41513190d5475b8d6acb68
    0x42c97b293287d8c4a0cb109491588df055722b34
    0xdd899d68aa49b48b12afda64c5b19f70f5205088
    0x250ba3156dfc5531d51650dd0d8d7d090758b08c
    0x701d16b65b9e2f39c0792e2c9cdf6b18283a486b
    0x5a45481627ad413db466841f477a8ecce4fae72a
    0xda5c94a9d70c000aa498bafdd1df363eb27b3eae
    0x6d3161cc084fbdb2e2c08c441af9c73a18966d39
    0xee7af18727e01cf2ecaab029c0db01d5454fa28a
    0x4f7151890c9458aa31933e99a06d7bb0baaaa672
    0x73af151bdf5fe264991a0aad6462343ee71411ee
    0xca5859d2c3db9a4d44f183efea50299799e2281f
    0xc8615e5708135a4c93331e6f3314f830337e4d07
    0xff45a1bdeb7d1a882406daab73a013d54fd1b8f0
    0xeb1379c139bc2cd4cbfad64803e4b1a3738c85cd
    0xf089c5c70b9d38256485e1d8af094163eeeca364
    0x15fc1662280f5a49beeb135a6104734f05af7071
    0x73d9bba2ec3357d8e913f2f7040055ba251bfd0c
    0xe6f5fb4d9a18bddb6e44e71073cba85938b1edf3
    0x8795a4c760914e66ee83661173ffbf269b1dcdce
    0x009036a1945a94d46e51e9da77202e58bf426fe3
    0xa37da9f20653e30f82fb4b0d2365349aa216ad7c
    0x03bd7122f1b94f5271d07bcce77a19a57a3bbd25
    0x0fe68651af87bce43a4e962b5a65c661589c8f1f
    0x6ddc9885cea2025bd891b27f166f6bc9caf3d317
    0x80546c69d2752c9d9155546edb1b9d6a29dae1da
    0x194bb0d97f33004b1a459a701dcdcaf573a62e68
    0x2c0533a0ff611c45977ab25f34530cbd88890df4
    0xd47cade0184e134f5ced736805d64247891d1bbf
    0x8653fea497a7913e3cb5db1c2022ed4fa635b484
    0x6842af932a7151263df2a2efc9ac5f3fcb2b8577
    0x07e52cc212b5c72a13c3c58796323cbdf1f799c0
    0xf73034a766ca7f097409c487389b8df50ed49b44
    0xd5dd7af8b718ddd290fb922c375f884002aaa8d7
    0x2ada99a8626356312c8c5d5486d76233327e0557
    0x1f93ca387c43bc6fd5d0b0e7c006b8d82a29d9b6
    0x8cf248ca2e9f9170f75c1a84433d8ef3dbfa6bb3
    0xa42898e555c7f78978b89360b5ec41ad17418619
    0x5837567cb248b2aed9df12cc717d4aa4e667ed49
    0xbdad4b9804df3311f64ea0d46046f19fa1512f34
    0xd19ec2e091cbaed87f4ff3170233e5abaf68a203
    0x4dd1672fca5b7f9442641941ad36852ea7c03d21
    0xffd7b0b9aacfc37068b366929734bebb99fed561
    0x084da95fb185f4fd3112cd796459db195c1594c4
    0x184acdafaa923f377fae0690f219d5611b7ca134
    0x46956ac4b584d0595546dd6890a9a75fbb64b9e8
    0x47bc0594bcdc9f5e9266e44aee5399e51a1ac40c
    0x39a137e4b924d82562289d0efeff0ef9af2a9c72
    0x082b1695ca1da6fbbf357487cd4a05c14f604aa9
    0x34532cb0ad27c5f2792d127cc05fd686bc2ec9b1
    0xe02e9f0fa9cd06850c6ef16f2704de8df68cff3c
    0x21dd377a48f11fabb2f01a9fee38b1c0b118e163
    0x4c2a67dc98e6e1f0a998ca4a9086a25725e6a653
    0xcda6bc68dba5c595bd15bc9d43f7ccd399dd5ab9
Done in 20.621 seconds

What does 0x80ab5bcd943419b8988234e8e19b83389edc542a mean?  It's not a Bitcoin address in the typical format.

[SgtSpike]

What does 0x80ab5bcd943419b8988234e8e19b83389edc542a mean?  It's not a Bitcoin address in the typical format.

Yeah, sorry about that, I should have dropped the 0x in front. Once
you remove the 0x, it's the ripemd-160 hash of the public key.

So:

    . Copy everything but the '0x' , in your example, that would be : 80ab5bcd943419b8988234e8e19b83389edc542a
    . Go paste that number in http://blockchain.info
    . You'll land on a page that has the usual representation, in this case: 1CjLmm25ty4aSjxzG8Er4qSiQ63bqaQGA8

Got it, thanks.

I have to ask though, why not display the hashed address to begin with?  Just an extra step of complication?

Also, want to run it on my public address?  I am curious what other addresses are correlated with it...    If not, that's fine, I'll download it and attempt to compile myself.

[SgtSpike]

What does 0x80ab5bcd943419b8988234e8e19b83389edc542a mean?  It's not a Bitcoin address in the typical format.

Yeah, sorry about that, I should have dropped the 0x in front. Once
you remove the 0x, it's the ripemd-160 hash of the public key.

So:

    . Copy everything but the '0x' , in your example, that would be : 80ab5bcd943419b8988234e8e19b83389edc542a
    . Go paste that number in http://blockchain.info
    . You'll land on a page that has the usual representation, in this case: 1CjLmm25ty4aSjxzG8Er4qSiQ63bqaQGA8

Got it, thanks.

I have to ask though, why not display the hashed address to begin with?  Just an extra step of complication?

Not sure what you mean by hashed addresses.
What I am displaying *is* a hashed address.
You mean base58-encoded addresses like "1CjLmm25ty4aSjxzG8Er4qSiQ63bqaQGA8" ?
If so, then : these things are way too expensive to compute, this was designed for spped.

Also, want to run it on my public address?  I am curious what other addresses are correlated with it...    If not, that's fine, I'll download it and attempt to compile myself.

Give me a sec

Yes, sorry, I meant the base58 encoded address.  That makes sense though, I was just curious. 

[SgtSpike]

Also, want to run it on my public address?  I am curious what other addresses are correlated with it...    If not, that's fine, I'll download it and attempt to compile myself.

Here you go.
Even translated the result to the usual format.
I'd be curious to see if it worked. Let me know.

Code:

18TKNbSLTrd3a2W8mtoH5uNzFhWRWNcuHU
1HCPvWwBuUp7wqY69pgVeQZVv7KWj2foTC
15QeFbpe3o1rweuB4vHFG6FZHwZw6SKTe6
1EScB6WzkeDVUZwLSsAHxUNuRr8jN38MLm
17sUs5wQ9VrqxNwVg2ZGu2f77XW8QWzbAF
1MTvpvsZgjsWK588kgvm9RS35MQW46eyAW
1Ah5SFitc1zeWFck4CAdFaheAdU5RxiGq7
1M3z9w1tGaRKeGgoNCqXHLrfBRfyWh4WT6
14vWo67QCcWy6ET1Be7fsF4RW1QWNoJTwv
19yKfxN1xCcuk9nC6nDSV4PbpQifudjP4r
15mBZw6cjujDh61NBQu3pton78RAxdugmN
1MJ7EF8cDdkJY2YEYDJNFPiMg6oprMwb2K
1Ht1M1ndrRuKGWGuzBjxKfj6uWfrdVWQQr
1Q8mYsQhXREsHAmYWLqiD1iQjRns6pEE3v
13ViQh8GcaXHcSgwiM2szYfAHogdj2rjdu
17rgBsV5vhgFoXM67Xw9xLor5ZDG4Bbc5E
1LKbGSSZsBMixyLzKwt8k45VXY3Jxq6YBo
1Ce2FpRAgzqTPNVRE9oSTFtyHsjFQAeBwo
16jZP9v3RwoiVHYKTu89W2MvuHYHJjhV6n
1GVshgfm7dKWQaKnGfZpVfUgHMc667SSy2
16oGSr3q8H7m7mhFiRZgLZiBFPSLWxYJ6H
1MmRmJY9Kkepn98tUDw6mgRu8FDtqRv3Sq
1GcJTf4xWe8eNXmg37pJEYQz9zfynaWR1F
1EFwce8G1jEM384H4XAmgDg4oKS4CNYmjc
16EN1R3GUgTKPDbwYG8MuorjebKBqRfQQC
1L76oj8piiQjZTX81LvJTY8nqJCRQme24M
19YgjvW48TFLh7VDx4kqtoQdUntr6nbwvb
1M4Px4n2mudnCvMur56MgJmBoVmJBsUE1e
1BP5oZMunWkn4AumtVidxvSgdprMtc7TiS
14rh6vEzfpCz7KhiiMnBVFDPbSDtk5XhU8
11bUZzLby1Z1SAgS9UcHC3V2FXLTkJ5u73
1QBWU6Wzg7ER9ryiTqDsxsj452m7bMnGwK
13XPKVAR7AWzoiN1Pph7fGjcx7o1iMP6KK
16Fc6U3K9YBToqifj43jpiqynj7ug3MNye
1DwMrunRyaYPq94577j7Kj6zBf3uLhZDpW
18eWPzRWH12L4CmdeBc66eimdAKeyZWSi6
1N1VFUQLdQp6L2Z25H4zswoxc5L8NppSSJ
1K8F2dzKzvxixkMQhb8dq7LBCuKxCPFgW8
11Eqw5T99VsNtSzjBTrWGQHPsRrk4RmB2B
18LzGvrFokVvWZJAWw5wDsaErX5VrovGeQ
137DULuj5JhNc9V3Q4MEi1AJQX65QAG4xf
1EZp1rmPjBhZ5FvuAvfUqrZWM3uoF8p3h8
12pL3fwPGz9GsVE8L8v5agiQDpZhWYHGXV
1KxJd5v5orAoFuSkdUZSXc4UXmDa95bAND
1HjzgWVCkCXYW6LWX8THBtNwbKbYxUdpFe
1BfSfnbz7hbSgcXerakXuyf7FHRKcmpJcA
1GJFN1SN8Y9SJdSDK16uzGjV5wsGmEgMvn
1FAkUHN2rzf2PvLJ1E4NZ6r7zZsjw7Xa32
12LwQUq9HKsbaHmtPH46NgABYcp8A3sXWB
11FNssaGrVp9amTvi8sG6qb7mZB4d1VQYQ
1BrXK4M4c6xihr3gJj5mBht6xVoFzgtFvK
11D1dSTNJdhfpAf98USCJdUJp3917q3TJy
152SqwUX8R6PoEmJkxprU8Hr3GGbF5QpPn
1PRQk473DBQrdBhgAY6GcTXeezkyhPqvBx
1Hd3F12RoDQfHYShYBKPDC4NrxyPmk24Ne
1K5oHkRCZJeLB7p1E8yMEa4X7LAz3fbsCU
18Vv4whHTnGAe4f582trW4fLz3T3RUFhnC
1DP4bm1xBkKaRMcmZk996hsYSXcfmxH687
16TxHT4Par4xrKEoTYUDGGqcmZhBijTC2q
1CeADvshNvXVoxw1t1nqhwczYh5CefvuJB
1N5iVMVEEv6G2iSmJFPrPZ4mxkBCFdCmkA
14Z6Qa7Z1QNUTmEP5LkAjsXDzy151Jbwya
12ALSWvaVPekEkDWc5CrDDZvtSpDiWnWM8
1BG5Rbhc88VFLGCk5S3AmumgCa2ro1y1uz
17oFBQJau137iTPjkzyzayvTm7Wouu5DGn
1Ghu4ACM8UGtFxUmGx4BSJ3cA4hkB1iafy
1AcPdXqmJw67Mf6KvMo6d2iPZXoWLN2snM
18etxowpqFzYVbh2DwikcRvijfBZ6VWvjC
1D6FXvnELTfXSsZ8H5MSdfxFMofq1n4LBy
1JbkiKFGEqDHje1wSfHwiRbfxsbodmTK5s

Wow, that's quite a lot!  I can say that the below is my BitMinter payout address, so I know that that much worked.  Not at my wallet computer at the moment, but I'll check more of them out when I can.

1EScB6WzkeDVUZwLSsAHxUNuRr8jN38MLm

[etotheipi]

The C++ utilities I wrote for Armory are almost that fast (but in linux only, have no idea why windows is so damned slow!).  I buffer blk000X.dat into RAM 50 MB at a time, and then pass around pointers to this data and scan it directly -- to avoid any extra copies.  It's a lot of pointers...

Looks like you went and re-did by hand pretty much what
the OS does for free when you use memory-mapped files ...

I used memory-mapped files in version 0.60 to 0.77 of Armory.  Not only was it slow as dirt in Windows (though it worked brilliantly in Linux), it won't work on any 32-bit systems once the blockchain reaches 4GB+.  32-bit processes can't allocate enough virtual memory.  Hell, Win32 wouldn't take it at all even with 4 GB RAM, and a 1 GB blockchain (hence why Armory never worked in Win32).  Not only that, but mmap is OS-dependent.  If you want to use Map-Of-File-Views (the mmap equiv in Windows) you're going to deal with windows.h and LPWCSTRs.  

I'm so glad I got rid of that.  That's why I emulate it now, and it cleaned up my code a bit not having to branch base on OS.

I understand about the python wrappers.  You don't have to use the python, there's a pure C++ library underneath, though it's not obvious how to use it.  BlockUtilsTest.cpp has a lot of examples, but it's not intended for educational purposes:  it's really for me to do my unit-testing.

And in the end, I always approve of people digging in and implementing pieces of Bitcoin.  There's no better way to really undrestand it.

[SgtSpike]

Now what you need to do is group addresses based on the inputs and create a list of "wallets" in the same fashion. I have a tool that does that, to a point...

It already does that: that's what the "closure" callback does.

As a matter of fact, second teaser: here's the transitive closure of the Linode hack address
where slush lost BTC 3K+ (all these addresses are controlled by the same person).

Edit : changed it to look like usual addresses instead of hash160

Code:

Starting method "closure" on whole block chain

Building address equivalence graph ...
done.

Clustering 3824484 addresses ... 
done.

Showing address cluster for address eb1379c139bc2cd4cbfad64803e4b1a3738c85cd

113ym88TLsNUgQomGpVP97K1kwVLGk5r7V
11ikGgzoMdHAED9zW4NXFAN2ftDqvnABuq
11j58CBch6wHCDAVNCsckgHvDz6kD9aNHq
11k4H81T4ZRRDV7gFDcpfq2YcrSHp8AHrS
11kC2MhWdJyasnAQXRNzq3aUYxS5RZCESz
11kuSZPHuN7xCVtDSaow2UKLbJ3hX8T93Q
11LmyEU8KHVLj7rpE2zowZ5Je6WWtCJMhG
11ND26Juq3MkR7idBvqoJpxH3TppmbAYcR
11VinxmU2KPe2VkPaw8ouMNqSn8fqZNcfX
11W3eKv1E91P78tySdsgjtXfSN1DV1PNfc
12fDonKoU5hFpSJsFLEFRYrA3fVhCwEzDy
12oDTWFdtYwyUJfM1kGKcb3boQPxajgD58
12T5J1miwQgTK7mkqAhKm9nbdxMFP6SCE9
12XawgQWSrJFXwyoFtYdH5YBdkRBZMJ6oY
12yThcNZMuh8DQmw8MfX3vsdf89A5rMMec
131FBNVNsbzNwdh6uem9S9UBME3oiyiKT7
131ULLsD2zN7AB9KMuA9gH6vGv8QSoF39s
13DSq3J6u73rw9z5zXc2XBjSbZyZ5xarMQ
13JkZtPevDBeY6BNigxVrWQBuFBrqap927
13sxza6sdLr6qzEDA4raCZSbSGUq795wm7
145zQpb3wrJivNFQXpnxdEwjrisismDJLQ
1464JjjmsVfynGEykPVoKdmp1ChmYj7GsW
14fq1GqCLX4YayWz9pWiUoyBicCtLaMsGh
14J2Pkyne7XbCoT4aHsvUXM6913A4kNcwc
14NswevWjkamPqX4uQZatvYLGyvpLCpxsS
14ubFfgCjfKWts9wq9pGjGAtfvzi5ikrpm
151kxrqcoiquNUmkubCHFF1U7yEe72fnRz
15c3bKSX1UhuuSntNCUsWsBvzKxPw2PWW5
15gwFHzpTvAjFsJPDHG2QDwkQHYg9nSn3p
15mfn3efeGjmDZQEc5SZxXVqM3TLEDSk4d
15PW28638KQLFLWN64DyFXL9FkbSomBoP2
15RJx3MuMQY7kwmoP3yM8Fhc18c8XF54VA
16B31S2Q7oUsTnS1W65ycEnoRCeWEc3yVv
16Fieafx2vMyeqpkoBWaWXvNLf6BsrpuCM
1768xp9WcjTigs1H9XNdXq9nCfbCmZkz9V
17SDJ7FSmriKLe9yqsbNYJ94N65MwL8NbN
17wj9yoynVvhgi11svbK6PDDxtMxhQzJzQ
17YJDc1sUBqGg1rEQk2G2eLJtBXVFaN2HB
186TtMPXXQA1AFJrL3KQj4Veku872Q3Uqu
18dseD6wzXNnh2xWZg2L4LKdzLF5LG77gX
18F48fPRZEXWC2CHUvHbeeGUc16DPz6ByV
18NUp8Faisnip5MDTP7Xah7CzdPCFtJBWy
193SkpewDQXavnfxSJL2wecnx8udaTvWqP
19EJp2xvst7DrKmRyyLNK2xRsmeA3KkEXN
1ANL1cBJqdcaGxGCmsDc7e9xYGLeqsNkR9
1AUrDgnJ4qhA4eUaX2UPFUMpPmsNY6FeoH
1AWHAN6WySSCWir71YNSzGYfMDFA3qEUzb
1AxMp2swgVKbycg5sgNwwCh59PKhSQ7Bji
1B1tvB5mR8AdMxfY88P3EYQqic1vnPgvHP
1BDoXc5F4c3B7xhq9Ti67vTKGBL4VvuGee
1BYgT1Aa8uqepeDUDwygC1wKQQVA7iHLUr
1BZZYQrzUHEpSiJYFPGvZi2aGBCY1mhUvr
1CDcfFif32Eqnr1rZRRyJe9jvYS1wWMgrD
1ChYdQGcrXUmbjfXnKX5Ws8NC7QVgCMnsf
1CjLmm25ty4aSjxzG8Er4qSiQ63bqaQGA8
1CnJdd97uKz72kiHgwV3FrieDVsmKVvGXN
1D68upM6ad9GDZX1XRRvouRQSBG8FTUZ64
1D7eHcVqfmMsk6oLjBy5xNmojMAeafgqFd
1DFGAm3CXhsWNxRqFDcUru3RyTaG5KRN7U
1DGxyMnGgQwjD5YTTRaCHCUrDGY7GExEsi
1DMuVKe9PKpx3dbs2b2MnXuVmLfA4drHif
1DrFr4tKxzwW8pyiaET8B6ezMR1fUbxXBm
1FdkVB5vycm5AwJPfi2TRoi7kbpuspBUYK
1FHKfBVSUpCdipCjP4xANHtho8ZGpauTay
1FuTgYeeaLTW7UofPhJ7ePvV4sNPvoSFvr
1FWQ1pCEsJ2HLccn6ZX7PjHPuwjqxXEHw3
1FxzTCyknhEMwmvZVQkWSHJPuRkTAxLcrZ
1GGPVq6s7LCL4NoKHwLfovuWib6JqXgKp6
1Gh2gFT23asTnXH5EvQN87MeK8nZwZSgbf
1H2Ez99VeyYp9mCiUkLwLq8xFLHxJZuyLq
1HVEoKcpFp6EaCDEKNj1pkCvUwKpXGeY5R
1JdYpuFFEBcS6BnRpfeJj6z4a9ha553cBX
1JHvK2NeBnw3CoK7p5Vp7MxvwPCMc59e8b
1JLBnSF8kYD5f3b6fN5vvS3NddXi4ZZdmE
1Jrw9x9jvbVW872fqjLjGdpXd8jphxuQS9
1Kd74Jb8k3XicEcomvcdHJRnJ33Y2tnZCQ
1KGWneein7cyg8enNbDWTiX9EKbZZAdLyt
1KHebJWXvyeE1YB7h1sMzzywU6g4CXAenY
1KkPGLwopJMcFvpKdNMc3ew3PBzecm3QVp
1KnFe5ZCSmR5Pqs2fchRahX6ybhJNQHnJv
1KrV7UeoAJUQjCT672MsuuE3rDjTZZNpU9
1KSuLUcAJBe4Q75NJimYMBM1j7g272EJJc
1KUUQccsbw4thehTw1VeeyyE3T8225LBCC
1L1Y4H1scyWZCHmx2gXiM2guL7wR3zxmRB
1L7NR1yebk1KDNz3u1bK2cYNAspf2CGjL5
1LNXcXnYqyEB1vssC5s91hLj8yxFT4Rki8
1LubDSX88v16HSTHPZAgC9spSAoDz5bnwz
1LVpFavdTY2DHXExuC1JADh8kKMSz8ZBEo
1M7KnJG7Qe7Ymu4EsfEdPoU8jJv3tRvMBP
1MCPC3YfSG91FF9U2ndntUq1rBDFUHsV8Y
1MDQzpxEjFBt9cX8A8Lhr2a5taJynKCy5Y
1MdzcSx5bYxUPvFTKNHp5v7P7UbqiMHqKv
1MHqULM4JiMUYf9o9v19rMEJhxrhcUjHLV
1MisTTJvY7G1kWLG6vtHBMGDZKAwBfSiUB
1MKR29gF8YiF22FHZeQ5NeVVaN3F1Dewz9
1MSNDH4qWyps9Ch4z8WMk5kEDJJUXyZxt7
1N4D6E6ot99iTKNRprD6qqdu5vH6E2tVCj
1Njy6vDn8F163PhHvm744w6ehw5jj8bpzT
1NRy8GbX56MymBhDYMyqsNKwW9VupqKVG7
1NvrDeRNQ4be58s16tLvrEJGQPdrkakEpq
1P1y6bQ4psazd93XdvLMYuQGuEGyxqYM6G
1PubqviWyeYy9khFWZsGQ8g44tTSgBeRVq
1PY1emk6tJjtq6LAUSY3cbvBLJtQwpxk1c
1Q58P9vnMY3dfCeaftioox57V5NAJ9SQiE
1QGkj282ZygUnJq6XSfuzLhWscj2qRRjEF
1QKmhLwdstrEDxdyyzwirvEDGFGKu31xeF
Done in 20.621 seconds

Some of those addresses (for instance, 11k4H81T4ZRRDV7gFDcpfq2YcrSHp8AHrS) do not show up in any of the block explorers.  What's up with those?

[mrb]

Sub.
(Forum admins: please install the SMF Bookmark Mod to avoid these noisy "sub" posts.)

[SgtSpike]

Also, want to run it on my public address?  I am curious what other addresses are correlated with it...    If not, that's fine, I'll download it and attempt to compile myself.

Here you go.
Even translated the result to the usual format.
I'd be curious to see if it worked. Let me know.

Code:

18TKNbSLTrd3a2W8mtoH5uNzFhWRWNcuHU
1HCPvWwBuUp7wqY69pgVeQZVv7KWj2foTC
15QeFbpe3o1rweuB4vHFG6FZHwZw6SKTe6
1EScB6WzkeDVUZwLSsAHxUNuRr8jN38MLm
17sUs5wQ9VrqxNwVg2ZGu2f77XW8QWzbAF
1MTvpvsZgjsWK588kgvm9RS35MQW46eyAW
1Ah5SFitc1zeWFck4CAdFaheAdU5RxiGq7
1M3z9w1tGaRKeGgoNCqXHLrfBRfyWh4WT6
14vWo67QCcWy6ET1Be7fsF4RW1QWNoJTwv
19yKfxN1xCcuk9nC6nDSV4PbpQifudjP4r
15mBZw6cjujDh61NBQu3pton78RAxdugmN
1MJ7EF8cDdkJY2YEYDJNFPiMg6oprMwb2K
1Ht1M1ndrRuKGWGuzBjxKfj6uWfrdVWQQr
1Q8mYsQhXREsHAmYWLqiD1iQjRns6pEE3v
13ViQh8GcaXHcSgwiM2szYfAHogdj2rjdu
17rgBsV5vhgFoXM67Xw9xLor5ZDG4Bbc5E
1LKbGSSZsBMixyLzKwt8k45VXY3Jxq6YBo
1Ce2FpRAgzqTPNVRE9oSTFtyHsjFQAeBwo
16jZP9v3RwoiVHYKTu89W2MvuHYHJjhV6n
1GVshgfm7dKWQaKnGfZpVfUgHMc667SSy2
16oGSr3q8H7m7mhFiRZgLZiBFPSLWxYJ6H
1MmRmJY9Kkepn98tUDw6mgRu8FDtqRv3Sq
1GcJTf4xWe8eNXmg37pJEYQz9zfynaWR1F
1EFwce8G1jEM384H4XAmgDg4oKS4CNYmjc
16EN1R3GUgTKPDbwYG8MuorjebKBqRfQQC
1L76oj8piiQjZTX81LvJTY8nqJCRQme24M
19YgjvW48TFLh7VDx4kqtoQdUntr6nbwvb
1M4Px4n2mudnCvMur56MgJmBoVmJBsUE1e
1BP5oZMunWkn4AumtVidxvSgdprMtc7TiS
14rh6vEzfpCz7KhiiMnBVFDPbSDtk5XhU8
11bUZzLby1Z1SAgS9UcHC3V2FXLTkJ5u73
1QBWU6Wzg7ER9ryiTqDsxsj452m7bMnGwK
13XPKVAR7AWzoiN1Pph7fGjcx7o1iMP6KK
16Fc6U3K9YBToqifj43jpiqynj7ug3MNye
1DwMrunRyaYPq94577j7Kj6zBf3uLhZDpW
18eWPzRWH12L4CmdeBc66eimdAKeyZWSi6
1N1VFUQLdQp6L2Z25H4zswoxc5L8NppSSJ
1K8F2dzKzvxixkMQhb8dq7LBCuKxCPFgW8
11Eqw5T99VsNtSzjBTrWGQHPsRrk4RmB2B
18LzGvrFokVvWZJAWw5wDsaErX5VrovGeQ
137DULuj5JhNc9V3Q4MEi1AJQX65QAG4xf
1EZp1rmPjBhZ5FvuAvfUqrZWM3uoF8p3h8
12pL3fwPGz9GsVE8L8v5agiQDpZhWYHGXV
1KxJd5v5orAoFuSkdUZSXc4UXmDa95bAND
1HjzgWVCkCXYW6LWX8THBtNwbKbYxUdpFe
1BfSfnbz7hbSgcXerakXuyf7FHRKcmpJcA
1GJFN1SN8Y9SJdSDK16uzGjV5wsGmEgMvn
1FAkUHN2rzf2PvLJ1E4NZ6r7zZsjw7Xa32
12LwQUq9HKsbaHmtPH46NgABYcp8A3sXWB
11FNssaGrVp9amTvi8sG6qb7mZB4d1VQYQ
1BrXK4M4c6xihr3gJj5mBht6xVoFzgtFvK
11D1dSTNJdhfpAf98USCJdUJp3917q3TJy
152SqwUX8R6PoEmJkxprU8Hr3GGbF5QpPn
1PRQk473DBQrdBhgAY6GcTXeezkyhPqvBx
1Hd3F12RoDQfHYShYBKPDC4NrxyPmk24Ne
1K5oHkRCZJeLB7p1E8yMEa4X7LAz3fbsCU
18Vv4whHTnGAe4f582trW4fLz3T3RUFhnC
1DP4bm1xBkKaRMcmZk996hsYSXcfmxH687
16TxHT4Par4xrKEoTYUDGGqcmZhBijTC2q
1CeADvshNvXVoxw1t1nqhwczYh5CefvuJB
1N5iVMVEEv6G2iSmJFPrPZ4mxkBCFdCmkA
14Z6Qa7Z1QNUTmEP5LkAjsXDzy151Jbwya
12ALSWvaVPekEkDWc5CrDDZvtSpDiWnWM8
1BG5Rbhc88VFLGCk5S3AmumgCa2ro1y1uz
17oFBQJau137iTPjkzyzayvTm7Wouu5DGn
1Ghu4ACM8UGtFxUmGx4BSJ3cA4hkB1iafy
1AcPdXqmJw67Mf6KvMo6d2iPZXoWLN2snM
18etxowpqFzYVbh2DwikcRvijfBZ6VWvjC
1D6FXvnELTfXSsZ8H5MSdfxFMofq1n4LBy
1JbkiKFGEqDHje1wSfHwiRbfxsbodmTK5s

It got nearly all of my often-used addresses, but had a LOT of false positives.  Not sure why that would be...
As an example, 11bUZzLby1Z1SAgS9UcHC3V2FXLTkJ5u73 is an address that is not in my address book.  And interestingly, a google search turns up nothing but this post, indicating that once again, you're somehow getting addresses that don't exist in the blockchain yet.

Hmmm...

[SgtSpike]

It got nearly all of my often-used addresses, but had a LOT of false positives. 

Also: with the satoshi client there are quite a few addresses that actually are
in your wallet but that you never "see". These are called "change" address,
they're created automatically by the client when you send money.

The way to check if they're yours is to run ./bitcoind validateaddress <address>
and to check the "ismine" field in the JSON output.

Ah yeah, good point.  I wasn't looking for change addresses.

[check_status]

Will it work with yED to graph the addresses visually?

For Ubuntu 11.04 64 bit

To download yED:
http://www.yworks.com/en/products_download.php?file=yEd-3.9.2.zip

To run yED:
Start yEd from commandline using,

Code:

java -Xmx512m -jar /path/to/yed.jar

[check_status]

I wouldn't mind doing something like this:

[check_status]

The vertices are hash160 or, in the case of the pic, base58 bitcoin addresses. How big a vertex appears, like in the pic, is probably determined by number of transactions. The edges are transactions, send and receive, colored by amounts transferred, small amounts grey, large amounts colors.

I was thrown out of geometry class in highschool, so I didn't take it.

[check_status]

OK, now that's clear: what you want is a full dump of the blockchain in graph form.

So, as it is today, the code can't directly generate that, but it'd be a feature super simple
to add to the 'allBalances' callback since the code basically walks that exact graph to generate
the final balances.

That being said, this is going to be a *huge* dump. To get an idea, here's the current
blockchain stats gathered by blockparser:
     - 4.407 Million unique addresses received BTC
     - the blockchain contains 17.826 Million address spends

In other words, not even taking into account the complex kind of transactions
(send fromMany toMany) your graph would have on the order of 4.5M vertices
and on the order of 18M edges ...

Unless your graph software was optimized to handle very large graphs, you're not
going to get much out of it.

Hmm...I guess it might be better to dump as text and then crawl for the specific info needed. I certainly can't load the whole blockchain into the grapher, I don't have a Cray nearby.

Another way would be to somehow figure out a way to restrict the graph dump to
a nearby neighborhood of a given address (for example N hops).

This sounds a bit more reasonable.

Feel free to submit a patch.

That means it's somewhat difficult or tedious to implement.

I was thrown out of geometry class in highschool, so I didn't take it.

Since when is graph theory taught during geometry classes ?

Neither had I computer science. Biology, tables yes, graphing no.

[check_status]

We'll see how far past 'hello world!' I get first.

[EnergyVampire]

Very nice!

[muyuu]

Looks good. Cheers.

[Sukrim]

Also, want to run it on my public address?  I am curious what other addresses are correlated with it...    If not, that's fine, I'll download it and attempt to compile myself.

Here you go.
Even translated the result to the usual format.
I'd be curious to see if it worked. Let me know.

Code:

18TKNbSLTrd3a2W8mtoH5uNzFhWRWNcuHU
1HCPvWwBuUp7wqY69pgVeQZVv7KWj2foTC
15QeFbpe3o1rweuB4vHFG6FZHwZw6SKTe6
1EScB6WzkeDVUZwLSsAHxUNuRr8jN38MLm
17sUs5wQ9VrqxNwVg2ZGu2f77XW8QWzbAF
1MTvpvsZgjsWK588kgvm9RS35MQW46eyAW
1Ah5SFitc1zeWFck4CAdFaheAdU5RxiGq7
1M3z9w1tGaRKeGgoNCqXHLrfBRfyWh4WT6
14vWo67QCcWy6ET1Be7fsF4RW1QWNoJTwv
19yKfxN1xCcuk9nC6nDSV4PbpQifudjP4r
15mBZw6cjujDh61NBQu3pton78RAxdugmN
1MJ7EF8cDdkJY2YEYDJNFPiMg6oprMwb2K
1Ht1M1ndrRuKGWGuzBjxKfj6uWfrdVWQQr
1Q8mYsQhXREsHAmYWLqiD1iQjRns6pEE3v
13ViQh8GcaXHcSgwiM2szYfAHogdj2rjdu
17rgBsV5vhgFoXM67Xw9xLor5ZDG4Bbc5E
1LKbGSSZsBMixyLzKwt8k45VXY3Jxq6YBo
1Ce2FpRAgzqTPNVRE9oSTFtyHsjFQAeBwo
16jZP9v3RwoiVHYKTu89W2MvuHYHJjhV6n
1GVshgfm7dKWQaKnGfZpVfUgHMc667SSy2
16oGSr3q8H7m7mhFiRZgLZiBFPSLWxYJ6H
1MmRmJY9Kkepn98tUDw6mgRu8FDtqRv3Sq
1GcJTf4xWe8eNXmg37pJEYQz9zfynaWR1F
1EFwce8G1jEM384H4XAmgDg4oKS4CNYmjc
16EN1R3GUgTKPDbwYG8MuorjebKBqRfQQC
1L76oj8piiQjZTX81LvJTY8nqJCRQme24M
19YgjvW48TFLh7VDx4kqtoQdUntr6nbwvb
1M4Px4n2mudnCvMur56MgJmBoVmJBsUE1e
1BP5oZMunWkn4AumtVidxvSgdprMtc7TiS
14rh6vEzfpCz7KhiiMnBVFDPbSDtk5XhU8
11bUZzLby1Z1SAgS9UcHC3V2FXLTkJ5u73
1QBWU6Wzg7ER9ryiTqDsxsj452m7bMnGwK
13XPKVAR7AWzoiN1Pph7fGjcx7o1iMP6KK
16Fc6U3K9YBToqifj43jpiqynj7ug3MNye
1DwMrunRyaYPq94577j7Kj6zBf3uLhZDpW
18eWPzRWH12L4CmdeBc66eimdAKeyZWSi6
1N1VFUQLdQp6L2Z25H4zswoxc5L8NppSSJ
1K8F2dzKzvxixkMQhb8dq7LBCuKxCPFgW8
11Eqw5T99VsNtSzjBTrWGQHPsRrk4RmB2B
18LzGvrFokVvWZJAWw5wDsaErX5VrovGeQ
137DULuj5JhNc9V3Q4MEi1AJQX65QAG4xf
1EZp1rmPjBhZ5FvuAvfUqrZWM3uoF8p3h8
12pL3fwPGz9GsVE8L8v5agiQDpZhWYHGXV
1KxJd5v5orAoFuSkdUZSXc4UXmDa95bAND
1HjzgWVCkCXYW6LWX8THBtNwbKbYxUdpFe
1BfSfnbz7hbSgcXerakXuyf7FHRKcmpJcA
1GJFN1SN8Y9SJdSDK16uzGjV5wsGmEgMvn
1FAkUHN2rzf2PvLJ1E4NZ6r7zZsjw7Xa32
12LwQUq9HKsbaHmtPH46NgABYcp8A3sXWB
11FNssaGrVp9amTvi8sG6qb7mZB4d1VQYQ
1BrXK4M4c6xihr3gJj5mBht6xVoFzgtFvK
11D1dSTNJdhfpAf98USCJdUJp3917q3TJy
152SqwUX8R6PoEmJkxprU8Hr3GGbF5QpPn
1PRQk473DBQrdBhgAY6GcTXeezkyhPqvBx
1Hd3F12RoDQfHYShYBKPDC4NrxyPmk24Ne
1K5oHkRCZJeLB7p1E8yMEa4X7LAz3fbsCU
18Vv4whHTnGAe4f582trW4fLz3T3RUFhnC
1DP4bm1xBkKaRMcmZk996hsYSXcfmxH687
16TxHT4Par4xrKEoTYUDGGqcmZhBijTC2q
1CeADvshNvXVoxw1t1nqhwczYh5CefvuJB
1N5iVMVEEv6G2iSmJFPrPZ4mxkBCFdCmkA
14Z6Qa7Z1QNUTmEP5LkAjsXDzy151Jbwya
12ALSWvaVPekEkDWc5CrDDZvtSpDiWnWM8
1BG5Rbhc88VFLGCk5S3AmumgCa2ro1y1uz
17oFBQJau137iTPjkzyzayvTm7Wouu5DGn
1Ghu4ACM8UGtFxUmGx4BSJ3cA4hkB1iafy
1AcPdXqmJw67Mf6KvMo6d2iPZXoWLN2snM
18etxowpqFzYVbh2DwikcRvijfBZ6VWvjC
1D6FXvnELTfXSsZ8H5MSdfxFMofq1n4LBy
1JbkiKFGEqDHje1wSfHwiRbfxsbodmTK5s

Wow, that's quite a lot!  I can say that the below is my BitMinter payout address, so I know that that much worked.  Not at my wallet computer at the moment, but I'll check more of them out when I can.

1EScB6WzkeDVUZwLSsAHxUNuRr8jN38MLm

Working from tat bitminter address I got

Code:

1GVshgfm7dKWQaKnGfZpVfUgHMc667SSy2
17oFBQJau137iTPjkzyzayvTm7Wouu5DGn
1N5iVMVEEv6G2iSmJFPrPZ4mxkBCFdCmkA
1K5oHkRCZJeLB7p1E8yMEa4X7LAz3fbsCU
1BP5oZMunWkn4AumtVidxvSgdprMtc7TiS
19yKfxN1xCcuk9nC6nDSV4PbpQifudjP4r
1Eqw5T99VsNtSzjBTrWGQHPsRrk4RmB2B
1HCPvWwBuUp7wqY69pgVeQZVv7KWj2foTC
18TKNbSLTrd3a2W8mtoH5uNzFhWRWNcuHU
1Ce2FpRAgzqTPNVRE9oSTFtyHsjFQAeBwo
17rgBsV5vhgFoXM67Xw9xLor5ZDG4Bbc5E
15QeFbpe3o1rweuB4vHFG6FZHwZw6SKTe6
1PRQk473DBQrdBhgAY6GcTXeezkyhPqvBx
1N1VFUQLdQp6L2Z25H4zswoxc5L8NppSSJ
16jZP9v3RwoiVHYKTu89W2MvuHYHJjhV6n
1QBWU6Wzg7ER9ryiTqDsxsj452m7bMnGwK
16oGSr3q8H7m7mhFiRZgLZiBFPSLWxYJ6H
1GcJTf4xWe8eNXmg37pJEYQz9zfynaWR1F
1Q8mYsQhXREsHAmYWLqiD1iQjRns6pEE3v
1Ah5SFitc1zeWFck4CAdFaheAdU5RxiGq7
137DULuj5JhNc9V3Q4MEi1AJQX65QAG4xf
18Vv4whHTnGAe4f582trW4fLz3T3RUFhnC
1GJFN1SN8Y9SJdSDK16uzGjV5wsGmEgMvn
15mBZw6cjujDh61NBQu3pton78RAxdugmN
1BfSfnbz7hbSgcXerakXuyf7FHRKcmpJcA
1FNssaGrVp9amTvi8sG6qb7mZB4d1VQYQ
1EFwce8G1jEM384H4XAmgDg4oKS4CNYmjc
17sUs5wQ9VrqxNwVg2ZGu2f77XW8QWzbAF
18etxowpqFzYVbh2DwikcRvijfBZ6VWvjC
1BrXK4M4c6xihr3gJj5mBht6xVoFzgtFvK
1Hd3F12RoDQfHYShYBKPDC4NrxyPmk24Ne
12pL3fwPGz9GsVE8L8v5agiQDpZhWYHGXV
1Ghu4ACM8UGtFxUmGx4BSJ3cA4hkB1iafy
1MTvpvsZgjsWK588kgvm9RS35MQW46eyAW
1FAkUHN2rzf2PvLJ1E4NZ6r7zZsjw7Xa32
1M3z9w1tGaRKeGgoNCqXHLrfBRfyWh4WT6
12ALSWvaVPekEkDWc5CrDDZvtSpDiWnWM8
1EZp1rmPjBhZ5FvuAvfUqrZWM3uoF8p3h8
16EN1R3GUgTKPDbwYG8MuorjebKBqRfQQC
1M4Px4n2mudnCvMur56MgJmBoVmJBsUE1e
13XPKVAR7AWzoiN1Pph7fGjcx7o1iMP6KK
1MJ7EF8cDdkJY2YEYDJNFPiMg6oprMwb2K
1L76oj8piiQjZTX81LvJTY8nqJCRQme24M
1JbkiKFGEqDHje1wSfHwiRbfxsbodmTK5s
18LzGvrFokVvWZJAWw5wDsaErX5VrovGeQ
16Fc6U3K9YBToqifj43jpiqynj7ug3MNye
19YgjvW48TFLh7VDx4kqtoQdUntr6nbwvb
1D1dSTNJdhfpAf98USCJdUJp3917q3TJy
1EScB6WzkeDVUZwLSsAHxUNuRr8jN38MLm
12LwQUq9HKsbaHmtPH46NgABYcp8A3sXWB
152SqwUX8R6PoEmJkxprU8Hr3GGbF5QpPn
1MmRmJY9Kkepn98tUDw6mgRu8FDtqRv3Sq
13ViQh8GcaXHcSgwiM2szYfAHogdj2rjdu
1LKbGSSZsBMixyLzKwt8k45VXY3Jxq6YBo
1CeADvshNvXVoxw1t1nqhwczYh5CefvuJB
1BG5Rbhc88VFLGCk5S3AmumgCa2ro1y1uz
14Z6Qa7Z1QNUTmEP5LkAjsXDzy151Jbwya
16TxHT4Par4xrKEoTYUDGGqcmZhBijTC2q
1DP4bm1xBkKaRMcmZk996hsYSXcfmxH687
1DwMrunRyaYPq94577j7Kj6zBf3uLhZDpW
1KxJd5v5orAoFuSkdUZSXc4UXmDa95bAND
1AcPdXqmJw67Mf6KvMo6d2iPZXoWLN2snM
1K8F2dzKzvxixkMQhb8dq7LBCuKxCPFgW8
14rh6vEzfpCz7KhiiMnBVFDPbSDtk5XhU8
1D6FXvnELTfXSsZ8H5MSdfxFMofq1n4LBy
14vWo67QCcWy6ET1Be7fsF4RW1QWNoJTwv
18eWPzRWH12L4CmdeBc66eimdAKeyZWSi6
1HjzgWVCkCXYW6LWX8THBtNwbKbYxUdpFe
1Ht1M1ndrRuKGWGuzBjxKfj6uWfrdVWQQr
1bUZzLby1Z1SAgS9UcHC3V2FXLTkJ5u73

with a current balance of 10.46293721 BTC

I still need to change my own script to output hashes instead of addresses though, so I can compare with this (much faster but limited to linux) program.

[flatfly]

Is it possible to answer the following question with this tool?

Show all addresses currently containing coins from the (latest) Bitcoinica hack.

[copumpkin]

I haven't had a chance to look at your code yet, but do you use Tarjan's union-find to compute the "wallet-sets"? I've been meaning to write a tool that does that for a while.

Edit: oh, you're using boost's connected_components. I think they use union-find for incremental_components.

[copumpkin]

I haven't had a chance to look at your code yet, but do you use Tarjan's union-find to compute the "wallet-sets"? I've been meaning to write a tool that does that for a while.

Edit: oh, you're using boost's connected_components. I think they use union-find for incremental_components.

Yes I do because I build the entire graph in the first place.

Not sure what algorithm they use in there.

Also, given that I build the whole graph, not sure why using
TUFA has any advantage over a dumb [D|B]FS mark and sweep ...

Yeah, it doesn't. I wasn't planning on maintaining the entire graph, so can get away with it.

[copumpkin]

I haven't had a chance to look at your code yet, but do you use Tarjan's union-find to compute the "wallet-sets"? I've been meaning to write a tool that does that for a while.

Edit: oh, you're using boost's connected_components. I think they use union-find for incremental_components.

Yes I do because I build the entire graph in the first place.

Not sure what algorithm they use in there.

Also, given that I build the whole graph, not sure why using
TUFA has any advantage over a dumb [D|B]FS mark and sweep ...

Yeah, it doesn't. I wasn't planning on maintaining the entire graph, so can get away with it.

Turns out that the graph will grow so fat at some point that it might
very well be necessary to move to some sort of incremental data
structure ... that being said, since there is no way to know that two
disjoin subsets will or won't merge until the very last tx in the chain,
I don't think it's possible to get away with _not_ building the whole
graph in the first place.

Well, I haven't tried implementing it yet, but my thinking was that the union-find structure represents just as much of the graph as we need. We basically have an equivalence of "has been used as an input in the same transaction as" and we're building equivalence classes.

So basically, you make a pass over the data building up an associative map structure from each address to something your union-find algorithm knows how to deal with (I'm language-agnostic ). As you process transactions in the block building these things up, you make sure to call union on your union-find structure to join classes together if you see them as inputs to the same transaction. This is linear in the number of inputs, not quadratic, because the structure knows what's already in the class. Once you've made your pass over the blockchain, you're left with a big map containing all input addresses ever seen and your union-find structure. Now all you need to do is traverse your map of addresses and look up (using find to find a representative element of your equivalence class) which set each element is in, and group on that. The resulting sets should be your connected components. You could then make your map from addresses point to the set it belongs to for easy lookups, or do whatever.

If I'm not spouting nonsense, that would allow us to run the set-building in O(n * alpha(n)) time, where n is the total number of inputs to all transactions. You'd use O(total unique input addresses) space for it, and it should have fairly low constant factors.

I'll probably try implementing that in a bit, but I need to finish decoding the scripts first

[copumpkin]

I'll probably try implementing that in a bit, but I need to finish decoding the scripts first

I'll have to read your explanation about the algorithm you describe when my
mind is a little more alert than right now (4am here).

However, for the script decoding part, I would suggest looking at my code,
in file util.cpp, routing solveOutputScript and showScript. They're very simple
and pretty much self-contained (unlike the satoshi client code which has lots
of layers of abstraction and makes things a tad hard to follow linearly).

Cool, will do. I'm doing it in a completely different language (I dare not say which, lest people judge me! ), but having some nicely written code to look at will be a great help, thanks.

[pancake]

can't compile on linux-x86-32:

$ make
c++ -- cb/closure.cpp
c++ -- cb/simpleStats.cpp
c++ -- cb/taint.cpp
c++ -- cb/transactions.cpp
c++ -- cb/allBalances.cpp
c++ -- callback.cpp
c++ -- opcodes.cpp
c++ -- parser.cpp
cb/allBalances.cpp:1:0: sorry, unimplemented: 64-bit mode not compiled in
opcodes.cpp:1:0: sorry, unimplemented: 64-bit mode not compiled in
callback.cpp:1:0: sorry, unimplemented: 64-bit mode not compiled in
cb/simpleStats.cpp:1:0: sorry, unimplemented: 64-bit mode not compiled in
cb/transactions.cpp:1:0: sorry, unimplemented: 64-bit mode not compiled in
parser.cpp:1:0: sorry, unimplemented: 64-bit mode not compiled in
cb/closure.cpp:1:0: sorry, unimplemented: 64-bit mode not compiled in
cb/taint.cpp:1:0: sorry, unimplemented: 64-bit mode not compiled in
make: *** [.objs/transactions.o] Error 1
make: *** Waiting for unfinished jobs....
make: *** [.objs/opcodes.o] Error 1
make: *** [.objs/simpleStats.o] Error 1
make: *** [.objs/taint.o] Error 1
make: *** [.objs/allBalances.o] Error 1
make: *** [.objs/callback.o] Error 1
make: *** [.objs/closure.o] Error 1
make: *** [.objs/parser.o] Error 1

[pancake]

can't compile on linux-x86-32:

The fix is to remove the -m64 flag from the Makefile. Should I do a pull request?

[runlinux]

I really need to play with this. My tool pings blockexplorer for its data... lol

[organofcorti]

Anyone managed to get this working on BSD or whatever it is that macs use?

[gweedo]

Anyone managed to get this working on BSD or whatever it is that macs use?

if your on lion you need to use Xcode and inside of xcode install the command line tools to get it working
and also use homebrew to install dependencies macports doesn't work as well for me at least.

[organofcorti]

Anyone managed to get this working on BSD or whatever it is that macs use?

if your on lion you need to use Xcode and inside of xcode install the command line tools to get it working
and also use homebrew to install dependencies macports doesn't work as well for me at least.

Great! Thanks for that. I'm on snow leopard and have installed compiling tools, so I'll give it a go.

[gweedo]

Anyone managed to get this working on BSD or whatever it is that macs use?

if your on lion you need to use Xcode and inside of xcode install the command line tools to get it working
and also use homebrew to install dependencies macports doesn't work as well for me at least.

Great! Thanks for that. I'm on snow leopard and have installed compiling tools, so I'll give it a go.

actually I forgot you need to update your gcc to gcc 4.5
so do...

Code:

% brew tap homebrew/versions
% brew install gcc45

[Jouke]

I am having trouble compiling.
http://pastebin.com/TrfSecqA

[Jouke]

I am having trouble compiling.
http://pastebin.com/TrfSecqA

From the README:

Code:

sudo apt-get install libssl-dev build-essential g++-4.4 libboost-all-dev libsparsehash-dev git-core perl

Did you do that ?

yes, updated better pastebin btw

http://pastebin.com/tQUq28Zk

[Jouke]

almost complete fresh install of ubuntu 64 bit.

[runlinux]

installing 12.04 x64 in VMWare player.

I really want to play with this. I need to brush up on my C++ skills... haven't used it in about 10 years..

[runlinux]

yay! compiled on my first try!

EDIT:

snap... I need to DL the blockchain?! ungh... (What was I thinking? lol)

copy and paste from host machine...

[runlinux]

yup, i found that out...

[Jouke]

xubuntu 12.10 or something like that

Code:

In file included from cb/allBalances.cpp:4:0:
./util.h: In instantiation of ‘void GoogMap<Key, Value, Hasher, Equal>::Map::setEmptyKey(const Key&) [with Key = const unsigned char*; Value = Addr*; Hasher = Hash160Hasher; Equal = Hash160Equal]’:
cb/allBalances.cpp:67:37:   required from here
./util.h:122:21: error: ‘set_empty_key’ was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive]
./util.h:122:21: note: declarations in dependent base ‘google::dense_hash_map<const unsigned char*, Addr*, Hash160Hasher, Hash160Equal, google::libc_allocator_with_realloc<std::pair<const unsigned char* const, Addr*> > >’ are not found by unqualified lookup
./util.h:122:21: note: use ‘this->set_empty_key’ instead
In file included from cb/taint.cpp:25:0:
./util.h: In instantiation of ‘void GoogMap<Key, Value, Hasher, Equal>::Map::setEmptyKey(const Key&) [with Key = const unsigned char*; Value = long double; Hasher = Hash256Hasher; Equal = Hash256Equal]’:
cb/taint.cpp:70:35:   required from here
./util.h:122:21: error: ‘set_empty_key’ was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive]
./util.h:122:21: note: declarations in dependent base ‘google::dense_hash_map<const unsigned char*, long double, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std::pair<const unsigned char* const, long double> > >’ are not found by unqualified lookup
./util.h:122:21: note: use ‘this->set_empty_key’ instead
In file included from cb/transactions.cpp:5:0:
./util.h: In instantiation of ‘void GoogMap<Key, Value, Hasher, Equal>::Map::setEmptyKey(const Key&) [with Key = const unsigned char*; Value = int; Hasher = Hash160Hasher; Equal = Hash160Equal]’:
cb/transactions.cpp:64:37:   required from here
./util.h:122:21: error: ‘set_empty_key’ was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive]
./util.h:122:21: note: declarations in dependent base ‘google::dense_hash_map<const unsigned char*, int, Hash160Hasher, Hash160Equal, google::libc_allocator_with_realloc<std::pair<const unsigned char* const, int> > >’ are not found by unqualified lookup
./util.h:122:21: note: use ‘this->set_empty_key’ instead
make: *** [.objs/taint.o] Error 1
make: *** Waiting for unfinished jobs....
make: *** [.objs/allBalances.o] Error 1
make: *** [.objs/transactions.o] Error 1
In file included from parser.cpp:2:0:
./util.h: In instantiation of ‘void GoogMap<Key, Value, Hasher, Equal>::Map::setEmptyKey(const Key&) [with Key = const unsigned char*; Value = const unsigned char*; Hasher = Hash256Hasher; Equal = Hash256Equal]’:
parser.cpp:449:29:   required from here
./util.h:122:21: error: ‘set_empty_key’ was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive]
./util.h:122:21: note: declarations in dependent base ‘google::dense_hash_map<const unsigned char*, const unsigned char*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std::pair<const unsigned char* const, const unsigned char*> > >’ are not found by unqualified lookup
./util.h:122:21: note: use ‘this->set_empty_key’ instead
./util.h: In instantiation of ‘void GoogMap<Key, Value, Hasher, Equal>::Map::setEmptyKey(const Key&) [with Key = const unsigned char*; Value = Block*; Hasher = Hash256Hasher; Equal = Hash256Equal]’:
parser.cpp:450:32:   required from here
./util.h:122:21: error: ‘set_empty_key’ was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive]
./util.h:122:21: note: declarations in dependent base ‘google::dense_hash_map<const unsigned char*, Block*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std::pair<const unsigned char* const, Block*> > >’ are not found by unqualified lookup
./util.h:122:21: note: use ‘this->set_empty_key’ instead
make: *** [.objs/parser.o] Error 1
In file included from cb/closure.cpp:4:0:
./util.h: In instantiation of ‘void GoogMap<Key, Value, Hasher, Equal>::Map::setEmptyKey(const Key&) [with Key = const unsigned char*; Value = long unsigned int; Hasher = Hash160Hasher; Equal = Hash160Equal]’:
cb/closure.cpp:43:38:   required from here
./util.h:122:21: error: ‘set_empty_key’ was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive]
./util.h:122:21: note: declarations in dependent base ‘google::dense_hash_map<const unsigned char*, long unsigned int, Hash160Hasher, Hash160Equal, google::libc_allocator_with_realloc<std::pair<const unsigned char* const, long unsigned int> > >’ are not found by unqualified lookup
./util.h:122:21: note: use ‘this->set_empty_key’ instead
make: *** [.objs/closure.o] Error 1

[Jouke]

Ok, it does not compile on g++ 4.7, it does on g++ 4.5

[Jouke]

Ok, it does not compile on g++ 4.7, it does on g++ 4.5

OK.
Let me install 4.7 and see what's what.

Fixed and fix pushed to github

Compiles nicely

[Jouke]

Is it possible to point it to an other dir then ~/.bitcoin ?


Tried it some other way.

I have a specific problem. I had a blk0001.dat already when I first started the bitcoin-client. Instead of using that file it made a blk0002.dat and recently it made a blk0003.dat. The parser won't open the blockchain file: "Invalid argument: failed to open block chain file /home/jouke/.bitcoin/blk0001.dat"

I guessed that 0002 was actually 0001 and tried to symlink 0002 to 0001 and 0003 to 0002 in a separate directory, but your parser still won't open the file.

Any sugestions?

[dooglus]

Is it possible to point it to an other dir then ~/.bitcoin ?


Tried it some other way.

I have a specific problem. I had a blk0001.dat already when I first started the bitcoin-client. Instead of using that file it made a blk0002.dat and recently it made a blk0003.dat. The parser won't open the blockchain file: "Invalid argument: failed to open block chain file /home/jouke/.bitcoin/blk0001.dat"

I guessed that 0002 was actually 0001 and tried to symlink 0002 to 0001 and 0003 to 0002 in a separate directory, but your parser still won't open the file.

Any sugestions?

Temporarily set HOME to wherever you want it to read the blockchain from:

$ HOME=/path/to/folder ./parser closure
[...]
fatal: No such file or directory: failed to open block chain file /path/to/folder/.bitcoin/blk0001.dat

I doubt the parser made any blockchain files.  It opens the files readonly, which won't create any files.

About a week ago the blockchain file grew bigger than 2GB and so a 2nd file was created by bitcoin-qt:

$ ls -l blk*
-rw------- 1 chris chris 2097307549 2012-07-12 20:07 blk0001.dat
-rw------- 1 chris chris  104659251 2012-07-19 02:52 blk0002.dat
-rw------- 1 chris chris  751022080 2012-07-19 02:52 blkindex.dat

I don't know where the 0003 file came from though!

[Jouke]

Temporarily set HOME to wherever you want it to read the blockchain from:

Already did that. That's where I symlinked the file to.

I doubt the parser made any blockchain files.  It opens the files readonly, which won't create any files.

I didn't say it did?

About a week ago the blockchain file grew bigger than 2GB and so a 2nd file was created by bitcoin-qt:

$ ls -l blk*
-rw------- 1 chris chris 2097307549 2012-07-12 20:07 blk0001.dat
-rw------- 1 chris chris  104659251 2012-07-19 02:52 blk0002.dat
-rw------- 1 chris chris  751022080 2012-07-19 02:52 blkindex.dat

I don't know where the 0003 file came from though!

I Know a 2nd file was created.

Because I already had a blk0001.dat, my client started with blk0002.dat.

[runlinux]

How hard would it be to add address balance to the closure method? (and a total balance at the bottom with address count)

I can't figure it out. My C++ is old and unused... - at least I can read the code!

[runlinux]

Thanks for that! i'll modify the output so its just the address. from there a simple script should do what i need

[runlinux]

OK, got a question now:

Why does this address: 12BdKh5HHUyGFduGm6y8nY4KS1LxzsduXo

Link to this address: 12cbQLTFMXRnSzktFkuoG3eHoMeFtpTu3S

Closure reports it has no outputs then returns that one.

On a side note, script written and code modified to take a wallet address, and then just output the balance of the final wallet

Code:

runlinux@ubuntu:~/blockparser$ ./wallet.sh 1MrPonziaM4QT2S7SdPEKQH88BGa4LRHJU

    transactions  = 211
    received      =      207.30710185
    spent         =      207.30680287
    balance       =        0.00029898

[dooglus]

Why does this address: 12BdKh5HHUyGFduGm6y8nY4KS1LxzsduXo

Link to this address: 12cbQLTFMXRnSzktFkuoG3eHoMeFtpTu3S

Closure reports it has no outputs then returns that one.

I see the same.

It tells me:

Code:

error: specified key was never used in a TX output

but this transaction has 12BdKh5HHUyGFduGm6y8nY4KS1LxzsduXo as an output, so the message is a little confusing.

And then the address isn't in its own reported cluster - that's got to be a bug.

[Sukrim]

What's your definition of "taint"?

If there is 6% taint from pirate in that huge address, does that mean 6% of these BTC were at one point in time owned by a pirate address or that pirate deposited 6% of these coins to that address?

If it is the first definition, isn't there the possibility to have >100% total taint? E.g. pirate sends BTC to MtGox and then MtGox sends these to the huge address - now they are "tainted" by pirate AND "tainted" by MtGox...

What I would still like to do is to enter an address, get the closure of that and then get the closures of every address where money was sent to or received from that cluster (ideally I could enter how many levels deep, with "0" being the current closure setting).

[etotheipi]

I'm sure you know I've spent a lot of time doing blockchain scanning, and attempting to optimize it in various ways in Armory.  In the end, the fastest I could achieve was about 20-25s on a full blk0001.dat (2GB) on Linux-64bit using memory-mapping.  I couldn't achieve the same thing in Windows (mmap'ing seems to be garbage there).   I just wanted to share my experience from playing with leveldb, how awesomely optimized it is.

First, I iterated through blk0001.dat and stored all transactions in a leveldb database using Key:TxHash-->Value:RawTx.   I plugged 4 addresses into a set<BinaryData> object.  I then iterated (out of order) through the entire database, pulling each transaction into memory from the DB, and then searching for TxOuts matching any of those 4 addresses.  This took 8.0 seconds! (but was only the TxOuts).  This included putting the TxOuts into a map<OutPoint,uint64_t> (OutPoint, nBTC), so that I could then scan the entire DB again and search for TxIns. 

At the end of both scans, I had the balance and unspent TxOut list from those four addresses.  It took approximately 12-13 seconds!  And this is with a non-SSD drive:  it seems like it shouldn't even be possible to go that quickly... I had to quadruple-check everything to verify that I wasn't missing something or accidentally supplying the answer.  Perhaps there was some OS-level caching of the DB files, though I would've expected such caching to improve my custom blockchain utilities performance. 

I don't intend to discourage or insult your efforts (in fact, I think you said you were getting similar speeds).  Plus, I think that writing such a parser/scanner is both useful and educational.  I just bring it up because I would guess that this is about as good as anyone can do with a non-SSD drive.  And it's especially impressive considering I did no optimizations.  I'm especially curious how leveldb works under the hood that it could be so efficient.  Maybe there's something to learn from it.

[jgarzik]

I don't intend to discourage or insult your efforts (in fact, I think you said you were getting similar speeds).  Plus, I think that writing such a parser/scanner is both useful and educational.  I just bring it up because I would guess that this is about as good as anyone can do with a non-SSD drive.  And it's especially impressive considering I did no optimizations.  I'm especially curious how leveldb works under the hood that it could be so efficient.  Maybe there's something to learn from it.

LevelDB provides the backend to Google's services, so it is not surprising that it is so heavily optimized.  One of the key optimizations is that writes go to a log, and then another thread comes along in the background and "optimizes" the log data back into the main database, using hueristics to hopefully place the data in a useful, locality-friendly location on disk.

BitcoinJ author Mike Hearn has proposed replacing BDB with LevelDB in the Satoshi client, and we are looking seriously at that.  Here is his pull request: https://github.com/bitcoin/bitcoin/pull/1619

[etotheipi]

Finally, it's a good idea to do this: in the end the blockchain will anyway grow
so big that even the hashmaps won't fit in RAM, and my approach will start
swapping like crazy. Pre-indexing in a disk-level DB hash table is the right thing
to do.

Well that's why I started looking into this.  My original blockchain utilities design in Armory were more of an educational adventure to see how fast I could get the scanning from scratch (no external libraries), with lots of pointers, etc.  It was educational, and I was happy with the results.  But I had the same realization that even the index will eventually not fit in RAM, and thus there is no way to make the library scalable without using some kind of disk-based engine.  To that end, I had no interest re-inventing disk-based DB operations, etc.  When I heard the devs talking about LevelDB, I looked into and quickly realized why they chose it.  It is very simple, portable... and apparently fast!

However, gmaxwell just told me how to clear my disk cache and then retest.  It was dramatically slower for sure.  Probably by a factor of 5-10x.  However, I never ran the same test using my other blockchain utilities, so I don't know the actual comparison.  Given that modern OS'es are good at doing that caching, the LevelDB version is clearly a very good solution.

I'm currently investigating how I can transition Armory to maintain it's own blockchain data now with LevelDB

[jgarzik]

FWIW the set of unspent tx-out's should fit nicely in RAM, for some time to come.

I know that's not helpful to applications doing historical queries, but it may be helpful for other queries.

[vampire]

znort987:

Could you add to parser.c:

#ifndef O_DIRECT
#define O_DIRECT 0x10000000
#endif

O_DIRECT isn't defined on OSX. Otherwise I was able to compile on my macbook with macports.

[thallium205]

Hey all.

I developed an online tool that does the exact same kind of operations (grouping addresses together through connected components) in O(1) while giving you the ability to visualize an ownership network that is built on top of it.  For that first address you calculated, you can check out the results by going here: http://blockviewer.com/#/owns/1CjLmm25ty4aSjxzG8Er4qSiQ63bqaQGA8 and then clicking on the "owner" node (they are sort of hard to click on... I'm working on that).  That will pull up all the addresses associated with it, along with other information.

If you think this is interesting, check out my announcement post here -> https://bitcointalk.org/index.php?topic=103609.0
I don't mean to thread jack you all, but I thought you all would find this interesting.  If you would like to check more addresses, simply type it in in the search bar and select "Owner Address".  Then, as usual, click the owner node and start traversing the block chain!

Enjoy!

[Chimsley]

I installed this and started doing some of the examples in the readme file and everything is working until I ran allbalances.

I get this output.

xxxxx@xxxxxxxx:~/blockparser$ ./parser allbalances >allbalances.txt

info: Starting command "allBalances"

info: analyzing blockchain ...
info:        0 blocks,    0.000 MegaMoves ,    0.000 MegaAddrs ,  0.00%
info:   118089 blocks,    1.049 MegaMoves ,    0.381 MegaAddrs , 59.56%
info:   129462 blocks,    2.097 MegaMoves ,    0.722 MegaAddrs , 65.29%
info:   134099 blocks,    3.146 MegaMoves ,    1.060 MegaAddrs , 67.63%
info:   138049 blocks,    4.194 MegaMoves ,    1.462 MegaAddrs , 69.62%
info:   142392 blocks,    5.243 MegaMoves ,    1.844 MegaAddrs , 71.81%
info:   146931 blocks,    6.291 MegaMoves ,    2.164 MegaAddrs , 74.10%
info:   152726 blocks,    7.340 MegaMoves ,    2.481 MegaAddrs , 77.02%
info:   158931 blocks,    8.389 MegaMoves ,    2.733 MegaAddrs , 80.15%
info:   164693 blocks,    9.437 MegaMoves ,    2.953 MegaAddrs , 83.06%
info:   169636 blocks,   10.486 MegaMoves ,    3.182 MegaAddrs , 85.55%
info:   174740 blocks,   11.534 MegaMoves ,    3.410 MegaAddrs , 88.13%
info:   178761 blocks,   12.583 MegaMoves ,    3.635 MegaAddrs , 90.15%
info:   180687 blocks,   13.631 MegaMoves ,    3.811 MegaAddrs , 91.13%
info:   181868 blocks,   14.680 MegaMoves ,    3.968 MegaAddrs , 91.72%
info:   183095 blocks,   15.729 MegaMoves ,    4.125 MegaAddrs , 92.34%
info:   184314 blocks,   16.777 MegaMoves ,    4.273 MegaAddrs , 92.96%
info:   185046 blocks,   17.826 MegaMoves ,    4.407 MegaAddrs , 93.32%
info:   186241 blocks,   18.874 MegaMoves ,    4.558 MegaAddrs , 93.93%
info:   187702 blocks,   19.923 MegaMoves ,    4.754 MegaAddrs , 94.66%
info:   189164 blocks,   20.972 MegaMoves ,    4.931 MegaAddrs , 95.40%
info:   190250 blocks,   22.020 MegaMoves ,    5.094 MegaAddrs , 95.95%
info:   191531 blocks,   23.069 MegaMoves ,    5.262 MegaAddrs , 96.60%
info:   192657 blocks,   24.117 MegaMoves ,    5.449 MegaAddrs , 97.16%
info:   193834 blocks,   25.166 MegaMoves ,    5.630 MegaAddrs , 97.76%
info:   194764 blocks,   26.214 MegaMoves ,    5.815 MegaAddrs , 98.23%
info:   195960 blocks,   27.263 MegaMoves ,    6.006 MegaAddrs , 98.83%
info:   197082 blocks,   28.312 MegaMoves ,    6.172 MegaAddrs , 99.39%
fatal: o2i_ECPublicKey failed
Aborted (core dumped)

This is a fresh Ubuntu 12.04 install with 8Gb of ram. Any thoughts on why I got this message? The allbalances.txt file is created but is empty so I assume the file is not written until the command is completed.

***Disclaimer***
I am just barely above noob level on linux.

[EskimoBob]

I am having trouble compiling.
http://pastebin.com/TrfSecqA

From the README:

Code:

sudo apt-get install libssl-dev build-essential g++-4.4 libboost-all-dev libsparsehash-dev git-core perl

Did you do that ?

I am going to attempt building it in Gentoo.
(most of this stuff is probably installed)

Code:

emerge -atvu1 dev-libs/openssl dev-libs/boost dev-cpp/sparsehash dev-vcs/git dev-lang/perl

[Chimsley]

Cool thanks znort987 for the future fix. I am just tinkering but I have seen other people on the forums doing serious analysis with this tool and I applaud you for releasing this to everyone.

[JompinDox]

sorry for the newb question:

is it possible to find out how many addresses containing over 500 btc have NOT been used since January 2011, for instance?

(not used = not used for sending coins)

[flatfly]

Great, thanks!

Btw, are there any pre-built windows binaries available?

[dooglus]

Great, thanks!

Btw, are there any pre-built windows binaries available?

Nope.
And there's unlikely ever going to be any.
haven't touched windows in 10 years and don't plan to do so ever again.

+1

[organofcorti]

Great, thanks!

Btw, are there any pre-built windows binaries available?

Nope.
And there's unlikely ever going to be any.
haven't touched windows in 10 years and don't plan to do so ever again.

What, you're saying that we just have to work when we're at work? It's people like you that make it hard for me to be a slacker.

[Buffer Overflow]

haven't touched windows in 10 years and don't plan to do so ever again.

6 years for me; and counting.

[xchrix]

great work znort!
is it also possible to use this parser for a litecoin blockchain? i think so...

unfortunately i only have debian 32bit so i have to organize a 64bit server for testing this.

[xchrix]

ok i see. i am good at php coding but c++ is not my best friend i dont think that i am able to provide some patches.

[SgtSpike]

It'd be great if someone could host this on the web, maybe charging a small fee for each lookup or something.

[SgtSpike]

It'd be great if someone could host this on the web, maybe charging a small fee for each lookup or something.

Mmmh, not really : this is designed to run a full linear sweep of the entire
blockchain for every query ... that's truly heavy duty (both raw I/O and CPU).

The moment 5 users hit the web site at the same time, the server will basically die.

The way blockchain.info does it is the right approach for what you describe:
they dump the blockchain into a DB with a crapload of pre-computed indices
and update said DB incrementally to keep the queryable version up to date.

I guess I just don't like blockchain.info's way of displaying the data it finds.  Maybe I need to get used to it more.

Regardless, jobs could be queued, and the resource-intensiveness is why I suggested a small fee for each lookup.

There are those of us who do not know linux well enough to run something like what you have created here, but would still like it to be accessible. 

[dirtycat]

I installed this and started doing some of the examples in the readme file and everything is working until I ran allbalances.

I get this output.

xxxxx@xxxxxxxx:~/blockparser$ ./parser allbalances >allbalances.txt

info: Starting command "allBalances"

info: analyzing blockchain ...
info:        0 blocks,    0.000 MegaMoves ,    0.000 MegaAddrs ,  0.00%
info:   118089 blocks,    1.049 MegaMoves ,    0.381 MegaAddrs , 59.56%
info:   129462 blocks,    2.097 MegaMoves ,    0.722 MegaAddrs , 65.29%
info:   134099 blocks,    3.146 MegaMoves ,    1.060 MegaAddrs , 67.63%
info:   138049 blocks,    4.194 MegaMoves ,    1.462 MegaAddrs , 69.62%
info:   142392 blocks,    5.243 MegaMoves ,    1.844 MegaAddrs , 71.81%
info:   146931 blocks,    6.291 MegaMoves ,    2.164 MegaAddrs , 74.10%
info:   152726 blocks,    7.340 MegaMoves ,    2.481 MegaAddrs , 77.02%
info:   158931 blocks,    8.389 MegaMoves ,    2.733 MegaAddrs , 80.15%
info:   164693 blocks,    9.437 MegaMoves ,    2.953 MegaAddrs , 83.06%
info:   169636 blocks,   10.486 MegaMoves ,    3.182 MegaAddrs , 85.55%
info:   174740 blocks,   11.534 MegaMoves ,    3.410 MegaAddrs , 88.13%
info:   178761 blocks,   12.583 MegaMoves ,    3.635 MegaAddrs , 90.15%
info:   180687 blocks,   13.631 MegaMoves ,    3.811 MegaAddrs , 91.13%
info:   181868 blocks,   14.680 MegaMoves ,    3.968 MegaAddrs , 91.72%
info:   183095 blocks,   15.729 MegaMoves ,    4.125 MegaAddrs , 92.34%
info:   184314 blocks,   16.777 MegaMoves ,    4.273 MegaAddrs , 92.96%
info:   185046 blocks,   17.826 MegaMoves ,    4.407 MegaAddrs , 93.32%
info:   186241 blocks,   18.874 MegaMoves ,    4.558 MegaAddrs , 93.93%
info:   187702 blocks,   19.923 MegaMoves ,    4.754 MegaAddrs , 94.66%
info:   189164 blocks,   20.972 MegaMoves ,    4.931 MegaAddrs , 95.40%
info:   190250 blocks,   22.020 MegaMoves ,    5.094 MegaAddrs , 95.95%
info:   191531 blocks,   23.069 MegaMoves ,    5.262 MegaAddrs , 96.60%
info:   192657 blocks,   24.117 MegaMoves ,    5.449 MegaAddrs , 97.16%
info:   193834 blocks,   25.166 MegaMoves ,    5.630 MegaAddrs , 97.76%
info:   194764 blocks,   26.214 MegaMoves ,    5.815 MegaAddrs , 98.23%
info:   195960 blocks,   27.263 MegaMoves ,    6.006 MegaAddrs , 98.83%
info:   197082 blocks,   28.312 MegaMoves ,    6.172 MegaAddrs , 99.39%
fatal: o2i_ECPublicKey failed
Aborted (core dumped)

This is a fresh Ubuntu 12.04 install with 8Gb of ram. Any thoughts on why I got this message? The allbalances.txt file is created but is empty so I assume the file is not written until the command is completed.

***Disclaimer***
I am just barely above noob level on linux.

Yeah, I hit this too a couple of days back.

Turns out someone managed to include a few transactions in the blockchain
that send funds to invalid compressed keys, something my code assumed was
not possible.

I have a fix for it, I'll push it to the git repo sometimes today.

Fix pushed to github.

Im still getting this when I run allabalances.. 

[dirtycat]

now using a different machine.. I get

Code:

gato@50cal:~/blockparser$ ./parser all

info: Starting command "allBalances"

info: analyzing blockchain ...
fatal: Invalid argument: failed to open block chain file /home/gato/.bitcoin/blk0001.dat
Aborted (core dumped)
gato@50cal:~/blockparser$ 

everything is there.. soooooo.. I have no idea.
not my day lol

[dirtycat]

figured it out.  had my home directory encrypted.. disabled encrypt and all is well.. fun adventure.. btw great tool!

[runeks]

This looks cool. Do you have any plans to add a feature that allows the program to output the total balance for a specific bunch of addresses? Like:

./parser showBalance < addresses.txt

or alternatively

./parser showBalance -i addresses.txt

Where addresses.txt would contain a list of addresses (hash160/base58).

[runeks]

Cool. Just tried it. blockparser reports the balance of the address 991fd9bf50ad8d6a551efceb5c228d21bcc0fe92 to be 200.777, but blockchain.info says it's 351.06501343 BTC. Who's right?

bitcoin-qt says my block chain is up to date (last block was found 8 minutes ago).

Armory's block chain tool reports the same balance as blockparser (200.777) so it seems that this is valid. But why does this disagree with blockchain.info?

[runeks]

Great! Is it possible to read addresses from a file that just contains concatenated hash160 addresses as raw bytes? Ie. 20 bytes followed by 20 bytes etc. I use this format to save space when storing a large number of addresses. I tried modifying the code to do this, but it seems it would require substituting fgets with fread, and things started looking messy so I gave up.

[runeks]

The reason I ask is because I got curious over whether people have used insecure passwords to create brain wallets. For this purpose I've created 1.5 GB worth of addresses in that 20 byte raw format, out of a word list made by John the Ripper. So the reason is that even though there are only ~300 megs of different addresses in the block chain, there are (2^160)*20 bytes of possible addresses using the raw 20 byte format. Having the addresses lined up on 20 byte-boundaries also increases performance, I imagine, since I don't have to do any string parsing on the data.

[Xenland]

This might help my Pseudo Bitcoin Client Documentation project I'll be looking into this code shortly

[runeks]

The reason I ask is because I got curious over whether people have used insecure passwords to create brain wallets. For this purpose I've created 1.5 GB worth of addresses in that 20 byte raw format, out of a word list made by John the Ripper.

So, that same list in ASCII will be 3GB ... BFD, if you'll pardon my french.

When the block chain is loaded into memory, the operation will mostly be memory-bound. Loading an additional 3 GB into memory is not an option.

I've just ordered more RAM, to bring my machine up to 6 GB though. Although it hurt a bit to see that the same amount of DDR3 RAM was half the price of the DDR2 RAM my machine supports

So the reason is that even though there are only ~300 megs of different addresses in the block chain, there are (2^160)*20 bytes of possible addresses using the raw 20 byte format. Having the addresses lined up on 20 byte-boundaries also increases performance, I imagine, since I don't have to do any string parsing on the data.

IMO, the time it'll take to parse the ASCII will be dwarfed by
the time taken to chew through the blockchain while hammering
in a completely cache-incoherent manner at that 1.5GB hashtable
of restricts you just forced the parser to load and work with.

I'm currently working with 50 MB chunks (~2.6 million addresses) at a time, because of the aforementioned memory-restriction. How long do you think blockparser would take to parse that amount of addresses? Armory's Python tools does it in 120 seconds (not including the time it takes to load the block chain into memory).

Also ... I'm hoping you're planning to Do The Right Thing (tm) with
that attempt of yours. Nuff said.

Of course. I'm not a thief. Just curious.

[Buffer Overflow]

Code:

info: Starting command "simpleStats"
fatal: No such file or directory: failed to open block chain file /home/buffer/.bitcoin/blocks/blk00000.dat
Aborted (core dumped)

I don't have the blocks/ directory?

I only have:

blk0001.dat
blkindex.dat
db.log
peers.dat
blk0002.dat
database/
debug.log
wallet.dat

Using Bitcoin 0.7.1 & latest version of blockparser.

[Buffer Overflow]

Ok, thank you very much.

How do I change over to the new style blockchain files? I'm using 0.7.1.

[Buffer Overflow]

Oh dear more problems. Just cloned latest commit and tested with ./parser simpleStats and this appears:

warning: at depth 20221 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da
warning: at depth 12432 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da
warning: at depth 4315 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da
warning: at depth 14255 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da
warning: at depth 2975 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da
warning: at depth 13760 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da
warning: at depth 15626 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da
warning: at depth 15844 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da
warning: at depth 629 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da
warning: at depth 3965 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da
warning: at depth 3249 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da
warning: at depth 13269 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da
warning: at depth 8480 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da
warning: at depth 16582 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da
warning: at depth 18383 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da
warning: at depth 6801 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da
warning: at depth 14939 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da

I have to exit with ctrl-c.

[Buffer Overflow]

Ok, thank you very much.

How do I change over to the new style blockchain files? I'm using 0.7.1.

Code:

git clone git://github.com/bitcoin/bitcoin.git bitcoin-head
cd bitcoin-head
qmake-qt4
make -j4
./bitcoin-qt &

Be aware this is going to "upgrade" your datafiles
and that this "upgrade" is likely not reversible.

You might want to wait until this is officially supported
by the released client.

Oh, I see, it's in Beta state. Think I leave it then.

[Pieter Wuille]

Oh, I see, it's in Beta state. Think I leave it then.

Well, all bitcoin releases to date have been beta.

This is about the code that hasn't even been released, but will eventually become 0.8.

[Buffer Overflow]

Oh, I see, it's in Beta state. Think I leave it then.

Well, all bitcoin releases to date have been beta.

This is about the code that hasn't even been released, but will eventually become 0.8.

That's a good point. I wonder when Bitcoin will come out of beta? 1.0 I guess.

[Jouke]

Oh dear more problems. Just cloned latest commit and tested with ./parser simpleStats and this appears:

warning: at depth 20221 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da
warning: at depth 12432 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da
warning: at depth 4315 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da
warning: at depth 14255 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da
warning: at depth 2975 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da
warning: at depth 13760 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da
warning: at depth 15626 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da
warning: at depth 15844 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da
warning: at depth 629 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da
warning: at depth 3965 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da
warning: at depth 3249 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da
warning: at depth 13269 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da
warning: at depth 8480 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da
warning: at depth 16582 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da
warning: at depth 18383 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da
warning: at depth 6801 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da
warning: at depth 14939 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da

I have to exit with ctrl-c.

I have the same problem with 7.2

[Jouke]

I did a git reset 90a89c0373, but that didn't help.
nm. I am running an extra daemon now.

However:
info: Address cluster for address 16EC7Nx6THDZpGMCavLubbtt2UPXimyUx7:
warning: specified key was never used to spend coins

But it is used in this transaction:
https://blockchain.info/tx/49688f5183b3898d19de3904a09d75d1af52a14edd5f951f371f6823eb22b1f6

[Jouke]

You need to make sure that your local blockchain is up to date.

Hmm, strange, bitcoind getinfo told me it was at up2date.

[SilverLeaf]

Same "Failed to locate parent block..." problem.

Opened a GitHub issue here: https://github.com/znort987/blockparser/issues/1 .

[SilverLeaf]

Your blockchain files are likely corrupted.

Even if they aren't, I do not plan to support old style bitcoind blockchain in the future.

If you want to use blockparser, I would strongly advise moving to a recent version of the bitcoind
client with the levelDB backend (the one that has a directory called "chainstate" in ~/.bitcoin)

Thank you, I am regenerating the blockchain using bootstrap.dat and trying again.

blockparser doesn't support levelDB by now, right?

[SilverLeaf]

Thank you, now it's much more clear.

EDIT: I can confirm that it works with the git version of bitcoind.

[naima53]

Is compatible with version 0.8 bitcoin-qt ?

[Dabs]

Is there a pre-compiled windows version? binary? (which you can of course sign or produce a SHA so us newbies can be sure we have a clean binary.)

[Buffer Overflow]

Blockparser does not run on Windows and other legacy technologies from the 20th Century

LOL 
I couldn't agree more.

[ShadowOfHarbringer]

Is there a pre-compiled windows version? binary? (which you can of course sign or produce a SHA so us newbies can be sure we have a clean binary.)

Blockparser does not run on Windows and other legacy technologies from the 20th Century

Hahahahaha.
Splendid statement, milord.

[codro]

It would be really interesting to see a visualization (graph/chart/printout ) of all the bitcoins being traded in the past couple of years, how many of them (atomically) haven't exchanged hands at all in the past 2 years? Since all coins should be able to be traced back to the moment they were mined, it should be fairly trivial to see how many times and the dates when they were transacted. No?

I think this would be valuable for finding out how much is being hoarded, and how much is in circulation.

I haven't fully looked into this tool yet to see if it can do this, but any pointers would be appreciated.

[Sukrim]

Well, in theory it should be possible to let this run on Windows as well... in practice however I've only managed to compile boost and am still fighting with sparsehash as well as openssl and it's dependencies (openssl wants perl for example).

Sparsehash also seems to really not like to be compiled with MingW... Let's see if I give up and just install a VM on my big machine, because my small Linux Atom nettop probably will choke to death on the hash tables.

[Dabs]

Maybe there is a way to implement this differently so it doesn't use mmap files? Or windoze implementation could be improved? Or I'm asking too much?

If I had the ram, maybe blockparser could do it all in ram? Or I can try using a ram drive?

[Sukrim]

To deal with crappy retarded GPU drivers and UI in Linux? No thanks, sir...

Anyways, in theory it should(TM) be possible already to compile + run this parser for Windows. It's not znort987's resposibility to actually provide the means to do so, especially since he doesn't want to do it. It would be nice of him if somebody manages to actually come up with a solution how to do it, to integrate some minor patches (as far as I see it from the code, the blockchain location is hardcoded for example - having a setting for Windows or even Linux users there might be nice) and the actual way to do it, but even that is not his job at all (and you simply could just fork his project after all).

The code is public domain and uses OpenSource libraries. Either invest the time to compile that stuff yourself or pay somebody to do it for you. If mmap implementations on Windows are really that slow, you can also improve them, since this project is built on OpenSource software. Again, either by yourself or somebody you pay to do that.

[Dabs]

I guess I am asking too much, in the same way that a request to get rid of crappy OSes is also asking too much. Depends on who you talk to, if it's one individual, it is reasonable to ask them to switch OS, if you are talking to a corporation with hundreds of deployed crappy OS machines, that might be asking too much.

However, I do have the option to run Linux on my personal machine, so unless Windows gets improved (either the mmap implementation or the parser itself), then that's the route to take, for now. My guess is that, since this is a specialized tool, you really should use it on dedicated machines that are not being used for other crappy OS stuff.

Unfortunately for me, that's not a practical thing to do. I could dual-boot to experiment with it, but eventually I'd have to go back to my crappy OS. Thus my question.

[Buffer Overflow]

To deal with crappy retarded GPU drivers and UI in Linux? No thanks, sir...

The manufacturers make the drivers, blame them. Linux is more than capable of running most hardware.
Who wants to dance and jump about to Bill Gates tune? Playing the game by his rules, his way, at his (expensive) price.   No thanks, sir...

Unfortunately for me, that's not a practical thing to do. I could dual-boot to experiment with it, but eventually I'd have to go back to my crappy OS. Thus my question.

You could install a virtualization software package, like Virtual Box and install Linux inside that. That's assuming your hardware can take the extra load of course.



How come (and this isn't directed at anyone in particular) when a Linux user needs a program that only runs on Windows, he/she gets their head down, knuckles down and finds way around the problem. But on the other hand when a Windows user needs a program that only runs on Linux, there's a huge uproar, and expects everyone else to adjust to them? It's as if they assume Windows is the default system and everyone must whistle their tune.

[Sukrim]

To deal with crappy retarded GPU drivers and UI in Linux? No thanks, sir...

The manufacturers make the drivers, blame them.

Ok, let's blame AMDand nVidia! Evil you! You really should support Linux better!
Great to have done this, I guess that changes now everything...

I am also a Linux user and I never got my head down, knuckled down and tried to find a way around the problem - I simply installed a Windows VM or native machine and ran the program in it's intended OS. Please refrain from generalizing, I bet if you wrote this statement in a Linux heavy forum replacing Windows with Ubuntu and Linux with Gentoo it would also be in line there... especially taking into consideration that this software is said to only support 64 bit Ubuntu.

Anyways, there IS already the possibility to compile and run this software under Windows and as soon as I manage to compile it, I'll maybe even share the binary (for a hefty fee perhaps to cover my time). If this doesn't work, well - having a result in 3 minutes instead of one vs. having no result at all would still be an improvement, right?

[gmaxwell]

After interacting with znort987 on IRC I'd advise being very careful in using this software double check any results that you get from it.

I pointed out that he was making some (theoretically and practically) minorly incorrect claims about the analysis this tool produced and his response was confused first failing on technically grounds and then after the technical misunderstand was made clear he became uncivil— calling me and other folks names and such— quite bizarre. While its obviously useful, it's written by someone who apparently hasn't done your thinking for you, so you need to think for yourself.

[phantomcircuit]

After interacting with znort987 on IRC I'd advise being very careful in using this software double check any results that you get from it.

I pointed out that he was making some (theoretically and practically) minorly incorrect claims about the analysis this tool produced and his response was confused first failing on technically grounds and then after the technical misunderstand was made clear he became uncivil— calling me and other folks names and such— quite bizarre. While its obviously useful, it's written by someone who apparently hasn't done your thinking for you, so you need to think for yourself.

I can confirm this.

[ShadowOfHarbringer]

After interacting with znort987 on IRC I'd advise being very careful in using this software double check any results that you get from it.

I pointed out that he was making some (theoretically and practically) minorly incorrect claims about the analysis this tool produced and his response was confused first failing on technically grounds and then after the technical misunderstand was made clear he became uncivil— calling me and other folks names and such— quite bizarre. While its obviously useful, it's written by someone who apparently hasn't done your thinking for you, so you need to think for yourself.

This forum should have a red or orange warning box on the top of each topic (placeable by mods, admins & devs only) to warn people of scams and events such as this.

What do you think ?

[gmaxwell]

Ah, and since he changed this back— I suppose I should point this out here:

* Compute the list of addresses that provably belong to the same person

This is both theoretically and practically incorrect. The tool doesn't do this for a couple of reasons.

The most common violation of this comes from shared wallets (like mtgox and instawallet). Signatures on transactions do not not indicate "belonging"— mtgox signs on hundreds of thousands of coin that do not belong to them. In that case it only indicates control, and only at a point in time as people do transfer ownership of private keys to these services. (E.g. If give my old privkeys with some coins on them to mtgox, they'll end up as inputs on other people's txn— it's true that mtgox controls them from that point on, but not in the past).

Less commonly, but also more fundamentally, is this assumption is also invalidated by group transactions. E.g. it's quite possible for two people to each provide an input to a transaction. This is useful for making atomic shared payments to bounties and the like, but also for doing trustless mixing.  I believe the new "shared send" functionality on b.i (which replaced their old mixing service) works  this way, and so I wouldn't be surprised if a non-trivial number of transactions failed on this criteria.

Obviously, the importance of these details depend on how you are using the data. Common inputs are suggestive of common control at the time the transaction was authored. They are suggestive (though less so) of past control. They are far less suggestive of actual belonging because the belonging assumption is commonly violated by shared wallets and may be untrue for a significant fraction of coins in existence. They certainly do not constitute a 'proof' in any sense that we'd normally use the word in this community.

If you're interested in making soft guesses about some transactions, then these might be fine assumptions.  If you're looking to prosecute someone for a crime— certainly not. If you want to make statistical comments about populations of transactions it's highly questionable because no one knows how biased a statistic the result is, since mixed and shared transactions appear to be indistinguishable from other ones.

Cheers.

This forum should have a red or orange warning box on the top of each topic (placeable by mods, admins & devs only) to warn people of scams and events such as this.

I don't think it rises to that level of concern. I apologize if I came off sounding like it was that urgent.  It's not. In this case I wanted to throw out a caveat emptor because I thought it was odd... but really caveat emptor is generally good advice.   If it were something like I thought that the software would cause you to lose coin or something I would have made it a _lot_ more obvious.

[Buffer Overflow]

Ok, let's blame AMDand nVidia! Evil you! You really should support Linux better!
Great to have done this, I guess that changes now everything...

Correct me if I'm wrong, but don't these two quotes listed below contradict each other?

I am also a Linux user and I never got my head down, knuckled down and tried to find a way around the problem - I simply installed a Windows VM or native machine and ran the program in it's intended OS.

Anyways, there IS already the possibility to compile and run this software under Windows and as soon as I manage to compile it, I'll maybe even share the binary (for a hefty fee perhaps to cover my time).

[Sukrim]

Nope, as I would do the feats that were attributed to a brave Linux hacker as a Windows user, not as a Linux user. Anyways, this is actually besides the point.

The issue that transactions that spend from two addresses do not 100% mean that these addresses are controlled by the same person or entity is something more interesting than whether I manage to compile OpenSSL etc. under Windows.

Especially that constant cries to import private keys here and there might make it tough in the future to even analyze transactions at all.
I also can't think of any 100% sure ways to overcome this problem of people sharing inputs to fool software like this - maybe creating address pools and then looking for suspicious overlaps might be a solution. E.g. 500 transactions each in 2 sets of coupled addresses, then suddenly for a single transaction they bond (even though the "simulated wallet" containing both sets would contain better fitting outputs?) and then they start transacting on their own again never mixing coins... Also transaction timestamps could give indications about users (and adding a second set that has so far completely different transaction patterns might indicate someone trying to mix coins).

[Sukrim]

Oh, and for some measurements:

With larger hash maps: SimpleStats in ~850s
With smaller hash maps (WANT_DENSE commented out): SimpleStats in ~350s

Still a lot of time, but not too shabby for a little atom machine.

[Sukrim]

Hm, could be the case - I didn't reboot between runs + recompilation.

[Sukrim]

Oh btw. it works/compiles like a charm on Arch Linux, so no worries about needing Ubuntu (you don't even specify which release anyways...).

Now back to solving my Hash160 issues... :-/

[mrb]

No need to reboot.
Just flush the disk cache.
Something along the lines of:

Code:

dd if=/dev/zero bs=1024k count=4096 of=/dev/null

usually does the trick.

This has pretty much no effect on the buffercache (kernel knows /dev/zero does not need caching, and dd will use at most 1MB of resident memory). What will drop it is (as root):

Code:

$ echo 3 >/proc/sys/vm/drop_caches

[Dabs]

Windows is not the default OS. It's just, unfortunately the market leader, or the OS with the largest share of users. I'm sure that if I can't dual-boot, I can probably do a Live CD.

[Sukrim]

Windows is not the default OS. It's just, unfortunately the market leader, or the OS with the largest share of users. I'm sure that if I can't dual-boot, I can probably do a Live CD.

Good luck with catching up on the blockchain then... depending on your needs, you're faster to just run the stuff you need on armory-python.

Anyways, doing a closure run brings my poor little nettop to its knees (even with the "smaller" hash maps) - at least after a few hours it didn't show any kind of progress, so I killed the program. 1GB RAM is not cutting it at all apparently.

[wiggi]

Well, in theory it should be possible to let this run on Windows as well... in practice however I've only managed to compile boost and am still fighting with sparsehash as well as openssl and it's dependencies (openssl wants perl for example).

Sparsehash also seems to really not like to be compiled with MingW... Let's see if I give up and just install a VM on my big machine, because my small Linux Atom nettop probably will choke to death on the hash tables.

Another issue with VM's is that:

   a) you're going to have to copy the blockchain over to the VM
   b) blockparser uses mmap'd files extensively. windoze implementation of those is hugely inefficient
   c) blockparser is really I/O intensive (every run gobbles in the full blockchain). This is highly inefficient on VMs

A little test, how long does it take to do "./parser allBalances >allBalances.txt" in VirtualBox
with different amounts of ram assigned to the VM:
(host is win7 64bit, guest Linux Mint 64bit, 8GB physical ram)

1GB   eta 1200000s
3GB        eta 5000s
6GB         eta 200s, but actually >1500s, slows down alot @50-70%
7GB             850s

As long as the VM can consume more and more ram, blockparser is *very* fast, but I'm
not sure how much is really enough to let the OS buffer all I/O in ram, 32GB perhaps.

[Dabs]

I'm tempted to buy a machine with 32 Gigs of RAM. ... ... Maybe those so called "ram drives" (hardware implemented) might be useful.

@Sukrim, I can probably sync the blockchain with another computer that already has it within the network, or just copy the files over. That will save download time.

[mrb]

I'm tempted to buy a machine with 32 Gigs of RAM. ... ... Maybe those so called "ram drives" (hardware implemented) might be useful.

Do it, 32GB only costs $160, assuming you already have a machine with a motherboard supporting 4 x 8GB DDR3.
But don't buy hardware ram drives. They are all horribly overpriced, as in 10x more expensive per GB, or more.

[Dabs]

The hardware ram drives I know use old DDR2 memory. Maybe there are new ones that are horribly overpriced. Or I'll look around to see if someone has made one. The one I know used to use a SATA cable to emulate a hard drive. Real ram inside the motherboard is obviously going to work a lot faster.

[2112]

But don't buy hardware ram drives. They are all horribly overpriced, as in 10x more expensive per GB, or more.

Please pardon me for jumping in, but please post the links to those overpriced hardware RAM drives. My friends in IT operations are on the market and have hard time sourcing devices that don't wear out are reasonably non-volatile and can survice reboot/reconfig/reload of the OS.

[mrb]

But don't buy hardware ram drives. They are all horribly overpriced, as in 10x more expensive per GB, or more.

Please pardon me for jumping in, but please post the links to those overpriced hardware RAM drives. My friends in IT operations are on the market and have hard time sourcing devices that don't wear out are reasonably non-volatile and can survice reboot/reconfig/reload of the OS.

http://www.ddrdrive.com - it was designed for ZFS ZIL, and the RAM is backed by SLC NAND flash. It was priced $2000 when it launched ~3 years ago. The price has probably gone down a lot, but it is still way more than 10x the price per GB than system's RAM.

Part of the reason prices are so high is because there are so few vendors, so there is practically no competition.

[2112]

http://www.ddrdrive.com - it was designed for ZFS ZIL, and the RAM is backed by SLC NAND flash. It was priced $2000 when it launched ~3 years ago. The price has probably gone down a lot, but it is still way more than 10x the price per GB than system's RAM.

Part of the reason prices are so high is because there are so few vendors, so there is practically no competition.

Thanks. Unfortunately this is still a flash drive with RAM cache and still susceptible to write amplification abuse. Thus far we haven't found anything better than discontinued ACARD ANS-9010 (64GB with battery backup) and some other custom-made solutions.

[onelineproof]

This is nice. One feature I would like to see is the creation of a table that stores public key to balance pairs, and keeps updating from time to time. This way you can quickly look up the balance of an address. If I have time, I may use this code to do this, and I'll add it to my wallet parser: https://github.com/piratelinux/cwallet

[phelix]

Would it be possible to dig out what percentage of coins have ever been moved for any 2016 block period?

I would be interested in how many of the early coins have never been moved.

edit: ok, just found your post: https://bitcointalk.org/index.php?topic=98641.0  [2 Million unspent...]

[phathash]

I know this is developed under Ubuntu, but it would be nice to have working on RH type distros.

I have openssl with ecdsa working, but blockparser won't compile -

# make
c++ -- cb/dumpTX.cpp
cb/dumpTX.cpp: In member function 'virtual int DumpTX::init(int, const char**)':
cb/dumpTX.cpp:80: error: expected initializer before ':' token
cb/dumpTX.cpp:84: error: expected primary-expression before 'return'
cb/dumpTX.cpp:84: error: expected ';' before 'return'
cb/dumpTX.cpp:84: error: expected primary-expression before 'return'
cb/dumpTX.cpp:84: error: expected ')' before 'return'
make: *** [.objs/dumpTX.o] Error 1

Is this something worth pursuing further?

[phathash]

Thanks. gcc now at  4.6.3.

However, now receive -

# make
lnk -- parser
.objs/util.o: In function `decompressPublicKey(unsigned char*, unsigned char const*)':
util.cpp:(.text+0x804): undefined reference to `EC_KEY_new_by_curve_name'
util.cpp:(.text+0x820): undefined reference to `o2i_ECPublicKey'
util.cpp:(.text+0x834): undefined reference to `EC_KEY_set_conv_form'
util.cpp:(.text+0x843): undefined reference to `i2o_ECPublicKey'
util.cpp:(.text+0x84f): undefined reference to `EC_KEY_free'
util.cpp:(.text+0x896): undefined reference to `EC_KEY_free'
.objs/util.o: In function `compressPublicKey(unsigned char*, unsigned char const*)':
util.cpp:(.text+0x8b4): undefined reference to `EC_KEY_new_by_curve_name'
util.cpp:(.text+0x8d0): undefined reference to `o2i_ECPublicKey'
util.cpp:(.text+0x8e4): undefined reference to `EC_KEY_set_conv_form'
util.cpp:(.text+0x8f3): undefined reference to `i2o_ECPublicKey'
util.cpp:(.text+0x8ff): undefined reference to `EC_KEY_free'
util.cpp:(.text+0x946): undefined reference to `EC_KEY_free'
collect2: ld returned 1 exit status
make: *** [parser] Error 1

I believe I do have openssl compiled correctly with ECDSA.

Is this worth pursuing?

[willphase]

blockchain.info satoshidice profit/loss seems to be a bit flaky at the moment - so right now to work out my profit/loss I'm exporting my transactions then pulling tx info from blockchain.info for each of my transactions and working out if it contains a 1dice* address then assuming that's a bet and adding/removing from the total.

it would be nice to do this using your tool - so supplied with a list of all my addresses, parse blockchain for tx with either { my address } -> { 1dice* } or { 1dice* } -> { my address }

Is this possible?

Will

[willphase]

You can try this:

[snip]

ought to  give you what you want.

Sounds good - I'll give that a go.

Cheers

Will

[r.willis]

2112, your friends' best bet is http://www.ramsan.com/products/rackmount-ram-storage-line
But price can be quite high.

[2112]

2112, your friends' best bet is http://www.ramsan.com/products/rackmount-ram-storage-line
But price can be quite high.

Thank you very much. Those are primarily aimed at the "IOPS desperation" market niche. We aren't there yet. I would call our niche "rapid reboot & real reliability" for write-ahead-logging internal drive. The flash EEPROM storage is so full of misinformation about reliability that its is nearly impossible to make sensible product choices.

[r.willis]

If reliability is main concern, you can use RAID1 (preferably staggered in such a way that two disk in each array have different wear level). Also, larger disks will last longer before wearing.

[2112]

If reliability is main concern, you can use RAID1 (preferably staggered in such a way that two disk in each array have different wear level). Also, larger disks will last longer before wearing.

I wish the life would be so simple. The flash storage sector is so full of cheats and fly-by-nights, that the total cost of building a reliable and long lasting Unix server simply approaches or exceeds the cost of an equivalent mainframe solution. C.f. the recent trend of using supercapacitors to power hidden write-back caches. Combined with the various wear-leveling cheats it produces failures that are extremely hard to locate. Consider this trivialized example of a circular on-disk-buffer of 3 blocks:

1 2 3
4 5 6
7 8 9

If you read it after simple reboot you'll get "7 8 9". If you read it after proper power down with super-cap discharge, you'll get "7 5 9" or "7 2 9" or various other combinations. The culprit really isn't flash technology, it is nearly always firmware problem due to its extreme complexity and trivialized testing.

There are various options available on the market, but they aren't openly advertised because they tend to use some technologies that IBM has either patented or keeps a tight contractual grip on the channel partners. e.g. SCSI disk drives with AS/400-like formatting with 520 bytes per block for 512 bytes of user data per block.

On the other hand we've never suffered a bug in a true RAM drive, possibly because of the extreme simplicity of the required firmware.

I apologise for contributing to a derailment of this thread.

[mrb]

If reliability is main concern, you can use RAID1 (preferably staggered in such a way that two disk in each array have different wear level). Also, larger disks will last longer before wearing.

I wish the life would be so simple. The flash storage sector is so full of cheats and fly-by-nights, that the total cost of building a reliable and long lasting Unix server simply approaches or exceeds the cost of an equivalent mainframe solution.

Self-healing (ZFS) can make unreliable flash storage reliable.

[MeatPopsicle]

Compiles fine but then segafaults.

There's logic issues with the mmap'ing.

[MeatPopsicle]

ext3, and pretty sure it's crashing from a null ptr. Always dies once you try to operate on the last element in vecMap.

[MeatPopsicle]

When I didn't have any blockchain files it would crash on unmapping somewhere in cleanMaps, but with blockchain data it dies on:

Code:

while(i!=e) totalSize += (i++)->size;

in initHashtables.

Code:

ovrlrdq@thedarkcitadel:/tmp/blockparser$ ./parser stats

info: starting command "simpleStats"
Segmentation fault (core dumped)
ovrlrdq@thedarkcitadel:/tmp/blockparser$ gdb parser core
GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /tmp/blockparser/parser...done.
[New LWP 20039]

warning: Can't read pathname for load map: Input/output error.

warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fffb238a000
Core was generated by `./parser stats'.
Program terminated with signal 11, Segmentation fault.
#0  0x000000000044f2d3 in google::dense_hashtable<std::pair<unsigned char const* const, unsigned char const*>, unsigned char const*, Hash256Hasher, google::dense_hash_map<unsigned char const*, unsigned char const*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const, unsigned char const*> > >::SelectKey, google::dense_hash_map<unsigned char const*, unsigned char const*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const, unsigned char const*> > >::SetKey, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const, unsigned char const*> > >::dense_hashtable(google::dense_hashtable<std::pair<unsigned char const* const, unsigned char const*>, unsigned char const*, Hash256Hasher, google::dense_hash_map<unsigned char const*, unsigned char const*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const, unsigned char const*> > >::SelectKey, google::dense_hash_map<unsigned char const*, unsigned char const*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const, unsigned char const*> > >::SetKey, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const, unsigned char const*> > > const&, unsigned long) ()
(gdb) bt
#0  0x000000000044f2d3 in google::dense_hashtable<std::pair<unsigned char const* const, unsigned char const*>, unsigned char const*, Hash256Hasher, google::dense_hash_map<unsigned char const*, unsigned char const*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const, unsigned char const*> > >::SelectKey, google::dense_hash_map<unsigned char const*, unsigned char const*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const, unsigned char const*> > >::SetKey, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const, unsigned char const*> > >::dense_hashtable(google::dense_hashtable<std::pair<unsigned char const* const, unsigned char const*>, unsigned char const*, Hash256Hasher, google::dense_hash_map<unsigned char const*, unsigned char const*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const, unsigned char const*> > >::SelectKey, google::dense_hash_map<unsigned char const*, unsigned char const*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const, unsigned char const*> > >::SetKey, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const, unsigned char const*> > > const&, unsigned long) ()
#1  0x000000000044b8fc in initHashtables() ()
#2  0x0000000000404c4c in main ()

Valgrind output:

Code:

info: starting command "simpleStats"
==22223== Invalid write of size 8
==22223==    at 0x44EA23: google::dense_hashtable<std::pair<unsigned char const* const, unsigned char const*>, unsigned char const*, Hash256Hasher, google::dense_ha
sh_map<unsigned char const*, unsigned char const*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const, unsigned c
har const*> > >::SelectKey, google::dense_hash_map<unsigned char const*, unsigned char const*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std:
:pair<unsigned char const* const, unsigned char const*> > >::SetKey, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const, unsigne
d char const*> > >::dense_hashtable(google::dense_hashtable<std::pair<unsigned char const* const, unsigned char const*>, unsigned char const*, Hash256Hasher, google
::dense_hash_map<unsigned char const*, unsigned char const*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const,
unsigned char const*> > >::SelectKey, google::dense_hash_map<unsigned char const*, unsigned char const*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_re
alloc<std::pair<unsigned char const* const, unsigned char const*> > >::SetKey, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* cons
t, unsigned char const*> > > const&, unsigned long) (in /tmp/blockparser/parser)
==22223==    by 0x44B76D: initHashtables() (in /tmp/blockparser/parser)
==22223==    by 0x404C4B: main (in /tmp/blockparser/parser)
==22223==  Address 0x10 is not stack'd, malloc'd or (recently) free'd
==22223==
==22223==
==22223== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==22223==  Access not within mapped region at address 0x10
==22223==    at 0x44EA23: google::dense_hashtable<std::pair<unsigned char const* const, unsigned char const*>, unsigned char const*, Hash256Hasher, google::dense_ha
sh_map<unsigned char const*, unsigned char const*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const, unsigned c
har const*> > >::SelectKey, google::dense_hash_map<unsigned char const*, unsigned char const*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std:
:pair<unsigned char const* const, unsigned char const*> > >::SetKey, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const, unsigne
d char const*> > >::dense_hashtable(google::dense_hashtable<std::pair<unsigned char const* const, unsigned char const*>, unsigned char const*, Hash256Hasher, google
::dense_hash_map<unsigned char const*, unsigned char const*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const,
unsigned char const*> > >::SelectKey, google::dense_hash_map<unsigned char const*, unsigned char const*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_re
alloc<std::pair<unsigned char const* const, unsigned char const*> > >::SetKey, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* cons
t, unsigned char const*> > > const&, unsigned long) (in /tmp/blockparser/parser)
==22223==    by 0x44B76D: initHashtables() (in /tmp/blockparser/parser)
==22223==    by 0x404C4B: main (in /tmp/blockparser/parser)

[fluffypony]

I'd like to re-open https://github.com/znort987/blockparser/issues/1 - I'm getting the same problem when running an allBalances yesterday. Alternatively, if someone could do an allBalances for me as a precursor to solving the problem that would be greatly appreciated.

[hyh]

I saw tons of likely/unlikely in the code, what do they do?

Code:

    #if defined(__GNUC__)
        #define likely(x)   __builtin_expect((x), 1)
        #define unlikely(x) __builtin_expect((x), 0)
        #define ALWAYS_INLINE __attribute__ ((always_inline))
    #else
        #define likely(x)   (x)
        #define unlikely(x) (x)
        #define ALWAYS_INLINE
    #endif

I searched a bit and it says something like make a prediction etc.

Is this the dirty part to make the code fast?

How can I know if it works on my machine or not?

(I mean, how do I know which one is taken in the definition, #if defined(__GNUC__) or #else?)

[hyh]

Performance-wise for blockparser, they don't make much
difference anyways (most of the time is spent in I/O).

Actually on my computer most time is spent on parseLongestChain().
It takes about 10 min for my computer to do that.

[hyh]

Are you

   A) On a VM ?
   B) On an encrypted partition ?
   C) On an OS which doesn't generally implement mmap() properly ?

A) No.
B) No.
C) How can I tell? Can you tell me a way to bench mark this, or shall I check the version of some library?

[hyh]

Actually I'm using an SSD... don't know why it is slow.

[hyh]

Are you using a standard drive or an SSD ?
If the former, the times you are seeing are not really surprising: parsing
8G (the current size of the block chain) requires *reading* 8G of data from
disk. If your disk reads at 20 Mb/s then just the I/O already costs ~ 7mn.

I just tried, when I run the parser, the peak speed is about 56MB/s, although my SSD allows about 250MB/s.

(But that doesn't look like the key problem. It's just the step parseLongestchain(), steps before that takes about 25s, but parseLongestchain() takes about 600s.)

[andr3w321]

Anyone manage to compile on ec2 ubuntu server?  I tried from a fresh ubuntu 12.04 and 12.10 server and neither worked.  I managed to compile just fine on my laptop running 12.10.  I upgraded g++ to 4.7 from 4.6, but that didn't help.

Code:

$ make
c++ -- cb/allBalances.cpp
c++ -- callback.cpp
c++ -- cb/closure.cpp
c++ -- cb/dumpTX.cpp
c++ -- cb/help.cpp
c++ -- opcodes.cpp
c++ -- option.cpp
c++ -- parser.cpp
c++ -- cb/pristine.cpp
g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-4.7/README.Bugs> for instructions.
make: *** [.objs/callback.o] Error 4

[Sukrim]

znort what the hell?!

Why are all your posts deleted + OP censored?

[willphase]

can someone explain why all znort987 posts have been vanished?

Will

[andr3w321]

.2 bitcoins for anyone who can provide instructions for getting to compile on ec2 ubuntu server 12.04 image.  Been at this awhile now and can't figure out what's wrong.  I've tried installing 32 bit c libs and compiling 32 bit version and that didn't work out.  Tried various versions of g++.  4.7 error may be related to this https://bugs.launchpad.net/ubuntu/+source/gcc-4.7/+bug/1132899  I tried 4.5 out and it gave me an error of line 80 of dumpTX which I'm assuming is because of newer foreach syntax.  I tried modifying, but this did not seem to help either.  I tried copying my locally compiled version but it just segfaults on the server.  I tried upgrading to 13.04 but kept getting

/usr/bin/ld: cannot find -l-c
collect2: ld returned 1 exit status

[andr3w321]

No more bounty.  I finally successfully compiled.  Turns out I was on a micro instance and running out of memory during compile.  I was able to follow these instructions: https://bitcointalk.org/index.php?topic=110627.0 to get it working

Run "dmesg" to make sure its a memory failure.  If so you can create a 1gb swap file like so

Code:

sudo dd if=/dev/zero of=/swapfile bs=64M count=16
sudo mkswap /swapfile
sudo swapon /swapfile

When you're done optionally remove the swap file

Code:

sudo swapoff /swapfile
sudo rm /swapfile

On an unrelated note, before I figured this out I came very close to compiling on a CentOS instance before running into openssl issues found here: https://bitcointalk.org/index.php?topic=9476.0  Below are the commands I ran.  They may not all be necessary

Code:

#install dependencies
sudo yum groupinstall "Development Tools"
sudo yum install kernel-devel kernel-headers
sudo yum install openssl-devel
sudo yum install boost-devel

#update sources
sudo yum update

#install wget
sudo yum install wget

#install sparsehash
wget https://sparsehash.googlecode.com/files/sparsehash-2.0.2-1.noarch.rpm
sudo yum localinstall sparsehash-2.0.2-1.noarch.rpm

#update g++ to 4.7
sudo yum install devtoolset-1.0
export PATH=/opt/centos/devtoolset-1.0/root/usr/bin/:$PATH
ln -sf /opt/centos/devtoolset-1.0/root/usr/bin/g++ /usr/bin/g++

[dunand]

I can compile blockparser but I have this error when using it:

Code:

user@machine:~/blockparser$ ./parser allBalances

info: starting command "allBalances"
info: analyzing blockchain ...
fatal: Invalid argument: failed to open block chain file /home/user/.bitcoin/blocks/blk00000.dat
Aborted (core dumped)

I'm running 0.8. Is blockparser working with 0.8?

[dooglus]

I saw tons of likely/unlikely in the code, what do they do?

They are hints to the compiler's optimizer, and tell it whether a condition is usually true or false.

Code like:

Code:

if (condition)
  A();
rest_of_code();
exit();

Could be compiled in one of two ways, depending on whether the condition is usually true:

Code:

test condition
if false, goto rest;
A();
rest: rest_of_code();
exit();

or usually false:

Code:

test condition
if true, goto do_A;
rest: rest_of_code();
exit();

do_A:
A();
goto rest;

In both cases, when condition has its expected value no 'goto' is executed and so the code is faster in that case.

[ShadowOfHarbringer]

can someone explain why all znort987 posts have been vanished?

Will

It is probably some kind of SCAM, there was a warning on the beginning by a high ranked user (was it grondilu ?).

So use this code with care.

EDIT: Oh yeah, here it is:

After interacting with znort987 on IRC I'd advise being very careful in using this software double check any results that you get from it.

I pointed out that he was making some (theoretically and practically) minorly incorrect claims about the analysis this tool produced and his response was confused first failing on technically grounds and then after the technical misunderstand was made clear he became uncivil— calling me and other folks names and such— quite bizarre. While its obviously useful, it's written by someone who apparently hasn't done your thinking for you, so you need to think for yourself.

[Sukrim]

This was most likely because closure (2 inputs being viewed as belonging to the exact same person) is not a 100% measure, as demonstrated some time ago where someone asked others to sign some transactions with him to add his coins to their closure calculation and vice versa.

The code is open source, so it is not hard to find any scams in it... Please be careful with your accusations, maybe this is why he left?!

[ShadowOfHarbringer]

The code is open source, so it is not hard to find any scams in it... Please be careful with your accusations, maybe this is why he left?!

I am only saying that gmaxwell found this suspicious.

[dooglus]

This was most likely because closure (2 inputs being viewed as belonging to the exact same person) is not a 100% measure, as demonstrated some time ago where someone asked others to sign some transactions with him to add his coins to their closure calculation and vice versa.

That someone was gmaxwell.

This seems to be the IRC chat they're referring to:

--- Log opened Sat Mar 09 00:00:01 2013
12:23 -!- znort987 [~znort987@gateway/tor-sasl/znort987] has joined #bitcoin
12:24 < znort987> ;;gpg eauth znort987
12:24 <@gribble> Request successful for user znort987, hostmask znort987!~znort987@gateway/tor-sasl/znort987. Get your encrypted OTP from http://bitcoin-otc.com/otps/FD5AA2A5E26602B0
12:26 < znort987> ;;gpg everify freenode:#bitcoin-otc:f6c4d80bf1e5f7f84a1cae73581f3a7d824b23838dccc6a8e3715ac1
12:26 <@gribble> You are now authenticated for user znort987 with key FD5AA2A5E26602B0
13:50 <@gmaxwell> "Compute the list of addresses that provably belong to the same person"  = fail.
13:50 <@gmaxwell> znort987: can you fix that claim?
13:50 < znort987> gmaxwell: fail, "how so" ?
13:51 <@gmaxwell> Before your software results in someone getting prosecuted for someone elses crime or other facepalmery.
13:51 <@gmaxwell> znort987: being common authors on an input does not imply common address ownership.
13:51 < comboy> znort987: that was posted probably befoe multisig?
13:51 <@gmaxwell> even without multisig.
13:51 < comboy> without multisig it does not prove that?
13:51 <@gmaxwell> E.g. you can have a transaction where I provide one input and you provide another.
13:52 < znort987> All I'm hearing are claims, and not explanation forthcoming
13:52 <@gmaxwell> wtf.
13:52 < comboy> ah, indeed
13:52 <@phantomcircuit> znort987, let me save you some time
13:52 <@gmaxwell> znort987: wow. I just explained it.
13:52 <@phantomcircuit> if gmaxwell assets plainly that something is true, you can pretty much be sure that it is
13:53 < jouke> gmaxwell: just give him the address :>
13:53 < znort987> phantomcircuit: sure. That still doesn't explain it.
13:53 < Scrat> i've seen gmaxwell make typos. ha ha!
13:53 <@gmaxwell> znort987: Okay, I'm not sure what was unclear there.
13:53 < comboy> znort987: you could probably only assume same author if it was signed by the same priv key
13:54 <@phantomcircuit> znort987, valid transactions can be constructed in which multiple parties sign the input portion of the transaction
13:54 <@gmaxwell> znort987: We write a transaction it has two inputs. 1znort987 and 1gmaxwell.  Each of us respectively sign the transaction for our own inputs. This does not mean we are the same person.
13:54 < znort987> ok, problem is fixed
13:54 <@phantomcircuit> znort987, in short your assumption is wrong
13:54 <+TuxBlackEdo> znort987, when people make bitcoin transactions, sometimes two people that don't know eachother exchange private keys to make a transaction to someone else
13:54 <@gmaxwell> TuxBlackEdo: or just jointly sign
13:55 <@phantomcircuit> TuxBlackEdo, dont need to exchange private keys, just sign the same outputs
13:55 <+TuxBlackEdo> oh yeah
13:55 <+TuxBlackEdo> but still
13:55 <@gmaxwell> People also transfer ownership of privkeys.
13:55 <+TuxBlackEdo> how often does that happen
13:55 < Scrat> what about disassociative mixing? then it will assign ownership toi completely different people
13:55 < znort987> p(keys owned by same person) = count(weird tx assembled by folks who can understand the shit you guys are talking about)/count(all transactions)
13:56 <@phantomcircuit> TuxBlackEdo, often enough that assuming people are linked by same signed inputs is invalid
13:56 <@gmaxwell> E.g. I can take a key with a long history and load in into mtgox. Mtgox will then very likely sign with it in a transaction with keys from other people.
13:56 <+TuxBlackEdo> i am sure one could get in trouble just by being the first relayer of a transaction
13:56 < Scrat> znort987: b.i mixer. used by many people
13:56 < znort987> therefore, fix  = s/provably/very likely/
13:56 <+TuxBlackEdo> "oh you forwarded me a drug transaction, that must mean you must be a drug dealer"
13:56 < comboy> znort987: then s/provably/very likely
13:56 < znort987> Are sphincters relaxing now ?
13:57 <+TuxBlackEdo> ^
13:57 <@phantomcircuit> znort987, no actually because of the mtgox private key load it isn't even likely
13:57 <@phantomcircuit> there are very simple tools to upload your wallet.dat to mtgox
13:57 <@gmaxwell> znort987: It looks like the b.i 'shared send' (replacement for their mixer service) works this way normally.
13:57 <@phantomcircuit> so indeed any correlation is *highly* suspect
13:58 <@gmaxwell> So I wouldn't be surprised if a few percent of txn on the network are like that now.
13:58 <@gmaxwell> But sure, I agree that its likely, though it'll be less of an indicator in the future.
13:58 <@gmaxwell> But seriously, be more careful with the provably in the future.
13:59 <@phantomcircuit> gmaxwell, i suspect the amount of mixing sr engages in actually makes even current assumptions incorrect
13:59 <@phantomcircuit> i've moved coins through there to find out and i honestly couldn't figure out where they came out
13:59 < znort987> For the anal retentive crowd out there, in general, there is no such thing as a proof in the real world. Just high probabilities.
14:00 < Scrat> phantomcircuit: same goes for gox withdrawals sometimes
14:00 <@gmaxwell> znort987: there are orders of magnitude, however.
14:00 < Scrat> or using the green address
14:00 <@phantomcircuit> znort987, the point is that the probabilities are likely low but even so unknowable
14:02 < znort987> Nope. The point is : is the information extracted that way any kind of useful. I claim that as things stand today, they are, notwithstanding the language lwayers.
14:02 < znort987> *lawyers
14:02 <@gmaxwell> znort987: often we talk about proven when the chances are astronomical absent a mathmatical breakthrough. Your clam was likely invalid for a substantial percentage so of addresses holding coins. (due to your use of the word belong)
14:03 <@phantomcircuit> znort987, forget lawyering, your odds are just plain wrong
14:03 <@phantomcircuit> there is a lot more mixing going on than most people thing
14:03 <@gmaxwell> znort987: 'any kind of useful' sure. Proven absolutely not. Very likely?  eh. maybe.
14:03 <@phantomcircuit> think*
14:03 <@phantomcircuit> a *lot* more
14:04 <@phantomcircuit> gmaxwell, actually i would say the odds are very low unless it's a satoshi dice transaction
14:04 <@gmaxwell> I would not even be surprised if your claim was incorrect for a simple majority of coin ownership weighed addresses, simply due to shared wallet usage.
14:04 < znort987> You guys are in the wrong business. I'm thinking you ought to give a try to fly fucking or classifying dead invertebrates. If there was olympic games for nitpicking you'd likely be in it.
14:04 <@phantomcircuit> most people are really using shared wallets of some kind
14:04 <@phantomcircuit> lol
14:04 <@phantomcircuit> while that was hilarious
14:04 <@phantomcircuit> (im drunk enough that that was funny)
14:04 <+TuxBlackEdo> znort987, i think you need to calm down, these are smart people
14:04 <@phantomcircuit> you're still wrong
14:05 <@gmaxwell> znort987: 'nitpicking' when your claims are objectively incorrect on a substantial number of channels, and could cause people harm? give me a break.
14:05 < znort987> intelligent != smart. A textbook example if there ever was one.
14:05 <@gmaxwell> znort987: how about you behave responsibly with your software, and work on your civility a bit too? enh?
14:05 <@phantomcircuit> gmaxwell, sigh i really want to ban him but i know im too drunk to make such a decision
14:05 <@phantomcircuit> :|
14:05 <+TuxBlackEdo> znort987, the only one with a tight sphincer is you, right now.
14:06 <@gmaxwell> hm. s/channels/transactions/  (this is what I get for concurrently talking and typing)
14:06 <@phantomcircuit> as TuxBlackEdo says
14:06 <@phantomcircuit> my sphincter is very loose
14:06  * phantomcircuit lols away
14:06 <+TuxBlackEdo> phantomcircuit, i know
14:06 <@phantomcircuit> ahahah
14:07 <+TuxBlackEdo> no homo.
14:07 <@phantomcircuit> yeah sure
14:07 <@phantomcircuit> whatever you say
14:07 < znort987> gmaxwell: "responsibly" ? Are you actually dumb enough to believe that someone would use this in a court of law ? You must be living in the US.
14:08 <@phantomcircuit> zomg can i ban him now
14:08 <@phantomcircuit> pleassseeeee
14:08 -!- mode/#bitcoin [+b *!*znort987@gateway/tor-sasl/znort987] by gmaxwell
14:08 -!- znort987 was kicked from #bitcoin by gmaxwell [znort987]
14:08 <@phantomcircuit> HORRAAAYYY
14:08 <+TuxBlackEdo> i thought gmaxwell was from mexico
14:15 <@gmaxwell> tor user, why am I surprised.
14:15 -!- mode/#bitcoin [-b *!*znort987@gateway/tor-sasl/znort987] by gmaxwell
14:15 <@gmaxwell> https://bitcointalk.org/index.php?topic=88584.msg1603598#msg1603598 FWIW
14:16 <@gmaxwell> I notice he changed it back to " provably belong to the same person"
14:17 < valparaiso> yes, shame really. Anonymity in the hands of trolls...
14:17 < comboy> huh, that escalated quickly, he shouldn't say "provably", especially in bitcoin community, but still he has done really nice work on this parser
14:18 <@phantomcircuit> jesus crist i have 429 posts on the forum???
14:18  * phantomcircuit needs to get a life
14:18 <@phantomcircuit> :|

[dunand]

I can compile blockparser but I have this error when using it:

Code:

user@machine:~/blockparser$ ./parser allBalances

info: starting command "allBalances"
info: analyzing blockchain ...
fatal: Invalid argument: failed to open block chain file /home/user/.bitcoin/blocks/blk00000.dat
Aborted (core dumped)

I'm running 0.8. Is blockparser working with 0.8?

[wiggi]

I'm running 0.8. Is blockparser working with 0.8?

Yes, should definitely work with 0.8.1, just a bit slower than with the older (0.7.x) versions.

[gweedo]

After interacting with znort987 on IRC I'd advise being very careful in using this software double check any results that you get from it.

I pointed out that he was making some (theoretically and practically) minorly incorrect claims about the analysis this tool produced and his response was confused first failing on technically grounds and then after the technical misunderstand was made clear he became uncivil— calling me and other folks names and such— quite bizarre. While its obviously useful, it's written by someone who apparently hasn't done your thinking for you, so you need to think for yourself.

LMAO more censorship by the Core Dev team, first it was transaction now it is other people software. I mean when does it stop. Honestly power hungry much? So dumb, you, Gavin, and Luke Jr. I really hope another programming team, just takes over and we can be done, this what no competition does in a free market. Creates egotistical, self centered, power hungry people. Know what instead of policing the forum and irc actually make the full node work with a big blockchain instead of putting a band-aid on it. Never saw more power hungry people in my life. Nothing decentralized about bitcoins, it completely centralized with people like this running behind it.

[andr3w321]

I can compile blockparser but I have this error when using it:

Code:

user@machine:~/blockparser$ ./parser allBalances

info: starting command "allBalances"
info: analyzing blockchain ...
fatal: Invalid argument: failed to open block chain file /home/user/.bitcoin/blocks/blk00000.dat
Aborted (core dumped)

I'm running 0.8. Is blockparser working with 0.8?

Does this file exist and do you have permission to open it?  Print the output of

Code:

ls -l /home/user/.bitcoin/blocks

[wolongong]

How much is "*lots* of RAM" in the warning for --detailed?

[wiggi]

How much is "*lots* of RAM" in the warning for --detailed?

It likely will finish on 8GB but I wouldn't do this to my HDD,
and

Code:

./parser allBalances >allBalances.txt

(without -d) is about as memory-demanding as

Code:

./parser allBalances -a200000 -d >allBalances.txt

upgrade plan:
now:     16GB
Q1 2014: 32GB

[refer_2_me]

So I ran ./parser allBalances >allBalances.txt. It took a while to run, but didn't throw any errors and seemed to complete normally.
The first 100 lines or so of the output also seem fine, but after that the Bitcoin addresses are all X's: for example:

"35.09900000 0476042f5062d0f4092dcab601b142c0dcf339ef XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX      1 Thu May 16 13:26:12 2013       0 Thu Jan  1 00:00:00 1970"

Anyone have any idea why this is happening and how I can fix it? I glanced at the code, but don't really understand that is happening.

Thanks in advance!

[w00dy]

So I ran ./parser allBalances >allBalances.txt. It took a while to run, but didn't throw any errors and seemed to complete normally.
The first 100 lines or so of the output also seem fine, but after that the Bitcoin addresses are all X's: for example:

"35.09900000 0476042f5062d0f4092dcab601b142c0dcf339ef XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX      1 Thu May 16 13:26:12 2013       0 Thu Jan  1 00:00:00 1970"

Anyone have any idea why this is happening and how I can fix it? I glanced at the code, but don't really understand that is happening.

Thanks in advance!

use the -w / --withAddr switch.

(out of my head):

Code:

./parser allBalances -w 100000 >allBalances.txt

("only show address for top N results (default: N=%default)"

[refer_2_me]

Ah, thank you. I can't believe I missed that.

[emsit]

Hello, What is the minimum RAM need to perform a command?
I have 512MB and all command ended with error: Killed

Can you tell me some other parser that can display a one block of raw?

[Sukrim]

To load all the stuff in RAM you probably need by now far more than 512 MB...

bitcoind can give you raw blocks via its RPC interface and you can read the raw block files too.

[emsit]

Try bitcoind RPC but I found only this commands: getblock

I need information with including transactions. Something like https://blockchain.info/rawblock/1

EDIT:I must as first get getblock and get for each transaction getrawtransaction and, finally connected to one output

[barwizi]

any guide to use in windows? or does someone have it built for windows already?
 also can i have a few tips on how to apply it on a different chain?

[Sukrim]

any guide to use in windows? or does someone have it built for windows already?
 also can i have a few tips on how to apply it on a different chain?

znort never intended it to run on Windows, if you post a bounty maybe someone is able to compile it? Otherwise, just get going, theoretically the libraries used should be available on Windows too.

As it parses the transactions, it likely depends on how close your "different chain" implements these compared to Bitcoin.

[organofcorti]

any guide to use in windows? or does someone have it built for windows already?
 also can i have a few tips on how to apply it on a different chain?

znort never intended it to run on Windows, if you post a bounty maybe someone is able to compile it? Otherwise, just get going, theoretically the libraries used should be available on Windows too.

As it parses the transactions, it likely depends on how close your "different chain" implements these compared to Bitcoin.

I think it worked with cygwin when I tried it. Otherwise a dual boot with linux or a VM are going to be your only options.

[Sukrim]

Well, looking at RAM requirements, I guess I'd recommend a live system from a USB stick if you just want to try the parser instead of a VM.

[barwizi]

Thanks!! so if the code base is bitcoin, if i just copy the files i want to parse to .bitcoin it will work? or are there some parameters i need to change?

[jj30]

Can anyone help? Make went fine. When running the parser, it's just hanging...

info: starting command "simpleStats"

and then it just sits there indefinitely. I ctrl-c after a half hour or so, assuming nothing is happening (8 gb memory).

What have I done wrong? I think there is an "info: parsing blockchain" message supposed to come after that. I have checked and my blockchain is in ~/.bitcoin.

Can anyone help?

EDIT: SimpleStats came back in 5774 seconds ... a little over an hour and a half... so I guess I need more memory or something.

[gweedo]

EDIT: SimpleStats came back in 5774 seconds ... a little over an hour and a half... so I guess I need more memory or something.

Remember the blockchain is always growing, when Znort wrote this a while ago it was like 7 or 8gb now it is 14 or 15gb that is a lot to parse and keep in memory. So you don't need more memory it is just a lot of data.

[barwizi]

I want to hire someone to modify it for a different coin, if you can do it and are interested ,PM me and we can get to business. Need this asap.

[ShadowOfHarbringer]

I wonder...

- Is the author even maintaining/supporting this project in any way ?
- Does this thing even work ?
- Why did first post get deleted ?

And NO, i do not want to read-through all hundereds of posts to find this out.

[Sukrim]

No idea if it works, I don't have enough RAM and wrote my own parser based on armory-python.

Probably it works, try it out.

Seems like he wanted to delete all his posting history on bitcointalk (just like some others did too).

[Vinz87]

can someone help me? I'm on OS X 10.9, I have gcc 4.5 installed via homebrew, now I have to install the dependencies this software relies on, should I use homebrew as well?

but for example I get this error

Code:

$ brew install libssl-dev
Error: No available formula for libssl-dev 

[mullick]

I dont think i have the correct syntax here or something. Cant seem to get this to work

Code:

./parser simpleStats

info: starting command "simpleStats"
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_S_construct null not valid
Aborted (core dumped)

Tried a few commands. As well as declaring the block directory. Same result. Am I missing something here?