.

[Jouke]

almost complete fresh install of ubuntu 64 bit.

[runlinux]

installing 12.04 x64 in VMWare player.

I really want to play with this. I need to brush up on my C++ skills... haven't used it in about 10 years..

[runlinux]

yay! compiled on my first try!

EDIT:

snap... I need to DL the blockchain?! ungh... (What was I thinking? lol)

copy and paste from host machine...

[runlinux]

yup, i found that out...

[Jouke]

xubuntu 12.10 or something like that

Code:

In file included from cb/allBalances.cpp:4:0:
./util.h: In instantiation of ‘void GoogMap<Key, Value, Hasher, Equal>::Map::setEmptyKey(const Key&) [with Key = const unsigned char*; Value = Addr*; Hasher = Hash160Hasher; Equal = Hash160Equal]’:
cb/allBalances.cpp:67:37:   required from here
./util.h:122:21: error: ‘set_empty_key’ was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive]
./util.h:122:21: note: declarations in dependent base ‘google::dense_hash_map<const unsigned char*, Addr*, Hash160Hasher, Hash160Equal, google::libc_allocator_with_realloc<std::pair<const unsigned char* const, Addr*> > >’ are not found by unqualified lookup
./util.h:122:21: note: use ‘this->set_empty_key’ instead
In file included from cb/taint.cpp:25:0:
./util.h: In instantiation of ‘void GoogMap<Key, Value, Hasher, Equal>::Map::setEmptyKey(const Key&) [with Key = const unsigned char*; Value = long double; Hasher = Hash256Hasher; Equal = Hash256Equal]’:
cb/taint.cpp:70:35:   required from here
./util.h:122:21: error: ‘set_empty_key’ was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive]
./util.h:122:21: note: declarations in dependent base ‘google::dense_hash_map<const unsigned char*, long double, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std::pair<const unsigned char* const, long double> > >’ are not found by unqualified lookup
./util.h:122:21: note: use ‘this->set_empty_key’ instead
In file included from cb/transactions.cpp:5:0:
./util.h: In instantiation of ‘void GoogMap<Key, Value, Hasher, Equal>::Map::setEmptyKey(const Key&) [with Key = const unsigned char*; Value = int; Hasher = Hash160Hasher; Equal = Hash160Equal]’:
cb/transactions.cpp:64:37:   required from here
./util.h:122:21: error: ‘set_empty_key’ was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive]
./util.h:122:21: note: declarations in dependent base ‘google::dense_hash_map<const unsigned char*, int, Hash160Hasher, Hash160Equal, google::libc_allocator_with_realloc<std::pair<const unsigned char* const, int> > >’ are not found by unqualified lookup
./util.h:122:21: note: use ‘this->set_empty_key’ instead
make: *** [.objs/taint.o] Error 1
make: *** Waiting for unfinished jobs....
make: *** [.objs/allBalances.o] Error 1
make: *** [.objs/transactions.o] Error 1
In file included from parser.cpp:2:0:
./util.h: In instantiation of ‘void GoogMap<Key, Value, Hasher, Equal>::Map::setEmptyKey(const Key&) [with Key = const unsigned char*; Value = const unsigned char*; Hasher = Hash256Hasher; Equal = Hash256Equal]’:
parser.cpp:449:29:   required from here
./util.h:122:21: error: ‘set_empty_key’ was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive]
./util.h:122:21: note: declarations in dependent base ‘google::dense_hash_map<const unsigned char*, const unsigned char*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std::pair<const unsigned char* const, const unsigned char*> > >’ are not found by unqualified lookup
./util.h:122:21: note: use ‘this->set_empty_key’ instead
./util.h: In instantiation of ‘void GoogMap<Key, Value, Hasher, Equal>::Map::setEmptyKey(const Key&) [with Key = const unsigned char*; Value = Block*; Hasher = Hash256Hasher; Equal = Hash256Equal]’:
parser.cpp:450:32:   required from here
./util.h:122:21: error: ‘set_empty_key’ was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive]
./util.h:122:21: note: declarations in dependent base ‘google::dense_hash_map<const unsigned char*, Block*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std::pair<const unsigned char* const, Block*> > >’ are not found by unqualified lookup
./util.h:122:21: note: use ‘this->set_empty_key’ instead
make: *** [.objs/parser.o] Error 1
In file included from cb/closure.cpp:4:0:
./util.h: In instantiation of ‘void GoogMap<Key, Value, Hasher, Equal>::Map::setEmptyKey(const Key&) [with Key = const unsigned char*; Value = long unsigned int; Hasher = Hash160Hasher; Equal = Hash160Equal]’:
cb/closure.cpp:43:38:   required from here
./util.h:122:21: error: ‘set_empty_key’ was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive]
./util.h:122:21: note: declarations in dependent base ‘google::dense_hash_map<const unsigned char*, long unsigned int, Hash160Hasher, Hash160Equal, google::libc_allocator_with_realloc<std::pair<const unsigned char* const, long unsigned int> > >’ are not found by unqualified lookup
./util.h:122:21: note: use ‘this->set_empty_key’ instead
make: *** [.objs/closure.o] Error 1

[Jouke]

Ok, it does not compile on g++ 4.7, it does on g++ 4.5

[Jouke]

Ok, it does not compile on g++ 4.7, it does on g++ 4.5

OK.
Let me install 4.7 and see what's what.

Fixed and fix pushed to github

Compiles nicely

[Jouke]

Is it possible to point it to an other dir then ~/.bitcoin ?


Tried it some other way.

I have a specific problem. I had a blk0001.dat already when I first started the bitcoin-client. Instead of using that file it made a blk0002.dat and recently it made a blk0003.dat. The parser won't open the blockchain file: "Invalid argument: failed to open block chain file /home/jouke/.bitcoin/blk0001.dat"

I guessed that 0002 was actually 0001 and tried to symlink 0002 to 0001 and 0003 to 0002 in a separate directory, but your parser still won't open the file.

Any sugestions?

[dooglus]

Is it possible to point it to an other dir then ~/.bitcoin ?


Tried it some other way.

I have a specific problem. I had a blk0001.dat already when I first started the bitcoin-client. Instead of using that file it made a blk0002.dat and recently it made a blk0003.dat. The parser won't open the blockchain file: "Invalid argument: failed to open block chain file /home/jouke/.bitcoin/blk0001.dat"

I guessed that 0002 was actually 0001 and tried to symlink 0002 to 0001 and 0003 to 0002 in a separate directory, but your parser still won't open the file.

Any sugestions?

Temporarily set HOME to wherever you want it to read the blockchain from:

$ HOME=/path/to/folder ./parser closure
[...]
fatal: No such file or directory: failed to open block chain file /path/to/folder/.bitcoin/blk0001.dat

I doubt the parser made any blockchain files.  It opens the files readonly, which won't create any files.

About a week ago the blockchain file grew bigger than 2GB and so a 2nd file was created by bitcoin-qt:

$ ls -l blk*
-rw------- 1 chris chris 2097307549 2012-07-12 20:07 blk0001.dat
-rw------- 1 chris chris  104659251 2012-07-19 02:52 blk0002.dat
-rw------- 1 chris chris  751022080 2012-07-19 02:52 blkindex.dat

I don't know where the 0003 file came from though!

[Jouke]

Temporarily set HOME to wherever you want it to read the blockchain from:

Already did that. That's where I symlinked the file to.

I doubt the parser made any blockchain files.  It opens the files readonly, which won't create any files.

I didn't say it did?

About a week ago the blockchain file grew bigger than 2GB and so a 2nd file was created by bitcoin-qt:

$ ls -l blk*
-rw------- 1 chris chris 2097307549 2012-07-12 20:07 blk0001.dat
-rw------- 1 chris chris  104659251 2012-07-19 02:52 blk0002.dat
-rw------- 1 chris chris  751022080 2012-07-19 02:52 blkindex.dat

I don't know where the 0003 file came from though!

I Know a 2nd file was created.

Because I already had a blk0001.dat, my client started with blk0002.dat.

[runlinux]

How hard would it be to add address balance to the closure method? (and a total balance at the bottom with address count)

I can't figure it out. My C++ is old and unused... - at least I can read the code!

[runlinux]

Thanks for that! i'll modify the output so its just the address. from there a simple script should do what i need

[runlinux]

OK, got a question now:

Why does this address: 12BdKh5HHUyGFduGm6y8nY4KS1LxzsduXo

Link to this address: 12cbQLTFMXRnSzktFkuoG3eHoMeFtpTu3S

Closure reports it has no outputs then returns that one.

On a side note, script written and code modified to take a wallet address, and then just output the balance of the final wallet

Code:

runlinux@ubuntu:~/blockparser$ ./wallet.sh 1MrPonziaM4QT2S7SdPEKQH88BGa4LRHJU

    transactions  = 211
    received      =      207.30710185
    spent         =      207.30680287
    balance       =        0.00029898

[dooglus]

Why does this address: 12BdKh5HHUyGFduGm6y8nY4KS1LxzsduXo

Link to this address: 12cbQLTFMXRnSzktFkuoG3eHoMeFtpTu3S

Closure reports it has no outputs then returns that one.

I see the same.

It tells me:

Code:

error: specified key was never used in a TX output

but this transaction has 12BdKh5HHUyGFduGm6y8nY4KS1LxzsduXo as an output, so the message is a little confusing.

And then the address isn't in its own reported cluster - that's got to be a bug.

[Sukrim]

What's your definition of "taint"?

If there is 6% taint from pirate in that huge address, does that mean 6% of these BTC were at one point in time owned by a pirate address or that pirate deposited 6% of these coins to that address?

If it is the first definition, isn't there the possibility to have >100% total taint? E.g. pirate sends BTC to MtGox and then MtGox sends these to the huge address - now they are "tainted" by pirate AND "tainted" by MtGox...

What I would still like to do is to enter an address, get the closure of that and then get the closures of every address where money was sent to or received from that cluster (ideally I could enter how many levels deep, with "0" being the current closure setting).

[etotheipi]

I'm sure you know I've spent a lot of time doing blockchain scanning, and attempting to optimize it in various ways in Armory.  In the end, the fastest I could achieve was about 20-25s on a full blk0001.dat (2GB) on Linux-64bit using memory-mapping.  I couldn't achieve the same thing in Windows (mmap'ing seems to be garbage there).   I just wanted to share my experience from playing with leveldb, how awesomely optimized it is.

First, I iterated through blk0001.dat and stored all transactions in a leveldb database using Key:TxHash-->Value:RawTx.   I plugged 4 addresses into a set<BinaryData> object.  I then iterated (out of order) through the entire database, pulling each transaction into memory from the DB, and then searching for TxOuts matching any of those 4 addresses.  This took 8.0 seconds! (but was only the TxOuts).  This included putting the TxOuts into a map<OutPoint,uint64_t> (OutPoint, nBTC), so that I could then scan the entire DB again and search for TxIns. 

At the end of both scans, I had the balance and unspent TxOut list from those four addresses.  It took approximately 12-13 seconds!  And this is with a non-SSD drive:  it seems like it shouldn't even be possible to go that quickly... I had to quadruple-check everything to verify that I wasn't missing something or accidentally supplying the answer.  Perhaps there was some OS-level caching of the DB files, though I would've expected such caching to improve my custom blockchain utilities performance. 

I don't intend to discourage or insult your efforts (in fact, I think you said you were getting similar speeds).  Plus, I think that writing such a parser/scanner is both useful and educational.  I just bring it up because I would guess that this is about as good as anyone can do with a non-SSD drive.  And it's especially impressive considering I did no optimizations.  I'm especially curious how leveldb works under the hood that it could be so efficient.  Maybe there's something to learn from it.

[jgarzik]

I don't intend to discourage or insult your efforts (in fact, I think you said you were getting similar speeds).  Plus, I think that writing such a parser/scanner is both useful and educational.  I just bring it up because I would guess that this is about as good as anyone can do with a non-SSD drive.  And it's especially impressive considering I did no optimizations.  I'm especially curious how leveldb works under the hood that it could be so efficient.  Maybe there's something to learn from it.

LevelDB provides the backend to Google's services, so it is not surprising that it is so heavily optimized.  One of the key optimizations is that writes go to a log, and then another thread comes along in the background and "optimizes" the log data back into the main database, using hueristics to hopefully place the data in a useful, locality-friendly location on disk.

BitcoinJ author Mike Hearn has proposed replacing BDB with LevelDB in the Satoshi client, and we are looking seriously at that.  Here is his pull request: https://github.com/bitcoin/bitcoin/pull/1619

[etotheipi]

Finally, it's a good idea to do this: in the end the blockchain will anyway grow
so big that even the hashmaps won't fit in RAM, and my approach will start
swapping like crazy. Pre-indexing in a disk-level DB hash table is the right thing
to do.

Well that's why I started looking into this.  My original blockchain utilities design in Armory were more of an educational adventure to see how fast I could get the scanning from scratch (no external libraries), with lots of pointers, etc.  It was educational, and I was happy with the results.  But I had the same realization that even the index will eventually not fit in RAM, and thus there is no way to make the library scalable without using some kind of disk-based engine.  To that end, I had no interest re-inventing disk-based DB operations, etc.  When I heard the devs talking about LevelDB, I looked into and quickly realized why they chose it.  It is very simple, portable... and apparently fast!

However, gmaxwell just told me how to clear my disk cache and then retest.  It was dramatically slower for sure.  Probably by a factor of 5-10x.  However, I never ran the same test using my other blockchain utilities, so I don't know the actual comparison.  Given that modern OS'es are good at doing that caching, the LevelDB version is clearly a very good solution.

I'm currently investigating how I can transition Armory to maintain it's own blockchain data now with LevelDB

[jgarzik]

FWIW the set of unspent tx-out's should fit nicely in RAM, for some time to come.

I know that's not helpful to applications doing historical queries, but it may be helpful for other queries.

[vampire]

znort987:

Could you add to parser.c:

#ifndef O_DIRECT
#define O_DIRECT 0x10000000
#endif

O_DIRECT isn't defined on OSX. Otherwise I was able to compile on my macbook with macports.