Title: . Post by: znort987 on June 19, 2012, 09:41:01 PM .
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: runlinux on June 19, 2012, 11:20:55 PM Now what you need to do is group addresses based on the inputs and create a list of "wallets" in the same fashion. I have a tool that does that, to a point...
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: malaimult on June 19, 2012, 11:43:45 PM please stop posting my btc addresses :-[
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: bulanula on June 19, 2012, 11:46:07 PM please stop posting my btc addresses :-[ You have 8000+ coins and hack Linode ??? Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: etotheipi on June 19, 2012, 11:49:32 PM To try and understand the nitty-gritty of how bitcoin works, I've written a fairly fast blockchain parser in C++ (x86-64 ubuntu only). It can currently do the following things: . Compute simple blockchain stats, full chain parse (< 1 second) . Extract all known transactions for a popular address (20 seconds) . Compute the list of addresses that provably belong to the same person (20 seconds): . Compute and print the balance for all keys ever used since the beginning of time (30 seconds): . You can quite easily add your own data extraction code Nice! Mind sharing a top-level overview of how you do it? Data structures? Do you know how much that is I/O limited? The C++ utilities I wrote for Armory are almost that fast (but in linux only, have no idea why windows is so damned slow!). I buffer blk000X.dat into RAM 50 MB at a time, and then pass around pointers to this data and scan it directly -- to avoid any extra copies. It's a lot of pointers... Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: malaimult on June 19, 2012, 11:52:41 PM damn it. you got me ;D let's share the btc and don't tell anyone. will be our little secret ;D
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: malaimult on June 19, 2012, 11:53:03 PM dirty secret ::)
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: bulanula on June 19, 2012, 11:56:15 PM damn it. you got me ;D let's share the btc and don't tell anyone. will be our little secret ;D Send some love to 15uxVpSgWtfQ25ae6YYxG3US3eWRpYqKkp if that is the case. I sure do loves me some stolen coins :-* Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: malaimult on June 20, 2012, 12:03:35 AM i wish i could. :'( my mom got my wallet pasw. if you know someone who can torture my mom so she can tell the pasw i will pay 20.000 btc
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: SgtSpike on June 20, 2012, 12:11:57 AM Now what you need to do is group addresses based on the inputs and create a list of "wallets" in the same fashion. I have a tool that does that, to a point... It already does that: that's what the "closure" callback does. As a matter of fact, second teaser: here's the transitive closure of the Linode hack address where slush lost BTC 3K+ (all these addresses are controlled by the same person). Code:
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: SgtSpike on June 20, 2012, 12:20:24 AM What does 0x80ab5bcd943419b8988234e8e19b83389edc542a mean? It's not a Bitcoin address in the typical format. Yeah, sorry about that, I should have dropped the 0x in front. Once you remove the 0x, it's the ripemd-160 hash of the public key. So: . Copy everything but the '0x' , in your example, that would be : 80ab5bcd943419b8988234e8e19b83389edc542a . Go paste that number in http://blockchain.info . You'll land on a page that has the usual representation, in this case: 1CjLmm25ty4aSjxzG8Er4qSiQ63bqaQGA8 I have to ask though, why not display the hashed address to begin with? Just an extra step of complication? Also, want to run it on my public address? I am curious what other addresses are correlated with it... ;) If not, that's fine, I'll download it and attempt to compile myself. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: SgtSpike on June 20, 2012, 12:27:29 AM What does 0x80ab5bcd943419b8988234e8e19b83389edc542a mean? It's not a Bitcoin address in the typical format. Yeah, sorry about that, I should have dropped the 0x in front. Once you remove the 0x, it's the ripemd-160 hash of the public key. So: . Copy everything but the '0x' , in your example, that would be : 80ab5bcd943419b8988234e8e19b83389edc542a . Go paste that number in http://blockchain.info . You'll land on a page that has the usual representation, in this case: 1CjLmm25ty4aSjxzG8Er4qSiQ63bqaQGA8 I have to ask though, why not display the hashed address to begin with? Just an extra step of complication? Not sure what you mean by hashed addresses. What I am displaying *is* a hashed address. You mean base58-encoded addresses like "1CjLmm25ty4aSjxzG8Er4qSiQ63bqaQGA8" ? If so, then : these things are way too expensive to compute, this was designed for spped. Quote Also, want to run it on my public address? I am curious what other addresses are correlated with it... ;) If not, that's fine, I'll download it and attempt to compile myself. Give me a secTitle: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: SgtSpike on June 20, 2012, 12:36:08 AM Also, want to run it on my public address? I am curious what other addresses are correlated with it... ;) If not, that's fine, I'll download it and attempt to compile myself. Here you go. Even translated the result to the usual format. I'd be curious to see if it worked. Let me know. Code: 18TKNbSLTrd3a2W8mtoH5uNzFhWRWNcuHU 1EScB6WzkeDVUZwLSsAHxUNuRr8jN38MLm Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: etotheipi on June 20, 2012, 12:40:47 AM The C++ utilities I wrote for Armory are almost that fast (but in linux only, have no idea why windows is so damned slow!). I buffer blk000X.dat into RAM 50 MB at a time, and then pass around pointers to this data and scan it directly -- to avoid any extra copies. It's a lot of pointers... Looks like you went and re-did by hand pretty much what the OS does for free when you use memory-mapped files ... I used memory-mapped files in version 0.60 to 0.77 of Armory. Not only was it slow as dirt in Windows (though it worked brilliantly in Linux), it won't work on any 32-bit systems once the blockchain reaches 4GB+. 32-bit processes can't allocate enough virtual memory. Hell, Win32 wouldn't take it at all even with 4 GB RAM, and a 1 GB blockchain (hence why Armory never worked in Win32). Not only that, but mmap is OS-dependent. If you want to use Map-Of-File-Views (the mmap equiv in Windows) you're going to deal with windows.h and LPWCSTRs. I'm so glad I got rid of that. That's why I emulate it now, and it cleaned up my code a bit not having to branch base on OS. I understand about the python wrappers. You don't have to use the python, there's a pure C++ library underneath, though it's not obvious how to use it. BlockUtilsTest.cpp has a lot of examples, but it's not intended for educational purposes: it's really for me to do my unit-testing. And in the end, I always approve of people digging in and implementing pieces of Bitcoin. There's no better way to really undrestand it. :) Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: SgtSpike on June 20, 2012, 12:43:45 AM Now what you need to do is group addresses based on the inputs and create a list of "wallets" in the same fashion. I have a tool that does that, to a point... It already does that: that's what the "closure" callback does. As a matter of fact, second teaser: here's the transitive closure of the Linode hack address where slush lost BTC 3K+ (all these addresses are controlled by the same person). Edit : changed it to look like usual addresses instead of hash160 Code:
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: mrb on June 20, 2012, 06:49:10 AM Sub.
(Forum admins: please install the SMF Bookmark Mod (http://custom.simplemachines.org/mods/index.php?mod=350) to avoid these noisy "sub" posts.) Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: SgtSpike on June 20, 2012, 06:56:00 AM Also, want to run it on my public address? I am curious what other addresses are correlated with it... ;) If not, that's fine, I'll download it and attempt to compile myself. Here you go. Even translated the result to the usual format. I'd be curious to see if it worked. Let me know. Code: 18TKNbSLTrd3a2W8mtoH5uNzFhWRWNcuHU As an example, 11bUZzLby1Z1SAgS9UcHC3V2FXLTkJ5u73 is an address that is not in my address book. And interestingly, a google search turns up nothing but this post, indicating that once again, you're somehow getting addresses that don't exist in the blockchain yet. Hmmm... Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: SgtSpike on June 20, 2012, 08:11:59 AM It got nearly all of my often-used addresses, but had a LOT of false positives. Also: with the satoshi client there are quite a few addresses that actually are in your wallet but that you never "see". These are called "change" address, they're created automatically by the client when you send money. The way to check if they're yours is to run ./bitcoind validateaddress <address> and to check the "ismine" field in the JSON output. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: check_status on June 21, 2012, 08:48:45 AM Will it work with yED to graph the addresses visually?
For Ubuntu 11.04 64 bit To download yED: http://www.yworks.com/en/products_download.php?file=yEd-3.9.2.zip To run yED: Start yEd from commandline using, Code: java -Xmx512m -jar /path/to/yed.jar Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: check_status on June 21, 2012, 10:39:09 AM I wouldn't mind doing something like this:
http://1.bp.blogspot.com/-naxagbLRdSI/TihbllkcY3I/AAAAAAAATe8/1YuN2Gw46iQ/s1600/thief_public_key_2_steps_with_extras.png Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: check_status on June 21, 2012, 11:50:12 AM The vertices are hash160 or, in the case of the pic, base58 bitcoin addresses. How big a vertex appears, like in the pic, is probably determined by number of transactions. The edges are transactions, send and receive, colored by amounts transferred, small amounts grey, large amounts colors.
I was thrown out of geometry class in highschool, so I didn't take it. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: check_status on June 21, 2012, 01:21:08 PM OK, now that's clear: what you want is a full dump of the blockchain in graph form. Hmm...I guess it might be better to dump as text and then crawl for the specific info needed. I certainly can't load the whole blockchain into the grapher, I don't have a Cray nearby.So, as it is today, the code can't directly generate that, but it'd be a feature super simple to add to the 'allBalances' callback since the code basically walks that exact graph to generate the final balances. That being said, this is going to be a *huge* dump. To get an idea, here's the current blockchain stats gathered by blockparser: - 4.407 Million unique addresses received BTC - the blockchain contains 17.826 Million address spends In other words, not even taking into account the complex kind of transactions (send fromMany toMany) your graph would have on the order of 4.5M vertices and on the order of 18M edges ... Unless your graph software was optimized to handle very large graphs, you're not going to get much out of it. Another way would be to somehow figure out a way to restrict the graph dump to This sounds a bit more reasonable.a nearby neighborhood of a given address (for example N hops). Feel free to submit a patch. That means it's somewhat difficult or tedious to implement.Quote I was thrown out of geometry class in highschool, so I didn't take it. Since when is graph theory taught during geometry classes ? Neither had I computer science. Biology, tables yes, graphing no.Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: check_status on June 21, 2012, 01:37:37 PM We'll see how far past 'hello world!' I get first.
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: EnergyVampire on June 23, 2012, 08:21:35 PM Very nice!
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: muyuu on July 03, 2012, 12:29:25 AM Looks good. Cheers.
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Sukrim on July 05, 2012, 10:22:17 AM Also, want to run it on my public address? I am curious what other addresses are correlated with it... ;) If not, that's fine, I'll download it and attempt to compile myself. Here you go. Even translated the result to the usual format. I'd be curious to see if it worked. Let me know. Code: 18TKNbSLTrd3a2W8mtoH5uNzFhWRWNcuHU 1EScB6WzkeDVUZwLSsAHxUNuRr8jN38MLm Working from tat bitminter address I got Code: 1GVshgfm7dKWQaKnGfZpVfUgHMc667SSy2 I still need to change my own script to output hashes instead of addresses though, so I can compare with this (much faster but limited to linux) program. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: flatfly on July 06, 2012, 11:20:54 AM Is it possible to answer the following question with this tool?
Show all addresses currently containing coins from the (latest) Bitcoinica hack. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: copumpkin on July 06, 2012, 08:32:34 PM I haven't had a chance to look at your code yet, but do you use Tarjan's union-find to compute the "wallet-sets"? I've been meaning to write a tool that does that for a while.
Edit: oh, you're using boost's connected_components. I think they use union-find for incremental_components. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: copumpkin on July 06, 2012, 10:26:21 PM I haven't had a chance to look at your code yet, but do you use Tarjan's union-find to compute the "wallet-sets"? I've been meaning to write a tool that does that for a while. Edit: oh, you're using boost's connected_components. I think they use union-find for incremental_components. Yes I do because I build the entire graph in the first place. Not sure what algorithm they use in there. Also, given that I build the whole graph, not sure why using TUFA has any advantage over a dumb [D|B]FS mark and sweep ... Yeah, it doesn't. I wasn't planning on maintaining the entire graph, so can get away with it. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: copumpkin on July 06, 2012, 10:55:21 PM I haven't had a chance to look at your code yet, but do you use Tarjan's union-find to compute the "wallet-sets"? I've been meaning to write a tool that does that for a while. Edit: oh, you're using boost's connected_components. I think they use union-find for incremental_components. Yes I do because I build the entire graph in the first place. Not sure what algorithm they use in there. Also, given that I build the whole graph, not sure why using TUFA has any advantage over a dumb [D|B]FS mark and sweep ... Yeah, it doesn't. I wasn't planning on maintaining the entire graph, so can get away with it. Turns out that the graph will grow so fat at some point that it might very well be necessary to move to some sort of incremental data structure ... that being said, since there is no way to know that two disjoin subsets will or won't merge until the very last tx in the chain, I don't think it's possible to get away with _not_ building the whole graph in the first place. Well, I haven't tried implementing it yet, but my thinking was that the union-find structure represents just as much of the graph as we need. We basically have an equivalence of "has been used as an input in the same transaction as" and we're building equivalence classes. So basically, you make a pass over the data building up an associative map structure from each address to something your union-find algorithm knows how to deal with (I'm language-agnostic ;)). As you process transactions in the block building these things up, you make sure to call union on your union-find structure to join classes together if you see them as inputs to the same transaction. This is linear in the number of inputs, not quadratic, because the structure knows what's already in the class. Once you've made your pass over the blockchain, you're left with a big map containing all input addresses ever seen and your union-find structure. Now all you need to do is traverse your map of addresses and look up (using find to find a representative element of your equivalence class) which set each element is in, and group on that. The resulting sets should be your connected components. You could then make your map from addresses point to the set it belongs to for easy lookups, or do whatever. If I'm not spouting nonsense, that would allow us to run the set-building in O(n * alpha(n)) time, where n is the total number of inputs to all transactions. You'd use O(total unique input addresses) space for it, and it should have fairly low constant factors. I'll probably try implementing that in a bit, but I need to finish decoding the scripts first :( Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: copumpkin on July 06, 2012, 11:02:09 PM I'll probably try implementing that in a bit, but I need to finish decoding the scripts first :( I'll have to read your explanation about the algorithm you describe when my mind is a little more alert than right now (4am here). However, for the script decoding part, I would suggest looking at my code, in file util.cpp, routing solveOutputScript and showScript. They're very simple and pretty much self-contained (unlike the satoshi client code which has lots of layers of abstraction and makes things a tad hard to follow linearly). Cool, will do. I'm doing it in a completely different language (I dare not say which, lest people judge me! ???), but having some nicely written code to look at will be a great help, thanks. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: pancake on July 10, 2012, 10:04:48 AM can't compile on linux-x86-32:
$ make c++ -- cb/closure.cpp c++ -- cb/simpleStats.cpp c++ -- cb/taint.cpp c++ -- cb/transactions.cpp c++ -- cb/allBalances.cpp c++ -- callback.cpp c++ -- opcodes.cpp c++ -- parser.cpp cb/allBalances.cpp:1:0: sorry, unimplemented: 64-bit mode not compiled in opcodes.cpp:1:0: sorry, unimplemented: 64-bit mode not compiled in callback.cpp:1:0: sorry, unimplemented: 64-bit mode not compiled in cb/simpleStats.cpp:1:0: sorry, unimplemented: 64-bit mode not compiled in cb/transactions.cpp:1:0: sorry, unimplemented: 64-bit mode not compiled in parser.cpp:1:0: sorry, unimplemented: 64-bit mode not compiled in cb/closure.cpp:1:0: sorry, unimplemented: 64-bit mode not compiled in cb/taint.cpp:1:0: sorry, unimplemented: 64-bit mode not compiled in make: *** [.objs/transactions.o] Error 1 make: *** Waiting for unfinished jobs.... make: *** [.objs/opcodes.o] Error 1 make: *** [.objs/simpleStats.o] Error 1 make: *** [.objs/taint.o] Error 1 make: *** [.objs/allBalances.o] Error 1 make: *** [.objs/callback.o] Error 1 make: *** [.objs/closure.o] Error 1 make: *** [.objs/parser.o] Error 1 Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: pancake on July 10, 2012, 10:11:16 AM can't compile on linux-x86-32: The fix is to remove the -m64 flag from the Makefile. Should I do a pull request? Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: runlinux on July 10, 2012, 10:52:38 AM I really need to play with this. My tool pings blockexplorer for its data... lol
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: organofcorti on July 16, 2012, 11:57:37 PM Anyone managed to get this working on BSD or whatever it is that macs use?
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: gweedo on July 17, 2012, 12:11:35 AM Anyone managed to get this working on BSD or whatever it is that macs use? if your on lion you need to use Xcode and inside of xcode install the command line tools to get it workingand also use homebrew to install dependencies macports doesn't work as well for me at least. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: organofcorti on July 17, 2012, 12:17:37 AM Anyone managed to get this working on BSD or whatever it is that macs use? if your on lion you need to use Xcode and inside of xcode install the command line tools to get it workingand also use homebrew to install dependencies macports doesn't work as well for me at least. Great! Thanks for that. I'm on snow leopard and have installed compiling tools, so I'll give it a go. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: gweedo on July 17, 2012, 12:18:30 AM Anyone managed to get this working on BSD or whatever it is that macs use? if your on lion you need to use Xcode and inside of xcode install the command line tools to get it workingand also use homebrew to install dependencies macports doesn't work as well for me at least. Great! Thanks for that. I'm on snow leopard and have installed compiling tools, so I'll give it a go. actually I forgot you need to update your gcc to gcc 4.5 so do... Code: % brew tap homebrew/versions Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Jouke on July 18, 2012, 07:06:53 PM I am having trouble compiling.
http://pastebin.com/TrfSecqA Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Jouke on July 18, 2012, 07:18:16 PM I am having trouble compiling. http://pastebin.com/TrfSecqA From the README: Code: sudo apt-get install libssl-dev build-essential g++-4.4 libboost-all-dev libsparsehash-dev git-core perl Did you do that ? yes, http://pastebin.com/tQUq28Zk Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Jouke on July 18, 2012, 07:50:38 PM almost complete fresh install of ubuntu 64 bit.
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: runlinux on July 18, 2012, 09:35:15 PM installing 12.04 x64 in VMWare player.
I really want to play with this. I need to brush up on my C++ skills... haven't used it in about 10 years.. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: runlinux on July 18, 2012, 10:13:58 PM yay! compiled on my first try!
EDIT: snap... I need to DL the blockchain?! ungh... (What was I thinking? lol) copy and paste from host machine... Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: runlinux on July 18, 2012, 10:21:49 PM yup, i found that out...
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Jouke on July 19, 2012, 07:49:59 AM xubuntu 12.10 or something like that
Code: In file included from cb/allBalances.cpp:4:0: Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Jouke on July 19, 2012, 08:10:35 AM Ok, it does not compile on g++ 4.7, it does on g++ 4.5
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Jouke on July 19, 2012, 09:02:03 AM Ok, it does not compile on g++ 4.7, it does on g++ 4.5 OK. Let me install 4.7 and see what's what. Fixed and fix pushed to github Compiles nicely Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Jouke on July 19, 2012, 09:08:31 AM Tried it some other way. I have a specific problem. I had a blk0001.dat already when I first started the bitcoin-client. Instead of using that file it made a blk0002.dat and recently it made a blk0003.dat. The parser won't open the blockchain file: "Invalid argument: failed to open block chain file /home/jouke/.bitcoin/blk0001.dat" I guessed that 0002 was actually 0001 and tried to symlink 0002 to 0001 and 0003 to 0002 in a separate directory, but your parser still won't open the file. Any sugestions? Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: dooglus on July 19, 2012, 10:01:53 AM Tried it some other way. I have a specific problem. I had a blk0001.dat already when I first started the bitcoin-client. Instead of using that file it made a blk0002.dat and recently it made a blk0003.dat. The parser won't open the blockchain file: "Invalid argument: failed to open block chain file /home/jouke/.bitcoin/blk0001.dat" I guessed that 0002 was actually 0001 and tried to symlink 0002 to 0001 and 0003 to 0002 in a separate directory, but your parser still won't open the file. Any sugestions? Temporarily set HOME to wherever you want it to read the blockchain from: Quote $ HOME=/path/to/folder ./parser closure [...] fatal: No such file or directory: failed to open block chain file /path/to/folder/.bitcoin/blk0001.dat I doubt the parser made any blockchain files. It opens the files readonly, which won't create any files. About a week ago the blockchain file grew bigger than 2GB and so a 2nd file was created by bitcoin-qt: Quote $ ls -l blk* -rw------- 1 chris chris 2097307549 2012-07-12 20:07 blk0001.dat -rw------- 1 chris chris 104659251 2012-07-19 02:52 blk0002.dat -rw------- 1 chris chris 751022080 2012-07-19 02:52 blkindex.dat I don't know where the 0003 file came from though! Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Jouke on July 19, 2012, 10:09:06 AM Temporarily set HOME to wherever you want it to read the blockchain from: Already did that. That's where I symlinked the file to.Quote I doubt the parser made any blockchain files. It opens the files readonly, which won't create any files. I didn't say it did?Quote About a week ago the blockchain file grew bigger than 2GB and so a 2nd file was created by bitcoin-qt: Quote $ ls -l blk* -rw------- 1 chris chris 2097307549 2012-07-12 20:07 blk0001.dat -rw------- 1 chris chris 104659251 2012-07-19 02:52 blk0002.dat -rw------- 1 chris chris 751022080 2012-07-19 02:52 blkindex.dat I don't know where the 0003 file came from though! Because I already had a blk0001.dat, my client started with blk0002.dat. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: runlinux on July 23, 2012, 11:23:19 PM How hard would it be to add address balance to the closure method? (and a total balance at the bottom with address count)
I can't figure it out. My C++ is old and unused... :( - at least I can read the code! Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: runlinux on July 24, 2012, 07:45:03 PM Thanks for that! i'll modify the output so its just the address. from there a simple script should do what i need :)
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: runlinux on July 25, 2012, 02:29:54 AM OK, got a question now:
Why does this address: 12BdKh5HHUyGFduGm6y8nY4KS1LxzsduXo Link to this address: 12cbQLTFMXRnSzktFkuoG3eHoMeFtpTu3S Closure reports it has no outputs then returns that one. On a side note, script written and code modified to take a wallet address, and then just output the balance of the final wallet :) Code: runlinux@ubuntu:~/blockparser$ ./wallet.sh 1MrPonziaM4QT2S7SdPEKQH88BGa4LRHJU Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: dooglus on July 25, 2012, 04:02:51 AM Why does this address: 12BdKh5HHUyGFduGm6y8nY4KS1LxzsduXo Link to this address: 12cbQLTFMXRnSzktFkuoG3eHoMeFtpTu3S Closure reports it has no outputs then returns that one. I see the same. It tells me: Code: error: specified key was never used in a TX output but this transaction (http://blockchain.info/tx-index/13110367/a818dbb6d0d6df24abd890eea56cc1221ad46cdf7670d08481852ca8b5010b7f) has 12BdKh5HHUyGFduGm6y8nY4KS1LxzsduXo as an output, so the message is a little confusing. And then the address isn't in its own reported cluster - that's got to be a bug. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Sukrim on July 31, 2012, 12:56:05 PM What's your definition of "taint"?
If there is 6% taint from pirate in that huge address, does that mean 6% of these BTC were at one point in time owned by a pirate address or that pirate deposited 6% of these coins to that address? If it is the first definition, isn't there the possibility to have >100% total taint? E.g. pirate sends BTC to MtGox and then MtGox sends these to the huge address - now they are "tainted" by pirate AND "tainted" by MtGox... What I would still like to do is to enter an address, get the closure of that and then get the closures of every address where money was sent to or received from that cluster (ideally I could enter how many levels deep, with "0" being the current closure setting). Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: etotheipi on August 02, 2012, 12:23:44 PM I'm sure you know I've spent a lot of time doing blockchain scanning, and attempting to optimize it in various ways in Armory. In the end, the fastest I could achieve was about 20-25s on a full blk0001.dat (2GB) on Linux-64bit using memory-mapping. I couldn't achieve the same thing in Windows (mmap'ing seems to be garbage there). I just wanted to share my experience from playing with leveldb, how awesomely optimized it is.
First, I iterated through blk0001.dat and stored all transactions in a leveldb database using Key:TxHash-->Value:RawTx. I plugged 4 addresses into a set<BinaryData> object. I then iterated (out of order) through the entire database, pulling each transaction into memory from the DB, and then searching for TxOuts matching any of those 4 addresses. This took 8.0 seconds! (but was only the TxOuts). This included putting the TxOuts into a map<OutPoint,uint64_t> (OutPoint, nBTC), so that I could then scan the entire DB again and search for TxIns. At the end of both scans, I had the balance and unspent TxOut list from those four addresses. It took approximately 12-13 seconds! And this is with a non-SSD drive: it seems like it shouldn't even be possible to go that quickly... I had to quadruple-check everything to verify that I wasn't missing something or accidentally supplying the answer. Perhaps there was some OS-level caching of the DB files, though I would've expected such caching to improve my custom blockchain utilities performance. I don't intend to discourage or insult your efforts (in fact, I think you said you were getting similar speeds). Plus, I think that writing such a parser/scanner is both useful and educational. I just bring it up because I would guess that this is about as good as anyone can do with a non-SSD drive. And it's especially impressive considering I did no optimizations. I'm especially curious how leveldb works under the hood that it could be so efficient. Maybe there's something to learn from it. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: jgarzik on August 02, 2012, 02:06:36 PM I don't intend to discourage or insult your efforts (in fact, I think you said you were getting similar speeds). Plus, I think that writing such a parser/scanner is both useful and educational. I just bring it up because I would guess that this is about as good as anyone can do with a non-SSD drive. And it's especially impressive considering I did no optimizations. I'm especially curious how leveldb works under the hood that it could be so efficient. Maybe there's something to learn from it. LevelDB provides the backend to Google's services, so it is not surprising that it is so heavily optimized. One of the key optimizations is that writes go to a log, and then another thread comes along in the background and "optimizes" the log data back into the main database, using hueristics to hopefully place the data in a useful, locality-friendly location on disk. BitcoinJ author Mike Hearn has proposed replacing BDB with LevelDB in the Satoshi client, and we are looking seriously at that. Here is his pull request: https://github.com/bitcoin/bitcoin/pull/1619 Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: etotheipi on August 02, 2012, 02:33:52 PM Finally, it's a good idea to do this: in the end the blockchain will anyway grow so big that even the hashmaps won't fit in RAM, and my approach will start swapping like crazy. Pre-indexing in a disk-level DB hash table is the right thing to do. Well that's why I started looking into this. My original blockchain utilities design in Armory were more of an educational adventure to see how fast I could get the scanning from scratch (no external libraries), with lots of pointers, etc. It was educational, and I was happy with the results. But I had the same realization that even the index will eventually not fit in RAM, and thus there is no way to make the library scalable without using some kind of disk-based engine. To that end, I had no interest re-inventing disk-based DB operations, etc. When I heard the devs talking about LevelDB, I looked into and quickly realized why they chose it. It is very simple, portable... and apparently fast! However, gmaxwell just told me how to clear my disk cache and then retest. It was dramatically slower for sure. Probably by a factor of 5-10x. However, I never ran the same test using my other blockchain utilities, so I don't know the actual comparison. Given that modern OS'es are good at doing that caching, the LevelDB version is clearly a very good solution. I'm currently investigating how I can transition Armory to maintain it's own blockchain data now with LevelDB :) Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: jgarzik on August 02, 2012, 03:12:27 PM FWIW the set of unspent tx-out's should fit nicely in RAM, for some time to come.
I know that's not helpful to applications doing historical queries, but it may be helpful for other queries. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: vampire on August 02, 2012, 03:43:00 PM znort987:
Could you add to parser.c: #ifndef O_DIRECT #define O_DIRECT 0x10000000 #endif O_DIRECT isn't defined on OSX. Otherwise I was able to compile on my macbook with macports. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: thallium205 on August 27, 2012, 05:55:41 AM Hey all.
I developed an online tool that does the exact same kind of operations (grouping addresses together through connected components) in O(1) while giving you the ability to visualize an ownership network that is built on top of it. For that first address you calculated, you can check out the results by going here: http://blockviewer.com/#/owns/1CjLmm25ty4aSjxzG8Er4qSiQ63bqaQGA8 (http://blockviewer.com/#/owns/1CjLmm25ty4aSjxzG8Er4qSiQ63bqaQGA8) and then clicking on the "owner" node (they are sort of hard to click on... I'm working on that). That will pull up all the addresses associated with it, along with other information. If you think this is interesting, check out my announcement post here -> https://bitcointalk.org/index.php?topic=103609.0 (https://bitcointalk.org/index.php?topic=103609.0) I don't mean to thread jack you all, but I thought you all would find this interesting. If you would like to check more addresses, simply type it in in the search bar and select "Owner Address". Then, as usual, click the owner node and start traversing the block chain! Enjoy! Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Chimsley on September 11, 2012, 07:24:48 PM I installed this and started doing some of the examples in the readme file and everything is working until I ran allbalances.
I get this output. Quote xxxxx@xxxxxxxx:~/blockparser$ ./parser allbalances >allbalances.txt info: Starting command "allBalances" info: analyzing blockchain ... info: 0 blocks, 0.000 MegaMoves , 0.000 MegaAddrs , 0.00% info: 118089 blocks, 1.049 MegaMoves , 0.381 MegaAddrs , 59.56% info: 129462 blocks, 2.097 MegaMoves , 0.722 MegaAddrs , 65.29% info: 134099 blocks, 3.146 MegaMoves , 1.060 MegaAddrs , 67.63% info: 138049 blocks, 4.194 MegaMoves , 1.462 MegaAddrs , 69.62% info: 142392 blocks, 5.243 MegaMoves , 1.844 MegaAddrs , 71.81% info: 146931 blocks, 6.291 MegaMoves , 2.164 MegaAddrs , 74.10% info: 152726 blocks, 7.340 MegaMoves , 2.481 MegaAddrs , 77.02% info: 158931 blocks, 8.389 MegaMoves , 2.733 MegaAddrs , 80.15% info: 164693 blocks, 9.437 MegaMoves , 2.953 MegaAddrs , 83.06% info: 169636 blocks, 10.486 MegaMoves , 3.182 MegaAddrs , 85.55% info: 174740 blocks, 11.534 MegaMoves , 3.410 MegaAddrs , 88.13% info: 178761 blocks, 12.583 MegaMoves , 3.635 MegaAddrs , 90.15% info: 180687 blocks, 13.631 MegaMoves , 3.811 MegaAddrs , 91.13% info: 181868 blocks, 14.680 MegaMoves , 3.968 MegaAddrs , 91.72% info: 183095 blocks, 15.729 MegaMoves , 4.125 MegaAddrs , 92.34% info: 184314 blocks, 16.777 MegaMoves , 4.273 MegaAddrs , 92.96% info: 185046 blocks, 17.826 MegaMoves , 4.407 MegaAddrs , 93.32% info: 186241 blocks, 18.874 MegaMoves , 4.558 MegaAddrs , 93.93% info: 187702 blocks, 19.923 MegaMoves , 4.754 MegaAddrs , 94.66% info: 189164 blocks, 20.972 MegaMoves , 4.931 MegaAddrs , 95.40% info: 190250 blocks, 22.020 MegaMoves , 5.094 MegaAddrs , 95.95% info: 191531 blocks, 23.069 MegaMoves , 5.262 MegaAddrs , 96.60% info: 192657 blocks, 24.117 MegaMoves , 5.449 MegaAddrs , 97.16% info: 193834 blocks, 25.166 MegaMoves , 5.630 MegaAddrs , 97.76% info: 194764 blocks, 26.214 MegaMoves , 5.815 MegaAddrs , 98.23% info: 195960 blocks, 27.263 MegaMoves , 6.006 MegaAddrs , 98.83% info: 197082 blocks, 28.312 MegaMoves , 6.172 MegaAddrs , 99.39% fatal: o2i_ECPublicKey failed Aborted (core dumped) This is a fresh Ubuntu 12.04 install with 8Gb of ram. Any thoughts on why I got this message? The allbalances.txt file is created but is empty so I assume the file is not written until the command is completed. ***Disclaimer*** I am just barely above noob level on linux. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: EskimoBob on September 11, 2012, 07:42:19 PM I am having trouble compiling. http://pastebin.com/TrfSecqA From the README: Code: sudo apt-get install libssl-dev build-essential g++-4.4 libboost-all-dev libsparsehash-dev git-core perl Did you do that ? I am going to attempt building it in Gentoo. (most of this stuff is probably installed) Code: emerge -atvu1 dev-libs/openssl dev-libs/boost dev-cpp/sparsehash dev-vcs/git dev-lang/perl Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Chimsley on September 12, 2012, 05:13:02 PM Cool thanks znort987 for the future fix. I am just tinkering but I have seen other people on the forums doing serious analysis with this tool and I applaud you for releasing this to everyone. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: JompinDox on September 20, 2012, 09:14:38 AM sorry for the newb question:
is it possible to find out how many addresses containing over 500 btc have NOT been used since January 2011, for instance? (not used = not used for sending coins) Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: flatfly on September 21, 2012, 01:12:30 PM Great, thanks!
Btw, are there any pre-built windows binaries available? Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: dooglus on September 22, 2012, 12:02:03 AM Great, thanks! Btw, are there any pre-built windows binaries available? Nope. And there's unlikely ever going to be any. haven't touched windows in 10 years and don't plan to do so ever again. +1 Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: organofcorti on September 22, 2012, 12:44:04 AM Great, thanks! Btw, are there any pre-built windows binaries available? Nope. And there's unlikely ever going to be any. haven't touched windows in 10 years and don't plan to do so ever again. What, you're saying that we just have to work when we're at work? It's people like you that make it hard for me to be a slacker. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Buffer Overflow on September 22, 2012, 04:30:00 PM haven't touched windows in 10 years and don't plan to do so ever again. 6 years for me; and counting. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: xchrix on September 27, 2012, 12:33:01 PM great work znort!
is it also possible to use this parser for a litecoin blockchain? i think so... unfortunately i only have debian 32bit so i have to organize a 64bit server for testing this. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: xchrix on September 27, 2012, 02:57:42 PM ok i see. i am good at php coding but c++ is not my best friend :D i dont think that i am able to provide some patches.
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: SgtSpike on September 27, 2012, 09:17:14 PM It'd be great if someone could host this on the web, maybe charging a small fee for each lookup or something.
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: SgtSpike on September 27, 2012, 10:39:53 PM It'd be great if someone could host this on the web, maybe charging a small fee for each lookup or something. Mmmh, not really : this is designed to run a full linear sweep of the entire blockchain for every query ... that's truly heavy duty (both raw I/O and CPU). The moment 5 users hit the web site at the same time, the server will basically die. The way blockchain.info does it is the right approach for what you describe: they dump the blockchain into a DB with a crapload of pre-computed indices and update said DB incrementally to keep the queryable version up to date. Regardless, jobs could be queued, and the resource-intensiveness is why I suggested a small fee for each lookup. There are those of us who do not know linux well enough to run something like what you have created here, but would still like it to be accessible. ;) Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: dirtycat on October 06, 2012, 05:08:17 AM I installed this and started doing some of the examples in the readme file and everything is working until I ran allbalances. I get this output. Quote xxxxx@xxxxxxxx:~/blockparser$ ./parser allbalances >allbalances.txt info: Starting command "allBalances" info: analyzing blockchain ... info: 0 blocks, 0.000 MegaMoves , 0.000 MegaAddrs , 0.00% info: 118089 blocks, 1.049 MegaMoves , 0.381 MegaAddrs , 59.56% info: 129462 blocks, 2.097 MegaMoves , 0.722 MegaAddrs , 65.29% info: 134099 blocks, 3.146 MegaMoves , 1.060 MegaAddrs , 67.63% info: 138049 blocks, 4.194 MegaMoves , 1.462 MegaAddrs , 69.62% info: 142392 blocks, 5.243 MegaMoves , 1.844 MegaAddrs , 71.81% info: 146931 blocks, 6.291 MegaMoves , 2.164 MegaAddrs , 74.10% info: 152726 blocks, 7.340 MegaMoves , 2.481 MegaAddrs , 77.02% info: 158931 blocks, 8.389 MegaMoves , 2.733 MegaAddrs , 80.15% info: 164693 blocks, 9.437 MegaMoves , 2.953 MegaAddrs , 83.06% info: 169636 blocks, 10.486 MegaMoves , 3.182 MegaAddrs , 85.55% info: 174740 blocks, 11.534 MegaMoves , 3.410 MegaAddrs , 88.13% info: 178761 blocks, 12.583 MegaMoves , 3.635 MegaAddrs , 90.15% info: 180687 blocks, 13.631 MegaMoves , 3.811 MegaAddrs , 91.13% info: 181868 blocks, 14.680 MegaMoves , 3.968 MegaAddrs , 91.72% info: 183095 blocks, 15.729 MegaMoves , 4.125 MegaAddrs , 92.34% info: 184314 blocks, 16.777 MegaMoves , 4.273 MegaAddrs , 92.96% info: 185046 blocks, 17.826 MegaMoves , 4.407 MegaAddrs , 93.32% info: 186241 blocks, 18.874 MegaMoves , 4.558 MegaAddrs , 93.93% info: 187702 blocks, 19.923 MegaMoves , 4.754 MegaAddrs , 94.66% info: 189164 blocks, 20.972 MegaMoves , 4.931 MegaAddrs , 95.40% info: 190250 blocks, 22.020 MegaMoves , 5.094 MegaAddrs , 95.95% info: 191531 blocks, 23.069 MegaMoves , 5.262 MegaAddrs , 96.60% info: 192657 blocks, 24.117 MegaMoves , 5.449 MegaAddrs , 97.16% info: 193834 blocks, 25.166 MegaMoves , 5.630 MegaAddrs , 97.76% info: 194764 blocks, 26.214 MegaMoves , 5.815 MegaAddrs , 98.23% info: 195960 blocks, 27.263 MegaMoves , 6.006 MegaAddrs , 98.83% info: 197082 blocks, 28.312 MegaMoves , 6.172 MegaAddrs , 99.39% fatal: o2i_ECPublicKey failed Aborted (core dumped) This is a fresh Ubuntu 12.04 install with 8Gb of ram. Any thoughts on why I got this message? The allbalances.txt file is created but is empty so I assume the file is not written until the command is completed. ***Disclaimer*** I am just barely above noob level on linux. Yeah, I hit this too a couple of days back. Turns out someone managed to include a few transactions in the blockchain that send funds to invalid compressed keys, something my code assumed was not possible. I have a fix for it, I'll push it to the git repo sometimes today. Fix pushed to github. Im still getting this when I run allabalances.. :( Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: dirtycat on October 06, 2012, 10:40:11 AM now using a different machine.. I get
Code: gato@50cal:~/blockparser$ ./parser all everything is there.. soooooo.. I have no idea. not my day lol Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: dirtycat on October 06, 2012, 11:52:41 AM figured it out. had my home directory encrypted.. disabled encrypt and all is well.. fun adventure.. btw great tool!
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: runeks on October 07, 2012, 08:35:29 PM This looks cool. Do you have any plans to add a feature that allows the program to output the total balance for a specific bunch of addresses? Like:
./parser showBalance < addresses.txt or alternatively ./parser showBalance -i addresses.txt Where addresses.txt would contain a list of addresses (hash160/base58). Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: runeks on October 07, 2012, 10:10:06 PM Cool. Just tried it. blockparser reports the balance of the address 991fd9bf50ad8d6a551efceb5c228d21bcc0fe92 to be 200.777, but blockchain.info says it's 351.06501343 BTC (https://blockchain.info/address/991fd9bf50ad8d6a551efceb5c228d21bcc0fe92). Who's right?
bitcoin-qt says my block chain is up to date (last block was found 8 minutes ago). Armory's block chain tool reports the same balance as blockparser (200.777) so it seems that this is valid. But why does this disagree with blockchain.info? Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: runeks on October 08, 2012, 01:16:38 AM Great! Is it possible to read addresses from a file that just contains concatenated hash160 addresses as raw bytes? Ie. 20 bytes followed by 20 bytes etc. I use this format to save space when storing a large number of addresses. I tried modifying the code to do this, but it seems it would require substituting fgets with fread, and things started looking messy so I gave up.
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: runeks on October 08, 2012, 08:55:17 AM The reason I ask is because I got curious over whether people have used insecure passwords to create brain wallets. For this purpose I've created 1.5 GB worth of addresses in that 20 byte raw format, out of a word list made by John the Ripper. So the reason is that even though there are only ~300 megs of different addresses in the block chain, there are (2^160)*20 bytes of possible addresses using the raw 20 byte format. Having the addresses lined up on 20 byte-boundaries also increases performance, I imagine, since I don't have to do any string parsing on the data.
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Xenland on October 08, 2012, 10:01:02 PM This might help my Pseudo Bitcoin Client Documentation project :D I'll be looking into this code shortly :D
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: runeks on October 09, 2012, 10:40:32 AM The reason I ask is because I got curious over whether people have used insecure passwords to create brain wallets. For this purpose I've created 1.5 GB worth of addresses in that 20 byte raw format, out of a word list made by John the Ripper. So, that same list in ASCII will be 3GB ... BFD, if you'll pardon my french. I've just ordered more RAM, to bring my machine up to 6 GB though. Although it hurt a bit to see that the same amount of DDR3 RAM was half the price of the DDR2 RAM my machine supports So the reason is that even though there are only ~300 megs of different addresses in the block chain, there are (2^160)*20 bytes of possible addresses using the raw 20 byte format. Having the addresses lined up on 20 byte-boundaries also increases performance, I imagine, since I don't have to do any string parsing on the data. IMO, the time it'll take to parse the ASCII will be dwarfed bythe time taken to chew through the blockchain while hammering in a completely cache-incoherent manner at that 1.5GB hashtable of restricts you just forced the parser to load and work with. Also ... I'm hoping you're planning to Do The Right Thing (tm) with Of course. I'm not a thief. Just curious.that attempt of yours. Nuff said. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Buffer Overflow on November 29, 2012, 11:36:29 PM Code: info: Starting command "simpleStats" I don't have the blocks/ directory? I only have: blk0001.dat blkindex.dat db.log peers.dat blk0002.dat database/ debug.log wallet.dat Using Bitcoin 0.7.1 & latest version of blockparser. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Buffer Overflow on November 30, 2012, 09:40:38 AM Ok, thank you very much.
How do I change over to the new style blockchain files? I'm using 0.7.1. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Buffer Overflow on November 30, 2012, 09:45:47 AM Oh dear more problems. Just cloned latest commit and tested with ./parser simpleStats and this appears:
warning: at depth 20221 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da warning: at depth 12432 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da warning: at depth 4315 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da warning: at depth 14255 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da warning: at depth 2975 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da warning: at depth 13760 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da warning: at depth 15626 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da warning: at depth 15844 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da warning: at depth 629 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da warning: at depth 3965 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da warning: at depth 3249 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da warning: at depth 13269 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da warning: at depth 8480 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da warning: at depth 16582 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da warning: at depth 18383 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da warning: at depth 6801 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da warning: at depth 14939 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da I have to exit with ctrl-c. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Buffer Overflow on November 30, 2012, 09:46:41 AM Ok, thank you very much. How do I change over to the new style blockchain files? I'm using 0.7.1. Code: git clone git://github.com/bitcoin/bitcoin.git bitcoin-head Be aware this is going to "upgrade" your datafiles and that this "upgrade" is likely not reversible. You might want to wait until this is officially supported by the released client. Oh, I see, it's in Beta state. Think I leave it then. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Pieter Wuille on November 30, 2012, 09:23:06 PM Oh, I see, it's in Beta state. Think I leave it then. Well, all bitcoin releases to date have been beta. This is about the code that hasn't even been released, but will eventually become 0.8. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Buffer Overflow on November 30, 2012, 09:32:58 PM Oh, I see, it's in Beta state. Think I leave it then. Well, all bitcoin releases to date have been beta. This is about the code that hasn't even been released, but will eventually become 0.8. That's a good point. I wonder when Bitcoin will come out of beta? 1.0 I guess. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Jouke on January 23, 2013, 12:17:04 PM Oh dear more problems. Just cloned latest commit and tested with ./parser simpleStats and this appears: warning: at depth 20221 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da warning: at depth 12432 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da warning: at depth 4315 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da warning: at depth 14255 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da warning: at depth 2975 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da warning: at depth 13760 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da warning: at depth 15626 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da warning: at depth 15844 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da warning: at depth 629 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da warning: at depth 3965 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da warning: at depth 3249 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da warning: at depth 13269 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da warning: at depth 8480 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da warning: at depth 16582 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da warning: at depth 18383 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da warning: at depth 6801 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da warning: at depth 14939 in chain, failed to locate parent block 000000000000040167c69f7a835d124214316b72d9a9fc2ec2413a238b6ab6da I have to exit with ctrl-c. I have the same problem with 7.2 Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Jouke on January 23, 2013, 12:37:14 PM nm. I am running an extra daemon now. However: info: Address cluster for address 16EC7Nx6THDZpGMCavLubbtt2UPXimyUx7: warning: specified key was never used to spend coins But it is used in this transaction: https://blockchain.info/tx/49688f5183b3898d19de3904a09d75d1af52a14edd5f951f371f6823eb22b1f6 Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Jouke on January 24, 2013, 09:51:08 PM You need to make sure that your local blockchain is up to date. Hmm, strange, bitcoind getinfo told me it was at up2date. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: SilverLeaf on February 19, 2013, 04:10:18 PM Same "Failed to locate parent block..." problem.
Opened a GitHub issue here: https://github.com/znort987/blockparser/issues/1 :) . Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: SilverLeaf on February 19, 2013, 05:05:37 PM Your blockchain files are likely corrupted. Even if they aren't, I do not plan to support old style bitcoind blockchain in the future. If you want to use blockparser, I would strongly advise moving to a recent version of the bitcoind client with the levelDB backend (the one that has a directory called "chainstate" in ~/.bitcoin) Thank you, I am regenerating the blockchain using bootstrap.dat and trying again. blockparser doesn't support levelDB by now, right? Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: SilverLeaf on February 19, 2013, 05:58:01 PM Thank you, now it's much more clear.
EDIT: I can confirm that it works with the git version of bitcoind. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: naima53 on February 28, 2013, 06:54:58 AM Is compatible with version 0.8 bitcoin-qt ?
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Dabs on March 04, 2013, 12:49:45 PM Is there a pre-compiled windows version? binary? (which you can of course sign or produce a SHA so us newbies can be sure we have a clean binary.)
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Buffer Overflow on March 04, 2013, 07:44:27 PM Blockparser does not run on Windows and other legacy technologies from the 20th Century LOL :D I couldn't agree more. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: ShadowOfHarbringer on March 06, 2013, 10:47:51 AM Is there a pre-compiled windows version? binary? (which you can of course sign or produce a SHA so us newbies can be sure we have a clean binary.) Blockparser does not run on Windows and other legacy technologies from the 20th Century Hahahahaha. Splendid statement, milord. http://3.bp.blogspot.com/-vkrqp59BtHc/UNL67BngAPI/AAAAAAAAAp8/t97m5iJCOXI/s400/Milord.jpg Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: codro on March 06, 2013, 07:06:47 PM It would be really interesting to see a visualization (graph/chart/printout :) ) of all the bitcoins being traded in the past couple of years, how many of them (atomically) haven't exchanged hands at all in the past 2 years? Since all coins should be able to be traced back to the moment they were mined, it should be fairly trivial to see how many times and the dates when they were transacted. No?
I think this would be valuable for finding out how much is being hoarded, and how much is in circulation. I haven't fully looked into this tool yet to see if it can do this, but any pointers would be appreciated. :) Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Sukrim on March 07, 2013, 02:26:47 AM Well, in theory it should be possible to let this run on Windows as well... in practice however I've only managed to compile boost and am still fighting with sparsehash as well as openssl and it's dependencies (openssl wants perl for example).
Sparsehash also seems to really not like to be compiled with MingW... Let's see if I give up and just install a VM on my big machine, because my small Linux Atom nettop probably will choke to death on the hash tables. :( Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Dabs on March 08, 2013, 02:26:19 AM Maybe there is a way to implement this differently so it doesn't use mmap files? Or windoze implementation could be improved? Or I'm asking too much?
If I had the ram, maybe blockparser could do it all in ram? Or I can try using a ram drive? Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Sukrim on March 08, 2013, 12:21:44 PM To deal with crappy retarded GPU drivers and UI in Linux? ::) No thanks, sir...
Anyways, in theory it should(TM) be possible already to compile + run this parser for Windows. It's not znort987's resposibility to actually provide the means to do so, especially since he doesn't want to do it. It would be nice of him if somebody manages to actually come up with a solution how to do it, to integrate some minor patches (as far as I see it from the code, the blockchain location is hardcoded for example - having a setting for Windows or even Linux users there might be nice) and the actual way to do it, but even that is not his job at all (and you simply could just fork his project after all). The code is public domain and uses OpenSource libraries. Either invest the time to compile that stuff yourself or pay somebody to do it for you. If mmap implementations on Windows are really that slow, you can also improve them, since this project is built on OpenSource software. Again, either by yourself or somebody you pay to do that. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Dabs on March 09, 2013, 03:02:31 AM I guess I am asking too much, in the same way that a request to get rid of crappy OSes is also asking too much. Depends on who you talk to, if it's one individual, it is reasonable to ask them to switch OS, if you are talking to a corporation with hundreds of deployed crappy OS machines, that might be asking too much.
However, I do have the option to run Linux on my personal machine, so unless Windows gets improved (either the mmap implementation or the parser itself), then that's the route to take, for now. My guess is that, since this is a specialized tool, you really should use it on dedicated machines that are not being used for other crappy OS stuff. Unfortunately for me, that's not a practical thing to do. I could dual-boot to experiment with it, but eventually I'd have to go back to my crappy OS. Thus my question. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Buffer Overflow on March 09, 2013, 09:07:03 AM To deal with crappy retarded GPU drivers and UI in Linux? ::) No thanks, sir... The manufacturers make the drivers, blame them. Linux is more than capable of running most hardware.Who wants to dance and jump about to Bill Gates tune? Playing the game by his rules, his way, at his (expensive) price. ::) No thanks, sir... Unfortunately for me, that's not a practical thing to do. I could dual-boot to experiment with it, but eventually I'd have to go back to my crappy OS. Thus my question. You could install a virtualization software package, like Virtual Box and install Linux inside that. That's assuming your hardware can take the extra load of course.How come (and this isn't directed at anyone in particular) when a Linux user needs a program that only runs on Windows, he/she gets their head down, knuckles down and finds way around the problem. But on the other hand when a Windows user needs a program that only runs on Linux, there's a huge uproar, and expects everyone else to adjust to them? It's as if they assume Windows is the default system and everyone must whistle their tune. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Sukrim on March 09, 2013, 10:41:42 AM To deal with crappy retarded GPU drivers and UI in Linux? ::) No thanks, sir... The manufacturers make the drivers, blame them.Great to have done this, I guess that changes now everything... I am also a Linux user and I never got my head down, knuckled down and tried to find a way around the problem - I simply installed a Windows VM or native machine and ran the program in it's intended OS. Please refrain from generalizing, I bet if you wrote this statement in a Linux heavy forum replacing Windows with Ubuntu and Linux with Gentoo it would also be in line there... especially taking into consideration that this software is said to only support 64 bit Ubuntu. Anyways, there IS already the possibility to compile and run this software under Windows and as soon as I manage to compile it, I'll maybe even share the binary (for a hefty fee perhaps to cover my time). If this doesn't work, well - having a result in 3 minutes instead of one vs. having no result at all would still be an improvement, right? Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: gmaxwell on March 09, 2013, 01:13:25 PM After interacting with znort987 on IRC I'd advise being very careful in using this software double check any results that you get from it.
I pointed out that he was making some (theoretically and practically) minorly incorrect claims about the analysis this tool produced and his response was confused first failing on technically grounds and then after the technical misunderstand was made clear he became uncivil— calling me and other folks names and such— quite bizarre. While its obviously useful, it's written by someone who apparently hasn't done your thinking for you, so you need to think for yourself. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: phantomcircuit on March 09, 2013, 01:16:20 PM After interacting with znort987 on IRC I'd advise being very careful in using this software double check any results that you get from it. I pointed out that he was making some (theoretically and practically) minorly incorrect claims about the analysis this tool produced and his response was confused first failing on technically grounds and then after the technical misunderstand was made clear he became uncivil— calling me and other folks names and such— quite bizarre. While its obviously useful, it's written by someone who apparently hasn't done your thinking for you, so you need to think for yourself. I can confirm this. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: ShadowOfHarbringer on March 09, 2013, 01:33:17 PM After interacting with znort987 on IRC I'd advise being very careful in using this software double check any results that you get from it. I pointed out that he was making some (theoretically and practically) minorly incorrect claims about the analysis this tool produced and his response was confused first failing on technically grounds and then after the technical misunderstand was made clear he became uncivil— calling me and other folks names and such— quite bizarre. While its obviously useful, it's written by someone who apparently hasn't done your thinking for you, so you need to think for yourself. This forum should have a red or orange warning box on the top of each topic (placeable by mods, admins & devs only) to warn people of scams and events such as this. What do you think ? Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: gmaxwell on March 09, 2013, 01:37:14 PM Ah, and since he changed this back— I suppose I should point this out here:
Quote * Compute the list of addresses that provably belong to the same person This is both theoretically and practically incorrect. The tool doesn't do this for a couple of reasons.The most common violation of this comes from shared wallets (like mtgox and instawallet). Signatures on transactions do not not indicate "belonging"— mtgox signs on hundreds of thousands of coin that do not belong to them. In that case it only indicates control, and only at a point in time as people do transfer ownership of private keys to these services. (E.g. If give my old privkeys with some coins on them to mtgox, they'll end up as inputs on other people's txn— it's true that mtgox controls them from that point on, but not in the past). Less commonly, but also more fundamentally, is this assumption is also invalidated by group transactions. E.g. it's quite possible for two people to each provide an input to a transaction. This is useful for making atomic shared payments to bounties and the like, but also for doing trustless mixing. I believe the new "shared send" functionality on b.i (which replaced their old mixing service) works this way, and so I wouldn't be surprised if a non-trivial number of transactions failed on this criteria. Obviously, the importance of these details depend on how you are using the data. Common inputs are suggestive of common control at the time the transaction was authored. They are suggestive (though less so) of past control. They are far less suggestive of actual belonging because the belonging assumption is commonly violated by shared wallets and may be untrue for a significant fraction of coins in existence. They certainly do not constitute a 'proof' in any sense that we'd normally use the word in this community. If you're interested in making soft guesses about some transactions, then these might be fine assumptions. If you're looking to prosecute someone for a crime— certainly not. If you want to make statistical comments about populations of transactions it's highly questionable because no one knows how biased a statistic the result is, since mixed and shared transactions appear to be indistinguishable from other ones. Cheers. This forum should have a red or orange warning box on the top of each topic (placeable by mods, admins & devs only) to warn people of scams and events such as this. I don't think it rises to that level of concern. I apologize if I came off sounding like it was that urgent. It's not. In this case I wanted to throw out a caveat emptor because I thought it was odd... but really caveat emptor is generally good advice. If it were something like I thought that the software would cause you to lose coin or something I would have made it a _lot_ more obvious.Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Buffer Overflow on March 09, 2013, 02:51:20 PM Ok, let's blame AMDand nVidia! Evil you! You really should support Linux better! ???Great to have done this, I guess that changes now everything... Correct me if I'm wrong, but don't these two quotes listed below contradict each other? I am also a Linux user and I never got my head down, knuckled down and tried to find a way around the problem - I simply installed a Windows VM or native machine and ran the program in it's intended OS. Anyways, there IS already the possibility to compile and run this software under Windows and as soon as I manage to compile it, I'll maybe even share the binary (for a hefty fee perhaps to cover my time). Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Sukrim on March 09, 2013, 08:29:03 PM Nope, as I would do the feats that were attributed to a brave Linux hacker as a Windows user, not as a Linux user. Anyways, this is actually besides the point.
The issue that transactions that spend from two addresses do not 100% mean that these addresses are controlled by the same person or entity is something more interesting than whether I manage to compile OpenSSL etc. under Windows. Especially that constant cries to import private keys here and there might make it tough in the future to even analyze transactions at all. I also can't think of any 100% sure ways to overcome this problem of people sharing inputs to fool software like this - maybe creating address pools and then looking for suspicious overlaps might be a solution. E.g. 500 transactions each in 2 sets of coupled addresses, then suddenly for a single transaction they bond (even though the "simulated wallet" containing both sets would contain better fitting outputs?) and then they start transacting on their own again never mixing coins... Also transaction timestamps could give indications about users (and adding a second set that has so far completely different transaction patterns might indicate someone trying to mix coins). Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Sukrim on March 10, 2013, 09:12:44 AM Oh, and for some measurements:
With larger hash maps: SimpleStats in ~850s With smaller hash maps (WANT_DENSE commented out): SimpleStats in ~350s Still a lot of time, but not too shabby for a little atom machine. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Sukrim on March 10, 2013, 12:15:04 PM Hm, could be the case - I didn't reboot between runs + recompilation.
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Sukrim on March 10, 2013, 02:09:32 PM Oh btw. it works/compiles like a charm on Arch Linux, so no worries about needing Ubuntu (you don't even specify which release anyways...).
Now back to solving my Hash160 issues... :-/ Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: mrb on March 11, 2013, 02:50:18 AM No need to reboot. Just flush the disk cache. Something along the lines of: Code: dd if=/dev/zero bs=1024k count=4096 of=/dev/null This has pretty much no effect on the buffercache (kernel knows /dev/zero does not need caching, and dd will use at most 1MB of resident memory). What will drop it is (as root): Code: $ echo 3 >/proc/sys/vm/drop_caches Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Dabs on March 11, 2013, 04:17:48 AM Windows is not the default OS. It's just, unfortunately the market leader, or the OS with the largest share of users. I'm sure that if I can't dual-boot, I can probably do a Live CD.
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Sukrim on March 11, 2013, 10:02:30 AM Windows is not the default OS. It's just, unfortunately the market leader, or the OS with the largest share of users. I'm sure that if I can't dual-boot, I can probably do a Live CD. Good luck with catching up on the blockchain then... depending on your needs, you're faster to just run the stuff you need on armory-python.Anyways, doing a closure run brings my poor little nettop to its knees (even with the "smaller" hash maps) - at least after a few hours it didn't show any kind of progress, so I killed the program. 1GB RAM is not cutting it at all apparently. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: wiggi on March 11, 2013, 06:23:51 PM Well, in theory it should be possible to let this run on Windows as well... in practice however I've only managed to compile boost and am still fighting with sparsehash as well as openssl and it's dependencies (openssl wants perl for example). Sparsehash also seems to really not like to be compiled with MingW... Let's see if I give up and just install a VM on my big machine, because my small Linux Atom nettop probably will choke to death on the hash tables. :( Another issue with VM's is that: a) you're going to have to copy the blockchain over to the VM b) blockparser uses mmap'd files extensively. windoze implementation of those is hugely inefficient c) blockparser is really I/O intensive (every run gobbles in the full blockchain). This is highly inefficient on VMs A little test, how long does it take to do "./parser allBalances >allBalances.txt" in VirtualBox with different amounts of ram assigned to the VM: (host is win7 64bit, guest Linux Mint 64bit, 8GB physical ram) 1GB eta 1200000s :o 3GB eta 5000s 6GB eta 200s, but actually >1500s, slows down alot @50-70% 7GB 850s As long as the VM can consume more and more ram, blockparser is *very* fast, but I'm not sure how much is really enough to let the OS buffer all I/O in ram, 32GB perhaps. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Dabs on March 12, 2013, 01:18:50 AM I'm tempted to buy a machine with 32 Gigs of RAM. ... ... Maybe those so called "ram drives" (hardware implemented) might be useful.
@Sukrim, I can probably sync the blockchain with another computer that already has it within the network, or just copy the files over. That will save download time. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: mrb on March 12, 2013, 01:23:37 AM I'm tempted to buy a machine with 32 Gigs of RAM. ... ... Maybe those so called "ram drives" (hardware implemented) might be useful. Do it, 32GB only costs $160, assuming you already have a machine with a motherboard supporting 4 x 8GB DDR3. But don't buy hardware ram drives. They are all horribly overpriced, as in 10x more expensive per GB, or more. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: Dabs on March 12, 2013, 01:51:13 AM The hardware ram drives I know use old DDR2 memory. Maybe there are new ones that are horribly overpriced. Or I'll look around to see if someone has made one. The one I know used to use a SATA cable to emulate a hard drive. Real ram inside the motherboard is obviously going to work a lot faster.
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: 2112 on March 12, 2013, 02:10:26 AM But don't buy hardware ram drives. They are all horribly overpriced, as in 10x more expensive per GB, or more. Please pardon me for jumping in, but please post the links to those overpriced hardware RAM drives. My friends in IT operations are on the market and have hard time sourcing devices that don't wear out are reasonably non-volatile and can survice reboot/reconfig/reload of the OS.Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: mrb on March 12, 2013, 09:29:11 AM But don't buy hardware ram drives. They are all horribly overpriced, as in 10x more expensive per GB, or more. Please pardon me for jumping in, but please post the links to those overpriced hardware RAM drives. My friends in IT operations are on the market and have hard time sourcing devices that don't wear out are reasonably non-volatile and can survice reboot/reconfig/reload of the OS.http://www.ddrdrive.com - it was designed for ZFS ZIL, and the RAM is backed by SLC NAND flash. It was priced $2000 when it launched ~3 years ago. The price has probably gone down a lot, but it is still way more than 10x the price per GB than system's RAM. Part of the reason prices are so high is because there are so few vendors, so there is practically no competition. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: 2112 on March 12, 2013, 10:25:34 AM http://www.ddrdrive.com - it was designed for ZFS ZIL, and the RAM is backed by SLC NAND flash. It was priced $2000 when it launched ~3 years ago. The price has probably gone down a lot, but it is still way more than 10x the price per GB than system's RAM. Thanks. Unfortunately this is still a flash drive with RAM cache and still susceptible to write amplification abuse. Thus far we haven't found anything better than discontinued ACARD ANS-9010 (64GB with battery backup) and some other custom-made solutions.Part of the reason prices are so high is because there are so few vendors, so there is practically no competition. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: onelineproof on March 18, 2013, 08:01:28 AM This is nice. One feature I would like to see is the creation of a table that stores public key to balance pairs, and keeps updating from time to time. This way you can quickly look up the balance of an address. If I have time, I may use this code to do this, and I'll add it to my wallet parser: https://github.com/piratelinux/cwallet
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: phelix on March 19, 2013, 08:25:53 AM Would it be possible to dig out what percentage of coins have ever been moved for any 2016 block period?
I would be interested in how many of the early coins have never been moved. edit: ok, just found your post: https://bitcointalk.org/index.php?topic=98641.0 [2 Million unspent...] Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: phathash on March 24, 2013, 10:23:22 AM I know this is developed under Ubuntu, but it would be nice to have working on RH type distros.
I have openssl with ecdsa working, but blockparser won't compile - Quote # make c++ -- cb/dumpTX.cpp cb/dumpTX.cpp: In member function 'virtual int DumpTX::init(int, const char**)': cb/dumpTX.cpp:80: error: expected initializer before ':' token cb/dumpTX.cpp:84: error: expected primary-expression before 'return' cb/dumpTX.cpp:84: error: expected ';' before 'return' cb/dumpTX.cpp:84: error: expected primary-expression before 'return' cb/dumpTX.cpp:84: error: expected ')' before 'return' make: *** [.objs/dumpTX.o] Error 1 Is this something worth pursuing further? Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: phathash on March 27, 2013, 12:10:57 PM Thanks. gcc now at 4.6.3.
However, now receive - Quote # make lnk -- parser .objs/util.o: In function `decompressPublicKey(unsigned char*, unsigned char const*)': util.cpp:(.text+0x804): undefined reference to `EC_KEY_new_by_curve_name' util.cpp:(.text+0x820): undefined reference to `o2i_ECPublicKey' util.cpp:(.text+0x834): undefined reference to `EC_KEY_set_conv_form' util.cpp:(.text+0x843): undefined reference to `i2o_ECPublicKey' util.cpp:(.text+0x84f): undefined reference to `EC_KEY_free' util.cpp:(.text+0x896): undefined reference to `EC_KEY_free' .objs/util.o: In function `compressPublicKey(unsigned char*, unsigned char const*)': util.cpp:(.text+0x8b4): undefined reference to `EC_KEY_new_by_curve_name' util.cpp:(.text+0x8d0): undefined reference to `o2i_ECPublicKey' util.cpp:(.text+0x8e4): undefined reference to `EC_KEY_set_conv_form' util.cpp:(.text+0x8f3): undefined reference to `i2o_ECPublicKey' util.cpp:(.text+0x8ff): undefined reference to `EC_KEY_free' util.cpp:(.text+0x946): undefined reference to `EC_KEY_free' collect2: ld returned 1 exit status make: *** [parser] Error 1 I believe I do have openssl compiled correctly with ECDSA. Is this worth pursuing? Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: willphase on April 03, 2013, 12:42:16 AM blockchain.info satoshidice profit/loss seems to be a bit flaky at the moment - so right now to work out my profit/loss I'm exporting my transactions then pulling tx info from blockchain.info for each of my transactions and working out if it contains a 1dice* address then assuming that's a bet and adding/removing from the total.
it would be nice to do this using your tool - so supplied with a list of all my addresses, parse blockchain for tx with either { my address } -> { 1dice* } or { 1dice* } -> { my address } Is this possible? Will Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: willphase on April 03, 2013, 07:55:07 AM You can try this: [snip] ought to give you what you want. Sounds good - I'll give that a go. Cheers Will Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: r.willis on April 03, 2013, 08:23:31 AM 2112, your friends' best bet is http://www.ramsan.com/products/rackmount-ram-storage-line
But price can be quite high. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: 2112 on April 03, 2013, 05:54:58 PM 2112, your friends' best bet is http://www.ramsan.com/products/rackmount-ram-storage-line Thank you very much. Those are primarily aimed at the "IOPS desperation" market niche. We aren't there yet. I would call our niche "rapid reboot & real reliability" for write-ahead-logging internal drive. The flash EEPROM storage is so full of misinformation about reliability that its is nearly impossible to make sensible product choices.But price can be quite high. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: r.willis on April 03, 2013, 06:48:42 PM If reliability is main concern, you can use RAID1 (preferably staggered in such a way that two disk in each array have different wear level). Also, larger disks will last longer before wearing.
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: 2112 on April 03, 2013, 09:04:51 PM If reliability is main concern, you can use RAID1 (preferably staggered in such a way that two disk in each array have different wear level). Also, larger disks will last longer before wearing. I wish the life would be so simple. The flash storage sector is so full of cheats and fly-by-nights, that the total cost of building a reliable and long lasting Unix server simply approaches or exceeds the cost of an equivalent mainframe solution. C.f. the recent trend of using supercapacitors to power hidden write-back caches. Combined with the various wear-leveling cheats it produces failures that are extremely hard to locate. Consider this trivialized example of a circular on-disk-buffer of 3 blocks:1 2 3 4 5 6 7 8 9 If you read it after simple reboot you'll get "7 8 9". If you read it after proper power down with super-cap discharge, you'll get "7 5 9" or "7 2 9" or various other combinations. The culprit really isn't flash technology, it is nearly always firmware problem due to its extreme complexity and trivialized testing. There are various options available on the market, but they aren't openly advertised because they tend to use some technologies that IBM has either patented or keeps a tight contractual grip on the channel partners. e.g. SCSI disk drives with AS/400-like formatting with 520 bytes per block for 512 bytes of user data per block. On the other hand we've never suffered a bug in a true RAM drive, possibly because of the extreme simplicity of the required firmware. I apologise for contributing to a derailment of this thread. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: mrb on April 05, 2013, 01:02:57 AM If reliability is main concern, you can use RAID1 (preferably staggered in such a way that two disk in each array have different wear level). Also, larger disks will last longer before wearing. I wish the life would be so simple. The flash storage sector is so full of cheats and fly-by-nights, that the total cost of building a reliable and long lasting Unix server simply approaches or exceeds the cost of an equivalent mainframe solution.Self-healing (ZFS) can make unreliable flash storage reliable. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: MeatPopsicle on April 05, 2013, 11:01:43 PM Compiles fine but then segafaults.
There's logic issues with the mmap'ing. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: MeatPopsicle on April 06, 2013, 03:18:10 AM ext3, and pretty sure it's crashing from a null ptr. Always dies once you try to operate on the last element in vecMap.
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: MeatPopsicle on April 07, 2013, 12:33:46 AM When I didn't have any blockchain files it would crash on unmapping somewhere in cleanMaps, but with blockchain data it dies on:
Code: while(i!=e) totalSize += (i++)->size; in initHashtables. Code: ovrlrdq@thedarkcitadel:/tmp/blockparser$ ./parser stats Valgrind output: Code: info: starting command "simpleStats" Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: fluffypony on April 15, 2013, 05:00:25 AM I'd like to re-open https://github.com/znort987/blockparser/issues/1 - I'm getting the same problem when running an allBalances yesterday. Alternatively, if someone could do an allBalances for me as a precursor to solving the problem that would be greatly appreciated.
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: hyh on April 17, 2013, 09:51:24 PM I saw tons of likely/unlikely in the code, what do they do?
Code:
I searched a bit and it says something like make a prediction etc. Is this the dirty part to make the code fast? How can I know if it works on my machine or not? (I mean, how do I know which one is taken in the definition, #if defined(__GNUC__) or #else?) Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: hyh on April 18, 2013, 03:45:03 AM Performance-wise for blockparser, they don't make much difference anyways (most of the time is spent in I/O). Actually on my computer most time is spent on parseLongestChain(). It takes about 10 min for my computer to do that. Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: hyh on April 18, 2013, 04:55:57 PM Are you A) On a VM ? B) On an encrypted partition ? C) On an OS which doesn't generally implement mmap() properly ? A) No. B) No. C) How can I tell? Can you tell me a way to bench mark this, or shall I check the version of some library? Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: hyh on April 19, 2013, 05:19:10 AM Actually I'm using an SSD... don't know why it is slow.
Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: hyh on April 19, 2013, 05:36:53 AM Are you using a standard drive or an SSD ? If the former, the times you are seeing are not really surprising: parsing 8G (the current size of the block chain) requires *reading* 8G of data from disk. If your disk reads at 20 Mb/s then just the I/O already costs ~ 7mn. I just tried, when I run the parser, the peak speed is about 56MB/s, although my SSD allows about 250MB/s. (But that doesn't look like the key problem. It's just the step parseLongestchain(), steps before that takes about 25s, but parseLongestchain() takes about 600s.) Title: Re: . Post by: andr3w321 on May 07, 2013, 06:01:36 AM Anyone manage to compile on ec2 ubuntu server? I tried from a fresh ubuntu 12.04 and 12.10 server and neither worked. I managed to compile just fine on my laptop running 12.10. I upgraded g++ to 4.7 from 4.6, but that didn't help.
Code: $ make Title: Re: . Post by: Sukrim on May 07, 2013, 07:03:41 AM znort what the hell?! :-\
Why are all your posts deleted + OP censored? Title: Re: . Post by: willphase on May 07, 2013, 07:19:43 AM can someone explain why all znort987 posts have been vanished?
Will Title: Re: . Post by: andr3w321 on May 07, 2013, 09:30:50 AM .2 bitcoins for anyone who can provide instructions for getting to compile on ec2 ubuntu server 12.04 image. Been at this awhile now and can't figure out what's wrong. I've tried installing 32 bit c libs and compiling 32 bit version and that didn't work out. Tried various versions of g++. 4.7 error may be related to this https://bugs.launchpad.net/ubuntu/+source/gcc-4.7/+bug/1132899 I tried 4.5 out and it gave me an error of line 80 of dumpTX which I'm assuming is because of newer foreach syntax. I tried modifying, but this did not seem to help either. I tried copying my locally compiled version but it just segfaults on the server. I tried upgrading to 13.04 but kept getting
/usr/bin/ld: cannot find -l-c collect2: ld returned 1 exit status Title: Re: . Post by: andr3w321 on May 07, 2013, 08:41:01 PM No more bounty. I finally successfully compiled. Turns out I was on a micro instance and running out of memory during compile. I was able to follow these instructions: https://bitcointalk.org/index.php?topic=110627.0 to get it working
Run "dmesg" to make sure its a memory failure. If so you can create a 1gb swap file like so Code: sudo dd if=/dev/zero of=/swapfile bs=64M count=16 When you're done optionally remove the swap file Code: sudo swapoff /swapfile On an unrelated note, before I figured this out I came very close to compiling on a CentOS instance before running into openssl issues found here: https://bitcointalk.org/index.php?topic=9476.0 Below are the commands I ran. They may not all be necessary Code: #install dependencies Title: Re: . Post by: dunand on May 21, 2013, 02:27:49 AM I can compile blockparser but I have this error when using it:
Code: user@machine:~/blockparser$ ./parser allBalances I'm running 0.8. Is blockparser working with 0.8? Title: Re: [ANN] Fast blockchain C++ parser w/ source code Post by: dooglus on May 21, 2013, 06:20:28 AM I saw tons of likely/unlikely in the code, what do they do? They are hints to the compiler's optimizer, and tell it whether a condition is usually true or false. Code like: Code: if (condition) Could be compiled in one of two ways, depending on whether the condition is usually true: Code: test condition or usually false: Code: test condition In both cases, when condition has its expected value no 'goto' is executed and so the code is faster in that case. Title: Re: . Post by: ShadowOfHarbringer on May 21, 2013, 08:20:26 AM can someone explain why all znort987 posts have been vanished? Will It is probably some kind of SCAM, there was a warning on the beginning by a high ranked user (was it grondilu ?). So use this code with care. EDIT: Oh yeah, here it is: After interacting with znort987 on IRC I'd advise being very careful in using this software double check any results that you get from it. I pointed out that he was making some (theoretically and practically) minorly incorrect claims about the analysis this tool produced and his response was confused first failing on technically grounds and then after the technical misunderstand was made clear he became uncivil— calling me and other folks names and such— quite bizarre. While its obviously useful, it's written by someone who apparently hasn't done your thinking for you, so you need to think for yourself. Title: Re: . Post by: Sukrim on May 21, 2013, 09:35:09 AM This was most likely because closure (2 inputs being viewed as belonging to the exact same person) is not a 100% measure, as demonstrated some time ago where someone asked others to sign some transactions with him to add his coins to their closure calculation and vice versa.
The code is open source, so it is not hard to find any scams in it... Please be careful with your accusations, maybe this is why he left?! Title: Re: . Post by: ShadowOfHarbringer on May 21, 2013, 10:32:11 AM The code is open source, so it is not hard to find any scams in it... Please be careful with your accusations, maybe this is why he left?! I am only saying that gmaxwell found this suspicious. Title: Re: . Post by: dooglus on May 21, 2013, 11:59:55 AM This was most likely because closure (2 inputs being viewed as belonging to the exact same person) is not a 100% measure, as demonstrated some time ago where someone asked others to sign some transactions with him to add his coins to their closure calculation and vice versa. That someone was gmaxwell. This seems to be the IRC chat they're referring to: Quote --- Log opened Sat Mar 09 00:00:01 2013 12:23 -!- znort987 [~znort987@gateway/tor-sasl/znort987] has joined #bitcoin 12:24 < znort987> ;;gpg eauth znort987 12:24 <@gribble> Request successful for user znort987, hostmask znort987!~znort987@gateway/tor-sasl/znort987. Get your encrypted OTP from http://bitcoin-otc.com/otps/FD5AA2A5E26602B0 12:26 < znort987> ;;gpg everify freenode:#bitcoin-otc:f6c4d80bf1e5f7f84a1cae73581f3a7d824b23838dccc6a8e3715ac1 12:26 <@gribble> You are now authenticated for user znort987 with key FD5AA2A5E26602B0 13:50 <@gmaxwell> "Compute the list of addresses that provably belong to the same person" = fail. 13:50 <@gmaxwell> znort987: can you fix that claim? 13:50 < znort987> gmaxwell: fail, "how so" ? 13:51 <@gmaxwell> Before your software results in someone getting prosecuted for someone elses crime or other facepalmery. 13:51 <@gmaxwell> znort987: being common authors on an input does not imply common address ownership. 13:51 < comboy> znort987: that was posted probably befoe multisig? 13:51 <@gmaxwell> even without multisig. 13:51 < comboy> without multisig it does not prove that? 13:51 <@gmaxwell> E.g. you can have a transaction where I provide one input and you provide another. 13:52 < znort987> All I'm hearing are claims, and not explanation forthcoming 13:52 <@gmaxwell> wtf. 13:52 < comboy> ah, indeed 13:52 <@phantomcircuit> znort987, let me save you some time 13:52 <@gmaxwell> znort987: wow. I just explained it. 13:52 <@phantomcircuit> if gmaxwell assets plainly that something is true, you can pretty much be sure that it is 13:53 < jouke> gmaxwell: just give him the address :> 13:53 < znort987> phantomcircuit: sure. That still doesn't explain it. 13:53 < Scrat> i've seen gmaxwell make typos. ha ha! 13:53 <@gmaxwell> znort987: Okay, I'm not sure what was unclear there. 13:53 < comboy> znort987: you could probably only assume same author if it was signed by the same priv key 13:54 <@phantomcircuit> znort987, valid transactions can be constructed in which multiple parties sign the input portion of the transaction 13:54 <@gmaxwell> znort987: We write a transaction it has two inputs. 1znort987 and 1gmaxwell. Each of us respectively sign the transaction for our own inputs. This does not mean we are the same person. 13:54 < znort987> ok, problem is fixed 13:54 <@phantomcircuit> znort987, in short your assumption is wrong 13:54 <+TuxBlackEdo> znort987, when people make bitcoin transactions, sometimes two people that don't know eachother exchange private keys to make a transaction to someone else 13:54 <@gmaxwell> TuxBlackEdo: or just jointly sign 13:55 <@phantomcircuit> TuxBlackEdo, dont need to exchange private keys, just sign the same outputs 13:55 <+TuxBlackEdo> oh yeah 13:55 <+TuxBlackEdo> but still 13:55 <@gmaxwell> People also transfer ownership of privkeys. 13:55 <+TuxBlackEdo> how often does that happen 13:55 < Scrat> what about disassociative mixing? then it will assign ownership toi completely different people 13:55 < znort987> p(keys owned by same person) = count(weird tx assembled by folks who can understand the shit you guys are talking about)/count(all transactions) 13:56 <@phantomcircuit> TuxBlackEdo, often enough that assuming people are linked by same signed inputs is invalid 13:56 <@gmaxwell> E.g. I can take a key with a long history and load in into mtgox. Mtgox will then very likely sign with it in a transaction with keys from other people. 13:56 <+TuxBlackEdo> i am sure one could get in trouble just by being the first relayer of a transaction 13:56 < Scrat> znort987: b.i mixer. used by many people 13:56 < znort987> therefore, fix = s/provably/very likely/ 13:56 <+TuxBlackEdo> "oh you forwarded me a drug transaction, that must mean you must be a drug dealer" 13:56 < comboy> znort987: then s/provably/very likely 13:56 < znort987> Are sphincters relaxing now ? 13:57 <+TuxBlackEdo> ^ 13:57 <@phantomcircuit> znort987, no actually because of the mtgox private key load it isn't even likely 13:57 <@phantomcircuit> there are very simple tools to upload your wallet.dat to mtgox 13:57 <@gmaxwell> znort987: It looks like the b.i 'shared send' (replacement for their mixer service) works this way normally. 13:57 <@phantomcircuit> so indeed any correlation is *highly* suspect 13:58 <@gmaxwell> So I wouldn't be surprised if a few percent of txn on the network are like that now. 13:58 <@gmaxwell> But sure, I agree that its likely, though it'll be less of an indicator in the future. 13:58 <@gmaxwell> But seriously, be more careful with the provably in the future. :P 13:59 <@phantomcircuit> gmaxwell, i suspect the amount of mixing sr engages in actually makes even current assumptions incorrect 13:59 <@phantomcircuit> i've moved coins through there to find out and i honestly couldn't figure out where they came out 13:59 < znort987> For the anal retentive crowd out there, in general, there is no such thing as a proof in the real world. Just high probabilities. 14:00 < Scrat> phantomcircuit: same goes for gox withdrawals sometimes 14:00 <@gmaxwell> znort987: there are orders of magnitude, however. 14:00 < Scrat> or using the green address 14:00 <@phantomcircuit> znort987, the point is that the probabilities are likely low but even so unknowable 14:02 < znort987> Nope. The point is : is the information extracted that way any kind of useful. I claim that as things stand today, they are, notwithstanding the language lwayers. 14:02 < znort987> *lawyers 14:02 <@gmaxwell> znort987: often we talk about proven when the chances are astronomical absent a mathmatical breakthrough. Your clam was likely invalid for a substantial percentage so of addresses holding coins. (due to your use of the word belong) 14:03 <@phantomcircuit> znort987, forget lawyering, your odds are just plain wrong 14:03 <@phantomcircuit> there is a lot more mixing going on than most people thing 14:03 <@gmaxwell> znort987: 'any kind of useful' sure. Proven absolutely not. Very likely? eh. maybe. 14:03 <@phantomcircuit> think* 14:03 <@phantomcircuit> a *lot* more 14:04 <@phantomcircuit> gmaxwell, actually i would say the odds are very low unless it's a satoshi dice transaction 14:04 <@gmaxwell> I would not even be surprised if your claim was incorrect for a simple majority of coin ownership weighed addresses, simply due to shared wallet usage. 14:04 < znort987> You guys are in the wrong business. I'm thinking you ought to give a try to fly fucking or classifying dead invertebrates. If there was olympic games for nitpicking you'd likely be in it. 14:04 <@phantomcircuit> most people are really using shared wallets of some kind 14:04 <@phantomcircuit> lol 14:04 <@phantomcircuit> while that was hilarious 14:04 <@phantomcircuit> (im drunk enough that that was funny) 14:04 <+TuxBlackEdo> znort987, i think you need to calm down, these are smart people 14:04 <@phantomcircuit> you're still wrong 14:05 <@gmaxwell> znort987: 'nitpicking' when your claims are objectively incorrect on a substantial number of channels, and could cause people harm? give me a break. 14:05 < znort987> intelligent != smart. A textbook example if there ever was one. 14:05 <@gmaxwell> znort987: how about you behave responsibly with your software, and work on your civility a bit too? enh? 14:05 <@phantomcircuit> gmaxwell, sigh i really want to ban him but i know im too drunk to make such a decision 14:05 <@phantomcircuit> :| 14:05 <+TuxBlackEdo> znort987, the only one with a tight sphincer is you, right now. 14:06 <@gmaxwell> hm. s/channels/transactions/ (this is what I get for concurrently talking and typing) 14:06 <@phantomcircuit> as TuxBlackEdo says 14:06 <@phantomcircuit> my sphincter is very loose 14:06 * phantomcircuit lols away 14:06 <+TuxBlackEdo> phantomcircuit, i know ;) 14:06 <@phantomcircuit> ahahah 14:07 <+TuxBlackEdo> no homo. 14:07 <@phantomcircuit> yeah sure 14:07 <@phantomcircuit> whatever you say 14:07 < znort987> gmaxwell: "responsibly" ? Are you actually dumb enough to believe that someone would use this in a court of law ? You must be living in the US. 14:08 <@phantomcircuit> zomg can i ban him now 14:08 <@phantomcircuit> pleassseeeee 14:08 -!- mode/#bitcoin [+b *!*znort987@gateway/tor-sasl/znort987] by gmaxwell 14:08 -!- znort987 was kicked from #bitcoin by gmaxwell [znort987] 14:08 <@phantomcircuit> HORRAAAYYY 14:08 <+TuxBlackEdo> i thought gmaxwell was from mexico 14:15 <@gmaxwell> tor user, why am I surprised. 14:15 -!- mode/#bitcoin [-b *!*znort987@gateway/tor-sasl/znort987] by gmaxwell 14:15 <@gmaxwell> https://bitcointalk.org/index.php?topic=88584.msg1603598#msg1603598 FWIW 14:16 <@gmaxwell> I notice he changed it back to " provably belong to the same person" 14:17 < valparaiso> yes, shame really. Anonymity in the hands of trolls... 14:17 < comboy> huh, that escalated quickly, he shouldn't say "provably", especially in bitcoin community, but still he has done really nice work on this parser 14:18 <@phantomcircuit> jesus crist i have 429 posts on the forum??? 14:18 * phantomcircuit needs to get a life 14:18 <@phantomcircuit> :| Title: Re: . Post by: dunand on May 22, 2013, 02:40:05 AM I can compile blockparser but I have this error when using it:
Code: user@machine:~/blockparser$ ./parser allBalances I'm running 0.8. Is blockparser working with 0.8? Title: Re: . Post by: wiggi on May 26, 2013, 01:54:01 PM I'm running 0.8. Is blockparser working with 0.8? Yes, should definitely work with 0.8.1, just a bit slower than with the older (0.7.x) versions. Title: Re: . Post by: gweedo on May 26, 2013, 06:09:21 PM After interacting with znort987 on IRC I'd advise being very careful in using this software double check any results that you get from it. I pointed out that he was making some (theoretically and practically) minorly incorrect claims about the analysis this tool produced and his response was confused first failing on technically grounds and then after the technical misunderstand was made clear he became uncivil— calling me and other folks names and such— quite bizarre. While its obviously useful, it's written by someone who apparently hasn't done your thinking for you, so you need to think for yourself. LMAO more censorship by the Core Dev team, first it was transaction now it is other people software. I mean when does it stop. Honestly power hungry much? So dumb, you, Gavin, and Luke Jr. I really hope another programming team, just takes over and we can be done, this what no competition does in a free market. Creates egotistical, self centered, power hungry people. Know what instead of policing the forum and irc actually make the full node work with a big blockchain instead of putting a band-aid on it. Never saw more power hungry people in my life. Nothing decentralized about bitcoins, it completely centralized with people like this running behind it. Title: Re: . Post by: andr3w321 on May 29, 2013, 07:19:14 PM I can compile blockparser but I have this error when using it: Code: user@machine:~/blockparser$ ./parser allBalances I'm running 0.8. Is blockparser working with 0.8? Does this file exist and do you have permission to open it? Print the output of Code: ls -l /home/user/.bitcoin/blocks Title: Re: . Post by: wolongong on June 02, 2013, 10:18:11 PM How much is "*lots* of RAM" in the warning for --detailed?
Title: Re: . Post by: wiggi on June 04, 2013, 02:45:31 PM How much is "*lots* of RAM" in the warning for --detailed? It likely will finish on 8GB but I wouldn't do this to my HDD, and Code: ./parser allBalances >allBalances.txt Code: ./parser allBalances -a200000 -d >allBalances.txt upgrade plan: now: 16GB Q1 2014: 32GB Title: Re: . Post by: refer_2_me on June 11, 2013, 03:08:06 PM So I ran ./parser allBalances >allBalances.txt. It took a while to run, but didn't throw any errors and seemed to complete normally.
The first 100 lines or so of the output also seem fine, but after that the Bitcoin addresses are all X's: for example: "35.09900000 0476042f5062d0f4092dcab601b142c0dcf339ef XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 1 Thu May 16 13:26:12 2013 0 Thu Jan 1 00:00:00 1970" Anyone have any idea why this is happening and how I can fix it? I glanced at the code, but don't really understand that is happening. Thanks in advance! Title: Re: . Post by: w00dy on June 11, 2013, 05:45:39 PM So I ran ./parser allBalances >allBalances.txt. It took a while to run, but didn't throw any errors and seemed to complete normally. The first 100 lines or so of the output also seem fine, but after that the Bitcoin addresses are all X's: for example: "35.09900000 0476042f5062d0f4092dcab601b142c0dcf339ef XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 1 Thu May 16 13:26:12 2013 0 Thu Jan 1 00:00:00 1970" Anyone have any idea why this is happening and how I can fix it? I glanced at the code, but don't really understand that is happening. Thanks in advance! use the -w / --withAddr switch. (out of my head): Code: ./parser allBalances -w 100000 >allBalances.txt ("only show address for top N results (default: N=%default)" Title: Re: . Post by: refer_2_me on June 11, 2013, 05:49:23 PM Ah, thank you. I can't believe I missed that.
Title: Re: . Post by: emsit on November 17, 2013, 05:54:50 PM Hello, What is the minimum RAM need to perform a command?
I have 512MB and all command ended with error: Killed Can you tell me some other parser that can display a one block of raw? Title: Re: . Post by: Sukrim on November 17, 2013, 06:08:36 PM To load all the stuff in RAM you probably need by now far more than 512 MB...
bitcoind can give you raw blocks via its RPC interface and you can read the raw block files too. Title: Re: . Post by: emsit on November 17, 2013, 06:26:45 PM Try bitcoind RPC but I found only this commands: getblock
I need information with including transactions. Something like https://blockchain.info/rawblock/1 EDIT:I must as first get getblock and get for each transaction getrawtransaction and, finally connected to one output Title: Re: . Post by: barwizi on December 30, 2013, 11:01:57 PM any guide to use in windows? or does someone have it built for windows already?
also can i have a few tips on how to apply it on a different chain? Title: Re: . Post by: Sukrim on December 31, 2013, 12:23:10 AM any guide to use in windows? or does someone have it built for windows already? znort never intended it to run on Windows, if you post a bounty maybe someone is able to compile it? Otherwise, just get going, theoretically the libraries used should be available on Windows too.also can i have a few tips on how to apply it on a different chain? As it parses the transactions, it likely depends on how close your "different chain" implements these compared to Bitcoin. Title: Re: . Post by: organofcorti on December 31, 2013, 02:46:02 AM any guide to use in windows? or does someone have it built for windows already? znort never intended it to run on Windows, if you post a bounty maybe someone is able to compile it? Otherwise, just get going, theoretically the libraries used should be available on Windows too.also can i have a few tips on how to apply it on a different chain? As it parses the transactions, it likely depends on how close your "different chain" implements these compared to Bitcoin. I think it worked with cygwin when I tried it. Otherwise a dual boot with linux or a VM are going to be your only options. Title: Re: . Post by: Sukrim on December 31, 2013, 09:52:07 AM Well, looking at RAM requirements, I guess I'd recommend a live system from a USB stick if you just want to try the parser instead of a VM.
Title: Re: . Post by: barwizi on December 31, 2013, 10:38:33 AM Thanks!! so if the code base is bitcoin, if i just copy the files i want to parse to .bitcoin it will work? or are there some parameters i need to change?
Title: Re: . Post by: jj30 on January 03, 2014, 03:21:14 PM Can anyone help? Make went fine. When running the parser, it's just hanging...
info: starting command "simpleStats" and then it just sits there indefinitely. I ctrl-c after a half hour or so, assuming nothing is happening (8 gb memory). What have I done wrong? I think there is an "info: parsing blockchain" message supposed to come after that. I have checked and my blockchain is in ~/.bitcoin. Can anyone help? EDIT: SimpleStats came back in 5774 seconds ... a little over an hour and a half... so I guess I need more memory or something. Title: Re: . Post by: gweedo on January 03, 2014, 06:11:05 PM EDIT: SimpleStats came back in 5774 seconds ... a little over an hour and a half... so I guess I need more memory or something. Remember the blockchain is always growing, when Znort wrote this a while ago it was like 7 or 8gb now it is 14 or 15gb that is a lot to parse and keep in memory. So you don't need more memory it is just a lot of data. Title: Re: . Post by: barwizi on February 06, 2014, 11:28:11 AM I want to hire someone to modify it for a different coin, if you can do it and are interested ,PM me and we can get to business. Need this asap.
Title: Re: . Post by: ShadowOfHarbringer on February 06, 2014, 04:20:38 PM I wonder...
- Is the author even maintaining/supporting this project in any way ? - Does this thing even work ? - Why did first post get deleted ? And NO, i do not want to read-through all hundereds of posts to find this out. Title: Re: . Post by: Sukrim on February 06, 2014, 04:48:26 PM No idea if it works, I don't have enough RAM and wrote my own parser based on armory-python.
Probably it works, try it out. Seems like he wanted to delete all his posting history on bitcointalk (just like some others did too). Title: Re: . Post by: Vinz87 on March 23, 2014, 08:10:33 PM can someone help me? I'm on OS X 10.9, I have gcc 4.5 installed via homebrew, now I have to install the dependencies this software relies on, should I use homebrew as well?
but for example I get this error Code: $ brew install libssl-dev Title: Re: . Post by: mullick on July 25, 2015, 08:05:14 AM I dont think i have the correct syntax here or something. Cant seem to get this to work
Code: ./parser simpleStats Tried a few commands. As well as declaring the block directory. Same result. Am I missing something here? |