.

[Dabs]

Maybe there is a way to implement this differently so it doesn't use mmap files? Or windoze implementation could be improved? Or I'm asking too much?

If I had the ram, maybe blockparser could do it all in ram? Or I can try using a ram drive?

[1713570733]

1713570733

[1713570733]

1713570733

[Sukrim]

To deal with crappy retarded GPU drivers and UI in Linux? No thanks, sir...

Anyways, in theory it should(TM) be possible already to compile + run this parser for Windows. It's not znort987's resposibility to actually provide the means to do so, especially since he doesn't want to do it. It would be nice of him if somebody manages to actually come up with a solution how to do it, to integrate some minor patches (as far as I see it from the code, the blockchain location is hardcoded for example - having a setting for Windows or even Linux users there might be nice) and the actual way to do it, but even that is not his job at all (and you simply could just fork his project after all).

The code is public domain and uses OpenSource libraries. Either invest the time to compile that stuff yourself or pay somebody to do it for you. If mmap implementations on Windows are really that slow, you can also improve them, since this project is built on OpenSource software. Again, either by yourself or somebody you pay to do that.

[Dabs]

I guess I am asking too much, in the same way that a request to get rid of crappy OSes is also asking too much. Depends on who you talk to, if it's one individual, it is reasonable to ask them to switch OS, if you are talking to a corporation with hundreds of deployed crappy OS machines, that might be asking too much.

However, I do have the option to run Linux on my personal machine, so unless Windows gets improved (either the mmap implementation or the parser itself), then that's the route to take, for now. My guess is that, since this is a specialized tool, you really should use it on dedicated machines that are not being used for other crappy OS stuff.

Unfortunately for me, that's not a practical thing to do. I could dual-boot to experiment with it, but eventually I'd have to go back to my crappy OS. Thus my question.

[Buffer Overflow]

To deal with crappy retarded GPU drivers and UI in Linux? No thanks, sir...

The manufacturers make the drivers, blame them. Linux is more than capable of running most hardware.
Who wants to dance and jump about to Bill Gates tune? Playing the game by his rules, his way, at his (expensive) price.   No thanks, sir...

Unfortunately for me, that's not a practical thing to do. I could dual-boot to experiment with it, but eventually I'd have to go back to my crappy OS. Thus my question.

You could install a virtualization software package, like Virtual Box and install Linux inside that. That's assuming your hardware can take the extra load of course.



How come (and this isn't directed at anyone in particular) when a Linux user needs a program that only runs on Windows, he/she gets their head down, knuckles down and finds way around the problem. But on the other hand when a Windows user needs a program that only runs on Linux, there's a huge uproar, and expects everyone else to adjust to them? It's as if they assume Windows is the default system and everyone must whistle their tune.

[Sukrim]

To deal with crappy retarded GPU drivers and UI in Linux? No thanks, sir...

The manufacturers make the drivers, blame them.

Ok, let's blame AMDand nVidia! Evil you! You really should support Linux better!
Great to have done this, I guess that changes now everything...

I am also a Linux user and I never got my head down, knuckled down and tried to find a way around the problem - I simply installed a Windows VM or native machine and ran the program in it's intended OS. Please refrain from generalizing, I bet if you wrote this statement in a Linux heavy forum replacing Windows with Ubuntu and Linux with Gentoo it would also be in line there... especially taking into consideration that this software is said to only support 64 bit Ubuntu.

Anyways, there IS already the possibility to compile and run this software under Windows and as soon as I manage to compile it, I'll maybe even share the binary (for a hefty fee perhaps to cover my time). If this doesn't work, well - having a result in 3 minutes instead of one vs. having no result at all would still be an improvement, right?

[gmaxwell]

After interacting with znort987 on IRC I'd advise being very careful in using this software double check any results that you get from it.

I pointed out that he was making some (theoretically and practically) minorly incorrect claims about the analysis this tool produced and his response was confused first failing on technically grounds and then after the technical misunderstand was made clear he became uncivil— calling me and other folks names and such— quite bizarre. While its obviously useful, it's written by someone who apparently hasn't done your thinking for you, so you need to think for yourself.

[phantomcircuit]

After interacting with znort987 on IRC I'd advise being very careful in using this software double check any results that you get from it.

I pointed out that he was making some (theoretically and practically) minorly incorrect claims about the analysis this tool produced and his response was confused first failing on technically grounds and then after the technical misunderstand was made clear he became uncivil— calling me and other folks names and such— quite bizarre. While its obviously useful, it's written by someone who apparently hasn't done your thinking for you, so you need to think for yourself.

I can confirm this.

[ShadowOfHarbringer]

After interacting with znort987 on IRC I'd advise being very careful in using this software double check any results that you get from it.

I pointed out that he was making some (theoretically and practically) minorly incorrect claims about the analysis this tool produced and his response was confused first failing on technically grounds and then after the technical misunderstand was made clear he became uncivil— calling me and other folks names and such— quite bizarre. While its obviously useful, it's written by someone who apparently hasn't done your thinking for you, so you need to think for yourself.

This forum should have a red or orange warning box on the top of each topic (placeable by mods, admins & devs only) to warn people of scams and events such as this.

What do you think ?

[gmaxwell]

Ah, and since he changed this back— I suppose I should point this out here:

* Compute the list of addresses that provably belong to the same person

This is both theoretically and practically incorrect. The tool doesn't do this for a couple of reasons.

The most common violation of this comes from shared wallets (like mtgox and instawallet). Signatures on transactions do not not indicate "belonging"— mtgox signs on hundreds of thousands of coin that do not belong to them. In that case it only indicates control, and only at a point in time as people do transfer ownership of private keys to these services. (E.g. If give my old privkeys with some coins on them to mtgox, they'll end up as inputs on other people's txn— it's true that mtgox controls them from that point on, but not in the past).

Less commonly, but also more fundamentally, is this assumption is also invalidated by group transactions. E.g. it's quite possible for two people to each provide an input to a transaction. This is useful for making atomic shared payments to bounties and the like, but also for doing trustless mixing.  I believe the new "shared send" functionality on b.i (which replaced their old mixing service) works  this way, and so I wouldn't be surprised if a non-trivial number of transactions failed on this criteria.

Obviously, the importance of these details depend on how you are using the data. Common inputs are suggestive of common control at the time the transaction was authored. They are suggestive (though less so) of past control. They are far less suggestive of actual belonging because the belonging assumption is commonly violated by shared wallets and may be untrue for a significant fraction of coins in existence. They certainly do not constitute a 'proof' in any sense that we'd normally use the word in this community.

If you're interested in making soft guesses about some transactions, then these might be fine assumptions.  If you're looking to prosecute someone for a crime— certainly not. If you want to make statistical comments about populations of transactions it's highly questionable because no one knows how biased a statistic the result is, since mixed and shared transactions appear to be indistinguishable from other ones.

Cheers.

This forum should have a red or orange warning box on the top of each topic (placeable by mods, admins & devs only) to warn people of scams and events such as this.

I don't think it rises to that level of concern. I apologize if I came off sounding like it was that urgent.  It's not. In this case I wanted to throw out a caveat emptor because I thought it was odd... but really caveat emptor is generally good advice.   If it were something like I thought that the software would cause you to lose coin or something I would have made it a _lot_ more obvious.

[Buffer Overflow]

Ok, let's blame AMDand nVidia! Evil you! You really should support Linux better!
Great to have done this, I guess that changes now everything...

Correct me if I'm wrong, but don't these two quotes listed below contradict each other?

I am also a Linux user and I never got my head down, knuckled down and tried to find a way around the problem - I simply installed a Windows VM or native machine and ran the program in it's intended OS.

Anyways, there IS already the possibility to compile and run this software under Windows and as soon as I manage to compile it, I'll maybe even share the binary (for a hefty fee perhaps to cover my time).

[Sukrim]

Nope, as I would do the feats that were attributed to a brave Linux hacker as a Windows user, not as a Linux user. Anyways, this is actually besides the point.

The issue that transactions that spend from two addresses do not 100% mean that these addresses are controlled by the same person or entity is something more interesting than whether I manage to compile OpenSSL etc. under Windows.

Especially that constant cries to import private keys here and there might make it tough in the future to even analyze transactions at all.
I also can't think of any 100% sure ways to overcome this problem of people sharing inputs to fool software like this - maybe creating address pools and then looking for suspicious overlaps might be a solution. E.g. 500 transactions each in 2 sets of coupled addresses, then suddenly for a single transaction they bond (even though the "simulated wallet" containing both sets would contain better fitting outputs?) and then they start transacting on their own again never mixing coins... Also transaction timestamps could give indications about users (and adding a second set that has so far completely different transaction patterns might indicate someone trying to mix coins).

[Sukrim]

Oh, and for some measurements:

With larger hash maps: SimpleStats in ~850s
With smaller hash maps (WANT_DENSE commented out): SimpleStats in ~350s

Still a lot of time, but not too shabby for a little atom machine.

[Sukrim]

Hm, could be the case - I didn't reboot between runs + recompilation.

[Sukrim]

Oh btw. it works/compiles like a charm on Arch Linux, so no worries about needing Ubuntu (you don't even specify which release anyways...).

Now back to solving my Hash160 issues... :-/

[mrb]

No need to reboot.
Just flush the disk cache.
Something along the lines of:

Code:

dd if=/dev/zero bs=1024k count=4096 of=/dev/null

usually does the trick.

This has pretty much no effect on the buffercache (kernel knows /dev/zero does not need caching, and dd will use at most 1MB of resident memory). What will drop it is (as root):

Code:

$ echo 3 >/proc/sys/vm/drop_caches

[Dabs]

Windows is not the default OS. It's just, unfortunately the market leader, or the OS with the largest share of users. I'm sure that if I can't dual-boot, I can probably do a Live CD.

[Sukrim]

Windows is not the default OS. It's just, unfortunately the market leader, or the OS with the largest share of users. I'm sure that if I can't dual-boot, I can probably do a Live CD.

Good luck with catching up on the blockchain then... depending on your needs, you're faster to just run the stuff you need on armory-python.

Anyways, doing a closure run brings my poor little nettop to its knees (even with the "smaller" hash maps) - at least after a few hours it didn't show any kind of progress, so I killed the program. 1GB RAM is not cutting it at all apparently.

[wiggi]

Well, in theory it should be possible to let this run on Windows as well... in practice however I've only managed to compile boost and am still fighting with sparsehash as well as openssl and it's dependencies (openssl wants perl for example).

Sparsehash also seems to really not like to be compiled with MingW... Let's see if I give up and just install a VM on my big machine, because my small Linux Atom nettop probably will choke to death on the hash tables.

Another issue with VM's is that:

   a) you're going to have to copy the blockchain over to the VM
   b) blockparser uses mmap'd files extensively. windoze implementation of those is hugely inefficient
   c) blockparser is really I/O intensive (every run gobbles in the full blockchain). This is highly inefficient on VMs

A little test, how long does it take to do "./parser allBalances >allBalances.txt" in VirtualBox
with different amounts of ram assigned to the VM:
(host is win7 64bit, guest Linux Mint 64bit, 8GB physical ram)

1GB   eta 1200000s
3GB        eta 5000s
6GB         eta 200s, but actually >1500s, slows down alot @50-70%
7GB             850s

As long as the VM can consume more and more ram, blockparser is *very* fast, but I'm
not sure how much is really enough to let the OS buffer all I/O in ram, 32GB perhaps.

[Dabs]

I'm tempted to buy a machine with 32 Gigs of RAM. ... ... Maybe those so called "ram drives" (hardware implemented) might be useful.

@Sukrim, I can probably sync the blockchain with another computer that already has it within the network, or just copy the files over. That will save download time.

[mrb]

I'm tempted to buy a machine with 32 Gigs of RAM. ... ... Maybe those so called "ram drives" (hardware implemented) might be useful.

Do it, 32GB only costs $160, assuming you already have a machine with a motherboard supporting 4 x 8GB DDR3.
But don't buy hardware ram drives. They are all horribly overpriced, as in 10x more expensive per GB, or more.