.

[Dabs]

The hardware ram drives I know use old DDR2 memory. Maybe there are new ones that are horribly overpriced. Or I'll look around to see if someone has made one. The one I know used to use a SATA cable to emulate a hard drive. Real ram inside the motherboard is obviously going to work a lot faster.

[1714878727]

1714878727

[1714878727]

1714878727

[1714878727]

1714878727

[1714878727]

1714878727

[2112]

But don't buy hardware ram drives. They are all horribly overpriced, as in 10x more expensive per GB, or more.

Please pardon me for jumping in, but please post the links to those overpriced hardware RAM drives. My friends in IT operations are on the market and have hard time sourcing devices that don't wear out are reasonably non-volatile and can survice reboot/reconfig/reload of the OS.

[mrb]

But don't buy hardware ram drives. They are all horribly overpriced, as in 10x more expensive per GB, or more.

Please pardon me for jumping in, but please post the links to those overpriced hardware RAM drives. My friends in IT operations are on the market and have hard time sourcing devices that don't wear out are reasonably non-volatile and can survice reboot/reconfig/reload of the OS.

http://www.ddrdrive.com - it was designed for ZFS ZIL, and the RAM is backed by SLC NAND flash. It was priced $2000 when it launched ~3 years ago. The price has probably gone down a lot, but it is still way more than 10x the price per GB than system's RAM.

Part of the reason prices are so high is because there are so few vendors, so there is practically no competition.

[2112]

http://www.ddrdrive.com - it was designed for ZFS ZIL, and the RAM is backed by SLC NAND flash. It was priced $2000 when it launched ~3 years ago. The price has probably gone down a lot, but it is still way more than 10x the price per GB than system's RAM.

Part of the reason prices are so high is because there are so few vendors, so there is practically no competition.

Thanks. Unfortunately this is still a flash drive with RAM cache and still susceptible to write amplification abuse. Thus far we haven't found anything better than discontinued ACARD ANS-9010 (64GB with battery backup) and some other custom-made solutions.

[onelineproof]

This is nice. One feature I would like to see is the creation of a table that stores public key to balance pairs, and keeps updating from time to time. This way you can quickly look up the balance of an address. If I have time, I may use this code to do this, and I'll add it to my wallet parser: https://github.com/piratelinux/cwallet

[phelix]

Would it be possible to dig out what percentage of coins have ever been moved for any 2016 block period?

I would be interested in how many of the early coins have never been moved.

edit: ok, just found your post: https://bitcointalk.org/index.php?topic=98641.0  [2 Million unspent...]

[phathash]

I know this is developed under Ubuntu, but it would be nice to have working on RH type distros.

I have openssl with ecdsa working, but blockparser won't compile -

# make
c++ -- cb/dumpTX.cpp
cb/dumpTX.cpp: In member function 'virtual int DumpTX::init(int, const char**)':
cb/dumpTX.cpp:80: error: expected initializer before ':' token
cb/dumpTX.cpp:84: error: expected primary-expression before 'return'
cb/dumpTX.cpp:84: error: expected ';' before 'return'
cb/dumpTX.cpp:84: error: expected primary-expression before 'return'
cb/dumpTX.cpp:84: error: expected ')' before 'return'
make: *** [.objs/dumpTX.o] Error 1

Is this something worth pursuing further?

[phathash]

Thanks. gcc now at  4.6.3.

However, now receive -

# make
lnk -- parser
.objs/util.o: In function `decompressPublicKey(unsigned char*, unsigned char const*)':
util.cpp:(.text+0x804): undefined reference to `EC_KEY_new_by_curve_name'
util.cpp:(.text+0x820): undefined reference to `o2i_ECPublicKey'
util.cpp:(.text+0x834): undefined reference to `EC_KEY_set_conv_form'
util.cpp:(.text+0x843): undefined reference to `i2o_ECPublicKey'
util.cpp:(.text+0x84f): undefined reference to `EC_KEY_free'
util.cpp:(.text+0x896): undefined reference to `EC_KEY_free'
.objs/util.o: In function `compressPublicKey(unsigned char*, unsigned char const*)':
util.cpp:(.text+0x8b4): undefined reference to `EC_KEY_new_by_curve_name'
util.cpp:(.text+0x8d0): undefined reference to `o2i_ECPublicKey'
util.cpp:(.text+0x8e4): undefined reference to `EC_KEY_set_conv_form'
util.cpp:(.text+0x8f3): undefined reference to `i2o_ECPublicKey'
util.cpp:(.text+0x8ff): undefined reference to `EC_KEY_free'
util.cpp:(.text+0x946): undefined reference to `EC_KEY_free'
collect2: ld returned 1 exit status
make: *** [parser] Error 1

I believe I do have openssl compiled correctly with ECDSA.

Is this worth pursuing?

[willphase]

blockchain.info satoshidice profit/loss seems to be a bit flaky at the moment - so right now to work out my profit/loss I'm exporting my transactions then pulling tx info from blockchain.info for each of my transactions and working out if it contains a 1dice* address then assuming that's a bet and adding/removing from the total.

it would be nice to do this using your tool - so supplied with a list of all my addresses, parse blockchain for tx with either { my address } -> { 1dice* } or { 1dice* } -> { my address }

Is this possible?

Will

[willphase]

You can try this:

[snip]

ought to  give you what you want.

Sounds good - I'll give that a go.

Cheers

Will

[r.willis]

2112, your friends' best bet is http://www.ramsan.com/products/rackmount-ram-storage-line
But price can be quite high.

[2112]

2112, your friends' best bet is http://www.ramsan.com/products/rackmount-ram-storage-line
But price can be quite high.

Thank you very much. Those are primarily aimed at the "IOPS desperation" market niche. We aren't there yet. I would call our niche "rapid reboot & real reliability" for write-ahead-logging internal drive. The flash EEPROM storage is so full of misinformation about reliability that its is nearly impossible to make sensible product choices.

[r.willis]

If reliability is main concern, you can use RAID1 (preferably staggered in such a way that two disk in each array have different wear level). Also, larger disks will last longer before wearing.

[2112]

If reliability is main concern, you can use RAID1 (preferably staggered in such a way that two disk in each array have different wear level). Also, larger disks will last longer before wearing.

I wish the life would be so simple. The flash storage sector is so full of cheats and fly-by-nights, that the total cost of building a reliable and long lasting Unix server simply approaches or exceeds the cost of an equivalent mainframe solution. C.f. the recent trend of using supercapacitors to power hidden write-back caches. Combined with the various wear-leveling cheats it produces failures that are extremely hard to locate. Consider this trivialized example of a circular on-disk-buffer of 3 blocks:

1 2 3
4 5 6
7 8 9

If you read it after simple reboot you'll get "7 8 9". If you read it after proper power down with super-cap discharge, you'll get "7 5 9" or "7 2 9" or various other combinations. The culprit really isn't flash technology, it is nearly always firmware problem due to its extreme complexity and trivialized testing.

There are various options available on the market, but they aren't openly advertised because they tend to use some technologies that IBM has either patented or keeps a tight contractual grip on the channel partners. e.g. SCSI disk drives with AS/400-like formatting with 520 bytes per block for 512 bytes of user data per block.

On the other hand we've never suffered a bug in a true RAM drive, possibly because of the extreme simplicity of the required firmware.

I apologise for contributing to a derailment of this thread.

[mrb]

If reliability is main concern, you can use RAID1 (preferably staggered in such a way that two disk in each array have different wear level). Also, larger disks will last longer before wearing.

I wish the life would be so simple. The flash storage sector is so full of cheats and fly-by-nights, that the total cost of building a reliable and long lasting Unix server simply approaches or exceeds the cost of an equivalent mainframe solution.

Self-healing (ZFS) can make unreliable flash storage reliable.

[MeatPopsicle]

Compiles fine but then segafaults.

There's logic issues with the mmap'ing.

[MeatPopsicle]

ext3, and pretty sure it's crashing from a null ptr. Always dies once you try to operate on the last element in vecMap.

[MeatPopsicle]

When I didn't have any blockchain files it would crash on unmapping somewhere in cleanMaps, but with blockchain data it dies on:

Code:

while(i!=e) totalSize += (i++)->size;

in initHashtables.

Code:

ovrlrdq@thedarkcitadel:/tmp/blockparser$ ./parser stats

info: starting command "simpleStats"
Segmentation fault (core dumped)
ovrlrdq@thedarkcitadel:/tmp/blockparser$ gdb parser core
GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /tmp/blockparser/parser...done.
[New LWP 20039]

warning: Can't read pathname for load map: Input/output error.

warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fffb238a000
Core was generated by `./parser stats'.
Program terminated with signal 11, Segmentation fault.
#0  0x000000000044f2d3 in google::dense_hashtable<std::pair<unsigned char const* const, unsigned char const*>, unsigned char const*, Hash256Hasher, google::dense_hash_map<unsigned char const*, unsigned char const*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const, unsigned char const*> > >::SelectKey, google::dense_hash_map<unsigned char const*, unsigned char const*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const, unsigned char const*> > >::SetKey, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const, unsigned char const*> > >::dense_hashtable(google::dense_hashtable<std::pair<unsigned char const* const, unsigned char const*>, unsigned char const*, Hash256Hasher, google::dense_hash_map<unsigned char const*, unsigned char const*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const, unsigned char const*> > >::SelectKey, google::dense_hash_map<unsigned char const*, unsigned char const*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const, unsigned char const*> > >::SetKey, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const, unsigned char const*> > > const&, unsigned long) ()
(gdb) bt
#0  0x000000000044f2d3 in google::dense_hashtable<std::pair<unsigned char const* const, unsigned char const*>, unsigned char const*, Hash256Hasher, google::dense_hash_map<unsigned char const*, unsigned char const*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const, unsigned char const*> > >::SelectKey, google::dense_hash_map<unsigned char const*, unsigned char const*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const, unsigned char const*> > >::SetKey, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const, unsigned char const*> > >::dense_hashtable(google::dense_hashtable<std::pair<unsigned char const* const, unsigned char const*>, unsigned char const*, Hash256Hasher, google::dense_hash_map<unsigned char const*, unsigned char const*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const, unsigned char const*> > >::SelectKey, google::dense_hash_map<unsigned char const*, unsigned char const*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const, unsigned char const*> > >::SetKey, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const, unsigned char const*> > > const&, unsigned long) ()
#1  0x000000000044b8fc in initHashtables() ()
#2  0x0000000000404c4c in main ()

Valgrind output:

Code:

info: starting command "simpleStats"
==22223== Invalid write of size 8
==22223==    at 0x44EA23: google::dense_hashtable<std::pair<unsigned char const* const, unsigned char const*>, unsigned char const*, Hash256Hasher, google::dense_ha
sh_map<unsigned char const*, unsigned char const*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const, unsigned c
har const*> > >::SelectKey, google::dense_hash_map<unsigned char const*, unsigned char const*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std:
:pair<unsigned char const* const, unsigned char const*> > >::SetKey, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const, unsigne
d char const*> > >::dense_hashtable(google::dense_hashtable<std::pair<unsigned char const* const, unsigned char const*>, unsigned char const*, Hash256Hasher, google
::dense_hash_map<unsigned char const*, unsigned char const*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const,
unsigned char const*> > >::SelectKey, google::dense_hash_map<unsigned char const*, unsigned char const*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_re
alloc<std::pair<unsigned char const* const, unsigned char const*> > >::SetKey, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* cons
t, unsigned char const*> > > const&, unsigned long) (in /tmp/blockparser/parser)
==22223==    by 0x44B76D: initHashtables() (in /tmp/blockparser/parser)
==22223==    by 0x404C4B: main (in /tmp/blockparser/parser)
==22223==  Address 0x10 is not stack'd, malloc'd or (recently) free'd
==22223==
==22223==
==22223== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==22223==  Access not within mapped region at address 0x10
==22223==    at 0x44EA23: google::dense_hashtable<std::pair<unsigned char const* const, unsigned char const*>, unsigned char const*, Hash256Hasher, google::dense_ha
sh_map<unsigned char const*, unsigned char const*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const, unsigned c
har const*> > >::SelectKey, google::dense_hash_map<unsigned char const*, unsigned char const*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std:
:pair<unsigned char const* const, unsigned char const*> > >::SetKey, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const, unsigne
d char const*> > >::dense_hashtable(google::dense_hashtable<std::pair<unsigned char const* const, unsigned char const*>, unsigned char const*, Hash256Hasher, google
::dense_hash_map<unsigned char const*, unsigned char const*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* const,
unsigned char const*> > >::SelectKey, google::dense_hash_map<unsigned char const*, unsigned char const*, Hash256Hasher, Hash256Equal, google::libc_allocator_with_re
alloc<std::pair<unsigned char const* const, unsigned char const*> > >::SetKey, Hash256Equal, google::libc_allocator_with_realloc<std::pair<unsigned char const* cons
t, unsigned char const*> > > const&, unsigned long) (in /tmp/blockparser/parser)
==22223==    by 0x44B76D: initHashtables() (in /tmp/blockparser/parser)
==22223==    by 0x404C4B: main (in /tmp/blockparser/parser)

[fluffypony]

I'd like to re-open https://github.com/znort987/blockparser/issues/1 - I'm getting the same problem when running an allBalances yesterday. Alternatively, if someone could do an allBalances for me as a precursor to solving the problem that would be greatly appreciated.

[hyh]

I saw tons of likely/unlikely in the code, what do they do?

Code:

    #if defined(__GNUC__)
        #define likely(x)   __builtin_expect((x), 1)
        #define unlikely(x) __builtin_expect((x), 0)
        #define ALWAYS_INLINE __attribute__ ((always_inline))
    #else
        #define likely(x)   (x)
        #define unlikely(x) (x)
        #define ALWAYS_INLINE
    #endif

I searched a bit and it says something like make a prediction etc.

Is this the dirty part to make the code fast?

How can I know if it works on my machine or not?

(I mean, how do I know which one is taken in the definition, #if defined(__GNUC__) or #else?)