[ANNOUNCE] Abe 0.7: Open Source Block Explorer Knockoff

[unick]

Is there a reason why my questions don't get answered ?

Two possible reasons: bad timing (I am busy) and no obvious answer.

ok thanks, no offense, it seemed as my questions were overlooked, but I understand 

Is there a reason why my questions don't get answered ?

You have several questions, and it is not clear what you are currently using in terms of configuration and modifications to the code.  You mentioned commenting out the Merkle root check, but your latest error shows it still occurring.  Are you able to load blocks without the check?

I have converted your hex dump back to binary, commented out the Merkle root check, installed the ltc_scrypt module, and loaded the block with this command:

Code:

python -m Abe.abe --dbtype sqlite3 --connect-args :memory: --port 2750 --datadir '[{"dirname":".", "chain":"GrowthCoin", "policy":"NovaCoin"}]'

I wasn't able to load after commenting out the merkle root check, I'll give it a shot like you did, maybe I passed too many parameters on datadir option.

[1714329814]

1714329814

[1714329814]

1714329814

[1714329814]

1714329814

[unick]

Ok I'm still trying to figure this thing out... I've manage to get passed the invalid address error (not sure why but I tried putting the chain specifcation into the config file instead of the python file).  But now moving forward on this I get yet another error!

Code:

root@tmp:~/bitcoin-abe# python -m Abe.abe --config /home/abe/abe-mysql.conf --commit-bytes 100000 --no-serve
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/root/bitcoin-abe/Abe/abe.py", line 2276, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/root/bitcoin-abe/Abe/abe.py", line 2270, in main
    store = make_store(args)
  File "/root/bitcoin-abe/Abe/abe.py", line 153, in make_store
    store = DataStore.new(args)
  File "Abe/DataStore.py", line 3446, in new
    return DataStore(args)
  File "Abe/DataStore.py", line 165, in __init__
    store.init_conn()
  File "Abe/DataStore.py", line 226, in init_conn
    store.conn = store.connect()
  File "Abe/DataStore.py", line 237, in connect
    conn = store._connect(cargs)
  File "Abe/DataStore.py", line 269, in _connect
    return store.module.connect(**cargs)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
    return Connection(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 187, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
_mysql_exceptions.OperationalError: (1045, "Access denied for user 'abe'@'localhost' (using password: YES)")
root@tmp:~/bitcoin-abe# nano /home/abe/abe-mysql.conf
root@tmp:~/bitcoin-abe# python -m Abe.abe --config /home/abe/abe-mysql.conf --commit-bytes 100000 --no-serve
ddl_implicit_commit=true
create_table_epilogue=''
Abe/DataStore.py:536: Warning: Converting column 'a' from VARCHAR to TEXT
  store.cursor.execute(stmt)
Abe/DataStore.py:536: Warning: Converting column 'b' from VARCHAR to TEXT
  store.cursor.execute(stmt)
max_varchar=4294967295
clob_type=LONGTEXT
Abe/DataStore.py:536: Warning: Converting column 'test_varbit' from VARCHAR to TEXT
  store.cursor.execute(stmt)
binary_type=hex
int_type=int
Created silly table abe_dual
sequence_type=mysql
limit_style=native
Abe/DataStore.py:536: Warning: Converting column 'txout_scriptPubKey' from VARCHAR to TEXT
  store.cursor.execute(stmt)
Abe/DataStore.py:536: Warning: Converting column 'txin_scriptSig' from VARCHAR to TEXT
  store.cursor.execute(stmt)
Assigned chain_id 1 to GrowthCoin
Opened /root/.GrowthCoin/blk0001.dat
Chain 1 genesis tx: 020000001d0bfa51010000000000000000000000000000000000000000000000000000000000000000ffffffffb904ffff001d020f274caf4a756c792033312c2032303133203132706d204544543a2054686520552e532e2065636f6e6f6d79206772657720312e372520696e20746865207365636f6e6420717561727465722c20616964656420627920736f6c696420636f6e73756d6572207370656e64696e6720616e6420612072616d702d757020696e20627573696e65737320696e766573746d656e742c2074686520676f7665726e6d656e742073616964205765646e65736461792effffffff0100000000000000000000000000
Exception at 368
Failed to catch up {'blkfile_offset': 0, 'blkfile_number': 1, 'chain_id': 1L, 'loader': None, 'conf': u'GrowthCoin.conf', 'dirname': u'/root/.GrowthCoin', 'id': 1L}
Traceback (most recent call last):
  File "Abe/DataStore.py", line 2687, in catch_up
    store.catch_up_dir(dircfg)
  File "Abe/DataStore.py", line 2951, in catch_up_dir
    store.import_blkdat(dircfg, ds, blkfile['name'])
  File "Abe/DataStore.py", line 3083, in import_blkdat
    store.import_block(b, chain = chain)
  File "Abe/DataStore.py", line 1805, in import_block
    raise MerkleRootMismatch(b['hash'], tx_hash_array)
MerkleRootMismatch: Block header Merkle root does not match its transactions. block hash=d5e4fc54c53a9a20bbe01e70d8b47ce66f15c1bd3504f7e80390bf5601150ded

I've read to comment out in DataStore.py

Code:

# Verify Merkle root.
        #if b['hashMerkleRoot'] != util.merkle(tx_hash_array):
        #raise MerkleRootMismatch(b['hash'], tx_hash_array)

But as you can image if I post this... it didn't work!

It gave me a few of these

Code:

Skipped 73 bytes at block end
block_tx 970 970
Skipped 71 bytes at block end
block_tx 971 971
Skipped 72 bytes at block end
block_tx 972 972
Skipped 72 bytes at block end
Exception at 1677721600398062
Failed to catch up {'blkfile_offset': 206488, 'blkfile_number': 1, 'chain_id': 1, 'loader': None, 'conf': u'GrowthCoin.conf', 'dirname': u'/root/.GrowthCoin', 'id': 2L}
Traceback (most recent call last):

Then

Code:

error: unpack_from requires a buffer of at least 4 bytes

Witch I had earlier in my attempts to get Abe read the blockchain.  It looks to me as the magic bytes are causing troubles again.

Any inputs/help/solutions on this would be appreciated  

Ok tried what you said John, but I reposted that quote since I get the same message as previously when I tryed to comment out the MerkleRoot check.

It goes pass the 370th byte but fails at byte 1677721600398062

[dasource]

Shows only the first block. The thing I do not understand.

When Abe loads many blocks but shows only the genesis block, it means that Abe is using the wrong algorithm to calculate the block header hash as it appears in the next block's hashPrev field.  By default, Abe uses double-SHA256, the same as Bitcoin.  Many coins use scrypt.  I would like to make this selectable by datadir=[{...,"hashPrev":"scrypt"}], but for now it requires a few lines of code.  Look for "NovaCoin" in the sources for an example.

Hi John,
Could you elaborate abit further on this? much appreciated. What few lines of code are needed?

[John Tobey]

Ok tried what you said John, but I reposted that quote since I get the same message as previously when I tryed to comment out the MerkleRoot check.

It goes pass the 370th byte but fails at byte 1677721600398062

"Exception at <impossibly huge file offset>" means an unexpected block format.  Where Abe expects a block size or script length, there is some other data.  Abe interprets it as a huge number, uses it as the blockfile pointer, and reports the error as coming from there.  See: https://github.com/bitcoin-abe/bitcoin-abe/issues/4.  Try either the ppcoin patch or RPC mode.

[John Tobey]

Shows only the first block. The thing I do not understand.

When Abe loads many blocks but shows only the genesis block, it means that Abe is using the wrong algorithm to calculate the block header hash as it appears in the next block's hashPrev field.  By default, Abe uses double-SHA256, the same as Bitcoin.  Many coins use scrypt.  I would like to make this selectable by datadir=[{...,"hashPrev":"scrypt"}], but for now it requires a few lines of code.  Look for "NovaCoin" in the sources for an example.

Hi John,
Could you elaborate abit further on this? much appreciated. What few lines of code are needed?

Come to think of it, if the format is exactly NovaCoin's, you might not need a code change.  In datadir, add "policy":"NovaCoin" along with "chain" etc.

The changes are mostly in Abe/Chain.py: add a class defined like NovaCoin (or CryptoCash) and a corresponding conditional in the create function.  In Abe/DataStore.py, where it says

Code:

    {"chain":"NovaCoin"},
    {"chain":"CryptoCash"},

add a similar line.

Edit: and if it works, post your changes, or better yet, create a GitHub pull request so I can add them.

[dasource]

Shows only the first block. The thing I do not understand.

When Abe loads many blocks but shows only the genesis block, it means that Abe is using the wrong algorithm to calculate the block header hash as it appears in the next block's hashPrev field.  By default, Abe uses double-SHA256, the same as Bitcoin.  Many coins use scrypt.  I would like to make this selectable by datadir=[{...,"hashPrev":"scrypt"}], but for now it requires a few lines of code.  Look for "NovaCoin" in the sources for an example.

Hi John,
Could you elaborate abit further on this? much appreciated. What few lines of code are needed?

Come to think of it, if the format is exactly NovaCoin's, you might not need a code change.  In datadir, add "policy":"NovaCoin" along with "chain" etc.

The changes are mostly in Abe/Chain.py: add a class defined like NovaCoin (or CryptoCash) and a corresponding conditional in the create function.  In Abe/DataStore.py, where it says

Code:

    {"chain":"NovaCoin"},
    {"chain":"CryptoCash"},

add a similar line.

Edit: and if it works, post your changes, or better yet, create a GitHub pull request so I can add them.

Thanks John, will give this a shot. Actually trying this on HiroCoin which is a fork of litecoin.

EDIT : Am I being dumb or have I missed something ?

Code:

ImportError: No module named ltc_scrypt

Here is where I think the issue is (block_id 2) :

Code:

*************************** 1. row ***************************
             block_id: 1
           block_hash: ae94c51e61c87f32a2a6c755df00b1ffa130c73b1bd165aaccaa3f90f8d243a9
        block_version: 1
 block_hashMerkleRoot: b0019d92bc054f7418960c91e252e7d24c77719c7a30128c5f6a827c73095d2a
          block_nTime: 1394723131
          block_nBits: 504365040
         block_nNonce: 1234746574
         block_height: 0
        prev_block_id: NULL
      search_block_id: NULL
     block_chain_work: 0000000000000000000000000000000000000000000000000000000000000000000000100010
       block_value_in: 0
      block_value_out: 40000000000
 block_total_satoshis: 40000000000
  block_total_seconds: 0
block_satoshi_seconds: 0
       block_total_ss: 0
         block_num_tx: 1
   block_ss_destroyed: 0
*************************** 2. row ***************************
             block_id: 2
           block_hash: 54a642a1aabb71a5b9d57a6455ba4ab7f69641e2ebe24ca18dfdc4bd9d98e198
        block_version: 2
 block_hashMerkleRoot: cf42adf0dbda83114f23f3c771700b1daf107aaf57f935f1d6bfcc568f38aa4f
          block_nTime: 1394867933
          block_nBits: 504365040
         block_nNonce: 264378
         block_height: NULL
        prev_block_id: NULL
      search_block_id: NULL
     block_chain_work: NULL
       block_value_in: 0
      block_value_out: 40000000000
 block_total_satoshis: -1
  block_total_seconds: NULL
block_satoshi_seconds: NULL
       block_total_ss: NULL
         block_num_tx: 1
   block_ss_destroyed: NULL

[John Tobey]

Thanks John, will give this a shot. Actually trying this on HiroCoin which is a fork of litecoin.

EDIT : Am I being dumb or have I missed something ?

Code:

ImportError: No module named ltc_scrypt

Install ltc_scrypt as per README.txt:

NovaCoin and CryptoCash support depends on the ltc_scrypt module
available from https://github.com/CryptoManiac/bitcoin-abe (see
README-SCRYPT.txt).

Here is where I think the issue is (block_id 2) :

Code:

*************************** 1. row ***************************
             block_id: 1
           block_hash: ae94c51e61c87f32a2a6c755df00b1ffa130c73b1bd165aaccaa3f90f8d243a9
        block_version: 1
 block_hashMerkleRoot: b0019d92bc054f7418960c91e252e7d24c77719c7a30128c5f6a827c73095d2a
          block_nTime: 1394723131
          block_nBits: 504365040
         block_nNonce: 1234746574
         block_height: 0
        prev_block_id: NULL
      search_block_id: NULL
     block_chain_work: 0000000000000000000000000000000000000000000000000000000000000000000000100010
       block_value_in: 0
      block_value_out: 40000000000
 block_total_satoshis: 40000000000
  block_total_seconds: 0
block_satoshi_seconds: 0
       block_total_ss: 0
         block_num_tx: 1
   block_ss_destroyed: 0
*************************** 2. row ***************************
             block_id: 2
           block_hash: 54a642a1aabb71a5b9d57a6455ba4ab7f69641e2ebe24ca18dfdc4bd9d98e198
        block_version: 2
 block_hashMerkleRoot: cf42adf0dbda83114f23f3c771700b1daf107aaf57f935f1d6bfcc568f38aa4f
          block_nTime: 1394867933
          block_nBits: 504365040
         block_nNonce: 264378
         block_height: NULL
        prev_block_id: NULL
      search_block_id: NULL
     block_chain_work: NULL
       block_value_in: 0
      block_value_out: 40000000000
 block_total_satoshis: -1
  block_total_seconds: NULL
block_satoshi_seconds: NULL
       block_total_ss: NULL
         block_num_tx: 1
   block_ss_destroyed: NULL

If you mean all those NULLs, they result from the incorrect block hash algorithm.  Abe does not see block_id 1 as the parent of block_id 2, so it can not compute chain statistics or even height (block number).  See if ltc_scrypt helps.

[dasource]

<snip>
If you mean all those NULLs, they result from the incorrect block hash algorithm.  Abe does not see block_id 1 as the parent of block_id 2, so it can not compute chain statistics or even height (block number).  See if ltc_scrypt helps.

Ok tried this with ltc_scrypt and policy:novacoin and now get this..
Dies at block 9590 rather than before it was getting to full 26000 .... Same output on the web interface, only shows block 0.

Code:

block_tx 9590 1
Skipped 92 bytes at block end
commit
Exception at 5777995176649507762
Failed to catch up {'blkfile_offset': 1917723, 'blkfile_number': 100000, 'chain_id': 11, 'loader': None, 'conf': None, 'dirname': u'/home/hiro/.hirocoin', 'id': 4L}
Traceback (most recent call last):
  File "Abe/DataStore.py", line 2789, in catch_up
    store.catch_up_dir(dircfg)
  File "Abe/DataStore.py", line 3053, in catch_up_dir
    store.import_blkdat(dircfg, ds, blkfile['name'])
  File "Abe/DataStore.py", line 3175, in import_blkdat
    b = chain.ds_parse_block(ds)
  File "Abe/Chain.py", line 121, in ds_parse_block
    d = Chain.ds_parse_block(chain, ds)
  File "Abe/Chain.py", line 59, in ds_parse_block
    d['transactions'].append(chain.ds_parse_transaction(ds))
  File "Abe/Chain.py", line 118, in ds_parse_transaction
    return deserialize.parse_Transaction(ds, has_nTime=True)
  File "Abe/deserialize.py", line 94, in parse_Transaction
    d['txOut'].append(parse_TxOut(vds))
  File "Abe/deserialize.py", line 67, in parse_TxOut
    d['value'] = vds.read_int64()
  File "Abe/BCDataStream.py", line 72, in read_int64
    def read_int64  (self): return self._read_num('<q')
  File "Abe/BCDataStream.py", line 110, in _read_num
    (i,) = struct.unpack_from(format, self.input, self.read_cursor)
error: unpack_from requires a buffer of at least 8 bytes
Abe initialized.

[dasource]

<snip>
If you mean all those NULLs, they result from the incorrect block hash algorithm.  Abe does not see block_id 1 as the parent of block_id 2, so it can not compute chain statistics or even height (block number).  See if ltc_scrypt helps.

Ok tried this with ltc_scrypt and policy:novacoin and now get this..
Dies at block 9590 rather than before it was getting to full 26000 .... Same output on the web interface, only shows block 0.

Code:

block_tx 9590 1
Skipped 92 bytes at block end
commit
Exception at 5777995176649507762
Failed to catch up {'blkfile_offset': 1917723, 'blkfile_number': 100000, 'chain_id': 11, 'loader': None, 'conf': None, 'dirname': u'/home/hiro/.hirocoin', 'id': 4L}
Traceback (most recent call last):
  File "Abe/DataStore.py", line 2789, in catch_up
    store.catch_up_dir(dircfg)
  File "Abe/DataStore.py", line 3053, in catch_up_dir
    store.import_blkdat(dircfg, ds, blkfile['name'])
  File "Abe/DataStore.py", line 3175, in import_blkdat
    b = chain.ds_parse_block(ds)
  File "Abe/Chain.py", line 121, in ds_parse_block
    d = Chain.ds_parse_block(chain, ds)
  File "Abe/Chain.py", line 59, in ds_parse_block
    d['transactions'].append(chain.ds_parse_transaction(ds))
  File "Abe/Chain.py", line 118, in ds_parse_transaction
    return deserialize.parse_Transaction(ds, has_nTime=True)
  File "Abe/deserialize.py", line 94, in parse_Transaction
    d['txOut'].append(parse_TxOut(vds))
  File "Abe/deserialize.py", line 67, in parse_TxOut
    d['value'] = vds.read_int64()
  File "Abe/BCDataStream.py", line 72, in read_int64
    def read_int64  (self): return self._read_num('<q')
  File "Abe/BCDataStream.py", line 110, in _read_num
    (i,) = struct.unpack_from(format, self.input, self.read_cursor)
error: unpack_from requires a buffer of at least 8 bytes
Abe initialized.

Not sure if that was going to work anyway since its using NovaCoin which HiroCoin is not a fork of.
Trying the method on defined class with :

Code:

class HiroCoin(NvcChain):                
    def __init__(chain, **kwargs):
        chain.name = 'HiroCoin'
        chain.code3 = 'HIC'       
        chain.address_version = "\x28"        
        chain.magic = "\xfe\xc3\xb9\xde"
        Chain.__init__(chain, **kwargs)

    datadir_conf_file_name = "hirocoin.conf"      
    datadir_rpcport = 9347

And adding "policy: HiroCoin" to the datadir made no difference .... it runs through the chain perfectly but again lots of NULLS in the DB.

P.S. HiroCoin is not POS like NovaCoin so not sure if I am going down the wrong path here. Its based around DRK Coin and X11.

[unick]

Update:

So I tried to use ppcoin branch to get Abe to read the GrowthCoin database without success.

I also tried to manually load the genesis transaction without success (I had to hexdump the blk0001.dat file since getblock 0 returned and error in the client, similar to the bitcoin issue).  ABE complained of an odd lenght transaction with import-tx parameter in abe.conf.

So if I load with the blkfile loader and merkleroot check commented out, I get up to 971 blocks then it fails with

Code:

block_tx 969 969
Skipped 72 bytes at block end
block_tx 970 970
Skipped 73 bytes at block end
block_tx 971 971
Skipped 73 bytes at block end
Exception at 266991
Failed to catch up {'blkfile_offset': 264462, 'blkfile_number': 1, 'chain_id': Decimal('8'), 'loader': None, 'dirname': u'/root/.GrowthCoin', 'id': 2L}
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2631, in catch_up
    store.catch_up_dir(dircfg)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2895, in catch_up_dir
    store.import_blkdat(dircfg, ds, blkfile['name'])
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 3022, in import_blkdat
    b = store.parse_block(ds, chain_id, magic, length)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 3053, in parse_block
    d['transactions'].append(deserialize.parse_Transaction(ds))
  File "/usr/local/lib/python2.7/dist-packages/Abe/deserialize.py", line 89, in parse_Transaction
    d['txIn'].append(parse_TxIn(vds))
  File "/usr/local/lib/python2.7/dist-packages/Abe/deserialize.py", line 45, in parse_TxIn
    d['scriptSig'] = vds.read_bytes(vds.read_compact_size())
  File "/usr/local/lib/python2.7/dist-packages/Abe/BCDataStream.py", line 59, in read_bytes
    result = self.input[self.read_cursor:self.read_cursor+length]
MemoryError

With current master version of ABE, If I try to load with rpc, it will work but the frontend crashes as int(ss) is of the NoneType.  I checked the database and I have NULL values after ~3k blocks mark.... So it will load the blocks but it misses some info.

With PPCOIN branch, it was prior to ABE's NovaCoin upgrade so I assumen datadir wasn't implemented yet at that point and it doesn't accept my growthcoin.conf file (it says bitcoin.conf)  still trying to find were can I input the correct conf file but I am not sure it will work since I guess the block format is a lot different from NovaCoin yet it is a descendent of it (and Diamond).

Trying to contact the dev to have more info on that block format

EDIT: does this decription give you any clue if ABE can parse the blockchain data correctly
GrowthCoin - a fork of Diamond, with several algorithms fixed and improved. It is decendent of Novacoin/PPCoin with Pow/PoS. It combines the great features from Luckycoin (random blocks) and Florincoin (transaction message).

[John Tobey]

Folks,

The block format is defined in C++ in a call to the IMPLEMENT_SERIALIZE macro in classes CBlockHeader, CTransaction, CTxIn, and CTxOut.  The relevant sections of Bitcoin source code are:

https://github.com/bitcoin/bitcoin/blob/v0.9.0/src/core.h#L363-369
https://github.com/bitcoin/bitcoin/blob/v0.9.0/src/core.h#L201-205
https://github.com/bitcoin/bitcoin/blob/v0.9.0/src/core.h#L87-89
https://github.com/bitcoin/bitcoin/blob/v0.9.0/src/core.h#L134-135

That is what we translate into Python here:

https://github.com/bitcoin-abe/bitcoin-abe/blob/v0.7.2/Abe/deserialize.py#L163-168
https://github.com/bitcoin-abe/bitcoin-abe/blob/v0.7.2/Abe/deserialize.py#L82-91
https://github.com/bitcoin-abe/bitcoin-abe/blob/v0.7.2/Abe/deserialize.py#L42-45
https://github.com/bitcoin-abe/bitcoin-abe/blob/v0.7.2/Abe/deserialize.py#L65-66

If you can figure out the correspondence between those two sets of code snippets, you can read any chain that uses Bitcoin's serialization facility.

[John Tobey]

With current master version of ABE, If I try to load with rpc, it will work but the frontend crashes as int(ss) is of the NoneType.

My knee-jerk fix for this type of bug is to replace "int(ss)" with "None if ss is None else int(ss)".

[ShadesOfMarble]

I haven't kept up with the development of Abe, but is P2SH/Multisig supported now?

[dasource]

Folks,

The block format is defined in C++ in a call to the IMPLEMENT_SERIALIZE macro in classes CBlockHeader, CTransaction, CTxIn, and CTxOut.  The relevant sections of Bitcoin source code are:

https://github.com/bitcoin/bitcoin/blob/v0.9.0/src/core.h#L363-369
https://github.com/bitcoin/bitcoin/blob/v0.9.0/src/core.h#L201-205
https://github.com/bitcoin/bitcoin/blob/v0.9.0/src/core.h#L87-89
https://github.com/bitcoin/bitcoin/blob/v0.9.0/src/core.h#L134-135

That is what we translate into Python here:

https://github.com/bitcoin-abe/bitcoin-abe/blob/v0.7.2/Abe/deserialize.py#L163-168
https://github.com/bitcoin-abe/bitcoin-abe/blob/v0.7.2/Abe/deserialize.py#L82-91
https://github.com/bitcoin-abe/bitcoin-abe/blob/v0.7.2/Abe/deserialize.py#L42-45
https://github.com/bitcoin-abe/bitcoin-abe/blob/v0.7.2/Abe/deserialize.py#L65-66

If you can figure out the correspondence between those two sets of code snippets, you can read any chain that uses Bitcoin's serialization facility.

Hi John, double checked all that and the DarkCoin/HiroCoin code has had no alterations on any of those classes on IMPLEMENT_SERIALIZE macro.
Any other ideas on what could be the issue?

[dasource]

This is how I setup the class...

Code:

class HiroCoin(LtcScryptChain):          
    def __init__(chain, **kwargs):
        chain.name = 'HiroCoin'
        chain.code3 = 'HIC'       
        chain.address_version = "\x28"        
        chain.magic = "\xfe\xc3\xb9\xde"
        Chain.__init__(chain, **kwargs)

    datadir_conf_file_name = "hirocoin.conf"      
    datadir_rpcport = 9347         

EDIT: This may or may not help, but all the block past block 0 are all in orphan_block table.
Checking the code concludes this is because it cannot see the previous block hash correctly.

[John Tobey]

I haven't kept up with the development of Abe, but is P2SH/Multisig supported now?

Apart from an issue affecting Namecoin and the question whether to include "escrow" outputs in address balances, yes, on the multisig branch, thanks to Jouke's sponsorship.

[John Tobey]

This is how I setup the class...

Code:

class HiroCoin(LtcScryptChain):          
    def __init__(chain, **kwargs):
        chain.name = 'HiroCoin'
        chain.code3 = 'HIC'       
        chain.address_version = "\x28"        
        chain.magic = "\xfe\xc3\xb9\xde"
        Chain.__init__(chain, **kwargs)

    datadir_conf_file_name = "hirocoin.conf"      
    datadir_rpcport = 9347         

EDIT: This may or may not help, but all the block past block 0 are all in orphan_block table.
Checking the code concludes this is because it cannot see the previous block hash correctly.

Please see: https://bitcointalk.org/index.php?topic=22785.msg5785774#msg5785774 (and note that "policy":"NovaCoin" has an effect similar to the proposed "hashPrev":"scrypt").

If the algorithm is neither Bitcoin's nor NovaCoin's, we have to override block_header_hash in the new class:

Code:

class Sha256Chain(Chain):
    def block_header_hash(chain, header):
        return util.double_sha256(header)

Code:

class LtcScryptChain(Chain):
    def block_header_hash(chain, header):
        import ltc_scrypt
        return ltc_scrypt.getPoWHash(header)

[molecular]

I have a question about mempool transactions regarding performance:

So I'm running http://blockexplorer.auroracoin.eu and because I have allocated quite a machine to the task everything has been zappy and fine this morning.

However when I checked back from work an hour later I saw loads of exceptions saying "error: [Errno 32] Broken pipe", the nginx I have in front reporting gateway timeouts. I'm hypothesizing the db queries are the bottleneck.

I tried rebuilding the database (drop it completely and rebuild)... that didn't help, it started again right away.

There are loads of mempool transactions in AURoracoin because we're being pool-hopping-attacked to the point where there hasn't been a block for 6 hours or so.

Another instance I run with same setup, but on quite weak machine (vm) was able to cope with the load quite well and didn't suffer broken pipes.

What fixed it was to re-initialize BOTH the blockchain of the AuroraCoind AND the db (just db didn't help).

Now the question: how are mempool transactions handled and could the existance of many mempool transactions have considerable impact on db (or abe.py) performance?

I'm a bit confused... does anyone have an idea what could've been causing this?

[jeffmthomas]

Just posted a bounty for Memorycoin

https://bountify.co/get-memorycoin-working-with-bitcoin-abe

[John Tobey]

I have a question about mempool transactions regarding performance:

So I'm running http://blockexplorer.auroracoin.eu and because I have allocated quite a machine to the task everything has been zappy and fine this morning.

However when I checked back from work an hour later I saw loads of exceptions saying "error: [Errno 32] Broken pipe", the nginx I have in front reporting gateway timeouts. I'm hypothesizing the db queries are the bottleneck.

I tried rebuilding the database (drop it completely and rebuild)... that didn't help, it started again right away.

There are loads of mempool transactions in AURoracoin because we're being pool-hopping-attacked to the point where there hasn't been a block for 6 hours or so.

Another instance I run with same setup, but on quite weak machine (vm) was able to cope with the load quite well and didn't suffer broken pipes.

What fixed it was to re-initialize BOTH the blockchain of the AuroraCoind AND the db (just db didn't help).

Now the question: how are mempool transactions handled and could the existance of many mempool transactions have considerable impact on db (or abe.py) performance?

I'm a bit confused... does anyone have an idea what could've been causing this?

It could be transactions being left open, suboptimal SQL, or   If you have a collection of Python stack traces or database process lists from during the timeouts, they might point to the offender(s).