[ANNOUNCE] Abe 0.7: Open Source Block Explorer Knockoff

[diatonic]

I was able to get it to run from Python with Bitgems & BottleCaps (NovaCoin/PPcoin forks) this way. Thought I would post it here in hopes that it helps someone else.

Check out https://github.com/CryptoManiac/bitcoin-abe

My abe.conf:

Code:

dbtype MySQLdb
connect-args {"user":"abe","db":"btg","passwd":"yourpassword"}
upgrade
port 2750
host 70.98.114.237
address-history-rows-max 10000
datadir = [{
        "dirname": "/home/chillman/.bitgem",
        "chain":   "Bitgem",
        "code3":   "BTG",
    "address_version":"\u0062",
    "magic":"\u00e4\u00e8\u00e9\u00e5"
     }]

Find/Replace /Bitgems/NovaCoin/ in abe.py (four instances)

cd litecoin_scrypt and run:

python setup.py install

After all that it should work  I'd like to run it from FastCGI but I think it would take some refactoring to get it to run along side standard Abe.

You can check mine out at http://btg.webboise.com:2750/chain/Bitgem and http://bot.webboise.com:2760/chain/Bottlecaps

[John Tobey]

I was able to get it to run from Python with Bitgems & BottleCaps (NovaCoin/PPcoin forks) this way. Thought I would post it here in hopes that it helps someone else.

Great, thanks!

[Jouke]

I am trying to get Abe to connect through RPC, but I am getting the following error. The initial database loading was trough blkfile.

2013-06-27 23:39:43,338 [5344:MainThread] DataStore DEBUG - no chain_id
2013-06-27 23:39:43,339 [5344:MainThread] DataStore ERROR - Failed to catch up {'blkfile_offset': 40774615, 'blkfile_number': 100068, 'chain_id': None, 'loader': None, 'dirname': '/home/node/.bitcoin-fullindex', 'id': Decimal('1')}
Traceback (most recent call last):
  File "/home/node/bitcoin-abe/bitcoin-abe-master/Abe/DataStore.py", line 2594, in catch_up
    raise Exception("RPC load failed")
Exception: RPC load failed

[John Tobey]

2013-06-27 23:39:43,338 [5344:MainThread] DataStore DEBUG - no chain_id

Does datadir have "chain":"Bitcoin"?

[diatonic]

For some crazy reason I'm trying to load a ton of chains in Abe. It seems to error after 25 chains. See http://alt.webboise.com:2750/chains

This is an example (trying to add CHNCoin)

Code:

Opened /home/diatonic/.chncoin/blk0001.dat
Exception at 8
Failed to catch up {'blkfile_offset': 0, 'blkfile_number': 1, 'chain_id': Decimal('29'), 'loader': None, 'dirname': u'/home/diatonic/.chncoin', 'id': 84L}
Traceback (most recent call last):
  File "Abe/DataStore.py", line 2596, in catch_up
    store.catch_up_dir(dircfg)
  File "Abe/DataStore.py", line 2854, in catch_up_dir
    store.import_blkdat(dircfg, ds, blkfile['name'])
  File "Abe/DataStore.py", line 2978, in import_blkdat
    if not store.offer_existing_block(hash, chain_id):
  File "Abe/DataStore.py", line 2465, in offer_existing_block
    store.offer_block_to_chains(b, frozenset([chain_id]))
  File "Abe/DataStore.py", line 2349, in offer_block_to_chains
    b['top'] = store.adopt_orphans(b, 0, chain_ids, chain_ids)
  File "Abe/DataStore.py", line 2015, in adopt_orphans
    if b['total_ss'] is None or b['satoshis'] is None:
KeyError: 'total_ss'

[John Tobey]

Code:

  File "Abe/DataStore.py", line 2465, in offer_existing_block
    store.offer_block_to_chains(b, frozenset([chain_id]))
  File "Abe/DataStore.py", line 2349, in offer_block_to_chains
    b['top'] = store.adopt_orphans(b, 0, chain_ids, chain_ids)
  File "Abe/DataStore.py", line 2015, in adopt_orphans
    if b['total_ss'] is None or b['satoshis'] is None:
KeyError: 'total_ss'

Fixed, please retry with the latest.  Thanks.

[diatonic]

I really appreciate your help with this.

I updated, but now I get a different error:

Code:

Opened /home/diatonic/.chncoin/blk0001.dat
Exception at 8
Failed to catch up {'blkfile_offset': 0, 'blkfile_number': 1, 'chain_id': Decimal('30'), 'loader': None, 'dirname': u'/home/diatonic/.chncoin', 'id': 92L}
Traceback (most recent call last):
  File "Abe/DataStore.py", line 2599, in catch_up
    store.catch_up_dir(dircfg)
  File "Abe/DataStore.py", line 2857, in catch_up_dir
    store.import_blkdat(dircfg, ds, blkfile['name'])
  File "Abe/DataStore.py", line 2981, in import_blkdat
    if not store.offer_existing_block(hash, chain_id):
  File "Abe/DataStore.py", line 2468, in offer_existing_block
    store.offer_block_to_chains(b, frozenset([chain_id]))
  File "Abe/DataStore.py", line 2350, in offer_block_to_chains
    b['top'] = store.adopt_orphans(b, 0, chain_ids, chain_ids)
  File "Abe/DataStore.py", line 2097, in adopt_orphans
    next_ret = store.adopt_orphans(nb, new_work, None, chain_mask)
  File "Abe/DataStore.py", line 2029, in adopt_orphans
    util.get_search_height(height), int(block_id))
  File "Abe/util.py", line 91, in get_search_height
    if n & 1:
TypeError: unsupported operand type(s) for &: 'Decimal' and 'int'

[John Tobey]

I really appreciate your help with this.

I updated, but now I get a different error:

Code:

Opened /home/diatonic/.chncoin/blk0001.dat
Exception at 8
Failed to catch up {'blkfile_offset': 0, 'blkfile_number': 1, 'chain_id': Decimal('30'), 'loader': None, 'dirname': u'/home/diatonic/.chncoin', 'id': 92L}
Traceback (most recent call last):
  File "Abe/DataStore.py", line 2599, in catch_up
    store.catch_up_dir(dircfg)
  File "Abe/DataStore.py", line 2857, in catch_up_dir
    store.import_blkdat(dircfg, ds, blkfile['name'])
  File "Abe/DataStore.py", line 2981, in import_blkdat
    if not store.offer_existing_block(hash, chain_id):
  File "Abe/DataStore.py", line 2468, in offer_existing_block
    store.offer_block_to_chains(b, frozenset([chain_id]))
  File "Abe/DataStore.py", line 2350, in offer_block_to_chains
    b['top'] = store.adopt_orphans(b, 0, chain_ids, chain_ids)
  File "Abe/DataStore.py", line 2097, in adopt_orphans
    next_ret = store.adopt_orphans(nb, new_work, None, chain_mask)
  File "Abe/DataStore.py", line 2029, in adopt_orphans
    util.get_search_height(height), int(block_id))
  File "Abe/util.py", line 91, in get_search_height
    if n & 1:
TypeError: unsupported operand type(s) for &: 'Decimal' and 'int'

Fixed, please try again.  Sorry if there are more errors, this code path is untested.  The lack of a test suite is what keeps me from calling it 1.0.

[diatonic]

Thanks again for the help tracking this down.

New errors!

Code:

Opened /home/diatonic/.chncoin/blk0001.dat
Exception at 8
Failed to catch up {'blkfile_offset': 0, 'blkfile_number': 1, 'chain_id': Decimal('30'), 'loader': None, 'dirname': u'/home/diatonic/.chncoin', 'id': 98L}
Traceback (most recent call last):
  File "Abe/DataStore.py", line 2599, in catch_up
    store.catch_up_dir(dircfg)
  File "Abe/DataStore.py", line 2857, in catch_up_dir
    store.import_blkdat(dircfg, ds, blkfile['name'])
  File "Abe/DataStore.py", line 2981, in import_blkdat
    if not store.offer_existing_block(hash, chain_id):
  File "Abe/DataStore.py", line 2468, in offer_existing_block
    store.offer_block_to_chains(b, frozenset([chain_id]))
  File "Abe/DataStore.py", line 2350, in offer_block_to_chains
    b['top'] = store.adopt_orphans(b, 0, chain_ids, chain_ids)
  File "Abe/DataStore.py", line 2097, in adopt_orphans
    next_ret = store.adopt_orphans(nb, new_work, None, chain_mask)
[deleted about 1000 occurrnaces of the last 2 lines repeating]
....
  File "Abe/DataStore.py", line 2097, in adopt_orphans
    next_ret = store.adopt_orphans(nb, new_work, None, chain_mask)
  File "Abe/DataStore.py", line 2029, in adopt_orphans
    util.get_search_height(height), int(block_id))
  File "Abe/DataStore.py", line 1668, in get_block_id_at_height
    block = store._load_block(descendant_id)
  File "Abe/DataStore.py", line 1654, in _load_block
    WHERE block_id = ?""", (block_id,))
  File "Abe/DataStore.py", line 636, in selectrow
    store.sql(stmt, params)
  File "Abe/DataStore.py", line 508, in sql
    store._execute(cached, params)
  File "Abe/DataStore.py", line 485, in _execute
    store.cursor.execute(stmt, params)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
RuntimeError: maximum recursion depth exceeded while calling a Python object

[Jouke]

2013-06-27 23:39:43,338 [5344:MainThread] DataStore DEBUG - no chain_id

Does datadir have "chain":"Bitcoin"?

Code:

datadir += [{
        "chain": "Bitcoin",
        "dirname": "/home/node/.bitcoin-fullindex",
        "loader": "rpc"    # See the comments for default-loader below.
      }]

[John Tobey]

I am trying to get Abe to connect through RPC, but I am getting the following error. The initial database loading was trough blkfile.

2013-06-27 23:39:43,338 [5344:MainThread] DataStore DEBUG - no chain_id
2013-06-27 23:39:43,339 [5344:MainThread] DataStore ERROR - Failed to catch up {'blkfile_offset': 40774615, 'blkfile_number': 100068, 'chain_id': None, 'loader': None, 'dirname': '/home/node/.bitcoin-fullindex', 'id': Decimal('1')}
Traceback (most recent call last):
  File "/home/node/bitcoin-abe/bitcoin-abe-master/Abe/DataStore.py", line 2594, in catch_up
    raise Exception("RPC load failed")
Exception: RPC load failed

Run this SQL and see if it is fixed:

Code:

UPDATE datadir SET chain_id = 1, loader = 'rpc' WHERE datadir_id = 1;

It's a bug, Abe should take the chain and loader from the config, not the database.

[diatonic]

The errors I posted above happen when I try to load Litecoin, CHNCoin or GameCoin. I've got about 27 chains loaded now I think at http://altcha.in .

[John Tobey]

The errors I posted above happen when I try to load Litecoin, CHNCoin or GameCoin. I've got about 27 chains loaded now I think at http://altcha.in .

I am working on a fix for this old bug.

[John Tobey]

@diatonic Please try the latest code and report back.

The errors I posted above happen when I try to load Litecoin, CHNCoin or GameCoin. I've got about 27 chains loaded now I think at http://altcha.in .

I am working on a fix for this old bug.

[John Tobey]

where you able to find out what was wrong with testnet3?

Not yet, thanks for the reminder.

I think this is fixed in the latest commit (ecfe350).  Thanks for the report.

[Jouke]

I am trying to get Abe to connect through RPC, but I am getting the following error. The initial database loading was trough blkfile.

2013-06-27 23:39:43,338 [5344:MainThread] DataStore DEBUG - no chain_id
2013-06-27 23:39:43,339 [5344:MainThread] DataStore ERROR - Failed to catch up {'blkfile_offset': 40774615, 'blkfile_number': 100068, 'chain_id': None, 'loader': None, 'dirname': '/home/node/.bitcoin-fullindex', 'id': Decimal('1')}
Traceback (most recent call last):
  File "/home/node/bitcoin-abe/bitcoin-abe-master/Abe/DataStore.py", line 2594, in catch_up
    raise Exception("RPC load failed")
Exception: RPC load failed

Run this SQL and see if it is fixed:

Code:

UPDATE datadir SET chain_id = 1, loader = 'rpc' WHERE datadir_id = 1;

It's a bug, Abe should take the chain and loader from the config, not the database.

In the datadir table I have the following columns:
datadir_id
dirname
blkfile_number
blkfile_number
blkfile_offset
chain_id
datadir_loader


How does Abe keep track of blockfiles and blockfile offsets when it is updating from RPC?

[diatonic]

@diatonic Please try the latest code and report back.

The errors I posted above happen when I try to load Litecoin, CHNCoin or GameCoin. I've got about 27 chains loaded now I think at http://altcha.in .

I am working on a fix for this old bug.

Thanks John, I really appreciate the time you've put in to this. I was able to load litecoin by itself in a fresh database, so I dropped the abe database and loaded from scratch. CHNcoin loaded, but I got errors when litecoin loaded. Perhaps they share a genesis block or something that is leading to the duplicate key error I'm seeing:

Code:

failed to load /home/diatonic/.litecoin/bitcoin.conf: [Errno 2] No such file or directory: u'/home/diatonic/.litecoin/bitcoin.conf'
catch_up_rpc: abort
Opened /home/diatonic/.litecoin/blk0001.dat
Exception at 8
Failed to catch up {'blkfile_offset': 0, 'blkfile_number': 1, 'chain_id': Decimal('19'), 'loader': None, 'dirname': u'/home/diatonic/.litecoin', 'id': 33L}
Traceback (most recent call last):
  File "Abe/DataStore.py", line 2632, in catch_up
    store.catch_up_dir(dircfg)
  File "Abe/DataStore.py", line 2890, in catch_up_dir
    store.import_blkdat(dircfg, ds, blkfile['name'])
  File "Abe/DataStore.py", line 3014, in import_blkdat
    if not store.offer_existing_block(hash, chain_id):
  File "Abe/DataStore.py", line 2501, in offer_existing_block
    store.offer_block_to_chains(b, frozenset([chain_id]))
  File "Abe/DataStore.py", line 2383, in offer_block_to_chains
    b['top'] = store.adopt_orphans(b, 0, chain_ids, chain_ids)
  File "Abe/DataStore.py", line 1970, in adopt_orphans
    stack.pop()()
  File "Abe/DataStore.py", line 1975, in doit
    store._adopt_orphans_1(stack)
  File "Abe/DataStore.py", line 2086, in _adopt_orphans_1
    store._populate_block_txin(int(next_id))
  File "Abe/DataStore.py", line 1919, in _populate_block_txin
    (block_id, txin_id, oblock_id))
  File "Abe/DataStore.py", line 508, in sql
    store._execute(cached, params)
  File "Abe/DataStore.py", line 485, in _execute
    store.cursor.execute(stmt, params)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
IntegrityError: (1062, "Duplicate entry '577171-1214092' for key 'PRIMARY'")

I'll probably just pick one of those chains. I didn't suspect trying to get 30 chains loaded in Abe would require you to fix a bunch of stuff

[John Tobey]

Run this SQL and see if it is fixed:

Code:

UPDATE datadir SET chain_id = 1, loader = 'rpc' WHERE datadir_id = 1;

In the datadir table I have the following columns:
datadir_id
dirname
blkfile_number
blkfile_number
blkfile_offset
chain_id
datadir_loader

Whoops, I guess I meant:
UPDATE datadir SET chain_id = 1, datadir_loader = 'rpc' WHERE datadir_id = 1;

How does Abe keep track of blockfiles and blockfile offsets when it is updating from RPC?

Putting chain_id and datadir_loader in the table was bad design.  I think they get their values from the "chain" and "loader" (or "default-loader") config elements when the datadir is first used, and thereafter, the config values are ignored.  I plan to make it use the config values only, but meanwhile, I think you can copy the correct values into the table row, and it will work.

[diatonic]

In trying to find out the cause of my issue, I found that LTC, FTC, CNC & GME all have the same genesis block hash. Would that cause my issue?

Line 32 of src/main.cpp on all of those:

Code:

uint256 hashGenesisBlock("0x12a765e31ffd4059bada1e25190f6e98c99d9714d334efa41a195a7e7e04bfe2");

[John Tobey]

In trying to find out the cause of my issue, I found that LTC, FTC, CNC & GME all have the same genesis block hash. Would that cause my issue?

Yes, probably, but it is a bug in Abe, and I'll try to fix it.  Thanks.