[ANNOUNCE] Abe 0.7: Open Source Block Explorer Knockoff

[John Tobey]

What would be faster, using one DB for all the blocks
or each chain one DB

I guess you should use a separate database per chain.  It complicates the HTTP side but simplifies deleting a corrupt (or abandoned) chain and spreading an increasing load.  I don't know if it would be much faster.

[1714303089]

1714303089

[unick]

Hi John,

I have a quick question (don't know if the answer would be as quick and easy) so here's the deal in a nutshell.

I am trying to setup a block explorer for FairCoin.

it reads the blockchain just fine but on the HTTP side, I can't get it to read the FairCoin chain name. It defaults to NovaCoin no matter what I do.

It seems that the magic number for NovaCoin is the same as the one for FairCoin. could that be it?

how can this be fixed/where should I look?

thanks in advance

[unick]

ok I found where NovaCoin took it's config from in Chain.py... and like I said they both have the same magic number so I hardcoded the FairCoin names into that file, witch I find is the unelegant way to go but it works for now... If you have a better solution, I would love to know

edit: well after putting more thoughts to it... I just added a conditional statement for the FairCoin name, it's a bit more clean this way... still if there is a better solution (as an update to the code will wipe this setting) let me know (or maybe update the github with this conditional statement)

[elcoco]

I'm trying to run a blockchain explorer for HeavyCoin, by using the newest bitcoin-abe, but i keep getting this error
Opened /home/azureuser/.heavycoin/blocks/blk00000.dat
Exception at 433754408174527358
Failed to catch up {'blkfile_offset': 40726, 'blkfile_number': 100000, 'chain_id': 19, 'loader': u'blkfile', 'conf': None, 'dirname': '/home/azureuser/.heavycoin', 'id': Decimal('84')}
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2686, in catch_up
    store.catch_up_dir(dircfg)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2950, in catch_up_dir
    store.import_blkdat(dircfg, ds, blkfile['name'])
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 3072, in import_blkdat
    b = chain.ds_parse_block(ds)
  File "/usr/local/lib/python2.7/dist-packages/Abe/Chain.py", line 59, in ds_parse_block
    d['transactions'].append(chain.ds_parse_transaction(ds))
  File "/usr/local/lib/python2.7/dist-packages/Abe/Chain.py", line 52, in ds_parse_transaction
    return deserialize.parse_Transaction(ds)
  File "/usr/local/lib/python2.7/dist-packages/Abe/deserialize.py", line 90, in parse_Transaction
    d['txIn'].append(parse_TxIn(vds))
  File "/usr/local/lib/python2.7/dist-packages/Abe/deserialize.py", line 46, in parse_TxIn
    d['sequence'] = vds.read_uint32()
  File "/usr/local/lib/python2.7/dist-packages/Abe/BCDataStream.py", line 71, in read_uint32
    def read_uint32 (self): return self._read_num('<I')
  File "/usr/local/lib/python2.7/dist-packages/Abe/BCDataStream.py", line 110, in _read_num
    (i,) = struct.unpack_from(format, self.input, self.read_cursor)
error: unpack_from requires a buffer of at least 4 bytes


I installed the ltc_scrypt module, by doing python setup.py install, but i'm not sure what to do next.
at first I was getting
raise MerkleRootMismatch(b['hash'], tx_hash_array)
but I changed lines in     
# Verify Merkle root.
    if b['hashMerkleRoot'] != util.merkle(tx_hash_array):
        raise MerkleRootMismatch(b['hash'], tx_hash_array)
from the Abe/DataStore.py

but now i'm getting the error posted

[jeffmthomas]

So I was finally able to get Dogecoin working.  I was using the wrong magic.  I have everything up on a live server now and it doesn't seem to be polling the database for the queries, instead it is opening up the block file every single time.

I am on the no-statistics branch using this config file:

Code:

dbtype sqlite3
connect-args doge.sqlite
port 2750
host 0.0.0.0
#upgrade
datadir = [{
        "dirname": "/home/ubuntu/.dogecoin",
        "chain": "Dogecoin",
        "code3": "DOGE",
        "address_version": "\u001e",
        "magic": "\u00c0\u00c0\u00c0\u00c0",
        "conf": "dogecoin.conf",
        "trim_depth": 40
        }]
commit-bytes = 100000
keep-scriptsig = false
int-type str

Here is what shows up on the server:

Code:

[13/Mar/2014 02:06:12] "GET /q/addressbalance/D9f7RRygjA7N91TLDoshLwNVoFxqCjvv7i HTTP/1.1" 200 83
failed to load /home/ubuntu/.dogecoin/bitcoin.conf: [Errno 2] No such file or directory: u'/home/ubuntu/.dogecoin/bitcoin.conf'
catch_up_rpc: abort
Opened /home/ubuntu/.dogecoin/blocks/blk00022.dat

[John Tobey]

So I was finally able to get Dogecoin working.  I was using the wrong magic.  I have everything up on a live server now and it doesn't seem to be polling the database for the queries, instead it is opening up the block file every single time.

Please search for "no-serve" in this thread.

[lunalauncher]

I'm transferring Abe from one server to the next. After restoring the MySQL dump from the previous server and starting Abe, the block number no longer increases and it is stuck at block 289302 for Bitcoin. The MySQL dump was made for at block height 289302. The previous server's copy of Abe is able to go past block 289302.

On another pair of servers, the same transfer was made and Ltc had no problems updating the new blocks.

Any idea what has happened?

[unick]

Hi, I'm trying to setup GrowthCoin block explorer.

This is what I get

Code:

root@grw:~/bitcoin-abe# python -m Abe.abe --config /home/abe/abe-mysql.conf --commit-bytes 100000 --no-serve
Opened /root/.GrowthCoin/blk0001.dat
Chain not found for magic number a5efdbfd in block file /root/.GrowthCoin/blk0001.dat at offset 0.

But we can clearly see the firt byte from

Code:

hexdump -n 300 -C /root/.GrowthCoin/blk0001.dat

are correct!

Code:

00000000  a5 ef db fd 69 01 00 00  01 00 00 00 00 00 00 00  |....i...........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 bf f2 79 9b  |..............y.|
00000030  bf ec 52 41 32 ea 94 54  5f 53 83 7c 5c eb 46 26  |..RA2..T_S.|\.F&|
00000040  b6 3d 18 70 76 01 0c cb  96 10 53 14 31 0b fa 51  |.=.pv.....S.1..Q|
00000050  ff ff 0f 1e ba 6e 15 01  01 02 00 00 00 1d 0b fa  |.....n..........|
00000060  51 01 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |Q...............|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000080  00 00 ff ff ff ff b9 04  ff ff 00 1d 02 0f 27 4c  |..............'L|
00000090  af 4a 75 6c 79 20 33 31  2c 20 32 30 31 33 20 31  |.July 31, 2013 1|
000000a0  32 70 6d 20 45 44 54 3a  20 54 68 65 20 55 2e 53  |2pm EDT: The U.S|
000000b0  2e 20 65 63 6f 6e 6f 6d  79 20 67 72 65 77 20 31  |. economy grew 1|
000000c0  2e 37 25 20 69 6e 20 74  68 65 20 73 65 63 6f 6e  |.7% in the secon|
000000d0  64 20 71 75 61 72 74 65  72 2c 20 61 69 64 65 64  |d quarter, aided|
000000e0  20 62 79 20 73 6f 6c 69  64 20 63 6f 6e 73 75 6d  | by solid consum|
000000f0  65 72 20 73 70 65 6e 64  69 6e 67 20 61 6e 64 20  |er spending and |
00000100  61 20 72 61 6d 70 2d 75  70 20 69 6e 20 62 75 73  |a ramp-up in bus|
00000110  69 6e 65 73 73 20 69 6e  76 65 73 74 6d 65 6e 74  |iness investment|
00000120  2c 20 74 68 65 20 67 6f  76 65 72 6e              |, the govern|
0000012c

What did I miss to get this message?

thanks

[Remember remember the 5th of November]

Hi, I'm trying to setup GrowthCoin block explorer.

This is what I get

Code:

root@grw:~/bitcoin-abe# python -m Abe.abe --config /home/abe/abe-mysql.conf --commit-bytes 100000 --no-serve
Opened /root/.GrowthCoin/blk0001.dat
Chain not found for magic number a5efdbfd in block file /root/.GrowthCoin/blk0001.dat at offset 0.

But we can clearly see the firt byte from

Code:

hexdump -n 300 -C /root/.GrowthCoin/blk0001.dat

are correct!

Code:

00000000  a5 ef db fd 69 01 00 00  01 00 00 00 00 00 00 00  |....i...........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 bf f2 79 9b  |..............y.|
00000030  bf ec 52 41 32 ea 94 54  5f 53 83 7c 5c eb 46 26  |..RA2..T_S.|\.F&|
00000040  b6 3d 18 70 76 01 0c cb  96 10 53 14 31 0b fa 51  |.=.pv.....S.1..Q|
00000050  ff ff 0f 1e ba 6e 15 01  01 02 00 00 00 1d 0b fa  |.....n..........|
00000060  51 01 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |Q...............|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000080  00 00 ff ff ff ff b9 04  ff ff 00 1d 02 0f 27 4c  |..............'L|
00000090  af 4a 75 6c 79 20 33 31  2c 20 32 30 31 33 20 31  |.July 31, 2013 1|
000000a0  32 70 6d 20 45 44 54 3a  20 54 68 65 20 55 2e 53  |2pm EDT: The U.S|
000000b0  2e 20 65 63 6f 6e 6f 6d  79 20 67 72 65 77 20 31  |. economy grew 1|
000000c0  2e 37 25 20 69 6e 20 74  68 65 20 73 65 63 6f 6e  |.7% in the secon|
000000d0  64 20 71 75 61 72 74 65  72 2c 20 61 69 64 65 64  |d quarter, aided|
000000e0  20 62 79 20 73 6f 6c 69  64 20 63 6f 6e 73 75 6d  | by solid consum|
000000f0  65 72 20 73 70 65 6e 64  69 6e 67 20 61 6e 64 20  |er spending and |
00000100  61 20 72 61 6d 70 2d 75  70 20 69 6e 20 62 75 73  |a ramp-up in bus|
00000110  69 6e 65 73 73 20 69 6e  76 65 73 74 6d 65 6e 74  |iness investment|
00000120  2c 20 74 68 65 20 67 6f  76 65 72 6e              |, the govern|
0000012c

What did I miss to get this message?

thanks

Maybe it's an endiannes thing?

[John Tobey]

Hi, I'm trying to setup GrowthCoin block explorer.

This is what I get

Code:

root@grw:~/bitcoin-abe# python -m Abe.abe --config /home/abe/abe-mysql.conf --commit-bytes 100000 --no-serve
Opened /root/.GrowthCoin/blk0001.dat
Chain not found for magic number a5efdbfd in block file /root/.GrowthCoin/blk0001.dat at offset 0.

This coin is new to Abe.  Please follow the example "datadir" syntax in abe.conf, and add datadir = [{....}] to your abe-mysql.conf file.  If the currency is scrypt-based, it may need more changes discussed recently in this thread.

[John Tobey]

ok I found where NovaCoin took it's config from in Chain.py... and like I said they both have the same magic number so I hardcoded the FairCoin names into that file, witch I find is the unelegant way to go but it works for now... If you have a better solution, I would love to know

edit: well after putting more thoughts to it... I just added a conditional statement for the FairCoin name, it's a bit more clean this way... still if there is a better solution (as an update to the code will wipe this setting) let me know (or maybe update the github with this conditional statement)

I'd put the FairCoin parameters in abe.conf datadir=[{...}] and not in Python.  That should survive upgrades.  See the datadir examples in abe.conf.

Given a simple directory name (datadir="...") Abe relies on the magic number.  We could support two chains with the same magic number, but we would need some way to distinguish them.  That's why I introduced the expanded syntax for datadir, a JSON array of objects.

[jeffmthomas]

I am attempting to get this running on a postgresql database.  I have followed the instructions to the letter and this is the error that I get:

Code:

admin@ip-10-196-49-5:~$ python -m Abe.abe --config doge.conf --commit-bytes 100000 --no-serve
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/local/lib/python2.7/dist-packages/Abe/abe.py", line 2276, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/usr/local/lib/python2.7/dist-packages/Abe/abe.py", line 2270, in main
    store = make_store(args)
  File "/usr/local/lib/python2.7/dist-packages/Abe/abe.py", line 153, in make_store
    store = DataStore.new(args)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 3445, in new
    return DataStore(args)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 199, in __init__
    store._init_datadirs()
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 753, in _init_datadirs
    dircfg.get('policy', chain_name), decimals))
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 522, in sql
    store._execute(cached, params)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 499, in _execute
    store.cursor.execute(stmt, params)
psycopg2.DataError: value too long for type character(3)

I am on Debain Wheezy running python 2.7 and postgresql 9.1

[John Tobey]

Code:

psycopg2.DataError: value too long for type character(3)

Either shorten your chain's "code3" to 3 characters or widen the database column:

Code:

ALTER TABLE chain ALTER COLUMN chain_code3 TYPE VARCHAR(4);

[unick]

Maybe it's an endiannes thing?

I'm not familiar with the concepet of endiannes. After relooking at the code, I had made a typo on the class name

This coin is new to Abe.  Please follow the example "datadir" syntax in abe.conf, and add datadir = [{....}] to your abe-mysql.conf file.  If the currency is scrypt-based, it may need more changes discussed recently in this thread.

Well I modified the Chain.py file as I did for FairCoin (it worked for FairCoin) so the abe-mysql.conf file would be a better option on the perspective of future upgrades, but the specs of the GrowthCoin blockchain were indeed entered. I will revert this to the config file when I sorted out the issue.

Meanwhile I triple checked the code and found that I made a typo on the class name.

That being said, I have another issue now. This time with the address version

Code:

root@grw:~/bitcoin-abe# python -m Abe.abe --config /home/abe/abe-mysql.conf --commit-bytes 100000 --no-serve
ddl_implicit_commit=true
create_table_epilogue=''
Abe/DataStore.py:536: Warning: Converting column 'a' from VARCHAR to TEXT
  store.cursor.execute(stmt)
Abe/DataStore.py:536: Warning: Converting column 'b' from VARCHAR to TEXT
  store.cursor.execute(stmt)
max_varchar=4294967295
clob_type=LONGTEXT
Abe/DataStore.py:536: Warning: Converting column 'test_varbit' from VARCHAR to TEXT
  store.cursor.execute(stmt)
binary_type=hex
int_type=int
Created silly table abe_dual
sequence_type=mysql
limit_style=native
Abe/DataStore.py:536: Warning: Converting column 'txout_scriptPubKey' from VARCHAR to TEXT
  store.cursor.execute(stmt)
Abe/DataStore.py:536: Warning: Converting column 'txin_scriptSig' from VARCHAR to TEXT
  store.cursor.execute(stmt)
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/root/bitcoin-abe/Abe/abe.py", line 2276, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/root/bitcoin-abe/Abe/abe.py", line 2270, in main
    store = make_store(args)
  File "/root/bitcoin-abe/Abe/abe.py", line 153, in make_store
    store = DataStore.new(args)
  File "Abe/DataStore.py", line 3446, in new
    return DataStore(args)
  File "Abe/DataStore.py", line 181, in __init__
    store.initialize()
  File "Abe/DataStore.py", line 1262, in initialize
    store.insert_chain(chain)
  File "Abe/DataStore.py", line 1301, in insert_chain
    chain.name, chain.decimals))
  File "Abe/DataStore.py", line 523, in sql
    store._execute(cached, params)
  File "Abe/DataStore.py", line 500, in _execute
    store.cursor.execute(stmt, params)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
_mysql_exceptions.OperationalError: (1048, "Column 'chain_address_version' cannot be null")

if I decode the address, this is what I get

Code:

root@grw:~/bitcoin-abe# python -m Abe.abe --query /q/decode_address/GTjYKganLhDtMTcWCinzbQZpsrSjXfB29m
26:6c7c51acb76a0b62c10b6cb17b39cc0982f5d289

and this is my GrowthCoin class

Code:

class GrowthCoin(NvcChain):
    def __init__(chain, **kwargs):
        chain.name = 'GrowthCoin'
        chain.code3 = 'GRW'
        chain.address_version = "\x26"
        chain.magic = "\xa5\xef\xdb\xfd"
        Chain.__init__(chain, **kwargs)

    datadir_conf_file_name = "GrowthCoin.conf"
    datadir_rpcport = 17178

I also added CHAIN_CONFIG = [{"chain":"GrowthCoin"}] in the DataStore.py file...

Everything looks ok, what could be the issue?

[crabel]

I developed a method for analyzing the bitcoin blockchain using very simple analytical tools. While my initial work was with bitcoin it works with altcoins in general. I wrote a post describing the method and theory. In it I posted the MatLab files I used to perform the analysis.

I am not familiar with programing in general. I do not see any reason why these functions and methods can't be incorporated into abe's source code. They provide powerful description of these currencies and are very easily implemented.

I used abe to parse the bitcoin blockchain and greatly appreciate the effort that you put into developing the source code. Please take my work as a thank you for the work that you did.

http://statisticaleconomics.org/2014/03/13/quantifying-the-value-of-bitcoin-2/

[lunalauncher]

Abe is giving me the error

Code:

block_tx 290732 34916394
block_tx 290732 34916395
block_tx 290732 34916396
block_tx 290732 34916397
block_tx 290732 34916398
block_tx 290732 34916399
commit
RPC service lacks full txindex
catch_up_rpc: abort
Opened /root/.bitcoin/blocks/blk00123.dat
RPC service lacks full txindex
catch_up_rpc: abort
Opened /root/.bitcoin/blocks/blk00123.dat
RPC service lacks full txindex
catch_up_rpc: abort
Opened /root/.bitcoin/blocks/blk00123.dat
RPC service lacks full txindex
catch_up_rpc: abort
Opened /root/.bitcoin/blocks/blk00123.dat

Made bitcoind do a reindex but it's still the same. Any ideas?

[unick]

Ok I'm still trying to figure this thing out... I've manage to get passed the invalid address error (not sure why but I tried putting the chain specifcation into the config file instead of the python file).  But now moving forward on this I get yet another error!

Code:

root@tmp:~/bitcoin-abe# python -m Abe.abe --config /home/abe/abe-mysql.conf --commit-bytes 100000 --no-serve
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/root/bitcoin-abe/Abe/abe.py", line 2276, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/root/bitcoin-abe/Abe/abe.py", line 2270, in main
    store = make_store(args)
  File "/root/bitcoin-abe/Abe/abe.py", line 153, in make_store
    store = DataStore.new(args)
  File "Abe/DataStore.py", line 3446, in new
    return DataStore(args)
  File "Abe/DataStore.py", line 165, in __init__
    store.init_conn()
  File "Abe/DataStore.py", line 226, in init_conn
    store.conn = store.connect()
  File "Abe/DataStore.py", line 237, in connect
    conn = store._connect(cargs)
  File "Abe/DataStore.py", line 269, in _connect
    return store.module.connect(**cargs)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
    return Connection(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 187, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
_mysql_exceptions.OperationalError: (1045, "Access denied for user 'abe'@'localhost' (using password: YES)")
root@tmp:~/bitcoin-abe# nano /home/abe/abe-mysql.conf
root@tmp:~/bitcoin-abe# python -m Abe.abe --config /home/abe/abe-mysql.conf --commit-bytes 100000 --no-serve
ddl_implicit_commit=true
create_table_epilogue=''
Abe/DataStore.py:536: Warning: Converting column 'a' from VARCHAR to TEXT
  store.cursor.execute(stmt)
Abe/DataStore.py:536: Warning: Converting column 'b' from VARCHAR to TEXT
  store.cursor.execute(stmt)
max_varchar=4294967295
clob_type=LONGTEXT
Abe/DataStore.py:536: Warning: Converting column 'test_varbit' from VARCHAR to TEXT
  store.cursor.execute(stmt)
binary_type=hex
int_type=int
Created silly table abe_dual
sequence_type=mysql
limit_style=native
Abe/DataStore.py:536: Warning: Converting column 'txout_scriptPubKey' from VARCHAR to TEXT
  store.cursor.execute(stmt)
Abe/DataStore.py:536: Warning: Converting column 'txin_scriptSig' from VARCHAR to TEXT
  store.cursor.execute(stmt)
Assigned chain_id 1 to GrowthCoin
Opened /root/.GrowthCoin/blk0001.dat
Chain 1 genesis tx: 020000001d0bfa51010000000000000000000000000000000000000000000000000000000000000000ffffffffb904ffff001d020f274caf4a756c792033312c2032303133203132706d204544543a2054686520552e532e2065636f6e6f6d79206772657720312e372520696e20746865207365636f6e6420717561727465722c20616964656420627920736f6c696420636f6e73756d6572207370656e64696e6720616e6420612072616d702d757020696e20627573696e65737320696e766573746d656e742c2074686520676f7665726e6d656e742073616964205765646e65736461792effffffff0100000000000000000000000000
Exception at 368
Failed to catch up {'blkfile_offset': 0, 'blkfile_number': 1, 'chain_id': 1L, 'loader': None, 'conf': u'GrowthCoin.conf', 'dirname': u'/root/.GrowthCoin', 'id': 1L}
Traceback (most recent call last):
  File "Abe/DataStore.py", line 2687, in catch_up
    store.catch_up_dir(dircfg)
  File "Abe/DataStore.py", line 2951, in catch_up_dir
    store.import_blkdat(dircfg, ds, blkfile['name'])
  File "Abe/DataStore.py", line 3083, in import_blkdat
    store.import_block(b, chain = chain)
  File "Abe/DataStore.py", line 1805, in import_block
    raise MerkleRootMismatch(b['hash'], tx_hash_array)
MerkleRootMismatch: Block header Merkle root does not match its transactions. block hash=d5e4fc54c53a9a20bbe01e70d8b47ce66f15c1bd3504f7e80390bf5601150ded

I've read to comment out in DataStore.py

Code:

# Verify Merkle root.
        #if b['hashMerkleRoot'] != util.merkle(tx_hash_array):
        #raise MerkleRootMismatch(b['hash'], tx_hash_array)

But as you can image if I post this... it didn't work!

It gave me a few of these

Code:

Skipped 73 bytes at block end
block_tx 970 970
Skipped 71 bytes at block end
block_tx 971 971
Skipped 72 bytes at block end
block_tx 972 972
Skipped 72 bytes at block end
Exception at 1677721600398062
Failed to catch up {'blkfile_offset': 206488, 'blkfile_number': 1, 'chain_id': 1, 'loader': None, 'conf': u'GrowthCoin.conf', 'dirname': u'/root/.GrowthCoin', 'id': 2L}
Traceback (most recent call last):

Then

Code:

error: unpack_from requires a buffer of at least 4 bytes

Witch I had earlier in my attempts to get Abe read the blockchain.  It looks to me as the magic bytes are causing troubles again.

Any inputs/help/solutions on this would be appreciated  

[adversor]

John Tobey, great job with Abe. Thanks for sharing this work with the community!

I'm trying to set up a website with several altcoin block explorers. The first one I tried was Klondikecoin, this works already like a charm.

Now I'm trying Noblecoin, but get same error, same as unick, just one post above.

Code:

$ python -m Abe.abe --config NobleCoin.conf --commit-bytes 100000 --no-serve
Opened /home/adversor/.noblecoin/blk0001.dat
Chain 2 genesis tx: 02000000010000000000000000000000000000000000000000000000000000000000000000ffffffff4604ffff001d01043e36204a616e20323031353a20434e4e3a204d696c6c696f6e7320696e20552e532e20627261636520666f72207265636f7264206465657020667265657a65ffffffff010000000000000000434104dcba12349012341234900abcd12223abcd455abcd77788abcd000000aaaaabbbbbcccccdddddeeeeeff00ff00ff00ff001234567890abcdef0022446688abc11ac00000000
Exception at 286
Failed to catch up {'blkfile_offset': 0, 'blkfile_number': 1, 'chain_id': 2, 'loader': None, 'conf': u'noblecoin.conf', 'dirname': u'/home/adversor/.noblecoin', 'id': 3L}
Traceback (most recent call last):
  File "Abe/DataStore.py", line 2686, in catch_up
    store.catch_up_dir(dircfg)
  File "Abe/DataStore.py", line 2950, in catch_up_dir
    store.import_blkdat(dircfg, ds, blkfile['name'])
  File "Abe/DataStore.py", line 3082, in import_blkdat
    store.import_block(b, chain = chain)
  File "Abe/DataStore.py", line 1804, in import_block
    raise MerkleRootMismatch(b['hash'], tx_hash_array)
MerkleRootMismatch: Block header Merkle root does not match its transactions. block hash=dbf8ee8b91b5f03d7786cb1d58f2588538b2d48d8c23a7913db5aac7bb58c481

After I have commented out the Merkle root verification in DataStore.py, it reads 214 blocks from chain and this is the output:

Code:

.....
Skipped 1 bytes at block end
block_tx 212 212
Skipped 1 bytes at block end
block_tx 213 213
Skipped 1 bytes at block end
block_tx 214 214
Skipped 1 bytes at block end
Exception at 60825391604920033
Failed to catch up {'blkfile_offset': 0, 'blkfile_number': 1, 'chain_id': 2, 'loader': None, 'conf': u'noblecoin.conf', 'dirname': u'/home/adversor/.noblecoin', 'id': 4L}
Traceback (most recent call last):
  File "Abe/DataStore.py", line 2686, in catch_up
    store.catch_up_dir(dircfg)
  File "Abe/DataStore.py", line 2950, in catch_up_dir
    store.import_blkdat(dircfg, ds, blkfile['name'])
  File "Abe/DataStore.py", line 3072, in import_blkdat
    b = chain.ds_parse_block(ds)
  File "Abe/Chain.py", line 59, in ds_parse_block
    d['transactions'].append(chain.ds_parse_transaction(ds))
  File "Abe/Chain.py", line 52, in ds_parse_transaction
    return deserialize.parse_Transaction(ds)
  File "Abe/deserialize.py", line 90, in parse_Transaction
    d['txIn'].append(parse_TxIn(vds))
  File "Abe/deserialize.py", line 46, in parse_TxIn
    d['sequence'] = vds.read_uint32()
  File "Abe/BCDataStream.py", line 71, in read_uint32
    def read_uint32 (self): return self._read_num('<I')
  File "Abe/BCDataStream.py", line 110, in _read_num
    (i,) = struct.unpack_from(format, self.input, self.read_cursor)
error: unpack_from requires a buffer of at least 4 bytes

I'm using

Code:

default-loader = blkfile

Someone has a solution?

[wing_hk]

Well I have a ABE working well for a particular Altcoin now.

Is there any existing data mining tool for ABE? (i.e. largest coin holder distribution etc. ) 

[gdassori]

Hi

commit
block_tx 54506 88356
block_tx 54507 88357
Exception at 22941369
Failed to catch up {'blkfile_offset': 21942607, 'blkfile_number': 100000, 'chain_id': 2, 'loader': None, 'dirname': u'/home/guido/testnet/testnet3', 'id': 1L}
Traceback (most recent call last):
  File "Abe/DataStore.py", line 2689, in catch_up
    store.catch_up_dir(dircfg)
  File "Abe/DataStore.py", line 2948, in catch_up_dir
    store.import_blkdat(dircfg, ds, blkfile['name'])
  File "Abe/DataStore.py", line 3072, in import_blkdat
    store.import_block(b, chain = chain)
  File "Abe/DataStore.py", line 1784, in import_block
    tx['tx_id'] = store.import_tx(tx, pos == 0)
  File "Abe/DataStore.py", line 2248, in import_tx
    store.binin(txout['scriptPubKey']), pubkey_id))
  File "Abe/DataStore.py", line 521, in sql
    store._execute(cached, params)
  File "Abe/DataStore.py", line 498, in _execute
    store.cursor.execute(stmt, params)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
OperationalError: (2006, 'MySQL server has gone away')
Reconnecting after rollback error: (2006, 'MySQL server has gone away')
Reconnecting to database.

happened twice, block_tx 54507 88357
blockchain is bitcoin testnet.
mysql server is still there.
ideas?


Enabled log-sql:

EXEC:
            INSERT INTO tx (tx_id, tx_hash, tx_version, tx_lockTime, tx_size)
            VALUES (%s, %s, %s, %s, %s) (88379L, 'c333a53f0174166236e341af9cad795d21578fb87ad7a1b6d2cf8aa9c722083c', 1, 0, 998171)
EXEC: INSERT INTO txout_seq () VALUES () ()
EXEC: SELECT LAST_INSERT_ID() ()
FETCH: (115923L,)
EXEC:
                INSERT INTO txout (
                    txout_id, tx_id, txout_pos, txout_value,
                    txout_scriptPubKey, pubkey_id
                ) VALUES (%s, %s, %s, %s, %s, %s) (115923L, 88379L, 0, 0, 'fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff ffff
fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff fffff [...cut... many, many, MANY, 'f']

This is the query that screws everything up


Edit: WELL, found the problem: http://blockexplorer.com/testnet/tx/c333a53f0174166236e341af9cad795d21578fb87ad7a1b6d2cf8aa9c722083c (warning, it could crash or hang the browser)
a crazy block in testnet blockchain. Now I have to figure out how to skip this block.


Edit2: Asked myself "why skipping the block if you can include it, also if its crazy?". Answer was setting max_allowed_packet in my.cnf from 1MB to 8MB. Now it works.