[ANNOUNCE] Abe 0.7: Open Source Block Explorer Knockoff

[John Tobey]

Hello John again lol could you please look at this I'm going crazy on how to fix it mega
http://blockexplorer.coinworld.us/

Do you mean errors like this starting at MegaCoin block 41798?  It puzzles me.  From what I can piece together (without direct access to your database) this block could not figure out its block_value_in and/or block_value_out, yet it knows its parent block.  This is not supposed to happen.  Maybe one of the transactions failed to load because of bugs affecting parallel loading.

You might try deleting the block and its descendants and rescanning the chain.  Perhaps I can write an tool to help with this, but I have another priority at the moment.

[1713593878]

1713593878

[1713593878]

1713593878

[1713593878]

1713593878

[1713593878]

1713593878

[1713593878]

1713593878

[MobGod]

John so your telling me to delete the whole db? Is there any other way to do this

[John Tobey]

John so your telling me to delete the whole db? Is there any other way to do this

Sorry for the long delay, but I am just busy with other things.  Try pulling the latest, then run:

Code:

python -m Abe.admin --config abe.conf delete-chain-blocks MegaCoin

replacing abe.conf as appropriate, and show me the output.  If it says that it deleted "from block", run Abe with --rescan to reload the chain.  That ought to wipe just the block headers from just that chain, but it may be slow due to a MySQL issue.  I suggest first trying it on a copy of the live database.

Also, try to configure your system so that only process loads chain data at a time.

[pisces1999]

Looks good, can't wait for a more complete API, are you going to match block explorer's urls

[John Tobey]

Looks good, can't wait for a more complete API, are you going to match block explorer's urls

Well, I do not spend much time on new features these days, but if you have a particular function in mind, I'll at least comment on how best to do it.  Several Block Explorer API functions already work more or less compatibly.

[toolbag]

After having Abe running for weeks with no issues at all, I suddenly ran into this:

Code:

Failed to catch up {'blkfile_offset': 0, 'blkfile_number': 1, 'chain_id': Decimal('1'), 'loader': u'rpc', 'dirname': u'/home/xxxxxxx/.bitcoin', 'id': 295286L}
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2592, in catch_up
    if not store.catch_up_rpc(dircfg):
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2765, in catch_up_rpc
    store.import_block(block, chain_ids = chain_ids)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 1731, in import_block
    tx['tx_id'] = store.import_and_commit_tx(tx, pos == 0)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2220, in import_and_commit_tx
    tx_id = store.import_tx(tx, is_coinbase)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2154, in import_tx
    pubkey_id = store.script_to_pubkey_id(txout['scriptPubKey'])
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2532, in script_to_pubkey_id
    for opcode, data, i in deserialize.script_GetOp(script):
  File "/usr/local/lib/python2.7/dist-packages/Abe/deserialize.py", line 236, in script_GetOp
    opcode |= ord(bytes[i])
IndexError: string index out of range

Any ideas? It's using MySQL as the back.

[John Tobey]

After having Abe running for weeks with no issues at all, I suddenly ran into this:

Code:

Failed to catch up {'blkfile_offset': 0, 'blkfile_number': 1, 'chain_id': Decimal('1'), 'loader': u'rpc', 'dirname': u'/home/xxxxxxx/.bitcoin', 'id': 295286L}
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2592, in catch_up
    if not store.catch_up_rpc(dircfg):
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2765, in catch_up_rpc
    store.import_block(block, chain_ids = chain_ids)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 1731, in import_block
    tx['tx_id'] = store.import_and_commit_tx(tx, pos == 0)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2220, in import_and_commit_tx
    tx_id = store.import_tx(tx, is_coinbase)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2154, in import_tx
    pubkey_id = store.script_to_pubkey_id(txout['scriptPubKey'])
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2532, in script_to_pubkey_id
    for opcode, data, i in deserialize.script_GetOp(script):
  File "/usr/local/lib/python2.7/dist-packages/Abe/deserialize.py", line 236, in script_GetOp
    opcode |= ord(bytes[i])
IndexError: string index out of range

Any ideas? It's using MySQL as the back.

I suspect this comes from new script types.  I have a possible fix here in diff format.  I'll push it to master soon.

I am curious what transaction triggered the error.  I'll ask more about that if the fix doesn't work.

Thanks for the report.  Please mention the git revision (git rev-parse HEAD) and any local code changes.

[toolbag]

I really know nothing about git except how to use it to grab a copy of some code from Github. Running that git command in either my bitcoin-abe directory or its parent isn't showing anything, so I must have moved things around enough that git became confused. I can see in my command history where I fetched Abe using "git clone," so I know that's how I originally got it.

I did think to save a slice of the transaction history. I was running Abe in an infinite loop where it was using RPC to fetch the latest blocks/transactions from bitcoind, sleeping 60 seconds, and doing it again. The last block I got successfully was 251552 ("block_tx 251552 21898522") followed by several minutes of successful "mempool tx" commits, the last one being "mempool tx 21898948", then one of my script's 60-second sleeps, then the error I posted above. There were no successful transactions after that last "mempool tx" one.

I have made no local modifications to the code at all.

[John Tobey]

I really know nothing about git except how to use it to grab a copy of some code from Github. Running that git command in either my bitcoin-abe directory or its parent isn't showing anything, so I must have moved things around enough that git became confused. I can see in my command history where I fetched Abe using "git clone," so I know that's how I originally got it.

Please pull the latest and see if it gets you past the error.  I would have pushed it earlier but did not have my github keys handy.

Thanks.

[toolbag]

The latest update seems to have done the trick! Thanks!

[antonimasso]

Hello,

I have a few questions:

- Do I need to wait for bitcoind to download the complete block files or can I run them in parallel?
- Once the initial data load finishes do I need to stop it and run it with python -m Abe.abe --config abe-my.conf?
- Will running python -m Abe.abe --config abe-my.conf insert new incoming blocks?

Thanks

[John Tobey]

Hello,

I have a few questions:

- Do I need to wait for bitcoind to download the complete block files or can I run them in parallel?
- Once the initial data load finishes do I need to stop it and run it with python -m Abe.abe --config abe-my.conf?
- Will running python -m Abe.abe --config abe-my.conf insert new incoming blocks?

Thanks

You can run bitcoind and Abe in parallel.

The instructions call for --no-serve on the initial load, and this makes it exit as soon as it loads the last block, so you don't have to stop it.  The instructions say to run in web-server mode after the initial load, but you can run both at the same time, as long as you add "--datadir=[]" to the end of the web server command:

Code:

python -m Abe.abe --config abe-my.conf "--datadir=[]"

The "--datadir=[]" prevents the two processes from competing to insert blocks, which tends to cause trouble.

Unfortunately, Abe does not automatically insert new blocks every so often.  By default, (without "--datadir=[]") it inserts new blocks when you browse the site, but if you haven't for a long time, it can take Abe quite a while to catch up.  If this is a problem, I suggest you schedule a job to trigger this at regular intervals.  For example, I use this in crontab to check every minute:

Code:

* * * * * /usr/bin/wget -q -O /dev/null http://localhost:2750/

If you want to run a server as opposed to just personal use, I suggest configuring the public server not to load anything (by adding "--datadir=[]") and have a separate process listening on the local interface with a job such as the above to trigger loading new blocks.  Using a local listener keeps the loader single-threaded.  (This information should be in a readme, thanks for asking.)

[antonimasso]

Thanks for the detailed reply John. Once the initial load finishes I'll run some tests.
BTW, the last 50.000 blocks seem to take forever, I guess there are many more TXs.

[weex]

Does Abe work with leveldb ok? I have an instance that was running on Litecoin 0.6.3 and am trying to upgrade to 0.8.3.7 which uses leveldb.

If you can give me a quick idea on what I need to do to rebuild the db with this new client, that would be great.

Edit: Looks good. I just had to make a new database.

[John Tobey]

Thanks for the detailed reply John. Once the initial load finishes I'll run some tests.
BTW, the last 50.000 blocks seem to take forever, I guess there are many more TXs.

Yes.  You can get an idea of how far it has to go by examining the datadir table.  "blkfile_number" corresponds to a file in the blocks directory (e.g. 100048 = blk00048.dat) and "blkfile_offset" shows how far in the file Abe has loaded.

[antonimasso]

Hello John,

Is it easy to implement N confirmations to addressbalance?

Also, is is possible to not show the results of the second SQL in the result set of the first SQL?

mysql> SELECT b.block_nTime, cc.chain_id, b.block_height, 0, b.block_hash, tx.tx_hash, txout.txout_pos, txout.txout_value FROM chain_candidate cc JOIN block b ON (b.block_id = cc.block_id) JOIN block_tx ON (block_tx.block_id = b.block_id) JOIN tx ON (tx.tx_id = block_tx.tx_id) JOIN txout ON (txout.tx_id = tx.tx_id) JOIN pubkey ON (pubkey.pubkey_id = txout.pubkey_id) WHERE pubkey.pubkey_hash = '645677d0601371051c9f723fecdd466c317a01cb';
+-------------+----------+--------------+---+------------------------------------------------------------------+------------------------------------------------------------------+-----------+-------------+
| block_nTime | chain_id | block_height | 0 | block_hash                                                       | tx_hash                                                          | txout_pos | txout_value |
+-------------+----------+--------------+---+------------------------------------------------------------------+------------------------------------------------------------------+-----------+-------------+
|  1375776420 |        1 |       250493 | 0 | 000000000000003dcdc75f30fd1c732944fb62693ac76fc217ec142207ab6ae3 | ac772eed99449ec15254a92f8292da1460ed650691b5b282521948a221dbbe78 |         0 |    18700000 |
|  1375886710 |        1 |       250732 | 0 | 000000000000001cb840f80614ed360324cf9073f0ffce25975a456c0ad41ea2 | f25f9a8fdb83ec5eb8ef99a3532e62dba4c158a6529d919cf5998462d20223a9 |         1 |    17650000 |
|  1376079367 |        1 |       251180 | 0 | 000000000000004d8a953fd3ce48c12f88c5e266b72aa816332e45a740fc377d | 0cd22a486659df7a7435edd426805016f0b26ed2779d947bba3212c9f29f4edd |         1 |   500000000 |
|  1376767233 |        1 |       252686 | 0 | 000000000000000af30ed5b0af8414ad2837c07218b45af00a1c2931f1f9700f | 4f345a39e4af5456f3e03f0322188c1701204df169d9db04844559d4afdcecbf |         1 |   489950000 |
|  1376812393 |        1 |       252796 | 0 | 0000000000000025271bf2ac0e0466ae77d00ab12baad1338bd5fc54468ae6b0 | f3f4eb433ea379b494ce1c22867217008f23da2dc6839668f434d0032665c184 |         1 |   479900000 |
|  1377100725 |        1 |       253425 | 0 | 000000000000004d69bfa91a524c5123192d2c93f84d24f4dc30e7d273e17949 | 0320695bf487ec494734c572a29e0e05521f70eb3e636e3673c810f4e2d6aaa3 |         0 |    10609000 |
|  1377330046 |        1 |       253965 | 0 | 000000000000002f6c0ac43224314d21b95139d8dd76893aa7b231a03d905fe8 | 01b3a6e93f5862ee00a4184b4851b84e6a9e5c63d0a4c122c2221f21bd41d315 |         1 |   480459000 |
|  1377446268 |        1 |       254181 | 0 | 000000000000000363f7e2298cdf320e1dd4867788a49b92bb3c366ad0b4f72c | 16a56d242e5bd5fd35a59f27decce1e0bb1b9f4cbc3ce670552bfd6f639f50b7 |         0 |     5000000 |
|  1377525653 |        1 |       254324 | 0 | 0000000000000024593882cf59758b73160b79042731ea16f4c4963f4da5fa04 | a5ccd0bf756a9b867d3ff95543974ebebf4bbe9c478108a2ee829b28bd63b94c |         0 |     1000000 |
|  1377527510 |        1 |       254325 | 0 | 00000000000000040bfcc8600878065faa2db3eb0d3a9434e922b2620fa8f133 | 11517d88ab64fffcc8779b341512b4e8a307d73e7aa00ce01733acb0cf90a99a |         0 |     1000000 |
|  1377527510 |        1 |       254325 | 0 | 00000000000000040bfcc8600878065faa2db3eb0d3a9434e922b2620fa8f133 | 7710da0657a055eeadf8f45b25530ecf07f4bd2b3871c19a0f17e44d9514e91d |         0 |     1000000 |
+-------------+----------+--------------+---+------------------------------------------------------------------+------------------------------------------------------------------+-----------+-------------+
11 rows in set (0.01 sec)

mysql> SELECT b.block_nTime, cc.chain_id, b.block_height, 1, b.block_hash,tx.tx_hash, txin.txin_pos, -prevout.txout_value FROM chain_candidate cc JOIN block b ON (b.block_id = cc.block_id) JOIN block_tx ON (block_tx.block_id = b.block_id) JOIN tx ON (tx.tx_id = block_tx.tx_id) JOIN txin ON (txin.tx_id = tx.tx_id) JOIN txout prevout ON (txin.txout_id = prevout.txout_id) JOIN pubkey ON (pubkey.pubkey_id = prevout.pubkey_id) WHERE pubkey.pubkey_hash = '645677d0601371051c9f723fecdd466c317a01cb';
+-------------+----------+--------------+---+------------------------------------------------------------------+------------------------------------------------------------------+----------+----------------------+
| block_nTime | chain_id | block_height | 1 | block_hash                                                       | tx_hash                                                          | txin_pos | -prevout.txout_value |
+-------------+----------+--------------+---+------------------------------------------------------------------+------------------------------------------------------------------+----------+----------------------+
|  1375886710 |        1 |       250732 | 1 | 000000000000001cb840f80614ed360324cf9073f0ffce25975a456c0ad41ea2 | f25f9a8fdb83ec5eb8ef99a3532e62dba4c158a6529d919cf5998462d20223a9 |        0 |            -18700000 |
|  1375887925 |        1 |       250734 | 1 | 00000000000000214294e44ef349ccb337b5038d5cfecc2c7a9b086d89d08f20 | 0244b7597b986d9433f719a4decb0ce46f636182099471f982c0cb0eb5dee674 |        0 |            -17650000 |
|  1376767233 |        1 |       252686 | 1 | 000000000000000af30ed5b0af8414ad2837c07218b45af00a1c2931f1f9700f | 4f345a39e4af5456f3e03f0322188c1701204df169d9db04844559d4afdcecbf |        0 |           -500000000 |
|  1376812393 |        1 |       252796 | 1 | 0000000000000025271bf2ac0e0466ae77d00ab12baad1338bd5fc54468ae6b0 | f3f4eb433ea379b494ce1c22867217008f23da2dc6839668f434d0032665c184 |        0 |           -489950000 |
|  1377330046 |        1 |       253965 | 1 | 000000000000002f6c0ac43224314d21b95139d8dd76893aa7b231a03d905fe8 | 01b3a6e93f5862ee00a4184b4851b84e6a9e5c63d0a4c122c2221f21bd41d315 |        0 |           -479900000 |
|  1377330046 |        1 |       253965 | 1 | 000000000000002f6c0ac43224314d21b95139d8dd76893aa7b231a03d905fe8 | 01b3a6e93f5862ee00a4184b4851b84e6a9e5c63d0a4c122c2221f21bd41d315 |        1 |            -10609000 |
+-------------+----------+--------------+---+------------------------------------------------------------------+------------------------------------------------------------------+----------+----------------------+
6 rows in set (0.00 sec)

Thanks

[antonimasso]

Reading your code I found this function, do_unspent, which makes exactly what I needed.

[John Tobey]

Hello John,

Is it easy to implement N confirmations to addressbalance?

Yes, it should be pretty easy, since the underlying functions get_sent_and_last_block_id and get_received_and_last_block_id both accept a block_height.  Just make the new API pass N less than the current block number.  Please note that I have not had much time to work on new features lately.

Also, is is possible to not show the results of the second SQL in the result set of the first SQL?

I am not sure which API function you mean, but there is commingling of sent and received amounts, and to separate them would take a couple of hours of coding.  Can you point to another block explorer that does it how you want?  I usually try to copy their interfaces rather than invent my own.

[HeroC]

Looks good, I bet it will be used with a lot of new altcoins. 

[antonimasso]

I'm currently using blockchain.info API. Check out this URL to get the unspent outputs:

http://blockchain.info/unspent?address=1A9YA62QvpmE32wu9h4tgC3GHQiUhiTAXi

and this URL to get the total sent and received amounts (much clearer than blockexplorer.com IMO):

http://blockchain.info/address/1A9YA62QvpmE32wu9h4tgC3GHQiUhiTAXi?format=json