John Tobey (OP)
|
|
July 31, 2013, 05:16:11 PM |
|
Do you mean errors like this starting at MegaCoin block 41798? It puzzles me. From what I can piece together (without direct access to your database) this block could not figure out its block_value_in and/or block_value_out, yet it knows its parent block. This is not supposed to happen. Maybe one of the transactions failed to load because of bugs affecting parallel loading. You might try deleting the block and its descendants and rescanning the chain. Perhaps I can write an tool to help with this, but I have another priority at the moment.
|
|
|
|
|
|
|
|
|
"Your bitcoin is secured in a way that is physically impossible for others to access, no matter for what reason, no matter how good the excuse, no matter a majority of miners, no matter what." -- Greg Maxwell
|
|
|
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction.
|
|
|
MobGod
|
|
August 07, 2013, 04:44:23 PM |
|
John so your telling me to delete the whole db? Is there any other way to do this
|
|
|
|
John Tobey (OP)
|
|
August 07, 2013, 09:30:59 PM |
|
John so your telling me to delete the whole db? Is there any other way to do this
Sorry for the long delay, but I am just busy with other things. Try pulling the latest, then run: python -m Abe.admin --config abe.conf delete-chain-blocks MegaCoin
replacing abe.conf as appropriate, and show me the output. If it says that it deleted "from block", run Abe with --rescan to reload the chain. That ought to wipe just the block headers from just that chain, but it may be slow due to a MySQL issue. I suggest first trying it on a copy of the live database. Also, try to configure your system so that only process loads chain data at a time.
|
|
|
|
pisces1999
Newbie
Offline
Activity: 14
Merit: 0
|
|
August 08, 2013, 03:54:54 PM |
|
Looks good, can't wait for a more complete API, are you going to match block explorer's urls
|
|
|
|
John Tobey (OP)
|
|
August 08, 2013, 04:52:17 PM |
|
Looks good, can't wait for a more complete API, are you going to match block explorer's urls Well, I do not spend much time on new features these days, but if you have a particular function in mind, I'll at least comment on how best to do it. Several Block Explorer API functions already work more or less compatibly.
|
|
|
|
toolbag
Newbie
Offline
Activity: 42
Merit: 0
|
|
August 11, 2013, 07:26:58 PM |
|
After having Abe running for weeks with no issues at all, I suddenly ran into this: Failed to catch up {'blkfile_offset': 0, 'blkfile_number': 1, 'chain_id': Decimal('1'), 'loader': u'rpc', 'dirname': u'/home/xxxxxxx/.bitcoin', 'id': 295286L} Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2592, in catch_up if not store.catch_up_rpc(dircfg): File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2765, in catch_up_rpc store.import_block(block, chain_ids = chain_ids) File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 1731, in import_block tx['tx_id'] = store.import_and_commit_tx(tx, pos == 0) File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2220, in import_and_commit_tx tx_id = store.import_tx(tx, is_coinbase) File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2154, in import_tx pubkey_id = store.script_to_pubkey_id(txout['scriptPubKey']) File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2532, in script_to_pubkey_id for opcode, data, i in deserialize.script_GetOp(script): File "/usr/local/lib/python2.7/dist-packages/Abe/deserialize.py", line 236, in script_GetOp opcode |= ord(bytes[i]) IndexError: string index out of range
Any ideas? It's using MySQL as the back.
|
|
|
|
John Tobey (OP)
|
|
August 12, 2013, 02:19:59 AM |
|
After having Abe running for weeks with no issues at all, I suddenly ran into this: Failed to catch up {'blkfile_offset': 0, 'blkfile_number': 1, 'chain_id': Decimal('1'), 'loader': u'rpc', 'dirname': u'/home/xxxxxxx/.bitcoin', 'id': 295286L} Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2592, in catch_up if not store.catch_up_rpc(dircfg): File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2765, in catch_up_rpc store.import_block(block, chain_ids = chain_ids) File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 1731, in import_block tx['tx_id'] = store.import_and_commit_tx(tx, pos == 0) File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2220, in import_and_commit_tx tx_id = store.import_tx(tx, is_coinbase) File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2154, in import_tx pubkey_id = store.script_to_pubkey_id(txout['scriptPubKey']) File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2532, in script_to_pubkey_id for opcode, data, i in deserialize.script_GetOp(script): File "/usr/local/lib/python2.7/dist-packages/Abe/deserialize.py", line 236, in script_GetOp opcode |= ord(bytes[i]) IndexError: string index out of range
Any ideas? It's using MySQL as the back. I suspect this comes from new script types. I have a possible fix here in diff format. I'll push it to master soon. I am curious what transaction triggered the error. I'll ask more about that if the fix doesn't work. Thanks for the report. Please mention the git revision ( git rev-parse HEAD) and any local code changes.
|
|
|
|
toolbag
Newbie
Offline
Activity: 42
Merit: 0
|
|
August 12, 2013, 02:52:59 AM |
|
I really know nothing about git except how to use it to grab a copy of some code from Github. Running that git command in either my bitcoin-abe directory or its parent isn't showing anything, so I must have moved things around enough that git became confused. I can see in my command history where I fetched Abe using "git clone," so I know that's how I originally got it.
I did think to save a slice of the transaction history. I was running Abe in an infinite loop where it was using RPC to fetch the latest blocks/transactions from bitcoind, sleeping 60 seconds, and doing it again. The last block I got successfully was 251552 ("block_tx 251552 21898522") followed by several minutes of successful "mempool tx" commits, the last one being "mempool tx 21898948", then one of my script's 60-second sleeps, then the error I posted above. There were no successful transactions after that last "mempool tx" one.
I have made no local modifications to the code at all.
|
|
|
|
John Tobey (OP)
|
|
August 12, 2013, 05:47:02 PM |
|
I really know nothing about git except how to use it to grab a copy of some code from Github. Running that git command in either my bitcoin-abe directory or its parent isn't showing anything, so I must have moved things around enough that git became confused. I can see in my command history where I fetched Abe using "git clone," so I know that's how I originally got it.
Please pull the latest and see if it gets you past the error. I would have pushed it earlier but did not have my github keys handy. Thanks.
|
|
|
|
toolbag
Newbie
Offline
Activity: 42
Merit: 0
|
|
August 13, 2013, 10:51:06 PM |
|
The latest update seems to have done the trick! Thanks!
|
|
|
|
antonimasso
Member
Offline
Activity: 73
Merit: 10
|
|
August 20, 2013, 05:56:26 PM |
|
Hello,
I have a few questions:
- Do I need to wait for bitcoind to download the complete block files or can I run them in parallel? - Once the initial data load finishes do I need to stop it and run it with python -m Abe.abe --config abe-my.conf? - Will running python -m Abe.abe --config abe-my.conf insert new incoming blocks?
Thanks
|
|
|
|
John Tobey (OP)
|
|
August 21, 2013, 01:14:35 PM |
|
Hello,
I have a few questions:
- Do I need to wait for bitcoind to download the complete block files or can I run them in parallel? - Once the initial data load finishes do I need to stop it and run it with python -m Abe.abe --config abe-my.conf? - Will running python -m Abe.abe --config abe-my.conf insert new incoming blocks?
Thanks
You can run bitcoind and Abe in parallel. The instructions call for --no-serve on the initial load, and this makes it exit as soon as it loads the last block, so you don't have to stop it. The instructions say to run in web-server mode after the initial load, but you can run both at the same time, as long as you add "--datadir=[]" to the end of the web server command: python -m Abe.abe --config abe-my.conf "--datadir=[]" The "--datadir=[]" prevents the two processes from competing to insert blocks, which tends to cause trouble. Unfortunately, Abe does not automatically insert new blocks every so often. By default, ( without "--datadir=[]") it inserts new blocks when you browse the site, but if you haven't for a long time, it can take Abe quite a while to catch up. If this is a problem, I suggest you schedule a job to trigger this at regular intervals. For example, I use this in crontab to check every minute: * * * * * /usr/bin/wget -q -O /dev/null http://localhost:2750/ If you want to run a server as opposed to just personal use, I suggest configuring the public server not to load anything (by adding "--datadir=[]") and have a separate process listening on the local interface with a job such as the above to trigger loading new blocks. Using a local listener keeps the loader single-threaded. (This information should be in a readme, thanks for asking.)
|
|
|
|
antonimasso
Member
Offline
Activity: 73
Merit: 10
|
|
August 22, 2013, 06:20:01 AM |
|
Thanks for the detailed reply John. Once the initial load finishes I'll run some tests. BTW, the last 50.000 blocks seem to take forever, I guess there are many more TXs.
|
|
|
|
weex
Legendary
Offline
Activity: 1102
Merit: 1014
|
|
August 22, 2013, 11:09:57 PM Last edit: August 22, 2013, 11:36:22 PM by weex |
|
Does Abe work with leveldb ok? I have an instance that was running on Litecoin 0.6.3 and am trying to upgrade to 0.8.3.7 which uses leveldb.
If you can give me a quick idea on what I need to do to rebuild the db with this new client, that would be great.
Edit: Looks good. I just had to make a new database.
|
|
|
|
John Tobey (OP)
|
|
August 23, 2013, 12:43:41 AM |
|
Thanks for the detailed reply John. Once the initial load finishes I'll run some tests. BTW, the last 50.000 blocks seem to take forever, I guess there are many more TXs.
Yes. You can get an idea of how far it has to go by examining the datadir table. "blkfile_number" corresponds to a file in the blocks directory (e.g. 100048 = blk00048.dat) and "blkfile_offset" shows how far in the file Abe has loaded.
|
|
|
|
antonimasso
Member
Offline
Activity: 73
Merit: 10
|
|
August 26, 2013, 06:04:44 AM Last edit: August 26, 2013, 05:15:16 PM by antonimasso |
|
Hello John,
Is it easy to implement N confirmations to addressbalance?
Also, is is possible to not show the results of the second SQL in the result set of the first SQL?
mysql> SELECT b.block_nTime, cc.chain_id, b.block_height, 0, b.block_hash, tx.tx_hash, txout.txout_pos, txout.txout_value FROM chain_candidate cc JOIN block b ON (b.block_id = cc.block_id) JOIN block_tx ON (block_tx.block_id = b.block_id) JOIN tx ON (tx.tx_id = block_tx.tx_id) JOIN txout ON (txout.tx_id = tx.tx_id) JOIN pubkey ON (pubkey.pubkey_id = txout.pubkey_id) WHERE pubkey.pubkey_hash = '645677d0601371051c9f723fecdd466c317a01cb'; +-------------+----------+--------------+---+------------------------------------------------------------------+------------------------------------------------------------------+-----------+-------------+ | block_nTime | chain_id | block_height | 0 | block_hash | tx_hash | txout_pos | txout_value | +-------------+----------+--------------+---+------------------------------------------------------------------+------------------------------------------------------------------+-----------+-------------+ | 1375776420 | 1 | 250493 | 0 | 000000000000003dcdc75f30fd1c732944fb62693ac76fc217ec142207ab6ae3 | ac772eed99449ec15254a92f8292da1460ed650691b5b282521948a221dbbe78 | 0 | 18700000 | | 1375886710 | 1 | 250732 | 0 | 000000000000001cb840f80614ed360324cf9073f0ffce25975a456c0ad41ea2 | f25f9a8fdb83ec5eb8ef99a3532e62dba4c158a6529d919cf5998462d20223a9 | 1 | 17650000 | | 1376079367 | 1 | 251180 | 0 | 000000000000004d8a953fd3ce48c12f88c5e266b72aa816332e45a740fc377d | 0cd22a486659df7a7435edd426805016f0b26ed2779d947bba3212c9f29f4edd | 1 | 500000000 | | 1376767233 | 1 | 252686 | 0 | 000000000000000af30ed5b0af8414ad2837c07218b45af00a1c2931f1f9700f | 4f345a39e4af5456f3e03f0322188c1701204df169d9db04844559d4afdcecbf | 1 | 489950000 | | 1376812393 | 1 | 252796 | 0 | 0000000000000025271bf2ac0e0466ae77d00ab12baad1338bd5fc54468ae6b0 | f3f4eb433ea379b494ce1c22867217008f23da2dc6839668f434d0032665c184 | 1 | 479900000 | | 1377100725 | 1 | 253425 | 0 | 000000000000004d69bfa91a524c5123192d2c93f84d24f4dc30e7d273e17949 | 0320695bf487ec494734c572a29e0e05521f70eb3e636e3673c810f4e2d6aaa3 | 0 | 10609000 | | 1377330046 | 1 | 253965 | 0 | 000000000000002f6c0ac43224314d21b95139d8dd76893aa7b231a03d905fe8 | 01b3a6e93f5862ee00a4184b4851b84e6a9e5c63d0a4c122c2221f21bd41d315 | 1 | 480459000 | | 1377446268 | 1 | 254181 | 0 | 000000000000000363f7e2298cdf320e1dd4867788a49b92bb3c366ad0b4f72c | 16a56d242e5bd5fd35a59f27decce1e0bb1b9f4cbc3ce670552bfd6f639f50b7 | 0 | 5000000 | | 1377525653 | 1 | 254324 | 0 | 0000000000000024593882cf59758b73160b79042731ea16f4c4963f4da5fa04 | a5ccd0bf756a9b867d3ff95543974ebebf4bbe9c478108a2ee829b28bd63b94c | 0 | 1000000 | | 1377527510 | 1 | 254325 | 0 | 00000000000000040bfcc8600878065faa2db3eb0d3a9434e922b2620fa8f133 | 11517d88ab64fffcc8779b341512b4e8a307d73e7aa00ce01733acb0cf90a99a | 0 | 1000000 | | 1377527510 | 1 | 254325 | 0 | 00000000000000040bfcc8600878065faa2db3eb0d3a9434e922b2620fa8f133 | 7710da0657a055eeadf8f45b25530ecf07f4bd2b3871c19a0f17e44d9514e91d | 0 | 1000000 | +-------------+----------+--------------+---+------------------------------------------------------------------+------------------------------------------------------------------+-----------+-------------+ 11 rows in set (0.01 sec)
mysql> SELECT b.block_nTime, cc.chain_id, b.block_height, 1, b.block_hash,tx.tx_hash, txin.txin_pos, -prevout.txout_value FROM chain_candidate cc JOIN block b ON (b.block_id = cc.block_id) JOIN block_tx ON (block_tx.block_id = b.block_id) JOIN tx ON (tx.tx_id = block_tx.tx_id) JOIN txin ON (txin.tx_id = tx.tx_id) JOIN txout prevout ON (txin.txout_id = prevout.txout_id) JOIN pubkey ON (pubkey.pubkey_id = prevout.pubkey_id) WHERE pubkey.pubkey_hash = '645677d0601371051c9f723fecdd466c317a01cb'; +-------------+----------+--------------+---+------------------------------------------------------------------+------------------------------------------------------------------+----------+----------------------+ | block_nTime | chain_id | block_height | 1 | block_hash | tx_hash | txin_pos | -prevout.txout_value | +-------------+----------+--------------+---+------------------------------------------------------------------+------------------------------------------------------------------+----------+----------------------+ | 1375886710 | 1 | 250732 | 1 | 000000000000001cb840f80614ed360324cf9073f0ffce25975a456c0ad41ea2 | f25f9a8fdb83ec5eb8ef99a3532e62dba4c158a6529d919cf5998462d20223a9 | 0 | -18700000 | | 1375887925 | 1 | 250734 | 1 | 00000000000000214294e44ef349ccb337b5038d5cfecc2c7a9b086d89d08f20 | 0244b7597b986d9433f719a4decb0ce46f636182099471f982c0cb0eb5dee674 | 0 | -17650000 | | 1376767233 | 1 | 252686 | 1 | 000000000000000af30ed5b0af8414ad2837c07218b45af00a1c2931f1f9700f | 4f345a39e4af5456f3e03f0322188c1701204df169d9db04844559d4afdcecbf | 0 | -500000000 | | 1376812393 | 1 | 252796 | 1 | 0000000000000025271bf2ac0e0466ae77d00ab12baad1338bd5fc54468ae6b0 | f3f4eb433ea379b494ce1c22867217008f23da2dc6839668f434d0032665c184 | 0 | -489950000 | | 1377330046 | 1 | 253965 | 1 | 000000000000002f6c0ac43224314d21b95139d8dd76893aa7b231a03d905fe8 | 01b3a6e93f5862ee00a4184b4851b84e6a9e5c63d0a4c122c2221f21bd41d315 | 0 | -479900000 | | 1377330046 | 1 | 253965 | 1 | 000000000000002f6c0ac43224314d21b95139d8dd76893aa7b231a03d905fe8 | 01b3a6e93f5862ee00a4184b4851b84e6a9e5c63d0a4c122c2221f21bd41d315 | 1 | -10609000 | +-------------+----------+--------------+---+------------------------------------------------------------------+------------------------------------------------------------------+----------+----------------------+ 6 rows in set (0.00 sec)
Thanks
|
|
|
|
antonimasso
Member
Offline
Activity: 73
Merit: 10
|
|
August 26, 2013, 07:27:19 PM |
|
Reading your code I found this function, do_unspent, which makes exactly what I needed.
|
|
|
|
John Tobey (OP)
|
|
August 26, 2013, 08:42:39 PM |
|
Hello John,
Is it easy to implement N confirmations to addressbalance?
Yes, it should be pretty easy, since the underlying functions get_sent_and_last_block_id and get_received_and_last_block_id both accept a block_height. Just make the new API pass N less than the current block number. Please note that I have not had much time to work on new features lately. Also, is is possible to not show the results of the second SQL in the result set of the first SQL?
I am not sure which API function you mean, but there is commingling of sent and received amounts, and to separate them would take a couple of hours of coding. Can you point to another block explorer that does it how you want? I usually try to copy their interfaces rather than invent my own.
|
|
|
|
HeroC
Legendary
Offline
Activity: 858
Merit: 1000
|
|
August 27, 2013, 02:56:18 AM |
|
Looks good, I bet it will be used with a lot of new altcoins.
|
|
|
|
|
|