Bitcoin Forum
December 11, 2016, 04:42:01 AM *
News: To be able to use the next phase of the beta forum software, please ensure that your email address is correct/functional.
 
   Home   Help Search Donate Login Register  
Pages: « 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 [27] 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 »
  Print  
Author Topic: [ANNOUNCE] Abe 0.7: Open Source Block Explorer Knockoff  (Read 175900 times)
John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
July 31, 2013, 05:16:11 PM
 #521

Hello John again lol could you please look at this I'm going crazy on how to fix it mega
http://blockexplorer.coinworld.us/

Do you mean errors like this starting at MegaCoin block 41798?  It puzzles me.  From what I can piece together (without direct access to your database) this block could not figure out its block_value_in and/or block_value_out, yet it knows its parent block.  This is not supposed to happen.  Maybe one of the transactions failed to load because of bugs affecting parallel loading.

You might try deleting the block and its descendants and rescanning the chain.  Perhaps I can write an tool to help with this, but I have another priority at the moment.

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
1481431321
Hero Member
*
Offline Offline

Posts: 1481431321

View Profile Personal Message (Offline)

Ignore
1481431321
Reply with quote  #2

1481431321
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1481431321
Hero Member
*
Offline Offline

Posts: 1481431321

View Profile Personal Message (Offline)

Ignore
1481431321
Reply with quote  #2

1481431321
Report to moderator
1481431321
Hero Member
*
Offline Offline

Posts: 1481431321

View Profile Personal Message (Offline)

Ignore
1481431321
Reply with quote  #2

1481431321
Report to moderator
1481431321
Hero Member
*
Offline Offline

Posts: 1481431321

View Profile Personal Message (Offline)

Ignore
1481431321
Reply with quote  #2

1481431321
Report to moderator
MobGod
Hero Member
*****
Offline Offline

Activity: 518



View Profile WWW
August 07, 2013, 04:44:23 PM
 #522

John so your telling me to delete the whole db? Is there any other way to do this

BTC: 1RLMpP6GydCUtEW3B3WHijYCy6E3Sp435    DGC: DMmXuiP4nNP2z7R3BCVdhKV7svz5e3iMyi 
  MEC: MPAs8ged36rJQXCsm96rwwbpk2SopTAi1S  LTC: Lao7GR161qAYCRxaiWN2akBEaCBkaVpo6z
GPL: RXXgMYQPNkuATdoYA7odM9xJHyJbVma74V Coffee: BznEeTLbKBAVLgu1tpE1ZDC99S6KVoQr2j
John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
August 07, 2013, 09:30:59 PM
 #523

John so your telling me to delete the whole db? Is there any other way to do this

Sorry for the long delay, but I am just busy with other things.  Try pulling the latest, then run:

Code:
python -m Abe.admin --config abe.conf delete-chain-blocks MegaCoin

replacing abe.conf as appropriate, and show me the output.  If it says that it deleted "from block", run Abe with --rescan to reload the chain.  That ought to wipe just the block headers from just that chain, but it may be slow due to a MySQL issue.  I suggest first trying it on a copy of the live database.

Also, try to configure your system so that only process loads chain data at a time.

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
pisces1999
Newbie
*
Offline Offline

Activity: 14



View Profile
August 08, 2013, 03:54:54 PM
 #524

Looks good, can't wait for a more complete API, are you going to match block explorer's urls Grin

BTC.sx (https://btc.sx) - Leveraged Bitcoin Trading.
Get paid weekly for your signature. (https://bitcointalk.org/index.php?topic=233434.0)
John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
August 08, 2013, 04:52:17 PM
 #525

Looks good, can't wait for a more complete API, are you going to match block explorer's urls Grin

Well, I do not spend much time on new features these days, but if you have a particular function in mind, I'll at least comment on how best to do it.  Several Block Explorer API functions already work more or less compatibly.

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
toolbag
Jr. Member
*
Offline Offline

Activity: 42


View Profile
August 11, 2013, 07:26:58 PM
 #526

After having Abe running for weeks with no issues at all, I suddenly ran into this:

Code:
Failed to catch up {'blkfile_offset': 0, 'blkfile_number': 1, 'chain_id': Decimal('1'), 'loader': u'rpc', 'dirname': u'/home/xxxxxxx/.bitcoin', 'id': 295286L}
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2592, in catch_up
    if not store.catch_up_rpc(dircfg):
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2765, in catch_up_rpc
    store.import_block(block, chain_ids = chain_ids)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 1731, in import_block
    tx['tx_id'] = store.import_and_commit_tx(tx, pos == 0)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2220, in import_and_commit_tx
    tx_id = store.import_tx(tx, is_coinbase)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2154, in import_tx
    pubkey_id = store.script_to_pubkey_id(txout['scriptPubKey'])
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2532, in script_to_pubkey_id
    for opcode, data, i in deserialize.script_GetOp(script):
  File "/usr/local/lib/python2.7/dist-packages/Abe/deserialize.py", line 236, in script_GetOp
    opcode |= ord(bytes[i])
IndexError: string index out of range

Any ideas? It's using MySQL as the back.

Tips weclome and appreciated: 1LRxTZT76SE1UqB1ijvGp46qWJXV7BQczk
John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
August 12, 2013, 02:19:59 AM
 #527

After having Abe running for weeks with no issues at all, I suddenly ran into this:

Code:
Failed to catch up {'blkfile_offset': 0, 'blkfile_number': 1, 'chain_id': Decimal('1'), 'loader': u'rpc', 'dirname': u'/home/xxxxxxx/.bitcoin', 'id': 295286L}
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2592, in catch_up
    if not store.catch_up_rpc(dircfg):
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2765, in catch_up_rpc
    store.import_block(block, chain_ids = chain_ids)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 1731, in import_block
    tx['tx_id'] = store.import_and_commit_tx(tx, pos == 0)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2220, in import_and_commit_tx
    tx_id = store.import_tx(tx, is_coinbase)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2154, in import_tx
    pubkey_id = store.script_to_pubkey_id(txout['scriptPubKey'])
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2532, in script_to_pubkey_id
    for opcode, data, i in deserialize.script_GetOp(script):
  File "/usr/local/lib/python2.7/dist-packages/Abe/deserialize.py", line 236, in script_GetOp
    opcode |= ord(bytes[i])
IndexError: string index out of range

Any ideas? It's using MySQL as the back.

I suspect this comes from new script types.  I have a possible fix here in diff format.  I'll push it to master soon.

I am curious what transaction triggered the error.  I'll ask more about that if the fix doesn't work. Smiley

Thanks for the report.  Please mention the git revision (git rev-parse HEAD) and any local code changes.

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
toolbag
Jr. Member
*
Offline Offline

Activity: 42


View Profile
August 12, 2013, 02:52:59 AM
 #528

I really know nothing about git except how to use it to grab a copy of some code from Github. Running that git command in either my bitcoin-abe directory or its parent isn't showing anything, so I must have moved things around enough that git became confused. I can see in my command history where I fetched Abe using "git clone," so I know that's how I originally got it.

I did think to save a slice of the transaction history. I was running Abe in an infinite loop where it was using RPC to fetch the latest blocks/transactions from bitcoind, sleeping 60 seconds, and doing it again. The last block I got successfully was 251552 ("block_tx 251552 21898522") followed by several minutes of successful "mempool tx" commits, the last one being "mempool tx 21898948", then one of my script's 60-second sleeps, then the error I posted above. There were no successful transactions after that last "mempool tx" one.

I have made no local modifications to the code at all.

Tips weclome and appreciated: 1LRxTZT76SE1UqB1ijvGp46qWJXV7BQczk
John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
August 12, 2013, 05:47:02 PM
 #529

I really know nothing about git except how to use it to grab a copy of some code from Github. Running that git command in either my bitcoin-abe directory or its parent isn't showing anything, so I must have moved things around enough that git became confused. I can see in my command history where I fetched Abe using "git clone," so I know that's how I originally got it.

Please pull the latest and see if it gets you past the error.  I would have pushed it earlier but did not have my github keys handy.

Thanks.

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
toolbag
Jr. Member
*
Offline Offline

Activity: 42


View Profile
August 13, 2013, 10:51:06 PM
 #530

The latest update seems to have done the trick! Thanks!

Tips weclome and appreciated: 1LRxTZT76SE1UqB1ijvGp46qWJXV7BQczk
antonimasso
Member
**
Offline Offline

Activity: 73


View Profile
August 20, 2013, 05:56:26 PM
 #531

Hello,

I have a few questions:

- Do I need to wait for bitcoind to download the complete block files or can I run them in parallel?
- Once the initial data load finishes do I need to stop it and run it with python -m Abe.abe --config abe-my.conf?
- Will running python -m Abe.abe --config abe-my.conf insert new incoming blocks?

Thanks

John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
August 21, 2013, 01:14:35 PM
 #532

Hello,

I have a few questions:

- Do I need to wait for bitcoind to download the complete block files or can I run them in parallel?
- Once the initial data load finishes do I need to stop it and run it with python -m Abe.abe --config abe-my.conf?
- Will running python -m Abe.abe --config abe-my.conf insert new incoming blocks?

Thanks

You can run bitcoind and Abe in parallel.

The instructions call for --no-serve on the initial load, and this makes it exit as soon as it loads the last block, so you don't have to stop it.  The instructions say to run in web-server mode after the initial load, but you can run both at the same time, as long as you add "--datadir=[]" to the end of the web server command:

Code:
python -m Abe.abe --config abe-my.conf "--datadir=[]"

The "--datadir=[]" prevents the two processes from competing to insert blocks, which tends to cause trouble.

Unfortunately, Abe does not automatically insert new blocks every so often.  By default, (without "--datadir=[]") it inserts new blocks when you browse the site, but if you haven't for a long time, it can take Abe quite a while to catch up.  If this is a problem, I suggest you schedule a job to trigger this at regular intervals.  For example, I use this in crontab to check every minute:

Code:
* * * * * /usr/bin/wget -q -O /dev/null http://localhost:2750/

If you want to run a server as opposed to just personal use, I suggest configuring the public server not to load anything (by adding "--datadir=[]") and have a separate process listening on the local interface with a job such as the above to trigger loading new blocks.  Using a local listener keeps the loader single-threaded.  (This information should be in a readme, thanks for asking.)

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
antonimasso
Member
**
Offline Offline

Activity: 73


View Profile
August 22, 2013, 06:20:01 AM
 #533

Thanks for the detailed reply John. Once the initial load finishes I'll run some tests.
BTW, the last 50.000 blocks seem to take forever, I guess there are many more TXs.

weex
Legendary
*
Offline Offline

Activity: 1062


Give me your tired, your poor, your huddled...


View Profile
August 22, 2013, 11:09:57 PM
 #534

Does Abe work with leveldb ok? I have an instance that was running on Litecoin 0.6.3 and am trying to upgrade to 0.8.3.7 which uses leveldb.

If you can give me a quick idea on what I need to do to rebuild the db with this new client, that would be great.

Edit: Looks good. I just had to make a new database.

John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
August 23, 2013, 12:43:41 AM
 #535

Thanks for the detailed reply John. Once the initial load finishes I'll run some tests.
BTW, the last 50.000 blocks seem to take forever, I guess there are many more TXs.

Yes.  You can get an idea of how far it has to go by examining the datadir table.  "blkfile_number" corresponds to a file in the blocks directory (e.g. 100048 = blk00048.dat) and "blkfile_offset" shows how far in the file Abe has loaded.

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
antonimasso
Member
**
Offline Offline

Activity: 73


View Profile
August 26, 2013, 06:04:44 AM
 #536

Hello John,

Is it easy to implement N confirmations to addressbalance?

Also, is is possible to not show the results of the second SQL in the result set of the first SQL?

mysql> SELECT b.block_nTime, cc.chain_id, b.block_height, 0, b.block_hash, tx.tx_hash, txout.txout_pos, txout.txout_value FROM chain_candidate cc JOIN block b ON (b.block_id = cc.block_id) JOIN block_tx ON (block_tx.block_id = b.block_id) JOIN tx ON (tx.tx_id = block_tx.tx_id) JOIN txout ON (txout.tx_id = tx.tx_id) JOIN pubkey ON (pubkey.pubkey_id = txout.pubkey_id) WHERE pubkey.pubkey_hash = '645677d0601371051c9f723fecdd466c317a01cb';
+-------------+----------+--------------+---+------------------------------------------------------------------+------------------------------------------------------------------+-----------+-------------+
| block_nTime | chain_id | block_height | 0 | block_hash                                                       | tx_hash                                                          | txout_pos | txout_value |
+-------------+----------+--------------+---+------------------------------------------------------------------+------------------------------------------------------------------+-----------+-------------+
|  1375776420 |        1 |       250493 | 0 | 000000000000003dcdc75f30fd1c732944fb62693ac76fc217ec142207ab6ae3 | ac772eed99449ec15254a92f8292da1460ed650691b5b282521948a221dbbe78 |         0 |    18700000 |
|  1375886710 |        1 |       250732 | 0 | 000000000000001cb840f80614ed360324cf9073f0ffce25975a456c0ad41ea2 | f25f9a8fdb83ec5eb8ef99a3532e62dba4c158a6529d919cf5998462d20223a9 |         1 |    17650000 |
|  1376079367 |        1 |       251180 | 0 | 000000000000004d8a953fd3ce48c12f88c5e266b72aa816332e45a740fc377d | 0cd22a486659df7a7435edd426805016f0b26ed2779d947bba3212c9f29f4edd |         1 |   500000000 |
|  1376767233 |        1 |       252686 | 0 | 000000000000000af30ed5b0af8414ad2837c07218b45af00a1c2931f1f9700f | 4f345a39e4af5456f3e03f0322188c1701204df169d9db04844559d4afdcecbf |         1 |   489950000 |
|  1376812393 |        1 |       252796 | 0 | 0000000000000025271bf2ac0e0466ae77d00ab12baad1338bd5fc54468ae6b0 | f3f4eb433ea379b494ce1c22867217008f23da2dc6839668f434d0032665c184 |         1 |   479900000 |
|  1377100725 |        1 |       253425 | 0 | 000000000000004d69bfa91a524c5123192d2c93f84d24f4dc30e7d273e17949 | 0320695bf487ec494734c572a29e0e05521f70eb3e636e3673c810f4e2d6aaa3 |         0 |    10609000 |

|  1377330046 |        1 |       253965 | 0 | 000000000000002f6c0ac43224314d21b95139d8dd76893aa7b231a03d905fe8 | 01b3a6e93f5862ee00a4184b4851b84e6a9e5c63d0a4c122c2221f21bd41d315 |         1 |   480459000 |
|  1377446268 |        1 |       254181 | 0 | 000000000000000363f7e2298cdf320e1dd4867788a49b92bb3c366ad0b4f72c | 16a56d242e5bd5fd35a59f27decce1e0bb1b9f4cbc3ce670552bfd6f639f50b7 |         0 |     5000000 |
|  1377525653 |        1 |       254324 | 0 | 0000000000000024593882cf59758b73160b79042731ea16f4c4963f4da5fa04 | a5ccd0bf756a9b867d3ff95543974ebebf4bbe9c478108a2ee829b28bd63b94c |         0 |     1000000 |
|  1377527510 |        1 |       254325 | 0 | 00000000000000040bfcc8600878065faa2db3eb0d3a9434e922b2620fa8f133 | 11517d88ab64fffcc8779b341512b4e8a307d73e7aa00ce01733acb0cf90a99a |         0 |     1000000 |
|  1377527510 |        1 |       254325 | 0 | 00000000000000040bfcc8600878065faa2db3eb0d3a9434e922b2620fa8f133 | 7710da0657a055eeadf8f45b25530ecf07f4bd2b3871c19a0f17e44d9514e91d |         0 |     1000000 |
+-------------+----------+--------------+---+------------------------------------------------------------------+------------------------------------------------------------------+-----------+-------------+
11 rows in set (0.01 sec)

mysql> SELECT b.block_nTime, cc.chain_id, b.block_height, 1, b.block_hash,tx.tx_hash, txin.txin_pos, -prevout.txout_value FROM chain_candidate cc JOIN block b ON (b.block_id = cc.block_id) JOIN block_tx ON (block_tx.block_id = b.block_id) JOIN tx ON (tx.tx_id = block_tx.tx_id) JOIN txin ON (txin.tx_id = tx.tx_id) JOIN txout prevout ON (txin.txout_id = prevout.txout_id) JOIN pubkey ON (pubkey.pubkey_id = prevout.pubkey_id) WHERE pubkey.pubkey_hash = '645677d0601371051c9f723fecdd466c317a01cb';
+-------------+----------+--------------+---+------------------------------------------------------------------+------------------------------------------------------------------+----------+----------------------+
| block_nTime | chain_id | block_height | 1 | block_hash                                                       | tx_hash                                                          | txin_pos | -prevout.txout_value |
+-------------+----------+--------------+---+------------------------------------------------------------------+------------------------------------------------------------------+----------+----------------------+
|  1375886710 |        1 |       250732 | 1 | 000000000000001cb840f80614ed360324cf9073f0ffce25975a456c0ad41ea2 | f25f9a8fdb83ec5eb8ef99a3532e62dba4c158a6529d919cf5998462d20223a9 |        0 |            -18700000 |
|  1375887925 |        1 |       250734 | 1 | 00000000000000214294e44ef349ccb337b5038d5cfecc2c7a9b086d89d08f20 | 0244b7597b986d9433f719a4decb0ce46f636182099471f982c0cb0eb5dee674 |        0 |            -17650000 |
|  1376767233 |        1 |       252686 | 1 | 000000000000000af30ed5b0af8414ad2837c07218b45af00a1c2931f1f9700f | 4f345a39e4af5456f3e03f0322188c1701204df169d9db04844559d4afdcecbf |        0 |           -500000000 |
|  1376812393 |        1 |       252796 | 1 | 0000000000000025271bf2ac0e0466ae77d00ab12baad1338bd5fc54468ae6b0 | f3f4eb433ea379b494ce1c22867217008f23da2dc6839668f434d0032665c184 |        0 |           -489950000 |
|  1377330046 |        1 |       253965 | 1 | 000000000000002f6c0ac43224314d21b95139d8dd76893aa7b231a03d905fe8 | 01b3a6e93f5862ee00a4184b4851b84e6a9e5c63d0a4c122c2221f21bd41d315 |        0 |           -479900000 |
|  1377330046 |        1 |       253965 | 1 | 000000000000002f6c0ac43224314d21b95139d8dd76893aa7b231a03d905fe8 | 01b3a6e93f5862ee00a4184b4851b84e6a9e5c63d0a4c122c2221f21bd41d315 |        1 |            -10609000 |
+-------------+----------+--------------+---+------------------------------------------------------------------+------------------------------------------------------------------+----------+----------------------+
6 rows in set (0.00 sec)

Thanks

antonimasso
Member
**
Offline Offline

Activity: 73


View Profile
August 26, 2013, 07:27:19 PM
 #537

Reading your code I found this function, do_unspent, which makes exactly what I needed.

John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
August 26, 2013, 08:42:39 PM
 #538

Hello John,

Is it easy to implement N confirmations to addressbalance?

Yes, it should be pretty easy, since the underlying functions get_sent_and_last_block_id and get_received_and_last_block_id both accept a block_height.  Just make the new API pass N less than the current block number.  Please note that I have not had much time to work on new features lately.

Also, is is possible to not show the results of the second SQL in the result set of the first SQL?

I am not sure which API function you mean, but there is commingling of sent and received amounts, and to separate them would take a couple of hours of coding.  Can you point to another block explorer that does it how you want?  I usually try to copy their interfaces rather than invent my own.

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
HeroC
Hero Member
*****
Offline Offline

Activity: 742



View Profile WWW
August 27, 2013, 02:56:18 AM
 #539

Looks good, I bet it will be used with a lot of new altcoins. 

GPG: FA122C1A | IRC: HeroCC
antonimasso
Member
**
Offline Offline

Activity: 73


View Profile
August 27, 2013, 05:55:07 AM
 #540

I'm currently using blockchain.info API. Check out this URL to get the unspent outputs:

http://blockchain.info/unspent?address=1A9YA62QvpmE32wu9h4tgC3GHQiUhiTAXi

and this URL to get the total sent and received amounts (much clearer than blockexplorer.com IMO):

http://blockchain.info/address/1A9YA62QvpmE32wu9h4tgC3GHQiUhiTAXi?format=json


Pages: « 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 [27] 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 »
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!