Bitcoin Forum
December 07, 2016, 12:41:09 PM *
News: To be able to use the next phase of the beta forum software, please ensure that your email address is correct/functional.
 
   Home   Help Search Donate Login Register  
Pages: « 1 2 3 4 5 6 7 8 9 10 11 12 [13] 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 »
  Print  
Author Topic: [ANNOUNCE] Abe 0.7: Open Source Block Explorer Knockoff  (Read 175716 times)
John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
February 15, 2012, 05:30:35 PM
 #241

I've seen somewhere an Abe install with the SolidCoin2 chain but I can't seem to get it going on my install.
Code:
  File "Abe/DataStore.py", line 1493, in import_block
    raise MerkleRootMismatch(b['hash'], tx_hash_array)
MerkleRootMismatch: Block header Merkle root does not match its transactions. block hash=725ca2009ba1b949a820d3a48511dc21c7f8d76d06d8afb6abb5988f5a911a8b

I won't be surprised if SC2 has changed the Merkle tree hashing algorithm.  If you trust your copy of the block chain, you may get around this by simply deleting these lines from Abe/DataStore.py:

Code:
        # Verify Merkle root.
        if b['hashMerkleRoot'] != util.merkle(tx_hash_array):
            raise MerkleRootMismatch(b['hash'], tx_hash_array)

A long-term solution would be to add support for different algorithms per chain, or at least an option to skip the verification in a given chain.

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
1481114469
Hero Member
*
Offline Offline

Posts: 1481114469

View Profile Personal Message (Offline)

Ignore
1481114469
Reply with quote  #2

1481114469
Report to moderator
1481114469
Hero Member
*
Offline Offline

Posts: 1481114469

View Profile Personal Message (Offline)

Ignore
1481114469
Reply with quote  #2

1481114469
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1481114469
Hero Member
*
Offline Offline

Posts: 1481114469

View Profile Personal Message (Offline)

Ignore
1481114469
Reply with quote  #2

1481114469
Report to moderator
1481114469
Hero Member
*
Offline Offline

Posts: 1481114469

View Profile Personal Message (Offline)

Ignore
1481114469
Reply with quote  #2

1481114469
Report to moderator
1481114469
Hero Member
*
Offline Offline

Posts: 1481114469

View Profile Personal Message (Offline)

Ignore
1481114469
Reply with quote  #2

1481114469
Report to moderator
tiker
Sr. Member
****
Offline Offline

Activity: 459



View Profile WWW
February 15, 2012, 05:41:12 PM
 #242

Thanks.  Don't know if I trust it or not.. I'll send some SC from my windows client to this linux install and see if they show up.. guess that's the only way to tell.

Thanks.
eurekafag
Full Member
***
Offline Offline

Activity: 186


View Profile
February 15, 2012, 05:46:34 PM
 #243

Please run the script in this post: https://bitcointalk.org/index.php?topic=22785.msg707302#msg707302

I've committed a small change that I hope will stop these MySQL duplicate key errors.

Thanks, now it works. Just out of curiosity why don't you support MyISAM engine? I tried replacing innodb with myisam in the source but then abe complains about inability to create a transactional database. Isn't MyISAM faster than InnoDB? I don't know just read some articles comparing them so I'm asking you.
John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
February 15, 2012, 05:57:10 PM
 #244

Just out of curiosity why don't you support MyISAM engine? I tried replacing innodb with myisam in the source but then abe complains about inability to create a transactional database. Isn't MyISAM faster than InnoDB? I don't know just read some articles comparing them so I'm asking you.
As I understand, MyISAM is non-transactional.  It does not support rollback.  See: http://en.wikipedia.org/wiki/ACID

Abe relies on the ability to roll back the current transaction and recover when interrupted or in the event of an error.  You may find MyISAM works for a while, but when something goes wrong, typically you will have to rebuild the database.  With transactional tables, it normally suffices to restart Abe.

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
eurekafag
Full Member
***
Offline Offline

Activity: 186


View Profile
February 15, 2012, 06:56:21 PM
 #245

Hmm, I didn't experience something like this but it could be crucial for abe. Thanks for your help and answers, keep up the good work!
John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
February 15, 2012, 07:19:41 PM
 #246

Simple answer, Abe runs on many databases, all of them transactional.  This knowledge permits one to code with a certain set of assumptions that may not hold in a MyISAM environment.  Fearing inconsistencies in users' MyISAM tables, I put in the check that you discovered.

I think the changes needed to support MyISAM's semantics safely would be too invasive.  To use MyISAM unsafely is easy, just disable the check in Abe.DataStore._test_transaction, but don't expect support when you start to see wrong data.

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
tiker
Sr. Member
****
Offline Offline

Activity: 459



View Profile WWW
February 15, 2012, 08:32:54 PM
 #247

I've seen somewhere an Abe install with the SolidCoin2 chain but I can't seem to get it going on my install.
Code:
  File "Abe/DataStore.py", line 1493, in import_block
    raise MerkleRootMismatch(b['hash'], tx_hash_array)
MerkleRootMismatch: Block header Merkle root does not match its transactions. block hash=725ca2009ba1b949a820d3a48511dc21c7f8d76d06d8afb6abb5988f5a911a8b

I won't be surprised if SC2 has changed the Merkle tree hashing algorithm.  If you trust your copy of the block chain, you may get around this by simply deleting these lines from Abe/DataStore.py:

Code:
        # Verify Merkle root.
        if b['hashMerkleRoot'] != util.merkle(tx_hash_array):
            raise MerkleRootMismatch(b['hash'], tx_hash_array)

A long-term solution would be to add support for different algorithms per chain, or at least an option to skip the verification in a given chain.


Just an FYI, after doing some checking it turns out the block chain for SC2 isn't downloading correctly and is getting stuck.  The windows client is stuck at block 17999 (like the other posts on this board) so I can't trust anything coming down for SC2 at the moment.  I'm leaving the above lines in the code.

Thanks.
kstepyra
Jr. Member
*
Offline Offline

Activity: 52


View Profile
February 18, 2012, 02:05:47 AM
 #248

Hi there,

I've started using abe(initial check right now Smiley ), had to quit app and rerun it and after using ctrl-c  i lack one record :

Code:
mysql> SELECT count(*) FROM `block` WHERE block_id=116245;
+----------+
| count(*) |
+----------+
|        1 |
+----------+
1 row in set (0.00 sec)

mysql> SELECT count(*) FROM `block` WHERE block_id=116246;
+----------+
| count(*) |
+----------+
|        0 |
+----------+
1 row in set (0.00 sec)

mysql> SELECT count(*) FROM `block` WHERE block_id=116247;
+----------+
| count(*) |
+----------+
|        1 |
+----------+
1 row in set (0.00 sec)
commit should prevent that, right? i am ussing InnoDB.

Is here any possibility to recheck for missing data? EDIT: commits are only for tx's so maybe just wrong id given while selecting last block in app after starting it? like 116247 instead of 116246 while having 116245 put in mysqlDb?

Another thing - when i have bitcoin running - can i rerun app to check for new blocks, or i have to stop bitcoin?

and most important: how is this tool different from Bitcointools ? i am new to this tools, was using crappy json-rpc methods given in bitcoind : p
John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
February 18, 2012, 02:33:10 AM
 #249

had to quit app and rerun it and after using ctrl-c  i lack one record :
More likely it seems to me, you have one unassigned block_id.  This is not the same as a missing block.  block_id is an internal identifier, not visible in any HTML or API output.  So it doesn't really matter what the block_id is.  It will vary from one Abe instance to another for the same block.  Abe allocated a block_id but hadn't used it when you stopped it, so it remains unused.  This is not a problem.

By the way, the visible "block number" is called "height" internally.  Often height and block_id are numerically close, depending on how you load the data, but they will diverge if, for example, your block file contains side chains or you load Testnet or alt chain data along with the main BTC block chain.

Is here any possibility to recheck for missing data?
It would be nice to have a validation tool that checks for missing data.  We don't have that.  Part of it is covered by the foreign key constraints, and there is a tool (Abe/verify.py) that verifies the transaction Merkle hashes, but it could do much more.

If you haven't done anything like delete table rows, and barring any bugs, if Abe shows a block number, it has all data from the genesis through that block.

Another thing - when i have bitcoin running - can i rerun app to check for new blocks, or i have to stop bitcoin?
I have never noticed a problem running both at once.

And btw - how is this tool different from Bitcointools ? i am new to this tools, was using crappy json-rpc methods given in bitcoind : p
Abe borrows some low-level code from Bitcointools and adds many features, notably SQL database support, speed improvements through indexing, and the HTML interface similar to Bitcoin Block Explorer.

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
tiker
Sr. Member
****
Offline Offline

Activity: 459



View Profile WWW
February 22, 2012, 02:42:56 PM
 #250

Another question here...

http://blockexplorer.funkymonkey.org/block/00000000bc06d31f6fc31e99e3be7adfbc47c04e9bf56cddf1accda6cac69529

What does it mean when there are 2 "next block" links, one looking to be a valid block and 1 appearing to be part of a different chain?  I'm would take a guess and say 1 got orphaned and the other was accepted but the incorrect "to" address makes it look like it's part of another chain.


Edit:  Now that I look at it a bit more.. I think it is an orphaned block but also a bug where the block version code isn't being applied to properly show the correct addresses.
John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
February 22, 2012, 03:36:14 PM
 #251

Edit:  Now that I look at it a bit more.. I think it is an orphaned block but also a bug where the block version code isn't being applied to properly show the correct addresses.
Correct you are.

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
tiker
Sr. Member
****
Offline Offline

Activity: 459



View Profile WWW
February 26, 2012, 02:11:48 PM
 #252

I've run into a new problem.  Trying to import the bitcoin testnet, was working fine then this came up...

Code:

Failed to catch up {'blkfile_number': 1, 'dirname': '/home/ed/.bitcoin/testnet', 'chain_id': 20, 'id': Decimal('95'), 'blkfile_offset': 30108327}
Traceback (most recent call last):
  File "Abe/DataStore.py", line 2146, in catch_up
    store.catch_up_dir(dircfg)
  File "Abe/DataStore.py", line 2167, in catch_up_dir
    store.import_blkdat(dircfg, ds)
  File "Abe/DataStore.py", line 2279, in import_blkdat
    b = deserialize.parse_Block(ds)
  File "Abe/deserialize.py", line 176, in parse_Block
    d['auxpow'] = parse_AuxPow(vds)
  File "Abe/deserialize.py", line 153, in parse_AuxPow
    d = parse_MerkleTx(vds)
  File "Abe/deserialize.py", line 103, in parse_MerkleTx
    d = parse_Transaction(vds)
  File "Abe/deserialize.py", line 86, in parse_Transaction
    d['txIn'].append(parse_TxIn(vds))
  File "Abe/deserialize.py", line 45, in parse_TxIn
    d['sequence'] = vds.read_uint32()
  File "Abe/BCDataStream.py", line 71, in read_uint32
    def read_uint32(self): return self._read_num('<I')
  File "Abe/BCDataStream.py", line 110, in _read_num
    (i,) = struct.unpack_from(format, self.input, self.read_cursor)
OverflowError: Python int too large to convert to C long

Other chains continue to import fine so far from what I can tell.  It seems to be just this current spot in this chain.
John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
February 27, 2012, 01:17:07 AM
 #253

The testnet contains a block with an unusual value in the version field.  Abe thinks it is a merged-mining block.  I mean to provide a workaround, especially since the main BTC chain is vulnerable to it, but haven't got to it yet.  Abe will need a new kind of chain metadata with flags for features such as merged mining, rather than rely on a universal interpretation of block version.

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
tiker
Sr. Member
****
Offline Offline

Activity: 459



View Profile WWW
February 27, 2012, 01:46:07 AM
 #254

So I'm stuck at block 42156 on the testnet for now then?  It doesn't sound like a quick upgrade / change to Abe.  Smiley
John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
February 27, 2012, 02:06:02 AM
 #255

So I'm stuck at block 42156 on the testnet for now then?  It doesn't sound like a quick upgrade / change to Abe.  Smiley

To be compatible with Testnet, but incompatible with Namecoin and chains that use its proof-of-work format, remove or comment out these lines in Abe/deserialize.py, lines 175-176 in the latest commit:
Code:
 if d['version'] & (1 << 8):
    d['auxpow'] = parse_AuxPow(vds)

To be compatible with both, until I have a better fix, I suggest you run this query:
Code:
SELECT DISTINCT c.chain_name, b.block_version
FROM block b
JOIN chain_candidate cc ON (b.block_id = cc.block_id)
JOIN chain c ON (cc.chain_id = c.chain_id)

You will see something like:
Code:
chain_name | block_version
------------+---------------
 NcTestnet  |         65537
 NcTestnet  |             1
 Namecoin   |             1
 Namecoin   |         65537
 Namecoin   |         65793
 Bitcoin    |             1
 NcTestnet  |         65793

Make a list of the values under block_version that belongs to a network with Namecoin-style merged mining.  For the above output, the list would be (1, 65537, 65793) corresponding to Namecoin and NcTestnet.  Then add " and d['version'] in (LIST)" after "d['version'] & (1 << Cool" on line 175, e.g.:
Code:
  if (d['version'] & (1 << 8)) and d['version'] in (1, 65537, 65793):
    d['auxpow'] = parse_AuxPow(vds) 
Let us know how it goes.

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
February 27, 2012, 02:18:33 AM
 #256

A less fragile workaround would be to run a separate Abe process for loading Testnet using --no-serve and a copy of the software with the lines commented out.  Your server process would not be able to add new testnet blocks at every request the way it does for other chains, and you would have to schedule the Testnet process to run every minute or so.  But you might prefer this, since it should survive any new block versions introduced by the chains.

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
tiker
Sr. Member
****
Offline Offline

Activity: 459



View Profile WWW
February 27, 2012, 02:45:07 AM
 #257

The output from the SQL query is as follows:

Code:
chain_name block_version
Bitchip 1
Bitcoin 1
Bitcoin Testnet 1
Coiledcoin 1
Coiledcoin 1048577
Coiledcoin 1048833
Devcoin 1
Devcoin 196609
Devcoin 262145
Devcoin 262401
Fairbrix 1
GeistGeld 1
GeistGeld 257
i0coin 1
i0coin 131073
i0coin 131329
Ixcoin 1
Ixcoin 196609
Ixcoin 196865
Liquidcoin 1
Litecoin 1
Namecoin 1
Namecoin 65537
Namecoin 65793
Rucoin 1
Rucoin 196609
Rucoin 458753
Tenebrix 1

When you say namecoin like merged mining do you mean any chain that's compatible to be merged with namecoin?  In this case, it would be namecoin, ixcoin, i0coin, geistgeld, coiledcoin, devcoin and soon groupcoin (not imported yet).

That would be values  1, 1048577, 1048833, 196609, 262145, 262401, 257, 131073, 131329, 196609, 196865, 65537, 65793; and then whatever groupcoin reports and any other future chains.  Is this correct?


Would commenting out those lines to get the import running again, then un-commenting afterwords be an option or will every block from this point onwards have the problem?
John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
February 27, 2012, 03:00:40 AM
 #258

The output from the SQL query is as follows:

(snipped)
Wow!  I guess everybody creates their own block version to go along with the magic number and address version.

When you say namecoin like merged mining do you mean any chain that's compatible to be merged with namecoin?  In this case, it would be namecoin, ixcoin, i0coin, geistgeld, coiledcoin, devcoin and soon groupcoin (not imported yet).
That list is probably right.  In practice, it's whoever started with a copy of Namecoin's merged-mining code and didn't change it too much.  Small chains try to be compatible with tools like Abe.  Abe tries to be compatible with big chains. Smiley

That would be values  1, 1048577, 1048833, 196609, 262145, 262401, 257, 131073, 131329, 196609, 196865, 65537, 65793; and then whatever groupcoin reports and any other future chains.  Is this correct?
That looks about right.

Would commenting out those lines to get the import running again, then un-commenting afterwords be an option or will every block from this point onwards have the problem?
I don't know.  It depends whether the Testnet block was a one-time test or that miner solved more blocks.  Strange things happen on Testnet.

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
February 27, 2012, 05:06:50 PM
 #259

So I'm stuck at block 42156 on the testnet for now then?  It doesn't sound like a quick upgrade / change to Abe.  Smiley
I've committed a fix.  By default, only Bitcoin and Testnet are allowed to have any block version number.  If there are more such chains, uncomment ignore-bit8-chains and add them to the list in abe.conf.

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
tiker
Sr. Member
****
Offline Offline

Activity: 459



View Profile WWW
February 27, 2012, 07:54:18 PM
 #260

That worked.  Smiley

Uncommented the line, removed bitcoin so it shows "Bitcoin Testnet" (which is how i've named it here) and started.

First line reads:
Ignored bit8 in version 0x558d394b of chain_id 20

The import blocks are listed after that.

Thanks!   Smiley
Pages: « 1 2 3 4 5 6 7 8 9 10 11 12 [13] 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 »
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!