Bitcoin Forum
December 05, 2016, 04:32:37 AM *
News: Latest stable version of Bitcoin Core: 0.13.1  [Torrent].
 
   Home   Help Search Donate Login Register  
Pages: [1]
  Print  
Author Topic: [update] ABE: AGPL Block Explorer on github  (Read 5174 times)
John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
June 13, 2011, 04:32:03 AM
 #1

Edit: latest features & discussion: https://bitcointalk.org/index.php?topic=22785

Tired of waiting for theymos to release the original Block Explorer code, just before khal's Namecoin explorer announcement, I went and started my own based on Gavin's bitcointools.

Code: https://github.com/jtobey/bitcoin-abe
Service: http://abe.john-edwin-tobey.org/ and anywhere you host it

June 26 updates in orange
Features:
  • free and open source (GNU Affero General Public License)
  • block, transaction, and address history pages similar to http://blockexplorer.com/
  • block/tx/address search box
  • does not require patching bitcoind (uses slightly modified bitcointools, not getblock)
  • low block delay: often shows blocks not yet on http://blockexplorer.com/
  • chain page can list more than the 20 newest blocks
  • can handle several chains in the same database (Bitcoin, Testnet, Namecoin, ...)
  • real-time Bitcoin Days Destroyed and Average Coin Age data
  • written in Python and portable SQL, tested with PostgreSQL and Sqlite

Coming later, maybe:

Enjoy!

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
1480912357
Hero Member
*
Offline Offline

Posts: 1480912357

View Profile Personal Message (Offline)

Ignore
1480912357
Reply with quote  #2

1480912357
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1480912357
Hero Member
*
Offline Offline

Posts: 1480912357

View Profile Personal Message (Offline)

Ignore
1480912357
Reply with quote  #2

1480912357
Report to moderator
1480912357
Hero Member
*
Offline Offline

Posts: 1480912357

View Profile Personal Message (Offline)

Ignore
1480912357
Reply with quote  #2

1480912357
Report to moderator
1480912357
Hero Member
*
Offline Offline

Posts: 1480912357

View Profile Personal Message (Offline)

Ignore
1480912357
Reply with quote  #2

1480912357
Report to moderator
AntiVigilante
Member
**
Offline Offline

Activity: 98



View Profile
June 13, 2011, 07:34:48 AM
 #2

Egg celery ant.

Proposal: http://forum.bitcoin.org/index.php?topic=11541.msg162881#msg162881
Inception: https://github.com/bitcoin/bitcoin/issues/296
Goal: http://forum.bitcoin.org/index.php?topic=12536.0
Means: Code, donations, and brutal criticism. I've got a thick skin. 1Gc3xCHAzwvTDnyMW3evBBr5qNRDN3DRpq
just_someguy
Full Member
***
Offline Offline

Activity: 125


View Profile
June 23, 2011, 08:15:20 PM
 #3

Very nice!
Do you plan on implementing the json return option for queries?
John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
June 23, 2011, 09:20:35 PM
 #4

Very nice!
Thanks!

Quote
Do you plan on implementing the json return option for queries?
Do you mean the "raw" block and transaction pages?  It's on the to-do, but not near the top.  Patches are welcome.  Smiley

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
just_someguy
Full Member
***
Offline Offline

Activity: 125


View Profile
June 23, 2011, 09:53:37 PM
 #5

Yes, the raw stuff.

I ask because I believe writing against blockexplorer like implementations will be more common than writing against bitcoin itself... especially for ecommerce when you are generally only on the receiving side.

Not that theymos's isn't trustworthy but I wouldn't want to count on it being around if I was relying on it to run my software. Its also not fair to him.

Even for sending I think it will be a lost easier to have a small program decrypt an offline wallet, query your addresses, gather inputs for transaction creation, send, then shutdown.
It relies on having a trusted (and most likely private) db to query against so I think an open source version is very important.
Since it seems like blockexplorer won't be open sourced I'm glad to see someone is taking this on.


John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
June 27, 2011, 07:26:13 PM
 #6

Announced under Project Development: http://forum.bitcoin.org/index.php?topic=22785.0

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
ama
Member
**
Offline Offline

Activity: 98


View Profile
September 27, 2011, 07:00:17 PM
 #7

If a chain changes the directory where it keeps the block chain file, is it possible to change the configuration of Abe to get it continuing reading from the new directory without needing to completely rebuild the database?  I've tried changing it both in the abe.conf file and directly on the datadir table in the database to no avail.

If not, is it at least possible to start again with such a chain, leaving the rest of the chains as they are in the database?  Thank you!

Buy me a coffee: 18NMyRMSkCjk8FnmdfAuxbuSSET4ASwBL5
Or a decaf: Lcn3xp5iVwo6aSFx8VYaRxp8qJyPGvzgah
Bitcoin Pyramid
John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
September 27, 2011, 08:51:25 PM
 #8

If a chain changes the directory where it keeps the block chain file, is it possible to change the configuration of Abe to get it continuing reading from the new directory without needing to completely rebuild the database?  I've tried changing it both in the abe.conf file and directly on the datadir table in the database to no avail.
You shouldn't need to rebuild the database or even the chain.  If blk0001.dat is the same, except for possibly being appended to (by bitcoind) it should have no problem when you update datadir.dirname and restart the process.  Even if blk0001.dat is different (bitcoind restarted from an empty dir or copied blk0001.dat from another host) the load should go 10-100x faster, because Abe assumes that it has loaded a block when it recognizes the header hash and skips ahead in the file.

Please provide details of what you did and what error you got.

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
ama
Member
**
Offline Offline

Activity: 98


View Profile
September 27, 2011, 11:31:57 PM
 #9

Please provide details of what you did and what error you got.

I changed the datadir both in abe.conf and in the datadir table in the database.  The chain block was downloaded new.  No new blocks were added to the database.  I didn't get any error, it simply didn't update anymore.  Then I tryed replacing the contents of the datadir by the old ones, restarted the chain daemon, which continued adding blocks to the chain, and Abe worked fine too.  So, finally it worked out, with the old blk0001.dat file, but not with a new blk0001.dat file.

Buy me a coffee: 18NMyRMSkCjk8FnmdfAuxbuSSET4ASwBL5
Or a decaf: Lcn3xp5iVwo6aSFx8VYaRxp8qJyPGvzgah
Bitcoin Pyramid
John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
September 28, 2011, 01:02:51 AM
 #10

I changed the datadir both in abe.conf and in the datadir table in the database.  The chain block was downloaded new.  No new blocks were added to the database.  I didn't get any error, it simply didn't update anymore.  Then I tryed replacing the contents of the datadir by the old ones, restarted the chain daemon, which continued adding blocks to the chain, and Abe worked fine too.  So, finally it worked out, with the old blk0001.dat file, but not with a new blk0001.dat file.
This needs documenting, but in the log perhaps it mentioned forcing a rescan with "UPDATE datadir SET blkfile_number=1, blkfile_offset=0 WHERE dirname='...'"

(Replace "..." with the dirname.)

This ought to let it use the new file, and it will scan it 10x faster (or more) than if it had to rebuild the data structures.  I hope one day we'll have an administrative web interface to perform this task and others, like configuring chains, etc.

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
ama
Member
**
Offline Offline

Activity: 98


View Profile
September 28, 2011, 03:01:11 AM
 #11

This needs documenting, but in the log perhaps it mentioned forcing a rescan with "UPDATE datadir SET blkfile_number=1, blkfile_offset=0 WHERE dirname='...'"

(Replace "..." with the dirname.)

This ought to let it use the new file, and it will scan it 10x faster (or more) than if it had to rebuild the data structures.  I hope one day we'll have an administrative web interface to perform this task and others, like configuring chains, etc.

Yes, I see.  In the README.txt file it says:

"If you rebuild your block file, you must rebuild Abe's database or at least set datadir.blkfile_offset = 0 to force a rescan.  Abe does not currently handle block file changes gracefully."

It wasn't very clear to me how or where to set it, I didn't interpret it as updating a field in the database, so I tried setting datadir.blkfile_offset = 0 in the conf file, which made the daemon to fail to launch, of course.   I shpuld have known better!   Grin

Buy me a coffee: 18NMyRMSkCjk8FnmdfAuxbuSSET4ASwBL5
Or a decaf: Lcn3xp5iVwo6aSFx8VYaRxp8qJyPGvzgah
Bitcoin Pyramid
ama
Member
**
Offline Offline

Activity: 98


View Profile
September 30, 2011, 06:58:26 PM
 #12

Is it possible to rebuild the data of a single chain or maybe delete a chain and add it again?

I'm having some problems with a duplicated key in a chain which prevents the new blocks to be added to the database.

Buy me a coffee: 18NMyRMSkCjk8FnmdfAuxbuSSET4ASwBL5
Or a decaf: Lcn3xp5iVwo6aSFx8VYaRxp8qJyPGvzgah
Bitcoin Pyramid
ama
Member
**
Offline Offline

Activity: 98


View Profile
September 30, 2011, 10:09:53 PM
 #13

I have tried seting blkfile_offset to 0 in the database, delete all the data directory and download the block chain again, and it hasen't helped.  It keeps saying there is a duplicated key and refuses to save any more data into the database.

Buy me a coffee: 18NMyRMSkCjk8FnmdfAuxbuSSET4ASwBL5
Or a decaf: Lcn3xp5iVwo6aSFx8VYaRxp8qJyPGvzgah
Bitcoin Pyramid
ama
Member
**
Offline Offline

Activity: 98


View Profile
October 01, 2011, 02:32:07 AM
 #14

According to http://www.ruby-forum.com/topic/64428, the problem is caused by a bug in PostgreSQL.  They also explain what the solution is, but I cannot understand how they know what to do with the information provided by the OP, so I cannot really apply it to my problem here.  Apparently simply reindexing a table should fix it, but I can't see what table needs reindexing from the error I'm getting:

Warning: failed to catch up /home/ama/.tenebrix/tenebrix_data:  duplicate key violates unique constraint «block_txin_pkey»
{'blkfile_number': 1, 'dirname': '/home/ama/.tenebrix/tenebrix_data', 'chain_id': 11, 'id': Decimal('77'), 'blkfile_offset': 2386329}


I've reindexed block_txin, to no avail.  Any idea, please?

Buy me a coffee: 18NMyRMSkCjk8FnmdfAuxbuSSET4ASwBL5
Or a decaf: Lcn3xp5iVwo6aSFx8VYaRxp8qJyPGvzgah
Bitcoin Pyramid
ama
Member
**
Offline Offline

Activity: 98


View Profile
October 01, 2011, 03:19:31 AM
 #15

I've reindexed block_txin, to no avail.  Any idea, please?

I've asked in #PostgreSQL @ Freenode and they say there is nothing to fix in the database, because there is no index being used.  They say it's an application problem.  I guess I made a mistake while trying to run two instances of ABE (one for adding a chain and another one for the web service).

So, I'm back at the begining.  Can I somehow eliminate the chain to add it again later or do I need to destroy the database and build it again entirely (all chains)?

Buy me a coffee: 18NMyRMSkCjk8FnmdfAuxbuSSET4ASwBL5
Or a decaf: Lcn3xp5iVwo6aSFx8VYaRxp8qJyPGvzgah
Bitcoin Pyramid
ama
Member
**
Offline Offline

Activity: 98


View Profile
October 01, 2011, 06:41:50 PM
 #16

I've tried to build a different database with this chain with the idea that if it worked fine, I could then add the rest of the chains and replace the original database with the new one, but I'm facing the same problem.  So, it doesn't seem as if I made a mistake the first time, but as if the chain has some blocks with duplicated transactions.  I don't think there is anything which can be done at the chain level.  Is there a way to cope with such problem in Abe, please?

Buy me a coffee: 18NMyRMSkCjk8FnmdfAuxbuSSET4ASwBL5
Or a decaf: Lcn3xp5iVwo6aSFx8VYaRxp8qJyPGvzgah
Bitcoin Pyramid
ama
Member
**
Offline Offline

Activity: 98


View Profile
October 01, 2011, 07:54:50 PM
 #17

The chain is Tenebrix.  It seems that block 7177 and 7320 have dupe coinbases and the second transaction in block 9971 spends one of those.  It's while trying to insert this when it fails. 

Buy me a coffee: 18NMyRMSkCjk8FnmdfAuxbuSSET4ASwBL5
Or a decaf: Lcn3xp5iVwo6aSFx8VYaRxp8qJyPGvzgah
Bitcoin Pyramid
John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
October 02, 2011, 01:28:05 AM
 #18

The chain is Tenebrix.  It seems that block 7177 and 7320 have dupe coinbases and the second transaction in block 9971 spends one of those.  It's while trying to insert this when it fails. 
Good sleuthing!  Indeed, this is a bug.  While I think about the right fix, you might get past it with this:

Code:
diff --git a/Abe/DataStore.py b/Abe/DataStore.py
index a04333c..5eec013 100644
--- a/Abe/DataStore.py
+++ b/Abe/DataStore.py
@@ -1426,12 +1426,13 @@ store._ddl['txout_approx'],
 
         # Create rows in block_txin.
         for row in store.selectall("""
-            SELECT txin.txin_id, obt.block_id
+            SELECT txin.txin_id, MAX(obt.block_id)
               FROM block_tx bt
               JOIN txin ON (txin.tx_id = bt.tx_id)
               JOIN txout ON (txin.txout_id = txout.txout_id)
               JOIN block_tx obt ON (txout.tx_id = obt.tx_id)
-             WHERE bt.block_id = ?""", (block_id,)):
+             WHERE bt.block_id = ?
+             GROUP BY txin.txin_id""", (block_id,)):
             (txin_id, oblock_id) = row
             if store.is_descended_from(block_id, oblock_id):
                 store.sql("""

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
ama
Member
**
Offline Offline

Activity: 98


View Profile
October 02, 2011, 02:40:58 AM
 #19

Good sleuthing!  Indeed, this is a bug.  While I think about the right fix, you might get past it with this:

Thank you for the quick fix, John.  It worked fine.   Smiley

Buy me a coffee: 18NMyRMSkCjk8FnmdfAuxbuSSET4ASwBL5
Or a decaf: Lcn3xp5iVwo6aSFx8VYaRxp8qJyPGvzgah
Bitcoin Pyramid
Binford 6100
Hero Member
*****
Offline Offline

Activity: 504


PGP OTC WOT: EB7FCE3D


View Profile
October 02, 2011, 02:43:38 PM
 #20

https://bitcointalk.org/index.php?topic=22785.0
the other thread has longer 'proof of work' ; )

You can't build a reputation on what you are going to do.
Pages: [1]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!