[December 6, 2012:]

This is a Newbies-friendly copy of: https://bitcointalk.org/index.php?topic=22785.0

!!! Announcing Abe, the open-source block chain browser !!!

While theymos' Bitcoin Block Explorer has served us well, there are benefits that can only be had by an open-source version of this critical Bitcoin infrastructure.

• Privacy: you can view information without revealing your interest in it.
• Extensibility: you can add features that you want.
• Trust: you can audit the source code.
• Reliability: you can run it on your own server free of charge.

Abe provides much of Block Explorer's interface, including the list of recent blocks, a search tool, and detailed block, transaction, and address history pages.  Abe can also present multiple currencies such as Bitcoin, Namecoin, Tenebrix, and whatever FooCoin tomorrow may bring.  Abe lets you page back and forth in the block list and presents some new statistics like Average Coin Age.

Abe's interactive performance approaches Block Explorer's, though it is untested under BBE loads.  Abe supports a subset of Block Explorer's API.

Source code on Github: https://github.com/bitcoin-abe/bitcoin-abe
Demonstration site: http://abe.john-edwin-tobey.org/ (a 512MB VPS, can't handle heavy loads)
Requirements: Python and an RDBMS; tested with PostgreSQL, SQLite, MySQL, Oracle, ODBC, and DB2.
Donations: 1PWC7PNHL1SgvZaN7xEtygenKjWobWsCuf (BTC) and NJ3MSELK1cWnqUa6xhF2wUYAnz3RSrWXcK (NMC).

Enjoy!

December 6, 2012: Version 0.7.2 released.
August 31, 2011: Version 0.6 released.
August 16, 2011: Version 0.5 released.
July 15, 2011: See changes since July 4.

[Newbies post here:]

!!! Announcing Abe, the open-source block chain browser !!!

Newbies post here: https://bitcointalk.org/index.php?topic=51139.0

While theymos' Bitcoin Block Explorer has served us well, there are benefits that can only be had by an open-source version of this critical Bitcoin infrastructure.

• Privacy: you can view information without revealing your interest in it.
• Extensibility: you can add features that you want.
• Trust: you can audit the source code.
• Reliability: you can run it on your own server free of charge.

Abe provides much of Block Explorer's interface, including the list of recent blocks, a search tool, and detailed block, transaction, and address history pages.  Abe can also present multiple currencies such as Bitcoin, Namecoin, Tenebrix, and whatever FooCoin tomorrow may bring.  Abe lets you page back and forth in the block list and presents some new statistics like Average Coin Age.

Abe's interactive performance approaches Block Explorer's, though it is untested under BBE loads.  Abe supports a subset of Block Explorer's API.

Source code on Github: https://github.com/bitcoin-abe/bitcoin-abe
Demonstration site: http://abe.john-edwin-tobey.org/ (a 512MB VPS, can't handle heavy loads)
Requirements: Python and an RDBMS; tested with PostgreSQL, SQLite, MySQL, Oracle, ODBC, and DB2.
Donations: 1PWC7PNHL1SgvZaN7xEtygenKjWobWsCuf (BTC) and NJ3MSELK1cWnqUa6xhF2wUYAnz3RSrWXcK (NMC).

Enjoy!

December 6, 2012: Version 0.7.2 released.
August 31, 2011: Version 0.6 released.
August 16, 2011: Version 0.5 released.
July 15, 2011: See changes since July 4.

[Edit:]

Edit: latest features & discussion: https://bitcointalk.org/index.php?topic=22785

Tired of waiting for theymos to release the original Block Explorer code, just before khal's Namecoin explorer announcement, I went and started my own based on Gavin's bitcointools.

Code: https://github.com/jtobey/bitcoin-abe
Service: http://abe.john-edwin-tobey.org/ and anywhere you host it

June 26 updates in orange
Features:

• free and open source (GNU Affero General Public License)
• block, transaction, and address history pages similar to http://blockexplorer.com/
• block/tx/address search box
• does not require patching bitcoind (uses slightly modified bitcointools, not getblock)
• low block delay: often shows blocks not yet on http://blockexplorer.com/
• chain page can list more than the 20 newest blocks
• can handle several chains in the same database (Bitcoin, Testnet, Namecoin, ...)
• real-time Bitcoin Days Destroyed and Average Coin Age data
• written in Python and portable SQL, tested with PostgreSQL and Sqlite

Coming later, maybe:

• speedups to match http://blockexplorer.com/, I hope
• stats pages/API like http://blockexplorer.com/q
• see the TODO.txt file
• support for indirect block hashes along these lines
• Whatever you implement

Enjoy!

[adopters]

Suppose in 2008, in the middle of the credit crunch and bailout talk, an anonymous crypto expert had publicly offered to cure several of the world's monetary problems with a new currency much like Bitcoin.

Suppose, however, that he/she/they had made this offer contingent on a promise to pay US$10 million in tax money if the plan were to succeed.  Leaving aside the unlikelihood of TPTB allowing such a measure to come to a vote, would you have voted for the tax?  As for me, perhaps yes, $10 million seems about right, given the time and effort needed, plus a risk premium of a few hundred percent.

Now suppose the offer had been for $100 BILLION (10¹¹) rather than $10 million.  I don't know about you, but I would draw the line well below that figure.

It is hard to bring this up without being shouted down as "jealous" and ungrateful for the courage of the "early adopters".  So let us make a distinction between the early adopters who developed code and infrastructure and promoted Bitcoin in the hope of alleviating the world's ills, and early speculators who simply accumulated BTC in the hope that the block chain started by Satoshi Nakamoto in 2009 would eventually be worth a fortune.  Granted, some individuals may fall into both categories.

When we offer newcomers BTC at today's price (around $8) or suggest that they start mining at current difficulty/price ratios, we implicitly assign a value of $20 million to the first 50,000 blocks, which were solved at very low difficulty (under 0.001% of today's network strength) and have mostly not entered circulation (based on tracking a sample of block generation outputs through theymos' Block Explorer).  Some suspect that the private keys to much of this wealth were destroyed, but we can't know that, and I consider it unlikely.

As the BTC price rises, so does this implied value, until it acquires the character of an "entry tax" payable to the early speculators who hold large sums of (relatively) cheaply acquired BTC.

Does this mean I hate or envy Satoshi or Bitcoin?  No.  I think the world of Satoshi and would not wish to endanger his/her/their life/lives by pumping the value of his/her/their private keys to a ridiculous level.  Do you believe Satoshi's goal was to become filthy rich?  I rather think he wanted to reform a broken system and spare people from its abuses.  If Satoshi and the early speculators make a few tens of million USD out of it, I'll be happy for them.

So, can anything be done?  Yes, as I outlined, with backing from enough miners, one could start a block chain using more or less the same rules as 2009/Nakamoto/BTC but with a much higher initial difficulty and a genesis block based on the then-current BTC block.

Would such an action hurt Bitcoin?  Not at all, if by "Bitcoin" you mean the system Satoshi designed prior to 2009 and helped promote and debug during 2009-2010.  Would it hurt BTC, i.e., the exchange value of today's "bitcoin" (lowercase "b")?  Perhaps yes, if successful.  It would compete against BTC as an alternative currency with similar characteristics but less of an "early speculator tax".  Of course, those early speculators are probably savvy enough to sell some BTC and invest in the new currency if it bids well to outcompete BTC.

Would anyone like to work with me on software changes to facilitate the creation of "child" block chains as I outlined?

Would any miners run such software?  It would continue to build BTC blocks until it detects a signal of support for a child chain, and then it would devote 25% (configurable) of hashing power to the new chain as long as the new chain has at least 5% of BTC's strength.  I believe this kind of chain could coexist with BTC on the same network and even share BTC's blk*.dat files.

[201085 of 544296 tx freeable]

"nobody has done studies of the existing block chain to see how much space could be reclaimed."  --https://en.bitcoin.it/wiki/Scalability

So I gave it a go.  I hope someone will check my logic, especially as I am new to Python.

$ time src/bitcointools/freeable.py
loaded  125925  blocks
201085 of 544296 tx freeable (corrected: 387616 of 544296 tx)
63328760 of 168627001 tx bytes freeable (corrected: 123285900 of 168627001 tx bytes)

real   0m47.131s
user   0m43.799s
sys   0m2.744s
$ ls -l .bitcoin/blk0*.dat
-rw------- 1 jtobey jtobey 180119823 2011-05-23 00:42 .bitcoin/blk0001.dat

(edited: fixed bug http://forum.bitcoin.org/index.php?topic=9461.msg137059#msg137059) diff of bitcointools:

Code:

diff --git a/deserialize.py b/deserialize.py
index fe0cb09..a67645b 100644
--- a/deserialize.py
+++ b/deserialize.py
@@ -75,6 +75,7 @@ def deserialize_TxOut(d, owner_keys=None):
 
 def parse_Transaction(vds):
   d = {}
+  start = vds.read_cursor  # XXX breaks BCDataStream abstraction, do we care?
   d['version'] = vds.read_int32()
   n_vin = vds.read_compact_size()
   d['txIn'] = []
@@ -85,6 +86,7 @@ def parse_Transaction(vds):
   for i in xrange(n_vout):
     d['txOut'].append(parse_TxOut(vds))
   d['lockTime'] = vds.read_uint32()
+  d['tx'] = vds.input[start:vds.read_cursor]
   return d
 def deserialize_Transaction(d, transaction_index=None, owner_keys=None):
   result = "%d tx in, %d out\n"%(len(d['txIn']), len(d['txOut']))
diff --git a/freeable.py b/freeable.py
new file mode 100755
index 0000000..c79f458
--- /dev/null
+++ b/freeable.py
@@ -0,0 +1,87 @@
+#!/usr/bin/env python
+#
+# Read the block database, find out how many transactions
+# could be purged and how many bytes they take up.
+# TODO: find out how many Merkle hashes could be purged.
+#
+from bsddb.db import *
+import logging
+import os
+import sys
+
+from BCDataStream import *
+from block import scan_blocks
+from deserialize import parse_Block
+from util import determine_db_dir, create_env
+import Crypto.Hash.SHA256 as SHA256
+
+import binascii  # debugging
+
+def main():
+  import optparse
+  parser = optparse.OptionParser(usage="%prog [options]")
+  parser.add_option("--datadir", dest="datadir", default=None,
+                    help="Look for files here (defaults to bitcoin default)")
+  (options, args) = parser.parse_args()
+
+  if options.datadir is None:
+    db_dir = determine_db_dir()
+  else:
+    db_dir = options.datadir
+
+  try:
+    db_env = create_env(db_dir)
+  except DBNoSuchFileError:
+    logging.error("Couldn't open " + db_dir)
+    sys.exit(1)
+
+  blockfile = open(os.path.join(db_dir, "blk%04d.dat"%(1,)), "rb")
+  block_datastream = BCDataStream()
+  block_datastream.map_file(blockfile, 0)
+
+  blocks = []
+  def gather_stats(block_data):
+    block_datastream.seek_file(block_data['nBlockPos'])
+    blocks.append(parse_Block(block_datastream))
+    return True
+
+  scan_blocks(db_dir, db_env, gather_stats)
+  blocks.reverse()
+  print 'loaded ', len(blocks), ' blocks'
+
+  tx = {}
+  total_tx = 0
+  freeable_tx = 0
+  total_bytes = 0
+  freeable_bytes = 0
+
+  for data in blocks:
+    coinbase = True
+    for txn in data['transactions']:
+      tx_hash = SHA256.new(SHA256.new(txn['tx']).digest()).digest()
+      #print '> ', binascii.hexlify(tx_hash)
+      tx_bytes = len(txn['tx'])
+      tx[tx_hash] = (tx_bytes, len(txn['txOut']))
+      total_tx += 1
+      total_bytes += tx_bytes
+
+      if coinbase:
+        coinbase = False
+      else:
+        for txin in txn['txIn']:
+          #print '< ', binascii.hexlify(txin['prevout_hash'])
+          (bytes, live) = tx[txin['prevout_hash']]
+          if live == 1:
+            freeable_bytes += bytes
+            freeable_tx += 1
+            del tx[txin['prevout_hash']]
+          else:
+            tx[txin['prevout_hash']] = (bytes, live - 1)
+
+  db_env.close()
+
+  print freeable_tx, 'of', total_tx, 'tx freeable'
+  print freeable_bytes, 'of', total_bytes, 'tx bytes freeable'
+
+if __name__ == '__main__':
+    main()

Yet another Feeping Creature, but it's tame, oh so tame...

https://github.com/bitcoin/bitcoin/pull/245

import key from PEM file

Motivation: Create keys on a secure system using only OpenSSL or similar software. Receive BTC, then when ready to spend them, use importkey in a running client.

importkey <file> <account>
Reads a PEM-encoded keypair from file and adds it to the wallet.
To create a keypair with OpenSSL, use:
openssl ecparam -name secp256k1 -out NEW_KEY.pem -genkey
Returns the key's bitcoin address.

$ bitcoind importkey ~/NEW_KEY.pem TestAcct
mt5M3Qa7fXsUV3bK6WtWTZEvXY2M1UPEgv

Bug: I'd like to safeguard against overwriting a key in the wallet with bogus data.
Bug: I don't understand what has to be mutexed.
Bug: Should do anything possible to make sure the imported key is valid.

Note: I did not implement the corresponding export function, because my use case does not require it.  I will be happy to implement it if this will improve the patch's chance of acceptance.

Note: To be convenient, one (I) would write a little script to read one of these PEM files and print a Bitcoin address.  All key handling would be possible offline, and certainly in the absence of a block chain, until spend time.  For my next trick, expect offline transaction signing and an importtx function.

Kind regards,
John