Lookup bitcoin transaction

[mtomcdev]

All bitcoin blockchain explorers provide very quick transaction lookup functionality. I heard from someone in this forum that the the merkle tree allows the quick transaction lookup. I also read that the blockchain explorers actually maintains a relational database to store the the transaction data and that allows a quick lookup. My understanding was that the merkle tree itself is there to store the transactions in a verifiable manner instead of for quick transaction lookup. How the blockchain explorers actually works?

[deepceleron]

Previous versions of Bitcoin indexed all transactions, allowing them to be looked up by RPC commands. The new database only indexes transactions pertinent to your wallet. If you want this functionality, you can add the txindex=1 option to your bitcoin.conf file and reindex the blockchain.

Other functionality such as arbitrary address lookups are not available in Bitcoin. The blockchain explorers do these queries by pulling all the blockchain info out into a separate database that then can be used by the front end. You can look at Abe as an open-source blockchain explorer to see how it is done: https://bitcointalk.org/index.php?topic=22785.0

[mtomcdev]

Previous versions of Bitcoin indexed all transactions, allowing them to be looked up by RPC commands. The new database only indexes transactions pertinent to your wallet. If you want this functionality, you can add the txindex=1 option to your bitcoin.conf file and reindex the blockchain.

Other functionality such as arbitrary address lookups are not available in Bitcoin. The blockchain explorers do these queries by pulling all the blockchain info out into a separate database that then can be used by the front end. You can look at Abe as an open-source blockchain explorer to see how it is done: https://bitcointalk.org/index.php?topic=22785.0

Thanks for your reply. That's what I thought that blockchain explorers use a database at the back-in.

What is the implementation for the local database that indexes the blockchain using the txindex=1 option? Is it leveldb or a something similar key-value store?

[deepceleron]

Yes, it is exactly leveldb:

Bitcoin-Qt version 0.8.0 is now available from:
  http://sourceforge.net/projects/bitcoin/files/Bitcoin/bitcoin-0.8.0/

This is a major release designed to improve performance and handle the
increasing volume of transactions on the network.

Please report bugs using the issue tracker at github:
  https://github.com/bitcoin/bitcoin/issues

How to Upgrade
--------------

If you are running an older version, shut it down. Wait
until it has completely shut down (which might take a few minutes for older
versions), then run the installer (on Windows) or just copy over
/Applications/Bitcoin-Qt (on Mac) or bitcoind/bitcoin-qt (on Linux).

The first time you run after the upgrade a re-indexing process will be
started that will take anywhere from 30 minutes to several hours,
depending on the speed of your machine.

Incompatible Changes
--------------------

This release no longer maintains a full index of historical transaction ids
by default, so looking up an arbitrary transaction using the getrawtransaction
RPC call will not work. If you need that functionality, you must run once
with -txindex=1 -reindex=1 to rebuild block-chain indices (see below for more
details).

Improvements
------------

...

LevelDB, a fast, open-source, non-relational database from Google, is
now used to store transaction and block indices.  LevelDB works much better
on machines with slow I/O and is faster in general. Berkeley DB is now only
used for the wallet.dat file (public and private wallet keys and transactions
relevant to you).

Pieter Wuille implemented many optimizations to the way transactions are
verified, so a running, synchronized node uses less working memory and does
much less I/O. He also implemented parallel signature checking, so if you
have a multi-CPU machine all CPUs will be used to verify transactions.

[mtomcdev]

Yes, it is exactly leveldb:

Bitcoin-Qt version 0.8.0 is now available from:
  http://sourceforge.net/projects/bitcoin/files/Bitcoin/bitcoin-0.8.0/

This is a major release designed to improve performance and handle the
increasing volume of transactions on the network.

Please report bugs using the issue tracker at github:
  https://github.com/bitcoin/bitcoin/issues

How to Upgrade
--------------

If you are running an older version, shut it down. Wait
until it has completely shut down (which might take a few minutes for older
versions), then run the installer (on Windows) or just copy over
/Applications/Bitcoin-Qt (on Mac) or bitcoind/bitcoin-qt (on Linux).

The first time you run after the upgrade a re-indexing process will be
started that will take anywhere from 30 minutes to several hours,
depending on the speed of your machine.

Incompatible Changes
--------------------

This release no longer maintains a full index of historical transaction ids
by default, so looking up an arbitrary transaction using the getrawtransaction
RPC call will not work. If you need that functionality, you must run once
with -txindex=1 -reindex=1 to rebuild block-chain indices (see below for more
details).

Improvements
------------

...

LevelDB, a fast, open-source, non-relational database from Google, is
now used to store transaction and block indices.  LevelDB works much better
on machines with slow I/O and is faster in general. Berkeley DB is now only
used for the wallet.dat file (public and private wallet keys and transactions
relevant to you).

Pieter Wuille implemented many optimizations to the way transactions are
verified, so a running, synchronized node uses less working memory and does
much less I/O. He also implemented parallel signature checking, so if you
have a multi-CPU machine all CPUs will be used to verify transactions.

I should lookup the release note and find out this, so thanks for your time to answer my questions.

[micax1]

Previous versions of Bitcoin indexed all transactions, allowing them to be looked up by RPC commands. The new database only indexes transactions pertinent to your wallet. If you want this functionality, you can add the txindex=1 option to your bitcoin.conf file and reindex the blockchain.

Other functionality such as arbitrary address lookups are not available in Bitcoin. The blockchain explorers do these queries by pulling all the blockchain info out into a separate database that then can be used by the front end. You can look at Abe as an open-source blockchain explorer to see how it is done: https://bitcointalk.org/index.php?topic=22785.0

anybody have any estimates how long reindex take on avereage machine i7 16gb ram?

[rubensayshi]

afaik enabling txindex=1 in bitcoind takes a few hours, but not days like downloading the whole blockchain does.

the merkleroot (and basicly almost everything in the bitcoin protocol and bitcoind) is made to be fast / good at verifying and finding data of which you already know something (ie. getting a transaction by hash, getting the balance of your already 'imported' addresses etc).

a blockexplorer (such as www.blocktrail.com which I'm the CTO of) can't work using bitcoind directly, it just doesn't provide what you need.
I think most blockexplorers use a relational (or graph or nosql) database that allows them to look up data in a (way) more flexible manner.

https://github.com/bitpay/insight is the only one that I know of that follows a schema more similar to bitcoind, using leveldb and basicly only storing data in a way that they need it directly.
which is a pretty neat project for small usage, but it doesn't scale very well and you won't be able to get any data out of it that it's not already designed for (so it's hard to query for statistics etc).