|
Title: Lookup bitcoin transaction Post by: mtomcdev on September 23, 2014, 11:23:13 AM All bitcoin blockchain explorers provide very quick transaction lookup functionality. I heard from someone in this forum that the the merkle tree allows the quick transaction lookup. I also read that the blockchain explorers actually maintains a relational database to store the the transaction data and that allows a quick lookup. My understanding was that the merkle tree itself is there to store the transactions in a verifiable manner instead of for quick transaction lookup. How the blockchain explorers actually works?
Title: Re: Lookup bitcoin transaction Post by: deepceleron on September 23, 2014, 12:36:38 PM Previous versions of Bitcoin indexed all transactions, allowing them to be looked up by RPC commands. The new database only indexes transactions pertinent to your wallet. If you want this functionality, you can add the txindex=1 option to your bitcoin.conf file and reindex the blockchain.
Other functionality such as arbitrary address lookups are not available in Bitcoin. The blockchain explorers do these queries by pulling all the blockchain info out into a separate database that then can be used by the front end. You can look at Abe as an open-source blockchain explorer to see how it is done: https://bitcointalk.org/index.php?topic=22785.0 Title: Re: Lookup bitcoin transaction Post by: mtomcdev on September 23, 2014, 03:48:51 PM Previous versions of Bitcoin indexed all transactions, allowing them to be looked up by RPC commands. The new database only indexes transactions pertinent to your wallet. If you want this functionality, you can add the txindex=1 option to your bitcoin.conf file and reindex the blockchain. Other functionality such as arbitrary address lookups are not available in Bitcoin. The blockchain explorers do these queries by pulling all the blockchain info out into a separate database that then can be used by the front end. You can look at Abe as an open-source blockchain explorer to see how it is done: https://bitcointalk.org/index.php?topic=22785.0 Thanks for your reply. That's what I thought that blockchain explorers use a database at the back-in. What is the implementation for the local database that indexes the blockchain using the txindex=1 option? Is it leveldb or a something similar key-value store? Title: Re: Lookup bitcoin transaction Post by: deepceleron on September 23, 2014, 03:55:35 PM Yes, it is exactly leveldb:
Bitcoin-Qt version 0.8.0 is now available from: http://sourceforge.net/projects/bitcoin/files/Bitcoin/bitcoin-0.8.0/ This is a major release designed to improve performance and handle the increasing volume of transactions on the network. Please report bugs using the issue tracker at github: https://github.com/bitcoin/bitcoin/issues How to Upgrade -------------- If you are running an older version, shut it down. Wait until it has completely shut down (which might take a few minutes for older versions), then run the installer (on Windows) or just copy over /Applications/Bitcoin-Qt (on Mac) or bitcoind/bitcoin-qt (on Linux). The first time you run after the upgrade a re-indexing process will be started that will take anywhere from 30 minutes to several hours, depending on the speed of your machine. Incompatible Changes -------------------- This release no longer maintains a full index of historical transaction ids by default, so looking up an arbitrary transaction using the getrawtransaction RPC call will not work. If you need that functionality, you must run once with -txindex=1 -reindex=1 to rebuild block-chain indices (see below for more details). Improvements ------------ ... LevelDB, a fast, open-source, non-relational database from Google, is now used to store transaction and block indices. LevelDB works much better on machines with slow I/O and is faster in general. Berkeley DB is now only used for the wallet.dat file (public and private wallet keys and transactions relevant to you). Pieter Wuille implemented many optimizations to the way transactions are verified, so a running, synchronized node uses less working memory and does much less I/O. He also implemented parallel signature checking, so if you have a multi-CPU machine all CPUs will be used to verify transactions. Title: Re: Lookup bitcoin transaction Post by: mtomcdev on September 23, 2014, 06:25:47 PM Yes, it is exactly leveldb: Bitcoin-Qt version 0.8.0 is now available from: http://sourceforge.net/projects/bitcoin/files/Bitcoin/bitcoin-0.8.0/ This is a major release designed to improve performance and handle the increasing volume of transactions on the network. Please report bugs using the issue tracker at github: https://github.com/bitcoin/bitcoin/issues How to Upgrade -------------- If you are running an older version, shut it down. Wait until it has completely shut down (which might take a few minutes for older versions), then run the installer (on Windows) or just copy over /Applications/Bitcoin-Qt (on Mac) or bitcoind/bitcoin-qt (on Linux). The first time you run after the upgrade a re-indexing process will be started that will take anywhere from 30 minutes to several hours, depending on the speed of your machine. Incompatible Changes -------------------- This release no longer maintains a full index of historical transaction ids by default, so looking up an arbitrary transaction using the getrawtransaction RPC call will not work. If you need that functionality, you must run once with -txindex=1 -reindex=1 to rebuild block-chain indices (see below for more details). Improvements ------------ ... LevelDB, a fast, open-source, non-relational database from Google, is now used to store transaction and block indices. LevelDB works much better on machines with slow I/O and is faster in general. Berkeley DB is now only used for the wallet.dat file (public and private wallet keys and transactions relevant to you). Pieter Wuille implemented many optimizations to the way transactions are verified, so a running, synchronized node uses less working memory and does much less I/O. He also implemented parallel signature checking, so if you have a multi-CPU machine all CPUs will be used to verify transactions. I should lookup the release note and find out this, so thanks for your time to answer my questions. Title: Re: Lookup bitcoin transaction Post by: micax1 on October 29, 2014, 08:14:04 PM Previous versions of Bitcoin indexed all transactions, allowing them to be looked up by RPC commands. The new database only indexes transactions pertinent to your wallet. If you want this functionality, you can add the txindex=1 option to your bitcoin.conf file and reindex the blockchain. Other functionality such as arbitrary address lookups are not available in Bitcoin. The blockchain explorers do these queries by pulling all the blockchain info out into a separate database that then can be used by the front end. You can look at Abe as an open-source blockchain explorer to see how it is done: https://bitcointalk.org/index.php?topic=22785.0 anybody have any estimates how long reindex take on avereage machine i7 16gb ram? Title: Re: Lookup bitcoin transaction Post by: rubensayshi on October 30, 2014, 12:06:31 PM afaik enabling txindex=1 in bitcoind takes a few hours, but not days like downloading the whole blockchain does.
the merkleroot (and basicly almost everything in the bitcoin protocol and bitcoind) is made to be fast / good at verifying and finding data of which you already know something (ie. getting a transaction by hash, getting the balance of your already 'imported' addresses etc). a blockexplorer (such as www.blocktrail.com which I'm the CTO of) can't work using bitcoind directly, it just doesn't provide what you need. I think most blockexplorers use a relational (or graph or nosql) database that allows them to look up data in a (way) more flexible manner. https://github.com/bitpay/insight is the only one that I know of that follows a schema more similar to bitcoind, using leveldb and basicly only storing data in a way that they need it directly. which is a pretty neat project for small usage, but it doesn't scale very well and you won't be able to get any data out of it that it's not already designed for (so it's hard to query for statistics etc). |