|
Title: Reading the block chain with a library? Post by: Xenland on April 22, 2012, 06:54:57 AM How do I read the Bitcoin blockchain with an api or library with a programming language like php, python or java?
Title: Re: Reading the block chain with a library? Post by: ThomasV on April 22, 2012, 06:58:17 AM How do I read the Bitcoin blockchain with an api or library with a programming language like php, python or java? use libbitcoin Title: Re: Reading the block chain with a library? Post by: kokjo on April 22, 2012, 06:58:30 AM libbitcoin: https://github.com/libbitcoin/libbitcoin
Title: Re: Reading the block chain with a library? Post by: arby on April 22, 2012, 07:07:40 AM libbitoin as i see is in cpp, not php or java, python as the OP mentioned.
I think he is interested in using this on linux, and not using any executables. I am searching for this also, but have not found anything yet. What I found that you can reliably use with PHP is: https://blockchain.info/api https://blockchain.info/q To explore the blockchain. Title: Re: Reading the block chain with a library? Post by: kokjo on April 22, 2012, 07:09:42 AM libbitoin as i see is in cpp, not php or java, python as the OP mentioned. swig!I thin he is interested in using this on linux, and not using any executables. Title: Re: Reading the block chain with a library? Post by: arby on April 22, 2012, 07:14:45 AM libbitoin as i see is in cpp, not php or java, python as the OP mentioned. swig!I thin he is interested in using this on linux, and not using any executables. Ok I will give that a try also, thanks for the info Title: Re: Reading the block chain with a library? Post by: ThomasV on April 22, 2012, 07:54:46 AM libbitoin as i see is in cpp, not php or java, python as the OP mentioned. yes, it has a python frontend Title: Re: Reading the block chain with a library? Post by: Xenland on April 22, 2012, 08:35:55 PM Nice responses mates, libbitcoin should work fine for what i need
Title: Re: Reading the block chain with a library? Post by: Xenland on April 23, 2012, 05:54:10 AM So how do I use this libbitcoin thing, for example something simple like a balance of a particular address, or a list of details of a particular address? (with python)
Title: Re: Reading the block chain with a library? Post by: kokjo on April 23, 2012, 07:31:27 AM So how do I use this libbitcoin thing, for example something simple like a balance of a particular address, or a list of details of a particular address? (with python) don't know, try read the api... im not a libbitcoin guru.Title: Re: Reading the block chain with a library? Post by: Xenland on April 23, 2012, 08:49:31 AM Libbitcoin is quite confusing even looking through the source code, are there any alternatives to libbitcoin that could help me query the blockchain?
Title: Re: Reading the block chain with a library? Post by: etotheipi on April 23, 2012, 02:30:11 PM Armory is perfect this. It is C++ accessed in Python (via SWIG). You only ever touch python.
I'm not at my development computer now but there is example code on my github page (https://github.com/etotheipi/BitcoinArmory) (though many of the examples were created before RAM reduction, they will mostly still work) Code: from armoryengine import * When I get home later, I can post more example code for how to access what you want. If you tell me what you're trying to do with it, I can write more directed code samples for you. Title: Re: Reading the block chain with a library? Post by: Xenland on April 23, 2012, 08:31:30 PM Armory is perfect this. It is C++ accessed in Python (via SWIG). You only ever touch python. I'm not at my development computer now but there is example code on my github page (https://github.com/etotheipi/BitcoinArmory) (though many of the examples were created before RAM reduction, they will mostly still work) Code: from armoryengine import * When I get home later, I can post more example code for how to access what you want. If you tell me what you're trying to do with it, I can write more directed code samples for you. Nice I heard about armory but I thought it was just a GUI client with advanced features. I'll try it out with python see how it is. Title: Re: Reading the block chain with a library? Post by: etotheipi on April 24, 2012, 01:05:10 AM Here you go! I just put together a script that not only scans the blockchain, it does something useful! (it's at the bottom of the post)
Timings will vary depending on RAM. If you have a lot of RAM, just about the whole blockchain will be cached from the first scan, and rescans will be nearly instantaneous. If you want to avoid rescans, make sure everything is added to your wallet and registered with TheBDM.registerWallet before BDM_LoadBlockChain(). The sample python script at the bottom of this post produced the following output (the difficulty changes): Code: Collect all difficulty changes... And here is the script itself. It illustrates a variety of ways you can access block data. Mainly, scanning for addresses in the blockchain with balances and unspent outputs, and walking through every TxOut of every Tx of every Block -- we grab the address from ever standard TxOut and add it to a set() object which only allows unique addresses. Code: from armoryengine import * This scanning is lower-level than what happens in Armory, so you are mainly using C++ objects. You can find a full list of all the methods in BlockObjRef.h and BlockUtils.h. Please ask questions if something isn't clear, or if you want to use some functionality not exemplified here. Title: Re: Reading the block chain with a library? Post by: Sukrim on April 24, 2012, 01:11:23 AM If you tell me what you're trying to do with it, I can write more directed code samples for you. I personally want to get the timestamp of blocks with a certain block height once per day, but didn't get around to looking at either libbitcoin or Armory.Ideally it would be something like "TheBDM.getHeaderByHeight(12345).getTimestamp()"... Also for armory I still have to run bitcoind to get the blockchain updated, right? Edit: Wow - this is awesome, you manage to answer my posts before I even post them! :o Edit2: And I even guessed both function names correctly to the letter! This is getting creepy... Title: Re: Reading the block chain with a library? Post by: etotheipi on April 24, 2012, 01:13:40 AM If you tell me what you're trying to do with it, I can write more directed code samples for you. I personally want to get the timestamp of blocks with a certain block height once per day, but didn't get around to looking at either libbitcoin or Armory.Ideally it would be something like "TheBDM.getHeaderByHeight(12345).getTimestamp()"... Also for armory I still have to run bitcoind to get the blockchain updated, right? To use Armory, you need bitcoind running to receive blockchain updates and send&receive transactions. For the example code above, you only need to have the blk0001.dat file produced by the bitcoind, but it doesn't need to be running when you run the script. Edit: Wow, I must have telepathy. Glad I could answer your question... before you asked it! Title: Re: Reading the block chain with a library? Post by: Xenland on April 24, 2012, 05:55:41 AM How does one just query the balance of a Bitcoin address with armory? is there any calculations involved(like looping over tx history and sum up a balance) or can I just call it and print out a simple balance?
Im' sure a lot of programmers could benefit the solution to this question to write their own software to use armory for like making their own Blockchain.info website or w/e. Title: Re: Reading the block chain with a library? Post by: etotheipi on April 24, 2012, 03:53:52 PM How does one just query the balance of a Bitcoin address with armory? is there any calculations involved(like looping over tx history and sum up a balance) or can I just call it and print out a simple balance? Im' sure a lot of programmers could benefit the solution to this question to write their own software to use armory for like making their own Blockchain.info website or w/e. The code was in my example, but I guess my example was too long. So here's an extraction of it. Code: from armoryengine import * Title: Re: Reading the block chain with a library? Post by: Xenland on April 25, 2012, 12:02:20 AM How does one just query the balance of a Bitcoin address with armory? is there any calculations involved(like looping over tx history and sum up a balance) or can I just call it and print out a simple balance? Im' sure a lot of programmers could benefit the solution to this question to write their own software to use armory for like making their own Blockchain.info website or w/e. The code was in my example, but I guess my example was too long. So here's an extraction of it. Code: from armoryengine import * Oh yeah I noticed that part of your code, I just assumed that there was a quick way of going about this since it takes a few seconds to scan the block chain for one address and I'm looking for a way to scan and update the database for multiple addresses. Does that sound possible or am I stuck with this route you presented me? Thanks for your help btw I appreciate it. Title: Re: Reading the block chain with a library? Post by: etotheipi on April 25, 2012, 01:20:07 AM Oh yeah I noticed that part of your code, I just assumed that there was a quick way of going about this since it takes a few seconds to scan the block chain for one address and I'm looking for a way to scan and update the database for multiple addresses. Does that sound possible or am I stuck with this route you presented me? Thanks for your help btw I appreciate it. You can use as many addAddress_1_() calls as you want before a rescan, and the rescan will still take the same amount of time. Load 20 addresses into your wallet before loading the blockchain, and it will take the same time as if you loaded in 10,000. After the blockchain is loaded, if you add any new addresses to the wallet, the chain will be re-scanned on the "scanBlockchainForTx()" call, and again, it won't matter how many addresses that is, it'll take the same amount of time. Title: Re: Reading the block chain with a library? Post by: Pieter Wuille on April 25, 2012, 01:55:28 AM If you tell me what you're trying to do with it, I can write more directed code samples for you. I personally want to get the timestamp of blocks with a certain block height once per day, but didn't get around to looking at either libbitcoin or Armory.Ideally it would be something like "TheBDM.getHeaderByHeight(12345).getTimestamp()"... libbitcoin or armory's library will surely give you a much more flexible interface, but you *can* just do this using bitcoind as well. See the getblockhash and getblock RPC commands. Title: Re: Reading the block chain with a library? Post by: Xenland on April 25, 2012, 03:57:50 AM Oh yeah I noticed that part of your code, I just assumed that there was a quick way of going about this since it takes a few seconds to scan the block chain for one address and I'm looking for a way to scan and update the database for multiple addresses. Does that sound possible or am I stuck with this route you presented me? Thanks for your help btw I appreciate it. Oh yeah I noticed that part of your code, I just assumed that there was a quick way of going about this since it takes a few seconds to scan the block chain for one address and I'm looking for a way to scan and update the database for multiple addresses. Does that sound possible or am I stuck with this route you presented me? Thanks for your help btw I appreciate it. You can use as many addAddress_1_() calls as you want before a rescan, and the rescan will still take the same amount of time. Load 20 addresses into your wallet before loading the blockchain, and it will take the same time as if you loaded in 10,000. After the blockchain is loaded, if you add any new addresses to the wallet, the chain will be re-scanned on the "scanBlockchainForTx()" call, and again, it won't matter how many addresses that is, it'll take the same amount of time. Nice work btw Title: Re: Reading the block chain with a library? Post by: etotheipi on April 25, 2012, 04:40:06 PM I started the whole project because I wanted blockchain access from python. I quickly found out that it just wasn't possible to scan the blockchain with python, at least if I wanted it to finish within an hour. So the C++ layer does that and SWIG makes it possible to use it from python. Unfortunately, SWIG has some quirks, such as difficulties with function overloading (hence the addAddress_1_() method instead of addAddress(), etc). But, in general, it's been extraordinarily reliable. Most of the C++ classes and structures are available from python as if they were native.
That's why I recommended looking at BlockUtils.h and BlockObjRef.h, since that lists the all the functions available for most of the object types you will need. Please let me know if there's more things you want to do with it, and I'll add more example code. I had always wanted to setup some kind of tutorial for describing how to use armoryengine like this, but I never had a good excuse until now :) P.S. - Pieter is right... a lot of the much simpler stuff could be done by just running bitcoind and scripting RPC calls to it. But if you ever want to expand the functionality, I think a full library will be worth the time investment. Title: Re: Reading the block chain with a library? Post by: Xenland on April 25, 2012, 08:17:19 PM I tried the bitcoind getblockhash thing... I had no idea what information it was presenting me or how'd I utilize that information.
I ended up using Abe since it was best for my situation. Which was for me to be able to query things with PHP and MySql databases. At least myself and others have a thread to checkout later when they need some good bitcoin libraries. Title: Re: Reading the block chain with a library? Post by: Pieter Wuille on April 26, 2012, 12:11:24 PM $ ./bitcoind getblockhash 100000 000000000003ba27aa200b1cecaad478d2b00432346c3f1f3986da1afd33e506 $ ./bitcoind getblock 000000000003ba27aa200b1cecaad478d2b00432346c3f1f3986da1afd33e506 { "hash" : "000000000003ba27aa200b1cecaad478d2b00432346c3f1f3986da1afd33e506", "size" : 957, "height" : 100000, "version" : 1, "merkleroot" : "f3e94742aca4b5ef85488dc37c06c3282295ffec960994b2c0d5ac2a25a95766", "time" : 1293623863, "nonce" : 274148111, "bits" : "1b04864c", "difficulty" : 14484.16236123, "tx" : [ "8c14f0db3df150123e6f3dbbf30f8b955a8249b62ac1d1ff16284aefa3d06d87", "fff2525b8931402dd09222c50775608f75787bd2b87e56995a7bdd30f79702c4", "6359f0868171b1d194cbee1af2f16ea598ae8fad666d9b012c8ed2b79a236ec4", "e9a66845e05d5abc0ad04ec80f774a7e585c6e8db975962d069a522137b80c1d" ], "previousblockhash" : "000000000002d01c1fccc21636b607dfd930d31d01c3a62104612a1719011250", "nextblockhash" : "00000000000080b66c911bd5ba14a74260057311eaeb1982802f7010f1a9f090" } First call getblochash to find the hash of a block at a particular height, then use getblock to query information about the block with that given hash. The "time" field will tell you the block's timestamp (in seconds since epoch). Title: Re: Reading the block chain with a library? Post by: Xenland on April 26, 2012, 11:49:54 PM $ ./bitcoind getblockhash 100000 000000000003ba27aa200b1cecaad478d2b00432346c3f1f3986da1afd33e506 $ ./bitcoind getblock 000000000003ba27aa200b1cecaad478d2b00432346c3f1f3986da1afd33e506 { "hash" : "000000000003ba27aa200b1cecaad478d2b00432346c3f1f3986da1afd33e506", "size" : 957, "height" : 100000, "version" : 1, "merkleroot" : "f3e94742aca4b5ef85488dc37c06c3282295ffec960994b2c0d5ac2a25a95766", "time" : 1293623863, "nonce" : 274148111, "bits" : "1b04864c", "difficulty" : 14484.16236123, "tx" : [ "8c14f0db3df150123e6f3dbbf30f8b955a8249b62ac1d1ff16284aefa3d06d87", "fff2525b8931402dd09222c50775608f75787bd2b87e56995a7bdd30f79702c4", "6359f0868171b1d194cbee1af2f16ea598ae8fad666d9b012c8ed2b79a236ec4", "e9a66845e05d5abc0ad04ec80f774a7e585c6e8db975962d069a522137b80c1d" ], "previousblockhash" : "000000000002d01c1fccc21636b607dfd930d31d01c3a62104612a1719011250", "nextblockhash" : "00000000000080b66c911bd5ba14a74260057311eaeb1982802f7010f1a9f090" } First call getblochash to find the hash of a block at a particular height, then use getblock to query information about the block with that given hash. The "time" field will tell you the block's timestamp (in seconds since epoch). Title: Re: Reading the block chain with a library? Post by: Sukrim on April 27, 2012, 12:01:20 AM I think that was rather related to my question about block timestamps... thanks by the way!
Title: Re: Reading the block chain with a library? Post by: Xenland on April 27, 2012, 12:49:32 AM I think that was rather related to my question about block timestamps... thanks by the way! Still either way it helped me understand what those functions are used for and I'm still quite curious too. Apologies to any awkwardness felt from my responses.Title: Re: Reading the block chain with a library? Post by: Sukrim on May 08, 2012, 10:23:01 PM This scanning is lower-level than what happens in Armory, so you are mainly using C++ objects. You can find a full list of all the methods in BlockObjRef.h and BlockUtils.h. Please ask questions if something isn't clear, or if you want to use some functionality not exemplified here. I'm currently trying to write a program that connects multiple addresses to "entities". If there are 2 (or more) inputs used in a transaction, I assume that these belong to the same person. Atm all I want to do is to get a transaction and then get a list consisting of either ["Mined"] (if it was a coinbase) or the input address(es). My code so far: Code: def getListOfInputAdresses(tx): getSenderAddrIfAvailable() however seems not to catch a lot of input addresses, compared to your "count all addresses" that just counts the txouts I get only a fraction. I already read the comment Code: // Not all TxIns have sendor info. Might have to go to the Outpoint and get but I'm kinda lost after calling txin.getOutPoint() - how do I then get to the OutTx object again and how do I know which of it's (possibly multiple) output addresses is the input address I'm looking for? Title: Re: Reading the block chain with a library? Post by: etotheipi on May 09, 2012, 05:11:58 AM I'm currently trying to write a program that connects multiple addresses to "entities". If there are 2 (or more) inputs used in a transaction, I assume that these belong to the same person. Atm all I want to do is to get a transaction and then get a list consisting of either ["Mined"] (if it was a coinbase) or the input address(es). ... getSenderAddrIfAvailable() however seems not to catch a lot of input addresses, compared to your "count all addresses" that just counts the txouts I get only a fraction. I already read the comment Code: // Not all TxIns have sendor info. Might have to go to the Outpoint and get but I'm kinda lost after calling txin.getOutPoint() - how do I then get to the OutTx object again and how do I know which of it's (possibly multiple) output addresses is the input address I'm looking for? I wrote a special method in the BlockDataManager specifically for this situation (and I use it all the time). It's because you can't always determine the identity behind a TxIn without actually retrieving the previous Tx that is being spent. Thus, you need the help of the BlockDataManager, because a Tx object knows nothing about other transactions! What you're looking for is this: Code: theTxIn = tx.getTxInRef(i) Method "getSenderAddrIfAvailable()" does exactly what it says: if it can deduce it from the TxIn, then it will return it. However, coinbase TxIns do not have that info (though it could usually be deduced using some fancy math, but I never got around to implementing that). Title: Re: Reading the block chain with a library? Post by: Mike Hearn on May 09, 2012, 05:46:43 AM You can also use bitcoinj for this kind of task, from either Java or C++. Take a look at the documentation here:
http://code.google.com/p/bitcoinj/ Title: Re: Reading the block chain with a library? Post by: runeks on August 06, 2012, 05:47:34 PM How does one just query the balance of a Bitcoin address with armory? is there any calculations involved(like looping over tx history and sum up a balance) or can I just call it and print out a simple balance? Im' sure a lot of programmers could benefit the solution to this question to write their own software to use armory for like making their own Blockchain.info website or w/e. The code was in my example, but I guess my example was too long. So here's an extraction of it. Code: from armoryengine import * Code: rune@rune-desktop:~/Desktop$ /home/rune/Programming/BitcoinArmory/armory-scan.py Any idea what this is about? bitcoin-qt is running and accepting connections over RPC. EDIT: Running the newest version from git. On the master branch. Title: Re: Reading the block chain with a library? Post by: etotheipi on August 06, 2012, 05:59:59 PM Any idea what this is about? bitcoin-qt is running and accepting connections over RPC. EDIT: Running the newest version from git. On the master branch. Did you "make swig" after pulling the 0.81 code? A bunch of stuff changed under the hood with 0.80+, including the way that blockchain scanning is done. This error looks like you're using the latest python code, but the underlying C++ utilities are still the 0.77-compiled versions. Title: Re: Reading the block chain with a library? Post by: runeks on August 13, 2012, 05:31:37 PM ^ If I 'make swig' from the BitcoinArmory root directory I get an error ("make: *** No rule to make target `swig'. Stop."). If I enter the "cppForSwig" directory and 'make', I still get the error:
Code: rune@rune-desktop:~/Programming/BitcoinArmory/cppForSwig$ make Title: Re: Reading the block chain with a library? Post by: etotheipi on August 13, 2012, 05:43:26 PM ^ If I 'make swig' from the BitcoinArmory root directory I get an error ("make: *** No rule to make target `swig'. Stop."). If I enter the "cppForSwig" directory and 'make', I still get the error: Make in the root directory should call "make swig" in the cppForSwig directory. Or you can go into the cppForSwig directory and "make swig". In both cases, it will recompile all the stuff and re-run swig to provide armoryengine.py access to the latest C++ utilities. The error you are reporting is indicative of a version mismatch between the C++ and python code. I recommend doing a fresh checkout, then go into the cppForSwig directory and type "make swig" (to be sure), and then try running it again. If that doesn't work, then... ? Title: Re: Reading the block chain with a library? Post by: runeks on August 13, 2012, 08:00:36 PM Did it all from scratch, works perfectly now :). Thanks!
Title: Re: Reading the block chain with a library? Post by: kayrice on August 14, 2012, 01:21:44 AM For what it's worth I wrote some code in PHP that read the entire block-chain as downloaded from a "daily source" in tgz format early in development. I needed the block-chain and stats from it but didn't want to wrestle with APIs or wait for bitcoind to finish. Downloading the tgz is fast and pretty recent.
The binary format of blk0001.dat is well documented and you can parse the entire blockchain and validate it easily. My quad-core system under a highly limited VM can parse the entire block chain and validate the double SHA-256 hashes within about 10 seconds. Title: Re: Reading the block chain with a library? Post by: Xenland on August 14, 2012, 02:33:44 AM For what it's worth I wrote some code in PHP that read the entire block-chain as downloaded from a "daily source" in tgz format early in development. I needed the block-chain and stats from it but didn't want to wrestle with APIs or wait for bitcoind to finish. Downloading the tgz is fast and pretty recent. The binary format of blk0001.dat is well documented and you can parse the entire blockchain and validate it easily. My quad-core system under a highly limited VM can parse the entire block chain and validate the double SHA-256 hashes within about 10 seconds. Hmm I haven't seen this "well documented" information quite yet would you mind posting a link so I can dive in? Title: Re: Reading the block chain with a library? Post by: etotheipi on August 14, 2012, 02:53:41 AM For what it's worth I wrote some code in PHP that read the entire block-chain as downloaded from a "daily source" in tgz format early in development. I needed the block-chain and stats from it but didn't want to wrestle with APIs or wait for bitcoind to finish. Downloading the tgz is fast and pretty recent. The binary format of blk0001.dat is well documented and you can parse the entire blockchain and validate it easily. My quad-core system under a highly limited VM can parse the entire block chain and validate the double SHA-256 hashes within about 10 seconds. Hmm I haven't seen this "well documented" information quite yet would you mind posting a link so I can dive in? The format is stupid simple: Magic Bytes (4) Num Block Bytes (4) Header (80) Num Tx in Block (VAR_INT) Tx1 Tx2 ... TxN <Repeat> Title: Re: Reading the block chain with a library? Post by: runeks on October 02, 2012, 10:14:36 PM How does one just query the balance of a Bitcoin address with armory? is there any calculations involved(like looping over tx history and sum up a balance) or can I just call it and print out a simple balance? Im' sure a lot of programmers could benefit the solution to this question to write their own software to use armory for like making their own Blockchain.info website or w/e. The code was in my example, but I guess my example was too long. So here's an extraction of it. Code: from armoryengine import * I have some code where I want to be able to query balances as quickly as possible, and would prefer not to wait while loading the now 3 GB block chain. This way doesn't work for me. The first scan works fine, but the second scan hangs at TheBDM.scanBlockchainForTx(cppWallet). Code: from armoryengine import * Code: (gdb) where So I guess I'd like to know:
Title: Re: Reading the block chain with a library? Post by: runeks on October 07, 2012, 08:55:12 PM Never mind. I was structuring my code wrong so the previous cppWallet wasn't deleted from memory before starting with the next (I think that's what was wrong). This works:
Code: from armoryengine import * |