In a given transaction they are unrelated.   For each TxIn you look at its OutPoint and go fetch the TxOut it references from a previous transaction in the block database.  You use that TxOut.  The TxOut in this Tx will be used as inputs in a future transaction (when the recipient spends the money)

You shouldn't be surprised there's no timestamps on transactions - that's the whole point of Bitcoin which is for a whole network of thousands of independent nodes to all agree on the ordering of transactions.   In essence, a transaction doesn't even really exist until it is included in a block.  Therefore, the block timestamp is when the TX became "real".  Everyone has different clocks and receives the tx in different orders and times.  The point of Bitcoin is for this network with no central authority to agree on which ones actually happened.

As for C++ code, go look at the Armory code base which had code that does exactly what you're looking for in the  cppForSWIG directory.  BlockUtils cppForSWIG.   Should be a function called parseEntireBlockchain of something like that.

If you redownload the blockchain, they will be 134 MB.  But if your blockfiles were mostly downloaded prior to version 0.8, then it used a 2 GB blkfile size, and just kept them when they switched it in 0.8.

If you were to wipe the directory and redownload with 0.8+, it would be about 70 files of 134 MB each.  If you redownloaded with version 0.7 or prior, it would be 5 files of 2 GB each.

Oh, I understood that part, but I forgot about the part where they don't have to provide a large signature in order to spend it to themselves.  And I missed the part that they can mine all of them in a single transaction and produce only one extra output for themself.  Okay, it's kind of cool.

Although it's a neat idea, it underestimates the blockchain bloat that comes with it.  If this became standard, and we had 1000 tx per block, that's potentially 1,000 extra UTXOs we wouldn't otherwise have.  Adding 1000 UTXOs per block would be pretty rough, and the total blockchain size would increase pretty dramatically in an effort to spend all those. 

It seems that you would end up with much more frequent "double-spends", for the same reason coinbase transactions require 100 blocks to become valid.  Miners could spend those fees immediately, meaning that orphaned blocks could lead to countless "accidental" double-spends.

The TxIns and TxOuts are independent.  Think of the network as containing a bunch of floating [unspent coins].  Each of those unspent TxOuts is like a bill (like $20 bill).  The TxIn is simply a reference to a particular bill with a signature proving you are authorized to spend it.  If you want to send someone 10 BTC, you need to collect enough bills (say, a 4 BTC bill/TxOut, a 2 BTC bill/TxOut, and a 7 BTC bill/TxOut).  10BTC output, and one output sending the remainder back to yourself (3 BTC).  

You can think of transactions as eating unspent TxOuts, and producing out new TxOuts (and you are authorizing the spending of the old TxOuts by the signatures in the TxIns). A transaction takes TxOuts of some ownership, and converts them into TxOuts of a new ownership.  You can split them or merge them in any quantity, as long as you have at least one TxIn (corresponding to a previous TxOut being eaten) and one new TxOut.

Here's a random picture I made a long time ago that attempts to illustrate it.  



The dotted lines for the TxIns illustrates that they are really kind of "transient" data.  They're simply for the network to see that you were authorized to move those coins, but in the far future, the old spent TxOuts, and the TxIns that moved them can all be pruned off.

For reference, when Armory implements multi-sig and linked wallets, P2SH will become a staple.  It's not that there isn't use for them, it's that you need existing, mature wallet software to build on top of, and there just isn't a whole lot of options for that.  If I hadn't made such poor scalability decisions in Armory two years ago when it was started as a pet project, I probably would've done it by now...

I wouldn't say it's useless.  You could always create a transaction to someone (as Akka pointed out, it can be done with Armory), and then email it to them instead of broadcasting it.  Then it would be the same as writing a check from your bank checking account -- checks aren't guaranteed to have funds available to them, but if it's lost in the mail or the person moved (i.e. lost their wallet, etc), the coins never left your wallet.  

I had even thought about doing something like this in Armory as another kind of payment option.  

(1) Recipient gives the user his payment address.
(2) Sender, perhaps at a far later time doesn't know if that address is still valid, etc
(3) Sender creates a signed transaction and emails it to the recipient
(4) Recipient opens it in Armory and it says "This transaction sends 13.24 BTC to Wallet ABC" or "This transaction does not send coins to any wallet you own"
(5) Recipient presses "Broadcast" if they are happy with it.  Or replies with "this isn't valid, please use this address instead: ..."

It also makes MitM address swapping attack much less potent, because the person broadcasting is much more capable than the sender, of recognizing whether the transaction goes to their own wallet (edit: okay so this isn't actually true: because if the MitM can swap the addresses, he can probably also intercept the signed transaction and broadcast it himself... maybe only relevant if the tx is sent to the recipient through a different channel).  It also means that the recipient must actually act to broadcast the tx, which could be seen as actively acknowledging that payment was accepted.

http://www.cbsnews.com/8301-205_162-57592240/winklevoss-trust-will-test-bitcoin-security-concerns/

More discussion about the Winklevoss ETF, and then some educational discussion about Bitcoin... based on an interview with me!  

Hmm... good question.  I never bothered putting in something like that, because I always just used testnet if I needed to operate on less data.  I suppose you could create a temporary directory with just the first couple blk files and point your script to that using --satoshi-datadir, and then disable it when you're ready for the full blockchain.

I know sipa was working on an optimized ECDSA verifier that would outperform SSL about 4x.  I don't know if he finished it (and got it into Bitcoin-Qt/bitcoind yet).

But I do remember that a single core of my i5-2500K did something like 200-500 ECDSA verifications per sec.  So with 4 cores that would be like 1000-2000/s.  And probably something like 5,000/sec when using sipa's optimization. 

However, it still doesn't change the fact that it's a couple orders of magnitude slower than hashing, which can execute about 15,000,000/s on the same hardware.  In the grand scheme of things, the address-indexed DB will not consume much time when implemented properly.  Each block only updates a couple MB of DB values which should take far less than a second, even with synchronous writes.

Had a chat with "Chenda" from CBS news this morning.  Apparently she used our conversation to write the whole last half of the CBS article! 

http://www.cbsnews.com/8301-205_162-57592240/winklevoss-trust-will-test-bitcoin-security-concerns/

Worrying about block validation is a whole new world you don't want to get into.  For now you can simply leverage the fact that if the block is in the blk*.dat files it is valid along with all the transactions in it.   As TierNolan said though, you'll probably have a few orphans in there, so you probably need your scanning code to either compute longest chain (which is a good exercise in itself) or just not assume that 100% of the blocks in those files are in the main chain.

For a first exercise, you don't need a full script evaluation engine.  Just program in enough to recognize common TxOut scripts and ignore the non - std scripts.  Although in terms of computing balances, it's kind of arbitrary... You can store every TxOut in a map<OutPoint, RawTxOut> and remove them as you see TxIns that spend them.  When you're done, you should have a map containing all unspent TxOuts on the network.

Also for your analysis of looking for dormant coins being spent, look up "bitcoin days destroyed"

Being someone who wrote a blockchain parser from scratch (which is what Armory does on every load/rescan), I can tell you that everything about that is normal (except for the second point below).   And 95 seconds is pretty good.  It depends what else you're doing while you're scanning.  In my case, my code scans in about 150-240 seconds depending on HDD caching, but it's also identifying all the transaction scripts in order to search for coins that are related to my wallet (so I additionally parse and identify individual scripts and then do a lookup in a set<Address160> structure to see if it's relevant).

• The zeros at the end of files are due to pre-allocation.  Since Bitcoin-Qt version 0.8, they now always allocate blk*.dat files in chunks of 16 MB to reduce disk thrashing.  i.e. if you keep appending to a file over and over again, the OS is forced to keep reallocating that space, or fragmenting it.  By preallocating with zero-padding, the number of disk reallocations are cut down dramatically
• I'm not sure what would cause the numBytes field to be incorrect.  How often do you see it?  Do you have an example?

I think doing exactly what you did is one of the best ways to learn the ins and outs of Bitcoin.  If you want to do more, I recommend you pick some addresses and try adding code to watch for that address and accumulate its balance and spendable UTXOs.

Just a quick comment about progress:  No, it's not done yet.  But I am finally getting some time to do it.  And I'm doing it... slowly.

Now that Armory (and Bitcoin) is getting more serious, I decided I needed to adopt a more-serious testing framework for my code.  I mean, I have tests... thousands of lines of tests.  But they're not automated, and perhaps not as exhaustive as they should be.  Now seemed like a good time to do the upgrade, because I'm really tearing things up and putting them back together.   Again, it's one of those short-term-sacrifice-for-long-term-gain things.   Also, I'm writing a lot of new code that needs to be tested, and I wanted to start doing it "The Right Way."

So I recently started playing with GoogleTest.  In usual Google fashion, what they have made is super impressive.  They have produced a versatile, cross-platform, well-documented tool that is remarkably powerful yet stupid simple to use (once you get past the compiling issues).  In a day I've been able to translate a bunch of my tests into a GoogleTest setup.  And it's pretty.  It even times things for you.



That is all.  Back to my coding cave...

Relax  The coins are not lost.  There is no way to lose to coins unless you sent them to the wrong address.  Worst case, they just get a little stuck

Close Armory and delete your mempool.bin file.  In windows:

C:\Users\<username>\AppData\Roaming\Armory\mempool.bin

or Linux

/home/<username>/.armory/mempool.bin.

Then restart both Armory and Bitcoin-Qt.  That should be it.

(1) If you could not successfully broadcast a transaction despite being online, I need your help.  I've heard about a couple other people with this problem, and I need to narrow it down.  I suspect it may have to do with the more-recent versions of Bitcoin-Qt not liking certain signatures made by older versions of Armory.  If you have any more information about why it's not working, please let me know.  If it's that Armory can't even synchronize at all... well I know about that problem already (but since you got an unsigned transaction to be able to be signed, I suspect you did synchronize properly...)


(2) If you switch Armory to "Expert" usermode (might be in "Advanced" too, I don't know), on the window where you would normally broadcast your transaction, there's a button that says "Copy Final Tx (Hex)".  This should give you the raw transaction that can be plugged into any broadcast utility.  You're right that the raw text shown in the text box is not compatible with anything else, but the raw hex should be.  Let me know if that doesn't work.

Armory does exactly this.  It implements "comments" individually for addresses and transactions.  On the main window, you have a list of transactions.  You can double-click them and individually set the comment for that transaction.  If there is no transaction comment (tied to the tx ID), then it displays the address comment instead.  You can modify the address comments in the wallet properties address list the same way.

Oh I missed the part about no tx on each mini block.  So all this does is effectively eliminates the probability of one-block reversals.  Hmm, I'm not so much in favor of this direction.

My only point was that as your average confirm time moves closer to the header propagation time, you're dramatically increasing time spent by miners mining on forks.  Sure you don't get one-regular-confirmation reversals, but you'll end up with a totally fragmented network all working on different chains because most of the network can't get the longest chain in time to switch to it.    It may be that the header propagation times are, in fact, much smaller than 10s, in which case it wouldn't be so bad for the majority of the network. 

But consider a miner on a slow, high-latency connection.  If they have 10s latency to receive data, they would probably spend >50% of their time mining on forks.  Basically, they wouldn't be able to mine. 

I haven't fully thought it through, but don't forget that although a fork only takes 10 seconds to resolve in your scheme, forks happen a whole ton more often.  I don't know what all the terms add up to, but it wouldn't surprise me if this turned out to cause miners to spend more time mining on forks than less.  Especially as you lower that average-time-between-blocks closer to the average header propagation time.  I don't know what propagation times are, but if they were on average 30s to reach 99% of the network, you could have X% (X>10) of the network always mining on [soon-to-be] dead chains.    

At the very least it would lower the effective hashrate, making the network more vulnerable.  If you have 100 units of hashing power building one blockchain, an attacker needs 101 to outpace it.  If you have 4 groups of 25 units each mining on different forks, an attacker only need 26 to outpace it, even though the total network hashrate is the same.

Satoshi picked 10 minutes, because he was concerned about the network dynamics of slow block propagation and what dragons it would create.  He picked 10 minutes to be conservative, knowing that if it's long enough to absolutely be sure that propagation can happen reliably and consistently.  Though I don't have any links off-hand to previous discussions about that.  I'd be curious to re-read it myself.

Though, I'm going off on a tangent: your idea doesn't rely on 10s... it can be calibrated to whatever is "safe."  I do like the gist of the idea, since we already know that safety is basically a asymptotically increasing as the number of confirmations goes up, so if you can get the first couple confirmations out the door immediately 64 mini-confirmations would most definitely be better than 1 regular confirmation.