Are disabled OpCodes considered invalid or are they treated as NOP?  Link to where that is enforced in the code would be appreciated.

The recent sybil attack on the network by Chainalysis got me thinking about identity being leaked by the high correlation between the first relayed IP address and the original creator.  IP can be a very powerful piece of identity information.  It alone can be useful but then combined with other measured can be used to build a real world identity and link transactions to it.  Granted the first relayed node may not be the sender but with enough nodes and some heuristics and multiple datapoints it becomes possible to start seeing the information from the noise.

The good news is that Chainanlysis created non-responding nodes and used a sequential range of IP addresses from a given subnet.  So it was both suspicious and did not appear as normal nodes to other nodes on the network.  A smarter attacker however would use a few high capacity nodes hidden behind a larger number geo diverse IP addresses acting as transparent proxies.  It would be possible to aggregate the same information without appearing any differently than other bitcoin nodes.  So I guess we can be thankful that Chainalysis did such a horribly bad job out of the gate that they brought attention to the issue before someone competent tries.  Hiding the creator's IP address alone does not provide perfect privacy but it is a start.  

1) Nodes should support split routing
TOR, I2P, and VPN proxy are options to decouple your IP address from your transactions but bitcoin generates a lot of traffic when only a tiny portion of that is privacy related. Routing all of it through privacy protecting services is inefficient.  Split routing would allow the client to be configured to relay some of its traffic (like locally created transactions) to one network and the rest of the traffic (like relayed blocks) to another network.  By routing only a node's outbound txns the bandwidth requirements over the secure network can be reduced significantly.  It may be more secure to randomly pick a small portion of the received transaction to relay over the secure network and then add locally created transactions to that stream.  This makes exit node analysis more difficult in the event the exit node is compromised.   Still it should be possible to reduce the TOR bandwidth requirements by 99% or more.  Likewise users of VPN proxy services could take advantage of split routing to reduce cost as their bandwidth requirements will be low.

2) Merchants and payment service providers should directly accept signed transactions.
The person you are paying already knows 'you' are paying.  How much of you they know depends on the circumstances it may be a little or it may be a lot but there is some element of trust with any counterparty.  If I buy a gold coin from an online dealer who accepts bitcoins they already know at a minimum my name, address, and probably my IP address unless I used privacy protecting service.  It would leak less information to provide the signed transaction directly to the recipient rather than to an anonymous peer who may be malicious.

3) Wallet developers should consider making it possible for a user to create a signed transaction without broadcasting and to import a signed transaction.
Armory already makes it easy to sign a transaction without broadcasting in offline mode (cold storage) but this could be useful feature for an online wallet as well and also implemented in Bitcoin core.  User creates a transaction in the wallet, checks an option to not broadcast and when signed receives the hex encoded raw transaction which he can send directly to the receiver.  Merchants and other service providers will likely use automated system to accept and process this transactions but end users may not.  Clients should make it easy to 'import' a signed transaction for validation and relaying.

In the stock trading world, fill or kill is trade type which indicates the order should either execute immediately or be deleted.   Could something like this be used in a blockchain based system to prevent "stuck" transactions?

The problem:
Due to outdated priority rules, user misunderstanding, or wallet bugs users can create transactions which are very unlikely to confirm or even be relayed through most nodes.  However most wallets have no good method to cancel.  Cancelling txns is complex.  A simple cancel button would creating confusion and unexpected results.  The wallet software could remove the txn locally but it can't force other nodes to remove their copies.  This means the txn could be 'confirmed' long after it is 'canceled'.  Waiting is also not a good solution and can be confusing to users.  The time before a txn is removed from a node's memory pool is not static and the txn may be removed from some but not all nodes.  Many clients will continue to rebroadcast unconfirmed txns which 'reset' the clock.  Users also have no way of knowing if copies of the txn remain.  This means the network appears to have 'inconsistent' behavior.   Something more robust is needed to provide deterministic behavior.

The solution:
What about including a max block height flag?  This could be setup very similar to nlocktime except that it would designate the last block the txn can be confirmed in, instead of the first one.  Once the kof height has been created the txn would no longer be valid for inclusion in a block.  To simplify dealing with reorgs it would be a good idea for nodes to keep the txn in the memory pool but not considered for inclusion in a block until the current block height is a small number of blocks ahead of the kof height and then delete the txn.  The user would have a high level of confidence that after the kof height the txn could be resubmitted to the network with a more appropriate fee.  The user's client would change the status of the txn to canceled once the kof height has been passed and increase the 'available balance' which would be consistent with user expectations and help to hide the network behavior in a deterministic manner.

Any potential attack vectors or complications?

[When you as an individual have been reduced to a position where you must outbid a 'bank' for space in the blockchain then you have effectively lost access to the blockchain.]

Permanently keeping the 1MB (anti-spam) restriction is a great idea, if you are a bank. Those favoring a permanent 1MB cap, whilst asserting that Bitcoin can still be a financial backbone of sorts, don't know how right they are. The problem isn't a limit in general but that 1MB provides so little transaction capacity that, under any meaningful adoption scenario, it will push all individual users off the blockchain to rely on trusted third parties. 1MB is insufficient for end to end direct user access, but it is sufficient for a robust inter-‘bank’ settlement network.

If the cap is not raised to some higher limit; allowing a larger number of users to maintain direct access, then individuals will be priced out of the blockchain. When that happens Bitcoin becomes yet another network with no direct (peer) access; like FedWire, SWIFT, and other private closed transfer networks.  There is no realistic scenario where a network capped permanently at 1MB can have meaningful adoption whilst still maintaining direct access to the blockchain by individuals. To be clear, by ‘direct access’ I mean both parties transacting directly on-chain without relying on an intermediary or trusted third party.

Bitcoin ... A purely peer-to-peer version of electronic cash would allow online payments to be sent directly from one party to another without going through a financial institution.

Satoshi Nakamoto - Bitcoin Whitepaper

Finding a balance
If the transaction capacity of a peer to peer network is very low then the resource requirements of running a full node will also be low but the cost of a transactions will be high.  A network that keeps computing requirements low but which has priced direct access to that network out of the hands of individuals doesn't help decentralization. Likewise if the transaction capacity of the network is very high then the cost of transactions will be low but the resource requirements for running a full node will be high.  A network that has sufficient transaction capacity for every possible transaction but which requires resources that put operation of a full node beyond the capabilities of individuals also doesn't help decentralization.

There is no "perfect" transaction rate.  It is a compromise between two different centralization risks.  Lowering the risk of transaction centralization raises the risk of node centralization and the reverse is also true.  A robust network means minimizing overall risk not minimizing one risk at the expense of the other.  This means that neither extreme is optimal.  What level of transaction capacity (in terms of block size) provides the optimum balance will be saved for future discussions. Having a discussion on how to change the network first requires an acceptance that the network needs to change.  So lets start with why the network must change.


Can we stop talking about a cup of coffee please?
If you have been following the discussion you may have heard a claim such as "Every $5 coffee doesn't need to be on the blockchain so there is no need to raise the limit".   The implied meaning is that while the cost of permanently keeping the limit will exclude 'trivial' transactions you will still have direct access to the blockchain for everything else.   This is misleading as the 1MB restriction is so restrictive that larger more meaningful transactions will eventually become uneconomical as well.  

I don't really care if my morning coffee is paid for using a centralized service.  Centralization and trusted third parties aren't always bad. <gasp>.  Context matters so try to read the rest before freaking out.  Have you ever given or received a gift card?   Can't get more centralized than that.  If I put $100 in Bitcoins under the control of a third party say in a ewallet or bitcoin debit card service, the risk and scope of that centralization are limited and manageable.  As long as direct access to the network remains economical I can securely store and transfer my wealth using on-chain transactions and use centralized solutions where the risk is low such as day to day purchases.  On the other hand if the imbalance between transaction demand and capacity makes individual transactions uneconomical I will lose direct access all together and that risk is more severe and the consequences more significant.

Sidechains, payment channels, and cross-chain atomic transactions are decentralized system that can move some of the transaction volume off the primary chain.  In essence like centralized solutions they can act as a multiplier allowing a higher number of total transactions than the number of direct on-chain transactions.   It is important to realize they still rely on the primary chain having sufficient transaction capacity or they aren't trustless.  As an example a payment channel could allow hundreds or even thousands of off chain transactions but it requires periodic on-chain transactions to create, adjust, and takedown.  If the individual user loses direct access to the primary chain they also lose trust free solutions like payment channels.  If direct access becomes prohibitively expensive then only alternative which provides sufficient scale is using trusted third parties.

When demand significantly exceeds capacity it increases the utility and value of centralized solutions
If the transaction demand exceeds capacity by a magnitude or more it will lead to direct users being replaced by trusted third parties acting as aggregators.   There are a lot of disadvantages to centralized services but they are more efficient and if the artificial limit is kept it will play right into the advantages of centralized services.  A third party can facilitate instant off-chain transactions.  If demand outstrips the very limited capacity it will force transactions off-chain using trusted third parties which I will call processors.  

Today these processors would include online wallet providers, exchanges, merchant payment service providers, and services where the user maintains a balance under the control of the merchant (casino, poker room).  In time even traditional financial companies and banks could be processors.  The one thing they all have in common is that customers of these services do not have direct control over private keys and do not directly make transactions on the network.  The processor has control over the private keys, keeps the Bitcoins in reserve and maintains an internal ledger of user transactions and balances.   Processors can trivially facilitate transactions between two customers of their service.   Since they control the private keys they simply update the ledger balances of the two customers and many of them do this today.  

However transactions can still occur off-chain even if they involve customers of different processors.  The process is not trustless but risk can be managed.  Two processors would aggregate all the transactions which occur between their customers and then periodically settle the difference.  When a payment from a customer of one processor is made to a customer of another processor the sending processor will notify the receiving processor.  Both processors will update their internal ledgers.  Over time this will result in an accumulated balanced owed by one processor to the other which can be settled with a single on-chain transaction.  The key thing is that there is a one to many relationship between the settlement transactions and underlying customer transactions.   While the 1MB block limit does not provide sufficient capacity for a large number of direct user transactions, third party processors can facilitate a very large number of off-chain transactions using a smaller number of on-chain settlements.  Blocks of a finite size can support a nearly unlimited amount of off-chain transaction capacity with the limitation that it involves the use of trusted third parties.

You can't competing with a 'bank'
You might be considering the point above but dismissing it because you still 'could' submit a transaction to the network.  The processors described above don't have the ability to close the network but a network that you have technical access to but which is uneconomical is effectively no access at all.  The current block size realistically limits capacity to no more than two to four transactions per second.  Two transactions per second is roughly 64 million transactions per year.  A finite number of transactions can't support an infinite number of direct users.  Say at some point there are ten million users and they wish to make two transactions per month.  That is 240 million transactions but only 64 million will fit in the blocks.  What happens to the excess.  If third party processors are attractive the difference will be handled by them. When you consider a settlement network would allows these third party processors to offer instant, 'no risk' transactions at significantly lower fees than on chain transactions the excess demand will be processed off-chain.   If the network continues to grow the profitability of these companies will grow and that will lead to more third party companies.   Those settlement transactions allow more off-chain transactions but at the same time compete with direct user transactions for the limited on-chain capacity.  In a settlement network the upper limit on the number of settlements required grows exponentially with the number of trusted peers.  Just two hundred trusted peers (crypto banks) performing hourly settlements would fill the blocks, all the blocks perpetually.  There may be billions of 'Bitcoin' transactions but they would be nothing more than updates on centralized ledgers.  The blockchain would just handle the settlement between massive financial service providers.  As these entities are collecting fees, and on chain transactions are a necessity of doing business, they can and will pay far more than you to ensure timely inclusion in a block.   When you as an individual have been reduced to a position where you must outbid a 'bank' for space in the blockchain then you have effectively lost access to the blockchain.

Wait I don't get how off blockchain transactions could occur across entities
Imagine two large financial service providers where thousands of customers make payments to customers of the other entity.  These may not be banks in the traditional sense, they can be any entity which acts as a third party to manage the bitcoins and transactions of customer.  Today it could be major exchanges, payment processors, and ewallet providers but tomorrow it could include traditional financial service companies or even banks.   For this example lets call the two entities Chase and HSBC.   Chase and HSBC can notify each other when one of their customers makes a payment to a customer of the other entity.  Both would update their internal ledgers and the payments would appear to occur instantly.  Most importantly none of them would require an on-chain transaction.  It is just updating numbers in a ledger.  If you are a Coinbase customer and pay another coinbase customer this happens today.  We are only taking it a step further and handling cross entity transactions.  Now the entities have no real cost to perform these payments.  They are just sharing a few bytes of information with their counterparty and updating numbers in a database.  However over time, the net amount of the thousands of transactions will result in one entity accumulating a balanced owed to the other.  This is why settlement networks require some level of trust.   It requires trusted peers to extend mutual lines of credit to each other.  The more they trust each other, the larger the lines of credit and the less frequently they need to settle.  It is also why you will never be a peer on this network.  The entities would enter into a legally binding agreements which set up the conditions of the settlement.  The amount of funds the entities are risking is limited.  The entities will limit the amount of credit they will extend and the terms are usually very short.  These aren't long term loans, in the traditional banking world settlement might occur the next business day.  The efficiency of the blockchain allows for lower capital requirements and lower risk by settling more frequently maybe even hourly.

Imagine that in a particular hour HSBC customers make thousands of payments to Chase customers totalling 10,000 BTC and Chase customers make thousands of payments to HSBC customers totalling 3,000 BTC.  In total transaction volume is worth 13,000 BTC.   As these payments occur Chase and HSBC notify the other party.  This allows both to update their ledgers.  A customer making a payment would see their balance be reduced 'instantly' and the customer receiving a payment would see their balance increase 'instantly'.  The net flows however are not balanced.  Chase has increased their customer balances 10,000 BTC and only reduced their customer balances 3,000 BTC.  On the books both entities have a liability called 'Customer Deposits'.  They keep reserves (hopefully >=100% to cover those liabilities).  However Chase has seen its liability increase by 7,000 BTC and HSBC has seen its liability decrease by the same amount.  To reconcile this HSBC will make a single on-chain transaction to Chase for 7,000 BTC.  This will increase Chase's reserves by 7,000 BTC and decrease HSBC's reserves by 7,000 BTC.  Everything is balanced again.   Yes it did require a limited amount of trust between settlement peers and a single on-chain transaction but it facilitated thousands of off chain transactions.  As soon as the next cross entity transaction happens a balance is owed by one entity and the net amount owed will increase and decrease until the next settlement which balances the books again and the cycle perpetually continues.

Now when demand for transactions exceeds what is possible who do you think can pay more in fees?  You or a bank?  If transaction demand exceeds capacity then some transactions will not make it into a block.  Those paying the highest fee will be the ones who retain access to the blockchain and those unable or unwilling to will be excluded.  It is delusional to think it will be the 'banks' that suffer in a situation like that.

The reported 7tps transaction capacity does not exist.
There is a myth that without raising the limit the network could handle 7tps.  It can't.  The limit is 1MB per block the actual transaction capacity depends on the average transaction size and realistically that provides no more than 2 to 4 tps. To achieve 7 tps using one block of 1 MB every 600 seconds means that the average transaction size must be 240 bytes (1,000,000 bytes / 600 seconds / 7tps = 240 bytes).  If you have a Bitcoin wallet handy take a look at your last dozen transactions and if you don't have a wallet handy use a website to lookup the transactions in the most recent block.  How many of the transactions were under 240 bytes? Not very many.  I am going to say the majority of your transactions were probably between 300 and 700 bytes.  

Can you form a 240 byte transaction?  Sure as long as you only have only a single input.  A transaction input is requires at least 147 bytes so an average of 240 bytes per transaction is not possible unless the average number of inputs is less than 2.  While some transactions may have one input on average they are going to have more.   The total number of inputs in the blockchain will roughly equal the the total number of outputs.  As the number of blocks approaches infinity the ratio of inputs to outputs in a well functioning blockchain will approach 1:1.

Since most outputs will eventually become inputs it makes more sense to look at block capacity using a balanced transactions as a template for transaction size.  A balanced transaction is one where the number of inputs equals the number of outputs.  Single input, single output exceptions are both rare and have limited use.   A 2 input, 2 output transaction using all compressed keys and P2PKH scripts is typical and weighs in at 373 bytes.  At 373 bytes per transaction and 1MB per block the network will not exceed 4.4 tps..  This is already 37% less than claimed but it is still unrealistic as it represents the smallest possible balanced transaction.

Most transactions are going to be larger than 373 bytes due to the use of uncompressed keys being used, more complex scripts, and more inputs and outputs per transaction.  Looking at the last million transactions in the blockchain I found the average txn size was 560 bytes.  At 560 bytes per transaction and 1MB per block the network will not exceed 3.0 tps.  So we have already lost over half of this claimed capacity but this is very likely to decrease over time as transaction sizes creep higher.  Multisig and other more complex scripts are being used more frequently and that trend will continue.  A good estimate for the network throughput when limited to 1MB blocks would be 2 to 4 tps depending on how optimistic you want to be.

Here is a direct comparison of the combined script size for some different types of scripts.  The scriptPubKey is encoded in a transaction output and the scriptSig is encoded in the transaction that "spends" that output.  Since outputs eventually become inputs in new transactions the combined size of the scriptPubKey and scriptSig represents the "roundtrip" script cost of a transaction.

Code:

        P2PkH:   131 bytes per script round trip (25 byte scriptPubKey +   106 byte scriptSig)
  2-of-3 P2SH:   253 bytes per script round trip (22 byte scriptPubKey +   231 byte scriptSig)  
  3-of-5 P2SH:   383 bytes per script round trip (22 byte scriptPubKey +   361 byte scriptSig) 
15-of-15 P2SH: 1,481 bytes per script round trip (22 byte scriptPubKey + 1,459 byte scriptSig) 

How many transactions are possible per megabyte of block capacity?  Below is a the maximum capacity of the network at various average transaction sizes.  Realistically 2 to 4 tps is all that is supported by a 1MB block and the lower end of that range is a far more likely.

Code:

Txn Size  Upper Bound   Example  
373       4.4 tps       (2in, 2out, P2PkH)
416       4.0 tps
520       3.3 tps       Average of the last 1,000,000 transactions
555       3.0 tps
616       2.7 tps       (2in, 2out, 2-of-3 P2SH)
833       2.0 tps

This same metric also applies to larger blocks.  Advocates of larger blocks will often overestimate the capacity of larger blocks.  It is realistic to estimate getting 2 to 4 tps per MB of block space regardless of the block size. If all blocks were 20MB that would provide a realistic throughput of 40 to 80 tps not 140 tps.  Still 40 to 80 tps would be sufficient for 100 million users to make one or two transactions per month.

1MB can not support a sufficient number of direct users even if transaction frequency is very low
One argument made by those favoring the cap is that Bitcoin doesn't need to be used as a transactional currency to be successful.  Users could primarily acquire Bitcoins as a way to secure store of value (savings) and continue to use other currencies for routine purchases.  Bullion and other stores of value have a much lower velocity than transactional currencies.  This means a block of the same size could support more more users.  While the user of a non-transaction currency may not make dozens of transactions a day, meaningful access would at least require access on the order of dozens of transactions per year.  If your savings or brokerage account restricted you to only one deposit per quarter and one withdrawal per year I don't think you would find that acceptable.  Future users of Bitcoin will not find it any more acceptable if they are forced to transaction as infrequently.

Code:

Maximum supported users based on transaction frequency.
Assumptions: 1MB block, 821 bytes per txn
Throughput:  2.03 tps, 64,000,000 transactions annually

Total #        Transactions per  Transaction
direct users     user annually    Frequency
       <8,000       8760          Once an hour
      178,000        365          Once a day
      500,000        128          A few (2.4) times a week
    1,200,000         52          Once a week
    2,600,000         24	  Twice a month
    5,300,000         12	  Once a month
   16,000,000          4	  Once a quarter
   64,000,000          1          Once a year
  200,000,000          0.3        Less than once every few years
1,000,000,000          0.06       Less than once a decade

As you can see even with an average transaction frequency of just once a week or once a month the network can't support more than a token number of users.  When someone advocates a permanent cap of 1MB what they are saying is I think Bitcoin will be great if it is never used by more than a couple million users making less than one transaction per month.  Such a system will never flourish as a store of value as it is eclipsed by alternatives which are more inclusive.  To support even 100 million direct users making an average of one transaction every two weeks would require a throughput of 82 tps and an average block size of 20 to 40 Megabytes.

1MB doesn't can't even keep up with existing non-retail payment networks.
Going back to that coffee meme, the implied message is that 1MB is fine unless for everything else.  You know substantial stuff like paying your mortgage, business deals, major capital expenditures, or paying a supplier for inventory.  This just isn't the case though.  Do you know anyone who pays for coffee with a bank wire? The FedWire service (run by US federal reserve) processes ~150 million bank wires annually.   The FedWire service only operates in the US.  Internationally the largest clearinghouse is SWIFT and it processes more than 5 billion transfers annually.  The US ACH network is even larger with 19 billion transactions annually (excluding converted checks).  There are also about 2 billion international remittances annually (western union, moneygram, and other networks).  A 1MB restricted Bitcoin network couldn't even keep up with these transfer networks even if you forget about retail sales completely.  The idea keeping the 1MB restriction, only keeps limits the utility of small payments is simply incorrect.

Code:

Bitcoin block size to reach comparable network volume based on average txn size
Network    txn volume        Average transaction size
           annually (mil)    373 bytes   560 bytes   833 bytes
FedWire       150               1.1 MB      1.7 MB      2.3 MB
Remittance  2,000              14.2 MB     21.3 MB     31.7 MB
SWIFT       5,000              35.5 MB     53.3 MB     79.3 MB
ACH        19,000             134.8 MB    202.4 MB    301.0 MB

On a transaction fee basis.
Currently the cost of the network is roughly $300 million annually. The users of the network are collectively purchasing $300 mil worth of security each year.  If users paid $400 million the network would be more secure and if they paid $200 million it would be less secure. Today the majority of this cost is paid indirectly (or subsidized) through the creation of new coins but it is important to keep in mind the total unsubsidized security cost.  At 2 tps the network the unsubsidized cost per transaction would be about $5. At 100 tps it would be $0.05.  If Bitcoin was widely adopted, more users purchasing more coins should mean a higher exchange rate and thus the value of potential attacks also rises.  The future cost of the network will need to rise to ensure that attacks are not economical and non-economic attacks are prohibitively expense relative to the benefit for the attacker.   It may not rise linearly but it will need to rise.   If someday one Bitcoin is worth $10,000 and we are still only spending $300 million a year on security we probably are going to have a problem.  Now advocates of keeping the limit may argue that the majority of the network cost won't be paid by fees for many years but the reality is that with a low maximum transaction rate you can choose either much higher fees or much lower security.

Conclusion
Restricting the size of blocks to 1MB permanently is great if you are a major financial services company.   You could co-opt a very robust network, act as a trusted intermediary and force direct users off the chain onto centralized services.  For the same reasons, it is a horrible idea if you even want to keep open the possibility that individuals will be able to participate in that network without using a trusted third party as an intermediary.

On edit: fixed some typos.  Cleaned up some poorly worded statements.  Yeah there are a lot more.  It is a work in progress.

[Given a "y"]

Is there any added benefit to using SHA256d over SHA256 in Bitcoin?

For the computing of transaction hashes, block hashes, and merkle trees Bitcoin uses SHA256d

Code:

SHA256d(x) = SHA256(SHA256(x))

Effectively all hashing operations are taking twice as long.   This isn't as bad as it sounds because ECDSA verification takes up the majority of the CPU cycles when verifying a block but is there any benefit to the work done in double hashing?  Or should we just accept that this was a poor implementation decision by Satoshi?  Simple if always better in cryptography so Satoshi must have had a reason to go with a double hash but I am wondering if that reason was based on flawed understanding of the limited benefits of double hashing.

Over the years I have heard a couple theories but I don't think they hold up to any scrutiny.

1) Double hashing prevents length extension attacks
This is true but length extension attacks are not useful against Bitcoin.  Length extension attacks involve signature spoofing in authentication when the send and receiver are using a shared secret.   There are no share secret communications in the Bitcoin protocol and thus there are no possible length extension attacks.

2) Double hashing provides a fallback if preimage resistance is weakened
This is also true but only for first preimage resistance which involves the input being unknown.  As such there is some merit to this rationale in using HASH160 (or any double hash) to produce the PubKeyHash (or ScriptHash) from the PubKey (or Script).   Any benefit to double hashing is lost if the address is reused as the input becomes known.   It would also only apply if due to cryptanalysis second preimage resistance was degraded but first preimage resistance was not.

This doesn't apply to any use of SHA256d because they are used in instances where the input is known.  For those unclear on first and second preimage resistance:
  

First-preimage resistance:  it is computationally infeasible to find any input which hashes to a pre-specified output
Given a "y" such that h(x) = y it is difficult to find any preimage x .

Second-preimage resistance: it is computationally infeasible to find a second input which has the same output as a specified input.
Given x, it is difficult to find a second preimage x' ≠ x such that h(x) = h(x′)

The key difference to the two scenarios is what is known to the attacker.  In the first the attacker only has the hash. A good example would be cracking a password.  In the second the attacker has the original input. A good example would be producing a "counterfeit" txn/block/merkletree/pubkey which results in the same hash as an existing one to spoof the network and steal funds.

In Bitcoin every use of SHA256 relies on second not first preimage resistance to provide security.  The input is already known so the interim hash can be computed.  The second hashing step provides no security because if the attacker finds a second input which produces the same interim hash as the target then they both will obviously produce the same final hash.  It is possible that double hashing may harden a hash against first preimage attack but that doesn't enhance the security of Bitcoin.

3) Double hashing may break a backdoor in SHA256
I believe a backdoor in a public open algorithm like SHA256 to be very unlikely.  It would have to be hidden in plain sight.   Still even if one did exist the use of a double hash could only provide protection in a first preimage scenario.   Similar to the reasons above, in a second preimage scenario the input is known and thus the adversary can separate out the two steps.

Is the security of the trusted root cert the weak link in BIP-70?  While BIP-70 prevents a MITM attack or a substitution attack on the receiver's end it fails if the user's system is compromised.  If the user's system is compromised by malware (common way to steal bitcoins) then the attacker can feed the user misinformation by a number of ways including providing a false "trusted root cert".

It would seem to me that the trusted root cert would need to be inaccessible to the attacker to provide any real security from the most obvious attack vector. I assume the unstated assumption is that the user will be using some type of hardware device. Either a secure hardware bitcoin wallet, some general purpose PKI hardware device i.e. TPM (trusted platform module), or even HSM.  Am I missing anything?

Is it possible to keep non-transaction data out of a blockchain? This doesn't necessarily need to be Bitcoin for the sake of the thread lets just consider a generic blockchain like system.  Also for the sake of the thread lets not get into "if" it is a good idea to limit the blockchain to only transactional data (that is a good debate for another thread).

For partial solutions which make it more difficult lets give higher value to solutions which keep non-transactional data out of the UTXO.  Spent or provably unspendable transactions can be pruned.

One idea would be to make all transaction outputs being P2SH.  This could even be implicit with no opcodes in the output at all where outputs are simply a hash of the redeem script and a value.   As a side note this has an added benefit of making the UTXO (but not the full blockchain) smaller with fixed length records, making lookups simpler and cheaper.  Of course this by itself would not significantly limit inclusion of non-transactional data as encoded data (the 20 bytes "hash" could be put any encoded data without verification).  This by itself would actually be a step back because unlike OP_RETURN outputs the output would be unspendable but not provably so leading to UTXO bloat.

To secure the output against misuse one could require that the "pre-hash" be included when a transaction is relayed or when a block is published.  For Bitcoin the ScriptHash = RIPEMD160(SHA256(script)).  So the RIPEMD160 hash would be in the output and the transmitting node would also include the prehash (output of the SHA256 function) with the transaction.   Nodes would take the SHA256 "prehash" hash it and verify it matches the output to confirm the output has a valid hash.  Once a txn was sufficiently deep in the blockchain to make re-orgs unlikely, nodes could discard the prehash.

Any other ideas?

I used the export to csv functionality in bitcoin-qt to export txns from wallet for reconciliation of accounting ledger.


Here is a subset which shows the types of entries.  The values have been replaced with placeholders to protect the details of the actual transactions.

Code:

Confirmed Date              Type                Address                             Amount          TxId
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
TRUE      12/18/2013 00:04  Payment to yourself                                       -0.00010000   596bd0f1a1f6943005ec686ae0e3b4443359f90bcfe7ffd6cd575a3e9e480846
TRUE      09/26/2012 18:46  Sent to             1G3udQm7bS74Yp4kTjnMX5GuZRXN6WyWe6  -248.00400000   67707a92ed98819fa7fe5060572c9db735f1db37be05b421ac5bd7f3e8cfd1ac
TRUE      09/26/2012 18:46  Sent to             12PduCSQ48ZM1qDJdKiphjdbifTdA4BUKK  -306.00000000   67707a92ed98819fa7fe5060572c9db735f1db37be05b421ac5bd7f3e8cfd1ac
TRUE      07/10/2012 09:28  Received with       1PA6v4KWZvehUAhHtu2b4RSAR6htq7vwkR     8.00000000   deac39699e0a66894080c0f8fe54c0fd2c281d7f3555719e29c6f00fb584379d
TRUE      07/10/2012 20:52  Sent to             1qvyovB7WUmiyEdoTBgQ26ZYyrFjNc4uKG   -85.27738000   7e411a1e70de5148ec931d35fd23fe652814b718952e0a4d 42a7de54b0e5c30
TRUE      06/05/2012 22:43  [Other]                                                   -0.12461652   5b40192065586ba30ece8abd7c9253c5d021bce0da9f89a138547a01cbe7559c
TRUE      03/22/2012 19:18  Mined               14KWZvehUAhHtu2tMyykv6PiRMycPz8KtM     0.03337541   3a08a00cf04433423cadd4c5c6688765dccc66f272bbf8e0a642150bed61dd21
TRUE      02/28/2012 11:48  Mined               14KWZvehUAhHtu2tMyykv6PiRMycPz8KtM     0.19822471   0ae3e24993a6ac56a5dcde559189abadc484012165b36027e81afb5e2fcf6899

Since the export has the value of all "inbound" txns and "outbound" txns the sum of the amount column should equal the current balance shown in the client (i.e new wallet "receives" 100 BTC, "spends" 20 BTC and 30 BTC would have a "balance" of 50 BTC.  The ledger would show +100, -30, -20 in the amount column.  The sum of the ledger would also be 50 BTC).

This isn't the case when I did an export.  There is a difference of about 1% between the current "balance" in the client (sum of UTXO) and the sum of the amount column in the ledger.  So why don't the numbers match?  I figured there was a simple reason and five hours later I can't come up with a rational explanation. The number "should" match so even a discrepancy of 1 satoshi would bother me.

Here are some potential explanations for the discrepancy but they turned out to be dead ends.

• The txn history contains unconfirmed or double spent txns.  Verified this isn't the case.
• The txn history is corrupt or has incorrect entries due to a reorg.  I used "zapwallettxes" to clear the txn history.  The client then rebuilds it from the current blockchain.
• I sent funds from one address in the wallet to another and the history reports this incorrectly.   The history reports it correct as the "Payment to yourself" entries only show the net change (i.e. fees as a negative amount).
• I dumped a private key and used it in another wallet.  Upon a rescan after zapwallettxes the wallet should have picked these up.
• I deleted a private key (using pywallet).  This would mean txn records would not show but it wouldn't be reflected in the client balance (i.e. any unspent outputs would be unknown to the client) so the running sum and current balance should still match.
• I imported a private key into the wallet after it had been used in another wallet (this I know I did with a very old vanity address).  The wallet may not have shown that were created prior to the import but after zapwallettxes it should "see" the older txns from the previous wallet as its own.
• The export does not correctly report txns with multiple outputs.  Verified it does.  They are listed as unique entries.
• The export does not correctly handle "change".  It does.  Change doesn't affect "balance" so they are not included in the export.
  

I am sure there is an rational explanation but I just can't see it.  Any ideas?

[34 alphanumeric characters]

A Bitcoin address, or simply address, is an identifier of 27-34 alphanumeric characters, beginning with the number 1 or 3, that represents a possible destination for a Bitcoin payment. Addresses can be generated at no cost by any user of Bitcoin. For example, using Bitcoin-Qt, one can click "New Address" and be assigned an address. It is also possible to get a Bitcoin address using an account at an exchange or online wallet service.

An example of a Bitcoin address is 3J98t1WpEZ73CNmQviecrnyiWrnqRhWNLy.

https://en.bitcoin.it/wiki/Address


Decoded an address consists of
20 byte payload/hash
4 byte checksum
1 byte version
--------------------
total: 25 bytes

34 symbols * Log2(58)/8 bytes  per symbol = 24.896 bytes

So 35 digits?

For custom application development it would be nice to use bitcoind as a node only.  Bitcoind supports "no wallet" parameter and txns can be created rather easily using createrawtransaction, signrawtransaction, and sendrawtransaction.  Unless i am missing something there is no easy way to find the unspent outputs of an arbitrary address.  For now lets ignore non-P2SH/P2PkH outputs.

listunspent only returns unspent outputs for addresses in the wallet.

All the necessary information is in the UTXO (chainstate folder) it just is not accessible from RPC.  Am I missing something?

Green Dot Ending MoneyPak PIN Reloads to Fight Fraud
Green Dot Corp. is changing the way money can be loaded onto its popular prepaid debit card. It’s phasing out the MoneyPak PIN — which allows these transactions to take place online or over the phone — and switching to a “swipe at the register” method that requires the cardholder to be present. Because they were so widely available and easy to use, MoneyPaks had become a favorite way for scammers to steal their victims’ money. Green Dot told NBC News that phasing out the MoneyPak will “eliminate the opportunity for con artists to use it to conduct confidence scams against seniors and other vulnerable populations.” John Breyault, who runs the Fraud.org website, applauded the move, but cautioned that “people need to remain vigilant” because scammers soon will find another way to get their money. The MoneyPak has already been removed from Wal-Mart and many other retail stores. It should be completely gone by early next year. This week, Green Dot told the U.S. Senate Committee on Aging that the vast majority of reloads are now done using the swipe method.

http://www.nbcnews.com/business/consumer/green-dot-ending-moneypak-pin-reloads-fight-fraud-n159536

Someone mentioned using the random ordering of a deck of cards as the entropy for an HD wallet seed (I can't recall if it was electrum developer or armory developer but great idea).  It had never occured to me but there are 52! possible permutations (52 * 51 * 50 * 49 .... *1) to the ordering of a shuffled deck of cards.  That is ~80,658,175,170,943,900,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000 or ~225 bits of entropy.  It is faster, and easier than rolling a number of dice.  Unlike dice there is no issue of bias (a dice which favors higher numbers).  I wouldn't recommend it but you could even leave a backup of the seed in plain sight by keeping the deck in the proper order in the box.

So it got me thinking could you design a wallet which made absolutely no PRNG calls.  CSPRNG are potential weak link in any cryptographic system and as most rely on deep OS level calls flaws or intentional backdoors can be difficult to detect (or at least provable state that they don't exist).  The only client operations that I can think that require the use of cryptographically secure random numbers are key generation, transaction signing, and wallet encryption (salt in key derivation function).

HMAC(card sequence) -> seed
BIP38(seed) -> master pubkey and master key (and indirectly all derived keys and pubkeys)
RFC6979(key, message_hash) -> k value
LEFT16BYTES(HMAC(seed)) -> salt  (on password change LEFT16BYTES(HMAC(old_salt)) -> new_salt)

So one deck of cards, a lifetime of secure transactions, and no additional sources of entropy required.  Anything I am missing?

["asm" : "","hex" : "",]

https://blockchain.info/tx/87d8ed6aae8ad82b01e2b9d7d7ca21e55e83e8fa2ae587892f12b06d3a12253c

getrawtransaction 87d8ed6aae8ad82b01e2b9d7d7ca21e55e83e8fa2ae587892f12b06d3a12253c 1
{
"hex" : "0100000002a08d75950a62c4a7d3b2ce5d5d451f340d26e27ab88c8d1d6a4d1e875846035a00000 000fd600100493046022100f078988f0448183b2439e6a210420a0cf6bd62b11fe6b1ec5bfe8366 a28cfac4022100f6ef61d2888b74dd4d3eb065a853644f9dbd372e312d25f89eb339b7081ca4bc0 1493046022100c99192c8eb5fc7da5a73136e92ecb38e20541a3fbb31cd7a488c8182e899cef702 2100a9427dbe6c878fffb0ece6cf7f89250e46f6f4582f809bc18fecfe273f5138a9014cc952410 4a7a95441bb281f9f851556b8dcae7d5d85746e4382a852dcc8faa3a2320f442fed7d5b4f50d368 cf4e18fb1642cfc4a5091e54d240aba9aeb74b07bcf95ad1d4410417b756f5562aa8b48d38fa256 14391904dbfd7ef5a56e1d8b1e1c0b31200852a0ae3689350ed38be3d6de0c2472205f5ad697208 784aa54174660cdf0b649b814104e35613163e133615ab272c30532c137f61344e14f36a146c3d4 461863e5e5fcdc430206d08119771a18977d3a476f4889e5fb6c6eb5fbdd6ab7519e66b096fdb53 aeffffffff4ba60a7b2dc4ba89f10bb7939a9b8a6d6735642e655ca173625594432a006acd01000 000fd5e0100493046022100b7aa571bfb81fa57e0d366f41be7e1d451fbf9529cbb5b5f0df3e2b9 edc59ad6022100c02138c790e4c84288a7f9338f217a035fb87c4c0716c2dcc3d4a950704f32300 1473044022041c21d0fa28adeab92a355b285f62fdc2b1bee4f16b0d78aac0d3be2c6e986b10220 2e73c88015ae7966aa3fd0e06d7f0bb1d2e231c502ee333cd1f5f3c5f6f3a320014cc9524104a7a 95441bb281f9f851556b8dcae7d5d85746e4382a852dcc8faa3a2320f442fed7d5b4f50d368cf4e 18fb1642cfc4a5091e54d240aba9aeb74b07bcf95ad1d4410417b756f5562aa8b48d38fa2561439 1904dbfd7ef5a56e1d8b1e1c0b31200852a0ae3689350ed38be3d6de0c2472205f5ad697208784a a54174660cdf0b649b814104e35613163e133615ab272c30532c137f61344e14f36a146c3d44618 63e5e5fcdc430206d08119771a18977d3a476f4889e5fb6c6eb5fbdd6ab7519e66b096fdb53aeff ffffff0100000000000000000000000000",
"txid" : "87d8ed6aae8ad82b01e2b9d7d7ca21e55e83e8fa2ae587892f12b06d3a12253c",
"version" : 1,
"locktime" : 0,
"vin" : [
{
"txid" : "5a034658871e4d6a1d8d8cb87ae2260d341f455d5dceb2d3a7c4620a95758da0",
"vout" : 0,
"scriptSig" : {
"asm" : "0 3046022100f078988f0448183b2439e6a210420a0cf6bd62b11fe6b1ec5bfe8366a28cfac402210 0f6ef61d2888b74dd4d3eb065a853644f9dbd372e312d25f89eb339b7081ca4bc01 3046022100c99192c8eb5fc7da5a73136e92ecb38e20541a3fbb31cd7a488c8182e899cef702210 0a9427dbe6c878fffb0ece6cf7f89250e46f6f4582f809bc18fecfe273f5138a901 524104a7a95441bb281f9f851556b8dcae7d5d85746e4382a852dcc8faa3a2320f442fed7d5b4f5 0d368cf4e18fb1642cfc4a5091e54d240aba9aeb74b07bcf95ad1d4410417b756f5562aa8b48d38 fa25614391904dbfd7ef5a56e1d8b1e1c0b31200852a0ae3689350ed38be3d6de0c2472205f5ad6 97208784aa54174660cdf0b649b814104e35613163e133615ab272c30532c137f61344e14f36a14 6c3d4461863e5e5fcdc430206d08119771a18977d3a476f4889e5fb6c6eb5fbdd6ab7519e66b096 fdb53ae",
"hex" : "00493046022100f078988f0448183b2439e6a210420a0cf6bd62b11fe6b1ec5bfe8366a28cfac40 22100f6ef61d2888b74dd4d3eb065a853644f9dbd372e312d25f89eb339b7081ca4bc0149304602 2100c99192c8eb5fc7da5a73136e92ecb38e20541a3fbb31cd7a488c8182e899cef7022100a9427 dbe6c878fffb0ece6cf7f89250e46f6f4582f809bc18fecfe273f5138a9014cc9524104a7a95441 bb281f9f851556b8dcae7d5d85746e4382a852dcc8faa3a2320f442fed7d5b4f50d368cf4e18fb1 642cfc4a5091e54d240aba9aeb74b07bcf95ad1d4410417b756f5562aa8b48d38fa25614391904d bfd7ef5a56e1d8b1e1c0b31200852a0ae3689350ed38be3d6de0c2472205f5ad697208784aa5417 4660cdf0b649b814104e35613163e133615ab272c30532c137f61344e14f36a146c3d4461863e5e 5fcdc430206d08119771a18977d3a476f4889e5fb6c6eb5fbdd6ab7519e66b096fdb53ae"
},
"sequence" : 4294967295
},
{
"txid" : "cd6a002a4394556273a15c652e6435676d8a9b9a93b70bf189bac42d7b0aa64b",
"vout" : 1,
"scriptSig" : {
"asm" : "0 3046022100b7aa571bfb81fa57e0d366f41be7e1d451fbf9529cbb5b5f0df3e2b9edc59ad602210 0c02138c790e4c84288a7f9338f217a035fb87c4c0716c2dcc3d4a950704f323001 3044022041c21d0fa28adeab92a355b285f62fdc2b1bee4f16b0d78aac0d3be2c6e986b102202e7 3c88015ae7966aa3fd0e06d7f0bb1d2e231c502ee333cd1f5f3c5f6f3a32001 524104a7a95441bb281f9f851556b8dcae7d5d85746e4382a852dcc8faa3a2320f442fed7d5b4f5 0d368cf4e18fb1642cfc4a5091e54d240aba9aeb74b07bcf95ad1d4410417b756f5562aa8b48d38 fa25614391904dbfd7ef5a56e1d8b1e1c0b31200852a0ae3689350ed38be3d6de0c2472205f5ad6 97208784aa54174660cdf0b649b814104e35613163e133615ab272c30532c137f61344e14f36a14 6c3d4461863e5e5fcdc430206d08119771a18977d3a476f4889e5fb6c6eb5fbdd6ab7519e66b096 fdb53ae",
"hex" : "00493046022100b7aa571bfb81fa57e0d366f41be7e1d451fbf9529cbb5b5f0df3e2b9edc59ad60 22100c02138c790e4c84288a7f9338f217a035fb87c4c0716c2dcc3d4a950704f32300147304402 2041c21d0fa28adeab92a355b285f62fdc2b1bee4f16b0d78aac0d3be2c6e986b102202e73c8801 5ae7966aa3fd0e06d7f0bb1d2e231c502ee333cd1f5f3c5f6f3a320014cc9524104a7a95441bb28 1f9f851556b8dcae7d5d85746e4382a852dcc8faa3a2320f442fed7d5b4f50d368cf4e18fb1642c fc4a5091e54d240aba9aeb74b07bcf95ad1d4410417b756f5562aa8b48d38fa25614391904dbfd7 ef5a56e1d8b1e1c0b31200852a0ae3689350ed38be3d6de0c2472205f5ad697208784aa54174660 cdf0b649b814104e35613163e133615ab272c30532c137f61344e14f36a146c3d4461863e5e5fcd c430206d08119771a18977d3a476f4889e5fb6c6eb5fbdd6ab7519e66b096fdb53ae"
},
"sequence" : 4294967295
}
],
"vout" : [
{
"value" : 0.00000000,
"n" : 0,
"scriptPubKey" : {
"asm" : "",
"hex" : "",
"type" : "nonstandard"
}
}
],
"blockhash" : "0000000000000000005ed62331546b26c7f660c4150db84dcf21414dceee29b2",
"confirmations" : 8241,
"time" : 1400451717,
"blocktime" : 1400451717
}

It has an empty/null PkScript ("output script").  Would a ScriptSig ("input signature") of just OP_TRUE lead to a valid input?
Also is it still valid to create outputs with zero value (other than OP_RETURN)?

[It is only defined for non-hardened child keys.]

What is the purpose of "hardened key" in BIP32? The BIP seems to talk around the concept without actually addressing the reason for using a hardened key over a normal (non-hardened) key.

We represent an extended private key as (k, c), with k the normal private key, and c the chain code. An extended public key is represented as (K, c), with K = point(k) and c the chain code.

Each extended key has 2^31 normal child keys, and 2^31 hardened child keys. Each of these child keys has an index. The normal child keys use indices 0 through 2^31-1. The hardened child keys use indices 2^31 through 2^32-1. To ease notation for hardened key indices, a number iH represents i+2^31.

Ok so using an index > 2,147,483,647 results in a hardened key otherwise it is a normal key.

Given a parent extended key and an index i, it is possible to compute the corresponding child extended key. The algorithm to do so depends on whether the child is a hardened key or not (or, equivalently, whether i ≥ 231), and whether we're talking about private or public keys.

...
The function CKDpub((Kpar, cpar), i) → (Ki, ci) computes a child extended public key from the parent extended public key. It is only defined for non-hardened child keys.

So hardened keys (which are defined as a child [private]Key produced from an index > 2^31-1) can not be derived from the parent's extendedPubKey.  Is the reason for "hardening" a key to prevent recomputation of the parent's extended[Private]Key (and thus all derived [private]Keys) in the event of the accidental disclosure of a)parent's extendedPubKey AND b) any private key that is a child of that parent?  Beyond that key leakage scenario is there any risk or limitation of using normal (non-hardened) keys over hardened ones?

The reason I ask is because an obvious usage of BIP32 is to create a "watching wallet".  A server which is processing incoming payments would only need the extendedPubKey and an indexer.  With just those elements (and no [private]Key) the server could deterministically create a number of PubKeys as needed.  The obvious advantage is the server has no knowledge of the extended[Private]Key for security reasons. Am I correct to assume this isn't possible to with hardened keys?

If the prior two assumptions are correct then I am going to infer that the rationale for using hardened keys is to gain added security in a scenario where the extended[Private]Key is present and thus there is no need to use/share the extendedPubKey.  An example would be internal usage by a desktop wallet for computing change addresses.  The wallet can derive PubKeys indirectly from the derived [private]Keys.  Is this a correct assumption?

As a recommendation for the BIP32 document, it may be useful to early on define normal vs hardened keys and articulate a rationale for selecting one over the other.

In validating a UTXO parser I started looking at various outputs which are provably unspendable.  As of block #305303 2,745.22283996 BTC have been provably lost.  The total number of coins lost is higher potentially much higher but most of those losses can't be proven.   Funds sent to outputs that can never be redeemed can be provably shown to be lost.

Code:

Category       NumOutputs    AmountLost
-----------------------------------------
BugOpFalse            23   2,609.36304319
BugP2Pool            182       0.60280235
BugInvalidOpcode      14       0.04520008
BugInvalidPubKey  17,112       0.00242288
BugParseError          1       0.00040000
ZeroValue *        3,080       0.00000000
MissingFromUTXO **   ---     135.20897146
-----------------------------------------
Total             20,412   2,745.22283996 BTC

* Zero value unprunable outputs are not invalid outputs but they are undesirable.  I was surprised to see there are over three thousand in the UTXO.  In the future the creation of new zero value outputs (with the exception of the prunable OP_RETURN) could be made invalid and potentially even these outputs pruned off by a hard fork.

** As of block 305,303 the coin supply is limited to 12,882,575 BTC.   This is based on the max subsidy per block and the block height.  However the UTXO (set of all unspent outputs) is only 12,882,439.79102854 BTC.  Some of the difference may be due to OP_RETURN outputs (which are unspendable by protocol) having a value set.  This could be accidental or intentional.  Another source of lost coins is due to miners taking less than the maximum block reward which in effect "de-mines" an amount of coins equal to the difference between the allowed reward and the taken reward.

[OP_FALSE]

In building a UTXO parser I came across a number of high value "custom" scripts and started looking at each of them individually.   They all look like really expensive mistakes.

They all have the same output signature.  Here is a link to the most expensive one (497 BTC)
https://blockchain.info/tx/03acfae47d1e0b7674f1193237099d1553d3d8a93ecc85c18c4bec37544fe386

OP_DUP OP_HASH160 OP_FALSE OP_EQUALVERIFY OP_CHECKSIG

If they indeed are unspendable (which I believe they are because it would require a signature from a pubkey which hashes to zero) and you didn't make one of these 23 txns well the good news is that is 2609.36304319 BTC destroyed, so your BTC are worth a little more.

Code:

TxId:Index                                                          Type      Length         Value
--------------------------------------------------------------------------------------------------
03acfae47d1e0b7674f1193237099d1553d3d8a93ecc85c18c4bec37544fe386:1  RawScript      5  497.00000000
aa62bdd690de061a6fbbd88420f7a7aa574ba86da4fe82edc27e2263f8743988:0  RawScript      5  367.75849319
2d00ef4895f20904d7d4c0bada17a8e9d47d6c049cd2e5002f8914bfa7f1d27b:1  RawScript      5  200.00000000
aebe39a99114f1b46fc5a67289545e54cbfec92d08fc8ffc92dc9df4a15ea05a:1  RawScript      5  143.62000000
6a86e6a5e8d5f9e9492114dafe5056c5618222f5042408ad867d3c1888855a31:0  RawScript      5  100.00000000
15ad0894ab42a46eb04108fb8bd66786566a74356d2103f077710733e0516c3a:1  RawScript      5  100.00000000
9edab6e7fadf1d6006315ff9394c08a7bf42e19cf61502200a1f73994f8da94b:1  RawScript      5  100.00000000
3ab5f53978850413a273920bfc86f4278d9c418272accddade736990d60bdd53:1  RawScript      5  100.00000000
835d4dcc52e160c23173658de0b747082f1937d1184e8e1838e9394bc62c0392:1  RawScript      5  100.00000000
5bd88ab32b50e4a691dcfd1fff9396f512e003d7275bb5c1b816ab071beca5ba:1  RawScript      5  100.00000000
0ca7f7299dc8d87c26c82badf9a303049098af050698c694fbec35c4b08fc3df:0  RawScript      5  100.00000000
07d33c8c74e945c50e45d3eaf4add7553534154503a478cf6d48e1c617b3f9f3:0  RawScript      5  100.00000000
81f591582b436c5b129f347fe7e681afd6811417973c4a4f83b18e92a9d130fd:1  RawScript      5  100.00000000
305fbc2ec7f7f2bc5a21d2dfb01a5fc52ab5d064a7278e2ecbab0d2a27b8c392:0  RawScript      5   98.48055000
6d39eeb2ae7f9d42b0569cf1009de4c9f031450873bf2ec84ce795837482e7a6:0  RawScript      5   98.00000000
633acf266c913523ab5ed9fcc4632bae18d2a7efc1744fd43dd669e5f2869ce5:0  RawScript      5   65.00000000
6d5088c138e2fbf4ea7a8c2cb1b57a76c4b0a5fab5f4c188696aad807a5ba6d8:0  RawScript      5   45.82000000
f0137a6b31947cf7ab367ae23942a263272c41f36252fcd3460ee8b6e94a84c1:0  RawScript      5   39.81000000
ddddf9f04b4c1d4e1185cacf5cf302f3d11dee5d74f71721d741fbb507062e9e:0  RawScript      5   37.00000000
3be0ac3dc1c3b7fa7fbe34f4678037ed733a14e801abe6d3da42bc643a651401:1  RawScript      5   35.78400000
7ad47a19b201ce052f98161de1b1457bacaca2e698f542e196d4c7f8f45899ab:0  RawScript      5   35.78000000
111291fcf8ab84803d42ec59cb4eaceadd661185242a1e8f4b7e49b79ecbe5f3:1  RawScript      5   24.31000000
64c01fedd5cf6d306ca18d85e842f068e19488126c411741e089be8f4052df09:1  RawScript      5   21.00000000

[020000000000000000000000000000000000000000000000000000e000000000001]

I am looking for a good way to create more than one P2SH address (address is based on the hash of the script) which is encumbered by the same set of private keys.

"Standard" multisig script is <OP_m> [PubKeys] <OP_n> <OP_CHECKMULTISIG>

So with the following PubKeys:

02171f8058701e94efbbca609509b858f0ef1e32923d51991eac403ade22952421
03694411c22d04a72ced42e79d584de695e322e67b9f2bafc1f2805146dda20f60
036b033de1664219216408262a9a32dfb801f09b0bfb53dd83f036fedf1a644488

The script would be:

<OP_2> <"OP_PUSH_33"> <PubKey1:02171f8058701e94efbbca609509b858f0ef1e32923d51991eac403ade22952421> <"OP_PUSH_33"> <PubKey2:03694411c22d04a72ced42e79d584de695e322e67b9f2bafc1f2805146dda20f60> <"OP_PUSH_33"> <PubKey3:036b033de1664219216408262a9a32dfb801f09b0bfb53dd83f036fedf1a644488> <OP_3> <OP_CHECKMULTISIG>

In hex:

522102171f8058701e94efbbca609509b858f0ef1e32923d51991eac403ade22952421210369441 1c22d04a72ced42e79d584de695e322e67b9f2bafc1f2805146dda20f6021036b033de166421921 6408262a9a32dfb801f09b0bfb53dd83f036fedf1a64448853ae

The base58 encoded (0x05 prefix byte) HASH-160 hash of the script above is the address 3KKVrgqN2AYAsLQoJQchT5GK2KnKQ3dAPL.  The simplest way to get a new address would be to change the keys but lets say I want a set of addresses which use the same keys.  I could make one of the keys bogus and use it as a counter value (obviously this requires more keys i.e. for a real 2-of-3 script would require creating a pseudo 2-of-4 script).  

As an example, in the two scripts the following key the first key is invalid and just used to create a unique hash output so this is effectively a 2-of-2 script despite being 2-of-3.

script: 5221020000000000000000000000000000000000000000000000000000e0000000000012103694411c22d04a72ced42e79d584de695e322e67b9f2bafc1f2805146dda20f6021036b033de 1664219216408262a9a32dfb801f09b0bfb53dd83f036fedf1a64448853ae
address: 3DmfeKGpHVsC7HHf2GumUE31NG14f4P8SM

script: 52210200000000000000000000000000000000000000000000000000000000000000022103694411c22d04a72ced42e79d584de695e322e67b9f2bafc1f2805146dda20f6021036b033de 1664219216408262a9a32dfb801f09b0bfb53dd83f036fedf1a64448853ae
address: 3AeD9PWp4nCi5x1BKFhyX4qHVkiejfTbq8

Addresses can be created on demand by incrementing the counter value in the pseudo-key and recomputing the scriptHash.  This naive approach however is rather wasteful (and cludgy). It requires 34 additional bytes when only a few bytes are needed.  If we assume a 3 byte counter (>16 million sequential addresses) then this has 31 bytes of overhead.

Any ideas on a more space efficient way to accomplish the same task?  I am thinking maybe pushing the nonce to the stack and then using an OP_DROP?

<"OP_PUSH_3"> <0x010000> <OP_DROP> <OP_2> <"OP_PUSH_33"> <PubKey1> <"OP_PUSH_33"> <PubKey2> <OP_2> <OP_CHECKMULTISIG>

Any other ideas?  Any way to make it pass IsStandard()?

[Some rough numbers of the current storage requirements]

Many times in the past people have suggested using a DHT (either by name or by reinventing the concept) to store the entire blockchain but this breaks the trust-free nature of blockchain verification.  To verify blocks a node must have the full block history.  With the TxIds from the longest chain however you can not be spoofed if you ask an arbitrary node for the details of that txn.   To create a different txn with the same TxId would require a preimage attack on the hash and that is considered infeasible as long as SHA-256 is cryptographically strong.   This would allow txns to be stored in a distributed trust free manner using a DHT (Distributed Hash Table).  The primary reason would be to reduce the minimum storage requirements of each node and allow nodes to support the network on a partial basis.   Currently there is an all or nothing requirement.  You are either a full node (cost is ~25GB) or you do not help the network in any way (SPV nodes do not improve the security of the network).  DHT of Transactions, a Distributed Transaction Table (DTT).  The required changes not be significant and could be done in a backwards compatible manner.

1) Block structure for DHT nodes would be changed such that it contains just the block header and tx hashes (TxId).
2) A new block message would be created which allows relaying this "reduced block" (also useful in reducing the orphan cost of blocks).    
3) The full transaction would be stored in the DTT.  Individual nodes would store a subset of the DTT.
4) Current "full nodes" could be converted into DHT nodes by simply maintaining a 100% share of the DHT.

The naive solution would be for individual nodes to drop all txns which aren't in their share and use the DTT as needed.  This would be rather inefficient as the probability of a node needing a particular txn (either internally or because it was requested by a peer) is not consistent for all txns.   Certain txn would always be maintained in the local cache those would include:
a) Txns in "recent" blocks.  In the event of a reorg updating the UTXO requires knowing the details of the txns of the block being orphaned.
b) Txns in the UTXO.  This is txns which have at least one unspent output.  The UTXO is needed to validate new txns and new blocks.
c) Txns in the memory pool.  This is the set of txns which are valid and known to the node but not yet included in a block.  If block messages include txns hashes only the memory pool will be needed to validate a block.
d) Txns involving the user's keys.  This isn't a requirement but a user has a vested interest in ensuring copies of his txns are maintained.  If nothing else this may be psychologically useful to improve trust in the DHT concept.

Understand however the integrity of txns comes from their txn hash (TxId) so a "badly optimized" DHT would have suboptimal performance but not reduced security.

Some rough numbers of the current storage requirements
Full raw blockchain = ~18.9 GB
Undo chain = ~2.5 GB (can this be reduced? seems undo information past say the last 144 blocks would be of little value)
Blockchain Indexes = ~2.0 GB (I haven't decoded these but find it interesting it is so large relative to the blocks)
Chainstate (UTXO) = ~400 MB (compressed format contains unspent outputs & txn metadata only)
Memory Pool = <1MB (unconfirmed txns, 538 at time of post)
Total: ~23.8 GB

The current blockchain stats
Number of blocks: 307,394
Number of transactions: ~41,182,000
Number of unspent txns: 3,347,562  (a txns with at least one unspent output)
Number of unspent outputs: 11,648,626

Breaking this down
Size of hash-only blocks: 1,320 MB (includes headers & txn hashes)
Size of the UTXO: 400 MB (unspent outputs only)
Size of the Unspent Txns in entirety: ~1,500 MB

So the local cache storage requirements for a DHT node would be: 1.7 GB (or 2.8 GB if full txns of UTXO elements are retained).  If we assume the average node maintains a 5% DHT share of the remaining txns (bulk of the block bodies) that would be another 1GB.  DHT nodes would also keep a local copy of all memory pool but that is a rounding error on storage requirements.

This wouldn't reduce the bootstrap time or (or amount of bandwidth used) for bootstrapping full nodes.  As a full node you would still need to download 100% of the txns of 100% of the blocks to verify the longest chain is the longest valid chain.  However once bootstrapped the ongoing resource requirements would be reduced.  This would work best if the protocol contained a block message which just sent blockheader & txns hashes.  Currently when a full node receives and verifies a new block which extends the longest chain it stores the full block and removes the now spent txns from the UTXO.  DHT nodes would instead record the reduced block (header & hashes only), then saved its DHT share of the spent txns and discard the rest.  To reduce the overhead from reorgs and provide better coverage for syncing nodes needing only the tip of the blockchain it may make sense for DHT nodes to retain all txns for the most recent x blocks (144? one block day?).

If a structure like this was used for all nodes then "full nodes" would simply be nodes participating in the txn DHT that retain a 100% copy of the full txn set.  The best comparison would be to the status of a "seeder" in the bittorrent protocol.  Since all DHT nodes would retain a full copy of the UTXO these nodes could still support SPV clients.  SPV clients could actually support the network by retaining a portion of the txn set.  Retaining older txns would be especially beneficial in that they could reduce the load on full nodes when the network bootstraps new full nodes.

Is there a relatively easy way to extract the UTXO from Bitcoin Core?

Code:

{
"height" : 305134,
"bestblock" : "00000000000000003f0bb243b026fa904946f262eebf0684614ebb2622e4e554",
"transactions" : 3304525,
"txouts" : 11337577,
"bytes_serialized" : 393434666,
"hash_serialized" : "4132c6a9bdb7f2492348ba9cf8ca1c1fb954f619805de0c3cd01847f969ece65",
"total_amount" : 12878214.79102854
}

I would like the set of these 11,337,577 outputs so they can be parsed and dumped into a database for analysis.  Any links to information or relevant code on how Bitcoin-Core stores the UTXO?  I understand I could use a blockchain parser to build the UTXO and that is probably happening next (as I want to see how it has changed over time) but a dump would be a good start.  Even just a dump of the index (tx_out_hash & tx_out_index) would be great.

[1 of 1 MULTISIG]

1 of 1 MULTISIG
Is a 1 of 1 "multisig" valid?
OP_1 <PubKey> OP_1 OP_CHECK_MULTISIG

Example hasn't been spent but w/ a 1 satoshi output I don't think it will be.  Is output 20 otherwise valid?
http://webbtc.com/tx/b96af3b69b48a82c5eae3e44ebb6ef93f30d7764b1d5b40243e11b0d374ac1b7

P2SH "Spend" without a signature?

http://webbtc.com/tx/6a26d2ecb67f27d1fa5524763b49029d7106e91e3cc05743073461a719776192

Code:

getrawtransaction 6a26d2ecb67f27d1fa5524763b49029d7106e91e3cc05743073461a719776192 1

{
"hex" : "0100000001f6ea284ec7521f8a7d094a6cf4e6873098b90f90725ffd372b343189d7a4089c0100000026255121029b6d2c97b8b7c718c325d7be3ac30f7c9d67651bce0c929f55ee77ce58efcf8451aeffffffff0130570500000000001976a9145a3acbc7bbcc97c5ff16f5909c9d7d3fadb293a888ac00000000",
"txid" : "6a26d2ecb67f27d1fa5524763b49029d7106e91e3cc05743073461a719776192",
"version" : 1,
"locktime" : 0,
"vin" : [
{
"txid" : "9c08a4d78931342b37fd5f72900fb9983087e6f46c4a097d8a1f52c74e28eaf6",
"vout" : 1,
"scriptSig" : {
"asm" : "5121029b6d2c97b8b7c718c325d7be3ac30f7c9d67651bce0c929f55ee77ce58efcf8451ae",
"hex" : "255121029b6d2c97b8b7c718c325d7be3ac30f7c9d67651bce0c929f55ee77ce58efcf8451ae"
},
"sequence" : 4294967295
}
],
"vout" : [
{
"value" : 0.00350000,
"n" : 0,
"scriptPubKey" : {
"asm" : "OP_DUP OP_HASH160 5a3acbc7bbcc97c5ff16f5909c9d7d3fadb293a8 OP_EQUALVERIFY OP_CHECKSIG",
"hex" : "76a9145a3acbc7bbcc97c5ff16f5909c9d7d3fadb293a888ac",
"reqSigs" : 1,
"type" : "pubkeyhash",
"addresses" : [
"19E6FV3m3kEPoJD5Jz6dGKdKwTVvjsWUvu"
]
}
}
],
"blockhash" : "00000000000002dc756eebf4f49723ed8d30cc28a5f108eb94b1ba88ac4f9c22",
"confirmations" : 134376,
"time" : 1331137983,
"blocktime" : 1331137983
}

Ok Input references unspent previous output 9c08a4d78931342b37fd5f72900fb9983087e6f46c4a097d8a1f52c74e28eaf6:0 (txid:index) which is
"scriptPubKey": "OP_HASH160 19a7d869032368fd1f1e26e5e73a4ad0e474960e OP_EQUAL"

"Pay to the Script which hashes to 19a7d869032368fd1f1e26e5e73a4ad0e474960e".  Ok good so far.

The input ScriptSig is 5121029b6d2c97b8b7c718c325d7be3ac30f7c9d67651bce0c929f55ee77ce58efcf8451ae

The HASH-160(5121029b6d2c97b8b7c718c325d7be3ac30f7c9d67651bce0c929f55ee77ce58efcf8451ae) = 19a7d869032368fd1f1e26e5e73a4ad0e474960e
So redeemScript is correct (hashes to ScriptHash).  Looks good so far.

The bolded 0x21 means push the next 33 bytes (the compressed PubKey).  Right?

So decoded we have:
OP_1 0x029b6d2c97b8b7c718c325d7be3ac30f7c9d67651bce0c929f55ee77ce58efcf84 OP_1 OP_CHECK_MULTISIG

Ok another 1 of 1 multisig except a P2SH version this time.  We need the tx signed by 1 signatures from the set of 1 PubKey(s) {029b6d2c97b8b7c718c325d7be3ac30f7c9d67651bce0c929f55ee77ce58efcf84}.  Right?

Except there is no signature.

Here is the Raw Transaction:

0100000001f6ea284ec7521f8a7d094a6cf4e6873098b90f90725ffd372b343189d7a4089c0100000026255121029b6d2c97b8b7c718c325d7be3ac30f7c9d67651bce0c929f55ee77ce58efcf8451aeffffffff0130570500000000001976a9145a3acbc7bbcc97c5ff16f5909c9d7d3fadb293a888ac00000000

The redeemScript is bolded.  The blue is the TxOutHash, the green the TxOutIndex, and the red is the Sequence (end of input).  No signature for the 1 of 1 multisig.

Tx was included in a block so what am I missing?