There's a lot of data there, so it's tough for me to parse all of it by eye.  But it looks like you're appending a big-endian hashcode instead of little-endian.  Also, try switching the endianness after the sha256^2 operation.  You might have to do some guess-and-check for this part.

Also don't forget to add the 1-byte hashcode to the end of the signature you generated.  It may not be relevant for OpenSSL, but if you're signing transactions to be put in the blockchain, you'll need that 0x01 byte at the end of the pubkey script.

I'm at work, and don't have my python/btc tools here, but from manual inspection that a3 appears to be part of the signature.  The signature is represented as an r and s value, and in this case the s-value is actually 33 bytes, not 32.  I haven't counted it all up, but I could see how that would lead you to believe there's an extra byte (if you look at the s-value, you'll notice it's actually 32-bytes, but has an extra 0x00 leading byte to remove sign ambiguity).

As for signing:  don't forget that the process for generating a signature involves a random number.  Every signature you generate on the exact same data will produce a completely different-looking signature.  The only way to know for sure whether you did it right is if you used the same random number both times, or (more likely) execute your verification algorithm to check if the signature is valid. 

It's the very last byte of the 72-byte signature (bolded below):

304502205ea291ce55ecc95f346f6be2c198993dcb1a72cc4eddf520f173ed9ac85a1e03022100d 0ae6c394d014de8fecb44d034904a0c6142e6335a394aa4629d7a839aaaa2a301

Expand that 0x01 to four bytes, little-endian, and attach it to the modified, serialized txCopy before signing.

Sorry to break the news to you:  but you're asking about OP_CHECKSIG which is really complicated.  But, see my post here for a description of how it works (about halfway down the page).  It tells you exactly how to construct the string to be signed.

And everything should be in little-endian unless otherwise stated, including the hashcode.  Just make sure the final hash is big-endian before signing.  Yeah, it's ridiculous...that's why I made this diagram!

-Eto

Nibor,

First of all: as for the 0.005 payment/bribe () I already thought about something like that in the past, but I just don't see how to make it work.  First of all, if I add an output less than 0.01 BTC to my tx, then it's considered dust and cannot satisfy the AllowFree requirement.  Sure, sometimes it gets propagated without a fee, but there's no guarantee if any of the outputs are less than 0.01.   And most people hate paying transaction fees...

On the other hand, if your fee is >=0.01, the person can do it, as long as they don't mind breaking off some of their change output for you.  But for whatever reason, despite being practically insignificant, people are really averse to paying such fees.  I think it's fair for your service, but admittedly, the Tx construction algorithm has to be modified to include an extra output just for the forwarding node.

SelectCoins is the algorithm that takes all the unspent TxOuts that you can spend, and figures out a "good" subset of them for a desired transaction.  For instance, I have 5 unspent TxOuts to my wallet {10.3, 0.003, 0.148, 29.0, 2.2}.  If I want to send someone 2.5 BTC, what is the best way to do it?  Or 0.002?  Or 31?  etc.   There's actually too many solutions, and a lot of them require fees (if a lot of outputs are used, and/or young, small outputs), or link a lot of addresses together on the input side (since all addresses on the input side are clearly owned by the same person, you want to limit how many are used).

One of the criteria for anonymity of the transaction is the outputs -- usually there's two outputs: reicipient, and change-back-to-self.  The sender usually picks a new address for the change part, which means, if both outputs are the same, then it's not actually clear which is the recipient and which is change.  If you're concerned about someone tracking the flow of your money, having two outputs, one of 10.00 and change of 1.93283, it's pretty obvious which is which.  However, in the case of the tx I sent you, it's got 20 outputs of exactly 0.5 BTC.  An investigator who really wants to know who I'm sending money to, would now have to investigate the pool of 20 different addresses to find the actual recipient since there's no information to distinguish them in the tx itself.    Most SelectCoins algs could not come up with this Tx on their own, but I wouldn't mind figuring out how to get mine to do it

Perhaps we're both right.  I see use in both applications, and it seems like the kind of thing the user should decide.  I think, by default, the no-sensitive-data-regardless-if-it's-encrypted method would be the default.  Just make sure there's a unique salt so that users who pick the same passphrase don't end up sharing funds.    

At the same time, I think there are plenty of users who are not concerned about encrypted key data, and should have the ability to use the more flexible, wallet (can change passphrase and import keys).  Personally, if the US gov't is okay with encrypting top secret data with AES-256, I'm happy protecting my funds with it.  Of course, only if it's done right -- and I think it can be as long as we write the code from the outset to be secure and done properly.

Perhaps it's too much change for the Satoshi client, in which case, we should divert our discussion back to how to deal with the current Satoshi implementation issue.  Personally, I believe the best thing to do is rewrite the entire file, "born encrypted", and try to zero out the old file before deleting it.  It sounds like Gavin has pretty much proposed this, already.

I know that when you delete/overwrite files, it doesn't necessary remove the old data (depending on FS), but there's a good chance if you actually open the file in "rw" mode and seek to the keys to overwrite the bytes in place?  I would expect that if we manually find all the private keys, and zero/randomize them, that it will almost be guaranateed to modify the file in-place.  As Gavin said, it's outside of scope to worry too much about this data left on the HDD, but it doesn't hurt to try.

I have been doing some work on this myself.  I don't see how we can get away with not storing any key data in the wallet at all.  What if someone wants to change their passphrase?   If the passphrase is actually determining their wallet keys, then they're stuck with that passphrase even if there's evidence of compromise.  They have to create a new wallet from a new passphrase and throw away all their old addresses.  Not to mention, some people might want to import non-generated keys in their wallet (there's a variety of reasons for this).  I think the answer is in "simple" wallets, with straightforward binary representations, and all handling of data is controlled by the developer, not some magical DB library.

For this reason, I'm doing the following in my client:  The determinstic generator is in the header of the wallet file:  

This is AES-encrypted by a key generated from a passphrase by a memory-hungry, compute-heavy key-derivation function.  Once unencrypted, you can get to any of your generated private keys.  AND this same encryption key can be used to encrypt/decrypt imported keys into the wallet.  They would be appended to the file the same way as the privkey generator, but with unique bytes identifying the encryption state and IV of this new key.  

By doing it this way, the wallet is much more flexible/versatile, there's no ambiguity about where key data is and isn't, you can support arbitrary key generation as well as importing keys, and you control the file format, instead of using an external library with unknown behaviors under-the-hood.  Maybe I'm naive, but I think this is the way to go.

I don't really understand your example anymore.  Did you give Bob address X and Y?  Why is Bob sending you two outputs to two different addresses?

If you gave X to Bob, and Y to Alice, and you got a transaction with two outputs, one to each address... the only reasonable explanation is that Bob and Alice are the same person, or are doing something very creative -- Bob and Alice would have to get together, and create a single joint transaction using inputs from both of their wallets, and sending to outputs both X and Y in the same transaction.  That just doesn't happen, and the software has no incentive to try to distinguish this situation (if it were possible).

Or am I still misunderstanding?

This is actually a feature, not a bug, for the sake of "let's dumb it down for the average user."    If you look at it in block explorer, you'll see what you expect.

Let's look at a simpler example: basically every transaction that is executed by the client.  You want to send someone 72 BTC.  So your client combines 4 different inputs that add up to 85, constructs an output transaction with 72 BTC and 13 BTC change back to yourself (at a different address).  

If what you suggest is a bug were fixed, the user would see 5 new transactions in their client... 4 transactions of arbitrary amounts (but adding up to 85) and one transaction in for 13.  The user never authorized 5 different transactions of any of those amounts:  he only issued one transaction for 72 BTC.  

By accumulating the data in the transaction ledger to only show the net outcome of a transaction, the user sees what they expected:  which is the total net debit to their wallet is 72 BTC.  The alternative would be ridiculously confusing.  It would even be annoying for users that understand it:  they know that multiple addresses were used, but they only want to see the net, too.

In your example, this is really the same thing.  It's just with multiple output addresses instead of input addresses.

This one is for you, Nibor!  I decided to have fun with high-anonymity transactions and used your webpage to make sure I did it right.  One of those outputs is yours   I just gotta figure out how to get my SelectCoins to do that automatically...

If you read the previous posts in this thread, you'd see the exact conditions under which you are required to pay a fee.  Most casual users do not run into this case very often, and can get away without ever paying a fee.  

That's just not feasible.  Remove all fees and then there's nothing stopping 'bad people' from flooding the network with millions of tiny transactions.  With the current limits in place, they could add about 1 GB of data to the permanent blockchain every week (about 150 MB per day).  It's not just for rewarding miners, it's also a security thing.  The current fee schedule is actually designed for this purpose, instead of for rewarding miners. 

This is a 0.4.0 issue.  Gavin was trying to release 0.5 in the immediate future, but I think he wants to hold it up until we find a fix for this problem.  We should get this fixed ASAP.

Has anyone verified this?  I don't want to stop any shows unless someone else can confirm it.

BTW, I am on Ubuntu 10.04, 64-bit, if that matters...

This is where casascius' recommendation for "sweepprivkey" becomes extremely useful.  

2012 Update:  It's now been a few months since this event, so I think it's safe to describe, for the record, what actually happened.  The Bitcoin wallet file is managed by BSDDB, which is a database engine.  Presumably, a database engine was used, in order to ensure ACID transactions (atomicity, consistency, isolation, durability), which makes sure that the database won't be corrupted by inconveniently-timed power outtages, etc.  Most importantly, it should protect your private keys from ever getting corrupted.

The problem with a database manager, though, is that it does its own thing with the file storage.  It's a black box in terms of its behavior when you tell it to "delete" something.  You'd hope it would actually remove the data, or at least blank it out, but that's not necessarily true.  In the case of BSDDB, when Satoshi client says "delete this key and replace it with the encrypted key" it seems to not actually do that.  Instead, it simply marks the entry deleted, and then appends the new data to the end of the file.  It doesn't even overwrite the original data.  

From the user perspective, this is completely transparent, because the user must ask the BSDDB interface for the data, and BSDDB reports that it's gone, since the pointer to the data was discarded.  But, it's still in the binary file!

Before wallet encryption, I had written a pure-binary key-extractor script, to pull my private keys out of the wallet file (it was an elaborate search using the ECDSA library to find public-keys and find the associated private-key nearby).  After Bitcoin 0.4.0 was released with wallet encryption, I re-ran my script for fun.  To my surprise, it found 36 of 112 private keys!  Whoops!

I wasn't too involved in the fix for the problem, since I have no experience with BSDDB.  But the solution had to do with convincing the database to actually remove/overwrite the data, and allow for wallets to be "born" encrypted to avoid any risk of it happening again.   I don't know the gory details though, you'd have to ask the devs for specific details about how they fixed it.




Original Post:
I think I might've found a bug in the wallet encryption.

Edit: This problem has been fixed in the client version 0.5.0.  Please see the release notes for information on fixing your wallet.   The most important information is:

Actually, that's not correct.  I copied that equation directly out of the source code.  wallet.cpp:985.  For whatever reason, the source code uses 1000 instead of 1024.   Although in their defense, they never mention "kB" or "kilobytes", so they can easily argue they never intended to measure kB.

You're close:  it is:


Which is like roundUp(txSizeInKb), using nBaseFee==MIN_TX_FEE(==0.0005).  However, if I understand correctly, that is the min fee for Satoshi-based miners to accept your tx to be included in a block.  You only need nBaseFee==MIN_RELAY_TX_FEE (==0.0001) to get the tx propagated through the network: and that should be enough for it to find its way to a miner that will accept sub-standard transaction fees.  Depending on what proportion of miners accept this fee, it could take a few blocks to get into the blockchain.  If you make 0.0005 the base fee, then all miners should try to include your tx in the next block.

Also, if you have any outputs that are strictly less than 0.01, you will need to include that MIN_TX_FEE or MIN_RELAY_TX_FEE regardless of tx size.  Similarly, if the size is small but not sub-cent outputs, you will still pay a fee for low-prioirty transactions (summing to less than 1-BTC-1-day old).

Well then, that answers my question -- these tx fee rules need to be followed, unless you plan to seek out specific miners who will accept these transactions.  For a regular client, there's no guarantees you will be connected to a node that accepts them, and thus the user will be very confused when their tx says "Sent!" but then an hour later nothing has happened.

By the way, I just tried to send 0.007 BTC without a fee and it didn't show up in the blockchain, even after 12 hours.  I then created a competing transaction, using the same outputs but not requiring a fee, and it went through immediately.  So it would seem that nodes didn't even add it to their memory pool without the fee (which is good, otherwise that output would've been in purgatory).

The client won't let you do this at all, so you have to construct the tx by hand in order to test test.  It looks like the client is using these rules, or something very close to determine when to allow a free tx through. 

One other question about this is:  what is the deal with zero-confirmation txs?  I didn't see anything about those requiring a fee.  Or is it only that they contribute nothing to your priority, so it would be okay as long as you have other inputs with priority above COIN*144/250?