Thanks.  I did a lot of this because of the learning curve to get into this stuff.  Bitcoin is complicated.   But if you want to dig into it, you'll have to do a little messing around on your own.  I don't want to ruin all the fun for you

The rest of the serialization is trivial once you know the transactions.  The header is exactly 80 bytes, 6 values.  And you can look up how VAR_INTs are read from that protocol page.  The magic bytes are f9beb4d9.  The protocol page isn't pleasant, but it's not useless, either.

I can vouch that they are extremely consistent.  Armory does a raw scan of them on every load (which will be changed soon, thank god).  Despite being used by thousands of people over the course of the last 12-18 months, there is rarely any problems with it.   Some of the recent Armory problems have actually had to do with memory usage on 32-bit architectures, and not due to actual blk*.dat corruption (switching to the 64-bit version solved it).

The serialization of the headers and the transactions are documented on the protocol specification page.  What I wrote above is how those individual pieces come together in the blk*.dat files which contain the entire blockchain.  I did make an exact binary map of the transactions, though it was made before P2SH.  You can see it here.  There's a few other nice inkscape drawings here. 

The only caveat is that now the blk*.dat files are padded to 16MB chunks, so the end of the files may contain lots of zeros.  But this is easy to detect, because blocks are never split between files.  So you only have to check for (isEndOfFile() OR magicBytes==0x00000000) to determine that there are no more blocks in the file.

This patch makes the best case the same as the worst case: zero-confirmation transactions can be "overridden" if another transaction with a higher fee is broadcast before it makes it into a block.  We are arguing that this unquestionably how miners will operate in the future, anyway, so we change the default now to prevent any further adaptation to a  false sense of security/reality.  i.e. users currently experience transactions not being replaced so readily, so they build around an assumption that zero-conf are okay for zero-trust situations.  They're not. 

I agree this is not inline with the original intention of the software, but I don't think there's a way around it.  Miners will do this eventually, even if most of them aren't, yet.  Let's not pretend they won't.

Plus, this patch comes with the nice benefit that "stuck" tx can be easily fixed.  Though, I do agree that a non-deterministic "undo" button is a bad idea.  It would be unreliable in many ways, and would also give a false sense of reality to users.  Instead, I'd like to see a button/dialog that says "This transaction has been waiting more than one hour without being accepted.  Would you like to increase the fee to try to speed up its acceptance?" 

Good questions/observations.

I had meant to make the font configurable, but forgot to implement it.

I also did not think to draw a border around the QR codes, but I totally should have.  I basically just create a QR matrix and draw black squares for every cell that is 1.  I guess I should lay down a white square that is (N+2)x(N+2) before drawing the squares.  Good call!  Is one of your apps failing to read a QR code?  Or is it just aesthetics?  Are you using a dark background theme?

But you might have to remind me of these things later.  I have a few other priorities at the moment  

The blk*.dat files stored in the Bitcoin home directory are quite simple -- just a straight concatenation of blocks of the form:


The "BlockSizeBytes" field includes the BlockHeader and NumTx, so you only have to read that and skip that many bytes if you want to jump to the next block.  This is is also how blocks are serialized peer-to-peer. 

Yes, print-to-PDF works to save the paper backup digitally.  It's built-in to most Linux distros (no need for a specific app, just install "cups-pdf" package).  Just keep in mind that you need to make a couple backups if you save it digitally like this.  One of the most useful benefits of a paper backup is that it will last decades.  The same cannot be said for USB keys, CDs, DVDs, etc, which only have a "shelf-life" of a few years (they can last longer, but not guaranteed by any means).  And just about anything that would destroy paper, would destroy the digital media, too.  Also, you cannot verify visually whether the digital backup is still working.  i.e. If you keep it in a safe-deposit box, how do you know that your USB key still works?  Take it home and check it once a year?  Sounds inconvenient.   On the other hand, you can simply look at your paper backup and know that it's good.  Even if it was severely faded after 50 years, it would still work.

This is why I highly recommend a paper backup.  The 144 characters is inconvenient to type in, but it's only a couple minute's work and should only be needed ... once or twice in your life.  Obviously, you are more comfortable with digital backups.  Just make sure you protect yourself by making a couple copies.  

You just completely ignored the entire discussion in this thread.  You don't get to decide what transactions get to go in a block.  I don't, no one else does, either.  Only the miners get to decide for themselves what transactions go in the block.  And if it involves more money, they will do it eventually (though they generally are not doing it, yet).  Because they can do so while operating completely within the rules of the system.  You are welcome to add your own mining power to the system and refuse to replace transactions with your share of power.  But you don't get to tell others what they get to do with their share.

As the number of confirmations goes to infinity, transactions are irreversible.  But zero-confirmation transactions are, by definition, not confirmed, and thus there is no such thing as "reversing them."  They were never confirmed to begin with, and should never have been trusted at all until they had confirmations.   (Note:  this applies only to trustless transactions:  zero-conf can still be useful if there is a persistent relationship between the two parties, but Bitcoin was never designed to do instantaneous transactions with zero-trust)

Armory does not understand those types of keys. They just don't fit into the wallet format and armory does not know how to compute the addresses for them.  however, if you use on compressed keys, you can import them into the offline computer and the public wallet that you export will have those public keys.

I just wanted to make the point that if you are holding $5 million, you want a solution that leads to 100% chance of recovery.  The M-of-N solution is exactly that.  Do it once. Ever.  No shenanigans.  No maintenance.  No chance of failure.  But what you are proposing leaves room for things to go wrong.  No matter how diligent you are, it's going to get annoying to do anything with your funds, even once a year, if you have to revisit safe-deposit boxes, redistribute transactions to your will, or whatever.  You'll take shortcuts, you'll be in a hurry and say you'll deal with it tomorrow.  For periods of time, only 0/5, or maybe 3/5 of your money will actually be recoverable if something bad were to happen to you.  In a way, it kind of defeats the purpose.

I appreciate you are keeping an open mind.  As you can tell, I've spent a lot of time thinking about these things   And I've also spent a lot of time dealing with users who didn't bother to even make a single-sheet backup and then lost all their coins when they forgot their passphrase.  And people who chose to backup their Bitcoin-Qt wallet and then didn't realize they had to re-backup, or were too lazy to do so.  And they can't backup securely, because they have to do it often.  That's why deterministic wallets are so valuable -- anything that requires regular maintenance is going to fail.  Even if you think you can do it yourself, and that you'll do it right--others won't.  And even you, in all your diligence to do it right, may screw that up because you're in a hurry and don't have time to stop by the bank today to replace the inheritance transaction in your safe deposit box.  Etc.    But one thing is for sure:  you make paper backups, you secure them hardc0re, once.  And you never have to worry about it again.

Okay.  I've beaten this dead horse enough.  You get my point   I'm happy to walk through thought experiments with you about this, but I'm doubtful anything is going to beat the combined convenience and security of a 2-of-3, 2-of-4, 3-of-4 or 3-of-5 backup.

I believe that what you are proposing is completely unsustainable.  You can't reissue, re-distribute, and re-secure all these different transactions after every single incoming and outgoing transaction.  It's going to make your workflow terribly inconvenient, that will lead to taking shortcuts, probably missing coins, etc.  How fantastic would that be if you had a heart attack when you were trying to fix whatever mess just happened and all your existing transactions are invalid, etc.  Now, your family will spend days digging through old transactions, and trying to figure out what the heck happened, and in the end no one gets your coins.  Ever.

There is most definitely a place for multi-sig transactions, but you can't do with them what you want.  The M-of-N solution is designed for this, and it has the security that you want.  If you want to split your coins between multiple family members, make multiple wallets.  Make 2-of-3 for each.  Give the respective family member one piece, and put 1 fragment of each in your safe deposit box.  That way, whoever eventually gets access to your safe deposit box can't take the coins, except for the single family member that already has a piece.   

This is done exactly once, and never has to be done again.  It is pretty much like writing a will.  Every time your deposit a check, do you go back to your lawyer and remake your will to say "My family now gets $3,406.74" instead of "$2,722.11"?  No, you leave your "bank account" to them, and they get whatever's in there.  You can leave each person a different bank account.  Using the M-of-N solution with different wallets is like setting beneficiaries of each bank account.

If you try it your way, keep us posted.  I'd be thoroughly impressed if you succeeded, but I've wrong before   However, I suspect that the maintenance hassle is just too extreme for any "regular" user to want to use this service, when the M-of-N solution is so comparable and so much easier.

Also, there is no way to pre-sign a transaction like you suggested, unless you plan to never touch your money again.  If you make a single transaction after you have pre-signed the "inheritance" transaction, that inheritance transaction will become invalid, and you will have to reissue it.  This makes it very sensitive to failure (something happens before you've had a chance to reissue and re-distribute it).

The solution is the M-of-N backups.  Even if you did a 7-of-30 backup, someone who has only 6 pieces might as well have 0.  They have no brute-force advantage over someone who has 0 pieces (except of course, they only need one more piece).  That's the nice thing about Shamir's Secret Sharing: you have 0% access until you have 100%.  Giving your family one piece gives them nothing, except for requiring them to find one less piece than anyone else (which they will, when they get to your safe-deposit box).

Just to echo TierNolan,

I'm working on the M-of-N shared secret stuff now.  It is exactly what you want.  And if you have a little bit of command-line patience, you can use it right now, and I have promised to support it forever (I have multiple wallets personally backed up with the current utility).  You can read about it here.

The fact that it is is "split" is what prevents "attacks of opportunity", such as someone breaking into your house and stealing your jewelry box which happened to contain your paper backup.  Or a bank employee with too much access rooting safe-deposit boxes for jewelry and finding your backup.  In the M-of-N case, they would have to target you to compromise your wallet.

Thank you for promoting responsible Bitcoin savings practices.  I've always pushed people against brainwallets for exactly the reasons you describe.

I have considered an upgrade that will reduce it to half the amount of data.  It's extremely simple in concept, and still secure.  But I really don't want to mess with it if it isn't broken.

However, when I'm done with the persistent blockchain stuff, and I get back to the new wallets, those wallets will have about 1/3 the amount of data (160-bit seeds instead of 2x256-bit key&chaincode).

The fragmented backups, will also be the same size as the single-sig backups, with only about 5 extra bytes.  The first version of the script I made doubled the amount of data, but I later realized I missed an important optimization that has no effect on security.  So that means that future fragmented backups of current wallets will only be 4 lines plus a couple bytes, and future wallets that use only 160 bits will be only two, shorter lines. 

I agree that the current backups are not "pleasant" to type in, but they aren't terrible either.  And you typically won't be using the backup... ever... or maybe twice a year for various reasons.  So convenience is not worth any risk to brekaing something that works already.

There's not good documentation about it.  It's a mathematical trick, made possible by elliptic curve math, that requires a little bit of math background to really understand. 

It allows you to create one one private key, compute that one public key from it, and then give them both the same random number (chaincode).  From that, you can produce a [near-]infinite string of private keys on the offline computer, and the [near-]infinite string of public keys on the online computer, knowing that they are the public keys associated with the offline computer private keys.  The online computer only needs the root public key and chaincode to do the computation.

This won't be much of an issue.  I started reworking the paper backup page, and added PrinterSafe™ which encrypts the backup that is printed, and then has you write down the 64-bit encryption key ON the paper backup.  This encryption key is generated with the same keystretching as the wallet encryption, which means that, even though 64-bits doesn't seem like a lot, it will take the attacker like 0.25 sec and 8MB per guess. 

This should be sufficient for keeping your printer in the dark.

I woudln't mind a slightly better name... PrinterMask, PrinterSafe... PrintSafe?

Unfortunately, no.  I had thoroughly tested the wallet file and it was in use by lots of people, before I'd even heard of compressed public keys.  I didn't want to take any risks with the wallet file to support them.  Instead, I'm making a new wallet file format that supports a whole host of new features, and compressed public keys are one of them.  Though, that has been put on hold while I deal with the resource usage.   Hopefully the new wallets will become "real" in a month or two.

Sure.  I assume jackjack will be making some python blackboxes, one for each version.  I saw your other thread about this, and it does sound like a good match.  You can easily invoke the python directly.  Alternatively, I have all the ECDSA operations wrapped up nicely in the C++ code which is wrapped up in the python.  You could actually access the C++ code directly in a main.cpp file, but you do need a little bit extra to get it working.  Just a thought...

Thanks for doing research before posting!  It's great to see people jump in and find previous issues so I don't always have to repeat it.

I used to get about 1 wallet-corruption complaint every 4-6 months... now I'm getting about 2 per month.    I wonder if the wallet code was somehow affected.  Or may have just have 10x the users I had before! 

I've intentionally tried not to mess with the wallet code, because it is thoroughly tested and everyone is happy.  Maybe it's corruption-resistence isn't as good as I thought it was...?  Granted, I've crashed Armory and endless number of times, tested these features endlessly, and never got corruption.  Maybe it's more likely to happen on Windows? 

Maybe I should be putting up scary in-app warnings about backing up your wallet BEFORE you use it for anything.  I have some warnings that pop up periodically after you've created it, but it doesn't help if you somehow corrupt your wallet between creation and making that backup.

Glad to hear that it worked out in the end.  Now that you have a paper backup, if that ever happens, you can just delete the wallet and restore from paper backup.  That was my intention to begine with anyway

All addresses you ever generate in your wallet are all derived from that root key and chaincode.  Change address is chosen by just picking the next unused address in the sequence.  For instance, if you have received Bitcoins to addresses 1-10, then you send a transaction it will use address 11 as the change address, then address 12 will be the next one to pop up when you click "Receive Bitcoins".