Bitcoin Forum
December 10, 2016, 07:24:51 AM *
News: To be able to use the next phase of the beta forum software, please ensure that your email address is correct/functional.
 
   Home   Help Search Donate Login Register  
Pages: « 1 [2]  All
  Print  
Author Topic: fuck you wallet format!!!! (RANT)  (Read 2830 times)
splatster
Full Member
***
Offline Offline

Activity: 175



View Profile
May 06, 2012, 02:35:16 AM
 #21

I think the fact that it is a database in the first place is a silly design decision.
I couldn't agree more.

Why can't we use a simple format?

S² Capital Management | #bitcoin-otc ratings | 1M5j2g4iz4mSwkngrYkqtcmKNGmyDAQzk2
1481354691
Hero Member
*
Offline Offline

Posts: 1481354691

View Profile Personal Message (Offline)

Ignore
1481354691
Reply with quote  #2

1481354691
Report to moderator
1481354691
Hero Member
*
Offline Offline

Posts: 1481354691

View Profile Personal Message (Offline)

Ignore
1481354691
Reply with quote  #2

1481354691
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1481354691
Hero Member
*
Offline Offline

Posts: 1481354691

View Profile Personal Message (Offline)

Ignore
1481354691
Reply with quote  #2

1481354691
Report to moderator
1481354691
Hero Member
*
Offline Offline

Posts: 1481354691

View Profile Personal Message (Offline)

Ignore
1481354691
Reply with quote  #2

1481354691
Report to moderator
etotheipi
Legendary
*
expert
Offline Offline

Activity: 1428


Core Armory Developer


View Profile WWW
May 06, 2012, 02:48:25 AM
 #22

I think the fact that it is a database in the first place is a silly design decision.
I couldn't agree more.

Why can't we use a simple format?

It's because the wallet is sensitive stuff, and one of the benefits of a database engine is the ACID/atomic operations:  it guarantees that data is written as intended or not written at all  to the database.  No matter what nanosecond the power goes out, it's supposed to be impervious to corruption.   When you're talking about private keys protecting millions of dollars, it's a very good idea to have atomic operations... but it comes with the downside that the database is a kind of blackbox and you don't always know what it's doing (hence the 0.4.0 wallet-not-actually-encrypted bug).

In my wallet format, I created atomic operations using a backup file and some flag files that detect when corruption has happened and to be able to detect and restore an uncorrupted version automatically.  It's probably only 90-95% as effective as a real ACID/atomic database, but I've tested the heck out of it and it does work.   In the end, it's pretty rare that this logic would even be triggered because Armory doesn't keep the wallet file open.  It only does open-modify-close operations occasionally.

I shared this experience with the devs, and while some of them thought it was interesting, their attitude was "let's not reinvent the wheel -- this is a solved problem, let's use an existing solution."  I understand that attitude.  But to me, the simplicity of the file format with 100% control over the data is worth every ounce of effort I put into it.

Founder and CEO of Armory Technologies, Inc.
Armory Bitcoin Wallet: Bringing cold storage to the average user!
Only use Armory software signed by the Armory Offline Signing Key (0x98832223)

Please donate to the Armory project by clicking here!    (or donate directly via 1QBDLYTDFHHZAABYSKGKPWKLSXZWCCJQBX -- yes, it's a real address!)
splatster
Full Member
***
Offline Offline

Activity: 175



View Profile
May 06, 2012, 02:59:45 AM
 #23

I think the fact that it is a database in the first place is a silly design decision.
I couldn't agree more.

Why can't we use a simple format?

It's because the wallet is sensitive stuff, and one of the benefits of a database engine is the ACID/atomic operations:  it guarantees that data is written as intended or not written at all  to the database.  No matter what nanosecond the power goes out, it's supposed to be impervious to corruption.   When you're talking about private keys protecting millions of dollars, it's a very good idea to have atomic operations... but it comes with the downside that the database is a kind of blackbox and you don't always know what it's doing (hence the 0.4.0 wallet-not-actually-encrypted bug).

In my wallet format, I created atomic operations using a backup file and some flag files that detect when corruption has happened and to be able to detect and restore an uncorrupted version automatically.  It's probably only 90-95% as effective as a real ACID/atomic database, but I've tested the heck out of it and it does work.   In the end, it's pretty rare that this logic would even be triggered because Armory doesn't keep the wallet file open.  It only does open-modify-close operations occasionally.

I shared this experience with the devs, and while some of them thought it was interesting, their attitude was "let's not reinvent the wheel -- this is a solved problem, let's use an existing solution."  I understand that attitude.  But to me, the simplicity of the file format with 100% control over the data is worth every ounce of effort I put into it.

So you have a solution which combines the safety and the simplicity of each option, yet it hasn't been implemented into the mainstream client?  I might have to make armory my main wallet if something similar never hits the main client.

S² Capital Management | #bitcoin-otc ratings | 1M5j2g4iz4mSwkngrYkqtcmKNGmyDAQzk2
etotheipi
Legendary
*
expert
Offline Offline

Activity: 1428


Core Armory Developer


View Profile WWW
May 06, 2012, 03:10:07 AM
 #24

So you have a solution which combines the safety and the simplicity of each option, yet it hasn't been implemented into the mainstream client?  I might have to make armory my main wallet if something similar never hits the main client.

Bear in mind that my solution is not a perfect replacement for ACID operations.  It's only a "good" solution (as far as I can tell).  However, I think it's a lot less important with deterministic wallets.  Would-be wallet corruption is so rare to begin with.  Now take into account that 90% of that will be recoverable.  Now take into account that your wallet is completely recoverable from your very first paper or digital backup*.

And most users will make a first backup.  The issue with the Satoshi client is that it requires a persistent backup solution, and end-users are very bad at that.

There's some semi-frequent discussion on IRC about what the devs would like to do with the wallet, but I haven't heard any consensus.  However, I do know that they will be implementing deterministic wallets soon, too.  So regardless of disk corruption/failure protections, the Satoshi wallets may have the same only-one-time-backup-needed property.  But of course, it will probably be a while before they get around to it...

*Exception: this isn't true if you imported keys after making the first backup. 


Founder and CEO of Armory Technologies, Inc.
Armory Bitcoin Wallet: Bringing cold storage to the average user!
Only use Armory software signed by the Armory Offline Signing Key (0x98832223)

Please donate to the Armory project by clicking here!    (or donate directly via 1QBDLYTDFHHZAABYSKGKPWKLSXZWCCJQBX -- yes, it's a real address!)
casascius
Mike Caldwell
VIP
Legendary
*
Offline Offline

Activity: 1344


The Casascius 1oz 10BTC Silver Round (w/ Gold B)


View Profile WWW
May 06, 2012, 11:46:04 AM
 #25

It's because the wallet is sensitive stuff, and one of the benefits of a database engine is the ACID/atomic operations:  it guarantees that data is written as intended or not written at all  to the database.  No matter what nanosecond the power goes out, it's supposed to be impervious to corruption.   When you're talking about private keys protecting millions of dollars, it's a very good idea to have atomic operations... but it comes with the downside that the database is a kind of blackbox and you don't always know what it's doing (hence the 0.4.0 wallet-not-actually-encrypted bug).

The ability of a database engine to provide atomic operations is no stronger than the underlying OS's ability to reliably report that all writes to a certain point have been flushed to disk when asked - using operations that anybody can call from any application, not just a database engine.

The magic that brings an atomic operation to a database is nothing more complicated than replacing "write a record" with the following flowchart:

1. Write a record somewhere that says you intend to make a particular write, including details of the substance of the write, so the write can be repeated if this is the only record of it.
2. Ensure that that record is committed to disk before continuing.
3. Make the write as intended.
4. Make sure the write done in step 3 is committed to disk before continuing.
5. Eliminate the record you created in step 1.
6. Ensure that the elimination done in step 5 has completed before allowing step 1 to occur on a future write.
7. When your program starts up, have it so that it looks for any unremoved records similar to the one created in step 1.  Confirm that they were written completely.  If so, simply perform the write operation that the record says you planned to make (which will have no effect if the prior write was successful).  If such records were not written completely, discard them.

This is simple enough for computer science students to implement in their homework.

The only magic that a database engine brings to the table is the ability for these seven steps to run at a high level of performance with lots of concurrent operations, in an effort to mitigate the performance penalty of tripling the burden of doing writes.

Since a Bitcoin wallet is only updated eternities apart (in terms of compute time, especially when it is limited only to data created or changed by the user), the perceived performance penalty of doing 3 writes instead of 1 ought to be so negligible as to make a full blown database engine completely unnecessary even when ACID properties are desirable.


Companies claiming they got hacked and lost your coins sounds like fraud so perfect it could be called fashionable.  I never believe them.  If I ever experience the misfortune of a real intrusion, I declare I have been honest about the way I have managed the keys in Casascius Coins.  I maintain no ability to recover or reproduce the keys, not even under limitless duress or total intrusion.  Remember that trusting strangers with your coins without any recourse is, as a matter of principle, not a best practice.  Don't keep coins online. Use paper wallets instead.
Remember remember the 5th of November
Legendary
*
Offline Offline

Activity: 1526

Reverse engineer from time to time


View Profile
May 06, 2012, 12:01:02 PM
 #26

Thread title made me laugh. A user in YT going by the name TheRadBrad often says such phrases. He is funny Cheesy

BTC:1AiCRMxgf1ptVQwx6hDuKMu4f7F27QmJC2
etotheipi
Legendary
*
expert
Offline Offline

Activity: 1428


Core Armory Developer


View Profile WWW
May 06, 2012, 12:56:43 PM
 #27

It's because the wallet is sensitive stuff, and one of the benefits of a database engine is the ACID/atomic operations:  it guarantees that data is written as intended or not written at all  to the database.  No matter what nanosecond the power goes out, it's supposed to be impervious to corruption.   When you're talking about private keys protecting millions of dollars, it's a very good idea to have atomic operations... but it comes with the downside that the database is a kind of blackbox and you don't always know what it's doing (hence the 0.4.0 wallet-not-actually-encrypted bug).

The ability of a database engine to provide atomic operations is no stronger than the underlying OS's ability to reliably report that all writes to a certain point have been flushed to disk when asked - using operations that anybody can call from any application, not just a database engine.

The magic that brings an atomic operation to a database is nothing more complicated than replacing "write a record" with the following flowchart:

1. Write a record somewhere that says you intend to make a particular write, including details of the substance of the write, so the write can be repeated if this is the only record of it.
2. Ensure that that record is committed to disk before continuing.
3. Make the write as intended.
4. Make sure the write done in step 3 is committed to disk before continuing.
5. Eliminate the record you created in step 1.
6. Ensure that the elimination done in step 5 has completed before allowing step 1 to occur on a future write.
7. When your program starts up, have it so that it looks for any unremoved records similar to the one created in step 1.  Confirm that they were written completely.  If so, simply perform the write operation that the record says you planned to make (which will have no effect if the prior write was successful).  If such records were not written completely, discard them.

This is simple enough for computer science students to implement in their homework.

The only magic that a database engine brings to the table is the ability for these seven steps to run at a high level of performance with lots of concurrent operations, in an effort to mitigate the performance penalty of tripling the burden of doing writes.

Since a Bitcoin wallet is only updated eternities apart (in terms of compute time, especially when it is limited only to data created or changed by the user), the perceived performance penalty of doing 3 writes instead of 1 ought to be so negligible as to make a full blown database engine completely unnecessary even when ACID properties are desirable.

Well this is exactly what I do in my wallet, except with an extra step of repeating 1-5 on a backup immediately after the main file is updated.  I do this so that if the file gets corrupted I have a guaranteed working backup, and the flag files tell me which one is the corrupted one...

HOWEVER: one of the criticisms of this technique (which I would think equally applies to any app trying to do atomic operations) is that you don't have control over when data actually gets written to disk.  And there's no guarantee that the writes happen in the same order that you issued them. 


Founder and CEO of Armory Technologies, Inc.
Armory Bitcoin Wallet: Bringing cold storage to the average user!
Only use Armory software signed by the Armory Offline Signing Key (0x98832223)

Please donate to the Armory project by clicking here!    (or donate directly via 1QBDLYTDFHHZAABYSKGKPWKLSXZWCCJQBX -- yes, it's a real address!)
Mike Hearn
Legendary
*
expert
Offline Offline

Activity: 1526


View Profile
May 06, 2012, 06:41:49 PM
 #28

I'm not sure the performance impact of maintaining key/hash160 -> tx indexes is really worth having a simpler wallet format. People who feel really strongly about this can adjust the code to build such an index and measure the impact. Block chain download/processing is already expensive enough, IMHO.
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
May 06, 2012, 07:34:44 PM
 #29

    What do people think about other wallet formats?

    For reference, I went as far in the opposite direction as I could, when creating the Armory wallet format.  I hate the Satoshi wallet format as much as kokjo.  Armory uses a simple binary format, easy to read, and only two operations on it are ever used:  append, or overwrite-in-place-with-same-data-size.   I documented it here: 

    http://bitcoinarmory.com/index.php/armory-wallet-files

    I had two goals in mind when I made the wallet format:

    • I want 100% control of what happens in the wallet file.  Inspired by the wallet-not-actually-encrypted bug in 0.4.0
    • I want it to be dead simple for other developers to be able to read (and maybe modify) the wallet files

    There's quite a bit of extra wallet-management code to protect against corruption & errors, and enforce atomic operations, but that's in code -- it doesn't affect the simplicity for other developers to read the files.    The most important feature is that when I encrypt my wallet, the encrypted key is guaranteed to overwrite the original unencrypted key, which prevents any leaks happening when I back it up to Dropbox, etc.  Same with deleting data:  it's overwritten with zeros in-place.  I know the overwrite may not happen in-place on-disk, but there's nothing I can do about that -- at least when someone copies the wallet file from my HDD, the binary file will not have any surprises in it.
    [/list]

    Your format seems pretty good. Where would I be able to find out more about the error-correcting checksums. You say they can fix up to one byte. Sounds good enough to me but what do I know? Would people say that's the only error correction needed?

    Bitcoin Extra Wallet | Peercoin Android Wallet
    BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
    etotheipi
    Legendary
    *
    expert
    Offline Offline

    Activity: 1428


    Core Armory Developer


    View Profile WWW
    May 06, 2012, 07:43:13 PM
     #30

    Your format seems pretty good. Where would I be able to find out more about the error-correcting checksums. You say they can fix up to one byte. Sounds good enough to me but what do I know? Would people say that's the only error correction needed?

    I used dumb error-correction:  it's just regular checksums as seen elsewhere in the Bitcoin protocol.  All I do is hash the field, and add the first four bytes to the end of that field.  If a byte goes bad, I just iterate through the field changing single bytes until it matches the checksum again.

    Hashing like this is not really intended for error correction, but 4 bytes is enough to do it reliably, and dead simple to use it.  And given the remarkably-low frequency of hard-drive errors, one-byte should be enough.  If more than one byte in the same checksummed field went bad, there's probably bigger problems.   

    I decided not to use something more appropriate like Reed-Solomon because I thought it would obfuscate the fields (i.e. -- 256 bytes of data has to be converted to 268 bytes of coefficients, meaning you need another library in order to read the data even if you don't care about the error correction).  I just recently found out that's not true under my circumstances, so I may consider switching to it on my next wallet upgrade. 




    Founder and CEO of Armory Technologies, Inc.
    Armory Bitcoin Wallet: Bringing cold storage to the average user!
    Only use Armory software signed by the Armory Offline Signing Key (0x98832223)

    Please donate to the Armory project by clicking here!    (or donate directly via 1QBDLYTDFHHZAABYSKGKPWKLSXZWCCJQBX -- yes, it's a real address!)
    MatthewLM
    Legendary
    *
    Offline Offline

    Activity: 1092



    View Profile WWW
    May 06, 2012, 08:08:34 PM
     #31

    There seems to be a few simple Reed Solomon algorithms available online so I don't think it would be a major problem implementing, assuming these algorithms work.

    Bitcoin Extra Wallet | Peercoin Android Wallet
    BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
    Pages: « 1 [2]  All
      Print  
     
    Jump to:  

    Sponsored by , a Bitcoin-accepting VPN.
    Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!