Bad security advice again: shred

[bcearl]

After the bullshit advice that security noobs give here all the time about VMs and TrueCrypt, this time I will discuss shred.


Problem:
Destroying wallet.dat files with shred does not work.

Why?
That's not shred's fault. shred is just from another millenium. Modern operating systems use modern file systems, that don't store files at a fixed place any more. That has a lot of reasons that reach from performance improvement to better error correction after system crashes.
When you use shred on these filesystems (anything more modern than FAT and ext2), shred will write random data to the file - but that does not actually hit the disk at the spot where the file used to be. The original data of the file may survive that. For more details see the man page quotes below.

What does work?
shred is still useful - for example if you wipe out entire drives before creating new filesystems. If you don't want to wipe out your whole disk every time, you have to choose between a prehistoric file system or cryptography (which is not trivial).



Read the f****ing manual!

CAUTION: Note that shred relies on a very  important  assumption:  that
       the  file system overwrites data in place.  This is the traditional way
       to do things, but many modern file system designs do not  satisfy  this
       assumption.   The following are examples of file systems on which shred
       is not effective, or is not guaranteed to be effective in all file sys‐
       tem modes:

       * log-structured or journaled file systems, such as those supplied with
       AIX and Solaris (and JFS, ReiserFS, XFS, Ext3, etc.)

       * file systems that write redundant data and  carry  on  even  if  some
       writes fail, such as RAID-based file systems

       *  file  systems  that  make snapshots, such as Network Appliance's NFS
       server

       * file systems that cache in temporary locations, such as NFS version 3
       clients

       * compressed file systems

       In  the  case  of  ext3 file systems, the above disclaimer applies (and
       shred is thus of limited  effectiveness)  only  in  data=journal  mode,
       which  journals  file  data  in addition to just metadata.  In both the
       data=ordered (default) and data=writeback modes, shred works as  usual.
       Ext3  journaling  modes  can  be  changed  by adding the data=something
       option to the mount  options  for  a  particular  file  system  in  the
       /etc/fstab file, as documented in the mount man page (man mount).

       In  addition, file system backups and remote mirrors may contain copies
       of the file that cannot be removed, and that will allow a shredded file
       to be recovered later.

GNU shred manual from Ubuntu 11.04

[1714940203]

1714940203

[1.21gigawatts]

so this is good news for the guy that used the shred command on his wallet?

[kinghajj]

Well according to the manual it does work in ext4 except in data=journal mode, so most Linux users should be alright then, no?

[willphase]

The guy who shredded his wallet was tuning linux or else I'm sure someone would have already advised him that he could have just looked in his volume shadow copy of the directory (on Windows). Hint: I haven't looked at what GNU shred does on Windows but unless it's deleting these shadow copies it's not really shredding anything.

Use a truecrypt drive or wait for ekey support in the next bitcoin release.

Will

[BeeCee1]

Why?
That's not shred's fault. shred is just from another millenium. Modern operating systems use modern file systems, that don't store files at a fixed place any more. That has a lot of reasons that reach from performance improvement to better error correction after system crashes.
When you use shred on these filesystems (anything more modern than FAT and ext2), shred will write random data to the file - but that does not actually hit the disk at the spot where the file used to be. The original data of the file may survive that. For more details see the man page quotes below.

If you're using a solid state disk, even FAT or ext2 won't make shred useful.  SSD's do lots of stuff underneath the filesystem to speed things up and for wear leveling, so even if the filesystem things it is overwriting the file it probably isn't.  (On the bright side, many SSD's are agressive about reclaiming deleted blocks, so if your OS deletes it instead of moving it to a trash directory, it will get overwritten quickly.)

[Capitan]

I probably missed the thread where TrueCrypt was shown to be useless. Can you describe why? is it simply that as soon as you mount an encrypted volume its contents become vulnerable?

AND, can you  please give an overview of proper security measures one can take (without becoming a security expert)?

[JohnDoe]

       In  the  case  of  ext3 file systems, the above disclaimer applies (and
       shred is thus of limited  effectiveness)  only  in  data=journal  mode,
       which  journals  file  data  in addition to just metadata.  In both the
       data=ordered (default) and data=writeback modes, shred works as  usual.
       Ext3  journaling  modes  can  be  changed  by adding the data=something
       option to the mount  options  for  a  particular  file  system  in  the
       /etc/fstab file, as documented in the mount man page (man mount).

       In  addition, file system backups and remote mirrors may contain copies
       of the file that cannot be removed, and that will allow a shredded file
       to be recovered later.

Since data=ordered is the default then what's the problem?

Use a truecrypt drive or wait for ekey support in the next bitcoin release.

Don't know much about the new feature being worked on, will it basically make manual GPG or TrueCrypt solutions obsolete or will they still be superior?

I probably missed the thread where TrueCrypt was shown to be useless. Can you describe why? is it simply that as soon as you mount an encrypted volume its contents become vulnerable?

As far as I know, with any solution you'll have to temporarily have the wallet unencrypted when you want to use it. I think the main issue with TrueCrypt was the in-memory password storage.

http://en.wikipedia.org/wiki/Truecrypt#Passwords_stored_in_memory

[JohnDoe]

I'd be really interested to hear the security solutions from exchangers/merchants that need to maintain their wallet unencrypted all the time to send and receive payments.

[Nescio]

(On the bright side, many SSD's are agressive about reclaiming deleted blocks, so if your OS deletes it instead of moving it to a trash directory, it will get overwritten quickly.)

Not anymore. As the flash process shrinks, longevity (number of erase cycles) drops. This drives controller manufacturers to ever more lazy trim command handling (and what you are probably talking about is garbage collection, which has to be filesystem aware). The currently most popular controllers (Sandforce 12xx and 22xx series) do not have active GC and also employ compression, meaning there is a lot more actual free space so trims are executed even lazier. Active GC depends on filesystem bitmap snooping, which usually excludes most other filesystems besides NTFS.

Lazy trim handling means that when you shred LBA 500 to 1000, the controller remaps these to a different flash region from the fresh overprovisioning pool, then notes the old physical location in its internal bitmap. If your SSD is brand new, it will never delete anything until you have written all flash at least once, at which point it will clean out as much as needed for the OP pool, i.e. your data may stick around for a very long time. For wear leveling purposes it might even not erase that particular flash region until a multiple of the whole device's capacity has been written. Increase that by another factor of 2 to 3 due to compression.

Also, ATA trim commands do not propagate through most current RAID controllers. And many systems do not support trim altogether (Windows XP, MacOS 10.5 and older, Linux before 2.6.36 IIRC, Windows 7 with default (buggy PoS msahci) driver partially, it swallows large trims etc.).

The only way to erase data on an SSD with 100% certainty is issueing a secure erase ATA command, this puts 21V on the substrate, emptying all flash cells at the same time. Needless to say this will delete all data, including partitioning and MBR, and shorten the longevity of the device by one cycle (usually out of 1500 to 3000 now for 25 nm MLC).

Using encryption on your wallet is pointless in this scenario unless no unencrypted version of it is ever written, including as part of swap or a hibernation file.

The bottom line is that if you are paranoid, don't use an SSD

[Chick]

so this is good news for the guy that used the shred command on his wallet?

As far as I'm concerned, this is the only reason why I opened up this thread. 

[bcearl]

Well according to the manual it does work in ext4 except in data=journal mode, so most Linux users should be alright then, no?

No. Journal is the essential feature that makes the advantage of ext3/4 over ext2. If you disable that, the only remainig difference is details like maximum file size. You have a slow old file system then.

[bcearl]

Why?
That's not shred's fault. shred is just from another millenium. Modern operating systems use modern file systems, that don't store files at a fixed place any more. That has a lot of reasons that reach from performance improvement to better error correction after system crashes.
When you use shred on these filesystems (anything more modern than FAT and ext2), shred will write random data to the file - but that does not actually hit the disk at the spot where the file used to be. The original data of the file may survive that. For more details see the man page quotes below.

If you're using a solid state disk, even FAT or ext2 won't make shred useful.  SSD's do lots of stuff underneath the filesystem to speed things up and for wear leveling, so even if the filesystem things it is overwriting the file it probably isn't.  (On the bright side, many SSD's are agressive about reclaiming deleted blocks, so if your OS deletes it instead of moving it to a trash directory, it will get overwritten quickly.)

Same is true for magnetic disks. But that a different issue, that's on the hardware level. That's only relevant if somebody gets your drive.

The problem I was talking about is that there remains data that is accessible via software.

[bcearl]

I probably missed the thread where TrueCrypt was shown to be useless. Can you describe why? is it simply that as soon as you mount an encrypted volume its contents become vulnerable?

AND, can you  please give an overview of proper security measures one can take (without becoming a security expert)?

Because people ignore that TrueCrypt is no easier than any other encryption method. You can easily setup something, but it's not trivial to ensure that it is secure.

http://forum.bitcoin.org/index.php?topic=16246.0

[bcearl]

so this is good news for the guy that used the shred command on his wallet?

As far as I'm concerned, this is the only reason why I opened up this thread. 

What? You? It was me!



But yes, that was an inspiration. Not because he destroyed his wallet, but because people seem to have told him that deleting with shred is all you need.

[bcearl]

       In  the  case  of  ext3 file systems, the above disclaimer applies (and
       shred is thus of limited  effectiveness)  only  in  data=journal  mode,
       which  journals  file  data  in addition to just metadata.  In both the
       data=ordered (default) and data=writeback modes, shred works as  usual.
       Ext3  journaling  modes  can  be  changed  by adding the data=something
       option to the mount  options  for  a  particular  file  system  in  the
       /etc/fstab file, as documented in the mount man page (man mount).

       In  addition, file system backups and remote mirrors may contain copies
       of the file that cannot be removed, and that will allow a shredded file
       to be recovered later.

Since data=ordered is the default then what's the problem?

It's the default of the file system creation tools. It not the default of any serious linux distro.

On todays drive dimensions that would make file system checks after violent system shutdowns take decades.

[BeeCee1]

       In  the  case  of  ext3 file systems, the above disclaimer applies (and
       shred is thus of limited  effectiveness)  only  in  data=journal  mode,
       which  journals  file  data  in addition to just metadata.  In both the
       data=ordered (default) and data=writeback modes, shred works as  usual.
       Ext3  journaling  modes  can  be  changed  by adding the data=something
       option to the mount  options  for  a  particular  file  system  in  the
       /etc/fstab file, as documented in the mount man page (man mount).

       In  addition, file system backups and remote mirrors may contain copies
       of the file that cannot be removed, and that will allow a shredded file
       to be recovered later.

Since data=ordered is the default then what's the problem?

It's the default of the file system creation tools. It not the default of any serious linux distro.

On todays drive dimensions that would make file system checks after violent system shutdowns take decades.

I think it is still the default.  with data=ordered, the data is written to the filesystem but the metadata is journaled, fsck only has to replay the journal.  If you use data=journal then the data, as well as the metadata, is written to the journal.

[JohnDoe]

How can I verify if I'm using data=ordered or data=journal? This is my partition on archlinux:

/dev/sda4  /home  ext4  defaults,noatime  0  2

@bcearl: I assume you don't use shred right? If so then how can you securely use GPG encryption? It sounds useless to me if I'm leaving traces behind in my disk whenever I decrypt to use my wallet.

[bcearl]

How can I verify if I'm using data=ordered or data=journal? This is my partition on archlinux:

/dev/sda4  /home  ext4  defaults,noatime  0  2

@bcearl: I assume you don't use shred right? If so then how can you securely use GPG encryption? It sounds useless to me if I'm leaving traces behind in my disk whenever I decrypt to use my wallet.

You don't want anything other than data=journal. You should not store secret information on unencrypted volumes in the first place.

[theymos]

How can I verify if I'm using data=ordered or data=journal? This is my partition on archlinux:

/dev/sda4  /home  ext4  defaults,noatime  0  2

@bcearl: I assume you don't use shred right? If so then how can you securely use GPG encryption? It sounds useless to me if I'm leaving traces behind in my disk whenever I decrypt to use my wallet.

Code:

dumpe2fs -h /dev/sda4 |grep 'Default mount options'

Using journal=data really hurts performance. It's useful mostly for home users, who have low reliability. Most other file systems (NTFS, ReiserFS, XFS, etc.) don't even support data journaling.

[LokeRundt]

I feel that all the talk of how bitcoin is for some higher-nerd class are proving correct.  I'm not the average dumbass windows user, and I have used truecrypt for a couple of years before coming across btc. . .but damn man.  It feels like I have to make a choice between using a cool awesome currency like btc (and being secure) and HAVING A LIFE