Bit-error in Block 108009, Tx 23 ?

[casascius]

Random observation from perusing the raw block chain file: for the number of times it contains messages directed to Luke-Jr stating that "god does not exist", from the view of a hex editor, the sequence "G0D" (in ascii) sure appears a god-awful lot in the block chain...

(it seems to occur each place a ScriptSig starts out with the bytes 0x30 0x44)...

by the same logic, the block chain is full of H0E's too.

[bitrick]

I've seen enough errors to believe that hardware is a perfectly reasonable possible source of error in this case.

I am fully in agreement that hardware could possibly be the cause.  Perhaps what I'm really trying to say could be summarized like this.

• Because this software is in beta, it too could be the cause, in fact this is very likely for that reason alone.  I have had it crash for no reason at all (usually it complaining about its own database files after an improper shutdown).  This could be happening to everybody and we may just not know it.
• RAID is not a solution to the specific conjecture offered.  If this were a hardware error under identical circumstances, RAID would have given no benefit, just because of what RAID is and isn't.
• Software issues that could contribute to this include the following: misuse of stray pointers, accessing freed memory, threading-related issues, buffer overruns.  Or, it could be hardware.
• A potential tool to help rule out software issues might be to distribute this blockchain verification code and have others run it.  I'd run it.  Who knows.  Maybe my copy of the block chain will have a similar kind of corruption in a different block.  If it did, then there's likely a software gremlin lurking.

The key is the nature of the corruption, which was described as a bit error. Honestly, I did not read the initial post closely enough, and thought it was a bit flipped in a hash value, but rereading I see the error is a single bit value that should be a zero across four bytes.  That could have be an _overwrite_ of a 32 bit memory location, since zero is an extremely common value and single bit values are fairly common. There is no clear connection between the two.

A hash bit error, in contrast, would increase my suspicion of hardware. Otherwise software must have read the value, flipped a specific bit (or ORd it against a mask), and wrote it back. The crypto stuff might do that but other type of software errors are unlikely to produce that effect. But that debate can be tabled until hash bit errors or other obvious bit errors are reported.

I seriously doubt that anyone would lose money as a result of this kind of problem but investing in a light background continuous validation process and some sort of reporting to the end user like a message box that provides instructions for reporting the presence of corrupted blocks. If that reveals a significant problem (beta testing would probably be enough if that is the case) then the remedy would be an error correction method to replace corrupted blocks from the network (and more instrumentation to try to isolate any software causes).

[bitrick]

I seriously doubt that anyone would lose money as a result of this kind of problem but investing in a light background continuous valdation process and some sort of reporting to the end user like a message box that provides instructions for reporting the presence of corrupted blocks...

.. might be a good idea.  (sorry forgot to finish that sentence)

[maaku]

by the same logic, the block chain is full of H0E's too.

That made me smile  

EDIT: It is possible to lose money, if the bitflip happens in the receiving address hash of the TxOut in the time between which the script is composed and the hash of the entire transaction is computed. But *is* exceedingly unlikely to occur, and could be checked for in the client anyway before broadcasting the tx. If I were running an exchange, it'd be the least of my concerns.

[nhodges]

If this were a hardware error, even a 1-in-a-trillion one, I would expect his machine to be unstable, locking up constantly, his file system growing errors on its own and constantly needing to be fsck'd/repaired/etc.  Software errors can be responsible of errors that occur with literally any magnitude of frequency, from one in a zillion all the way to constantly.

I think the experience of myself and maaku differs from yours. I had a similar belief during my first 15 years of working in the computer industry.  I did a lot of things with a lot of data and never encountered an obvious internal bit error.

However, in the last ten years I have worked more closely with hardware and I've seen enough errors to believe that hardware is a perfectly reasonable possible source of error in this case. The recent DEFCON paper on bitsquatting awakened a lot of people to the potential for hardware bit errors. Bandwidth of various transmission types, external and internal, and storage capacity has increased so dramatically, that bit errors are popping up in places that previously were so rare as to be ignorable.

I have personally diagnosed a repeatable bit error which flipped a particular bit of the destination address of a DMA transfer on a PCI bus. It only occurred once every several hours on a heavily loaded network device. It caused one connection to be dropped and there were no other obvious ill effects.

Error correction technology is a slam dunk solution for these types of problems, but it has not yet been applied to all the systems that need it.




 

Here's the paper this man is referring to: http://good.net/dl/k4r3lj/DEFCON19/DEFCON-19-Dinaburg-Bit-Squatting.pdf

As a geek I thoroughly endorse this conversation.

[ctoon6]

would putting your hardware inside a properly grounded faraday cage reduce the number of errors? alon

[maaku]

No, that would neither stop cosmic rays (too much energy), nor atmospheric neutrons (no charge). No simple solution I'm afraid

[bcforum]

would putting your hardware inside a properly grounded faraday cage reduce the number of errors? alon

Purchasing a server from a respected company instead of a desktop computer from e-machines would go a long way to reduce the number of errors. Bit rot is a common problem on desktop computers, particularly if they are overclocked or you are pushing the memory to it's limit. Just because it 'seems' stable you don't have any guarantee random memory errors aren't occurring.