MysteryHex.py 1.0: Figure out unidentified binary/hex data!

[etotheipi]

Introducing MysteryHex v1.0

Ever had a chunk of hex or binary from the BTC network/files that you wanted help identifying?  Well, here's a script that does it for you!   Will find all main-network block headers, tx's, hashes, public keys, addresses, scripts and common binary strings (magic numbers, verack, addr, etc).     Clone my git repo at https://github.com/etotheipi/PyBtcEngine , or just download the two files: pybtcengine.py and mysteryHex.py.  Here's a simple example:

Code:

./mysteryHex.py 04fc8ad7af7e5bbc66d1bac090855d9d4eedc59a38bba2847e3ee23a1d3d4270ed71f5596409ab7c9601482671216e381fd506be2cccbb2350f44cc3aaa2bd0fbb012345f9beb4d9
$ 

Found:

   0x0000:  AAAAAAAA AAAAAAAA AAAAAAAA AAAAAAAA AAAAAAAA AAAAAAAA AAAAAAAA AAAAAAAA 
   0x0020:  AAAAAAAA AAAAAAAA AAAAAAAA AAAAAAAA AAAAAAAA AAAAAAAA AAAAAAAA AAAAAAAA 
   0x0040:  AA------ BBBBBBBB 

   A: BarePublicKey    : 1HdjH5Rgnsrn5PPWAQDWwmU2cxk5huXqm2
   B: MagicNum         : Main network magic bytes (f9beb4d9)

If nothing else, you can use this tool convert public key strings into the associated address.  But, that's kind of boring... let's look at something more complex.  Download dataIDTest.hex from my repo:

Code:

$ ./mysteryHex.py -s -f dataIDTest.hex 

####################################################################################################

Found:  BlockHeader
Size: 80 bytes
Bytes: 4 to 84  (0x00000004 to 0x00000054)

   0x0000:  -------- 01000000 55bd840a 78798ad0 da853f68 974f3d18 3e2bd1db 6a842c1f 
   0x0020:  eecf222a 00000000 ff104ccb 05421ab9 3e63f8c3 ce5c2c2e 9dbb37de 2764b3a3 
   0x0040:  175c8166 562cac7d 51b96a49 ffff001d 283e9e70 -------- -------- -------- 
   0x0060:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x0080:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x00a0:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x00c0:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x00e0:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x0100:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x0120:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x0140:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x0160:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x0180:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x01a0:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x01c0:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x01e0:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x0200:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x0220:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x0240:  -------- -------- -------- -------- -------- -------- ------ 

   BlockHeader:
      Hash:       00000000d1145790a8694403d4063f323d499e655c83426834d4ce2f8dd4a2ee (BE)
      Version:    1
      PrevBlock:  000000002a22cfee1f2c846adbd12b3e183d4f97683f85dad08a79780a84bd55 (BE)
      MerkRoot:   7dac2c5666815c17a3b36427de37bb9d2e2c5ccec3f8633eb91a4205cb4c10ff (BE)
      Timestamp:  1231731025
      Difficulty: ffff001d
      Nonce:      1889418792
####################################################################################################
Found:  Transaction
Size: 134 bytes
Bytes: 150 to 284  (0x00000096 to 0x0000011c)

   0x0000:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x0020:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x0040:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x0060:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x0080:  -------- -------- -------- -------- -------- ----0100 00000100 00000000 
   0x00a0:  00000000 00000000 00000000 00000000 00000000 00000000 000000ff ffffff07 
   0x00c0:  04ffff00 1d0102ff ffffff01 00f2052a 01000000 434104d4 6c4968bd e02899d2 
   0x00e0:  aa096336 7c7a6ce3 4eec332b 32e42e5f 3407e052 d64ac625 da6f0718 e7b30214 
   0x0100:  0434bd72 5706957c 092db538 05b821a8 5b23a7ac 61725bac 00000000 -------- 
   0x0120:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x0140:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x0160:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x0180:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x01a0:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x01c0:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x01e0:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x0200:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x0220:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x0240:  -------- -------- -------- -------- -------- -------- ------ 

   Transaction:
      TxHash:    82501c1178fa0b222c1f3d474ec726b832013f0a532b44bb620cce8624a5feb1
      Version:   1
      nInputs:   1
      nOutputs:  1
      LockTime:  0
      Inputs: 
         TxIn:
            OutPoint:
               PrevTxHash: 0000000000000000000000000000000000000000000000000000000000000000 (BE)
               TxOutIndex: 4294967295
            Script:  (SignatureForCoinbaseTx)
            Seq:     4294967295
      Outputs: 
         TxOut:
            Value:    5000000000 ( 50.0 )
            Script:   PubKey(1PSSGeFHDnKNxiEyFrD1wcEaHr9hrQDDWc) OP_CHECKSIG
####################################################################################################
Found:  Transaction
Size: 275 bytes
Bytes: 284 to 559  (0x0000011c to 0x0000022f)

   0x0000:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x0020:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x0040:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x0060:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x0080:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x00a0:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x00c0:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x00e0:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x0100:  -------- -------- -------- -------- -------- -------- -------- 01000000 
   0x0120:  01c997a5 e56e1041 02fa209c 6a852dd9 0660a20b 2d9c3524 23edce25 857fcd37 
   0x0140:  04000000 00484730 4402204e 45e16932 b8af5149 61a1d3a1 a25fdf3f 4f7732e9 
   0x0160:  d624c6c6 1548ab5f b8cd4102 20181522 ec8eca07 de4860a4 acdd1290 9d831cc5 
   0x0180:  6cbbac46 22082221 a8768d1d 0901ffff ffff0200 ca9a3b00 00000043 4104ae1a 
   0x01a0:  62fe09c5 f51b1390 5f07f06b 99a2f715 9b2225f3 74cd378d 71302fa2 8414e7aa 
   0x01c0:  b37397f5 54a7df5f 142c21c1 b7303b8a 0626f1ba ded5c72a 704f7e6c d84cac00 
   0x01e0:  286bee00 00000043 410411db 93e1dcdb 8a016b49 840f8c53 bc1eb68a 382e97b1 
   0x0200:  482ecad7 b148a690 9a5cb2e0 eaddfb84 ccf97444 64f82e16 0bfa9b8b 64f9d4c0 
   0x0220:  3f999b86 43f656b4 12a3ac00 000000-- -------- -------- -------- -------- 
   0x0240:  -------- -------- -------- -------- -------- -------- ------ 

   Transaction:
      TxHash:    169e1e83e930853391bc6f35f605c6754cfead57cf8387639d3b4096c54f18f4
      Version:   1
      nInputs:   1
      nOutputs:  2
      LockTime:  0
      Inputs: 
         TxIn:
            OutPoint:
               PrevTxHash: 0437cd7f8525ceed2324359c2d0ba26006d92d856a9c20fa0241106ee5a597c9 (BE)
               TxOutIndex: 0
            Script:  (SignatureForCoinbaseTx)
            Seq:     4294967295
      Outputs: 
         TxOut:
            Value:    1000000000 ( 10.0 )
            Script:   PubKey(1Q2TWHE3GMdB6BZKafqwxXtWAWgFt5Jvm3) OP_CHECKSIG
         TxOut:
            Value:    4000000000 ( 40.0 )
            Script:   PubKey(12cbQLTFMXRnSzktFkuoG3eHoMeFtpTu3S) OP_CHECKSIG
####################################################################################################
Other assorted things:

   0x0000:  AAAAAAAA -------- -------- -------- -------- -------- -------- -------- 
   0x0020:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x0040:  -------- -------- -------- -------- -------- BBBBBBBB BBBBBBBB BBBBBBBB 
   0x0060:  BBBBBBBB BBBBBBBB BBBBBBBB BBBBBBBB BBBBBBBB BBBBBBBB BBBBBBBB BBBBBBBB 
   0x0080:  BBBBBBBB BBBBBBBB BBBBBBBB BBBBBBBB BBBBBBBB BB------ -------- -------- 
   0x00a0:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x00c0:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x00e0:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x0100:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x0120:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x0140:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x0160:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x0180:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x01a0:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x01c0:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x01e0:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x0200:  -------- -------- -------- -------- -------- -------- -------- -------- 
   0x0220:  -------- -------- -------- -------- -------- -------- ------CC CCCCCCCC 
   0x0240:  CCCCCCCC CCCCCCCC CCCCCCCC CCCCCCCC CCCCCCCC CCCCCCCC CCCCCC 

   A: MagicNum         : Main network magic bytes (f9beb4d9)
   B: BarePublicKey    : 15VihAPHSVKRyCbWLBbysJnXhmpta8MBJJ
   C: HeaderHash(BE)   : 000000000000043dd94eb922fbf4e21eedf05f7af1ec5477470955c523bfdd6c

Notice on the last line that a "HeaderHash" was flagged as identified data.  This is because the script will scan your blk0001.dat file in your bitcoin directory, and construct an up-to-the-minute dictionary of known header/merkle/tx hashes to use in the search.  This option (-s flag) is disabled by default because it will take about 4 minutes to process the 1.5 million transactions in blk0001.dat the first time it is done.  However, it will remember its progress (stores everything in ./knownHashes.bin) and only updates based on new blocks your client has received since you last ran the script.  There is no danger in doing this while the client is open, and will update hashlist nearly instantly on all future executions.  Use the -s flag on every call if you want this functionality.

Known issues:

• Automatic BTC directory detection has been implemented for Windows, Linux and Mac.  It works for me in Linux, but I have not tested it in Windows or Mac.  Please let me know if it doesn't work.
• Testnet addresses can be handled by the library, but it's not implemented in a way easily accessed by the script.  You must change the "USE_TESTNET" variable in pybtcengine.py and then use '-r -k /path/to/testnet/blk0001.dat' to rescan the testnet blockchain for hashes.  I will work on a more-flexible solution in the near future.
  
• To maximize usefulness, I have set the library to print all hashes in big-endian, so that the hex strings can be copied directly into blockexplorer.com.  This may not be the preferred behavior for everyone.  In the future I will add a flag that lets you select your endianness. (actually, I'd much prefer to just always print in little-endian, and lobby block-explorer to allow searching with LE strings)

In progress:

• Search unidentified bytes for private keys that match public keys found
• Enable a -t flag to work with test-network
• Partial hash searches

I've spent a lot of time polishing this code, so it should be pretty robust.  Please let me know (via PM) if you find any bugs, or post here if you have suggestions for improving the script.  And of course, please donate!  Putting this together was not easy!  1Gffm7LKXcNFPrtxy6yF4JBoe5rVka4sn1

-Eto

[jackjack]

I don't have the time to test it now but I sure will torture your tool

Automatic detection of bitcoin directory works in linux, but fails in other OS's.  If -s flag is used but blk0001.dat cannot be found, script will continue without populating/updating the hash file.  Use -k /path/to/blk0001.dat to help the script find it on other OS's.  Please let me know how I can support automatic detection on other OS's.  

Did you look at pywallet code? It should work pretty well. Actually I didn't check on something else than Linux but as it's one of the Joric's functions I think he should have tested it

Code:

def determine_db_dir():
	import os
	import os.path
	import platform
	if platform.system() == "Darwin":
		return os.path.expanduser("~/Library/Application Support/Bitcoin/")
	elif platform.system() == "Windows":
		return os.path.join(os.environ['APPDATA'], "Bitcoin")
	return os.path.expanduser("~/.bitcoin")

[etotheipi]

Thanks Jackjack.  I used the os.getenv('APPDATA') call for Windows instead of os.getenv('HOME') and now it looks like windows auto-detection works. 

Also, I updated the code to properly calculate non-main-network addresses.  I had written the code before I totally understood the algorithm and had written it in a way that only worked for main-network.  I re-ran all my unit tests and confirmed that test-network addresses are calculated correctly.

If you want to use it for the test network, you will have to flip the "USE_TESTNET" var in pybtcengine, and then use

Code:

$ ./mysteryHex.py -r -k /path/to/testnet/blk0001.dat -f somefile.hex

which will erase your current knownHashes.bin file and rescan the blockdata from the testnet blockfile.  I haven't tested it, but I don't see why it wouldn't work.