Crypto Compression Concept Worth Big Money - I Did It!

[Jace]

Yes. Every known file in the universe is somewhere inside of Pi. (As a matter of fact, even every finite-length unknown file is located inside Pi.

This isn't even certain, it is not known whether Pi is a normal number or not.

That means every wikipedia article that will be published, description of every invention that will be invented on Earth or elsewhere, every novel, short-story or book that was, is or will be written. Incuding all variants of ending, all misspellings, etc.)

So, even the algorithm that B(asic)Miner claims to have written, is located somewhere in Pi already. Therefore it must exist. Or, ehh... wait

[rigel]

I've created a path through Pi like taking footsteps on certain numbers.  The footsteps taken are irrelevant, what is relevant are the changes between those steps.  For 0s, we hop from the first 4 in Pi to every other 4 in Pi only.  But if we encounter a 1 in our binary data, then the Hunt Value increments +1, so now we are only hopping on every 5 in Pi.  This is what keeps the record of our original bits.  All the other numbers in Pi are useless as we are encoding.  Here is an example:   001      We would be looking for the first 4 in Pi then the 2nd 4 in Pi and now we must increment +1 and hop to the next 5 in Pi.  We keep hopping on 5s as long as our data is 0s, but if we encounter another 1, we increment and begin hopping along 6s.  In this way, our pathway is totally unique to the data we are feeding into it, and thus our arrival point (end point) can let us figure out the path taken to reach itself by knowing how many 1s were used and then attempting to solve backwards to the decimal point using the precise number of steps it took to encode it, the original file size recorded in bits.

I'm not sure I got it right...

You store in your 4k file final position and path length... how can you say the path is unique?

Take this part of pi:
3.141592653589793238462643383279502884197169399375

These two sequences have same lenght and end on final 5:
00010
00001

[Jace]

But if you read his page, you will see that he has rational support behind that idea that every known file in the universe can fit inside of Pi.  Go read his arguments yourself,

I happen to have done so recently:

One of the properties that π is conjectured to have is that it is normal,

Ah, see, conjectured. This is NOT AT ALL proven to be true whatsoever. To this very day, no mathematical proof exists that states whether or not Pi is normal.

(and even if it is, the idea is still flawed for obvious reasons already described above)

[Kazimir]

B(asic)Miner, how about this: wanna bet? Walk your talk? Put your money where your mouth is? After all, money talks and bullshit walks

I give you a few 1 MB files (1048576 bytes). If you can compress them to 950 KB (972800 bytes) or less, I pay you 100 BTC. If you are unable to pull this off within a reasonable time limit, let's say one or two months from now, you pay 1 BTC to any charity of your choice (must post tx here )

Deal?

[Professor James Moriarty]

B(asic)Miner, how about this: wanna bet? Walk your talk? Put your money where your mouth is? After all, money talks and bullshit walks

I give you a few 1 MB files (1048576 bytes). If you can compress them to 950 KB (972800 bytes) or less, I pay you 100 BTC. If you are unable to pull this off within a reasonable time limit, let's say one or two months from now, you pay 1 BTC to any charity of your choice (must post tx here )

Deal?

 I thought this was possible? Aren't we able to rip games and all? I dont get it why cant he compress 1mb into 950kb ?

 This is not 'of course it can be done' message , it is more like 'I really dont know if we are able to do this or not , I tought we were able to do it but I might be wrong' type of message

[ZephramC]

Yes. Every known file in the universe is somewhere inside of Pi. (As a matter of fact, even every finite-length unknown file is located inside Pi.

This isn't even certain, it is not known whether Pi is a normal number or not.

That means every wikipedia article that will be published, description of every invention that will be invented on Earth or elsewhere, every novel, short-story or book that was, is or will be written. Incuding all variants of ending, all misspellings, etc.)

So, even the algorithm that B(asic)Miner claims to have written, is located somewhere in Pi already. Therefore it must exist. Or, ehh... wait

True. I forgot about unproven normality. So lets rephrase this and say that it is instead in sequence of truly random numbers. (Well, actually you can only prove, that every finite-lenght sequence is encountered [i random number sequence] with probability which converges to 1.)

Yes, the B(asic)Miner's algorithm is there. That does not mean it does work :-]. All algorithms exists there even non-functiong ones.


But there is interesting question... How many different computer files are there? I mean... the number of possible files is easily calculated given length limit. But how many different files truly exist up today?
So imagine central database with one indexed copy of every file. (Every time any new file is created or existing file is somehow changed in the slightest, not yet encountered way, it is stored into central database under new index.) Such set of indexes would be manageable. However, the process of upgrading the database and downloading files based on index would be not.

Another interesting question: Is it possible to create "supercompression" as non-deterministic probabilistic algorithm? That means one compressed file can produce multiple outputs but the probability of correct one is >99.999997% for long enough processing.

[ZephramC]

B(asic)Miner, how about this: wanna bet? Walk your talk? Put your money where your mouth is? After all, money talks and bullshit walks

I give you a few 1 MB files (1048576 bytes). If you can compress them to 950 KB (972800 bytes) or less, I pay you 100 BTC. If you are unable to pull this off within a reasonable time limit, let's say one or two months from now, you pay 1 BTC to any charity of your choice (must post tx here )

Deal?

 I thought this was possible? Aren't we able to rip games and all? I dont get it why cant he compress 1mb into 950kb ?

 This is not 'of course it can be done' message , it is more like 'I really dont know if we are able to do this or not , I tought we were able to do it but I might be wrong' type of message

Entropy is the answer! :-]

[bumpk1nK]

But if you read his page, you will see that he has rational support behind that idea that every known file in the universe can fit inside of Pi.

Yes. Every known file in the universe is somewhere inside of Pi. (As a matter of fact, even every finite-length unknown file is located inside Pi. That means every wikipedia article that will be published, description of every invention that will be invented on Earth or elsewhere, every novel, short-story or book that was, is or will be written. Incuding all variants of ending, all misspellings, etc.)

Your right, but finding the position where it is is the problem, and the position index will be likely even bigger (in Bytes) than the data you get (and dont start the loop - we will encode this way the positin as well  )

[Kazimir]

B(asic)Miner, how about this: wanna bet? Walk your talk? Put your money where your mouth is? After all, money talks and bullshit walks

I give you a few 1 MB files (1048576 bytes). If you can compress them to 950 KB (972800 bytes) or less, I pay you 100 BTC. If you are unable to pull this off within a reasonable time limit, let's say one or two months from now, you pay 1 BTC to any charity of your choice (must post tx here )

Deal?

 I thought this was possible? Aren't we able to rip games and all? I dont get it why cant he compress 1mb into 950kb ?

 This is not 'of course it can be done' message , it is more like 'I really dont know if we are able to do this or not , I tought we were able to do it but I might be wrong' type of message

Oh, he'll probably be able to compress some 1MB files to 950KB. But not the ones I'm supplying

See, compression is all about information density. Quite some everyday files of 1MB contain much less than 1MB of actual information. A text file, for example, or an image with lots of 'empty' spaces, will not have a very high information density. Meaning they use 1MB of disk space to store a lower amount of actual data.

But if an 1MB file actually holds (close to) 1MB of information, there's no way to represent that same information in significantly less disk space. Brilliant compression algorithm or crypto key magic or not. And to get the proper persective: the vast majority (like 99.9999%) of all 1MB files actually do hold pretty close to 1MB of information.

Either way, I have enough faith in these principles to set a 100 BTC bounty to be proven otherwise. Hope to hear from B(asic)Miner soon.

[Professor James Moriarty]

 Well , if I had 1 bitcoin to bet , I would try to get you on that bet but I dont. I would tought take you on , if I can compress one 1mb file you send me into 950kb you will pay me 1 bitcoin , if I fail I will do something of your choice that doesnt involve any sorts of money (since I cope with poverty after some bad investments :/)

[Kazimir]

Well , if I had 1 bitcoin to bet , I would try to get you on that bet but I dont. I would tought take you on , if I can compress one 1mb file you send me into 950kb you will pay me 1 bitcoin , if I fail I will do something of your choice that doesnt involve any sorts of money (since I cope with poverty after some bad investments :/)

Sure, I'll be happy to do that.

Let's be clear about the definition of 'compress' though. Either of these conditions is fine with me:

1) I provide the 1MB file in advance. Then you provide a smaller ('compressed') file in return, plus some decompression program or script or anything, which can recreate the original 1MB file. The size of the compressed file and your program or script together, may not exceed 950K.

or

2) I provide the 1MB file in advance, in encrypted form. You provide two programs or scripts or whatever. The first is able to take an input file and reduce it to 950K or less. The second is able to take the compressed file and restore the original 1MB file. These two programs or scripts can be as large as you want. May also be just one program that does both things. Then I provide the password to get access to the 1MB file I supplied earlier.

These two approaches are to avoid nonsense solutions where the data is actually just moved into the decompression program. E.g. simply cutting off the last 50K of my file and storing that in the 'decompression' program.

In both cases, things must still work if I rename the program and/or compressed file. This is to avoid nonsense solutions where data is stored in the filename.

Let me know if you want some 1MB files to test

1 BTC for every success. No action (other than the effort you put into compressing them) required in return if you fail.

[Professor James Moriarty]

 Let me put this out there , I have no technological or math skills , I was merely joking and would probably put whatever you send me int o a .rar file or search google for some methods and if i fail i fail

 So basically , we could already see where this is going , I wont win , but I will try just for the heck of it

 I do not understand your propositions , both of them . Am I allow to try to .rar whatever you send me? How much time do I have? Its like asking for a 12 year old to score a touchdown at an NFL game , there might be a sheer luck where a 12 year old may run 1 yard and score , very low chance but it worths the try for 1 bitcoin since there is nothing I could lose

 Btw that means : sure mail me some stuff and I will try to compress them , you can find my mail on my profile.

[ZephramC]

Can you compress compressed file? Can you RAR .rar file? :-]
Even 99% compression (1% reduction of file size) working for ALL files would be impossible as you could run the file through this compression multiple times.

[BurtW]

I find your idea described here:

https://bitcointalk.org/index.php?topic=288152.msg3120662#msg3120662

very interesting from a purely mathematical, theoretical point of view.

Could a very large out of band reproducible random data set (like pi) be used in this way?

Can we uniquely encode a binary input stream as a very small set of metadata in this way?

Correct me if I am wrong but your theory can be summarized as follows:

The metadata (your very small file description) consists of an end point in the number pi and number of hops it took you to reach that end point into the number pi.

Your encoding is just finding the end point and counting the number of hops.

Your decoding is finding the one unique path that ends up at that specified endpoint using that specific number of hops.

First, I must say that I do not think the general purpose decoder for anything other than trivial examples is possible by current computational means.  As an exercise it might be fun to code up a simple encoder and decoder that could be used to try this out on trivial (up to a hundred bits or so) examples.  Who knows, something interesting might come out of the exercise of trying to find an efficient way to find the path (or paths see the next point below).

Second, you would need to prove that given an endpoint and a number of hops there is only one solution (path).  I have not put too much thought into this but this looks to be a very hard thing to prove.  However, the good/bad news is that the opposite assertion:  given an endpoint and a number of hops there is not always one unique path looks to be very easy to prove.  We just need one single example where we find more than one path to an endpoint that takes the same number of hops.  The simple program described above could be used to prove this.  All you have to do is pick random input data sets, calculate the encoding and then run the decoder to find all possible solutions.  If there is more than one solution in a few cases (my expectation) then we have proved the negative assertion.

I for one am very glad you thought this up and have spent so much time and effort on it and want to personally thank you for bringing it to my attention.  It is fun to dream up new things and share them.  You could have saved yourself a lot of grief and flames by just publishing your idea in your original post but I know from personal experience that is very hard to do when you think you have really got something new and valuable.

[bit-joker]

Oh God, please just give OP a bitch-slap!

[BurtW]

Everyone:

I also know how much fun it can be to bait, flame and troll some of the losers and scammers who post here.  I think this guy is simply trying to find out if his idea will work.  I know I could be wrong and he may just be having fun or trying to pull a scam - it is so hard to tell sometimes but so far he looks genuine to me.

Carry on...

[Kazimir]

I for one am very glad you thought this up and have spent so much time and effort on it and want to personally thank you for bringing it to my attention.  It is fun to dream up new things and share them.  You could have saved yourself a lot of grief and flames by just publishing your idea in your original post but I know from personal experience that is very hard to do when you think you have really got something new and valuable.

Although I highly appreciate creativity and thinking out of the box, this whole idea is fundamentally flawed, as could have been proven in advance, regardless of the exact implementation details.

Not because of 'stubborn thinking' or refusing to look beyond known methods, but simply because it is mathematically impossible to create a compression (or 'encoding' or whatever you wanna call it) algorithm that reduces any file to less than its own size. No matter if you use Pi, or black magic, or voodoo, or whatever.

Let's repeat the same argument one more time: there are N possible files of 1MB. There are M possible 4K crypto keys. N is much, much, MUCH larger than M. So, by definition, the vast majority of 1MB files (that is, way over 99.9999999999999999999999%) can NOT be represented by a 4K crypto key. Period.

Even if you would create a HUGE (de)compression program that contains a vast library of ALL possible 1MB files, and when compressing a file it just stores some index to specify the exact 1MB file you need, it's still impossible. There's much more possible input files, than the number of unique index keys (or 'compressed files') you can output.

What riddles me, is why people who are obviously not stupid, keep believing in fairy tales and ignoring plain simple logic.

[BurtW]

That is why I called it a "very large out of band reproducible random data set (like pi)"

That is why it is an encoding/decoding system, not a compression system.

Very large amounts of information can be encoded in very small metadata:  ISBN numbers, URLs, etc.

He is not talking about data compression at all.  He just does/did not know the proper terminology is all.

[murraypaul]

Listen, if I sat down with a programmer who was able to listen to what I'm saying and not throw up objections that have no relevance to my method, and we could sit and work on this until it fits my theory exactly, then I know it would work.  You are right, I cannot do it myself, but I have been very clear about that from the very first post.  I have asked for help.  But it could be done soon.  Very very soon.  A few days for the encoding portion, it's just a file lines really.  The decoding portion would take a lot of brain work, research, testing, modification, etc ...  and we'd have to make our choices about how to resolve the solution of this software by seeing the results of those tests.  If I had the money to hire a coder myself, I would not have even come here at all.  Again, not asking for money now, but I am asking for someone to just TRY to do this with me.  How much time could it take to prove me wrong?  Maybe less than the time it's taking you to actually type out these responses to belittle me so you can go on believing everything is impossible.

Ok, to go the extra mile, I have written a toy program which does what I think you are describing:
- Set the first Pi search digit to 4
- Start at the beginning of the digits of Pi, and the beginning of an input file
- Repeatedly:
  - Take the next byte of the input file and convert it into binary
  - For each binary bit:
    - if it is is 1 increment the Pi search digit, wrapping round to 0 as necessary
    - find the next digit of Pi which matches the search digit
- Print the resulting index digit of Pi where the last binary bit of the last byte of the input file was found.

The program is extremely slow, so I'm not going to run it on a huge file.
For the first just over 1.5KB of its own source file, the program produces:

After byte 1610, pi position is 130339
After byte 1611, pi position is 130407
After byte 1612, pi position is 130468
After byte 1613, pi position is 130531
After byte 1614, pi position is 130666
After byte 1615, pi position is 130747
After byte 1616, pi position is 130857
After byte 1617, pi position is 130915
After byte 1618, pi position is 130995

I hope you can see that your 50MB file is never going to have a final index position of 500,000, if I have already reached 131,000 with only 1.5KB.

[Note that you haven't demonstrated any decompression routine, and this 'compression' routine is not proved to be reversible, so that two input files may give the same index value, which means that decompression is not possible.]

[BurtW]

OOPS, reading back through the previous posts I see this has already been shown here:

https://bitcointalk.org/index.php?topic=288152.msg3120822#msg3120822

Oh well, might as well post my work here anyway....

-----------------------------------------------------------------

Thanks for writing the program!  Cool, I was just about to do the same.

Here is a quick example showing that the paths are not unique for the idea as described.  We may not have the full description and there may be more to it than we have been told but using the start at 4, jump to next 4 as long as the input is zero and increment when the input is one idea:

The following two sequences must have different paths:

00001000
00010000

Assume what if we are in an area of pi with the following sequence of digits:

   4 ... 4 ... 4 ... 5 ... 4 ... 5 ... 5 ... 5 ... 5  (where ... means digits that do not matter in this example)

If my understanding of the idea is correct both inputs land at the same end location.

00001000 would map to 4 4 4 4 5 5 5 5
00010000 would map to 4 4 4 5 5 5 5 5

but with the totally possible distribution given we end up at the same ending location.

Is our understanding of your idea complete and correct or is there more to it than described so far?