Alright since it seems that you have rsa-key.der, you don't need chmod or an editor...
This rsa-key.der file is a binary file, it should not be directly opened in an editor. But the fact that Windows recognizes it as a security certificate, I have to look it up.
So it turns out that DER files are just a binary encoding of a certificate (
https://wiki.openssl.org/index.php/DER). DER files are not supposed to have arbitrary bytes, such as the hex input we are trying to create a private key from.
That means the second command, the one with openssl, is not creating a private key from random bytes at all. It looks like it's extracting the private key from a DER certificate, and my input was apparently a certificate encoded in binary, not the random bytes I thought it was.
Certificates, DER, these have nothing to do with our problem. I tried finding the openssl command that uses our binary input as entropy for an RSA private key, but the best it can do is
openssl genrsa -out <OUTFILE> -rand <INFILE> <N>, OUTFILE being the private key, INFILE being our binary numbers file, and N is 2048 or some other key size. This reads bytes from INFILE to seed it's random number generator, but it still collects entropy used in the private key from the operating system. I don't think this is what you are looking for because you want a deterministic way to generate private keys from bytes.
But this stack overflow answer looks promising:
https://stackoverflow.com/questions/19850283/how-to-generate-rsa-keys-using-specific-input-numbers-in-opensslWe basically have to use some of the bytes to create primes, exponents and coefficients which is the syntax of an RSA private key.
(Note that generating these things by hand is a completely insecure way to make private keys if the numbers happen to be easy to factor, but at this point we are doing this for demonstration purposes only.)
I will study it out later today.