Thanks for answering my questions.
Viginere string encryption has been used since the 18th centry it is basically a caesar encryption but a lot more secure, I wrote my own implementation in the Delphi language and was featured on a large Delphi web site called about.delphi.com, The reason for "Custom" algo is to reduce the risk of cracking, Most "crackers" are very fimilar with the common knowen algo's and they them self have created tools which can crack these algo's, So to me using common everyday algo's makes it more of chance to get cracked rather than a custom algo, which mearly a combination of algo's put together.
Your cipher might act as slight obfuscation, but I wouldn't rely on it. Make sure that the security of your program does not depend on it.
For number 2 question... It is very possiable to use multiple iterations and substitutions to streach the key length(reducing the chance of repeated key being generated),Like I stated in first post I have only just started this project for public use and at the momment relys on Vigenere encryption to prevent collisions, which in real life is not even issue considering the chances of a MD5 collision(very rare),
Yes, MD5 collisions are rare and I believe MD5 is still usable for key derivation. I just feel uncomfortable depending on such a weakened hash when we have plenty of better alternatives.
It also has the option of SHA1 incase people are aware of the MD5 collision issues, It can also use Both to create a hash of the file which would really stretch out the hash and key length. We could also implement any variations of SHA if people wanted including the most common SHA256 thats used in the bitcoin community.
I think SHA-256 would be preferred, although SHA1 is reasonable.
I have never read anything about RC4 being difficult to use securely. After all it only encryption a string(Wallet.dat coverted to string and then encrypted) once the string is created it is then appended to the created file. The begining of that created file is like a header which will contain information like the custom hash of the decrypted wallet(original wallet.dat).
Improper use of RC4 made WEP Wi-Fi encryption trivial to break. RC4 has a few weaknesses you have to watch for, which I will explain further down. They are documented here:http://aboba.drizzlehosting.com/IEEE/rc4_ksaproc.pdfhttp://en.wikipedia.org/wiki/Fluhrer,_Mantin_and_Shamir_attack
for you last 2 questions I think that is explained in the above explaination, no bytes are dropped from the RC4 stream(or maybe I have mistake your question regarding this)?
As described in the paper above, the initial bytes produced in RC4's keystream can leak information about the key. The first bytes in the keystream should be dropped prior to encryption. This is just one of the little things that makes RC4 difficult to implement securely.
Ohh and the RC$ implemetation I am using is from friend who converted it from C++ and to include the changes to encrypt as a string and not a stream. Its a very simple implementation and can be found on my blog or I can post it here if you wanted to see it.
Implementing encryption securely is very hard. You have to worry about the weird nuances of the algorithms and various side-channel attacks. I don't really trust an implementation unless it has been widely used, tested, and reviewed by many professionals. Even prominant open-source implementations created by experts sometimes turn out to have issues. I'd recommend using a library like OpenSSL. It has been highly scrutenized and it will handle many of the little issues for you.
The one question you didn't answer is the nonce question. RC4 is a stream cipher and should never be used to encrypt two different things with the same key unless you use a unique nonce or IV which is combined with the key prior to encryption. This is yet another thing that a crypto library could help with.