Well you have to separate the JVM (a just in time compiler and class library), from Java the language, from Java the failed attempt to sandbox downloadable code.
The JVM is not very interesting, it's just a big runtime library. There are techniques for making it much smaller. For instance Avian (
http://oss.readytalk.com/avian/) shows how to make GUI apps that compile down to standalone executables that are less than a megabyte in size, no dependency on having Java installed. Unfortunately it can be a pain to compile on modern MacOS versions, but obviously the binaries work.
If you work in any kind of modern language you'll end up needing a big runtime library anyway. This is true of apps written in Python or using Qt as well.
Java the language is mediocre at best. It's main redeeming feature is that it's simple and old so everyone knows it. But you don't have to use Java to access libraries written in it. You can write code in JavaScript, Ruby, Python (2.7), Lisp, and a lot of more modern languages like Kotlin, Scala or Ceylon that have lots of interesting features and type systems.
Then there's the failed attempt to do sandboxing (applets). Well, basically every attempt at this has gone badly wrong. Web browsers still routinely have serious exploits in their rendering engines, that's why Chrome sandboxes the sandbox! But for writing regular apps this doesn't really matter to you - applets can be completely disabled and it makes no difference.
So if you wrote your own wallet on top of bitcoinj, what would it mean?
- You could write your code in lots of modern languages (but not C/C++/Objective-C without binding work).
- Your app can be standalone and look/feel like a normal app. See JWrapper for a tool that makes native installers for each platform.
- The end user never needs to know that there's any Java under the hood.