Very hard to estimate, it depends on who you are dealing with. I've had a Debian server rooted simply because it was running a vulnerable exim package for which no fix was available at the time. I required mail receive capability, so without watching Full Disclosure 24h/day or running an IPS there's no way I could have prevented it.
If you are talking about a desktop machine and local attacks, the attack surface is huge. For example someone might send you a crafted pdf that smashes KDE/Gnome when it tries to generate a thumbnail. Someone might send you a crafted USB stick that smashes the userland file system driver or even the kernel when plugged into a port.
A headless machine that's not listening to any ports and it's not used is probably secure, even if connected to the internet. Anything else, it depends who you are dealing with and how far they are willing to go (find new exploits, compromise other devices with which you exchange physical media etc.)
BTW, I bet there are many programs listening to outside connections on your Ubuntu machine. Use this command to list them:
Anything with 0.0.0.0:[portno] or :::[portno] as local bind address is a potential remote vulnerability.