But how about all the other coins stores in all the other addresses no one else knows about, of if they do, it's only one or two people. And unless I sign a message using the private key, there is no proof that that particular address belongs to me.
...
You can guess that a person might control or own an address, but you can never prove it.
In short, a wallet will combine addresses if it needs to. Thus, if you know that the address of one output in a transaction belongs to a person, you know that the addresses of the other outputs also belong to that person.