I've been playing with using one worker for multiple computers and it seems to work fine, is there any drawbacks to this other than not knowing which computer is down?
No, unless you are using more than 20 miners with long polling per single worker account.
If you don't use LP then there is no limit (for deepbit pool).