The difficulty that the pool gives you has nothing to do with network difficulty, it creates a barrier below which shares are not submitted - you have to make a calculation/share over your personal diff given by the pool so that it'll be submitted. It can become a nuisance if the pool gives you high diffs due to bad estimation of your hashrate, but in the end it balances around yours.
Would it be more efficient to add all of my cards to one miner in hopes of a lower collective difficulty?