The answer I was looking for was that there is more processing power required by the higher difficulty level and if that stays the same, it takes longer for the bounty to be paid out.
The difficulty determines the probability of "solving" a block. Your hash rate determines how many solutions you can try over a period of time. If the difficulty goes up and your hash rate stays the same, then the number of times you solve a block goes down. The result is that you make less money over that period of time.