Users != price
However, more users implies more utility for those who use it; the network effect, bitcoin is more useful to me when my business partner accepts it than when he does not.
It depends on how they are using it. If they are using it to replace paypal for sending transactions, and they only buy bitcoins right when they need them and cash out the bitcoins as soon as they get them, then it is true the number of users has a small effect on the price. But if they are using bitcoins to replace their savings accounts, then absolutely the number of users will affect the price, since each added person adds their own amount of savings to the money supply.
How long on average do you hold your bitcoins before selling/using them?