Every investor counts and holds its own role. The impact of those who buy or sell big amounts is indeed greater than those who play in points but still if a lot of small investors will sell out, it is going to leave same impression on market value of the coin as that done by a big whale. Moreover, it is not only whales or investors that are only factor. The market depends on government movements too
There is even a theory that there are no whales (only big holders who are not trading)
And thus there are no market manipulations either, so all price movements are only caused by changes in aggregate demand as well as supply. Personally, I'm not very much sold on this theory, but that's not my point here. The main problem with the current market is that even relatively small investors and traders can move the market on their own (i.e. not collectively). In this way, they are the new whales, with truly big whales hiding deep under the surface
Anyone that has large amount of money to move in and out of the market at the shortest possible time is to me termed as a whale whether they are individual or group of people, so whales actually do exist and they started too when the wall street entered the market, it was from there that the manipulation of this bitcoin started
It is only a theory, and I do not necessarily endorse it
Also, it is not every time like you said we see all forms of dip and rise of bitcoin as manipulation, some are definitely caused by the activities of people which sometime depend on the way they place demand and supply for these coins
And this is where things get tricky
It should be obvious that everyone in this market, in fact in any market for that matter, is pursuing their own interests, which consist in earning dough, as simple as it gets. Another obvious thing is that cryptocurrency markets are essentially a zero-sum game, i.e. there is only so much money, and if you win, someone else necessarily loses. And the bigger your win is going to be, the more people will have to lose to provide for that profit
In this way, if someone, someone whom we can rightfully call a whale, uses his financial leverage to his advantage, and does his job of earning money pretty well, which is only possible if it is someone else's money (since it is a zero-sum game), it means that many people have to lose their coins and cents. Can we then call that an instance of market manipulation as this is how all these people will inevitably feel (kind of robbed)?