Let's get back to the birth of BTC to make my question clearer. At the very first beginning, nobody was mining it.
Well, before the very beginning nobody was mining it.
AT the very beginning Satoshi was mining it. There isn't a beginning until someone mines the first block. Hal Finney was likely mining it as well.
And then some miners get involved, first blocks of transaction were validated.
In the case of bitcoin, the earliest blocks contained ONLY the 1 transaction that pays the miner his block reward. There were no other transactions yet.
At a point I guess, there were less than 20 miners.
Very likely true for a while. Yes.
Where was the trust in the BTC network with such a small amount of miners?
There wasn't any.
Bitcoin was considered to be an interesting experiment. Since there wasn't any trust, bitcoins didn't have any value. You could get them for free, and people sent them to others for free. Trust developed over time. As trust increased, the demand for bitcoins increased, and as a result the exchange rate increased.
They could agree together on how to store the transaction history and for example send all the BTC to one specific address.
This is not possible. Miners cannot include invalid transactions in their blocks. If they do, then all nodes, wallets, merchants, etc would reject the invalid block. Miners can only choose which VALID transactions to include in their block.
So it would be possible for colluding miners to engage in a double-spend attack, or to prevent transactions from being confirmed. However, it would not be possible to just take bitcoins from others (unless they start over with a new blockchain from the earliest block that mined any of the bitcoins that the person is holding).