What makes you think that they are worried? When Warren Buffet or Bill Gates attack Bitcoin, it's not necessarily because they are afraid of it. They are influential people and they share their thoughts on global events, especially if someone asks them. And if they criticize Bitcoin, it's not automatically because they are rich. There are also rich people who defend it, or poor people who don't like it.
We can't really generalize to everyone. Rich people spend more on investment, crypto-currencies is one of their options. Some are already accepted but others hesitate to buy. Even Elon Musk promotes doge coins.