Take a look at Bisq. Decentralized exchange, pretty much what you're explaining here.
The idea is very similar to Bisq itself, I found this company very nice.
Only problem of them is to use software for desktop, if take this for web and app can grow the number of users. Thanks for the answer!
Can it even be done with the current advancements in this field?
Yes you can! In this formula the danger is the malicious people, containing people who actually work in the business or accept rigid terms against fraud. It is possible to create the system.