If you are referring to Bard (the Google version of Chat GPT), it will have safeguards in place to prevent these types of outright scams from being generated by their product with no legitimate use case. Obviously these are not foolproof....
There will always be tools not safeguarded well enough, there will be free API or free tools one can configured for doing such malicious data. Maybe not as easy and at hand as Vod tells, but in the "right" circles I expect it'll be easy to find by the script kiddies open for fraud.
It will be difficult to protect against fraudulent emails, and really already is difficult to spot them by the content alone. A good first start would be to use a unique private email for each service or website you will potentially receive email from, which will result in you knowing if an email is from a particular company based on the email address they send the email to.
This is a very good idea and easy too to set up. And you don't even have to have your domain or knowledge.
I use for free for almost a year the services of anonaddy.com, an email redirect tool helping me to do exactly this: one email per subscription/service.
Of course, this also means that (at least in my setup) those emails are available for some more eyes, but in 99.9% of the cases this is not a problem.
Email providers will probably need to do a really good job at detecting and filtering out fraudulent emails.
Your expectations are highly unrealistic in this. Do you remember by chance how bad was the spam filtering in the first decade of spam mails?