I have a loved one who is battling mild schizophrenia on his psychotherapy appointment, I happened to be there with him and he revealed that ChatGPT agreed with him that he need to leave alone, he has recently been seeking to leave alone now that ChatGPT agreed with him on that he is insisting that the professionals are wrong by telling him not to stay alone.
ChatGPT when in chat with anyone should direct them for and towards a real life professional advice, they can't keep giving out harmful suggestions and advice and it almost always what the victims chatting wants to get from them.
I don’t really know for others but to the best of my knowledge for the times I have used ChatGPT, it does not recommend medication or give medical advice because I have tried using it for that purpose but the response I get from it is that I should see a doctor for consultation or prescription. It can actually tell you about a medication and how it works but for consultation, this is strange to me.
It is also weird as to why someone will take Ai advice over their doctors advice, it is a sad news a life was lost from this effect and I believe others will learn too.
Using AI to know the name of a medicine and whether AI should be used for one's own physical condition are two completely different things. While a medicine may be beneficial for one person, it may be harmful for another. Treatment depends on various factors, such as age, physical condition, and body weight. If someone uses AI for their own physical condition, AI does not want to know their age, weight, body temperature, or whether they have any major problems. Then if AI gives them any advice, it will definitely not be good for them. Otherwise, if you seek the help of a doctor, the doctor will look at you carefully and try to know whether they have any problems, body weight, temperature, or whether they have any problems before, and then they will make a decision.
It is often seen that people give the information provided by AI the same importance as the doctor's advice, especially those who cannot go to the doctor due to lack of money. They can easily rely on AI excessively. AI can give them basic ideas or information about the time to take the medicine, but it is not the right thing to do as an alternative to a doctor.