AI-generated post discussion thread: how to identify & report

[nutildah]

Seems AI-assisted posting is becoming more rampant, and we should work together to brainstorm some concrete criteria for identification of AI-written posts.

For me, the best indicator if a post is AI-generated is if it is written in flawless English. Even the best posters who are native English speakers don't necessarily write every post using the precision in punctuation, spelling & grammar executed by ChatGPT. If a user's native language isn't English (as can be evidenced by their posting history), and they suddenly start posting in flawless English, it means one of two things:

1. The text is plagiarized, or
2. The text is AI-generated.

Its worth noting that not all plagiarized texts are written with the same mastery of English as ChatGPT. In fact, they seldom are.

The second best indicator is taking an average of separate AI text identification systems; the more, the better. This is the one created by the makers of ChatGPT and is thus probably the best for identifying such text:

https://platform.openai.com/ai-text-classifier

Here are 6 other ones.

https://writer.com/ai-content-detector/ (max. 1500 chars)
https://gptzero.me/ (min. 250 chars)
https://contentatscale.ai/ai-content-detector/ (min. 25 words)
https://x.writefull.com/gpt-detector (50-2000 words)
https://hivemoderation.com/ai-generated-content-detection (max. 8192 chars)
https://paraphrasingtool.ai/ai-content-detector/ (max. 4900 chars)

Seems like too much work to run a text through all 7, and some probably use extremely similar methods.

Of course, they are not always accurate, but if the majority of them say "likely" or "over 50% likely", then I'd say its compelling evidence the text is AI generated.

For example, this post - which I'm almost certain is AI-generated - comes back as "unlikely" to be AI-generated according to OpenAI's Text Classifier, yet it is "99% likely" AI generated according to the Hive Moderation checker.

Here is how you ask ChatGPT to generate such a list:

Prompt:

Can you write me a list of bullet points about Bitcoin, with each one starting with a letter of the alphabet? For example:

"A- All your crypto journey could begin here and set you on the right path if you ask questions and follow up question and threads of your interest

B- Bitcoin should be your first coin of choice when you're mentally ready to enter into the market because it's one of the best coins and generally accepted one.

C- consistency is the easiest if not the only way to achieve everything you desire in everything you do and it's very evident over here in the forum."

etc.

It generated an extremely similar list which comes back as "unclear" according to the OpenAI Text Classifier and "99% likely to contain AI generated text" according to the Hive Moderation site.


The problem is people can now be lazier than ever on the forum and get paid for it, which actually hurts them in the long run. Instead of thinking or learning anything for themselves, they just plug in a prompt, and prompt-plugging is the only actual skill they are gathering. Has nothing to do with Bitcoin, cryptocurrency, the blockchain, etc.


TL;DR - What are the rock-solid criteria for detection & definition of AI-generated text?


I suggest we put examples of what are believed to be AI-written posts in this thread to help evaluate them and come up with a refined consensus on how to detect them.

[cryptoaddictchie]

I suggest we put examples of what are believed to be AI-written posts in this thread to help evaluate them and come up with a refined consensus on how to detect them.

Thats good idea but maybe we should keep it as lenient as possible or dont put names within the suspected since w arent sure yet if the content or written post are AI generated. It could be worse if we accuse someone of doing this and later on proven his innocent. Maybe a rule to what grounds a post can be submitted here. Should the post itself must be quoted including a member's name, or his/her plain post only.

[Rikafip]

TL;DR - What are the rock-solid criteria for detection & definition of AI-generated text?

I personally avoid using those detection tools because of false positive and (at least for now) I think that the sudden increase in English level is probably the best piece of evidence that we can provide to prove someone is using chatbot.

Its probably harder to catch those brand new account that use chatbots right from the start, but so far I had some success with those as well.

For example, this post (https://bitcointalk.org/index.php?topic=5424029.msg62038551#msg62038551) - which I'm almost certain is AI-generated - comes back as "unlikely" to be AI-generated according to OpenAI's Text Classifier, yet it is "99% likely" AI generated according to the Hive Moderation checker.

AI generated text for sure. 4 days ago he managed to wrote that and then yesterday he wrote the post below. There's simply no plausible explanation for such a drastic decrease of his English.

As you are saying that increasing number winning prizes will for sure increase number for your chances to increase your winning chances but I will say that this gambling algorithm don't work that way. As when there will be more winners than there might be more traffic and it would be more difficult for us win prize and probability of winning could increase of decrease by a lot of factors.
And also I think that we should predict on each side if you know that probability of your winning I'd more than 50% and it will increase your chances of winning too. I don't like those games that offer only 50% ratio as that's not good for us.

[joker_josue]

Of course, they are not always accurate, but if the majority of them say "likely" or "over 50% likely", then I'd say its compelling evidence the text is AI generated.

Written with the help of: Google Translate
The idea itself is good in general and we have to be aware of the posts that arise that may become suspicious.

But this statement is not the best. I remember reading recently that LoyceV did a test of its own text and those detectors said it was AI text. Therefore, these systems saying that the text has a 50% or more probability of being written by an AI, could be 100% wrong.

Another point is that not everyone is fluent in English or has the ability to write in English, and can therefore use translation and writing help systems. For example, to write here in the forum in English, I use Google Translate and sometimes I use the Grammarly tool to try to improve my text. The probability of being considered a text written by AI is high.

That's why I wrote this post, using 4 methods: write only using Google Translate; write with ChatGPT doing the translation; write only using Google Translate with Grammarly help; write with ChatGPT translation and Grammarly help. Do your analysis.




Written with the help of: Google Translate & Grammarly
The idea itself is good in general and we have to be aware of the posts that arise that may become suspicious.

But this statement is not the best. I recently read that LoyceV did a test of its text and those detectors said it was AI text. Therefore, these systems saying that the text has a 50% or more probability of being written by an AI, could be 100% wrong.

Another point is that not everyone is fluent in English or can write in English, and can therefore use translation and writing help systems. For example, to write here in the forum in English, I use Google Translate and sometimes the Grammarly tool to try to improve my text. The probability of being considered a text written by AI is high.

That's why I wrote this post, using 4 methods: write only using Google Translate; write with ChatGPT doing the translation; write only using Google Translate with Grammarly help; write with ChatGPT translation and Grammarly help. Do your analysis.




Written with the help of: ChatGPT
The idea itself is generally good and we need to be attentive to posts that may become suspicious.

However, this statement is not the best. I remember reading recently that LoyceV tested their own text and those detectors said it was AI-generated. Therefore, these systems claiming that a text has a 50% or higher probability of being written by an AI may be 100% wrong.

Another point is that not everyone is fluent in English or has the ability to write in English and may therefore use translation and writing assistance systems. For example, I use Google Translate to write here on the forum in English and sometimes use the Grammarly tool to try to improve my text. The probability of being considered an AI-generated text is high.

That's why I wrote this post, using four methods: writing only using Google Translate; writing with ChatGPT doing the translation; writing only using Google Translate with Grammarly's help; and writing with ChatGPT's translation and Grammarly's help. Make your own analysis.




Written with the help of: ChatGPT & Grammarly
The idea itself is generally good and we need to be attentive to posts that may become suspicious.

However, this statement is not the best. I recently read that LoyceV tested their text and those detectors said it was AI-generated. Therefore, these systems claim that a text that has a 50% or higher probability of being written by an AI maybe 100% wrong.

Another point is that not everyone is fluent in English or can write in English and may therefore use translation and writing assistance systems. For example, I use Google Translate to write here on the forum in English and sometimes use the Grammarly tool to try to improve my text. The probability of being considered an AI-generated text is high.

That's why I wrote this post, using four methods: writing only using Google Translate; writing with ChatGPT doing the translation; writing only using Google Translate with Grammarly's help; and writing with ChatGPT's translation and Grammarly's help. Make your analysis.




In Portuguese

Code:

A ideia em si é boa de um modo geral e temos de estar atentos aos posts que vão surgindo que se podem tornar suspeitos.

Mas esta afirmação não é a melhor. Lembro-me de ter lido recentemente que o LoyceV fez um teste do seu próprio texto e esses detetores dizerem que era texto de IA. Por isso, esses sistemas dizerem que o texto tem 50% ou mais de probabilidade ter sido escrito por um IA, pode estar 100% enganado.

Outro ponto, é que nem todos são fluentes em Inglês ou tem capacidade de escrever em inglês, e podem por isso usar sistemas de tradução e de ajuda na escrita. Eu por exemplo, para escrever aqui no fórum em Inglês, uso o Google Tradutor e as vezes uso a ferramenta Grammarly para tentar melhor o meu texto. A probabilidade de ser considerado um texto escrito por IA é alta.

Por isso escrevi esse post, usando 4 métodos: escrever apenas usando o Google Tradutor; escrever com o ChatGPT a fazer a tradução; escrever apenas usando o Google Tradutor com ajuda Grammarly; escrever com a tradução do ChatGPT e ajuda do Grammarly. Façam a vossa analise.

[nutildah]

Thats good idea but maybe we should keep it as lenient as possible or dont put names within the suspected since w arent sure yet if the content or written post are AI generated. It could be worse if we accuse someone of doing this and later on proven his innocent.

That's true. From now on we'll just post links with no names. With this exception: if you report a post for being "AI spam" and the report gets marked as Good, then share the archived version of the post here so we can get an idea of what "AI spam" looks like according to the mods.

I personally avoid using those detection tools because of false positive and (at least for now) I think that the sudden increase in English level is probably the best piece of evidence that we can provide to prove someone is using chatbot.

That's also true. What I have so far is:

1) sudden shift to perfect English is the #1 indicator
2) detection tools can provide additional evidence if the majority of them indicate "probable AI," or >50% likelihood.

Of course running them through all 7 is too time consuming. Perhaps in the future we can narrow it down as we gain a better understanding of their accuracy.

However, this statement is not the best. I remember reading recently that LoyceV tested their own text and those detectors said it was AI-generated. Therefore, these systems claiming that a text has a 50% or higher probability of being written by an AI may be 100% wrong.

When I have a bit more time I will put LoyceV's text through all 7 of the detectors.

[rat03gopoh]

TL;DR - What are the rock-solid criteria for detection & definition of AI-generated text?

I suggest we put examples of what are believed to be AI-written posts in this thread to help evaluate them and come up with a refined consensus on how to detect them.

I think I know the criteria for AI generated posts. For some reason I suspect posts that contains this sentence: "it's important to note/remember that..."
That's like saying in my mind, it's a part of the disclaimer that must always be included in every fenerated answers.

Look https://ninjastic.space/search?content=it%27s%20important%20to%20note%20that
Lately, this sentence has been used a lot by newbie accounts.

[NeuroticFish]

I do like the idea, but think that in most cases it will not good enough for using beyond doubt.
Of course, there are exceptions:
* sudden change in the quality (and fluence) of the user's posts
* sudden change in how knowledgeable somebody is in various fields

And here there can be a doubt too between use of AI or account changing hands, for example, however both should come at least with red tag.


... Maybe this new development would also trigger equivalating the gravity of plagiarism, use of AI and buying accounts, to make everybody's life easier.

----
I like rat03gopoh logic, but that will probably not work on long term (AI will start writing even more human, the users will learn to edit such keywords, ...) and I'm not sure it could be proven (although I totally agree it's unnatural to talk here like that).

[lovesmayfamilis]

I would add a moment to how quickly posts written by AI are published. Sometimes you can see quite long posts being posted in a short amount of time when the average person would not be able to write as fast.

From now on we'll just post links with no names. With this exception: if you report a post for being "AI spam" and the report gets marked as Good, then share the archived version of the post here so we can get an idea of what "AI spam" looks like according to the mods.

And yes, I have several examples of good reports where the moderators marked the reports as good.


https://loyce.club/archive/posts/6203/62036246.html
https://loyce.club/archive/posts/6203/62036255.html
https://loyce.club/archive/posts/6203/62036260.html
https://loyce.club/archive/posts/6199/61993288.html

https://loyce.club/archive/posts/6188/61885663.html
https://loyce.club/archive/posts/6188/61885806.html
https://loyce.club/archive/posts/6188/61885793.html
https://loyce.club/archive/posts/6188/61885849.html
https://loyce.club/archive/posts/6188/61885810.html
https://loyce.club/archive/posts/6188/61885733.html


https://ninjastic.space/search?author=Cathy_57&board=34

[Stalker22]

For me, the best indicator if a post is AI-generated is if it is written in flawless English. Even the best posters who are native English speakers don't necessarily write every post using the precision in punctuation, spelling & grammar executed by ChatGPT. If a user's native language isn't English (as can be evidenced by their posting history), and they suddenly start posting in flawless English, it means one of two things:

1. The text is plagiarized, or
2. The text is AI-generated.

It is not fair to assume that someone who writes in flawless English is necessarily plagiarizing or using AI-generated text, and I disagree with this assumption because I believe there could be a third possibility.

In my opinion, @joker_josue makes a valid point. It is difficult to differentiate between content generated by ChatGPT and text that has been translated from a non-English native language to English or edited using ChatGPT and similar tools. I do not think there is a clear way to distinguish between the two, and I suspect that most AI Content Detectors would classify the text as AI-generated, even if it is not the case.

[joker_josue]

In my opinion, @joker_josue makes a valid point. It is difficult to differentiate between content generated by ChatGPT and text that has been translated from a non-English native language to English or edited using ChatGPT and similar tools. I do not think there is a clear way to distinguish between the two, and I suspect that most AI Content Detectors would classify the text as AI-generated, even if it is not the case.

In the example of my post, when using one of the recommended detectors, it indicates the 4 options written by human. What is correct.
But I found it interesting that in the first 3 options he was 97-99% sure he was human. While in the last option (Written with the help of: ChatGPT & Grammarly) it identified as 100% written by human.

That is, if the person uses the right tools, with half a dozen clicks and one or another adjustment, the detectors are no longer effective.

Using an automatic system to detect text written by AI will only be effective if that AI uses a standard element. Thing, that doesn't happen. Now, you can use terms or expressions that normally a human, in a written conversation, is not used. And this can be an element that draws attention, but even so, you have to be careful about classifying it as done by AI or not.

[Adbitco]

AI detecting tools are tools created by an ordinary person which are not certain to be entirely correct while detecting, there are every possibility of scanning through a post and it would results as AI generated post.
For instance, I use myself as an example i don't usually like using Abbreviation while posting and i make sure words are correctly spelled, why?
Because i know too well this site (Forum) is a reference site and is open for everyone to source information, so i tries all my best to make sure i limits the rates of using those unlikely terms and words. Although i know too well that there are some addicts who loves using shorter words or abbreviations to what they are writing, but all text and words being correctly spelled doesn't really guaranteed someone using AI to generate  those words because there are no facts to detect is 100 percent correct.

Otherwise i can say, if a post is created and immediately has a reply without even a seconds after posting then it could be rest assured that such person is using AI to write in my opinion. Please i am not in any way countering your words or arguing words with you but the fact remains uncertain, if there were other ways then that would be more better to detect if someone is a Bot.

[NotATether]

Should I turn JARVIS into a shooting range where I make it randomly alternate between human posts and AI generated posts so that you guys have have a target to practice your methods on? (Just don't get it banned, that's all )

I will use a deterministic random number generator using the post number as the seed to determine whether I should make it post an AI or human post.

[joker_josue]

~~

I wanted to do some more tests, based on the example I made.


Written with the help of: ChatGPT, using other words
The overall idea is commendable and warrants careful monitoring of potential suspect posts.

However, the aforementioned statement is suboptimal. Recently, I recall LoyceV testing their own text and those detectors marking it as AI-generated. Consequently, these systems asserting a 50% or greater probability of AI-generated text may be 100% mistaken.

Furthermore, not all possess fluency in English nor possess the capacity to write in English, hence they may utilize translation and writing assistance systems. Personally, I use Google Translate to compose posts on this forum in English and sometimes employ the Grammarly tool to refine my writing. The chance of being perceived as AI-generated text is significantly elevated.

Thus, I have composed this post employing four methods: writing solely with Google Translate, composing with ChatGPT translation, utilizing Google Translate and Grammarly, and using ChatGPT and Grammarly in tandem. Please assess accordingly.


Written with the help of: ChatGPT, in just 200 words
The idea is good, but suspicious posts must be closely monitored. However, some systems claiming a 50% or higher chance of AI-generated text could be wrong. Recently, LoyceV's self-tested text was marked as AI-generated by detectors, highlighting the issue. Additionally, not everyone is fluent in English or has the ability to write well in English. Some may use translation and writing assistance systems like Google Translate or Grammarly. This raises the likelihood of being perceived as AI-generated text. To demonstrate this point, I wrote this post using four methods: Google Translate only, ChatGPT translation, Google Translate with Grammarly, and ChatGPT with Grammarly. It is important to analyze posts carefully, especially in the case of suspect text.


In both cases the detector says it was written 100% by humans.

[elevates]

Every AI needs a prompt. It is easy to master it after playing with prompts on most AI. I do not think those who use AI to reply or create topics are so naive. gptzero according to me still good to detect an AI. I have used it to detect many submission. Almost everytime it cam up with a correct analysis.

For me, the best indicator if a post is AI-generated is if it is written in flawless English. Even the best posters who are native English speakers don't necessarily write every post using the precision in punctuation, spelling & grammar executed by ChatGPT. If a user's native language isn't English (as can be evidenced by their posting history), and they suddenly start posting in flawless English, it means one of two things:

I also agree that an AI generated response has flawless English. Still, there are many ways to disguise flawless English if you know how to write a prompt. Even there are a few platforms that can help you with it for free. Even Google is finding is difficult to detect AI written articles.

I would say the best way to recognise if an article is written by an AI is to see whether it uses pronoun and also noun in it. As we humans tend to use it more often for communicating with each other. This where a communication between humans become to and fro.

[NotATether]

Every AI needs a prompt. It is easy to master it after playing with prompts on most AI. I do not think those who use AI to reply or create topics are so naive. gptzero according to me still good to detect an AI was used to create a topic or reply.

Not only that, but there's even starting to be job offers for "ChatGPT Prompt Engineer" , as if that was a tech job

[nutildah]

For the sake of research, let's look at the LoyceV post in all 7 detectors. Here is the snippet he analyzed that came back 97% likely to be AI according to detector.dng.ai:

Freedom!
Welcome to Bitcointalk! I'll give you a very short introduction of what makes Bitcointalk unique. First: note that scams are not moderated! If something looks too good to be true, you are going to get scammed. Use common sense, and don't trust random strangers on the internet with your money.
Bitcointalk offers more freedom than any other forum I know. You can choose to be an asshole or you can be a nice person. It's entirely up to you how you want to come across. If you're a snowflake, you might get offended. That's okay.
You're allowed to earn money or do business here. Your freedom is mainly limited by the forum rules.
Warning: unlike most websites, Bitcointalk does not allow plagiarism. This will get you banned! It's very easy to prevent: if you quote someone, give them the credit they deserve.

1. OpenAI Text Classifier: (doesn't meet minimum character count)
2. Writer.com: 98% human-generated content
3. GPTZero.me: Your text may include parts written by AI
4. Contentscale.ai: 89% - Highly likely to be Human!
5. Writefull.com: 4% likely this comes from GPT-3, GPT-4 or ChatGPT.
6. Hivemoderation.com: 0% - not likely to contain AI Generated Text
7. Paraphrasingtool.ai: Text may contain AI-written sentences

So based on this singular sample, I would say detectors 2, 4, 5 and 6 are the most accurate, while the one used by LoyceV seems to be terribly wrong. All the ones that offer percentages (except for detector.dng.ai) returned pretty accurate results.


Now let's take a look at a post that we already believe is likely to be written by AI, assuming you agree with rat03gopoh's logic about posts with the phrase "It's important to note..." in them:

There are many reasons why traders may prefer to trade in the Bitcoin market over Forex.
First, the bitcoin market is not regulated, which means that its prices and trading opportunities can change much faster than in regulated markets. This may provide more opportunities for profit, but it may also involve more risk.
Secondly, the bitcoin market is still relatively young and rapidly developing, which may attract traders looking for new opportunities and risks.
Third, the bitcoin market is more decentralized than the forex market, which may appeal to those looking for a freer, more independent market.
Finally, it's important to note that not all bitcoin exchanges are "shady" or "unsafe", and many of them provide a high quality service to their clients. However, as with any market, there are risks associated with Bitcoin trading and traders should be prepared for this risk and have a risk management plan in place.

1. OpenAI Text Classifier: (doesn't meet minimum character count)
2. Writer.com: 53% human-generated content
3. GPTZero.me: Your text is likely to be written entirely by AI
4. Contentscale.ai: 26% - Highly likely to be AI generated!
5. Writefull.com: 54% likely this comes from GPT-3, GPT-4 or ChatGPT.
6. Hivemoderation.com: 99.9% - The input is likely to contain AI Generated Text
7. Paraphrasingtool.ai: It's likely that AI wrote this text

So this time around, detectors 3, 4, 5, 6 and 7 all gave the expected answer (though #5 at 54% isn't exactly "convincing"). So based on a sample size of 2, detectors 4, 5 and 6 are thus far the most accurate, with Hivemoderation.com being the most accurate. Will need to do more such tests before coming to any conclusions.

One thing for certain is detector.dng.ai shouldn't be used. It didn't even make the cut in this article which is where I found these.

[WillyAp]

Seems AI-assisted posting is becoming more rampant, and we should work together to brainstorm some concrete criteria for identification of AI-written posts.

bitcointalk needs to find ways to stop in its tracks those texts from pretrained software. Reason being that search engines are not very fond of these texts which translates into less visibility.

There are ways to enhance good posts. I know of a forum who gives 1 cent for each sensitive post.
He uses USDT for that.

[cryptoaddictchie]

There are ways to enhance good posts. I know of a forum who gives 1 cent for each sensitive post.
He uses USDT for that.

Isnt it through practicing daily and based on personal experience? Also what you mean by gives 1cent per each post? Isnt it a normal way to have discussion on normal ways? Plus AI generated topics copied are probably noticeable than self improve posting of someone.

[WillyAp]

Isnt it through practicing daily and based on personal experience? Also what you mean by gives 1cent per each post? Isnt it a normal way to have discussion on normal ways? Plus AI generated topics copied are probably noticeable than self improve posting of someone.

It is used also as a payment for services. Its a spanish spoken forum from Mexico. Yes, I have used parts of the balance. It is limited to 30 Post though.

[jokers10]

...
In both cases the detector says it was written 100% by humans.

If someone gave AI good enough criteria and AI made a post which should look like written by human we probably can't find out if it is not written by human. But usually those ones who've been caught on highly likely using AI for posting are not so thorough, the texts they post look unnatural, written in common words and having not much sense. Of course some real users write text the same way by themselves... but should we really care of the texts which have low quality and look like made by AI?