The Hidden Security Cost of "Free" AI

[pooya87]

I agree with the most part but to be fair this type of scams have been around even before AI was a thing. The scammers used various tactics to fool people into giving them their money willingly. For example through various methods they regularly convince people to install some innocent looking software (like a fun game or one of those image editing softwares) on their phone that contains something malicious. Then that malicious code used their phone to silently send messages to their contacts asking for money from their number but to the scammer's account.

[memehunter]

I can detect them within a few seconds.

Maybe you can as of now but if you look at the speed at which AI is getting better (or worse for some matters), what about couple of years later? Even today, what about your grandparents? Are they able to differentiate? Usually, scammers target previous generation for such scams.
I have a first hand experience of this. One my relative got a call from the police saying that they have arrested me for kidnapping/raping a girl and if they want to save me they have to pay bribe. On the phone call, it was even my voice (screaming for help in the back ground). Luckily they verified and the scam did not work but it could have if my phone is out of reach/batter dead etc. 
 

To actually verify identity you need multiple channels, code phrases, callback protocols;

Hmm nice idea, you can set a code phrase offline among your loved ones so that when the situation arise when scammer force you to act immediately (by a compelling fake story), you can ask for that code. Of course, you dont have to use it in your normal phone calls, and keep changing it after every incident.

[Strongkored]

I'd love to read the news about this case to get the full picture. It's quite unbelievable how $25 million could have changed hands so easily, as I believe it should have gone through several checks before the funds could be released.
However, I can't deny that AI is a disruptive technology these days, as it seems to be increasingly being used for nefarious purposes.
I remember a few weeks ago, my boss assigned me to call a lot of customers, and many of them chose not to answer my calls because there were quite a lot of news stories about phone scams. So, AI can sometimes be helpful, but it can also sometimes be a hindrance to our work.

[Shishir99]

Maybe you can as of now but if you look at the speed at which AI is getting better (or worse for some matters), what about couple of years later? Even today, what about your grandparents? Are they able to differentiate? Usually, scammers target previous generation for such scams.

Yes, I understand that AI is continually improving. However, any tech genius working online should understand the distinction between AI and real. Yes, I agree that older generations of people don't understand the difference. I see some older people come to me and show me AI-generated videos, saying, 'Look how amazing it is,' 'How terrible it is,' etc. But we are talking about a person who had a meeting online, who worked with the CEO and other bosses for a while, but wasn't able to understand that it was AI. Moreover, we didn't see any source of this news yet. I feel like this is not real.

[Faisal2202]

Who pays for this? Right now, it's individuals and small business that can't afford sophisticated security systems. The big players will adapt and transfer the costs to customers. But what about everyone else? What happens to people that can't afford the new cost of trust? Has anybody here changed the way they deal with financial verification as a result?

I have been learning AI content generation course for a long time now, making videos and images usign different models, like imagen, banana, for images and sora and veo 3.1 for videos, and used many others but liked the results of these, sora 2 has changed the world completely. I made few videos did some research from that perspective and realize why such videos are trending on social media.

Because they look surreal, especially that ring camera angles. The point is, someone in the comment sections of a videos said, these narrative (trend) will only be trending until these tools are used by some people only, but when more people start using it like every 2nd person then it will be overrated and not liked by most well that's another thing but till then many people are going to get scammed.

These are nothing, there are some uncencored ai bots, that are too deep and to be honest I love to use them but they can be mis-used. I can't agree more with you on the fact that, world is changing and now AI is not steping into malware section, so we can say the next bull run we are going to have AI security products in trend haha.

[Hispo]

That is something I have realized since I started to see news from people getting scammed because of sophisticate deep fakes being generated in real time with the help of artificial intelligence. Eventually, we will reach a point when we won't longer be able to afford to believe that we see on screens, and we will need to sacrifice time and energy to verify whether we are getting scammed or not.
This generation of young people who are tech-savvy will adapt faster then previous generations, which unfortunately will get scammed more often.

As stands for now, we don't have options but to adapt and evolve to protect our capital, to grow more skeptical about anything we see on television and on our phones.

[Josefjix]

SO keep the numbers saved in your contact and don't asnwer unless they call from that number.

What if they call from another number, not even the solution, these same group of scammers will definitely generate new numbers for more jobs, or contact more victims who will def get trapped. The best solution is not picking any strange numbers that call at all.
It is not always easy to protect oneself from scammers, they might apply next strategy and bypass new protocol to scam more victims through the same means.

[retreat]

Artificial intelligence is like a double-edged sword. It offers incredible benefits and is incredibly helpful, but at the same time, it can pose significant risks if misused. I remember a major case in my country where a student used numerous photos of women on campus and then created AI-generated videos that looked like they were performing pornographic acts. He used these AI-generated videos to sell to numerous groups and even used them to intimidate several female students into handing over money. It was truly heartbreaking that he used thousands of these videos to drain these female students mentally and financially.

This is one example of why AI can pose a serious threat if not properly regulated.

[Yaunfitda]

Who pays for this? Right now, it's individuals and small business that can't afford sophisticated security systems. The big players will adapt and transfer the costs to customers. But what about everyone else? What happens to people that can't afford the new cost of trust? Has anybody here changed the way they deal with financial verification as a result?

I have been learning AI content generation course for a long time now, making videos and images usign different models, like imagen, banana, for images and sora and veo 3.1 for videos, and used many others but liked the results of these, sora 2 has changed the world completely. I made few videos did some research from that perspective and realize why such videos are trending on social media.

I'm also trying to learn AI in the last couple of years and yeah, it's scary on who those videos are generated, like using Kling 2.0 version  or Veo 31 specially when they are released last Auguest or even higher as they are very realistic. But one advantage of us learning this kind of videos being generated, is that there could be subtle clues that we can see and then know that this is AI. But still it is very scary to see this kind of videos and they are going to improved maybe in just 6 months to be realistic as possible and very hard to what fakes and real. And so what happened here could still happened in the future and there could be scammers that are going to used this advanced technology to be able to steal huge money from unsuspecting victims.

[slapper]

Transferred 25 million dollars over the video call? For that big amount there must be a legal process involved with all the paper and documentation so it is not that easy unless someone is stupid enough and given that much money to manage.

Yeah, it sounds insane. This was a multi-national engineering firm and they had all the processes that you are talking about. The employee that got scammed knew the protocols. And they still approved fifteen separate transfers on five different bank accounts because they SAW their colleagues on video that it was legitimate.

Critically, the scam utilised exact the legal processes. The deepfakes were good enough that the employee thought they were following proper authorization procedures. They believed that they were being careful in demanding the video call.

But for the smaller amounts, it is possible, I heard a few stories that someone over the phonecall sounded exactly like a father or husband asked for an emergency need and asked to send money to an bank account but only later the person realized that they have been scammed.

SO keep the numbers saved in your contact and don't asnwer unless they call from that number.

Caller ID can be faked. Video can be faked. Even if the call is coming from the right number and you see the right face, you don't still know whether it is real.

You're not wrong smaller amounts are more common. And as the technology is becoming cheaper, what's preventing these syndicates from doing hundreds of smaller scams, instead of one big one? The barrier to entry is coming down quickly.

This is indeed becoming very serious. With artificial intelligence becoming ubiquitous and potentially misused, there will be more problems like this in the future.

I remember reading a news article about two years ago about Turkish security forces arresting someone who used  AI to impersonate President Recep Tayyip Erdoğan's voice, attempting to deceive select companies and high-ranking government officials.
https://www.trtworld.com/article/14515799

That was two years ago. Now, the situation is much more complex and dangerous, as it's now possible to create realistic videos of people who look real, not just voices.

The Turkey case is interesting because it shows how fast this was to move, right? Anyway, they caught the person. Security forces arrested them. There was accountability. But I wonder how many times there were attempts that we didn't hear. How many worked. How many government officials or companies got fooled but never say anything like it's embarrassing.

Voice to video is a huge leap. With voice you can be suspicious anyway. But video? That's where our brains just... accept it. We tend to believe what we see rather than what we hear.

Do you think that the institutional targets (governments, high-ranking officials) are rather more or less vulnerable than the regular people? Because on the one hand, they have security teams and protocols. On the other hand, they're high value targets worth the effort.

Like, the individual who tried to impersonate Erdogan probably spent a lot of time and money to do so because the payoff would be massive. But now that the technology is getting cheaper, are those economics changing in terms of who gets targeted?

I find it extremely hard to believe.
I see a lot of AI-generated videos on Social media, and I can detect them within a few seconds. We are talking about a video conference here where AI had to match the voice, face, and everything, which is nearly impossible. Even if AI can do this, it should not be that hard to notice the difference.

The social media deepfakes are pretty obvious. But, the deepfakes that you see on the social media are created for fun or low-effort scams. They're not targeting you specifically. They don't have to be perfect because they're casting a wide net.

This was a targeted attack on a specific company with preparation time. The scammers knew who the CFO was, what he looked like, how he talked. They probably scraped hours of video from company presentations, earnings calls, public appearances. Then they created the deepfake - specifically for that employee. And they impersonated several executives during a real-time video conferencing.

If this is a real news, I would expect some referance link so I can have a read.

Click to the image for fast transfering to google

If a 3rd person can get the data, that means you didn't care about your privacy and you are liable for that.

The privacy angle is interesting though. You're correct, if a third party has your data, then something went wrong. But in this case, the data they needed was in the public domain. Company website. LinkedIn. Earnings calls. Press conferences. All public information. They didn't have to hack anything. So they just had to scrape out what was already out there. And moreover, He is CFO

The employee in this case did everything right by traditional security standards. They were suspicious about the email. They demanded video verification. They still got destroyed. So what's the standard now? Never trust video? Never trust voice? Only perform financial transactions in person?

Anti-AI software is going to become big, I think.

I am quite tired of the AI slop and advanced scamming.
We need a way to filter these things.

AI has its limitations and those limitations should be poked and prodded at until something gives way.

Who pays for it? Because if the solution is "buy software to detect fakes", then we're just creating another problem from our own advancement. People who have money get protection. Everyone else continues to be vulnerable.

The economics of this arms race do not favor detection. Generating deepfakes is becoming less and less expensive by the month. Detection requires constant updating, constant investment. The attacker does not have to win all the time. The defender must win each and every time.

I've been thinking about this a lot since I wrote the post. The limitation of AI that you're talking about, yeah, it exists now. But it's shrinking. Every time detection gets better, generation gets better too. And generation's always gonna be cheaper because you're creating something, you're not analyzing something.

So even if anti-AI software becomes big, I don't know if it solves the underlying issue. Which is: trust used to be free. Now it costs money. And that cost is going to continue to increase.

[WillyAp]

This generation of young people who are tech-savvy will adapt faster then previous generations, which unfortunately will get scammed more often.
As stands for now, we don't have options but to adapt and evolve to protect our capital, to grow more skeptical about anything we see on television and on our phones.

Many of this young generation are not able to wash their clothes, or find a girl/boy to mate.

[Hispo]

This generation of young people who are tech-savvy will adapt faster then previous generations, which unfortunately will get scammed more often.
As stands for now, we don't have options but to adapt and evolve to protect our capital, to grow more skeptical about anything we see on television and on our phones.

Many of this young generation are not able to wash their clothes, or find a girl/boy to mate.

How does any of that have anything to do with artificial intelligence or being tech-savvy?
The bulk of young people who understand how to use Articial intelligence are smart and have enough emotional intelligence to be functional members of society and have successful careers and families. You are just applying an stereotype to this argument about AI for no constructive reason.
Those who are growing with the implementation of AI, are the one who will be able to identify it's threats in the short and mid term, while the oldest generations will continue to be fooled and getting their wallets emptied by scammers.

[davis196]

In January of 2024, an employee gave twenty-five million dollars to scammers in the midst of a video call. The employee saw his/her boss and colleagues on screen. Everyone looked real. But all of the individuals were AI-generated.

Verification once used to be cheap. You see somebody's face, hear their voice, you know it's them. That verification cost was essentially nothing. Now? The same verification is worthless. To actually verify identity you need multiple channels, code phrases, callback protocols; all of which cost time and money. The cost of trust just got higher, and higher, for all people.

Videocalls will become obsolete as a method for verification. There are other ways to verify something, without having to rely on video and audio. Verification with public and private keys for example. I keep thinking that AI will make ID verification obsolete as well. How easy it is for AI to create a fake ID card with a fake picture of some non-existing person? What about AI-generated selfies of someone holding an ID card? Will third-party ID verification services like Jumio and Onfido be able to catch such fake AI-generated ID cards and selfies? Maybe the best method would be to use one AI technology to battle another AI technology. I'm sure that AI can definitely track fake AI-generated video and audio.

[crwth]

If it were something with verification, like blockchain-based verification or a zero-trust network, this might not have happened. I think there should always be a Training and Awareness for employees to have solutions before there are any problems for this.

Cybersecurity is very important and the tools now are getting better so there's a chance for it to be used badly.

[EarnOnVictor]

The truth is that this could happen to anybody, and the major reason is because of a "failed company system."

Companies' hierarchy often likes to breach security protocol, this is one of the uncommon bad results of that. I'm sure that almost all companies, if not all, would be guilty of that. So long as they see the boss or a higher power in the office, they can obey their order before doing the needful (signature and other security passes/verification). I believe it's for the fear/honour of the higher hierarchy that caused the employee to over-relied on the visual recognition, and I tell you, that would have been their way of doing things for a long time, which was why the Deepfake AI could be successful.

Besides, an approval of such a huge amount of money should never be allowed to be processed by a single person. At least 2 or 3 physically present approvals should recorded, and with official phone verification(s).

[EluguHcman]

Transferred 25 million dollars over the video call? For that big amount there must be a legal process involved with all the paper and documentation so it is not that easy unless someone is stupid enough and given that much money to manage.

You also can not tell if there had been some necessary quality looking and convinceable legal documents were also tendered leading to the fall of the victim.

I think technology has really become a specific tool being admired for scamming.
On the quote below is a formal director of one of the commerical banks in Nigeria who successfully sold a fake (inexistent) airport to a bank in Brazil in the late 90's for $242 M.
The scammers was said to had snitched to get some legal documents following his purporting as the Boss of the Central Banks of the Federation Nigeria (CBN) just to make the other party believe every provided documents is being official.

Nwude and his team of accomplices used their convincing false identities and access to confidential documents to string Sakaguchi along. By the time anyone suspected foul play, irreparable damage had already been done.

https://www.thetravel.com/con-artist-sold-airport-that-did-not-exist/?

So it is very possible that similarly that employee who lost such huge amount of money to the scammers may had been tricked with documents provided because I also believe such a range amount of money can not be proceeded to be successful unless there had been a long mile of convinceable.

[justinlamode]

If it were something with verification, like blockchain-based verification or a zero-trust network, this might not have happened. I think there should always be a Training and Awareness for employees to have solutions before there are any problems for this.

Cybersecurity is very important and the tools now are getting better so there's a chance for it to be used badly.

Companies invest millions into training of their workers against issues like this. I have gone through many of such trainings and for organization that is of this scale, such trainings will be mandatory for their workers. Which is why I still maintain that carelessness from the employee is a plausible explanation for what happened. There is no way you will get approval of $25 million over a video call because that is not any evidence, email correspondence is more formal as that can be referenced.

[slapper]

It's a good point, but I'd still be wary about believing some of the news stories that come out. There are a lot of silly people who will fall for scams quite easily, even with plenty of training given, but we can all be susceptible if caught at the wrong time with the right combination of factors. We don't actually know the full details of what was involved in this particular scam, sometimes it will be glamorized by the victim because otherwise they might lose even more business and reputation. There will need to be an evolution in protocols as you say, but unless there are offline or separate secure channels for communicating, then all the same vulnerabilities will continue to exist. It does make for interesting discussions though, as AI has come along rather quickly and can be very realistic even in these early stages.

If companies are lying about how sophisticated the attacks were, that means they're hiding the real vulnerabilities. Which means other companies can't learn from it. Which means the same dumb mistakes continue to be made over and over. So whether it's actually a deepfake or some regular social engineering in a bodysuit to sound cooler, the economic damage is real. $25 million is $25 million.

I agree with the most part but to be fair this type of scams have been around even before AI was a thing. The scammers used various tactics to fool people into giving them their money willingly. For example through various methods they regularly convince people to install some innocent looking software (like a fun game or one of those image editing softwares) on their phone that contains something malicious. Then that malicious code used their phone to silently send messages to their contacts asking for money from their number but to the scammer's account.

Social engineering has always been the real attack vector. All the malicious app things you mentioned is just exploiting the same human vulnerability: We trust what looks familiar. However, those old scams required effort which was linear. To send fake SMS messages from someone's phone you had to compromise that particular phone. To run a fake tech support operation, you needed call centers, people with accents that sounded right, scripts. Each victim needed individual work.

What's different now (I believe didn't explain this well enough in the op) is that with AI the cost of deception is significantly dropped. You do not need to compromise someone's phone to send messages which look like they are from them. You need some voice samples (which is everywhere) and you can produce a call that sounds exactly like them.

The human vulnerability is the same. People still fall for scams because they believe in what is real. But the scale and sophistication available to the attackers just changed dramatically. And I think that alters the economics in a way that we haven't quite caught on to yet.

I'd love to read the news about this case to get the full picture.

I have added the link in the OP. Forgot about that

~Because they look surreal, especially that ring camera angles. The point is, someone in the comment sections of a videos said, these narrative (trend) will only be trending until these tools are used by some people only, but when more people start using it like every 2nd person then it will be overrated and not liked by most well that's another thing but till then many people are going to get scammed.

These are nothing, there are some uncencored ai bots, that are too deep and to be honest I love to use them but they can be mis-used. I can't agree more with you on the fact that, world is changing and now AI is not steping into malware section, so we can say the next bull run we are going to have AI security products in trend haha.

The scam problem disappears when everybody has access to the tools? I'm not quite sure I follow that logic. It is the surreal nature that makes deepfakes work at the moment. People look at it and think "wow that's weird" but they don't instantly think "that's fake". But when everyone can make this stuff, doesn't that make the problem worse, not better? Like, you said uncensored AI bots that can be misused. When those tools become easier and cheaper (which they will), the bar to running a scam is lowered. Right now it requires some technical knowledge to deepfake a CFO in a video call. In two years? Maybe it's just an app. The trend dying don't mean the danger dies. It simply means that the danger becomes invisible.

That is something I have realized since I started to see news from people getting scammed because of sophisticate deep fakes being generated in real time with the help of artificial intelligence. Eventually, we will reach a point when we won't longer be able to afford to believe that we see on screens, and we will need to sacrifice time and energy to verify whether we are getting scammed or not.
This generation of young people who are tech-savvy will adapt faster then previous generations, which unfortunately will get scammed more often.

As stands for now, we don't have options but to adapt and evolve to protect our capital, to grow more skeptical about anything we see on television and on our phones.

Older generations have something the younger people don't: established trust networks. Younger people? We're digital-native, sure. But we're also transacting with strangers all the time. We don't have those decades long relationships. So even if we're able to do maybe a better job than our parents at spotting a deepfake, we're open to a lot more attack surface.

And the price of being skeptical is compounded with time. You're right that we need to be a little bit more skeptical. But with skepticism everything slows down. If I have to validate every financial transaction across any number of channels, that's time I'm not making money. For a person who has a consistent salary, that's frustrating but not too bad. For someone who's doing freelance work, making money in crypto, working cross-border, that's a lot of friction, and the cost is expensive.

[icebar]

Yes, it is true that people become more advanced, the cost of trust increases. Especially after the advent of AI technology, people are also being harmed as well as benefited. Earlier, we used to consider video evidence as real, but nowadays AI technology has developed so much that a copy of that person can be created with an AI. In case of such a problem, people now have to spend a lot of money and waste a lot of time to prove the truth and falsehood. Due to the benefits of AI, I am now having trouble understanding which is real and which is fake. People have to spend a lot of money and time now than before. Now they have to be extremely careful in doing every work.

[Ultegra134]

I've read multiple articles and posts regarding AI or deepfake scams, focusing elderly or generally people who are unaware of such scams. The technology is so advanced, that it can even fool knowledgeable people, there has been a great improvement when it comes to AI video generation (my parents will literally believe the majority of AI videos on Facebook). However, seeing a tech firm (employee's fault but still) is extremely concerning. Who would have thought that their employer were AI generated and this was all a scam?

Just like the OP said, verification used to be simple, seeing and hearing someone used to be enough, but now isn't. Drastic verification measures should be taken to avoid similar cases in the future.