Railway.com breached

[coinlary]

I got an email from Railway customer sypport yesterday about a breach in some of their systems handling customer environment  variables, maybe more but thats all i got  :

Earlier this morning, Railway detected anomalous activity involving systems that store customer environment variables. After further investigation, our investigation to date indicates that an unauthorized third party may have accessed encrypted environment variables and, for some of them, obtained the decrypted contents. Environment variables may contain secrets such as API keys or database and registry credentials.

We believe this is part of a series of attacks recently targeting platform companies with the aim of compromising AI inference providers. We have notified these upstream providers to revoke all keys proactively, as we take these kinds of events extremely seriously.

What you should do

Immediately revoke and reissue all environment variables, including any API keys, stored in Railway.
If you have any credentials on an external registry, .
Review logs for those services for unusual activity.
Until builds are restored, you will not be able to roll out new versions. We apologize for this inconvenience.

So if you use Railway or have used Railway for any of your deploymesnts , you may want to start doing something about those secret keys & env vars before something  goes wrong with your deployed apps/services.

You can follow up here

For mods: I hope this is the right board to post this. If it's not, please move to the appropriate board.
Thank you!

[nc50lc]

Has it something to do with the recently discovered PHP vulnerabilities that'll allow an attacker to do remote code execution?
The link doesn't have hits when searching for "PHP" keyword, though.

For more info: /index.php?topic=5597537.0

[coinlary]

Has it something to do with the recently discovered PHP vulnerabilities that'll allow an attacker to do remote code execution?
The link doesn't have hits when searching for "PHP" keyword, though.

For more info: /index.php?topic=5597537.0

Thanks for the link.

It could be but I'm not sure if they use PHP internally, i do know that Node js is among the stack used internally so maybe NodeJs or  a different  loophole entirely in their system. So far no reliable  info regarding how it happened other than the same report from Jacob C. informing users to take actions.

They also mentioned that there would be a preliminary report by noon EST.

[noorman0]

They also publish articles recommending specific actions for each case. I noticed 13 different articles published on the same date.
https://blog.railway.com/engineering

I don't know what the OP has built with this platform, but if you received an email, you are an affected user.

As we confirm which accounts were affected, we will contact those customers directly by email.

[348Judah]

With the rate many platforms are getting attacked these days and even considering the recent PHP attack as well, we should be more careful on using a random platform for anything, while using centralized storage should also be forbidden by us if we want to maintain a high standard of preventing any form of attack from taking place. Most of them will not give prior notice until we are finally under attack.

[yhiaali3]

I don’t know what is happening; the internet is no longer a safe place at all.
The leaking of API keys and database data unfortunately gives hackers the ability to access and tamper with external developer services and databases, or steal information from them.

Affected users should immediately revoke and reissue all environment variables and API keys stored on the Railway platform and review the activity logs of external services associated with the application to ensure there is no unusual or unauthorized activity.

[Upgrade00]

They just posted a preliminary report; https://x.com/Railway/status/2113671961624682722

It gives a bit more detail to what they think happened, what you should do and the users most at risk. "We recommend that all customers who signed up prior to October 21 rotate their keys stored on Railway. We have notified the customers that have been directly impacted."

The last weeks have made me treat security a bit more proactively. We some of the times get comfortable with thinking a system is safe that we stop testing them or taking actions to protect ourselves. We should act always as if we were under attack.

[avp2306]

Has it something to do with the recently discovered PHP vulnerabilities that'll allow an attacker to do remote code execution?
The link doesn't have hits when searching for "PHP" keyword, though.

For more info: /index.php?topic=5597537.0

I think not, there's preliminary report about the incident happened https://blog.railway.com/p/preliminary-report-october-security-incident

There's also a article discussing about two incidents https://www.traversymedia.com/blog/a-bad-week-for-the-web-php-railway-and-what-to-do

Looking at recent incident happened on Railway, it looks like hacking issues became so rampant now. I guess its good for people to pay close attention with patterns or methods done by hackers, to have an idea and evade experiencing any possible issues.

[Accardo]

With the rate many platforms are getting attacked these days and even considering the recent PHP attack as well, we should be more careful on using a random platform for anything, while using centralized storage should also be forbidden by us if we want to maintain a high standard of preventing any form of attack from taking place. Most of them will not give prior notice until we are finally under attack.

Opencode faced same API key breach few days back, it's strange agents are getting way ahead of advanced system securities, out of multiple methods, Open AI just found out their agent could extract and share information through a cache timing technique executed through the processor cache where recently accessed data by apps are held. Some attacks in the future by these agents won't require an internet connection. Devs are seriously under attack.

[coinlary]

I don’t know what is happening; the internet is no longer a safe place at all.

The internet has never been a safe place, it is only getting worse.

They just posted a preliminary report; https://x.com/Railway/status/2113671961624682722

Yes, I got another email yesterday but this time it's personalized based on my workspace. Almost same advice from the general announcement, rotate credentials and revoke the old ones.

I think not, there's preliminary report about the incident happened https://blog.railway.com/p/preliminary-report-october-security-incident

I already checked the preliminary report, they haven't  given any clear statement regarding what caused it, that was as at yesterday when i checked. I don’t know if they've publish another report today.