Huge site outage!

[shahzadafzal]

It was the same weird thing which caused a few hours of downtime 6-12 months ago, and I think it's what causes the few minutes of downtime people observe every few weeks. I've never been able to figure out what causes it. It looks like the network gets cut between the web server and the database for a while. It seems to always happen at around the same time, so I suppose it's probably some automated maintenance process messing something up, but I don't have anything running at this time, so I wonder if it's something in the web-host's networking infrastructure, or something like that. It's difficult to diagnose, since it happens rarely, and usually when I'm sleeping.

Ok so theymos made the "I" bold... "I don't have anything running at this time"... does it mean someone else have been running things but theymos don't want to name him

This reminds me of working on one project where daily backup of the database used to take ages (3-4 hours) and during this time no one was allowed to run any thing on the server. But as soon as we use to start backup someone will run some other tasks on the server everyone in db team started running around and asking who's running that? who's doing this? and everybody will be like... no I didn't run anything, but we knew who's doing that

(The few minutes of downtime just now was due to me doing something, and is unrelated.)

So for those few minutes of downtime you caused, hey, we all have our moments, right? At least we can blame that on you.

Anyway thanks for keeping the ship sailing, even when those rare downtimes. We appreciate your hard work!

I'm planning on upgrading the main servers sometime in the next few months. (Just as part of standard maintenance: there will be no user-visible changes other than perhaps slightly improved speed.) Maybe that'll fix it just by virtue of moving things to different areas.

Also it's great to hear you're planning an upgrade in the moving things around might just be the secret sauce to overcome the mysterious downtime.  And "slightly improved speed" – it's huge, every millisecond counts in the bitcoin world!

"there will be no user-visible changes" yeah thank you, but when? When there will be user-visible changes

[1715415545]

1715415545

[1715415545]

1715415545

[dkbit98]

It's a good thing that admins were at hand and got the servers back online. But may I ask what was the cause of the outage in the first place?

Maybe they are working on adding new 2FA forum patch, but I am just speculation and hoping for this

I think something strange is also happening with cloudflare, today I got a message that I am in some waiting line and that I have to wait for one minute to continue to website... don't know if this is related or not.

I was online at that time and I was waiting boringly for almost an hour, pressing the refresh button LOL. It was fully down in everywhere.

Crazy!

It's difficult to diagnose, since it happens rarely, and usually when I'm sleeping.

Gremlins attack

[Rikafip]

Maybe they are working on adding new 2FA forum patch, but I am just speculation and hoping for this

Hah that would be great but unfortunately that wasn't the reason for this and it looks like even theymos is not really sure why that keeps happening every once in a while.

It was the same weird thing which caused a few hours of downtime 6-12 months ago, and I think it's what causes the few minutes of downtime people observe every few weeks. I've never been able to figure out what causes it. It looks like the network gets cut between the web server and the database for a while. It seems to always happen at around the same time, so I suppose it's probably some automated maintenance process messing something up, but I don't have anything running at this time, so I wonder if it's something in the web-host's networking infrastructure, or something like that. It's difficult to diagnose, since it happens rarely, and usually when I'm sleeping.

[The Sceptical Chymist]

I was online at that time and I was waiting boringly for almost an hour, pressing the refresh button LOL. It was fully down in everywhere.

Years ago I probably would've been in the same boat, eager to see what kind of drama was playing out or browsing the Speculation section during one of the raging bull markets.  Maybe it's a change in attitude or enthusiasm or the state of the forum, but while I did in fact notice that bitcointalk was down for a lot longer than it normally goes down for, I just got up from my busted-up chair, stretched my legs, and went for a bike ride.

That was hours ago, and *yay!* we're back.  I don't know much about cloudfare, but I thought there might be some kind of DDOS attack (and if that were to happen, some other screen would probably show up, right?).  So let's get back to the fun and games, kids.

[GxSTxV]

This morning, I got worried when I noticed an unusual delay that took hours and different than previous time where it took only few minutes. I texted some friends and tried using different devices thinking the issue might be only on my device or my home network. But after checking websites like isitdownrightnow, I discovered that the problem is with everyone. In my opinion this is a serious issue, and even the administrators here are not aware of the main reason and why it took longer this time. I'm wondering if the problem is related to Cloudflare or the hosting servers, and I believe the hosting company can provide an answer for this error and suggest ways to prevent it.
 Some of us can't live without Bitcointalk for even few hours LOL!

[Upgrade00]

In my opinion this is a serious issue, and even the administrators here are not aware of the main reason and why it took longer this time. I'm wondering if the problem is related to Cloudflare or the hosting servers, and I believe the hosting company can provide an answer for this error and suggest ways to prevent it.

The hosting company will not be providing an answer to what caused the downtime or ways to prevent it.
It is impossible to have a website or app with zero downtimes, but when it last as long as 2-3 hours then it is a problem that you will not want to repeat itself.

Hopefully the update on the server end will be able to fix whatever issues that caused the downtime.

[virasog]

This morning, I got worried when I noticed an unusual delay that took hours and different than previous time where it took only few minutes. I texted some friends and tried using different devices thinking the issue might be only on my device or my home network. But after checking websites like isitdownrightnow, I discovered that the problem is with everyone. In my opinion this is a serious issue, and even the administrators here are not aware of the main reason and why it took longer this time. I'm wondering if the problem is related to Cloudflare or the hosting servers, and I believe the hosting company can provide an answer for this error and suggest ways to prevent it.
 Some of us can't live without Bitcointalk for even few hours LOL!

Luckily or unluckily (don't know what to call it), I did not experience this big outage today and the reason for this was that I wasn't online on bitcointalk at this moment.

However, i know that these days we often experience outages in bitcointalk, the previous recent one was experienced by me when Re: Bitcointalk was down .

It is true that we cannot live without bitcointalk but what worries me is that theymos is also not sure why the forum is facing too many of these downtimes recently.

It was the same weird thing which caused a few hours of downtime 6-12 months ago, and I think it's what causes the few minutes of downtime people observe every few weeks. I've never been able to figure out what causes it.

[philipma1957]

I slept for the entire time. completely unaware of it until I read this thread.

best possible way to deal with down time just sleep right through it.

[vapourminer]

Theymos just forgot to pay the bills this month. Better make sure to remind him to set some reminders.

his btc tx for hosting bounced

[Zoomic]

I slept for the entire time. completely unaware of it until I read this thread.

best possible way to deal with down time just sleep right through it.

I don't actually know about the forum down time as I was totally not online and not aware of it, if not for this thread. I remember those days I was involved with the Ponzi scheme sites, whenever they go down and say about site maintenance, they might not return from there. But with the longevity of the forum and the calibre of personalities in the forum, I do not think there will be a time this forum will go down and theymos will not choose to restore the site.
In as much as theymos is willing to always renew the domain of this site, it will as long as bitcoin is lasting

[JollyGood]

It seems

I slept for the entire time. completely unaware of it until I read this thread.

best possible way to deal with down time just sleep right through it.

Some of us maybe would have slept through it but we are not all in the same time zones. I had no idea about the outage but I have noted from reading threads in meta, it seems to be happening a little more regularly than we would have expected.

Theymos just forgot to pay the bills this month. Better make sure to remind him to set some reminders.

his btc tx for hosting bounced

Now, imagine if that was something that really did happen  

[Jessie2121]

This is one of those crazy experience everyone wouldn't love to have in a long while, like 3hrs and was still counting until it came back online. Hold on, is it that the next one might take a whole day? Maybe is time to start expecting it every month.

[LoyceV]

(The few minutes of downtime just now was due to me doing something, and is unrelated.)

Not again

(source)

Some of us can't live without Bitcointalk for even few hours LOL!

I was thinking we need a backup plan. What if some day the forum goes offline and doesn't get back up? It would be really nice if we can "start over" at another location, where only the real users can claim the same username. But to start, everyone would need to know that location and it can't be found without Bitcointalk.

[joker_josue]

Some of us can't live without Bitcointalk for even few hours LOL!

I was thinking we need a backup plan. What if some day the forum goes offline and doesn't get back up? It would be really nice if we can "start over" at another location, where only the real users can claim the same username. But to start, everyone would need to know that location and it can't be found without Bitcointalk.

In reality, starting over is not difficult, you and TryNinja already have a copy of all forum activity.

The question now is, how do you find the location, if it was only possible to find it via bitcointalk?

[LoyceV]

In reality, starting over is not difficult, you and TryNinja already have a copy of all forum activity.

You have too much confidence in me. In reality, we have an outdated and incomplete copy of only posts before they were edited, without user feedback, without associated email addresses, without full ban information and without forum infrastructure.

[joker_josue]

In reality, starting over is not difficult, you and TryNinja already have a copy of all forum activity.

You have too much confidence in me. In reality, we have an outdated and incomplete copy of only posts before they were edited, without user feedback, without associated email addresses, without full ban information and without forum infrastructure.

You never did anything to make me distrust you!

Yes, but to have this type of information, only if @theymos provides a regular bakcup, someone. This already seemed more complicated to me. Either way, I think he has worked out mechanisms so that if something happens to him, the project will continue to live.

[digaran]

Yesterday I noticed internet archive was down at the same time, is there a way to check all the outages in the past and see which other sites were affected at the same time?
I'm saying this because LoyceV mentioned that their scrappers were down as well.  Maybe they are related?

[LoyceV]

I'm saying this because LoyceV mentioned that their scrappers were down as well.  Maybe they are related?

My scraper didn't miss a post when the forum was down yesterday. I did miss posts in the past when Cloudflare blocked it.
Of course this is related: if the forum is down, obviously I can't scrape it.

[virasog]

I'm saying this because LoyceV mentioned that their scrappers were down as well.  Maybe they are related?

My scraper didn't miss a post when the forum was down yesterday. I did miss posts in the past when Cloudflare blocked it.
Of course this is related: if the forum is down, obviously I can't scrape it.

Well, i guess no one may have noticed but it happened again. I think if I hadn't taken this screenshot, no one would have believed me that bitcointalk was down for a sometime again a few minutes ago   

[JollyGood]

You did not need to attach the screenshot because I did notice it, I was making a post at the time and was using the preview button at the time. From what is being posted by others, I think this is occurring far more regularly now than at any other time I have known before.

Well, i guess no one may have noticed but it happened again. I think if I hadn't taken this screenshot, no one would have believed me that bitcointalk was down for a sometime again a few minutes ago