Upcoming downtime

[theymos]

In a few hours I'm going to switch to nginx from Apache, so there might be a long period of downtime while I get it working.

[1715510403]

1715510403

[1715510403]

1715510403

[1715510403]

1715510403

[1715510403]

1715510403

[1715510403]

1715510403

[ingrownpocket]

Thanks.

[greyhawk]

Now watch Avalon Batch 2 being sold right during that time. 

[Phinnaeus Gage]

Did the downtime occur yet, and does that explain why I don't see any avatars in threads?

[mufa23]

...explain why I don't see any avatars in threads?

Avatars are temporarily disabled.

Thread --> https://bitcointalk.org/index.php?topic=140345.0

[Stephen Gornick]

there might be a long period of downtime while I get it working.

Is this now completed, or will there be further adjustments [edit: affecting site availability] made?

[Red Emerald]

Wait. You are switching TO apache?  I don't see that often.

Words are hard.

[theymos]

OK, it's done. There were more complications than I expected. Tell me if you see any problems.

[Akka]

Works fine (so far), thank you very much for upgrading.

Can we expect the forum to lag less in high traffic peaks now?

[gweedo]

Works fine (so far), thank you very much for upgrading.

Can we expect the forum to lag less in high traffic peaks now?

yes nginx, uses a different way of handling each connection then apache so the performance should be better 

[mufa23]

Seems to work. Thanks

[Lethn]

Seems to be working quite quickly now compared to before, definite improvement in speed, nice one.

[wtfvanity]

There were some slow spots but things have picked up. Hope my suggestions URL I posted didn't cause too many problems, doesn't sound like the avatar function helped that much.

You may want to consider some nginx tuning as well if you haven't already tried.

[wtfvanity]

Site had a bad gateway for a second but it's even faster now.

[theymos]

There were some slow spots but things have picked up. Hope my suggestions URL I posted didn't cause too many problems, doesn't sound like the avatar function helped that much.

You may want to consider some nginx tuning as well if you haven't already tried.

No, it was a good idea. I just wasn't able to do it properly with the old Apache setup (which was somewhat broken). I also converted some tables to InnoDB as recommended at your link. Most of the suggestions there involved disabling useful features, though, which I'm not going to do.

Thanks for the tip.

[wtfvanity]

I think there is an improvement, but it might just some kind of placebo effect since I knew changes were being made!

There were a few minutes of down time. Maybe some people left. Wait until another peek time.

Theymos, do you have any say bandwidth stats that show when the forums are the most heavily used?

I know they all weren't good or usable, but something like the icons the way it explained it seemed like it could be super helpful.

[repentance]

Still getting a LOT of 502 errors here.

[theymos]

Still getting a LOT of 502 errors here.

Randomly interspersed with working pages?

[davout]

No, it was a good idea. I just wasn't able to do it properly with the old Apache setup (which was somewhat broken). I also converted some tables to InnoDB as recommended at your link. Most of the suggestions there involved disabling useful features, though, which I'm not going to do.

InnoDB has lots of tuning options that can dramatically impact performance, especially the buffer pool size, but you probably already know that.

[davout]

Still getting a LOT of 502 errors here.

I get none.

[davout]

And yes, bitcointalk does feel much more responsive.

[wtfvanity]

Still getting a LOT of 502 errors here.

Randomly interspersed with working pages?

Theymos I'm getting them too. Before my previous post, the entire website was down, 502 for 20-30 minutes. It has just become available again after another 10-15 minutes of only 502 errors.

[repentance]

Still getting a LOT of 502 errors here.

Randomly interspersed with working pages?

It will work fine and then suddenly I'll get 502 errors for 5 minutes straight.

Not sure if it's relevant but I'm using mobile broadband and I'm pretty sure my ISP uses an open proxy.

[Phinnaeus Gage]

I just got notified via the normal popup that I had a new PM, but when I went to my PMs, nothing new. Not sure if there's a bug or what.

Somebody send me a PM to see if it's still doing it, or if I was seeing things.

Edit: Somebody did send me a PM and all seems fine. It's possible I may have been seeing things, for something was implemented where I now get micro-popups, like you see when a page automatically refreshes.

[Akka]

In a few hours I'm going to switch to nginx from Apache, so there might be a long period of downtime while I get it working.

Are there problems with nginx or is the recent outage unrelated?

[repentance]

Tons of connection errors this morning theymos.

In addition to the 502 I was getting yesterday, I'm also getting a 504 (bad gateway) error and "Sorry, SMF was unable to connect to the database. This may be caused by the server being busy. Please try again later."

[davout]

Getting these a lot too.

[theymos]

I allowed mysql to take too much memory, and it was killed by the oom killer. I didn't notice for a long time. This particular thing shouldn't happen again.

By the way, the current ads will get at least one extra day due to all this downtime and slowness.

[theymos]

There should not be any errors now.

[repentance]

There should not be any errors now.

Yep.  Stable again now.

[greyhawk]

Still getting intermittent errors as shown below every 15 minutes or so for some seconds

Connection Problems

Sorry, SMF was unable to connect to the database. This may be caused by the server being busy. Please try again later.

[Akka]

I was just getting a 502, a 504 and then SMF was saying database connection error, but now it seems to be working just letting theymos know

Me too. Seems there are still some problems.

This was about at the same time as yesterday.

Maybe a problem with high traffic peaks?

[repentance]

This was about at the same time as yesterday.

Maybe a problem with high traffic peaks?

Yep, I was getting a "connection refused" (102) error this morning for a fair while.

[theymos]

I adjusted things to hopefully stop that from happening.

[Stephen Gornick]

From another thread:

2013-02-02 Page views number wrong?

Seems way off as other recent days are closer to 400k.

[Phinnaeus Gage]

For lack of a better place to post this concern, short of starting a new thread:

For the past couple three days, I seem to be getting more than my fair share of 502 BAD GATEWAY errors.

Just a heads-up, theymos. Oh, and BTW, thanks for fuckin' up my hobby thread. Some people may have had pleasure viewing all those animated GIF avatars in the OP, but not anymore thanks to their URL links being changed.

Later, bud.

~Bruno K~

[theymos]

I haven't been seeing any. Do they appear at any particular times, or on any particular sorts of pages?

[wtfvanity]

I haven't been seeing any. Do they appear at any particular times, or on any particular sorts of pages?

Theymos, I've been seeing them too. But not a huge amount of them. They will pop in between 12-4 EST. Lasting maybe 5 minutes. When I return there are lots of posts like it wasn't effecting everyone while I was gone.

[ingrownpocket]

I haven't been seeing any. Do they appear at any particular times, or on any particular sorts of pages?

I see them about 2-3 times in a busy day.
Specially here: https://bitcointalk.org/index.php?action=unread

[wtfvanity]

For the last few minutes site has been crawling for me, but no errors. Each page taking 30-45 seconds to load.

[Akka]

Connection Problems
Sorry, SMF was unable to connect to the database. This may be caused by the server being busy. Please try again later.

 

Only for a few seconds though.

[theymos]

Yeah, I restarted the database so I can maybe see what's happening.

It looks like there's more than enough free memory. free reports 1250 MB free memory minus buffers, and smem gives similar stats. But for some reason Linux uses swap a lot (causing slowness) and occasionally the OOM killer kills php-fpm processes (causing 502 errors), so I guess there isn't actually enough memory. Anyone know what might be causing this?

[paraipan]

Yeah, I restarted the database so I can maybe see what's happening.

It looks like there's more than enough free memory. free reports 1250 MB free memory minus buffers, and smem gives similar stats. But for some reason Linux uses swap a lot (causing slowness) and occasionally the OOM killer kills php-fpm processes (causing 502 errors), so I guess there isn't actually enough memory. Anyone know what might be causing this?

Try mysqltuner theymos, you may have improper size buffers set. After that just install memcached, method is different depending on the OS, and enable the option in the SMF admin area. Mysql will be more than grateful after that 

[theymos]

I'm already using that.

I may have improved the situation by decreasing the maximum amount of memory that php-fpm and mysql can use.

[dree12]

I'm already using that.

I may have improved the situation by decreasing the maximum amount of memory that php-fpm and mysql can use.

Why not just install more memory with the forum's funds?

[cedivad]

A properly configured 100€/month ovh server would fit whatever load this website has... Please admin, give us something we can build a currency on.

[paraipan]

I'm already using that.

I may have improved the situation by decreasing the maximum amount of memory that php-fpm and mysql can use.

Got it

[greyhawk]

502s again from 21:01 to 21:02 CET

[Akka]

502s again from 21:01 to 21:02 CET

Me too.

[Phinnaeus Gage]

Another 502 about 30 seconds ago.

[Maged]

502

[theymos]

Strangely enough I have only ever gotten one 502 in the last few days. What browsers are you all using?

I'm going to find some way of upgrading the hardware. There's probably some way to get more performance out of this hardware, but I don't know how.

[21after2]

I experienced a lot of 502 errors about 2-3 days ago, but since then I've had a perfectly fine connection. Maybe even quicker than usual, though it may just seem that way since it was loading pretty damn slow during those couple days.

[theymos]

The error log has too many trivial errors. I wasn't able to immediately find the 502s. I may look into it more later.

[repentance]

Strangely enough I have only ever gotten one 502 in the last few days. What browsers are you all using?

I'm going to find some way of upgrading the hardware. There's probably some way to get more performance out of this hardware, but I don't know how.

Chrome Version 24.0.1312.57 m with Win 7.

I have the most problem with errors between about 1900 and 2200 UTC but the forum is often very laggy during the day my time (which is overnight UTC).

[greyhawk]

Strangely enough I have only ever gotten one 502 in the last few days. What browsers are you all using?

Chrome 24.0.1312.57 on Win7 Home Premium SP1
Chrome 18.0.1025469 on Android 4.2.1 GNex Build JOP40D

When the 502 problem appears, it's on both devices.

[cedivad]

I'm going to find some way of upgrading the hardware. There's probably some way to get more performance out of this hardware, but I don't know how.

If you tell us everything you know i would be happy to help.
And yes, upgrading the hardware is certainly the right thing to do.

The error log has too many trivial errors. I wasn't able to immediately find the 502s. I may look into it more later.

Most probably the 502s are caused by the php-fpm daemon, you won't find a lot of info on nginx/logs/error.log, it will just say "your php-fpm sucks".

[greyhawk]

502s again at 15:15 CET on IE 8.0.7601.17514 (don't judge  ) / Windows 7 Enterprise SP1

[Akka]

502s again at 15:15 CET on IE 8.0.7601.17514 (don't judge   ) / Windows 7 Enterprise SP1

That's interesting.

I have none. Yesterday I had some during the same time as you with my normal connection (from Germany)

Today US proxy and none.

Probably just coincidence.

[greyhawk]

I don't think it's connection related. I'm connected via X-WiN/GÉANT 2/Internet2 right now.

[wtfvanity]

I use chrome as well. I've tested it when I get the 502's and I'll open Firefox 19 and get the same error. Only lasts for a minute or two. When you first switched, it was longer, now, just momentarily.

[sounds]

Just got a 502 for a few seconds

[21after2]

The last two days I've been getting major lag again. I just had a couple errors on Chrome saying that the server shut down the connection, or something like that. First time I had seen the error.

[theymos]

I improved some more settings and restarted the web server.

I will probably switch to a much more powerful server in a week or two. I've been making arrangements.

[21after2]

That would do it then, lol.

[cedivad]

I improved some more settings and restarted the web server.

I will probably switch to a much more powerful server in a week or two. I've been making arrangements.

Can we know the details so that we can help?

[Scrat Acorns]

Getting the 502 error every couple of hours or so.

Spawn more overlords fastcgi instances. PHP-FPM has this adaptive spawning feature to handle load spikes. If you are memory starved you can do a custom PHP build (--disable-all and enable only what SMF is using) which will make the processes a lot leaner.

[Akka]

The forum is neraly unsualble today.

I hope we are ddosed or something. If not this really needs to be fixed ASAP. It's not OK if the forum collapses everytime we get a huge amount of visitors.

[cedivad]

The point is that we could help... Hell I manage 50M pageviews daily... But there is no collaboration at all from the admin.

It must be a plot to make bitcoin look less professional so that we have more time unitil it explodes.

[21after2]

The forum is neraly unsualble today.

I hope we are ddosed or something. If not this really needs to be fixed ASAP. It's not OK if the forum collapses everytime we get a huge amount of visitors.

I agree. I had multiple 500 errors today (500, 502, and 503), and it seems like the forum was completely down for quite a while.

theymos, was this because of new changes again? We need to be more pro-active about forum stability and software if we want to match forum growth, and the growth of Bitcoin as a whole.

[Phinnaeus Gage]

The forum is neraly unsualble today.

I hope we are ddosed or something. If not this really needs to be fixed ASAP. It's not OK if the forum collapses everytime we get a huge amount of visitors.

I agree. I had multiple 500 errors today (500, 502, and 503), and it seems like the forum was completely down for quite a while.

theymos, was this because of new changes again? We need to be more pro-active about forum stability and software if we want to match forum growth, and the growth of Bitcoin as a whole.

Same here. This is the first time I've been able to stay on BT for an extended period of time, and have no guarantee that I'll be able to post this post. I've had the 500, 502, and 503 errors as well all day. Along with an SSL warning of some type via Chrome (using Vista on laptop) and another really weird warning I've never seen before, nor bother to remember. Taxing my memory, it read similar to an SSL type warning, yet completely different.

Any chance the meteor had anything to do with this?  

Later, bud.

~Bruno K~

EDIT: Had a 504 (first time) Gateway Problem when I posted this and this is the first time I've been able to get back on BT. Compare timestamps with original and edit version of this post to see the difference.

[Scrat Acorns]

theymos, it is time to make a choice. This forum is running on the crappiest forum software out there. SMF 1 was released in 2003, that's 10 years ago. It is still using <tables> for formatting for crying out loud. This dinosaur cannot support a community like this. Even SMF 2 would be an improvement at this point. A faster server won't make the problem go away, it's just like patching a wound with duct tape.

Your other post about developing forum software makes absolutely no sense. Even if you paid 3 people to develop custom forum software it would take months for a working solution and at least a year for it to become stable enough. It's only a case with big companies that they develop their own forum software, the reason being that it's a huge undertaking to do something like that. There are many commercial options available right now which are already deployed on websites handling 50K+ concurrent visitors. They all support plugins so bitcoin specific parts can be developed by the community after the fact.

[21after2]

EDIT: Had a 504 (first time) Gateway Problem when I posted this and this is the first time I've been able to get back on BT. Compare timestamps with original and edit version of this post to see the difference.

Just had one myself. It's taking upwards of two minutes to load a page, if it loads at all.

[humanitee]

Just became usable for me.

[paraipan]

theymos please enable cloudflare caching and ddos protection, the forum is not usable lately.

[theymos]

theymos, was this because of new changes again? We need to be more pro-active about forum stability and software if we want to match forum growth, and the growth of Bitcoin as a whole.

No. Traffic is much higher, but I don't know exactly why it gets so slow. Memory usage seems fine now. Things improve after I restart mysql, php-fpm, etc., but then the site gradually slows down over time.

As I said, I'm making arrangements to move to a better server. This isn't as easy as it sounds because I don't (and won't) control any of the forum's critical assets other than the BTC. So I need to get someone trustworthy to buy the server (this part seems OK, but not done yet), and I need to get Sirius to update the DNS (I haven't heard from him yet).

[paraipan]

... and I need to get Sirius to update the DNS (I haven't heard from him yet).

Damn, didn't know that, so cloudflare is out of the question.

I would be more than happy to help out if needed, just ask.

[21after2]

theymos, was this because of new changes again? We need to be more pro-active about forum stability and software if we want to match forum growth, and the growth of Bitcoin as a whole.

No. Traffic is much higher, but I don't know exactly why it gets so slow. Memory usage seems fine now. Things improve after I restart mysql, php-fpm, etc., but then the site gradually slows down over time.

As I said, I'm making arrangements to move to a better server. This isn't as easy as it sounds because I don't (and won't) control any of the forum's critical assets other than the BTC. So I need to get someone trustworthy to buy the server (this part seems OK, but not done yet), and I need to get Sirius to update the DNS (I haven't heard from him yet).

Any estimated time frame on the server buy? I knew traffic was getting higher, but I'm surprised it's had this much of an affect on the site. I guess there's a silver lining in all the lag, at least.

[theymos]

I don't think that it's a bandwidth/DDoS problem. I think that the server's hardware is just too weak to handle this much traffic.

Any estimated time frame on the server buy?

Less than a week, hopefully.