Bitcoin Forum

Recently it has loaded very slow these past few days.

I wish it was, it's my Wireless provider that's been crap. I live in a rural area temporarily and the internet is just crap, I ain't even getting my promised speeds.

I'm having this problem too.  Got a database error as well the other day when I tried to post a reply.

Was just going to post about this very same thing, yes the database seems to be very slow at random times.  ???

noticed the slowness too

Very sluggish.  Last time this happened site crashed for a short time.

You're not the only ones...

Seems better now. Must be an intermittent server or internet connection issue at their end.

I'm also having difficulty accessing BitcoinTalk, very slow and frequent database errors.

Pretty sluggish here as well.

It's actually starting to feel faster than usual if you ask me.

I also feel it's been quite slow recently. May be it's time for the admins to manufacture some hero status accounts for sale to fund server upgrades.  :D

Super slow on my end as well

Still very slow...

It isn't really slow. It's more like erratic.
2 clicks out of 10 result in a lightning fast render.
But 8 clicks out of 10 end up with a timeout, or a really long loading time (>=5 seconds).

I think that the site becomes slow every 90 seconds when the modSettings cache expires and every visitor then needs to do an expensive query. Does this seem to match your observations?

SMF handles modSettings really badly.

Based on a grep of SMF source, it seems the modSettings contains only forum-wide settings. If it's global, can't the cache be set to indefinite, and manually cleared when needed?

That's how modSettings should work, but it's actually also used for things like a timestamp of the last ban added, so it changes pretty frequently.

So, where's that new forum software? You've only got a fraction of a million dollars sitting in theymos' pocket.

Yes that sort of matches for me - I can be clicking with sub 0.5 page loads times then suddenly 10+ second page load on the next click. When a page does load slow it's always over 10 seconds for me - I'm not seeing any variation like some pages loading 5 or 6 seconds etc. It's either really fast or 10+ seconds. Now you mention the 90 secs cache expiry I will look for that.

Why does every visitor need to refresh the cache though? It's still a global array, so I'm confused as to why it's not just refreshed globally through a cron script, dumped into a mod_settings_cache.php file, and loaded.

And this has only been happening for about a week - something must have changed recently...

Maybe we should get rid of threads like the one "Lets post 21 million photos"

Why the hell do we need that garbage on the forums anyways? Its just getting out of hand, reduce the noise, that leads to reduced traffic and load on the servers = less problems.

I don't think that costs the forum anything. Storage is dirt cheap—highly doubtful that's the issue—and it makes an insignificant portion of the posting volume.

^^^_______^^^^^^

lol

Same thing is happening with me, forum is slow.  :(

I don't think its post volume. If anything I'll bet posts have decreased since member level was de-coupled from post count. Something else is wrong.

I too have been experiencing slow pageloads occasionally, while most of the time the loads only take less than a second. But I'm running into these slowdowns much more frequently than every 90 seconds.

I just tried, and I had two very slow (~10s) pageloads in a row, then multiple sub-second pageloads for 10-20 seconds, and then another slow load.

EDIT: Submitting this post also took ~15 seconds after I pressed the button :)

I beg to differ. I'm getting the slow ones most of the times, the fast ones are pretty scarce.
And I'm not quite sure about the 90 seconds, yet, but it sure looks like it's a good lead.

Yep - it's getting worse now - slow responses on nearly every click...

Seems to have become much worse today. That was my experience for the past week or so, but it indeed seems to be even worse now. I just got some database errors as well, when I tried to edit my post.

Welcome back to 2011

I also noticed slow loading times something like a week ago. Few days it was fine, today it was slow again...

Other web sites load as usual, so it's not ISP.

Any ideas?

This is the only decent lead so far, and it looks promising, imo. Hopefully reports will pile up and this will be adressed.

Theymos, do you need any kind of reporting about the issue, can we help in any way?

I'm no longer experiencing the slow-downs, seems to be fixed.
Thanks!

Sorry, but still very slow.  More slow when hitting back button for previous pages than on clicking new topics.

It's around the 3-second loading mark for me at the moment.

Lot faster now, for me anyway.

Same here, seems better

Feels better especially when I used a VPN, for no reason at all.

Maybe you used the same VPN the forum is hosted on?

Nope, I didn't. Maybe it's just that theymos fixed something up with the forums.

Actually, the random - though very frequent - stallings are back for me, guess it's linked to the peak hours, which is probably not a good sign.  :-\

Same

I still have slowdowns too.

Random slowdowns, and slowdowns on DB actions (posting, search, et al) reported here. :(

Do you guys think that it is easy to keep so many mirrors of the backend database: NSA, CIA, FBI and now SEC?  ;)  I hope SEC will finally publish the most interesting deleted posts.

Or maybe the SSD disks storing the backends are getting worn out? Nearly random and multi-second delays on writes are common symptoms when using flash SSD drives to store databases with write-ahead-logging, which is the worst wear pattern for them: many writes and no reads.

I can take slow, but dread the disappearance of Bitcointalk. I remember loss of fuckedcompany.com post database and it was dreadfull loss to the information technology folklore.

Is there any way to test for this? I thought that SSDs were built to last several years, even under extremely heavy load.

The site is slowing down from all the avalon customers that have it on auto-refresh awaiting word from Yifu. 

Or maybe that's just my contribution  :'(

They can last years if you have a decent mix of reads and writes for the flash technology. Typical write-ahead-log is a worst case write-amplification pattern: writes 512-byte blocks, forces physical sync/flush to the drive, which in turn forces full page (about 4096-bytes) relocate and erase in the flash controller. And to add the insult to the injury that data is never read unless the database or the OS crashes.

I don't know of any way to check it while the disks are online, besides continuously monitoring the S.M.A.R.T. parameters with smartmontools and hope that the "old-age" statistics shown by them aren't lying.

The folks I know would simply run a full backup, full low-level security erase of the flash drives and then full restore. The security erase may help fix the bottlenecks in the wear-leveling software in the flash controller that is causing those random delays.

But they also aggresively use the warranties on the flash drives, preemptively returning them before the warranty expires. This is only workable if one orders such drives in quantities large enough to compile some sensible wear statistics.

Due to the extremely competitive nature of this business it is quite hard to get true and correct information about the wear-leveling algorithms used and their interaction with file systems. I'm sorry about not being able to be more specific: on one hand there are NDAs, on the other hand the flash controller firmware changes very frequently.

To get flash SSD last several years you'll need to mix them with normal spinning drives. Since database logs are almost exclusively sequentially written and almost never read they won't bottleneck the whole machine.

Edit: If you are willing and able to rebuild your kernel there ware some patches that use ATA TRIM and optional bitwise negation to significantly reduce the flash wear on some controllers. The frequent operation of "allocate disk blocks sectors and initialize to zeros" is replaced with simply "erase block" which initializes flash to all 0xFF and then stores all data complemented. Folks who created and maintained those patches would presumably know for which controllers they are worth applying and maybe if there are other custom patches for similar circumstances. The other strings worth seaarching may be "Deterministic Read Zero after Trim" DZAT, which basically refers to the same idea implemented closer to the actual hardware.

Edit2: I keep forgetting about the partition table issue. Many OS installers continue to create the "compatible" disk partition layout with integral number of "cylinders" (C/H/S == x/255/63). With 255 "heads" per "cylinder" and 63 "sectors" per "track" you would be assured that the partitions have odd sizes and alignments. Therefore the most common I/O operation 4kB is assured to cross into 2 physical 4kB sectors. Various optimizations then become ineffective and they automatically disable itself for the sake of safety. This is related, but non identical, to the "long sector" "advanced format" on the mechanical drives. Just make sure that you didn't accidently set your system into "512 byte sector emulation" (512e) mode.

I do not think a hardware problem should be investigated until all software problems are ruled out. Try attaching counters to suspect functions. Don't bother with making a SQL table for them; just make a bunch of flat files (possibly mounted on RAM, but shouldn't matter that much), one for each function. All that's needed are two calls to system time, a subtraction and addition, and a file read/write (possibly to tmpfs)—all should be extremely cheap, and the data is invaluable.

SSDs were built to last for so many write cycles.  If your hard drive is almost full, then the same sectors will get re-written faster, making them fail faster.  SSD capacities are determined giving some lee-way (a 100GB drive might have 120GB of space) for sector failures.

When using SSD's you should make sure your drives never become full - always have lots of free space.

I improved modSettings. It's only updated once every few minutes now instead of once every 5-10 seconds. Hopefully that'll make things faster.

I'm sorry to be the bad news messenger, but it has not improved anything on my side.
I also got a few whole blank pages, with no error, but this was just before your post, so I'm guessing it's related to your fix deployment.

Still the same for me.

I'm seeing much better results

Yes.  Quicker today.

Just read a bunch of pages, smooth surfing!

Much better.

Yes. Good again. Thanks

Confirmed

It seems to be much faster now most of the time, although I did have a few long loading times today.

Running like a top.

So far it has been fine.  However, occasionally, the site will fail to load at all for minutes at a time, as if it were offline.  Other times, it's just reliably slow, but now, it's speedy.

Noticed the same on my phone, thought it was just my crappy 3G, a quick refresh brings it straight back usually though.

Same, I got several Time-Outs when posting. Loading is for the most part fine, though.

Running slow again this evening.  :(

Confirmed, sluggy again... If we can help you track down the issue, just shoot instructions away!
Good luck.

That's weird. It has been running completely fine for a week already - at least for me.