Why has this site been going so slow?

[dree12]

Feels better especially when I used a VPN, for no reason at all.

Maybe you used the same VPN the forum is hosted on?

[John (John K.)]

Feels better especially when I used a VPN, for no reason at all.

Maybe you used the same VPN the forum is hosted on?

Nope, I didn't. Maybe it's just that theymos fixed something up with the forums.

[Kouye]

I'm no longer experiencing the slow-downs, seems to be fixed.
Thanks!

Actually, the random - though very frequent - stallings are back for me, guess it's linked to the peak hours, which is probably not a good sign. 

[jackjack]

I'm no longer experiencing the slow-downs, seems to be fixed.
Thanks!

Actually, the random - though very frequent - stallings are back for me, guess it's linked to the peak hours, which is probably not a good sign. 

Same

[x^2]

I'm no longer experiencing the slow-downs, seems to be fixed.
Thanks!

Actually, the random - though very frequent - stallings are back for me, guess it's linked to the peak hours, which is probably not a good sign. 

Same

I still have slowdowns too.

[whiskers75]

Random slowdowns, and slowdowns on DB actions (posting, search, et al) reported here.

[2112]

Do you guys think that it is easy to keep so many mirrors of the backend database: NSA, CIA, FBI and now SEC?    I hope SEC will finally publish the most interesting deleted posts.

Or maybe the SSD disks storing the backends are getting worn out? Nearly random and multi-second delays on writes are common symptoms when using flash SSD drives to store databases with write-ahead-logging, which is the worst wear pattern for them: many writes and no reads.

I can take slow, but dread the disappearance of Bitcointalk. I remember loss of fuckedcompany.com post database and it was dreadfull loss to the information technology folklore.

[theymos]

Or maybe the SSD disks storing the backends are getting worn out? Nearly random and multi-second delays on writes are common symptoms when using flash SSD drives to store databases with write-ahead-logging, which is the worst wear pattern for them: many writes and no reads.

Is there any way to test for this? I thought that SSDs were built to last several years, even under extremely heavy load.

[s40ward]

The site is slowing down from all the avalon customers that have it on auto-refresh awaiting word from Yifu. 

Or maybe that's just my contribution 

[2112]

Is there any way to test for this? I thought that SSDs were built to last several years, even under extremely heavy load.

They can last years if you have a decent mix of reads and writes for the flash technology. Typical write-ahead-log is a worst case write-amplification pattern: writes 512-byte blocks, forces physical sync/flush to the drive, which in turn forces full page (about 4096-bytes) relocate and erase in the flash controller. And to add the insult to the injury that data is never read unless the database or the OS crashes.

I don't know of any way to check it while the disks are online, besides continuously monitoring the S.M.A.R.T. parameters with smartmontools and hope that the "old-age" statistics shown by them aren't lying.

The folks I know would simply run a full backup, full low-level security erase of the flash drives and then full restore. The security erase may help fix the bottlenecks in the wear-leveling software in the flash controller that is causing those random delays.

But they also aggresively use the warranties on the flash drives, preemptively returning them before the warranty expires. This is only workable if one orders such drives in quantities large enough to compile some sensible wear statistics.

Due to the extremely competitive nature of this business it is quite hard to get true and correct information about the wear-leveling algorithms used and their interaction with file systems. I'm sorry about not being able to be more specific: on one hand there are NDAs, on the other hand the flash controller firmware changes very frequently.

To get flash SSD last several years you'll need to mix them with normal spinning drives. Since database logs are almost exclusively sequentially written and almost never read they won't bottleneck the whole machine.

Edit: If you are willing and able to rebuild your kernel there ware some patches that use ATA TRIM and optional bitwise negation to significantly reduce the flash wear on some controllers. The frequent operation of "allocate disk blocks sectors and initialize to zeros" is replaced with simply "erase block" which initializes flash to all 0xFF and then stores all data complemented. Folks who created and maintained those patches would presumably know for which controllers they are worth applying and maybe if there are other custom patches for similar circumstances. The other strings worth seaarching may be "Deterministic Read Zero after Trim" DZAT, which basically refers to the same idea implemented closer to the actual hardware.

Edit2: I keep forgetting about the partition table issue. Many OS installers continue to create the "compatible" disk partition layout with integral number of "cylinders" (C/H/S == x/255/63). With 255 "heads" per "cylinder" and 63 "sectors" per "track" you would be assured that the partitions have odd sizes and alignments. Therefore the most common I/O operation 4kB is assured to cross into 2 physical 4kB sectors. Various optimizations then become ineffective and they automatically disable itself for the sake of safety. This is related, but non identical, to the "long sector" "advanced format" on the mechanical drives. Just make sure that you didn't accidently set your system into "512 byte sector emulation" (512e) mode.

[dree12]

Or maybe the SSD disks storing the backends are getting worn out? Nearly random and multi-second delays on writes are common symptoms when using flash SSD drives to store databases with write-ahead-logging, which is the worst wear pattern for them: many writes and no reads.

Is there any way to test for this? I thought that SSDs were built to last several years, even under extremely heavy load.

I do not think a hardware problem should be investigated until all software problems are ruled out. Try attaching counters to suspect functions. Don't bother with making a SQL table for them; just make a bunch of flat files (possibly mounted on RAM, but shouldn't matter that much), one for each function. All that's needed are two calls to system time, a subtraction and addition, and a file read/write (possibly to tmpfs)—all should be extremely cheap, and the data is invaluable.

[Vod]

Or maybe the SSD disks storing the backends are getting worn out? Nearly random and multi-second delays on writes are common symptoms when using flash SSD drives to store databases with write-ahead-logging, which is the worst wear pattern for them: many writes and no reads.

Is there any way to test for this? I thought that SSDs were built to last several years, even under extremely heavy load.

SSDs were built to last for so many write cycles.  If your hard drive is almost full, then the same sectors will get re-written faster, making them fail faster.  SSD capacities are determined giving some lee-way (a 100GB drive might have 120GB of space) for sector failures.

When using SSD's you should make sure your drives never become full - always have lots of free space.

[theymos]

I improved modSettings. It's only updated once every few minutes now instead of once every 5-10 seconds. Hopefully that'll make things faster.

[Kouye]

I improved modSettings. It's only updated once every few minutes now instead of once every 5-10 seconds. Hopefully that'll make things faster.

I'm sorry to be the bad news messenger, but it has not improved anything on my side.
I also got a few whole blank pages, with no error, but this was just before your post, so I'm guessing it's related to your fix deployment.

[btcton]

I improved modSettings. It's only updated once every few minutes now instead of once every 5-10 seconds. Hopefully that'll make things faster.

I'm sorry to be the bad news messenger, but it has not improved anything on my side.
I also got a few whole blank pages, with no error, but this was just before your post, so I'm guessing it's related to your fix deployment.

Still the same for me.

[MiningBuddy]

I improved modSettings. It's only updated once every few minutes now instead of once every 5-10 seconds. Hopefully that'll make things faster.

I'm seeing much better results

[dwdoc]

Yes.  Quicker today.

[pedrog]

Just read a bunch of pages, smooth surfing!

Much better.

[nottm28]

Yes. Good again. Thanks

[wachtwoord]

I improved modSettings. It's only updated once every few minutes now instead of once every 5-10 seconds. Hopefully that'll make things faster.

I'm seeing much better results

Confirmed