My (donated) server got an upgrade, so creating the top-navigation-bar should be a bit faster now. It's still a lot of data to edit though.
Amazingly, since my upgrade to a faster server, updating the top-navigation-bar takes a lot longer. It's as if it only uses half a CPU core, it now causes the server load to go to 0.50 instead of 1. I'm still ill, but once I'm okay again (which takes a lot longer than expected), I'll add multithreading to the navigation-update.
So... there's an update on this: I think I fixed it
When I started this project, I had Trust data from a few weeks. So when I added the navigation bar, creating them all from scratch every week didn't take too long. By the time I had about 50 weeks and about 600,000 pages to update, I started caching temporary files to speed things up. When I got the first Xeon server (4 cores), it got a bit faster again.
But, by now I have data from 250 weeks, and close to 3.5 million pages, and there were still some old "loops" left to go through all of them: that meant that one loop had almost 1 billion operations. The newer and faster Xeon server (6 cores) should have been faster per core, but my script got slower. It now took almost 4 full days to update the top-navigation-bar on all pages, and it took the better part of Saturday to update navigation on the latest week.
So... time to overhaul this old spaghetti code. I got rid of the recursive loops, and now it scales more or less linear instead of taking exponentially more computing time. I also added multithreading, so it uses all CPU cores. There are still some time delays because all this happens on a different server than the one serving loyce.club, but I expect the navigation bar to be added to the latest week within an hour. At the same time it updates the navigation on the 6 most recent weeks. After that, the real fun starts and it crunches through the older data in 4 hours.
Part of my update means I no longer need to cache weekly
temp files (it got to 1.9 GB and growing), which makes things easier but came with a small performance penalty. All in all I expect it to take about 4 hours now instead of 4 days, and most users shouldn't even notice it anymore as it only means "week 5" will only link up to "week 250" and not "week 251" until the update is done.
If anything fails, let me know. I'll use my new setup for the first time starting on Saturday.
Update: it's not working yet
Update 2: I think I got it now
