Forum down

[Caras pet]

Is it just me?


# 504 Gateway Timeout ERROR

## The request could not be satisfied.

---

We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner. 
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation. 

---

Generated by cloudfront (CloudFront) HTTP3 Server
Request ID: ...


I get this on a regular basis, the forum is unreachable for hours when this happens.

[teanndaorsa]

Hmm, I'll take a look at the logs, but it does suggest the server is getting itself into a state and then fixing itself after a while. Thanks for letting me know; I'd be interested how many other folks are seeing this.

[bentbliss]

I've seen this message early and there have been a few times over the past week where it wasn't accessible and had a Cloudflare message. That one happened with another site as well.

[teanndaorsa]

I suspect the other site is just a coincidence.

I've just watched the Plaza (or at least the dynamic parts of it) be down for an hour or two. It doesn't appear to be the server being overloaded, or even properly offline, it just gets itself into a state where PHP requests take too long and the CDN that sits in front of the site gives up waiting. So I suspect either a misconfiguration on my part, or sporadic attacks that hog all of the connections to the PHP part without actually making it work hard. I say attacks because I've enabled some more detailed analysis that breaks down the traffic by country, and it's showing Brazil and Iraq as generating four times as much traffic between them as the US does, when I know that typically the US is our biggest visitors by far. And even with the basic monitoring looking back I can see this is something that has crept up in frequency since January.

I'm afraid there's no quick fix here though, because I don't want to just block those two countries, and there isn't an easy configuration change (that I know of) that I can make that will address the root cause. I've something I can try which may make the site more efficient and less vulnerable to the problem, but I won't have the time to test it properly until the weekend or some time next week. So you'll have to bear with it for now, sorry.

[Caras pet]

Thanks for investigating the issue.

Yes, the server is just fine when the forum is down, at least I can ping it and get stable responses.

The PHP logs didn't contain any clues?

[edit]
Forum was down again as I tried to post a reply; looked like the reply wasn't sent, preview didn't work either.
Just checked: The reply was indeed saved, so the page just couldn't reload itself afterwards - maybe that helps?

[rbbrlvr]

Forum has been up and down for me ongoing for about a week or so.

[Eido]

I've had similar issues, as recently as 26 May, USA date. The problems have been the inability to connect and also long time delay connecting. They have been lessening in frequency recently.
Eido

[SwitchMan]

I have also experienced this in the last week or so.

[teanndaorsa]

Right, so to give folks an update, I've been digging in various ways to diagnose the underlying issue. That will have increased the flakiness of the site a little but there's no denying that there are big swathes of time where it's effectively offline anyway so it's probably not noticeably worse for me breaking it here and there for 5-10m at a time.

For those who don't care about the technical aspects, the tl;dr is that the part of the server that handles the forum, searching, and the index pages, is going into a broken state for hours at a time, usually around early afternoon, UTC. Because of the way the site is set up, the index pages and most of the recent stories will still seem perfectly accessible, that's a deliberate design choice and it makes us seem more robust than we are. But you can't do that with the forum or search functionality, so it's there we're obviously offline. It's not a simple broken site, nor is it something we've recently changed. To stop it from happening, without just throwing money at the problem, I first need to understand the root cause, and that is taking some time.

For a more technical perspective: it looks for all the world like this is fundamentally a capacity issue, once the server reaches a certain level of load, instead of just responding more slowly, it tips over into a state where every request times out and gives an error, and only recovers once load goes away to almost nothing. My theory is that is happening more regularly now because we're being hit with a lot more traffic these days, and that the point in the day where it tips over is when we start to get busy.

I don't want to simply throw more power at the problem because a) that costs more, and b) I don't believe that the increased load is actually more people visiting. A large number of the requests appear to be from bots of various flavours, though it is impossible for me to tell if that proportion has been growing over time. But I am pretty sure that there aren't hundreds of real people making requests every second, which is what the logs suggest at peak times, just like I'm sure that we don't suddenly have a massive following in Iraq.

Instead I am focusing on correcting the behaviour of the server so that it slows down rather than tipping over, and I am looking into how effective blocking some of the bots might be. If it turns out this is just what it's going to be like running a Simple Machines Forum site, and it's constantly under attack by bots sniffing for weaknesses, then I may have to separate out the forum from the site functionality and just let it degrade while it's under attack. As much as the authors enjoy getting feedback, I don't want the costs of running the site to end up dominated by the costs of running the forum.

[Caras pet]

Thanks for your efforts and the info.

I'm curious to see what solution you can come up with - maybe some kind of rate limit per IP?

In my subjective opinion, the problem is not as frequent at the moment as it was at the beginning?