View Topic

As you may know, last night our webhost temporarily shut us down due to extremely high CPU usage after the Summersend spam wars game opened up. In the few hours that it was open, the game produced 1400 posts and at the time we were shut down was using 80% of the server CPU and around 120MB of bandwidth per minute.

Since then, I have been monitoring our CPU usage, and even with the game closed our usage has been borderline, as you can see here:

Because of this, I've started looking at contributing factors and have found that it's likely that the issue wasn't the spam game itself, but the heavy load required to display the topic pages, which only became apparent during the spam game because they were being loaded so quickly.

Specifically, I've found two particular issues:

1) HTTP Images: When we switched to using https, we started using SMF's image proxy to make sure all images were served as https. This means for images in avatars, posts, and signatures, if it uses a http url the site downloads it, caches it, and serves it to you as https. However, this creates overhead depending on the size of the image, and some images were using upwards of 4MB of bandwidth per image per page load.

The solution is to simply switch those http urls to https, and I've already done so for some avatars and sigs that I founded to be particularly large. It's likely in the future that http image links will no longer be accepted for avatars or [img] tags to minimize use of the proxy.

This issue probably needs to be revisited sometime in the future,

2) Notifications: Every time you load a page on the website, it pulls the full list of notifications you haven't dismissed along with the graphic for dismissing them, and for some people (including me) this amounts to hundreds of notifications. I have dismissed all my notifications personally, but in the future I will make it so only the latest 25 or so notifications will appear, with a link to see all of them on a separate page.

3) Bots: At any given time, you will see that we usually have many more guests than registered members on the forums, and usually most of them are bots such as search engine web-crawlers. These bots use as many resources as an actual person viewing the website, and having many of them crawling around the forums can create quite drain even if actual forum activity is low. I have instituted a robots.txt file limiting crawlers to 1 action every 20 seconds. If this doesn't help, I may have to look into banning certain bots via ip address.

It will be several hours before I know if the preliminary measures I have taken will have an effect using our webhost's tools, but if they do, I will try to implement the other measures tonight and have Spam Wars back up tomorrow.

Ah, thanks for the information!

1. In the past, https images were not accepted by the forum and would display a blank image. I'm guessing this has been fixed?

2. Oh crap, sorry, I was lazy and never dismissed my notifications. There must have been thousands... I see that you've dismissed all my notifications except the last 25, thanks! And I also see that the Dismiss button is no longer an image but rather text, probably a good idea.

And now that I think about it, that massive number of notifs probably contributed to much of the lag I sometimes experienced when browsing Wintreath...

3. Will this robots.txt still continue to allow search engine indexing? (For me personally, it's quite convenient to search Wintreath via Google, lol.)

Ah, thanks for the information!

1. In the past, https images were not accepted by the forum and would display a blank image. I'm guessing this has been fixed?

2. Oh crap, sorry, I was lazy and never dismissed my notifications. There must have been thousands... I see that you've dismissed all my notifications except the last 25, thanks! And I also see that the Dismiss button is no longer an image but rather text, probably a good idea.

And now that I think about it, that massive number of notifs probably contributed to much of the lag I sometimes experienced when browsing Wintreath...

3. Will this robots.txt still continue to allow search engine indexing? (For me personally, it's quite convenient to search Wintreath via Google, lol.)

1) Yeah, it was fixed in the updates that allowed us to switch to https.

2) I made it so that only the last 25 notifications display, but I'll add a button to mass-dismiss notifications soon. And actually yeah, I notice that the forum actually seems to load faster now. Before it could take as much as 8MB of bandwidth for a topic page to download, with notifications, avatars, sigs, etc, and I'm sure that was killer for mobile data plans. >_>

3) It doesn't deny access to index the forums, just specifies that they should do so more slowly. I don't think Googlebot follows that particular directive, anyways.

And actually yeah, I notice that the forum actually seems to load faster now. Before it could take as much as 8MB of bandwidth for a topic page to download, with notifications, avatars, sigs, etc, and I'm sure that was killer for mobile data plans. >_>

Yep, definitely feels much faster now.

And I think a certain iOS adblocker was getting confused by all the notifications and blocking the site from loading at all, and from what I can see this longer appears to be an issue on my phone, which is great.

Update 2
After the last update, I noticed that our bandwidth usage had significantly gone down, but our CPU usage was still borderline, so I worked late into the night last night (until 5am) to narrow down and fix or optimize issues. Specifically, I fixed any errors that showed up in the error logs, fixed an issue where the NSAPI script to check if nations in the recruitment database were open to recruitment messages ran every time it was checked rather than every minute, and condensed the query that pulled the names of people who liked a post from running once for each post to once per topic page.

Today I noticed these steps had made a decent impact, so I implemented the 30 second wait requirement for Spam Wars and reopened it.

This is where I promote the donation page/Patron.
Spam Wars might seem like a simple game, but I've spent nearly 11 hours between actually creating the game and fixing the issues that contributed to it getting our site shut down. However, I'm happy that it has proven to be so popular, because I'd like to believe that there's a future for both Wintreath and myself personally in building unique games that people have fun with, and I'd like to work on more games in the future after Summersend is over.

If you're interested in things going in this direction and you're able to, consider checking out the donation page and sending in a donation or becoming a Patron! The money raised pays for Wintreath's hosting costs first and foremost, but it also helps justify the time that I spend on things like this. We even have a $1 Patron tier now, and every little bit helps! Thank you for your consideration!

NOOOO 30 SECONDS  


Just kidding it needed it

Also I think there's a bug:  After I get the 30 seconds warning, I think that the 30 seconds timer resets...

And here I thought it was part of the game to trick your opponent into resetting their 30s timer

I was playing Civ 5 yesterday and spamming in the steam overlay browser. The Steam overlay clock counts seconds, so I used that to keep track of when to post next.

I was using my computer's clock to do it.  While simultaneously trying to do homework, guess which one was more successful...

Another possible bug: does someone posting before you reset the timer as well?

Another possible bug: does someone posting before you reset the timer as well?

In my experience it hasn't. However, someone posting before you when you're on the post page will trigger the new posts have been made message, which will require you to submit the post again.

Another possible bug: does someone posting before you reset the timer as well?

In my experience it hasn't. However, someone posting before you when you're on the post page will trigger the new posts have been made message, which will require you to submit the post again.

Yeah, it's just that the submitting again led to the 30 second warning. But I think it might just be that I hadn't reached the 30 seconds yet, or had reset it earlier.