View Topic

As many of you know, the website/forums were down for an extended period of time yesterday into today.

When the site went down, I received an email from our host indicating that site access had been cut due to extremely high CPU use. There was no obvious cause for this since we haven't had a surge in activity or new features or changes, so I had to investigate. Checking the raw access logs, I discovered that bingbot (the crawler for Bing) was accessing a single script dozens of times a minute, ignoring the restrictions for web-crawlers specified in robots.txt (a file that among other things specifies how often bots can access the site). Apparently, instead Bing requires you to create a webmaster tools account with them and from there specify how often it can crawl your website. I think that's shit, so I resolved the issue by banning bingbot from the site entirely. Nobody uses Bing anyways.

I then replied to the ticket requesting access be restored, and when I didn't hear back this morning contacted their support about it again, resulting in the ticket being escalated and access being restored just now.

tl;dr the website was down because bingbot was being a resource hog. bingbot is now banned from the site.

Good old Bing, continuing to show why it's both useless and unwanted.

So it wasn't the Chinese government? 

So it wasn't the Chinese government? 

Nope, it was Redmond.

Nobody uses Bing anyways.

I feel like this ended up being the message you were trying to convey, lol.

I feel like this ended up being the message you were trying to convey, lol.

I literally wrote that in the ticket.

Good afternoon,

After looking at the raw access logs, I believe this is due to bingbot repeatedly accessing a script dozens of times a minute. Nobody uses Bing, so I have blocked the bot in .htaccess, which should resolve an issue (and punish a bad bot for not following robots.txt). Please let me know if there's anything else I need to do to restore site availability.

Thanks,
-Jason

I literally wrote that in the ticket.

I don't know if it's relevant but I feel like this might have some bearing on the site as it relates to robots.txt: https://searchengineland.com/google-to-stop-supporting-noindex-directive-in-robots-txt-319003

Thanks for the concern!

I'm fine with crawlers indexing the site, so long as they do it responsibly. I'm not sure why Bing is so attracted to a single page that it's trying to pull it dozens of times a minute (even a full day after it's been blocked), but we've never had any trouble with other crawlers. I don't think Google's changes should be a problem.

Yeah, I mean crawlers are exceptionally typical on any forum.  They're usually just much more discreet.

Thanks for the concern!

I'm fine with crawlers indexing the site, so long as they do it responsibly. I'm not sure why Bing is so attracted to a single page that it's trying to pull it dozens of times a minute (even a full day after it's been blocked), but we've never had any trouble with other crawlers. I don't think Google's changes should be a problem.

I can't help but be curious about this. What page was it so desperately accessing?

Thanks for the concern!

I'm fine with crawlers indexing the site, so long as they do it responsibly. I'm not sure why Bing is so attracted to a single page that it's trying to pull it dozens of times a minute (even a full day after it's been blocked), but we've never had any trouble with other crawlers. I don't think Google's changes should be a problem.

I can't help but be curious about this. What page was it so desperately accessing?

  Clearly it really wanted into the Adult Forum.  But it needs to fill out the paperwork first...we don't make exceptions for bots or crawlers.

So it wasn't the Chinese government? 

Nope, it was Redmond.

Same thing.