Pages: [1]

Concerning Forum Downtime
Posts: 12 Views: 924

Wintermoot
  • Regional Stability Squad
  • The Greyscale Magi-Monk
  • As many of you know, the website/forums were down for an extended period of time yesterday into today.

    When the site went down, I received an email from our host indicating that site access had been cut due to extremely high CPU use. There was no obvious cause for this since we haven't had a surge in activity or new features or changes, so I had to investigate. Checking the raw access logs, I discovered that bingbot (the crawler for Bing) was accessing a single script dozens of times a minute, ignoring the restrictions for web-crawlers specified in robots.txt (a file that among other things specifies how often bots can access the site). Apparently, instead Bing requires you to create a webmaster tools account with them and from there specify how often it can crawl your website. I think that's shit, so I resolved the issue by banning bingbot from the site entirely. Nobody uses Bing anyways.

    I then replied to the ticket requesting access be restored, and when I didn't hear back this morning contacted their support about it again, resulting in the ticket being escalated and access being restored just now.

    tl;dr the website was down because bingbot was being a resource hog. bingbot is now banned from the site.


    I went all the way to Cassadega to commune with the dead
    They said "You'd better look alive"
    Wintermoot
    • The Greyscale Magi-Monk
    • Posts: 19,453
    • Karma: 9,677
    • Weather: ❄️
    • Regional Stability Squad
    • Pronouns
      He/Him/His
      Orientation
      Demisexual
      Wintreath Nation
      Logged
    Michi
  • Regional Stability Squad
  • Level 167 Caticorn God of Destruction
  • Good old Bing, continuing to show why it's both useless and unwanted.
    6 people like this post: Treeguard, Marzipan, taulover, _zM, Wintermoot, Red Mones
    My Wintreath Resumé
    Michi
    • Level 167 Caticorn God of Destruction
    • Posts: 7,195
    • Karma: 4,052
    • Wintreath's Official Video Game Enthusiast
    • Regional Stability Squad
    • Pronouns
      Any except it/its
      Orientation
      Michisexual <3
      Familial House
      Valeria
      Wintreath Nation
      Logged
    Treeguard
  • Ambassador
  • Your friendly Forest emissary
  • So it wasn't the Chinese government?  :o
    2 people like this post: Wintermoot, Michi
    “Better a cruel truth than a comfortable delusion.”--Edward Abbey
    Treeguard
    • Your friendly Forest emissary
    • Posts: 11
    • Karma: 18
    • Ex Arboribus, Vita
    • Ambassador
    • Pronouns
      He/Him/His
      Representing
      Logged
    Wintermoot
  • Regional Stability Squad
  • The Greyscale Magi-Monk
  • So it wasn't the Chinese government?  :o
    Nope, it was Redmond. :P
    1 person likes this post: Treeguard


    I went all the way to Cassadega to commune with the dead
    They said "You'd better look alive"
    Wintermoot
    • The Greyscale Magi-Monk
    • Posts: 19,453
    • Karma: 9,677
    • Weather: ❄️
    • Regional Stability Squad
    • Pronouns
      He/Him/His
      Orientation
      Demisexual
      Wintreath Nation
      Logged
    Emoticonius
  • Former Citizen
  • Supreme Grand Admiral of The Emoticonian Army
  • Nobody uses Bing anyways.
    I feel like this ended up being the message you were trying to convey, lol.
    3 people like this post: Red Mones, Michi, Wintermoot
    “I support anyone’s right to be who they want to be. My question is: to what extent do I have to participate in your self-image?” - Dave Chappelle
    7:42 PM <Govindia> eh, i like the taste of nuts in my mouth



    Potential clients should PM or Query Emoticonius
    Emoticonius
    • Supreme Grand Admiral of The Emoticonian Army
    • Posts: 993
    • Karma: 453
    • Crasher Endorsed, Native Approved
    • Former Citizen
    • Pronouns
      He/Him/His
      Familial House
      Kaizer
      Representing
      Wintreath Nation
      Logged
    Wintermoot
  • Regional Stability Squad
  • The Greyscale Magi-Monk
  • I feel like this ended up being the message you were trying to convey, lol.
    I literally wrote that in the ticket.

    Quote
    Good afternoon,

    After looking at the raw access logs, I believe this is due to bingbot repeatedly accessing a script dozens of times a minute. Nobody uses Bing, so I have blocked the bot in .htaccess, which should resolve an issue (and punish a bad bot for not following robots.txt). Please let me know if there's anything else I need to do to restore site availability.

    Thanks,
    -Jason
    5 people like this post: Emoticonius, Gerrick, _zM, taulover, Michi


    I went all the way to Cassadega to commune with the dead
    They said "You'd better look alive"
    Wintermoot
    • The Greyscale Magi-Monk
    • Posts: 19,453
    • Karma: 9,677
    • Weather: ❄️
    • Regional Stability Squad
    • Pronouns
      He/Him/His
      Orientation
      Demisexual
      Wintreath Nation
      Logged
    Emoticonius
  • Former Citizen
  • Supreme Grand Admiral of The Emoticonian Army
  • I literally wrote that in the ticket.
    I don't know if it's relevant but I feel like this might have some bearing on the site as it relates to robots.txt: https://searchengineland.com/google-to-stop-supporting-noindex-directive-in-robots-txt-319003
    2 people like this post: Wintermoot, taulover
    “I support anyone’s right to be who they want to be. My question is: to what extent do I have to participate in your self-image?” - Dave Chappelle
    7:42 PM <Govindia> eh, i like the taste of nuts in my mouth



    Potential clients should PM or Query Emoticonius
    Emoticonius
    • Supreme Grand Admiral of The Emoticonian Army
    • Posts: 993
    • Karma: 453
    • Crasher Endorsed, Native Approved
    • Former Citizen
    • Pronouns
      He/Him/His
      Familial House
      Kaizer
      Representing
      Wintreath Nation
      Logged
    Wintermoot
  • Regional Stability Squad
  • The Greyscale Magi-Monk
  • Thanks for the concern!

    I'm fine with crawlers indexing the site, so long as they do it responsibly. I'm not sure why Bing is so attracted to a single page that it's trying to pull it dozens of times a minute (even a full day after it's been blocked), but we've never had any trouble with other crawlers. I don't think Google's changes should be a problem.
    2 people like this post: taulover, Michi


    I went all the way to Cassadega to commune with the dead
    They said "You'd better look alive"
    Wintermoot
    • The Greyscale Magi-Monk
    • Posts: 19,453
    • Karma: 9,677
    • Weather: ❄️
    • Regional Stability Squad
    • Pronouns
      He/Him/His
      Orientation
      Demisexual
      Wintreath Nation
      Logged
    Michi
  • Regional Stability Squad
  • Level 167 Caticorn God of Destruction
  • Yeah, I mean crawlers are exceptionally typical on any forum.  They're usually just much more discreet.
    My Wintreath Resumé
    Michi
    • Level 167 Caticorn God of Destruction
    • Posts: 7,195
    • Karma: 4,052
    • Wintreath's Official Video Game Enthusiast
    • Regional Stability Squad
    • Pronouns
      Any except it/its
      Orientation
      Michisexual <3
      Familial House
      Valeria
      Wintreath Nation
      Logged
    Emoticonius
  • Former Citizen
  • Supreme Grand Admiral of The Emoticonian Army
  • Thanks for the concern!

    I'm fine with crawlers indexing the site, so long as they do it responsibly. I'm not sure why Bing is so attracted to a single page that it's trying to pull it dozens of times a minute (even a full day after it's been blocked), but we've never had any trouble with other crawlers. I don't think Google's changes should be a problem.
    I can't help but be curious about this. What page was it so desperately accessing?
    “I support anyone’s right to be who they want to be. My question is: to what extent do I have to participate in your self-image?” - Dave Chappelle
    7:42 PM <Govindia> eh, i like the taste of nuts in my mouth



    Potential clients should PM or Query Emoticonius
    Emoticonius
    • Supreme Grand Admiral of The Emoticonian Army
    • Posts: 993
    • Karma: 453
    • Crasher Endorsed, Native Approved
    • Former Citizen
    • Pronouns
      He/Him/His
      Familial House
      Kaizer
      Representing
      Wintreath Nation
      Logged
    Michi
  • Regional Stability Squad
  • Level 167 Caticorn God of Destruction
  • Thanks for the concern!

    I'm fine with crawlers indexing the site, so long as they do it responsibly. I'm not sure why Bing is so attracted to a single page that it's trying to pull it dozens of times a minute (even a full day after it's been blocked), but we've never had any trouble with other crawlers. I don't think Google's changes should be a problem.
    I can't help but be curious about this. What page was it so desperately accessing?
      Clearly it really wanted into the Adult Forum.  But it needs to fill out the paperwork first...we don't make exceptions for bots or crawlers.
    1 person likes this post: Red Mones
    My Wintreath Resumé
    Michi
    • Level 167 Caticorn God of Destruction
    • Posts: 7,195
    • Karma: 4,052
    • Wintreath's Official Video Game Enthusiast
    • Regional Stability Squad
    • Pronouns
      Any except it/its
      Orientation
      Michisexual <3
      Familial House
      Valeria
      Wintreath Nation
      Logged
    Treeguard
  • Ambassador
  • Your friendly Forest emissary
  • So it wasn't the Chinese government?  :o
    Nope, it was Redmond. :P
    Same thing.  :P
    “Better a cruel truth than a comfortable delusion.”--Edward Abbey
    Treeguard
    • Your friendly Forest emissary
    • Posts: 11
    • Karma: 18
    • Ex Arboribus, Vita
    • Ambassador
    • Pronouns
      He/Him/His
      Representing
      Logged
     
    Pages: [1]