by, 2012-11-13 at 07:45 AM (6147 Views)
Over the past few months, I've been getting an ever increasing number of emails from my site with errors due to overloading. Fortunately my friend Simon is really good with websites, and is very helpful too, so I've been able to put in some changes that I hope will fix things.
I think it's fair to say that I'm pretty accomplished with Excel, but man... websites are a different matter. Every day I touch this stuff I realize how little I know. Granted, I'm learning all the time, but I've got a LONG way to go before web stuff is as straight forward as Excel to me!
The first thing that we focussed on was banning a great deal of web spiders from the site. For those who don't know, spiders are generally sent out be search engines to "crawl" the web, indexing the pages on websites. This is how Google, Bing, Yahoo and others know which pages to serve up to you during a search. We also see spiders from Facebook, Feedburner (RSS feeds) and other sources. I was always under the impression that spiders were good for websites, but this isn't always the case. Some spiders just cause traffic without any real benefit, and others, while good, are good for a market that you don't serve. Spiders trolling my site and indexing it for an audience that doesn't speak or read English, for example, don't serve me much good. They cause a significant load with very little benefit since none of my content is in their market's native language.
Something else that has been happening here is that I've been experiencing a massive amount of spam registrations. While I do have tools that block registrations for known spammers, I've still had way too many slip the net and post on the site. Granted you may have only seen the one or two that made it through the automatic moderation, but I've been wiping out tons of them behind the scenes.
Now interestingly enough, as I was working through the process of blocking spiders, I also discovered that I'm getting a large amounts of hits from one specific country on a daily basis. And further investigation helped me to the realization that 100% of spam posts are coming from within their borders.
After a lot of investigation, evaulation of spam prevention methods and soul searching, I have made the difficult decision to ban any traffic originating their country to my website. It was not an easy decision to make, believe me. As an MVP, my philosophy is to help people work with technology, no matter their race, skin color or religious beliefs. What unites us is our need to get things done, and I like helping people with Excel. Implementing a country wide block is sure to affect a legitimate user at some point.
At the end of the day, however, I also have to realize that time spent moderating spam and dealing with performance issues to serve a country that has such a small percentage of English speaking natives is counter productive. The reality is that very few people there will be affected, at the cost of everyone else who does actually use my site for it's intended purpose.
While I haven't successfully been able to ban all IP's from their country yet, (the site I used to source my list claims 99.5% accuracy,) I did ban all their know IP ranges a few hours ago. (So if you're reading this, then I'd say your country wasn't the one I've blocked!)
Time well tell how effective this is in the long run, but one thing I can see is that the site performance has already improved drastically.