Where do I start?

As most of you know… we had severe down time last week that presented a challenge unlike anything we’ve experienced to date. I want to go into that in a bit, but first, let me address the communication failure.

As the site administrator and owner, it’s my responsibility to communicate with each of you when something as significant as this happens. While I wasn’t personally aware of the “source” of the problem for several days, I understand it is my duty to let you know what is happening and I failed to do that properly.

Where the communication went wrong

Unfortunately, we haven’t kept a list of emails for blogivists. I’m not really sure why we never went through the database and did a proper address export, but the fact is… we never did. So, because of this, I didn’t have your email addresses when we were hit two weeks ago. I had an old list, but constant contact gave me problems because there were a LOT of bouncebacks and spam reports on the last blogivists message we sent out. That list was scrapped.

In short, I had no way to make an announcement. Like you, I couldn’t get into the site, and I couldn’t get email addresses from it either.

By the time the problems hit critical mass, we were getting pummeled by emails about blogivists and just couldn’t keep up.

But the reason is not really relevant at this point. The result is. And the result is that we didn’t properly communicate with you, our bloggers, and this caused a lot of problems within the community.

For that, I personally want to apologize. You have my word… it won’t happen again.

The problem explained

Running a website as large as blogivists is no easy task. And, unlike organizations that are well funded and overstaffed, we’re an all volunteer team with zero funding. We pay for the servers that host blogivists out of pocket, and when a huge problem comes along, we have to pay developers to fix it. I’m not suggesting that we’re not capable of getting problems solved, but I will admit that solutions to problems as large as the one we recently experienced will take longer to find for us.

That’s the bad news. The good news is that we’ll always eventually be able to find the solution.

Anyway… so over the past few months we’ve seen a tremendous rise in spam blogs. At the same time this was happening, our volunteer staff was dealing with several large scale campaigns such as TaxDayTeaParty.com and a few other time consuming efforts. A few weeks ago, when we were at our weakest with regards to time dedication to blogivists, the site got hit with thousands of spam blogs.

The spam blogs were being created so quickly that it eventually overwhelmed the server, shutting down the entire system. Allen Fuller had to immediately freeze the site to get the database interaction and memory use under control before the server would/could be damaged.

This is why everything went dark. And this is why we couldn’t access anything.

The solution

First, we had to get a bigger server with more RAM. The blogivists database is so large that the site will require an entire server just for itself. So we had to pay for a brand new server, and buy extra RAM, before we could bring the site back online.

Allen Fuller and his team, with the help of Lennie Jarratt, carefully moved all of the data over to the new server, and we believe the site is finally starting to stabilize. There are still some slow moments and there may be a few glitches here and there, but we’re working on them and we hope to be running smoothly soon.

The future

We now have a brand new server that is fitted just for blogivists. We’re currently having a custom design built for the new homepage, and we’re going through to delete all spam blogs and implement new spam fighting features.

Also, we’re hoping to be able to hire an intern who can work with the community and help us know what needs to be changed/fixed/implemented.

It was unrealistic for us, as volunteers, to assume we would be able to manage a large community like blogivists as a 100% volunteer effort. We understand this is a problem, and we’re trying to work out a few ways to address the situation and make sure we’re better positioned for system problems moving forward.

I hope you’ll continue blogging with us here at Blogivists.com. Please know that even though there have been some challenging times in the past, we’re working hard to make things better and strengthen the community.

-Eric Odom
Blogivists.com