Emergency Maintenance 5 Jun 11

[wolf_bluejay]

Hmmm, just swo you know -- I don't think I'm the only other IT related person on here that would offer free advice

I do work in the public sector or things, so I don't do consulting anymore -- which translates to some reasonable advise without trying to make a buck.....


Anyways, I currently run a pretty large server setup (about 10 racks full at the moment). As we ONLY run open source solutions mostly due to cost, the solutions are kinda neat.

The first off advise is this: If you are running your own servers, virtualize. This allow you to expand, move things, and as well as not waste on extra server until you really really need it.

My current new and shiny setup -- pushing about 100G daily.
8 VM's total 8 cores/16 GB RAM
1) squid as a reverse proxy,load balancer -- handles 50% of the load at about 5% of 1 CPU, 256 MB RAM
2) bulk storage using NFS for load balanced apache
3,4,5) matching apache web servers
6) memcache server 128Mb RAM less tiny CPU usage
7) Mysql server -- on fast disk, everything tweaked for fast DB access
8) mysql slave -- allows read only access, and backup run from here without slowing down sites

Backups --- we have a MASSIVE backup system, but it runs "rsbackup". we use ZFS on FreeBSD, with de-duplication, and filesystem level snapshots.
The backup server calls to the NFS/DB servers and does an rsync that pulls only the changed data, then does a snapshot, then replicates the backs to a second FreeBSD box in a separate data center. Files are stored as a copy of the file system and are compressed by ZFS filesystem.
Backups are small on size (only stores the diff of the files), snapshots happen in a few seconds, and we can roll any server back to any snapshot time. In a few cases the backups happen about every 10 min or so.
Of course the backup server holds about 40Tb of storage, but it backs up about 90 servers as a full backup daily for about 1 year. (these cost us about $4000 each, but are a 5U case).

Whew -- that is what I get for being a linux geek.
BTW -- your slow transfers were due to SSH slow downs -- It doesn't perform well over long high speed links -- look into the HPN patches (10X speed).

Originally Posted by titus

Hello,

Just for the record, this past weekend's fiasco was totally embarrassing. This has to be the longest outage we have sustained, despite being an announced planned outage with data loss.

Ultimately it's the hosting company's fault in not being able to get the RAID array built despite having replaced a faulty drive on the old server, and basically required me to enter into another year's contract just to get a server built but didn't follow through to assist me in transferring unless I chased and called and chatted.

Yet having said that, I did make a rather embarrassing mistake (not going to comment further here) resulting in about 18 hrs of data loss, followed by having to rebuild two servers from scratch. That was the easy part.

Then there was an attempt to transfer the web server across and believe me, we tried just about everything under the radar from using normal scp, fast scp, mounting eSATA, direct cross cable connection, etc but each time it's crawling at an average rate of 3Mb/s. Now given the server is about 250GB one can imagine the time it's going to take.

In the end, I rebuilt it as well. It turned out had I made the decision to rebuild, I could have brought the server back up by the target time at 18:00 PST on Sunday. Yes 3 servers rebuilt by hand. All in all not bad in terms of recovery time given everything has to be re-installed. I did get save somewhat because I have multiple back ups available on hand.

Anyway, as a result of this, I'm going to look seriously into taking the data protection and service resiliency to another level. There are a lot of work to do for this as there are many solutions available. Do we use two sets of servers sharing a SAN, do we use an additional server doing CDP, do we use two sets of servers with web load balanced and database replication, etc. This needs to tie into a plan that I have to use mem cache as well. Other considerations would be how to address uploaded files appearing on both web servers if we go that route.

So this will side track on the recent efforts on the reference library and the vBulletin 4 and wiki upgrade.

One step at a time.

Titus