The Joy of DR
Everyone hates it but it is a necessary evil that we have to go through at least once a year, some twice others, god help their minds, several times.
In previous years we have had a truck turn up with a roof that went up to create a two story office, a bit like a low rent Formala 1 paddock truck with a bad paint job. You had the office upstairs and a bunch of racks with a HP EVA SAN and a shed load of servers of your choice and some tape libraries. We would get our backup tapes and recover as many systems as possible pretending we were doing a good job and if it came to it we could recover all our systems within a week, whilst knowing full well we wouldn't have a hope in hell.
Now my life is made a lot simpler. We have visualized as many of our systems as possible, these are all stored on some cheap EqualLogics (compared to EVA and all the licenses) and replicated to a remote site. I now sit at my desk and bring online replicated copies of our VMFS partitions, present to our hosts at the remote site, find the servers and add to inventory, boot up and job's a good'un.
No more sitting in a freezing cold or roasting hot trailler waiting for huge DBs to restore from tape, all servers up and running in short order.
Problems? Well there are always a few flies in the ointment.People keep coming up to my desk and asking me to fix stuff in the real world. A couple of configuration issues, Our DCs were mysteriously unavailable to service logons for 30 minutes (Win 2k8 if any one has any ideas) and some problems getting our physical exchange servers in to the test and err... that's about it.
The exchange issue was sorted by pulling a system drive and P2V it then replicating it to the DR site, this was then booted and joind to the copy of the database which was already on the EqualLogics.
It was so easy we are planing to change the way we do our DR test, instead of doing as many as we can twice a year we will just recover systems, possibly bring them up in the DR site, test and even test patch them before doing it in live.
It has been almost a happy experience rather then the drudgery of previous tests.
We do still have number of issues to resolve not least of which is testing recovery of the servers that remain physical. Exchange will not be a problem once we finish migrating to 2010 as we have a replicated infrastructure, but providing email services in the test DR environment will need some planning. Others will either need the systems owners to front up the external developers and tell them it make or break or some black magic on our part possibly involving backup to disk and replicating that, but it is never as simple as saying it, timing is everything and data sizes can be huge when dealing with backups rather than continual replication of changes.
Next up my focus will be moving to completely overhaulling our backups, at the moment we are still using the old style weekly full and daily differential to tape using Backup Exec remote agents and LTO 3, but my initial feeling is that it would be better to move away from this. I feel we are at the top end of Backup Exec capabilities and to be honest GRT is good when it works and pants when it doesn't, unfortunately it is the latter more often than not. The step up to Netbackup is a big one and could prove too expensive. Other options are Veeam and Quest both of which seem to have some good offerings in the area of VMware.
It will be interesting to see some costs and compare with our current BE maintenance which is not exactly small change.
In previous years we have had a truck turn up with a roof that went up to create a two story office, a bit like a low rent Formala 1 paddock truck with a bad paint job. You had the office upstairs and a bunch of racks with a HP EVA SAN and a shed load of servers of your choice and some tape libraries. We would get our backup tapes and recover as many systems as possible pretending we were doing a good job and if it came to it we could recover all our systems within a week, whilst knowing full well we wouldn't have a hope in hell.
Now my life is made a lot simpler. We have visualized as many of our systems as possible, these are all stored on some cheap EqualLogics (compared to EVA and all the licenses) and replicated to a remote site. I now sit at my desk and bring online replicated copies of our VMFS partitions, present to our hosts at the remote site, find the servers and add to inventory, boot up and job's a good'un.
No more sitting in a freezing cold or roasting hot trailler waiting for huge DBs to restore from tape, all servers up and running in short order.
Problems? Well there are always a few flies in the ointment.People keep coming up to my desk and asking me to fix stuff in the real world. A couple of configuration issues, Our DCs were mysteriously unavailable to service logons for 30 minutes (Win 2k8 if any one has any ideas) and some problems getting our physical exchange servers in to the test and err... that's about it.
The exchange issue was sorted by pulling a system drive and P2V it then replicating it to the DR site, this was then booted and joind to the copy of the database which was already on the EqualLogics.
It was so easy we are planing to change the way we do our DR test, instead of doing as many as we can twice a year we will just recover systems, possibly bring them up in the DR site, test and even test patch them before doing it in live.
It has been almost a happy experience rather then the drudgery of previous tests.
We do still have number of issues to resolve not least of which is testing recovery of the servers that remain physical. Exchange will not be a problem once we finish migrating to 2010 as we have a replicated infrastructure, but providing email services in the test DR environment will need some planning. Others will either need the systems owners to front up the external developers and tell them it make or break or some black magic on our part possibly involving backup to disk and replicating that, but it is never as simple as saying it, timing is everything and data sizes can be huge when dealing with backups rather than continual replication of changes.
Next up my focus will be moving to completely overhaulling our backups, at the moment we are still using the old style weekly full and daily differential to tape using Backup Exec remote agents and LTO 3, but my initial feeling is that it would be better to move away from this. I feel we are at the top end of Backup Exec capabilities and to be honest GRT is good when it works and pants when it doesn't, unfortunately it is the latter more often than not. The step up to Netbackup is a big one and could prove too expensive. Other options are Veeam and Quest both of which seem to have some good offerings in the area of VMware.
It will be interesting to see some costs and compare with our current BE maintenance which is not exactly small change.
Comments
Post a Comment