Exchange 2010 Update Rollup 6
Those of us who have bought in to the Microsoft experience also buy in to the regular updating of our servers and the software that runs on them.
Updating is a good thing and needs to be done regularly, but for many, myself included, it can be an incredibly stressful experience after so many failed updates and upgrades, roll back and long nights recovering from a bad patch or uppdate, not just to Microsoft products but also to others applications, drivers and in the case of EqualLogics firmware. One bad update can bring hours or days of fun.
Down to the nitty, gritty of the problem. We have an Exchange 2010 system which has taken around 12 months to get to an almost usable state. The main problem has been the way our Domain is made and the requirement to set up a very resilient infrastructure spread across 4 sites in the UK.
The root .com domain is in the US and there are 5 sub-domains for each country below that. Each country is fairly autonomous and to be honest if I had my time again I would go for a multi domain model with trusts between. Still it is what it is. The US and Canada domains runs Exchange 2007, Spain runs 2003 and Australia/NZ moved to 2010 at the same time as us but settled for a much simpler set up, however their choices have affected in many ways.
Trying to get everything up to date before we start moving users mailboxes, we installed Roll up 6. this went well on the 8 CAS/HUB servers and on 6 of the mailbox servers but the two mailbox servers on one site it failed with 1603 error.
The patch would run and get to the bit where it was stopping services then roll back. No error message just a 1603 in the event log.
We reached for Google and came up with a load of possibilities. Dotnet needed to be running, Offload on the NICs should be turned off, run from elevated command prompt, run with logging enabled, and many other bits that we tried. The Log file didn't show much either. All to no avail.
Time to get Microsoft to earn their money. The call went in and two days were spent with Microsoft picking apart the servers and they tried everything we had and also turning off UAC, checking AV etc but still no joy, they couldn't find anything in the install log to give a clue either.
Some more ordinary Windows patches came along and installed with out a hitch but Roll up 6 was a no show. Now it was only about 6 weeks ago we installed Roll up 5 so what had gone wrong. We also could not uninstall Roll up 5, we got the same error.
We had previously found a problem with DNS, this is the DNS server listed on the NIC not the one you name in some bit of Powershell. The problem we had was that the DNS service was running but not happy and need to be crashed out and restarted, this was causing issues with routing, adding mailboxes and all sorts of weird issues. A restart fixed the problem that time but it was not this, however putting another DNS server as the primary and hey presto the patch went on.
Job 1 done but now what was wrong with DNS on this server? DCDiag came back with some missing SRV records for the server on the domain. I checked fro these and could clearly see them, there was also an error for the root domain. I checked this on the server and it errored, I couldn't get it to load, so this was struggling to provide links to the root domain which is probably a bad thing for Exchange. I deleted the domain and added it back in which fixed the one problem but still left me with the SRV record error.
So I started digging around all records in the root domain and I found there was duplicate entry in the _msdcs of the root domain.
This was probably a hang over from when this DC was rebuilt after the server crashed and burned, yes we ran the metadata clean up and waited 2 days checked everything to do with AD but there you go. It could also explain the problems we have been seeing generally with DNS. I have asked the US to get the duplicate entry for the old server removed and some other things I found to be tidied up. Hopefully this will fix the problem but I suspect we may have to kill this DC and build another one with a different name, though that has consequences with our VMS systems and Samba so is not something we can just do.
MS were grateful for us finding the solution to this and it goes to show once again how heavily Windows now relies on DNS.
Just need to SP2 now, ho hum.
Updating is a good thing and needs to be done regularly, but for many, myself included, it can be an incredibly stressful experience after so many failed updates and upgrades, roll back and long nights recovering from a bad patch or uppdate, not just to Microsoft products but also to others applications, drivers and in the case of EqualLogics firmware. One bad update can bring hours or days of fun.
Down to the nitty, gritty of the problem. We have an Exchange 2010 system which has taken around 12 months to get to an almost usable state. The main problem has been the way our Domain is made and the requirement to set up a very resilient infrastructure spread across 4 sites in the UK.
The root .com domain is in the US and there are 5 sub-domains for each country below that. Each country is fairly autonomous and to be honest if I had my time again I would go for a multi domain model with trusts between. Still it is what it is. The US and Canada domains runs Exchange 2007, Spain runs 2003 and Australia/NZ moved to 2010 at the same time as us but settled for a much simpler set up, however their choices have affected in many ways.
Trying to get everything up to date before we start moving users mailboxes, we installed Roll up 6. this went well on the 8 CAS/HUB servers and on 6 of the mailbox servers but the two mailbox servers on one site it failed with 1603 error.
The patch would run and get to the bit where it was stopping services then roll back. No error message just a 1603 in the event log.
We reached for Google and came up with a load of possibilities. Dotnet needed to be running, Offload on the NICs should be turned off, run from elevated command prompt, run with logging enabled, and many other bits that we tried. The Log file didn't show much either. All to no avail.
Time to get Microsoft to earn their money. The call went in and two days were spent with Microsoft picking apart the servers and they tried everything we had and also turning off UAC, checking AV etc but still no joy, they couldn't find anything in the install log to give a clue either.
Some more ordinary Windows patches came along and installed with out a hitch but Roll up 6 was a no show. Now it was only about 6 weeks ago we installed Roll up 5 so what had gone wrong. We also could not uninstall Roll up 5, we got the same error.
We had previously found a problem with DNS, this is the DNS server listed on the NIC not the one you name in some bit of Powershell. The problem we had was that the DNS service was running but not happy and need to be crashed out and restarted, this was causing issues with routing, adding mailboxes and all sorts of weird issues. A restart fixed the problem that time but it was not this, however putting another DNS server as the primary and hey presto the patch went on.
Job 1 done but now what was wrong with DNS on this server? DCDiag came back with some missing SRV records for the server on the domain. I checked fro these and could clearly see them, there was also an error for the root domain. I checked this on the server and it errored, I couldn't get it to load, so this was struggling to provide links to the root domain which is probably a bad thing for Exchange. I deleted the domain and added it back in which fixed the one problem but still left me with the SRV record error.
So I started digging around all records in the root domain and I found there was duplicate entry in the _msdcs of the root domain.
This was probably a hang over from when this DC was rebuilt after the server crashed and burned, yes we ran the metadata clean up and waited 2 days checked everything to do with AD but there you go. It could also explain the problems we have been seeing generally with DNS. I have asked the US to get the duplicate entry for the old server removed and some other things I found to be tidied up. Hopefully this will fix the problem but I suspect we may have to kill this DC and build another one with a different name, though that has consequences with our VMS systems and Samba so is not something we can just do.
MS were grateful for us finding the solution to this and it goes to show once again how heavily Windows now relies on DNS.
Just need to SP2 now, ho hum.
Comments
Post a Comment