The Grumpy IT Admin

I do a little bit of scripting now and again, simple stuff to set permissions as part of silent install, copy some files or bulk update AD entries but nothing too drastic, I don't have the right kind of mind for all that, I am a visual guy and work best with images, long lines of text just don't do it for me, unless they can save me a lot of time. Since we moved to Exchange 2010 and due to being involved in VMware, more and more of my time is wasted trying to work out what I am supposed to be typing in to some command window. I’m sure if I spent all day every day making Exchange or VMware bend to my will I would probably get quite used to it and be able to reel off the required string in seconds but in a world where I have to also fix printers, deal with users, set up new server, make Backup Exec work and all the other stuff, trying work out a couple of lines of text with some really poor documentation and unhelpful error messages is not really a good use of my time. I c...

We are struggling with one 3rd party supplier that insist that our iSCSI solution is around 5000% slower than our previous solution. They have also stated that they would not support any speed related issues with the product if we moved the DB on to iSCSI. No just to fill a few details, the old storage was a HP EVA 8000, a big beastie if ever there was one and the current is a Dell EqualLogic which is more utilitarian to say the least. Now anyone with half a brain cell will say the EVA is going to be quicker than the EQ. Any test you throw at the the two solutions will back that up to varying degrees of excess. Now everyone accepts that, so where may you ask is the problem? Well, what we should be asking is not how fast is our choice of storage but is it fast enough and how does it affect the end user experience? If the storage is runs at half the speed but this represents only 1% of the time an application takes to respond to the user then to be honest the user is not really ...

I didn't expect to have to revisit this but my recent DR test has highlighted another issue along the same lines. Trying to recover the same server again and the drives once again were not writable. No problem I know how to fix that. In to Diskpart and ...oh no hidden volumes. What is going on? A quick check and the volumes are set to read only, but are not showing as hidden. A quick look through the list of commands available in Diskpart revealed the Detail command, running Det Vol on an affected volume gives a listing of all the attributes something like the following Read-only                          : Yes Hidden                               : No No Default Drive Letter    : Yes Shadow Copy             ...

I have not been terribly nice about Dell, but a recent problem with a disk that failed which started badly with some poor communication got much better and ended with a complete star called Anne from Ireland who really couldn't do enough for me. She even got a list of all the contact details I would ever need, checked they still worked and tested the correct key press to get through the automated queuing rubbish that seem so prevalent these days. It was all above and beyond and fills me with a little hope for the future. I just hope Dell can now live up to my expectations that have been so expertly raised by the rather nice Anne from Ireland.

Everyone hates it but it is a necessary evil that we have to go through at least once a year, some twice others, god help their minds, several times. In previous years we have had a truck turn up with a roof that went up to create a two story office, a bit like a low rent Formala 1 paddock truck with a bad paint job. You had the office upstairs and a bunch of racks with a HP EVA SAN and a shed load of servers of your choice and some tape libraries. We would get our backup tapes and recover as many systems as possible pretending we were doing a good job and if it came to it we could recover all our systems within a week, whilst knowing full well we wouldn't have a hope in hell. Now my life is made a lot simpler. We have visualized as many of our systems as possible, these are all stored on some cheap EqualLogics (compared to EVA and all the licenses) and replicated to a remote site. I now sit at my desk and bring online replicated copies of our VMFS partitions, present to our ho...

As my little foible, I decided to do a rebuild of my PC yesterday using media with SP integrated. Back in the old days you could use a tool like nLite to integrate SPs in XP media, but no more. You now have to download the media from Microsoft via Digital River here are the links 32-bit Windows 7 Home Premium x86 SP1 (bootable) Digital River: http://msft.digitalrivercontent.net/win/X17-24208.iso 64-bit Windows 7 Home Premium x64 SP1 (bootable) Digital River: http://msft.digitalrivercontent.net/win/X17-24209.iso 32-bit Windows 7 Professional x86 SP1 (bootable) Digital River: http://msft.digitalrivercontent.net/win/X17-24280.iso 64-bit Windows 7 Professional x64 SP1 (bootable) Digital River: http://msft.digitalrivercontent.net/win/X17-24281.iso 32-bit Windows 7 Ultimate x86 SP1 (bootable) Digital River: http://msft.digitalrivercontent.net/win/X17-24394.iso 64-bit Windows 7 Ultimate X64 SP1 (bootable) Digital River: http://msft.digitalrivercontent.net/win/X17-24395.iso Now I have norma...

As part of our DR strategy we replicate or servers to our DR site. This now includes a couple of 2008 R2 Domain Controllers. For DR testing we will bring these replicas up and then the other servers in the isolated environment. However, bring the DC up in isolated seems to cause a problem. this varies fom never being able to log on to having to wait for an hour before we can logon. Once we have logged on thy event log seems to indicate a problem with DNS, and yet I can't as nail what the problem is. One of the problems we face is that we set up or domain wrong in the first place. For those of you that remember the Windows 2000 AD design exam we should have gone for the multi forest Ski lodge model but ended up with a single domain and bunch of autonomous sub domains, this was fine under Windows/Exchange 2000 (just) and 2003 (better) but under Windows 2008/R2 and Exchange 2010 is starting to look severely broken or at least more awkward to mange then it needs to be. Now whether ...

After some dancing with developers, I managed ot get some time to upgrade our EqualLogics. The main reason for teh delay was teh small number of servers that had direct attached iSCSI drives that needed to be updated to the latest HIT Kit and would therefore probably need a reboot. As it turns out, upgrading from 3.3.1 or higher to 3.5.1 didn't require a reboot. The Firmware updates went quite smoothly, this was a case of getting various polls on different updates on to the latest 5.0.7 version, including some stuck on the last version 4 release. After all the updates and server reboots I found that two servers would no longer replicate using the ASM. One was fine replicating at the EqualLogics level and just contained a bunch of flat files so I have left it as is. the other is our 2003 Exchange system. This was rather weird as the ASM was showing the iSCSI drive as "Unknown Type" and shows Exchange as being on unsupported drives. When I tried replicating the Datab...

Well no upgrades happening this weekend due to compatibilty issues and trying to schedule down time for servers. Due to a lack of foresight by the powers that be, we do not have a scheduled period where we can take servers down for maintenance, to my mind this needs to be set in stone in the following way "between x time and y time on this day you need to expect your server to be restarted", this gives us a windows when we don't need to worry about whether data is be imported or worked on and there is no other real activity of note being carried out. Just because it is there doesn't mean we will use it, but it is nice to have the opportunity without jumping through hoops trying to get permission from a bunch of system owners to update drivers and reboot a server. This time it was both VMware and EqualLogics updates that needed to reboot servers. Some of our servers have direct attached iSCSI drives on the EqualLogic, these run the HIT kit and since version 5.0.2...

Hopefully if plans go well I will be doing a bunch of upgrades this weekend. Firstly I will be doing the EqualLogic firmware, which, due to some rather buttock clenching moments, I have got different versions on a couple of our systems so I need to get them all in line. I will probably go for 5.0.5 rather than the latest 5.0.7 as it has been out for a while and seems to be relatively safe and as I get older safest seems more bearable than latest. This also means I need to get all the iSCSI connected servers on to the latest version of the HIT kit which is always a worry as most of these are physical servers so you can't just snapshot and then roll back if there is a problem, I will probably pull one of the mirrored system drives before updating to give a quick recovery if it all goes horribly wrong. The other is an update to do is VMware up to 4.1, this is a straight forward thing for the most part, however it is made more problematic by the need to update the VMware tools on ...

Had a conference call planned today with a Dell boffin and our external support people to discuss firmware and all stuff EqualLogic. Unfortunately said boffin was OOO but no one thought to mention this until we had been waiting on the phone for 10 minutes and then emailed our Dell contact (who was on paternity leave). Rescheduled for Monday but once again Dell fall over on simple Customer Service. We have about 30 or 40 companies we deal with, either support or development and to be honest there are only three that I feel are actually any good. One is called PDQ which we have hardware support with for our servers and printers, they are efficient and always follow up any deliveries to check they have arrived and for generally leave you with that nice warm glow of feeling wanted, we also have always dealt with the same people which makes a huge difference. Second we have Voyager our networking people, the account mangers are salesmen and can be irritating but they have a diamond in t...

As we rush headlong over to Exchange 2010 (don't get me started) one of the jobs I've had to do was upgrade our Enterprise Vault to version 9.0.2, this also brings with it support for Outlook 2010 at the client end allegedly, but for now we have a 2003 Exchange environment. The first attempt at an upgrade only just managed to get out of the starting blocks before going pear shaped. I patched the server installed SQL reporting and moved the Message Queue to a different location as recommended by the Vault Deployment Scanner and after reboot we had server that was crawling along a wouldn't start the vault services. Our support company were as much use a the snowball in the Sahara. We managed to recover by going back to the just finished backups of the systems drive and all was well again. I did a copy of the server and tested everything again and was unable to replicate the problem so tried again on the live server with out doing any patching or other stuff and all was wel...

There is nothing like doing patching or firmware upgrades to really get those buttocks clenching. After so many that have gone wrong it has become one of those jobs I'd rather go on holiday when they are due and leave to someone else who is younger and more care free and has yet to be scarred. The latest round of EqualLogic patching at one of our sites went a bit tom tit. Upgrading from 4.3.7 to 5.0.4 should have been a fairly simple thing, though previously I did experience a small problem with one of the controllers restarting about 15 minutes after the upgrade which was a bit weird I wasn't expecting any issues. However this went completely bonkers after the upgrade. Lots of errors which were triggered by cloning a replicated drive. Replicating would start to fail, drives would disappear and then reappear, the only solution that Dell could come up with was to move to version 5.0.5. as 5.0.4. This was weird as I had been told previously that I should move to 5.0.4 as it w...

Another one of those weird things that happen to us admins. This time it was our HP C7000 chassis. On and off we have been getting some errors where the Chassis reports some problem then 1 millisecond later all is well again, also we have had VMware reporting loss of redundancy on the iSCSI connections and packet floods logged on the chassis. Whilst I was off enjoy the sunshine this went completely ballistic to the extent I was getting emails at the rate of 1 every 1 to 10 minutes. I returned to work to find 4000+ emails, not a bad score for 4 days off. A quick Google and the best advice I could find mentioned reboots can make it go away for month but best to do a firmware update, though these mentioned old versions we were on 3.11 and the latest was 3.30 so maybe worth a shot. Given my experiences with EqualLogics (did I mention that another site upgraded their EQ to 5.0.4 and lost all their replication, barely managed to make it through the day with drive intact and had to upda...

I read a few blogs and tech sites that started of as blogs (sometimes before the world discovered blogs) and the one thing that tends to strike me is that whilst the content is often very useful and informative the command of the English language of some bloggers is really quite poor. There are, perhaps, several reasons for this, things like lack of a spell checker in the built in blogging tools on websites, the need to blog there and then, phone texting and street language all spring to mind. Twitter is is prime site for poor grammar and misuse of words with too and to and your and you're seemingly being interchangeable and having no separate meanings. Whilst I can forgive the immediacy of Twitter and the like, Blogs should be a more considered place for one's writing and bloggers should be putting more effort in to their work. A recent post I read went something along the lines of "....suppliers were trying their up most to...". Up most, from the word upperm...

Had the pleasure of install our first 2008 R2 Domain Controller today. All went splendidly, after following through the various MS documents on how to do it and what to check beforehand. Adprep had been run over a year ago as the project was sidelined several times due to other things getting in the way. Only glitch was that after running DCPromo on the new DC it restarted and whilst I could logon I couldn't get a desktop just a normal blue wallpaper. Only option was Ctrl+Alt+Del and to log off, even task manager wouldn’t run.  A reboot to safe mode with networking and all was well.  So I had a look in the event logs and spotted the following error The Windows logon process has failed to spawn a user application. Application name: . Command line parameters: C:\Windows\system32\userinit.exe Lots of Googling lead me to a MS kb article ( http://support.microsoft.com/kb/970879 ) which explains that if the Interactive and Authenticated users were not in the Local Users grou...

Update and more info available in part 2   Hit a weird problem with our EqualLogics drives. The drives themselves are fine but replicas and snapshots of some direct attached NTFS volumes would mount onto a machine but would not get a drive letter automatically, I could add one manually but it wouldn't stick after a reboot and services like SQL would not run. I could copy the data off the drive and this was data was fine. I was able to delete a volume, create a new one but not format it, this is where I got a read only error. Trying another drive I found I couldn't copy anything on to it, I got the same error This caused a lot of head scratching till I noticed the word hidden next to the drive in Diskpart. A quick Google found a Microsoft kb article that described the problem and how Diskpart could fix it. After attaching the drives, go in to Diskpart and list volumes and you will see that the Volumes are given the Hidden attribute. To clear this...

Just to be clear on how much I don't like developers here's a question for you. Why does sliced bread never fit in toasters? Because designers never use them. Its the only explanation I can think of, if they used them in the real world with loaves of bread brought from Tesco they would realise that normal sliced bread is 2 or 3 cm longer or deeper than the meager slots they provide. Developers are much the same, they never seem to test anything in the real world with real users. That is why even nowadays you come across so many applications that assume the user is in the local admins group and can write to the Program Files and Windows directories. If they tested these things properly they would pick these things up and wouldn't need to rely on Admins to fix it so their programs can actually be used. Web apps are supposed to be the answer to this. They can be run anywhere by anyone. Or can they? If the use ActiveX then no, we are talking windows only and you nee...

We use SAP and when moving from the 710 client to the 720 we came across an odd error. We install using the a SAP server distribution point and then change the SAPLogon.ini file in the Windows directory so the user get the correct links available to them depending on their role. This worked fine for 710 but on some systems users were getting the default settings only and it seems that sometime the install defaults to looking for the saplogon.ini file in the user's Appdata\Common\SAP folder. this in itself is a good thing but a bit of a nightmare to mange across several hundred PC, before we would just copy an ini file to the windows directory and all users on that PC got the same things now we potentially had to manage that ini file in multiple users accounts which gets bit more involved on teh scripting side. I then came across an article that mentioned the SAPLOGON_INI_FILE system variable. You can set this so that it looks at a central location on the PC and then your life ...

Did I mention about Windows CALs for printers? Well not printer specifically but those all in one big photocopier, come printers come scanners. Often referred to as MFDs or multi-function devices. If they scan stuff and copy it to a network share on a Windows server you need to have a server CAL for it. I wouldn't be surprised if you need a CAL if something on your network is doing DNS lookups or claiming a DHCP address. I am not particular lover of Linux but there more I learn about the more restrictive licencing practices Microsoft are in to the more I yearn for a simpler life. CALs are probably the most irksome form of licencing known to man. It is shame that the EU and US justice system seem hell bent on removing useful bits of Windows for some imagined slight on providers of free software but do nothing about onerous licencing practices that are hidden behind an impenetrable forest of words that are at forefront of mumbo jumbo legalese. Plain licensing terms in plain E...

It's probably every Admins least favourite topic. Licensing is a head ache all round. From reading EULAs to trying to maintain some kind of record of what is installed where, it is a difficult and painful part of our job but it has to be done and done meticulously and with great diligence and it is always a fight to get those further up the food chain to appreciate the cost and restrictions that are imposed. Trying to get any kind of financial support for some kind of Software Asset Management (SAM) system is both difficult to justify and a difficult sell, where's the return on investment on what can be quite a large investment? Well you just wait until Microsoft come knocking on your door. If you have any kind of volume licensing agreement you will have signed something to say it's ok for Microsoft to come in at 30 days notice and do an audit of all the software on all your PCs and Servers and Macs. Oh and you will pay for the privilege. In our case that could be som...

Every now and then I have the need to perform a simple task across a large number of PCs. This could be rolling out an update of Flash or updating a registry entry. All of this is made simple by having some desktop management software like Altiris or Landesk to hand. Write a quick script to run a silent install or import the registry settings in silent mode and job done. However occasionally you come a across a desire that is little more difficult to fulfill. This normally involves applying a registry entry or deleting a folder in all of the user profiles on a PC. This is often trickier than it first seems. For example say you wanted to empty the temp folder in every user's profile. If you want to delete all files that have a .tmp extension anywhere on the C: drive then you can simple run the following Del C:\*.tmp /s Simple, job done. But when you want to delete all the files in the Temp folder that appears in C:\Documents and Settings\ Username\ Local settings\Temp. The ...