Life of a Sysadmin

Don't Panic, or the burning smell is perfectly normal.

Our facilities liason sent out a company wide email today.

"Please do not be alarmed by the burning smell in the hallway/stairwell/bathroom area. We're aware of the issue."

Good to hear that the plastic burning smell that quickly causes a headache is nothing to be concerned about.

MovinCool, or You should know where to rent one on short notice

The picture above is a "portable" air conditioner. This particular one is rated to cool 17000 btu/hr and is pretty much the largest air conditioner you can run off a standard 20A circuit (it pulls 16A when running). It's one of two (the second one is larger) attempting to keep our second server room cool.

While we had planned on just renting this particular unit until the permanent AC system was installed, it became obvious quite quickly that planning and permitting would extend the rental past the point where it made sense to just buy the thing. We now own two of these (the other requires a 30A circuit), and will hopefully never need to rent one again.

That said, anyone that runs a server room should know where to rent one of these, how long it would take to get it delivered, and how it can be paid for. Our local provider (where we both rented and purchased our units) is Atlas Sales and Rentals. Great guys if you need such a thing in and around Silicon Valley.

Connectors, or the IEC and NEMA make things simple

I was in need of a bunch of specific lengths of power cables to clean up the wiring on one of our racks. I asked my salesman at CDW for "normal PC power cords to go from a typical wall outlet to a typical PC power supply in lengths of 1, 2, 3, and 4 ft".

Apparently that wasn't clear as I needed as I was presented with a quote for not what I was seeking. A little searching (and questioning of my engineer brother), and I learned that I sought NEMA 5-15 plug to IEC c13 cords. And in case you need to figure out the names of a power connector you find on the wall or on a piece of computer equipment, checkout, Wikipedia's pages NEMA and IEC connectors.

Up front pricing, or Stop wasting my time

I was at a presentation the other night where several different vendors were presenting their "virtualization" (complaints about the creeping usage of the term virtualization by marketing people will be saved for another day) products to a group of mostly IT professionals.

After one company finished their presentation, I asked for general pricing information. The representative from marketing deflected my question and suggested I talk with them after the meeting. I could have spoken with them after the other presentations were finished, but I already knew all I needed to know. Their product was expensive, damned expensive. A quick search online confirms this -- pricing starts at $20k, and is realistically $50k for all the pieces anyone consider the product would actually want.

Now, this avoidance of public pricing was in direct contrast to another company;, who answered all of the basic pricing questions one might have with their last slide. They also have it clearly on their website. Good for them.

I wish marketing departments would realize that hiding the cost of their product only annoys technical people. Companies need to provide at public pricing that is at least in the correct ballpark of what I would pay. Sure, if they must they can do silly things like MAP pricing or have an MSRP that is noticeably more than customers actually pay. But at least it lets me get an understanding if I can even consider your product. If you hide your prices and you became legitimately interesting in the marketplace, your price lists will likely end up somewhere like Storage Mojo's Pricing Guide page.

So marketers, please don't waste my time or your time and let me at least figure out if your product is even within my budget before I have to talk to you.

Damaged Users, or Geez does QuickBooks suck

The office manager was updating from QuickBooks 2005 to QuickBooks 2008. With a fresh backup of the data from QuickBooks 2005, we went to import the data in to the new version. "An error occurred when QuickBooks tried to access the company file" when converting a file to QuickBooks 2008 (Error -13,0)".

I guess it's not going to be as simple of an upgrade as I had hoped (or the manual led us to believe). Good thing Intuit has a knowledge base article on Error -13,0. Let's look at the three suggested problems and the related solutions.

Shortcuts on the icon bars to features that are no longer in the product such as the Open PO List: So, if they can recognize that there is a bad shortcut, why can't they fix this automatically? More fun, is that they provide only one possible shortcut that could be causing the problem. Couldn't they at least tell me which shortcut is causing the error?
Damaged Admin user password: Wait a minute, so the password I provided moments earlier to access the backup file (and the step that presented an error when provided the wrong password) isn't good? And they ask me to fix it by simply changing the admin password? How lame.
Damaged Users: I certainly wouldn't have called the office managed damaged. In fact, she's a pretty smart cookie. Oh, I see, it's one of the user accounts that's causing the problem. Once again, couldn't they tell me which user caused the problem? Do they really think it's acceptable to "Delete all users but the Admin user"? I sure look forward to getting all of the users to re-enter their passwords.

With user experiences and support documents like this, I think I can safely predict more posts about the QuickBooks in my future.

Sysadmins Law 38, or Windows vs Linux

When given a choice, the vast majority of the users that I support choose Windows over Linux as their desktop OS. To be more specific, many of the users I support are assigned both a respectably powerful Windows laptop and a pretty darned nice dual screen Linux workstation setup.

The majority of those users use their Windows laptop as their main computer. Nevermind that most of their time is spent in terminal windows connected to Unix systems Graphical apps on small laptop screens. Using windows as little more than a way to display a bunch of windows at once.

Users like what is known, that almost always means Windows.

Evaluation Licenses, or How to annoy a sys admin

I was considering a major upgrade to our backup system (for more robust backups of laptops), but before I committed to making such a large purchase, I sought an evaluation license that would add the extra features to our backup software. With the evaluation license installed, I poked, prodded, and happily tested.

We ended up deciding against the upgrade, and after the 30 day evaluation license expired, my backup software simply stopped working. You see when I installed the evaluation license (which gave permission to use nearly every feature of the software), it overwrote the permanent license that was installed.

Sure there were emailed reminders every two hours for the 7 days before it stopped working. As I had stopped using all the extra features by the end of the 30 days, why couldn't the software just revert to my previous permanent license? Why did I specifically have to go re-install my license?

Promises, or Zimbra down to the wire

Version 5 of Zimbra was originally announced to be released late in 2007. The first official date put forth by the company was the 17th of Decemeber. That date was pushed back the 31st. The open source version was released on the 20th of December. A variety of forum posts from Zimbra employees said that the commercial version would be released before 2008.

The announcement of the commercial version of Zimbra 5 was made about 4pm on the last day of 2007 That's kinda cutting it close.

A classy tin of cookies, or more vendor loot

CDW sent me this classy tin of cookies for the holidays to thank us for being a customer. Let's see, over $100k in purchases and we get a $50 tin of cookies.

The office manager (to whom the bills are sent) thought that it should have had her name on it. My boss (who's name is on the account) thought that it should have been address to him. Not that it really mattered, as there were plenty to share; although they didn't make it through the day in the kitchen.

Data recovery, or a review of Kroll Ontrack for disk recovery

A manager came to me with a report of what sounded like a failing hard drive. It didn't take me long to see that there was something seriously wrong with the drive. I offered him three options; expensive ($500-3000 with Kroll Ontrack), the inexpensive ($300-700 with Gillware), or the extremely cheap (me trying things that have worked for me in the past).

A call to Kroll Ontrack got me the address to ship the drive to and the information that needed to be included with the drive. It would be $100 to diagnosis the drive. That gets a list of files they can recover and a quote for what it would cost to get the data back.

For this particular drive, it was $1500 to recover the data. It was extra for media to get the data back on. $10/dvd, $2/cd, or usb/firewire drives at about twice market price. The drive turned out to be made by AMS and was shipped in a custom labeled box just for Ontrack.

Between first call and recieving the data back on a drive it was 14 days; several of those days were spent by me getting approval to possibly spend the money, by our lawyer reading their paperwork, and by our accounts payable person arranging payment.

Everything about my experience with Ontrack was well executred. I had a single point of contact for all matters. I was kept in the loop about the processing of the job through the various stages of work. All in all great experience and I would happily recommend them.

Dell laptop ram, or idiocy by some system builders

Let's say you order a laptop with a gig of ram. Let's further suggest that you paid extra to have that gig in the form of one stick of 1gb instead of two sticks of 512mb.

Now consider that many laptops have one ram slot under the keyboard and that slot is often annoying to get to, while the other slot is often readily accessible with the removal of a single screw on the bottom of the laptop.

With this knowledge, which ram slot would you expect the ram to be installed in? Can you guess which one Dell puts it in about half the time?

Fine Tec Computer Wine, or woot vendor loot!

The delivery driver for our whitebox vendor came by earlier this week with three small boxes. A coworker looked at the boxes, looked at me and commented, "Nothing we ordered could fit in those boxes."

Turns out each box was a bottle of wine from the private celler of Fine Tec Computer (it's actually a nice bottle from Kendall Jackson with a custom label).

If you are in the Bay Area and need computers for a business, I would happily recommend Fine Tec. Prices are great. The warranty is great. They deliver. They will happily build very custom systems. The one warning I would give is that you need to double check that their suggested configurations will indeed meet your requirements.

Server video cards, or a cheap hack

That picture is indeed of a half height AGP video card with the mounting bracket removed and the connector cut off. The surgery was performed to cope with an annoying problem of many x86 systems; that of requiring a video card to be present to boot the system.

I have of course never used this card in a production system.

BakBone Customer Support, or support done right

About once a week I need to talk to a technical support group with one company or another. Few of these calls are what I would describe as pleasant experiences. One company however has always proven themselves to understand what good customer service is supposed to be.

When I call, the call is answered by a nice lady (I have never been on hold for more than two or three minutes). She wants a serial number to verify your support eligibility (she can look up serial numbers if need be based on company or a person's name), followed by a brief description of the problem. While not a support technician herself, she understands more than enough to collect the needed pieces to create a detailed description of the problem at hand (she does this by taking what I say and repeating what she understands the problem to be in a different way). At no point does she make me feel stupid, nor does she make me feel like I am wasting my time dictacting to a drone who simply types what I say.

From there, she arranges how the technician will get in touch with me and can arrange a specific time if need be. I have always been called or emailed back promptly by a tech. They have always been polite and understand how valuable my time is. As such, they believe me when I tell them I have already tried something and don't ask me to repeat these tests unless there really is valuable information to be obtained that I did not take note of the first time. They have never sent me on wild information gathering quests like many vendors do. The techs always follow-up when they say they will, and they keep me in the loop when a problem is sent up to the engineers for further analysis.

Great job BakBone. Sure most of my support requests end with them being marked "won't fix; works as intended" or some similar such condition; but you sure understand how to run a support group. Thanks for not being another source of blood pressure raising anger in the sea of mediocrity that technical support usually is.

Battery Recalls, or Dell tries to hide recalls

In the last two months a bunch of users have come to me telling me that there laptops were giving them warnings about their batteries reaching the end of their useful life. When the first two were reported, I assumed they were unusual flukes just gave the users new batteries. When the third came in, I suspected something was up.

With six basically dead batteries in front of me (the oldest only 14 months old) I give Dell technical support a call. Once again I learn that Dell can't track an items serial number back to an original system or order. I also learn that Dell has a one year warranty on batteries. The support rep suggests I check to see if the batteries are part of a recall at the Dell Battery Program.

Nope, none of the batteries are part of a recall. With service tags for some of the batteries, the support rep looks to see if the batteries would still be under warranty. Nope, not under warranty. Requesting his manager gets me the response "There are some recalls involving runtime issues that I can look up". I guess the Dell Battery Program page is only for recalls that might be dangerous to the user.

It turns out to be my lucky day, as the rep can replace all six batteries under one of those apparently top-secret recalls. With a case number and a promise for new batteries to be on my doorstep within two business days, I was off the phone in under 30 minutes.

Virus scanners, or An Overly Concerned Engineer

A paraphased portion of the conversation between myself and an engineer about his new laptop;

engineer: ... and where is the virus program?

me: There isn't one at the moment. In the next few weeks we are migrating from one vendor to another. As it's kind of a pain to uninstall the current one, I was trying to save myself some time and effort. Plus, we are out of licenses for our current product.

engineer: There needs to be a virus scanner.

me: Were you planning to run attachments received via email from strangers?

engineer: I just won't use this for email until I get a virus scanner. It's really important.

sigh I guess I should be happy he was concerned about the matter.

OEM Licenses, or Trying to make a non sysadmin understand software licensing

"I bought alaptop. I could return my company issued one if I could get a copy of Office for it." an engineer stopping by my office tells me.

"Ok, I need an email from your manager authorizing a $370 license."

"What?! Office was only like $150 from HP."

"Yes, and that license wasn't for Office Professional, plus it would be for Office 2007 and not Office 2003."

"What's wrong with 2007?" the engineer naively asked.

"In short, it would cause plenty of headaches for our OpenOffice users."

"Couldn't we just move the copy installed on my company laptop?" the engineer continues hopefully.

"Nope. OEM licenses purchased from Dell and the like are tied to the machine they were purchased on. $370 gets us permission to both install older versions and the ability to move a license from one machine to another."

Value Added Resellers, or Where is the added value exactly?

About half of what I purchase at work requires going through a value-added reseller.

In theory a VAR is supposed to add value by selling entire turnkey solutions. Or they might specialize in providing knowledge and support for a particular industry. At the very least a VAR should have in depth knowledge of the products they are selling.

In practice, I find they can rarely answer my questions off the top of their head. I have yet to have one actually provide useful planning assistance for my long term needs. They tend to take three or four days to get me quotes. Where is the added value for me as a customer?

Why do companies wish to share the profit with another party? This should be really simple. Provide decent marketing material; provide fast turnaround for pricing requests; and when a customer is going to drop a few grand, let them talk to tech support to answer the tough questions the docs and sales people can't. Sure this means that they need to have a few more sales people on staff, but I have to believe that the increased profit per sale offsets those costs.

I find this all particularly frustrating, because I have walked into nearly every one of these purchases knowing exactly what I wanted and not needing anything more than a few minutes with technical support to verify some fiddly bits.

Sysadmins Law 7, or Never scrimp on the interconnects for high speed devices

While getting our new tape jukebox working, I was running into all sorts of seemingly random and odd behaviors. The new jukebox (run by WinCE) couldn't consistenty recognize all of it's drives. On boot, the server did not always find all of the devices connected. Tapes would show up as unreadable in some drives some of the time. General oddness all around.

I checked the cabling, the scsi id's, and the termination. Everything was as it should be. I even added up the various lengths of cable to make sure I was within spec for total cable length. Through a tedious process of elimination, I was left with a seemingly bad cable. Which lead me to sysadmins Law 7;

Never scrimp on cables for expensive systems. Buy good cables.

Server Lifts, or How to mount heavy server in a rack

Sitting in our server room right now is a new 250 lb toy. I spent a few minutes this afternoon contemplating how exactly we were going tolift that thing onto the rails (it doesn't help that I'm not particularly confident that the provided sliding rails will work as advertised).

I brought this problem up to a friend who works in a large datacenter. He told me of the existance of lifts built to ease the process of hefting large servers into a rack. The only such lift that I could find advertised for sale is one from Server Life Corporation (which seems to cost about $8k). IBM makes reference to a support document pdf doc. Not quite designed for racking servers, but R on I's Lift-o-flex line looks to be a reasonable compromise of function to cost (under $1000 for the largest ones).

Not having a large enough serverroom to justify even the Lift-O-Flex (and certainly not being patient enough to attempt to acquire one before using the new toy), I recruited a couple of burly co-workers and it was heaved 10 inches up and two feet back onto the rails without much difficultly.

tee, or A useful Unix utility

Nearly a year ago, my boss introduced me to the Unix utility tee. Seeing as at least one of my unix using friends was unaware of the utility, I figured others must not know of it either.

As the documentation puts it; "reads standard input, then writes the output of a program to standard output and simultaneously copies it into the specified file or files." Pretty straightforward.

multixterm, or An entertaining and useful X app

I believe every unix shop has a locally written script that will run a specified script on a collection of hosts. I've certainly written that script before. Sometimes however I can't write a script that encapsulates my needs. For these needs, I can usually use multixterm.

As the name implies, multixterm runs multiple xterms. The neat part, is that it provides a way to provide the same input to all of those xterms at the same time.

Running the command; multixterm -xc "ssh %n" foo bar baz (where foo, bar, and baz are hostnames), will open three xterm terms. Each window will have already run the command "ssh hostname" (with one hostname per xterm). You will also have a small input window where you can type, and that text appears in all of the spawned xterms. Or, if you have an exception in a particular window, you can go type in that one alone.

Doesn't that sound useful?

Photos of colocated machines, or How to not forget a needed piece

Useful tip: take photos of the internal and external wiring of any computers that you install far from workplace/home. I realized how useful this would be on a late night race to Fry's Electronics (a race because Fry's was 3 miles away and was closing in 10 minutes) from my colo provider.

Had I had a picture of the inside of a particular server, I would have known that there were no free power plugs for hard drives, and that I would need to bring a y splitter. Instead, I broke a multitude of laws to acquire one last night. .

User Access Control, or A user annoynance

Today was my first chance using Windows Vista. One of our consultants got a new laptop and needed some assistance with it. After guiding him for awhile, I took the drivers seat and went about installing and configuring some more complex matters.

While installing the VPN client, I encountered four instances of User Account Control asking for permission to continue. I asked the owner of the laptop if he had grown annoyed by the UAC messages yet. He responded "I have yet to see one yet."

In the 15-20 minutes I spent watching and helping him before taking the mouse, I saw him click through at least a half dozen of these messages. It seems Microsoft's goals of alerting the user of possibly conerning changes becomes ineffective after just two days with the OS.

DHL Tracking information, or I dislike DHL

About that package that DHL claims they couldn't deliver because my workplace was closed. The next business day came and went without my package. Monday (the second business day after the package should have been delivered) morning rolls around and I look up the status of the package on DHL's site.

The most recent entry is from the original day the package was supposed to arrive; no updates for the past three days. A phone call to DHL reveals that the driver claims to have attempted to deliver the package around 10am and again around 1:13pm. And as they made two delivery attempts, I must arrange for another delivery myself.

What part of this tracking report says they attempted to deliver it at 10am. What part of that report says that I the consignee must arrange another delivery attempt?

A series of firm requests got me the package today.

DHL Delivery men suck, or Seriously what the hell is wrong with our delivery dudes?

As we don't get too many visitors, we do not yet have a receptionist in our lobby. We do have a phone with a sign next to it indicating that visitors should either call the extension of who they are hear to meet, or pick up the phone and wait to be connected to the office manager. It seems our delivery dudes are illiterate however.

Incident 1: We moved across the parking lot to a new building (in the same complex). The door to our old suite has a sign indicating our new address and with a little map of which building we are in now. I was expecting a package one day. It doesn't show up and I call the FedEx up and see what's up. Sure the company that shipped the box screwed up and didn't correct our shipping address as I requested, but couldn't the delivery guy read the damn sign, and walk the extra 50 yards?

Incident 2: A DHL guy wanders into an ongoing meeting in the conference room off of our lobby shouting about having a package to be delivered.

Incident 3: DHL shows up to pick-up a laptop that is being returned for repair. Part of the pick-up is that they will pack the laptop for transport. He did indeed bring a box. One that allowed the laptop to rattle around in transit. This is the same box that is used to return the laptop.

Incident 4: I was expecting a package via DHL yesterday. When I checked on the package this morning, I saw that the most recent message was from yesterday and read "1:13 pm Consignee premises closed. Will attempt again next business day.". My workplace was of course not closed at 1 pm on a Thursday.

To paraphrase a friend of mine; There is only one shipping company.

Dell Replacement part tracking, or Why is this the customer's responsibility?

So in a tale similar to this one, I once again was in possession of both a new laptop and a broken one that needed to be returned to Dell. In the box with the new laptop was a prepaid DHL label to return the broken machine. I needed to put the RMA number and our return address on the label, and call up DHL to arrange a pick-up.

It took me a little while to get around to calling DHL (I wasn't about to give up the broken machine until I had verified the working status of the new one). The day it was finally picked up by DHL was the day the office manager came into my office with a letter from Dell indicating 1) they were grumpy with me for not having returned the broken machine and 2) that if I have already had the machine picked up by DHL that I needed to contact Dell with the tracking number for the package. Once I provide the tracking number (and it's in transit) Dell will stop pestering me.

It would have been nice of Dell to fill out the prepaid shipping label with address and the RMA number, but I can forgive them for that. It is however pretty sad that they can't keep track of the tracking numbers themselves. Apparently Dell doesn't understand the little touches about customer service.

A demo user account, or perhaps a rodent remover

For a demo system I was setting up, I created a user with the name "demouser". Upon showing the system to the salesman who would be using it; he commented that the username looked like "de-mouser". I chuckle everyime I see the username now.

Loading docks, or How not to make an order for 500 lbs of stuff

The office manager (who sits in the cube next to me) recieved a call from the lobby. It was a truck driver saying he had a shipment that required a loading dock. It seems that our keyboard trays had shown up. And as they weighed nearly 500 pounds in total, I guess it is of no surprise that it was shipped as freight.

Tangent: To find these keyboard trays, I ordered five different keyboard trays (A few more options if you wish to see what else I looked at; 1 2 3). Four of those were returned. All that effort was to avoid paying $400/tray for the Herman Miller keyboard trays that match our cube system. It was worth it. We found a keyboard that was better than the Herman Miller one for almost a quarter of the cost.

So the lesson learned here is that when ordering 500lbs of stuff, make sure it is shipped via a method that does not require a loading dock if you don't have one.

Windows Genuine Advantage, or What it's like to have a "non genuine" copy of Windows

Ever wonder what a Windows Genuine Advantage failure looks like? I can't say that I had. Too bad I was presented the opportunity to see first hand what it was like.

We first got the warning that our "copy of Windows is not genuine" about two months ago (long story as to why we didn't deal with it sooner). We were not concerned about the machine not recieving updates from Microsoft as the machine segregated from the rest of our network. With our Volume license Key in hand (we couldn't find the original license for the system), and a use of that key marked on our spreadsheet tracking such things, I aproached the machine.

(click any of the images for full screenshots) Where I am greeted immediately by a reminder that I "may be a victim of software counterfeiting".

In case I missed the warning on the login screen, Microsoft turns Windows nagware .

And just in case I missed the two previous messages, Microsoft thoughtfully provides another warning. Clicking this one takes you to Microsoft's site for further explanation.

That page provided me a link to "Find out if you can update your product key without purchasing a new copy of windows". I could also have given Microsoft $149 to "Get Genuine Now".

I am told to download and run the Key Update Tool. It downloads quickly and once run presents a wizard to change your product key.

Unfortunately it didn't work.

Thankfully I knew of a tool to change the key (or registration information) for Windows XP. This tool simply provides a nice gui to change the product, as Microsoft helpfully documents.

A reboot, a manual validation, and a successful trip to Windows Update for updates leaves me confident that the process was a success.

How contactless keycards work, or "I don't know what it stopped working"

The office manager brings me this card today. The cleaning crew reported to her that it wasn't working.

A swipe past a card reader shows that the card is indeed not functioning. If you look carefully at the hole in the picture above, you can see a few copper wires sticking out. I guessed that they cut through the wire loop that makes the contactless part of the keycard work. I setup a new card for cleaning crew and went about pulling the back off the card.

Well will ya look at that. The spindler hit pretty much the key spot not to on the card; the link between the wire loop and the circuit that has the unique identifier in it.

About that call to Dell Customer Service, or a Latitude Lemon

The first report of a problem with the laptop was that the screen went "funky". Our advice was to hibernate the system and power it back on. If it occured again, the user should call us when it occured. The problem of course recurred, and I was called. The drive was moved to a new laptop and the user continued on their merry way. A call to technical support went smoothly, with the result being the laptop being shipped to the repair depot, where they replaced the screen.

Fast forward a month, I am setting up the laptop for a new user and the screen problems return. I deploy a different laptop the new user and make another service call. The laptop takes another trip to the depot, where the motherboard is replaced.

Fast forward a month, the laptop is out on short term loan to an engineer on vacation. He calls me up and tells me the screen is "woogie". He was able to work around the problem for the week by simply suspending and awaking the machine whenever the screen went south. A trip to the depot replaces the screen (again in theory).

The laptop is returned to us in a worse state than it was sent it; some of the damage likely from the poor packing job by the repair depot. My call to technical support reports a broken wireless switch (it falls off), a flakey screen (same problem), and a horrible whine. It only took me 20 minutes to get them to send a technican onsite to make these repairs (as opposed to sending it back to the depot again). I am told that I will be contacted by the repair person the next day and they would be out to do the repairs the following day.

When I haven't heard from the repair person by the end of the second day, I call Dell. Who can't escalate the matter as the people it would be escalated to have gone for the weekend. I am promised a call on Monday morning from the support escalation team. Monday morning passes and I finally hear from the repairman. He asks when would be convenient for him to come out. He doesn't catch the sarcasm when I reply "Three days ago". We setup an appointment for the following day.

The next day arrives, I show the repairman into a conference room, where he opens the boxes with the parts and learns what work he will be performing on the laptop. In about an hour he replaces the screen, motherboard, and wireless switch. I check the laptop out and everything appears to be in working order. About 30 minutes into a Windows install, the screen problem comes back. About 30 minutes later the whine comes back. The trip from the repairman however did yield one useful piece of information; after the second repair, I should have called Customer Service instead of Technical Support.

That's just what I did. They listened to my complaint, they offered to transfer for me to Technical Support, I asked for a new laptop, and 20 minutes later the agreed. Four days later a new laptop shows up.

Too bad the laptop only vaguely resembles our standard issue laptops. The CPU is faster (great), there is more ram (no problem), the hard drive is bigger (sure), it has the Intel wireless card instead of the Dell one (annoying as I like consistency), the Nvidia graphics card instead of the Intel one (seriously annoying for the automated installer we use), has a fingerprint reader (annoying in that it is a driver that I now need to deal with for just one laptop), has a dvd writer instead of a dvd/cd writer (sure, whatever), has the extended battery (eww, it makes the laptop bigger), it has a more expensive version of Office (not cool as it is yet another special case I have to track).

Sigh. I guess this will become my test/development machine.

Prescription Drug, or Backup software company

While researching backup solutions for our Windows laptops, a coworker recommend I look into the products of Avamar. My first through was "That sounds like the name of a prescription drug.". My first glance at the website only provided further evidence that this was the name of a drug and not a company that makes backup software

Turns out that they make some really cool software that works out how to not backup multiple copies of the same thing (across different systems even). It also turns out that they are well out of the price range of this particular project.

x11vnc, or How to Recover from a wedged X session

An engineer came to me complaining that his X session was wedged. My immediate reaction (without looking up from my work) has become a standard response; "Ctrl, Alt, Backspace to kill the X server and once you log back in set the screensaver to just power off the monitors."

This paricularly engineer however had a couple of long running tasks in terminal windows that he didn't wish to lose, as they had been running for nearly 8 hours; and could I please help him fix the X session? He couldn't just let the machine sit until the tasks were done, and than kill the X server as he needed to see the output of the jobs that were running.

Tangent: Why do so few people understand that when running tasks that are going to take awhile, the output should be redirected to a file (or perhaps a file and stdout with the tee command) so that if the computer were to crash mid run, at least the output would be saved. Not even a bad experience or two (resulting in lost output) seems to convince people of this need.

In this particular instance, the X server wasn't horked far enough to stop x11vnc (which takes a running X session and exports it as a VNC session) from working. This technique won't work on all X server breakages (or even most in my experience), but it does work on occassion.

Low Disk Space and VMWare, or Corrupting a VMs ram

Yesterday evening, I recieve a report that a user can't log into a particular virtual machine. A quick look and it turns out that the virtual machine is stalled awaiting an answer to a question.

For those without access to the image above, the relevant portion is "The directory /vmware/virtual_machine/ has less than 150 MB of free space. Running out of free space in this directory may corrupt the virtual machine's RAM."

sidenote: You can see if a VM in VMWare Server has a question to be answered by eithe connecting to the console of the VM (this is what generated the image above) or with the command line "vmware-cmd vmwareconfig.vmx answer"

This became a problem only because we do not pre-allocate the disk space of virtual machines upon creation. As users of the VM added data to the VM, the hosts's disk filled up and we got this error message.

Doh

VI Keybindings and tcsh, or Oh the humanity

For historical reasons the default shell for the engineers I support is tcsh. The default cshrc sources a half dozen other csh scripts. One of those scripts issues the command bindkey -v which sets the shell to use vi style keybindings.

I learned this little joy when I first started working here, but as I use VI regularly I could adapt (and there was the little fact that I use bash as my day to day shell and enter tcsh as little as possible). I had quickly forgotten about it until I recieved a bug report complaining about "oddly" terminal windows.

Sitting down at the user's machine shortly after the bug report hit my inbox I realized quickly what the problem was; vi keybindings. After explaining what was going on to her, we added the line bindkey -e to the end of her cshrc and her world was better. For the rest of the week, I kept receiving thank you's from people who learned about my "fix" through the grapevine.

It seems few people like many of my coworkers dislike vi keybindings on the shell.

For more information on bindkey, man tcsh

Dell Customer Service, or Not an acceptable message

Written 2007-03-14

The story of why I was calling Dell Customer Service will be saved for another day, suffice it to say I had issues that the technical support group clearly could not resolve.

A few minutes navigating the general phone tree, a few minutes with a call manager (aka an operator), I get to the Customer Service phone tree and select the option to talk to a human. A moment later, I recieve the following message;

"Dell Customer Service is currently closed for a company meeting. Please try calling again in one hour." I was speecheless. Which is good since Dell hung up on me at that point.

Slipstreaming, or Customizing a Windows XP install CD Part 1

In a quest to automate and standardize our Windows installs, I have begun working to make a custom Windows Install DVD that will hopefully install Windows and all our standard software. This is the first of likely many posts describing the journey. This post describes how to slipstream a service pack and hotfixes into install media.

Slipstreaming is the process of integrating a service pack and hotfixes into a Windows install cd. The end result of this will be a install cd that will have very few updates to download from Microsoft Update after a fresh install. Note that the slipstream process only works with un-customized Windows install media. Thus, you will likely have issues if you attempt this with the media provided by most major hardware vendors. This is really intended for those with a Volume Licensing Agreement or with retail media.

First we need to incorporate Service Pack 2 into the install media. Note, that much of the media shipped in the past two years already has SP2 incorporated. If however, you have media that does not have it, go follow the instructions at the Supersite for Windows; Slipstreaming Windows XP with Service Pack 2.

With install media containing SP2, you now likely want to install the 60+ updates that have been released since Service Pack 2 was released.

Copy your SP2 loaded install cd to a working directory on your hard drive.
Extract the boot file from the install media. See the instructions in the slipstreaming guide linked above for assistance.
Download all relevant hotfixes. This is much easier said than done. I did it by finding a fully patched Windows XP machine, and running QFECheck to obtain a list of Knowledge Base articles (the six digit numbers e.g. KB909520). With that list, put them into urls of the form http://support.microsoft.com/?kbid=xxxxxx and download each hotfix referenced.
You now need to integrate each hotfix into the working directory. Use a command of the form ⟨hotfix.exe⟩ /integrate:C:\⟨workingdir⟩ for each hotfix. You could simplify this process with the command FOR %%f IN (*.exe) DO "%%f" /Integrate:C:\⟨workingdir⟩D, which must be executed in the same directory as the hotfixes.
Use your favorite cd burning application to create a bootable cd. Once again, see the slipstreaming guide above for specifics.

If you are one of those without virgin install media, or you want a simpler method with a gui, check out nLite which puts a GUI frontend on this (and much more).

Air Conditioner Failure, or always install them in pairs

My boss recieved a phone call from our alarm company about 8pm one evening. He was informed that the temperature sensor in the server room tripped. They didn't know what temperature it was in the room, they only knew that the sensor had tripped and that it was still tripped. When he arrived, the room was 86 degrees. It seems our newly installed AC unit had failed.

As the backup (original) AC unit could not even begin to keep up with our processing machines, he relocated all of those machines to the secondary server room. Thinking the problem temporarily solved, he retired for the night. When I arrived the next morning, that secondary room was almost 80. I arrived about the same time that all of the engineers were arriving and were beginning to use those processing machines more seriously. Within an hour, the room was over 90 and a few of the cases were quite hot to the touch. Once again at a critical temperature, I powered off the machines and investigated my options.

All of our servers are powered by Opteron's, and nearly all of the motherboards support PowerNow, we double-checked that all of our processing machines were configured to take advantage of this dynamically adjusting clock speed (HP's PowerNow for linux instructions Gentoo instructions. With the primary room, maintaining a mid 70s, and the secondary room staying above 80, we forcibly set all of the processors to the slowest speeds available.

Later that day, we had new circuits installed in each of our server rooms to handle our two ton Movincool. And our primary AC unit had been inspected, with a slated time to fix it being the next day once new parts arrive.

Now a portable AC unit like the Movincool is a reasonable backup unit (it had previously been used as the primary cooler in a previous server room), it is not really the best solution. The correct solution is to install two HVAC systems in the server room, either of which could handle the full load of the room. Then, run each of them at half utilization. One day.

Sysadmins Law 92, or Never second guess the amout of power a single circuit can provide

We had just recieved a new server and my coworker and I discussed which server room to put it in; the as yet incomplete one where we were plugging servers into regular wall outlets, or the original server room which we would be moving out of within the month but had multiple circuits dropped above each rack.

The matter of not having proper power in the new server room was discussed, but our desire to not move the server later that month caused us to decide to put the system into the new server room. Shelves were quickly installed, and the server was racked. Power and ether were hooked up and the power button was hit. Nothing blew up, so I left the room for my coworker to finish getting the OS install under way.

Not 15 steps out of the room and I hear a sudden decrease in the noise from the room. And while it was mildly cool to hear 15 loud machines be powered off at once; I present a firmly learned lesson;

Sysadmin Law 92: Always double check that you aren't about to overload a circuit before plugging in a new server.