Life of a Sysadmin 2005

Mandatory Contracts, or Salesmen, not the Smartest Trees in the Ocean

About a month ago, the State formalized a series of contracts for purchasing computer accessories and peripherals. These contracts effectively state that such purchases must be made through one of four approved vendors. Amusingly I didn't hear about this through official channels, but instead heard of it as it was complained about on a technical mailing list.

A call to the purchasing department of the university clarified a few things for me. As far as the university is concerned, there are many exceptions to the contract (Yes, I have this in writing. My reading of the contracts says there are no exceptions.). If I get price quotes from the four vendors and am able to find what I am purchasing cheaper at another store, I can put those quotes into my purchasing records and buy from the cheaper source. If I need an item now I can go ahead and purchase from any vendor that can meet my immediate need. If none of the vendors could provide the items I wanted, I could order from elsewhere (assuming of course I documented all this in case the department is audited).

So with an immediate need for a replacement battery for a UPS I head to the uber-secret-special webstores for the vendors. The first joy is getting an account on the vendors normal stores and than getting the account marked so that I see I could see the special state pricing. That process was in theory simple (more on that later). Create an account and send them an email to get special pricing applied to the account. With my accounts supposedly setup, I go perusing the stores for prices.

Looking for both the battery I need now and my planned purchases for the next few months , I find that all in the all the prices are bad. Actually what I find first is that the webstores all suck. When logged with my special account most of what I am looking for isn't available. Logging out and using the public versions of the webstores is what reveals almost all of the products I seek and the bad prices mentioned earlier. Perhaps bad is a little harsh. The prices are quite similar to CDW, who never has the best prices, but rarely has the worst (and as a result I almost never order from them).

Amusing Tangent: Finished with being frustrated with a god awful webstore while attempting to make a purchase, I call up the salesrep for assistance. After a couple of rings the other end is answered by a plesant sounding gentleman on what is unquestionably a cellphone. Asking me how he can help me, I explain what I am looking to purchase. He apologies and says it is going to take a moment for him to find a spot to pull over and look up the prices I am after. While finding that place he is chatting about where I work and such. When he hears where I work, he explains he is in town at the moment. He goes on to say that he is currently driving down State Street -- pointing out that he likely shouldn't be driving there as it is a restricted access street gets a pause and then "Oh, I was wondering why there were no cars on the street."

Calling the salesmen from each of the vendors I am able to reasonably promptly get quotes for the various things I am looking for prices for (apparently setting up my accounts to view the State's pricing is much more difficult than it should be as many others at the university are having the same problem). In nearly every case the various vendors do indeed have the best price (often the price is the same as what I can get elsewhere, but the free shipping gives the contract vendors the win). I guess I will end up doing what I do for Dell and after figuring out what I want, I will call the sales rep for the actual price.

The State made up these contracts in an effort to save money. The general belief being if the State promises to give a select group of vendors all of their business, the vendors will provide better discounts. This appears to be true. While there are cheaper places online for nearly everything that I priced, for various reasons I would be unwilling to purchase from those places.

So all in all these new requirements aren't overly onerous, but they are a bit of a pain. The obvious is that they can create more paperwork for some orders. A much greater concern for many is that we will have to forge new relationships and trust with salespeople and that we will be doing our product research and pricing elsewhere since the webstores for the four vendors suck in ways that haven't been common on webstores since the turn of the century.

Corel's Exclusive Offer, or Marketing Taking Over an Update System

I like the concept of self updating programs. In practice however they are usually implemented poorly (ala Acrobat Reader) and/or co-opted by marketing for nefarious purposes.

While checking for updates for our installation of Corel WordPerfect Office Suite 12 I found that some manager in the marketing division of WordPerfect thought they were being clever when they decided to use the update system to send out what amounts to an advertisement.

For an update system to be effective and trusted by users it can not be co-opted by marketing in an attempt to make money. Whatever manager that approved this "update" message should be fired.

Next Business Day Support, or Perhaps Not

Surprised that a $70 unmanaged 16 port switch has a one year next business day replacement warranty, I called up Dell Support about a PowerConnect 2016 switch that I have that flakes out (all the lights come on and it stops moving packets) on a regular basis.

Tangent: Dell has apparently started using call managers to route calls. It is their job to collect enough information to route your call to the appropriate place without needing to spend 10 minutes going through a phone maze. The first ironic point was that I spent five minutes in a phone maze before talking to a call manager, and another few minutes in a phone maze after speaking with them. The second was that I had more troubles understanding the call manager than I have had with any other Dell employee (I think it is time for me to learn a phonetic alphabet) and thus spent several minutes repeatedly correcting the lady on the other end of the line.

Once I finally got to a support technician, he quickly acknowledged that my switch had a problem and then told me his connection to the support databse was broken. This is not really something I should care about, but it meant that he might not be able to submit the shipment request to get me a new switch the next day. He did apologize and indicated he would email me immediately with his contact information and would email me again when he was able to setup the shipment. I note that 12 hours after my call (most definately after the end of the technician's shift) I have not recieved that second email. I also note that standing in place of this $70 switch right now is a $3000 Cisco Catalyst 3550.

Campus wide Technical Emergency Broadcast System, or Writing Web Apps is Hard Take Two

So the central IT organization on campus is implementing a system to enable them to send out emergency messages to the primary departmental technical contacts. This system can call a set list of numbers to notify you of various campus wide issues. I am not entirely sold that this is a good idea, but what the hell, I follow the instructions to update my contact information. While doing as the instructions indicate, it comes as no real surprise to me that I have a few concerns.

Problems with the system; 1) The database of contact info is not part of the campuswide directory system and must be maintained and updated by hand. 2) To update your own contact information you use a shared username and password. 3) You need to allow cookies to login, but instead of providing an error message, the login script will simply redirect you back to the login page. 4) Instead of just editing your own entry, you can edit the entry of anyone. 5) If you put a home phone number into your entry, it is made available via the public interface to the list.

And the central IT organization wonders why they get so little respect.

Hello 1995, or Software That Sucks

Two of the machines I support are used for accessing datasets from a sizable collection of cdroms. These cds generally cost a few hundred dollars each and come with the data wrapped up in a propreitary brinary format which requires a poorly written custom application to extract. Complaints about the usability of these applications (which is almost always poor) will be saved for another day. My complaint today is the number of companies that clearly don't wish to waste money on programmers.

We have a collection of CensusCD products from GeoLytics. The installer defaults to wanting to install to a c:\CDIDENTIFIER, changing it to c:\program files\censuscd\CDIDENTIFIER, the installer completes succesfully. Thinking the program is installed, I run the program and all appears happy. Setting up a query to extract some data works goes smoothly. Actually extracting the data however creates a cryptic error message.

A little experimentation and reading the manual (heaven forbid) show that the program needs to be installed to a path that has no spaces and no part of the path is more then 8 characters. Why then did the installer allow me to install to that directory? I know testing for this is possible in the installer, in fact while working out possible solutions I found that the ArcView installer did exactly this. This type of silliness was acceptable from small software companies when we were still transitioning from DOS to Windows (I would say up until about 1998), but it is completely unacceptable from a product released in 2002, even from a company that does not have a multi-person software development team.

Authenticated Downloading, or Writing Web Applications is Hard

Written 2005-11-16, updated 2005-11-22

The central campus IT organization manages site licenses for a handful of products. Back in the days of yore, they provided this software on floppy and then cdrom. Now it is provided by a web site which authenticates the downloader with the campus directory. After giving the site my username and password, I click the link to download the software and am presented with--

"Illegal Access!!!
 
Microsoft OLE DB Provider for ODBC Drivers error '80040e14'

[Microsoft][ODBC driver for Oracle][Oracle]ORA-00947: not enough values

/programs/download.asp, line 24"

Having seen similar messages before, I allow cookies from the site, re-login and all is fine. This is of course broken, I should have been presented a nice error message explaining something like "You seem to not have the correct cookie perhaps you are blocking cookies from this site." Well, all is fine until I try to download the second part of the package I am after, at which point I am told--

"Central Campus Download Site
 
There was an error with your download attempt.

You have already downloaded this program. Please re-register if you need to download the program again."

Ignoring the incorrectness of this message (I was downloading a different piece of the same package), this is a message I should simply not have recieved. There is no reason to make the user go through extra steps to assist them logging of information when they could have handled it inside the application.

These are mistakes that I could tolerate from small time departments, but not from one that spends more money on trial projects in a year than I have in my entire year's tech budget.

Update 2005-11-22: I filed this matter as a bug report with the group that maintains the download system. The case was closed with the following note--

"While not a critical issue, this seems to be resolved by restarting the browser and re-logging back in."

sigh

Trackpoints, or as Dell calls them "Pointing Stick Covers"

My primary system at work is a laptop. It has both a trackpoint (the IBM name used for the pointing device in the home row) and a touchpad. I find I am slow and clumsy with touchpads and thus greatly prefer trackpoints (Dell seems to call them Pointing Sticks). I really like the grippy feel of a fresh pointing stick cap. When new, using the trackpoint is an absolute joy. But as they get filled with human goo and dirt moving the cursor becomes less and less accurate.

The Dell Latitudes I use at work each come with a spare trackpoint cap. Now seeing as how 4-6 months of heavy use wears one down to the point of leaving me unhappy, I clearly need a supply of more. Dell sells Pointing Stick Covers for $11 for two. The saleswoman I spoke with said that quantity discounts were not available for that part. Thinkpads come with two different types of trackpoints. One is grippy like the Dell's and the other is textured rubber. The rubber one is washable and lasts lasts much longer than the grippy one that once filled with dirt is unsalvageable.

A month or two ago, while at the University surplus store I came across a small cache of new in package ones for fifty cents a piece. I purchased the 9 that I could find in the box. That will keep me happy for a good while.

Beeping from the Server Room, or Is a Distended UPS Battery a Bad Thing?

I heard beeping from the room next door. It was not a beeping I heard before. It almost sounded sickly. Poking my head into the server room I hear for certain that the beeping was not just an angry piece of equipment, but was an unhealthy angry one. The beeping fluctuated, growing quieter and quieter before making a feeble attempt to shout for attention at its former volume.

It turns out the beeping is from a UPS. Further investigation found that the UPS had failed in a serious way as the monitor and kvm plugged into it were no longer getting power. Moving those devices over to the backup UPS (we are not big enough to justify a proper failover system), I take note of the serial and model number of the dead UPS.

I am on hold with APC's technical support line for less than five minutes before speaking with someone. After hearing my explanation he asks for the serial number and asks if the battery pack has been replaced in this unit. The tech was quite surprised to hear that it had not been considering the unit was manufacturered in late 2000. He went on to explain my options, providing me part numbers for each; purchase a new battery pack (about $50), get a ChargeUPS pack for the unit (about $90 and extends the warranty of the entire UPS), or trade the unit in for a larger model ($50-100 discount off of MSRP). Kudos to APC for a well run support group. That phone call went exactly like it should have.

With a new battery in hand I set about pulling out the old one. After a good deal of effort, I finally get the dead battery pack out, only to find the top of the battery seriously distended and out of whack.

The new battery went in fine, charged, and went through the basic tests provided by PowerChute Personal Edition.

Updating Acrobat Reader, or Another Reason to Hate Acrobat Reader

While updating a machine from Acrobat Reader 7.0 to 7.0.5 I encountered a another annoyance I have with the program.

There is no reason an application should require a system reboot for a software update.

Update 2005-11-04

So a friend who programs under Windows on occasion pointed out that the recommended way for programs to handle replacing in use files under Windows is set it up to be done at next reboot. So there is apparently a reason for an application to require a reboot to install. I point out however that it would have been preferred to inform the user what programs are currently using inuse files and asking the user to close them so the install can continue without a reboot. I note that plugins for web browsers do this. Updates for CorelDraw work this way.

So I change my objection only slightly: There are few reasons an application should require a system reboot to perform a software update. If it needs to update in use files, it should make every effort to do so without requiring a reboot.

Wall Warts, or Something that Does Not Belong on Anything Considered Enterprise

I really dislike wall warts. Primarily because they take up so much space near plugs. But also because; they seem prone to fail more frequently than integrated power supplies (although this is likely because most wall warts are cheap linear ones), are hot (also likely because of cheap build quality), waste a good deal of electricity (once again the fault of cheap warts usually), and easily come unconnected from the device they are powering when wires are accidentally (or intentionally) jiggled.

Under my desk at home I have 9 wall warts (Palm, camera, usb hub, battery charger, external hard drive, cordless phone, dsl router/modem, and two ethernet switches), connected to a single 7 outlet power strip via Power Strip Liberators. Even with my nearly obsessive need to tie up excess wire, it is a mess.

At work I deal with a handful of wall warts at my desk for things like my USB hub and ethernet switch. But it is not these that caused my displeasure to bubble forth out of my brain. My complaint is about things marketed toward businesses and things designed to be mounted in racks Even more specifically, my complaint is about a Belkin OmniView 8 Port KVM I have.

Now it is my understanding (gleaned mostly from my electrical engineering brother) that wall warts are used because; companies can use off the shelf power supplies, the design of the device is easier as they don't need to deal with interferance or heat from the power supply, and they don't need to get their device certified by places like the Underwriters Laboratory since it is a low voltage device.

Wall warts on my server shelves are a pain. UPS's are not designed to accommodate them. They are more difficult to tie up neatly (I purchase cables of the correct length so there is little to tie up normally). And perhaps the biggest gripe, the damned barrel plug on the KVM falls out at the slightest nudge (well, it did until I applied a dab of hot glue to the top of it).

As I now look for environmental monitoring hardware I see wall warts everywhere. Wall warts on $500 products simply do no make sense. So I make a plea to all electronics makers; please eliminate wall warts whenever possible. If not possible at least use high quality inline warts. While I go out of my way to purchase products without warts for work, I will file a complaint with any product maker that uses them when the size of the device could accommodate an internal power supply.

Powercycling equipment, or The Case of a Nonexistant Battery Charger

So along with 40 laptops we use as a mobile instructional lab, we got four external battery chargers for the batteries used in those laptops. I have been using one of them for a few months now. This morning I dropped four batteries into it. Coming back a few hours later expecting the green lights to be blinking (meaning the batteries were charged), I was surprised to see three of the four lights red. Now I had understood that that indicated the batteries were bad in some way but didn't quite know what it actually meant.

A little bit of trial and error found that any batteries put into those slots were now deemed bad even if they had previously charged fine. At this point I go to the Dell website looking for the manual for the charger. Searching for battery charger didn't come up with much. Finding a part number (0F0075), a service tag, and an express service code, I go back to the support site and try to look up the device by those. Support.dell.com tells me that the Express Service code and the service tag are not valid. A general search of the site brings up a page with an interesting tidbit.

Q: Does Dell sell external battery chargers?
A: Dell does not currently offer external battery chargers.
from Dell Notebook Battery Center FAQ

Wow, this must be a pretty serious hallucination I am having. Searching the internet brings up nothing but replacement batteries. So I call up Dell.

A brief tangent on phone mazes: I have dealt with two phone mazes today, one at American Airlines and the other at Dell. Both systems allowed (and basically required) voice response to the automated questions. The one at Dell could barely handle exact responses as requested. The one at American Airlines on the otherhand could handle all sorts of extraneous noise and responses. For example it could figure out that "yep", "yes", and "correct" were the same thing. The one at Dell could only handle a nice short enunciated "yes".

While upgrading their software, Dell should invest in a decent audio to hold music adapter. The hold music was pleasant generic light classicalesque, but it faded in and out and was usually crackly. It would also be good if they normalized the volume of the music to the lady that continualy informs me that "All of our representatives are assisting other callers. Please remain on the line and a representative will be with you as quickly as possible." was a similar volume to the music.

After asking and confirming my express service code, the phone maze directs me to Dell Plasma TV support. I quickly get bounced to another phone queue followed by another, and another. Before I actually speak with someone that can help, I have spent over 2 hours on the phone and have spoken with at least 4 people.

During this enjoyable time on hold (cordless phones with speaker phones are required for dealing with tech support) I discovered another interesting issue with the battery charger; apparently it needs to be powercycled. After it was unplugged for a minute all of the charging slots once again seemed to work and all of the batteries were once again deemed good

Sure enough, once I finally did get ahold of the manual(pdf), the troubleshooting section on page 12 suggests powercycling as a solution to the red light indicators.

Daily Virus Definitions, or Deep Dark Batch File Magic

In a world where a new virus/worm can sweep around the world in under 48 hours, prompt virus definition updates are a requirement. For some inexplicable reason however, there is no built in way for a Symantec Corporate Edition 9 server to download updates daily.

Symantec Corporate Edition has server side bits that allow a company to manage the client anti-virus software. You can setup scheduled scans, configure how on access scanning works, push out new virus definitions, and configure pretty much anything you would want to configure on a client machine all from one place.

One would think that the server side programs would have the ability to schedule checking for new virus definitions from Symantec. You can easily do this. Well, you can if you want updates weekly. Weekly isn't acceptable in this day and age of sweeping virus outbreaks. Thoughtfully, Symantec has a solution, the XDBdown.cmd script downloads Intelligent Updates (which are updated daily). Ignoring the fact that this script uses some batch file voodoo that could be used to scare first year computer science students; why is this needed? Why isn't this included in the basic functions of the server side software?

I have been told version 10 allows the admin to choose any definition download schedule they wish. I have also been told not to rush to install it as the upgrade process is not quiet painless. When there is a major virus outbreak in the wild, I can run LiveUpdate manually and it will grab new definitions more often then weekly. can manually run LiveUpdate whenever I wish and it will grab a daily update. For something as simple and repetitive as this it should be automated. Until this upgrade happens, I shall have to survive with a script scheduled by Scheduled Tasks on my server.

Support Cases, or A Numbers Game for the Support Technicians

I am currently working with two companies to resolve two completely unrelated problems I am having with their respective software packages. One company is fairly small, doesn't use a case tracking system, and has only two technical support staff. There are no forms, I can call or email them to start the process. The other company is a big one. They have a web interface to submit support requests. They have a case tracking system. There are lots of support staff (they have shifts they work).

I always enjoy calling the small company. The first thing we do is make sure both they and I have the same understanding of the problem. Once that is handled we get to work actually solving the problem. This process has never once included them asking me to reboot machines just to see if it solves the problem. It has included requests for me to run cryptic debugging commands and provide them with the results. Aside from the extensive debugging abilities they built into their software, the most intelligent thing they do to make support easier is to have a license to use Citrix's web based meeting software to enable them to see and interact with their customer's computers. I have dealt with them on perhaps a dozen issues with this company, and all but one was resolved within a few days.

On the other hand, I have grown to hate asking for support from the big company. I have opened nearly ten cases with them and have only had two end with what I would call a good resolution. Those two were simply reports of minor nuisances, one was already known about and fixed by them in betas, the other was also known about, but a solution had not yet been implemented. The other cases ended with varying degrees of failure.

Most of the cases I resolved on my own through nearly blind trial and error. A couple of thse wild experiments were actually guided by the VMWare support tech monkey. Most of the time however, the tech attempted to shift the blame to other companies. One tried to claim my hardware was faulty (nevermind that the hardware went through the manufacturer's diagnostics properly and what I was trying to do inside the virtual machine worked outside of it). Another time I was asked to verify that the problem occured on a fresh install of the OS with no other software installed (yeah, as if the problem is likely to occur in that state). Several of the techs asked me to reboot the host system (which is kind of a pain as there are several virtual machines on the host) to hopefully fix the problem. My most recent support request was closed by the support tech as I couldn't provide any further information about the problem. Now this case should have been put on hold as the problem as it happens randomly, but when it does happen it makes the virtual machine unusuable for some period of time (normally a dayish). If it weren't for the fact that the yearly support license gets me free upgrades I don't think I would renew next year.

Spec'ing a firewall, or how to scare off a salesman

Having decided upon m0n0wall for our firewall, I set it up in the building machine room with a small form factor Dell and an Intel Pro/1000 MT Dual Port NIC. Even before getting grief from the campus IT group about installing a non rack mount machine into the network rack, I knew I would need to replace it with a real rackmount server at some point.

I created a requirements list for the server.

1 or 2U rackmount case
Sliding rack rails would be nice but are not required
4 network connections minimum, preferably 6. No more than two can be 10/100, the rest must be gigabit.
The gigabit connections should be connected to the motherboard via pci-x or pci express.
Preference given to Intel NIC's, however Broadcom and 3com are likely acceptable as well.
The server should be able to boot from cdrom or floppy.
The motherboard must have pata ide on it.
The hard drive should have the cheapest drive that can be purchased, as it will be replaced a compact flash ide adapter. (like http://www.pcengines.ch/cflash.htm)
The motherboard and processor should be considered servergrade.
The processor should be at least 2ghz.
512megs of memory is fine. This should be in the form of one 512meg stick of ram, unless the motherboard can make use of dual channel memory, in which case two 256 meg sticks is acceptable.
The system should cost less than $1500

Knowing that Dell couldn't really provide what I wanted, and wanting an excuse to look into some of the whitebox builders out there, I shopped my requirements list around. In general I recieved quotes in just a day or two. For every quote, I needed to follow up with a list of questions clarifying or verifying various items on my requirements list. The contentious issue was almost always how the gigabit ports were connected to the processor.

One of the responses surprised me. I had asked for the specific model number of the motherboard used so I could look up the chipsets used and verify the bus used to connect the onboard network controllers to the cpu. Instead of those answers, the salesman apologized and said that his company could not provide the machine I requested. While this was the most extreme "wrong" answer I recieved, I wasn't particularly happy with any of the quotes I recieved. Mostly I found that salespeople don't seem to like answering questions that are "hard". While I still have a few months to check more possible whitebox companies, currently it looks like I will be putting together my own machine based on a SuperMicro bare bones system.

Lab Assistant, or How I became a Hall Monitor

I sit in the hall writing this (thankfully several offices on the floor are being remodeled and there is a comfortable chair out here). I have been relegated to lab assistant for our mobile laptop lab.

As this is the first semester for us to be running this mobile lab and there were still kinks to work out, it would be run almost entirely by staff and not student lab assistants. And seeing as how I had the best technical skills to handle difficulties, it made sense for me to be the babysitter for the laptops.

It is halfway through the class, and I have been relegated to hall monitor it seems. "Where is room 4208?", "How do I get to the second floor?", "Where is the registrar's office?"

I hear the class starting to wrap up. Time to trade student ids for laptops, packup the rest of the cart, and manhandle the damned cart back to the storeroom.

The answers to the above questions by the way are; there is no 4208, through the double set of doors behind me and down the stairs to the right, and Peterson Building.

Projectors and remote X, or This can't be happening

A week before classes, I did a run through of setting up and using our new mobile laptop lab. Most of the issues were related to the logistics of setting up a projector in a classroom not designed for a projector. There was however a problem that I had a hard time believing was real.

One of the classes that will be using the laptops will be using them as dumb terms to connect to a remote x server. Everything was working as expected until we tried to run SAS. That generated the following error in the console window;

ERROR: Floating Point Zero Divide.
ERROR: Generic critical error.
ERROR: Explorer failed to initialize.
WARNING: Protected resource may be inconsistent
WARNING: Protected resource may be inconsistent

The campus has a site license to X-Win32 which we use as the local X server. It provides a properly setup ssh client to connect to a remote x server. All other X windows applications (xeyes of course being one of the more important ones tested) displayed correctly and worked properly. It was only SAS which caused a problem.

Through a good deal of trial and error (testing different computers, different accounts, different servers), we discover that the error only occurs when a projector is plugged into the laptop prior to turning on and logging into the laptop. To further add to the oddness, it is only an issue when an "intelligent" (which is to say a modern one that does autodetection and automatic image optimization) is used. If a video splitter (which will strip any communication between the display device and the laptop) is put between the laptop and the projector, SAS will run without error.

SAS's tech support provided the following answer and suggested solution

"We've seen this problem only with Sharp so far, but I can tell you what is happening. The video drivers in your projectors are corrupting the SASUSER profile catalog. It does sound far-fetched, I know, but it's true.

Some things we've found that work are: 1. Starting the projector first, THEN invoking SAS. 2. Turning down the hardware acceleration on the projector's video card. "

For the time being, we will be using an older projector which doesn't cause this error to occur. Bug reports have been filed with both SAS and Starnet, not that I expect much to come from either report as both companies will likely blame each other.

A Digital Camera, or an Odd Request

The lab assistant brought a young lady back to my office to solve a login problem. That problem was easy enough to solve and she started to walk away.

"Do you by chance have a digital camera?" she inquired, stepping back into the doorway.

It seems she has a ring that she wishes to sell on eBay (having learned that used diamonds aren't actually worth much to jewelery stores) and needs some pictures of it.

This is certainly not the oddest request that has been made of me (that is currently held by the person that asked for help logging into a horse race betting site). Seeing as how she asked nicely (and is a good user whose requests have always been polite and reasonable) how could I refuse?

Locks, or A Necessary Annoyance Made Worse by Big Companies

Note to those reading via rss/atom, there are images in this post that you likely can't see.

Computer locks are a necesary evil in many places. The computer lab I support is one such place. We do not go so far as to lockdown the keyboards and mice (I have seen places that do this, and it usually makes the working conditions for users much worse), but we do lockdown the computers, monitors, printers, and such.

Right now, we use like keyed normal Master Lock padlocks, and like keyed Master Lock padlock cables (as shown here)

The computer is locked closed with the padlock, which has the cable lock run through it. The cable is looped through the stand of the monitor and locked to the desk. (shown below)

Now those familar with Dell flat panels most likely realize that this is not secure, as the flat panel itself is attached to the stand by screws that would stop a thief for maybe a minute or three. Dell would have us use the thoughtfully supplied Kensington Security Slot (slightly blurry picture below).

Now the lock required to use that security slot (picture of a typical example below) would cost a normal consumer $40. A like keyed set of 25 would cost about $30 a lock. That's quite a bit of a difference than the $2 padlocks and $15 cables we bought in the past.

A slight tangent: I don't actually dislike the Kensington Security Lock system. I think it is wonderful that there is a sane non-mandated standard for locking down portable devices. I even carry one of their retractable locks with me in my laptop goodie bag. What I object to, is the price, and the instance on using it for larger devices. A monitor is easily large enough to accommodate a security hole large enough for a standard padlock. It would certainly be trivial to include a decent locking point on a 100 pound server, yet Dell did not in the most recent server we purchased from them.

Unfortunately it does not appear as if we are going to have much choice in the future. The current generation of monitors from Dell have a stand that can be removed at the touch of a button (and thus necessitate use of the Kensington locks), and their desktops use custom lock mechanisms (see below for an example) (that custom lock mechanism from Dell costs $30 btw).

Apparently I will have to factor in a few thousand dollars for locks and related bits and pieces when we next upgrade the lab.

VMWare Workstation, or how I can test linux firewall distributions with just my laptop

This week I tested a half dozen linux and bsd based firewall distributions (ClarkConnect, M0n0wall, Smoothwall Express, SME Server, IPCop, and RedWall if you care). I tested each with three windows clients, a windows servers, and a linux server behind them. I did this all from the comfort of my laptop. I did this with VMWare Workstation

VMWare makes virtualization products (more info here, here, and here). I first learned of VMWare Workstation (version 2 if I recall) in college when they succesfully lured me into their shinyness with a $99 academic license. I toyed with it through college (mostly running CorelDraw while my machine was booted into Linux), yet forgot about it for a few years.

About a year into the job, I purchased and began using VMWare GSX Server for server consolidation I used it quite conservatively (more on that process some other day). I didn't quite learn how much glee VMWare could bring me until I recieved a copy of Workstation for attending one of their sales seminars. But this isn't a piece on all the things I have done with VMWare Workstation, this is a description of how I used it to test Firewalls.

The first thing I needed was a handful of basic virtual machine setups. That was easy, considering I already have standard virtual machine images setup for a linux server, a windows server, and the three windows workstation setups I use.
Next I installed a firewall, configured it, and connected it to a virtual network. VMWare's virtual network setup enables you to connect virtual machines in various ways without having to get a real switch involved.
Next I installed a firewall, configured it, and connected it to a virtual network. VMWare's virtual network setup enables you to connect virtual machines in various ways without having to get a real switch involved.
Finally I powered up a handful of virtual machines and tested the actual behavior of the firewall. I experimented with the admin interface (which are mostly web based); watched the logs as I did things that should and should not show up there; tested general performance; and timed reboot times. Now I couldn't test throughput this way. While I was doing these tests, my laptop was hammered in pretty much everyway. While I could get an idea at how well a firewall could handle lower traffic levels (in the 10mg range), I couldn't test it at 100 (most linux and bsd distributions do not have the needed drivers to enable a faster than 10mb network card within VMWare). These tests will have to happen in the real world.

With this general setup I could perform basic testing on a firewall setup in about 30 minutes. Having performed this type of testing with real machines in the past, I would estimate a savings of about 4 hours for the initial setup and about an hour per firewall. I wouldn't have to go through that initial setup if I actually had the funds, space, and assistance to have a proper test lab with a variety of spare. The testing of each firewall was sped up in ways that couldn't be done with physical machines. A key item for this testing was the ability to take snapshots (a save of the state of a virtual machine) of both the firewall and the test machines. With these snapshots, I could bring back the exact same setup over and over again in just a minute or two.

The story of what firewall I choose and why is for another week.

Server Room on a Budget, or Considerations involving racks

When I arrived in this job the half dozen servers were spread across four tables in a cramped room with a window air conditioning unit. The room was a tangle of wires; the UPS that was physically closest to a server was not necessarily the one it was plugged into. There were ethernet cables being strung in from other offices via the hung ceiling.

Doing something about the room was immediately put onto my list of big projects. While I am pretty sure every good admin dreams of nicely rackmounting everything, it can be an expensive route to take. With a good rack running $800 and rack mount equipment for servers costing $100-$200 per server, the costs add up quickly. The per server cost is reduced if the rack mounting hardware is purchased with the server. Racks are unfortunately not really an option for me however.

The initial cost (which I estimated at $2200) was only a small part of my decision to not rackmount everything. My biggest reasons for going with strong shelving instead of a rack was the need for flexibility and the lack of need for density. Racks make packing many servers into a very small space easy. I have plenty of space and no need for more than about 10 servers at any one time. Flexibility was really the deal breaker for me. We had (and still have) a strange collection of servers and related hardware. With good shelves I don't need to worry about how or where I will store something when I consider a purchase.

It has taken a good deal of patience and nearly a year, but the room is almost organized.

The shelving is Metro Super Erecta. It's 30 inches deep, 74 inches high, and 60 inches wide. In theory each shelf can hold 600 pounds. That does actually hold pretty true in the real world. The heaviest loads should be closest to the poles to avoid sagging in the shelves. Also important to note, is that this is the commercial version of the Metro shelving and not the consumer stuff you may have seen at your local mega-homestore.

The back of the eight port KVM with neatly tied down cables. Nearly every wire on the shelf is tied down with cable ties (yellow ones seen in this picture) and labeled at both ends (white tags visible in this picture). The labels were made with a Brother Ptouch 2600.

There is an admin console and a KVM switch to access all of the servers and computers on the shelf. The KVM is an 8 port Belkin (nothing designed for rack mounting should use a wall wart).

There are five UPSes on the middle shelf, four 1500A APC SmartUPS and one 750VA one. We only need two of the four 1500VA UPSes to run all of the servers off of batteries for around 30 minutes. Having four means we can run off of batteries longer and it means that we can have up to two fail without loss of protection (this does assume as in our case that all of your servers have dual power supplies and they are plugged into different UPSes). The 750VA UPS is used to power the console and the in-room ethernet switch.

There are still some wires to tie up and neaten. That should be easy to fix, as soon I find a source for various lengths of decent power cords. The window air conditioner is another matter all together. That is a subject that shall be saved for another time.

Making bulk changes in AD, or my windows has some icky command lines

I have an Active Directory with about 900 users. The vast majority (all but about 15) have a single mandatory roaming profile. Because of some inconsistencies in the creation of user accounts over the years, how the profile location is specified in an account varies. Some accounts have "\\servername\profiles\normal\" some have "%logonserver%\profiles\normal". I needed to standardize these to "\\newservername\profiles\normal\".

The easy way would be with the graphical tools. Select multiple users in Active Directory Users and Computers, right click, and select Properties.

With this form it is relatively trivial to change a huge number of accounts. While I changed nearly the profile path listing to "\\newservername\profiles\normal\", I changed some (those accounts that have their own profiles) to "\\newservername\profiles\%username%". There are a variety of other environment variables available.

The hard way would use the Directory Service command-line tools from Microsoft that were included with Windows 2003 Server. They are quite powerful tools that allow you to query, modify, add, or whatnot.

The command I ended up with, after a great deal of experimentation (most of it was simply getting comfortable with the tools and toying with examples provided in Microsoft's documentation), was

dsget group "CN=groupname,DC=ads,DC=example,DC=com" -members -expand | dsmod user -profile "\\servername\profiles\normal"

the dsget command returns one per line a list of users that belong to the group "groupname". dsmod takes that output and changes the profile setting.

Other interesting examples of the DS tools.

This will get you a list of all members (recursively expanded if you have nested groups) of group groupname.

dsget group "CN=groupname,DC=ads,DC=example,DC=com" -members -expand

To create a new user

dsadd user "cn=username,DC=ads,DC=example,DC=com"

Much of my experimentation with the DS tools was done with thoughts of finally scripting account creation floating through my head. Let me just slide this into a slot near the top of to do list.

Dell Pricing, or how to get me not to consider your product

I just bought a Dell Server. I have had my eyes on a server for over six months now. It is going to be the other half of a pair of VMWare GSX Server hosts. It will provide a means for reasonable disaster recovery, a good deal of room for expansion, and the means with which to eliminate at least one physical server.

I originally spec'd the machine with dual 3.6Ghz Xeon's, 6gb of ram, four 146gb 15k drives, two 73gb 15k drives, and 3 years of gold service. The Dell website told me it would cost $11,000. That price was reached through the Wisconsin State Dell Store. The same machine configured through the Small Business store put the price at $12,500. The Medium and Large Business store didn't give me the options to configure a machine with the same specs. We ended up paying less then $8,000 after speaking with the Wisconsin Dell representative.

In a way, I like buying from Dell. It's always a pleasant surprise to spec a machine at the price I want to buy it at and have it cost much less. But it is also an exercise in abusive selling tactics. I really dislike it when companies screw with their customers.

I would love to be rebellious and say that I have no intention of purchasing from Dell again (this time I had no choice as I had to have a server that matched certain specs of my other VMWare Server host), but it is not quite that simple when purchasing for a university. There is political pressure to purchase from University approved sources. There is an economic advantage (Dell provides us excellent rates) of purchasing from the State's primary computer supplier. And finally, the State contract provides warranty and service from Dell that simply can not be had for a similar price.

One day when I actually put out most of the fires that are raging, I will have the time to persue purchasing from other vendors, perhaps whitebox ones where I get to choose the specific components.

Acrobat 7, or why I hate Adobe Reader

Contrary to any bitching you may hear from me, I actually like PDF's. Adobe Reader is a different matter though. I dispise Adobe Reader. I despise it with a passion I normally hold for people trying to sell jewelry by informing me my wife will love me more if I spend more money on jewelery (and how they will be happy to explain their financing options). Today I bring up the first of what I expect to be many pieces on user abusive decisions made by Adobe in Adobe Reader 7.

This version added Javascript as a scripting language. The scripting and programming available in previous version of Reader weren't easy enough for value added enterprise solution providers apparently (more on them here). I have a great dislike and distrust for javascript. This is of course because of the evils that have been perpetrated upon webusers by any number of websites. As such I have Javascript disabled by default in Firefox and Internet Explorer. Following that trend, I simply disabled Javascript within the Preferences.

I did this immediately after turning off automatic updates, which is something I do immediately after installation of Adobe Reader. Upon closing the program I was greeted with

Figuring this was a one time warning I ignored it. But no, I get this message each and every time I close Reader. If I open a document that I know to not have javascript I get this message upon exiting the program. Even if I open the program, don't open a pdf, and close the program, I am still informed that "This Document contains Javascripts".

The correct thing to do would have of course been to warn the user on load that a PDF had Javascript in it (and not present any warning if the document didn't have Javascript in it) and advise that it might be needed to view the document "properly" (where properly means according to the ways set forth and desired by the document creator). Or maybe they can learn from the mistakes made by Microsoft with macros in Office and warn users to not enable Macros (Javascript in this case) unless the document came from a trusted source. Whether or not that would be worthwhile is a discussion for another time.

Licenses for free software, or how to encourage people to not give you money

Part of my job entails making sure that we are in compliance with the licenses for the software we use. While not a difficult task, it is tedious and quite dull at times. Reading that much leagelese when one is not employeed as a lawyer or paralegal can not possibly be good for one's sanity. .

In most instances the issue for us has to do with us what is required to allow us to have concurrent licenses. Usually it is simply a matter of running a license server. Some involve stipulations that require the software run off of a file server. Those are hurdles that can be overcome without too much difficulty.

We use a good number of free (as in beer) programs, including a handful from Sysinternals. Most free software has limitations such as; a license is required for commercial use or the software is free for instructional use but not research use. Sysinternals however has some very interesting restrictions for their software.

The license from Sysinternals starts off saying anyone can use Sysinterals tools for home or work so long as the user downloads it from the Sysinternals website. This sounds like you aren't supposed to provide a mirror of their installers. It goes on to state quite explicitly " A commercial license is required to redistribute any of these utilities directly (whether by computer media, a file server, an email attachment, etc.)". I guess that means I shouldn't have it on our administrative file server.

A quick email to the supplied licensing address came back with a response that I almost couldn't believe. Their response was quite clear; (paraphrased) "We aren't interested in the hassle for licenses of less than $1000."

A thousand dollars is quite alot of money for my lab. I sent a reply to the email clarifying the conditions we wished to use the software; We wanted to keep copies of the programs on a file server for use by three staff members to be used in troubleshooting and debugging. The answer to that email was again quite clear in stating that we would require a license for this suggested use and that the minimum license is $1000.

Instead of having the programs on our file server, I have a series of scripts that when run will download the software, install it, run it, and once exited delete the software. I wouldn't have blinked had they asked for a one time license fee of $150. I would have blinked but not really hesitated had they asked for a one time license fee of $250. They probably could have gotten away with asking $400 even. Their tools really are that useful to us. This way however instead of getting some money from us they got none.

BgInfo or How not to forget what computer you are on.

How many times have you rebooted a server thinking you were on one machine when in reality you were on another? Or have you ever been looking for something that you just know is on the server, but you can't find, only to find you are not on the server you think you are on. This is actually a problem when you are connected to servers remotely, or are working through a kvm.

Under unix, I rarely had this problem as I set my prompt to include the hostname (Checkout the Bash Prompt HOWTO for more than you probably want to know about prompts and bash). This unfortunately doesn't help me under Windows, which most of my servers are.

My first thought was to simply use a different background on each system. With just different pictures or patterns it wouldn't scale well. I clearly needed to create a custom bitmap with the name of the computer and set it as the background. I never did get around to doing anything about that.

This morning, while at a VMWare Users Group Meeting, I saw what looked like a perfect solution to my problem. A program that displayed basic system information on the background. It turns out the program is cooler than I had thought.

BgInfo provides a simple way to autogenerate a custom background image. By default, it runs once and creates a custom image that is than set for the background. What is displayed in the image is highly customizable. There are a bunch of default options like cpu type, hostname, ip address, but it can also display information from a script, a text file, or the registry.

Wiki's are cool, or Documentation is hard

I was lucky to be shown in college how important documentation is to a project. The course was on operating system design, the project involved writing a simplistic operating system. The real challenge came from the fact that we would be working with code that had been created and worked upon for over 3 years by students in past iterations of the class.

The first year I was in the job, I learned a good deal about how well I document things. Which is to say, I learned that when making small changes and doing systems maintance it is really hard to take good notes and make proper documentation. In a class where the documentation was part of the grade it wasn't very difficult to make the time to do it. In the real world I found that unless documentation is really easy to add/update it won't be done. Conversely if the documentation that is not readily available may not be consulted until one's head has met the desk at least once. I needed to do something that would encourage me to use and write and update documentation more frequently.

What we had: When I started, all of the records were in a file cabinet. It was pretty cool actually. There were several hundred files in three drawers. There was a folder for each server, hardware purchase, software package, and such. I could often be experiencing a problem with a software package or a server or a piece of hardware, pull the folder for it and find hints about my immediate issue.

Goal: I needed a consistent and easy to follow procedure that would enable me to write how-tos, working notes, change logs, and whatever other documentation was needed. It would need to be viewable by others (limited by user or group permissions) and if possible edited by others as well. It needed to be searchable. It would need to be able to handle images. Perference would be given to a solution that didn't use a database and wasn't difficult to setup.

Implementation: It didn't take long for me to decide that this was an obvious candidate for a wiki was the way to go for this. Perusing a few lists of Wiki Engines, I settled on PodWiki. It had user and group permissions with which I could restrict view and editing access to sections, didn't use a database, had revision control built in, could handle images easily, and it had a simple install. It was installed an up and running within an hour.

Actual results: Having used my little wiki (which others at my job don't actually know about yet) for nearly three months no, I can say I am quite happy with the results.

ServiceTags, or a really neat concept killed in it's implementation

So every Dell produced since at least 1998 has had a service tag (which I faintly recall being called express service codes back in 98), a 5-7 letter and number code that uniquely identifed the machine. The code is printed on a sticker somewhere on the outside of the case. Newer machines have a sticker on the side which includes the service tag, date machine was produced, and other such useful tidbits.

Now with a unique identifier one would think that it would be easy to lookup the exact specs of ones system as it was purchased. Dell's support site does indeed let you do this. Sort of, if you can figure out what lines like this mean...

1	K3444	CARD (CIRCUIT), NETWORK, MINI PCI CARD, INTEL2200, NOT APPLICABLE

In this instance it means an Intel 2200 minipci wireless card. Usually dechipering these descriptions isn't too hard. Here is an example spec sheet for a full desktop system.

Service Tag:	7KYPQ01
System Type:	OptiPlex GX400
Ship Date:	7/19/2001
Dell IBU:	Americas
1	43YVH	PROCESSOR, 80528, 1.4GHZ, 0K, 400FSB, SOCKET W
1	57589	CABLE, AUDIO, MOLEX TO MOLEX
1	6F067	PRINTED WIRING ASSY, PLANAR (MOTHERBOARD), ASTRO, NETWORK INTERFACE CARD/CONTROLLERS, 4RIMM, OPPLX
1	9019C	ASSEMBLY, CABLE, DORADO/ATHENS/TUALATIN/ALMODOR, 34P, FLOPPY DRIVE, 1DROP, HUNNICUT/MEDIUM DESKTOP
1	9809T	ASSEMBLY, CABLE, 40P, IDE (INTEGRATED DRIVE ELECTRONICS), 2DROP, LS120 FLOPPY DRIVE
1	5120P	CORD, POWER, 125V, 6FT, SPT2, UNSHIELDED
1	25PGG	KEYBOARD, 104, 6P, UNITED STATES, NMB, MIDNIGHT GRAY
1	735HE	MOUSE, PERSONAL SYSTEM 2, 6P, 2BTN, WHEEL, 1.3A, MICROSOFT, MIDNIGHT GRAY
2	1561P	RAMBUS INLINE MEMORY MODULE, 256, 400M, 128X18, ERROR CORRECTION CODE, 16C
2	9578D	CARD (CIRCUIT), MEMORY BOARD, MEMORY, PRINTED WIRING BOARD, CONTINUITY, RAMBUS
1	70NYT	COMPACT DISK DRIVE, 128K, I, 5.25" FORM FACTOR, 48X, LENGTH/LONG, 8482B, MIDNIGHT GRAY
1	06HRM	DISPLAY, FLAT, D-INTFC, 17, 1701FP, UNITED STATES, MIDNIGHT GRAY
1	34MCW	CARD (CIRCUIT), GRAPHICS, VIDEO, 16MB, TNT2, PROFESSIONAL
1	5828D	ASSEMBLY, CABLE, ATA66, 2DROP, KLINGER
1	903DP	HARD DRIVE, 40GB, I, 7.2K, 20/P, IBM-ERC
1	4C496	FLOPPY DRIVE, 1.44M, 3.5" FORM FACTOR, 3MD, NO BEZEL, NEC CORPORATION, MIDNIGHT GRAY, 418
1	0C138	REMOVABLE MEDIA STORAGE, ZIP DRIVE, 250M, I, 3.5" FORM FACTOR, IOMEGA, V4, MIDNIGHT GRAY
1	97MNT	KIT, COMPACT DISKETTE, OPERATING SYSTEM, WINDOWS 98 SECOND EDITION, ENGLAND/ENGLISH, A01

If you tell the Dell support site a service tag, your experience will be customized to what is appropriate for that machine. Well, that's my dream anyway. It is tailored to that machine type, not the specific machine. This is an important distinction, as when looking at the downloads available for the machine type you are shown all the possible downloads for all the combinations of standard hardware the machine sold with. This is annoying when presented with a half dozen different wireless card choices for a laptop. You can usually work out what part you have from looking at the somewhat cryptic system spec sheet.

Sometimes it isn't easy though. I have a laptop, that I know has an Nvidia video card in it. Dell presents three choices for me to download. And from past experience, installing the wrong driver will create graphical weirdness that is only curable with a windows reinstall. A look at the system spec list doesn't help that much.

1	2Y833	ASSEMBLY, CARD (CIRCUIT), GRAPHICS, 32M, NV28, 32 BITS

My choices for files to download are for an nVidia GeForce FX G05650, GeForce FX Go5200, or a GeForce4 4200 Go. From going through this once before (and needing to reimage the machine between mistakes), I know that it is the GeForce4 4200 Go.

To add to this silliness, the download page will frequently list a LONG list of optional perhipherals for the machine. For example, the downloads page for the desktop listed above shows 26 different monitors.

Would it really have been so hard to give me a list of just the applicable software for me?

About Me, or Why am I doing this

The title on my Position Description (PD) is Associate Information Processing Consultant. That's what my business cards say. I call myself Systems Administrator. I handle pretty much everything computer related for about 85 machines (which includes a 50 seat computer lab) and 8 staff members. The workstations and desktops are all Windows based (2k at the time of this writing). The servers are a collection of win2k server, w2k3 server, and Redhat Enterprise Linux derivatives (Tao and Whitebox at the moment).

This blog will hopefully become; a collection of interesting tricks, lessons learned, results of research into products or problems, and other technical tidbits from my life as a sysadmin. Hopefully it will be useful to others.

Blosxom, or My there are a lot of blogging tools out there

So I'm starting a blog. I could write my own software. This wouldn't unreasonable as the rest of my site is maintained with custom perl code using Mason. But seeing as how I have enough projects on my plate (at both work and home), adding another seemed like a bad idea.

A bunch of my friends (seemingly all of them actually) have blogs of some sort. Most of them use a hosted blogging package (like LiveJournal or Blogger. This wasn't an option as I wanted it hosted on my own site. Of self-hosted blogging packages WordPress and MovableType are both used by a couple of my friends. Some research was required.

The MicroContent News Blogging Software Roundup and the Blog Software Breakdown gave me what I felt to be a reasonable overview of what was out there. Seeing as how I didn't want a database involved and I don't care for PHP, I gave Blosxom. It also helped there was a series of articles on it a few months back in Linux Journal, or maybe it was Linux Magazine, that gave me a warm feeling.

So I installed it (which took all of 5 minutes), and here I am. So far so good.