Life of a Sysadmin 2006

Server floppy drive, or Where exactly is the floppy drive?

Below is a picture of a server (click the picture for a bigger version). This particular server has space for six sata drives, a slimline optical drive, and a floppy drive. Guess where the floppy drive is.

Answer: here.

Java 1.3, or I really thought I left Java version hell when I left the computer lab

A user asked to have Java 1.3 installed on his linux workstation. Suspicious of a request for a 5 year old version of Java I stopped by the requester's cube to learn more.

It turned out that one of the companies we were working with provided access to a custom application via a Tarantella setup. Tarantella is a terminal services platform (think VNC or Citrix), and while the web/java client for this particular version of the Tarantella server would load with a modern Java runtime, the application wasn't actually usable because of screen redrawing issues. The user was told that he should use a 1.3 release of Java from Sun.

It of course would have been my preference to tell this company to get with the times (after cursing the creators of Java for the fact that I seem to have more negative experiences with Java than positive ones). But as that was not an option, I went to work out how to have multiple versions of Java available to a browser under linux.

While investigating the feasibilty of this (short answer: While doable under Windows, the solution under linux involves multiple copies of a web browser). It dawned upon me that there was likely a locally installable client that could connect to the server. A quick email to the company hosting the Tarantella server we were trying to connect to got me a copy of the program and I was able to close requestor's bug ticket.

lmcheck.sh, or a script to warn you of license expirations

I needed a way to know when our various flexlm licenses would expire. I suppose I could have simply added the dates to my calendar whenever I added/updated the license files. But since I am not the only one to update the license files (and since sometimes the licenses are updated before the actual expiration), I figured a script that checked for soon to expiring licenses would be the correct solution.

Some experimentation with lmstat and some clarifications from the FlexLM Manual led to the creation of lmcheck.sh. The script should work on any unix system with a modern sh.

The output looks like;

/opt/lmgrd/bin/lmcheck.sh running on hostname at Tue 12 Dec 2006 06:53:16PM EST

The following licenses have expired or are expiring within 7 days

VENDOR      FEATURE                       VERSION             DATE                
------      -------                       -------             ----                
VENDORNAME  FeatureFoo                    1.2.3               4-dec-2006          


The following configs could not be tested
----------
27001@hostname

Cron runs this daily with the command lmcheck.sh | mail -S "lmcheck on hostname" email@example.com

Power outage tip, or How do you get in when the power is out and you only have a swipe card to get in?

We had a power outage at work today. Power was out for at least an hour. My boss was called a few minutes after the power outage happened. Upon arriving, he encoutered a problem. While the keycard reader seemed to have power and even blinked properly upon him waving his card at it, the door lock did not disengage. It seems the card reader system has a battery in it, but the mechanism to power the door lock release is powered by standard building power. Thankfully there were already people in the building to let him in.

We will be obtaining keys shortly.

A Failed CPU Cooler Fan, or heatsinks on modern processors get hot

I received a report that a machine was powering off erratically. After interrogating the user who made the report, I was reasonably sure that the problem was a matter of an overheating processor. Upon opening the case and powering the machine back on, the cpu fan was indeed malfunctioning. It tried desperatly to spin up, but could never quite work out a full rotation. Prodding provided the final bit of needing information; the bearing had fallen out of alignment.

I still felt the need to double check that the cpu was actually overheating. Instead of doing something sensible like booting into the bios menu and looking at the hardware monitor to see the temperature, I touched the heatsink. Ow.

Printing from Linux, or My that's a large test page

The image above is letter sized test page from CUPS printed on an inkjet printer from one of our linux workstations. This is the what happens when you print a test page to our 42 inch wide wide format inkjet. Note the letter sized test page in the upper left of the large image.

Dell Flat Panels, or an impressive failure rate

We have nearly one hundred 19 in. Dell flat panel monitors across four revisions of the hardware. A month ago I would have happily recommended the monitors to anyone. I just boxed up the 21st monitor that has been replaced because of screen burn in issues. There were bad monitors across nearly every batch of monitors purchased over a two year span, and across all four hardware revisions. I no longer recommend Dell flat panels.

My first indication of the epidemic occured nearly two months ago when I swapped out a pair of badly burned-in images for a user (which prompted a few other people to complain of the problem). With a half dozen bad monitors sitting in my office I finally got around to calling Dell. I went into the call expecting this to be simple. The monitors have unique serial numbers. I assumed I would spend some time on hold and then give the nice support technician a list of serial numbers. They would then tell me which monitors were still under warranty and which were not, and set up delivery of replacements for the ones that were. If only it were so easy.

Over the course of 4 hours, 3 phone calls, and about 10 different support technicians I learned a few things. 1) Dell does not repair monitors. 2) Support technicians can not look up warranty information on anything but express service tag (noting that express service tag numbers only come with computers and laptops). 3) Dell does repair monitors. After all of that, I still didn't know if any of the monitors were or were not under warranty.

The next day, my boss gave it a shot. He lasted about 30 minutes before giving up on tech support. In the end, we asked our salesperson to resolve the matter for us. With no response from the salesperson, we rejected the delivery of a fairly sizable order that was being delivered from Dell. We were then put in touch with a very helpful customer service representative who has helped replace all of our bad monitors without hestitation or further wasted time.

It's too bad all of our replacement monitors had been previously loved and half of them were sent to us in conditions that should never had been it through Dell's quality assurance group. Nothing serious, just things like damaged cables, poorly packed for shipping, not including cables. As you might have guessed by this point, we are no longer purchasing Dell monitors.

Mentor Graphics SupportNet, or Super Secret Support Documents

I was seeking documentation on FlexLM usage by products from Mentor Graphics. Their support site was easy enough to find, but every time I clicked a link to a document that looked relevant, I was taken to a login page.

Usability annoyance tangent: The link entitled Learn how to use SupportNet, opens a new browser window with a full window flash applet, and in my case a dialog box explaining "This tutorial was designed to work on screens of 1024x768 or greater, and therefore you may have trouble seeing the entire screen. Note: the tutorial control is located on the bottom of the window". I note that I was doing this from a laptop with a 1024x768 screen.

Not actually wanted an account on the support site, but seeing no other option, I follow the link to Sign Up. The first thing I see is a warning in red "Registration requests are processed within 24 hours of receiving email verification." Sigh, I was hoping to resolve this matter today. I fill out the form and moments later recieve an email asking me to verify my email address and reminding me that they are the only EDA vendor that has 5 STAR support. Who would have guessed that requiring your customers to jump through meaningless hoops is one of the requirements of the STAR awards. I can understand requiring registration to download software, but there is no excuse to lock up the knowledge base and how-to documents.

The link that verified my email address did take me to a page saying I could peruse SupportNet as a lowly guest. I wasn't able to download the updated Mentor specific Flexlm pieces I needed, but I did much of the information I needed. Why must software companies make the lives of systems administrators more difficult?

Virtual MAC Addresses, or Perhaps I should have been more subtle

We use a good of software that is locked up by FlexLM. FlexLM is a license management and enforcement system sold by Macrovision (formerlly Globetrotter) to makers of software. The system can enforce all sorts of policies; most of the time it either locks a program to only run a specific computer (tied to mac address, hardware dongle, ip address, etc.) or allows a vendor daemon running on a server to provide a certain number of client workstations to run the software concurrently.

Each of the nine application suites that we use that use FlexLM have a license file that is tied to a MAC Address. As part of our efforts to clean and make sane our critical infrastructure we made plans to move the FlexLM daemons to a virtual machine. Since VMware does not by default guarantee that a MAC address for a virtual machine will never change, I followed the best practices laid out by VMware to manually set a MAC.

The short version of that best practices document is that the range 00:50:56:00:00:00-00:50:56:3F:FF:FF is available for assignment by the end user. I choose 00:50:56:00:00:01. It seems one of the vendors of an application thought it was fake and questioned it. Oops, I hadn't thought about that issue. Cutting and pasting the output from the ifconfig command put an end to the complaint.

NetApp Service, or Glee at a failed disk

A few Friday's ago, at 11:48pm I recieved an email from our new Network Appliance filer indicating that a hard disk had failed. The subject was "FILESYSTEM DISK NOT RESPONDING". Shortly there after, I recieved an email from my boss (who was at the time in the process of transitioning to said new filer);

"Score! A disk failure in the middle of the rsync."

A bit later (at 1:35am) we recieved an email from Netapp asking us to confirm the address we wanted the new drive sent to and to confirm that someone would be there for the next several hours. It seems we have four hour repair service for our filer, and that includes getting us replacements on weekends and in middle of the night.

Slight Tangent: The drive traveled less than 10 miles from a UPS logistics warehouse to the company. Had I as a random person paid for that UPS SonicAir service, it would have cost nearly $150. Woweee

Locked in the bathroom, or This is an IT problem?

If you are a fire marshall, building inspector, or hold a similar position, please skip this post. I received an email from the office manager explaining that she and several others had gotten trapped in the bathroom hallway ealier in the day and asking me if I could do anything about it.

Background: The company I work for shares the bottom floor of a building with another tenant. The bathrooms are in a T shaped hallway between the two suites. An interesting property of this hallway, is that when you enter the hallway, you get locked into the hallway and need either a keycode (to get into the other suite) or a security card (to get into our suite). There is an emergency release (looks like a fire alarm, only it's yellow) next to the door into our suite.

Now you might be wondering why, what sounds like a facilities problem was being brought to the attention of IT. She assumed that there was something wrong with the card reader and thus we needed to fix it since we manage the security system. This was my first chance to seriously poke about at the computer running the alarm and keycard system. Unfortunately I found nothing wrong. So off to investigate the door I went.

It took me about five minutes to work out the problem. The card reader was indeed reading each and every swipe of a card, and you could always hear a noise from the lock mechanism. What was odd however was that the lock mechanism made two different noises. One when the lock actually opened properly, and another when it didn't. It was pretty clear that that the solenoid that released the lock and allowed the door to open was not working correctly; probably sticking at times.

It was as I stood there experimenting with the door that the office manager came and explained how the emergency release was supposed to work. After activating the emergency release a few times, it became quiet clear however that this release sends the same electrical signal to the lock as the card reader, and thus if the problem is with the lock itself, the emergency release won't actually let you out of the hallway.

My days certainly are never routine.

Time Synchronization, or Why is the default so complex?

Before this experience, I was under the impression that the reference implementation of NTP by the NTP Project was the bees knees. I have since come to have a very different opinion of the program. It all started with the need to setup a pair of NTP servers.

First off I needed to get the correct time on the servers. After changing the default server entries from the global pool.ntp.org entries to the country specific us.pool.ntp.org and adding entries to the step-tickers file (this enables the init script for ntpd to specifically set the time from the listed servers upon daemon startup. Why exactly isn't this the default?) I had the correct time on my servers. This step was easy enough.

Next, I had to get the server to accept requests from other machines on the network. Redhat kindly commented up ntp.conf. The relevant section is;

 # -- CLIENT NETWORK -------
 # Permit systems on this network to synchronize with this
 # time service.  Do not permit those systems to modify the
 # configuration of this service.  Also, do not use those
 # systems as peers for synchronization.
 # restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

Wait a minute, to allow a network to request time from the server, I start a line with restrict? You have got to be kidding me. Anyway, I make the change to make it applicable to the network, and from another machine, queried the server for the time. The request was not met with an answer that pleased me though.

[root@server ~]# ntpdate -q 192.168.0.1
server 192.168.0.1, stratum 0, offset 0.000000, delay 0.00000
2 Sep 22:27:26 ntpdate[2674]: no server suitable for synchronization 
found

Which leads to perhaps the biggest problem with the project: some of the worst documentation I have encountered. You can't blame them for lack of documentation. They have lots of documentation. Lots and lots of documentation. That is really the problem. I can assure the authors of all that written material that very very few people care about the gritty details of how NTP works. People just want a simple, straight-forward, and reliable way to get sub second accurate time on all of their machines.

I spent nearly four hours reading nearly all of that documentation. I not only found the answer to my problem; ntpd won't respond to requests for time until it is confident that it has the accurate time, which takes a few minutes after each restart of the daemon. In the end, I found that someone had already gone through the pain of working out the correct config for ntp.conf and Redhat ships that nicely documented config that will work for the majority of administrators.

After all of this, I am left with the feeling that there has got to be a simpler and more straightforward ntp daemon. Thankfully, there are indeed alternatives to the reference implementation of NTP. There is Chrony, or OpenNTPD from the fine OpenBSD group. I have begun using the later at home. Next time I need to solve this problem at work, perhaps I will move away from the reference implementation.

Setting up Conference Rooms in Outlook/Exchange, or You have got to be kidding me

We were expanding into a new building and I was tasked with setting up the new conference rooms so that they could be scheduled through Outlook/Exchange. I recall from many years ago that there was much idiocy to setting up such things, so I asked the interweb for assistance.

Early in my searches I came across a page from Microsoft TechNet entitled Set Up a Conference Room as an Outlook 2000 Resource (another set of instructions doing the same thing is here). I followed the instructions (Ignoring how silly it is to need to create a profile in Outlook for each resource you wish to manage) and surprise surprise it works. Great, on to test it with Evolution and the Exchange Connector. Nope, it doesn't work. But clearly it should work from the Outlook Web Interface provided by the Exchange Server, right? Hmmm, no that doesn't work there either.

I guess the page was serious with the prerequisite of "You must be using Microsoft Outlook 2000 and Microsoft Exchange Server 5.5." Further investigation confirms that that solution only works when using Outlook 2000 or greater as the client to setup the meeting. More info here (note Windows IT Pro magazine subscription required to access).

Still further investigation yields an interesting page entitled Scheduling Resources for Microsoft Outlook that says that there are two primary ways to allow Outlook users to schedule shared resources automatically. 1) setup each resource as an Exchange Server mailbox and do various bits of trickery to make it auto accept meeting requests or 2) create a public folder that holds appointment items and allow various groups of users permission to read and write to it.

In the end I find that the conference rooms we already have in our Exchange system were implemented using the first option with a tool called the Microsoft Exchange Server Auto Accept Agent (download here. I created new users (this does of course use up a CAL) to represent each of the new conference rooms, setup Outlook profiles for each so I could change permissions on the new calendars, and finally, using the command line VB scripts from the Auto Accept Agent, added the new conference rooms to the monitored mailboxes list.

The Auto Accept Agent basically snags incoming meeting requests to registered mailboxes and processes them based on criteria (if the room is available, if the event is in the future, etc.). Registering mailboxes to check is done through a trio of command line VB scripts. Managing the behavior is done through editing an xml file. Another more full featured option (which I would have used had MS's solution not already been configured on the server) appears to be the open source AutoAccept Sink for Exchange

The sad part, is that about half way through the day (about the time I learned that Outlook and the Outlook Web Interface behaved differently) I took a break from fighting with Exchange and Outlook and voluntarily went to read the Sun Grid Engine documentation. When a product is so frustratingly annoying that I voluntarily go to read a very very dull manual to take a break, there is clearly something wrong.

A SATA Back Plane, or A small annoyance

Last week, I ordered a server to run a tape jukebox and perform backups for all our servers. To save a little money, it was ordered without drives, as we have a small pile of 250 gb drives sitting around after having upgraded a decent sized sata array.

The server (in one of these chassis) showed up, I set it in a rack, and went to install a pair of hard drives. For some reason the drives weren't being recognized. Upon further investigation, it seems the layout of the bays up-front do not match the locations of the sata connectors on the back of the back plane.

The bays up front are numbered

0 3
1 4
2 5

The ports at the back of the back plane are numbered

4 5
2 3
0 1

How annoying. How dumb.

A Change in Job Duties, or Silicon Valley here I come

For the past few months, my department has been a pawn in a varity of political games. This has effectively derailed or delayed every project I was working on or would have started at the end of the spring semester. About three weeks ago, decisions were made, changes were coming. I was faced with the prospect of having to assist in the dismantling of the technical infrastructure I had built up. Once finished with that, I would be left with a very different set of duties than I had been doing over the last two years.

No longer would I would be the person making all (or even most) of the technical decisions. No longer would I be able to dabble in every aspect of IT. No longer would I be researching and developing the policy. Worse than all that though, was that I would be asked to do work that didn't interest me intellectually.

With encouragement and assistance from a good friend, an opportunity was presented to me that my wife and I were unable to pass up; a job that would be a challenge, in a land of nearly perpetually nice weather. So with less than 3 weeks notice, I find myself leaving America's Dairyland and heading for Silicon Valley.

This of course means that I am leaving the happy-go-lucky world of academic freedom and entering the world of non-disclosure agreements. What this means for this blog is yet to be worked out. I would expect to continue to be able to continue to write the types of pieces I have been writing. There will definitely be a break for awhile as I find my feet in a new job and in a new city.

Fedora Core, or An inappropriate linux distribution for a server

I simply don't understand why anyone would use Fedora Core in a workplace, let alone on servers. I can understand wanting to avoid paying yearly per system licensing fees for Redhat Enterprise Linux, but major upgrades and bleeding edge software every 6-12 months is not something that should be done in a business.

There are however alternatives to those two extremes. There are a bunch of RHEL Forks. Each project builds a distribution based on the source rpms made available by Redhat for Redhat Enterprise Linux. Each project has slightly different goals. Scientific Linux for example endeavors to be RedHat comatible while still adding in various clustering goodies used by researchers. My current choice for a straight forward, staying true to RHEL distribution is CentOS.

Using this type of project however, is not for everyone. Red Hat provides support options and the percieved stability of security updates and patches coming from a company; these might be an issue for some managers. The other big issues, are all about mitigating your level of risk when using a completely volunter community project.

What would happen if the project weren't to put out security patches as fast as you need? Do you have the knowledge and skills to rebuild the source rpms yourself? What if the project collapses? How quickly would you be able to migrate to another distribution? (One option, is to migrate in place with these instructions as a guide.) And the most devasting of possible issues; What happens if Redhat stops releasing source rpms? Would you be able to hand patch the services you maintain until you could migrate to another distribution?

If those risks scare you, perhaps you will be more willing to pay for the licenses from Redhat. These risks don't bother me because 1) we don't have to be a 100% uptime workplace and 2) I have the skills needed to maintain it all myself as I worked on a migration plan.

Private Server Networks, or A Great Step for Security

For the best possible security, servers should be on a seperate network from any machines that connect to them and the traffic to and from the servers should be restricted by a firewall with active intrusion detection monitoring.

That type of firewall is complex to manage and likely to be quite expensive (in general, throughput is a major factor in the cost of a firewall). The benefits of such a setup are unlikely to surpass the limitations and expenses encurred. The opposite end of the spectrum is to plop your servers onto the same network as all of your machins and do everything on that one network.

A good in-between setup is to place your servers on two separate networks and move all services that you can from the network shared with the workstations to the server only network (effectively setting up an out-of-band network).

Each of my servers has at least two network interfaces (mostly dual port Intel Pro 1000/MT Server Adapters). One of those interfaces is connected at 100 megabit to the general network shared with all of the workstations. The other uses a private ip addressThis setup has provided performance improvements and increased security. The performance is only real noticed when performing backups, although it has given me the bandwidth needed to experiment with the idea of moving my VMWare images to a NAS like device.

For the security improvements, I needed to move services from the public network to the private one. I was able to relatively easily move my snmp queries, backup process, and ssh access to be accessible to only the private network. Now if only I could work out how to only enable Windows Remote Desktop on just one interface.

Tabasco Sauce, or To bad I'm not in the market for the Product

There was a box in my mailbox the other day. At first I thought it was a new batch of CDs for TechNet subscription. The box turned out to be the wrong size, and most definately the wrong color.

Now I have become pretty calloused in throwing out out flashy advertisements, but this one had a shiny red box! It turned out to be an advertisement for ExaGrid's disk-based backup system.

Now I'm not in the market for a backup system; and I am a bit uneasy with black box backup system hardware. But I did get a little bottle of Tabasco Sauce, so it wasn't a total loss.

Colophon, or What I use to make this blog go

As I tweak various bits on the blog, I thought I should share what all makes this blog go. The webserver is Apache running on Solaris on Sun hardware with an UltraSPARC processor. The blog software is Blosxom.

Tangent: Now Blosxom hasn't really actively been developed since 2003. And the author migrated away from the package in early 2006. I don't let such things bother me though, as I am used to choosing software packages and products that aren't really the most popular or mainstream. There is however an active User Group, Yahoo group, and a SourceForge Group.

Now blosxom is a darned simple package, less than 400 lines of perl parsing text files in a simple directory structure. That simplicity is part of what attracted me to the package, but it does mean I have a few plugins to add or refine various features.

categorytree: This provides the category list on the right.
flatarchives: This plugin provides the archive list on the right.
preview: A plugin that provides a way for me to see posts on the blog that I am working on, without showing those posts to the general public.
strip_unix_comments: I use unix style comments in my entries to put various meta data for my own benefit. It wouldn't be bad if the public saw them, but they are really only beneficial to me.
date_fullname: A simple plugin that provides templates with the full month name.
flavourdir: Allows me to put my flavour files (the templates used for the html and rss versions of this page) in a different folder than the default.
imagesizer: This plugin automatically puts in the height and width tags for images.

Self Destruct Button, or What a strange ISA card

Written 2006-05-03

click image for a full view of the card

This card amused me and has puzzled many friends and acquaintances over the years. It is indeed as simple as it looks, an ISA card with a momentary switch connecting two pins on the card slot. The back is just as simple as the front. I not sure as to what it was use for, my best guess is that it was used to manually trigger an interrupt for hardware developers.

If you have a good idea as to what it is, please contact me so I can update this post.

Contacting me, or You no longer need to dig up an email address

I have been suprised at the number of people that have gone to the trouble of finding an email address for me to provide comments and compliments about various entries. So I suppose I should provide an easily accessible address.

So consider this an invitation to email me with comments, criticisms, and what not about this blog. The email address is sysadmin followed by the @ sign, with the domain "fief.org" after it.

I apologize for the annoyance of presenting my email address this way, but spammers are doing their best to make email useless, and I must fight back to keep my email a useful communication tool.

Outlook Express, or A reminder of why I avoid the program

A user came into my office this morning saying that he was having troubles with Outlook Express. A few questions later, and I learn that the program crashed whenever he tried to open his inbox. I had seen the problem before, it was undoubtebly because of a corrupt dbx file (the format used by Outlook Express to save folders full of messages).

After making a copy of the Outlook Express folder from his Application Data directory, we tried compacting the folder followed by compacting all folders. Outlook Express would still crash upon opening his inbox. We deleted the folders.dbx file hoping that the central index was the problem. That didn't solve the problem either. Searching for assistance from Microsoft, I come across the page: The Other E-Mail Threat: File Corruption in Outlook Express

Tangent: I find databases and other binary structures for storing mail to be overkill and a bad idea. The primary argument used for why it is a good idea is to make searching and manipulating large mailboxes faster. Sure, it can be faster, but plenty of email clients do a fine job without storing your mail away in a binary blob. Mail should be stored in a nice simple mbox related format. While mbox certainly has it's own problems, at least I have never seen a mail client crash from a corrupt mail file, and when I did see an instance of a client breaking the file, I was able to recover nearly all of the messages by hand with a simple text editor. Plus, text files make it much easier to migrate your mail to another client should that become necessary.

I was horrified by the article. They were advocating purchasing software to solve what is apparently a common fault with Outlook Express. Besides DBXtract (the product recommend in the article above), there are many, other tools to recover corrupt dbx files. There is simply no excuse for this. If this problem is common enough to have spawned that many products to fix it, Microsoft needs to get it's act together, fix Outlook Express and/or ship as part of the program a method to repair corrupt db files.

With no intention to purchase software to support software that isn't on our supported software list, I provided him with the most recent backup of the files and he was able to get back up and running.

Drive carriers, or I'm being charged for screws?

I wanted to add a pair of hard drives to a server. I had the drives, but I needed a few of Dell's custom mounting trays to use them. My sales rep sent me a quote for the parts (not available through the website without a hard drive it seems). The trays would be $10.95 each. There was another item with on the quote; 8 "SCR,6-32X1/4,FLH,MS,ZPS,CTSK" at $.05 each. I asked the sales rep about it, and he said they were screws. I can't believe Dell is going to bother to charge me for 20 cents worth of screws. Why didn't they just add another buck to the cost of the drive tray and call it done. Heck, they could charge $20 per tray and I wouldn't think much of it.

Amusing Tangent 1: The quote came with a from address "@del.com". I can understand them owning that domain, which they do, it should be a silent redirect, and it looks quite unprofessional to use it for email.

Amusing Tangent 2: I placed the order and recieved a confirmation email a few minutes later. This confirmation showed that I had ordered 2 hard drive carriers, 8 screws, and 5 of my salesrep. I sure hope he can share with himself as the only space I have for him at work is a small paper closet.

What the heck, or Yes it really is what you think it is

On one end is a parallel port connection. The other has a an rj45 ethernet jack and a barrel plug connecting to an inline ps2 style connector. Yep, this is indeed a parallel port ethernet adapter. The ps2 plug provides power from the keyboard port to the adapter. A particularly nice touch, is that the red stripe is a rubber belt with notches on it that when spun around the body turns the screws that lock the adapter to the port.

I last used this in college on a Windows ME laptop with a dead pcmcia slot. It wasn't particularly fast, it chewed up the processor, but it provided enough of a network connection to transfer all the data off of the machine. It's a neat device, but considering USB has basically been standard since Pentium IIs, and USB flash disks and ethernet adapters are so cheap, this has been relegated to my shelf of cool old stuff.

Intel (who purchased Xircom in 2001) has a support site (including drivers) up for the product.

Boot CDs, or How to shrink your cd wallet

The majority of cds that come into my office get ripped to an iso, stored on the file server, and put into a an ugly cd storage box, hopefully not to be touched again. Unfortunately, not all of my cds can be put away, I had nearly 20 cds that I still needed use to for installation and troubleshooting.

With the Ultimate Boot CD, I reduced the number of cds on my desk to 6 (that includes 4 OS install discs). Before customizing it is simply a collection of free bootable disk images with a menu system to select between them (note, I recommend that ALL people who work with computers have this cd). After my additions, it eliminates nearly all of the cds I used to have to keep around by putting them all on one.

My custom additions include;

Acronis Disk Director
4 boot discs for Altiris Deployment Solution for various systems
4 install discs for Altiris Deployment Solution for various systems
BIOS flashers for all of the systems we support

I created the ISO of Disk Director with LC ISO Creator. The Altiris Deployment Solution created those ISOs. The bios flashers were made from the boot floppy creators provided by Dell and saved using a Virtual floppy Drive. After making my changes, I created a new iso with Nero and tested the changes under VMWare.

Note on Making Your Custom Disc Bootable: To make the disc bootable has different settings under different burning packages. Under Nero, you need to change a few things in the boot tab of the disc properties; The image file is in "UBCDdir\boot\loader.bin\". Under the expert settings; kind of emulation should be set to "no emulation", Load segment of sectors is "07C0", and number of loaded sectors is 4.

ELDump, or How to automate extraction of log data under windows

I have been looking for a way to easily (and cheaply) acquire statistics on users of my lab. I want to know things like; How many unique users use the lab get each day/week/month/semester? How often does the average student stay logged in? Do all of our users login in a given month/semester?

A bit of searching by a coworker found that events were logged to the primary domain controller's security log with event id 680 whenever someone attempts to login. He was further able to work out from an export of the log answers to some of the questions we sought answers to.

My coworker was on vacation last week, and he tasked me with exporting the logs on Monday; I forgot. So this morning (when I was reminded a week late (user error put it on the wrong date) by my Palm of the task), I sought a way to make a scheduled task of it. With the program ELDump, I was able to construct a command line to perform the export. It was then trivial to wrap it in a batch file and set it up as a scheduled task.

The batch file:
SET TODAY=%DATE% SET YEAR=%TODAY:~-4% SET DAY=%TODAY:~-7,-5% SET MONTH=%TODAY:~-10,-8% "c:\Program Files\ELDump\ELdump.exe" -e 680 -m Security -l security -c , -M -A 192 -O "dtus" > "c:\logs\event680_%YEAR%%MONTH%%DAY%.csv"

That batch file spits out a csv file that tells the who (what user), where (from what machine), and when that we care about for each login. With some appropriate crunching, my coworker can now tell us when the lab is most used, how many unique people use the lab in a span of time, what the average number of users per day we see, and answers to other similar questions. While none of the results were a real surprise to us, it is nice to know that we can now provide actual numbers to the powers above and grant submissions.

Magazine Subscriptions, or Gosh I got a lot of crap

Each month I receive dozens of magazines (the picture below shows the pile created by one months worth of magazines that arrived in my mailbox), the vast majority of them are free advertising paid for drivel (my predcessor had a thing for free tech rags). I actually pay for and read regularly just four technical magazines.

Windows IT Pro: I first read this magazine in college when it was called Windows NT Magazine (since 1999 it has gone through the names Windows 2000 Magazine and Windows and .NET Magazine before settling on the current name). Previews of new Microsoft software, reviews of all sorts of enterprise software, and indepth how-to articles continue to make this a must read for all Windows administrators.

SysAdmin: A magazine geared toward the professional unix adminstrator (with details for Solaris and Linux most frequently). Each month is obstensibly filled with articles centering around a theme. While the articles don't always relate too closely to the theme, they are always filled with serious technical know how and real world experiences from the authors.

2600: The Hacker Quarterly: Not a magazine that has much immediately applicable knowledge for my job, but one that continues to encourage me to be paranoid and think cynically about businesses and the world.

Computer Power User: This magazine aims for the gaming, modder, and obsessive tweaker audiences. While they do focus a good deal on the latest and greatest videocards and processors, there are plenty of articles on useful utilities and troubleshooting tips that make it a worthwhile read. If I weren't interested in the rest of the articles for my non-work life it probably wouldn't be worth the subscription though.

StudioMX Activation Woes, or Macromedia Blames the User for Their Inadequacies

I installed Macromedia Studio MX 2004 and all of the relevant updates on a coworker's machine. After rebooting, I was asked to, and did, activate the product. Logging in as a normal user verified that all was happy.

It was a surprise to me when later that week, my coworker stopped by saying that Dreamweaver was asking to be activated again. I had her reactivate and noted that I should check up on the matter in a later that week. A few days later, she tells me that she is asked to reactivate the software each time she reboots the computer.

While perusing the Activation Support Center, I call up the support number and quickly get through to a member of the "Product Activation Team".

Call one: After learing that the computer boots into two different operating systems from the same drive, I am told that Macromedia does not support this configuration and the tech quite simply states that she can not offer any further help. She points me to Service Note 18789, entitled "Partitioning and emulation software". Since I still needed to deal with other matters today, I put it on hold until the next day.

Call two: Explaining that I am being asked to reactivate the software upon each reboot, the "activation suport specialist" learns that there are two hard drives in the system and indicates that Macromedia does not support dual hard drive configurations. He points me to the EULA and the support representative says that all he can do is "increase my activation install quota by one notch". When I point out that I am aware of the brain dead limitations of Macromedia's activation system in regards to RAID configurations, and that the two drives in the system are not in such a setup. He points me to the EULA saying the issue is clarified there.

Scanning the EULA quickly, all I find that seems to be relevant is paragraph 'i' in section 2: "You agree that Macromedia may use those measures and you agree to follow any requirements regarding such technological measures." Inquires to learn what those requirements are leads nowhere. Thankully the tech from call number two provided me with a number to reach the activation team directly (800-945-9049), instead of going through the technical support phone maze.

A week goes by with activation happening a handful of times as my coworker uses the software. Seeking an answer to the question "Is there anything wrong with repeatedly activating on the same hardware?" I make another call to activation support.

Call Three: Teh tech, upon hearing the situation, asks what version of version I have installed. Upon hearing it is 7.2, the support technician suggests installing Service Note 19468 entitled "Reactivation failure after upgrading to Flash 7.2". I quickly install the hotfix referenced in the service note, reboot, and reactivate the software. Several reboots later, and it appears as if the problem is solved. The nice activation support represetantive does also answer my question; At some point, continuously reactivating would cause an error that would need to be resolved by speaking with technical support.

An Overheated Server Room, or Data in the form of pretty graphs

I have always felt my server room has been hot. Informal observations with a simple thermometer showed temperatures hovering in the mid to high 70s, with not-infrequent forays into the low 80s and the rare spike to nearly 90 on days when the air conditioner stops spitting out cold. With money in this years budget that has not yet been planned for it was time to consider replacing the inadequate window air conditioning unit with something more appropriate. But before I make plans to spend to spend a few thousand dollars on an air conditioner and installation labor, I needed more solid data.

I sought an inexpensive (less than $500) device that could handle at least four temperature sensors, required no server side software, and could be queried by my cacti host (preferably via snmp). As far as I could find, there were two options; the APC Environmental Monitoring Unit and the IT Watchdogs WeatherGoose. (Note: it seems APC is replacing their own environmental monitoring line with the products of the acquired company NetBotz

I ended up choosing the WeatherGoose (online demo) as it more easily handled more than two remote sensors and it provided a cleaner interface and simpler ways to ge log data out of the device. With a 30 day satisfaction guarantee, I placed an order for the base unit, a door sensor, and two remote temperature sensors, all for a little over $400.

Installation would have been painless, had I not had to fish some of the probes through a suspended ceiling without the appropriate tools. Not including pulling the sensor cables through the ceiling, I was seeing data on the web interface (demo) in under 30 minutes. All in all, my only real complaint is that the unit has a damned wall wart. More on more Cacti setup real soon.

A Survey, or I sure feel valued

Written 2006-02-24

"Dear Valued" the email began. Not "Dear Valued Customer", just "Dear Valued". I was being invited to participate in a customer satisfaction survey. Following the provided link, I was presented with a page that didn't give me much faith in the company VMWare had hired.

I can't say that I went any further.

Updated late on 2006-02-24

Hours after I made this post, a product manager from the VMTN sent me an email telling me he has passed along this silliness to the appropriate folks within VMWare and thanking me for using their products.

QuarkXPress Activation Woes, or I Would Be Happy to Make a Directory World Writeable

The installation of QuarkXPress 6.1 and the upgrade to 6.5 went smoothly enough (QuarkXPress 6.5 was released in November of 2004, why isn't there a single integrated installer?), but upon logging in and running Quark as a regular user, I recieve the message "The activation file for this copy of QuarkXPress 6.0 has been corrupted. You will need to reinstall this copy of QuarkXPress 6.0."

The very thought that I would need to reinstall the software to solve an activation problem really irked me. Since Quark runs as expected as the administrative user, I assume that the issue is with permissions of some files and I go diving into the Quark technical support database for help. Not finding much of use immediately, I call the tech support number and hang out on hold while I continue searching their site.

Tangent: I can only assume it is complete carelessness that allows the hold music of most companies to be so painful. Quark seems to have recorded their hold music with a kids tape recorder stuck in the cone of a Victrola phonograph playing in the backseat of a Hummer undergoing field testing. Worse is that the loop is less than a minute.

Finally (after enjoying 22 minutes of hold music) reaching someone in technical support, I explain the problem and the tech immediately knows what is wrong. He instructs me to provide "Full Control" to "Everyone" to the folder c:\Documents and Settings\All Users\Application Data\Quark. Inquires for more details confirmed that 1) just the users who wish to run QuarkXPress need "Full Control" and 2) Quark is unconcerned that they are recommending settings that should make most systems administrators cringe.

RF Choke, or What a Quaint Little Accessory

From my shelf of amusing old stuff; an accessory kit from ATI to add ferrite cores to "non-ferrited" video cables.

In the box are some ferrite cores, zip ties, and instructions (front, back). While there is no date on any of the material, the instructions do give an idea as to when the product was made, the image of the video connector appears to show a 9 pin one, likely for a CGA monitor. That is all.

Mandatory Vendors Disappoint, or Outrageous Prices Abound

Needing a variety of power, ethernet, and kvm cables, I send emails to the three vendors I am legally allowed to purchase from. One particular vendor responded very promptly, although it required a few additional emails to get the quote right.

To:salesrep From: sysadmin Subject:Cords and Cables Oh My It's time to neaten my server benches. Can I get a quote for the following cable order? 16 6ft standard computer power cords 4 3ft cat5e rated ethernet patch cables 4 10ft cat5e rated ethernet patch cables in the same color as the 4 3ft ones. 16 15ft cat5e rated ethernet patch cables 4 3ft ps2 kvm cables 4 10ft ps2 kvm cables Thanks Sysadmin

To:Sysadmin From: salesrep Subject:Quote No: AA-1234 date: 1/27/06 Attachment:Quote.pdf Hi Brian, By standard computer power cords, you mean USB cables, correct? Regards, Sales Rep

To:salesrep@example.com From: sysadmin Subject:Quote No: AA-1234 date: 1/27/06 > By standard computer power cords, you mean USB cables, correct? I mean the cord that plugs a computer into a wall outlet. Sincerely Sysadmin

To:salesrep@example.com From: sysadmin Subject:Quote No: AA-1234 date: 1/27/06 Hi Sysadmin, To quote that, I would need the PC make & model. Regards, Sales Rep Team Lead

To:salesrep@example.com From: sysadmin Subject:Quote No: AA-1234 date: 1/27/06 I would like the quote to includes prices for these... http://www.cyberguys.com/templates/searchdetail.asp?T1=120+2140

The response to that was a correct quote. This exchange didn't exactly give me confidence in the company, but to make matters worse, the quote I recieved did not include the needed legalese that matched their agreement with the State, and had someone elses name and address in the Bill to and Ship To fields. The final price would have been $531.

The other two vendors were much better in terms of service, but weren't even in the correct ballpark for what I wanted to pay. One gave me a quote for $333 (this being the price after I removed the incorrect inclusion of sales tax; which is wrong both because I am in another state and because my purchases are tax exempt). The other gave me an accurate quote on the first try but wanted $488 for the order.

Going through Cyberguys (where I personally order computer cables) I assembled a shopping cart with all the needed parts in less than 10 minutes for a grand total of $153.

Sysadmins Law 119, or fscks Always Happen at the Most Inconvenient Time

You would think I would learn; pretty much everytime I am do hardware maintenance on a linux system, I happen to time one of my reboots so that one of the automated fscks is trigged. Either too much time has passed since the last one or the partition has been mounted too many times (both can be set/reset by tune2fs). It is almost always one of my large partitions which takes a good while to check, meaning I sit around twiddling my thumbs not wanting to start anything else until I finish the maintenance. Thus I present sysadmins law 119.

Automated file system checks (those not triggered by an error) always happen at the most inopportune times. Either reset the counters or do a check before starting maintenance.

Floppy Disk Protection, or Computer Paraphernalia You Could Find In My Office

My office is littered with random computer paraphernalia. Some are parts that are still useful, many others are antiquated and would be considered trash by most people. This bit is one that I find too amusing to even consider throwing out. In the box is A Devoke Data Products Disk-Pro-Tek Floppy Disk Reinforcing Kit. As the instructions (scan) state, they "Extend the life of your flexible discs and mini-flexible discs by strengthening the spindle hole and thereby substantially reducing the chances of disc dimpling, coating removal, and permanent distortion."

While I never actually used 8 in. floppies, I used plenty of 5 in. ones and rarely had issues that this kit would solve. Overly paranoid computer users do all sorts of silly things (for example d_skin Protective Disc Skins). If a particular disk is that valuable, a copy should be made and the copy should be the one used on a regular basis.

These are the actual reinforcement labels. They basically work like the reinforcements thatare used with standard paper hole punches.

This kit includes a double sided (one side is for floppies (8 in.) and the other for mini-floppies (5 in.)) applicator.

Offsite Backup, or Another Layer of Protection

While I am confident in my onsite backup system, to handle the possibility of my server room going up in my smoke, I need offsite backups. I have about 600 gigs used on my primary backup server, but I only need to store about 300 of that at an offsite location to be able to recover from a complete server room meltdown.

Taking tapes offsite is of course the classic way to solve this problem. This means user intervention on a regular basis (which means it is more likely to be skipped), and it means using tape, something with which I have a great dislike (mostly caused by bad experiences with QIC tapes in the early 90s).

I could contract out to an outsider service, and the central IT group on campus will sell me storage on a massively redundant Tivoli managed backup system for two dollars per gig per month. That's $600 a month or $7200 a year for my 300 gigs.

Or, I could build a server and host it some place else. Turns out I wasn't the only person on campus who had a desire to host a server in an "offsite" (meaing not in the same or an adjacent building) location. I easily found another systems administrator on campus who was willing to swap space in our respective server rooms.

For $2500 I can build a server that will meet my expected future needs for at least three years. Sure the server likely wouldn't be as robust as the Tivoli managed service, but I don't really need that level of service, I simply need another layer of protection.

xcacls.vbs, or Microsoft's Command Line NTFS tools sure do suck

I have been working on a single script that will do user account creation. I'll cover the script itself later, right now I feel I call attention to the awful programs xcacls.exe and xcacls.vbs.

After experimenting with xcacls.exe (download) to modify the ACLs of home directories, I thought I had down everything I needed to do. Opening the GUI to verify that my carefully crafted command lines did what I expected them to do, I was presented with an interesting message-- The permissions on FolderName are incorrectly ordered, which may cause some entries to be ineffective. Press OK to continue and sort the permissions correctly, or Cancel to reset the permissions. It seems there is a problem with xcacls.exe. Apparently using the program in a way that is concistent with the instructions is not supported,

So I download xcacls.vbs and start experimenting with it. It took about thirty minutes of experimentation to work out the new features and the differences with the previous version, but it seems the script does solve my problem. That is not to say it is a well written script. An annoyance: you can't designate the username in the active directory form of username@domainname; just NT style ntdomain\username. A problem: the program takes 5-10 seconds to process the ACLs on about a dozen files/folders.

I can only wonder if had the script had been implemented and compiled to a native binary it would have been as fast as the original program. This is also another reminder that Microsoft expects me to learn and code all of my utilites in VBScript. I think not.

Port Configuration, or A Lack of Imagination on the Part of the Campus Network Architects

The only access I have to do network configuration for ports under my control is via a custom campus written web application. For each port I can configure things like rate, duplex setting, and what vlan it is on. This system has been in place for nearly two years, and just last month they finally made it possible to lock specific jacks to specific MAC addresses.

Tangent: MAC address filtering is not secure in and of itself. Spoofing the MAC address a card responds to is possible with pretty much every network card and OS I have used in the past several years; it can even be done in the bios on some motherboards. It is however quite an effective deterrant against casual attempts to hook non-sanctioned equipment.

Now this new feature only allows you to lock a single port to a single MAC address. This is a useful thing for most systems administrators on campus. Being able to limit which computers professors plug into the network jack in their office will most definitely improve the overall well being of campus networks. I had hoped however for a system where I could setup lists of addresses and I could specify that a port should be restricted to one of the lists (the simplest form of course being a single address being locked to a single port). My hopes however were dashed with the last section of the introductory document announcing the MAC address locking feature.

It seems the campus-wide network architecure team feels there are political and logistical reasons (which they choose not to share) not to provide list based locking. The only explanation they provide is that it is better network design to provide each device with its own jack (this is a concept I do generally agree with).

Clearly the campus-wide network architecture team needs some more creative thinkers on it. I can think of a few situations where it would be useful.

My lab solves serious workstation troubles (be them software or hardware) by replacing the problem machine with a spare sitting in the closet. Without having a list the person doing the replacing would need to be able to access the configuration application to change what MAC address is locked to that port (and this can't be done as the lab assistants that would be doing the replacing do not have the technical knowledge to use the tool safely).
I partially manage the ports in several public classrooms. I would like to restrict use of those ports to my collection of laptops that instructors may borrow.
A faculty member wishes to be able to use their laptop in their office. It's not worth installing a second physical jack for this occassional use, but the sysadmin still wishes to limit the jack to only the desktop and laptop.

I am annoyed at this mostly because of what should be possible, and not by what I actually need now. I have much bigger fish to fry before I get around to MAC address locking at the switch.

Silly Putty, or How to Pick Up Chicks

I have sitting on my desk about two pounds of Dow Corning 3179 Dilatant Compound, known to most people as Silly Putty

Anyone that knows me would agree that if I don't have something to fidget with, I will find something to fidget with (or worse, take apart). I have had this wad of Silly Putty in my office for about 6 months now. It acts as a medium for temporary artistic endeavors, provides unbreakable stress relief, and can be a relaxing focus as it oozes from whatever form I give it to its natural blob like form.

There is one other benefit that I had never considered before. a blob of Silly Putty can apparently be used to pick up women. Now I have found that woman are much more likely to ask to play with one of the desk toys as I solve whatever problem they came to see me about it. But for some reason the Silly Putty is more attractive to them than any of the other gadgets, gizmos, and stress toys I have. So to all those single guys out there-- Silly Putty, better than a cheesy pickup line.