Life of a Sysadmin

The occassional trials and tribulations of a jack of all tr ades sysadmin in a startup in Silicon Valley

March 2008

Damaged Users, or Geez does QuickBooks suck

The office manager was updating from QuickBooks 2005 to QuickBooks 2008. With a fresh backup of the data from QuickBooks 2005, we went to import the data in to the new version. "An error occurred when QuickBooks tried to access the company file" when converting a file to QuickBooks 2008 (Error -13,0)".

I guess it's not going to be as simple of an upgrade as I had hoped (or the manual led us to believe). Good thing Intuit has a knowledge base article on Error -13,0. Let's look at the three suggested problems and the related solutions.

  1. Shortcuts on the icon bars to features that are no longer in the product such as the Open PO List: So, if they can recognize that there is a bad shortcut, why can't they fix this automatically? More fun, is that they provide only one possible shortcut that could be causing the problem. Couldn't they at least tell me which shortcut is causing the error?
  2. Damaged Admin user password: Wait a minute, so the password I provided moments earlier to access the backup file (and the step that presented an error when provided the wrong password) isn't good? And they ask me to fix it by simply changing the admin password? How lame.
  3. Damaged Users: I certainly wouldn't have called the office managed damaged. In fact, she's a pretty smart cookie. Oh, I see, it's one of the user accounts that's causing the problem. Once again, couldn't they tell me which user caused the problem? Do they really think it's acceptable to "Delete all users but the Admin user"? I sure look forward to getting all of the users to re-enter their passwords.

With user experiences and support documents like this, I think I can safely predict more posts about the QuickBooks in my future.

[2008/03/17 | /software | permanent link]

Evaluation Licenses, or How to annoy a sys admin

I was considering a major upgrade to our backup system (for more robust backups of laptops), but before I committed to making such a large purchase, I sought an evaluation license that would add the extra features to our backup software. With the evaluation license installed, I poked, prodded, and happily tested.

We ended up deciding against the upgrade, and after the 30 day evaluation license expired, my backup software simply stopped working. You see when I installed the evaluation license (which gave permission to use nearly every feature of the software), it overwrote the permanent license that was installed.

Sure there were emailed reminders every two hours for the 7 days before it stopped working. As I had stopped using all the extra features by the end of the 30 days, why couldn't the software just revert to my previous permanent license? Why did I specifically have to go re-install my license?

[2008/03/05 | /software | permanent link]

August 2007

Virus scanners, or An Overly Concerned Engineer

A paraphased portion of the conversation between myself and an engineer about his new laptop;

engineer: ... and where is the virus program?

me: There isn't one at the moment. In the next few weeks we are migrating from one vendor to another. As it's kind of a pain to uninstall the current one, I was trying to save myself some time and effort. Plus, we are out of licenses for our current product.

engineer: There needs to be a virus scanner.

me: Were you planning to run attachments received via email from strangers?

engineer: I just won't use this for email until I get a virus scanner. It's really important.

sigh I guess I should be happy he was concerned about the matter.

[2007/08/14 | /software | permanent link]

OEM Licenses, or Trying to make a non sysadmin understand software licensing

"I bought alaptop. I could return my company issued one if I could get a copy of Office for it." an engineer stopping by my office tells me.

"Ok, I need an email from your manager authorizing a $370 license."

"What?! Office was only like $150 from HP."

"Yes, and that license wasn't for Office Professional, plus it would be for Office 2007 and not Office 2003."

"What's wrong with 2007?" the engineer naively asked.

"In short, it would cause plenty of headaches for our OpenOffice users."

"Couldn't we just move the copy installed on my company laptop?" the engineer continues hopefully.

"Nope. OEM licenses purchased from Dell and the like are tied to the machine they were purchased on. $370 gets us permission to both install older versions and the ability to move a license from one machine to another."

[2007/08/11 | /software | permanent link]

June 2007

tee, or A useful Unix utility

Nearly a year ago, my boss introduced me to the Unix utility tee. Seeing as at least one of my unix using friends was unaware of the utility, I figured others must not know of it either.

As the documentation puts it; "reads standard input, then writes the output of a program to standard output and simultaneously copies it into the specified file or files." Pretty straightforward.

[2007/06/19 | /software | permanent link]

multixterm, or An entertaining and useful X app

I believe every unix shop has a locally written script that will run a specified script on a collection of hosts. I've certainly written that script before. Sometimes however I can't write a script that encapsulates my needs. For these needs, I can usually use multixterm.

As the name implies, multixterm runs multiple xterms. The neat part, is that it provides a way to provide the same input to all of those xterms at the same time.

Running the command; multixterm -xc "ssh %n" foo bar baz (where foo, bar, and baz are hostnames), will open three xterm terms. Each window will have already run the command "ssh hostname" (with one hostname per xterm). You will also have a small input window where you can type, and that text appears in all of the spawned xterms. Or, if you have an exception in a particular window, you can go type in that one alone.

Doesn't that sound useful?

[2007/06/12 | /software | permanent link]

May 2007

User Access Control, or A user annoynance

Today was my first chance using Windows Vista. One of our consultants got a new laptop and needed some assistance with it. After guiding him for awhile, I took the drivers seat and went about installing and configuring some more complex matters.

While installing the VPN client, I encountered four instances of User Account Control asking for permission to continue. I asked the owner of the laptop if he had grown annoyed by the UAC messages yet. He responded "I have yet to see one yet."

In the 15-20 minutes I spent watching and helping him before taking the mouse, I saw him click through at least a half dozen of these messages. It seems Microsoft's goals of alerting the user of possibly conerning changes becomes ineffective after just two days with the OS.

[2007/05/26 | /software | permanent link]

April 2007

Windows Genuine Advantage, or What it's like to have a "non genuine" copy of Windows

Ever wonder what a Windows Genuine Advantage failure looks like? I can't say that I had. Too bad I was presented the opportunity to see first hand what it was like.

We first got the warning that our "copy of Windows is not genuine" about two months ago (long story as to why we didn't deal with it sooner). We were not concerned about the machine not recieving updates from Microsoft as the machine segregated from the rest of our network. With our Volume license Key in hand (we couldn't find the original license for the system), and a use of that key marked on our spreadsheet tracking such things, I aproached the machine.


(click any of the images for full screenshots) Where I am greeted immediately by a reminder that I "may be a victim of software counterfeiting".


In case I missed the warning on the login screen, Microsoft turns Windows nagware .


And just in case I missed the two previous messages, Microsoft thoughtfully provides another warning. Clicking this one takes you to Microsoft's site for further explanation.


That page provided me a link to "Find out if you can update your product key without purchasing a new copy of windows". I could also have given Microsoft $149 to "Get Genuine Now".


I am told to download and run the Key Update Tool. It downloads quickly and once run presents a wizard to change your product key.


Unfortunately it didn't work.


Thankfully I knew of a tool to change the key (or registration information) for Windows XP. This tool simply provides a nice gui to change the product, as Microsoft helpfully documents.


A reboot, a manual validation, and a successful trip to Windows Update for updates leaves me confident that the process was a success.

[2007/04/20 | /software | permanent link]

March 2007

x11vnc, or How to Recover from a wedged X session

An engineer came to me complaining that his X session was wedged. My immediate reaction (without looking up from my work) has become a standard response; "Ctrl, Alt, Backspace to kill the X server and once you log back in set the screensaver to just power off the monitors."

This paricularly engineer however had a couple of long running tasks in terminal windows that he didn't wish to lose, as they had been running for nearly 8 hours; and could I please help him fix the X session? He couldn't just let the machine sit until the tasks were done, and than kill the X server as he needed to see the output of the jobs that were running.

Tangent: Why do so few people understand that when running tasks that are going to take awhile, the output should be redirected to a file (or perhaps a file and stdout with the tee command) so that if the computer were to crash mid run, at least the output would be saved. Not even a bad experience or two (resulting in lost output) seems to convince people of this need.

In this particular instance, the X server wasn't horked far enough to stop x11vnc (which takes a running X session and exports it as a VNC session) from working. This technique won't work on all X server breakages (or even most in my experience), but it does work on occassion.

[2007/03/23 | /software | permanent link]

Low Disk Space and VMWare, or Corrupting a VMs ram

Yesterday evening, I recieve a report that a user can't log into a particular virtual machine. A quick look and it turns out that the virtual machine is stalled awaiting an answer to a question.

For those without access to the image above, the relevant portion is "The directory /vmware/virtual_machine/ has less than 150 MB of free space. Running out of free space in this directory may corrupt the virtual machine's RAM."

sidenote: You can see if a VM in VMWare Server has a question to be answered by eithe connecting to the console of the VM (this is what generated the image above) or with the command line "vmware-cmd vmwareconfig.vmx answer"

This became a problem only because we do not pre-allocate the disk space of virtual machines upon creation. As users of the VM added data to the VM, the hosts's disk filled up and we got this error message.

Doh

[2007/03/16 | /software | permanent link]

VI Keybindings and tcsh, or Oh the humanity

For historical reasons the default shell for the engineers I support is tcsh. The default cshrc sources a half dozen other csh scripts. One of those scripts issues the command bindkey -v which sets the shell to use vi style keybindings.

I learned this little joy when I first started working here, but as I use VI regularly I could adapt (and there was the little fact that I use bash as my day to day shell and enter tcsh as little as possible). I had quickly forgotten about it until I recieved a bug report complaining about "oddly" terminal windows.

Sitting down at the user's machine shortly after the bug report hit my inbox I realized quickly what the problem was; vi keybindings. After explaining what was going on to her, we added the line bindkey -e to the end of her cshrc and her world was better. For the rest of the week, I kept receiving thank you's from people who learned about my "fix" through the grapevine.

It seems few people like many of my coworkers dislike vi keybindings on the shell.

For more information on bindkey, man tcsh

[2007/03/16 | /software | permanent link]

February 2007

Slipstreaming, or Customizing a Windows XP install CD Part 1

In a quest to automate and standardize our Windows installs, I have begun working to make a custom Windows Install DVD that will hopefully install Windows and all our standard software. This is the first of likely many posts describing the journey. This post describes how to slipstream a service pack and hotfixes into install media.

Slipstreaming is the process of integrating a service pack and hotfixes into a Windows install cd. The end result of this will be a install cd that will have very few updates to download from Microsoft Update after a fresh install. Note that the slipstream process only works with un-customized Windows install media. Thus, you will likely have issues if you attempt this with the media provided by most major hardware vendors. This is really intended for those with a Volume Licensing Agreement or with retail media.

First we need to incorporate Service Pack 2 into the install media. Note, that much of the media shipped in the past two years already has SP2 incorporated. If however, you have media that does not have it, go follow the instructions at the Supersite for Windows; Slipstreaming Windows XP with Service Pack 2.

With install media containing SP2, you now likely want to install the 60+ updates that have been released since Service Pack 2 was released.

  1. Copy your SP2 loaded install cd to a working directory on your hard drive.
  2. Extract the boot file from the install media. See the instructions in the slipstreaming guide linked above for assistance.
  3. Download all relevant hotfixes. This is much easier said than done. I did it by finding a fully patched Windows XP machine, and running QFECheck to obtain a list of Knowledge Base articles (the six digit numbers e.g. KB909520). With that list, put them into urls of the form http://support.microsoft.com/?kbid=xxxxxx and download each hotfix referenced.
  4. You now need to integrate each hotfix into the working directory. Use a command of the form ⟨hotfix.exe⟩ /integrate:C:\⟨workingdir⟩ for each hotfix. You could simplify this process with the command FOR %%f IN (*.exe) DO "%%f" /Integrate:C:\⟨workingdir⟩D, which must be executed in the same directory as the hotfixes.
  5. Use your favorite cd burning application to create a bootable cd. Once again, see the slipstreaming guide above for specifics.

If you are one of those without virgin install media, or you want a simpler method with a gui, check out nLite which puts a GUI frontend on this (and much more).

[2007/02/15 | /software | permanent link]

December 2006

Java 1.3, or I really thought I left Java version hell when I left the computer lab

A user asked to have Java 1.3 installed on his linux workstation. Suspicious of a request for a 5 year old version of Java I stopped by the requester's cube to learn more.

It turned out that one of the companies we were working with provided access to a custom application via a Tarantella setup. Tarantella is a terminal services platform (think VNC or Citrix), and while the web/java client for this particular version of the Tarantella server would load with a modern Java runtime, the application wasn't actually usable because of screen redrawing issues. The user was told that he should use a 1.3 release of Java from Sun.

It of course would have been my preference to tell this company to get with the times (after cursing the creators of Java for the fact that I seem to have more negative experiences with Java than positive ones). But as that was not an option, I went to work out how to have multiple versions of Java available to a browser under linux.

While investigating the feasibilty of this (short answer: While doable under Windows, the solution under linux involves multiple copies of a web browser). It dawned upon me that there was likely a locally installable client that could connect to the server. A quick email to the company hosting the Tarantella server we were trying to connect to got me a copy of the program and I was able to close requestor's bug ticket.

[2006/12/18 | /software | permanent link]

lmcheck.sh, or a script to warn you of license expirations

I needed a way to know when our various flexlm licenses would expire. I suppose I could have simply added the dates to my calendar whenever I added/updated the license files. But since I am not the only one to update the license files (and since sometimes the licenses are updated before the actual expiration), I figured a script that checked for soon to expiring licenses would be the correct solution.

Some experimentation with lmstat and some clarifications from the FlexLM Manual led to the creation of lmcheck.sh. The script should work on any unix system with a modern sh.

The output looks like;

/opt/lmgrd/bin/lmcheck.sh running on hostname at Tue 12 Dec 2006 06:53:16PM EST

The following licenses have expired or are expiring within 7 days

VENDOR      FEATURE                       VERSION             DATE                
------      -------                       -------             ----                
VENDORNAME  FeatureFoo                    1.2.3               4-dec-2006          


The following configs could not be tested
----------
27001@hostname

Cron runs this daily with the command lmcheck.sh | mail -S "lmcheck on hostname" email@example.com

[2006/12/15 | /software | permanent link]

October 2006

Mentor Graphics SupportNet, or Super Secret Support Documents

I was seeking documentation on FlexLM usage by products from Mentor Graphics. Their support site was easy enough to find, but every time I clicked a link to a document that looked relevant, I was taken to a login page.

Usability annoyance tangent: The link entitled Learn how to use SupportNet, opens a new browser window with a full window flash applet, and in my case a dialog box explaining "This tutorial was designed to work on screens of 1024x768 or greater, and therefore you may have trouble seeing the entire screen. Note: the tutorial control is located on the bottom of the window". I note that I was doing this from a laptop with a 1024x768 screen.

Not actually wanted an account on the support site, but seeing no other option, I follow the link to Sign Up. The first thing I see is a warning in red "Registration requests are processed within 24 hours of receiving email verification." Sigh, I was hoping to resolve this matter today. I fill out the form and moments later recieve an email asking me to verify my email address and reminding me that they are the only EDA vendor that has 5 STAR support. Who would have guessed that requiring your customers to jump through meaningless hoops is one of the requirements of the STAR awards. I can understand requiring registration to download software, but there is no excuse to lock up the knowledge base and how-to documents.

The link that verified my email address did take me to a page saying I could peruse SupportNet as a lowly guest. I wasn't able to download the updated Mentor specific Flexlm pieces I needed, but I did much of the information I needed. Why must software companies make the lives of systems administrators more difficult?

[2006/10/26 | /software | permanent link]

August 2006

Time Synchronization, or Why is the default so complex?

Before this experience, I was under the impression that the reference implementation of NTP by the NTP Project was the bees knees. I have since come to have a very different opinion of the program. It all started with the need to setup a pair of NTP servers.

First off I needed to get the correct time on the servers. After changing the default server entries from the global pool.ntp.org entries to the country specific us.pool.ntp.org and adding entries to the step-tickers file (this enables the init script for ntpd to specifically set the time from the listed servers upon daemon startup. Why exactly isn't this the default?) I had the correct time on my servers. This step was easy enough.

Next, I had to get the server to accept requests from other machines on the network. Redhat kindly commented up ntp.conf. The relevant section is;

 # -- CLIENT NETWORK -------
 # Permit systems on this network to synchronize with this
 # time service.  Do not permit those systems to modify the
 # configuration of this service.  Also, do not use those
 # systems as peers for synchronization.
 # restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

Wait a minute, to allow a network to request time from the server, I start a line with restrict? You have got to be kidding me. Anyway, I make the change to make it applicable to the network, and from another machine, queried the server for the time. The request was not met with an answer that pleased me though.

[root@server ~]# ntpdate -q 192.168.0.1
server 192.168.0.1, stratum 0, offset 0.000000, delay 0.00000
2 Sep 22:27:26 ntpdate[2674]: no server suitable for synchronization 
found

Which leads to perhaps the biggest problem with the project: some of the worst documentation I have encountered. You can't blame them for lack of documentation. They have lots of documentation. Lots and lots of documentation. That is really the problem. I can assure the authors of all that written material that very very few people care about the gritty details of how NTP works. People just want a simple, straight-forward, and reliable way to get sub second accurate time on all of their machines.

I spent nearly four hours reading nearly all of that documentation. I not only found the answer to my problem; ntpd won't respond to requests for time until it is confident that it has the accurate time, which takes a few minutes after each restart of the daemon. In the end, I found that someone had already gone through the pain of working out the correct config for ntp.conf and Redhat ships that nicely documented config that will work for the majority of administrators.

After all of this, I am left with the feeling that there has got to be a simpler and more straightforward ntp daemon. Thankfully, there are indeed alternatives to the reference implementation of NTP. There is Chrony, or OpenNTPD from the fine OpenBSD group. I have begun using the later at home. Next time I need to solve this problem at work, perhaps I will move away from the reference implementation.

[2006/08/24 | /software | permanent link]

Setting up Conference Rooms in Outlook/Exchange, or You have got to be kidding me

We were expanding into a new building and I was tasked with setting up the new conference rooms so that they could be scheduled through Outlook/Exchange. I recall from many years ago that there was much idiocy to setting up such things, so I asked the interweb for assistance.

Early in my searches I came across a page from Microsoft TechNet entitled Set Up a Conference Room as an Outlook 2000 Resource (another set of instructions doing the same thing is here). I followed the instructions (Ignoring how silly it is to need to create a profile in Outlook for each resource you wish to manage) and surprise surprise it works. Great, on to test it with Evolution and the Exchange Connector. Nope, it doesn't work. But clearly it should work from the Outlook Web Interface provided by the Exchange Server, right? Hmmm, no that doesn't work there either.

I guess the page was serious with the prerequisite of "You must be using Microsoft Outlook 2000 and Microsoft Exchange Server 5.5." Further investigation confirms that that solution only works when using Outlook 2000 or greater as the client to setup the meeting. More info here (note Windows IT Pro magazine subscription required to access).

Still further investigation yields an interesting page entitled Scheduling Resources for Microsoft Outlook that says that there are two primary ways to allow Outlook users to schedule shared resources automatically. 1) setup each resource as an Exchange Server mailbox and do various bits of trickery to make it auto accept meeting requests or 2) create a public folder that holds appointment items and allow various groups of users permission to read and write to it.

In the end I find that the conference rooms we already have in our Exchange system were implemented using the first option with a tool called the Microsoft Exchange Server Auto Accept Agent (download here. I created new users (this does of course use up a CAL) to represent each of the new conference rooms, setup Outlook profiles for each so I could change permissions on the new calendars, and finally, using the command line VB scripts from the Auto Accept Agent, added the new conference rooms to the monitored mailboxes list.

The Auto Accept Agent basically snags incoming meeting requests to registered mailboxes and processes them based on criteria (if the room is available, if the event is in the future, etc.). Registering mailboxes to check is done through a trio of command line VB scripts. Managing the behavior is done through editing an xml file. Another more full featured option (which I would have used had MS's solution not already been configured on the server) appears to be the open source AutoAccept Sink for Exchange

The sad part, is that about half way through the day (about the time I learned that Outlook and the Outlook Web Interface behaved differently) I took a break from fighting with Exchange and Outlook and voluntarily went to read the Sun Grid Engine documentation. When a product is so frustratingly annoying that I voluntarily go to read a very very dull manual to take a break, there is clearly something wrong.

[2006/08/23 | /software | permanent link]

June 2006

Fedora Core, or An inappropriate linux distribution for a server

I simply don't understand why anyone would use Fedora Core in a workplace, let alone on servers. I can understand wanting to avoid paying yearly per system licensing fees for Redhat Enterprise Linux, but major upgrades and bleeding edge software every 6-12 months is not something that should be done in a business.

There are however alternatives to those two extremes. There are a bunch of RHEL Forks. Each project builds a distribution based on the source rpms made available by Redhat for Redhat Enterprise Linux. Each project has slightly different goals. Scientific Linux for example endeavors to be RedHat comatible while still adding in various clustering goodies used by researchers. My current choice for a straight forward, staying true to RHEL distribution is CentOS.

Using this type of project however, is not for everyone. Red Hat provides support options and the percieved stability of security updates and patches coming from a company; these might be an issue for some managers. The other big issues, are all about mitigating your level of risk when using a completely volunter community project.

What would happen if the project weren't to put out security patches as fast as you need? Do you have the knowledge and skills to rebuild the source rpms yourself? What if the project collapses? How quickly would you be able to migrate to another distribution? (One option, is to migrate in place with these instructions as a guide.) And the most devasting of possible issues; What happens if Redhat stops releasing source rpms? Would you be able to hand patch the services you maintain until you could migrate to another distribution?

If those risks scare you, perhaps you will be more willing to pay for the licenses from Redhat. These risks don't bother me because 1) we don't have to be a 100% uptime workplace and 2) I have the skills needed to maintain it all myself as I worked on a migration plan.

[2006/06/04 | /software | permanent link]

April 2006

Outlook Express, or A reminder of why I avoid the program

A user came into my office this morning saying that he was having troubles with Outlook Express. A few questions later, and I learn that the program crashed whenever he tried to open his inbox. I had seen the problem before, it was undoubtebly because of a corrupt dbx file (the format used by Outlook Express to save folders full of messages).

After making a copy of the Outlook Express folder from his Application Data directory, we tried compacting the folder followed by compacting all folders. Outlook Express would still crash upon opening his inbox. We deleted the folders.dbx file hoping that the central index was the problem. That didn't solve the problem either. Searching for assistance from Microsoft, I come across the page: The Other E-Mail Threat: File Corruption in Outlook Express

Tangent: I find databases and other binary structures for storing mail to be overkill and a bad idea. The primary argument used for why it is a good idea is to make searching and manipulating large mailboxes faster. Sure, it can be faster, but plenty of email clients do a fine job without storing your mail away in a binary blob. Mail should be stored in a nice simple mbox related format. While mbox certainly has it's own problems, at least I have never seen a mail client crash from a corrupt mail file, and when I did see an instance of a client breaking the file, I was able to recover nearly all of the messages by hand with a simple text editor. Plus, text files make it much easier to migrate your mail to another client should that become necessary.

I was horrified by the article. They were advocating purchasing software to solve what is apparently a common fault with Outlook Express. Besides DBXtract (the product recommend in the article above), there are many, other tools to recover corrupt dbx files. There is simply no excuse for this. If this problem is common enough to have spawned that many products to fix it, Microsoft needs to get it's act together, fix Outlook Express and/or ship as part of the program a method to repair corrupt db files.

With no intention to purchase software to support software that isn't on our supported software list, I provided him with the most recent backup of the files and he was able to get back up and running.

[2006/04/19 | /software | permanent link]

March 2006

Boot CDs, or How to shrink your cd wallet

The majority of cds that come into my office get ripped to an iso, stored on the file server, and put into a an ugly cd storage box, hopefully not to be touched again. Unfortunately, not all of my cds can be put away, I had nearly 20 cds that I still needed use to for installation and troubleshooting.

With the Ultimate Boot CD, I reduced the number of cds on my desk to 6 (that includes 4 OS install discs). Before customizing it is simply a collection of free bootable disk images with a menu system to select between them (note, I recommend that ALL people who work with computers have this cd). After my additions, it eliminates nearly all of the cds I used to have to keep around by putting them all on one.

My custom additions include;

I created the ISO of Disk Director with LC ISO Creator. The Altiris Deployment Solution created those ISOs. The bios flashers were made from the boot floppy creators provided by Dell and saved using a Virtual floppy Drive. After making my changes, I created a new iso with Nero and tested the changes under VMWare.

Note on Making Your Custom Disc Bootable: To make the disc bootable has different settings under different burning packages. Under Nero, you need to change a few things in the boot tab of the disc properties; The image file is in "UBCDdir\boot\loader.bin\". Under the expert settings; kind of emulation should be set to "no emulation", Load segment of sectors is "07C0", and number of loaded sectors is 4.

[2006/03/20 | /software | permanent link]

ELDump, or How to automate extraction of log data under windows

I have been looking for a way to easily (and cheaply) acquire statistics on users of my lab. I want to know things like; How many unique users use the lab get each day/week/month/semester? How often does the average student stay logged in? Do all of our users login in a given month/semester?

A bit of searching by a coworker found that events were logged to the primary domain controller's security log with event id 680 whenever someone attempts to login. He was further able to work out from an export of the log answers to some of the questions we sought answers to.

My coworker was on vacation last week, and he tasked me with exporting the logs on Monday; I forgot. So this morning (when I was reminded a week late (user error put it on the wrong date) by my Palm of the task), I sought a way to make a scheduled task of it. With the program ELDump, I was able to construct a command line to perform the export. It was then trivial to wrap it in a batch file and set it up as a scheduled task.

The batch file:
SET TODAY=%DATE%
SET YEAR=%TODAY:~-4%
SET DAY=%TODAY:~-7,-5%
SET MONTH=%TODAY:~-10,-8%

"c:\Program Files\ELDump\ELdump.exe" -e 680 -m Security -l security -c , -M -A 192 -O "dtus" > "c:\logs\event680_%YEAR%%MONTH%%DAY%.csv"

That batch file spits out a csv file that tells the who (what user), where (from what machine), and when that we care about for each login. With some appropriate crunching, my coworker can now tell us when the lab is most used, how many unique people use the lab in a span of time, what the average number of users per day we see, and answers to other similar questions. While none of the results were a real surprise to us, it is nice to know that we can now provide actual numbers to the powers above and grant submissions.

[2006/03/20 | /software | permanent link]

February 2006

StudioMX Activation Woes, or Macromedia Blames the User for Their Inadequacies

I installed Macromedia Studio MX 2004 and all of the relevant updates on a coworker's machine. After rebooting, I was asked to, and did, activate the product. Logging in as a normal user verified that all was happy.

It was a surprise to me when later that week, my coworker stopped by saying that Dreamweaver was asking to be activated again. I had her reactivate and noted that I should check up on the matter in a later that week. A few days later, she tells me that she is asked to reactivate the software each time she reboots the computer.

While perusing the Activation Support Center, I call up the support number and quickly get through to a member of the "Product Activation Team".

Call one: After learing that the computer boots into two different operating systems from the same drive, I am told that Macromedia does not support this configuration and the tech quite simply states that she can not offer any further help. She points me to Service Note 18789, entitled "Partitioning and emulation software". Since I still needed to deal with other matters today, I put it on hold until the next day.

Call two: Explaining that I am being asked to reactivate the software upon each reboot, the "activation suport specialist" learns that there are two hard drives in the system and indicates that Macromedia does not support dual hard drive configurations. He points me to the EULA and the support representative says that all he can do is "increase my activation install quota by one notch". When I point out that I am aware of the brain dead limitations of Macromedia's activation system in regards to RAID configurations, and that the two drives in the system are not in such a setup. He points me to the EULA saying the issue is clarified there.

Scanning the EULA quickly, all I find that seems to be relevant is paragraph 'i' in section 2: "You agree that Macromedia may use those measures and you agree to follow any requirements regarding such technological measures." Inquires to learn what those requirements are leads nowhere. Thankully the tech from call number two provided me with a number to reach the activation team directly (800-945-9049), instead of going through the technical support phone maze.

A week goes by with activation happening a handful of times as my coworker uses the software. Seeking an answer to the question "Is there anything wrong with repeatedly activating on the same hardware?" I make another call to activation support.

Call Three: Teh tech, upon hearing the situation, asks what version of version I have installed. Upon hearing it is 7.2, the support technician suggests installing Service Note 19468 entitled "Reactivation failure after upgrading to Flash 7.2". I quickly install the hotfix referenced in the service note, reboot, and reactivate the software. Several reboots later, and it appears as if the problem is solved. The nice activation support represetantive does also answer my question; At some point, continuously reactivating would cause an error that would need to be resolved by speaking with technical support.

[2006/02/28 | /software | permanent link]

QuarkXPress Activation Woes, or I Would Be Happy to Make a Directory World Writeable

The installation of QuarkXPress 6.1 and the upgrade to 6.5 went smoothly enough (QuarkXPress 6.5 was released in November of 2004, why isn't there a single integrated installer?), but upon logging in and running Quark as a regular user, I recieve the message "The activation file for this copy of QuarkXPress 6.0 has been corrupted. You will need to reinstall this copy of QuarkXPress 6.0."

The very thought that I would need to reinstall the software to solve an activation problem really irked me. Since Quark runs as expected as the administrative user, I assume that the issue is with permissions of some files and I go diving into the Quark technical support database for help. Not finding much of use immediately, I call the tech support number and hang out on hold while I continue searching their site.

Tangent: I can only assume it is complete carelessness that allows the hold music of most companies to be so painful. Quark seems to have recorded their hold music with a kids tape recorder stuck in the cone of a Victrola phonograph playing in the backseat of a Hummer undergoing field testing. Worse is that the loop is less than a minute.

Finally (after enjoying 22 minutes of hold music) reaching someone in technical support, I explain the problem and the tech immediately knows what is wrong. He instructs me to provide "Full Control" to "Everyone" to the folder c:\Documents and Settings\All Users\Application Data\Quark. Inquires for more details confirmed that 1) just the users who wish to run QuarkXPress need "Full Control" and 2) Quark is unconcerned that they are recommending settings that should make most systems administrators cringe.

[2006/02/20 | /software | permanent link]

January 2006

xcacls.vbs, or Microsoft's Command Line NTFS tools sure do suck

I have been working on a single script that will do user account creation. I'll cover the script itself later, right now I feel I call attention to the awful programs xcacls.exe and xcacls.vbs.

After experimenting with xcacls.exe (download) to modify the ACLs of home directories, I thought I had down everything I needed to do. Opening the GUI to verify that my carefully crafted command lines did what I expected them to do, I was presented with an interesting message-- The permissions on FolderName are incorrectly ordered, which may cause some entries to be ineffective. Press OK to continue and sort the permissions correctly, or Cancel to reset the permissions. It seems there is a problem with xcacls.exe. Apparently using the program in a way that is concistent with the instructions is not supported,

So I download xcacls.vbs and start experimenting with it. It took about thirty minutes of experimentation to work out the new features and the differences with the previous version, but it seems the script does solve my problem. That is not to say it is a well written script. An annoyance: you can't designate the username in the active directory form of username@domainname; just NT style ntdomain\username. A problem: the program takes 5-10 seconds to process the ACLs on about a dozen files/folders.

I can only wonder if had the script had been implemented and compiled to a native binary it would have been as fast as the original program. This is also another reminder that Microsoft expects me to learn and code all of my utilites in VBScript. I think not.

[2006/01/18 | /software | permanent link]

December 2005

Corel's Exclusive Offer, or Marketing Taking Over an Update System

I like the concept of self updating programs. In practice however they are usually implemented poorly (ala Acrobat Reader) and/or co-opted by marketing for nefarious purposes.

While checking for updates for our installation of Corel WordPerfect Office Suite 12 I found that some manager in the marketing division of WordPerfect thought they were being clever when they decided to use the update system to send out what amounts to an advertisement.

For an update system to be effective and trusted by users it can not be co-opted by marketing in an attempt to make money. Whatever manager that approved this "update" message should be fired.

[2005/12/09 | /software | permanent link]

November 2005

Hello 1995, or Software That Sucks

Two of the machines I support are used for accessing datasets from a sizable collection of cdroms. These cds generally cost a few hundred dollars each and come with the data wrapped up in a propreitary brinary format which requires a poorly written custom application to extract. Complaints about the usability of these applications (which is almost always poor) will be saved for another day. My complaint today is the number of companies that clearly don't wish to waste money on programmers.

We have a collection of CensusCD products from GeoLytics. The installer defaults to wanting to install to a c:\CDIDENTIFIER, changing it to c:\program files\censuscd\CDIDENTIFIER, the installer completes succesfully. Thinking the program is installed, I run the program and all appears happy. Setting up a query to extract some data works goes smoothly. Actually extracting the data however creates a cryptic error message.

A little experimentation and reading the manual (heaven forbid) show that the program needs to be installed to a path that has no spaces and no part of the path is more then 8 characters. Why then did the installer allow me to install to that directory? I know testing for this is possible in the installer, in fact while working out possible solutions I found that the ArcView installer did exactly this. This type of silliness was acceptable from small software companies when we were still transitioning from DOS to Windows (I would say up until about 1998), but it is completely unacceptable from a product released in 2002, even from a company that does not have a multi-person software development team.

[2005/11/23 | /software | permanent link]

October 2005

Updating Acrobat Reader, or Another Reason to Hate Acrobat Reader

While updating a machine from Acrobat Reader 7.0 to 7.0.5 I encountered a another annoyance I have with the program.

There is no reason an application should require a system reboot for a software update.


Update 2005-11-04

So a friend who programs under Windows on occasion pointed out that the recommended way for programs to handle replacing in use files under Windows is set it up to be done at next reboot. So there is apparently a reason for an application to require a reboot to install. I point out however that it would have been preferred to inform the user what programs are currently using inuse files and asking the user to close them so the install can continue without a reboot. I note that plugins for web browsers do this. Updates for CorelDraw work this way.

So I change my objection only slightly: There are few reasons an application should require a system reboot to perform a software update. If it needs to update in use files, it should make every effort to do so without requiring a reboot.

[2005/10/27 | /software | permanent link]

Daily Virus Definitions, or Deep Dark Batch File Magic

In a world where a new virus/worm can sweep around the world in under 48 hours, prompt virus definition updates are a requirement. For some inexplicable reason however, there is no built in way for a Symantec Corporate Edition 9 server to download updates daily.

Symantec Corporate Edition has server side bits that allow a company to manage the client anti-virus software. You can setup scheduled scans, configure how on access scanning works, push out new virus definitions, and configure pretty much anything you would want to configure on a client machine all from one place.

One would think that the server side programs would have the ability to schedule checking for new virus definitions from Symantec. You can easily do this. Well, you can if you want updates weekly. Weekly isn't acceptable in this day and age of sweeping virus outbreaks. Thoughtfully, Symantec has a solution, the XDBdown.cmd script downloads Intelligent Updates (which are updated daily). Ignoring the fact that this script uses some batch file voodoo that could be used to scare first year computer science students; why is this needed? Why isn't this included in the basic functions of the server side software?

I have been told version 10 allows the admin to choose any definition download schedule they wish. I have also been told not to rush to install it as the upgrade process is not quiet painless. When there is a major virus outbreak in the wild, I can run LiveUpdate manually and it will grab new definitions more often then weekly. can manually run LiveUpdate whenever I wish and it will grab a daily update. For something as simple and repetitive as this it should be automated. Until this upgrade happens, I shall have to survive with a script scheduled by Scheduled Tasks on my server.

[2005/10/03 | /software | permanent link]

July 2005

VMWare Workstation, or how I can test linux firewall distributions with just my laptop

This week I tested a half dozen linux and bsd based firewall distributions (ClarkConnect, M0n0wall, Smoothwall Express, SME Server, IPCop, and RedWall if you care). I tested each with three windows clients, a windows servers, and a linux server behind them. I did this all from the comfort of my laptop. I did this with VMWare Workstation

VMWare makes virtualization products (more info here, here, and here). I first learned of VMWare Workstation (version 2 if I recall) in college when they succesfully lured me into their shinyness with a $99 academic license. I toyed with it through college (mostly running CorelDraw while my machine was booted into Linux), yet forgot about it for a few years.

About a year into the job, I purchased and began using VMWare GSX Server for server consolidation I used it quite conservatively (more on that process some other day). I didn't quite learn how much glee VMWare could bring me until I recieved a copy of Workstation for attending one of their sales seminars. But this isn't a piece on all the things I have done with VMWare Workstation, this is a description of how I used it to test Firewalls.

With this general setup I could perform basic testing on a firewall setup in about 30 minutes. Having performed this type of testing with real machines in the past, I would estimate a savings of about 4 hours for the initial setup and about an hour per firewall. I wouldn't have to go through that initial setup if I actually had the funds, space, and assistance to have a proper test lab with a variety of spare. The testing of each firewall was sped up in ways that couldn't be done with physical machines. A key item for this testing was the ability to take snapshots (a save of the state of a virtual machine) of both the firewall and the test machines. With these snapshots, I could bring back the exact same setup over and over again in just a minute or two.

The story of what firewall I choose and why is for another week.

[2005/07/12 | /software | permanent link]

June 2005

Making bulk changes in AD, or my windows has some icky command lines

I have an Active Directory with about 900 users. The vast majority (all but about 15) have a single mandatory roaming profile. Because of some inconsistencies in the creation of user accounts over the years, how the profile location is specified in an account varies. Some accounts have "\\servername\profiles\normal\" some have "%logonserver%\profiles\normal". I needed to standardize these to "\\newservername\profiles\normal\".

The easy way would be with the graphical tools. Select multiple users in Active Directory Users and Computers, right click, and select Properties.

With this form it is relatively trivial to change a huge number of accounts. While I changed nearly the profile path listing to "\\newservername\profiles\normal\", I changed some (those accounts that have their own profiles) to "\\newservername\profiles\%username%". There are a variety of other environment variables available.

The hard way would use the Directory Service command-line tools from Microsoft that were included with Windows 2003 Server. They are quite powerful tools that allow you to query, modify, add, or whatnot.

The command I ended up with, after a great deal of experimentation (most of it was simply getting comfortable with the tools and toying with examples provided in Microsoft's documentation), was

dsget group "CN=groupname,DC=ads,DC=example,DC=com" -members -expand | dsmod user -profile "\\servername\profiles\normal"

the dsget command returns one per line a list of users that belong to the group "groupname". dsmod takes that output and changes the profile setting.

Other interesting examples of the DS tools.

This will get you a list of all members (recursively expanded if you have nested groups) of group groupname.

dsget group "CN=groupname,DC=ads,DC=example,DC=com" -members -expand

To create a new user

dsadd user "cn=username,DC=ads,DC=example,DC=com"

Much of my experimentation with the DS tools was done with thoughts of finally scripting account creation floating through my head. Let me just slide this into a slot near the top of to do list.

[2005/06/16 | /software | permanent link]

May 2005

Acrobat 7, or why I hate Adobe Reader

Contrary to any bitching you may hear from me, I actually like PDF's. Adobe Reader is a different matter though. I dispise Adobe Reader. I despise it with a passion I normally hold for people trying to sell jewelry by informing me my wife will love me more if I spend more money on jewelery (and how they will be happy to explain their financing options). Today I bring up the first of what I expect to be many pieces on user abusive decisions made by Adobe in Adobe Reader 7.

This version added Javascript as a scripting language. The scripting and programming available in previous version of Reader weren't easy enough for value added enterprise solution providers apparently (more on them here). I have a great dislike and distrust for javascript. This is of course because of the evils that have been perpetrated upon webusers by any number of websites. As such I have Javascript disabled by default in Firefox and Internet Explorer. Following that trend, I simply disabled Javascript within the Preferences.

I did this immediately after turning off automatic updates, which is something I do immediately after installation of Adobe Reader. Upon closing the program I was greeted with

Figuring this was a one time warning I ignored it. But no, I get this message each and every time I close Reader. If I open a document that I know to not have javascript I get this message upon exiting the program. Even if I open the program, don't open a pdf, and close the program, I am still informed that "This Document contains Javascripts".

The correct thing to do would have of course been to warn the user on load that a PDF had Javascript in it (and not present any warning if the document didn't have Javascript in it) and advise that it might be needed to view the document "properly" (where properly means according to the ways set forth and desired by the document creator). Or maybe they can learn from the mistakes made by Microsoft with macros in Office and warn users to not enable Macros (Javascript in this case) unless the document came from a trusted source. Whether or not that would be worthwhile is a discussion for another time.

[2005/05/24 | /software | permanent link]

Licenses for free software, or how to encourage people to not give you money

Part of my job entails making sure that we are in compliance with the licenses for the software we use. While not a difficult task, it is tedious and quite dull at times. Reading that much leagelese when one is not employeed as a lawyer or paralegal can not possibly be good for one's sanity. .

In most instances the issue for us has to do with us what is required to allow us to have concurrent licenses. Usually it is simply a matter of running a license server. Some involve stipulations that require the software run off of a file server. Those are hurdles that can be overcome without too much difficulty.

We use a good number of free (as in beer) programs, including a handful from Sysinternals. Most free software has limitations such as; a license is required for commercial use or the software is free for instructional use but not research use. Sysinternals however has some very interesting restrictions for their software.

The license from Sysinternals starts off saying anyone can use Sysinterals tools for home or work so long as the user downloads it from the Sysinternals website. This sounds like you aren't supposed to provide a mirror of their installers. It goes on to state quite explicitly " A commercial license is required to redistribute any of these utilities directly (whether by computer media, a file server, an email attachment, etc.)". I guess that means I shouldn't have it on our administrative file server.

A quick email to the supplied licensing address came back with a response that I almost couldn't believe. Their response was quite clear; (paraphrased) "We aren't interested in the hassle for licenses of less than $1000."

A thousand dollars is quite alot of money for my lab. I sent a reply to the email clarifying the conditions we wished to use the software; We wanted to keep copies of the programs on a file server for use by three staff members to be used in troubleshooting and debugging. The answer to that email was again quite clear in stating that we would require a license for this suggested use and that the minimum license is $1000.

Instead of having the programs on our file server, I have a series of scripts that when run will download the software, install it, run it, and once exited delete the software. I wouldn't have blinked had they asked for a one time license fee of $150. I would have blinked but not really hesitated had they asked for a one time license fee of $250. They probably could have gotten away with asking $400 even. Their tools really are that useful to us. This way however instead of getting some money from us they got none.

[2005/05/24 | /software | permanent link]

BgInfo or How not to forget what computer you are on.

How many times have you rebooted a server thinking you were on one machine when in reality you were on another? Or have you ever been looking for something that you just know is on the server, but you can't find, only to find you are not on the server you think you are on. This is actually a problem when you are connected to servers remotely, or are working through a kvm.

Under unix, I rarely had this problem as I set my prompt to include the hostname (Checkout the Bash Prompt HOWTO for more than you probably want to know about prompts and bash). This unfortunately doesn't help me under Windows, which most of my servers are.

My first thought was to simply use a different background on each system. With just different pictures or patterns it wouldn't scale well. I clearly needed to create a custom bitmap with the name of the computer and set it as the background. I never did get around to doing anything about that.

This morning, while at a VMWare Users Group Meeting, I saw what looked like a perfect solution to my problem. A program that displayed basic system information on the background. It turns out the program is cooler than I had thought.

BgInfo provides a simple way to autogenerate a custom background image. By default, it runs once and creates a custom image that is than set for the background. What is displayed in the image is highly customizable. There are a bunch of default options like cpu type, hostname, ip address, but it can also display information from a script, a text file, or the registry.

[2005/05/24 | /software | permanent link]

Wiki's are cool, or Documentation is hard

I was lucky to be shown in college how important documentation is to a project. The course was on operating system design, the project involved writing a simplistic operating system. The real challenge came from the fact that we would be working with code that had been created and worked upon for over 3 years by students in past iterations of the class.

The first year I was in the job, I learned a good deal about how well I document things. Which is to say, I learned that when making small changes and doing systems maintance it is really hard to take good notes and make proper documentation. In a class where the documentation was part of the grade it wasn't very difficult to make the time to do it. In the real world I found that unless documentation is really easy to add/update it won't be done. Conversely if the documentation that is not readily available may not be consulted until one's head has met the desk at least once. I needed to do something that would encourage me to use and write and update documentation more frequently.

What we had: When I started, all of the records were in a file cabinet. It was pretty cool actually. There were several hundred files in three drawers. There was a folder for each server, hardware purchase, software package, and such. I could often be experiencing a problem with a software package or a server or a piece of hardware, pull the folder for it and find hints about my immediate issue.

Goal: I needed a consistent and easy to follow procedure that would enable me to write how-tos, working notes, change logs, and whatever other documentation was needed. It would need to be viewable by others (limited by user or group permissions) and if possible edited by others as well. It needed to be searchable. It would need to be able to handle images. Perference would be given to a solution that didn't use a database and wasn't difficult to setup.

Implementation: It didn't take long for me to decide that this was an obvious candidate for a wiki was the way to go for this. Perusing a few lists of Wiki Engines, I settled on PodWiki. It had user and group permissions with which I could restrict view and editing access to sections, didn't use a database, had revision control built in, could handle images easily, and it had a simple install. It was installed an up and running within an hour.

Actual results: Having used my little wiki (which others at my job don't actually know about yet) for nearly three months no, I can say I am quite happy with the results.

[2005/05/24 | /software | permanent link]