Software Engineering (I focused on 'Release Engineering')

This document is not a resume. The intended audience is a person who may be interested in hiring me and is actively attempting to determine if I am a good candidate for the job at hand. I expect that it would be useful reading before a phone interview or as a basis for other such correspondence.

Table of Contents:

My work at Inktomi Corp (later Yahoo!).

I was hired in early 2000 by 'Inktomi' to work as a Software Engineer. Frankly, if the truth be known, one of the reasons I was hired was that the job market was extremely tight at the time. My training had consisted of a year of fairly intensive self-study on UNIX systems (primarily FreeBSD), and a handful of entry level courses. I had previously managed technical support for an internet service provider in Portland, OR. Inktomi's Release Engineering team needed someone to focus on assisting one of their divisions with multi-platform build requirements, and perform system administration on various Unix systems including Solaris, Irix, OSF/1 (Tru64), FreeBSD. My first task was to integrate Linux and HP-UX.

'Inktomi' was, at the time of my hire, the predominant search engine company (pre-google), and they had a very hot newish offering called 'Traffic Server' which was a caching proxy server for internet traffic. At that time, there was no where nearly as much fiber in the ground and there was a great need for such a solution. Additionally, every company in the sector was swamped with venture capital so we had many customers who purchased the software and didn't even get around to installing it. Many did, however, and so we had plenty of legacy support considerations. In short, the growth rate of the company was phenomenal with personnel being added to the existing divisions, and other companies being purchased and integrated.

After a little over a year, it was clear that major changes were in front of us. As the first round of lay-offs hit, I was retained, given a pay increase on the order of 35%, and promoted from a rating of 'software-I' to 'software-III'. On my performance review, I received a 'far exceeds' which was the highest rating. I later learned that this was quite rare though I didn't realize it at the time. Since 'Inktomi' is not completely defunct in all senses of the word, I will share this performance review with the reader only at special request. I am quite proud of it and believe that I earned it. I took pictures of the documents with my camera and converted them to two-color .png files so the size is reasonable while they remained vaguely legible.

  (Performance Review provided at special request)

There are a number of my projects and achievments mentioned in the review. I won't repeat them, but I must add that they were all very minor compared to what I later achieved. For one thing, each new development automated a portion of my work-load and freed me to focus on more important things. It is critical to understand that most of these efforts tied into one another, often to a very high degree. Here is a partial list of those which I developed _after_ the performance review which coincided with the first round of layoffs and my becoming the principle architect in my group.

Inktomi was in a downward spiral with remote development ceasing and layoffs happening regularly. At some point a decision was made to entirely offshore continuing support, maintenance and development of the products my division was responsible for. At this point a huge block of layoffs occurred, I was retained as a "key employee" with a raise of half-again my salary (contingent on performance.) The primary task was to train the Indian Engineers and to build a set of machines which would allow them succeed. This was both fun and a remarkable success.

After that, I was several more times retained as a 'key employee' until the whole division was down to 8 people (from probably around 800 at a high point!) One of the task I undertook was to build a very much reduced system which would allow one large customer to continue development of a product they had purchased from us. Aside from that there was not a whole lot to do and I mainly focused on collecting hardware in case Yahoo! (who had purchased Inktomi for their web-search technology...a different division) decided to continue development of any of my division's products.

After I left, Yahoo! did indeed decide to continue development of one server product and hired a small team to do the development work. I came back on contract and built a tight development framework which included the following features:

I did another contract for Yahoo!/Inktomi which consisted entirely of a trip to India in order to check up on that effort, and one more contract to re-IP the machines I set up earlier and also add some additional features. As it happened, the system ran without intervention for around 6 months when a hardware failure occurred. I had left detailed instructions for how to perform basic tasks like adding new users. By now, we are in the fall of 2003

My Non-Inktomi/Yahoo! Work.

Near the end of 2003, I got a contract to consult on a configure/build system, and that lead to an extension of the contract to work on implement some of the ideas we discussed. I do not feel at liberty to name the company. They were small and young, but old enough to have been strained by issues with their existing system. My total time on this contract was 6 weeks.

The system that emerged had some very interesting features that I am pleased with. Their product consisted of a set of quasi-independent projects, some customized open-source code, and some non-customized open-source code. What I ended up with was a means of arbitrarily and easily defining any set of code bases where each was from an arbitrary branch. Mutual dependencies were automatically detected and merged, and instances where multiple versions of the same code base were defined were readily recognizable and resolved. Furthermore, the build procedure from the developer's point of view was very straightforward, uniform, and comfortable, and he/she could enter the hierarchy at any level (if, say, they had checked out the definition for large project.) I used GNU Make as a driver to handle the dependencies and it was fast and accurate.

Do to my previous experience, I specifically designed in the flexibility to drive a multitude of different build types, and possibly pre-built components in instances where that might make sense. Also, I was careful to ensure that multi-platform simultaneous builds would be possible.

My main regret about this project was that although I accomplished more than I initially thought possible and demonstrated real-world functionality, there was simply not enough time to develop a number of features that I envisioned as being possible and designed in support for. I also did not get every module ported. For multi-platform efficiency reasons (and development convenience), I'm a stickler for separate-source builds and much open-source software comes with defective Makefiles which require work.

My Skills.


My focus has been entirely on Unix, and that is where my interests lay. I never had a Windows workstation or a particularly active part in Windows system design so my skills are adjusted accordingly toward Unix and away from Windows environments.

I am proficient with Bourne Shell (driving sed, awk, etc.) Because of my work with autoconf (which uses m4 macros to generate portable Borne shell script,) and a general need to have cross-platform compatibility, I ended up using Bourne shell a lot.

For more complex tasks, when the situation allows, I tend to use Python. I appreciate it's object oriented support and focus on maintainability and have found both of these to pay great dividends.

Occasionally I found it necessary to use C to write certain utilities, but do not consider myself a C programmer. Similarly, I occasionally needed to do some C++ porting, but usually consulted someone more competent when possible. I spent a great deal of time _building_ programs in said languages and working through compiler issues, however, and am proficient at solving associated problems.

My experience with Perl and tclis limited to minor bug fixes of, and additions to existing tools that I use. I used Python for original work requiring a language in that category.

I am proficient with CVS administration and have devised various solutions making use of an understanding of it's internal implementation. Unfortunately, I have no experience with any other revision control system (RCS excluded.) At my next opportunity, I do intend to explore 'Subversions' which appears to be a promising open-source CVS replacement.

Other tools I made significant use of include autoconf, autoconf, ssh, rsync, procmail, tinderbox-II (a mozilla-developed display system), ViewCVS, curl, make tar, and numerous similar basic Unix tools.

System Administration:

I have always set up my own Unix workstation and it is not practical (or generally possible) to burden 'helpdesk' with the task, so I can solve basic sys-admin issues like Unix X11 setup, print-serves setup, etc. Additionally, I use FreeBSD which is not as polished as most of the Linux distributions and generally requires more hand work and understanding.

I am especially familiar with automounters (both 'autofs' and 'amd') and filesystem design in multi-platform environments. Also with NIS and NFS to the extent that they are somewhat inter-related, at least in an environment which is vaguely sophisticated.

I've worked out the details of using 'mail' for certain things (like launching certain processes) and using something else in it's place when it is not appropriate (like feeding huge amounts of data to Tinderbox-II using 'curl' instead of e-mail, and using other methods of feeding CVS monitoring databases.) In general, I pride myself in finding novel solutions to problems, and often before they become problems.


I consider myself to be a systems architect first and foremost. To my non-computer literate friends, I describe my job as being analogous to a person who develops and sets up the machines in a manufacturing facility. I enjoy such work and take pride developing systems which are efficient, reliable, maintainable, and coherent.

I am very interested in automating processes to the highest level practical. Ideally, once a system is in place I would like it to run without intervention. For instance, the build system I developed took care of various housecleaning tasks automatically that used to consume a significant amount of time. The less a system needs to be touched, the easier it is to forget the details of how it works, thus, I believe in complete and legible feedback (for debugging) and good documentation.

I'm not adverse to taking on tangential responsibilities when that is what it takes to get the job done. Arguably, our core system administration group at Inktomi should have been responsible for some of the sys-admin details that I performed, but in reality I was the (only) one in a position to appreciate the importance of certain facets of the job, and it just worked better if I handled it. On the other hand, I did try to push back when the shoe was on the other foot. As an example, if a developer wanted me to handle a third-party element and it would require me to learn significant aspects of how it interacted with our code, I would argue against it. Though I would have liked to learn these types of things, it would have been inefficient and interfered with other important projects which I was chronically anxious to attack.

My thoughts and observations about Release Engineering

I have spent considerable time contemplating the role of Release Engineering, and the things I've observed about the Software Engineering environments I've been associated with. At Inktomi, I sat on a number of 'core teams' where I absorbed a great deal of information about both the product and the process (at the expense of loosing time dedicated toward my 'real work'.) In retrospect, I believe that it was of critical value. Here are what I consider the most interesting thoughts on these matters:

Firstly, I see Release Engineering as being _ideally_ situated to mediate between the often conflicting demands of 'Development', 'QA', 'Project Management' and 'Support'. A good Release Engineer should be always mindful of the stresses on each of these parties, and always on the lookout for ways to reduce the problems that each may face.

Secondly, and more of a 'system analysis' issue, I think that it is imperative for efficiency and quality to pay close attention to how tasks and responsibilities are assigned. I've touched on this above. That is, as a Release Engineer, tended toward being more involved in system administration, and less involved in product development details. In this case, understanding the system administration details that were important to me would require an sys-admin to know a great deal about my requirements. (i.e., it may _not_ be desirable to patch a certain machine's OS for reasons related to reproducibility.) It would be inefficient, if even possible, for a sys-admin to understand the specifications and the reasons behind them. It was much more practical for me to learn the the basic sys-admin and system design skills that I needed to perform these tasks. On the other hand, while building third-party libraries is something that a Release Engineer should be quite familiar with, integration into the product often requires a good bit of understanding known most completely to the developer who determined that the inclusion was a solution to the problem he/she was facing. In this case, I would tend to work with the developer on integration issues, but leave it to the developer to work through build details which related to product/library interoperability. This is one simple example of an issue that crops up regularly, and addressing logically pays dividends in many areas throughout time. Release Engineering is well positioned to recognize a number of these.

- more to follow -

Contact: Tom Huppi  [ ]