The $60,000 "First Customer Ship" Clock
A co-worker at Apple recently asked about a detail of the Software Lifecycle. "What is 'Mastering?'"
I got to the Slack message a bit late; they had already followed up with, "Nevermind, someone else told me, lol."
I advised them that according to Apple's REJI standards that "Mastering" was a deprecated term, and they should use "Image Assembly, and further, the "Golden Master"or "GM"--formerly a revered label for something special in the software industry--should be called the "First Customer Ship" release.
It will take though, many years I believe, to purge those terms, "Mastering" and "Golden Master" not only from speech, but from lots of software production tools and documentation.
In software the final version of a release, being the “Release Candidate” (RC) which will become the First Customer Ship (“FCS” as noun or verb) formerly was known: https://en.wikipedia.org/wiki/Gold_master_(disambiguation) where this occured when media was used: https://en.wikipedia.org/wiki/Software_release_life_cycle#RTM
Golden Master because it became or was to be the source of all copies of the software. "Master" from the audio engineering world. Different from a printing press plate which was reverse and not of paper even though the source of perhaps millions of duplicates. It was the origin copy. And by software process and general rule, it was sacrosanct. "Golden": pure, not to be adulterer, modified, treated with all due care. And also, in some points of view, the real golden goose of each software release. What creates more gold for the company.
Which gets me to an interesting experience I had 30 years ago with this very process.
I have (rather, had, but recently tossed from a shelf of keepsakes in our efforts to downsize) a clock which was made from a “Golden Master” of an RC. They called it “The $60,000 Clock” and it sat on my desk at Frame and Adobe for the next 16 years.
It came to be that one day in 1994, after their sign-off of the requisite quality of the Release Candidate of our software, Quality Assurance had burned the GM CD-ROM from the Release Engineered Image Assembly for our new product FrameMaker v4. This was the young (just five years old) company's biggest release yet, and the one which was going to take the company to IPO and crush our main rival in long form document publishing software, Interleaf. Here's some more about FrameMaker 4:
From the GM the QA Lead had made a handful of dupes for final media testing. The GM had already left the building, having been couriered to the duplication factory in Gilroy. The factory was about to start the run and go to packaging in the morning,
One of the dupes was sent to me for final test off the media. A formality, really. The software had been streamed off the disk and byte-compared to what was on the network servers and the images were deemed to be The Same. What was tested and signed-off upon, was duplicate.
I wasn’t in QA but I knew more about HP workstations than most at the company, moreover I had a CD drive on my workstation, and this media needed verification on every platform that FrameMaker 4 would ship on. By clever engineering, the main Unix platforms were all on the one CD. SunOS and Solaris, the System V variant), HP, IBM (AIX), Apollo, Mentor Graphics. The Mac, NeXT and Windows (3.11!) updates shipped on their own media, as did the lesser platforms which needed tape drives (DEC) or 5.25" floppy disks (20! SCO).
I mounted the CD onto my HP-9000. I immedately found the installer wouldn’t launch off the CD-ROM on an HP because HPUX used a very strict ISO-9660 format, which required POSIX compatibility, with a version number for each file on R/O media. The top level installer could be initiated, but the sub-installer for the HPUX was read by the HPUX system, off the CD as “INSTALL.SH;1”. The hazards of multi-platform installs.
This was only an issue with CD ROM and with the HP, there was never an issue installing off of NFS, and the Sun mounted the disk differently--to a different standard, as did all the other unix platforms on the disk, to my knowledge.
Within minutes I had the VP of Software Development (Eddie Kessler ) in my cube as the alarm went up the chain.
VP: “Should we fix it?” and risk an expensive slip?
Me: “The only other thing which can be done to mitigate this is for an experienced SysAdmin to copy the CD off onto their network or an HD and run it from there. We could put an “ATTENTION HP USERS” flyer in the package--but that's not a great first experience for a user who has spent thousands for licensing.”
VP: “How long will it take to fix it? Who from the installers is still around?”
Me: “I can fix it. I'll put in a condition for HP to look for the installer with this specific name format. Two line change, 15 minutes."
QA Lead: I can test that in 5 minutes after 45 minutes to burn the CD. [yeah, this was 1994 - a 1X speed CD burner]
VP: [A very smart guy.] "Will this affect the other platforms? Does anything else need to be touched?"
Me: "Absolutely not. HP only and we don't support other HP systems which don't mount the CD this way. I can write the tech note that mounting on a different host that doesn't mount it this way and run it across the network is not supported. Acommodating that is a more complicated change." The QA Lead and I exchanged looks. More complicated equaled more risk, and from his point of view, more testing and more time.
VP: “Do it.”
And he left. Not to return until I got the new CD and he swung by as I had FrameMaker 4 running on my HP workstation, fresh off the install.
I happily recall that clear decision being made, authoritatively, without anyone else's say-so necessary and the right people were there to get it done. That happened at other times in my career, and it is always notable.
We did the things we said we would do and the test succeeded and so my boss and her BF (the QA Lead) drove the new GM CD down to the factory and met the contract requirement. A not insignificant disaster was averted.
A week later my boss shows up in my cube with a box. Folk gathered around casually as I was told to open it. Inside was the original GM CD--the one with the error. She recovered from the duplication factory, as it was about to be copied onto several thousand CDs--but they replaced at the last minute with my fix. It wasn't just the Golden Master (which was, in fact, gold mylar coated). My boss had the CD made into a clock. No numbers, just hands sweeping around the gold backing of the CD. I kept it on my desk as a reminder of the sarcastic aphorism “There’s always enough time to do it over, never enough time to do it right the first time.” Barely. Barely enough time to do it over. And save the company $60,000.
Perhaps related, perhaps not, it was the case that 3 years later I was writing and updating all the installers for all the unix versions of Adobe FrameMaker as a Release Engineer at Adobe (who had bought Frame Technology). Ushering the compiled code through Image Assembly and through First Customer Ship. Which I continued to do for many more years for many more unix projects in the Publishing division at Adobe and in the Adobe's Advanced Projects Labs, working on future Acrobat stuff.
And then launching Acrobat.com. Which doesn't have any installers, nor customer images, and certainly nothing Golden. Nothing immutable. Instead, web and service applications are thoroughly mutable and always--if not instantly--repairable.
Comments
Post a Comment