[ Back to Kevin's Homepage | Resumé in PDF ]

Over 25 years experience as a Devops Infrastructure Coder, Release Engineer, Linux Systems Engineer, and Software Engineer

Work Experience

On Sabbatical, Santa Clara, CA - Jan 2023-current

I volunteer as an event leader with http://bikex.org/ , foster kittens for the Humane Society of Silicon Valley, train to bike 100 miles, cook, bake, and perform hardware and software engineering feats.

Software Build & Release Engineer (DevOps Engineer) at Google, Mountain View, CA - June 2004-Jan 2023 (layoff)

As a Software Engineer, I was one of 4-9 developers on an end-to-end continuous integration and continuous deployment (CI/CD) pipeline from source to production deployment, implementing best practices (tests, code reviews, branches, monitoring, alerting) for over 28,000 projects.

As a DevOps Engineer, I worked on teams from 2 to 5 people, supporting 100 to 50,000 developers. Major releases included:

• Google Web Server aka www.google.com
• GMail Frontend
• Google Search Appliance
• Google+ (Social network)

Specific projects included:

• Expand Python testing fake; pyfakefs (https://code.google.com/archive/p/pyfakefs) - Add unit and regression tests to insure the fake matches the core Python functions (i.e. fake_filesystem_vs_real_test.py).
• Converted from manual build instructions to regular (cron-driven) automated build, test, and deployment
• Automated translation imports in over 100 languages. This eliminated manual toil of 3-4 hours per week and shortening GMail's weekly launch checklist
• Migrated off dedicated build machines to scalable cloud (i.e. Borg)
• Created standardized build policies such as common build flags to eliminate mindless config copying
• Accelerated time-to-deploy down for 80% of projects down to a couple of hours
• Design and implement policies for Android support. For example, unify build rules and flags, automatic incrementing of version codes, cross-compilation to multiple architectures (including x86 unit tests), integrate APK signing, deploy to Google Play Store, onboard first projects
• Moved users away from releases as scripts into submitted configurations (i.e. IaC/IaaC). This enabled transparency and standardization, reducing developer toil.
• Interviewed and onboarded new team members: wrote documentation, taught classes, performed individual mentoring
• Advocate with users to adopt CI/CD best practices

Software Build, Release, and Systems Integration Specialist at There, Inc., Menlo Park, CA - Mar 2003-May 2004 (50% layoff)

As a senior member of the build, release, and systems integration team of 4 people:

• Set policies for revision control (Perforce) for all (50+) users - engineering, 2d web developers, and 3d artists
• Kept 10 development branches in-sync with main development branch by designing and implementing hourly automatic inter-branch merge tool
• Tutored junior teammates in job-related technology, process, policies
• Reduced production cluster migration issues 500% by taking ownership of all major software releases
• Increased developer productivity and decreased QA downtime by increasing build frequency upto 6 times daily with automation; work smarter, not harder

Independent IT Consulting - Oct 2002-Mar 2003

Specified and completed several IT consulting projects including:

• Built qmail mail server with integrated virus scanning, authenticated and encrypted send, IMAPS, mailing lists, web based management, and fetchmail retrieval of E-mails
• Installed CVS & build server with commit logging, web based CVS browser, CruiseControl for continuous Java builds, and local customization of CruiseControl
• Replaced NT 3 file server with a RedHat Linux Samba server. Integrated with existing NT 4 domain to provide seamless authentication.

Build Engineer and Systems Administrator at Entise, Cupertino, CA - July 2002 through Oct 2002 (company closed)

As member of rapidly growing 25 person startup, supported developers by owning and managing build and release in addition to maintaining the internal and external network services.

• Integrated and automated separate builds into one coherent process: Linux kernel, Sun JVM, Trifork J2EE server, and Java management application
• Maintained UNIX, Windows, and networks including WLAN, PPtP VPN, Web, DNS, and firewalls

Engineering Troubleshooter at Nexprise, Inc., Santa Clara, CA (merged with Ventro in July 2001) - Sep 2000-July 2002 (company moved 400 miles away)

Major projects included:

• Increased performance and scalability of application 10-fold by tuning the operating system (Solaris), Apache, JServ, Oracle, SQL queries, and the Java application itself using RadView WebLOAD, Segue SilkTest, and the Solaris SE Performance Toolkit
• Developed a modular /bin/sh installer, independent of any third party tools, such as Perl
• Sped up build process by a factor of 10 by rearchitecting the automated build architecture. Tools used include CVS, CruiseControl, and Ant

Other projects include: load balancing, application scaling, performance benchmarking, customer solutions, project management, build and release, systems application architecture.

Silicon Graphics, Inc, Mountain View, CA - July 1996-Sep 2000

• Software Developer - C, C++, multi-platform porting from IRIX to Solaris and Linux, as part of a small 5-person team. Network throughput was particularly important for this streaming video project
• Project Manager - Organized 50+ engineers developing the SGI Freeware product consisting of 2 CDs and 250 installable packages. Created processes and automation software to make the release drive itself rather than requiring a person to organize the information by hand. (see: http://freeware.sgi.com/ on archive.org )
• Release Manager - Coordinated the product handoff from engineering to manufacturing. Developed and documented the process and contacts involved: customer support, royalties payable, contracts, buyers, export compliance, and the factory change-control board
• Software Build and Release Engineer - Responsible for source control, final compilation, packaging, and release of software. Projects sized from the smallest of system patches, through medium sized projects, to the large 50M+ lines and 2 GB of source IRIX OS builds. Source control management was done via internal CVS-like tools (ptools)
• System Administrator - 24/7 support for company-wide source code and distribution servers, network services, desktop support, and managed Legato Networker daily backups for servers and desktops totaling over 1 terabyte

Treasurer and Network Administrator for Punk Net Internet Cooperative, San Luis Obispo, CA - Jan 1995-Jul 1996

Designed, implemented, and administered a twenty site city-wide network. Responsible for IP address space management, PPP, modem bank, terminal server, billing, new hardware.

Unix System Administrator at Academic Computing Services, Cal Poly, San Luis Obispo, CA - Mar 1993-Jul 1996

Performed network and system administration, security updates, software porting and installation, and new equipment specification of NeXT, Sun, and SGI systems, on a base of 500-600 users. Later (Sep 1995-July 1996), I additionally, cared for Aeronautics Dept's seven SGI machines in a similar fashion.

Expertise

Programming

• Expertise in Python, Go (Golang), Protobufs, C, Bourne-Shell
• Experienced with Google Pub/Sub, Google gRPC, Google Spanner, HTML, CGIs, Java, SQL, Perl C++
• Design patterns in: Microservices, Python testing (mock, stub, fake, monkey patch)

Technologies and Applications

• Expertise with: Unix, Linux, particularly with Ubuntu (Debian), Perforce, Bazel (Blaze)
• Equivalent expertise: Google Borg -> Kubernetes, cloud computing and Google Stubby -> gRPC, rpc library
• Experienced with Linux (LVM, md, iptables, ipchains, GRUB, UEFI), Ubiquiti (EdgeRouter, UniFi), OpenWrt, Git/Github, Docker, Xen, SmokePing, Prometheus, Nagios, MacOS 10, Windows server (Samba Active Directory, Samba domain, 2000, NT 4, NT 3), Windows desktops (10 though 95), Cisco IOS, Fedore Core (RedHat) 1-10, RedHat 6.0-9.0, SGI IRIX 4.0.5-6.5, , Solaris, SunOS, NeXTSTEP 3.0, FreeBSD, OpenBSD, HP/UX, Macintosh, Novell Netware, DOS, RCS, CVS, Subversion (svn), make, Ant, gcc, cc, clang, CruiseControl 1.x SMTP (SPF, DKIM, DMARC, sendmail, sendmail.cf, qmail), POP, IMAP (Courier-IMAP, dovecot), fetchmail, Samba, VMware Workatation Player, SCSI, IEEE 1394 (FireWire, iLink), USB, TCP, UDP, NETBEUI, IPSec, portmap, FTP, ssh, OpenSSH, stunnel, rsync, NFS, DHCP, bootp, DNS (bind, named, djbdns), YP/NIS, dnscache, PPP, SLIP, RIP, Multicast (mrouted), Ethernet LAN, WLAN, WiFi, VLAN (802.1q), ISDN, DSL, PPtP, IPsec, L2TP PHP, JavaScript, Apache (mod_ssl, mod_jserv, mod_jk), httpd, EFF Certbot, HTTP/1.0, HTTP/1.1, HTTPS, SSL, JServ, Java, Servlets, Tomcat, Oracle Database, MySQL, Postgres

Education

• BS, Computer Science, Cal Poly State Univ., San Luis Obispo, CA, 1996
• Bellarmine College Prep, San Jose, CA, Class of 1990

Other Accomplishments and Hobbies

Unix system administrator classification

Based on the Usenix LISA Core Job Descriptions, I am qualified as Senior System Administrator of a large (1000+ hosts, 1000+ users) site. Experienced primarily with Linux. (see: LISA Core Job Descriptions) Contributed and reviewed content for the IRIX System Administration Certification courses in 2000.

Event Leader at the Silicon Valley Bicycle Exchange 2019 - current

Organize and run events for up to 16 volunteers to repair donated bicycles. Hundreds of bicycles are repaired every year (800 bikes in 2023) for donation back to the community.

Convention Staff - Various Science Fiction Conventions, San Jose - 1990 - current

Worked BayCon, TimeCon, SiliCon, and Fanime with attendance ranging from 1,000 to 34,000 members. Positions include registration desk manager, and operations staff. Responsibilities include crisis management, registration, cashiering, scheduling, and managing a staff of upto 25 volunteers.

Computational Performance - SETI@Home, Crypto Speed Challenges - 1998-2000

Developer, Coordinator, and Advocate for the SETI@Home participation at SGI, as well as some of the RSA Security Inc sponsored Cryptographic Speed Challenges, utilizing thousands of machines across the company intranet. The difficult aspects of this project were coordination and advocacy; the technical portions were straightforward.

Personal Info

I enjoy equestrian pursuits (Hunter/Jumper), road biking, electronics, woodworking, and yoga.