Hunting the Shmoo

I first joined Mozilla as an intern in 2010 for the “Tools and Automation Team” (colloquially called the “A-Team”). I always had a bit of difficulty describing our role. We work on tests. But not the tests themselves, the the thing that runs the tests. Also we make sure the tests run when code lands. Also we have this dashboard to view results, oh and also we do a bunch of miscellaneous developer productivity kind of things. Oh and sometimes we have to do other operational type things as well, but it varies.

Over the years the team grew to a peak of around 25 people and the A-Team’s responsibilities expanded to include things like the build system, version control, review tools and more. Combined with Release Engineering (RelEng), this covered almost all of the software development pipeline. The A-Team was eventually split up into many smaller teams. Over time those smaller teams were re-org’ed, split up further, merged and renamed over and over again. Many labels were applied to the departments that tended to contain those teams. Labels like “Developer Productivity”, “Platform Operations”, “Product Integrity” and “Engineering Effectiveness”.

Interestingly, from 2010 to present, one label that has never been applied to any of these teams is “DevOps”.

If you’re working with mozilla-central on Windows and followed the official documentation, there’s a good chance the MozillaBuild shell is running in the default cmd.exe console. If you’ve spent any amount of time in this console you’ve also likely noticed it leaves a bit to be desired. Standard terminal features such as tabs, splits and themes are missing. More importantly, it doesn’t render unicode characters (at least out of the box).

Luckily Microsoft has developed a modern terminal that can replace cmd.exe, and getting it set up with MozillaBuild shell is simple.

This post was co-authored with Marco Castelluccio, and was originally posted to the Mozilla Hacks Blog.

A browser is an incredibly complex piece of software. With such enormous complexity, the only way to maintain a rapid pace of development is through an extensive CI system that can give developers confidence that their changes won’t introduce bugs. Given the scale of our CI, we’re always looking for ways to reduce load while maintaining a high standard of product quality. We wondered if we could use machine learning to reach a higher degree of efficiency.

Mozilla recently announced that we are planning to de-commission irc.mozilla.org in favour of a yet to be determined solution. As a long time user and supporter of IRC, this decision causes me some melancholy, but I 100% believe that it is the right call. Moreover, having had an inside glimpse at the process to replace it, I’m supremely confident whatever is chosen will be the best option for Mozilla’s needs.

I’m not here to explain why deprecating IRC is a good idea. Other people have already done so much more eloquently than I ever could have. I’m also not here to push for a specific replacement. Arguing over chat applications is like arguing over editors or version control. Yes, there are real and important differences from one application to the next, but if there’s one thing we’re spoiled for in 2019 it’s chat applications. Besides, so much time has been spent thinking about the requirements, there’s little anyone could say on the matter that hasn’t already been considered for hours.

This post is about an unrelated, but adjacent issue. An issue that began when mozilla.slack.com first came online, an issue that will likely persist long after irc.mozilla.org rides off into the sunset. An issue I don’t think is brought up enough, and which I’m hoping to start some discussion on now that communication is on everyone’s mind. I’m talking about using two communication platforms at once. For now Slack and IRC, soon to be Slack and something else.

Different platform, same problem.

Mozilla uses a lot of Python. Most of our build system, CI configuration, test harnesses, command line tooling and countless other scripts, tools or Github projects are all handled by Python. In mozilla-central there are over 3500 Python files (excluding third party files), comprising roughly 230k lines of code. Additionally there are 462 repositories labelled with Python in the Mozilla org on Github (though many of these are not active). That’s a lot of Python, and most of it is Python 2.

With Python 2’s exaugural year well underway, it is a good time to take stock of the situation and ask some questions. How far along has Mozilla come in the Python 3 migration? Which large work items lie on the critical path? And do we have a plan to get to a good state in time for Python 2’s EOL on January 1st, 2020?

A talk I did for the Automationeer’s Assemble series on how Mozilla handles complexity in their CI configuration.

Have you ever needed to inspect the taskgraph locally? Did you have a bad time? Learn how to inspect the taskgraph like a PRO. For the impatient skip to the installation instructions below.

It’s no secret that I’m not a fan of try syntax, it’s a topic I’ve blogged about on several occasions before. Today, I’m pleased to announce that there’s a real alternative now landed on mozilla-central. It works on all platforms with mercurial and git. For those who just like to dive in:

$ mach mercurial-setup --update  # only if using hg
$ mach try fuzzy

This will prompt you to install fzf. After bootstrapping is finished, you’ll enter an interface populated with a list of all possible taskcluster tasks. Start typing and the list will be filtered down using a fuzzy matching algorithm. I won’t go into details on how to use this tool in this blog post, for that see:

$ mach try fuzzy --help  # or
$ man fzf

Imagine this scenario. You’ve pushed a large series of commits to your favourite review tool (because you are a believer in the glory of microcommits). The reviewer however has found several problems, and worse, they are spread across all of the commits in your series. How do you fix all the issues with minimal fuss while preserving the commit order?

I’ve previously blogged about why I believe try syntax is an antiquated development process that should be replaced with something more modern and flexible. What follows is a series of ideas that I’m trying to convert into a concrete plan of action to bring this about. This is not an Intent to Implement or anything like that, but my hope is that this outline is detailed enough that it could be used as a solid starting point by someone with enough time and motivation to work on it.