Hunting the Shmoo

Dry versus Wet

As programmers, we are taught early on that code duplication is bad and should be avoided at all cost. It makes code less maintainable, reusable and readable. The DRY principle is very basic and fundamental to how most of us approach software design. If you aren’t familiar with the DRY principle, please take a minute to read the wikipedia page on it. The counterpart of DRY, is WET (write everything twice). In general, I agree that DRY is good and WET is bad. But I think there are a class of problems where the DRY approach can actually be harmful. For these types of problems, I will make a claim that a WET approach can actually work better.

Getting Started

tl;dr - It is possible to add more mach to your B2G repo! To get started, install pip:

$ wget https://raw.github.com/pypa/pip/master/contrib/get-pip.py -O - | python

Install b2g-commands:

$ pip install b2g-commands

To play around with it, cd to your B2G repo and run:

$ git pull                 # make sure repo is up to date
$ ./mach help              # see all available commands
$ ./mach help <command>    # see additional info about a command

Details

Most people who spend the majority of their time working within mozilla-central have probably been acquainted with mach. In case you aren’t acquainted, mach is a generic command dispatching tool. It is possible to write scripts called ‘mach targets’ which get registered with mach core and transformed into commands. Mach targets in mozilla-central have access to all sorts of powerful hooks into the build and test infrastructure which allow them to do some really cool things, such as bootstrapping your environment, running builds and tests, and generating diagnostics.

Mach is an amazing tool which facilitates a large number of common user stories in the mozilla source tree. You can perform initial setup, execute a build, run tests, examine diagnostics, even search Google. Many of these things require an object directory. This can potentially lead to some confusion if you typically have more than one object directory at any given time. How does mach know which object directory to operate on?

This probably isn’t a groundbreaking revelation to most Mozillians, but it’s something I haven’t quite managed to put into words until now.

On the Tools and Automation team at Mozilla, the easiest way to describe what we do is to say “We try to make the lives of others within the community easier”. This is a vague statement, but it’s about as close as we can get without writing a small dissertation.

Before now running ‘classic’ unittests (mochitest, reftest, xpcshell, etc.) on B2G emulators has been a massive pain. The new recommended way of running them is through mach.

1. Update B2G repo if you haven’t already: git pull
2. Configure an emulator: BRANCH=master ./config.sh emulator
3. Build: ./build.sh
4. Run: ./mach mochitest-remote

OUTDATED

The steps in this article are no longer accurate. Pushing a custom command line is easier now. See here for more details.

Have you ever wanted to see the test results of a custom command line in try? Things like –test-manifest, –shuffle or –run-slower? Now you can! The process isn’t exactly optimized for the developer use case, but neither is it really difficult to do once you know how.

As B2G continues to trod onwards to its release, there is still a lot of confusion about the level and state of test coverage it has. Back in November we started running mochitests, reftests and marionette/webapi tests on ARM emulators. Now we’ve also added xpcshell tests and for the most part we have these nice green letters to look at on TBPL that make us feel good about ourselves. But what is really being run? What is the meaning behind these letters “M”, “R”, “Mn” and “X”? Are there any causes for concern? Are there other tests being run that don’t show up on TBPL? What are the current automation priorities? What are the next platforms to use after emulators?

This blog post aims to answer these questions and more. It is a comprehensive snapshot of the current state of automated testing on B2G.

Contrary to popular belief, we (the A-Team) have been running mochitests, reftests, marionette tests and webapi tests on B2G in some form of continuous integration or another for about 5 months now. They’ve been reporting results to a TBPL look-alike called autolog, and were run on Amazon EC2 VM’s with emulators. This was a temporary solution to get something stood up quickly while we moved towards our ultimate B2G automation goal - tests running on Pandaboards and reporting to TBPL.

As of this week, while there are still no tests running on Pandaboards, I’m happy to say we have emulators running mochitests, reftests and marionette/webapi tests, all reporting to TBPL.