Hunting the Shmoo

As B2G continues to trod onwards to its release, there is still a lot of confusion about the level and state of test coverage it has. Back in November we started running mochitests, reftests and marionette/webapi tests on ARM emulators. Now we’ve also added xpcshell tests and for the most part we have these nice green letters to look at on TBPL that make us feel good about ourselves. But what is really being run? What is the meaning behind these letters “M”, “R”, “Mn” and “X”? Are there any causes for concern? Are there other tests being run that don’t show up on TBPL? What are the current automation priorities? What are the next platforms to use after emulators?

This blog post aims to answer these questions and more. It is a comprehensive snapshot of the current state of automated testing on B2G.

Contrary to popular belief, we (the A-Team) have been running mochitests, reftests, marionette tests and webapi tests on B2G in some form of continuous integration or another for about 5 months now. They’ve been reporting results to a TBPL look-alike called autolog, and were run on Amazon EC2 VM’s with emulators. This was a temporary solution to get something stood up quickly while we moved towards our ultimate B2G automation goal - tests running on Pandaboards and reporting to TBPL.

As of this week, while there are still no tests running on Pandaboards, I’m happy to say we have emulators running mochitests, reftests and marionette/webapi tests, all reporting to TBPL.

This quarter I’ve been focusing on getting reftests running on B2G, triaging them and fixing various issues. The purpose of this post is to outline their status, go over the work that still needs to be done and point out where I will need some help.

Mozconfigwrapper is a tool inspired by Doug Hellman’s magnificent virtualenvwrapper. In a nutshell, mozconfigwrapper hides all of your mozconfigs into a configurable directory (defaults to ~/.mozconfigs), and lets you easily switch, create, remove, edit and list them. Mozconfigwrapper is Unix only for now.

While responsiveness is one of the main goals for Firefox this quarter, we still don’t quite have the means to measure and test our progress towards this goal. The good news is that there are, and have been for some time, several efforts to fix this problem. Back in June, Ted wrote some event tracing instrumentation that gives us a reasonable idea of when the browser becomes unresponsive. This event tracer is already being used by some Talos tests which gives us a good general idea of whether or not Firefox is more or less responsive than it was previously. What it doesn’t give us is a method for developers to write their own tests and determine whether a specific action or feature they are working on is causing unresponsivness.

At the beginning of September, I was asked to write yet another automated test harness for testing user responsiveness. Among other things, the harness needed to be capable of automating a wide range of user interactions in Firefox (such as opening context menus, clicking buttons etc). Oh and by the way this needs to be finished as quickly as possible.

Before I started interning at Mozilla back in May 2010, I really didn’t know what to expect. How does a non-profit company with an open source product operate? After working at giant corporations like IBM and McAfee I couldn’t fathom what the experience would be like.

Firefox is known for its extensibility. In fact, over 2.4 billion addons have been downloaded to date, meaning there are a lot of people using a lot of addons. While having 20+ addons can undoubtedly personalize your browsing experience, it can also be a pain in the ass to manually install them every time you set up a new Firefox profile. As a developer working on Firefox related automation tools, this is twice as true since I create a separate profile for each and every project I work on, installing a constant set of addons on each one.