Page MenuHomePhabricator

Exactness
Updated 2,175 Days AgoPublic

Exactness is a pixel perfect regression testing tool for Elementary based applications.

Exactness works by playing back a recording of an interaction with the
application, saving images from that playback, and comparing the images to the
set of "known to be correct" images. Returning a non-zero return value when it
detects images mismatches and creates "error.html", a file that includes the
list of tests failed, including the base images, the new images, and a visual
"diff" of the images. Exactness also fails in case the tests themselves exit
with a non-zero return value, e.g segfault.

Modes of operation

Exactness has four modes of operation, three of which are essential for normal
operation, and the fourth is helpful in some cases, and in general, just pretty.

List of Modes

  1. record (-r, --record): record a test macro.
  2. init (-i, --init): run the test macro and take images of the state.
  3. play (-p, --play): run the test macro, take images of the state, and compare against the base images (took with init).
  4. simulation (-s, --simulation): run the test macro, showing the process on screen.

The List file

The list file contains a list of tests to actually run.
The file format is:
<test name> <test command>
<test name> has to be one word (no spaces), there are no restrictions on the command. The command is executed using "system", so you can set environment variables and generally do whatever you'd like.
Empty lines and lines starting with # are ignored.

Example line:

box ./elm_edje_player tests/box.edj

Using exactness

For the rest of this section we'll assume the name of the list file is
"tests.txt".

Recording

The first thing to do when running exactness on a new test is generating the
recording file, this is done but running with "-r". New recordings overwrite
old existing recordings so it's always a good idea to backup recordings you are
happy with.
In order to record for a test list run:

exactness -r tests.txt

Just interact with the application in any way you'd like (it's better to have
short interactions as tests run in real time, so running the tests will be as
slow as your recording).
Press F2 to mark an image taking opportunity (as many times you'd like), this
will cause "init" and "play" mode to take images for comparison. Please refer
to the "Limitations" section below to read more about image taking
opportunities.
Running this will create the "base dir" (see command line options for more info
about how to change it) and populate it with one recording file per test.

Init

After having a recording file for a test you can go on with capturing the base
images. Those images are assumed to be "correct". You should update them when
this assumption is no longer valid (e.g fixed a bug, updated a test).
Run:

exactness -i tests.txt

This will look for recording files in the "base dir", run them, take images of
the application, and put them in "dest-dir/orig".

Play

This is the most common mode of execution. This should be run on a regular
basis, for example in a CI system.
Run:

exactness -p tests.txt

This will look for recording files in the "base dir", run them, take images of the application, put them in "dest-dir/current", and compare them against the
images in "dest-dir/orig". This will fail (and return a non-zero return value)
if any of the tests return a non-zero return value, or if any of the images
in the "current" dir don't match the ones in "orig" dir.
Detecting a failure is the most important role of exactness, so the return
value is very important. Additionally, an error.html file will be created in
the "current" dir, listing the issues in a human-readable form.

Simulation

This mode is a show off and a debug mode. Very useful with the "-v" (verbose)
option that shows the mouse cursor on the canvas and outputs information to
console.
Run:

exactness -s tests.txt

or for verbose mode:

exactness -v -s tests.txt

This does nothing other than run the tests to screen. No images are taken,
and no report is generated.

Command line options

  • -b, --base-dir: The location of the recording files.
  • -d, --dest-dir: The location of the base dir for the images (orig/current will be appended).
  • -w, --wrap: Use a custom command to launch the tests with, e.g valgrind.
  • -j, --jobs: Number of jobs to run in parallel. Very useful when running in init and play mode.
  • -r, --record: record mode.
  • -p, --play: play mode.
  • -i, --init: init mode.
  • -s, --simulation: simulation mode.
  • -v, --verbose: Show cursor location and output extra information to console.
  • -h, --help: Help, and information about more boring options.

Limitations

  • Animations are very time-specific, you can't really take snapshots of

applications that have animations.

  • Exactness is very susceptible to theme/scale/etc. changes, so make sure to

replicate the full environment.

  • Only works with Elementary applications.
Last Author
tasn
Last Edited
Jul 31 2013, 5:08 AM
Projects
None
Subscribers
None