Page MenuHomePhabricator

unit test / CI issues
Closed, ResolvedPublic


rEFLdefded25ebe6d16a675ee968f8fc0f785310ccdf had a lot of probably-unintended side effects:

  • disabled running unit tests on regular travis builds
    • see chunk from where the "options-enabled" build is also aborted
  • disabled running tests with wayland/drm/egl enabled
  • side note for @bu5hm4n: meson never builds pixman support on CI at present

Further, reenabling tests for the "options-enabled" build is not currently desirable:

  • image masking was never implemented for the pixman image rendering codepath so evas masking tests will always fail
  • this build would always use the pixman codepaths, which nobody uses anyway

I propose the following steps to resolve these issues:

  1. move "options-enabled" to a cron build
    • nobody cares about most of these options so there's no point in building it for every patch; it's fine to know once in a while if they work
  2. maintain state of not running unit tests on "options-enabled" build
    • unless someone wants to spend time implementing pixman masking, which nobody will ever use?
  3. move "options-disabled" to a cron build
    • nobody cares about these options either
  4. add back a "normal" fedora build to commit-triggered builds
  5. add back a "wayland" fedora build to commit-triggered builds

This will reenable running unit tests consistently, which is far more valuable than constantly running 4 builds for targets that literally nobody cares about.

zmike created this task.Jan 4 2019, 9:03 AM
zmike triaged this task as Showstopper Issues priority.

I understand the problematic of having the test suite run in the all options enabled build.
Agreed that we need to have a build back with the default options which will also run the test suite.

I am happy to have the options-disabled builds (autotools as well as meson) moved to the cron job to make room for this.
I still would like to keep the options-enabled around as I see a benefit in having reports on such breaks early. And we have 5 parallel builds with Travis anyway.

If we want to have the wayland build options part of the job whioch also runs the tests I would argue we should but them into the normal fedora build job and not having another one. (Compile testing is already done for wayland with the options-enabled builds and we are talking about the wayland test suite here only)

Does that sound like a solution you could live with?

zmike added a comment.Jan 9 2019, 8:55 AM

The issue with rolling wayland into the "normal" job is that it uses egl while the "normal" job uses full gl. Ideally we should (in the very near future) be adding headless unit tests for gl, which would require that both codepaths are tested and they cannot be enabled in the same build.

While it would be good to see the "enabled" options build run frequently to check for build breaks, the point of commit-triggered CI should be (imo) to verify runtime breaks, e.g., unit test failures, on commonly used build configurations, with verifying buildable state as a nice side benefit--this is due to the number of builds we have available on travis. To my knowledge, none of the options in the "enabled" build are used; if any of them are in widespread use then they should be added to the default build options.

I added jobs back for wayland as well as one with the default options. Both of them are also running make check / ninja test. That should cover the root of this problem, having wayland with egl and actually running the unit tests.

What one can also see from the run above is that the length of a full CI run after a push is dictated by the osx build. With 33min is set the upper bar of the CI run. All the other jobs, even when only being allowed with 4 of them in parallel, are nicely slotted to fit into the same time window.

While I agree that the push builds should also detect runtime breaks (I am also working on getting exactness working with CI to help on this) it should also detect build breaks for configurations developers are using on their systems. We had way to much build breaks for windows, osx or different config options over the years which nobody detected in time. CI should also help with that and I very much want to have this information directly after a push and not in the cron daily build.

To really reduce the load on the push builds we need to a) find a way to drastically reduce the time of the osx build and b) drop the autotools build system so we only need to run the, way faster, meson/ninja jobs.