Page MenuHomePhabricator

unit test / CI issues
Open, Showstopper IssuesPublic

Description

rEFLdefded25ebe6d16a675ee968f8fc0f785310ccdf had a lot of probably-unintended side effects:

  • disabled running unit tests on regular travis builds
    • see chunk from ci-make-check.sh where the "options-enabled" build is also aborted
  • disabled running tests with wayland/drm/egl enabled
  • side note for @bu5hm4n: meson never builds pixman support on CI at present

Further, reenabling tests for the "options-enabled" build is not currently desirable:

  • image masking was never implemented for the pixman image rendering codepath so evas masking tests will always fail
  • this build would always use the pixman codepaths, which nobody uses anyway

I propose the following steps to resolve these issues:

  1. move "options-enabled" to a cron build
    • nobody cares about most of these options so there's no point in building it for every patch; it's fine to know once in a while if they work
  2. maintain state of not running unit tests on "options-enabled" build
    • unless someone wants to spend time implementing pixman masking, which nobody will ever use?
  3. move "options-disabled" to a cron build
    • nobody cares about these options either
  4. add back a "normal" fedora build to commit-triggered builds
  5. add back a "wayland" fedora build to commit-triggered builds

This will reenable running unit tests consistently, which is far more valuable than constantly running 4 builds for targets that literally nobody cares about.

zmike created this task.Fri, Jan 4, 9:03 AM
zmike triaged this task as Showstopper Issues priority.

I understand the problematic of having the test suite run in the all options enabled build.
Agreed that we need to have a build back with the default options which will also run the test suite.

I am happy to have the options-disabled builds (autotools as well as meson) moved to the cron job to make room for this.
I still would like to keep the options-enabled around as I see a benefit in having reports on such breaks early. And we have 5 parallel builds with Travis anyway.

If we want to have the wayland build options part of the job whioch also runs the tests I would argue we should but them into the normal fedora build job and not having another one. (Compile testing is already done for wayland with the options-enabled builds and we are talking about the wayland test suite here only)

Does that sound like a solution you could live with?

zmike added a comment.Wed, Jan 9, 8:55 AM

The issue with rolling wayland into the "normal" job is that it uses egl while the "normal" job uses full gl. Ideally we should (in the very near future) be adding headless unit tests for gl, which would require that both codepaths are tested and they cannot be enabled in the same build.

While it would be good to see the "enabled" options build run frequently to check for build breaks, the point of commit-triggered CI should be (imo) to verify runtime breaks, e.g., unit test failures, on commonly used build configurations, with verifying buildable state as a nice side benefit--this is due to the number of builds we have available on travis. To my knowledge, none of the options in the "enabled" build are used; if any of them are in widespread use then they should be added to the default build options.