Page MenuHomePhabricator

optimize elm tests
Open, TODOPublic

Description

src/tests/elementary/elm_suite.log15.251

Related Objects

StatusAssignedTask
OpenNone
OpenNone
OpenNone
zmike created this task.Apr 6 2018, 11:04 AM
zmike triaged this task as TODO priority.

@stefan_schmidt I've discovered that the biggest bottleneck in test runtime, by FAR, is building with coverage enabled. I think we should be running coverage maybe once or twice a week at most, and all other test runs should be without coverage. The speed difference is actually unbelievable.

@zmike Not sure I follow you here. You mean make check vs make lcov-check runtime or do you mean the runtime of make check when configured with coverage vs regular?

The increased test time from make check vs make lcov-check is indeed huge (61s vs 152s) but to be expected.

I see only a marginal difference in running make check configured with coverage vs. regular (61s vs 59s)

For the normal test flow I would expect developers and CI tools to run only make check (and we can have it configured as regular as well). The coverage run could easily be done during a nightly or weekly build.

zmike added a comment.Apr 19 2018, 7:23 AM

Actually I do mean running make check --with-tests=coverage vs --with-tests=regular. The difference here is absolutely staggering. As an example, elm_suite takes ~6-7s to run with coverage enabled, but only 1s to run without it; fork() calls with coverage enabled take 0.03s, vs 0.0s without coverage. While this may seem like a trivial number, consider that when running parallelized, each test case has a fork() call, and then each test in the test case also has a fork--meaning that the total number of fork calls in a test suite is $num_test_cases * (1 + $num_tests). In elm_suite, this results in ~5s spent doing nothing but executing the fork calls. As coverage increases and the number of tests increases, this number will also continue to increase.

zmike edited projects, added Restricted Project; removed efl.Jun 11 2018, 6:55 AM
bu5hm4n edited projects, added efl: widgets, Restricted Project; removed Restricted Project.Jun 11 2018, 8:45 AM
bu5hm4n added a project: Restricted Project.Jun 12 2018, 8:14 AM