Page MenuHomePhabricator

Exactness: integration into EFL test process
Open, TODOPublic

Description

Hi guyz,

As we discussed by mail, the plan is to integrate Exactness test suite into the EFL test process. Not with make check, as it would increase the duration considerably but as make check-exactness.
In the Exactness repository, I did it. However, for the EFL repo, I feel a little lost.
Here are the steps I think should be done:

Are there any other steps we should consider?

Thanks

Herald closed this task as Invalid. · View Herald TranscriptMay 22 2018, 11:58 PM
Herald added a project: Restricted Project. · View Herald Transcript

The Enlightenment ticket system is currently receiving high amounts of spam tickets. This ticket has been closed as spam because it lacks both a project and an assignee. If this ticket is not spam, please reopen it after adding these items.

JackDanielZ edited projects, added efl, Testing; removed Restricted Project.
JackDanielZ reopened this task as Open.
zmike triaged this task as TODO priority.
zmike added a subscriber: bu5hm4n.
zmike added a comment.May 23 2018, 7:27 AM

I think it's worth discussing whether we want to go with a git submodule or a git subtree for this? After considering further, I do think it's valuable to have the exactness exe in-tree, as apps may want to provide their own exactness data and run it independently of the efl checking.

I think the best way to implement this, at least in the build system, would be something like:

  1. add configure flag to build exactness exe
  2. add configure flag to enable exactness checking in efl
  3. if both enabled, have some additional makefile rule which downloads/updates the exactness data and runs the checks

Well I didn't know about git subtree. I googled it and I don't think it fits our needs. AFAIU, it brings the history inside the super history. If this is the case, how do you disable that for non-exactness users? More than that, it would mean all the exactness-elm-data would be stored as commits into the EFL repo. People will kill us for the amount of data they will have to fetch!
From my short experience, Submodule seems a good way for that kind of data repo.

Concerning the steps:
1 - should be easy to do although I don't think it is necessary. Exactness is really fast to compile. It is just a few C files.
2 - easy too
3:

  • shouldn't the download/update stuff be done manually. If some file has been modified there (orig directory), it could make issues to update it.
  • in which file do I need to put this rule? I mean, if I put it in Makefile_Exactness.am, is 'make check-exactness' known by make? I ask because I didn't succeed to do it in the Exactness repository
zmike added a comment.May 24 2018, 6:14 AM

Well I didn't know about git subtree. I googled it and I don't think it fits our needs. AFAIU, it brings the history inside the super history. If this is the case, how do you disable that for non-exactness users? More than that, it would mean all the exactness-elm-data would be stored as commits into the EFL repo. People will kill us for the amount of data they will have to fetch!
From my short experience, Submodule seems a good way for that kind of data repo.

Sure, I've never used subtree I just know that it exists. Having at least some (public) statement about it here is good to ensure there is some documentation regarding why one was chosen over the other. submodule also seems to me like what would be desired for our use case.

Concerning the steps:
1 - should be easy to do although I don't think it is necessary. Exactness is really fast to compile. It is just a few C files.

Ok

2 - easy too
3:

  • shouldn't the download/update stuff be done manually. If some file has been modified there (orig directory), it could make issues to update it.

My idea here was based on experience from previous projects which used submodules; the build rules would automatically checkout/update the submodule during build. Having to do it manually seems a bit too...efl...

  • in which file do I need to put this rule? I mean, if I put it in Makefile_Exactness.am, is 'make check-exactness' known by make? I ask because I didn't succeed to do it in the Exactness repository

I think probably you just have a conditional block which adds something to check_PROGRAMS and TESTS when enabled? Then you just use that; a specific rule will be generated for it (e.g., running elm tests can be done from src/ with make tests/elementary/elm_suite.log). I would assume anyone who enables efl exactness checking will want to run it with make check anyway.

@bu5hm4n I added you here so you can keep track of build system stuff for possible future meson work

In T6949#114800, @zmike wrote:

Well I didn't know about git subtree. I googled it and I don't think it fits our needs. AFAIU, it brings the history inside the super history. If this is the case, how do you disable that for non-exactness users? More than that, it would mean all the exactness-elm-data would be stored as commits into the EFL repo. People will kill us for the amount of data they will have to fetch!
From my short experience, Submodule seems a good way for that kind of data repo.

Sure, I've never used subtree I just know that it exists. Having at least some (public) statement about it here is good to ensure there is some documentation regarding why one was chosen over the other. submodule also seems to me like what would be desired for our use case.

Concerning the steps:
1 - should be easy to do although I don't think it is necessary. Exactness is really fast to compile. It is just a few C files.

Ok

2 - easy too
3:

  • shouldn't the download/update stuff be done manually. If some file has been modified there (orig directory), it could make issues to update it.

My idea here was based on experience from previous projects which used submodules; the build rules would automatically checkout/update the submodule during build. Having to do it manually seems a bit too...efl...

Ok. I can understand your point. I don't know what is better between updating during the build or the check. At each check invocation, it seems more accurate but it will take a few seconds every time (most of the time, for nothing). Is it critical?
For the CI, it is not a big deal as build and check will be done anyway at each iteration.

  • in which file do I need to put this rule? I mean, if I put it in Makefile_Exactness.am, is 'make check-exactness' known by make? I ask because I didn't succeed to do it in the Exactness repository

I think probably you just have a conditional block which adds something to check_PROGRAMS and TESTS when enabled? Then you just use that; a specific rule will be generated for it (e.g., running elm tests can be done from src/ with make tests/elementary/elm_suite.log). I would assume anyone who enables efl exactness checking will want to run it with make check anyway.

Sorry but I am a little confused. I thought we said exactness should not be invoked in make check. Or maybe you meant that for regular users, it should not be run except when it is enabled by some --enable-exactness-check? FYI, I am ok with that too. I just thought you wanted to stay under 2s :D
Anyway, your solution seems simple and makes the job.

@stefan_schmidt what do you think?

@bu5hm4n I added you here so you can keep track of build system stuff for possible future meson work

I am a bit late on this, sorry.

As for 1) I think we should have the exactness executable in tree. Its small and quick to compile and it will make it easier to be used. Please create a Makefile_Exactness.am and put it under src/bin/exactness. It does only have efl internal dependencies (or did I miss an external dependency?) so I think we should compile it without any configure flag.

For 2) we definitely need a configure option to enable the data use. If we agree on git submodules for this I am ok with trying it out for this purpose. What I would expect would be a configure flag like --enable-exactness-check and a build target make check-exactness. You can define this local rule in the Makefile_Exactness.am and make it conditional behind the configure flag. See how we did it in the main Makefile.am and the conditionals for EFL_ENABLE_COVERAGE.

As for 3) having it updated automatically (the rule should still work though without network access) is a good idea imho. We need to see if that really works for us. The local changes might be one problem. I honestly know to little on git submodules to say here what to do best. If I would approach it I would aim for the following: a) when make check-exactness is called check if the submodule is already setup, if not do so b) check if the submodule data is up to date, if not update (here migth be dragons on how to handle update fails) c) if it is upd top date run the exactness binary for comparison with the given recordings.

To make this all a bit easier I think we can safely split off item number one form the rest. getting the exactness bianry in tree is the first step and could be done any time now. It has no impact on library code or other parts of EFL so I think it is still fine to get it in this late before the freeze.
The whole integration of the data as git submodule and how it will be update/run, etc from the build system might be trickier and can be done as s second step once the binary is in. Before this can land we need to make sure it works on all kind of awkward system efl is being used on. :-)

How does that sound? @JackDanielZ does thsi strategy work for you? @zmike could you give us a pointer to the build system logic used for git submodules in the other project you mentioned? I think some example could be helpful for Daniel when he starts poking around at this.

I am a bit late on this, sorry.

As for 1) I think we should have the exactness executable in tree. Its small and quick to compile and it will make it easier to be used. Please create a Makefile_Exactness.am and put it under src/bin/exactness. It does only have efl internal dependencies (or did I miss an external dependency?) so I think we should compile it without any configure flag.

I created it in src because I have a library that supplies APIs to read the Exactness files. There are no external dependency. Once there was imagemagick to compare but it is now done internally.

For 2) we definitely need a configure option to enable the data use. If we agree on git submodules for this I am ok with trying it out for this purpose. What I would expect would be a configure flag like --enable-exactness-check and a build target make check-exactness. You can define this local rule in the Makefile_Exactness.am and make it conditional behind the configure flag. See how we did it in the main Makefile.am and the conditionals for EFL_ENABLE_COVERAGE.

Already done in my branch 'exactness'

As for 3) having it updated automatically (the rule should still work though without network access) is a good idea imho. We need to see if that really works for us. The local changes might be one problem. I honestly know to little on git submodules to say here what to do best. If I would approach it I would aim for the following: a) when make check-exactness is called check if the submodule is already setup, if not do so b) check if the submodule data is up to date, if not update (here migth be dragons on how to handle update fails) c) if it is upd top date run the exactness binary for comparison with the given recordings.

Same for me, I don't know what is the best choice between making the submodule update automatic or not. Anyway, I don't think this is so hard to make it work in the makefile.

To make this all a bit easier I think we can safely split off item number one form the rest. getting the exactness bianry in tree is the first step and could be done any time now. It has no impact on library code or other parts of EFL so I think it is still fine to get it in this late before the freeze.

I will wait for the EFL/Enlightenment to be stable before pushing Exactness in the tree. It is ready.

The whole integration of the data as git submodule and how it will be update/run, etc from the build system might be trickier and can be done as s second step once the binary is in. Before this can land we need to make sure it works on all kind of awkward system efl is being used on. :-)

Yep. All is done except the submodule update. If the base is ok, we could even push it, as it can make issues only if it is enabled.

Speaking of ackward, we need to figure out why shots are so different between machines. I should speak with Carsten, he may have some good ideas.

How does that sound? @JackDanielZ does thsi strategy work for you? @zmike could you give us a pointer to the build system logic used for git submodules in the other project you mentioned? I think some example could be helpful for Daniel when he starts poking around at this.

This strategy sounds good to me. KISS always works :-)

For the moment, the submodule is in data/exactness/elm/... Is it ok?

Another important thing: exactness executes the apps (e.g elementary_test) by checking $PATH, meaning dist-check should not work well with that if we want it supported, right?

Hi guys,

I got an update concerning the fonts. It seems that the main shots difference we have between computers is due to the installation of ttf-bitstream-vera fonts. They are loaded prior on ttf-dejavu-vera fonts. AFAIU, dejavu is an extended and more recent version of bitstream and therefore preferrable.
I succeeded to force the loading of fonts from a specific directory. So, for now, the new version of Exactness (present in my branch) supports the overriding of the system fonts via a -f/--fonts option. What I planned to do is to store the dejavu fonts in a the exactness-elm-data repository, per datestamp i.e fonts/YYYYMMDD. You can check my 'fonts' branch in the repo for a better understanding.
I don't know if it is the best solution but it is very flexible. The idea is that exu using older versions of ttf could use them, as long as the fonts datestamp is specified inside (e.g 20180601 is the directory in fonts that have been gathered on the 1st of June 2018). If the exu doesn't specify it, then the newest fonts are loaded.

It doesn't solve the freetype/harfbuzz upgrades but it should help.

What do you think?

YOhoho added a subscriber: YOhoho.Jun 3 2018, 6:06 PM
zmike added a comment.Jun 4 2018, 4:38 AM

I missed a couple replies to this in my inbox, so I'm coming in a bit late. Going to avoid quoting here in order to keep the comments from becoming too large and unreadable:

  • My idea was to not have a separate make check rule for exactness, and to instead just add it to the TESTS variable when exactness checking is enabled at configure time (a separate option from just enabling building the exactness binary). If someone enables it at configure then we should assume they'll want to run it, so this simplifies the makefile. I realize this is different from the original discussion, and I'm not strongly opposed to having a separate rule for it, but I think there is something to be said for potentially removing some complexity from the makefiles?
  • I think the best example that I have of submodules in use is probably from Servo, which at this time used custom makefiles to handle a LOT of submodules. We can probably at least get some ideas from this; the primary file to look at is 'configure' in the base directory. The submodule should be in data/exactness as has been proposed I think.

@JackDanielZ can you link to the changeset which adds the font handling? This could be useful for people to reference in the future. Nice work figuring it out! As for handling freetype/hb upgrades, do they have libraries to determine the runtime version? We may have to maintain separate data for when a breaking change occurs...

In T6949#115284, @zmike wrote:

I missed a couple replies to this in my inbox, so I'm coming in a bit late. Going to avoid quoting here in order to keep the comments from becoming too large and unreadable:

  • My idea was to not have a separate make check rule for exactness, and to instead just add it to the TESTS variable when exactness checking is enabled at configure time (a separate option from just enabling building the exactness binary). If someone enables it at configure then we should assume they'll want to run it, so this simplifies the makefile. I realize this is different from the original discussion, and I'm not strongly opposed to having a separate rule for it, but I think there is something to be said for potentially removing some complexity from the makefiles?

I am ok with that. make check runs all the tests, including exactness suite if enabled in configure. Additionnally, make check-exactness from the src directory runs only exactness suite. Both launch the same script that invokes exactness.

  • I think the best example that I have of submodules in use is probably from Servo, which at this time used custom makefiles to handle a LOT of submodules. We can probably at least get some ideas from this; the primary file to look at is 'configure' in the base directory. The submodule should be in data/exactness as has been proposed I think.

I will take a look

@JackDanielZ can you link to the changeset which adds the font handling? This could be useful for people to reference in the future. Nice work figuring it out! As for handling freetype/hb upgrades, do they have libraries to determine the runtime version? We may have to maintain separate data for when a breaking change occurs...

You mean the Exactness commit (https://phab.enlightenment.org/rTEXACT0e1f0cf1c5f7766f5228e74bbbc718a4e898660c) that handles the font handling?

zmike added a comment.Jun 5 2018, 6:41 AM

Yup, thanks!

zmike edited projects, added Restricted Project; removed efl.Jun 11 2018, 6:55 AM
zmike edited projects, added efl; removed Restricted Project.Jun 11 2018, 8:50 AM
bu5hm4n edited projects, added Restricted Project; removed efl.Jun 12 2018, 8:14 AM

Hi,

@stefan_schmidt, how was your vacation/leave? When do you think you will have time to review my plans to integrate Exactness into EFL?

Daniel

zmike added a comment.Aug 2 2018, 6:15 AM

@JackDanielZ Feel free to post about it here and everyone can review...