Meson has a nice feature for dependencies called "Wrap
Dependency", which
downloads dependency source-code and builds it using Meson. This way, it
becomes easier to assume library support for some dependencies such as zlib,
libjpeg, libcheck and etc. natively on some platforms such as Windows.
If the dependency is found, then the build-system will get things from it. It
it is not found, then the subproject will be used.
This is being used to help with T8760.
In a Linux environment, ensure libcheck is uninstalled or
unreachable (as other deps with wraps added here will probably be tied with the
system, UI's packages, etc.) and then build EFL and run tests (e.g. `meson test
-C build`). Meson should download libcheck's (or any other wrap that is not
installed or is unreachable) source code and then build them when trying to
build EFL.
| No Linters Available |
| No Unit Test Coverage |
| Buildable 17077 | |
| Build 11355: arc lint + arc unit |
i'm not very fond of wrapping in meson, because you don't have any control on how the lib is built. For example, is zlib built using the asm files ?
and all that for just 3 libs (one being optional). I'm not very convinced .
note that pcre has not fnmatch, which is required, while ewpi is providing everything
Well, it's not that we had too much control with those installed from a package manager anyway. But see: the wrap is the fallback, not the prioritary dependency provider, so if one must have full control on how the lib is built, that one can then build and install the library on its own and just make sure it's visible to the compiler. But if it is not the main priority, the wrap will handle the missing dependency out of the box.
Other solution is submitting a patch to https://github.com/mesonbuild's respective repo adding options for these compilation settings (or even use the source - that is open - to check if this is actually done).
and all that for just 3 libs (one being optional). I'm not very convinced .
It's just a start. There are plenty of other libs that we could just wrap and they'll be done. For example, libjpeg can be added from now on (specially since https://github.com/mesonbuild/meson/pull/7381 has landed to meson's master) and it's just adding a "jpeg.wrap" and setting it as a fallback.
note that pcre has not fnmatch, which is required, while ewpi is providing everything
That'll come in a later patch for fnmatch implementation.
and I honestly think that it is not the purpose of the EFL build system to build the dependencies. It will add too much stuff to meson
If you want, i can help you adding such framework in ewpi
This patch greatly reduces the burden for developers to compile EFL in Windows and possibly even in some linux distros. Meson knows how to compile those libraries through its own wrap dependency feature. There's no reason we shouldn't use this feature, which also helps compiling with the correct configuration because everything is compiled together. If, OTOH, the user doesn't want to use wrap dependencies, the user can very well specify the dependency through pkg-config or CMake dependencies if they want. This patch is very very useful for people building with hard to use toolsets such as Visual Studio or Clang-CL.
people who are compiling the EFL are very few : packagers (unix world), and devs. Packagers don't need wrap stuff at all. They represent 99% of the people who compile
the others are devs. And the devs are those who compile on unix and those who compile on Windows
and on Windows, there are those who compile with gcc, and the others for Visual Studio
so again, you want to add dozens of wrap stuff for all your needed dependencies. that is, you are going to modify and complexify the build system for not a lot of people
projects like glibs2 of gst-plugins-base have 3, maybe 4 wrap stuff at most.
it can also lead to problems : https://www.collabora.com/news-and-blog/blog/2019/11/26/gstreamer-windows/ here they say to remove libxml2 subproject, and openssl subprojetc has problems... And the post about openssl is recent (some weeks ago)
I propose that you add what you need to ewpi and i propose my help if you want it
Hi,
I am not really a fan of using subprojects here either, not because subprojects per se are bad, but they dont fit the scheme of efl and how distros work. Just from the top of my head a idea on why i dont think its a good idea:
I dont want that this all sounds like a bad idea, wraps are fantastic for some things, they simply do not really fit here good IMO. What i think would serve here the purpose way better would be if we turn the whole thing simply arround. So if you really insist on using wraps, and dont want to have ewpi, you could create another repository, which uses subprojects directly like zlib etc. etc. you get the dependency project and add it as a dependency override. After that you can also suck in efl itself as a subproject, and efl will suck in the deps from the wraps, without having the downsides above on efl itself. This way efl stays clean for packagers, easy on release maintanance, and you can use the advantages of efl. How does that idea sound to you ?
(Also, from the view of development, the efl tree would then simply life in a folder in the subproject subdir, which would simply allow development as there is the .git directoy as well)
Ok, took my time to think of the arguments here. So let me reorganize the ideas...
*Skipping this because I was going to ask to explain further, but I think you already did in your comment below - but feel free to reason a little more about what you meant*
If you want, i can help you adding such framework in ewpi
If we end up not using wraps, sure, I would appreciate some help there :)
I agree with making things easier. Other else, in Windows we'll have some lots of trouble coming from making the compiler know where libs are located and etc. (we had a previous workaround for this using meson_options, but wraps served way better and saved us from having a lot of options for directories (which are also a problem in Windows, since cmdline have a length limit - even if you use tweaks to increase that limit, it has become a problem sometimes before we moved to wraps).
I would agree with a -Dwraps={true,false}, or even better:
option('wraps',
type : 'array',
description : 'Which dependencies to use meson\'s wrap-dependency as fallback.',
choices : ['check', 'freetype2', 'jpeg', ..., 'zlib'],
value : []
)Which would be just a "you can enable a wrap for <these libs> using -Dwraps=lib1,lib2,lib3....
Indeed, I agree that the target wouldn't be so many users.
projects like glibs2 of gst-plugins-base have 3, maybe 4 wrap stuff at most.
I didn't get what you meant here. Mind to explain?
it can also lead to problems : https://www.collabora.com/news-and-blog/blog/2019/11/26/gstreamer-windows/ here they say to remove libxml2 subproject, and openssl subprojetc has problems... And the post about openssl is recent (some weeks ago)
openssl is out of question to use a wrap for now. It's building system is way to complex (mixing makefiles, perl, etc. in a single build-system for a security lib is just...ugh) to make a wrap so soon and I don't even think the result would be that satisfactory.
About problems: I don't think they'll bother us so much if we have clear in our mind that it's not our job to give maintance to wraps. They're a fallback (and we could make it optional, as stated in this thread), so most of the answers would be: "that's meson team's job, open an issue in <repo> so they know the problem exists, but for your specific case, download the dependency's sources and built it yourself following the steps here: <reliable link if we have it>".
I propose that you add what you need to ewpi and i propose my help if you want it
[answered before]
Hi bu5hm4n.
I am not really a fan of using subprojects here either, not because subprojects per se are bad, but they dont fit the scheme of efl and how distros work. Just from the top of my head a idea on why i dont think its a good idea:
- In case we are having subprojects, we are defining the version of a dependency, not the min version of anything,
True and I hadn't realize this before. If wrap-git supported meson patches as well, this could be way more reliable (even if it wouldn't solve the problem of "updates in source upstream breaking wraps").
the version itself is then defined, in case there is a CVE or some major bug, we need to proactivly reroll another release of efl bump versions etc.. Not sure we have the capacity in the release department for that CC @stefan_schmidt )
Now the CVE part got me and I have to totally agree. Didn't crossed my mind that a wrap (the way they are implemented now) would mean sticking to a version and then would mean sticking to a CVE until proper reroll.
To aim this issue, we wouldn't release EFL prebuilt binaries with wraps since it wouldn't be nice due to the possibility of discovering CVEs later, and actually no EFL provider would have the wraps enabled (just users who want to ease their work, actively - and not silently -, then knowing that they're doing it).
- Not all meson projects can be used as subprojects, some require external dependencies, in case some code compiler is compiled, cross compilation will fail.
We'll be testing this until trying to merge code that adds a wrap. But apparently you're right, considering our current issue with dftables for PCRE for some reason.
- Sometimes wraps do disappear (example stb, there is still the entry in the wrapdb, but the wrap in gone) in case we ever rely on wrap, we also need to take a share on maintaining the wraps, or we might loose them
About maintaining them: my answer to vtorri. About they disappearing: indeed, it could be a problem and I wouldn't like to be in this "come and go" of creating patches adding/removing a certain wrap.
- Subprojects bring the harm of 2 providers, so in case zlib-wrap is now sucked in as a dependency instead of a libzlib.so on the system, then the subproject wlll be compiled, and with the next ninja install it will be installed. In case you use any other project doing the same thing, or the distro shipping zlib as standalone, you will end up in a ugly conflict...
I was really sad to find out that this was true, but later I also found out that actually wraps have install : false by default for that reason, so wrap subprojects aren't actually installed with ninja install (from what I understood talking in #mesonbuild IRC channel).
I dont want that this all sounds like a bad idea, wraps are fantastic for some things, they simply do not really fit here good IMO. What i think would serve here the purpose way better would be if we turn the whole thing simply arround. So if you really insist on using wraps, and dont want to have ewpi, you could create another repository, which uses subprojects directly like zlib etc. etc. you get the dependency project and add it as a dependency override. After that you can also suck in efl itself as a subproject, and efl will suck in the deps from the wraps, without having the downsides above on efl itself. This way efl stays clean for packagers, easy on release maintanance, and you can use the advantages of efl. How does that idea sound to you ?
This actually sound nice, and if we move out of wraps (or, in this case, not even move in) from EFL, it could be a possible solution if we find that EWPI for some reason don't fit our purpose - that could be a later discussion.
(Also, from the view of development, the efl tree would then simply life in a folder in the subproject subdir, which would simply allow development as there is the .git directoy as well)
Sure, sounds pretty clean to be honest.
i have to say that i also am not a fan of this. it's controversial at best. i see the value but it also quickly becomes a "can of worms" you open up and then the mess is everywhere... if we were to build dependencies e should do it like we do static deps - we explicitly suck in code and explicitly compile it as part of our tree with explicitly chosen options etc. so we'd ne more like chromium/blink and such codebases then... we should do this very selectively only when there is a really good reason. right now i don't see one here... :(
Hi bu5hm4n.
Hi :)
I am not really a fan of using subprojects here either, not because subprojects per se are bad, but they dont fit the scheme of efl and how distros work. Just from the top of my head a idea on why i dont think its a good idea:
- In case we are having subprojects, we are defining the version of a dependency, not the min version of anything,
True and I hadn't realize this before. If wrap-git supported meson patches as well, this could be way more reliable (even if it wouldn't solve the problem of "updates in source upstream breaking wraps").
the version itself is then defined, in case there is a CVE or some major bug, we need to proactivly reroll another release of efl bump versions etc.. Not sure we have the capacity in the release department for that CC @stefan_schmidt )
Now the CVE part got me and I have to totally agree. Didn't crossed my mind that a wrap (the way they are implemented now) would mean sticking to a version and then would mean sticking to a CVE until proper reroll.
To aim this issue, we wouldn't release EFL prebuilt binaries with wraps since it wouldn't be nice due to the possibility of discovering CVEs later, and actually no EFL provider would have the wraps enabled (just users who want to ease their work, actively - and not silently -, then knowing that they're doing it).
Okay, but if you say that a release would not use the subprojects, then everything gets overly complex, either you need a sh**load of if's or patch in in and out for every release. Looking at that, vs. having just a repo doing that out of tree, i would prefer the repo doing that magic out of tree, (and that repo simply never gets released)
- Not all meson projects can be used as subprojects, some require external dependencies, in case some code compiler is compiled, cross compilation will fail.
We'll be testing this until trying to merge code that adds a wrap. But apparently you're right, considering our current issue with dftables for PCRE for some reason.
If the repo is not the efl repo where all this is done, then there is no reason in making it work for cross compilation ... :)
- Sometimes wraps do disappear (example stb, there is still the entry in the wrapdb, but the wrap in gone) in case we ever rely on wrap, we also need to take a share on maintaining the wraps, or we might loose them
About maintaining them: my answer to vtorri. About they disappearing: indeed, it could be a problem and I wouldn't like to be in this "come and go" of creating patches adding/removing a certain wrap.
- Subprojects bring the harm of 2 providers, so in case zlib-wrap is now sucked in as a dependency instead of a libzlib.so on the system, then the subproject wlll be compiled, and with the next ninja install it will be installed. In case you use any other project doing the same thing, or the distro shipping zlib as standalone, you will end up in a ugly conflict...
I was really sad to find out that this was true, but later I also found out that actually wraps have install : false by default for that reason, so wrap subprojects aren't actually installed with ninja install (from what I understood talking in #mesonbuild IRC channel).
Mhm, while that is true, zlib is a .so, not installing it will result in efl not beeing able to be run ... that is not a solution ... :)
I dont want that this all sounds like a bad idea, wraps are fantastic for some things, they simply do not really fit here good IMO. What i think would serve here the purpose way better would be if we turn the whole thing simply arround. So if you really insist on using wraps, and dont want to have ewpi, you could create another repository, which uses subprojects directly like zlib etc. etc. you get the dependency project and add it as a dependency override. After that you can also suck in efl itself as a subproject, and efl will suck in the deps from the wraps, without having the downsides above on efl itself. This way efl stays clean for packagers, easy on release maintanance, and you can use the advantages of efl. How does that idea sound to you ?
This actually sound nice, and if we move out of wraps (or, in this case, not even move in) from EFL, it could be a possible solution if we find that EWPI for some reason don't fit our purpose - that could be a later discussion.
So you go that path ?
I don't want to install an application to be able to compile or download dependencies. This makes no sense at all IMO. If we're not going to have wrap dependencies, we still need to be able to tell where dependencies are without pkg-config. We have neither, and wrap dependencies is a meson feature which, IMO, helps a lot. Lots of other libraries use it (gstreamer for example). And this doesn't need to be on by default. I wouldn't need subprojects in Linux (though I've used subprojects for gstreamer when developing for it, and was much easier when using git versions).
BTW, subprojects are used for development in-tree. I think that may be misunderstanding here. People are thinking about releases? This has nothing to do with doing proper releases. This helps EFL developers, 6 developers are using right now with great success.
Just that i have written that somewhere: gstreamer itself does not use subprojects. They use subprojects in gst-build, which also uses gstreamer as a subproject. Dev repos itself however do not. And that setup is 1:1 the thing i proposed above.