Page MenuHomePhabricator

edje_cc: make builds reproducable
Closed, ResolvedPublic

Description

Various distro people would like to build packages in a reproducible format currently edje_cc doesn't do that so this is just a tracker task for this effort should someone want to work on it.

simotek created this task.Jan 17 2017, 6:36 PM

Could you highlight a bit more what issues come up for reproducable builds?
I would suspect generated versions, timestamps and other parts that are generated from our build system during a build. That is just my guessing though and having some actual facts might help here. Also, is it only edje_cc that makes trouble?

Seems to just be edje_cc, I think we have ways of handing timestamps etc if they exist. The enlightenment package is fine as it doesn't include .edj files, but every package with one is affected.

https://www.zq1.de/~bernhard/linux/reproducibleopensuse/compare.factory-20170110/efl-compare.out

I brought this up with zmike, Cedric and JP and they pointed me in the right direction.

Looking at the hexdumps you can see that the one of the differences comes form the name of the edje programs inside the edj file. What happens is that we use the memory address of the edje_program within the generated name. That obviously changes has a big chance of being different during the next build.
src/bin/edje/edje_cc_handlers.c:14092
sprintf(def_name, "program_%p", ep);

We should be able to change this to use some incrementing counter to avoid having this random address leaking into the builds.

@simotek I coded up a patch which avoids the memory address for the program name and used a easier to reproduce counter for uniqueness.
https://git.enlightenment.org/core/efl.git/commit/?h=devs/stefan/reproducible-edj-T5113&id=8800eb0ba29fe56855e3d521387635fc3aebd515

I would very much appreciate if you could test the patch from the branch above. Its not yet in master as I wanted to get some feedback first.

There is one bug in your patch: you allocate space for sizeof(unsigned int) which is 4 or 8, but then you do sprintf %i which is a decimal ASCII representation of an int, so needs 10 or 20 bytes to represent numbers like 4294967295

apart from that, the patch helps to make .edj files build reproducibly.
For the patch, this should do the trick:

+   def_name = alloca(strlen("program_") + sizeof("FFFFFFFFFFFFFFFF") + 1);
+   sprintf(def_name, "program_%X", pc->programs.total_count);

Did you mean strlen("FFFFFFFFFFFFFFFF") instead of sizeof here?

Thanks a lot for testing. I will fix this up and get it pushed to the tree.

More detailed testing showed that more than half of openSUSE packages with .edj files now build reproducibly with the patch:
elemines
eperiodique
ephoto
epymc
rage
terminology
terminology-theme-openSUSE

but some issues remain in efl and enlightenment-theme-* packages
http://rb.zq1.de/compare.factory-20170807/efl-compare.out
http://rb.zq1.de/compare.factory-20170807/enlightenment-theme-cerium2-compare.out
http://rb.zq1.de/compare.factory-20170807/enlightenment-theme-openSUSE-compare.out
http://rb.zq1.de/compare.factory-20170807/enlightenment-theme-openSUSE-ice-compare.out

some of that seems to be randomly ordered lists (extract from the last link):

 000405f0  6a 65 2f 73 63 72 69 70  74 73 2f 65 6d 62 72 79  |je/scripts/embry|
-00040600  6f 2f 73 6f 75 72 63 65  2f 34 32 36 2f 31 36 00  |o/source/426/16.|
+00040600  6f 2f 73 6f 75 72 63 65  2f 34 31 31 2f 32 38 00  |o/source/411/28.|
 00040610  65 64 6a 65 2f 73 63 72  69 70 74 73 2f 65 6d 62  |edje/scripts/emb|

@bmwiedemann i'm in the process of creating new efl updates this week, i'll backport the above patch as part of it as there won't be another feature release for 3 months