Page MenuHomePhabricator

EFL memory usage - dirty pages from SO files
Open, NormalPublic

Description

Take a look at this (numbers in Kb - these are Private Dirty Pages of memory - i.e. pages written to):

                                               /home/raster/hello 4
                                                           [heap] 4120
                                                                - 4
                                                     [stack:1939] 8
                                                     [stack:1938] 8
                                      /usr/lib/libOSMesa.so.8.0.0 84
                                      /usr/lib/libOSMesa.so.8.0.0 16
                                                                - 12
                                                                - 4
                 /usr/local/lib/edje/modules/elm/v-1.18/module.so 28
/usr/local/lib/evas/modules/engines/software_x11/v-1.18/module.so 4
                                                                - 4
             /usr/local/lib/ecore_evas/engines/x/v-1.18/module.so 8
      /usr/local/lib/elementary/modules/web/none/v-1.18/module.so 4
         /usr/local/lib/elementary/modules/prefs/v-1.18/module.so 8
                                  /usr/lib/libibus-1.0.so.5.0.513 8
                                  /usr/lib/libibus-1.0.so.5.0.513 4
                                                                - 4
           /usr/local/lib/ecore_imf/modules/ibus/v-1.18/module.so 4
            /usr/local/lib/ecore_imf/modules/xim/v-1.18/module.so 4
                                    /usr/lib/libnss_files-2.23.so 4
                                    /usr/lib/libnss_files-2.23.so 4
                                        /usr/lib/libltdl.so.7.3.1 4
                                        /usr/lib/libltdl.so.7.3.1 4
                                    /usr/lib/libscim-1.0.so.8.2.6 12
                                    /usr/lib/libscim-1.0.so.8.2.6 64
           /usr/local/lib/ecore_imf/modules/scim/v-1.18/module.so 8
         /usr/local/lib/eeze/modules/sensor/fake/v-1.18/module.so 4
         /usr/local/lib/eeze/modules/sensor/udev/v-1.18/module.so 4
                                                     [stack:1937] 8
             /usr/local/lib/ecore/system/systemd/v-1.18/module.so 4
              /usr/local/lib/ecore/system/upower/v-1.18/module.so 4
                              /usr/lib/libgmodule-2.0.so.0.4600.2 4
                              /usr/lib/libgmodule-2.0.so.0.4600.2 4
                                  /usr/lib/libgio-2.0.so.0.4600.2 16
                                  /usr/lib/libgio-2.0.so.0.4600.2 8
                                                                - 4
                                          /usr/lib/libnsl-2.23.so 4
                                          /usr/lib/libnsl-2.23.so 4
                              /usr/lib/libgobject-2.0.so.0.4600.2 4
                              /usr/lib/libgobject-2.0.so.0.4600.2 4
                                   /usr/lib/libgudev-1.0.so.0.2.0 4
                                   /usr/lib/libgudev-1.0.so.0.2.0 4
                                     /usr/lib/libasyncns.so.0.3.1 4
                                     /usr/lib/libasyncns.so.0.3.1 4
                                       /usr/lib/libXdmcp.so.6.0.0 4
                                       /usr/lib/libXdmcp.so.6.0.0 4
                                         /usr/lib/libXau.so.6.0.0 4
                                         /usr/lib/libXau.so.6.0.0 4
                                  /usr/lib/libgpg-error.so.0.17.0 4
                                  /usr/lib/libgpg-error.so.0.17.0 4
                                     /usr/lib/libgcrypt.so.20.0.5 4
                                     /usr/lib/libgcrypt.so.20.0.5 20
                                         /usr/lib/liblz4.so.1.7.1 4
                                         /usr/lib/liblz4.so.1.7.1 4
                                        /usr/lib/liblzma.so.5.2.2 4
                                        /usr/lib/liblzma.so.5.2.2 4
                                        /usr/lib/libpcre.so.1.2.6 4
                                        /usr/lib/libpcre.so.1.2.6 4
                                         /usr/lib/libbz2.so.1.0.6 8
                                   /usr/lib/libgraphite2.so.3.0.1 8
                                   /usr/lib/libgraphite2.so.3.0.1 4
                                       /usr/lib/libresolv-2.23.so 4
                                       /usr/lib/libresolv-2.23.so 4
                                          /usr/lib/libcap.so.2.25 4
                                        /usr/lib/libuuid.so.1.3.0 4
                                        /usr/lib/libuuid.so.1.3.0 4
                                       /usr/lib/libblkid.so.1.1.0 16
                                       /usr/lib/libblkid.so.1.1.0 4
                                       /usr/lib/libwacom.so.2.4.5 4
                                       /usr/lib/libwacom.so.2.4.5 4
                                      /usr/lib/libevdev.so.2.1.12 24
                                      /usr/lib/libevdev.so.2.1.12 4
                                       /usr/lib/libmtdev.so.1.0.0 4
                                       /usr/lib/libmtdev.so.1.0.0 4
                                         /usr/lib/libffi.so.6.0.4 4
                                         /usr/lib/libffi.so.6.0.4 4
                                       /usr/lib/libexpat.so.1.6.0 12
                                       /usr/lib/libexpat.so.1.6.0 4
                                      /usr/lib/libjson-c.so.2.0.1 4
                                      /usr/lib/libjson-c.so.2.0.1 4
                        /usr/lib/pulseaudio/libpulsecommon-8.0.so 4
                        /usr/lib/pulseaudio/libpulsecommon-8.0.so 8
                                  /usr/lib/libvorbisenc.so.2.0.11 136
                                  /usr/lib/libvorbisenc.so.2.0.11 4
                                      /usr/lib/libvorbis.so.0.4.8 4
                                      /usr/lib/libvorbis.so.0.4.8 4
                                         /usr/lib/libogg.so.0.8.2 4
                                         /usr/lib/libogg.so.0.8.2 4
                                        /usr/lib/libFLAC.so.8.3.0 4
                                        /usr/lib/libFLAC.so.8.3.0 4
                                         /usr/lib/libxcb.so.1.1.0 4
                                         /usr/lib/libxcb.so.1.1.0 4
                                           /usr/lib/libgcc_s.so.1 4
                                     /usr/lib/libstdc++.so.6.0.21 40
                                     /usr/lib/libstdc++.so.6.0.21 8
                                                                - 16
                                      /usr/lib/libunwind.so.8.0.1 4
                                      /usr/lib/libunwind.so.8.0.1 4
                               /usr/lib/libunwind-x86_64.so.8.0.1 4
                               /usr/lib/libunwind-x86_64.so.8.0.1 4
                                        /usr/lib/libjpeg.so.8.0.2 4
                                        /usr/lib/libjpeg.so.8.0.2 4
                                           /usr/lib/libz.so.1.2.8 4
                                           /usr/lib/libz.so.1.2.8 4
                                      /usr/lib/libcrypto.so.1.0.0 112
                                      /usr/lib/libcrypto.so.1.0.0 48
                                                                - 4
                                         /usr/lib/libssl.so.1.0.0 16
                                         /usr/lib/libssl.so.1.0.0 28
                                /usr/local/lib/libemile.so.1.18.0 4
                                     /usr/lib/libpng16.so.16.21.0 4
                                     /usr/lib/libpng16.so.16.21.0 4
                                /usr/local/lib/libector.so.1.18.0 12
                                                                - 4
                                 /usr/lib/libglib-2.0.so.0.4600.2 4
                                 /usr/lib/libglib-2.0.so.0.4600.2 4
                                                                - 4
                              /usr/lib/libgthread-2.0.so.0.4600.2 4
                              /usr/lib/libgthread-2.0.so.0.4600.2 4
                                  /usr/lib/libluajit-5.1.so.2.0.4 8
                                  /usr/lib/libluajit-5.1.so.2.0.4 4
                                   /usr/lib/libfreetype.so.6.12.3 24
                                   /usr/lib/libfreetype.so.6.12.3 4
                                  /usr/lib/libfontconfig.so.1.8.0 8
                                  /usr/lib/libfontconfig.so.1.8.0 4
                                     /usr/lib/libfribidi.so.0.3.6 4
                                     /usr/lib/libfribidi.so.0.3.6 4
                                /usr/lib/libharfbuzz.so.0.10200.6 4
                                /usr/lib/libharfbuzz.so.0.10200.6 4
                                       /usr/lib/libmount.so.1.1.0 4
                                       /usr/lib/libmount.so.1.1.0 4
                                 /usr/local/lib/libeeze.so.1.18.0 4
                     /usr/local/lib/libecore_input_evas.so.1.18.0 4
                                     /usr/lib/libdbus-1.so.3.14.6 4
                                     /usr/lib/libdbus-1.so.3.14.6 4
                                                                - 4
                                   /usr/lib/libxkbcommon.so.0.0.0 8
                                   /usr/lib/libxkbcommon.so.0.0.0 4
                                      /usr/lib/libinput.so.10.7.3 4
                                      /usr/lib/libinput.so.10.7.3 4
                                /usr/local/lib/libelput.so.1.18.0 4
                                         /usr/lib/libdrm.so.2.4.0 4
                                         /usr/lib/libdrm.so.2.4.0 4
                                       /usr/lib/libglapi.so.0.0.0 16
                                       /usr/lib/libglapi.so.0.0.0 4
                              /usr/lib/libwayland-server.so.0.1.0 8
                              /usr/lib/libwayland-server.so.0.1.0 4
                              /usr/lib/libwayland-client.so.0.3.0 8
                              /usr/lib/libwayland-client.so.0.3.0 4
                                         /usr/lib/libgbm.so.1.0.0 4
                                         /usr/lib/libgbm.so.1.0.0 4
                           /usr/local/lib/libecore_drm2.so.1.18.0 4
                       /usr/local/lib/libecore_imf_evas.so.1.18.0 4
                               /usr/local/lib/libembryo.so.1.18.0 4
                            /usr/local/lib/libecore_ipc.so.1.18.0 4
                                   /usr/lib/libLinearMath.so.2.83 4
                                   /usr/lib/libLinearMath.so.2.83 8
                              /usr/lib/libBulletCollision.so.2.83 24
                              /usr/lib/libBulletCollision.so.2.83 4
                                                                - 4
                               /usr/lib/libBulletDynamics.so.2.83 8
                               /usr/lib/libBulletDynamics.so.2.83 4
                                                                - 4
                               /usr/lib/libBulletSoftBody.so.2.83 8
                               /usr/lib/libBulletSoftBody.so.2.83 4
                             /usr/local/lib/libephysics.so.1.18.0 8
                                      /usr/lib/libpulse.so.0.19.0 4
                                      /usr/lib/libpulse.so.0.19.0 4
                                    /usr/lib/libsndfile.so.1.0.26 8
                                    /usr/lib/libsndfile.so.1.0.26 4
                          /usr/local/lib/libecore_audio.so.1.18.0 8
                               /usr/local/lib/libethumb.so.1.18.0 4
                            /usr/local/lib/libelocation.so.1.18.0 4
                                          /usr/lib/libXi.so.6.1.0 4
                                          /usr/lib/libXi.so.6.1.0 4
                                         /usr/lib/libXss.so.1.0.0 4
                                         /usr/lib/libXss.so.1.0.0 4
                                        /usr/lib/libXtst.so.6.1.0 4
                                        /usr/lib/libXtst.so.6.1.0 4
                                     /usr/lib/libXrender.so.1.3.0 4
                                     /usr/lib/libXrender.so.1.3.0 4
                                      /usr/lib/libXrandr.so.2.2.0 4
                                      /usr/lib/libXrandr.so.2.2.0 4
                                    /usr/lib/libXinerama.so.1.0.0 4
                                    /usr/lib/libXinerama.so.1.0.0 4
                                      /usr/lib/libXfixes.so.3.1.0 4
                                      /usr/lib/libXfixes.so.3.1.0 4
                                        /usr/lib/libXext.so.6.4.0 4
                                        /usr/lib/libXext.so.6.4.0 4
                                     /usr/lib/libXdamage.so.1.1.0 4
                                     /usr/lib/libXdamage.so.1.1.0 4
                                  /usr/lib/libXcomposite.so.1.0.0 4
                                  /usr/lib/libXcomposite.so.1.0.0 4
                                         /usr/lib/libX11.so.6.3.0 8
                                         /usr/lib/libX11.so.6.3.0 20
                                                                - 4
                                     /usr/lib/libXcursor.so.1.0.2 4
                                     /usr/lib/libXcursor.so.1.0.2 4
                              /usr/local/lib/libecore_x.so.1.18.0 12
                                                                - 4
                             /usr/local/lib/libecore_fb.so.1.18.0 12
                              /usr/lib/libwayland-cursor.so.0.0.0 4
                              /usr/lib/libwayland-cursor.so.0.0.0 8
                            /usr/local/lib/libecore_wl2.so.1.18.0 8
                                            /usr/lib/libc-2.23.so 16
                                            /usr/lib/libc-2.23.so 8
                                                                - 16
                                           /usr/lib/librt-2.23.so 4
                                           /usr/lib/librt-2.23.so 4
                                           /usr/lib/libdl-2.23.so 4
                                           /usr/lib/libdl-2.23.so 4
                                            /usr/lib/libm-2.23.so 4
                                            /usr/lib/libm-2.23.so 4
                                  /usr/local/lib/libeio.so.1.18.0 8
                         /usr/local/lib/libefreet_trash.so.1.18.0 4
                          /usr/local/lib/libefreet_mime.so.1.18.0 4
                               /usr/local/lib/libefreet.so.1.18.0 4
                                                                - 4
                               /usr/local/lib/libeldbus.so.1.18.0 8
                            /usr/local/lib/libecore_con.so.1.18.0 8
                            /usr/local/lib/libecore_imf.so.1.18.0 4
                              /usr/local/lib/libemotion.so.1.18.0 8
                        /usr/local/lib/libethumb_client.so.1.18.0 4
                                   /usr/local/lib/libeo.so.1.18.0 4
                                                                - 8
                                 /usr/local/lib/libedje.so.1.18.0 24
                                                                - 4
                          /usr/local/lib/libecore_input.so.1.18.0 4
                           /usr/local/lib/libecore_file.so.1.18.0 4
                           /usr/local/lib/libecore_evas.so.1.18.0 8
                                /usr/local/lib/libecore.so.1.18.0 8
                                                                - 12
                                 /usr/local/lib/libevas.so.1.18.0 72
                                                                - 120
                                  /usr/local/lib/libeet.so.1.18.0 4
                                      /usr/lib/libpthread-2.23.so 4
                                      /usr/lib/libpthread-2.23.so 4
                                                                - 4
                                 /usr/local/lib/libeina.so.1.18.0 8
                                                                - 8
                                  /usr/local/lib/libefl.so.1.18.0 20
                                                                - 8
                           /usr/local/lib/libelementary.so.1.18.0 184
                                                                - 40
                                                                - 12
                                                                - 120
                                    /usr/lib/libsystemd.so.0.14.0 12
                                    /usr/lib/libsystemd.so.0.14.0 4
                                                                - 20
                                        /usr/lib/libudev.so.1.6.4 4
                                        /usr/lib/libudev.so.1.6.4 4
                                                                - 88
                                                                - 96
                                              /usr/lib/ld-2.23.so 4
                                              /usr/lib/ld-2.23.so 4
                                                                - 4
                                                          [stack] 80

Total All Libraries: 2012Kb
Total EFL libraries: 576Kb

Stack isn't bad - a few more stacks for threads, etc. .. heap is 4M - could be better considering it's the hello world example from our docs (a tiny window with a button in it):

https://www.enlightenment.org/docs/efl/start

But heap we can easily dissect with massif. Some other empty mappings (-) i think are mmap anonymous memory e.g. for eo object tables or something else. But I want to focus here on mappings from shared libraries or modules. Some of these are huge. 184Kb from elementary. That almost definitely is eo classes as globals somehow being written to. That's bad. osmesa is eating up like 100Kb. and it's not being actually used there. libvorbisenc... 140Kb! libcrypto - 160Kb! i can go on. you get the drift. so how do i capture this data. a very dirty:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>

int main(int argc, char **argv)
{
   int pid = atoi(argv[1]);
   char buf[4000];
   char file[4000];
   char key[4000];
   char val[4000];
   snprintf(buf, sizeof(buf), "/proc/%i/smaps", pid);
   FILE *f = fopen(buf, "r");
   int entry = 0;
   int total = 0;
   int total2 = 0;

   while (fgets(buf, sizeof(buf) - 1, f))
     {
        int len = strlen(buf);
        if ((len > 0) && (buf[len - 1] == '\n')) buf[len - 1] = 0;
        if (entry == 0)
          {
             file[0] = 0;
             sscanf(buf, "%*s %*s %*s %*s %*s %s", file);
             entry++;
          }
        else
          {
             key[0] = 0;
             sscanf(buf, "%s %s", key, val);
             if (!strcmp(key, "Private_Dirty:"))
               {
                  if (!file[0])
                    {
                       file[0] = '-';
                       file[1] = 0;
                    }
                  int vali = atoi(val);
                  if (vali > 0) printf("%70s %s\n", file, val);
                  if (strstr(file, "/lib/")) total += vali;
                  if (strstr(file, "/local/lib/")) total2 += vali;
               }
             else if (!strcmp(key, "VmFlags:")) entry = -1;
             entry++;
          }
     }
   fclose(f);
   printf("==================== %i | %i\n", total, total2);
   return 0;
}

just run it like:

gcc dirty.c -o dirty && ./dirty PID

where PID is the process id to inspect.

We really need to focus on this because right now 1/3rd of the base mem usage of a process is this stuff. it's pretty damned bad. Many of the bigger ones we can solve by dlopen()ing the libraries on an as-needed basis. if not used, don't open them. have efl act as the shim in between. ecore_x could reduce a lot of the libX* libraries by only opening if used. also merging efl libs will take several 4k pages and merge into single ones. eo classes i think need a serious look at to find out what fields are written to and organize them to keep these fields in a single block of memory separate from the rest of the "constant" class data. There is probably more to hunt too.

So things needing doing:

  • dlopen() dependencies runtime we need when we need them and otherwise don't link or load
    • vorbis libs
    • bullet libs
    • osmesa
    • ssl
    • pulse
    • evdev
    • ecore_x, ecore_fb, ecore_drm2, ecore_wl2 etc. that elm only needs when displaying in that display system...
  • look into eo classes and mem layout and "redesign" to keep writes localized in a single memory area
  • merge efl shared libs
  • don't load our own modules until needed - not even as a probe
    • ecore system modules are bad this way
    • ecore_imf loads all of them too and then probes the one to use
  • go look into more global variables we have that may be written to and refactor/clean or remove
    • e.g. take our init counters and just move them to init libefl.so only so evas_init calls efl_init() - for small libs with little or no globals we should do this.
    • e.g. collect globals together and assign some slot in libefl to store a void * and every efl lib uses a constant pre-defined slot number and all globals for that lib go into a single malloced struct and we store the ptr to this in this slot array thing... and idea?
  • more digging to see - maybe some of these are symbol tables so merging libs may help

We need to trim down our memory footprint and these are obvious pieces of bloat that do not turn up in massif. We need to pay attention to them because they do not come for free. They do cost and are often pretty much invisible to normal profiling tools. I have been thinking about even how to make such a tool to help us fix this.

raster added a comment.Aug 1 2016, 3:12 AM

more fixes:

1d0b500fa8ce93ac6cd9211abd35ea8972be74b2

non-efl: 1780Kb -> 1712Kb

efl: 576Kb -> 560Kb

raster added a comment.Aug 1 2016, 5:14 AM

37735d8b4fef461d669975bf66d031b6dd5cf184

non-efl: 1712Kb -> 1692Kb

efl: 560Kb -> 552Kb

tasn added a comment.Aug 16 2016, 8:41 AM

I just fixed heap usage a bit, but I wonder what this is and where it's coming from.
I still haven't dug deep, been dealing with other more urgent matters. Will hopefully manage to find the issue soon.

i also just improve heap usage - but in edje. about 300k saved. anyway this specifically here is stuff not in heap - not stuff that massif or regular memory debuggers find, but it is real memory usage. :( probably you need to find addresses of eo data structs (classes, etc.) - anything that's a global var, and match those up with the memory address ranges from pmap and the above to see if they are inside those regions. :)

tasn added a comment.Aug 17 2016, 1:26 AM

Oh, I didn't see you said it was private pages (though now I remember we actually talked about it a while back). Hm... OK, I will have to check. Annoying that there's no compiler option/tool to easily see what is mapped to private pages. Maybe I will have to write one. :|

my above dirty.c is a start on getting kernel info on what is dirty. it'd need some more parsing to get address range then yes - need to find addresses of tonnes of global vars and match them up.

maybe there's info in gdb debug DWARF info. seriously. i looked for tools and they are thin on the ground for this. i seriously think almost no one looks or cares about this stuff.

tasn added a comment.Aug 17 2016, 3:10 AM

I looked into it a bit. I managed to make some progress, but not a lot.
I managed to reduce a page for each library by removing some static variables in class creation. Just because it's only a page (and to be honest, the amount of memory these variables should take) I don't think that's the culprit. However, after looking a bit more into it, I saw something very suspicious. According to pmap on libefl:

00007f7719323000 212K r-x-- libefl.so.1.18.99
00007f7719358000 2044K ----- libefl.so.1.18.99
00007f7719557000 20K rw--- libefl.so.1.18.99

As you can see, there are no read-only non-executable pages, which means that probably GCC doesn't map things correctly to ro and rw pages and maybe mixes const with non-const which invalidates pages.
The class_desc (static const) for class Efl_Config for example is at 0x7f77195577f0 which is well within the the rw pages of libefl.so. So it seems that indeed, gcc is not mapping pages correctly. :((

More to come, I need to talk to some compiler people.

tasn added a comment.Aug 17 2016, 3:34 AM

OK, after giving it a second look, it seems that it just maps the global consts into the "rx" section, so if it's a global and constant page it would be there. I thought it would separate to non-executable pages, but I guess it's fine to put it in an executable page. So not the end of the world.

we can't have 20kb of written to global data. that sounds insane... we'd be lucky to write to 1 page of that memory... or that's what it SHOULD be...

tasn added a comment.Aug 17 2016, 4:44 AM

You misunderstood me, the "not the end of the world" was referring to putting RO data in executable pages. The memory issue is real, and I'm still looking into it. I solved a minor case. Working on the rest.

tasn added a comment.Aug 17 2016, 5:13 AM

So... More info.

Running this program:

#include <stdio.h>
#include <string.h>
#include <unistd.h>

typedef struct
{
   const void *invalider;
   const char data[4096 * 1000];
} Invalid;

static const char ro[4096 * 1000];
static const Invalid rw = { NULL, { 0 }};

int main()
{
   printf("%zd - %p %p\n", (size_t) getpid(), ro, &rw);
   scanf("\n");
   return 0;
}

I get with gcc:

0000000000400000   4004K r-x-- a.out
00000000009e8000      4K rw--- a.out
00000000009e9000   4000K rw---   [ anon ]

clang:

0000000000400000   8004K r-x-- a.out
0000000000dd0000      4K rw--- a.out

So in the most basic sense, GCC sucks. However, the issue you are describing in this ticket is also happening to me with clang, so there's something else at play. I thought maybe clang couldn't map the pages to RO because LD needed to write the symbol addresses. Testing this with libc yielded no results, and neither using symbols from elementary (changing the NULL in the example to "elm_init" or "strlen").

After all of this though, I managed to crack it, I was missing one simple thing, I was missing the fact that our libraries (like all) use PIC. So changing the example from NULL to strlen and compiling with "PIC" caused the intended "break" that caused the mapping to be wrong. I guess GCC does some sort of indirection (or assumes one? Maybe because they assume ASLR? Dunno) so it already maps it in RW pages by default?

tl;dr: the problem is that because we use symbols in arrays and etc, the compiler has to map them to RW pages so the linker can (runtime) put the correct addresses in place. This means we need to redesign some of the structures and APIs there to fix this. I'm happy I know exactly what's wrong, though I'm sad because it's my responsibility to fix it. :( I have some ideas though, so everyone can be happy. :)

To be continued...

tasn added a comment.Sep 12 2016, 1:11 AM

There have been a few fixes, we are now at:
00007f661ecea000 116K r---- libelementary.so.1.18.99
00007f661ed07000 32K rw--- libelementary.so.1.18.99

Both are dirty.

I have a way to make it even smaller but that would mean ugly as hell API. Thinking about other solutions and doing something else in the meanwhile.

jpeg lowered the priority of this task from High to Normal.Jul 11 2017, 10:22 PM

This was worked on a lot at some point and then completely abandonned. Remains partly relevant but not high priority IMHO.

it is relevant. i think it's waiting on us to merge .so's just to go removing the number of mappings and so on... and then some detailed look-and-see

zmike edited projects, added Restricted Project; removed efl.Jun 11 2018, 6:54 AM
segfaultxavi edited projects, added efl; removed Restricted Project.Jun 11 2018, 8:58 AM
bu5hm4n edited projects, added Restricted Project; removed efl.Jun 11 2018, 10:48 AM