Page MenuHomePhabricator

edi: large text file memory consumption
Closed, ResolvedPublic

Description

Hi, so am using an ASCII KJV bible and have watched it seemingly load the file (interupted earlier watching the text load [in memory not in the UI]).

Seems forever stuck here (eating all the RAM) and nothing bogus about this with valgrind: here is the backtrace:

Wait a minute and then interrupt - same backtrace.

(gdb) bt
#0  eina_list_data_get (list=0x81bdff640) at eina_inline_list.x:46
#1  0x00000008046bb5aa in evas_object_box_children_get (o=0x80000062cc337f1a)
    at evas_object_box.c:1922
#2  0x0000000800d30a6a in _focus_order_flush (obj=0x80000062c4337f19,
    pd=0x80ec55bb0) at elm_box.c:29
#3  0x0000000800d334f0 in _elm_box_pack_end (obj=0x80000062c4337f19,
    pd=0x80ec55bb0, subobj=0x800000c05c338b56) at elm_box.c:490
#4  0x0000000800d31b07 in elm_obj_box_pack_end (obj=0x80000062c4337f19,
    subobj=0x800000c05c338b56) at elm_box.eo.c:44
#5  0x0000000800d329dd in elm_box_pack_end (obj=0x80000062c4337f19,
    subobj=0x800000c05c338b56) at elm_box.eo.c:218
#6  0x0000000800d5bc31 in _elm_code_widget_ensure_n_grid_rows (
    widget=0x8000005d2c337ed1, rows=31102) at elm_code_widget.c:1853
#7  0x0000000800d5b87c in _elm_code_widget_resize (widget=0x8000005d2c337ed1,
    newline=0x0) at elm_code_widget.c:1900
#8  0x0000000800d5eed3 in _elm_code_widget_fill_update (
    widget=0x8000005d2c337ed1, first_row=1, last_row=31102, newline=0x0)
    at elm_code_widget.c:423
#9  0x0000000800d5eea5 in _elm_code_widget_fill (widget=0x8000005d2c337ed1)
    at elm_code_widget.c:472
#10 0x0000000800d5fdb1 in _elm_code_widget_file_cb (data=0x8000005d2c337ed1,
    event=0x7fffffffde90) at elm_code_widget.c:494
#11 0x0000000808afdcd9 in _event_callback_call (obj_id=0x8000005d2c337ed1,
    pd=0x81646d740, desc=0x801256ce0, event_info=0x815a83e20,
    legacy_compare=1 '\001') at eo_base_class.c:1496
#12 0x0000000808afb6c3 in _efl_object_event_callback_legacy_call (
    obj_id=0x8000005d2c337ed1, pd=0x81646d740, desc=0x801256ce0,
    event_info=0x815a83e20) at eo_base_class.c:1569
#13 0x0000000808af641b in efl_event_callback_legacy_call (
    obj=0x8000005d2c337ed1, desc=0x801256ce0, event_info=0x815a83e20)
    at eo_base_class.c:1572
#14 0x00000008046a7823 in _efl_canvas_object_efl_object_event_callback_legacy_call () at evas_object_main.c:1016
#15 0x0000000808af641b in efl_event_callback_legacy_call (
    obj=0x8000005d2c337ed1, desc=0x801256ce0, event_info=0x815a83e20)
    at eo_base_class.c:1572
#16 0x0000000800d6108e in elm_code_callback_fire (code=0x815b29500,
    signal=0x801256ce0, data=0x815a83e20) at elm_code.c:61
#17 0x0000000800d51222 in elm_code_file_open (code=0x815b29500,
    path=0x8162461fc "/usr/home/netstar/code/webcz/bible.txt")
    at elm_code_file.c:160
#18 0x000000000040f56b in edi_editor_add (parent=0x80000059dc337eb0,
    item=0x815826300) at editor/edi_editor.c:1140
#19 0x000000000041fe96 in _content_load (item=0x815826300)
    at mainview/edi_mainview_panel.c:191
#20 0x000000000041f9b9 in edi_mainview_panel_item_select (panel=0x8160d2300,
    item=0x815826300) at mainview/edi_mainview_panel.c:244
#21 0x0000000000420b8d in _edi_mainview_panel_item_tab_add (panel=0x8160d2300,
    options=0x815b6f3e0, mime=0x800753c5a "text/plain")
    at mainview/edi_mainview_panel.c:341
#22 0x00000000004209a5 in _edi_mainview_panel_tab_stat_done (data=0x815b6f3e0,
    handler=0x815ac2860, stat=0x815ac28b0) at mainview/edi_mainview_panel.c:649
#23 0x00000008063fcee8 in _eio_file_stat_done (data=0x815ac2860,
    thread=0x815ec4060) at eio_single.c:179
#24 0x000000080814504f in _ecore_thread_kill (work=0x815ec4060)
    at ecore_thread.c:229
#25 0x0000000808144fd8 in _ecore_thread_handler (data=0x815ec4060)
    at ecore_thread.c:256
#26 0x0000000808110e7e in _ecore_main_call_flush () at ecore.c:1033
#27 0x000000080810f6b8 in _thread_callback (data=0x0, buffer=0x81580d5e0,
    nbyte=4) at ecore.c:1044
#28 0x000000080813e55a in _ecore_pipe_handler_call (p=0x80be75050,
    buf=0x81580d5e0 "*", len=4) at ecore_pipe.c:646
#29 0x000000080813d83f in _ecore_pipe_read (data=0x80be75050,
    fd_handler=0x80be76050) at ecore_pipe.c:772
#30 0x0000000808120cc2 in _ecore_call_fd_cb (
    func=0x80813d500 <_ecore_pipe_read>, data=0x80be75050,
    fd_handler=0x80be76050) at ecore_private.h:341
#31 0x00000008081201d8 in _ecore_main_fd_handlers_call () at ecore_main.c:2005
#32 0x000000080811ca62 in _ecore_main_loop_iterate_internal (once_only=0)
    at ecore_main.c:2392
#33 0x000000080811cbe1 in ecore_main_loop_begin () at ecore_main.c:1299
#34 0x0000000800e877b9 in elm_run () at elm_main.c:1337
#35 0x0000000000425654 in elm_main (argc=1, argv=0x7fffffffe7a0)
    at edi_main.c:1614
#36 0x00000000004256cf in main (argc=1, argv=0x7fffffffe7a0) at edi_main.c:1627
(gdb)
netstar created this task.Jul 27 2017, 6:49 AM

I've found a few optimisations but it still takes a long time to do anything with a file this size.
I have not pushed a partial fix as I want to get to the bottom of where these inefficiencies happen so I can understand why it is so slow - this was designed to be a window on a file so long iteration of the content should not happen nearly as much as it currently does!

Sounds good! keep us posted!

ajwillia.ms edited projects, added Restricted Project; removed Restricted Project.Aug 17 2017, 8:12 AM
ajwillia.ms edited projects, added Restricted Project; removed Restricted Project.
ajwillia.ms edited projects, added Restricted Project; removed Restricted Project.
ajwillia.ms moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.Aug 17 2017, 8:18 AM

This is from opening the project with the 1M file already open when last closed.

Opening from the welcome panel (previously opened files)

#0  0x00007f8f40cc40a3 in _eina_list_mempool_list_new (list=0x2765c3c0) at lib/eina/eina_list.c:226
#1  0x00007f8f40cc4ef9 in eina_list_append (list=0x2765c3c0, data=0x800000ba48b9d18f) at lib/eina/eina_list.c:591
#2  0x00007f8f45749742 in evas_object_box_children_get (o=0x8000007ce0b9c931) at lib/evas/canvas/evas_object_box.c:1923
#3  0x00007f8f48e9e7d0 in _focus_order_flush (obj=0x8000007cd8b9c930, pd=0x1402dc0) at lib/elementary/elm_box.c:29
#4  0x00007f8f48ea02a9 in _elm_box_pack_end (obj=0x8000007cd8b9c930, pd=0x1402dc0, subobj=0x80000115b8b9dcf7)
    at lib/elementary/elm_box.c:488
#5  0x00007f8f48ea1f4d in elm_obj_box_pack_end (obj=0x8000007cd8b9c930, subobj=0x80000115b8b9dcf7) at lib/elementary/elm_box.eo.c:44
#6  0x00007f8f48ea2ba7 in elm_box_pack_end (obj=0x8000007cd8b9c930, subobj=0x80000115b8b9dcf7) at lib/elementary/elm_box.eo.c:218
#7  0x00007f8f48ec425d in _elm_code_widget_ensure_n_grid_rows (widget=0x8000007a98b9c8e8, rows=14330)
    at lib/elementary/elm_code_widget.c:1883
#8  0x00007f8f48ec450d in _elm_code_widget_resize (widget=0x8000007a98b9c8e8, newline=0x0) at lib/elementary/elm_code_widget.c:1930
#9  0x00007f8f48ec06fc in _elm_code_widget_fill_update (widget=0x8000007a98b9c8e8, first_row=1, last_row=0, newline=0x0)
    at lib/elementary/elm_code_widget.c:423
#10 0x00007f8f48ec0913 in _elm_code_widget_fill (widget=0x8000007a98b9c8e8) at lib/elementary/elm_code_widget.c:476
#11 0x00007f8f48ec0974 in _elm_code_widget_file_cb (data=0x8000007a98b9c8e8, event=0x7fffffffd0e0)
    at lib/elementary/elm_code_widget.c:498
#12 0x00007f8f40f47fda in _event_callback_call (obj_id=0x8000007a98b9c8e8, pd=0x13ed980,
    desc=0x7f8f4938e0b0 <ELM_CODE_EVENT_FILE_LOAD_DONE>, event_info=0x13ed900, legacy_compare=1 '\001') at lib/eo/eo_base_class.c:1502
#13 0x00007f8f40f48362 in _efl_object_event_callback_legacy_call (obj_id=0x8000007a98b9c8e8, pd=0x13ed980,
    desc=0x7f8f4938e0b0 <ELM_CODE_EVENT_FILE_LOAD_DONE>, event_info=0x13ed900) at lib/eo/eo_base_class.c:1575
#14 0x00007f8f40f4842a in efl_event_callback_legacy_call (obj=0x8000007a98b9c8e8, desc=0x7f8f4938e0b0 <ELM_CODE_EVENT_FILE_LOAD_DONE>,
    event_info=0x13ed900) at lib/eo/eo_base_class.c:1578
#15 0x00007f8f4572b4b1 in _efl_canvas_object_efl_object_event_callback_legacy_call (eo_obj=0x8000007a98b9c8e8, obj=0x13ed9c0,
    desc=0x7f8f4938e0b0 <ELM_CODE_EVENT_FILE_LOAD_DONE>, event_info=0x13ed900) at lib/evas/canvas/evas_object_main.c:1045
#16 0x00007f8f40f4842a in efl_event_callback_legacy_call (obj=0x8000007a98b9c8e8, desc=0x7f8f4938e0b0 <ELM_CODE_EVENT_FILE_LOAD_DONE>,
    event_info=0x13ed900) at lib/eo/eo_base_class.c:1578
#17 0x00007f8f48ecad6b in elm_code_callback_fire (code=0x13ed8d0, signal=0x7f8f4938e0b0 <ELM_CODE_EVENT_FILE_LOAD_DONE>,
    data=0x13ed900) at lib/elementary/elm_code.c:61
#18 0x00007f8f48ebc82e in elm_code_file_open (code=0x13ed8d0, path=0x13df8c4 "/home/netstar/code/scriptune/kjv.book")
    at lib/elementary/elm_code_file.c:160
#19 0x000000000041108f in edi_editor_add (parent=0x8000007990b9c8c7, item=0x13df3b0) at editor/edi_editor.c:1187
#20 0x000000000041fdfd in _content_load (item=0x13df3b0) at mainview/edi_mainview_panel.c:194
#21 0x00000000004200f7 in edi_mainview_panel_item_select (panel=0x1026450, item=0x13df3b0) at mainview/edi_mainview_panel.c:248
#22 0x000000000041fc4b in edi_mainview_panel_tab_select (panel=0x1026450, id=1) at mainview/edi_mainview_panel.c:159
#23 0x0000000000426e03 in _edi_open_tabs () at edi_main.c:1347
#24 0x0000000000427474 in edi_open (inputpath=0x7442ec "/home/netstar/code/scriptune") at edi_main.c:1438
#25 0x00000000004122c2 in _edi_welcome_project_open (path=0x7442ec "/home/netstar/code/scriptune", _edi_creating=0)
    at screens/edi_welcome.c:90
#26 0x0000000000413879 in _recent_project_mouse_down (data=0x7442ec, evas=0x8000000208b9b878, obj=0x8000000840b9b93f,
    event_info=0x8bae60) at screens/edi_welcome.c:523
#27 0x00007f8f456eab33 in _eo_evas_object_cb (data=0xacbe70, event=0x7fffffffd6a0) at lib/evas/canvas/evas_callbacks.c:197
#28 0x00007f8f40f47fda in _event_callback_call (obj_id=0x8000000840b9b93f, pd=0xb3f3d0, desc=0x7f8f45b6fd30 <_EFL_EVENT_POINTER_DOWN>,
    event_info=0x8000000238b9b87e, legacy_compare=1 '\001') at lib/eo/eo_base_class.c:1502
#29 0x00007f8f40f48362 in _efl_object_event_callback_legacy_call (obj_id=0x8000000840b9b93f, pd=0xb3f3d0,
    desc=0x7f8f45b6fd30 <_EFL_EVENT_POINTER_DOWN>, event_info=0x8000000238b9b87e) at lib/eo/eo_base_class.c:1575
#30 0x00007f8f40f4842a in efl_event_callback_legacy_call (obj=0x8000000840b9b93f, desc=0x7f8f45b6fd30 <_EFL_EVENT_POINTER_DOWN>,
    event_info=0x8000000238b9b87e) at lib/eo/eo_base_class.c:1578
#31 0x00007f8f4572b4b1 in _efl_canvas_object_efl_object_event_callback_legacy_call (eo_obj=0x8000000840b9b93f, obj=0xb3f410,
    desc=0x7f8f45b6fd30 <_EFL_EVENT_POINTER_DOWN>, event_info=0x8000000238b9b87e) at lib/evas/canvas/evas_object_main.c:1045
#32 0x00007f8f40f4842a in efl_event_callback_legacy_call (obj=0x8000000840b9b93f, desc=0x7f8f45b6fd30 <_EFL_EVENT_POINTER_DOWN>,
    event_info=0x8000000238b9b87e) at lib/eo/eo_base_class.c:1578
#33 0x00007f8f456eb53c in evas_object_event_callback_call (eo_obj=0x8000000840b9b93f, obj=0xb3f410, type=EVAS_CALLBACK_MOUSE_DOWN,
    event_info=0x8000000238b9b87e, event_id=10383, efl_event_desc=0x7f8f45b6fd30 <_EFL_EVENT_POINTER_DOWN>)
    at lib/evas/canvas/evas_callbacks.c:404
#34 0x00007f8f456eb727 in evas_object_event_callback_call (eo_obj=0x8000000c40b9b9bf, obj=0xbc7ec0, type=EVAS_CALLBACK_MOUSE_DOWN,
    event_info=0x8000000238b9b87e, event_id=10383, efl_event_desc=0x7f8f45b6fd30 <_EFL_EVENT_POINTER_DOWN>)
    at lib/evas/canvas/evas_callbacks.c:449
#35 0x00007f8f456f5f8e in _canvas_event_feed_mouse_down_internal (e=0x76ee30, ev=0x7d9700) at lib/evas/canvas/evas_events.c:1566
#36 0x00007f8f456ff6b4 in _evas_canvas_event_pointer_cb (data=0x76ee30, event=0x7fffffffdaa0) at lib/evas/canvas/evas_events.c:4057
#37 0x00007f8f40f47ef7 in _event_callback_call (obj_id=0x8000000208b9b878, pd=0x76edf0, desc=0x7f8f45b6fd30 <_EFL_EVENT_POINTER_DOWN>,
    event_info=0x8000000238b9b87e, legacy_compare=1 '\001') at lib/eo/eo_base_class.c:1479
#38 0x00007f8f40f48362 in _efl_object_event_callback_legacy_call (obj_id=0x8000000208b9b878, pd=0x76edf0,
    desc=0x7f8f45b6fd30 <_EFL_EVENT_POINTER_DOWN>, event_info=0x8000000238b9b87e) at lib/eo/eo_base_class.c:1575
#39 0x00007f8f40f4842a in efl_event_callback_legacy_call (obj=0x8000000208b9b878, desc=0x7f8f45b6fd30 <_EFL_EVENT_POINTER_DOWN>,
    event_info=0x8000000238b9b87e) at lib/eo/eo_base_class.c:1578
#40 0x00007f8f45fe7aa3 in _direct_mouse_updown (ee=0x7a1440, info=0x7a4d70, action=EFL_POINTER_ACTION_DOWN)
    at lib/ecore_evas/ecore_evas.c:4677
#41 0x00007f8f45fe7af6 in _direct_mouse_down_cb (ee=0x7a1440, info=0x7a4d70) at lib/ecore_evas/ecore_evas.c:4687
#42 0x00007f8f45fe86c2 in _ecore_evas_input_direct_cb (window=0x7a1440, type=77, info=0x7a4d70) at lib/ecore_evas/ecore_evas.c:4991
#43 0x00007f8f45dc8521 in _ecore_event_evas_mouse_button (e=0x7a4d70, press=ECORE_DOWN, faked=0 '\000')
    at lib/ecore_input_evas/ecore_input_evas.c:540
#44 0x00007f8f45dc8b84 in ecore_event_evas_mouse_button_down (data=0x0, type=77, event=0x7a4d70)
    at lib/ecore_input_evas/ecore_input_evas.c:655
#45 0x00007f8f418ece64 in _ecore_call_handler_cb (func=0x7f8f45dc8b5b <ecore_event_evas_mouse_button_down>, data=0x0, type=77,
    event=0x7a4d70) at lib/ecore/ecore_private.h:325
#46 0x00007f8f418ede3d in _ecore_event_call () at lib/ecore/ecore_events.c:518
#47 0x00007f8f418f7452 in _ecore_main_loop_iterate_internal (once_only=0) at lib/ecore/ecore_main.c:2398
---Type <return> to continue, or q <return> to quit---frame 7
#48 0x00007f8f418f4f7c in ecore_main_loop_begin () at lib/ecore/ecore_main.c:1299
#49 0x00007f8f48fcd297 in elm_run () at lib/elementary/elm_main.c:1334
#50 0x0000000000427920 in elm_main (argc=1, argv=0x7fffffffdf88) at edi_main.c:1614
#51 0x0000000000427998 in main (argc=1, argv=0x7fffffffdf88) at edi_main.c:1627
(gdb) frame 7
#7  0x00007f8f48ec425d in _elm_code_widget_ensure_n_grid_rows (widget=0x8000007a98b9c8e8, rows=14330)
    at lib/elementary/elm_code_widget.c:1883
1883            elm_box_pack_end(pd->gridbox, grid);
(gdb) sizoef
Undefined command: "sizoef".  Try "help".
(gdb) sizeof
Undefined command: "sizeof".  Try "help".
(gdb) size
Undefined command: "size".  Try "help".
(gdb) len
Undefined command: "len".  Try "help".
(gdb) print existing
$1 = 0
(gdb) print rows
$2 = 14330
(gdb)

Found the culprit:

#include <Evas.h>
#include <Elementary.h>

int main(void)
{
   int i;
   evas_init();
   elm_init(0, NULL);
   Evas *ev = evas_new();
   Evas_Object *canvas = evas_object_evas_get(ev);
   Evas_Object *box = elm_box_add(canvas);
   for (i = 0; i < 14330; i++) {
      Evas_Object *grid = evas_object_textgrid_add(canvas);
      elm_box_pack_end(box, grid);
   }
  sleep(100);
}

This is the cause of it:

from elm_code_widget.c

h = elm_code_file_lines_get(pd->code->file);

EINA_LIST_FOREACH(pd->code->file->lines, item, line)
  {
     line_width = elm_code_widget_line_text_column_width_get(widget, line);
     if ((int) line_width + gutter + 1 > w)
       w = (int) line_width + gutter + 1;
  }

_elm_code_widget_ensure_n_grid_rows(widget, h);
_elm_code_widget_cell_size_get(widget, &cw, &ch);

2 things.

  1. dont use 15000 textgrid objects. just dont use that many. it WILL eat memory.
  2. you did find a leak. well done! it's new focus manager code from @bu5hm4n.

static void
_focus_order_flush(Eo *obj, Elm_Box_Data *pd EINA_UNUSED)
{

Elm_Widget_Smart_Data *wpd = efl_data_scope_get(obj, ELM_WIDGET_CLASS);
Eina_List *order = evas_object_box_children_get(wpd->resize_obj);

efl_ui_focus_manager_calc_update_order(wpd->focus.manager, obj, order);

}

getting children returns an ALLOCATED list that is built specifically for the call.

As i just wrote on the mailinglist, the free there is prettymuch harmfull and wrong.

The leak shown above should be fixed with 63393d344b3203194ba28d77f2bc301384c7176c.

actually the api design is harmful. efl_ui_focus_manager_calc_update_order() if called on a wrong obj type will leak the list (as you expect the list to be consumed by the object and here it is not). i just reverted ... the free fixed the leak... but it created new issues with focus manager. now it seems lists are vanishing somewhere and not being freed...

They are not vanishing, they are kept internal, and reapplied once a new child appears, so the order is really kept, and not lost once a child is added or changed, so we dont have to spam update_order calls for keeping the order correctly while children are getting visible or invisible, which is happening often, and alot.

in fact both update_children and update_order are the same problem. they "take ownership" of the children. what if the method fails at runtime? who frees the children. look at _efl_ui_focus_manager_calc_update_children() - how do i know if this method successfully took ownership of the children. the return value doesn't tell me if it does. who frees them then? how do we know to free or not? what if eo itself fails to dispatch the method (NULL object?) before even this method content is called... up in eo core dispatch land. who frees the children then?

this design of the api is pretty problematic. the only way through this i see is to not take ownership and duplicate the list always.

No thats actually not a problem.

If a api is defined as own, then the caller passes the ownership to the called context, which is at first eo, eo then searches for the function and the object. If that fails then eo needs to free it, which it totally doable since eolian has the free function for each type (or at least it should if it does not have it, it should error). If eo gets teh pointer and function, then the ownership is continued to the implemented function. Then this function needs to free it. So all in all, no problem at all.
But there is currently a leak for the case that the node is invalid, have a patch here, will push later.

ajwillia.ms moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.Aug 31 2017, 1:57 AM
ApB added a comment.Sep 2 2017, 2:26 AM

FWIW ecrire -which uses elm code- still seems to have trouble opening files small and large.

Hey, yeah the issue was in EFL, so AFAIK for now the current fix is available in git master (EFL), but i'm sure it'll be backported into 1.20 when everyone is happy with the fix.

Also there should be some improvements done in later releases of EFL that will reduce the overhead even more. Not for now, concentrating on Edi 0.6 release and E22.

ApB added a comment.Sep 2 2017, 3:04 PM

Hey, yeah the issue was in EFL, so AFAIK for now the current fix is available in git master (EFL), but i'm sure it'll be backported into 1.20 when everyone is happy with the fix.

Also there should be some improvements done in later releases of EFL that will reduce the overhead even more. Not for now, concentrating on Edi 0.6 release and E22.

I tested on git.

I misread what you wrote. Both small and large files aren't loading in ecrire? I've never used it, I guess I could try. The ecrire from git.enlightenment.org???

ApB added a comment.Sep 2 2017, 4:34 PM

I misread what you wrote. Both small and large files aren't loading in ecrire? I've never used it, I guess I could try. The ecrire from git.enlightenment.org???

nope. the one from github. https://github.com/Obsidian-StudiosInc/ecrire

I don't know what the situation is there. Maybe take the issue up with the person who forked ecrire ?

If ecrire is having issues with files large and small its likely due to changes in EFL. It is possible its in other stuff, ecrire mime handing, setting document title, or other. But basically just does a elm_code_file_open.