Page MenuHomePhabricator

Issues of ea_gen
Closed, ResolvedPublic

Description

We found some issues while doing a internal dev-day.
And I got some feedback.
I think some issues must be fixed for stable work.

  1. The problem of tree depth of generated ML

There is a problem with the generated ML tree.
This is mostly the case when ML is generated.

window
 - container
  - content
    ...
  - content
 - container
  - content
    ...
  - content

There is no case where the container is the chlid of another container.
db was built on all ea files.
The Elm.Box in the DB has many container widget in Containers.
However, a box using container will not be created.

This may be a frequency issue. But at least it did not happen in the tests so far.
Eventually, the more the DB, the more overlapping widgets will occur.
I think this must be fixed.

  1. Too many items

This is probably due to the implementation of test_genlist.
test_genlist creates a very large number of items for stress testing.
ml_gen does not necessarily follow this.
Sometimes, App gen generates a very large number of items and allocates a lot of memory.
I think it needed that the tool user can to adjust the frequency of item creation. or number of item's max occurence need fixed

  1. Meaningless scenario

If all the shots in the generated scenario are the same, then this scenario is not necessary.
Of course, even if there is no change between shots, the scn can be a test unit.
But I think this is the problem.

If you do not have enough time, we can develop this feature.
If all the shots generated are the same, they can be displayed in name.
ex)
Scenario #1 - same
Scenario #2 - diff

  1. Event play

For example, a button widget, Click event is created in button ML.
This clicked event is called with callback_call via exactness_play.
This is an internal operation only and can not check that the pressed and unpressed edc of the button is well-behaved.
The checkbox is also called "changed" but it does not change checkbox image in player.
Maybe most elementary widgets have this problem.

We need to let the player generate a hardware click event rather than a legacy call.

The commented that in exactness/src/bin/player.c _feed_event의 EXACTNESS_ACTION_EFL_EVENT needs to be reviewed.
(If the event is a problem when the widget overlaps, I think overlaping's a problem.)
What do you think?

There are some other feedbacks when I received dev-day.
Graphics API test support, lottie(vector) animation test support, recorder GUI support, etc.
I think we should review these feedback

Please let me know what your opinion about issues :)

jsuya created this task.Nov 23 2018, 1:50 AM
jsuya triaged this task as High priority.

Hi,

Thank you for the feedbacks. I didn't have too much time to work on the DB these last days. So now, I see some of the issues.

  1. I checked and I found e.g boxes inside boxes... I don't see that issue.
  2. This one is tricky. I am working on it right now. There are two issues related to the big number of items:
    • When selecting in ea_gen a big app to display the ML, the entry deals in something that seems to be an infinite loop. Seems related to the markups but not sure
    • exactness_play to run this big app takes a lot of CPU and memory until the PC totally stucks (no memory)

I need to find what makes it so hard for ea_gen.

  1. If you can handle this one, it will be with pleasure. However, I don't think you need to show that shots are different. You could for example give more tries to obtain a different one. If you always get the same, just inform the user or disable it with some tooltip
  2. Do you mean the events triggered by the player don't work?
  3. Other feedbacks:
    • Graphics API test support: showing Exactness results graphically?
    • Animations: can you give more details?
    • recorder GUI support: you mean a GUI to record an app? inside ea_gen or exactness?
jsuya added a comment.Nov 26 2018, 4:45 AM

Thank you for your reply :)

1.You are right, I finally found it. But it is very infrequently generated. :(
3.Ok. We will review with @myoungwoon
I will create a checkbox in "Generator" tab. If checkbox is true and scn_gen is called, check the finished shot and if all shots are the same, remove the scn and run scn_gen again.
4.It does not mean that it does not work.
We need to exactness_player generate a hardware click event rather than event call.

The ml created from the app clicking on the checkbox will have a "changed" event.
An app created from this ml call "changed" callback.
But it is only calling callback. It does not change its check appearance.
item's selected callback, button's click callbacks, etc... is also only calling callback.
The actual item is not selected and the button's press image can not see.

I think this is because the widget of the elementary is separate from the event and the edc's signal emit.
Perhaps it is hard to solve in current structure.

So while reviewing the code in player.c, I found a commented line from lines 495 to 520.
I think that at least some events can be resolved by generating a mouse click.

5.- graphics and animiations
Graphics-team asked if they could test their application using their API.
I think we can support an api preload.
Or if graphics-team provide specific applications, we can create an exu that only takes shots without app_gen.
vector animiation also the same.

-recorder GUI
Yes. Some users want to make their own rec.

Perhaps they want to use exu and ea_gen tool in a variety of ways.
But I think these feedbacks are hard to implement right now.

Hi Junsu,

1 - Really weird cause many of the MLs I looked at were with boxes containing boxes... If I have time, I will try to investigate.
2 - I found the bug that was killing my PC in the player. It is related to flipselector min max when step is 0 -> infinite creation of items. I pushed a fix in EFL.
4 - I understand your point. I thought about something that may limit but that should work for 99% of the cases

  • In config.json, add "Klass"->"DB"->"Events"->"clicked":"CLICK" / "changed,user":"KEY / "selected":"CLICK".

This indicates the event type for a certain event.

  • In DB ML gen, remove the candidates events not listed in config.json before the events generation so we support only those that the Exactness player supports
  • In scn gen, use EXACTNESS_ACTION_EVENT with CLICK/wdg_name, KEY/key
  • In Exactness player,
    • on CLICK, look for wdg, get x/y/w/h and append new actions (MOVE, UP, DOWN)
    • on KEY, append new actions (KEY_DOWN, KEY_UP)

This should work. I suppose that this is what you meant by hardware events, right? What do you think?
IIRC, the commented code didn't work. I think that what was missing is the delay between events. I will work on that this week.
5 - You mean that we would have to support exu generation on a binary?

jsuya added a comment.Nov 27 2018, 4:55 AM

5.Yes it is similar.
If test person have some test app code, he should be able to create an exu without reconfigure the app or ML.

summarize again
current
(app binary + rec = ml)-> db -> ml -> (app code + scn) = exu

feedback
app code + rec = exu

+)


I found another issue from scn_gen.
Please check attached files
The exu have same shot.
(exactness_inspect eagen_181127_2056_005.001.exu)
But the two shots should be different.
(exactness_play -s -t ./eagen_181127_2056_005.001.exu -- ./eagen_181127_2056_005)

scn have call the first shot, change the text(check_1's changed callback), and call the second shot.

I think it is because the player call the shot event and the next event is called right away.
Can you make some delays for shot event or have the shot completed and the next event called?

please check :)

Hi Junsu,

4 - What is your opinion on this events design?
5 - Sorry for my incoming stupid question, but why can't the tester run the Exactness recorder on its application? He can tell when to take a shot (F2) and to save the scenario (F3).

Concerning the issue you found, this is weird because:

  • it works for me, I mean the exu you provided and the exu output of the player are the same. I compiled my own binary and then used your binary to be sure there are no issues too. Both are identical to the source exu.
  • If you look at the exu, there is the action STABILIZE that is supposed to wait 500ms and then check that the app is graphically stable (no changes at all)

Are you totally updated? Exactness too?

4.I agree your opinion.
Now I can't predict what kind of conflict will happen.
But i think this is the best way.

5.Yes. user can use exactness_recorder.
However, the exu (rec) that is created does not have the source code.
It also has no other information required for testing.
If the exu(rec) and user app are managed separately within the CI, there is no need to package them.
but, if user can easily create rec and edit and package it, I expect this project(ea_gen, exactness and regression test CI) to be more usable.

+)
Concerning the issue. (efl, ea_gen, and exactness are latest.)
I said that the two shots made on this exu are the same.
The two shots should not be the same.

Efl.Ui.Check {
  id: check_1; legacy
  text: "Lorem ipsum do"
  visible: true
  propagate_events: false
  My.Check.state: false
  on changed {
    text: "Lorem ipsum dolor"
  }
}
Efl.Ui.Check {
  id: check_2; legacy
  hint_weight: [1, 1]
  text: "Lorem ipsum d"
  visible: true
  My.Check.state: false
  on changed {
    My.Check.state: true
  }
}

The scenario calls the event of check2 and takes a first shot.
The check2 event makes the state of check2 true.
The text of check1 should be "Lorem ipsum do" and check2 should be true.
And event of check1 is called.
The event of check1 changes the text of check1 to "Lorem ipsum dolor"
The second shot is a shot whose check1 text is "Lorem ipsum dolor" and check2 is true.
But text of check1 of the first shot is "Lorem ipsum dolor"


there is another sample. this is same issue.
The text of the button in the first shot must be "Lorem ipsu"
But it is "Lor"

4 - I will implement it in the next days
5 - There is a way to pack the source code inside the exu itself as well as the command to convert it to a binary. Can it fit? I don't think it would be a good idea to pack the binary directly
+ - I will check it after (4)

jsuya added a comment.Nov 29 2018, 4:37 AM

We have issue.
@YOhoho set to scale 1.3 at elementary_config and did test all tests. a lot of test were failed.

I analyzed the problem and found something strange like the one below.

ELM_SCALE=1 ELM_FINGER_SIZE=10 LD_PRELOAD+=" /usr/local/lib/libea_spy.so" exactness_play --stabilize-shots -t '/home/junsu/dev/os/exactness-elm-data/default-profile/recordings/button.rec' -o './/current' -- EA_SPY_OUTPUT=button_orig.ea elementary_test --test-win-only button

ELM_SCALE=1 ELM_FINGER_SIZE=10 exactness_play --stabilize-shots -t '/home/junsu/dev/os/exactness-elm-data/default-profile/recordings/button.rec' -o './/orig' -- elementary_test --test-win-only button

The first is the command to create a ML.
The second is the command to create a SHOT.

In elementary_config, set the scale to 1.3 and execute the above two commands for button.

The first command did not generate the click event of the 5th button(Label only button).
But obviously the elm_object_text_set of the click callback has been called.
The second command did call elm_object_text_set.

However, if i test again with the command below, the scale will be 1.

LD_PRELOAD+=/usr/local/lib/libea_spy.so ELM_SCALE=1 ELM_FINGER_SIZE=10 elementary_test --test-win-only button

ELM_SCALE=1 ELM_FINGER_SIZE=10 elementary_test --test-win-only button

I did not know the exact reason.
So when I tested some of the tests, I changed the --test-win-only of the first command to -to and tested it.
(

ELM_SCALE=1 ELM_FINGER_SIZE=10 LD_PRELOAD+=" /usr/local/lib/libea_spy.so" exactness_play --stabilize-shots -t '/home/junsu/dev/os/exactness-elm-data/default-profile/recordings/button.rec' -o './/current' -- EA_SPY_OUTPUT=button_orig.ea elementary_test -to button

)
"on clicked" event is hooked normally.

I'm still analyzing, but I still do not know.
Please check this issue with us.

Tell me if D7388 solves the issue

jsuya added a comment.Nov 29 2018, 6:48 PM

The issue is solved.
I completely misunderstood this issue. haha....
Thank you :)

jsuya added a comment.Dec 3 2018, 2:22 AM

there is another some issues

Shot image broken.
Sometimes, generated shot is broken.

All three scenarios contain broken images.
This problem seems to be random.
but I think... It seems to happen when the window size is very large or the window size changes.
Please check this problem.

Concerning ea_spy_check test failure

Test ctxpopup - Failure (7/10)
Test evas_map_3d - Failure (8/8)
Test file_selector - Failure (3/3)
Test file_selector_button - Failure (4/19)
Test genlist_7 - Failure (4/5)
Test genlist_decorate_item_mode - Failure (2/4)
Test genlist_decorate_modes - Failure (2/5)
Test genlist_del - Failure (3/5)
Test genlist_item_search_by_text - Failure (4/4)
Test genlist_item_styles - Failure (11/12)
Test genlist_tree - Failure (4/7)
Test notify - Failure (1/17)
Test photocam - Failure (4/5)
Test popup - Failure (17/18)
Test scroller - Failure (1/6)
Test toolbar - Failure (7/9)
Test toolbar_2 - Failure (8/10)
Test toolbar_3 - Failure (8/10)
Test toolbar_4 - Failure (8/10)
Test toolbar_6 - Failure (7/8)
Test tooltip - Failure (8/17)
Test window_states_2 - Failure (0/0)

In the past, some tests did not fail.
I think it is because of several reasons below.
Please check this.

1.Selected callback of item of genlist and gengrid is not created.

2.Selected callback of Toolbar's item also is not created.
(I cannot test for item's event)

3.popup widget is not hooked. So popup test is fail. (ctxpopup also)

Environment
efl : d652231(latest) + efl01, efl02 patch
ea_gen : 5b6218c (latest)
exactness : ede4ff10 (latest)

Hi Junsu,

Concerning the ea_gen tests, I will check this week

Concerning the spy checks:

  • most of the failures were due to changes in EFL that modifies the shots. I pushed updates. I haven't update yet the genlist_group, gengrid and gengrid_group shots because I want to understand whether it is right or not.
  • The fileselectors and panel tests use the local directory instead of some tmp dir not related to the PC where it is run on

I updated the tests status section so you can see the failure reason for most of them

Hi,

The 3 tests that were failing yesterday needed the shots to be updated. Btw, I pushed in Exactness the pause capability. When you press F2 during the scenario playing, it will pause it until you press F2 again.

The best way would be to add a small GUI next to the played application, with some buttons such as pause/speed + -... Maybe some day I will have time for this :-)

jsuya closed this task as Resolved.Dec 30 2018, 9:57 PM

Most of the issues listed above have been resolved.
If additional issues occur, proceed to the new task.

Thank you :)