Page MenuHomePhabricator

Improving Sandbox Gadgets for Python and Other Languages
Open, TODOPublic

Description

I wanted to create a forum where we could discuss improving sandbox gadget uses for languages other than C such as Python. The goal here is to limit the interpreters running. I thought everyone could respond with ideas they have for handling this.

stephenmhouston triaged this task as TODO priority.

One idea I've had is going to be similar to how the sandboxed sysinfo gadgets work ... If they run by themselves they use the main loop, but they also have a separate function to create the gadget if they are run by the parent sysinfo gadget. This separate function could also be something that can be called as an event when E already realizes a process exists for a like minded gadget and would rather reuse that gadget process to create a new version than create a new process. The idea here being that Similar gadgets don't use multiple processes.

@DaveMDS it would be great to get some input from you here since you are the only developer of gadgets in a non-C language...

I only have one clean solution for this problem, and it's already coded and perfectly working: it is called edgar :)

edgar automatically manage different instances of the same gadget for you and everything is in the same process.

stephenmhouston added a comment.EditedJan 23 2018, 11:30 AM

Well Edgar doesn't apply to sandboxed gadgets @DaveMDS so we are more interested in hearing how something similar can work for that case. I.E. Explain what edgar does, can it be converted to sandboxed gadgets, if not, how we can do something similar, etc...

Sorry, but I really have no idea how the same performance and easy-of-use of edgar can be obtained with the sandbox infra.

This is also the reason why I'm not taking sandboxed stuff into serious consideration for the moment.

My plans is to port edgar to use the new bryce API as soon as bryce will provide all the features of the "shelve" system. Note that this port will be really easy for me to implement and will require zero (or near zero) modifications at the existing python gadgets.

Sorry, but I really have no idea how the same performance and easy-of-use of edgar can be obtained with the sandbox infra.

This is also the reason why I'm not taking sandboxed stuff into serious consideration for the moment.

My plans is to port edgar to use the new bryce API as soon as bryce will provide all the features of the "shelve" system. Note that this port will be really easy for me to implement and will require zero (or near zero) modifications at the existing python gadgets.

Bryce is now renamed Gadget bar. The new gadget api and gadget bar already support all of the features of the "shelve" system as you call it and I don't see any new features being added so you are good to go for converting over in that respect.

sandbox stuff doesn't have to have the same performance as inner process gadgets. That's not the point because inner process gadgets are not going away. Ease of use wise, sandbox gadgets are already leaps and bounds easier to use than the inner process gadget stuff so that argument is invalid.

We are working to find a solution but you are literally the only non C gadget developer and it would be really nice to have some input from you regardless of whether or not you think it will be better or worse than inner process gadgets.

Performance is not an issue with sandboxing at the moment; there are a number of projects underway which should improve this situation. With that said, there are only noticeable differences at present if you plan on displaying hundreds of actively-rendering gadgets or running on a system with less than 1GB ram.

While I can appreciate that you've put a lot of time into your project over the years, I think that sandboxing will be the way forward for gadgets of all kinds; for ease of development, time required, ease of installation, and stability, sandboxing is just infinitely better. My current plan, as I've said in the past, is to eventually remove the ability for non-core modules to register any new gadgets and only allow sandboxed gadgets.

I've said repeatedly that I would like to incorporate some of the quality work that you've done into the gadget infrastructure and I've asked you to help with suggesting improvements to it, but you've continued to sit on the sidelines without contributing anything other than comments about how your project is superior. If there are valid technical reasons why this cannot work for Python gadgets then I'm open to discussion, but currently I have no details to go on.

@stephenmhouston:

  1. The main feature I miss in bryce is the ability to move/resize gadgets as it was before (without having to know obscure keybindings)
  2. I was comparing sanboxed gadgets to Edgar gadgets (not modules written in C). So my point is valid, and (trust me) edgar gadgets are much more easier that sandoxed ones.

@zmike:
As you have said in the ml thread you are not doing any benchmarking on the sandboxing stuff, so I don't have any number to trust you when you say there are no performance issue. The main performance issue I'm feared about is memory consumption, not rendering time. Can you tell me how much memory a simple gadget will require? A rough top list here show that Ephoto as a gadget eat up 60Mb of ram. (I know this is not benchmarking, but I'm not going to benchmark your code, and this is the only number I have)

I tried lots of time in IRC with you both to explain why I think the sanboxing infra is not going to be as efficient and easier as Edgar, and you always ignored my comments and suggestions. Your are going on your route without even discussing on ML the direction to follow, and this is inacceptable for me in an open community.

You guys already choose that your sandboxing infra is good and better and there is no way to make you change your mind.

Finally: I'm really tired to discuss with an unmovable wall, so please stop asking me what I think about your sandbox stuff that I don't like.

zmike added a comment.Jan 23 2018, 2:38 PM

As I said, there's no point wasting time doing benchmarks before optimizations are done. Why would you benchmark something that's only API complete and has not had work done to improve performance? This is just a strawman argument.

My recollection of any discussions in IRC has been basically you saying "edgar is better" on a number of occasions as well as "edgar is so much easier to use" on others. Looking at the history of the gadget_design page, you also chose not to contribute to that page which has existed since 2013. You similarly have not sent a single mail to the list related to gadgets since before 2013.

Why is it that despite this evidence that you've shown zero interest in contributing over many years that now I am the one responsible for your lack of contribution? I have to be the one to discuss everything on the mailing list? There have been numerous blog posts, consistent mentions on IRC, posts on social media, and hundreds of commits, but none of this was sufficient for you to get involved in any significant capacity?

More than anything I'm just confused at this point. You call me an "unmovable wall" but it seems to me like you haven't even tried to do any moving? I've reached out again and again to try and get you more involved on IRC, but you seem content enough to sit back and say "edgar is the king of gadgets" without offering any suggestions for improvements outside of your personal project silo. If that's your final position then that's fine--as you've said, this is an open community, so you're free to continue doing what you want.

All I've tried to do is get you to contribute to core Enlightenment infrastructure. If you're not interested and want to continue working on your own projects, just let me know so that I can avoid bothering you in the future.

I'm going to tag @cedric here and @q66 because I know they both mentioned to me they had their own ideas about how the sandbox gadgets can utilize languages other than C.

@zmike

a little preamble: I'm not speaking about the bryce work, but just about the sandbox infra (as per thread title)

About opensource community:
Seems we have a different view of what an opesource community is. IMO how to implement stuff must be discussed before start the development so that there is a consensus on the direction. Instead you have developed the sandbox stuff behind the closed doors of your office, without a prior consensus of the community.

About the page: gadget_design
I'm following that page from when it was first written and I agree with all the points written there, that is why I did not write anything there, because I agree. Nothing about "a single process for each gadget" is written in that page.

About my contributions to Enlightenment itself
I don't have the material time to work on Enlightenment, I have a full time job (not E related) and all the free time I have in the nights and in the weekends is used to develop and maintain the python bindings for efl. I simpy don't have the time to code in E.

About the unmovable wall
As you said we spoken lots of time in IRC about the sandbox implementation, and the discussion always goes in the same direction: I always said that a single process for each gadget is not going to be efficient (primarily for memory consumption) and you always ignored this.

Now you are asking me (as per this tread) how we can improve the sadbox infra for python and other (not yet existant) bindings... Sorry but I really don't have any idea. A super simple python-efl app consume aroud 100MB of ram (rought top output) that is more or less the double of a simple C-efl app....
But wait... I'm again talking about performance and you don't want to speak about performance... so why we are discussing at all?

All the discussion we have had about sandbox multiprocess has gone this way:

  1. You ask me what I think about multiprocess sandbox
  2. I say they will not be efficient (compared to C module/gadgets or compared to Edgar)
  3. You ignore my concern and go along your way

So how can I help if you ignore my primary concern?

A last side note:

@zmike: My current plan, as I've said in the past, is to eventually remove the ability for non-core modules to register any new gadgets and only allow sandboxed gadgets.

I will "fight with all the power I have" against this because this will "trash" lots of my works and quite all the gadgets I use everyday since years without any issue.

zmike added a comment.Jan 24 2018, 6:20 AM

@zmike

a little preamble: I'm not speaking about the bryce work, but just about the sandbox infra (as per thread title)

About opensource community:
Seems we have a different view of what an opesource community is. IMO how to implement stuff must be discussed before start the development so that there is a consensus on the direction. Instead you have developed the sandbox stuff behind the closed doors of your office, without a prior consensus of the community.

It was initially developed over the course of 5 months in a public repo which sends commits to the mailing list.

About the page: gadget_design
I'm following that page from when it was first written and I agree with all the points written there, that is why I did not write anything there, because I agree. Nothing about "a single process for each gadget" is written in that page.

About my contributions to Enlightenment itself
I don't have the material time to work on Enlightenment, I have a full time job (not E related) and all the free time I have in the nights and in the weekends is used to develop and maintain the python bindings for efl. I simpy don't have the time to code in E.

My point here was not about your lack of material contributions (i.e., code); not everyone uses C or has the time to write lots of code, but contributing ideas to focus development is always valuable.

About the unmovable wall
As you said we spoken lots of time in IRC about the sandbox implementation, and the discussion always goes in the same direction: I always said that a single process for each gadget is not going to be efficient (primarily for memory consumption) and you always ignored this.

Now you are asking me (as per this tread) how we can improve the sadbox infra for python and other (not yet existant) bindings... Sorry but I really don't have any idea. A super simple python-efl app consume aroud 100MB of ram (rought top output) that is more or less the double of a simple C-efl app....
But wait... I'm again talking about performance and you don't want to speak about performance... so why we are discussing at all?

All the discussion we have had about sandbox multiprocess has gone this way:

  1. You ask me what I think about multiprocess sandbox
  2. I say they will not be efficient (compared to C module/gadgets or compared to Edgar)
  3. You ignore my concern and go along your way

    So how can I help if you ignore my primary concern?

I understand that you care about memory usage. Everyone cares about memory usage, including me. There's no point in doing benchmarks or performance testing at this point since more work is being done to improve this, so why would I bother? I'm not ignoring your concern, I'm saying to wait until doing this testing is worthwhile.

A last side note:

@zmike: My current plan, as I've said in the past, is to eventually remove the ability for non-core modules to register any new gadgets and only allow sandboxed gadgets.

I will "fight with all the power I have" against this because this will "trash" lots of my works and quite all the gadgets I use everyday since years without any issue.

Naturally if I'm proposing a new system that's incompatible with a large-scale prior work such as your gadget series then I'm also willing to help do the work to migrate it so that it remains functional.

I said in a previous post that I was looking for technical reasons why moving to a sandbox-style architecture using Python was impossible; I consider memory usage a valid technical reason in this case. If it's impossible to reach approximately the same memory usage as your current gadgets then naturally I wouldn't remove the ability to write "more efficient" gadgets. I had thought this would be obvious since it's a core goal of the EFL community/project, but it seems not to be the case and now everyone believes that I am trying to make EFL/Enlightenment into chromium.