revised proposal still violates run-to-completion execution semantics
allenwb opened this issue · comments
The revised HostImportModuleDynamically says
If the operation succeeds, then either immediately or at some future time the host environment must perform FinishDynamicImport(referencingScriptOrModule, specifier, promiseCapability, NormalCompletion(undefined)).
The phrase "some future time" does not appear to place any restriction on the timing of performing FinishDynamicImport. So, it could occur in the middle of any ES job without waiting for completion of the current job (or even assume atomic operations). FinishDynamicImport calls HostResolveImportedModule which may start evaluation of a module body. So, we are still potentially preemptively interrupting ES code execution and observably running other ES code.
You could avoid this by scheduling a job to do FinishDynamicImport. As I stated in #24 (comment) ES spec. jobs provide the spec. mechanisms to schedule the execution of asynchronous ES code without violating the run to completion invariants
Other issues
then either immediately or at some future time.
That means module initialization code might run now or might run latter. In other parts of ES it was decided we would always defer to at least the "next turn" in such situations in order to improve deterministic behavior. If it is a maybe now/maybe latter we choose to make it be always latter.
If the operation succeeds, a subsequent call to HostResolveImportedModule after FinishDynamicImport has been performed, given the arguments referencingScriptOrModule and specifier, must complete normally.
-"has been performed" or "has been completed"? If this is happening asynchronously and concurrently FinishDynamicImport may have started but not yet processed to determination of success or failure. Can't multiple FinishDynamicImport for the same thing be inflight at the same time?
- The meaning of "If the operation succeeds" and "If the operation fails also seems fuzzy for similar reasons.
I think you are trying to address consistency and idempotency requirements but the more I try to understand the language the less certain I am as to what it actually means. Consider the following:
import("perhaps this file doesn't exist").then(m1=>console.log("succeed 1",_=>console.log("fail 1"));
import("perhaps this file doesn't exist").then(m2=>console.log("succeed 2",_=>console.log("fail 2"));Ignoring asychnc ordering, can we ever get "succeed 1 fail2" or "fail 1 succeed 2". If the result is "succeed 1 succeed 2" do m1 and m2 every have different values?
Hi Allen,
It seems you are getting at a more general issue with ES and host environment integration, and not one related to this proposal. As such I'd appreciate if you didn't characterize this as the proposal violating run to completion semantics, but instead, the proposal allowing violation of run to completion semantics---just like the rest of the language already does. For example,
The phrase "some future time" does not appear to place any restriction on the timing of performing FinishDynamicImport. So, it could occur in the middle of any ES job without waiting for completion of the current job (or even assume atomic operations).
Sure, but the host environment could also call other arbitrary ES operations in the middle of any ES job. As far as I know, ES does not place any restrictions on hosts there. This proposal does not introduce anything new, and I don't think it's this proposal's job to start creating such restrictions.
I think there's an interesting discussion to be had about constraining host environments in such a way. From my recollection, we have in the past talked about codifying the run-to-completion semantics as a restriction on host environments, and in this way disallowing Nashorn from claiming to be a compliant ECMAScript implementation and so on. That's a larger discussion, and one I support having, but not one we should block this proposal on.
That means module initialization code might run now or might run latter. In other parts of ES it was decided we would always defer to at least the "next turn" in such situations in order to improve deterministic behavior. If it is a maybe now/maybe latter we choose to make it be always latter.
As discussed at the last meeting, those semantics are a no-go for Node.js (and for HTML). That's why no host environment currently uses the script/module evaluation jobs, or even the EnqueueJob abstract operation as-written in general. There's definitely work to do reforming that rotten foundation, and I don't plan to build this proposal on top of it.
-"has been performed" or "has been completed"? If this is happening asynchronously and concurrently FinishDynamicImport may have started but not yet processed to determination of success or failure. Can't multiple FinishDynamicImport for the same thing be inflight at the same time?
Again this hinges on the larger question of whether host environments must "queue a task back on to the main thread" (to use browser/Node.js parlance) in order to respect run to completion. I'm happy to clarify this to "completed" though if that can help.
The meaning of "If the operation succeeds" and "If the operation fails also seems fuzzy for similar reasons.
This is a fair criticism. The meaning was meant to be left up to the host environment, as in, the host environment gets to choose one of two paths to go down, and gets a different bundle of requirements depending on which they choose. I can try to rephrase this in that fashion, to make it clearer and rely less on undefined terms.
Consider the following:
This is a great example! I think you're right we want to guarantee the properties you imply (never mix success and failure, and always guarantee the same object is returned). But even my above "choose one of two paths" does not require this. Let me attempt a rewrite that imposes these requirements.
Let me attempt a rewrite that imposes these requirements.
This is now live. I'm curious to see what you think. I believe the same-object-if-success part is already enforced by the idempotency requirement on HostResolveImportedModule, so I didn't add anything for that. But I separated out the success and failure paths as two sets of requirements, and stated that all calls must go down the same path as previous ones with the same referencingScriptOrModule, specifier pair.
No, I'm not commenting on general issues. I'm commenting specifically on your proposal and its proposed spec. language. As I have pointed out, the ES spec. has existing mechanisms for expressing run-to-completion semantics but your proposed spec. changes don't use those mechanisms and instead have vague prose which, to me, allow for dynamic import to violate run to completion.
Sure, but the host environment could also call other arbitrary ES operations in the middle of any ES job. As far as I know, ES does not place any restrictions on hosts there.
I think you misunderstand the role of the ES spec. and it's relationship to hosts or host specifications. The ES spec. is essentially a large set of requirements that an conforming implementation must respect. If an implementation violates those requirements it does not conform to the standard. Many of the requirements are expressed in terms of "abstract operations" that are invoked at specific places in the ES specification. The overall requirements of the spec. are encode via a combination of both the algorithmic steps of those operations and the contexts from which they are invoked. The existence of an abstract operation within the ES specification does not imply that any arbitrary invocation of it in a proposed feature specification is valid. In many cases, such invocation could violate other requirements or invariants of the ES specification. All uses have to be reviewed from that perspective.
Sure, but the host environment could also call other arbitrary ES operations in the middle of any ES job. As far as I know, ES does not place any restrictions on hosts there.
There are many ways an actual a host environment, or an external agent could corrupt an otherwise conforming ES implementation. At the extreme, it might simply write over the memory used by the implementation to represent the ES program or its runtime state. A host or agent arbitrarily (and perhaps asynchronous or even concurrently) "calling" (however that might be possible) an ES abstract operation, uncoordinated with the ES conforming behavior of an ES implementation, is the moral equivalent of stomping on the ES engine's memory. A conforming host implementation must not do this. A specification of a host extension to ES must not do that.
...we have in the past talked about codifying the run-to-completion semantics as a restriction on host environments, ... That's a larger discussion, and one I support having, but not one we should block this proposal on.
Run to completion has long been a strongly held principle of ES. That could change, but for now that principle stands and new feature proposals need to conform to it. If you want your proposal to quickly advance you need to conform to it. If you aren't in a hurry, you can defer your proposal and work on another proposal relating to eliminating or changing run to completion requirements.
That's why no host environment currently uses the script/module evaluation jobs, or even the EnqueueJob abstract operation as-written in general.
I think this statement reflects another misunderstanding of the role of the ES specification. Implementations don't "use" abstract concepts and operations from the ES spec. Rather they are expected to conform to the requirements expressed via those abstractions. The abstractions are just a framework for expressing requirements. If you think the spec. currently is expressing invalid or unintended requirements you should identify those so that can be vetted by TC39.
There's definitely work to do reforming that rotten foundation, and I don't plan to build this proposal on top of it.
Either advance your feature proposal by using the available mechanisms. Or defer that proposal while you work on developing a better foundation. Whichever the spec. mechanism for expressing such requirments to be applied consistently across the entire ES spec. One feature should not say, " I won't play by the current rules". I would active oppose advancement if this is the approach you want to take.
Again this hinges on the larger question of whether host environments must "queue a task back on to the main thread" (to use browser/Node.js parlance) in order to respect run to completion.
Here are the requirements that ES spec is trying to express via the jobs abstraction:
- ES semantics divides computational work in to a set of "jobs"
- Only one job may be actively undergoing evaluated at any point in time. (jobs never perform concurrent computation)
- Once evaluation of a job starts active evaluation, it must run to completion before evaluation of any other job starts.
- Once a job that has evaluated to completion it is never re-evaluated.
- When the current job completes, another job is selected for evaluation.
- There are some very limited constraints on the order in which jobs are selected for evaluation.
Nothing here talks about concepts like "main thread". It is up to an implementation how it maps these requirements onto implementation concepts such as OS threads. For example, there is nothing in the ES requirements that says that the implementation mechanism of job evaluation can't migrate among OS threads between jobs (or even during a job).
Also, host environment are certainly free to use other abstractions (I'm think of you "task") to describe their behavior. The only ES expectation is that if (from the host perspective) they are subsuming the role of an ES abstraction a ES conforming host abstraction must have the ES requirements as a subset of the host behavior. If there are widely implemented hosts that can't conform to the above requirements we should have serious discussions about it in TC39. We might relax the ES spec. or we might say: sorry, you're non-conforming and probably should work harder to get into conformance.
Run to completion has long been a strongly held principle of ES. That could change, but for now that principle stands and new feature proposals need to conform to it. If you want your proposal to quickly advance you need to conform to it.
I strongly disagree with the characterization of my proposal as not conforming to run to completion. It conforms to it exactly as much as the rest of the language does. As you say, a host could violate it in many ways. My proposal introduces no new ways to do so, anymore than any other proposal that introduces new abstract operations that the host might call at the wrong time.
It sounds like we'll have to take this discussion to committee, which is disappointing.
I think this statement reflects another misunderstanding of the role of the ES specification. Implementations don't "use" abstract concepts and operations from the ES spec. Rather they are expected to conform to the requirements expressed via those abstractions. The abstractions are just a framework for expressing requirements. If you think the spec. currently is expressing invalid or unintended requirements you should identify those so that can be vetted by TC39.
No, I perfectly well understand that. Implementations cannot implement the semantics as specified. That has indeed been previously identified by TC39, and the conclusion is to remove those invalid requirements. The fact that we haven't done so yet is not reason to block this proposal, in my opinion, but I'm happy to send a PR removing the script/module evaluation jobs and converting EnqueueJob to HostEnqueueJob, as previously discussed and agreed upon, if you think that's a requirement.
Either advance your feature proposal by using the available mechanisms. Or defer that proposal while you work on developing a better foundation.
I reject this dichotomy. The relation between an ES spec and the host environment is complex and we're still evolving the best way to specify it. Currently there are some bad aspects to the spec. That shouldn't prevent us from doing useful feature work, or force us to build on those bad aspects when doing useful feature work.
I would active oppose advancement if this is the approach you want to take.
That's disappointing, but I am confident the committee will be able to overcome such an objection.
Here are the requirements that ES spec is trying to express via the jobs abstraction:
Yes, these are fine requirements. But as discussed previously, the current spec's implementation of them is overconstraining on hosts, and EnqueueJob and the module/script evaluation jobs as such are not useful expressions of these requirements.