TCK Challenge: TCK should not require JPMS module-info.class
brideck opened this issue · comments
Challenged Tests:
build.xml
javasoft.sqe.tests.jakarta.activation.Module.UnnamedModule_Test
TCK Version:
Jakarta Activation 2.1.0
Tested Implementation:
Open Liberty
Description:
Per section 13.3 of the EE 10 specification, "there is no requirement around testing of JPMS in API jar signature tests or TCKs." However, there is a test in this TCK (and in truth, the entire automation of this TCK) that is doing exactly that.
Open Liberty does not provide API jars containing a module-info.class and as such is unable to run this TCK as it is currently written. Attempts to do so fail with the following:
[javatest.batch] java.lang.module.FindException: Unable to derive module descriptor for /jakarta/vi/wlp/dev/api/spec/io.openliberty.jakarta.activation.2.1_1.0.70.jar
[javatest.batch] Caused by: java.lang.IllegalArgumentException: io.openliberty.jakarta.activation.2.1.1.0.70: Invalid module name: '2' is not a Java identifier
[javatest.batch] Java Result: 1
If the automation is updated to not require JPMS in the API jar (by removing the module stuff from lines 191 & 192 in build.xml), then the UnnamedModule_Test still fails since it is explicitly testing that the API jar is on the modulepath.
In my opinion and based on the original intent of the requirement for JPMS descriptors in the spec API artifacts and their testability
"there is no requirement around testing of JPMS in API jar signature tests or TCKs."
means, that while the TCK is not required to test the JPMS, it is not forbidden to do so if it knows how to test the requirement, where the requirement is that a jar with Jakarta Activation APIs contains module descriptor with appropriate module name (jakarta.activation in this particular case). The other content of the module descriptor (ie list of exported packages, defined opens etc) is the part which does not have to be explicitly covered as long as all tests in the TCK pass (ie a vendor can offer an all-in-one - api+impl - bundle).
To me, this challenge is invalid.
@scottmarlow what do you think?
Note that if io.openliberty.jakarta.activation.2.1.1.0.70 is an API jar which does not have module name defined, implementations using/relying on Activation spec are not able to use it as a replacement of the API jar produced by the Jakarta Activation specification project
Ach, realizing that per the TCK process this should have been opened against jaf instead of jaf-tck. I have a replacement issue open there, if we want to migrate the conversation -- #97
re: "implementations using/relying on Activation spec are not able to use it as a replacement of the API jar"
If you keep reading section 13.3 it clearly states "Vendors are free to create their own API jars that pass the signature tests, but include no JPMS module-info.class files or JPMS module-info.class files with different or conflicting contents." There is no requirement to have module-info at all. My understanding is that this was a requirement that was considered earlier in the EE 10 cycle, but ultimately dropped from the final plan.
@scottmarlow what do you think?
implementations using/relying on Activation spec are not able to use it as a replacement of the API jar produced by the Jakarta Activation specification project
Is the Activation dependency on JPMS explained in https://jakarta.ee/specifications/activation/2.1/jakarta-activation-spec-2.1.html or https://jakarta.ee/specifications/activation/2.1?
Quoting the entire section from https://jakarta.ee/specifications/platform/10/jakarta-platform-spec-10.0.html#java-platform-module-system-jpms which is partially quoted above:
The EE10 release introduced a requirement for every component specification API jar included a JPMS module-info.class suitable for use with OpenJDK tools like jlink and jdeps. The contents of module-info.class files are not standard, portable, may change without notice and there is no requirement around testing of JPMS in API jar signature tests or TCKs. Vendors are free to create their own API jars that pass the signature tests, but include no JPMS module-info.class files or JPMS module-info.class files with different or conflicting contents. It is a future task to determine whether EE containers should support deployments that make use of JPMS information.
I think that the relevant text is but include no JPMS module-info.class files in the context of the containing sentence about what vendors may do.
I wonder if someone wants to send email to the Platform mailing list to discuss how the Activation JPMS requirement for finding the implementation via jakarta.activation module name (if there is one) is asserted by this test versus what it means if the Activation spec is incorrectly depending on JPMS with regard to the quoted Platform spec section.
I'm thinking that the Platform requirements outweigh the Activation Spec requirements, if they do, what does that mean with regard to stated Activated Spec requirements with regard to discovery of implementation?
Is the Activation dependency on JPMS explained [..]
Given EFSP 1.2 adopted by JESP 1.3 defines Specification as:
A collection of Application Programming Interface (API) definitions, descriptions of semantic behavior, data formats, protocols, and/or other referenced specifications, along with its TCK.
and collection of APIs published at https://jakarta.ee/specifications/activation/2.1/apidocs defines the module, and the lookup itself in https://jakarta.ee/specifications/activation/2.1/apidocs/jakarta.activation/jakarta/activation/spi/package-summary.html
I really don't think there is an option for the custom implementation of the API for component specification to not provide module-info. Note that component != platform.
Quoting the entire section from [EE spec]
I do not think this section captures the intent regarding JPMS Module Info classes the right way. Based on discussions on platform and TCK calls - at least how I recall those I've been part of, the section should have made it clear that:
- component APIs must have module info (covering jakarta package namespace)
- platform API does not have to have module info
- component TCKs as well as platform TCK are not required to test JPMS
Where the last bullet regarding TCKs was a late addition to relax the requirement of covering all new features with tests as it became clear that creation of these tests, especially for the platform, would cause unnecessary delay of the release.
I'm thinking that the Platform requirements outweigh the Activation Spec requirements
That is something I have nothing against as long as the Platform requirements are not changed after Activation Spec finalization.
and collection of APIs published at https://jakarta.ee/specifications/activation/2.1/apidocs defines the module, and the lookup itself in https://jakarta.ee/specifications/activation/2.1/apidocs/jakarta.activation/jakarta/activation/spi/package-summary.html
I really don't think there is an option for the custom implementation of the API for component specification to not provide module-info. Note that component != platform.
Pasting contents from https://jakarta.ee/specifications/activation/2.1/apidocs/jakarta.activation/jakarta/activation/spi/package-summary.html for easy reference:
Provides interfaces which implementations will be used as service providers for other services that used by Jakarta Activation.
Implementation of Jakarta Activation must implement interfaces declared in this package. Jakarta Activation uses
ServiceLoaderclass to discover and load implementations of the interfaces from this package using standard Java SPI mechanism.Provides interfaces which implementations will be used as service providers for other services that used by Jakarta Activation.
Interface Description This interface defines a factory for MailcapRegistry. This interface defines a factory for MimeTypeRegistry. Implementation of Jakarta Activation must implement interfaces declared in this package. Jakarta Activation uses ServiceLoader class to discover and load implementations of the interfaces from this package using standard Java SPI mechanism.
Interface Summary Interface Description [MailcapRegistryProvider](https://jakarta.ee/specifications/activation/2.1/apidocs/jakarta.activation/jakarta/activation/spi/MailcapRegistryProvider.html) This interface defines a factory for MailcapRegistry. [MimeTypeRegistryProvider](https://jakarta.ee/specifications/activation/2.1/apidocs/jakarta.activation/jakarta/activation/spi/MimeTypeRegistryProvider.html) This interface defines a factory for MimeTypeRegistry.
^ seems to describe the lookup that is expected by the Activation Java Doc description but it looks like the actual lookup action is in jakarta.activation.FactoryFinder which is not part of the Activation Spec as far as I can tell.
Should the UnnamedModule_Test be considered invalid since it assumes how the Activation API provided by implementations will handle what is being done with jakarta.activation.FactoryFinder?
From the FactoryFinder description:
Verify that API on module path is able to find and use an implementation, which is provided on the classpath.
The test seems to validate using JPMS but I don't see JPMS mentioned in the Activation spec as a requirement for how MailcapRegistry/MimeTypeRegistry locates the implementation.
^ seems to describe the lookup that is expected by the Activation Java Doc description but it looks like the actual lookup action is in jakarta.activation.FactoryFinder which is not part of the Activation Spec as far as I can tell.
Jakarta Activation uses ServiceLoader class ... using standard Java SPI mechanism.
ServiceLoader is java.util.ServiceLoader - that is the API satisfying the standard Java SPI mechanism part of the sentence - granted, that should have been made more explicit. In any case, are you saying that a private helper class calling another helper class which does the actual call to ServiceLoader itself must not be part of the spec API jar and everything must be implemented in public classes and methods?
standard Java SPI mechanism means something different in SE 8 and in SE 11 (adds support for service discovery on JPMS) and may mean something else in the future (one may decide to drop classpath in SE 31, who knows)
Should the UnnamedModule_Test be considered invalid since it assumes how the Activation API provided by implementations will handle what is being done with jakarta.activation.FactoryFinder?
That is not what the test is doing. The test is checking that the API jar was passed in on the module path and correctness of the module name of the spec API jar at runtime - if the API jar is passed in on the classpath, its name will never be the one defined by the API.
I do not think this section captures the intent regarding JPMS Module Info classes the right way. Based on discussions on platform and TCK calls - at least how I recall those I've been part of, the section should have made it clear that:
* component APIs must have module info (covering jakarta package namespace) * platform API does not have to have module info * component TCKs as well as platform TCK are not required to test JPMSWhere the last bullet regarding TCKs was a late addition to relax the requirement of covering all new features with tests as it became clear that creation of these tests, especially for the platform, would cause unnecessary delay of the release.
This does not match what was documented in the EE wiki on this topic (https://github.com/eclipse-ee4j/jakartaee-platform/wiki/Modularized-Jars) nor what was discussed on the Oct 19 2021 platform call (Agenda). In both cases, it is very clear that vendors don't need module-info in their API jars and that there should be nothing in a TCK that provides validation in that space.
There was a discussion and agreement on the platform call back on October 19th, 2021 (minutes at https://docs.google.com/document/d/1EJ2ilaPhMnQqa3aw6AmwjRbBPGL3_np4uuwklgfqPZI) and that resulted in this on the wiki:
https://github.com/eclipse-ee4j/jakartaee-platform/wiki/Modularized-Jars
In particular is this under testing requirements:
For implementation/vendor provided API jars, there are NO requirements. An implmentation does not need to provide a module-info, and they may provide an alternate module-info. It will not be validated as part of compatibility testing.
There was a discussion and agreement on the platform call back on October 19th, 2021
right, this testing was merged ~week before that to cover the outcome from 2021-06-29 (~3.5 months earlier) at the standalone spec level
In particular is this under testing requirements:
For implementation/vendor provided API jars, there are NO requirements. An implmentation does not need to provide a module-info, and they may provide an alternate module-info. It will not be validated as part of compatibility testing.
this makes compete sense within the context of the platform and/or its profiles (aka servers/products like Tomcat, WildFly, OpenLiberty, etc and final platform/profile level API jars they produce) and that is how I understood the outcome of that meeting - nobody has ever wanted the platform itself to define module descriptor or require vendors to (fully) adopt it in their platform based products (for EE 10 at least), the platform spec itself wrt deployments is explicitly built on top of Class-Path after all - so in this particular context, requirement for module descriptor/module-path handling would mean a lot of work for everyone (and no one really wants that).
The main idea was to allow those who want to (probably only Jetty these days) to have a building block in standalone specs they can rely on, experiment with, provide feedback and/or share ideas, yet do not restrict the exact content of the module descriptor too much, and open the room for incremental evolution/innovation in this area, should there be a wish/will for it.
But the thing is - and it seems to me that the platform team is often either not seeing it or just ignoring it - is that usage of some specifications (including activation) is not limited to the EE platform and corresponding runtime. If I stick to the Activation spec itself here, the spec has no restrictions on how exactly the API jar is passed to the application as long as it runs on Java SE 11 (or newer), so passing the API jar to the app (consider some desktop app to be an example) through class path is expected to have the similar effect as passing it to the app through the module path (assuming that what is not explicitly forbidden is allowed here) and one even does not need a test to see that usage of io.openliberty.jakarta.activation.2.1_1.0.70.jar without module descriptor or at least automatic module name in the manifest on the module path will fail.
Moving one level higher - to XML Binding spec - it defines explicit dependency on Activation, it even defines explicit integration requirements with JPMS (see the note at the end of section 4.2) beyond of what's in javadoc, it makes no restrictions on how exactly the API or user defined classes are made available to the application and it has been defined this way for 5+ years already (since 2.3). Should this spec do a step back to comply with and fully support platform's Vendors are free to create their own API jars that pass the signature tests, but include no JPMS module-info.class files statement and/or fix it's API jar to be able to work with non-modularized Activation API jar a vendor is allowed to produce?
Should we discuss ^ on the Platform mailing list to get wider input on the asked questions? IMO it would be good to include the possible options.
IMO, I think the options for this TCK challenge are:
- Accept challenge and exclude the test
- Reject challenge
If the challenge is accepted but we want to update the test instead, that is another option but requires a Specification Committee vote.
IMO, we should also discuss the JPMS requirements for Activation + XML Binding SPECs and what should happen to those requirements (e.g. stay as is or step back to comply with and fully support platform's Vendors are free to create their own API jars that pass the signature tests, but include no JPMS module-info.class files statement). I think the question to answer is whether Activation/XML Binding SPEC API implementations that don't support JPMS can meet the current SPEC requirements?
But the problem here is that the Activation TCK (and the Mail TCK, which shares this exact same design) are now required in order to certify EE 10 at the platform level. If they're going to serve this dual purpose, they need to also support what was decided at the platform level, i.e. having no module-info in the API jar is valid.
This can be achieved without too much trouble by providing an alternate execution path by way of 1) adding additional javatest.* targets to the Ant script that don't rope in a module path, 2) providing a Java test invocation in ts.jte.* that also doesn't rope in a module path (the Mail TCK already does this), and 3) isolating the UnnamedModule_Test so there's a set of tests that can be run for platform validation that doesn't execute it.
But doing all of that first requires acknowledgement that this challenge is valid from the perspective of platform certification.
IMO, I think the options for this TCK challenge are:
* Accept challenge and exclude the test * Reject challengeIf the challenge is accepted but we want to update the test instead, that is another option but requires a Specification Committee vote.
@scottmarlow Unfortunately, merely excluding the one test does not resolve the challenge. The entire TCK cannot currently be executed unless the API jar being tested contains a module-info. It is a fundamental requirement for even using the TCK.
FYI, we are currently working around this in our automation by adding the Jakarta API jar's module-info into our API jar and then running the TCK using that. If the community agrees that that is an acceptable workaround for certification, then we could go forward with that, but we should not have to do that for platform certification.
@scottmarlow Unfortunately, merely excluding the one test does not resolve the challenge. The entire TCK cannot currently be executed unless the API jar being tested contains a module-info. It is a fundamental requirement for even using the TCK.
@brideck
Updated as test result is already above for identified test.
Could you please attach or paste the test failure observed with one of the other tests just to show an example of how the other tests fail. This will be useful information.
@scottmarlow Unfortunately, merely excluding the one test does not resolve the challenge. The entire TCK cannot currently be executed unless the API jar being tested contains a module-info. It is a fundamental requirement for even using the TCK.
@brideck Updated as test result is already above for identified test.
Could you please attach or paste the test failure observed with one of the other tests just to show an example of how the other tests fail. This will be useful information.
The error given in the initial description is what happens when you try to run the TCK. No tests run at all, and that is the message.
I do not have an example of the specific UnnamedModule_Test failure handy at the moment, as it requires tinkering with the TCK scripts to produce. I think it just fails with the "jakarta.activation is not visible" failure message.
This can be achieved without too much trouble by providing an alternate execution path by way of 1) adding additional
javatest.*targets to the Ant script that don't rope in a module path, 2) providing a Java test invocation ints.jte.*that also doesn't rope in a module path (the Mail TCK already does this), and 3) isolating theUnnamedModule_Testso there's a set of tests that can be run for platform validation that doesn't execute it.
To be clear, here's an example of the kind of change I'm talking about. The Ant script currently has:
<presetdef name="javatest">
<java classname="com.sun.javatest.tool.Main"
failonerror="${failOnError}" fork="yes" classpath="${run.class.path}" modulepath="${module.path}">
<jvmarg line="--add-modules ${api.module.name}"/>
<sysproperty key="testDebug" value="${tests.debug}"/>
</java>
</presetdef>
To support this, we would need an additional target like this:
<presetdef name="javatest.noMod">
<java classname="com.sun.javatest.tool.Main"
failonerror="${failOnError}" fork="yes" classpath="${run.class.path}">
<sysproperty key="testDebug" value="${tests.debug}"/>
</java>
</presetdef>
And then additional javatest.batch and run.x targets that would in turn use the new noMod target.
Note: The above does not resolve the issue of the specific UnnamedModule_Test. There would still need to be a resolution to the issue of whether and how to exclude that particular test.
13.3. Java Platform Module System (JPMS)
The EE10 release introduced a requirement for every component specification API jar included a JPMS module-info.class suitable for use with OpenJDK tools like jlink and jdeps. The contents of module-info.class files are not standard, portable, may change without notice and there is no requirement around testing of JPMS in API jar signature tests or TCKs. Vendors are free to create their own API jars that pass the signature tests, but include no JPMS module-info.class files or JPMS module-info.class files with different or conflicting contents. It is a future task to determine whether EE containers should support deployments that make use of JPMS information.
Given that this is clearly stating that vendors are free to provide API jars that include no module info, and that there are no guarantees/portability of what the contents of the module-info.class are, there is no way that we can require any testing of module-infos in signature or TCK tests.
As I understand it, a workaround is to provide an alternative run mode for the TCK which allows for the API jars to be on the class path rather than a module path. That is a simple service release level change to the TCK that requires no test changes or exclusions.
As I understand it, a workaround is to provide an alternative run mode for the TCK which allows for the API jars to be on the class path rather than a module path. That is a simple service release level change to the TCK that requires no test changes or exclusions.
As stated in my previous post, there will still be an exclusion/solution needed for the UnnamedModule_Test. It is explicitly testing that the API classes are in a module named jakarta.activation.
Can the test be modified to pass if it is not on modulepath?
I would say yes, but the current TCK process is very anal and says changes can only be made for Java SE version issues. There is an ongoing discussion in the specification committee to relax the TCK process. Your proposed change makes sense.
as I see it, the API for Activation spec defined by the platform is different from the one defined by its standalone version (applies to all specs) and therefore possible solution is to have different setup/config for the platform runs. Ideally, since it is the platform who opted in to use modified version of the standalone spec, it would make sense to me to let the platform TCK somehow to pull out the TCK, apply some platform TCK specific in/exclusions and do the run itself as part of the platform TCK run. For now, it may be easier to have this extra platform config here, in this TCK.
It should also be made clear that:
- if the API implementation passes standalone TCKs, then it's usage is certified for SE as well as the Jakarta EE runtimes
- if the API implementation passes the platform profile TCK run, then its usage is certified for Jakarta EE runtime but NOT for SE runtime
resolution:
- for the platform runs, new test configuration config file (
ts-platform.jte) gets created UnnamedModule_Testgets updated to include...getModule().isNamed()check before checking the actual name- instructions in the user guide get updated with section how to execute the TCK for the EE platform
- user guide explicitly mentions that certification for the EE platform does not imply certification for SE environment
In order to achieve certification for EE 10, Open Liberty is currently working around this issue in our TCK automation by adding the module-info.class from jakarta.activation:jakarta.activation-api:2.1.0 to our shipped API/impl jar (io.openliberty.jakarta.activation.2.1_<version>.jar).
Could we please get formal approval of this approach for certification purposes? If approved, I will reference this challenge issue from our certification compatibility request. We will, of course, change to using the updated TCK as it becomes available.
Done
should be OK now