Comment on state of Project Jigsaw

Module declarations control the visibility and accessibility of types in
all phases of development.

If that's not sufficient to make them a language construct, then what is
your criterion?

- Mark

Eric Johnson

2011-12-21 22:11:58 UTC

Is this really just a comparison with a language like Ruby, wherein a
"module" could be done without adding any additional syntax to the language?

That is, I take the original point to be that "module" definitions
effectively look like a new DSL, and one that requires changes to the
Java compiler to accommodate, rather than one that can be accommodated
as-is.

Keeping true to this principle of using what is already defined for the
language might lead one, for example, to leveraging the MANIFEST.MF file
already defined for Java, but adding new fields to that file.

Alternately, following the principle in question, it could be done as a
standard Java class file leveraging static initialization and/or
annotations.

-Eric.

Post by mark.reinhold

Module declarations control the visibility and accessibility of types in
all phases of development.
If that's not sufficient to make them a language construct, then what is
your criterion?
- Mark

Neal Gafter

2011-12-21 22:15:22 UTC

A language construct is a syntactic construct that appears within source
files of the programming language.

In this case (if I understand it) the language "addition" is distinct from
and never intermixed with syntax of the underlying programming language.
It is therefore a distinct language.

Post by Neal Gafter

Post by Neal Gafter
My main high-level comment on project Jigsaw is that is fails its first
principle: modularity appears in the draft not to be a language

construct,

Post by Neal Gafter
but at best a completely separate language from the Java programming
language.

Module declarations control the visibility and accessibility of types in
all phases of development.
If that's not sufficient to make them a language construct, then what is
your criterion?
- Mark

mark.reinhold

2011-12-21 22:30:27 UTC

A language construct is a syntactic construct that appears within source files
of the programming language.
In this case (if I understand it) the language "addition" is distinct from and
never intermixed with syntax of the underlying programming language. It is
therefore a distinct language.

In purely syntactic terms module declarations are no less intermixed than
package, import, or type declarations [1], and I don't think anyone would
argue that those are not part of the language.

- Mark

[1] http://openjdk.java.net/projects/jigsaw/doc/lang-vm.html#jigsaw-1.3.2

Rémi Forax

2011-12-21 22:39:37 UTC

Post by mark.reinhold

A language construct is a syntactic construct that appears within source files
of the programming language.
In this case (if I understand it) the language "addition" is distinct from and
never intermixed with syntax of the underlying programming language. It is
therefore a distinct language.

In purely syntactic terms module declarations are no less intermixed than
package, import, or type declarations [1], and I don't think anyone would
argue that those are not part of the language.
- Mark
[1] http://openjdk.java.net/projects/jigsaw/doc/lang-vm.html#jigsaw-1.3.2

Java is a programming languages with several sources files
and jigsaw introduces a rule to find the module declaration
from a source file. This rule used a convention instead of a
declaration i.e. you don't have to edit all class files
to declare that they are part of a module but you have to use
a precise source layout.

If this layout is defined in the language spec so I don't see the problem.

R?mi

Alex Buckley

2011-12-21 23:11:27 UTC

Post by RÃ©mi Forax
Java is a programming languages with several sources files
and jigsaw introduces a rule to find the module declaration
from a source file. This rule used a convention instead of a
declaration i.e. you don't have to edit all class files
to declare that they are part of a module but you have to use
a precise source layout.
If this layout is defined in the language spec so I don't see the problem.

As Mark said, a module declaration is just another kind of declaration
within a compilation unit. The JLS has never made normative statements
about the physical form or layout of compilation units. (Nor, for that
matter, has the JVM Spec with regard to class files).

A compiler for the Java language is free to make up rules for the
placement on a file system of files that represent compilation units.
The file system placement of a compilation unit containing a module
declaration should no more be specified in the JLS than the placement of
a compilation unit containing a type declaration.

I am still curious to know how Neal would integrate statements or
declarations into the Java language that determine the visibility of
types ("observability", in JLS terms) for code within the scope of the
statements or declarations.

Alex

Rémi Forax

2011-12-21 23:30:49 UTC

Post by Alex Buckley

As Mark said, a module declaration is just another kind of declaration
within a compilation unit. The JLS has never made normative statements
about the physical form or layout of compilation units. (Nor, for that
matter, has the JVM Spec with regard to class files).
A compiler for the Java language is free to make up rules for the
placement on a file system of files that represent compilation units.
The file system placement of a compilation unit containing a module
declaration should no more be specified in the JLS than the placement
of a compilation unit containing a type declaration.
I am still curious to know how Neal would integrate statements or
declarations into the Java language that determine the visibility of
types ("observability", in JLS terms) for code within the scope of the
statements or declarations.
Alex

section 7.2.1 of the JLS3 defines the package layout, while it's
presented as an example, it's in the JLS :)
and Java had the chance that Visual Age was written before.

R?mi

Alex Buckley

2011-12-22 00:18:01 UTC

Post by RÃ©mi Forax
section 7.2.1 of the JLS3 defines the package layout, while it's
presented as an example, it's in the JLS :)
and Java had the chance that Visual Age was written before.

JLS3 7.2.1 is non-normative ("As an extremely simple example..."). It is
not part of conformance testing and may as well not exist.

The intertwining of normative statements and non-normative
examples/discussion in the JLS was an unfortunate legacy that I have
taken significant steps to fix in JLS7. (See Annex 3 of the JSR 336
Final Release. Before you ask, the PDFs and HTML will be hosted at a
friendlier location, and the books will be published, sometime in 2012.)

Alex

Neal Gafter

2011-12-21 23:57:35 UTC

Post by Alex Buckley
I am still curious to know how Neal would integrate statements or
declarations into the Java language that determine the visibility of types
("observability", in JLS terms) for code within the scope of the statements
or declarations.

I recommend having the module system define a new form of accessibility,
not "observability". The JLS has many examples of language-defined
accessibility.

Alex Buckley

2011-12-22 00:13:23 UTC

On Wed, Dec 21, 2011 at 3:11 PM, Alex Buckley <alex.buckley at oracle.com
I am still curious to know how Neal would integrate statements or
declarations into the Java language that determine the visibility of
types ("observability", in JLS terms) for code within the scope of
the statements or declarations.
I recommend having the module system define a new form of accessibility,
not "observability". The JLS has many examples of language-defined
accessibility.

Accessibility is a declaration-site mechanism. A provider (typically a
type) defines its accessibility to the world. For the module system, we
need a use-site mechanism, whereby a consumer (typically a module)
defines its requirements on the rest of the world. Where do you say
that? It clearly can't be up to a providing module to say all the places
where it'll be observed/visible from.

Alex

Neal Gafter

2011-12-22 01:06:58 UTC

A use-site protection mechanism provides no protection, as one can just say
"yes, it's accessible to me" at any place you want to use the thing
supposedly protected. But you do want the modularity mechanism to provide
a measure of protection, don't you?

Post by Alex Buckley

Accessibility is a declaration-site mechanism. A provider (typically a
type) defines its accessibility to the world. For the module system, we
need a use-site mechanism, whereby a consumer (typically a module) defines
its requirements on the rest of the world. Where do you say that? It
clearly can't be up to a providing module to say all the places where it'll
be observed/visible from.
Alex

Alex Buckley

2011-12-22 01:32:12 UTC

Post by Neal Gafter
A use-site protection mechanism provides no protection, as one can just
say "yes, it's accessible to me" at any place you want to use the thing
supposedly protected.

I'm not getting my question across. _How_ do you say this at the place
you want to use the thing?

If the answer is "Just say the name of the thing!", then I ask: "How do
you know which module the thing comes from?"

Even with package dependences rather than module dependences, you still
need to say at the use-site which version of the package you want.
Perhaps you are proposing "import java.util.List @ 8.0;" ?

Alex

Neal Gafter

2011-12-22 18:07:07 UTC

Post by Neal Gafter
A use-site protection mechanism provides no protection, as one can just
say "yes, it's accessible to me" at any place you want to use the thing
supposedly protected.

I'm not getting my question across. _How_ do you say this at the place you
want to use the thing?
If the answer is "Just say the name of the thing!", then I ask: "How do
you know which module the thing comes from?"
Even with package dependences rather than module dependences, you still
need to say at the use-site which version of the package you want. Perhaps

I don't think a protection mechanism that involves use-site access control
(or version control) is an appropriate language extension at all. That is
more properly a feature of a build and packaging system. While it may well
involve a packaging specification language (e.g. a package specification
file *or* a compiler command-line or even the layout and organization of
files on the disk drive), that language need have no relationship to the
underlying programming languages of the system under construction. Jigsaw
appears to have largely taken that approach, with the exception that Jigsaw
pretends it is an extension of the Java programming language.

Alex Buckley

2011-12-23 01:05:20 UTC

Post by Neal Gafter
I don't think a protection mechanism that involves use-site access
control (or version control) is an appropriate language extension at
all. That is more properly a feature of a build and packaging system.
While it may well involve a packaging specification language (e.g. a
package specification file /or/ a compiler command-line or even the
layout and organization of files on the disk drive), that language need
have no relationship to the underlying programming languages of the
system under construction. Jigsaw appears to have largely taken that
approach, with the exception that Jigsaw pretends it is an extension of
the Java programming language.

OK, now I understand. You do not object to a module system being part of
the Java SE platform, or to dependences at the granularity of a module.
You object to including a module declaration in the grammar of the Java
language, specifically in the CompilationUnit production, and having the
effects of a module declaration be noticeable in the language's
visibility and accessibility rules.

Many presentations on Jigsaw have covered the need for modules to _look_
familiar to Java programmers, leading to the open requirement:

"Java-like syntax ? The syntax of module declarations must be similar to
that of declarations of a like nature in the Java programming language."

From that open requirement flows a module declaration built from the
traditional input elements of the Java language, and included under
CompilationUnit in the Java grammar. Yes, the requirement is open, but
it's the current direction of the Jigsaw prototype. When modules are as
much a part of the platform as packages and types, it's not too
surprising that their declarations live close together.

If the Java SE Platform Specification took it upon itself to define a
module declaration in a sibling of CompilationUnit, say ModuleUnit, and
mandated that a ModuleUnit be detected and interpreted at compile-time
and runtime according to the module system's rules, then I think you
would be marginally happier. At least the module declaration would be
"outside" the language.

You would still object to the language's sense of visibility and
accessibility being determined by another part of the platform, namely
the module system. But the language has always existed in the context of
the overall platform, and the omnipresence of the module system is a
fundamental principle in the requirements. Not sure we have a zone of
agreement here.

Alex

Neal Gafter

2011-12-23 01:55:21 UTC

Post by Alex Buckley
Many presentations on Jigsaw have covered the need for modules to _look_
"Java-like syntax ? The syntax of module declarations must be similar to
that of declarations of a like nature in the Java programming language."

In what sense are module declarations "of a like nature" to anything
currently in the Java programming language?

-Neal

Alex Buckley

2011-12-23 02:37:47 UTC

On Thu, Dec 22, 2011 at 5:05 PM, Alex Buckley <alex.buckley at oracle.com
Many presentations on Jigsaw have covered the need for modules to
"Java-like syntax ? The syntax of module declarations must be
similar to that of declarations of a like nature in the Java
programming language."
In what sense are module declarations "of a like nature" to anything
currently in the Java programming language?

I take it the rest of my email accurately summarized your views.

You are querying whether the _semantics_ of module declarations are like
the semantics of anything currently in the language. Obviously not. The
module concept is new, even though the concepts of visibility (a.k.a.
observability) and accessibility are old.

Precisely because a module is a first-class concept in the language
(affecting visibility and accessibility), it is not unreasonable to
represent it syntactically using the traditional input elements of the
Java programming language.

The fact that a type declaration doesn't refer to a module name seems to
be causing people to think that modules are not a first-class concept in
the language. But they are.

Wouldn't it be odd if some declarations of first-class language concepts
were made in Java-like syntax while others were made in, say, JSON?
After all, a package is a different thing to a field, and yet both exist
in the Java grammar. Just because a module is a different thing to a
field doesn't mean a module should be excluded from the Java grammar.

Alex

Neal Gafter

2011-12-23 04:20:35 UTC

Post by Alex Buckley
I take it the rest of my email accurately summarized your views.
You are querying whether the _semantics_ of module declarations are like
the semantics of anything currently in the language. Obviously not. The
module concept is new, even though the concepts of visibility (a.k.a.
observability) and accessibility are old.

The fact that a concept is old is not evidence in favor of supporting it in
a programming language.

Observability is not a concept currently part of the Java language. The
language spec mentions observability explicitly as something essentially
outside of its scope (JLS3 7.3).

Post by Alex Buckley
Precisely because a module is a first-class concept in the language
(affecting visibility and accessibility), it is not unreasonable to
represent it syntactically using the traditional input elements of the Java
programming language.

I don't see module-level accessibility as a language construct in the
latest draft (but I may well have missed it).

Post by Alex Buckley
The fact that a type declaration doesn't refer to a module name seems to
be causing people to think that modules are not a first-class concept in
the language. But they are.

What does it mean for something to be first-class in the language if it
cannot be named or referred to in code in any way? Where is the syntax for
*anything *in the Java language extended to be module-aware? Adding a new,
separate language (and allowing the .java extension for that language)
doesn't make that language's concepts first-class in the Java language.

Wouldn't it be odd if some declarations of first-class language concepts

Post by Alex Buckley
were made in Java-like syntax while others were made in, say, JSON? After
all, a package is a different thing to a field, and yet both exist in the
Java grammar. Just because a module is a different thing to a field doesn't
mean a module should be excluded from the Java grammar.

No, I do not find it odd that Observability is defined by the javac command
line (e.g. you can't see a class which isn't involved in the compilation at
all), even though the javac command line language is something completely
different from the Java programming language.

I do not consider something to be "first-class" in the language where that
thing cannot be referred to by name *or in any other way* in Java code.

In short, the rest of your email did not accurately reflect my views.

Alex Buckley

2011-12-23 16:59:14 UTC

One of the fundamental principles of the module system is fidelity
between compile-time and runtime. Conceptually, to align the language
with the module system, the scope of observability will grow to
encompass module dependences and the scope of accessibility will grow to
encompass module exports. Practically, a developer will know which
module his code belongs to and which modules he depends on, so there is
no need for module names within type declarations. But since you are
unwilling to accept this growth in the platform and its constituent
parts, I don't think there's any point in continuing with this thread.

Alex

On Thu, Dec 22, 2011 at 6:37 PM, Alex Buckley <alex.buckley at oracle.com
I take it the rest of my email accurately summarized your views.
You are querying whether the _semantics_ of module declarations are
like the semantics of anything currently in the language. Obviously
not. The module concept is new, even though the concepts of
visibility (a.k.a. observability) and accessibility are old.
The fact that a concept is old is not evidence in favor of supporting it
in a programming language.
Observability is not a concept currently part of the Java language. The
language spec mentions observability explicitly as something essentially
outside of its scope (JLS3 7.3).
Precisely because a module is a first-class concept in the language
(affecting visibility and accessibility), it is not unreasonable to
represent it syntactically using the traditional input elements of
the Java programming language.
I don't see module-level accessibility as a language construct in the
latest draft (but I may well have missed it).
The fact that a type declaration doesn't refer to a module name
seems to be causing people to think that modules are not a
first-class concept in the language. But they are.
What does it mean for something to be first-class in the language if it
cannot be named or referred to in code in any way? Where is the syntax
for /anything /in the Java language extended to be module-aware? Adding
a new, separate language (and allowing the .java extension for that
language) doesn't make that language's concepts first-class in the Java
language.
Wouldn't it be odd if some declarations of first-class language
concepts were made in Java-like syntax while others were made in,
say, JSON? After all, a package is a different thing to a field, and
yet both exist in the Java grammar. Just because a module is a
different thing to a field doesn't mean a module should be excluded
from the Java grammar.
No, I do not find it odd that Observability is defined by the javac
command line (e.g. you can't see a class which isn't involved in the
compilation at all), even though the javac command line language is
something completely different from the Java programming language.
I do not consider something to be "first-class" in the language where
that thing cannot be referred to by name /or in any other way/ in Java code.
In short, the rest of your email did not accurately reflect my views.

Peter Kriens

2011-12-23 18:38:54 UTC

I think that fidelity between build and run time is red herring. When I build a house I need scaffolding and lots of stuff I do not need when I live in the house. The build time should help me build robust components that work in different places and do not require me to change metadata all the time.

I usually do not know where my code ends so I fail to see how I can get fidelity ... The trick is to manage your dependencies and then minimize them.

Kind regards,

Peter kriens

-----
Sent while on the road, mobile +33698332260

One of the fundamental principles of the module system is fidelity between compile-time and runtime. Conceptually, to align the language with the module system, the scope of observability will grow to encompass module dependences and the scope of accessibility will grow to encompass module exports. Practically, a developer will know which module his code belongs to and which modules he depends on, so there is no need for module names within type declarations. But since you are unwilling to accept this growth in the platform and its constituent parts, I don't think there's any point in continuing with this thread.
Alex

Jesse Glick

2011-12-23 18:52:14 UTC

to align the language with the module system, the scope
of observability will grow to encompass module dependences
and the scope of accessibility will grow to encompass module exports.

I fail to see how the tradition of the JLS forces such a system to represent modules directly in Java source files. Following the style of chapter 7, the JLS could simply
say something to the effect of:

- A host system may group compilation units [or types?] into sets called "modules". The host system must then determine which other modules are observable from a given
module, and may specify that some types [or packages? or individual elements?] within those modules are to be considered inaccessible despite the other provisions in the
section "Access Control".

JSR 199's JavaFileManager would need to provide a way to specify module identity, observability, and accessibility. At runtime, java.lang.module would define opaque
module identifiers with no details such as version, just informational toString() and a boolean hook for the VM to conduct its access checks; and permit
ClassLoader.defineClass to pass a module identifier.

Then it would be up to the javac command line and the module-aware Java launcher to interpret a physical format of modules - which could, for example, be based on JARs
with special manifest attributes defining Jigsaw semantics. Alternate 199-based build tools and java.lang.module-based runtime containers could just as easily interpret
OSGi or various other existing formats, or some hybrid system capable of hosting and cross-wiring multiple containers; unlike with older JDKs, they could benefit from
compiler- and runtime-enforced module accessibility checks.

For that matter, a Maven plugin could easily produce a Jigsaw-compatible binary using a natural POM to determine observability (from <dependencies>), as well as other
module metadata (from <artifactId>, <version>, etc.), with either plugin <configuration> or source annotations to determine other information such as exports. Making a
Jigsaw plugin would be awkward under the current proposal, as it would need to either create target/generated-sources/jigsaw/module-info.java in the generate-sources
phase; or (like Tycho?) inject dependencies dynamically by parsing src/main/java/module-info.java, breaking many popular workflows and complicating IDE support.

Of course you could argue that module-info.java as currently proposed is simply more readable than manifests and is what Java developers want, especially those who use no
IDE or build framework or module system currently. That does not preclude the above approach - it just means that the syntax of module-info.java would be specified by
Jigsaw, not the JLS, and could just as easily be module.yaml.

David M. Lloyd

2011-12-23 19:14:15 UTC

Post by Jesse Glick

to align the language with the module system, the scope
of observability will grow to encompass module dependences
and the scope of accessibility will grow to encompass module exports.

I fail to see how the tradition of the JLS forces such a system to
represent modules directly in Java source files. Following the style of
- A host system may group compilation units [or types?] into sets called
"modules". The host system must then determine which other modules are
observable from a given module, and may specify that some types [or
packages? or individual elements?] within those modules are to be
considered inaccessible despite the other provisions in the section
"Access Control".

It would be even better if this could be refined to draw a 1:0..1
equivalence between a class loader and a module, and define the access
control in terms of the class loader. This also expands the visibility
rule to non-modular code which could be quite useful, and closely
matches existing package accessibility rules as well.

The only reason given to allow multiple modules to share a class loader
is for the purposes of code which expects JDK classes to have a "null"
class loader. But I don't see this as a serious hurdle - there are many
reasons that code may need updating after Java 8 and singling out this
one as the one unacceptable change seems a bit unrealistic.

--
- DML

Alex Buckley

2011-12-23 20:46:57 UTC

Post by Jesse Glick
I fail to see how the tradition of the JLS forces such a system to
represent modules directly in Java source files. Following the style of
- A host system may group compilation units [or types?] into sets called
"modules". The host system must then determine which other modules are
observable from a given module, and may specify that some types [or
packages? or individual elements?] within those modules are to be
considered inaccessible despite the other provisions in the section
"Access Control".

Yes, this is pretty much what the JLS needs to say. The host system
_must_ attempt to identify a module declaration among its observable
compilation units, and then based on the module system's interpretation
of that declaration, make other compilation units observable. (The JLS
only knows about Java code in compilation units, so there may be some
finessing here to cover code in class files.)

It doesn't even have to be in the JLS. The package-info javadoc for the
java.lang.module package would do fine. Both are part of the overall
Java SE Platform Specification. The accessibility rules embedded in the
JLS can refer to module system constructs (e.g. exports) whether those
constructs are in the JLS or in the SE API.

(This is how JSR 292 is specified. The JVM Spec's text for the
invokedynamic instruction is deeply dependent on concepts and rules
defined in the spec of the java.lang.invoke package, because the
instruction is fundamentally a front-end to operations on intrinsic
types like MethodHandle.)

Note we are not yet specifying the syntax of a module declaration, or a
physical placement of a module declaration on a file system.

Post by Jesse Glick
JSR 199's JavaFileManager would need to provide a way to specify module
identity, observability, and accessibility. At runtime, java.lang.module
would define opaque module identifiers with no details such as version,
just informational toString() and a boolean hook for the VM to conduct
its access checks; and permit ClassLoader.defineClass to pass a module
identifier.

Basically yes. (I don't want to get into versioning here.)

Post by Jesse Glick
Then it would be up to the javac command line and the module-aware Java
launcher to interpret a physical format of modules - which could, for
example, be based on JARs with special manifest attributes defining
Jigsaw semantics. ...
Of course you could argue that module-info.java

Stop right there. A file called module-info.java is an implementation
detail, not something for the spec. You should say: "Of course you could
argue that a module declaration having Java-like syntax" ... OK, let's
proceed.

Post by Jesse Glick
as currently proposed is
simply more readable than manifests and is what Java developers want,
especially those who use no IDE or build framework or module system
currently. That does not preclude the above approach - it just means
that the syntax of module-info.java would be specified by Jigsaw, not
the JLS, and could just as easily be module.yaml.

Again, to be fully correct: "it just means that the syntax of a module
declaration would be specified by Jigsaw, not the JLS".

Sure, the module system could define a YAML-based grammar for a module
declaration, and the host system would still be mandated (by text
_somewhere_ in the Platform Specification) to find and interpret a
module declaration at compile-time. This is where we remember there's
another requirement, currently open, that says the grammar should use
Java-like syntax.

Alex

Neal Gafter

2011-12-23 19:49:04 UTC

One of the fundamental principles of the module system is fidelity between
compile-time and runtime. Conceptually, to align the language with the
module system, the scope of observability will grow to encompass module
dependences and the scope of accessibility will grow to encompass module
exports. Practically, a developer will know which module his code belongs
to and which modules he depends on, so there is no need for module names
within type declarations.

I agree that module names need not be mentioned in Java code. That was
precisely my point!

But since you are unwilling to accept this growth in the platform and its

constituent parts, I don't think there's any point in continuing with this
thread.

Actually, I *want* the *platform* to grow to add support for modules.
Since we agree that there is no place for the mention of module names or
versions in Java code (i.e. the domain of the language specification),
there isn't any point in continuing with this thread.

David Bosschaert

2012-01-04 13:57:12 UTC

I'd say these presentations represent only a limited section of the community.
There have been many arguments made by a variety of people arguing for
a DSL or other mechanism which is more suited for storing module
declarations. This email trail is an example of that.

Just to give one example where it doesn't fit:
A module declaration applies to the artifact as a whole, i.e. the
module jar file, so it doesn't fit naturally alongside other java
classes in any java file - it needs to be in a special file of its
own. And that special file will look different from all the other java
files so hence it doesn't make sense to make it part of the java
syntax as this syntax is only used in this special place, effectively
creating a separate language, as Neal mentioned above.
Making things worse, having it in a .java file implies that it will be
compiled into a .class file which makes thing unnecessarily complex
for tools and other module systems to read this thing at runtime.
Additionally, having it in a .class file compromises the extensibility
of the module metadata, which is needed for module systems that may
have more features and need extensions to JSE module metadata.

Post by Alex Buckley
From that open requirement flows a module declaration built from the
traditional input elements of the Java language, and included under
CompilationUnit in the Java grammar. Yes, the requirement is open, but it's
the current direction of the Jigsaw prototype.

Its an open requirement clearly because opinions diverge over this
issue. We need to stress that Jigsaw is a prototype and that this
aspect of the prototype can't 'just make it' into JSE8 without proper
discussion in the upcoming modularity JSR.

Best regards,

David

Rémi Forax

2012-01-04 14:46:40 UTC

The module descriptor is not the .java but the .class,
the .java is a convenient way to specify the module descriptor.

This allow to have a readable syntax for the user and
a module descriptor that contains synthetized informations
by example the module descriptor can contain a set of exported types,
having a .java let users to use a * (wildcard) to specify the exported
types.

I expect all tools will use the jigsaw reflective API or ASM.

R?mi

I'd say these presentations represent only a limited section of the community.
There have been many arguments made by a variety of people arguing for
a DSL or other mechanism which is more suited for storing module
declarations. This email trail is an example of that.
A module declaration applies to the artifact as a whole, i.e. the
module jar file, so it doesn't fit naturally alongside other java
classes in any java file - it needs to be in a special file of its
own. And that special file will look different from all the other java
files so hence it doesn't make sense to make it part of the java
syntax as this syntax is only used in this special place, effectively
creating a separate language, as Neal mentioned above.
Making things worse, having it in a .java file implies that it will be
compiled into a .class file which makes thing unnecessarily complex
for tools and other module systems to read this thing at runtime.
Additionally, having it in a .class file compromises the extensibility
of the module metadata, which is needed for module systems that may
have more features and need extensions to JSE module metadata.

David Bosschaert

2012-01-04 15:23:31 UTC

Post by RÃ©mi Forax
The module descriptor is not the .java but the .class,
the .java is a convenient way to specify the module descriptor.

I have nothing against a convenient way of specifying a module
description, I actually see that as a requirement :) but that
requirement doesn't imply that it needs to be shoe-horned into .java
and .class files!

Post by RÃ©mi Forax
This allow to have a readable syntax for the user and
a module descriptor that contains synthetized informations
by example the module descriptor can contain a set of exported types,
having a .java let users to use a * (wildcard) to specify the exported
types.

I don't follow why you would only get this feature if you describe the
module metadata in a .java file. You could just as easily define a
textual file format that allows the specification of wildcards...

Post by RÃ©mi Forax
I expect all tools will use the jigsaw reflective API
or ASM.

Well, ahem, requiring people to use ASM isn't really user-friendly IMHO...

David

Rémi Forax

2012-01-04 16:02:15 UTC

Post by RÃ©mi Forax
The module descriptor is not the .java but the .class,
the .java is a convenient way to specify the module descriptor.

The .class format is already an extensible versioned container format
with a dictionnary which is able to store string and class name.
No shoehorn here.

Post by RÃ©mi Forax
I expect all tools will use the jigsaw reflective API
or ASM.

Well, ahem, requiring people to use ASM isn't really user-friendly IMHO...

Don't ask me if ASM use user friendly :)
http://asm.ow2.org/team.html

Post by David Bosschaert
David

R?mi

Eric Johnson

2012-01-04 20:49:50 UTC

Comment below...

Post by RÃ©mi Forax
The module descriptor is not the .java but the .class,
the .java is a convenient way to specify the module descriptor.

The .class format is already an extensible versioned container format
with a dictionnary which is able to store string and class name.
No shoehorn here.

OK, perhaps the format is reusable, but how will existing tools fare
with it? Will existing IDEs choke on the file unless modified because it
has unexpected content? Will code obfuscation tools obfuscate it by
default? Will Ant need to be updated to treat these files specially?
Will any ASM client ever written fall over because the module definition
file doesn't actually include byte codes? What about tools like "bnd"
that introspect ".class" files looking for referenced symbols, so that
they can generate OSGi information?

I think the notion of reusing a format might work, but I suspect reusing
the extension ".class" will cause a lot of churn in downstream tooling,
and it doesn't sound like that's been considered here. (And I worry that
I've written code that would succumb to that churn.)

If you're arguing that it is a useful container format, though, there
are plenty of other "container formats" - JSON and XML jump to mind -
that can equally well satisfy the requirements that have been stated so
far. Since modules must be "installed" (if I've understood what I've
read correctly), then any notions of pre-compiling them are likely not
essential, because the installation process can do what it wishes, and
the files themselves have no "external" dependencies that need to be
controlled in a compilation process (unlike, say, the existing Java
classes that we compile.)

-Eric.

Rémi Forax

2012-01-05 14:25:35 UTC

Post by Eric Johnson
Comment below...

Post by RÃ©mi Forax
The module descriptor is not the .java but the .class,
the .java is a convenient way to specify the module descriptor.

The .class format is already an extensible versioned container format
with a dictionnary which is able to store string and class name.
No shoehorn here.

The classfile format is versioned, each time a new version is created,
like 1.5 and 1.7 did,
you have to update your tools. most of the tools, first check the
version and just exit
if the version number is unknown.

Post by Eric Johnson
I think the notion of reusing a format might work, but I suspect
reusing the extension ".class" will cause a lot of churn in downstream
tooling, and it doesn't sound like that's been considered here. (And I
worry that I've written code that would succumb to that churn.)
If you're arguing that it is a useful container format, though, there
are plenty of other "container formats" - JSON and XML jump to mind -
that can equally well satisfy the requirements that have been stated
so far.

If you use a text format, you introduce several stupid problems like the
charset encoding
and the fact that a user can update the module descriptor without
re-validating it.

Post by Eric Johnson
Since modules must be "installed" (if I've understood what I've read
correctly), then any notions of pre-compiling them are likely not
essential, because the installation process can do what it wishes, and
the files themselves have no "external" dependencies that need to be
controlled in a compilation process (unlike, say, the existing Java
classes that we compile.)

I don't agree.
You still need to validate the module descriptor before distributing it
so a long time before installing it.
And the module descriptor contains references to packages and types that
need to be checked.

Post by Eric Johnson
-Eric.

R?mi

Eric Johnson

2012-01-05 16:25:04 UTC

Hi R?mi,

Post by Eric Johnson
Comment below...

Post by RÃ©mi Forax
The module descriptor is not the .java but the .class,
the .java is a convenient way to specify the module descriptor.

The .class format is already an extensible versioned container format
with a dictionnary which is able to store string and class name.
No shoehorn here.

I understood that. I think you're overlooking my point about how many
additional changes tools might need to make. Think of the universe of
existing tools that crawl Java .class files for whatever reason. How
many of those existing tools need to care about module information
(answer 0%)? How many of those tools will need to be updated - beyond
accepting the newer version - if the module information is loaded into
an existing .class file format, with a .class extension.

If I were drawing a picture, I'd have a circle with three pie slices:

* tools that need to change merely to accept the new version of the
class file format
* tools that don't care about the module information but will need to
change to work around/ignore said data
* tools that actually care about the new module information.

Question is, how big are you making that middle slice? If you put the
module information into an alternate format, I *know* that the second
slice is an empty set.

Post by Eric Johnson
I think the notion of reusing a format might work, but I suspect
reusing the extension ".class" will cause a lot of churn in
downstream tooling, and it doesn't sound like that's been considered
here. (And I worry that I've written code that would succumb to that
churn.)
If you're arguing that it is a useful container format, though, there
are plenty of other "container formats" - JSON and XML jump to mind -
that can equally well satisfy the requirements that have been stated
so far.

If you use a text format, you introduce several stupid problems like
the charset encoding
and the fact that a user can update the module descriptor without
re-validating it.

XML deals quite nicely with character encodings.

As for updating, it is merely that the tooling is simpler. Re-validating
might be required in either case.

Post by Eric Johnson
Since modules must be "installed" (if I've understood what I've read
correctly), then any notions of pre-compiling them are likely not
essential, because the installation process can do what it wishes,
and the files themselves have no "external" dependencies that need to
be controlled in a compilation process (unlike, say, the existing
Java classes that we compile.)

I don't agree.
You still need to validate the module descriptor before distributing
it so a long time before installing it.
And the module descriptor contains references to packages and types
that need to be checked.

My point was that any "compilation" process you can do on a module
descriptor can be done on the file itself without reference to those
external entities. /Validation/ is a separate problem.

For an analogy, a Python file can be compiled to a ".pyc" file without
any knowledge of external dependencies. However, it cannot be run unless
those external dependencies are satisfied.

-Eric.

Alex Buckley

2012-01-05 20:22:42 UTC

Post by Eric Johnson
* tools that need to change merely to accept the new version of the
class file format
* tools that don't care about the module information but will need to
change to work around/ignore said data
* tools that actually care about the new module information.
Question is, how big are you making that middle slice? If you put the
module information into an alternate format, I *know* that the second
slice is an empty set.

(In what follows, I assume 52.0 is the ClassFile version for Java SE 8.)

The various ClassFile attributes proposed by Jigsaw (Module,
ModuleRequires, etc) should have no impact on tools which only know
about pre-52.0 ClassFiles. The JVM Spec has always been very clear that
new attributes can be defined and must be silently ignored by parts of
the system which pre-date them. We already have two new
annotation-related attributes proposed for 52.0 by JSR 308.

The worst-case scenario is a pre-52.0 tool ignoring ClassFile version
and attempting to interpret a 52.0 ClassFile as if it was an earlier
version. This tool is broken (consider an 'invokedynamic' instruction in
a 51.0 ClassFile that's interpreted as 50.0) ... but still, new module
attributes will just be ignored. What _could_ be a problem is any
redefinition of non-attribute artifacts in a 52.0 ClassFile v. 51.0.

Is there any such redefinition? I tried to avoid it, but there are two.
They take effect only when the new ACC_MODULE flag is set. (I regard a
new flag much like a new attribute; it must be safe to add constructs
which in and of themselves don't overlap with existing constructs.) They
are:

1) Non-setting of ACC_INTERFACE no longer means "this is a class".
(Though this is moot because interpreting the ClassFile as a class
anyway will reveal a well-formed but very boring class.)

2) Setting of super_class to 0 no longer means "my supertype is an Object".

So that's the effect of using the ClassFile structure to reify module
information. Tools which properly recognize 52.0 ClassFiles but that
don't care about module information will have to work around (2). Since
such a tool would already be switching on the presence of ACC_MODULE,
(2) is almost trivial.

Alex

Alex Buckley

2012-01-05 21:56:22 UTC

Post by Alex Buckley
The various ClassFile attributes proposed by Jigsaw (Module,
ModuleRequires, etc) should have no impact on tools which only know
about pre-52.0 ClassFiles.

Actually I can strengthen this sentence: the various attributes should
have no impact on tools, period. (Or "full stop", as some would say.)
They can be ignored by _any_ ClassFile reader which doesn't care about
module information, whether it's from 1996 and reads 45.3 or it's from
2013 and reads 52.0.

Post by Alex Buckley
2) Setting of super_class to 0 no longer means "my supertype is an Object".

To phrase this better: it no longer means "my superclass is Object (if
I'm a class), or my superinterface is represented as Object (if I'm an
interface)".

Alex

David Bosschaert

2012-03-23 11:42:09 UTC

Post by Eric Johnson
Hi R?mi,

Post by Eric Johnson
Comment below...

Post by RÃ©mi Forax
The module descriptor is not the .java but the .class,
the .java is a convenient way to specify the module descriptor.

The .class format is already an extensible versioned container format
with a dictionnary which is able to store string and class name.
No shoehorn here.

The classfile format is versioned, each time a new version is created,
like 1.5 and 1.7 did,
you have to update your tools. most of the tools, first check the version
and just exit
if the version number is unknown.

I understood that. I think you're overlooking my point about how many
additional changes tools might need to make. Think of the universe of
existing tools that crawl Java .class files for whatever reason. How many of
those existing tools need to care about module information (answer 0%)? How
many of those tools will need to be updated - beyond accepting the newer
version - if the module information is loaded into an existing .class file
format, with a .class extension.
?* tools that need to change merely to accept the new version of the
? class file format
?* tools that don't care about the module information but will need to
? change to work around/ignore said data
?* tools that actually care about the new module information.
Question is, how big are you making that middle slice? If you put the module
information into an alternate format, I *know* that the second slice is an
empty set.

Post by Eric Johnson
I think the notion of reusing a format might work, but I suspect reusing
the extension ".class" will cause a lot of churn in downstream tooling, and
it doesn't sound like that's been considered here. (And I worry that I've
written code that would succumb to that churn.)
If you're arguing that it is a useful container format, though, there are
plenty of other "container formats" - JSON and XML jump to mind - that can
equally well satisfy the requirements that have been stated so far.

If you use a text format, you introduce several stupid problems like the
charset encoding
and the fact that a user can update the module descriptor without
re-validating it.

XML deals quite nicely with character encodings.
As for updating, it is merely that the tooling is simpler. Re-validating
might be required in either case.

Post by Eric Johnson
Since modules must be "installed" (if I've understood what I've read
correctly), then any notions of pre-compiling them are likely not essential,
because the installation process can do what it wishes, and the files
themselves have no "external" dependencies that need to be controlled in a
compilation process (unlike, say, the existing Java classes that we
compile.)

I don't agree.
You still need to validate the module descriptor before distributing it so
a long time before installing it.
And the module descriptor contains references to packages and types that
need to be checked.

My point was that any "compilation" process you can do on a module
descriptor can be done on the file itself without reference to those
external entities. /Validation/ is a separate problem.
For an analogy, a Python file can be compiled to a ".pyc" file without any
knowledge of external dependencies. However, it cannot be run unless those
external dependencies are satisfied.
-Eric.

Just FYI I started modifying OpenJDK/Jigsaw in the context of the
Penrose project to see if something like module-info.xml instead of
module-info.class/.java would work. From the prototype that I made it
seems to make a lot of sense. See here:
http://mail.openjdk.java.net/pipermail/penrose-dev/2012-March/000023.html

Best regards,

David

Eric Johnson

2012-03-23 12:07:16 UTC

Hi David,

Post by David Bosschaert
Just FYI I started modifying OpenJDK/Jigsaw in the context of the
Penrose project to see if something like module-info.xml instead of
module-info.class/.java would work. From the prototype that I made it
http://mail.openjdk.java.net/pipermail/penrose-dev/2012-March/000023.html
Best regards,
David

Thanks for exploring this question. I'm excited to see that someone has
taken up this work.

For the trivial case you note, the XML, of course, looks mildly more
complicated, if only because the namespace declarations add to the
verbosity (and I note that you left off the <?xml version="1.0" ?>
prologue). However, I think that is vastly outweighed by the significant
increase in the number of tools that can immediately process the data -
for example, I can write a script that extracts the module file from
every module in a set, and then walks through the XML to generate an
HTML report of the same. All without having to know a thing about Java
class file formats.

For another example, I can edit the XML in Eclipse today, and apply the
schema so that I know I am doing it correctly. And I can do that without
having to wait for the Eclipse Java toolkit to improve to handle the new
module-info.java syntax. Of course, as noted in previous discussion,
this won't encompass the runtime validity of the data (can the required
modules be resolved?), but it will ease the development of the contents
of said files. As I know developers who still insist on using Emacs and
its derivatives, I think this is enormously useful.

For yet another scenario, when a Linux distribution needs to patch
upstream module metadata for whatever reason (for example, changing a
choice of security module dependency), it will be far easier to alter an
XML file (treated as text, most likely), than to patch a
module-info.java file, then recompiling said file.

And, as you note, extending the data you can put into the module file is
trivial, in the sense that you can simply add new namespace declarations.

In other words, I see many reasons for a hearty +1 to your prototype!

-Eric.

Jaroslav Tulach

2012-03-26 20:51:22 UTC

## 23.?3.?2012 08:07:16 ##<
Hi David,

I cannot speak for the Jigsaw team, but (while my organization also needs
concurrently extensible module-info[1]) there may be a reason why not use XML
instead of module-info.class[2].

It is my understanding that one goal of Jigsaw is to make core Java really
small and allow it to execute on very limited devices. The core needs to
understand module-info. Right now the core contains ~3000 classes. If you
bring full XML parsing infrastructure in, then you increase the size by ~4000
new classes (that was my observation when I worked on splitting JDK last
time[3]).

You could argue, that XML need not be fully featured one, but some simplified
variant. Something that can be parsed by 30KB parser and not bloated Xerces.
But that is not much different from using binary module-info.class[4].

-jt

[1] http://wiki.netbeans.org/Jigsaw
[2] please note that I am not talking about module-info.java
[3] http://wiki.apidesign.org/wiki/Modular_Java_SE
[4] If module-info.class format was made truly extensible

For yet another scenario, when a Linux distribution needs to patch
upstream module metadata for whatever reason (for example, changing a
choice of security module dependency), it will be far easier to alter an
XML file (treated as text, most likely), than to patch a
module-info.java file, then recompiling said file.

PS: Linux distributions always recompile upstream projects from sources, so
they would be patching module-info.java - which would be simple textual diff.

David Bosschaert

2012-03-27 00:21:59 UTC

Hi Jaroslav,

## 23.?3.?2012 08:07:16 ##<
Hi David,

You certainly don't need a full blown XML parser. I think a restricted
one is fine. In my prototype I modified the (ASL2-licensed) Apache
Piccolo project to get things to work. I didn't optimize at all.
Piccolo.jar is about 160KB and around 50 classes in its original form
but that includes all the SAX and JAXP interfaces that go with it.

Right, I have seen XML parsers implemented in jars < 15Kb so if we
decide to go this route a dedicated parser could be implemented in
such a small footprint.
I think it would be *very* different from parsing the module-info.class because
* XML can be authored by a myriad of existing tools out there
* XML can be read by humans
* XML provides a well-defined extensibility model (using custom name-spaces)
* XML can be parsed easily by other module systems (as they most
likely have an XML parser at their disposal)
* XML can be parsed easily by other tools that wish to operate on
module information

On the other hand, I'm not married on XML - another textual format
might also do the job. I'm open to ideas here.

Cheers,

David

Neil Bartlett

2012-03-27 02:27:18 UTC

## 23.?3.?2012 08:07:16 ##<
Hi David,

I agree that bloating the core with an XML parser would be undesirable, but at the same time support David's desire for an extensible, textual format. If there were a textual format with a parser already supported in the core then perhaps this could be used. For example the MANIFEST.MF parsing in java.util.jar is currently part of core, I believe.

Regards,
Neil

Post by Jaroslav Tulach
You could argue, that XML need not be fully featured one, but some simplified
variant. Something that can be parsed by 30KB parser and not bloated Xerces.
But that is not much different from using binary module-info.class[4].
-jt
[1] http://wiki.netbeans.org/Jigsaw
[2] please note that I am not talking about module-info.java
[3] http://wiki.apidesign.org/wiki/Modular_Java_SE
[4] If module-info.class format was made truly extensible

PS: Linux distributions always recompile upstream projects from sources, so
they would be patching module-info.java - which would be simple textual diff.

Jaroslav Tulach

2012-03-27 03:25:57 UTC

Hello David, Neil,
if I was supposed to marry the benefits of XML (tooling, extensibility,
encoding) with ability to parse the info, I would claim that module-info.xml
is a source format which gets compiled by javac into module-info.class.
module-info.class would contain all the information needed by Jigsaw runtime
(so there is no XML processing in the core) and somehow stored the rest of the
information in the XML for other systems to process later (most trivially as
whole XML kept in an attribute of module-info.class, but possibly in better
way).

If there is a utility to read MANIFEST.MF in core, it is not bad idea to reuse
it. Properties like format (which manifests are using) is almost as extensible
as XML, although validation tools are hard to find[1].

Anyway I seem to like the approach of having two extensible formats. One which
is easily editable and audit-able (in spite of requiring sophisticated tools
with high memory and CPU consumption) and one that is easy to process with
minimal CPU cycles (generated during compilation).
-jt

[1] This would probably not happen in XML:
http://wiki.apidesign.org/wiki/PropertyFiles

## 26.?3.?2012 22:27:18 ##<

## 23.?3.?2012 08:07:16 ##<
Hi David,

I agree that bloating the core with an XML parser would be undesirable, but
at the same time support David's desire for an extensible, textual format.
If there were a textual format with a parser already supported in the core
then perhaps this could be used. For example the MANIFEST.MF parsing in
java.util.jar is currently part of core, I believe.
Regards,
Neil

Post by Jaroslav Tulach
You could argue, that XML need not be fully featured one, but some
simplified variant. Something that can be parsed by 30KB parser and not
bloated Xerces. But that is not much different from using binary
module-info.class[4].
-jt
[1] http://wiki.netbeans.org/Jigsaw
[2] please note that I am not talking about module-info.java
[3] http://wiki.apidesign.org/wiki/Modular_Java_SE
[4] If module-info.class format was made truly extensible

For yet another scenario, when a Linux distribution needs to patch
upstream module metadata for whatever reason (for example, changing a
choice of security module dependency), it will be far easier to alter
an XML file (treated as text, most likely), than to patch a
module-info.java file, then recompiling said file.

PS: Linux distributions always recompile upstream projects from sources,
so they would be patching module-info.java - which would be simple
textual diff.

David Bosschaert

2012-03-28 18:09:50 UTC

Hi Jaroslav,

I think combining module-info.class with an XML appendix really is the
worst of both worlds, because it doesn't allow you to use XML tools to
edit/view, as you end up with a combination binary/XML file. I would
be much more in favour of staying within a textual format (XML,
MANIFEST as Neil proposed, or something else) in the ultimate module
where the actual JRE can cache a binary representation of this
internally if it wishes to.

WRT to he MANIFEST.MF, yes I think the core contains a parser for
that, so it could be used. OSGi has leveraged the Manifest for about
14 years now without problems. I used XML in my prototype as it seems
like a more versatile format than MANIFEST.MF and since we're
designing something new here it might make sense to move to a more
generally understood format. I don't think the implementation size of
an XML parser which is only focused on parsing this file should be a
big issue.

I have to say I don't like having 2 different official representations
of the module info at all (sorry ;) as it carries all kinds of
synchronization issues and possibly doubles the burden of supporting
it. If you want a binary format I think the way to go is let the JRE
implementation generate that under the covers and cache it to optimize
speed etc.

Best regards,

David

Post by Jaroslav Tulach
Hello David, Neil,
if I was supposed to marry the benefits of XML (tooling, extensibility,
encoding) with ability to parse the info, I would claim that module-info.xml
is a source format which gets compiled by javac into module-info.class.
module-info.class would contain all the information needed by Jigsaw runtime
(so there is no XML processing in the core) and somehow stored the rest of the
information in the XML for other systems to process later (most trivially as
whole XML kept in an attribute of module-info.class, but possibly in better
way).
If there is a utility to read MANIFEST.MF in core, it is not bad idea to reuse
it. Properties like format (which manifests are using) is almost as extensible
as XML, although validation tools are hard to find[1].
Anyway I seem to like the approach of having two extensible formats. One which
is easily editable and audit-able (in spite of requiring sophisticated tools
with high memory and CPU consumption) and one that is easy to process with
minimal CPU cycles (generated during compilation).
-jt
http://wiki.apidesign.org/wiki/PropertyFiles

## 26.?3.?2012 22:27:18 ##<

## 23.?3.?2012 08:07:16 ##<
Hi David,

I agree that bloating the core with an XML parser would be undesirable, but
at the same time support David's desire for an extensible, textual format.
If there were a textual format with a parser already supported in the core
then perhaps this could be used. For example the MANIFEST.MF parsing in
java.util.jar is currently part of core, I believe.
Regards,
Neil

Post by Jaroslav Tulach
You could argue, that XML need not be fully featured one, but some
simplified variant. Something that can be parsed by 30KB parser and not
bloated Xerces. But that is not much different from using binary
module-info.class[4].
-jt
[1] http://wiki.netbeans.org/Jigsaw
[2] please note that I am not talking about module-info.java
[3] http://wiki.apidesign.org/wiki/Modular_Java_SE
[4] If module-info.class format was made truly extensible

For yet another scenario, when a Linux distribution needs to patch
upstream module metadata for whatever reason (for example, changing a
choice of security module dependency), it will be far easier to alter
an XML file (treated as text, most likely), than to patch a
module-info.java file, then recompiling said file.

PS: Linux distributions always recompile upstream projects from sources,
so they would be patching module-info.java - which would be simple
textual diff.

Rémi Forax

2012-03-28 18:52:31 UTC

Post by David Bosschaert
Hi Jaroslav,
I think combining module-info.class with an XML appendix really is the
worst of both worlds, because it doesn't allow you to use XML tools to
edit/view, as you end up with a combination binary/XML file. I would
be much more in favour of staying within a textual format (XML,
MANIFEST as Neil proposed, or something else) in the ultimate module
where the actual JRE can cache a binary representation of this
internally if it wishes to.
WRT to he MANIFEST.MF, yes I think the core contains a parser for
that, so it could be used. OSGi has leveraged the Manifest for about
14 years now without problems. I used XML in my prototype as it seems
like a more versatile format than MANIFEST.MF and since we're
designing something new here it might make sense to move to a more
generally understood format. I don't think the implementation size of
an XML parser which is only focused on parsing this file should be a
big issue.

There are issues with manifest, the two main points for my students
are that they publish manifest that don't even parse
and the second one is that they keep in the manifest
generated information that are not true anymore.

Post by David Bosschaert
I have to say I don't like having 2 different official representations
of the module info at all (sorry ;) as it carries all kinds of
synchronization issues and possibly doubles the burden of supporting
it. If you want a binary format I think the way to go is let the JRE
implementation generate that under the covers and cache it to optimize
speed etc.

As I already said, the best is to develop a small parser
that is able to parse a module-info.class and provide
XML events or properties.

Post by David Bosschaert
Best regards,
David

best regards,
R?mi

Alan Bateman

2012-03-29 12:05:32 UTC

Post by RÃ©mi Forax
As I already said, the best is to develop a small parser
that is able to parse a module-info.class and provide
XML events or properties.

FWIW, ModuleSystem already defines a parseModuleInfo method for parsing
module-info.class files.

On XML then the current prototype base module does not include JAXP or
any XML support.

-Alan.

David Bosschaert

2012-04-13 07:15:16 UTC

Post by RÃ©mi Forax
As I already said, the best is to develop a small parser
that is able to parse a module-info.class and provide
XML events or properties.

FWIW, ModuleSystem already defines a parseModuleInfo method for parsing
module-info.class files.

This is yet another reason why I think it's weird to use .class files.
Normally .class files are read by the VM, but in this case a special
java class is written to read the .class file. So it's used as just a
data file, which again underlines the point that using .class files is
not the right thing to do here.

Post by Alan Bateman
On XML then the current prototype base module does not include JAXP or any
XML support.

Right, so if we go with XML we'd have to embed a tiny parser that is
just good enough to read the XML definition, which isn't hard to do. I
have currently an implementation that (when jarred-up) is about 85kb,
but as mentioned before it should be possible to optimize that into
something smaller.

I've been thinking about alternative representations. Although I think
that using MANIFEST.MF is better than using module-info.class, there
are still some issues with it: the fixed line-length and other
esoteric formatting rules and the fact that when using it one
generally needs a second-level parser to parse the value of a header,
I mean the JRE Manifest parser could return the following from an OSGi
Import-Package header,
org.osgi.service.log;version="[1.3,2)",org.osgi.util.tracker;version="[1.4,2)"
but that will then need to be fed to another parser to be processed.
It would be nice that whatever format we come up with can use a single
parser to do all the parsing for us :)

So I've been thinking a little more about alternative representations.
Given the following XML (which I have working on my penrose clone)
<module name="org.astro" version="1.2"
xmlns="http://www.jcp.org/xmlns/modules/v1.0">
<exports name="org.astro.somepackage"/>
</module>

This could also be represented as JSON like this:
module {
"name" : "org.astro",
"version" : "1.2",
"exports" : [
{ "name" : "org.astro.somepackage"}
]
}
I don't think the core contains a JSON parser, but again it should be
possible to write a really small one (possibly by leveraging the
java.util.Scanner class). Using JSON one can then add extensibility by
introducing the idea of namespaces. I don't think this is standard
JSON at this point in time, but I have seen people do it in the
following way (e.g. to add OSGi versioning to the package export):
module {
"name" : "org.astro",
"version" : "1.2",
"exports" : [
{ "name" : "org.astro.somepackage", "org.osgi.version" : "1.0" }
]
}

I'd be happy to investigate the actual realization of this through
JSON if people think that's better than XML and MANIFEST.MF...

Cheers,

David

Eric Johnson

2012-04-13 09:07:21 UTC

I thought XML initially because that's what I'm familiar with, and I
don't know of all the JSON tools out there that could be applied.

JSON has some interesting properties - like now being readily able to
build a Javascript program running in a browser that can crawl metadata
about bundles.

In any case, if you think JSON works, that still seems better to me than
burying the information in a compiled "module-info.class" file.

As for the "OSGi" extensions you explore, you could equally well
"reserve" the "osgi-" prefix by fiat, and do this:

module {
"name" : "org.astro",
"version" : "1.2",
"exports" : [
{ "name" : "org.astro.somepackage"
"osgi-version" : "1.0"
"osgi-uses" : [ "org.astro.other.packageA",
"org.astro.otherpackageB" ] }
]
}

(My JSON syntax may be incorrect above, but hopefully you get the intent?)

-Eric.

Post by RÃ©mi Forax
As I already said, the best is to develop a small parser
that is able to parse a module-info.class and provide
XML events or properties.

FWIW, ModuleSystem already defines a parseModuleInfo method for parsing
module-info.class files.

Post by Alan Bateman
On XML then the current prototype base module does not include JAXP or any
XML support.

Right, so if we go with XML we'd have to embed a tiny parser that is
just good enough to read the XML definition, which isn't hard to do. I
have currently an implementation that (when jarred-up) is about 85kb,
but as mentioned before it should be possible to optimize that into
something smaller.
I've been thinking about alternative representations. Although I think
that using MANIFEST.MF is better than using module-info.class, there
are still some issues with it: the fixed line-length and other
esoteric formatting rules and the fact that when using it one
generally needs a second-level parser to parse the value of a header,
I mean the JRE Manifest parser could return the following from an OSGi
Import-Package header,
org.osgi.service.log;version="[1.3,2)",org.osgi.util.tracker;version="[1.4,2)"
but that will then need to be fed to another parser to be processed.
It would be nice that whatever format we come up with can use a single
parser to do all the parsing for us :)
So I've been thinking a little more about alternative representations.
Given the following XML (which I have working on my penrose clone)
<module name="org.astro" version="1.2"
xmlns="http://www.jcp.org/xmlns/modules/v1.0">
<exports name="org.astro.somepackage"/>
</module>
module {
"name" : "org.astro",
"version" : "1.2",
"exports" : [
{ "name" : "org.astro.somepackage"}
]
}
I don't think the core contains a JSON parser, but again it should be
possible to write a really small one (possibly by leveraging the
java.util.Scanner class). Using JSON one can then add extensibility by
introducing the idea of namespaces. I don't think this is standard
JSON at this point in time, but I have seen people do it in the
module {
"name" : "org.astro",
"version" : "1.2",
"exports" : [
{ "name" : "org.astro.somepackage", "org.osgi.version" : "1.0" }
]
}
I'd be happy to investigate the actual realization of this through
JSON if people think that's better than XML and MANIFEST.MF...
Cheers,
David

Shi Jun Zhang

2012-04-13 09:47:01 UTC

Post by RÃ©mi Forax
As I already said, the best is to develop a small parser
that is able to parse a module-info.class and provide
XML events or properties.

FWIW, ModuleSystem already defines a parseModuleInfo method for parsing
module-info.class files.

Post by Alan Bateman
On XML then the current prototype base module does not include JAXP or any
XML support.

Right, so if we go with XML we'd have to embed a tiny parser that is
just good enough to read the XML definition, which isn't hard to do. I
have currently an implementation that (when jarred-up) is about 85kb,
but as mentioned before it should be possible to optimize that into
something smaller.
I've been thinking about alternative representations. Although I think
that using MANIFEST.MF is better than using module-info.class, there
are still some issues with it: the fixed line-length and other
esoteric formatting rules and the fact that when using it one
generally needs a second-level parser to parse the value of a header,
I mean the JRE Manifest parser could return the following from an OSGi
Import-Package header,
org.osgi.service.log;version="[1.3,2)",org.osgi.util.tracker;version="[1.4,2)"
but that will then need to be fed to another parser to be processed.
It would be nice that whatever format we come up with can use a single
parser to do all the parsing for us :)
So I've been thinking a little more about alternative representations.
Given the following XML (which I have working on my penrose clone)
<module name="org.astro" version="1.2"
xmlns="http://www.jcp.org/xmlns/modules/v1.0">
<exports name="org.astro.somepackage"/>
</module>
module {
"name" : "org.astro",
"version" : "1.2",
"exports" : [
{ "name" : "org.astro.somepackage"}
]
}
I don't think the core contains a JSON parser, but again it should be
possible to write a really small one (possibly by leveraging the
java.util.Scanner class). Using JSON one can then add extensibility by
introducing the idea of namespaces. I don't think this is standard
JSON at this point in time, but I have seen people do it in the
module {
"name" : "org.astro",
"version" : "1.2",
"exports" : [
{ "name" : "org.astro.somepackage", "org.osgi.version" : "1.0" }
]
}
I'd be happy to investigate the actual realization of this through
JSON if people think that's better than XML and MANIFEST.MF...
Cheers,
David

+1

I think JSON is more readable than XML. And as MANIFEST.MF has
limitations, it seems not to be a good choice either. YAML is another
choice we can consider. The module info will be something like this:
module:
name: org.astro
version: 1.2
exports:
name: org.astro.somepackage
org.osgi.version: 1.0

--
Regards,

Shi Jun Zhang

Glyn Normington

2012-04-13 10:23:49 UTC

Post by Shi Jun Zhang

Post by RÃ©mi Forax
As I already said, the best is to develop a small parser
that is able to parse a module-info.class and provide
XML events or properties.

FWIW, ModuleSystem already defines a parseModuleInfo method for parsing
module-info.class files.

Post by Alan Bateman
On XML then the current prototype base module does not include JAXP or any
XML support.

Right, so if we go with XML we'd have to embed a tiny parser that is
just good enough to read the XML definition, which isn't hard to do. I
have currently an implementation that (when jarred-up) is about 85kb,
but as mentioned before it should be possible to optimize that into
something smaller.
I've been thinking about alternative representations. Although I think
that using MANIFEST.MF is better than using module-info.class, there
are still some issues with it: the fixed line-length and other
esoteric formatting rules and the fact that when using it one
generally needs a second-level parser to parse the value of a header,
I mean the JRE Manifest parser could return the following from an OSGi
Import-Package header,
org.osgi.service.log;version="[1.3,2)",org.osgi.util.tracker;version="[1.4,2)"
but that will then need to be fed to another parser to be processed.
It would be nice that whatever format we come up with can use a single
parser to do all the parsing for us :)
So I've been thinking a little more about alternative representations.
Given the following XML (which I have working on my penrose clone)
<module name="org.astro" version="1.2"
xmlns="http://www.jcp.org/xmlns/modules/v1.0">
<exports name="org.astro.somepackage"/>
</module>
module {
"name" : "org.astro",
"version" : "1.2",
"exports" : [
{ "name" : "org.astro.somepackage"}
]
}
I don't think the core contains a JSON parser, but again it should be
possible to write a really small one (possibly by leveraging the
java.util.Scanner class). Using JSON one can then add extensibility by
introducing the idea of namespaces. I don't think this is standard
JSON at this point in time, but I have seen people do it in the
module {
"name" : "org.astro",
"version" : "1.2",
"exports" : [
{ "name" : "org.astro.somepackage", "org.osgi.version" : "1.0" }
]
}
I'd be happy to investigate the actual realization of this through
JSON if people think that's better than XML and MANIFEST.MF...
Cheers,
David

+1
name: org.astro
version: 1.2
name: org.astro.somepackage
org.osgi.version: 1.0
--
Regards,
Shi Jun Zhang

Any text solution beats a binary file.

JSON seems preferable to XML. I think YAML is not as suitable as JSON as it's harder to parse and perhaps more error-prone when hand-written.

Regards,
Glyn

Debasish Ray Chawdhuri

2012-04-13 10:33:48 UTC

I agree with Glyn. Indentation based hierarchy will be error prone when
hand written.

--
Debasish Ray Chawdhuri
<http://www.geekyarticles.com/>

Neil Bartlett

2012-04-13 11:10:53 UTC

+1 for JSON as a first choice, XML as a second if a small enough parser can be found/written to fit into the core.

-1 for YAML and other significant-whitespace formats; I agree with the comments about fragility.

-1000 for module-info.class and other binary formats. The seems like a no-brainer to me. The only reason to use it appears to be that the parser is already present in the core, though as David points out that parser is ordinarily not accessible to applications.

Post by Debasish Ray Chawdhuri
I agree with Glyn. Indentation based hierarchy will be error prone when
hand written.
--
Debasish Ray Chawdhuri
<http://www.geekyarticles.com/>

Jeff MAURY

2012-04-13 22:16:39 UTC

+1 for JSON

Jeff

Post by Neil Bartlett
+1 for JSON as a first choice, XML as a second if a small enough parser
can be found/written to fit into the core.
-1 for YAML and other significant-whitespace formats; I agree with the
comments about fragility.
-1000 for module-info.class and other binary formats. The seems like a
no-brainer to me. The only reason to use it appears to be that the parser
is already present in the core, though as David points out that parser is
ordinarily not accessible to applications.

Post by Debasish Ray Chawdhuri
I agree with Glyn. Indentation based hierarchy will be error prone when
hand written.
--
Debasish Ray Chawdhuri
<http://www.geekyarticles.com/>

--
Jeff MAURY

"Legacy code" often differs from its suggested alternative by actually
working and scaling.
- Bjarne Stroustrup

http://www.jeffmaury.com
http://riadiscuss.jeffmaury.com
http://www.twitter.com/jeffmaury

Rémi Forax

2012-04-13 22:39:00 UTC

Post by Neil Bartlett
+1 for JSON as a first choice, XML as a second if a small enough parser can be found/written to fit into the core.
-1 for YAML and other significant-whitespace formats; I agree with the comments about fragility.
-1000 for module-info.class and other binary formats. The seems like a no-brainer to me. The only reason to use it appears to be that the parser is already present in the core, though as David points out that parser is ordinarily not accessible to applications.

Please re-read the mail of Alan.
and we can't use JSON because there is no way to specify the character
encoding.

R?mi

Jeff MAURY

2012-04-13 23:30:47 UTC

Why don't you force a specific encoding for this file (UTF-8?) ?
This should be the responsability of the compiler to generate the expected
encoding

Jeff

Please re-read the mail of Alan.
and we can't use JSON because there is no way to specify the character
encoding.
R?mi

Rémi Forax

2012-04-13 23:44:34 UTC

Post by Jeff MAURY
Why don't you force a specific encoding for this file (UTF-8?) ?
This should be the responsability of the compiler to generate the
expected encoding
Jeff

because UTF-8 is not the default encoding on some platforms.
I think people still want to use their Mac, by example, to code in Java.

R?mi

Post by Jeff MAURY
On Sat, Apr 14, 2012 at 12:39 AM, R?mi Forax <forax at univ-mlv.fr
+1 for JSON as a first choice, XML as a second if a small
enough parser can be found/written to fit into the core.
-1 for YAML and other significant-whitespace formats; I agree
with the comments about fragility.
-1000 for module-info.class and other binary formats. The
seems like a no-brainer to me. The only reason to use it
appears to be that the parser is already present in the core,
though as David points out that parser is ordinarily not
accessible to applications.
Please re-read the mail of Alan.
and we can't use JSON because there is no way to specify the
character encoding.
R?mi
--
Jeff MAURY
"Legacy code" often differs from its suggested alternative by actually
working and scaling.
- Bjarne Stroustrup
http://www.jeffmaury.com
http://riadiscuss.jeffmaury.com
http://www.twitter.com/jeffmaury

Jesse Glick

2012-04-14 01:06:57 UTC

Post by RÃ©mi Forax
UTF-8 is not the default encoding on some platforms.
I think people still want to use their Mac, by example, to code in Java.

In the rare case of someone using a computer with a legacy platform encoding who is editing a module descriptor using a plain text editor (rather than a Jigsaw-aware IDE
of some kind) and using non-ASCII characters in identifiers such as for modules/types/packages, it does not seem a major imposition for them to set their editor to use
UTF-8 for this file.

Keep in mind that JAR manifests have been mandated to use UTF-8 for years; I have never heard of anyone clamoring to use, say, ISO-8859-2 to specify the value of Main-Class.

Debasish Ray Chawdhuri

2012-04-14 01:49:11 UTC

1. How do your java files specify encoding? Similarly we might try to
externally specify encoding in case it is not our default UTF-8 with some
form of -encoding option.

2. JSON can specify encoding if we modify the parser a little. I guess we
would have to find something with a license compatible with GPL. We can
modify it to read an optional first line { "encoding":"UTF-8", if default
encoding is not used. But I still would prefer the external option being
passed to java compiler and java runtime.

3. The reason people do not complain about MANIFEST.MF being always in
UTF-8 is that people hardly ever try to have class names and package names
in non-ASCII characters, the encoding support for java files in mainly for
string literals.

It certainly does not make sense to have a complex parser like XML parser
just because we want to support different encoding and we want the class
names, package names and module names in non-ASCII characters.

--
Debasish Ray Chawdhuri
<http://www.geekyarticles.com/>

Jeff MAURY

2012-04-14 08:08:35 UTC

Even if the module file is using a specific encoding, the
compilation/packaging of the jmod file can convert the file to the expected
encoding just like what is done for .java --> .class files and probably
what is done for MANIFEST.MF as well.

Jeff

On Sat, Apr 14, 2012 at 3:49 AM, Debasish Ray Chawdhuri <

Post by Debasish Ray Chawdhuri
1. How do your java files specify encoding? Similarly we might try to
externally specify encoding in case it is not our default UTF-8 with some
form of -encoding option.
2. JSON can specify encoding if we modify the parser a little. I guess we
would have to find something with a license compatible with GPL. We can
modify it to read an optional first line { "encoding":"UTF-8", if default
encoding is not used. But I still would prefer the external option being
passed to java compiler and java runtime.
3. The reason people do not complain about MANIFEST.MF being always in
UTF-8 is that people hardly ever try to have class names and package names
in non-ASCII characters, the encoding support for java files in mainly for
string literals.
It certainly does not make sense to have a complex parser like XML parser
just because we want to support different encoding and we want the class
names, package names and module names in non-ASCII characters.
--
Debasish Ray Chawdhuri
<http://www.geekyarticles.com/>

David Bosschaert

2012-04-17 07:09:00 UTC

Seems like there is at least a majority for going with JSON as a
format. I'm not an expert in encodings but to me it looks like it
could be solved in the same way as currently done for the MANIFEST.MF.
I'll start prototyping the JSON module descriptor in Penrose.

Best regards,

David

Post by Jeff MAURY
Even if the module file is using a specific encoding, the
compilation/packaging of the jmod file can convert the file to the expected
encoding just like what is done for .java --> .class files and probably
what is done for MANIFEST.MF as well.
Jeff
On Sat, Apr 14, 2012 at 3:49 AM, Debasish Ray Chawdhuri <

Post by Debasish Ray Chawdhuri
1. How do your java files specify encoding? Similarly we might try to
externally specify encoding in case it is not our default UTF-8 with some
form of -encoding option.
2. JSON can specify encoding if we modify the parser a little. I guess we
would have to find something with a license compatible with GPL. We can
modify it to read an optional first line { "encoding":"UTF-8", ?if default
encoding is not used. But I still would prefer the external option being
passed to java compiler and java runtime.
3. The reason people do not complain about MANIFEST.MF being always in
UTF-8 is that people hardly ever try to have class names and package names
in non-ASCII characters, the encoding support for java files in mainly for
string literals.
It certainly does not make sense to have a complex parser like XML parser
just because we want to support different encoding and we want the class
names, package names and module names in non-ASCII characters.
--
Debasish Ray Chawdhuri
<http://www.geekyarticles.com/>

--
Jeff MAURY
"Legacy code" often differs from its suggested alternative by actually
working and scaling.
?- Bjarne Stroustrup
http://www.jeffmaury.com
http://riadiscuss.jeffmaury.com
http://www.twitter.com/jeffmaury

Alan Bateman

2012-04-17 09:22:56 UTC

Post by David Bosschaert
Seems like there is at least a majority for going with JSON as a
format. I'm not an expert in encodings but to me it looks like it
could be solved in the same way as currently done for the MANIFEST.MF.
I'll start prototyping the JSON module descriptor in Penrose.

None of my business as to the format that you use but I hope that
developers will be strongly discouraged from editing the module
declarations post compilation, otherwise it may result in issues at
runtime that are hard to diagnose. Note that I'm not talking about the
custom meta-data as I don't know if that can be validated at build time.
I'm also not talking about tool agents as they may need to augment the
access control rules for instrumentation purposes.

-Alan

Debasish Ray Chawdhuri

2012-04-17 15:36:58 UTC

If JSON is used, it would not be compiled, the source will contain the same
file I guess. Also even if compilation is done, it would give the errors
related to the syntax of the file. The errors can be made readable. If we
want to discourage editing the file, we should stick to binary format.

developers will be strongly discouraged from editing the module
declarations post compilation, otherwise it may result in issues at runtime
that are hard to diagnose. Note that I'm not talking about the custom
meta-data as I don't know if that can be validated at build time. I'm also
not talking about tool agents as they may need to augment the access
control rules for instrumentation purposes.
-Alan

--
Debasish Ray Chawdhuri
http://www.geekyarticles.com/
[A collection of advanced articles on java]

Eric Johnson

2012-04-17 20:20:12 UTC

Post by Debasish Ray Chawdhuri

Post by Debasish Ray Chawdhuri
If JSON is used, it would not be compiled, the source will contain the same
file I guess. Also even if compilation is done, it would give the errors
related to the syntax of the file. The errors can be made readable. If we
want to discourage editing the file, we should stick to binary format.

That last statement ("if we want to discourage....") seems a stretch.
Precisely one of the values of a textual format is its accessibility for
all sort of purposes that involve reading the file. Why would you want
to take that away?

In already compiled JARs, has rampant editing of MANIFEST.MF files ever
been a problem in the history of Java? How about for the history of
OSGi? That seems evidence enough that developers are already
appropriately disinclined to directly edit files of this ilk because of
the zipped archive that typically carries this kind of data.

-Eric.

Debasish Ray Chawdhuri

2012-04-18 01:12:26 UTC

I would not, it was in answer to the previous post by Alan. (And I
mentioned the post with my post.

Post by Eric Johnson

That last statement ("if we want to discourage....") seems a stretch.
Precisely one of the values of a textual format is its accessibility for
all sort of purposes that involve reading the file. Why would you want to
take that away?
In already compiled JARs, has rampant editing of MANIFEST.MF files ever
been a problem in the history of Java? How about for the history of OSGi?
That seems evidence enough that developers are already appropriately
disinclined to directly edit files of this ilk because of the zipped
archive that typically carries this kind of data.
-Eric.

Post by Debasish Ray Chawdhuri
On Tue, Apr 17, 2012 at 2:52 PM, Alan Bateman<Alan.Bateman at oracle.**com<Alan.Bateman at oracle.com>

Post by David Bosschaert
Seems like there is at least a majority for going with JSON as a

Post by David Bosschaert
format. I'm not an expert in encodings but to me it looks like it
could be solved in the same way as currently done for the MANIFEST.MF.
I'll start prototyping the JSON module descriptor in Penrose.
None of my business as to the format that you use but I hope that

--
Debasish Ray Chawdhuri
http://www.geekyarticles.com/
[A collection of advanced articles on java]

David Bosschaert

2012-04-19 12:09:00 UTC

For those who are interested, I've just posted a RFR with an initial
JSON-based implementation on penrose-dev:
http://mail.openjdk.java.net/pipermail/penrose-dev/2012-April/000049.html

Best regards,

David

Post by Debasish Ray Chawdhuri
1. How do your java files specify encoding? Similarly we might try to
externally specify encoding in case it is not our default UTF-8 with some
form of -encoding option.
2. JSON can specify encoding if we modify the parser a little. I guess we
would have to find something with a license compatible with GPL. We can
modify it to read an optional first line { "encoding":"UTF-8", ?if default
encoding is not used. But I still would prefer the external option being
passed to java compiler and java runtime.
3. The reason people do not complain about MANIFEST.MF being always in
UTF-8 is that people hardly ever try to have class names and package names
in non-ASCII characters, the encoding support for java files in mainly for
string literals.
It certainly does not make sense to have a complex parser like XML parser
just because we want to support different encoding and we want the class
names, package names and module names in non-ASCII characters.
--
Debasish Ray Chawdhuri
<http://www.geekyarticles.com/>

Peter Kriens

2012-01-05 13:23:55 UTC

Post by RÃ©mi Forax
The .class format is already an extensible versioned container format
with a dictionnary which is able to store string and class name.
No shoehorn here.

Hmmmmm. To me this is the biggest shoehorn I've seen in a long, long time ... Would you ever, ever, use this format if it had not been there before? Is there any reason to use the class file except that it happens already be there?

The worst thing is that you could view a class as a container of members, which on a higher level is also true for a module. However, the way the class format is used is ignoring this possibility and only (ab)uses the constant pool and the type's (!) attributes. Further:

a) It is not easy readable for inspection
c) You have to stopgap mandatory fields like super class, access, class name, etc.
d) It will confuse the heck out of any existing class parsers
e) It is not easy extendable for other module systems
f) There is just no need to use the class file
h) It unnecessarily complicates management systems

This is an almost perfect example of Conways law pointed out by Gafter. Once you're familiar with a technology you frame solutions in that technology.

Kind regards,

Peter Kriens

Well, ahem, requiring people to use ASM isn't really user-friendly IMHO...

Don't ask me if ASM use user friendly :)
http://asm.ow2.org/team.html

Post by David Bosschaert
David

R?mi

Rémi Forax

2012-01-05 14:14:21 UTC

Post by Peter Kriens

Post by RÃ©mi Forax
The .class format is already an extensible versioned container format
with a dictionnary which is able to store string and class name.
No shoehorn here.

javap module-info.class

Post by Peter Kriens
c) You have to stopgap mandatory fields like super class, access, class name, etc.

No, the access_flags will indicate that it's a module descriptor.

Post by Peter Kriens
d) It will confuse the heck out of any existing class parsers

No, the classfile is versioned. It has already changed twice (1.5 and 1.7)

Post by Peter Kriens
e) It is not easy extendable for other module systems

The attribute_info let you store any attributes you want.

Post by Peter Kriens
f) There is just no need to use the class file

a circular argument, sorry my brain overflow.

Post by Peter Kriens
h) It unnecessarily complicates management systems

why ?

Post by Peter Kriens
This is an almost perfect example of Conways law pointed out by Gafter. Once you're familiar with a technology you frame solutions in that technology.

A module descriptor has to be versioned, extensible, to contains
synthetized values (tagged as such),
be validated (by at least checking that referenced modules, packages and
types exist),
in my opinion, that's match what the classfile format is.

If you prefer another format, I have no problem but, in my opinion,
it will be enough close to the classfile format to not worth to create a
new format.

Post by Peter Kriens
Kind regards,
Peter Kriens

cheers,
R?mi

Eric Johnson

2012-01-05 16:05:03 UTC

At the risk of tilting at windmills, I'm going to try two responses
here. First this email.

Consider an alternate straw-man, wherein the module descriptor uses XML,
let's see how the responses change based on what R?mi proposes. Counter
responses sprinkled throughout.

<strawman>

Post by Peter Kriens

Post by RÃ©mi Forax
The .class format is already an extensible versioned container format
with a dictionnary which is able to store string and class name.
No shoehorn here.

javap module-info.class

cat module-info.xml (Java classfile tools need not be used - simpler!)

Post by Peter Kriens
c) You have to stopgap mandatory fields like super class, access, class name, etc.

No, the access_flags will indicate that it's a module descriptor.

The root element of the XML file will indicate it is a module
descriptor. If you add namespaces, a schema-aware XML editor can even
validate conformance of the "syntax" of expected elements, if not the
semantics.

Post by Peter Kriens
d) It will confuse the heck out of any existing class parsers

No, the classfile is versioned. It has already changed twice (1.5 and 1.7)

As a separate XML file, no existing class parsers need to be updated at all.

Post by Peter Kriens
e) It is not easy extendable for other module systems

The attribute_info let you store any attributes you want.

The attributes of XML elements can let you store any additional data you
want.

Post by Peter Kriens
f) There is just no need to use the class file

a circular argument, sorry my brain overflow.

Your argument seems to be "I have a hammer, therefore this is a nail" -
I have a container file format, therefore this module data can fit into it.

In other words, the argument is not circular. Peter and I are merely
pointing out that the use of a .class file, or even just a .class file
format under a different name is not a foregone conclusion based on the
requirements stated so far.

Post by Peter Kriens
h) It unnecessarily complicates management systems

why ?

Myriad tools can be used to edit/update an XML format file, not just the
specific tools designed to work with Java class files. The fact that you
*also* insist that it also end with ".class" means that all tools that
look for class files will now have to pay attention to when to ignore
these files. Examples include obfuscators, byte-code analysis tools, etc.

Post by Peter Kriens
This is an almost perfect example of Conways law pointed out by
Gafter. Once you're familiar with a technology you frame solutions in
that technology.

An XML file can be versioned, is extensible, can contain synthesized
values (which can be tagged as such), can be validated by a schema, and
use tools like SchemaTron to further check semantics. In my opinion
that's a perfect fit for schema-constrained XML file.

Post by RÃ©mi Forax
If you prefer another format, I have no problem but, in my opinion,
it will be enough close to the classfile format to not worth to create
a new format.

And in my opinion, you're overloading an existing container format with
surprising new content, when an existing meta-format works better for
myriad uses.

</strawman>

The above isn't all to argue that XML is the best choice here, merely
that it seems to work to counter the arguments put forth so far that the
classfile is the ideal format. At a minimum, the file deserves a
different extension so that existing tools that only care about actual
classes don't read it for byte codes.

-Eric.

Stephen McConnell

2012-01-04 14:50:45 UTC

Woops - sorry, need clarification.
Are your suggesting that the language is frozen and cannot evolve?
/SJM

-----Original Message-----
From: David Bosschaert
Sent: Thursday, January 05, 2012 12:27 AM
To: jigsaw-dev at openjdk.java.net
Subject: Re: What is a language construct? (was Re: Comment on state
ofProject Jigsaw)

I'd say these presentations represent only a limited section of the
community.
There have been many arguments made by a variety of people arguing for
a DSL or other mechanism which is more suited for storing module
declarations. This email trail is an example of that.

Just to give one example where it doesn't fit:
A module declaration applies to the artifact as a whole, i.e. the
module jar file, so it doesn't fit naturally alongside other java
classes in any java file - it needs to be in a special file of its
own. And that special file will look different from all the other java
files so hence it doesn't make sense to make it part of the java
syntax as this syntax is only used in this special place, effectively
creating a separate language, as Neal mentioned above.
Making things worse, having it in a .java file implies that it will be
compiled into a .class file which makes thing unnecessarily complex
for tools and other module systems to read this thing at runtime.
Additionally, having it in a .class file compromises the extensibility
of the module metadata, which is needed for module systems that may
have more features and need extensions to JSE module metadata.

David Bosschaert

2012-01-04 15:17:27 UTC

Not at all! I'm a big fan of the many features that are being worked
on for the language.

However, I think that specifying the module metadata in a Java source
file is the wrong choice and especially having it in a .class file at
runtime seems highly limiting and only focused on one particular
use-case. This type of information asks for a separate type of file
with its own format: a little DSL or something like that. It should
not be precompiled as a .class file for one particular use but rather
stay in its original format inside the module.
Java runtimes can, as a performance optimization, cache their own
binary version of it if they wish, but that's really a hidden
implementation detail...

Best regards,

David

Post by Stephen McConnell
Woops - sorry, need clarification.
Are your suggesting that the language is frozen and cannot evolve?
/SJM
-----Original Message----- From: David Bosschaert
Sent: Thursday, January 05, 2012 12:27 AM
To: jigsaw-dev at openjdk.java.net
Subject: Re: What is a language construct? (was Re: Comment on state
ofProject Jigsaw)

I'd say these presentations represent only a limited section of the community.
There have been many arguments made by a variety of people arguing for
a DSL or other mechanism which is more suited for storing module
declarations. This email trail is an example of that.
A module declaration applies to the artifact as a whole, i.e. the
module jar file, so it doesn't fit naturally alongside other java
classes in any java file - it needs to be in a special file of its
own. And that special file will look different from all the other java
files so hence it doesn't make sense to make it part of the java
syntax as this syntax is only used in this special place, effectively
creating a separate language, as Neal mentioned above.
Making things worse, having it in a .java file implies that it will be
compiled into a .class file which makes thing unnecessarily complex
for tools and other module systems to read this thing at runtime.
Additionally, having it in a .class file compromises the extensibility
of the module metadata, which is needed for module systems that may
have more features and need extensions to JSE module metadata.

Neal Gafter

2011-12-21 23:50:21 UTC

I know how to put package (and import) declarations in a Java source file
such that they apply to types declared in that source file.

There does not appear to be a way to put a module declaration in a Java
source file such that the module declaration applies to types declared in
that source file.

Module declarations are therefore very much less intermixed than package,
import, and type declarations.

Post by Neal Gafter
A language construct is a syntactic construct that appears within source

files

Post by Neal Gafter
of the programming language.
In this case (if I understand it) the language "addition" is distinct

from and

Post by Neal Gafter
never intermixed with syntax of the underlying programming language. It

Post by Neal Gafter
therefore a distinct language.

In purely syntactic terms module declarations are no less intermixed than
package, import, or type declarations [1], and I don't think anyone would
argue that those are not part of the language.
- Mark
[1] http://openjdk.java.net/projects/jigsaw/doc/lang-vm.html#jigsaw-1.3.2

Peter Kriens

2011-12-23 09:58:41 UTC

I think at JavaOne 2006 Gilad Bracha claimed that he did not trust the deployment engineers to touch his precious language and spun off JSR 294 from JSR 277 for exactly this reason :-)

And I actually now do agree with him after watching the story of Java modularity story for the last 6 years ... :-(

The purity of the separation between JLS and implementation of the environment has tremendous value in reducing conceptual complexity. Purity seems to have gotten a bad name but too many people are doing shortcuts only to get bitten later. Using a dummy class file's attributes to store Jigsaw data could be the archetypical example of Conway's law.

That said, I think there are two issues that are not always obvious to observers.

First, Jigsaw attempts to address the class path assembly problem (maven, unix package dependency managers) and not the software complexity problem. It fights for split packages and module friends, clearly the antithesis of modularity as a tool to reduce software complexity. With Jigsaw it will be easy to call your existing code modular but it will provide few of its benefits since these benefits require unfettered encapsulation.

Second, confusingly the JLS already has modules, they're called packages ... the only complaint I hear is about the granularity of packages, it is often too small.

This could easily be fixed with a module concept that leverages the package/type namespace. I.e. it encapsulates a set of packages that are below it in the namespace. The normal access keywords plus an additional module keyword can then be used to encapsulate packages and types.

module org.foo;
package bar;
class Baz {} org.foo.bar.Baz

One could call this concept super packages ... :-) Anyway, maybe in another life.

Kind regards,

Peter Kriens

Post by Neal Gafter
My main high-level comment on project Jigsaw is that is fails its first
principle: modularity appears in the draft not to be a language construct,
but at best a completely separate language from the Java programming
language. This appears to be an example of Conway's
law<http://en.wikipedia.org/wiki/Conway's_Law>
.
If you think this is the best approach, then it is not clear why it is
described as part of the Java programming language at all.

mark.reinhold

2011-12-23 21:26:22 UTC