Default enum literals for Xtext generated EMF models

Xtext allows to easily build fully-fledged editors for domain specific languages (DSL) which are based on the Eclipse Modeling Framework (EMF). While existing Ecore models may be used, Xtext is also able to infer an Ecore model from the Xtext grammar. For complex Ecore models, it should certainly be considered to manage them independently, e.g. with Xcore. However, a few simple tweaks of grammar rules can influence the automatically generated Ecore model quite remarkably (e.g. the inheritance hierarchy) and may delay the point in time where it may be better to manage them independently. This example shows how to influence the generation of enumerators (enums) and default literals with grammar rules.

The code example below assumes that there's a model element with an optional "visibility" attribute of type enum and that the DSL must be able to capture the case where no visibility enum at all was given.

As the default literal of a generated enum is the first literal, an enum rule is added to the grammar which uses, say, "unspecified" for that. However, the user of the DSL editor should simply be able to enter nothing at all instead of "unspecified" which would look odd in the editor. This can be achieved by adding a second rule which returns the aforementioned enum, but only contains the literals allowed for the user. Please note that the order of the enum rules is important for generated models.

MyModelElement:
 // ...
 // user may specify enum or nothing at all
 visibility=Visibility? 
 // ...
;

// rule for the generated Ecore model
enum VisibilityEnum:
 unspecified | // first literal is the default
 public |
 private
;

// rule for the DSL editor
enum Visibility returns VisibilityEnum:
 public | 
 private
;

Multiple Validators in Xtext

Early validation of domain specific languages (DSLs) and suggestions on how to fix these errors ('quickfixes') are key for a good end user experience and for avoiding problems in the toolchain, for instance in generators that use DSL models: The more editor validations, the better. This description shows how to improve the maintainability and modularity of validation code for Xtext DSLs by simply splitting them. In the generated [MyDsl]JavaValidator, an annotation referring to other custom validators has to be added as shown below:

@ComposedChecks(validators =
        {MyCustomValidator.class, MyOthercustomValidator.class})
public class MyDslJavaValidator extends AbstractMyDslJavaValidator {

    // check method as usual inside the generated Java Validator
    @Check
    public void checkMyDslElement(MyEntity myEntity) { // ...

Please note that the custom validator has to override the register()-method. Apart from that, the checks look exactly as they would if they were in the generated validator.

// Example for a split custom validator written in Xtend
public class MyCustomValidator extends AbstractDeclarativeValidator {

    override register(EValidatorRegistrar registrar) {
        //not needed for classes used as ComposedCheck
    }

    // additional check method in separate validator
    @Check
    def void checkMyDslElement(MyEntity myEntity) {
       [...] // validation code

Clean Eclipse Preferences Tree with Multiple DSLs

As it is easy to build new domain specific languages (DSLs) with Xtext for Eclipse, enterprise projects may have multiple languages, which are usually related. Each language comes with its own Eclipse preference page, which are by default mixed with other preference pages. A simple way to keep the Eclipse preferences tree clean is to add a root page which aggregates the preference pages for individual (but possibly related) Xtext languages. This can be done by adding the snippet below to a plugin.xml file, for instance to the one of the base DSL's UI project.

<extension
        point="org.eclipse.ui.preferencePages">
    <page
            class="my.dsl.ui.MyDslExecutableExtensionFactory:org.eclipse.xtext.ui.editor.preferences.LanguageRootPreferencePage"
            id="my.id.root.ui"
            name="My DSLs">
        <keywordReference id="my.id.root.ui.keyword_root_pref"/>
    </page>
</extension>

<!-- add keywords for the search in the preferences page -->
<extension
point="org.eclipse.ui.keywords">
<keyword
        id="my.id.root.ui.keyword_root_pref"
        label="other keywords"/>
</extension>

Now, the id of the root page, in this case my.id.root.ui just has to be added to the plugin.xml files of the UI projects of the languages whose preferences should be aggregated. This can also be done in the graphical plugin.xml editor of the DSL's UI projects by navigating to the tab Extensions, selecting the first child node under org.eclipse.ui.preferencePages (which should be the DSL preference page) and pasting my.id.root.ui into the category text box.

Custom Syntax Error Messages with Quick Fix

Xtext editors for domain specific languages (DSLs) provide many error messages out of the box, such as syntactical errors, duplicate name errors or unresolvable references. For an improved user experience, some technical error messages from the editor (or, more specifically, from the Antlr parser that is used by the editor) may be customized. In many DSLs, identifiers (for DSL concepts like packages, entities and so on) are expected to conform to the regular expression of the terminal rule ID:

terminal ID : '^'?('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'_'|'0'..'9')*;

In addition, any keyword that is defined in other rules of the DSL grammar may not be used as an identifier. Keywords may be escaped with the caret (^) symbol, which is certainly arguable. This would be similar to using "class" as a name for a class in Java (if that would be possible). Here are some snippets of a DSL with a package concept:

package package // the second word is a reserved keyword and therefore not be valid as identifier
package ^package // okay, the keyword was escaped
package myPackage // okay unless 'myPackage' is a grammar keyword

The default error message when using a reserved keyword where an identifier is expected looks like this.

mismatched input 'package' expecting RULE_ID

This  message can be customized using Xtext's SyntaxErrorMessageProvider (written in Xtend):

class SyntaxErrorMessageProviderCustom extends SyntaxErrorMessageProvider { public static val String USED_RESERVED_KEYWORD = "USED_RESERVED_KEYWORD" @Inject IGrammarAccess grammarAccess /** * Customized error message for reserved keywords */ override getSyntaxErrorMessage(IParserErrorContext context) { val unexpectedText = context?.recognitionException?.token?.text if (GrammarUtil::getAllKeywords(grammarAccess.getGrammar()).contains(unexpectedText)) { println(context.defaultMessage) return new SyntaxErrorMessage(''' "«unexpectedText»" is a reserved keyword which is not allowed as Identifier. Please choose another word or alternatively confuse your co-workers by escaping it with the caret (^) character like this: "^«unexpectedText»".''', USED_RESERVED_KEYWORD) } super.getSyntaxErrorMessage(context) } }

The customized error message provider has to be bound in MyDslRuntimeModule.java like this:

/** * custom error messages for syntax errors */ public Class<X extends ISyntaxErrorMessageProvider> bindISyntaxErrorMessageProvider() { return SyntaxErrorMessageProviderCustom.class; }

A simple quickfix in MyDslQuickfixProvider could look like this:

/** * Provide a fix when reserved keywords are used as identifiers */ @Fix(SyntaxErrorMessageProviderCustom::USED_RESERVED_KEYWORD) def public void reservedKeywordUsed(Issue issue, IssueResolutionAcceptor acceptor) { val unexpectedText = issue.data?.get(0) acceptor.accept(issue, '''Change '«unexpectedText»' to '«unexpectedText.generateUniqueIdentifier».' ''', ''' Change '«unexpectedText»' to '«unexpectedText.generateUniqueIdentifier»', which is not a reserved keyword.''', "correction_linked_rename.gif", [ IModificationContext context | val xtextDocument = context.getXtextDocument xtextDocument.replace(issue.offset, issue.length, unexpectedText.generateUniqueIdentifier) ]) } def String generateUniqueIdentifier(String it) { val candidate = 'my' + it?.toFirstUpper?:'Name' var count = 1 val reserved = GrammarUtil::getAllKeywords(grammarAccess.getGrammar()) if (reserved.contains(candidate)) { while (reserved.contains(candidate + count)) { count = count + 1 } return candidate + count } return candidate }

This kind of customization has been available for a long time now. For more information, see Customizing error messages from Sebastian Zarnekow.

Xtext Grammar Visualization

When developing EBNF-style grammars, e.g. for Xtext, syntax graphs can be a great help. One of the new features in Xtext 2 is the Xtext Syntax Graph which can be selected in the Eclipse menu under Views - Xtext - Xtext Syntax Graph. In case the grammar has ambiguities, users get an error message similar to the following when trying to generate the code for the grammar by running the MWE2 workflow file.

Decision can match input such as [...] using multiple alternatives 1,2

If the error isn't obvious, the alternatives can be displayed graphically with ANTLRWorks (the graph analysis can be expensive and thus is not included in the standard Xtext Syntax Graph). Here is an example of an Eclipse Xtext project with a simple, ambiguous grammar and the Xtext Syntax Graph.

To analyze the grammar in ANTLRWorks, the executable jar file from the ANTLR website can be downloaded and run using the JRE (java -jar antlrworks-1.x.x.jar). ANTLRWorks expects the ANTLR grammar file from the Xtext project, which is called Internal[...].g. A debug-friendly version of this file may be generated by adding fragment = parser.antlr.DebugAntlrGeneratorFragment { } to your MWE2 workflow file and running it again.  The file is located in the Eclipse Xtext project in the src-gen folder in the [...].antlr.internal package (see first screenshot).
After opening it, the grammar can be checked by selecting Grammar - Check Grammar (Ctrl-R) from the menu. It will show an error message. Selecting the incorrect rule ruleModel and ticking both checkboxes for the alternatives (on the lower right on the screenshot) shows the ambiguity graphically.

Quickly formatting DSLs with Xtext

One typical task of developing new domain specific languages with Xtext is to customize the formatting in order to have a nice text layout. If the majority of grammar keywords should be formatted in the same way and only a few in a specific way, a generic formatter could be used. As an example, for DSLs with many key-value pairs (e.g. domain models), one might want to indent everything between curly braces and start a new line before each keyword. Instead of manually listing the keywords for the findKeywords()-method of the IGrammarAccess, GrammarUtil.getAllKeywords() can be used to get all the keywords, so a generic formatting method could look like this:

public class GenericFormatter {

    /**
     * In your implementation of
     * {@link org.eclipse.xtext.formatting.impl.AbstractDeclarativeFormatter#configureFormatting(org.eclipse.xtext.formatting.impl.FormattingConfig)}
     * you may call this generic formatting method first. It indents blocks between curly braces and sets a linewrap
     * before each keyword. Add your own behavior afterwards, e.g.
     * 
     * keywords = grammar.findKeywords(...);
     * for (final Keyword keyword : keywords) {
     * config.setNoLinewrap().before(keyword);
     * }
     */
    public static void genericFormatting(final FormattingConfig config, final IGrammarAccess grammar) {
        for (final Pair pair : grammar.findKeywordPairs("{", "}")) { //$NON-NLS-1$ //$NON-NLS-2$
            // a space before the first '{'
            config.setSpace(" ").before(pair.getFirst()); //$NON-NLS-1$
            // indentation between
            config.setIndentation(pair.getFirst(), pair.getSecond());
            // and a linewrap before the last '{'
            config.setLinewrap(1).before(pair.getSecond());
        }

        // linewrap before all keywords
        final Set allKeywords = GrammarUtil.getAllKeywords(grammar.getGrammar());
        final List keywords = grammar.findKeywords(allKeywords.toArray(new String[allKeywords.size()]));
        for (final Keyword keyword : keywords) {
            config.setLinewrap().before(keyword);
        }

    }
}

It could be called inside the configureFormatting()-method in [NameOfTheDSL]Formatter. Formatting code that overrides this behavior for specific keywords can be added after the call to the generic method like this:

keywords = grammar.findKeywords([keywords w/o wrap before]);
for (final Keyword keyword : keywords) {
    config.setNoLinewrap().before(keyword);
}

Migrating from Xtext 0.7.x to Xtext 1.0 in Eclipse Galileo (3.5)

Are you stuck with Eclipse Galileo in your project but still like to use the new features of Xtext 1.0? Xtext 1.0 works fine not only in the newest Eclipse Helios (3.6), but also in Eclipse Galileo (3.5). Galileo users can also benefit from Xtext 1.0 features. The eclipse plugins can be updated by using the Eclipse update mechanism and adding the Xtext update site http://download.itemis.com/updates/releases/ to the update manager.

After the update of the Xtext plugins, developers may refer to the Xtext documentation (Chapter 11. Migrating from Xtext 0.7.x to 1.0) for a manual on what has to be changed. If projects are heavily customized, the cleanest way to migrate, which is starting a new Xtext project and moving the artifacts from the old to the new projects, might not be desired and developers may prefer following the Migrating Step by Step guidelines. Before follwing these guidelines, I recommend to quickly create a sample project (New|Project...|Xtext Project) with the default settings and running the workflow (GenerateMyDsl.mwe2). This can be a handy reference, e.g. on the new folder structure for the recommended "Rename Package" refactoring. I also recommend to read section 11.2.5. Noteworthy API Changes before fixing compilation errors.

Grammars are backwards compatible. With Xtext 1.0, working on the grammar as a developer has become more convenient. Among other improvements, the content assist was enhanced and more hints for developers were added to prevent grammars with potentially unintended side-effects, e.g. if a rule contains only optional elements and may be consumed from the parser without object instantiation, the developer is given a hint to add an action that creates an object (see Actions in the Xtext documentation).

I was doing this migration for a heavily customized Xtext modeling project, and it turned out it took less effort than expected. The customer is happy with the new features of Xtext 1.0 and the complex project now loads in a bit more than half of the time it took before.