Sunday, August 14, 2011

Xtext Grammar Visualization

When developing EBNF-style grammars, e.g. for Xtext, syntax graphs can be a great help. One of the new features in Xtext 2 is the Xtext Syntax Graph which can be selected in the Eclipse menu under Views - Xtext - Xtext Syntax Graph. In case the grammar has ambiguities, users get an error message similar to the following when trying to generate the code for the grammar by running the MWE2 workflow file.

Decision can match input such as [...] using multiple alternatives 1,2

If the error isn't obvious, the alternatives can be displayed graphically with ANTLRWorks (the graph analysis can be expensive and thus is not included in the standard Xtext Syntax Graph). Here is an example of an Eclipse Xtext project with a simple, ambiguous grammar and the Xtext Syntax Graph.



To analyze the grammar in ANTLRWorks, the executable jar file from the ANTLR website can be downloaded and run using the JRE (java -jar antlrworks-1.x.x.jar). ANTLRWorks expects the ANTLR grammar file from the Xtext project, which is called Internal[...].g and located in the Eclipse Xtext project in the src-gen folder in the [...].antlr.internal package (see first screenshot). After opening it, the grammar can be checked by selecting Grammar - Check Grammar (Ctrl-R) from the menu. It will show an error message. Selecting the incorrect rule ruleModel and ticking both checkboxes for the alternatives (on the lower right on the screenshot) shows the ambiguity graphically.



Monday, February 7, 2011

Quickly formatting DSLs with Xtext

One typical task of developing new domain specific languages with Xtext is to customize the formatting in order to have a nice text layout. If the majority of grammar keywords should be formatted in the same way and only a few in a specific way, a generic formatter could be used. As an example, for DSLs with many key-value pairs (e.g. domain models), one might want to indent everything between curly braces and start a new line before each keyword. Instead of manually listing the keywords for the findKeywords()-method of the IGrammarAccess, GrammarUtil.getAllKeywords() can be used to get all the keywords, so a generic formatting method could look like this:


public class GenericFormatter {

/**
* In your implementation of
* {@link org.eclipse.xtext.formatting.impl.AbstractDeclarativeFormatter#configureFormatting(org.eclipse.xtext.formatting.impl.FormattingConfig)}
* you may call this generic formatting method first. It indents blocks between curly braces and sets a linewrap
* before each keyword. Add your own behavior afterwards, e.g.
*
*

* keywords = grammar.findKeywords(...);
* for (final Keyword keyword : keywords) {
* config.setNoLinewrap().before(keyword);
* }
*

*/
public static void genericFormatting(final FormattingConfig config, final IGrammarAccess grammar) {
for (final Pair pair : grammar.findKeywordPairs("{", "}")) { //$NON-NLS-1$ //$NON-NLS-2$
// a space before the first '{'
config.setSpace(" ").before(pair.getFirst()); //$NON-NLS-1$
// indentation between
config.setIndentation(pair.getFirst(), pair.getSecond());
// and a linewrap before the last '{'
config.setLinewrap(1).before(pair.getSecond());
}

// linewrap before all keywords
final Set allKeywords = GrammarUtil.getAllKeywords(grammar.getGrammar());
final List keywords = grammar.findKeywords(allKeywords.toArray(new String[allKeywords.size()]));
for (final Keyword keyword : keywords) {
config.setLinewrap().before(keyword);
}

}
}


It could be called inside the configureFormatting()-method in [NameOfTheDSL]Formatter. Formatting code that overrides this behavior for specific keywords can be added after the call to the generic method like this:

keywords = grammar.findKeywords([keywords w/o wrap before]);
for (final Keyword keyword : keywords) {
config.setNoLinewrap().before(keyword);
}

Friday, September 24, 2010

Migrating from Xtext 0.7.x to Xtext 1.0 in Eclipse Galileo (3.5)

Are you stuck with Eclipse Galileo in your project but still like to use the new features of Xtext 1.0? Xtext 1.0 works fine not only in the newest Eclipse Helios (3.6), but also in Eclipse Galileo (3.5). Galileo users can also benefit from Xtext 1.0 features. The eclipse plugins can be updated by using the Eclipse update mechanism and adding the Xtext update site http://download.itemis.com/updates/releases/ to the update manager.

After the update of the Xtext plugins, developers may refer to the Xtext documentation (Chapter 11. Migrating from Xtext 0.7.x to 1.0) for a manual on what has to be changed. If projects are heavily customized, the cleanest way to migrate, which is starting a new Xtext project and moving the artifacts from the old to the new projects, might not be desired and developers may prefer following the Migrating Step by Step guidelines. Before follwing these guidelines, I recommend to quickly create a sample project (New|Project...|Xtext Project) with the default settings and running the workflow (GenerateMyDsl.mwe2). This can be a handy reference, e.g. on the new folder structure for the recommended "Rename Package" refactoring. I also recommend to read section 11.2.5. Noteworthy API Changes before fixing compilation errors.

Grammars are backwards compatible. With Xtext 1.0, working on the grammar as a developer has become more convenient. Among other improvements, the content assist was enhanced and more hints for developers were added to prevent grammars with potentially unintended side-effects, e.g. if a rule contains only optional elements and may be consumed from the parser without object instantiation, the developer is given a hint to add an action that creates an object (see Actions in the Xtext documentation).

I was doing this migration for a heavily customized Xtext modeling project, and it turned out it took less effort than expected. The customer is happy with the new features of Xtext 1.0 and the complex project now loads in a bit more than half of the time it took before.

Friday, July 16, 2010

Xtext Quick Fix Variants

One of my favorite Eclipse features is the quick fix functionality. When there is an error in your source code, Eclipse may have a quick fix available - it shows up with a bulb symbol on the left editor margin and may offer you several actions to fix it. You can also use the shortcut [Ctrl-1] to activate it. Xtext ships with a quick fix API which makes it easy to provide quick fixes for your own DSL, so you can provide quick fixes for your customized validation errors and warnings, which is really nice.



Quick fix actions in Xtext may either manipulate the Xtext document directly, or modify the underlying semantic model, and Xtext takes care of changing the document. Isn't that neat? For pure text manipulation, the IModification interface can be used, while for working on the model you may implement ISemanticModification.

The example below shows you how to use those two variants. In the example, we use a simple domain specific language for a tourist guide. The model may contain an arbitrary number of cities, and each city may contain zero or more sights, where a sight has a name and a description.

We implemented two validation warnings: One checks whether the city's name starts with a capital letter, the other warns the user when a city doesn't have any sights. Quick fixes could be to capitalize the first letter (by changing the Xtext document) and, for the sake of the example, just add a generic sight, but this time by modifying the semantic model.

To associate a quick fix with a certain validation, an identification code is used, so you have to define one for your validation like this:

public static final String INVALID_NAME = "xtext.workshop.advanced.quickfix.InvalidTypeName";

The check could look like this:

public class TouristguideDslJavaValidator extends
AbstractTouristguideDslJavaValidator {
[...]
@Check
public void checkTypeNameStartsWithCapital(City city) {
if (city.getName() == null || city.getName().length() == 0)
return;
if (!Character.isUpperCase(city.getName().charAt(0))) {
warning("Name should start with a capital letter.",
TouristguideDslPackage.CITY__NAME, INVALID_NAME,
city.getName());

}
}
[...]


Note that the warning takes additional parameters (in the example only one is supplied) with "user data". Here you can supply an arbitrary number of Strings with user data that may be useful for the quick fix.

Here is a step-by-step guide:
  1. Import the plug-ins with the Touristguide Language (download examples). If you only want to try the quick fix but not implement it yourself, just download the finished projects where quick fixes are already implemented. Have a quick look at the Touristguide.xtext file in to understand how a valid .guide-file looks like.
  2. Launch an Eclipse runtime application, create an new project, a new .guide-file and test if you get two validation warnings in the Problems view (Shift-Alt-Q X) for the following text:
    city "bonn" { }
  3. Switch back to the Eclipse development environment workbench and review the validator TouristguideDslJavaValidator.java (you may cf. section 7.3 Quick Fixes of the Xtext 1.0.0 documentation)
  4. Open the file TouristguideDslQuickfixProvider.java in the ui-Project. To implement the quick fix for the capital letters, you may implement methods like the ones below.


public class TouristguideDslQuickfixProvider extends DefaultQuickfixProvider {

@Fix(TouristguideDslJavaValidator.INVALID_NAME)
public void capitalizeName(final Issue issue,
IssueResolutionAcceptor acceptor) {

// retrieve the 'user data' from the validation warning
// upcase.png ... icon to display (in the icons folder)
acceptor.accept(issue, "Capitalize name", "Capitalize the name \""
+ issue.getData()[0] + "\".",
"upcase.png", new IModification() {
public void apply(IModificationContext context)
throws BadLocationException {
IXtextDocument xtextDocument = context
.getXtextDocument();
String firstLetter = xtextDocument.get(
issue.getOffset() + 1, 1);
xtextDocument.replace(issue.getOffset() + 1, 1,
firstLetter.toUpperCase());
}
});
}

@Fix(TouristguideDslJavaValidator.CITY_NOT_INTERESTING)
public void addSightToCity(final Issue issue,
IssueResolutionAcceptor acceptor) {
acceptor.accept(issue, "Add sight to make city more interesting",
"Add a random sight, to make the city look more interesting.",
// providing null for the icon name makes Eclipse use the
// standard quick fix icon
null, new ISemanticModification() {
public void apply(EObject element,
IModificationContext context) throws Exception {
// we know that the warning applies to cities
City c = (City) element;
// programmatic modification of the model
Sight sight = TouristguideDslFactory.eINSTANCE
.createSight();
sight.setName("Central Station");
sight.setDescription("The famous central station of "
+ Strings.toFirstUpper(c.getName()) + ".");
c.getSights().add(sight);
// Xtext automatically inserts text for the above
}
});
}
}

  1. Restart the runtime workbench and have fun testing your new quick fixes :-)

Tuesday, June 1, 2010

Bonner Vortragsreihe von itemis: Nach der WM ist vor der WM

Im Rahmen der Bonner Vortragsreihe, die gerade von Hanno Wendt (itemis) organisiert wird (siehe auch die allgemeinen itemis Veranstaltungen), erzählte gestern Thomas Fricke (viadee) von seinen Erlebnissen als Mitarbeiter von Franz Beckenbauer im Organisationskomitee der WM 2006 und davon, wie man die Erkenntnisse aus Projektmanagementsicht auf seine eigenen Projekte übertragen kann. Der Vortrag war, wie der Untertitel versprach, tatsächlich ein spannender Blick hinter die Kulissen einer Fußballweltmeisterschaft. Es wurden auch einige Zahlen genannt, die belegten, dass heutzutage der Ticketverkauf bei den Einnahmen kaum noch eine Rolle spielt, da der Löwenanteil durch Medienrechte und Sponsoren erzielt wird.

Der Vortrag war auch für Nicht-Fußballenthusiasten wie mich sehr unterhaltsam und mit lustigen Fußballer-Kommentaren gewürzt wie etwa "Fußball ist wie Schach - nur ohne Würfel." Neben der Unterhaltung wurden auch einige Parallelen zum IT-Geschäft gezogen. Nicht nur die Organisation alleine, sondern auch die Einbeziehung von Emotionen haben wohl dazu geführt, dass die WM als "Sommermärchen" anstatt nur als "gut organisiert" bezeichnet wird. Beispiele dafür sind der vorherige Besuch der Teilnehmerländer von hochrangigen Vertretern des Organisationskomitees, alternative Sicherheitskonzepte anstelle von Zäunen in Stadien, mehrsprachige Freiwillige für die ausländischen Fans und die "Freundlichkeitskampagne" mit der Deutschen Zentrale für Tourismus. Eine offene Hierarchie mit transparenter Kommunikation und aufrichtigem Lob (was in Deutschland generell viel zu kurz käme) hat laut Fricke ebenfalls wesentlich zum Gelingen beigetragen.

Beruhigend war für mich zu hören, dass dort bezüglich Methoden und Werkzeugen offenbar auch nur mit Wasser gekocht wurde. Diese Einfachheit sei aber konsequent durchgezogen worden. Es seien beispielsweise keine Ausnahmen bei den regelmäßigen Treffen gemacht worden, auch wenn man versucht war, diese zu streichen, "weil es doch gerade prima läuft". Auch wurden oftmals Office-Dokumente anstelle komplizierter Werkzeuge verwendet. Im Wesentlichen wurden also einfache und altbekannte Methoden eingesetzt. Fricke nannte das "die Perfektion des Naheliegenden".