Gosu’s Secret Sauce: The Open Type System

Earlier last week we finally released Gosu to the general public, the JVM language we use for configuring our enterprise applications at Guidewire.   Gosu has been in the works for several years here beginning at the dawn of our engineering history in 2002.  Over the years the language has evolved to meet the growing demands of our applications.  Initially we created Gosu because we needed an embeddable, statically typed, scripting language for business rules.  We needed static typing so that we could 1) statically verify business logic and 2) build deterministic tooling around the language via static analysis e.g., for code completion, navigation, usage searching, refactoring, etc.  Tooling was, and is now more than ever, a vital part of our technology offering.  For instance, we don’t want to burden our users with having to commit our entire API to memory in order to be productive with our platform.  But it doesn’t end there.

Our platform, like that of most large-scale enterprise application vendors, consists of a relatively tall stack of technologies, most are developed in-house.  The stack is comprised of the following subsystems in no particular order: Database Entity/OR layer, Web UI framework, Business Rules, Security/Permissions, Web Services, Localization, Naming services, Java API, Query model, XSD/XML object models, various application-dependent models of data, and Gosu.  A primary challenge with any enterprise application is to adapt to customer requirements through configuration of its subsystems.  As I’ve stated, Gosu is the language we use to accomplish that, but how does, say, the Gosu code in the Web UI or Rules access the Entity layer and XSD-modeled information?  In other words, if a primary goal of Gosu’s static type system is to help users confidently and quickly configure our applications, how can it possibly help with domains such as these which are external to the type system?

This is where Gosu separates from the pack.  Unlike most (all?) other programming languages, Gosu’s type system is not limited to a single meta-type or type domain.  For instance, Java provides one, and only one, meta-type, namely java.lang.Class.  If you want to directly represent some other type domain to Java, you’re out of luck.  Your only options are to write a class library to access information about the domain or resort to code generation.  Neither option is desirable or even adequate in many cases, but as Java programmers we don’t know any better.  Dynamic languages such as Ruby provide varying degrees of meta-programming, which can be a powerful way of dynamically representing other type domains.  But with dynamic  languages we escape type safety, deterministic tooling, and other advantages of static typing, which defeats the purpose of our primary goal of exposing other type domains to the language.  Our customers, for instance, would be at a terrible disadvantage without the ability to statically verify usages of various domains in their application code.  Thus, Gosu’s type system must provide a similar (or better) level of flexibility provided by dynamic meta-programming, but without sacrificing static typing — a tall order.  How did we do it?

Gosu’s type system consists of a configurable number of type domains.  Each domain provides a type loader as an instance of Gosu’s ITypeLoader interface.  A type loader’s primary responsibility is to resolve a type name in its domain and return an implementation of the IType interface.  This is what is most unique about Gosu — it’s type system is open to other domains to participate with first-class representation.  For instance, Gosu does not discriminate between a Gosu Class a Java Class or Entity Type or XSD Type or what have you; they’re all just types to Gosu’s compiler.  This unique and powerful feature extends all the benefits of static typing to a potentially huge range of otherwise inaccessible domains.

Before we consider a specific domain, it is important to understand that not all type loaders in Gosu are required to produce bytecode.  Those that do, like Gosu classes, implement an interface for getting at the corresponding Java bytecode. Those that don’t provide call handlers and property accessors for reflective MethodInfo and PropertyInfo evaluation.  Note all types provide TypeInfo, see IType.getTypeInfo().  For instance, the parser works against the TypeInfo, MethodInfo, etc. abstractions as the means for a level playing field between disparate types. At runtime, however, unless a type provides a Java bytecode class the MethodInfos and PropertyInfos also are responsible for handling calls.  Not requiring bytecode at runtime accommodates a potentially much wider range of type loaders and makes it possible  to quickly prototype loaders. 

With that understanding, we can jump to some specific examples.  Let’s take a look the Gosu Class type loader. It’s job is to resolve names that correspond to the domain of Gosu classes, interfaces, enumerations, enhancements, and templates (primarily .gs* family of files).  The result is an instance of GosuClass, which implements IType etc.  One of the methods in there is getBackingClass().  Essentially, this method cooperates with Gosu’s internal Java classloader to compile the parse tree from the Gosu class to a conventional Java class.  So it follows that Gosu’s compiler performs dynamic compilation direct from source; there is no separate build process or persisted classfiles. This does not imply that the Java class loader is transient, however.  It is not.  Since Gosu is a statically typed language it’s classes are compiled to conventional bytecode and, thus, don’t require any classloader shenanigans as with most dynamically typed languages.  But I digress.

What connects the Java bytecode class to its Gosu type is the IGosuObject implementation; all Gosu classes and most other types implicitly implement this. The getIntrinsicType() method is instrumental, for example, in making reified generics work in Gosu.  An instance of a Gosu class, or any type in the entire type system for that matter, can produce its underlying type via getIntrinsicType().  And from there Gosu’s type system takes over. For instance, you can reflectively access Gosu-level type information from Java code via TypeSystem.getFromObject( anyObject ).

It’s worth repeating that Gosu classes are no more important to Gosu’s type system than any other type.  For instance, XSD types resolve and load in the same manner as Gosu classes.  Given an XSD type loader (we have one internally), Gosu code can import and reference any element type defined in an XSD file by name.  No code generation, no dynamic meta-programming, and all with static name and feature resolution.  The XSD type defines constructors, properties, and methods corresponding with elements from the schema.  In addition it provides static methods for parsing XML text, files, readers, etc.  The result is an instance of the statically named XSD type corresponding with the element name, which can be used anywhere a Gosu type can be used, because it *is* a Gosu type!  Here’s a very simple example that demonstrates the use of an XSD type:

From a sample driver.xsd file:

<xs:element name=”DriverInfo”>
 <xs:complexType>
  <xs:sequence>
   <xs:element ref=”DriversLicense” minOccurs=”0″ maxOccurs=”unbounded”/>
   <xs:element name=”PurposeUse” type=”String” minOccurs=”0″/>
   <xs:element name=”PermissionInd” type=”String” minOccurs=”0″/>
   <xs:element name=”OperatorAtFaultInd” type=”String” minOccurs=”0″/>
  </xs:sequence>
  <xs:attribute name=”id” type=”xs:ID” use=”optional”/>
 </xs:complexType>
</xs:element>
<xs:element name=”DriversLicense”>
 <xs:complexType>
  <xs:sequence>
   <xs:element name=”DriversLicenseNumber” type=”String”/>
   <xs:element name=”StateProv” type=”String” minOccurs=”0″/>
   <xs:element name=”CountryCd” type=”String” minOccurs=”0″/>
  </xs:sequence>
  <xs:attribute name=”id” type=”xs:ID” use=”optional”/>
 </xs:complexType>
</xs:element>

Sample Gosu code:

uses xsd.driver.DriverInfo
uses xsd.driver.DriversLicense
uses java.util.ArrayList

function makeSampleDriver() : DriverInfo {
  var driver = new DriverInfo(){:PurposeUse = “Truck”}
  driver.DriversLicenses = new ArrayList<DriversLicense>()
  driver.DriversLicenses.add(new DriversLicense(){:CountryCd = “US”, :StateProv = “AL”})
  return driver
}

Notice the fully qualified names of the XSD types begin with xsd.  That’s just the way the XSD loader exposes the types; a type loader can name them any way it likes, but it should do its best to avoid colliding with other namespaces by using relatively unique names.  Next you may have discovered the name of the XSD file, driver, as the namespace containing all the defined element types.  That’s also a convention the loader chooses to define.  What’s most interesting, however, is how seamlessly the XSD types integrate with Gosu.  You can’t distinguish them from Gosu classes or Java classes, and that’s the whole idea!  All type domains are created equally.  Notice you can even parameterize Java’s ArrayList with the DriversLicense XSD type.  Isn’t that sweet?  And of course all the powerful static analysis tooling applies including code completion, feature usage, refactoring, etc.

Another interesting tidbit: Type domains aren’t limited to tangible resources like files on disk. For instance, one evening I decided to see what it would take to provide a dynamic type in Gosu, similar to C#’s dynamic type.  In theory, I thought,  the type system could handle a dynamic type as just another type domain.  And wouldn’t you know, it could. Well, in the spirit of full disclosure I did have to tweak the type system internals a bit to better handle the “placeholder” concept, but just a little.  Aaanyway, unlike most type domains this one consists of just a single type name, namely dynamic.Dynamic.  Note Gosu requires that a type live in a namespace, so we can’t name it just Dynamic.  It was surprisingly simple to implement the type loader, you can download the source from our website here (http://gosu-lang.org/examples.shtml).  In a future blog I’ll cover the implementation of the Dynamic type loader (or some other one) step-by-step.

Another super cool type loader is the one from the RoninDB project by Gus Prevas (http://code.google.com/p/ronin-gosu/).  Here Gus provides a zero-configuration JDBC O/R layer via Gosu type loader.  The tables are types, the columns are properties, and no code generation; it just works.  Pretty cool, eh?  Download it and give it a spin.

So far I’ve only touched on the basic concepts underlying Gosu’s open type system.  But understanding the core idea is critically important to appreciate Gosu’s full potential.  More so, if you plan to use Gosu in your personal or professional software projects.  Because once you see the light with respect to type domains and the benefits they provide, it’s hard to think of developing enterprise applications without them.  To that end I promise to blog more and shed more light on building type loaders and the type system in general.


19 Comments on “Gosu’s Secret Sauce: The Open Type System”

  1. Raoul Duke says:

    you go, girls. thanks for making all this available to the rest of us.

  2. The Open Type System sounds good, but I’m still pretty murky on the topic. I think you should do a blog on a very specific example. Simulating dynamic types, or types from property files would be cool.

    Thanks for sharing Gosu.

    • Scott McKinney says:

      I’m sure you’re not alone in your murkiness. Gosu’s open type system adds another dimension to the already intricate subject of type loading.

      It may help to think of the open type system as a metatype system. That’s essentially what it is; a system of type systems. In the article I use the term “type domain” instead of “type system” for a couple of reasons though. First, meta-things are too confusing to think about. Are we talking about the type or the type of the type? Discussions are clearer if we establish separate terminology. Second, the term domain carries with it more meaning; it better conveys the idea that the Open Type System is a federation of otherwise discrete systems.

      This is all new to us too. We’re still baking parts of the type system and sometimes struggle with expressing ideas. My article here is an example of that. But I hope it at least shed some light on *what* the open type system is. Next time I’ll pop the hood and dive into an example, sort of a mini tutorial. That should help more with understanding *how* it works.

      Thanks for the feeedback!

  3. Scott McKinney says:

    Good suggestion. What did you have in mind? Maybe demonstrate the difference between, say, Ruby meta-programming and Gous’s type loaders etc.? But ultimately, once you’re past the black art of dynamic meta-programming, that is probably more a lesson in the virtues of static typing, which I’ve kind of already established here… maybe not well enough though. I dunno.

    • Raoul Duke says:

      if you mean my comment: i mean just a simple list of similar things. you don’t have to provide examples. just “ruby’s meta-programming”, “xyz’s pluggable types”, so that it helps give people who know those other things a rough idea of what gosu’s is all about.

      • Scott McKinney says:

        Yep, I replied to your comment… user error.

        Gotcha. The problem is there aren’t any other static langauges to satisfy the “xyz’s pluggable types” template. But we’re okay with that :)

        In any case, I agree we need to provide more comparisons.

  4. Carson Gross says:

    Compare Rails introspecting the database schema and generating a bunch of properties/methods on the model classes. It does it dynamically at runtime by looking at the db’s metadata, and using Ruby’s metaprogramming abilities to fill out the classes (i.e. adding methods to classes on the fly.)

    In Gosu, you would create a custom TypeLoader that reads a database schema and emits custom types for the Gosu compiler to work with. (In fact, someone has done just that.) The core difference is that it would be at compile time and, therefore, statically verified and, presumably, tool-friendly, so you get the best of both worlds.

    Another example are the classes you might generate via Axis in Java to allow you to work with a WSDL. If the WSDL changes, you regenerate the files and recompile.

    In Gosu you can, instead, write a custom typeloader that creates types for the WSDL “on the fly” for the Gosu compiler to work with directly, without an intervening generation of class artifacts. In fact, soon we are going to release something that does exactly this: drop a WSDL in your source directory and, blam, go.

    So the Open Type System gives you a good mix of the dynamic language’s metaprogramming ability, but doesn’t sacrifice the tooling support and static verification of a statically typed environment.

    Hope that helps clarify,
    Carson

  5. Taras says:

    so speaking of tools, when are we likely to get tools like checkstyle etc for Gosu?

  6. [...] real innovation in Gosu is the Open Type System, which lets Gosu be an excellent host language for other external DSL’s, such as XSD, WSDL or [...]

  7. [...] I have just described is basically Gosu’s Open Type System. Gosu is not just “yet another JVM language”, it also enables the kind of type openness [...]

  8. [...] obviously can’t provide a general listener infrastructure: the Open Type System with all sorts of different types with different property implementations. *However*, using feature [...]

  9. [...] with some interesting features. (Not the least, the Open Type System, which Scott wrote about here.) LikeBe the first to like this [...]

  10. [...] Most of the Gosu Team will be at the JVMLS: Scott McKinney (Gosu Team Lead), Dumitru Petrusca (IDE Lead), David Green (XML Guru) and me. I’ll be giving a talk on the Protocols typeloader, and implementing language features as a library using The Open Type System. [...]

  11. [...] are big features in Gosu that make development more productive (type inference, closures, and the The Open Type System to name a few) it’s these little things that really make me enjoy programming in [...]

  12. [...] Gosu, talks about some of the history and features of the language, and once again points out the Open Type System as the core innovation of Gosu. LikeBe the first to like this [...]

  13. [...] the expressiveness of the syntax of Gosu. I’d rather see innovations in libraries, leveraging The Open Type System rather than much continued innovation in the core language [...]

  14. [...] I’ve written extensively on the subject as have others at Guidewire.  To my surprise it has been challenging to convey the idea.  So far it’s a technology you have to experience for yourself to realize its benefits.  Evangelizing it is akin to demonstrating the qualities of Blu-ray on DVD, but harder.  It demos so naturally that most people miss what just happened, even after we say, “Look! We just had an intelligent conversation with a toad!”  In programming we tend to concentrate on the conversation and not the participants (the types), because who cares, they’re just types and of course they can talk.  It’s difficult to get across that the types involved in our demonstrations are really just raw data with which you normally can’t communicate, let alone have the compiler tell you whether or not your conversation makes sense. [...]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 38 other followers