Gosu’s Inconceivable non-ClassLoader (Take 1)

Two feature requests stand out on Gosu’s short list for our next milestone release. Primarily they stand out because the demand for them overshadows all other requests combined. And it happens that both features are blocked on what appears to be a virtual impossibility. To give you a hint as to their nature, if you’ve considered using Gosu in any of your current Java projects and chose not to, the absence of at least one of these features is probably why you passed over Gosu. If you’re thinking, “Hard to use Gosu from Java” or “No class files”, you’ve nailed it — as if you’re not thinking that.

Yep, Gosu currently does not produce Java class files on disk. There’s no command line compiler, instead the runtime dynamically compiles source to bytecode and defines Java classes directly from memory. While there are benefits in doing it this way, there are significant drawbacks to consider. Performance is an obvious problem. The extra time it takes to compile Gosu may impact startup time and introduce lag in initial usage of an application’s features. Web applications can mitigate the issue by “warming up” the type system where critical parts are preloaded before users log in. Unfortunately, this strategy won’t work for other kinds of applications like scripts and command-line tools where startup time can be crucial.

Closely related to the problem of not having class files is basic Java interoperability. Gosu interoperates seamlessly with Java… if Java is being used inside Gosu. Turn that around, however, and we have a lot of room for improvement; specifically Gosu classes aren’t directly usable inside Java. I’ll go into detail about this in a bit, but basically the core problem deals with the fact that Java’s class loader has no knowledge of Gosu’s class loader. This is the nature of class loaders in general where a Java application loader is oblivious of custom loaders created after the application initializes. Consequently we have no way of instructing a Java app loader to delegate to our loader. As a result calling or referencing Gosu from Java involves low-road solutions such as reflection or extracting intermediary Java interfaces from Gosu classes. Not quite the seamless experience we’re looking for, hence the demand for better Java-to-Gosu interoperability. And it all stems from having to define a special class loader.

To illustrate the problem consider the following two classes:

package com.abc; 
public class JavaClass {
  private GosuClass obj;
}

package com.abc 
class GosuClass {
  var obj: JavaClass
}

    +-------------+
    | System      |       Loads system classes
    |             +------>
    | ClassLoader |       eg. java/lang/String.class
    +-------------+
           ^
Ask parent |
           |
    +------+------+
    | Application |       Loads application classes
    |             +------>
    | ClassLoader |       eg. com/abc/JavaClass.class
    +-------------+
           ^
Ask parent |
           |
    +------+------+
    | Gosu        |       Loads Gosu classes
    |             +------>
    | ClassLoader |       eg. com/abc/GosuClass.gs
    +-------------+

(For the purposes of this example let’s take it on faith we can compile JavaClass; we’ll touch on that later. Also let’s assume Gosu is initialized, its class loader locked and loaded.)

Ok. Regardless of what kind of application we run (command line, web, etc.) the host environment loads our compiled Java classes for us from the class path. In our generalized example the Application ClassLoader loads JavaClass, but JavaClass fails to verify because com.abc.GosuClass can’t be found. This is because the application loader doesn’t delegate to our Gosu ClassLoader, it delegates upward to its parent – it has no idea the Gosu loader even exists. Conversely, if we explicitly load com.abc.GosuClass using the Gosu ClassLoader, the JVM has no trouble verifying the reference to JavaClass because the Gosu loader involves its parent, the Application loader.

Now you might think that if we go ahead and write class files to disk, there’d be no need for a special loader because the normal Java loader could load our class files — two birds, one stone. And you’d be right if all of Gosu’s classes could be statically compiled to disk. Ah, but there’s the rub. First, and I’ll spare you the particulars, there are a few lingering implementation details with Gosu’s runtime that on occasion rely on compiling bits of code dynamically; these will go away eventually, but we’re stuck with them for now.

More critically, Gosu’s parser and compiler are necessarily incorporated into the runtime. A few corner cases in the language require these services e.g., eval(), but mostly the open type system drives this design. Types in Gosu aren’t required to resolve as Java classes; instead Gosu provides abstractions for type loaders and type information. At runtime Gosu classes compile to conventional Java classes, yet other custom types that are not directly source code oriented e.g., Guidewire’s web framework, aren’t represented this way. Instead such types provide runtime type information for method, constructor, and property invokers. For instance, a web page method typically wraps a user-defined chunk of Gosu. Instead of requiring the web page framework to manage a bytecode oriented type system for all the snippets of Gosu associated with web pages, Gosu allows the snippets to execute loosely at runtime as programs or fragments. Thus a method invoker can simply instruct Gosu’s runtime to execute its anonymous chunk of Gosu, directly from source. But that chunk of Gosu will ultimately compile down to an anonymous Java class which is not discoverable from the Java application’s primary class loader.

The problem isn’t quite as knotty as it may seem. Because these particular chunks of Gosu are anonymous — they’re just nameless scripts — there’s no opportunity for Java classes to reference them by name. Basically this means the Java application loader should never be required to load any of these classes. Therefore we can get away with defining our own special class loader for these, which is what Gosu has done all along. There’s a deeper problem, however. Our special class loader parses and compiles these classes, which involves locking in our type system. Java also has locks for its type system; have a look at ClassLoader, note the synchronized methods especially on loadClass():

protected synchronized Class<?> loadClass(String name, boolean resolve)

Suffice it to say we flirt with deadlock every time our special loader does its job. We lessen the likelihood of this deadlock by ensuring we acquire both our lock and our class loader’s monitor atomically, if not we release both, wait a bit, and try again:

while( !TypeSystem.tryLock() ) {
  wait(100); // release this class loader's monitor obtained from the enclosing synchronized method
}
try {
  compile();
}
finally {
  TypeSystem.unlock();
}

Mission accomplished! Well… we’re not quite out of the woods here. What if our loader compiles a class that references another class from our parent loader, the application loader? Its monitor must be acquired too. We’re back to square one. It turns out deadlock arising from this particular scenario is quite rare, but it’s there, waiting. The only way to avoid it is to take our special loader out of the picture and somehow have the application loader compile and load our classes. Inconceivable.

But wait, there’s more. The JVM considers classes, abc.A and abc.B to be in different packages if they’re loaded in separate loaders. Thus even though A and B are declared in the same logical package, if A is a Java class and B is a Gosu class, because our special loader loads B, B will not have access to A’s package-protected (“internal” in Gosu speak) members. Likewise, A will not have access to B’s internal members. We currently overcome this limitation by generating bytecode for internal method calls and so forth as reflective calls. It works, but it’s an ugly hack with performance ramifications we’d rather avoid. Again the only way to get this right is for the Java class loader to do our bidding. Inconceivable!

Lastly, and most importantly, although we understand and appreciate the benefits of deploying class files, part of the appeal of Gosu is that it doesn’t require them. You can simply write code, say in your favorite text editor, and run it, no IDE or build step necessary, an aspect of dynamic languages our static language currently enjoys — some might say the best of both worlds. What’s more, legacy Gosu libraries don’t have class files. We’d like to support them as they are. So we’re back to square one yet again. Are we sure Java’s application loader can’t be made to somehow load our Gosu classes? “As I told you, it would be absolutely, totally, and in all other ways inconceivable.” (Vizzini to Inigo Montoya)

It would seem so. A class loader can only load the classes it’s designed to load, right? Some of you might be thinking of an unforgivable hack involving AspectJ or some such where we rewrite a portion of the class loader’s bytecode, tailoring it to load our classes. I like the energy, but let’s not; too many different class loaders out there to screw up and they’re all moving targets, it just won’t hold. We couldn’t take that approach anyway because Oracle’s Binary Code License Agreement prohibits the modification of system classes such as sun.misc.Launcher$AppClassLoader which we desperately need to control. Here’s another idea. A lot of application class loaders derive from URLClassLoader, even Java’s standard AppClassLoader does this. We could reflectively call addURL() and modify the class path dynamically. But what URL would we add? The whole point here is to get the class loader to resolve Gosu class names from source, not class files. The URLs we normally add are file system directories and jar files. What we need is a URL for… what?

A better question to begin with is, what does URLClassLoader do with the URLs as it attempts to resolve a class name. For instance, let’s say my class path is comprised of a single directory:

-classpath /my/application/classes

The application loader will have the corresponding URL:

file://my/application/classes/

URLClassLoader finds classes in a given URL using the protocol handler associated with the URL’s protocol. Java provides stock protocol handlers for standard protocols such as file, http, ftp, etc. A protocol handler’s primary responsibility is to establish a connection to a given URL in the form of a java.net.URLConnection. Most protocol handlers extend java.net.URLStreamHandler and implement openConnection( URL ). So, essentially, URLClassLoader delegates responsibility of finding a class or resource on its class path to the openConnection() methods corresponding with the protocol handlers of the URLs on its path. Given our single directory example, let’s say URLClassLoader handles a call to findClass() with argument “com.abc.Foo”, let’s assume our directory contains this class. First, the loader transforms the class name to a relative file name: “com/abc/Foo.class”. Then it creates a new URL for the class by appending the relative name of the class file to the directory URL:

file://my/application/classes/com/abc/Foo.class

Then it calls URL.openConnection() to establish a connection to the class file on disk. Next, the URL finds the protocol handler associated with the “file” protocol and delegates to it. From there the resulting URLConnection provides a stream to the contents of the file and the rest is history. I’ve skipped over a lot of implementation details but that’s the gist.

Interesting. The nature of protocol handlers is to find content wherever it exists. In the case of the file protocol it looks for files on disk. Likewise, the http handler finds resources at an address on the web. In theory we could define a handler for finding gosu classes in a Java application. How would that work? Let’s say we define our protocol as “gosuclass”. Fine.  Now we need to define our subclass of java.net.URLStreamHandler. But wait. Looking over the rules for adding a protocol handler it’s not so simple. Check out the curious process for adding a new handler:

http://docs.oracle.com/javase/1.4.2/docs/api/java/net/URL.html#URL(java.lang.String, java.lang.String, int, java.lang.String)

There are three options. The first option suggests we define our own URLStreamHandlerFactory for creating protocol handlers. The intention, I think, is to centralize control of protocol handler creation so we have a consistent experience with… protocol handling. Fair enough if you’re after controlling all of protocol handler creation, which we aren’t. Adding insult to injury the way we go about adding our factory pretty much rules out its use. From the documentation for URL. setURLStreamHandlerFactory():

“This method can be called at most once in a given Java Virtual Machine.”

Well, I’m pretty sure most web servers call this method way before we’ll get a crack at it. Which is what we want because we need to get a handle to the server’s factory so our poor factory can delegate to it for finding the web server’s other handlers. Since it can be called only once, we’re out of luck. Strike one!

Next on the list describes a convoluted convention where the system property for ” java.protocol.handler.pkgs” provides a list of packages containing handlers. The handlers must be named as follows:

<package>.<protocol>.Handler

Where <package> is one of the packages in the list and <protocol> is the name of the protocol and then, of course, Handler is the name of your URLStreamHandler subclass. We do all of this and depending on the execution environment (web server, rich client, command line) we get mixed results. The worst of it, on the web server it flat just doesn’t work, ClassNotFoundExceptions abound. What’s the deal? It turns out that the URL class tries to load protocol handlers using Class.forName(), which uses the caller’s class loader. Well this means URL’s class loader since the call originates from that class. On a typical web server Java system classes such as URL are loaded in a different loader than our application classes, therefore it never finds our Handler class. Strike two!

Our third and final option is a dud. Basically the same as the second one except the “default system package” is consulted for the Handler. Strike three! Yeerrr Out…

…wait a sec…

We’re not asking for much, we’d just like to add a bleeping protocol handler.  Where and how are these handlers managed inside Java? I mean how complicated can this really be?  Following the debugger it’s all too easy. The URL class maintains a static Hashtable (how retro) mapping protocol name to handler. That’s it. Couldn’t they just have… I’ll spare you my emotional diatribe regarding pragmatism in software design, or the disregard of it in the Java libraries. Anyway, all we need to do here is resort to a bit of reflection hackery, feel the requisite amount of shame, and move on. For your viewing pleasure:

private static void addOurHandler( Handler handler ) throws Exception {
  Field field = URL.class.getDeclaredField( "handlers" );
  field.setAccessible( true );
  Method put = Hashtable.class.getMethod( "put", Object.class, Object.class );
  put.invoke( field.get( null ), handler.getProtocol(), handler );
}

Now that we’ve force fed our protocol handler to Java we can confidently add our gosuclass URL to the class loader’s path. This part is easy albeit not entirely guilt-free as it involves another reflective call to protected method, addURL():

private static void addOurUrl() {
  URLClassLoader urlLoader = findUrlLoader();
  if( !(Arrays.asList( urlLoader.getURLs() ).contains( url )) ) {
    Method addURL = URLClassLoader.class.getDeclaredMethod( "addURL", URL.class );
    addURL.setAccessible( true );
    addURL.invoke( urlLoader, url );
  }
}

The protocol handler itself is pretty simple. It’s only task is to produce a GosuClassURLConnection, which does all the work :

protected URLConnection openConnection( URL url ) throws IOException {
  GosuClassesUrlConnection connection = new GosuClassesUrlConnection( url );
  return connection.isValid() ? connection : null;
}

GosuClassUrlConnection “connects” with the URL for a Gosu class. Connecting doesn’t involve anything more than resolving the name of the class, which it accomplishes by a call to Gosu’s TypeSystem.getByFullName( className ). If the type exists and is an instance of a Gosu class, the connection is successful. Actual compilation can wait until the class loader asks for the content of the class, which it obtains via a call to our URL connection’s getInputStream() method. This is where we dynamically compile the class and return an input stream for the resulting bytecode and in the process complete the “inconceivable” task of making the application loader do our work for us. No more special loader. Gosu classes, both as class files and source files, are now for all intents and purposes regular Java classes to the JVM; they are usable directly from Java code as Java code.  Hallelujah!

As an added bonus our compiler generates Gosu classes with a static block to initialize Gosu. Initialization includes the critical steps to inject our protocol handler and add the gosuclass URL to the app class loader’s class path. This may seem chicken-and-egg-like because static block execution is part of class loading. Right, but recall the other more dynamic bits of Gosu that are not written to disk. Also recall we support both modes of execution: from source and from class files — the source file-based classes need to load too. The idea is that Java application developers incorporating Gosu, essentially meaning applications executed with Java.exe and not Gosu.exe, will likely choose to compile the Gosu classes they create in their projects to class files. As such with Gosu app code precompiled developers aren’t bothered with wedging Gosu initialization in their app’s to support dependencies on Gosu code not compiled to class files. In other words, the first Gosu class file that loads will automatically setup Gosu so that source-based classes and such will be loaded via our protocol handler. The only requirement for using Gosu in a Java app is that the core Gosu jars be included on the class path. Other than that there should be no difference whether your Java app’s classes are Java or Gosu.

That about wraps it up. Essentially what we’ve done is implement a virtual class loader in the form of a URLConnection. It does the work of finding and producing bytecode for a given Gosu source file; everything a real class loader does aside from actual class definition. Which brings us back to what were after all along. Unbeknownst to the Java application class loader, it is now loading and defining our Gosu classes simply by deferring to our clandestine gosuclass protocol handler. Gosu no longer has a “special” class loader” and now sits at the cool table with the other Java classes. “HE DIDN’T FALL?  INCONCEIVABLE.” (Vizzini to Inigo Montoya)

There were some things I deliberately glossed over or just didn’t cover here. Mostly I avoided compiler stuff dealing with how we compile Java with Gosu class references in it. Well, most of this is still a work in progress. I have a side project where I’m tricking javac into compiling Java with Gosu type references in it. I’ll write about that next, but in summary javac can be made to compile with Java and Gosu intertwined. In fact with the technique I’m using Gosu’s entire type system can be used from within Java with no modifications made to javac. I also skipped over IDE related specifics. Each IDE has its own way of supporting (or not supporting) multi-language integration. The IntelliJ IDE plugin we’re currently focused on has all the support we need — I have another side project with Gosu-in-Java integration limping along in IJ. In fact the guys at JetBrains graciously fixed a blocking bug for me in their most recent release — a two week turnaround!  Those guys are awesome.


What Differentiates Gosu From Other Languages?

A recent question posed on our forum at gosu-lang:

What is that differentiates Gosu from other ‘similar’ languages and that makes Gosu unique?

I couldn’t agree more regarding the importance of differentiating the language.  In fact there isn’t much point in producing a new language if it can’t set itself apart in a significant way.  And I mean measurable significance in terms of us day-job programmers who are looking to get stuff done faster, better, and cheaper for our customers.

Many of the newer JVM languages offer only grammatical improvements over Java.  Sure, Java is a dated language still without properties, closures, composition, extensions, etc., but we can still build the same systems using Java with only marginally more effort than with most of the other newer languages.  And I mean total effort including the impact of the language on design, testing, refactoring, and general ease of understanding, not just the insignificant effort in the number of characters you have to type.  In fact most of the newer languages, I’ll argue, make our lives harder in ways that trump whatever modern features they offer.  For instance, newer dynamic languages leave us in the dark ages of pre-IDE wisdom — no parser feedback to speak of, no deterministic code completion, limited navigation, almost no discoverability, no trustworthy refactoring, etc.  Not to mention no basic static analysis and type checking from the compiler.  Not that bad if you’re writing something smallish in a text editor.  Otherwise, over time I’ll wager a net loss in productivity for most development teams heavily invested in a dynamic language, not that they’ll ever know it. I can make this claim from experience here at Guidewire.  Our success in developing a huge stack of technology, including Gosu, can lend itself in part to state-of-the-art tooling.  Further I’ll say rather confidently that we would have likely failed as a company without it.

What’s striking is what lures a budding development team away from Java isn’t the dynamic languages per se, but the systems atop the languages.  For instance, Rails is the primary reason for the spike in Ruby usage.  Before this phenomenon dynamic languages were pretty much left for dead or a sideshow at best.  Like functional languages they had several opportunities to overtake the imperative OOP ones, but have always failed to gain lasting traction as general purpose languages. My observation is that most red-blooded programmers prefer imperative OOP because it makes sense to them.  That’s really all there is to it, it just makes plain sense to human beings because that’s how we naturally model the world around us.  But Java has given us such a horrible experience with building web applications that we are looking outside the marriage so to speak.  It’s that bad.  I blame the hodgepodge of half-baked tools that are the result of EJB and the JCP fiasco — the same bureaucracy that has delayed basic language features like properties, closures, etc.  The point I’m making is that it’s not the dynamic languages so much that are drawing folks from Java as it is the systems built around these languages.  They, at least in the fledgling stage of a project, provide real, measurable productivity gains, which is what matters.  As you’ll see this is the point I’m working toward with Gosu.

Before we move on to Gosu, though, I’d also like to share my disappointment with some statically typed JVM languages.  Of the few newer ones none significantly helps us programmers advance the ball.  Take Scala.  It’s probably the most popular new kid on the block, yet it’s feature-set and type system are so overwhelmingly complicated most salary paying companies rightly keep it out of their code bases.  Don’t get me wrong, in many ways I like and respect Scala; It’s designer understands the advantage of static typing and it has quite a bit in common with Gosu, but its roots are functional and its feature-set is academically driven (read, hard for us nut crackin, hard workin OO programmers to understand so we can get stuff done and keep it done).  A web search exposes more than enough rants on Scala’s complexity, basically the same reasons functional programming on the whole remains a niche – a lot of simple ideas combined in ways to form overwhelming complexity.

If complexity isn’t enough to deter you, Scala’s flexibility should.  Most language grammars leave wiggle room for expression resulting in mini-dialects, but successful languages thus far tend not to vary too much in dialect.  Java’s success, for instance, is due in part to this characteristic — if you’re an experienced Java programmer, you won’t have much trouble reading and understanding Java, no matter where you go.  Not so with Scala.  If you’re a Java programmer experimenting with Scala, you’ll tend to write Java-like OO code in it.  If you’re a “clever” programmer, guess what?  You can pretty much write Haskell in Scala… Type classes? Monads? Implicit parameters? No problem.  The point is that Scala, beyond its horrifyingly complicated set of features, or perhaps because of them, is also a Tower of Babel.  Again I’m more concerned with the 99-percenter OO programmers who’re paid to impress customers than with the 1-percenter programmers who’d rather impress their peers.  So if you can somehow hire yourself a steady stream of green-blooded Haskell hackers, maybe Scala is for you, but probably Haskell is more for you.

Still think maybe Scala could be in your future?  Witness Kotlin.  Scala is so danged complicated the wizards at JetBrains invented a new language to take its place, namely Kotlin.  I was there when its authors introduced the language at the JVM Language Summit last year where they proclaimed it a Scala-lite of sorts because they basically began with Scala and removed a lot of stuff until they were satisfied (I’m paraphrasing, but that’s the gist).  On the whole Kotlin feels like a reasonable language, but like most Java alternatives, it doesn’t really offer much beyond some grammatical improvements, some of which are difficult to argue as improvements, like multiple inheritance and operator overloading.  Have we not learned from our errors with C++?  Language designers should be made to fix at least one good hard bug in an ancient C++ MFC application before they form an opinion on multiple inheritance, operator overloading, or macros.  I’ve seen some things, man.  In any case it’s hard to argue Kotlin isn’t a net improvement over Scala.  But as I’ll explain, it doesn’t matter.

The intention here is to talk about Gosu and what it has to offer not only in terms of what sets it apart from other languages, but also, and more importantly, why Gosu can help normal development teams become significantly more productive.  As I pointed out earlier the other new languages provide only marginal gains via typical grammatical changes.  I know that’s rather bold of me to say, but consider any of the Java challengers.  None of them offer anything more than an improved Java and/or functional programming to a degree.  There really isn’t much there that warrants a massive shift from Java – a closure here and there reduces some typing and may (or may not) improve readability, properties in the architecture can help relieve some pain, class extensions are a breath of fresh air, but really these are just improvements on Java’s base OO design.  And Java’s going to have most of that stuff sooner or later; Java 8 is just around the corner.  Please don’t misunderstand.  I loathe Java’s slow progress.  No properties?  Really guys?  Wait for Java 8 for closures?  Ugh!  Defender methods?  Sigh.  So for now there’s a niche market for JVM languages to offer features at a faster pace and Gosu does not hold back in that department.  We think we have a nice balance of features, so if it’s closures or properties or enhancements or delegates or what have you, Gosu will not disappoint.  But that’s not important. The features are there because Gosu should have them, but they are not the raison d’être.  Why the language then?  We are so explosively enthusiastic about Gosu because of one giant, game changing idea: Gosu’s Open Type System.

I’ve written extensively on the subject as have others at Guidewire.  To my surprise it has been challenging to convey the idea.  So far it’s a technology you have to experience for yourself to realize its benefits.  Evangelizing it is akin to demonstrating the qualities of Blu-ray on DVD, but harder.  It demos so naturally that most people miss what just happened, even after we say, “Look! We just had an intelligent conversation with a toad!”  In programming we tend to concentrate on the conversation and not the participants (the types), because who cares, they’re just types and of course they can talk.  It’s difficult to get across that the types involved in our demonstrations are really just raw data with which you normally can’t communicate, let alone have the compiler tell you whether or not your conversation makes sense.

The idea itself is simple: programmatic expansion of the type system.  The general idea isn’t new, some call it meta-programming.  But meta-programming as it exists is reserved for dynamic languages where there’s little benefit for development since it exists only at runtime; the dynamic language parser can’t help me discover anything about the new type or the fact that the type even exists.  The unique and enormous achievement with Gosu is that we’ve unified two otherwise discrete concepts –data and static types.  Programmers can now work directly with raw data as first-class types and do it statically leveraging all the productivity features offered by modern, high quality IDEs.  But there’s a key piece of information missing in that last marketing slogan of a sentence – the batteries are not included.  Worse, we have to manufacture the batteries in the form of IDE plugins.

Deep down I’ve always felt the key to Gosu’s success as an open source language was to have a strong presence in one or more IDEs. After all that is where static languages really shine; tooling rules.  Our first foray into the IDE world took us through the gauntlet and ultimate dead-end for languages other than Java, namely Eclipse.  I won’t get into the details other than the JDT was designed exclusively for one thing, the Java language.  To those other static language providers who dare to build a truly useful and competitive plugin in Eclipse, I bid you well; make the best of your orgy with the JDT classes and AspectJ weaving.

IntelliJ IDEA is our IDE of choice here at Guidewire and has been for ten years.  As an experienced user of both IDEs I feel confident in my view that IntelliJ is notches above Eclipse in every way: features, quality, and architecture.  We didn’t integrate Gosu with IntelliJ at first for a couple of reasons: it wasn’t free and it wasn’t the market leader.  Well I suppose the main reason was that I had started the initial development of the Eclipse plugin to get the ball rolling in advance of our open sourcing Gosu; it seemed like the right direction at the time.  Live and learn.  As a company up until nearly a year ago we were still kidding ourselves that we could build our tooling around Eclipse.  Then after talking with our IDE team about the full-retard scenario they were facing with weaving etc. I became convinced we had to punch out, and quick.  As it happened shortly before that time I was also exploring IntelliJ’s plugin architecture as an alternative.  I had developed a simple Gosu plugin to provide debugging and basic highlighting for our internal devs.  It was a smallish project, but I learned enough to know that JetBrains designed a truly open architecture for custom language plugins.  Fast forward six months… without getting into details I began exploring a proof of concept project on fully integrating Gosu into the world of IntelliJ.  It wasn’t long before I was convinced that not only should I continue down this path, but also that Guidewire should redirect its Eclipse effort toward IntelliJ.  This was the IDE for Gosu and, therefore, at least to my caveman brain, for Guidewire.  It wasn’t an easily won battle, but it was well worth the fight.

Now Gosu is well on its way to achieving professional quality tooling in IntelliJ, which I have to say given JetBrains’ high bar is quite an achievement.  So thanks to our hard working IDE team we’ve finally released our first preview of the plugin.  As the “preview” qualifier suggests it ain’t perfect, but it’s at a stage now where you can be productive with it, which is a huge milestone for us.  Because until now Gosu’s powerful type system was not demonstrable to the outside world.  Now that it is I’m optimistic about Gosu’s future in the wide world of open source.  But as I pointed out earlier regarding Ruby’s success with Rails, Gosu’s success also depends on contributing systems such as web frameworks, OR layers, and the like.  But unlike Ruby on Rails, due to Gosu’s static typing and full IDE support it has quite a bit more potential to scale both in terms of performance and project size.  And to make matters worse for the other new languages (let’s not pretend we don’t keep score in this game), Gosu has an unfair advantage with its open type system.  And some smart people have been busy bees building impressive frameworks and systems leveraging every ounce of it.  This is where the rubber really meets the road.

Now with Ronin, the web framework for Gosu created by Gus Prevas, and Tosa, the SQL-focused OR layer for Gosu authored by Alan Keefer and Carson Gross, programmers can be extremely productive building hardcore web applications in Gosu with IntelliJ.  Where else can you build web applications where the links are statically verified?  Where else can you directly reference your SQL DDL and access queries with static types using code completion, all with no code generation? What other static language lets you modify types corresponding with your web content, even the structure of the types, and immediately see the results in the browser?  (I’m going to keep going with this.)  Can your JVM language directly expose arbitrary XML schemas where elements are types and attributes are properties with no code generation?  Unless you have Dana Lank’s XSD/XML framework and your language is Gosu, it can’t.  And even if it could, it couldn’t dream of having an IDE like IntelliJ refactor the name of an XML element and automatically rename all references to it in your code.

This is the kind of differentiation I’m talking about.  Sure, base language features like closures and properties enhance the experience for programmers on many levels, but they don’t increase overall productivity anywhere near the scale of the open type system.  With it Gosu offers something entirely new to programming; another dimension if you will.  The open type system breathes life into the otherwise dead world of raw data.  This is the world we struggle with as programmers.  This is where we waste precious time and resources reading and deciphering and parsing and mapping and reinventing, and we do this over and over. The cracks and crevices between data and our world of objects is where the bad bugs lie. With Gosu suddenly data has the potential to be directly accessible as typed objects, be it XML, WSDL, properties, SQL, data markets, business rules, or you name it.  As the list of data domains supported by Gosu’s type system grows, so does the potential for new and interesting applications to leverage them.  But most importantly, the potential for real productivity gains is available to everyone right now with Gosu, Ronin, Tosa and XML on IntelliJ.  I invite you to take a test drive and let us know what you think on our forum.  We’re always more than happy to help with questions and are excxited to respond to any feedback you might have.


Gosu’s Secret Sauce: The Open Type System

Earlier last week we finally released Gosu to the general public, the JVM language we use for configuring our enterprise applications at Guidewire.   Gosu has been in the works for several years here beginning at the dawn of our engineering history in 2002.  Over the years the language has evolved to meet the growing demands of our applications.  Initially we created Gosu because we needed an embeddable, statically typed, scripting language for business rules.  We needed static typing so that we could 1) statically verify business logic and 2) build deterministic tooling around the language via static analysis e.g., for code completion, navigation, usage searching, refactoring, etc.  Tooling was, and is now more than ever, a vital part of our technology offering.  For instance, we don’t want to burden our users with having to commit our entire API to memory in order to be productive with our platform.  But it doesn’t end there.

Our platform, like that of most large-scale enterprise application vendors, consists of a relatively tall stack of technologies, most are developed in-house.  The stack is comprised of the following subsystems in no particular order: Database Entity/OR layer, Web UI framework, Business Rules, Security/Permissions, Web Services, Localization, Naming services, Java API, Query model, XSD/XML object models, various application-dependent models of data, and Gosu.  A primary challenge with any enterprise application is to adapt to customer requirements through configuration of its subsystems.  As I’ve stated, Gosu is the language we use to accomplish that, but how does, say, the Gosu code in the Web UI or Rules access the Entity layer and XSD-modeled information?  In other words, if a primary goal of Gosu’s static type system is to help users confidently and quickly configure our applications, how can it possibly help with domains such as these which are external to the type system?

This is where Gosu separates from the pack.  Unlike most (all?) other programming languages, Gosu’s type system is not limited to a single meta-type or type domain.  For instance, Java provides one, and only one, meta-type, namely java.lang.Class.  If you want to directly represent some other type domain to Java, you’re out of luck.  Your only options are to write a class library to access information about the domain or resort to code generation.  Neither option is desirable or even adequate in many cases, but as Java programmers we don’t know any better.  Dynamic languages such as Ruby provide varying degrees of meta-programming, which can be a powerful way of dynamically representing other type domains.  But with dynamic  languages we escape type safety, deterministic tooling, and other advantages of static typing, which defeats the purpose of our primary goal of exposing other type domains to the language.  Our customers, for instance, would be at a terrible disadvantage without the ability to statically verify usages of various domains in their application code.  Thus, Gosu’s type system must provide a similar (or better) level of flexibility provided by dynamic meta-programming, but without sacrificing static typing — a tall order.  How did we do it?

Gosu’s type system consists of a configurable number of type domains.  Each domain provides a type loader as an instance of Gosu’s ITypeLoader interface.  A type loader’s primary responsibility is to resolve a type name in its domain and return an implementation of the IType interface.  This is what is most unique about Gosu — it’s type system is open to other domains to participate with first-class representation.  For instance, Gosu does not discriminate between a Gosu Class a Java Class or Entity Type or XSD Type or what have you; they’re all just types to Gosu’s compiler.  This unique and powerful feature extends all the benefits of static typing to a potentially huge range of otherwise inaccessible domains.

Before we consider a specific domain, it is important to understand that not all type loaders in Gosu are required to produce bytecode.  Those that do, like Gosu classes, implement an interface for getting at the corresponding Java bytecode. Those that don’t provide call handlers and property accessors for reflective MethodInfo and PropertyInfo evaluation.  Note all types provide TypeInfo, see IType.getTypeInfo().  For instance, the parser works against the TypeInfo, MethodInfo, etc. abstractions as the means for a level playing field between disparate types. At runtime, however, unless a type provides a Java bytecode class the MethodInfos and PropertyInfos also are responsible for handling calls.  Not requiring bytecode at runtime accommodates a potentially much wider range of type loaders and makes it possible  to quickly prototype loaders. 

With that understanding, we can jump to some specific examples.  Let’s take a look the Gosu Class type loader. It’s job is to resolve names that correspond to the domain of Gosu classes, interfaces, enumerations, enhancements, and templates (primarily .gs* family of files).  The result is an instance of GosuClass, which implements IType etc.  One of the methods in there is getBackingClass().  Essentially, this method cooperates with Gosu’s internal Java classloader to compile the parse tree from the Gosu class to a conventional Java class.  So it follows that Gosu’s compiler performs dynamic compilation direct from source; there is no separate build process or persisted classfiles. This does not imply that the Java class loader is transient, however.  It is not.  Since Gosu is a statically typed language it’s classes are compiled to conventional bytecode and, thus, don’t require any classloader shenanigans as with most dynamically typed languages.  But I digress.

What connects the Java bytecode class to its Gosu type is the IGosuObject implementation; all Gosu classes and most other types implicitly implement this. The getIntrinsicType() method is instrumental, for example, in making reified generics work in Gosu.  An instance of a Gosu class, or any type in the entire type system for that matter, can produce its underlying type via getIntrinsicType().  And from there Gosu’s type system takes over. For instance, you can reflectively access Gosu-level type information from Java code via TypeSystem.getFromObject( anyObject ).

It’s worth repeating that Gosu classes are no more important to Gosu’s type system than any other type.  For instance, XSD types resolve and load in the same manner as Gosu classes.  Given an XSD type loader (we have one internally), Gosu code can import and reference any element type defined in an XSD file by name.  No code generation, no dynamic meta-programming, and all with static name and feature resolution.  The XSD type defines constructors, properties, and methods corresponding with elements from the schema.  In addition it provides static methods for parsing XML text, files, readers, etc.  The result is an instance of the statically named XSD type corresponding with the element name, which can be used anywhere a Gosu type can be used, because it *is* a Gosu type!  Here’s a very simple example that demonstrates the use of an XSD type:

From a sample driver.xsd file:

<xs:element name=”DriverInfo”>
 <xs:complexType>
  <xs:sequence>
   <xs:element ref=”DriversLicense” minOccurs=”0″ maxOccurs=”unbounded”/>
   <xs:element name=”PurposeUse” type=”String” minOccurs=”0″/>
   <xs:element name=”PermissionInd” type=”String” minOccurs=”0″/>
   <xs:element name=”OperatorAtFaultInd” type=”String” minOccurs=”0″/>
  </xs:sequence>
  <xs:attribute name=”id” type=”xs:ID” use=”optional”/>
 </xs:complexType>
</xs:element>
<xs:element name=”DriversLicense”>
 <xs:complexType>
  <xs:sequence>
   <xs:element name=”DriversLicenseNumber” type=”String”/>
   <xs:element name=”StateProv” type=”String” minOccurs=”0″/>
   <xs:element name=”CountryCd” type=”String” minOccurs=”0″/>
  </xs:sequence>
  <xs:attribute name=”id” type=”xs:ID” use=”optional”/>
 </xs:complexType>
</xs:element>

Sample Gosu code:

uses xsd.driver.DriverInfo
uses xsd.driver.DriversLicense
uses java.util.ArrayList

function makeSampleDriver() : DriverInfo {
  var driver = new DriverInfo(){:PurposeUse = “Truck”}
  driver.DriversLicenses = new ArrayList<DriversLicense>()
  driver.DriversLicenses.add(new DriversLicense(){:CountryCd = “US”, :StateProv = “AL”})
  return driver
}

Notice the fully qualified names of the XSD types begin with xsd.  That’s just the way the XSD loader exposes the types; a type loader can name them any way it likes, but it should do its best to avoid colliding with other namespaces by using relatively unique names.  Next you may have discovered the name of the XSD file, driver, as the namespace containing all the defined element types.  That’s also a convention the loader chooses to define.  What’s most interesting, however, is how seamlessly the XSD types integrate with Gosu.  You can’t distinguish them from Gosu classes or Java classes, and that’s the whole idea!  All type domains are created equally.  Notice you can even parameterize Java’s ArrayList with the DriversLicense XSD type.  Isn’t that sweet?  And of course all the powerful static analysis tooling applies including code completion, feature usage, refactoring, etc.

Another interesting tidbit: Type domains aren’t limited to tangible resources like files on disk. For instance, one evening I decided to see what it would take to provide a dynamic type in Gosu, similar to C#’s dynamic type.  In theory, I thought,  the type system could handle a dynamic type as just another type domain.  And wouldn’t you know, it could. Well, in the spirit of full disclosure I did have to tweak the type system internals a bit to better handle the “placeholder” concept, but just a little.  Aaanyway, unlike most type domains this one consists of just a single type name, namely dynamic.Dynamic.  Note Gosu requires that a type live in a namespace, so we can’t name it just Dynamic.  It was surprisingly simple to implement the type loader, you can download the source from our website here (http://gosu-lang.org/examples.shtml).  In a future blog I’ll cover the implementation of the Dynamic type loader (or some other one) step-by-step.

Another super cool type loader is the one from the RoninDB project by Gus Prevas (http://code.google.com/p/ronin-gosu/).  Here Gus provides a zero-configuration JDBC O/R layer via Gosu type loader.  The tables are types, the columns are properties, and no code generation; it just works.  Pretty cool, eh?  Download it and give it a spin.

So far I’ve only touched on the basic concepts underlying Gosu’s open type system.  But understanding the core idea is critically important to appreciate Gosu’s full potential.  More so, if you plan to use Gosu in your personal or professional software projects.  Because once you see the light with respect to type domains and the benefits they provide, it’s hard to think of developing enterprise applications without them.  To that end I promise to blog more and shed more light on building type loaders and the type system in general.


Follow

Get every new post delivered to your Inbox.

Join 36 other followers