Gradle with "Service Provider" Pattern

I've started dusting off my old littleware project over the last month or so on a "dev" branch in github. As part of the cleanup I'm moving littleware from its old ANT-IVY build system to Gradle. When I started with ant-ivy it was a great alternative to Maven, and allowed me to extend the build setup put in place by a Netbeans IDE's project with transitive dependencies, jenkins integration, scala support - all that great stuff. The problem is that the ant-ivy setup is idiosyncratic, and Gradle and SBT are well designed, widely used systems. I wound up going with Gradle, because I've read a lot of good things about it, and I want to become more familiar with it, and I had used SBT a little in the past, and found SBT's DSL annoying and its extension mechanisms inscrutable, and of course Google is using Gradle to build Android, ...

Anyway - like I said, I'm dusting off the code, and trying to remember what the hell I was thinking with some of it, but so far the Gradle integration has gone well. One of the issues I'm dealing with is the IVY-centric mechanism littleware had setup to pull in optional run-time dependencies. With Gradle the default behavior of the java plugin is to include all compile-time dependencies at run-time - which is what most people expect. Littleware's ant-ivy setup excluded optional dependencies (like mysql-connector versus postgresql - something like that), and included add-on IVY configurations like "with_mysql", "with_postgres", "with_aws", ... that kind of thing, so a client project would have a dependency specification something like this:

<dependency org="littleware" name="littelware" ... conf="compile->compile;runtime->runtime,with_mysql" />

Of course nobody else works like that, so in the interest of clarity and doing what people expect - I'm going to try to rework things, so that littleware's base jar asset includes service provider interfaces (SPI's), that add-on modules implement, so a client that wants to hook up littleware's ability to wire in a MySQL DataSource to a Guice injector would add a dependency to the 'littleware_mysql_module' to their build - something like that. The Jackson JSON parser implements a module system like this, and the SPI pattern is all over java-EE (JDBC, Servlets, whatever); it's a good and widely understood pattern. We'll see how it goes.

testing YUI apps with phantomjs

It turns out that wiring up phantomjs to process a web page that runs a test suite based on yui's test module is pretty easy to do. I wish I did this sooner. Here's what I got working.

First, I'm already in the habit of setting up pages like this that run a javascript module through some tests. I just use yui's test framework, since I use yui for everything else, and yui test works basically the same way as junit - which I use when coding java. I wind up with the following code (on github here):

    littleware.littleYUI.bootstrap().use( 
 'littleware-littleUtil', 'test',
 function (Y) {
     // The modules are loaded and ready to use.
     // Your code goes here!
     var util = Y.littleware.littleUtil;
     var log = new util.Logger( "littleUtilTestSuite.html" );
     var suite = util.buildTestSuite();
     
     Y.Test.Runner.add( suite );
     
     if ( typeof( window.callPhantom ) != 'undefined' ) {
  // phantomjs environment!
  console.log( "Phantomjs detected!" );
  Y.Test.Runner.subscribe( Y.Test.Runner.COMPLETE_EVENT, window.callPhantom );
     }

     Y.Test.Runner.run();          
    });

The thing to notice is the window.callPhantom feature detection. Phantomjs integrates the V8 javascript engine with Webkit's DOM magic in a cool command-line javascript console. There's probably a better way to say that, but the important thing for me is that phantomjs offers a great way to run and evaluate the results of test web pages from the command line.

The phantomjs web site has a great list of references on its "headless testing" page, and I think my setup is pretty similar to what others do. My setup takes advantage of one trick that I wanted to mention - listening for YUI's test-run complete event with phantomjs' webpage.onCallback function to pass the web-page test results through to the phantomjs script (below) that runs through a series of test web pages. Ugh - that may not make much sense, but hopefully the "phantomTestRunner.js" script below (and in github) helps clarify what I mean.

/**
 * Little phantomjs script that opens a list of urls given
 * in the first argument as a comma-separated list.
 * Intended for running YUI test suites that callback to
 * phantom at test completion via:
 *    
 *    <pre>
 *       if ( typeof( window.callPhantom ) != 'undefined' ) {
 *           // phantomjs environment!
 *           console.log( "Phantomjs detected!" );
 *           Y.Test.Runner.subscribe( Y.Test.Runner.COMPLETE_EVENT, window.callPhantom );
 *       }
 *
 *    </pre>
 */

var system = require( 'system' );

if ( (system.args.length < 2) || system.args[1].match( /^-+/ ) ) {
    // use comma-separated list - easier to integrate with ant that way
    console.log( "script takes exactly one argument: a comma-separated list of test urls" );
    phantom.exit(1);
}

var urlList = system.args[1].split( /,+/ );
/*
for ( var i = 1; i < system.args.length; ++i ) {
    urlList.push( system.args[i] );
}
*/

var resultList = [];
var currentTest = 0;

var page = require( 'webpage' ).create();
page.onConsoleMessage = function(msg) { console.log( msg ); };

function runTest( url ) {
  page.open( url,
     function( status ) { 
         console.log( url + " status: " + status ); 
         if ( status != "success" ) {
             phantom.exit(1);
         } 
         page.evaluate( function() { console.log( "page location: " + window.location.href ); } );
     }
 );    
}

page.onCallback = function( testResult ) {
    resultList.push( testResult );
    currentTest += 1;
    if ( currentTest < urlList.length ) {
        runTest( urlList[currentTest] );
    } else {
        console.log( "-------------------------------------------------------------" );
        console.log( "-------------------------------------------------------------" );
        console.log( resultList.length + " Tests complete: " );
        console.log( JSON.stringify( resultList ) );
        phantom.exit( 0 );
    }
};


runTest( urlList[0] );

Anyway, long story short, running something like:
phantomjs phantomTestRunner.js http://apps.frickjack.com/littleware_apps/testsuite/littleUtilTestSuite.html
prints out the web-console messages from the page's test suite (the script can run multiple pages), and collects all the results at the end (I need to do more work with that). Pretty cool!

ant rules for typescript compile

I extended littleware's ant template with rules to compile typescript (*.ts) files in web projects using this ant-macro posted on stackoverflow. The typescript and yuidoc rules are listed further below.

Littleware's build system just extends netbeans' ant templates with IVY support. Netbeans' ant scripts are designed to support extension hooks that run before and after the different build stages (initialization, compile, package, ...). This is from the comment at the top of a new netbeans project's build.xml file:

    There exist several targets which are by default empty and which can be 
    used for execution of your tasks. These targets are usually executed 
    before and after some main targets. They are: 

      -pre-init:                 called before initialization of project properties 
      -post-init:                called after initialization of project properties 
      -pre-compile:              called before javac compilation 
      -post-compile:             called after javac compilation 
      -pre-compile-single:       called before javac compilation of single file
      -post-compile-single:      called after javac compilation of single file
      -pre-compile-test:         called before javac compilation of JUnit tests
      -post-compile-test:        called after javac compilation of JUnit tests
      -pre-compile-test-single:  called before javac compilation of single JUnit test
      -post-compile-test-single: called after javac compilation of single JUunit test
      -pre-dist:                 called before archive building 
      -post-dist:                called after archive building 
      -post-clean:               called after cleaning build products 
      -pre-run-deploy:           called before deploying
      -post-run-deploy:          called after deploying

...

Anyway - I'm glad for netbeans' extensible ant templates. Littleware's build system could benefit from another rework to better support scala and IVY, but it's good enough for now.

Finally - I also wanted to mention that Ian Obermiller posted on github a typescript brush for Alex Gorbatchev's syntax highlighter. It seems to work great ...

Here are those ant rules I mentioned earlier ...

<target name="config-check">
     <!--
    <available file="${ivy.jar.dir}/littlesettings.xml" property="settings.exists"/>
    -->
    <available file="${ivy.jar.file}" property="ivyjar.exists"/>
    <condition property="skip.download">
        <and>
            <isset property="ivyjar.exists" />
        </and>
    </condition>
    <available file="ivy/test" property="ivy.resolve.test.exists" />
    <available file="ivy/compile" property="ivy.resolve.compile.exists" />

    ...
    
    <condition property="ivy.resolve.exists">
        <and>
            <isset property="ivy.resolve.test.exists" />
            <isset property="ivy.resolve.compile.exists" />
        </and>
    </condition>
    
    <condition property="typescript.ready">
        <and>
            <isset property="build.web.dir" />
            <isset property="typescript.compiler.path" />
            <available file="${typescript.compiler.path}" />
        </and>
    </condition>    
    
    <condition property="ydoc.ready">
        <and>
            <isset property="build.web.dir" />
            <isset property="ydoc.path" />
            <available file="${ydoc.path}" />
        </and>
    </condition>    
    
</target>

<!-- web project stuff -->

<property name="typescript.compiler.path" value="${user.home}/AppData/Roaming/npm/tsc.cmd" />
<property name="ydoc.path" value="${user.home}/Documents/Code/yuidoc/bin/ydoc.bat" />


<target name="ydoc" depends="config-check" if="ydoc.ready" 
    description="run ydoc on build/ js"
    >
  <exec executable="${ydoc.path}">
    <arg value="${build.web.dir}/resources/js/littleware"/>
 </exec>    
</target>

    
<!--
    Recursively read a source directory for TypeScript files, generate a compile list in the
    format needed by the TypeScript compiler adding every parameters it take.
    Thanks Tekool.net:
       http://stackoverflow.com/questions/12799237/how-to-watch-and-compile-all-typescript-sources
-->
<macrodef name="TypeScriptCompileDir">

    <!-- required attribute -->
    <attribute name="src" />

    <!-- optional attributes -->
    <attribute name="out" default="" />
    <attribute name="module" default="" />
    <attribute name="comments" default="" />
    <attribute name="declarations" default="" />
    <attribute name="nolib" default="" />
    <attribute name="target" default="" />

    <sequential>

        <!-- local properties -->
        <local name="out.arg"/>
        <local name="module.arg"/>
        <local name="comments.arg"/>
        <local name="declarations.arg"/>
        <local name="nolib.arg"/>
        <local name="target.arg"/>
        <local name="typescript.file.list"/>
        <local name="tsc.compile.file"/>

        <property name="tsc.compile.file" value="@{src}compile.list" />

        <!-- Optional arguments are not written to compile file when attributes not set -->
        <condition property="out.arg" value="" else='--out "@{out}"'>
            <equals arg1="@{out}" arg2="" />
        </condition>

        <condition property="module.arg" value="" else="--module @{module}">
            <equals arg1="@{module}" arg2="" />
        </condition>

        <condition property="comments.arg" value="" else="--comments">
            <equals arg1="@{comments}" arg2="" />
        </condition>

        <condition property="declarations.arg" value="" else="--declarations">
            <equals arg1="@{declarations}" arg2="" />
        </condition>

        <condition property="nolib.arg" value="" else="--nolib">
            <equals arg1="@{nolib}" arg2="" />
        </condition>

        <!-- Could have been defaulted to ES3 but let the compiler uses its own default is quite better -->
        <condition property="target.arg" value="" else="--target @{target}">
            <equals arg1="@{target}" arg2="" />
        </condition>

        <!-- Recursively read TypeScript source directory and generate a compile list -->
        <pathconvert property="typescript.file.list" dirsep="\" pathsep="${line.separator}">

            <fileset dir="@{src}">
                <include name="**/*.ts" />
                <exclude name="**/*.d.ts" />
            </fileset>

            <!-- In case regexp doesn't work on your computer, comment <mapper /> and uncomment <regexpmapper /> -->
            <mapper type="regexp" from="^(.*)$" to='"\1"' />
            <!--regexpmapper from="^(.*)$" to='"\1"' /-->

        </pathconvert>


        <!-- Write to the file -->
        <echo message="Writing tsc command line arguments to : ${tsc.compile.file}" />
        <echo file="${tsc.compile.file}" message="${typescript.file.list}${line.separator}${out.arg}${line.separator}${module.arg}${line.separator}${comments.arg}${line.separator}${declarations.arg}${line.separator}${nolib.arg}${line.separator}${target.arg}${line.separator}--sourcemap" append="false" />

        <!-- Compile using the generated compile file --> 
        <echo message="Calling ${typescript.compiler.path} with ${tsc.compile.file}" />
        <exec dir="." executable="${typescript.compiler.path}">
            <arg value="@${tsc.compile.file}"/>
        </exec>

        <!-- Finally delete the compile file 
        <echo message="${tsc.compile.file} deleted" />
        <delete file="${tsc.compile.file}" />
        -->

    </sequential>

</macrodef>


<target name="typescript" depends="config-check" if="typescript.ready" 
    description="compile typescript files build/*.ts"
    >
      <TypeScriptCompileDir
          src="${build.web.dir}/resources/js/"
          module="amd"
      />      
      
    <!-- out="${build.web.dir}/resources/js" -->
</target>


<!-- post-compile rule kicks in if build.web.dir netbeans property is defined -->
<target name="-post-compile" depends="ydoc,typescript" if="build.web.dir">
      <copy todir="${build.web.dir}/WEB-INF/lib">
        <fileset dir="ivy/test"/>
      </copy>
</target>

scala 2.10 netbeans ant fix

I just upgraded the scala install on my laptop from 2.9.x to the latest 2.10.0 release candidate, and was depressed to discover that broke my crazy netbeans derived ant build scripts (ugh!).

> ant clean
Buildfile: C:\Users\pasquini\Documents\Code\littleware\WIP\littleware\webapp\lit
tleId\littleId\build.xml

config-check:

download-ivy:

install-ivy:

resolveIfNecessary:

-pre-init:

-init-private:

BUILD FAILED
C:\Users\pasquini\Documents\Code\littleware\WIP\littleware\webapp\littleId\littl
eId\nbproject\build-impl.xml:50: The following error occurred while executing th
is line:
jar:file:/C:/Program%20Files/Scala/scala-2.10.0-RC5/lib/scala-compiler.jar!/scal
a/tools/ant/antlib.xml:5: taskdef A class needed by class scala.tools.ant.FastSc
alac cannot be found: scala/reflect/internal/settings/MutableSettings$SettingVal
ue
 using the classloader AntClassLoader[C:\Program Files\Scala\scala-2.10.0-RC5\li
b\scala-compiler.jar;C:\Program Files\Scala\scala-2.10.0-RC5\lib\scala-library.j
ar]

Total time: 2 seconds

Fortunately - the fix was easy. Scala 2.10.x pushes some classes that its ant task depends on to a new scala-reflect jar file, so I added that to the classpath in one of the xml blocks in nbproject/build-impl.xml, and I was back in business:

        <property name="scala.compiler" value="${scala.home}/lib/scala-compiler.jar"/>
        <property name="scala.library" value="${scala.home}/lib/scala-library.jar"/>
        <property name="scala.lib" value="${scala.home}/lib"/>
        <taskdef resource="scala/tools/ant/antlib.xml">
            <classpath>
                <pathelement location="${scala.compiler}"/>
                <pathelement location="${scala.library}"/>
                <pathelement location="${scala.lib}/scala-reflect.jar"/>
            </classpath>
        </taskdef>

Anyway - scala 2.10 looks to be a great release. A lot of people are excited about the "macro" system, but I've been working with the new akka actors, and also look forward to playing around with the futures and promises APIs.