I spent way too long the last couple days patching the typescript compiler (tsc) to run with java's rhino javascript engine. I mentioned in an earlier post how my dell laptop died on me. I wound up buying an hp envy m4 laptop on clearance at BestBuy to replace it, but before that I did that I was goofing around for a couple days getting by with my Android phone and the old PowerBook G4 I pulled off the shelf that amazingly boot up for me. Anyway - I thought I'd try to get tsc running from the command-line. The compiler is javascript compiled from typescript code, so I first tried to build node from source, but node's build setup requires newer versions of python, make, and gcc than what the powerbook had, so then I got the idea to try to run tsc with rhino, since the powerbook did have java 1.5 installed (java 1.6+ ships with rhino and a jrunscript command line tool). Of course it didn't "just work", and I let myself get sucked into banging my head on it - even after I got the new laptop. Ugh.
Anyway - I eventually got tsc running with rhino (the patch listing is further below). The easy part of the project was implementing a rhino version of tsc's IIO interface for file IO. The rhino implementation just calls through to java.io classes, and tsc runs a few feature tests to figure out which javascript engine it's running under:
if (typeof ActiveXObject === "function")
return getWindowsScriptHostIO();
else if (typeof require === "function")
return getNodeIO();
else if ( typeof java != "undefined" )
return getRhinoIO();
else
return null; // Unsupported host
The IO code was straight forward; the painful part was working
around rhino's quirks.
The first quirk I ran into was that
rhino appears to treat the names of java's primitive types
as reserved words, so things like:
var byte = 0;
or
option.short = "v";
are illegal. Fortunately - that only popped up a couple places
in the tsc code, but it's an unfortunate "feature" for rhino to have.
Another problem I ran into was invoking "delete" on an instance of java.io.File. Javascript includes delete in its collection of reserved words, but it should be legal to include a "delete()" method on some class. Rhino's javascript grammar probably just needs some love. The workaround was to access the method via f["delete"]() instead of f.delete().
C:\Users\Reuben\Documents\Code\typescript\src\compiler
> node
> var Foo = function() { return this; }
undefined
> Foo.prototype.delete = function() { return "bla"; }
[Function]
> (new Foo()).delete();
'bla'
...
> C:\Users\Reuben\Documents\Code\typescript\src\compiler
> jrunscript
js> var Foo = function() { return this; }
js> Foo.prototype.delete = function() { return "bla"; }
script error: sun.org.mozilla.javascript.internal.EvaluatorException: missing name after . operator (<STN> at line number 1
js>
Another problematic feature of rhino is that it does not hide the distinction between javascript's string type (which tsc expects), and java's java.lang.String. I discovered that Rhino has methods for converting between java types and javascript types - including the String() method mentioned here:
resolvePath: function (path) {
return <string> String( (new java.io.File(path)).getCanonicalPath() );
},
There were one or two other small rhino quirks to work out, but the big one that surprised me was that rhino's regular expression objects apparently don't respect javascript's normal scoping rules. The tsc compiler was failing under rhino with various undefined types that weren't properly pulled in via the file-reference comments (see section 9.1.1 in the typescript spec). I eventually had a test setup, and found rhino would load every other referenced file, so a tsc run with rhino had this output:
(1)Reading code from C:/Users/Reuben/Documents/Code/typescript/src/compiler/typescript.ts Found code at C:/Users/Reuben/Documents/Code/typescript/src/compiler/typescript.ts file reference: diagnostics.ts file reference: nodeTypes.ts file reference: ast.ts file reference: astWalkerCallback.ts file reference: astLogger.ts file reference: base64.ts file reference: emitter.ts file reference: parser.ts file reference: scanner.ts file reference: scopeWalk.ts file reference: symbols.ts file reference: tokens.ts file reference: typeCollection.ts file reference: types.ts file reference: referenceResolution.ts file reference: incrementalParser.ts
The output with nodejs was:
(1)Reading code from C:/Users/Reuben/Documents/Code/typescript/src/compiler/typescript.ts
Found code at C:/Users/Reuben/Documents/Code/typescript/src/compiler/typescript.ts
file reference: diagnostics.ts
file reference: flags.ts
file reference: nodeTypes.ts
file reference: hashTable.ts
file reference: ast.ts
file reference: astWalker.ts
file reference: astWalkerCallback.ts
file reference: astPath.ts
file reference: astLogger.ts
file reference: binder.ts
file reference: base64.ts
file reference: sourceMapping.ts
file reference: emitter.ts
file reference: errorReporter.ts
file reference: parser.ts
file reference: printContext.ts
file reference: scanner.ts
file reference: scopeAssignment.ts
file reference: scopeWalk.ts
file reference: signatures.ts
file reference: symbols.ts
file reference: symbolScope.ts
file reference: tokens.ts
file reference: typeChecker.ts
file reference: typeCollection.ts
file reference: typeFlow.ts
file reference: types.ts
file reference: pathUtils.ts
file reference: referenceResolution.ts
file reference: precompile.ts
file reference: incrementalParser.ts
file reference: declarationEmitter.ts
Anyway, long story short, it turned out that the reference strings were each processed by a function with a regular expression, and rhino had this crazy behavior where regular expression objects appear to be global.
with rhino:
js> function doTest( s ) { var rx = /^\s*(\/\/\/\s*
Unbelievable. Anyway - the work around is to reset the rx.lastIndex before each run, so:
function getFileReferenceFromReferencePath(comment: string): IFileReference {
var referencesRegEx = /^(\/\/\/\s*<reference\s+path=)('|")(.+?)\2\s*(static=('|")(.+?)\2\s*)*\/>/gim;
referencesRegEx.lastIndex = 0; // work around ridiculous bug in rhino ...
var match = referencesRegEx.exec(comment);
...
Finally - I just tested this stuff by running the compiler on itself. The typescript repo on codeplex includes a bunch of test cases and an nmake based Makefile, but I was too lazy to download visual studio and get that working. In the end - rhino compiled tsc in 2 minutes, and node did it in 3 seconds. Ugh!
C:\Users\Reuben\Documents\Code\typescript\src\compiler > date; jrunscript tsc.js --out tsc2.js tsc.ts; date Monday, June 3, 2013 12:49:41 PM Monday, June 3, 2013 12:51:39 PM C:\Users\Reuben\Documents\Code\typescript\src\compiler > date; node tsc2.js --out tsc3.js tsc.ts; date Monday, June 3, 2013 12:58:40 PM Monday, June 3, 2013 12:58:43 PM
Update 2013/06/29: I tried java 8's new nashorn javascript engine (in a jdk8 pre-release) to see how it did. Nashorn currently runs the tsc compile about 10% faster than rhino - still a lot slower than node. Doh!
> date; & 'C:\Program Files\Java\jdk1.8.0\bin\jrunscript.exe' .\tsc2.js --out tsc2.js tsc.ts; date; Saturday, June 29, 2013 5:29:11 PM Saturday, June 29, 2013 5:30:57 PM
Anyway - I'll check to see if the typescript maintainers will accept this patch, but I'll be surprised if they want anything to do with rhino after reading this sad tale ...
diff --git a/src/compiler/base64.ts b/src/compiler/base64.ts
index ee2d3c5..b4fc315 100644
--- a/src/compiler/base64.ts
+++ b/src/compiler/base64.ts
@@ -67,20 +67,21 @@ module TypeScript {
var shift = 0;
for (var i = 0; i < inString.length; i++) {
- var byte = Base64Format.decodeChar(inString[i]);
+ // note: "byte" is reserved in java Rhino javascript environment - ugh
+ var bite = Base64Format.decodeChar(inString[i]);
if (i === 0) {
// Sign bit appears in the LSBit of the first value
- if ((byte & 1) === 1) {
+ if ((bite & 1) === 1) {
negative = true;
}
- result = (byte >> 1) & 15; // 1111x
+ result = (bite >> 1) & 15; // 1111x
} else {
- result = result | ((byte & 31) << shift); // 11111
+ result = result | ((bite & 31) << shift); // 11111
}
shift += (i == 0) ? 4 : 5;
- if ((byte & 32) === 32) {
+ if ((bite & 32) === 32) {
// Continue
} else {
return { value: negative ? -(result) : result, rest: inString.substr(i + 1) };
diff --git a/src/compiler/io.ts b/src/compiler/io.ts
index a5eb1ad..6e75bf2 100644
--- a/src/compiler/io.ts
+++ b/src/compiler/io.ts
@@ -13,6 +13,9 @@
// limitations under the License.
//
+declare var arguments:any;
+var javaArgs:any = arguments; // Rhino sets global arguments ... bla
+
interface IResolvedFile {
content: string;
path: string;
@@ -105,7 +108,10 @@ declare class Enumerator {
constructor (o: any);
}
declare function setTimeout(callback: () =>void , ms?: number);
+
declare var require: any;
+declare var java: any;
+
declare module process {
export var argv: string[];
export var platform: string;
@@ -123,7 +129,7 @@ declare module process {
}
var IO = (function() {
-
+
// Create an IO object for use inside WindowsScriptHost hosts
// Depends on WSCript and FileSystemObject
function getWindowsScriptHostIO(): IIO {
@@ -533,12 +539,251 @@ var IO = (function() {
},
quit: process.exit
}
- };
+ }
+ ;
+
+
+ function getRhinoIO():IIO {
+ var utf8 = java.nio.charset.Charset.forName( "UTF-8" );
+ var jscriptArgs = [];
+
+ for( var i=0; i < javaArgs.length; ++i ) {
+ //
+ // convert java string to javascript string (so javascript string methods work - ugh!)
+ // see https://groups.google.com/forum/?fromgroups#!topic/mozilla.dev.tech.js-engine.rhino/FV15_KJVLGM
+ //
+ jscriptArgs.push( String( javaArgs[i] ) );
+ }
+
+
+ /**
+ * Byte-order-mark detector - ugh.
+ * @param streamInn java.io.InputStream
+ * @return java.io.Reader
+ * @see http://blog.publicobject.com/2010/08/handling-byte-order-mark-in-java.html
+ */
+ function inputStreamToReader(streamIn) {
+ // buffered stream supports mark and reset
+ var stream = new java.io.BufferedInputStream( streamIn );
+ stream.mark(3);
+ var byte1 = stream.read();
+ var byte2 = stream.read();
+ if (byte1 == 0xFF && byte2 == 0xFE) {
+ return new java.io.InputStreamReader(stream, "UTF-16LE");
+ } else if (byte1 == 0xFF && byte2 == 0xFF) {
+ return new java.io.InputStreamReader(stream, "UTF-16BE");
+ } else {
+ var byte3 = stream.read();
+ if (byte1 == 0xEF && byte2 == 0xBB && byte3 == 0xBF) {
+ return new java.io.InputStreamReader(stream, "UTF-8");
+ } else {
+ stream.reset();
+ return new java.io.InputStreamReader(stream);
+ }
+ }
+ };
+
+ return {
+ readFile: function (file):string {
+ try {
+ var f = new java.io.File( file );
+ if( (! f.exists()) || (! f.isFile()) ) { return ""; }
+ var buffer = java.lang.reflect.Array.newInstance( java.lang.Character.TYPE, f.length() + 128 ); // 128 fudge
+ var reader = new java.io.BufferedReader(
+ inputStreamToReader(
+ new java.io.FileInputStream( f )
+ )
+ );
+ try {
+ var offset = 0;
+ for( var step = reader.read( buffer, offset, buffer.length - offset );
+ step >= 0; step = reader.read( buffer, offset, buffer.length - offset ) ) {
+ offset += step;
+ //java.lang.System.out.println( "Just read num bytes: " + step );
+ }
+ var javaString = new java.lang.String( buffer, 0, offset )
+ //java.lang.System.out.println( "Read: " + javaString );
+ // convert java string to javascript string ... ugh
+ return <string> String( new java.lang.String( buffer, 0, offset ) );
+ } catch (ex) {
+ reader.close();
+ ex.printStackTrace( java.lang.System.err );
+ throw ex;
+ }
+ } catch (e) {
+ IOUtils.throwIOError("Error reading file \"" + file + "\" - " + e.toString(), e );
+ }
+ },
+ writeFile: function( path, content ) {
+ var f = new java.io.File( path );
+ if( f.exists() && f.isFile() ) {
+ var writer = new java.io.OutputStreamWriter(
+ new java.io.FileOutputStream( f ), utf8
+ );
+ writer.write( content );
+ writer.close();
+ }
+ },
+ deleteFile: function (path) {
+ var f = new java.io.File( path );
+ if( f.exists() && f.isFile() ) {
+ // delete is reserved in javascript - confused Rhino parser - ugh
+ f["delete"]();
+ }
+ },
+ fileExists: function (path) {
+ var result:bool = (new java.io.File( path )).exists();
+ return result;
+ },
+ createFile: function (path, useUTF8?) {
+ var f = new java.io.File( path );
+ if ( f.exists() && (! f.isFile()) ) {
+ IOUtils.throwIOError("Error creating file \"" + path + "\".", null );
+ } else if ( ! f.exists() ) {
+ var dir = f.getParentFile();
+ dir.mkdirs();
+ }
+ try {
+ var writer = new java.io.OutputStreamWriter(
+ new java.io.FileOutputStream( f ), utf8
+ );
+ } catch (e) {
+ IOUtils.throwIOError("Couldn't write to file '" + path + "'.", e);
+ }
+ return new IOUtils.BufferedTextWriter( {
+ Write: function (str:string ) {
+ writer.write( str );
+ },
+ WriteLine: function (str:string) {
+ writer.write( str + "\n" );
+ },
+ Close: function () {
+ writer.close();
+ writer = null;
+ }
+ } );
+ },
+ dir: function dir(path, spec?, options?):string[] {
+ options = options || {
+ };
+ function filesInFolder(folder:any):string[] {
+ var paths = [];
+ var files = folder.listFiles();
+ for(var i = 0; i < files.length; i++) {
+ var f = files[i];
+ if(options.recursive && f.isDirectory()) {
+ paths = paths.concat(filesInFolder(f));
+ } else if(f.isFile() && (!spec || f.getName().match(spec))) {
+ paths.push( String( f.getPath() ) ); // convert to javascript String
+ }
+ }
+ return paths;
+ }
+ return filesInFolder( new java.io.File( path ) );
+ },
+ createDirectory: function (path) {
+ try {
+ if(!this.directoryExists(path)) {
+ (new java.io.File( path )).mkdirs();
+ }
+ } catch (e) {
+ IOUtils.throwIOError("Couldn't create directory '" + path + "'.", e);
+ }
+ },
+ directoryExists: function (path) {
+ var f = new java.io.File( path );
+ var result:bool = f.exists() && f.isDirectory();
+ return result;
+ },
+ resolvePath: function (path) {
+ return <string> String( (new java.io.File(path)).getCanonicalPath() );
+ },
+ dirName: function (path) {
+ return <string> String( (new java.io.File( path )).getCanonicalFile().getParent() );
+ },
+ findFile: function (rootPath, partialFilePath) {
+ var scan = new java.io.File( rootPath + "/" + partialFilePath ).getCanonicalFile();
+ while(true) {
+ if( scan.exists() ) {
+ try {
+ var content = this.readFile( scan.getPath() );
+ return {
+ content: <string> content,
+ path: <string> String( scan.getPath() )
+ };
+ } catch (err) {
+ }
+ } else {
+ // climb up the file system ... ?
+ var parent = (new java.io.File( rootPath )).getParent();
+ if( parent == null ) {
+ return null;
+ } else {
+ scan = new java.io.File( parent, partialFilePath );
+ }
+ }
+ }
+ },
+ print: function (str) {
+ java.lang.System.out.print( str );
+ },
+ printLine: function (str) {
+ java.lang.System.out.println( str );
+ },
+ arguments: <string[]> jscriptArgs,
+ stderr: {
+ Write: function (str) {
+ java.lang.System.err.print(str);
+ },
+ WriteLine: function (str) {
+ java.lang.System.err.println(str );
+ },
+ Close: function () {
+ }
+ },
+ stdout: {
+ Write: function (str) {
+ java.lang.System.out.print(str);
+ },
+ WriteLine: function (str) {
+ java.lang.System.out.println(str );
+ },
+ Close: function () {
+ }
+ },
+
+ /**
+ * Could implement watchFile() with java-7 nio2 code, but too lazy to bother,
+ * since WindowsScriptHost skips this method too ... :)
+ * @see http://docs.oracle.com/javase/tutorial/essential/io/notification.html
+ */
+ watchFile: null,
+ run: function(source, filename) {
+ try {
+ eval(source);
+ } catch (e) {
+ IOUtils.throwIOError("Error while executing file '" + filename + "'.", e);
+ }
+ },
+ getExecutingFilePath: function () {
+ return this.arguments[0];
+ },
+ quit: function (exitCode? : number = 0) {
+ try {
+ java.lang.System.lang.exit(exitCode);
+ } catch (e) {
+ }
+ }
+ };
+ }
+ ;
if (typeof ActiveXObject === "function")
return getWindowsScriptHostIO();
else if (typeof require === "function")
return getNodeIO();
+ else if ( typeof java != "undefined" )
+ return getRhinoIO();
else
return null; // Unsupported host
})();
diff --git a/src/compiler/optionsParser.ts b/src/compiler/optionsParser.ts
index a10fb8f..7a7eb73 100644
--- a/src/compiler/optionsParser.ts
+++ b/src/compiler/optionsParser.ts
@@ -18,7 +18,7 @@
interface IOptions {
name?: string;
flag?: bool;
- short?: string;
+ shorty?: string; // note: "short" is reserved in java
usage?: string;
set?: (s: string) => void;
type?: string;
@@ -34,7 +34,7 @@ class OptionsParser {
for (var i = 0; i < this.options.length; i++) {
- if (arg === this.options[i].short || arg === this.options[i].name) {
+ if (arg === this.options[i].shorty || arg === this.options[i].name) {
return this.options[i];
}
}
@@ -89,8 +89,8 @@ class OptionsParser {
var usageString = " ";
var type = option.type ? " " + option.type.toUpperCase() : "";
- if (option.short) {
- usageString += this.DEFAULT_SHORT_FLAG + option.short + type + ", ";
+ if (option.shorty) {
+ usageString += this.DEFAULT_SHORT_FLAG + option.shorty + type + ", ";
}
usageString += this.DEFAULT_LONG_FLAG + option.name + type;
@@ -110,27 +110,27 @@ class OptionsParser {
}
}
- public option(name: string, config: IOptions, short?: string) {
+ public option(name: string, config: IOptions, shorty?: string) {
if (!config) {
- config = <any>short;
- short = null;
+ config = <any>shorty;
+ shorty = null;
}
config.name = name;
- config.short = short;
+ config.shorty = shorty;
config.flag = false;
this.options.push(config);
}
- public flag(name: string, config: IOptions, short?: string) {
+ public flag(name: string, config: IOptions, shorty?: string) {
if (!config) {
- config = <any>short;
- short = null;
+ config = <any>shorty;
+ shorty = null;
}
config.name = name;
- config.short = short;
+ config.shorty = shorty;
config.flag = true
this.options.push(config);
diff --git a/src/compiler/precompile.ts b/src/compiler/precompile.ts
index 88adf32..c66f937 100644
--- a/src/compiler/precompile.ts
+++ b/src/compiler/precompile.ts
@@ -131,6 +131,7 @@ module TypeScript {
function getFileReferenceFromReferencePath(comment: string): IFileReference {
var referencesRegEx = /^(\/\/\/\s*<reference\s+path=)('|")(.+?)\2\s*(static=('|")(.+?)\2\s*)*\/>/gim;
+ referencesRegEx.lastIndex = 0; // work around ridiculous bug in rhino ...
var match = referencesRegEx.exec(comment);
if (match) {
@@ -294,6 +295,7 @@ module TypeScript {
if (!comment.isBlock) {
var referencedCode = getFileReferenceFromReferencePath(comment.getText());
+ //CompilerDiagnostics.debugPrint( "Considering comment as possible reference (" + (referencedCode ? "ok" : "no") + "): " + comment.getText() );
if (referencedCode) {
referencedCode.minChar = comment.startPos;
referencedCode.limChar = referencedCode.minChar + comment.value.length;
diff --git a/src/compiler/referenceResolution.ts b/src/compiler/referenceResolution.ts
index 442d8ec..52ae47c 100644
--- a/src/compiler/referenceResolution.ts
+++ b/src/compiler/referenceResolution.ts
@@ -102,7 +102,7 @@ module TypeScript {
// if the path is relative, or came from a reference tag, we don't perform a search
if (isRelativePath || isRootedPath || !performSearch) {
try {
- CompilerDiagnostics.debugPrint(" Reading code from " + normalizedPath);
+ CompilerDiagnostics.debugPrint(" (1)Reading code from " + normalizedPath);
// Look for the .ts file first - if not present, use the .ts, the .d.str and the .d.ts
try {
@@ -116,19 +116,19 @@ module TypeScript {
else if (isTSFile(normalizedPath)) {
normalizedPath = changePathToSTR(normalizedPath);
}
- CompilerDiagnostics.debugPrint(" Reading code from " + normalizedPath);
+ CompilerDiagnostics.debugPrint(" (2)Reading code from " + normalizedPath);
resolvedFile.content = ioHost.readFile(normalizedPath);
}
catch (err) {
normalizedPath = changePathToDSTR(normalizedPath);
- CompilerDiagnostics.debugPrint(" Reading code from " + normalizedPath);
+ CompilerDiagnostics.debugPrint(" (3)Reading code from " + normalizedPath);
try {
resolvedFile.content = ioHost.readFile(normalizedPath);
}
catch (err) {
normalizedPath = changePathToDTS(normalizedPath);
- CompilerDiagnostics.debugPrint(" Reading code from " + normalizedPath);
+ CompilerDiagnostics.debugPrint(" (4)Reading code from " + normalizedPath);
resolvedFile.content = ioHost.readFile(normalizedPath);
}
}
@@ -148,7 +148,10 @@ module TypeScript {
// if the path is non-relative, we should attempt to search on the relative path
resolvedFile = ioHost.findFile(parentPath, normalizedPath);
-
+ CompilerDiagnostics.debugPrint(" Attempting to resolve (" + parentPath + ", " + normalizedPath + ") got: " +
+ (resolvedFile == null) ? "null" : resolvedFile.path
+ );
+
if (!resolvedFile) {
if (isSTRFile(normalizedPath)) {
normalizedPath = changePathToTS(normalizedPath);
@@ -187,6 +190,10 @@ module TypeScript {
var resolvedFilePath = ioHost.resolvePath(resolvedFile.path);
sourceUnit.referencedFiles = preProcessedFileInfo.referencedFiles;
+ for (var i = 0; i < preProcessedFileInfo.referencedFiles.length; i++) {
+ var fileReference = preProcessedFileInfo.referencedFiles[i];
+ CompilerDiagnostics.debugPrint(" file reference: " + fileReference.path);
+ }
// resolve explicit references
for (var i = 0; i < preProcessedFileInfo.referencedFiles.length; i++) {
var fileReference = preProcessedFileInfo.referencedFiles[i];
No comments:
Post a Comment