JavaScript Builder from Falsy Values

Update: uglify-js as third option

Thanks to @kitgoncharov JSBuilder now includes the option to run uglify-js through Rhino ... still cross platform, a builder for any taste.

---

as promised during my workshop, I have created a Google Code Project with the builder used to demonstrate most advanced and common techniques to make a library, application, snippet, small and fast.

What Is JSBuilder

It's a truly simple Python module based over a certain hierarchy able to combine multiple files and produce a minified version of the application.
A similar approach has been used since ages with jQuery, and I am pretty sure there are tons of builders out there able to do similar task.
I personally created ages ago a similar project compatible with all windows platform.
However, that project was missing the possibility to put everything together, preserve some piece of code if necessary, replace some string runtime, and produce if necessary different outputs through the simplified module approach.
Here a build.py file example, the same you can find in the repository.

# JSBuilder example

# project name (from the root folder)
copyright = '(C) WebReflection JSBuilder from Falsy Values'
max_js = 'build/max.js'
min_js = 'build/min.js'

# file list (from the root/src folder)
files = [
    "intro.js",     #beginning of the closure
    "variables.js", #local scope variables
    "main.js",      #some application logic
    "functions.js", #functions declaration
    "debug.js",     #something useful or evil to debug
    "outro.js"      #end of the closure
]

# execute the task
import JSBuilder
JSBuilder.compile(
    copyright,
    max_js,
    min_js,
    files,
    # optional list of serches
    [   #mustaches in comments, hell yeah!
        '/*{namespace}*/'
    ],
    # with relative optional list of replaces
    [
        'var '  #produce var MyLib = ... 
        
        # it could be 'this.'
        # this.MyLib = ...
        
        # it could be 'my.personal.stuff.'
        # my.personal.stuff.MyLib = ... 
    ]
)

# let me read the result ...
import time
time.sleep(2)

Why Python

First of all the version is the 2.6 or higher but not compatible (yet) with version 3+.
Python to me is a sort of server side lingua-franca and it is missing by default only on Windows platform.
I would say the same about Java, with the slightly difference that in Python we don't need to create a class to perform a simple task as the one performed by JSBuilder.
If Python is not present in our system, we can easily find the installer in the python.org website while the build.py example code should be easy to understand even for non programmers.

Java As Well

Both YUICompressor and Google Closure Compiler come with a jar package which is a classic build out of one or more Java files. JSBuilder uses Java as well in order to call one compiler or another one with proper arguments and in a cross platform way so ... just in case you do not have Java installed, here the link to download what we need.

The Current Folder Structure

projectFolder
    build
    builder
    src

Above structure is exactly the same you can find in the repository but don't worry, as you can spot in the build.py file we are free to change destination folders or paths without problems.
JSBuilder assumes in any case relative paths from the root of the application but if you really need a different structure feel free to replace the string '../' with the one you prefer ... it is an easily replaceable part of the script, just find and replace and that's it.

Multiple Projects

The build.py file is just an example but obviously we can create as many examples as we want, even inside the build.py file itself, changing just the part to replace, destination files, and/or copyright if necessary.
JSBuilder should be suitable for all sort of cases ... at least for all cases I have been facing since I have started with JavaScript (10 years of cases, I hope it's enough)

The Concept Behind

As you can see there are different sources behind the scene and the application is distributed across multiple files.
While this concept may be not that familiar or weird, this has been used by jQuery library since ever and even if teams may think it's not easy to maintain such distributed code, a proper structure following some standard could be more than we need to make everything simple.
As example, the file called variables.js contains all local scope variables reused inside other files as well.
Of course if a variable has a meaning for more than a file, it makes perfect sense to put it in the precedent one, even as undefined variables, to eventually assign it later.

Closure Compiler VS YUICompressor

If you checkout the whole project and you try to build once with YUI and once with Closure Compiler, simply swapping comment on line 63 of JSBuilder.py file, you will realize these minifiers are able to produce a completely different output.
I leave you read all comments inside JSBuilder.py in order to understand what happens there exactly, and I hope you will realize there is even more than zou have imagined.

Debugging Code

Sometimes you may need a portion of code for debugging purpose, and the not minified version of the file is the only place where it should be.
When it comes to debug, we would like to be able to export outside private variables or functions, still understanding their original names rather than the minified one.
The debug.js files as example is able to modify internal variables thanks to evil eval function.
This piece of code is surrounded by this comment:

//^
... the debug/evil code ...
//$

If you are familiar with regular expressions you can spot that ^ as "start evil code" and $ as "end evil code" have semantic meaning ... and this is what will never be included in the final minified version of our app.

Give It A Try

I hope those present in my Falsy Values workshops got the advantages a build system could bring on our daily basis work but I have written as many comments as possible to let you understand what is going on, how does it work, and why this simple script is freaking cool and it could potentially become our favorite build system.

What's Next

In my personal projects I have included an extra line at the end of the build.py able to create, per each build, documentation.
This kinda helps me to do not forget I have to document new or modified stuff, plus it gives me extra hints if some documentation has been lazily copied and pasted rather than being rewritten ... well, I honestly don't really like jsDocToolkit since JavaScript syntax has nothing to do with Java one but I agree is, at least, a tool flexible enough and still widely adopted and/or extended so ... if you all agree, I will put the magic documentation part there inside a build-with-doc.py file example.

Google Closure ? I'm Not Impressed

We all know that Google is synonym of performances, simplicity, and again performances. A search engine that uses a truncated body for a not valid W3 markup should be the most bytes and performances maniac in the current web era, isn't it?
Well, Google Closure Tools has been a negative surprise, at least this is what I can tell about it after a first quick review.

Closure Compiler

It's since ages I am wondering what kind of tool Big G is using to produce their scripts and finally we got the answer: Closure Compiler ... ooooh yeah!
Packer, YUIC, it does not matter, when Google needs something, it creates something. This is almost intrinsic, as developers, in our DNA: we spot some interesting concept? We rewrite it from the scratch pretending we are doing it better!
This is not the case, or better, something could go terribly wrong!

// ==ClosureCompiler==
// @compilation_level ADVANCED_OPTIMIZATIONS
// @output_file_name default.js
// ==/ClosureCompiler==

(function(){
   "use strict";
   this.myLib = this.myLib || {};
}).call(this);

myLib.test = 123;

The produced output:

(function(){this.a=this.a||{}}).call(this);myLib.test=123;

And 2 warnings:

JSC_USELESS_CODE: Suspicious code. Is there a missing '+' on the previous line? at line 2 character 4
"use strict";
^
JSC_USED_GLOBAL_THIS: dangerous use of the global this object at line 3 character 4
this.myLib = this.myLib || {};
^

Excuse Me?
The sophisticated compiler is able to understand the "use strict" ES5 activation statement and the fact we are passing the global object as this reference in the closure. It does not matter, as showed in the produced code the result will be a broken library, thanks to its missed name, magically transformed into "a".
Advanced Optimization Could Fail to both give us right suggestions and fix or optimize the code, since in that case, as example, this.a won't perform anyhow faster than original this.myLib.
Advanced Optimization parameter could also be dangerous for lazy loaded libraries.
We need to be extremely careful with this option and, as result, rather than a Compiler, we will deal with a "code messer" where hard debug will become automatically the hardest ever.
Read Carefully This Page if you are planning to use this option because under the flag "best ratio" and "removed dead code" we could have massive surprises in the middle of the application.

As summary, SIMPLE_OPTIMIZATIONS as compilation_level directive is so far the recommended one, but at the same time it won't offer that different ratio compared against YUI Compressor or Dean's Packer (NO base62) produced outputs while ADVANCED_OPTIMIZATIONS could be tested for single stand alone files hoping these won't break the global namespace via renamed variables.
In this case a JavaScript closure, the real one, is an absolute must!

Closure Library

This is another part, not strictly related with the Compiler, but apparently able to work with it. The Closure Library is a wide namespace loads of core features and a cross browser User Interface. I have to admit this library is a massive piece of work, but techniques used to make it happen are often hilarious.
First of all, this is the first time I read protected variables called with an underscore at the end, rather than as first char:

// normal code
function Constructor(){
    this._protected = [];
};

// Closure Library Creativity
function Constructor(){
    this.protected_ = [];
};

Why On Earth? Python style a part, where the underscore has a concrete meaning, the technique to use underscore as first character has a reason to exists.

function copyOnlyPublic(o){
    var $o = {}, k;
    for(k in o){
        if(k.charAt() !== "_")
            $o[k] = o[k]
        ;
    };
    return $o;
};


var myC = new Constructor;
var $myC = copyOnlyPublic(myC);

No Way! To make the style "creative" the charAt method with optional 0 as argument needs to become:

if(k.charAt(k.length - 1) !== "_")

Is this what we would expect from the performances king? I don't think so.
Is this faster to read at least for human eyes? Neither!
Gotchas are everywhere in the library ... the most redundant stuff I've ever seen is the array namespace!

goog.array.indexOf = function(arr, obj, opt_fromIndex) {
  if (arr.indexOf) {
    return arr.indexOf(obj, opt_fromIndex);
  }
  if (Array.indexOf) {
    return Array.indexOf(arr, obj, opt_fromIndex);
  }

  var fromIndex = opt_fromIndex == null ?
      0 : (opt_fromIndex < 0 ?
           Math.max(0, arr.length + opt_fromIndex) : opt_fromIndex);
  for (var i = fromIndex; i < arr.length; i++) {
    if (i in arr && arr[i] === obj)
      return i;
  }
  return -1;
};

OMG, I can't believe performances matter only for a missed body tag in the layout ... it cannot be real, can it?
What we have there? 1 to 2 possibly missed checks for each call (IE) and everything just to emulate the native Array.indexOf which is present in basically every browser except truly old or Internet Explorer?

goog.array.indexOf = Array.indexOf || function(arr, obj, opt_fromIndex) {
  var fromIndex = opt_fromIndex == null ?
      0 : (opt_fromIndex < 0 ?
           Math.max(0, arr.length + opt_fromIndex) : opt_fromIndex);
  for (var i = fromIndex; i < arr.length; i++) {
    if (i in arr && arr[i] === obj)
      return i;
  }
  return -1;
};

Array.indexOf is used as fallback if for some unknown reason (and browser...) an Array has not indexOf but Array.indexOf is present ... well, if we can trust that case is there any valid reason to create a performance gap like that for almost every method?
forEach, lastIndexOf, every JavaScript 1.6 or greater emulated method contains redundant checks performed for each call ... where is the performance maniac here?
The feeling is that this library has been created by some Java guy, probably extremely skilled with Java, but definitively not that clever with JavaScript programming style. Google if you need skilled JS developers there are hundreds of us out there. What I mean is that it does not matter if a minifier is able to remove dead code because dead code should not be there at all, isn't it?

// WebReflection Suggestion
goog.array.clone = (function(slice){
try{slice.call(document.childNodes)}catch(e){
    slice = function(){
        var rv = [];
        if(this instanceof Object)
            // suitable for arguments
            // and every other ArrayLike instance
            return rv.slice.call(this)
        ;
        for (var i = 0, len = this.length; i < len; ++i)
            rv[i] = this[i]
        ;
        return rv;
    };
};
return function(arr){
    return slice.call(arr);
};
})(Array.prototype.slice);

Above suggested snippet is based on Features Detection, a technique apparently completely discarded in those files I've read in this library.

Features Detection Cons

• performed runtime, few milliseconds before the library or function is ready to use

Features Detection Pros

• performed once, and never again for the entire session
• best performances, browser independent and browser focused at the same time
• being usually based over most recent standards, features detections could cost a bit more only for deprecated, obsolete, or truly old browsers

Moreover!

If the problem is the wasted millisecond to perform a feature detection, we can always fallback into lazy feature detection.
I agree that web performances are more about download time and round trip, but if Google has V8 as engine monster, do we agree that better JavaScript practices could make even V8 faster?

Lack Of Creativity

Even if this library uses some weird practice, most logical and common techniques to speed up execution and reduce code are often not considered. As example, this is just one method we can find in the crypt file:

goog.crypt.byteArrayToString = function(array) {
  var output = [];
  for (var i = 0; i < array.length; i++) {
    output[i] = String.fromCharCode(array[i]);
  }
  return output.join('');
};

Oh Really? So now a JavaScript developer should create a function which accept an array in order to create another array to populate over a loop and a global method call to perform a join at the end? That's weird, I thought that method could have been written in this way:

// WebReflection Suggestion
goog.crypt.byteArrayToString = function(array) {
  return String.fromCharCode.apply(null, array);
};

... but maybe it's just me that noob that I cannot spot the difference except better performances over less code ... please enlighten me!

Closure Library, The Good Part

Is massive, for basically each file there is a test case so it is robust. For sure as we have seen before, this is not the fastest general purpose library we could find in the net, something I was expecting from Big G, but hey ... we cannot blame the excellent work done so far, can we?

Closure Templates

Well, here I am almost without a word ... I mean, the number 1 search engine able to create monster pages via JavaScript templates? I do hope those files will be performed everywhere but a web page and runtime, 'cause with incoming HTML5 I feel horrified thinking about a new web loads of document.write and nothing else. Where is the semantic? Where is the logic applied before and not during execution? Where are best practices? Looking at examples even V8CGI project is not able to perform that code ... what is that, exactly? And WHY???
The only thing I do like there is the definition file, something simple, clever, easy to parse, exactly the opposite of its implementation via JavaScript: please avoid it!

Closure Conclusion

No this is not another piece of the puzzle, just the end of this post. I've been probably too aggressive and it's only thanks to Google decision that I can write a post like this: Open Source is the key, or your libraries, operating systems, whatever, will be always behind 'cause developers able to help you are not only in your team.
I was expecting excellent ideas, new killer techniques unknown for everybody else, what I have found is "yet another toolkit". I am always up to contribute so if interested put my name in the committers list, I have already lot of stuff to optimize because a Compiler, whatever it does, cannot create a better code, it can simply try to make a bit better and shorter the existent one: never delegate skills to a machine until these will be able to take decisions for us!

Famous documentation and the dark side of "this" !

The good thing of internet is that you can find a lot of free documentation.
At the same time, the bad thing of internet, is that this documentation is rarely updated.

It could be a "guru documentation" or it could be a newbie documentation, but in both cases, it doesn't matter because it is probably wrong, not updated, or too generic.

This post has not enough space to describe every error you can find on, and off line ... so let me start with some true example about a common misunderstood referer, the this one.

Douglas Crockford and Private Members in JavaScript

This page is famous enough, and I suppose that every true JavaScript developer has read them at least once.
The main error in that page is described in this sentence

By convention, we make a private that parameter. This is used to make the object available to the private methods. This is a workaround for an error in the ECMAScript Language Specification which causes this to be set incorrectly for inner functions.

We know, or maybe we don't, that everything without a prefix will be executed in the global scope or in nested closure, if any. But even if a function is created inside a method, it fortunately does not make sense to call a function that has not been assigned as instance method and find a this reference inside.

// guys, this function ...
window.testMe = function(){};

// is exactly the same of this one
function testMe(){};

// or this one in a global scope
var testMe = function(){};

The main reason we do not need to use window when we call a global method, or function, is that everything in JavaScript is virtually executed like in a global window with statement (that's why write window.open is redundant and nothing else ... and Stargate code is totally redundant too :D).

// this code ...
window.onload = function(){};

// is exactly the same of this one
onload = function(){};

onload();

// that is virtually similar to ...
with(window){
    onload();
};

Let's focus on the onload example in the middle ... ok? Who will be the this referer if we do not use call or apply Function.prototype methods? window, of course.
And as window has a self property, that a link to window itself, if we use alert(this === self); it will be true every time we will use onload();
If we assign onload function as object method

onload = function(){
    alert(this === self);
    alert(this === window);
};
var o = {};
o.onload = onload;

... we will read two alerts with two false instead of two true value.
This simply means that to use a function that has not been assigned as method, or better, that is not called from a method, the this reference will be the global object.

onload = function(){
    alert(this === window);
};
onload(); // true
var o = {onload:onload};
o.onload(); // false

function useCallback(callback){
    callback();
};

useCallback(o.onload); // false again

I do not really know why Doug defined this an error in ECMA Specification, but if it really is, what could be the logic behaviour, an "unpredictable" default this referer even if the nested function is not inside an instance constructor?
I don't think so.

The intrinsic constructor Factory Design Pattern

Thanks to this "error", we could use a constructor in two ways or recognize when it has been called with new or as function.
If it is an error, it couldn't be possible to know this kind of useful information.

function Person(name, age){

    // this is true only if we used 
    // new before Person constructor
    if(this instanceof Person){
        this.name = name;
        this.age  = age;
    } else

    // otherwise we used the constructor
    // as factory design pattern (in this example)
        return new Person(name, age);
};

// these two lines do the same thing: create a Person instance
var me = Person("Andrea", 29), // doyou like python style, don't you?
 you = new Person("You", null);

alert(me instanceof Person);  // true
alert(you instanceof Person); // true

Anyway, as I wrote in my recent and updated documentation, that passed totally unobserved, we do not need a that variable for nested private function.

function Person(name, age){
    function setNameAndAge(){
        this.name = name;
        this.age  = age;
    };
    // to call setNameAndAge as private
    // method we do not need a that
    // but only call or apply Function.prototype
    // methods to inject instance as this referer
    setNameAndAge.call(this);
};
alert(
    (new Person("Andrea", 29)).name // Andrea
);

Finally, I have to admit that Douglas Crockford is my favourite JavaScript professor, and it's basically thanks to him that I know what I know about JavaScript. He wrote a lot of interesting JS stuff, and some document is truly updated, explaining for example errors wrote in old one.

My programming style has evolved since then, as any good programmer's should. I have learned to fully embrace prototypalism, and have liberated myself from the confines of the classical model.

Every programmer should evolve because in this sector knowledge is never enough. So thanks a lot Doug, but please update your 2001 doc writing something more about this, that, scope, injected this, and closures :D (a link to this post should be appreciated as well)

Ross and Dustin in Pro JavaScript Design Patterns

Differently, here we are in late 2007 ... December 2007
For some reason, few days ago someone posted again this book in Digg ... and for the first time, I read about them.

It seems to be really a good book, but without reading them, I've just downloaded examples trying to imagine what are these used for.

Fortunately, and thanks to Apress or authors decision, these sources are free and well organized inside chapter folders, each file with dedicated paragraph.

Well done ... but at the third chapter, I've read a pseudo "horror" in file 3.07 ...

var Class = (function() {
  
  // Constants (created as private static attributes).
  var UPPER_BOUND = 100;
  
  // Privileged static method.

  // ... that will never work, where is the function?
  this.getUPPER_BOUND() {
    return UPPER_BOUND;
  }
  // Return the constructor.
  return function(constructorArgument) {
    // 
  }
})();

Forgetting the missed = function, that could be a common error during quick development, getUPPER_BOUND is a privileged method of the global scope, aka window, and it is not static, it is like a window.getUPER_BOUND call, where this is obviously the window itself.
As I wrote few lines before, if we use this inside a function it will refer to window object. That's why after we can see a wrong example like this one:

var Class = (function() {
  var constants = {
    UPPER_BOUND: 100,
    LOWER_BOUND: -100
  }
  // again ...
  this.getConstant = function(name) {
    return constants[name];
  }
  return function(constructorArgument) {
    //...
  }
})();

// anyway ... Error!!!
Class.getConstant('UPPER_BOUND');

alert(Class.getConstant); // undefined
alert(getConstant); // expected function

I am pretty sure that if this code is the one showed to explain privileged, that chapter could be full of errors ... please update them putting correct code and free corrected page inside the zip ... otherwise, sorry for this correction.

Finally, Dustin and Ross are two JavaScript ninjas, and reading the rest of the code it seems that I absolutely need to find that book because I do like design patterns and I am so curious to know how are their implementation (good stuff guys!).

Conclusion

I am the first one that writes horrible code, and probably I've even forgot what I wrote one year ago and where ... so the only suggestion I could write is that if you find something interesting, or something that you did not know, look for something else and look every time for the date that document has been published.

Have fun with self instruction, and never stop ;)