gtuhl: startup technology » development

Recent Conference Reviews

Joe — Sat, 28 May 2011 13:17:58 +0000

I’ve attended a couple conferences recently and figured I would break the long silence since my last post with a quick list of thoughts on each of them. I’ve got all the numbers for a fun “Cloud IO Sucks Worse Than You Think” post but haven’t pulled it together yet.

Google IO

Definitely worth attending. It’s worth noting that I stuck with infrastructure and App Engine sessions (with one detour to hear about WebGL) and skipped introductory “101″ sessions after being told by my Google IO veteran coworkers they contained nothing more than could be read from the project overview pages online. You could have alternatively dove into Android for the entire conference.

Positive Thoughts

Well run, sessions started and ended on time, lines moved fast, registration was a breeze, lunches were excellent and served quickly.
Some really great content and sessions, the fireside chats were quite good.
San Francisco is a pretty cool place and a good place to have a conference. I worked out of a few different coffee shops and stopped by a meetup and the tech and startup vibe is pretty intense.
App Engine has come a long way since I last tinkered with it and I had an itch to build something new on it after leaving this conference. They have really filled out the feature set, improved the foundational pieces, and built some really powerful services around it. You can run Go on it now as well which is pretty cool.
The Go language has really come along and is very appealing for some of the stuff I end of working on. Read about it at golang.org, you can watch the Google IO session there as well. This was an excellent session, Rob Pike is a super smart guy.
The loot is ridiculous and spoiling. Tablets, Chromebooks, Verizon LTE hot spots with prepaid data plans, and additional phones and gadgets depending on which sessions you attended.

Negative Thoughts

“Breakfast” was pretty weak (bagels and coffee) and the wifi was as terrible as expected.
The demos at the after party were pretty bad. Basically 10 or so groups showing off “new technology” that boiled down to remote controlling something (poorly) with an android phone in all 10 cases.
The smaller session rooms were not set up very well. Specifically, the slide screens were positioned very low so that if you weren’t in the front row or two you could not see the bottom 3rd of the slides (depending on how large of a head the person in front of you had). This was a big deal in sessions involving code.

Percona Live 2011

This was a much smaller conference (hundreds instead of thousands). It was overall solid but I think is still trying to figure out their content and how to structure the sessions. From my perspective it started out really weak but finished strong with some great afternoon sessions and an excellent closing session by Facebook. If I could build my perfect idea of a database conference it would be 50% companies with large deployments talking about their setups and 50% deep diving into database internals and letting people ask questions.

Positive Thoughts

Some of the sessions were really good. I particularly enjoyed the sessions by TheLadders and ONEsite where they walked through their environments talking about servers, scripts, issues they had run into, etc. The ONEsite session got even cooler as they talked about bringing in Fusion-io drives and had numbers and discussion around the before and after.
The closing keynote/session by Facebook was fantastic. Again got to walk through a bit of their environment, how they build their servers, hear about production issues they had faced and how they worked around them, and got a sense of their scale. They run a single, huge cluster of MySQL boxes for the most part. 13mm QPS, ~15 people managing it all if you count their engineering, performance, and ops teams. Harrison Fisk (the closing keynote deliverer) was an excellent speaker and I appreciated how quickly and openly he answered questions from the crowd. There was none of the sense of dodging and hiding that I got from some of the Google IO sessions when asked about infrastructure details.
Walked away with a short but solid list of things I want to benchmark and try at work.

Negative Thoughts

The opening was not so good: not on schedule and the content was a walkthrough of high level press releases and sponsor pimping.
The sessions were too short (some only 30 minutes!) with no padding in between so no hard questions could be asked and details could not be explored unless you skipped sessions and could pin down somebody.
Many of the sessions were extremely introductory or high level, smacked of the “101″ sessions at Google IO but in this case there was no way for me to know ahead of time. As an example, the “MySQL on SSDs” session was essentially an overview of what SSDs are while I was hoping for a deep dive performance comparison that I could diff against my own testing (to be fair though, I think more details were coming but the session was so short he couldn’t get to them or answer any questions).
I didn’t get any laptops or tablets or awesome gadgets

Conclusion

I tried to provide my high and low points above. Don’t get me wrong while reading the negatives though, both were definitely worth attending.

Magento is Messy

Joe — Tue, 09 Nov 2010 01:54:28 +0000

Magento is a PHP Zend-based ecommerce solution. Go check out their site.

Looks great doesn’t it? Magento most certainly is NOT great and this post serves as an additional warning (there are a lot of them out there now so I am just piling on) to those that are considering it for anything. These notes were made against v1.3.2.4 so some of it may have improved though I doubt anything substantive.

Feature Dice

For all the chatter about how feature rich and impressively architected Magento is it lacks a lot of features. A sampling:

You cannot delete an order (without directly hitting dozens of tables in the DB to do it manually).
You cannot directly change the status of an order (without hacking on some core config files to make this possible in a buggy way).
You cannot edit an existing order directly.
You cannot control the initial order status of orders in a custom way.
Canceled orders still show up in all sales reports.
You cannot export orders in any format.

Technical Stuff

Lots of flawed stuff going on with the general structure of this thing as well:

Magento is SLOW. Check out this post where they proudly trumpet 5 requests per second in a new version. 5 requests per second? That is some horribly slow stuff.
Related to the above, they pointlessly used a poorly implemented EAV model for the database schema. In practice their schema generates dozens of joins on every query, makes it borderline impossible to figure out manual SQL queries, forces you to use their PHP objects to build up queries, and most importantly ensures that literally dozens of database queries run for every single page load.
The worst combination of convention and configuration imaginable is in play. You must configure EVERYTHING with countless, verbose objects and huge balls of XML to stitch everything together and then they put a vast, undocumented layer of conventions on TOP of all that configuration to ensure no one can pick up the system easily.
Theming Magento requires creating dozens of files in multiple folder hierarchies and modifying way too may configuration files. It is a really heavy weight process though at least this side of it is better documented than the rest.

The main overarching problem with Magento? It is overwhelmingly a vehicle for drumming up consulting and support contracts. I suspect most of the above pain points go away if you are willing to pay the $3k – $13k every year for a commercial version plus who knows how much for consulting on top of that. Its creators have no desire for anyone to pull down the software and do something useful with it without paying. The support forums are deadly silent, even the most trivial of plugins (or plugins to fill in sorely lacking features that should be built in) cost hundreds of dollars, and useful documentation is painfully difficult to find. Magento is an incredibly impressive accomplishment of architecture astronauts trying to see how complex they could make an ecommerce solution. This is the most closed and uninviting open source project I have ever encountered.

Side note, this guy’s blog is probably the best documentation I have been able to find so if you are stuck working with this platform his posts are absolutely worth a read.

That’s all I have to say about Magento. It absolutely has some positives and some very powerful features but you should stay away if you aren’t willing to pay big money and consider another solution. I would recommend checking out a hosted solution like Shopify or Core Commerce.

Using MATLAB Remotely on Windows

Joe — Sun, 13 Jun 2010 17:07:27 +0000

As part of the recent Hudson setup I needed to run MATLAB scripts remotely on Windows machines. Despite a fair amount of searching I could find instructions no where for making this work. There were however a lot of unanswered posts asking how to do it.

The main challenge is that MATLAB on Windows doesn’t put its output in the console. It instead opens another window (even if you use -nodesktop or any other param combo). That combined with the barbaric licensing they employ is the challenge.

Here is how I got it to work. This approach works great for doing remote running of scripts. It would be annoying if you were wanting to do prolonged interactive work with MATLAB given the requirement of using “-logfile”. I had to get more tricky to get MATLAB scripts hitting the GPU remotely but leaving those steps out of this post. If someone does need to accomplish that let me know and I can continue in the comments.

Why can this be helpful? I can think of a few reasons:

You have 1 machine licensed to run MATLAB that is shared by several people that only need occasional access. Linux would work better, but perhaps you are stuck with Windows by preference or circumstance.
You need to run bulk jobs, perhaps by cron or similar, on Windows machines automatically.
You want to write larger scripts of which M code is only a component while sticking with bash.
You want to do some sort of automated testing on Windows of MATLAB code.

I have tested this approach successfully in the 32bit and 64bit versions of Windows XP and Windows 7. Perform these steps as the user on the machine that is licensed to run MATLAB.

Install Cygwin

Go here, download the setup.exe file, and run that on your Windows machine.
In addition to the defaults install the openssh and cygrunsrv packages.

Setup sshd

In a cygwin prompt, run ssh-host-config -y. All prompts should get answered automatically and you will get a “Host configuration finished” message.
In a cygwin prompt, run cygrunsrv -S sshd.
Now open Control Panel -> Admin Tools -> Services.
Right click “CYGWIN sshd” and hit properties.
On the Log On tab, change the radio button to “This account” and specify the user name and password for the licensed MATLAB user on the Windows machine.
Hit Apply and OK and close the services dialog.
Open Control Panel -> Admin Tools -> Local Security Settings -> Local Policies
Make sure the licensed MATLAB user on the Windows machine is explicitly listed for each of the following permissions:
- Adjust the memory quotas for a process.
- Create a token object.
- Log on as a service.
- Replace a process level token.
In a cygwin prompt, chown [licensed user] /var/log/sshd.log and chown -R [licensed user] /var/empty
Now open the services dialog back up and restart CYGWIN sshd. Make sure it starts back up cleanly.

That should wrap up the setup.

General idea is we are using sshd to connect to the Windows machine and run our MATLAB scripts. The key piece is changing the user that runs the sshd process to be the same user that is licensed to run MATLAB.

Now you can connect to this machine and run MATLAB scripts like this:

matlab -nodesktop -nosplash -r "rand,quit" -logfile matlab_output.txt

That would run rand and put the output in matlab_output.txt. You can similarly (or more practically) run entire scripts:

matlab -nodesktop -nosplash -r "my_file,quit" -logfile script_output.txt

You can chain as many commands/scripts as you would like as part of the “-r” parameter by separating them with commas or semicolons. Note that there will be a delay between you issuing the command and when the specified log file gets written to. This is because in Windows a launcher/starter process runs, spawns the actual MATLAB process, and then exits while the MATLAB process continues to crank. You can use “-wait” to eliminate this and have the launcher process remain running until MATLAB terminates. This is particularly helpful when writing scripts that should not continue until your MATLAB call has completed.

With respect to Hudson this setup is really nice. You can just write bash scripts and use ssh to connect to the Windows slaves. Only change is the need to cat the output files if you want them to get logged and/or archived in Hudson.

Hibernate Criteria API Bug

Joe — Thu, 20 Dec 2007 00:19:14 +0000

I’ve talked about it before, but we’ve got a chunk of code in our primary product that allows permissioned users to build extremely complex queries using a GUI. The GUI inputs are converted to an intermediate format AST that is used for persistence and this gets converted into the Hibernate Criteria API equivalent by the DAO when the built up query is executed. The resulting feature is pretty dang powerful and I actually think it would be a cool thing to wrap up into a clean package and provide for others whenever I have time to do so.

In any case, this thing is complex enough that I’ve encountered many bugs or edge cases that aren’t handled cleanly in the Criteria API itself and we found a new one (new for us, the JIRA issue has been around for awhile) the other day that is worth throwing up a post about for others to find the workaround for more easily.

Sometimes databases require some denormalization for performance reasons (joins between 8+ digit row count tables don’t care how much RAM you have). If the column count gets too intense though and if you only really need a few values of each record the IO required to pull out all of the data can be fairly expensive and projections can be needed. With the Criteria API you can project specific columns with something like this:

public List getNamesOfPeople(){
    Criteria query = getSession().createCriteria(Person.class);

    ProjectionList projections = Projections.projectionList();
    projections.add(Projections.property("firstName"));
    projections.add(Projections.property("lastName"));

    query.setProjection(projections);

    return query.list();
}

One downside to this approach is you have to deal with Object arrays as the result from your queries. If you have manager or controller code its a bummer to have to write chunks of code for marshaling the values in the Object arrays into the actual objects (Person in this case) before doing things with it. Queries built with the Criteria API can have result transformers applied to them. Without going into specifics i’ll just say there is an AliasToBeanResultTransformer that can be used to automatically dump projected values into a instance of the specified class. Now the code would look like this:

public List getNamesOfPeople(){
    Criteria query = getSession().
        createCriteria(Person.class);

    ProjectionList projections = Projections.projectionList();
    projections.add(Projections.property("firstName"), "firstName");
    projections.add(Projections.property("lastName"), "firstName");

    query.setProjection(projections);
    query.setResultTransformer(
        new AliasToBeanResultTransformer(Person.class));

    return query.list();
}

The way the AliasToBeanResultTransformer works is to check the names of the properties of the object you want and write into them any values from the result with the same name. In the above example the second parameter to ProjectionList.add is the alias or the label for the column when the result hits the AliasToBeanResultTransformer. The above code will yield a list of Person objects with their firstName and lastName properties set. If the alias parameter to ProjectionList.add is omitted you will get a list of Person objects with nothing set (because Hibernate will use its own aliases and they won’t match up).

Finally the bug. When Hibernate generates the actual SQL that hits the database it aliases each projected column with its own internal scheme for the purposes of the SQL query. It will generate SQL that looks like this:

select this_.first_name as y0_, this_.last_name as y1_
from people this_;

If you add restrictions to your Criteria API query that involve the same properties and aliases you used in your projection list you can end up with generated SQL that looks like this:

select this_.first_name as y0_, this_.last_name as y1_
from people this_
where this_.y0_ = 'John' and this_.y1_ = 'Smith';

The key thing to note is that Hibernate is trying to be smart and reuse the alias specified in the select clause. The problem is that this isn’t valid SQL to any rdbms I know of. If you try to use the alias like that in the where clause you will get a “column ‘y0_’ does not exist” in PostgreSQL and similar messages in other databases.

This is a long known issue and the corresponding JIRA issue can be found here. Basically someone reports it, he gets cut off and told its Oracle’s problem because the SQL looks fine, and people pitch in about pretty much every other rdbms also not supporting it. Its still marked Minor and is Unassigned after 2 years so I doubt it will be fixed anytime soon. Thankfully, if you are having this problem you can look deeper in the comments and get a CustomPropertyAliasProjection class that when combined with a modification a little further down can be used to make it work with the AliasToBeanResultTransformer. The fix is essentially to ignore the alias in the select clause when generating the where clause. I won’t repost the code here as I didn’t write it and the JIRA page works just fine.

So if you run into this problem that is the fix. There are actually lots of bugs in the Criteria API but until someone is able to build an ORM tool better than Hibernate (and no such thing exists to my knowledge) I’m not going to switch and the fact that it is open source means you can usually find a workaround online.

Hibernate Colon (:) Bug

Joe — Sat, 13 Oct 2007 09:39:41 +0000

In our main product we’ve built a pretty heavy-weight, powerful block of code for allowing users to build and execute very complex queries through the GUI. The permutations covered by this widget recently uncovered a hibernate bug and want to quickly write up this overview so google can start indexing it

As a short explanation, the queries being generated by the code backing this GUI-based query builder was generating fragments that involved type casts. We are using PostgreSQL and they looked like this:

(some_column > 0)::integer

This has worked great for awhile but last night I uncovered a combination of factors that causes this to break. If you are:

Using the Criteria API to build up a query
Using sqlRestrictions in your query
Using colons in your sql restriction (in my case it was the ::type casting for PostgreSQL)
Have a filter applied to your queries. In our case it was a filter being set by a @FilterDef annotation on the model object

If you combine these factors you will get an error along the lines of Invalid filter-parameter name format. When I googled this I ran into the common scenario of only finding hits within public-facing code repositories for the project throwing the error. Hopefully the next searcher finds it here.

Basically I think what happens in this case is colons aren’t properly escaped and hibernate then expects named parameters. In my case it was seeing ::integer and thinking I had a named parameter called :integer and the code in SessionImpl.java of the hibernate project was blowing up because this doesn’t look like a valid parameter name. It would have been much harder to track down the problem if hibernate wasn’t open source.

If I disabled the filter it worked fine. In either case the complete workaround is thanks to PostgreSQL and involves using this alternate casting syntax:

CAST ((some_column > 0) AS integer)

Thank goodness PostgreSQL provides different types of syntax for this. I’ll probably post this to the Hibernate JIRA before too long, i’ll likely somehow be told it isn’t a bug.

Flex2 & DWR

Joe — Wed, 04 Apr 2007 12:08:26 +0000

In my last post I mentioned in passing some code we wrote to allow DWR calls inside Flex2 code. My goal with this post is to provide a short, clear example of doing this and then describe how we applied that to our project. This doesn’t handle timeouts, call batching, reverse Ajax/Comet, and probably some other DWR features I am forgetting. There are many improvements that could be made to it and it was only used in FireFox. You may want to be familiar with the ExternalInterface of Flash (which hopefully will have its bug fixed soon) and with Javascript and DWR before reading on. I apologize for any faulty formatting – still using vanilla, hosted wordpress and it doesn’t make code especially easy to work with.

There are 3 pieces to making it work:

An Html/JS container that your swfs must be run in.
An AS3 class that talks to the Html/JS container.
A layer of JS to make the DWR calls and route callbacks to AS3.

The Html/JS Container

For the sake of brevity here is a wrapper that will work. For a more robust wrapper take a look at the Html/JS wrappers that FlexBuilder generates and base your custom DWR wrapper files off of those. You need 2 wrappers – one for run mode and one for debug.

DWRContainer.html & DWRContainer-debug.html

The details to notice are the inclusion of dwrWrapper.js, the DWR engine.js file, and then any number of DWR controllers that you want to use in Flex2. Ideally these DWR controller JS files would have their content injected dynamically but they are small files so I don’t think it is a big issue to include them all and this makes the example simpler. Now to create the debug version just copy the above contents but use MyFlexApp-debug.swf for the src values. Once you have your wrappers built, you will need to tell FlexBuilder to use these when running/debugging. To set this go to the Run menu, click either the Run or Debug option that is expandable, then click Other. In the “Url or path to launch” section you can enter your run and debug wrapper locations.

Now lets look at the JS file.

The Javascript Layer

dwrWrapper.js


/**
 * Receives requests from AS3 for a Dwr call
 * @param wrapUUID The unique identifier for the wrapper
 *   used to make the call
 * @param jsMethodName Name of JS method to call
 * @param jsMethodArguments Arguments for JS method
 * @param asCallbacks The callback object containing AS3
 *   function UUIDs
 */
var dwrCall = function(wrapUUID, jsMethodName,
    jsMethodArguments, asCallbacks){

  var methodToCall = eval(jsMethodName);

  // add callback to arguments
  jsMethodArguments.push(buildDwrCallback(
    wrapUUID, asCallbacks.callback,
    asCallbacks.errorHandler));

  // call the desired JS method
  methodToCall.apply(null, jsMethodArguments);
}

/**
 * Constructs a callback map for a Dwr call.
 * @param wrapUUID The UUID of the AS3 wrapper
 * @param asCallback UUID of the AS3 function to call
 *   upon successful callback
 * @param asErrorHandler The UUID of the AS3 function to
 *   call if there is an error
 * @returns A JS callback object for a DWR call
 */
var buildDwrCallback = function(wrapUUID, asCallback,
    asErrorHandler){
  var jsCallback = handleDwrCallback;
  var jsErrorHandler = handleDwrError;

  var wrappedJsCallback = jsCallback;
  var wrappedJsErrorHandler = jsErrorHandler;

  // wrap the callbacks in functions that call them
  // with dwr response + AS3 callback info
  jsCallback = function(dwrResponse){
    return wrappedJsCallback.call(null, wrapUUID,
      dwrResponse, asCallback);
  }
  jsErrorHandler = function(dwrResponse){
    return wrappedJsErrorHandler.call(null, wrapUUID,
      dwrResponse, asErrorHandler);
  }

  return {callback: jsCallback,
    errorHandler: jsErrorHandler};
}

/**
 * Called when a DWR function returns -
 *   sends response to AS3 code
 * @param wrapUUID UUID for the wrapper
 * @param dwrResponse The actual response received
 *   from the server through Dwr
 * @param asCallback UUID for the AS3 function to call
 */
var handleDwrCallback = function(wrapUUID, dwrResponse,
    asCallback, extraArguments){

  var asCallbackCall = "document['flexObject'].
    handleJsCallback" + wrapUUID.replace(/-/gi, "") +
    "(dwrResponse, asCallback)";

  eval(asCallbackCall);
}

/**
 * Called when a DWR call fails -
 *   sends error to AS3 code
 * @param wrapUUID UUID for the wrapper
 * @param dwrResponse The actual error response received
 *   from the server through Dwr
 * @param asErrorHandler UUID for the AS3 function
 */
var handleDwrError = function(wrapUUID, dwrResponse,
    asErrorHandler){

  var asErrorCall = "document['flexObject'].
    handleJsError" + wrapUUID.replace(/-/gi, "") +
    "(dwrResponse, asErrorHandler)";

  eval(asErrorCall);
}

Some comments on this JS code:

Instead of passing the string names for the AS3 functions to use for callbacks and errors I used UUIDs. On the AS3 side a mapping from UUID to function reference is maintained.
I use eval above to call the AS3 functions registered on ExternalInterface because the apply function isn’t available.
I assigned each usage of the wrapper a UUID as well that is taken into account when calling back into AS3. By appending these wrapper UUIDs to the callback and errorHandler AS3 functions registered on the ExternalInterface we can ensure the Dwr callbacks are going to the correct AS3 code. Each DWR controller will have a separate wrapper instance on the AS3 side each with its own UUID.
In the buildDwrCallback function we are using the technique described here to pass the AS3 callback/errorhandler information through the DWR call such that we can use it in the handleDwrCallback/handleDwrError functions. If the above makes sense (and it will look weird to someone new to JS) you should be able to see how extra arguments could be passed on top of the callback info for the benefit of your AS3 code.
I strip the dashes from the wrapper UUIDs because they were causing problems with ExternalInterface and calling the registered AS3 functions. You will notice below that the dashes are stripped to match on the AS3 side of things.

So that covers the Html/JS wrappers for running your Flex2 code against DWR. The last step is the AS3 class that knows how to communicate with this wrapper.

The Actionscript Layer

DwrWrap.as

package someLib
{
  import flash.external.ExternalInterface;
  import mx.collections.ArrayCollection;
  import mx.utils.UIDUtil;

  /**
   * This class provides access to Dwr.  It assumes
   * that flex is being used in an Html container that
   * has access to dwrWrapper.js.
   * Each instance wraps a single Dwr controller.
   */
  public class DwrWrap
    private var functionMap:Array;
    private var wrapUID:String;

    public var controllerName:String;

    /**
     * Constructor for DwrWrap.  Initializes function
     * map and registers ExternalInterface callbacks
     * @param controllerName The name of the
     *   JS/Dwr controller to use
     */
    public function DwrWrap(controllerName:String){
      this.controllerName = controllerName;
      this.functionMap = new Array();
      this.wrapUID = UIDUtil.getUID(this);

      // register AS3 callbacks as name + UUID
      ExternalInterface.addCallback("handleJsCallback" +
        this.wrapUID.replace(/-/gi, ""),
        handleJsCallback);
      ExternalInterface.addCallback("handleJsError" +
        this.wrapUID.replace(/-/gi, ""),
        handleJsError);
    }

    /**
     * Method that calls JS using ExternalInterface.
     * @param jsMethodName JS/Dwr function to call
     * @param jsArgs A single argument or an
     *   ArrayCollection of arguments to use in JS call
     * @param callback An untyped object with callback
     *   and errorHandler properties
     */
    public function serverCall(jsMethodName:String,
        jsArgs:*, callback:Object = null):void{

      var jsCallback:Object = new Object();

      // map UUIDs to AS3 callback function references
      var callbackUID:String =
        UIDUtil.getUID(callback.callback);
      jsCallback.callback = callbackUID;
      functionMap[callbackUID] = callback.callback;

      var errorUID:String =
        UIDUtil.getUID(callback.errorHandler);
      jsCallback.errorHandler = errorUID;
      functionMap[errorUID] = callback.errorHandler;

      // make sure jsArgs is an array - assumption in JS
      if (jsArgs is ArrayCollection){
        jsArgs = jsArgs.toArray();
      }
      else if (!(jsArgs is Array)){
        var argArray:Array = new Array();
        argArray.push(jsArgs);
        jsArgs = argArray;
      }
      else if (jsArgs == null){
        jsArgs = new Array();
      }

      ExternalInterface.call("dwrCall", this.wrapUID,
        (controllerName + "." + jsMethodName), jsArgs,
        jsCallback);
    }  

    /**
     * This method receives the callback from JS and
     * then calls the appropriate AS3 callback
     * @param dwrResponse The response from Dwr
     * @param asCallback UUID of AS3 function to call
     */
    public function handleJsCallback(dwrResponse:*,
        asCallback:String):void{

      // use the UUID to look up the function reference
      var callbackFunction:Function =
        functionMap[asCallback];

      if (callbackFunction != null){
        delete functionMap[asCallback];
        callbackFunction(dwrResponse);
      }
    }

    /**
     * This method receives errors from JS and
     * then calls the appropriate AS3 errorHandler
     * @param dwrResponse The error response from Dwr
     * @param asErrorHandler UUID of the AS3 function
     */		
    public function handleJsError(dwrResponse:*,
        asErrorHandler:String):void{

      // use the UUID to look up the function reference
      var errorFunction:Function =
        functionMap[asErrorHandler];

      if (errorFunction != null){
        delete functionMap[asErrorHandler];
        errorFunction(dwrResponse);
      }
    }
  }
}

Some comments about the AS3 code:

Notice that by routing all callbacks and errorHandlers through 2 AS3 functions we only have to register those 2 function on the ExternalInterface. These two functions need to be registered for each DWR controller so we append a UUID to the names to ensure uniqueness of function names.
AS3 functions to use for DWR callbacks and errors are sent to JS as UUID strings. When we get calls back from the JS we can then look up the actual function references and call them directly. This is nice because it allows the AS3 callback object to contain actual references and not strings.

Finally, you can make DWR calls from AS3 that look like this. Assume you have a DWR controller exposed as someDWRController.js and that this controller has an addNumbers method that takes 2 arguments and returns the sum.

Usage Example

// instantiate a wrapper for the DWR controller
var wrap : DwrWrap = new DwrWrap("someDWRController");
// call a function on that controller
wrap.serverCall("addNumbers", 5, 6,
  {callback: someASCallback, errorHandler: someASErrorHandler});

// the callback and errorHandler
public function someASCallback(theSum): void{
  Alert.show("Should be 11: " + theSum);
}
public function someASErrorHandler(errorInfo): void{
  Alert.show("Stuff broke");
}

This should work for making basic DWR calls. In our project Mark cleaned this code up, allowed extra passthrough arguments to be passed to callback functions, injected the DWR Controller JS file content dynamically, and then integrated this logic into our Cairngorm-based application. DWR services were then listed and looked up from a Services.mxml file and executed by delegates like any other service. The callback and errorhandler functions for DWR calls were bound to the result/fault methods of our command objects and upon return the results from Dwr were translated into typed objects. I won’t share that code because I didn’t write it.

Hopefully just this will help someone stuck with Flex2 in an environment where DWR is being used. If you haven’t already read my opinions on Flex2 and you are trying to decide between Flex2 and Html/JS you should.

If you are determined to use Flex2 with DWR you really need to read about the ExternalInterface bug that may force you to refactor your value objects depending on their complexity.

Flex2 Opinions

Joe — Wed, 28 Mar 2007 12:00:12 +0000

At work we decided to give Flex2 a lengthy, honest trial run as a replacement of the Html/JS used for an important component of our application suite. It is an intranet sort of application where we controlled the environment and it seemed like a promising place to put Flex2 to work.

After months spent learning and becoming accustomed to Flex2 our team decided it simply is not a technology ready for building enterprise applications. We are now taking a pragmatic approach: continuing with Html/JS but using the Flex-Ajax bridge for charting, graphics, and multimedia. In my opinion these are things Flex2 is great for.

To anyone that may be considering the Flex2 platform, know that you will find large crowds of evangelists (many of them Adobe employees) on the web describing trivial applications and simple examples to make Flex2 sounds like a fantastic way to solve problems. When we starting dealing with hard problems our experience was extraordinarily negative.

My biggest issues with Flex2 are the following. My viewpoint here is largely a comparison of Flex2 against Html/JS for building rich internet applications. Though based on months of nontrivial development these are my individual opinions.

It is not production ready. It is not a stable product. We have an internal wiki page titled “Flex bugs & gotchas” and its content has grown to a substantial size. Take a look at the latest hotfix notes to see what sort of nontrivial issues are just being addressed.
The community is small and inexperienced. Flexcoders and Flexcomponents are great groups if you have trivial questions. We asked a half dozen hard questions over the last few months and those were met with either complete silence or confirmation from a Flex team member that we had bumped into another known bug with no confirmed fix date. Compare this to the mailing list and forums for Spring, Hibernate, JS/Dojo/Prototype/DWR, or PostgreSQL where your questions are answered so thoroughly that you feel questions far more difficult than the ones you face have already been conquered by people more knowledgeable. Patches to specific issues are also the norm in these communities. Maybe patching the Flex2 SDK is an easy process, I didn’t try it.
The livedocs are down more than they are up. These would be pretty helpful if they were reliable.
Existing examples of reasonably nontrivial Flex2 applications on the web are really buggy. Go visit the Flex2 search appliance or the golf store that someone made (can’t remember the url right now – will look it up) with a debug version of the flash player installed and watch the errors bubble out.
The tools are not established. Flex-Ajax bridge is bugged. ExternalInterface is profoundly bugged (see my earlier post about this). The Flex Ant tasks are bugged (only with the most recent update did a failed compile cause a build to fail and most of the command line arguments aren’t stubbed into their corresponding ant tasks). Want to display some HTML in your application? Get ready to wrap your application in an HTML container, invoke an iframe in that container, and then hack massage it into place over your application so that it looks like it belongs there. Granted, all of these tools are still in Adobe labs so it is fair to expect alpha-ish quality, but I need real tools now to build applications with.
Adobe’s priorities with this platform are exceedingly frustrating. They are working on Apollo and Flex3 while huge, show-stopping bugs exist in Flex2 and prevent it from being a viable alternative to more established technologies.
Flex Data Services might be a good thing. It might even make the platform reasonable for an enterprise-grade application but it costs $20k per CPU. I can buy a beefed out database server with dual procs, 16gb RAM, and a 16 disk sata raid array for less than the price of licensing one of its CPUs.
It is extremely difficult to find talent with Flex2 experience or a desire to obtain it. We want full-time, exceptionally intelligent software engineers. Our efforts to find Flex2 developers brought us contractors that refused to work onsite and people that think knowing individual, popular languages is more important that understanding the concepts that make them the same. We work with area recruiters and the vibe from them was much the same.
The evangelists drive me nuts. They are too worried about presenting Flex2 as the slickest piece of engineering ever created to discuss the real issues that prevent real work from being done with it.
While not unique to Flex2, the compile cycle is killer. I found over time, as our application grew, it completely crushed my productivity. Compare this to pressing F5 with the alternative.
FlexBuilder is worth $10 and not $749 (that’s with charting, but if you are using Flex2 you should take advantage of the charting). No refactoring, no find usages, no code generation, plugin problems (we had a hard time getting subclipse and some other plugins to work with it), awfully unimpressive automatic importing, ctrl-space completion, ctrl-click, etc, etc. Comparing the Java intelligence of IntelliJ/Eclipse to FlexBuilder isn’t fair or applicable, but comparing the Html/JS intelligence of those tools to FlexBuilder is and they put FlexBuilder to shame. The design view, though helpful in some situations for coders, is not a design tool. Our designer hated it. The debugger pales in comparison to a Firefox instance with Firebug and the web developer toolbar installed.
Flex2 without FDS does not play well with anything other than xml or web services. I can tell by the search terms used to find this blog that people are interested in using DWR with Flex2. We built an Html/JS container that provided a communication layer between Flex2 and our exposed DWR objects and though it worked for basic calls (did not handle timeouts, call batching, or reverse Ajax/Comet) it meant involving a 3rd debugger in development and it was really just very difficult to work with so many layers at the same time. Regardless I will try to post the code in the future to help anyone else in the same boat.
The whole platform feels disconnected. If you pull down the source code for the core Flex2 classes and take a peek, you will find big balls of logic commented out. It is as if they starting commenting out things that are broken until it was stable enough to ship. Also the impression I get from working with Flex2 is that Adobe has a grabbag of developers – some quite good, and some quite junior – and then turned them loose on different pieces of the compiler and FlexBuilder. The result is a language that is far less powerful than a fully dynamic language like Javascript and unawareness in some areas of the platform that other areas exist. A concrete example: Flex2 uses metadata internally ([Bindable], [ArrayElementType], etc) and essentially had a wonderfully powerful metadata construct written and working. You couldn’t use this metadata for your own code until 2.0.1 because they hadn’t thought to expose it. Even now you have to build with ANT and use some command line arguments that aren’t stubbed out in the Flex ANT tasks to make it work. Someone smart wrote the metadata code and it wasn’t leveraged in the rest of the platform. That feeling occurs repeatedly.

That said, Flex2 is great for making pretty things involving charting or multimedia. Our frustration began and grew as we tried to complete the nuts and bolts of a large enterprise application with it. If your application is heavy on the areas where Flex2 shines, then it may be worth the frustration. Perhaps your experience has been different? All I know is we are back to Html/JS using DWR for transport and Xulrunner/Firefox for the environment and getting work done again. Apollo sounds promising, but I am more excited about Xulrunner and Firefox3.

My advice is this: If your team has 1 or more of these following trends in place, I would strongly discourage a move to Flex2:

You are building an application for the public (you do not control the environment or user base). Flash player 9 is not ubiquitous and the 96% flash player penetration number that the evangelists trumpet include all versions of Flash player. Flash player 9 is no where near 96%. Though it wouldn’t be fair to not mention that MySpace and Youtube are going to help Flash player 9′s penetration number increase faster than version 8′s.
You have solid Html/Css/JS knowledge in place now. While I prefer this stack enourmously over Flex2 I also realize there is a learning curve involved. If you have already conquered that learning curve rest easy knowing you are using the superior tool.
You are building business applications or applications that aren’t heavy multimedia. Flex2 is great for multimedia and I don’t hesitate to admit its superiority here. Keep in mind that you can use the ExternalInterface or the Flex-Ajax bridge to use Flex2 widgets inside your Html/JS with relative ease.
You are building large applications. For smaller applications Flex2 is fine. Cairngorm helps (and in fact may help you keep your sanity) but it doesn’t solve all of the problems.
You don’t currently have nightmares about scrollbars and don’t want to start having them This will only make sense to someone who has worked with Flex2.

Needless to say, probably the last Flex-related post for this new blog except for the DWR bit or unless I have something interesting to share related to the charts and multimedia strengths we will continue to use. I tried to keep this post objective; it does represent 6 man months of experience. I love many of Adobe’s products and I recognize that there are several cases where Flex2 would be a great tool. Large, enterprise applications is not one of those areas.

That said, it is a new tool in my toolkit and because it has obvious strengths I can certainly see myself using it for those strengths in the future.

Generating DWR Javascript Files

Joe — Sun, 18 Mar 2007 16:17:30 +0000

I’ve talked about DWR before. It is a mature project that is well constructed and very useful. DWR generates Javascript files at runtime that you can include in your client-side code and use to call exposed Java methods. A few days ago I wanted to generate these files as part of our normal build process so that we could open and read them directly. More importantly (for me at least) I wanted to have them in my project directory so that IntelliJ could index them. This may seem trivial but I am obsessive about letting my IDE aware of as much as possible and I found this to be a huge help when writing client-side Javascript. Now I can ctrl-space complete, ctrl-click, and everything else IntelliJ can do with Javascript on these DWR Javascript files.

I posted my plan to the dwr mailing list and as usual the creator of DWR himself showed me exactly how to accomplish this when using a dwr.xml file for configuration and when not using Spring. If this is your environment I encourage you to search that list. There is a great community there and I have found it extremely helpful and responsive many times.

That said, I wanted to do this in our Spring project using DWR 2.0rc2 with an annotations based DWR configuration. It was pretty easy to take Joe’s example and adapt it to this case. Here is the core of the Java code to get started.

First the imports I am using:

import org.directwebremoting.util.FakeServletContext;
import org.directwebremoting.util.FakeServletConfig;
import org.directwebremoting.impl.ContainerUtil;
import org.directwebremoting.impl.DefaultContainer;
import org.directwebremoting.extend.Remoter;
import org.directwebremoting.spring.SpringContainer;
import org.springframework.context.support.ClassPathXmlApplicationContext;
import javax.servlet.ServletContext;
import javax.servlet.ServletConfig;

Now the code that sets up a DWR Remoter object which we can request Javascript content from:

ClassPathXmlApplicationContext appCtx =
  new ClassPathXmlApplicationContext("testContext.spring.xml");

Map initParameters = new HashMap();
// comma-delimited list of annotated classes
initParameters.put("classes", "FooAjaxController");
// telling DWR to use a different Container subclass
initParameters.put("org.directwebremoting.Container", 
  "org.directwebremoting.spring.SpringContainer");

ServletContext servletContext = new FakeServletContext();
ServletConfig servletConfig = new FakeServletConfig("dwr-test", 
  servletContext, initParameters);

DefaultContainer container = 
  ContainerUtil.createDefaultContainer(servletConfig);
((SpringContainer)container).setBeanFactory(appCtx);
ContainerUtil.setupDefaultContainer(container, servletConfig);
ContainerUtil.configureFromAnnotations(container);

Remoter remoter = (Remoter)container.getBean(Remoter.class.getName());

// do what you want with the Javascript content

appCtx.close();

In the above code, before the appCtx.close() line you can pull Javascript content out of the Remoter object like this:

remoter.generateInterfaceScript("fooAjaxController", "/fake/path");

The second parameter is the path you want the Javascript files to use for accessing corresponding Java servlets and in this case they didn’t matter. fooAjaxController is the Javascript name for the DWR Java class. My preference for naming of DWR classes is that all Java class names end with AjaxController and that the bean name and DWR Javascript name are the same and the camelcase version of the class name.

From here it is pretty straight forward to do whatever you want with this Javascript content from the remoter. In my case I pulled the bean definitions out of the Spring context, separated out all beans with a name ending in AjaxController and then built lists of bean names and class names at run-time to use for the classes parameter and for reading the content back and writing it out to files. This way nothing had to be hard-coded and changes made to the bean configurations in our Spring context were automatically accounted for.

Hopefully someone else finds this useful.

DWR Annotations & Spring

Joe — Thu, 15 Mar 2007 04:43:05 +0000

We recently upgraded DWR in our primary product from version 1.1.4 to version 2.0rc2. The newest version offers a pile of great improvements on an already excellent piece of code. One of my largest motivations for the upgrade was the new annotation-based configuration option and the resulting elimination of our dwr.xml file that had grown to be large and confusing. The signatures block can be completely dropped with Java 1.5 and the create and convert elements can now be represented 100% in annotations.

Our codebase was built with Spring in mind from the start and I hit a small block when trying to get the annotations to work. This page provides a start but doesn’t specifically mention Spring. Here is what an annotated RemoteProxy class looks like with Spring factored in:

import org.directwebremoting.annotations.RemoteProxy; import org.directwebremoting.annotations.Param; import org.directwebremoting.annotations.RemoteMethod; import org.directwebremoting.spring.SpringCreator;


/**

 * A Java controller exposed via DWR

 */

@RemoteProxy(

    creator = SpringCreator.class,

    creatorParams = @Param(name = "beanName", value = "someAjaxController"),

    name = "someAjaxController"

)

public class SomeAjaxController{

@RemoteMethod public int trivialExample(int param1, int param2){ return param1 + param2; } }

That’s it. As someone who lived mostly in the client-side of DWR I wasn’t familiar with the creator concept and it took me more than a single google run to find the creator/creatorParams being used clearly in the RemoteProxy annotation. In our project we standardized on making the beanName and Javascript name the same but they could be different in the above example.

Hopefully this saves someone else a few additional minutes of searching. I hope to post about Spring and DWR more in the future – they are both excellent projects with experienced, helpful communities.

Flex2 Resources

Joe — Thu, 08 Mar 2007 18:49:17 +0000

I mentioned flexcoders in my last post but didn’t provide a link or more information. I thought it would be helpful to list the Flex2 resources I rely on in one place as I know I struggled to find resources initially and the more places indexed in Google for people to find the better.

Point 1: Don’t use the forums at www.Adobe.com for Adobe Flex2 questions. I had 0 luck here and they are virtually silent.

Adobe Labs
This is a fairly helpful site both because it provides access to tools and utilities (like the Flex2 ANT Tasks, Flex-Ajax bridge, and lots more) and because the wiki pages associated with the various projects are very active.

flexcoders Yahoo Group
Why is Yahoo Groups being used? I don’t know, but this is probably the best place to post questions and look for answers. You’ll want to use Google to search the group though as my searches fail if I use more than 3 or 4 words (a fault of Yahoo).

flexcomponents Yahoo Group
Another good group – focused on component development.

The Livedocs
These can often be better (when they aren’t down) than the help system built into FlexBuilder due to the comment sections for each topic article and the fact that the Livedocs team can fix corrections immediately and keep them more up to date.

Cairngorm Documentation
Cairngorm is a solid framework for building substantial Flex2 applications. If you are working with Flex2 I highly recommend becoming familiar with it.

I have found all of the above to be quite helpful.