Dec 27, 2014

SQL benchmarking tool for Java

This article describes a simple java tool for recording and replaying MySQL queries while measuring their execution times. The replay mechanism makes the measurements comparable across different environments.

Why do we need another benchmarking tool? What's so special about this one?
  • It's capable of reading and replaying MySQL's 'general query logs', not 'slow query logs' and not tcpdumps. Percona Playback in version 0.3 doesn't supports 'general query logs', version 0.6 seems to also support 'general query logs'.
  • It's measuring execution times of all queries and aggregating identical executions. pt-query-digest supports general query log, but only without measuring execution times.
  • It's written in pure Java, making it the most realistic way of running queries against the server if you're developing a Java application.
  • It's small and easy to use, not a full blown full featured benchmarking suite.
  • It does not offer advanced filtering methods, just restriction to connection ids and prefix matchings.
  • Measurements can be exported in either CSV for easy Excel import or in JETM-Style.


In some cases, it's necessary to run a simple single-component benchmark on a MySQL server. In opposition to a full-stack benchmark, a single-component benchmark only targets the isolated database. It can easily be modified to target the network connection, too.

The benchmarking strategy presented in this article, records all queries during a real application run. Using a simple benchmarking tool written in java, allows you to re-run all recorded queries multiple times and record all query runtimes.

Besides slow query log and the built in profiling mechanisms, MySQL offers the often unnoticed general query log. You can use this mechanism to log all queries which run while executing various use cases in your application. My idea was to use the general query log to re-run all queries on the MySQL server and measure their timings. This enables me to reproduce the same use case multiple times on the same or on different MySQL servers or in different network configurations. Measuring the performance based on the same query execution makes measurements comparable.

This strategy is useful if you want to:
  • simply record and replay MySQL queries including all MySQL-parameters while executing a specific use case
  • compare MySQL performance on different server instances
  • compare MySQL performance in different network configurations to spot possible network bottlenecks
  • monitor MySQL performance continuously
  • spot and count slowest queries while executing a possibly complex use case

The following picture shows the whole benchmarking strategy in a sequential process: You execute a use case in your application with enabled general log in mysql. You can use this logfile to re-run all queries which were executed on the server during your usecase. The benchmark results can be viewed as text files or evaluated after an Excel import.


All I needed to do is writing a small tool which is able to parse the general log file and re-run all queries against a configurable mysql server. The output should be configurable in JETM-Style or csv-format which allows me to see all query timings in average, min, max, total and the count of each querie's excution.

Getting the log

Many developers know of the slow query log and the built in profiling mechanism which allow you to analyze and optimize single queries. But it seems not so many developers know about the general query log which is simply activated by setting one sql variable like this:

SET GLOBAL general_log = 'ON';

This method works on any platform and does not require a server restart.

To specify a log file name, use --general_log_file=file_name. To specify the log destination, use --log-output (as described by dev.mysql.com in " Selecting General Query and Slow Query Log Output Destinations")

The tool: MySQLQueryBenchmark

MySQLQueryBenchmark is a java console application, taking a slow query log, executing the contained queries and returning their execution times as JETM-measurements or in csv-format.

Usage for impatient:
# java -jar mysqlquerybenchmarking.jar -log LOG_FILE -db DATABASE -p MYSQL_USERNAME -u MYSQL_PASSWORD -ignore show,connect,set

The results will be stored in results.txt in JETM-Format by default. If nothing different is specified, localhost:3306 is being used as mysql server.

There are useful command line options which allow you to only execute specific queries, which match a prefix or are logged under one connection-id.


After executing all queries, the output in JETM-Format will look similar to this:



In some cases, you proably want to compare multiple results or evaluate the values in a program like Excel. The benchmarking tool supports csv output using the parameter -f csv. The output will look similar to this:


This file can easily be opened in Excel for further evaluation.

Digging deeper

As a developer, I cannot resist telling you how the tool is implemented.

  • For command line parsing, the tool uses jcommander. Parameters are simply stored in a parameter class like this:

  • The class QueryParser accepts one line from the log file, a restriction-id and a prefix to parse a query from that line if possible:

  • The class QueryBenchmark executes all queries and measures their timings using JETM:

  • For query execution, the class SQLStatementExecutor fires against the MySQL-server:

  • For csv output, I simply created a class CsvRenderer derived from JETM's MeasurementRenderer. I used the SimpleTextRenderer as template and adjusted it to generate a csv output.

Downloads

Git tepository
Precompiled jar
This tool is open source under Apache License, Version 2.0.


Enjoy benchmarking with this tool.
Have a nice day!

Nov 14, 2014

How to write a PostFilter for Solr 4.9


The problem

When working with Solr, there might be the time where a custom filter functionality is needed. Solr provides different possibilities on how to filter results such as ValueSourceParsers or QParserPlugins

In our own case we had stored a boolean expression in each document, reaching from simple true to very complex expressions. This string was post evaluated to check whether a document should be in the results or not.

Doesn't sound very fancy right? Just calculate it in advance!
But the boolean expression had to be interpreted differently depending on our filter query parameters. So the boolean's results could massively vary for each query.

This post is about explaining how to implement a QParserPlugin with PostFilter capabilities.
For the purpose of conveying the concept, the filter itself will be very simplified. In fact it will only be a modulo operation. Since Solr 4.0 there is already a mod function implemented but for the matter of understanding the idea it will be appropriate. 


What is a PostFilter

A PostFilter in Solr is a mechanism which will be executed after the query q and the filter queries fq as it may involve expensive logic or depend on document information.


What we will be doing

  • Write a basic PostFilter function for Solr 
  • Extend it to be adjustable
  • Give a thought on performance (using DocValues)


Implementing a basic PostFilter

It is necessary to implement two java classes a QueryParserPlugin and a Query.


1. ModuloQueryParserPlugin.java

Let's create the ModuloQueryParserPlugin class which extends the QParserPlugin. This is the class we will be linking to our Solr later. The QParserPlugin is abstract which forces us to implement two methods. For our filter we can leave the init method empty and focus on the createParser method.
public class ModuloQueryParserPlugin extends QParserPlugin {

  @Override
  public QParser createParser(String qstr, SolrParams localParams,
    SolrParams params, SolrQueryRequest req) {
     
     return new QParser(qstr, localParams, params, req) {
       
       @Override
       public Query parse() throws SyntaxError {
         return new ModuloQuery();
       }
     };
  }

  @Override
  public void init(NamedList args) {
    // Emtpy on purpose
  }
}


2. ModuloQuery.java

In our ModuloQueryParserPlugin we return an ModuloQuery (where all the filtering will be done). Create a new class called ModuloQuery which extends ExtendedQueryBase and implements the PostFilter interface.

Now we have to bend to the PostFilter API documentation which states two things:

Firstly, we need to make sure that our filter will always be executed after the query and the filter-query, for this purpose let's override the getCost method.
@Override
public int getCost() {
  // We make sure that the cost is at least 100 to be a post filter
  return Math.max(super.getCost(), 100);
}
With this functionality we will still be able to change the cost for the function but making sure that it will always be a post filter.

Secondly, the getCache method should always return false.

@Override
public boolean getCache() {
  return false;
}
From the PostFilter interface we need to implement the getFilterCollector method which
"returns a DelegatingCollector to be run after the main query and all of it's filters, but before any sorting or grouping collectors" PostFilter API Doc
The DelegatingCollector has one method necessary for our filtering purpose.
@Override
public DelegatingCollector getFilterCollector(IndexSearcher idxS) {

  return new DelegatingCollector() {

    @Override
    public void collect(int docNumber) throws IOException {
      // Our filter magic -> call super.collect()
    }
  };
}
Before we implement our filter logic, let me explain how Solr handles the filtering. Solr chains all the filter functions sorted by their cost parameter (functions with higher cost will be run later).
For instance, if there are two post filters (cost >= 100) the one with the lower cost will delegate its result set to the one with the higher cost.

We tell Solr to keep a specific document in the result set by collecting it. This is where the collect method comes to play. For us to make a document stay in the result set we need to call super.collect().


Now lets write our awesome filter logic. Only documents with their id being a multiple of 42 will be in our result.
@Override
public void collect(int docNumber) throws IOException {
  // To be able to get documents, we need the reader
  AtomicReader reader = context.reader();

  // From the reader we get the current document by the docNumber
  Document currentDoc = reader.document(docNumber);
  
  // We get the id field from our document
  Number currentDocId = currentDoc.getField("id").numericValue();

  // Filter magic
  if (currentDocId .intValue() % 42 == 0) {
    super.collect(docNumber);
  }
}

3. Important information about hashCode and equals

This is a static filter as we can't change the filter method. So we do not need to override hashCode and equals. In some further steps we are going to make our post filter adjustable and therefore those two methods must be overridden.

Side Note:
If you expect different results but always get the result from the first query. Take a look at those two methods.

4. Hooking up with Solr (small notes)

  • Build a jar file from our code
  • Put this jar in the lib folder
    • "...\solr-4.9.0\example\solr\collection1\lib"
  • Edit solrconfig.xml:
    • If your running stock Solr, there might be a comment saying Query Parsers after which you can add this:
<queryParser
name="ModuloPostFilter"
class="com.yourpath.ModuloQueryParserPlugin"/>
    • ModuloPostFilter is the name under which Solr will execute our ModuloQueryParserPlugin. 
  • Restart Solr


5. Applying our PostFilter

For running the filter I used the Solr admin panel and a stock Solr installation with 2 million documents each only with an unique id and no further information.

No filter

To make Solr use our PostFilter just add {!ModuloPostFilter} to the fq-field.


Post filter used

As we hoped, only documents with their id being a multiple of 42 are shown.



Extending our PostFilter to be adjustable

Now that we know how its done, our ModuloPostFilter seems to be boring, since filtering is always done the same way (id modulo 42).


1. ModuloQueryParserPlugin.java

Therefore ModuloQueryParserPlugin needs some editing in order to have access to the parameters in our collect method.
Before diving into code, let me explain what the parameters params and localParams in the createParser method are about.

params:

Includes all the Solr request parameters (q, wt, fq, etc).

For instance params.getParams("fq") will return an array of strings with one string for every fq-field. So we could possibly bind our filter to any given parameter.


localParams:
Includes all the parameters for our function. With one speciality, there are two defaults which are always set (type: the functions name; v: the part behind the closing curly bracket).

For example fq={!ModuloPostFilter}... will result in type=ModuloPostFilter and v=....

Any key=value pair can be defined within the curly brackets {!ModuloPostFilter key1=value1 key2=value2}.

This is the option we are going to use for the sake of our extension {!ModuloPostFilter modulo=x}.


Edit the createParser method to look like:

@Override
public QParser createParser(String qstr, SolrParams localParams,
  SolrParams params, SolrQueryRequest req) {
     
   return new QParser(qstr, localParams, params, req) {
       
     @Override
     public Query parse() throws SyntaxError {
       // The ModuloQuery knows the function parameters
       return new ModuloQuery(localParams);
     }
   };
}


2. ModuloQuery.java

We must add a constructor which checks for the desired key-value pair (modulo=x) in the localParams.
public class ModuloQuery extends ExtendedQueryBase
 implements PostFilter {
  private final int moduloX;

  public ModuloQuery(SolrParams localParams) {
    // We try to get the modulo pair
    // if there is none we will still be using 42
    moduloX = localParams.getInt("modulo", 42);
  }
  
  // previously added methods 
  // ...
}
The collect method must be adapted to use our new moduloX field.
@Override
public void collect(int docNumber) throws IOException {
  AtomicReader reader = context.reader();
  Document currentDoc = reader.document(docNumber);
  Number currentDocId= currentDoc.getField("id").numericValue();

  // new Filter magic
  if (currentDocId.intValue() % moduloX == 0) {
    super.collect(docNumber);
  }
}
Let your IDE implement hashCode and equals for ModuloQuery making use of the moduloX field.

3. Applying the adjustable PostFilter

We can now modify our filter call in the admin panel.

Using a different number for filtering

As you can see, we changed our filter behavior by setting the modulo pair.

Improving the performance

Iterating over 2 million document seems to be a bit of a performance issue even though our documents are structured very simple (in fact only having an id field).
As accessing the current document via reader.document(docNumber) is a relatively expensive call, the result set should be limited before running our post filter.


This illustration shows, that getting the document from the reader can lead to slow queries. For the case of improving the query time we are going to take a look at DocValues.


2. What are DocValues and how can we use them?

"With a search engine you typically build an inverted index for a field: where values point to documents. DocValues is a way to build a forward index so that documents point to values." - Solr Wiki
Let see how forwarded and inverted index are distinguished.

Forward-Index:
{
  'doc1': {'field-A':3, 'field-B':2, 'field-C':3},
  'doc2': {'field-A':1, 'field-B':3, 'field-C':4},
  'doc3': {'field-A':2, 'field-B':3, 'field-C':2},
  'doc4': {'field-A':4, 'field-B':4, 'field-C':4}
}
Inverted-Index:
{
  'field-A': {'doc1':3, 'doc2':1, 'doc3':2,'doc4':4},
  'field-B': {'doc1':2, 'doc2':3, 'doc3':3,'doc4':4},
  'field-C': {'doc1':3, 'doc2':4, 'doc3':4,'doc4':4}
}

The iteration over the forwarded index is faster in this context. It is not an all in one solution (but definitely worth mentioning), check the link for appropriate use cases.

To make a Solr field use this capability just add docValues="true" to that fields definition (in the schema.xml).

This is how our id field now looks like:

<field name="id" type="long" indexed="true" stored="true"
required="true" multiValued="false" docValues="true"/>
Usually indexing, storing and using docValues on the same field should not be done as Solr can't profit from it's optimization algorithms. For this post let's ignore that advise.

Dont forget, before we can use DocValues in our code Solr must re-index its data (Solr Wiki - HowToReindex).



3. ModuloQuery.java

We must change the collect method to make use of the new DocValues capability for the id-field.

@Override
public void collect(int docNumber) throws IOException {
  // SLOW: Document currentDoc = reader.document(docNumber);
  // FAST: gets the id field from the document with the docNumber
  long currentDocId = DocValues.getNumeric(context.reader(), "id")
                               .get(docNumber);

  if (currentDocId % moduloX == 0) {
    super.collect(docNumber);
  }
}


4. Testing the difference

Now let's apply the filter again and see the impact.

Using DocValues for accessing the id

Referring to the previous version, we can observe a decrease from ~3500ms to ~120ms.
The performance boost can not be seen representative as we were using a simplified schema and it heavily depends on your data structure. For our production system the impact was comparatively low.
It's just a hint at whats possibly worth looking at for tweaking a post filters performance.


5. Final words on DocValues

This approach has some downsides and it's usage depends on your overall data structure and environment. It is also possible to change how the forwarded index is stored by Solr via the "docValuesFormat" attribute.

The example shown in this post is very artificial and can not be seen too serious but as a first impression on DocValues. Also consider using FieldCache instead for accessing the document fields if your unable to reindex.



Sources

Oct 16, 2014

Migrating a JSF-Based Web Application from Spring 3 to Java EE 7 and CDI

This is a detailed case study about the migration of a JSF-based web application from Spring 3 to Java EE 7 and CDI. I presented this talk at the JavaOne 2014 conference.

At first sight this didn’t seem to be too difficult. Both technologies are based on similar concepts and provide similar mechanisms for dependency injection (DI). So migrating pure annotation based bean wiring from Spring to CDI was obvious and straight forward.


But the deeper we looked into the source code the more Spring specific code we found with no direct and easy mapping in neither of JEE7, EJB3 or CDI available. Some of the problems and challenges we were confronted with were:

  • How to migrate XML based bean wirings? 
  • How to migrate FactoryBeans? 
  • How to migrate AspectJ based AOP proxies? 
  • How to implement Spring profiles for environment based wiring? 
  • How to implement custom bean scopes? 
  • How to implement custom Spring XML schema notations and wiring? 
  • How to migrate code that is built against Spring APIs? 
  • How to migrate unit test based on Spring Test?

To answer these questions the talk will present the patterns and strategies used to map, transform and migrate the different concepts from Spring 3 to JEE7with CDI. The talk will discuss the changes and implications of the migration on the system’s architecture.
You can find more details abou the talk in the JavaOne content catalog. The final slides are also available at Speaker Deck and SlideShare.

Using CDI annotations from Spring

A question from the audience was, if it is possible to do the migration and still be able to implement upcoming features in parallel. The answer was: yes, this is technically possible.

First of all, my advice would be to perform the migration as quickly as possible. Also try to keep the change set of the newly developed features as small as possible to reduce the merge efforts to an absolute minimum. Use a separate SVN or Git branch for further development and perform the migration directly in your trunk or master branch.

The Spring Framework uses a ScopeMetadataResolver to resolve the Spring based scopes of any bean definition. Since Spring 3 there already is a JSR330 supporting implementation: Jsr330ScopeMetadataResolver. But, this implementation only supports the @Singleton annotation per default, so we need to extend it a little.
/**
 * A custom CDI scope metadata resolver implementation. 
 */
public class CdiScopeMetadataResolver 
                           extends Jsr330ScopeMetadataResolver {
    /**
     * Register additional CDI scope annotations with their Spring 
     * scope equivalent.
     */
    public CdiScopeMetadataResolver() {
       registerScope("javax.enterprise.context.ApplicationScoped", 
               BeanDefinition.SCOPE_SINGLETON);
       registerScope("javax.enterprise.context.Dependent", 
               BeanDefinition.SCOPE_PROTOTYPE);
       registerScope("javax.enterprise.context.SessionScoped", 
               "session");
       registerScope("javax.enterprise.context.RequestScoped", 
               "request");
    }

    @Override
    public ScopeMetadata resolveScopeMetadata(BeanDefinition bean) {
       ScopeMetadata metadata = super.resolveScopeMetadata(bean);
       String scopeName = metadata.getScopeName();

       // CDI always uses scoped proxies except for @Dependent
       if (!BeanDefinition.SCOPE_PROTOTYPE.equals(scopeName)) {
          metadata.setScopedProxyMode(ScopedProxyMode.TARGET_CLASS);
       }

       return metadata;
    }
}
Basically, what this implementation does it to translate the CDI scope annotations to their Spring scope equivalent. Your newly developed components can now be annotated with CDI scope annotations instead, but under the hood you are still using Spring. That is the trick. Once you are done with the migration, these components should work without any further migration efforts. All that is left to do in your Spring code is to activate the scope resolver in your component scan definition:
<context:component-scan base-package="your.package.here"
    scope-resolver="...CdiScopeMetadataResolver"/>

Using Arquillian with Spock and the Shrinkwrap Groovy DSL

In my talk I described the migration of Spring Testing Framework based integration tests towards Arquillian as quite painful and cumbersome. At the end of my session Reza Rahman asked me what my experiences with Arqillian were.

Don't get me wrong now: Arquillian is a great framework for in-container testing of your CDI enabled artifacts and is by far the best I found. But: the shrink wrap process to build a testable and isolated CDI archive is very elaborate. Shrinkwrap might have a nice fluent API to do this but still you have to write a lot of Java code. So how can we improve this?

Testing Java code using Groovy, and especially using the Spock Framework, is in my opinion the way how tests should be written nowadays. Your tests are more expressive, you need less code and it is more fun writing Spock specifications than it is writing plain old JUnit tests.

So after searching the WWW for a few minutes I discovered that there is an offical Arquillian Spock Testrunner. This already felt a lot better, but still not 100% perfect. After a few more minutes I then discovered that there is a Shrinkwrap Groovy DSL that enables you to describe Shrinkwrap archives using Groovy closures. Now have a look at this:
@RunWith(ArquillianSputnik.class)
class TestableCdiArchiveSpec extends Specification {
    @Deployment
    def static JavaArchive "Create testable CDI archive"() {
        ShrinkWrap.jar {
            packages true, "your.package.here"
            asManifestResource EmptyAsset.INSTANCE, "beans.xml"
        }.build()
    }
}

Awesome and Groovy!

Oct 1, 2014

Simplifying JavaScript/HTML5 Rich Clients with Java EE

Simplifying JavaScript/HTML5 Rich Clients with Java EE 

JEE7 Angular CRUD Demo
JavaOne 2014 - CON6170
Johannes Weigend

This year at the JavaOne 2014 in San Francisco I did a session together with Geertjan Wielenga. The session was about combining JavaScript/HTML5 Clients with JavaEE. I developed live a JEE7 based CRUD application with an AngularJS frontend. In the linked video you can see every step of the live programming demo: 
JavaOne2014-JEE7-Angular-CRUD

The source is available on Github: https://github.com/jweigend/JEEAngularCRUD.
It contains the customer list, detail, update and delete functionality.
Customer List
It is a simple CRUD Singlepage HTML5 Application developed with NetBeans 8, Glassfish 4.1 and AngularJS. We use a JEE REST Server Backend and a HTML/JS Frontend. The HTML Frontend communicates with REST Services. The Server is stateless. 

In the first step I developed a  REST Backend for JavaDB by using NetBeans, Maven and Glassfish. In the second step I developed a HTML5 Frontend with the Yeoman Angular Generator and showed debugging and development of a Angular Single Page Application
Finally everything was put together in a single Deployment Unit by adding the HTML5 Application as a Maven Resource using the Maven WAR-Plugin.

If you want to repeat the demo by yourself follow these steps:


Step 1: Customer JEE7 REST Server

  • Create a new Maven based Web Application in NetBeans: 
    • Project->New->Maven->Web Application 
  • Use the „Create REST Services from Database“ Wizard to create REST Services direct by connecting to the sample database and choose the Customer table to generate your services.
  • Test the „Count“ Service by clicking the method in the Projects/Webservices Section
  • Create a „Cross Origin Resource Sharing Filter (CORS)“ in NetBeans to make sure the external HTML5 app can access our Webservice (After step 3 - the filter could be later removed for production).
Thats all. Amazing. Thats a perfect starting point for rapid prototyping. Since most enterprises have JEE Application Servers this Application will also meet enterprise requirements. Now we are ready to develop the HTML5 Client.


Step 2: Customer HTML5 AngularJS Client

Use the yo angular generator to generate a Angular HTML5 application. 

yo angular

Prerequisite: Make sure Yeoman is installed. 

$> grunt serve  #opens the browser and shows the generated page


After successful generation you can use grunt to show the generated sources in your browser. „grunt serve“ starts a simple NodeJS server which is a good first test.
We are ready to open the generated project in NetBeans. We use the „HTML Project from existing sources“ type of project.
The NetBeans project has some errors because the generated structure is not direct supported. The problem here is that the downloaded bower dependencies are one directory lower than the index.html file. This is currently not supported by the Netbeans toolchain.
We can fix this easily be changing the directory in the .bowerrc file from bower-components to app/bower-components.
After successful editing of the bower component file we kann download the bower libraries by clicking „bower install“ direct to the app directory in the project tree.
Now we have professional template for our project. The template uses Grunt as build automation tool. It uses bower as library download tool. It has the standard angular structure with unit tests and integration tests.
We now want to have two views. A list view to display the list of customers. A detail view to update or create a single customer. With the Yeoman generator we can create the two views inclusive the corresponding controllers. The reference to the controller code is automatically added to the index.html page.

yo angular:route customer-list
yo angular:route customer-detail

To make a first test we edit customer-list.html to loop over the generated array in the controller.
<div ng-repeat="thing in awesomeThings">
    {{ thing }}    
</div>

We change the awesomeThings array of string to an array of customer objects which contains a name and an email property. 

// customer-list.js
$scope.customers = [{
                    name: "Johannes Weigend",
                    email: "johannes.weigend@weigend.de"
                }, {
                    name: "Jonathan Weigend",
                    email: "jonathan.weigend@weigend.de"
                }];

// customer-list.html 
<tr ng-repeat="customer in customers">
                <td>{{ customer.name }}</td>
                <td>{{ customer.email }}</td>


The complete HTML Code for the customer-list view looks like this:

// customer-list.html
<div class="col-xs-12">
    <h1>Customers</h1>
    <table class="table table-striped table-condensed">
        <thead>
            <tr>
                <th style="min-width: 80px;">First name</th>
                <th style="min-width: 80px;">Last name</th>
                <th style="width:20px;">&nbsp;</th>
                <th style="width:20px;">&nbsp;</th>
            </tr>
        </thead>
        <tbody>
            <tr ng-repeat="customer in customers">
                <td>{{ customer.name }}</td>
                <td>{{ customer.email }}</td>
                <td><a class="btn btn-small btn-primary">edit</a></td>
                <td><a class="btn btn-small btn-danger">delete</a></td>
            </tr>
        </tbody>
    </table>
    <a class="btn btn-default">create new customer</a>
</div>

Now we are ready to connect the UI with our REST service. 

First create a customer-svc factory. 

$ yo angular:factory customer-svc

We have to change the generated customer-svc.js file to return a angular resource object to access our webservice. 

// customer-svc.js
angular.module('customerServices', ['ngResource'])
        .factory('customerSvc', ['$resource',
            function ($resource) {
                return $resource(
                        'http://localhost:8080/customerserver/webresources/com.javaone.customerserver.customer/:customerId:id',
                        {},
                        {}
                        });
            }]);

We also have to add the new module „customerServices“ to our list of module dependencies of our application. Otherwise the module will not be accessible in our controller.

// app.js
angular
  .module('demoClientApp', [
    'ngAnimate',
    'ngCookies',
    'ngResource',
    'ngRoute',
    'ngSanitize',
    'ngTouch', 
    'customerServices'
  ])

We change now the customer list controller to use the Customer Service instead of an hard coded array.

// customer-list.js

$scope.customers = customerSvc.query();


Step 3: Putting HTML5 Client direct into the WAR Webapplication Archive

To include the HTML Project in our JEE Webapp we use the Maven WAR Plugin:

// pom.xml
<plugin>
     <groupId>org.apache.maven.plugins</groupId>
     <artifactId>maven-war-plugin</artifactId>
     <version>2.3</version>
     <configuration>
         <failOnMissingWebXml>false</failOnMissingWebXml>
         <webResources>
            <resource>
                <!-- this is relative to the pom.xml directory -->
                <directory>../CustomerClient/app</directory>
            </resource>
         </webResources>
     </configuration>
</plugin>

After new Maven build and deployment the page shows up when you start the Glassfish

Customer List View
To create the Update/Create dialog nothing special is needed. If you have this running it should be easy to extend the example as you wish.

Customer Create View
Customer Update View

Johannes Weigend





Aug 27, 2014

Integrating Grunt into an existing Maven build

We love automating things. Maven has been our build tool for quite some time now and it has served us very, very well. Ever-repeating tasks like setting up a new project from scratch, building it, testing it, running it, integrating it into our existing CI, etc. are a chore. Maven helps us focusing on the actually interesting bits while removing the necessity to manually deal with the mundane stuff.

But especially when developing web applications, not everything is shiny and golden with Maven. Back in the good ol’ days, web resources were simple files that lived somewhere in your src/main/webapp directory. Well, those days have long but passed: JS and CSS files are (finally) treated as “proper” sources that need some care during the project’s build phase, too:
  • Linting  JS code with tools like e.g. JSHint
  • Possibly generating JS code from another language, e.g. TypeScript
  • Testing JS code
  • Aggregating JS modules into a single, browser-consumable package
  • Pre-processing LESS or SASS into actual CSS
  • Auto-generating CSS sprites
  • Automatically adding the proper vendor prefixes to the CSS Output
  • Minifying / compressing JS and CSS
  • Optimizing Images
  • ...
Maven was not built with these kinds of tasks in mind and – naturally – often falls short of our expectations. There are some helpful Maven plugins, of course, but most of them sicken from the same basic problems:
  • being outdated or not maintained at all
  • possibly unstable
  • too opinionated / not opinionated enough
  • complicated configuration
  • long execution time
The web ecosystem around Node.js has spawned lots of tools that deal with all kinds of tasks. Grunt (and hundreds of plugins available via npm) is one of the most prominent ones. And since it runs on Node.js – which is known for its quick start-up and execution time – in many cases the performance is superior to the “classic” Maven plugins that use Rhino, custom Ant tasks or other means.

Integrating Grunt into Maven

In order to run Grunt tasks, you’d normally need Node.js and npm. Even though I strongly recommend setting those tools up on your dev machine, you don’t strictly have to: The great frontend-maven-plugin by Eirik Sletteberg allows us to integrate Grunt tasks into our Maven build even without a local installation of Node.js and npm. It also tries to download the correct Node.js binary for the current system - enabling the Maven build to run on your CI as well (as long as that machine has access to the internet).

In this example, we basically want to do two main things:
  • Handle Scripts
    • Running JSHint against the entire JS code
    • Minifying everything into a single file
  • Handle Styles
    • Generating the CSS from LESS
    • Running autoprefixer on the result to achieve dynamic cross-browser functionality
    • Minifying the result into a single file
Our goal is to generate the final JS / CSS files from the sources and then copy those file to the appropriate folder inside Maven’s target folder. This way, Maven will eventually pick up those files and put them into the final artifact (WAR, JAR, etc.).

First, you’ll need a package.json right next to your pom.xml. This file tells npm what packages you need for the build. Especially the devDependencies are the interesting part. Aside from the basics, we also require several grunt-contrib-* or grunt-* plugins. Those plugins can be loaded by Grunt at build time to do specific things (e.g. the LESS compiler).

Now, let’s configure the aforementioned maven plugin. Add it to the list of plugins in your pom.xml’s <build> section. That adds two additional executions to your lifecycle:
  1. The plugin will attempt to figure out the current environment and download the Node.js binary along with npm (if they haven’t already been downloaded before, of course). This happens in Maven’s initialize phase.
  2. In the generate-resources phase, the “default” Grunt task is run.
This basically completes the Maven part. Now we only need to set up the Grunt tasks.

Setting up Grunt Tasks

Add a file called gruntfile.js next to your pom.xml and package.json. This file basically serves three purposes: loading all required plugins, building the configuration object and defining Tasks.

Most of the time, plugins will simply be loaded via grunt.loadNpmTasks which uses the devDependencies in the package.json file. You can utilize the full Node.js module loading mechanism here as well because the gruntfile is just a Node.js module itself.

Every plugin expects to find its configuration under a meaningful key in the Grunt configuration object (e.g. grunt-contrib-jshint will look for the key “jshint”). You will need to look up the respective documentation for each plugin you want to use. Keep in mind that most plugins can (or must) have multiple named “tracks” in their configuration. We will use this later to run only a specific part of the grunt-contrib-copy’s plugin configuration – one for scripts and one for styles.

Finally, defining tasks is simply just giving a meaningful name to a sequence of tasks. You could call each task individually from the console, of course:

grunt jshint # run all tracks of the grunt-contrib-jshint plugin
grunt uglify # same for grunt-contrib-uglify
grunt copy:scripts # copy the minified JS file to the target Folder


But it’s obviously easier to simply define a “scripts” task that does all that for us. We recommend to have one top-level task for each pipeline – in our example that’s “scripts” and “styles” – as well as one “build” task that runs every of those top-level pipeline. Finally, the “default” task should basically just call the “build” task followed by a possible “clean” task that removes intermediate artifacts – in our case the dist Folder.

Here is one example of a gruntfile that’s doing all we need. As you can see, the sheer file size and complexity quickly gets out of hand. By utilizing the power of the Node.js module system and the API which is available to us, we can move each plugin’s configuration into its own file and clean up some repetitive clutter as well. This is a slightly refactored version of the same gruntfile. For example, the JSHint task’s configuration file would look like this. You don’t need to re-invent the wheel, of course: there are helper plugins and tutorials for effectively modularizing and speeding up Grunt tasks, too.

Jul 28, 2014

Emergent Design with Java Annotation Processors


The emergent design process in agile software development can be supported and accelerated using lightweight code generators. The wide spread use of annotations in all major Java APIs, standards and frameworks makes the Java Annotation Processing API a prime candidate to implement such generators in no time. This article will show the emergent design process from code refactoring to a working source code generator for a JPA based DAO implementation.

Background

When using agile software development methods the overall functionality of a system is implemented feature by feature spanning all software layers. The individual software components are being developed test-driven in small increments and undergo continuous refactoring. The internal micro-architecture of the system emerges during this TDD cycle; after a few iterations the commonalities of the overall OO design become apparent. This is when lightweight generators should be introduced to aid in the development of future features and to bring order and symmetry to the emerging design.


Most major Java APIs bring extensive annotation support. These annotations provide the required meta-data that can be used by custom Java annotation processors at compile time to dynamically generate further source code artifacts. By combining the capabilities of annotation processors with standard template engines such as FreeMarker or Velocity simple yet powerful source code generators can be implemented quickly and effortless. 


For example, when using the Java Persistence API you have to annotate POJOs with JPA annotations to define the details of the object-relational mapping.  All these annotations provide enough meta-data to be able to generate a suitable data access object for a JPA entity with CRUD functionality.




Refactoring and Abstraction

harvesting ideomatic patterns

Templating

using FreeMarker / Velocity

Code Generator

java annotation processors

Build Tool Integration





<plugin>  
   <groupId>org.apache.maven.plugins</groupId>  
   <artifactId>maven-compiler-plugin</artifactId>  
   <configuration>  
      <verbose>true</verbose>  
      <encoding>${project.build.sourceEncoding}</encoding>  
      <generatedSourcesDirectory>${project.build.directory}/generated-sources/annotations</generatedSourcesDirectory>  
      <annotationProcessors>  
         <annotationProcessor>de.qaware.blog.DaoSupportAnnotationProcessor</annotationProcessor>              
         <annotationProcessor>de.qaware.blog.PersistenceXMLAnnotationProcessor</annotationProcessor>  
      </annotationProcessors>  
      <compilerArguments>  
         <Ahibernate.dialect>${hibernate.dialect}</Ahibernate.dialect>  
         <Ahibernate.show_sql>${hibernate.show_sql}</Ahibernate.show_sql>  
         <Ahibernate.format_sql>${hibernate.format_sql}</Ahibernate.format_sql>  
         <Ahibernate.hbm2ddl.auto>${hibernate.hbm2ddl.auto}</Ahibernate.hbm2ddl.auto>  
      </compilerArguments>  
   </configuration>  
</plugin>  

maven compiler plugin

Links