Sep 20, 2017

How to develop Eclipse SmartHome Bindings with IntelliJ and Docker

Developing the Eclipse SmartHome (ESH) bindings with another IDE than Eclipse is difficult because the standard way of developing the bindings requires the tools provided by the Eclipse IDE. One can develop the ESH bindings with IntelliJ, however a crucial question remains open: how to test the developed bindings without having the Eclipse tools at hand?
Using IntelliJ to develop the ESH bindings is different to using Eclipse for two main reasons:
  • First: Eclipse uses a “manifest first” approach while the Java standard approach is “POM first” with a handwritten or generated manifest. In our example we have used a handwritten manifest.
  • Second: To debug and test in Eclipse, it is enough start the full environment using the target platform provided by ESH. This starts a JVM with equinox as a local process. To debug and test in IntelliJ, one should copy the binding jar built by MAVEN into a prepared Docker container and then start it.

Minimal Eclipse SmartHome for testing bindings in IntelliJ

Our approach is based on the packaging example of ESH in combination with Eclipse Concierge as OSGi Container.
A full MAVEN build with mvn clean package creates a new zip file within the /target directory. This zip file contains a full OSGi container, the Eclipse SmartHome basics, the Yahoo Weather Binding and an addons directory. The OSGi container scans the addons directory for bundles and deploys them automatically. This zip file can be used to create a Docker container.
In order to stop the container gracefully a small change in the start script is needed. A description of it can be found here: qaware/smarthome-packaging-sample.
We have updated and added some dependencies to the original repository. One of the updated dependencies is the Jetty webserver, which also includes its client in the latest version of the 9.3.x branch.

The Docker Container

All the following examples and commands are based on the qaware/smarthome-packaging-sample repository.
The Dockerfile creates an image which is runnable out of the box. It contains a full Eclipse SmartHome, which runs on an Alpine Linux with the current OpenJDK 8 Runtime Environment. The image has a final size of less than 140 MB.

FROM openjdk:8-jre-alpine
 
ENV DEBUG_PORT=5005 HTTPS_PORT=8443 HTTP_PORT=8080
 
WORKDIR /opt/esh/runtime/concierge
ADD /target/smarthome-packaging-sample-*.zip /tmp/
RUN apk add --update unzip \
 && mkdir -p /opt/esh/conf/services \
 && unzip /tmp/smarthome-packaging-sample-*.zip -d /opt/esh \
 && chmod +x /opt/esh/runtime/concierge/start.sh \
 && rm -f /tmp/smarthome-packaging-sample-*.zip \
 && apk del wget \
 && rm -rf /var/cache/apk/*
EXPOSE 5005 8080
ENTRYPOINT exec /opt/esh/runtime/concierge/start.sh

You can find the Dockerfile within the forked repository.
To build and tag the container as eclipse/smarthome:latest, you need to run:
./mvnw clean package && docker build . -t eclipse/smarthome:latest
Now it is possible to start the container:

docker run -dit \
  -p 127.0.0.1:8080:8080 \
  eclipse/smarthome:latest

You can subsequently open the PaperUI and the System Console (username & password: admin).
The container can also be started in debug mode, which allows connecting an external debugger to the container. To start the container in debug mode use this command:

docker run -dit \
  -p 127.0.0.1:8080:8080 \
  -p 127.0.0.1:5005:5005 \
  -e "REMOTE_DEBUG=true" \
  eclipse/smarthome:latest

You can now connect the container with a Java debugger (like IntelliJ) using the address 127.0.0.1:5005.

Run and Debug an Eclipse SmartHome Binding within the Docker container

This part builds upon our previous blog post about the usage of Docker and IntelliJ: How to use Docker within IntelliJ
You can use the previously created Docker image for the development of bindings. To do that you need another small Dockerfile which copies the binding into /opt/esh/addons/ of the container:

FROM eclipse/smarthome
# replaces org.eclipse.smarthome.binding.hue with your binding id
ADD target/org.eclipse.smarthome.binding.hue-*.jar /opt/esh/addons

You can place this Dockerfile into the main directory of the bundle, next to the MAVEN pom.xml, and subsequently use it for a new Docker deployment in IntelliJ as described in How to use Docker within IntelliJ. In IntelliJ you can directly start the Docker deployment with an attached debugger. Alternatively, you can create a volume mount to /opt/esh/addons and, after the successful completion of the build, copy the binding jar into the mount.

Recommended Volume Mount

We recommend to mount a volume into the Docker container to store the information about the bindings and the other configurations of the ESH: /opt/esh/userdata

Sep 18, 2017

Software Circus

Intro

We just came home from two days of talks, workshops and lots of fun during this year's Software Circus. "Cloudbusting" was the theme this year and it took place in the lovely city of Amsterdam. The organization team did a great job finding a unique venue: A festival location including a Big Top circus tent, a rusty hangar and a large outdoor area is not quite the setting you expect from a software conference!

Not your standard conference at all, the Circus provided some relaxed, fun and joyful atmosphere for learning new stuff, meeting new people and chatting to old friends of the cloud native community. The conference was embedded into a futuristic story arch that was moved forward by several great performances of actors, singers and dancers in between talks and sessions. Loud music, great food and Dutch beer rounded out the experience.

Maybe - just maybe, but don't tell our boss - especially the first day was a bit heavy on the show and too light on the content side. We did, however, get to talk tech, as there were several tracks throughout the day. If anything, we would wish for more talks and hands-on sessions during the next year's event!

The second day was reserved for workshops and some deep-dive sessions. Heavy rain and wet feet couldn't stop us from being there, not like many of the other attendants.

The following sections cover the most interesting topics and talks that we experienced this year.


Machine Learning/AI

One big topic at the Software Circus 2017 was Artificial Intelligence (AI), especially Machine Learning (ML).

In a practical part Google gave an intro into Tensorflow. It’s an Open Source library for AI and ML, that is developed by Google’s Brain Team. It performs operations on multidimensional arrays, so called tensors. As Google uses it for it’s search ranking, Tensorflow is worth a closer look.

In a theoretical talk Thiago de Faria spoke about ML, AI and DevOps. He introduced the history of AI and ML which goes back to the late 50’s and 60’s, where the first algorithms occurred. In the 90’s support vector machines mark another big step until 1997 IBM’s Deep Blue beat the World Champion at chess. Nowadays IBM’s Watson is one of the most famous AI / ML projects.

As Thiago is an ML practitioner he pointed out some very important questions concerning DevOps in AI and ML systems which still remain unanswered. A normal program is tested automatically in the context of Continuous Integration (CI). But how can you apply CI to AI and ML systems? Can you create automated tests for such a system? As even the smallest change in an AI has unpredictable effects and might break completely disjoint features, this is a very important point. Furthermore, a normal program is debuggable. You can set breakpoints and follow the program execution. But how can you debug AI and ML systems as there exists no traditional program flow? He hopes that those questions might be answered as AI and ML become more and more explainable.

At last he expressed his concerns and fears regarding ML. On the one hand existing biases might propagate into a system’s learned behavior and influence its decisions, on the other hand people might tend to delegate decisions to algorithms as they are too afraid to decide for themselves. Only time will reveal if his concerns were unfounded.

Software Architecture 

With many buzzwords flying around, it is sometimes forgotten that certain topics never loose their relevance. Independent technology consultant Simon Brown delivered an inspiring (re-) imagination of the modern Software Architect. He debunks the notion that a capable architect is only doing the up-front specification work (the seagull approach), and is an avid proponent of a hands-on approach to software architecture. An architect needs to have people skills as much as technological expertise and is an essential building block for well-performing dev teams.

Simon also advocates the use of modelling tools to support development. This does not mean UML necessarily, he introduced his own creation as an alternative: The C4 model for software architecture is a lightweight alternative to get started with a better architectural documentation.

DevOps 

Improving the software delivery process is still a key concern today and it was an important topic at the Software Circus as well. Often summarized under the 'DevOps' term, speakers and workshop organizers examined the matter from different perspectives, sharing success stories and cautionary tales. Maarten Dirkse of dutch online bookstore bol.com gave a workshop on doing continuous delivery - including automating canary deployments - using Gitlab-CI and Spinnaker.

DevOps is not about a particular technology, it is about culture and collaboration. This is the primary takeaway from Kris Buytaerts emotional talk in which he explained how "Docker Is Killing DevOps Efforts". He uses the Anti-Pattern of the "Enterprise Container", which contain an entire application-stack from Message-Queue to Database. By that he shows that simply adopting a particular technology does not solve the delivery problems and reminds everyone of the core values of the DevOps movement: culture, automation, measurement and sharing (CAMS).

DevOps principals are not only important in application development, as "DataOps" practitioner Thiago de Faria explained. With increasing importance, machine learning and AI projects need to think about their own delivery pipelines to tackle lock-in, onboarding and delivery problems.


Dealing with Legacy Applications 

Even if this year's Software Circus was using the theme "Cloudbusting", there was some good news for those of us dealing with monolithic legacy applications: David Pilato from Elastic Search demonstrated how legacy applications can easily adopt the Elastic stack with incredibly small effort (see github.com/dadoonet/legacy-search). Twelve brave attendants watched this morning's first demo on adopting Elastic Search, all of us defying rain, wind, cold and noise from trains and ice carving...

The new 6.0.0-beta version contains some really cool features: the now build-in client for Elastic's API can be integrated in your JAVA applications quite easily, comes with convenient query builders, and grants out-of-the-box access to the following features:

  • Bulk processing for high performance index operations
  • Custom analyzers for easy index token definitions
  • Easy data aggregations for your application specific needs
  • Fuzziness factor for typo tolerance
  • Easy Kibana integration

Good to know that Elastic Search cannot be used for time series only. Can't wait to try it out in our applications!

Conclusion

The Software Circus is worth a visit, especially if you tire a bit of the conventional conference setting. It is a community event in the truest sense, bringing the people together and creating a comfortable backdrop for talks and sessions. If you come for the conversations and the spirit, you will be delighted. If you are only in for the content, then you may leave longing for some more - even though the Circus had lots to offer in that regards as well!