Amdatu Bootstrap 1.0 release

It's finally here, the first public release of Amdatu Bootstrap! Amdatu Bootstrap is the easiest way to get started with OSGi development. It is a tool that helps you to setup Bndtools projects. Instead of Googling and copy-pasting your way to a new project, this tool sets up projects interactively.

Some of the things Amdatu Bootstrap can do:

• Add dependencies for often used libraries
• Setup run configuration for often used libraries
• Create template files to supercharge development
• Manage versions in your workspace
• Baseline bundle and package versions
• Create your own plugins

Amdatu Bootstrap is based on bnd and should be used together with Bndtools. Below are some videos that show what Bootstrap can do. If you just want to get started, download the release now.

Amdatu Bootstrap from Luminis Technologies on Vimeo.


Creating Amdatu Bootstrap Plugins from Luminis Technologies on Vimeo.


Sharing Amdatu Bootstrap Plugins from Luminis Technologies on Vimeo.

A bit of history

The birth of Amdatu Bootstrap was during my Christmas holidays in 2013. In the past I had worked on JBoss Forge, which is an interactive shell to setup Java EE projects. Because all my work had switched to OSGi I couldn't really use Forge anymore (Forge is Maven based). The great thing in Forge was that it was very easy to build plugins, which allowed me to create a plugin for anything that bothered me during development of projects. I missed this concept, because it makes just as much sense in an bnd/Bndtools world. Although Bndtools is great, it's based on Eclipse RCP (it's an Eclipse plugin), which makes it quite difficult to extend. 

During the holidays I felt like experimenting a bit, and decided to try to create a command line tool that was extremely pluggable, based on OSGi. New commands should be added by just adding new OSGi services. Based on the simple whiteboard pattern I had the plugin system itself working in no time. The Gogo shell already made it easy to add console commands, and I added some APIs to interact with the user using a prompt interface. 

To actually make some useful plugins, I created some more APIs to easily work with bnd workspaces; add dependencies, create run configurations etc. Plugins would just have to use those simple APIs, instead of the super powerful, but not so easy to understand bnd API.

At the end of the holiday, I had a proof of concept working. I started showing the project (which was really just an experiment so far) to colleagues at Luminis Technologies, and we decided it might have potential to actually become a useful tool, and made it an Amdatu project. In the next few months I added more APIs, more plugins and it soon started to become useful for doing real work on real projects. Luminis Technologies colleague Jago de Vreede implemented some very useful plugins, including the baselining plugin which is still one of the most used features. 

Out of nowhere (the project was never announced anywhere yet) we received a bunch of pull requests from Bram Pouwelse, which is now one of the most awesome Amdatu contributors, who has had an important role in all the Bootstrap development from that point on. With a small group of contributors we managed to really create something useful in the months that followed.

API design - taking the bitter pill

While more plugins were added, our ideas of what Bootstrap should do became more demanding. One of the important issues was that it should be easier to use Bootstrap from the shell, by adding things like command and argument completion. This was problematic with the API at the time, because all interaction with the user was done on-demand by invoking the Prompt API. This means we didn't have any meta model that described a plugin's arguments. Marcel Offermans and me started experimenting with a meta model that described plugins, which was based on Apache Commons CLI. This was a huge step forward, but was still mostly based on Strings. After showing the ideas to Peter Kriens (creator of bnd), he shared some great ideas about using interfaces to describe command arguments. After many iterations of fine tuning the API, we finally got to a point where it worked really easy for plugin developers, while containing all static metadata about plugins.

This new API did require refactoring all existing plugins, and rewriting a large part of the core of Bootstrap. This was a bitter pill to swallow, but looking back, I'm happy we took a few more months to do this work instead of rushing a release. New idea need lots of experimentation, and failing many times is part of that. Accepting this will make the end result much better however.

From command line to web based

Amdatu Bootstrap was designed to be used from the command line from the beginning. The main reason for this is that it is simple, and we didn't want plugin developers to care about UI. Although our API became better over time, it proofed to be really difficult to create a powerful console that worked well on all platforms. There are several libraries available to help with this, but it would still require a significant amount of work to really get this to work well. With our new API we had a much better meta model describing plugins, and I started wondering if it would be feasible to create a dynamic user interface in the browser. After lots of experimenting during traveling to conferences in the September-Oktober timeframe I created a web UI based on TypeScript, AngularJS, WebSockets based on Atmosphere and Rx. This worked so well that we decided to drop support for the console entirely and focus on the web UI. This allowed us to do things that were not possible in the console, making a much nicer user experience. The great thing about this was that we didn't have to change the plugin API at all, which also proofed that we finally got the API right.

The reason to create a Web UI instead of UI technology X (e.g. JavaFX) is simply experience. Most Bootstrap committers work with web technology on a daily base, while JavaFX would be something entirely unknown.

Architecture

OSGi makes building a tool like Bootstrap easy. Plugins are simply registered as OSGi services and this allows the core to know about the plugins. Plugins can not only provide end-user commands, but also provide services to other plugins. Again, by just registering more services in the Service Registry. We didn't have to create anything to make this plugin mechanism work. New plugins can also be added by just adding a new JAR file. No magic classpath scanning, no custom classloaders and never a restart during development. Not that this is any news, but OSGi really pays of here. On a high level, Bootstrap's architecture can be described as the diagram below.

Bootstrap Core provides APIs to access plugins, and APIs for plugins to make common tasks easy in code, such as adding dependencies, working with templates and navigating workspaces and projects.

From a technology perspective the architecture could be described in layers.

Of course you can also just look at the code yourself or learn more about creating plugins.

What's next for Bootstrap?

Now that the first version is released, the core API is stable and works well. This is why we took our time to get to a release. We can now focus on adding many more plugins, further enhancing the user experience, and offer even more powerful APIs to plugin developers. If you have ideas for plugins, we would be happy to hear; create issues in Jira, or join the mailing list. And even better if you decide to start hacking yourself, Bootstrap is designed for plugin developers after all! Don't hesitate to ask for directions if anything is unclear.

OSGi doesn't suck - You're just using it wrong

James Ward published a nice post "Java Doesn't Suck - You're Just Using It Wrong" recently. If you didn't read it yet, please do so first! I completely agree with James and thought it would be fun to see how this applies to the Amdatu stack, and how we used this on different projects in the past few years.

10 Page Wikis to Setup Dev Environments Suck

Even on our largest projects the dev environment setup is just a git clone away. All our projects are based on Bndtools, and a cloned workspace just needs to be imported in Eclipse. Headless builds based on Gradle are supported out of the box for each Bndtools workspace without any additional setup. A freshly cloned workspace contains all the configuration to run on the developer's machine.

Most developers install Mongo manually on their machines, but there's a hosted Mongo cluster for development as well.

For UI development we use tools from the JavaScript ecosystem such as Grunt, and those are installed automatically from the Gradle build.

Incongruent Deployment Environments Suck

Our build server does a fully automated deployment at each merge to master in git. In general master is pretty stable, because all work is done and reviewed on feature branches. Promoting a build to the more stable test server is done by starting a build on Bamboo. This will create a tag in git so that the release is reproducible and does the automated install on the test cluster. The same process applies to promote a build from test to production, it's just a single click on the build server.

When using OSGi you don't actually need an application server. The OSGi framework is part of the application, and you can start the application as an executable jar file. Things like a web server is just another bundle in the application (we use Jetty). On the cluster nodes (in any environment) we don't deploy the application directly either. A clean node starts a very bare bones OSGi application containing a Management Agent. The agent connects to a provisioning server, Apache Ace. The build server deploys bundles to the provisioning server, and when a new cluster node connects, it will receive the latest version of the application's bundles. This makes installations light weight, completely automated and reproducible. The important thing is, everything is completely automated, and there are no servers to maintain. When cluster nodes become unhealthy, a new one will be started automatically (by AWS AutoScaling) and will connect to the provisioning server to install the software. The videos below show more about deployments and Apache ACE.

Deploying with Apache ACE from Luminis Technologies on Vimeo.

Creating runnable JARs from Luminis Technologies on Vimeo.

Servers That Take More Than 30 Seconds to Start Suck

We don't even HAVE an application server to startup, so there's no waiting either... Starting an OSGi framework is extremely light weight (remember it was once designed for embedded environments). The only startup there is, is the actual application code. Even for very large applications (more than 500 bundles) this is a few seconds at max, while most applications start pretty much instantly. 

Besides that, you don't actually restart your application often during development. Using Bndtools you get hot code deployment; each time you compile code, the bundle containing that code is rebuilt and updated in the running framework. This process is so fast that you won't even notice it, coding feels like working in a dynamic language like Groovy or Ruby. 

Manually Managed Dependencies Suck

The very best way to manage dependencies is using bnd (which Bndtools is built on top of). Just like in Maven you declare dependencies by name and version. Because bnd is built for OSGi it understands things like semantic version ranges. You don't need POM files; OSGi bundles already contain all the metadata a POM file normally contains. This means that conceptually dependency management in bnd are not that different, but bnd is a lot easier because it's closer to OSGi. Similar to Maven there are online repositories, and you can host your own repositories.

There is one difference, bnd doesn't do transitive dependencies. Transitive dependencies are dangerous and can cause a lot of trouble. Also, the resolver in the OSGi framework will help you to make sure that all dependencies your application need are installed.

Bnd does integrate very well with Gradle, which we use for headless builds. But again, dependencies are managed by bnd, not by Gradle.

For UI development we often use Bower to manage dependencies. 

Unversioned & Unpublished Libraries Suck

Versioning is an important topic in OSGi. All the tools discussed understand semantic versioning, which helps a lot when checking for compatibility with newer versions of (external) bundles. API's can be baselined automatically, which will force correct semantic versioning of your packages.

baselining from Luminis Technologies on Vimeo.

Long Development / Validation Cycles Really Suck

Of all the things in OSGi, this is probably what I love most; an instant feedback cycle during development. Just check out this video.

Bndtools code reload from Luminis Technologies on Vimeo.

Monolithic Releases Suck

Completely automated deployments are required when releasing often, but as discussed above, all the tools are in place to do this. We deploy multiple times per day to development and often also to test, and multiple times per week to production. This definitely requires a different mind set in the team (and not just the development team), but once everyone got used to it, it's perfect. Fast feedback for the win!

James makes another excellent point about the need for monitoring. When deploying new code all the time, things can always go wrong once in a while. Make sure to monitor for this! We implemented health checks on all our (OSGi) services. The Load Balancer will check these health checks every few minutes. When there's something wrong, we receive notifications immediately. 

Another really useful tool that we use is New Relic. Specially when performance issues occur it gives a lot of helpful information.

Sticky Sessions and Server State Suck

All web user interfaces are based on AngularJS. Pretty much all session state is on the client side and this greatly reduces the need for server side sessions. The only state the server has is related to authentication on the RESTful resources, which is stored in Mongo. This gives horizontal scalability, and makes topics like failover and auto scaling a lot less complex.

Useless Blocking Sucks

Clients use a mix of RESTful web services and Web Sockets, where Web Sockets are used for asynchronous responses from the server. This makes asynchronous communication easy for wherever it applies. Also we use RabbitMQ and RX at several places. This all doesn't have much to do with OSGi, but it works perfectly together. 

The Java Language Kinda Sucks

I did agree with this for a long time. We used a mix of Groovy and Java for a while, but this turns out to introduce quite a few problems. The Groovy tooling in Eclipse is horrible, which is the main reason I'm not so eager to use it in new projects any more. Java got a LOT better with Java 8 as well. Once you go stream() you never go back (or something like that ;-) ). I'm really not sure anymore if polyglot really makes that much sense, even when I do enjoy playing with alternative languages. Groovy and Scala do work without any problem in OSGi, it's just another bundle... 

Making JavaFX better with OSGi

Why you should run JavaFX on OSGi

OSGi makes JavaFX better for three reasons:

1. Hot code reload during development
2. A services based architecture keeps UI features nicely isolated
3. Provisioning to devices

Let's take a look at all of them in detail before we see how to make JavaFX run on OSGi.

Hot code reload

Waiting for a build and restarting your application to see the effect of code changes is annoying. Even more so with UI related work, because it's less easy to work test-driven and you often need a bit of experimentation to get the UI to look the way you intended. 

OSGi is designed to be dynamic; in a running framework bundles can be added, removed and updated. Combined with an IDE that understands this, we can make updates to a running application without ever waiting for builds or restarts. See the following video for an example.

Services based architecture

In OSGi it's trivial to create plugin systems based on the so called "whiteboard pattern". Each part of the UI can be provided by a different bundle, which makes the code loosely coupled. This makes it easier to add features or change existing features without impacting the rest of the code. The same mechanism can be implemented using other Dependency Injection frameworks, but it comes most natural in OSGi. 

Provisioning

JavaFX is gaining popularity in the IoT space; it's very well suited to run on all kind of devices. If we have a lot of devices, it becomes a question how we install and update our application to those devices. Manually copying files to a devices is ok if we have one or two of them, but what if you roll out your software to many devices? Also, in the IoT space there might be limited bandwidth, so rolling out updates should preferably be efficient in this aspect as well. Because an OSGi framework can be updated, it's possible to create provisioning systems; a system that takes care of installing and updating the software running on a target (such as a device). Apache ACE is a great example of this. At JFokus, Sander Mak and me will present about exactly this topic in a lot more detail.

Running JavaFX on OSGi

Out of the box it seems a bit problematic to run JavaFX on OSGi. The reason for this is that starting a JavaFX application requires the use of a launcher, which wasn't designed to be used in a modular or dynamic environment. 

Typically the launch method is used to start a JavaFX application. Running this method from an OSGi bundle (e.g. from an Activator) produces two issues:

1. The method fails with a ClassNotFoundException
2.  The method may only be called once (this is explicitly checked), so updating/restarting the bundle that invokes the method is not possible.

Importing JavaFX packages

Besides these two problems there is something else to straighten out. When using the javafx packages, most OSGi frameworks will give resolver issues such as: "Unable to resolve 11.0: missing requirement [11.0] osgi.wiring.package; (osgi.wiring.package=javafx.application)".

In OSGi the packages made available from the JRE are explicitly exported by the "system bundle". The list of packages needs to be configured however, and most launchers available today don't export the javafx packages yet. This is no problem at all, we can do this manually with just a bit of configuration. When using Bndtools you can add the following to a .bndrun file: "-runsystempackages: javafx.application,javafx.scene". Just add other packages when you need them.

Fixing the ClassNotFoundException

Looking at the source code of the launch method, it uses the ContextClassLoader of the current thread to load the class that you are trying to launch. A common problem with an easy fix: Setting the ConextClassLoader to the bundle's classloader before invoking the method. 

Thread.currentThread().setContextClassLoader(
this.getClass().getClassLoader());
launch();

Working around the single invoke launch limitation

When we setup the UI in the same bundle as where the launch method is invoked, we run into a practical problem. Each time we edit our code, the bundle is updated in the running app (like we want it to). This fails because the launch will throw an exception. Not good. We can easily work around this problem by moving the code that actually sets up the UI to a separate bundle (or bundles). 

In the following example I register the Stage created by the launcher as a service. Another bundle can depend on this service to get the Stage and add UI elements to it. This way the launching code and the actual UI are separated, and we can happily restart and update the UI bundle without problem. 

Code in the launcher bundle.

import java.util.concurrent.Executors;

import javafx.application.Application;
import javafx.application.Platform;
import javafx.stage.Stage;

import org.apache.felix.dm.DependencyManager;
import org.apache.felix.dm.annotation.api.Component;
import org.apache.felix.dm.annotation.api.Start;
import org.apache.felix.dm.annotation.api.Stop;
import org.osgi.framework.BundleContext;
import org.osgi.framework.FrameworkUtil;

import example.javafx.launcher.StageService;

@Component
public class App extends Application {

 @Start
 public void startBundle() {

  Executors.defaultThreadFactory().newThread(() -> {
    Thread.currentThread().setContextClassLoader(
      this.getClass().getClassLoader());
    launch();
   }).start();
 }

 @Override
 public void start(Stage primaryStage) throws Exception {
  
  BundleContext bc = FrameworkUtil.getBundle(this.getClass()).getBundleContext();
  DependencyManager dm = new DependencyManager(bc);
  
  dm.add(dm.createComponent()
    .setInterface(StageService.class.getName(), null)
    .setImplementation(new StageServiceImpl(primaryStage)));
 }

 @Stop
 public void stopBundle() {
  Platform.exit();
 }
}

Code in the UI bundle.

@Component
public class UI {

 @ServiceDependency
 private volatile StageService m_stageService;
 
 @Start
 public void start() {
  Platform.runLater(() -> {
  
  Stage primaryStage = m_stageService.getStage();
   primaryStage.setTitle("Hello World!");
         Button btn = new Button();
         btn.setText("Say 'Hello'");
         btn.setOnAction(new EventHandler() {
  
             @Override
             public void handle(ActionEvent event) {
                 System.out.println("Hello World!");
             }
         });
         
         StackPane root = new StackPane();
         root.getChildren().add(btn);
         primaryStage.setScene(new Scene(root, 300, 250));
         primaryStage.show();
  });
 }
}

That's it! JavaFX now runs successfully in an OSGi container.

A pluggable architecture

Now that we are running on OSGi, we can make use of services to make the whole architecture more modular. As an example we take a UI that has multiple screens, divided by tabs. Let's try to create a mechanism that let you provide new tabs from different bundles, so that new features can be added to the UI without even changing anything in the main UI code. 

The main UI bundle just sets up the TabPane. It will than listen for any AppScreen services that are registered (this is an example of the whiteboard pattern). Each AppScreen represents a tab, and has a title and a Node that represent that tab. When a new AppScreen is found, it is added to the TabPane, and when an AppScreen is removed it is removed from the TabPane. Now we can add new parts to the UI by just installing new bundles. 

import java.util.Map;
import java.util.Optional;
import java.util.concurrent.ConcurrentHashMap;

import javafx.application.Platform;
import javafx.scene.Scene;
import javafx.scene.control.Tab;
import javafx.scene.control.TabPane;
import javafx.stage.Stage;

import org.apache.felix.dm.annotation.api.Component;
import org.apache.felix.dm.annotation.api.ServiceDependency;
import org.apache.felix.dm.annotation.api.Start;
import org.osgi.framework.ServiceReference;

import example.javafx.launcher.StageService;
import example.javafx.ui.AppScreen;

@Component
public class UI {

 @ServiceDependency
 private volatile StageService m_stageService;
 private volatile TabPane tabPane;

 private final Map screens = new ConcurrentHashMap<>();

 @Start
 public void start() {
  Platform.runLater(() -> {

   Stage primaryStage = m_stageService.getStage();
   primaryStage.setTitle("Tabs example!");
   tabPane = new TabPane();

   screens.values().forEach(this::createTab);

   primaryStage.setScene(new Scene(tabPane, 300, 250));
   primaryStage.show();

  });
 }

 private void createTab(AppScreen s) {
  Tab tab = new Tab(s.getName());
  tab.setContent(s.getContent());
  tabPane.getTabs().add(s.getPosition(), tab);
  tabPane.getSelectionModel().select(tabPane.getTabs().size()-1);
 }

 @ServiceDependency(removed = "removeScreen")
 public void addScreen(ServiceReference sr, AppScreen screen) {
  if (tabPane != null) {
   Platform.runLater(() -> {
    createTab(screen);
   });
  }

  screens.put(sr, screen);

 }

 public void removeScreen(ServiceReference sr) {
  Platform.runLater(() -> {
   AppScreen remove = screens.remove(sr);
   Optional findAny = tabPane.getTabs().stream()
     .filter(t -> t.getText().equals(remove.getName()))
     .findAny();
   if (findAny.isPresent()) {
    tabPane.getTabs().remove(findAny.get());
   }
  });
 }
}

A bundle that adds a new tab could contain the following code:

import javafx.scene.Node;
import javafx.scene.control.Button;
import javafx.scene.layout.VBox;

import org.apache.felix.dm.annotation.api.Component;

import example.javafx.ui.AppScreen;

@Component
public class OtherScreen implements AppScreen{

 @Override
 public String getName() {
  return "Other screen";
 }

 @Override
 public Node getContent() {
  VBox vbox = new VBox();
  Button button = new Button("Other screen");
  vbox.getChildren().add(button);
  
  return vbox;
 }

 @Override
 public int getPosition() {
  return 1;
 }

}

Tooling

To get the most out of OSGi we need an IDE that "understands" updating bundles in a running framework. This makes build tools like Maven and Gradle a less than optimal choice, although they can definitely be used for building bundles. A much better choice is Bndtools; an Eclipse plugin that makes OSGi development easy. There is also an open issue for support in Intellij, please vote! For JavaFX support in Eclipse I used e(fx)clipse.

Deploying

Just like any OSGi application built with Bndtools we can export the application as an executable JAR, as also shown in this video. Simply click the export button in the .bndrun configuration screen, or run gradle export with the out of the box available Gradle build.

For a more advanced IoT setup we would use Apache ACE for provisioning. Come see us at JFokus for more about this :-)

JMaghreb slides and resources

The JMaghreb conference in Casablanca was great! Lots of interesting people to talk to and a lot of good content. I had three talks and many people have asked for slides, code and more resources which are listed in this post.

Modularity Patterns with OSGi

The source code can be found here.

Lessons learned from a large scale OSGi web app

Tutorial: Introduction to OSGi

In this tutorial I showed building a complete chat application backend using OSGi and Amdatu. The code was already available here. 

More resources:

• Amdatu.org for introductions and the Amdatu components
• My book: Modular cloud apps with OSGi

First beta of Amdatu Bootstrap

For the past few months we have been working on a development tool: Amdatu Bootstrap. Amdatu Bootstrap makes OSGi development faster and easier by providing an interactive tool to automate common tasks like configuring a build path or run configuration and it integrates with many libraries. Amdatu Bootstrap is built on top of Bnd and is typically used together with Bndtools.

The video below gives an impression how you can use Amdatu Bootstrap.

Amdatu Bootstrap comes with a web based UI and an OSGi based backend. The reason for working with web technology in the frontend is because this makes it easy to develop a user friendly application, and it gives the possibility to integrate in different IDEs.

So why not just extend Bndtools with the functionality of Amdatu Bootstrap? First of all, because Bndtools is based on Eclipse, it is not very easy to extend Bndtools. A lot of knowledge about Eclipse RCP is required, even for relatively simple tasks. Also, we want a tool with the potential to be used with other IDEs in the future. Amdatu Bootstrap is designed to be extensible. It is completely based on OSGi services, and adding a plugin is as easy as implementing an interface.

Amdatu Bootstrap went through several iterations of APIs and ideas, while being used by a diverse group of early users. We are now really happy with the API and the way the tool works, and are announcing the first beta release. Please provide feedback! You can send feedback on the mailinglist or create issues and feature requests on JIRA. If you want to help out even more you can take a look at the plugin development guide and work on some awesome new plugins.

Links:

• Amdatu Bootstrap getting started guide
• Plugin development guide
• Download
• JIRA
• Mailinglist

Introducing Amdatu JPA

JPA is a popular way of working with relation databases in Java. It was designed for use in Java EE however, and it was problematic to use in OSGi. Wtih Amdatu JPA we fixed that problem!

Amdatu JPA makes JPA usable in OSGi using either Hibernate, EclipseLink or OpenJPA. It takes care of data source registration, declarative transaction management and makes EntityManagers available as an OSGi service to integrate it tightly to the programming model we are familiar with in OGSi. Amdatu JPA was first released a few months ago, and after initial feedback and experience in some large projects it is now time to start using it!

The following video shows how to use Amdatu JPA. The full documentation can be found on the Amdatu website. 

Join me at JDD Krakow

October 13th and 14th I will be speaking at JDD in Krakow. I was speaking at JDD last year I'm very happy to be back! The conference is not too large which give you great opportunity to actually meet and talk to people. There are some excellent speakers on the schedule already, and I'm expecting many more. The call for papers is still open, so you can be one of them: http://14.jdd.org.pl/cfp/cfp/

Krakow is also a great place to be, and seems to be a hotspot for software engineering.

My talk is a two hour introduction to OSGi:

Modularity is becoming more relevant each day. It is the key to maintainable code and the ultimate agile tool. OSGi is the only mature modularity solution available today. In this talk you will see OSGi development in action.
OSGi has a name of being hard to use and complex. With today’s tools and frameworks this is far from true! In this presentation you will see an OSGi application being built from scratch and learn about package imports and exports, dynamic services, dependency injection and integration with JAX-RS and MongoDB. This talk is both for developers new to OSGi that want to learn the OSGi basics, and for developers with some OSGi experience looking to optimize their workflow.

If you have the possibility: make sure to be there!