Visualizing OSGi services with D3.js

Because I couldn't resist writing a bit of code during my vacation I started playing with D3.js, a data visualization library for JavaScript, and used it to visualize dependencies between OSGi services. If you are already familiar with designing systems based on OSGi services you might just want to take a look at the video directly. If you need a little more introduction; continue reading.



We are using OSGi services for pretty much  everything; all our code lives within services. Services are the primary tool when implementing a modular architecture. By just using services, your code isn't necessarily modular yet however. A lot of thought has to go into the design of service interfaces and dependencies between services.

In a services based architecture it's obviously a good thing to re-use existing services when implementing new functionality. At the same time it's important to be careful with creating too many outgoing dependencies. If everything depends on everything else, it becomes very difficult to make any changes to the code (also known as "not modular"...). When implementing a system you will start to notice that some services are re-used by many other services. Although not an official name at all, I often call them core services; a service that plays a central role in your system. These services must be kept stable because major changes to it's interface requires a lot of work. Outgoing dependencies from these services should also be used with care. To guarantee stability of the service, it's good to keep the number of dependencies low. This prevents a ripple effect of changes when touching something. In practice, a core service should only depend on other services which are very stable. Do not depend on services that are likely to change often.

Many other services might not be used by any other services at all however, for example a scheduled job or a RESTful webservice that only serves a specific part of a user interface. These services can easily be replaced or even be discarded when no longer needed. In an agile environment this happens all the time. For these services it's not really a problem to depend on other services, specially not the core services of the system.

If your architecture is sound, you probably have a very clear idea about which services are your core services. Still, it's useful to actually visualize this to identify any services that have more incoming dependencies than you expected, or at least see which other services have a dependency on a certain service. And that's exactly what I did for this experiment.

We use Apache Felix Dependency Manager to create services and dependencies between them. Because of this I used the Apache Felix Dependency Manager API to create the list of dependencies between services. Note that this will not show services that are not created by Dependency Manager. The visual graph itself is created by D3.js based on this example.

The code is available on BitBucket: https://bitbucket.org/paul_bakker/dependency-graph/overview

Modularity at JavaOne

JavaOne is just a few weeks away, so it's time for a preview post! The road to JavaOne started last week for me with an interview by Stephen Chin in his Night Hacking show. We have been chatting about my sessions and the book by Bert Ertman and me that will be published right before JavaOne.

This year I will have four sessions, all around the modularity theme.

Building Modular Cloud Applications in Java: Lessons Learned [CON2020]

The first talk, together with Bert Ertman, is about our experiences with modularity in the past few years with building "cloud" applications. The talk is not so much about the cloud itself, but about building modern (web) applications using a modular approach in the "cloud era". We will discuss modular architecture based on OSGi, practical examples of leveraging OSGi services and cloud deployments. If you're interested in how modularity works in practice, this talk is for you.

Tutorial: Building Modular Enterprise Applications in the Cloud Age [TUT1978]
In this two hour talk, Bert Ertman and me are not just talking about building OSGi based cloud applications, we are going to actually show you. OSGi has had a name of being complex and hard to use, but with today's tools and frameworks this is far from true. During this talk we will show how to build modular applications based on dynamic OSGi services by implementing an application from scratch during the talk. If you are relatively new to OSGi, this is the perfect introduction to developing with OSGi. You will learn about modular architecture, package imports/exports, versioning, dynamic services and we will show how to use interact with MongoDB and create RESTful web services. Although the application that we will build during the talk is obviously small, it uses the exact same architecture and patterns that we use in large scale applications.

Modularity in the Cloud: A Case Study [CON2775]
While the previous two talks are mostly about building modular applications, this talk if focussed at deploying modular applications. "Cloud" features such as auto scaling and automated failover sound great, but actually setting this up for large deployments isn't trivial. In this talk, Marcel Offermans and me will share how we have been deploying a large scale OSGi application to the cloud in the past two years. We will also cover baselining and versioning of OSGi bundles, and applying modular deployments. The central component in our approach is Apache ACE; a provisioning server that fits cloud deployments perfectly. If you are either using OSGi and wonder about ways to deploy applications, or are facing the general challenge of deploying to the cloud this talk is for you.

Modular JavaScript [CON2959]
Me giving a talk about JavaScript at JavaOne. I didn't see that one coming! A few years ago I would probably have laughed at the idea, but a lot has change since that time. JavaScript is now a major part of the development I'm involved in. With that, I also started to notice the same problems with JavaScript that we have with Java; how do you keep a large code base maintainable? In this talk, Sander Mak and me will discuss possibilities to introduce modularity in JavaScript. We will be looking at module systems available in JavaScript, and will give detailed examples of RequireJS. Next we will discuss modularity at a higher level that requires services and dependency injection. We will give some examples based on AngularJS. We will also show how all of this fits a modular Java backend, so that we achieve modularity end-to-end.

Book signing session
Bert and me will have a book signing session during the JavaOne week. Rumour goes that there will be some free book give aways... I will tweet the exact time and location when I have the details.

Two years with: MongoDB

In this blog series I will discuss some of the languages, frameworks and components that we used building PulseOn in the past two years. 

PulseOn is an educational system that enables personalized learning. It's used by students and teachers in the classroom. PulseOn runs in the browser on a wide range of devices, and has to support large scale deployments. From a technology standpoint, PulseOn is a textbook example of a modern, cloud based, web application.

Introduction
While we started building PulseOn with a combination of MySQL and a semantic triple store, we now store most of our data in MongoDB. Having many years of experience with relational databases and object relational mapping frameworks in Java this was quite a conceptual leap; but one that I would recommend everyone to make.

Why MongoDB
There are a number of reasons why we use MongoDB.

• Document oriented storage fits a RESTful architecture well
• Schemaless
• Reasonably scalable
• Ease-of-use

Storage in a RESTful architecture

The User Interface of PulseOn is built with HTML5 / JavaScript. Our backend exposes data to clients using RESTful web services. This is interesting because if you look at the data produced and accepted by these services, these are in fact documents. Take a look at the following example; a JSON structure that represents a simple curriculum with learning objectives.

{
  published: true,
  previewLocation: "37289365.jpg",
  definition: "Example",
  learningObjectives: [
   {
     prerequisites: [ ],    
     autoComplete: true,
     subObjectives: [

        {

          prerequisites: [ ],    
          autoComplete: true,
          subObjectives: [ ], 
          title: "example",
        },

     ], 
     title: "example",
   },
   {
     prerequisites: [ ],    
     autoComplete: true,
     subObjectives: [ ], 
     title: "example 2",
   },
  ]
}

The structure is nested. The curriculum contains learning objectives, that may contain sub objectives. From a Java perspective, this is fine, and can easily be represented with classes. In a relational database however, not so much. Although we could easily map this structure using JPA, we would require multiple joins to actually retrieve the structure from the database. This is where performance of even the most trivial selects becomes tricky. And we didn't even start thinking about queries in these nested structures yet....

When using MongoDB instead, we can save this structure without any mapping. In general our Java code defines the data structure as part of a service API. In this example that would be a class Curriculum and a class LearningObjective. These API classes are used internally by our OSGi services, and stored directly (without any mapping) in Mongo. The RESTful resources, based on JAX-RS, produce and consume the exact same types (again without mapping). This makes it extremely natural to work with these "documents".

Schemaless flexibility
A characteristic of MongoDB that has both upsides and downsides is it's schemaless nature. Basically a  single collection can contain documents with totally different properties. First of all, you have to be careful with this. While MongoDB is happy to work with any document you put in a collection, your Java code probably doesn't.

This is a characteristic that you should use with great care. At the same time, for some specific scenarios it's extremely useful. For example we store profiling events for anything interesting a user is doing in the system. These events always have a set of fixed properties like the user id, a timestamp and type. To make these events really useful, most events also store some context information. For example, if a learning object is opened, we want to know which object, the objective it is related to, and the score from our content recommender. Each profiling event type has a different set of context properties. In MongoDB we can still store these different event types in a single collection. This way, we can query different types of events within a single query. Of course, our Java code has to work with untyped data in this case.

Scalability and failover
MongoDB is not the fastest, nor the most scalable datastore in the world. It is however, a very good tradeoff between functionality, speed and scalability. Scalability wise it can't compete with some of the key-value stores out there, but this is really comparing apples and oranges. The biggest difference is that MongoDB offers queries on documents, while key-value stores don't support this in most cases.

Scaling MongoDB is quite easy. You start by setting up a replica set. Even for someone new to MongoDB this should be quite straightforward. You can horizontally scale the cluster by just adding more nodes to the replica set. There is a big caveat to be aware of! By default, both read and write operations will have to go through the master node. This effectively kills horizontal scalability. You can define either at the driver level or operation level if secondaries may be used. This is something you have to think about about while developing code; is it ok for a query to read potentially stale data? When this is used correctly it is quite easy to scale. At least it's a lot easier than scaling a relational database.

Failover is achieved with replica sets as well. We have had several incidents where a server died for some reason (mostly not Mongo's fault), but the replica sets never went down.

Queries and data aggregation
Query functionality is probably the strongest point of MongoDB. Basically, you can do whatever you can do with SQL. In contrast to a relational database there is no concept of relations or joins between collections. In some cases this is a downside (more about this later), but the document oriented structure eliminates most of the reasons for joins. Not having joins makes queries a lot simpler. Queries can use nested properties as well, which is extremely useful in practice.

For data aggregation (e.g. counting and calculating values) you can either use map/reduce or the aggregation framework. The aggregation framework is very easy to use, and it makes a lot more sense than fiddling with having queries in a relational database. Map/reduce is slightly less convenient to work with, but is a powerful fallback. Every time I'm working with complex aggregation questions I'm surprised how easy it is to solve those.

The lack of joins
MongoDB doesn't support the concept of relations between documents in different collections, and with that, the concept of joins. In many cases where you would use joins in a relational database, you don't need them in MongoDB. Instead of joining data from collections together, you can often simply embed the data in another document. Documents can contain arrays of complex structures, so this can get you quite far, because many of the joins in relational databases are necessary just to work around the lack of this functionality. This is not a solution for all one-to-many or many-to-many problems however. The de-normalization of data  is easy and fast while performing queries, but you need extra code to keep different documents in sync, which can be quite difficult. In scenarios where multiple documents relate to the same data, we generally choose to link documents from different collections instead of embedding the data.

Links between documents can be created manually. This can be done by simply adding a field to a document that contains the id of another document (potentially in another collection). The linked document always has to be retrieved manually (a query can only be performed on a single collection) and you need to take care of updating documents when their related documents are deleted manually as well. The latter is not difficult, but easy to forget. We have had several bugs that were caused by this, simply because you don't run into it during development.

When retrieving related documents it is important to do this in an efficient way. Specifically avoid executing too many queries. For example do not do the following (pseudo code):

posts = db.blogposts.find()

while(post.hasNext()) {

   db.authors.findOne(post.authorId)

   //use author data

}

This would execute a query to retrieve the author data for each blog post, which is obviously very expensive. Instead do something like the following (pseude code):

post = db.blogpost.find()
authorIds = []

while(post.hasNext()) {

   authorIds.add({_id: post.authorId})

}

db.authors.find({_id: {$in: authorIds}})
//Use author data

The last example uses only two queries, but you will need to "join" the blogs and authors after retrieving both of them. This sometimes requires a little bit more code, but is mostly very straightforward.

The lack of transactions
Another big difference between MongoDB and relational databases is the fact that MongoDB doesn't support transactions. This feels strange in the beginning after using transactions for many years. In practice, it's really not as big of a problem. Think about how many actions in your system really need transactions. In most cases there is really only one document to update at a time (again, the nested document structure helps us here). Of course there are scenarios where transactions really apply; for example the classical banking example where money is transferred from one account to another. In these rare cases you could consider a polyglot storage solution, where a relational database is used for storing the data where transactions apply.

Data migration and scripting
In the MongoDB shell you use JavaScript to work with the database. The great thing about this is that you can script it. For example store a query result in a variable, iterate over it, and perform some operation for each document in the result. This makes it very easy to perform some data migration or some ad-hoc data analysis. In most relational databases you can do the same using proprietary SQL extensions, but using JavaScript just makes things so much easier (if you're a programmer). 

Indexes
Indexes in MongoDB are very similar to indexes in a relational database. You can create indexes on properties and sets of properties. Just like a relational database, queries are crucial for query performance. Forget to create them, and performance will be bad.

To figure out if any indexes are missing, MongoDB comes with built in profiling. When you turn this on, it collects slow queries. We often use this in combination with load tests to find any problem areas.

Using MongoDB with OSGi
Our stack is entirely based on OSGi. Using MongoDB from OSGi is trivial. The Java driver is already an OSGi bundle, so you can use it directly. To make things even better we created a MongoDB OSGi component in Amdatu. This component let's you configure a MongoDB driver using Configuration Admin and inject it as an OSGi service. For object mapping we use the MongoJack library, which uses Jackson for the actual mapping. This works well because we use Jackson for JSON mapping in JAX-RS as well. You should be able to use most other mapping libraries as well.

Conclusion
MongoDB is certainly not a silver bullet that solves every data related problem. Choosing a solution will always be a trade off between capabilities. MongoDB does come out very strong for general (web) use.

Will I use it again?
Definitely, it's a great fit for general use in web applications.

Forge Seam Persitence plugin

In this video I demonstrate how to add Seam Persistence to your project using Forge, and how Seam Persistence helps solving JPA lazy loading in views and how to add declarative transactions to non-ejb classes.

Seam Forge code quality plugin

In a previous post I demonstrated how to create a full blown Java EE 6 Maven project using Seam Forge. Forge can do much more than just creating project, you can use it for things like scaffolding and adding new stuff to your project.

In this video I demonstrate how to easily add Findbugs and Checkstyle configuration to your project using the Seam Forge Code Quality plugin. The project from the previous video is used as the starting point.
Watch the video fullscreen on HD quality to read the code easier.

CodeQualityPlugin from Paul Bakker on Vimeo.

Java EE 6 live coding presentation video

This live coding presentation shows how to create a complete, full stack Java EE 6 application starting with absolutely nothing. We use CDI, JSF 2.0, JPA 2.0, EJB 3.1, JAX-RS, Seam Forge and Arquillian on the JBoss Application Server and a MySQL database.

This video was recorded during a BEJUG evening session.

Creating a Java EE 6 Maven project using Forge and Intellij

In this screencast I show how to create a Java EE 6 Maven project using Seam Forge and Intellij. Because it's a Maven project it's possible to import the exact same project in other IDEs and it's possible to run a build on a CI server. For this reason you should never use IDE wizards to create projects, because they always create IDE specific project configurations.