Hazelcast JMS Provider

I'm currently using Apache ActiveMQ on a project in a "network of brokers" configuration with Apache Camel used to implement the enterprise integration patterns. From a development point of view, you work with a Camel generated proxy and setup routes to transform dispatch, and receive messages. This hides all the Java Messaging Service (JMS) complexities behind Camel's JMS (or ActiveMQ) components. My system primarily uses this setup for low volume, asynchronous work requests or RPC request/reply operations.

For the most part things work well but it has been a constant fight trying to get a stable system. The issues range from configuration complexity, network instability, deployment requirements, and just plain old bugs. Both ActiveMQ and Camel have a rather large set of dependencies and properly configuring a JMS broker and clients is rather tricky. A small configuration issue can cause your system to slowly grind to halt (usually only under production loads)! While ActiveMQ is one of the easier brokers to configure and use (it has Spring namespace bindings and can be used without JNDI), I could still imaging a full time position for broker management, configuration, monitoring, etc.

On another part of the system I've been prototyping some distributed locking using Hazelcast, an in-memory data grid that implements Java's Map, List, Queue, and Lock interfaces. I've been extremely impressed with how easy it is to get a Hazelcast cluster up and running and the fact that it has few, if any dependencies is another plus. As a data grid, clustering is a first class citizen in Hazelcast unlike a lot of JMS brokers which use a hub and spoke pattern with a "network of brokers" as a bit of an after thought for high availability.

While working on the project, I got to thinking about switching out the ActiveMQ broker with Hazelcast using queues and topics. There are a number of pros and cons to consider:

Pros

• I'm looking to use it for distributed locks so I could remove a dependency on ActiveMQ/JMS by using it for both locks and messages
• It is designed from the ground up for clustering and data distribution while clustering is a bit of an add-on for ActiveMQ (and other MQs)
• It is much simpler to configure and support than a JMS serves
• It has built in support for distributed queues
• I could leverage it in the future as a distributed cache
• I could continue to hide it behind Camel so there would be little to no code change in my code

Cons

• The distributed queues are in memory in Hazelcast so messages would be lost if all nodes went down (it should survive any single node failure). That being said, my queues are empty 95% of the time because the messages are consumed quickly. Hazelcast also has persistence support for maps which I believe back the queue implementation so some form of persistence may be possible.
• It exposes a custom REST API but not a generic STOMP API so I would need to either implement a STOMP server or rewrite some of our existing non-Java code that uses the STOMP interface on ActiveMQ
• Camel has basic Hazelcast support (as a cache) but I would need to write some non-trivial code to support message marshaling and transaction management
• Performance is a bit of an unknown. There are performance numbers for raw distributed map read/writes but not so much as a message passing system. That being said, I have very low performance requirements for the bus (10s - 100s of messages a second but not much more than that).

I first played around with writing a custom Apache Camel component but it quickly got complicated. Handling request/reply patterns with asynchronous clients isn't simple and properly using the internal Camel APIs definitely has a learning curve. That's when I got the idea to simply reuse Camel's JMS component and write a JMS provider for Hazelcast. The JMS APIs are well documented and relatively simple to understand.

After a day or so of work I have a simple, working implementation of a JMS provider on top Hazelcast which supports message producers and consumers on topics, queues, and temporary queues. I was even able to swap out the ActiveMQ ConnectionFactory for my HazelcastMQ ConnectionFactory with no code changes in my application. So far things are looking promising. The big advantage of implementing the JMS APIs is that it "just works" with Apache Camel, the Spring JmsTemplate API, or other JMS consumer.

There are some limitations that I don't have a good solution for yet. For example, implementing consumer message selectors isn't an option with Hazelcast queues (although they do support them with maps). Also, some of the JMS options like message persistence can't be honored because persistence would be all or nothing configured within Hazelcast. I'm hoping I can continue to work around these issues to support the majority of common usage patterns.

I'll post again soon with some more examples and hopefully I'll have the code available shortly if there is interest.