I’m sure that whoever follows our blog, is fairly familiar with Mule, but what is Waratek? Let’s cut to the chase here, and announce that for me, Waratek offers one of the coolest products I have ever seen. Waratek brought to the JVM, what VMWare brought to the hardware. Waratek offers a JVM that can be virtualised! It is like applying LPARS to your JVM. Suddenly your JVM becomes multi-tenant. How cool is that?
So how hard is it to start Mule on Waratek JVM? Well its very easy. Waratek employed the idea that the developer needs to do ZERO code changes in order to be able to run on their JVM. This is a huge relief, because it now means that to virtualise your JVM, you need zero development effort.
To test Mule on the Waratek JVM, we used the latest community edition of Mule 3.4.0 from here. As for Waratek, we downloaded the OVA image available here. This is an open format virtualisation image than can be run on several virtualisation packages such as Virtual Box and VMWare. It comes with CentOS 6 pre-configured with the Waratek Cloud VM and packaged with some demo applications .
Once we had everything installed, we just started Mule as we would normally do. Mule informs us on which JVM it is running in the boilerplate. This is what we got:
Shutting down Mule can be done just as simply; hit the Ctrl+C key combination. The Waratek JVM tell’s us that the interrupt is unsupported, but Mule will still shut down cleanly. The commands to start and stop Mule as a daemon, i.e mule start, and mule stop, also work out of the box on the Waratek JVM.
So what is happening underneath? When we start the Waratek JVM, it usually starts as a daemon process (although it can also be started as a foreground process), called javad. This is literally a hypervisor for Java, very similar in concept to the VMWare hypervisor. Waratek also provides a script called java, to replace the plain old java command. This script, instead of launching a normal JVM, will create a JVC (Java Virtual Container) inside the Waratek JVM, and kicks it off. In a few words, a JVC is a virtualised instance of a JVM, completely isolated from any other JVC. This compares to a VMWare image.
Waratek allows us to manage the hypervisor using an SSH connection. Once you SSH into the hypervisor, you can use jirsh. If we list the current JVCs while Mule is running, we get something like this:
One thing you have to be aware of, is that before Mule starts, it launches a Groovy script to prepare the classpath and perform some other preparations. Groovy scripts run on a JVM just like a normal Java application. So what happens here is that once you launch Mule, first a JVC is created to launch the Groovy script, and upon completion, another JVC is created to launch Mule. If you SSH into the hypervisor, and execute a list command while Mule is starting, you will get something like this:
Once the Groovy process is complete, the JVC is destroyed and you will have one JVC running Mule as shown above. This does come with one small caveat; Waratek limits the number of JVCs you can use depending on the licence you buy. If you have a 4 JVC licence, you can only start 3 Mule instances since the last one will not manage to start both the Groovy script and Mule at the same time. I am sure that this can be easily fixed. (Waratek have been consulted and have so far been unable to reproduce the behaviour so it may not affect all users. In other words; you should be able to run the number of JVCs you are licensed for.)
One suggestion would be to run the Groovy script on a normal JVM, like the Oracle JVM or OpenJDK, and run Mule on the Waratek JVM.
Once Mule is stopped, either by running mule stop if Mule is running as a daemon, or by pressing Ctrl+C if Mule is running in the foreground, when we inspect the hypervisor we should see the following. One thing to notice is that the JVC is not undefined from the hypervisor, it will just be in Shut Off state. This makes it very easy to restart it from the hypervisor itself by just invoking start jvc-id.
There is one thing you have to take care with when running Mule on the Waratek JVM. Mule uses the Tanuki Java Service Wrapper. This makes it very easy to run Mule as daemon process on different operating systems. It also offers other features like restarting your JVM if it hangs. The Tanuki wrapper will ping your JVM and if it does not receive a reply after a certain time, it will kill your JVM and start a new instance. This works perfectly well with the normal JVM, but it does not compliment the Waratek JVM.
If the JVM hangs, the Tanuki wrapper tries to kill your JVM by executing a shutdown process in your JVM itself (soft shutdown). However, if this fails, it performs a kill -9 on the process id.
The biggest issue here is that the Tanuki wrapper would not have the correct process id. Mule does not have a process id, its running in a JVC inside another JVM, the Waratek JVM. Luckily the wrapper does not try to kill the Waratek JVM, but instead it will try to kill the Waratek java command that launched off the JVC. This is not too bad, because it has no side effects.
However, this does not kill Mule, and the wrapper will start a new instance of Mule. This results in having 2 instances of Mule running in different JVCs, one hung and one running fine. As you can imagine, this can cause issues.
Our suggestion for this situation is to switch off this particular feature by adding the following line in the $MULE_HOME/conf/wrapper.conf:
This sets the timeout to infinity, hence the wrapper will never try to kill and restart a new instance of Mule. Of course this means that you have to put some monitoring in place, but that is normal with all enterprise applications like Mule.
We still haven’t explored one of the amazing features of the Waratek JVM: elastic memory. This feature allows us to assign memory to a JVC, plus share a pool of memory between different JVCs. A JVC will only use memory from the shared pool, when it would have exhausted its own. This is a very handy feature to avoid out of memory issues that usually appear only during peak times. On top of that, Waratek allows us to change the configuration of the JVC, like memory allocation, dynamically during runtime. There is no need to reboot your application if an expansion in heap is required.
What does this lead to? Well this means that now you can have Mule applications completely isolated from each other, running in different JVCs on separate Mule instances. A shared pool can be configured to allow each JVC access to extra memory during peak times. Moreover you can use the Waratek JVM features to limit each JVC (Mule app) in memory, network, CPU pinning, and more. No application will affect the other if it runs out of memory, or acts badly.
We tried running multiple Mules on the Waratek JVM, and when we explored the hypervisor, this is what we could see:
To conclude this blog, I re-iterate what I said in the beginning. The Waratek JVM is one of the coolest products I have seen. Its a very powerful and useful tool. Running Mule on it makes a lot of sense as you are able to apply a lot of control to each instance while having all instances completely isolated from each other.