WSO2 Enterprise Service Bus (ESB) Performance Testing - Round 3

This article is the third in a set of articles that measure and compare the performance of the WSO2 Enterprise Service Bus (ESB) against other leading implementations - both open source and proprietary.

Table of Content

1. Introduction
2. Sample Request Messages
3. Completeness of the Scenario Implementations and Issues Encountered
4. Benchmark Configurations and Adjustments Made
5. Results
6. Conclusions
7. Resources

Introduction

In this round of testing, we have compared the WSO2 Enterprise Service Bus (ESB) [6] v1.7 against the leading proprietary ESB (PROP-S/PROP-N), the proprietary version of an Open Source ESB (PROP-OS), the open source alternatives - Apache Service Mix 3.2.1 and Mule Community Edition 2.0.1 (Mule CE). The WSO2 Enterprise Service Bus (ESB) is an open source project (under the Apache License v2.0) developed by WSO2 based on the popular Apache Synapse Enterprise Service Bus (ESB) [7]. Since the WSO2 ESB v1.7 embeds the Apache Synapse ESB v1.2 release, the performance figures between these will be identical. However, the WSO2 ESB ships with better management and development support with a graphical Web interface, enhanced JMX support and a built-in WSO2 Registry instance.

Although, initially we planned to include the OpenESB and the JBoss ESB in the testing rounds, due to the complexity involved in understanding these products to build the simple scenarios we needed, we decided to go without them and to include a follow up article instead. You will recall from the first and second rounds of testing, that we had three specific scenarios benchmarked designed to test the processing performance of an Enterprise Service Bus (ESB) intermediary. Although these scenarios obviously aren't exhaustive, and we understand that there could be other scenarios where alternative benchmarks may show different results, these are some of the most common use-cases based on customer situations that we have encountered. The three scenarios are:

1. Virtualization, where the ESB hides the actual service and performs message routing

In this scenario we create a proxy service (defined by the ProxyWSDL) for the actual Echo service (defined by the EchoWSDL), which simply passes on the messages to the real service. This pattern allows users to hide real services behind proxies exposed over the ESB and thereby prevent direct application-to-application, application-to-service and/or service-to-service links from being created within an enterprise to bring order to a Service Oriented Architecture (SOA) deployment. This additionally allow users to perform authentication, authorization, validation, WS-Security, WS-Reliable Messaging and SSL decryption etc at the ESB layer, so that the real services within the organization could be simplified. This also allows real services available over different transports or schemas to be exposed over other transports and/or interfaces, and also allow WS-Policy attachments for consistent enforcement of enterprise wide policies.

2. Content-based Routing (CBR), where the ESB routes on data within the message

The Content Based Routing (CBR) proxy service performs an evaluation of an XPath expression over the payload of the messages, before they are routed to the real service. For this example, we use an XML payload with a list of 'order' elements, and check if the first order element is for the symbol "IBM". If this condition is satisfied, we forward it to the actual service implementation, else return an error. Typically payloads are routed on transport/SOAP headers and/or user defined header elements, or payload body elements.

3. Transformation, where the ESB transforms the request and response messages using XSLT-based transformations

In this scenario, we expose a different interface at the proxy service that expects all elements in reverse order. Thus a client querying the WSDL of the proxy service is presented a completely different WSDL with a different schema. The messages received from the client is in reverse order, and are then transformed as expected by the real service. The response from the real service is then transformed again (i.e. reversed) and sent back to the client. This is a typical use case when newer versions of a service are exposed, and a subset of its users now require backwards compatibility with the previous schemas etc.

Sample Request Messages

• A sample request message used for the Direct Proxy and CBR Proxy scenarios is given below. Refer[4] for actual requests used.

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://services.samples/xsd">
    <soapenv:Body>
        <xsd:buyStocks>
            <order><symbol>IBM</symbol><buyerID>asankha</buyerID><price>140.34</price><volume>2000</volume></order>
            <order><symbol>MSFT</symbol><buyerID>ruwan</buyerID><price>23.56</price><volume>8030</volume></order>
            <order><symbol>SUN</symbol><buyerID>indika</buyerID><price>14.56</price><volume>5000000</volume></order>
        </xsd:buyStocks>
    </soapenv:Body>
</soapenv:Envelope>

• A sample request message used for the XSLT Proxy scenario is given below. Refer[4] for actual requests used.

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://services.samples/xsd">
    <soapenv:Body>
        <xsd:skcotSyub>
            <redro><lobmys>IBM</lobmys><DIreyub>asankha</DIreyub><ecirp>140.34</ecirp><emulov>2000</emulov></redro>
            <redro><lobmys>MSFT</lobmys><DIreyub>ruwan</DIreyub><ecirp>23.56</ecirp><emulov>8030</emulov></redro>
            <redro><lobmys>SUN</lobmys><DIreyub>indika</DIreyub><ecirp>14.56</ecirp><emulov>500</emulov></redro>
        </xsd:skcotSyub>
    </soapenv:Body>
</soapenv:Envelope>

Completeness of the Scenario Implementations and Issues Encountered

We did not encounter any issues in implementing the scenarios within the WSO2 Enterprise Service Bus (ESB) or the proprietary ESB. However, Mule CE was not able to create any real proxy services (i.e. a service that can 'expose' a different interface from that of a real service they hide behind, and have the ability to forward those requests to the real service), without having to write a full service implementation class in Java. The proprietary version of the open source ESB was unable to expose the WSDL for the XSLT transformation scenario due to an error. Apache ServiceMix couldn't proxy the incoming SOAPAction to the real service implementation, and we found this to be a known issue with ServiceMix 3.2.1

Figure 1 - Ability to create real proxy services for the scenarios

Table 1 - Issues encountered with each ESB

Benchmark Configurations and Adjustments Made

Table 2 - Tuning of each of the systems

Backend Service Implementation

We developed a mock EchoService as a servlet which simulated a 10ms processing time, and deployed this on an Apache Tomcat 5.5.26 instance.We soon noticed Tomcat running out of resources to support the load test, with a message "SEVERE: All threads (150) are currently busy, waiting. Increase maxThreads (150) or check the servlet status". We then increased the maxThreads and acceptCount to 600, and allocated 512M of heap memory. The MockService WAR and its source code can be found at[2]

Client implementation

We used the excellent Apache HttpComponents projects' Benchmark code, which is a good replacement for the ApacheBench load generator, with support for HTTP 1.1 and SSL. This code has now been exposed as a simple to use binary JavaBench, and its source code/binary is available at[1]

Benchmarking methodology followed

We used the shell scripts available at[3] to drive each scenario. A warm up run of each of the three scenarios were executed by the script before the actual tests for each ESB began. We also re-started the Apache Tomcat backend before each test run. The output of the shell scripts were re-directed to a text file, and was fed through a utility to convert into a CSV file, which was then used to generate the tables and graphs.

Results

Important: In the following tables and charts, the proprietary ESB used, and the proprietary version of the open source ESB are indicated with the following keys:

• "PROP-S" - The proprietary ESB with its additional performance improvements turned on, and started in 'Production' mode
• "PROP-N" - The proprietary ESB without the additional performance improvements, and started in 'Production' mode
• "PROP-OS" - The proprietary version of an open source ESB

1. Direct Proxy Service - Virtualization

Observations:

1. Mule CE 2.0.1 and the proprietary version of the open source ESB crashed with a concurrency level over 80 (Also see [5])
2. Mule CE 2.0.1 consistently failed on ~1% of the requests for each and every scenario tested, even for the low concurrency tests.
3. Apache ServiceMix 3.2.1 failed to forward the incoming SOAPAction to the real service
4. The proprietary ESB demonstrated a clear leadership over all scenarios, with or without additional performance improvements turned on
5. The proprietary ESB performed 1.6~1.9 times TPS on average (with and without additional performance improvements) than the WSO2 ESB
6. WSO2 Enterprise Service Bus (ESB) v1.7 with its default configuration beat all other Open Source alternatives consistently
7. The WSO2 ESB average TPS was over 4 times than that of Mule CE and the proprietary version of the open source ESB, and over twice that of Apache ServiceMix

2. Content Based Routing

Observations:

1. Mule CE 2.0.1 and the proprietary version of the open source ESB crashed with a concurrency level over 80 (Also see [5])
2. Mule CE 2.0.1 consistently failed on ~1% of requests for each and every scenario tested, even for the low concurrency tests.
3. Apache ServiceMix 3.2.1 failed to forward the incoming SOAPAction to the real service
4. The WSO2 ESB performed better than the proprietary ESB (even with its additional performance improvements turned on) for 1/2K and 1K messages.
5. For 5K messages, the proprietary ESB did slightly better than the WSO2 ESB.
6. WSO2 Enterprise Service Bus (ESB) v1.7 with its default configuration beat all other Open Source alternatives consistently
7. The WSO2 ESB average TPS was roughly over 3 times than that of Mule CE, the proprietary version of the open source ESB, and Apache ServiceMix

3. XSLT Transformations

Notes:

• We turned on the default Apache Xalan translet transformation flag with the WSO2 ESB and this result is shown as "WSO2-*"
• The proprietary ESB did not have any additional performance optimizations for this scenario, where the message was being transformed during proxying

Observations:

1. Mule CE 2.0.1 and the proprietary version of the open source ESB crashed with a concurrency level over 80 (Also see [5])
2. Mule CE 2.0.1 consistently failed on ~1% of requests for each and every scenario tested, even for the low concurrency tests.
3. Apache ServiceMix 3.2.1 failed to forward the incoming SOAPAction to the real service
4. The WSO2 ESB performed better than the proprietary ESB for 1/2K messages with Xalan translet compilation enabled, but was beaten by it for the 1K and 5K messages.
5. WSO2 Enterprise Service Bus (ESB) v1.7 with Xalan translet compilation enabled, beat Mule CE and the proprietary version of the open source ESB for all scenarios.
6. Apache ServiceMix beats WSO2 ESB for XSLT transformation of 5K messages
7. Considering the average TPS, the WSO2 ESB with the Xalan translet compilation enabled, performed better than all ESB's with a close tie from the proprietary ESB, but with a clear lead against the open source options.

Conclusions

• WSO2 Enterprise Service Bus (ESB) displays excellent stability and performance on all scenarios
• WSO2 ESB v1.7 leads in performance over all other open source alternatives, as well as the proprietary version of the open source ESB benchmarked
• The performance of the proprietary ESB has improved drastically over the past year, as compared to numbers it gained during the rounds 1[8] and 2[9]

Notes

• All performance numbers were captured while running a single instance of each ESB, on a single JVM.
• All software (JavaBench, Tomcat, WSO2 ESB, Mule CE, the proprietary version of the open source ESB and ServiceMix) were run on the Sun JDK 1.5.0_15 on Red Hat Enterprise Linux Server release 5 (Tikanga)
• The proprietary ESB was run on its own JVM on the same Red Hat Enterprise Linux Server release 5 (Tikanga) server
• Hardware configuration
• Client / Load generator - Intel(R) Xeon(TM) 3.20GHz 2MB Cache Dual Core - 2 CPU system / 2GB RAM / 1G NW
• Enterprise Service Bus - Intel(R) Xeon(TM) 3.20GHz 2MB Cache Dual Core - 2 CPU system / 2GB RAM / 1G NW
• Backend service simulator - Intel(R) Xeon(R) 2.00GHz Dual core - 2 CPU system / 2 GB RAM / 1G NW
• All three systems were tuned at the Operating System level as follows:

/etc/sysctl.conf 
net.ipv4.ip_local_port_range = 1024 65535 
net.ipv4.tcp_fin_timeout = 30 
fs.file-max = 2097152 
net.ipv4.tcp_tw_recycle = 1 
net.ipv4.tcp_tw_reuse = 1 

/etc/security/limits.conf 
* soft nofile 4096 
* hard nofile 65535

Resources

[1] JavaBench - a Java clone of the popular Apache Bench load generator, implemented by Apache HttpComponents project

Source available at https://wso2.org/repos/wso2/trunk/commons/performance/esb/Tools/Client, with build instructions and a Maven project file

A complimentary binary build is available here: java-bench-0.1.zip

[2] MockServices - a high performance Echo Servlet that acts as a service simulator.

Source available at https://wso2.org/repos/wso2/trunk/commons/performance/esb/Tools/MockService/mock-services with build instructions and a Maven project file

A complimentary binary build is available here: MockServices.war

[3] Resources used for the load testing, including XSLT transformations, WSDL's, and shell scripts etc. available at https://wso2.org/repos/wso2/trunk/commons/performance/esb/Round_3/Resources

[4] Sample client requests of 0.5K, 1K and 5K can be found at https://wso2.org/repos/wso2/trunk/commons/performance/esb/Tools/Client/requests

[5] Mule CE 2.0.1 encountered the following exception when the number of concurrent users exceeded the maximum thread count configured

ERROR 2008-06-20 16:42:05,283 [HttpConnector.receiver.1] org.mule.providers.http.HttpConnector: Work caused exception on 'workCompleted'. 
   Work being executed was: HttpMessageReceiver{this=160c4b0, receiverKey=http://testb.wso2.com:8090/services/CBRProxy, endpoint=http://testb.wso2.com:8090/services/CBRProxy}


ERROR 2008-06-20 16:42:05,285 [HttpConnector.receiver.1] org.mule.impl.DefaultExceptionStrategy: Caught exception in Exception Strategy: null
edu.emory.mathcs.backport.java.util.concurrent.RejectedExecutionException
        at org.mule.util.concurrent.WaitPolicy.rejectedExecution(WaitPolicy.java:56)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:765)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1288)
        at org.mule.impl.work.ScheduleWorkExecutor.doExecute(ScheduleWorkExecutor.java:39)
        at org.mule.impl.work.MuleWorkManager.executeWork(MuleWorkManager.java:276)
        at org.mule.impl.work.MuleWorkManager.scheduleWork(MuleWorkManager.java:243)
        at org.mule.providers.tcp.TcpMessageReceiver.run(TcpMessageReceiver.java:177)
        at org.mule.impl.work.WorkerContext.run(WorkerContext.java:310)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1061)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:575)
        at java.lang.Thread.run(Thread.java:595)

[6] WSO2 Enterprise Service Bus (ESB) - http://wso2.org/esb

[7] Apache Synapse Enterprise Service Bus (ESB) - http://synapse.apache.org

[8] Performance Testing Round 1 = http://wso2.org/library/1721

[9] Performance Testing Round 2 = http://wso2.org/library/2259

Author

Asankha Perera, WSO2 Inc.

Project Management Commitee member and comitter of Apache Synapse Enterprise Service Bus (ESB), Apache Web Services and Apache HttpComponents projects

Architect and Product Manager of the WSO2 Enterprise Service Bus (ESB)

asankha at wso2 dot com

http://esbmagic.blogspot.com