Load testing Sitecore Experience Commerce with Taurus



Performance and load testing is a topic often talked about, but many of our customers have not yet seen a practical implementation of it on Sitecore Commerce. The requirements for an eCommerce website are a little different to that of a brochureware or plain content website. Additionally, we are typically dealing with a distributed system involving web servers, payment gateways and in Sitecore Commerce's case, a separate application that handles the eCommerce functionality.

This post aims to provide an overview of a load testing approach that can be applied to eCommerce implementations with a specific focus on Sitecore Commerce. In future posts I'll be expanding on the approach to address scenarios such as Checkouts, integrating with CI/CD pipelines, and running tests at-scale.

Terminology

There a few terms that need to be clarified before we proceed. If you're already familiar with load testing, then you'll be familiar with terms such as:
  • Response time (RT) a.k.a. latency
  • Time to first byte (TTFB)
  • Active users
The issue we've got with measuring everything in terms of requests and the metrics for each request is that an operation on an eCommerce website typically involves more than one request. For example, adding a product to the cart may involve:
  1. Navigating to the product page
  2. Clicking "Add to cart", which fires off an AJAX request
  3. The browser will request the new cart request using an additional AJAX request
For a scenario such as checkout, the number of steps would be even greater. So it then makes sense to measure performance in terms of transactions, which is either a single request, but more often a series of requests. The Transactions per second metric is therefore a more realistic measure of application performance in the context of an eCommerce website.

The scenario

The aim of this post is to provide a practical example using industry standard tools to load test an eCommerce website. The parameters of our scenario are as follows:
  • Sitecore Experience Commerce 9.1 hosted entirely on the local developer machine
  • Recording of test steps for the Add to cart flow on SXA Storefront
    • It is also possible to define the requests manually, but I believe that using a recording proxy or browser extension automates a lot of unnecessary work of manually defining each request that makes up our transaction.
  • Making adjustments to the test steps to support cookies and Anti-CSRF
  • Exploration of Transactions per second on the Add to cart flow on SXA Storefront
    • Interpretations of results to be covered in a future post.

Pre-requisites

If you're aiming to follow along with this post then the following software should be installed:
We'll be using some other tools of the trade that will need not be installed:

What is Taurus?

Taurus is an open-source test automation framework built by BlazeMeter, a company that builds a SaaS continuous testing platform. Taurus essentially abstracts away automation testing tools like Selenium, jMeter and Gatling behind a domain-specific language and also automatically downloads and configures these tools, leaving you to focus on the task at hand.

After you've built your Taurus tests you can scale them up by importing them into your BlazeMeter account and have them run from the cloud, or have them run continuously to ensure that your site is working as expected.

If you choose not to use the Taurus DSL, Taurus also supports running the respective tool's script directly e.g. Gatling's scala or jMeter JMX files.

Walkthrough

After you've installed the software mentioned in the pre-requisites, we're ready to record our first test for Add to cart, make the necessary adjustments then perform a few local test runs.

Recording the test

  1. Load the SXA Storefront in your Google Chrome browser.
  2. Navigate to a product page of a product that can be added to the cart.
  3. Click on the BlazeMeter extension icon and configure your test by giving it a name. There is no need to configure the advanced options, but you may review them for reference, ensuring "Record Ajax Requests" has been selected.
  4. Click the "Start recording" button. Re-load the page so that the request for loading the product page is captured. After this, the counter under "JMX" should be "4":
  5. Click "Add to cart". the counter under "JMX" should be "6". Click the "Stop recording" button.
  6. The Chrome extension should now show "Run", "Edit" and "Save" buttons. Click "Save" to save our test run in yaml format, including both Selenium and jMeter scripts.
Note that we use yaml instead of exporting a JMX file as later we're going to run the script in Taurus, which generates a JMX file at runtime for use with jMeter. It is also possible to run a JMX file directly with Taurus, but for the purposes of this demo we're going to define our test entirely in yaml format.

Making adjustments to the test script

In some basic cases we would be able to use the script right away, but in this scenario we will need to update our script to support submitting the .NET Anti-CSRF token in all of our requests. 
  1. Open the yaml file that was generated by the Chrome Extension. Remove the "- executor: selenium" and "Add to cart-Selenium" nodes.
  2. In the first request under the Add to cart transaction node, add the following code:
                extract-css-jquery:
                    csrf_token: 
                        expression: input[name=__RequestVerificationToken]
                        attribute: value
    
  3. Search and replace for the __RequestVerificationToken header value in each subsequent request with "${csrf_token}". e.g. "SGRFVsQg2baFEFrKV7ryMdwsuS-fFQ0rh-YmWWOFCiOS9q5DjUgScSdHdODSgdkSltLJ7Qae2kHH5FND7j6iQ-tKngt8tHnLo4tak7HcNpo1" becomes "${csrf_token}".

  4. Update the execution parameters so that the test will run with 1 concurrent user over 20 minutes.

Run the test

Once we have made the necessary adjustments to our script, it can be executed with Taurus. 
  1. Open a command line and navigate to where you have saved your test script. 
  2. To use BlazeMeter live reporting, type "bzt <your test script>.yaml -report". This will open a live report in your browser reporting on the performance stats.

  3. If you just want to use the built-in CLI Taurus reporting interface, simply type "bzt <your test script>.yaml".
  4. Once execution is complete, all artifacts, including error reports, the generated jMeter script and request / response log will be saved into a subdirectory. Refer to this directory if your test was unsuccessful or you had excessive errors.
  5. In subsequent executions you're going to want to adjust the concurrency and ramp-up settings to generate a more complete view of transaction throughput vs. number of concurrent users.


Conclusion

In this post we covered using the BlazeMeter Chrome extension to record our load test scenario and executing it with Taurus. In the next post I will cover how the results can be interpreted and what implications the results have for tuning your Sitecore Commerce architecture.

Files

The following file is a Taurus script that was recorded with the BlazeMeter Chrome extension and updated with the Anti-CSRF token support.

References

Comments