Assertions in JMeter Responses

Why should I care? #

Those of us who work with JMeter and performance testing know the importance of adding validations to the main HTTP Requests. Validations, assertions, something that automatically tells you if the response is OK or if there’s an error. If you’re not convinced by this idea, let’s think for a moment that you’re getting super-fast response times in your load test, and you generate a report to your client or manager saying that the site isn’t running, but rather is flying. Now imagine that, since there was something wrong with the test, the server’s responses were actually sending you a page that said “site down, please come back later.” And I’ll tell you more, all of us who have played with concurrency testing in applications have experienced the server notifying us of an error on a page with a response code of 200. Yes! 200, which means “OK,” means “no problem.” Curious, isn’t it? That is, we cannot rely solely on the response code, we need to do some minimal extra validation at each step to give us the peace of mind that we are simulating the test we want to simulate.

EIn this sense, validations are not only a way to corroborate that the server is responding as expected, but they also serve to verify that the script we are developing is acting as it should, that it is simulating user behavior as designed.

For example, if the impact of our script on the system involves creating a record in the database, we must ensure that after executing the script, the record is not created, and that the script states that it completed without problems.

These are examples of poor assertion or validation choices in our requests.

In other words, what we need is to avoid false negatives and false positives.

Avoiding false positives and false negatives #

Let’s first look at what these concepts mean:

• False Positive: The test says there is an error, but there isn’t, so we waste time searching for the cause of the error until we realize that the error was in the test, the environment, the data, etc.
• False Negative: There are bugs, and we’re so calm! Confident that those features are covered, that they’re being tested, and therefore bug-free (Watch out for this!)

That is, if there is an error, the script must show a failure; if the system is working correctly, the script must say there are no errors, and we must be able to trust that.

But how do we select good assertions?

Defining good assertions for our HTTP Requests #

The reality is that we haven’t found a clear guide on how to define assertions for each HTTP Request, and since this is something we always have to explain to new performance testers we recruit or train, we created our own guide.

This is why I would like to share with you the methods and criteria that have been most successful for us when selecting what to validate in a server’s response during a load test with JMeter (as we mentioned in this checklist to improve JMeter scripts).

1.- Validate a login #

Typically, after authenticating to a system, we see some text such as “Welcome Username” or our name appears somewhere on the screen, indicating that the login was successful. Therefore, in these cases, it is an excellent element to validate.

Remember that in JMeter, we can specify a variable in validations. So, in cases where the username is parameterized (read from a CSV file or defined as a user variable), I can validate the text ${UserName} and JMeter will replace the variable with its value at runtime.

2.- Validate window content #

Whenever the request we make returns an HTML page, we can choose text or titles that should appear in that response. For example, if I access the JMeter site ( http://jmeter.apache.org/), we can see the following:

So we know that if this site loads correctly, the title “What can I do with it?” should appear in response to the request made to the site. So, it’s a good assertion to add as validation that we’re receiving the expected page. The challenge with this type of validation is validating content so that we know that, in case of an error, that text doesn’t appear in the response. For this, we can turn to the developers for this type of information.

3.- Consider the script’s next steps #

If we know that the next action to perform is to click on button X, or access link Z, or select option A from combo box B, a very good strategy is to validate that the request response includes the element we want to act on in the next step. For example, if after loading the JMeter site we want to select the “Download Releases” option, we can validate this text and ensure the continuity of the flow. For this we resort to capturing the server response and check how that option is presented in the HTML, in the example we see this:

Download</div><ul><li><a href="./download_jmeter.cgi">Download Releases</a>

And then we create the validation in JMeter with >Download Releases< :

4.- Validate values to be extracted #

Sometimes, we extract variables from an HTTP Request response using regular expressions (Add->Post Processors->Regular Expression Extractor) with the intention of using that variable in a subsequent request. This is what we do in what is known as variable mapping.

We can then validate that this variable appears in the server’s response. To do this, we can use validations that utilize regular expressions. For example:

This way, if the regular expression doesn’t match the response received, we’ll get a validation error.

5.- Be careful with special characters #

You have to be careful when writing the assertion in JMeter because the text isn’t always received in the same format as it’s displayed on the screen. For example, if I wanted to validate the title Apache JMeter™, I can’t write it verbatim because the server response appears as Apache JMeter&trade; and not as it appears on the screen. If we don’t use the text as it appears in the HTML, the validation (as well as the regular expressions) won’t find it.

6.- The right number of assertions #

JMeter allows you to add as many assertions as the user wants, so each of the assertions mentioned above can be accompanied by other assertions, such as HTML response titles (<title>...</title>), to add robustness to the validation.

When adding more assertions, we must be careful that the script doesn’t become too overloaded. This is why it’s not recommended to add more than two assertions per request. Therefore, we must ensure that the ones we choose are decisive when defining the correctness of the response.

How are you going to validate your responses? #

We hope this list is helpful when adding assertions. In summary, I’d like you to keep this in mind:

• Add between one and two assertions for each important HTTP Request in the flow you’re testing.
• Choose to validate whatever you want: titles, text, values, or elements you’ll interact with in the following steps of the flow. The important thing is that if the script fails, it’s because the system fails, and if the system fails, the script fails (thus avoiding false positives and false negatives). If you have any other ideas, I’d love for you to share them so we can continually improve the way we do our load testing.

About this post #

We prepared this post a few years ago in English with several colleagues at Abstracta, but it remains relevant. It’s about understanding how to validate an HTTP request in JMeter. The original post in English can be accessed here. I say it’s still relevant because the HTTP protocol is quite stable, as is JMeter, so best practices around this haven’t changed much.

Federico Toledo
COO at www.abstracta.us
twitter.com/fltoledo
Bio: https://www.linkedin.com/in/federicotoledo/
Sitio: https://www.federico-toledo.com/