API Testing - without writing any code

In Redmart, we started out with a monolith which had all the eCommerce modules, in one service (and a single repository). All the data was created and managed using this monolith. But when this data needed to be sent to the consumer facing website, we used to do so by routing all the requests from this consumer site to an intermediate layer, called the API layer. This layer then talked to Mongo, fetched the required data, curated it and then sent it back.

Once the business was established, we slowly made changes to this monolith. We started to refactor and remove the very clearly defined and written eCommerce modules into different microservices. All that was left now, was the API layer which still remained as it is.

Now, in a typical microservices concept, a microservice should not, or is not supposed to update the collection (data) owned and operated by any other service, by itself. The API layer now, is in violation of this principle or standard or whatever you might want to call it.

Each engineer who owned a particular service was tasked to move the API endpoints that are ideally supposed to belong in their services. This meant refactoring and moving a lot of APIs that were sort of legacy code. This would definitely lead to a lot of regression testing of the API responses, once the refactoring was done. But this regression would be a huge and tedious task to be done manually and also, the QA engineers should have complete knowledge of each API and its response, which is a crazy expectation!

I am one of those engineers who own a service and few related APIs. I definitely did not want to do this testing manually. I am, for some reason, not comfortable doing this testing on an alpha environment or with some simulated data. These changes, though small, have high impact on business (customer experience, broken UI and worse, lost revenue and customers), if things go wrong. Hence my hesitation.

I had read about a recent performance and testing activity done by github when they refactored their merge algorithm. "It is pure genius", I thought. Fast-forward a couple of weeks, I started investigating more before starting the task at hand, and I came across another post by another really cool company, Twitter. They spoke of testing services without testing using Diffy. Curious enough blogpost title, to get me started.

There it was, precisely the kind of testing that I wanted to do and a tool for it. It met all my requirements. Here is the whole idea behind this testing:

  1. Deploy the old API service in an instance/box
  2. Once done, deploy the new refactored service with the newly added and implemented APIs, in another two instances
  3. Create an intermediary proxy that shall capture the incoming requests from the consumer site and multicasts the request to the instances setup in Step-1 and 2 (which in this case is done by Diffy)
  4. Continue to send the response from the API layer to the website
  5. But capture the response in step-4 along with the response sent by the new instances and compare them for any differences

If the schema of the responses and, the attributes and its values match, we can safely conclude that the responses generated by the new service are same as the old API layer (the tool has a feature to consider any noise in the responses).

The code that needs to be written to do the verification in step-5 would be simple and minimal. But, preventing the need for it, Diffy (twitter's new tool), provides this report in a very elegant manner, in a very neat, simple to use and elegant UI. Setting it up is very very easy too. It took about 5 minutes to actually clone the tool's repo, and another few minutes to start 3 instances of my application, mongodb and test (using curl in command-line) a simple API that returned a list of supported languages in my application.

I tweaked the config file in one of the application instances (that's supposed to mimic the newly refactored service), to remove one language. Then, I started 2 instances of this new service and one instance of the old service(as required by the example). I then started the diffy tool once it was was built using SBT. The only remaining thing left to be done now, is to invoke an API endpoint via the port exposed by the Diffy tool, and see the differences in responses returned by the application instances, and compared, in a nicely generated UI by the tool.

This UI can be accessed at a port exposed by the Diffy tool at: http://localhost:31149/ . Here's the report generated for the API I tested:


You can see better examples on the tool's website.

The moment one hooks up this set up in a production environment, we can start testing the API changes, realtime and cover all possible scenarios. We can go a step further and also perform performance testing in scenarios where it is deemed fit, for example, how long does it take to generate the API responses in the old service, compared to the new one, etc (which is what the engineering team at github did).

All this investigation is a precursor task before the refactoring task is taken up and, I believe is necessary, since any mistake here is a direct impact on consumer facing data and customer experience. I shall post much better examples and screenshots once the refactor task is done :) But the important point is that, next time if there is any changes to the code serving the APIs, it can be quickly tested in a matter of minutes.

Hope this helps others as well. Please do comment if you have come across similar tools, or even better! Any corrections or suggestions to this post are most welcome.