Running Postman Tests with CI/CD

Looking for a framework to perform integration tests, I’ve found that our friends of Postman implemented a really cool way to test your API calls. In any Postman collection, there is a Tests section where you can declare all the assertions you want to do on the response.

In the official example above, they are testing an scenario where they fetch all items, then create a new item, and finally verify if the item was created as expected. That’s great, but how can we run these tests from a command line, so we can add it as a step of our CI/CD pipeline? here it comes another great tool of the Postman team: Newman.

newman run examples/sample-collection.json

That’s it, that way we can run any Postman collection (previously exported as a JSON file). It seems we’ve found a nice solution for our requirement, but the real world is not that simple. Probably the API you want to test depends on a database, or even worse, on external APIs. We need to figure out how we can handle those scenarios.

Docker to the Rescue!

Thanks to Docker and Docker Compose we can run our API and its dependencies all together. I’m not writing this post to explain how docker compose works (there are a lot of great articles out there), so to simplify the example I’m assuming our API do not have any dependencies. Only the API and the Newman services would be needed.

version: '3'
services:
  cicd-postman-tests-api:
    container_name: cicd-postman-tests-api
    image: vad1mo/hello-world-rest
  cicd-postman-tests-newman:
    container_name: cicd-postman-tests-newman
    image: postman/newman
    entrypoint: sh
    command: /src/newman-wrapper.sh
    volumes:
      - ./src:/src
    depends_on:
      - cicd-postman-tests-api

Let me explain the configuration above:

• cicd-postman-tests-api is the API service itself. To reduce the amount of boiler plate code which is not interesting for this tutorial, I’m using the hello-world-rest image which contains a really simple API to interact with.
• cicd-postman-tests-newman is the service to run the tests, which is using the official newman image from the Postman team. Since we have to wait until the API is ready to handle requests, we go with this approach and use a script as a wrapper to wait for it. Note that it’s not enough to use the docker compose option depends-on, since this feature ignores the start up time of your service.

Here it’s the Newman wrapper script:

#!/bin/sh

set -eu pipefail

echo "Attempting to connect to cicd-postman-tests-api"
until $(nc -zv cicd-postman-tests-api 5050); do
    printf '.'
    sleep 5
done
echo "Connected to cicd-postman-tests-api!"

newman run /src/postman-collection.json

exit 0

We try to connect to the service running on port 5050, and wait 5 seconds before trying again. Finally, when the connection was successful, we execute the tests of our collection using the Newman binary.

Done! Now we can run the tests as simple as this:

docker-compose up --exit-code-from cicd-postman-tests-newman

We use the —-exit-code-from option to specify that after the cicd-postman-tests-newman container stops, all the other containers must be stopped too, and the exit code of the command will be the one returned from the newman container. This way, we force the CI/CD pipeline to break whenever a test fails.

But wait, what happen if my API has dependencies? well, you can simply add those dependencies (databases, mock servers, etc) as services in the docker-compose.yml, that’s the main reason why we introduce Docker to the stack. You can tune the wrapper script so it waits for those services too.

Conclusion

Thanks to Postman, Newman and Docker we can build and maintain integration tests that can be executed both in CI/CD pipelines and local environments, allowing the developers to test the full service on their own controlled space, in a faster and easier way.