Find flaky tests and check coverage

Two things quietly make a test suite less useful: flaky tests, which fail now and then for no clear reason so people stop trusting them, and gaps, endpoints or error cases nobody checks. Swastik Codes finds both.

Find flaky tests

1. Open the collection runner and pick the requests.
2. Switch to the Find flaky tests tab.
3. Choose the environment and how many Runs to do (5 to 20 is a good start).
4. Click Run.

The same requests are sent that many times in a row. Each request is then marked:

• Flaky: a test passed some times and failed others, the status code changed between runs, or there was sometimes no response. The reason says exactly what happened, for example "status is 2xx" passed 3 and failed 2 of 5.
• Failing: it failed every time. That is a real bug or a broken test, not flakiness.
• Stable: the same result every time. A latency spike is still mentioned if one run was far slower than the rest.

Common causes of flakiness: tests that depend on the order of a list, on the current time, on data another test created, or on a slow dependency that sometimes times out.

See your test coverage

Open a collection (right-click → Open overview) and go to the Coverage tab.

• Requests with tests: how many requests have their own tests, how many rely only on folder or collection tests, and a clickable list of the ones with none.
• Pick the API definition the collection implements (from APIs) to also see:
  • Endpoints with no request: operations in the definition that nothing in the collection calls;
  • Documented responses never seen: status codes in the definition, such as 401 or 404, that no saved example, status assertion or last run has shown.

The last list is the useful one: APIs break most often on error paths nobody tried. To cover a case, save an example of it or assert it in a test, for example swc.response.to.have.status(404) in a request that asks for an id that does not exist.

Download the summary with Report to track coverage over time.

At the bottom of the tab, Compare with code scans a public GitHub repository for its routes and lists undocumented routes, documented routes the code does not have, routes no request calls and requests for routes that are gone.

What you learned

• Find flaky tests repeats a run and separates flaky, failing and stable requests.
• The Coverage tab shows untested requests, uncalled endpoints and error cases nobody checks.

Next: make sure each kind of user can only do what they should.

Go deeper

• FeatureAPI testing
• LessonCompare responses and environments
• LessonCheck who can do what