Identifying Buggy Endpoints

Mayhem for API reports any issues it finds as well as the input that triggers them. In this chapter, we describe the different solutions you can use to see the issues Mayhem for API finds.

Web UI

Mayhem for API uploads findings from each fuzzing run to the ForAllSecure cloud service. Sign in to the UI at https://mayhem4api.forallsecure.com, where you can dig into the details of each run. This includes the details of any issues found, as well as per-endpoint statistics such as response code distribution and latency percentiles. Here's what the web UI looks like:

Web UI

Detailed Report in the Web UI

HTML Reports

Mayhem for API can generate static, local HTML reports using the --html argument to mapi run. These reports contain essentially the same information as their web UI counterparts. They can be saved as "build artifacts" in a CI system. Here's what an HTML report looks like:

HTML Report

Detailed Static HTML Report

JUnit Reports

Mayhem for API can generate industry standard, CI-friendly JUnit reports with the --junit argument to mapi run. These reports contain all of the detailed issues found—but not the additional statistical details from web and HTML reports!—in a machine-readable form, which most CI software can consume and render natively.

SARIF Reports

Mayhem for API can also generate SARIF files with the --sarif argument to mapi run. SARIF files are a great way to upload issues to GitHub, if you happen to use GitHub. See our “GitHub Integration" section for more details.

GitHub

Mayhem for API Issues in GitHub

SARIF files can also be loaded in VSCode using a SARIF extension from Microsoft:

VSCode

Mayhem for API Issues in VSCode

ℹ️ To give accurate file and line information for each issue, Mayhem for API requires your API to include stacktraces in the body of "500 Internal Server Error" responses. Only do that in dev & testing -- never in production!

Debugging tools

Of course, having Mayhem for API find an issue is, in some ways, just the start. Isolating, reproducing and ultimately fixing issues is hard work!

Here are a few approaches we hope will make it easier. (These are all completely independent of Mayhem for API.)

API Logs

The simplest solution is to look through your API logs when a 500 is triggered. Logs often include stack traces pinpointing which line of code triggered the issue. If you have centralized logging infrastructure, with ElasticSearch or Splunk for instance, you'll likely be able to filter the logs by response code to only see 500 responses.

The fuzzer-generated requests will contain the following headers, which can help you look through your logs and correlate log events with specific fuzzing inputs:

  • User-agent: mapi/X.X.X, where X.X.X represents the mapi CLI version.
  • X-Mapi-Program-UUID: <UUID>, where UUID corresponds to a fuzing input. This header allows you to correlate buggy requests from the mapi reports to your API logs.

Error Monitoring Tools

We believe that installing an error monitoring tool in your API is the best way to dig into bugs. They can monitor not only your running APIs, but also any other backend systems like other microservices, background workers, and so on. Those backend systems might error out without the API returning a 500 responses, and error monitoring tools can help you detect those issues triggered but undetected by the fuzzer.

Error monitoring tools detect errors, gather as much context as possible, group similar errors together, and make them available to you through their UI, which looks something like this for sentry.io:

Sentry UI

Plenty of third-party companies offer great error monitoring tools:

Odds are your organization is already using such a tool in production. You can leverage it for fuzzing to get deep insights into the bugs that the fuzzer triggered. Your error monitoring tool will now alert you when the fuzzer finds an issue. It will give you plenty of context like the stack trace to understand the issues. You will be alerted of those issues perhaps even before the bug gets deployed in production!