Mayhem for API

Develop reliable APIs faster

What is Mayhem for API?

Mayhem for API automates testing REST APIs by bringing the full might of fuzzing methodology to API testing. With the guidance of an API specification, Mayhem for API provides accurate and informative test coverage tailored to any REST API.

How does it work?

Mayhem for API uses a fuzzing engine to automatically generate a comprehensive suite of inputs used to test function and robustness of an application’s API infrastructure. By using fuzzing techniques to generate inputs and observing the response from the application, Mayhem for API can quickly iterate through multitudes of test cases to find weakness in an API’s functionality or security.

Why should I use it?

It is no secret that web APIs have become increasingly important to the operation of modern business. Many business models for new products and services are constructed based on APIs such as billing and identity providers. Trust has become a necessity for APIs. APIs that perform consistently and with high quality earn trust, and those that fail are abandoned. Mayhem for API helps you build trust in your APIs in a number of ways:

  • Resilience - Mayhem for API is great for discovering troublesome 500 Internal Server Error responses and server crashes automatically, before your clients do.
  • Security - With a growing list of Security Checkers, Mayhem for API can discover issues such as Server Side Request Forgery (SSRF) and SQL Injection.
  • Quality - Mayhem for API validates all the responses returned from your API with your specification to identify endpoints which are inconsistent with the spec, such as missing fields or incorrect response codes. Keeping your API synchronized with your specification ensures that API consumers are not caught of guard with unexpected behavior.
  • Performance - Latency statistics of every endpoint are recorded on every run to provide a clear picture of what endpoints have inconsistent or degraded performance.

The Mayhem for API website provides reproduction steps so that you can replay any issues either with curl or with the mapi CLI.

reproduce-curl

Mayhem for API is not a replacement for existing automated tests. It is complementary! With the help of fuzzing, Mayhem for API identifies blind spots in your existing testing, without the bias or the tedium of manually written tests.

Point and Shoot

Setting up the fuzzer is a breeze. Once you have signed up for a free account all you need to do is download the mapi CLI and start testing your API with a compatible specification such as OpenAPI or a Postman Collection.

Web UI

Reach out!

We aim to provide a great experience. Please reach out to us on Discord or by emailing support@forallsecure.com if you have any suggestions or run into any issues. We're more than happy to help!

Let's get started!

Getting Started

Let’s start your API fuzzing journey! There’s a lot to learn, but every journey starts somewhere. In this chapter, we’ll discuss:

  • Installing the Mayhem for API command-line client on Linux, macOS, or Windows.
  • Using the CLI to fuzz our first API
  • Monitoring progress

Signing up

If you haven't done so yet, the fastest way to get started is to sign up for a free plan at https://mayhem4api.forallsecure.com/signup. If you already have an account, then you are ready to go for the next steps!

Installation

The Mayhem for API CLI is available to download for various common platforms.

ℹ️ The CLI will automatically keep itself updated when used as we make fixes and bug improvements.

MacOS

curl -Lo mapi https://mayhem4api.forallsecure.com/downloads/cli/latest/macos/mapi \
  && chmod +x mapi

Here's an easy way to add the mapi executable to your path:

sudo mv mapi /usr/local/bin

You may download this SHA256 checksum or this SHA512 checksum to verify the integrity of the file you downloaded.

Linux (64-bit)

curl -Lo mapi https://mayhem4api.forallsecure.com/downloads/cli/latest/linux-musl/mapi \
  && chmod +x mapi

Here's an easy way to add the mapi executable to your path:

sudo mkdir -p /usr/local/bin/
sudo install mapi /usr/local/bin/

You may download this SHA256 checksum or this SHA512 checksum to verify the integrity of the file you downloaded.

Windows (64-bit)

From a Windows 10 terminal (PowerShell or cmd):

curl.exe -Lo mapi.exe https://mayhem4api.forallsecure.com/downloads/cli/latest/windows-amd64/mapi.exe

or download :

https://mayhem4api.forallsecure.com/downloads/cli/latest/windows-amd64/mapi.exe

You may download this SHA256 checksum or this SHA512 checksum to verify the integrity of the file you downloaded.

Test it out!

Make sure the CLI works by running:

mapi --help

Authentication

The mapi CLI communicates with our API using OAuth 2.0 Bearer Tokens. The token will be read by the environment variable, MAPI_TOKEN, if available.

To get a new token, visit the "Manage API Tokens page" to create a new token, <NEW_TOKEN>, and login:

$ mapi login <NEW_TOKEN>

Welcome to Mayhem for API! We have saved a new API token in
your local settings at '/Users/mapi_fuzzer/Library/Preferences/rs.mapi/mapi.toml':

           3BzW...

Setting the displayed API Token to the environment variable, MAPI_TOKEN, will allow you to run the CLI on other computers, such as part of your Continuous Integration build.

Now you can try contacting the API. Let's get the list of targets to which you have access:

mapi target list

You should see an (empty) list of API targets. Let's add our first target so that list won't stay empty for long.

Hello, Fuzz!

We have prepared an instance of the Swagger Petstore API for you to try fuzzing. The Swagger Petstore is a stand-alone REST API server which implements the OpenAPI 3 Specification.

Fuzz the Swagger Pestore API by running the following in your terminal...

mapi run \
     petstore auto "https://demo-api.mayhem4api.forallsecure.com/api/v3/openapi.json" \
     --url "https://demo-api.mayhem4api.forallsecure.com/api/v3/" \
     --interactive

...once the run completes, visit your dashboard to see the results!

What just happened?

Let's break that command down, to see what happened:

mapi run

Use the CLI to run a fuzzing job.

ℹ️ Check out mapi help if you want to explore some other things the CLI can do!

petstore

Use (and create, if it doesn't already exist) a "target" named petstore for this run. We'll come back to targets in more detail later. For now, a short name that uniquely identifies the API being tested is perfect.

ℹ️ Targets are used to group related fuzzing jobs.

auto

Runs the job until it is decided that enough endpoints have been exercised. You can specify a specific time period here if you wish, such as 30sec to run the job for 30 seconds. The longer the job is run, the more edge cases will be uncovered.

See How long to run for more information.

https://demo-api.mayhem4api.forallsecure.com/api/v3/openapi.json

Tells the fuzzer to base its testing on the OpenAPI 3 specification of the petstore demo API, which is hosted on our site for demo purposes.

--url "https://demo-api.mayhem4api.forallsecure.com/api/v3/"

Tells the fuzzer, which is running locally on your computer, to address its tests to the petstore demo API on our network.

--interactive

Runs the fuzzing job with an interactive, terminal-based user interface. This option should be omitted in a continuous integration environment.

You will see the following UI in interactive mode:

Status UI

The status UI contains a fair amount of information:

  • A progress bar at the top, based on the duration of the job and elapsed time.
  • The job logs. If the job doesn't start properly, or runs into trouble, this is where you should see information on how to resolve the issue.
  • The status codes that the fuzzer observed per endpoint. This will help you determine if the fuzzer is covering the endpoints and status code you expected. For instance, if you see only 401 responses, then that means the fuzzer probably needs help with authentication. We cover this in more details in the “API Authentication chapter".

The UI has a few navigation keys that might be handy:

  • q, ^C or <Esc> to quit
  • <Tab> to switch focus between widgets
  • <Space> to make the current widget full screen
  • Arrow keys for horizontal and vertical scrolling

In your run you should see a number of red endpoints which means that at least one 500 response code is observed. That means we just found some bugs! You should also see a number of green endpoints. These mean that at least one 2xx response has been observed.


You're already off to a great start on your API fuzzing journey! Hopefully, you're thinking to yourself: "this is neat! I wonder how I can use it to test my own APIs?"... because that's the next topic!

Fuzz Your Own API

You've seen what Mayhem for API can do in a demo. Now it's time to fuzz your own!

To start testing an API, you only need to provide two things: a specification describing the API, and a URL where it can be reached. You'll be running something like:

mapi run my-api 30 <specification> --url <url>

API Specification

ℹ️ Specifications can be passed to mapi as either local files or URLs.

Mayhem for API is built around OpenAPI 3 specifications. If you have an OpenAPI spec describing your API, you're all set. Pass either a URL or file system path directly to mapi as the <specification> argument!

If you don't have an OpenAPI spec, we support a few alternatives, described below.

OpenAPI 2.x (sometimes a.k.a. "Swagger")

mapi run will automatically detect and work with older OpenAPI/Swagger specs.

Alternatively, you can do a one-time conversion into OpenAPI 3 by running your old spec through the mapi convert swagger2 command (see mapi help convert swagger2 for details.)

Postman

Mayhem supports Postman collections. Read more about our Postman Collection support here.

HAR

If you don't have any of the above, you'll need to do a little bit more work to generate a spec for Mayhem for API to use. The process is documented in detail over here.

API URL

Mayhem for API needs to know the URL of your API server. Because the requests come directly from the mapi CLI tool running locally on your computer, this will work for APIs that are:

  • Behind a corporate firewall
  • On localhost
  • In a private network
  • On the public internet

If the machine running the CLI can access the API - then we can fuzz it!

Local Fuzz

Proximity and Latency

Although it'll work in just about any configuration, Mayhem for API works better the "closer" it is (in network terms) to the API server. For the best results, this means pointing the fuzzer at a locally-running instance of your server.

Never, Ever, Ever Production

Mayhem for API has one job: finding ways to break an API! You should absolutely not give Mayhem for API the URL of your production services to test against.

Keeping Results Local

Organizations with Enterprise plans can run the mapi CLI in 'local' mode by passing the --local flag to calls to mapi run. This will fuzz your Target API, as usual, with some caveats:

What is required to run Mayhem for API in local mode

  • An active 'Enterprise' plan for the Organization member running the mapi CLI.

What is uploaded to Mayhem for API servers for local runs?

  • Anonymized contributor seat counts
  • API Target placeholder (if a matching target does not exist)

What is NOT uploaded to Mayhem for API servers for local runs?

  • Job details (latency, covered endpoints)
  • Issues (requests sent, responses received)
  • Specifications for conversion (ie: HAR files will not be automatically uploaded and convert to a specification)
  • Custom issue rules from third party integrations, such as OWASP ZAP API Scan rules

How do I get the results of a local run?

All report formats supported by the mapi CLI will continue to work as described in the the chapter about reporting issues, except for 'Web UI' reporting -- because no results are uploaded the Mayhem for API service.

It is the responsibility of the caller to persist the results of local runs.

Files for conversion (no swagger2 → openapi)

All right, hopefully you've got Mayhem for API configured to test your API, congrats!

ℹ️ If we're already uncovering some issues in your API, you might want to take a detour into the chapter about issues, where we'll go into detail about those results.

For most real-world APIs, the next step is going to be telling Mayhem for API how to authenticate, which is what we'll cover next!

HTTP Archive (HAR)

Not all APIs have an OpenAPI/Postman/Swagger specification available. The only specification is the code itself. It is still possible to fuzz an API without a specification by recording transactions with the API as an HTTP Archive (.har file) and converting the recording into an OpenAPI specification. While this enables fuzzing of APIs without specifications, there is a trade-off in the accuracy of the converted specification compared with one generated from the code or maintained by hand.

HAR Files may contain sensitive information

⚠️ HAR files contain everything that was sent between your client/browser and the target API. This may include Authorization headers, cookies, etc. It is not advisable to persist or convert a .har file unless it has been stripped of sensitive information beforehand.

Converting a .HAR to an OpenAPI 3 specification

ℹ️ Requires mapi 2.6.16 or newer

The mapi convert command is used to convert from other formats to an OpenAPI 3 specification.

For example, the following command converts a recording, recording.har into an OpenAPI spec, openapi.yaml:

mapi convert har recording.har --host "127.0.0.1:8080" --base-path "api/v1" --out openapi.yaml

Let"s examine the arguments:

  • --host "127.0.0.1:8080" The conversion will attempt to infer the URL of your API from the recording. If the recording contains requests to multiple URLs, you can specify the --host option to restrict conversion to requests that are only from the specified host.
  • --base-path "api/v1" If your API has a common base path this option will strip the common prefix from all request URLs. This can significantly improve the accuracy of the converted spec.
  • --out opennapi.yaml This will overwrite any existing file, openapi.yaml, with the converted specification. Without this argument, the conversion will be printed to standard out in your terminal.

ℹ️ See all available options with the mapi convert --help command

Upon successful conversion you may inspect the converted file and make any modifications you feel are necessary or start a new run immediately.

Create a .HAR recording with your Browser

If you have a web frontend that interacts with your API, you can record an HTTP Archive by navigating your website and running through various use cases to build a recording of API interactions based on which features you access from the frontend.

Most modern browsers support recording your interactions and exporting a HAR file. See https://toolbox.googleapps.com/apps/har_analyzer/ for browser-specific details.

Create a .HAR recording with a proxy

A web frontend may not make sufficient requests to your API to generate a useful OpenAPI specification. You may have other tools such as a CLI, automated test suites, or a collection of manual curl invocations that can do this. In this case, you can use a proxy to record all of your traffic to produce a HAR file.

The following tools provide support for this capability:

💡 There are many other tools that can help you record your traffic to export as a HAR file. If you see any that should be included here, please let us know: support@forallsecure.com

Postman Collections

Specifying a postman api key and collection id enables Mayhem to read postman collections. You can optionally also specify a postman environment id or file.

mapi run my-api 30 <postman collection id> \
  --url <url> \
  --postman-api-key <postman api key> \
  --postman-environment <postman environment id or file>

Requests that use many of the Postman dynamic variables as path variables, headers, query parameters, or within a request body are handled by Mayhem and mapped to appropriate types during fuzzing.

Exported Postman File

mapi run will also automatically detect if it's given a Postman 2.x collection in place of an OpenAPI spec, and work with that.

Alternatively, you can do a one-time conversion from Postman into an OpenAPI 3 spec by running the collection through the mapi convert postman command (see mapi help convert postman for details.)

Postman Variables and Scripts

Mayhem supports Postman collection and environment variables.

Postman has a scripting engine that is not supported by Mahyem. Typically, these scripts are used to seed requests with values that are returned from another API's response. Mayhem uses API responses automatically during fuzzing.

It may be useful to fork a Postman collection that makes use of scripts, replace scripts with variables where possible, and target the forked collection with Mayhem.

Authenticating With Your API

For most APIs, the next step is setting up authentication. After all, without successfully authenticating, Mayhem for API can only test for very superficial problems! Giving the fuzzer a way to authenticate to the target API will enable it to exercise more endpoints and maximize coverage.

Mayhem for API has built-in support for basic authentication, header-based authentication (such as bearer tokens) and cookie-based authentication. If none of these are sufficient, our rewrite plugin system gives you a powerful option to implement whatever you need for your specific authentication scheme. All of these are described in more detail below. But first, a common gotcha...

Accidental Credential Invalidation

If the credentials you use in fuzzing can be invalidated through a logout endpoint, you will almost certainly need to prevent the fuzzer from issuing requests to that endpoint, using the --ignore-endpoint flag to mapi run, something like this:

mapi run [...] --ignore-endpoint "/api/logout"

ℹ️ For more details on this and other mechanisms for choosing which endpoints to fuzz, see Selective Route Testing.

Basic Authentication

Basic access authentication is a simple technique for enforcing access control to web resources, which is supported by most web servers. You can specify basic authentication credentials when you fuzz your target with mapi run with a command line option:

mapi run --basic-auth "username:password" <target> <duration> <specification>

... or as an environment variable:

export MAPI_BASIC_AUTH="username:password"
mapi run <target> <duration> <spec>

Note that basic authentication does not protect the username and password by itself. They are simply encoded with base64 in transit, but not encrypted or hashed. Basic authentication should be used in conjunction with HTTPS to protect the credentials, or on a trusted network.

Header Authentication (e.g. Bearer Tokens)

Bearer authentication (also called token authentication) is an HTTP authentication scheme that involves security tokens called bearer tokens The client must send this token in the Authorization header when making requests to protected resources:

mapi run --header-auth "Authorization: bearer <token>" <target> <duration> <spec>

In Mayhem for API, the same mechanism is generalized to work with any header-based authentication, for instance:

mapi run --header-auth "X-Custom: auth <token>" <target> <duration> <spec>

You can also use an environment variable to pass such headers:

export MAPI_HEADER_AUTH="Authorization:Bearer <token>"
mapi run <target> <duration> <spec>

Note that the authorization header does not protect the token by itself. It is not encrypted or hashed before sending it to the server. As with basic authentication, this authentication method should be used in conjunction with HTTPS to protect the credentials, or on a trusted network.

ℹ️ To specify custom headers that do not contain credentials, use --header instead of --header-auth. Mayhem for API treats --header-auth differently when probing for issues, and when redacting potentially sensitive data.

Cookie authentication uses HTTP cookies to authenticate client requests and maintain session information. Cookies are generally returned by the server after a successful login, and sent by the clients in subsequent requests. You can specify cookies when you fuzz your target with mapi run with a command line option:

mapi run --cookie-auth "PHPSESSID=abe67cd" <target> <duration> <spec>

... or as an environment variable:

export MAPI_COOKIE_AUTH="PHPSESSID=abe67cd"
mapi run <target> <duration> <spec>

Note that cookies are not encrypted or hashed before being sent to the server. As with basic and header authentication, this authentication method should be used in conjunction with HTTPS to protect the credentials, or on a trusted network.

Authentication Using Rewrite Plugins

For cases where the above built-in methods are insufficient (e.g. if the authentication is dynamic over the course of a fuzzing job), you can use our rewrite plugin system and code your own.

Rewrite plugins aren't authentication-specific and have lots of capabilities documented over here.

Postman Authentication

If the specification provided is a postman collection id, the Postman collection's authentication is used.

The authentication methods currently supported are

Authentication on folders or requests is not currently supported. Specifying authentication with arguments to Mayhem overrides Postman's authentication configuration.

More to come!

We are planning on adding built-in support for more authentication methods soon, and we'd love to hear from you. Please reach out to us on Discord or by emailing support@forallsecure.com to let us know what you'd like to see!


At this point, Mayhem for API should be able to exercise even the authenticated endpoints of your API. Excellent!

ℹ️ If we're already uncovering some issues in your API, you might want to take a detour into the chapter about issues, where we'll go into detail about those results.

We're not quite done, though. Most real-world APIs have other, non-authentication input requirements that are too specific to be reliably found through random fuzzing alone. The next section covers the why and the how of getting Mayhem for API to successfully exercise your entire API.

Successful Coverage

Once authentication is configured, the next step is to ensure that Mayhem for API is able to successfully cover as much of your API as possible.

What is Successful Coverage?

Mayhem for API considers any 2xx response from the API server to indicate success, and a single successful response from a given endpoint is enough for us to consider it successfully covered.

If your API has some other criteria for success: we don't currently support it, but we are interested in this use case, so please reach out to us on Discord or by emailing support@forallsecure.com, and let's see what we can figure out!

Why is Successful Coverage Important?

The absence of findings, for an endpoint which is not successfully covered, is meaningless.

Mayhem for API fuzzing is designed around this idea: that the most interesting test results come from requests that are almost correct. Some of the payloads we generate are purely random—that's important, too!—but once we find a successful payload for a given endpoint, we use that as a starting point for many future requests.

Consider a single POST endpoint which executes an INSERT statement against a relational database on success. If every single request issued by Mayhem for API is invalid, the INSERT statement has never been executed; if that INSERT statement has a SQL injection vulnerability, it won't be found.

If, on the other hand, we pass a valid payload (causing the INSERT statement to be executed) and get a 200 from this endpoint, we'll then use the successful payload as the basis for many future tests. If that INSERT statement has a SQL injection bug, we'll eventually find it.

That's why we emphasize "successful coverage": successfully covering an endpoint massively increases our confidence that the absence of findings is meaningful.

Reasons For Lack of Successful Coverage

Authentication, which is documented separately, is often the first barrier to successful coverage.

Mayhem for API (and fuzzing in general) usually does just fine, without intervention, generating values that satisfy simple data validations—things like "the 'page' parameter must be an integer".

What are other reasons Mayhem for API might be failing to successfully cover an endpoint?

Incorrect URLs

Sometimes the simplest solution is hard to see! If you're getting unexpected failures on endpoints that should be easy to exercise successfully—especially if the failures are 404s!—double-check that Mayhem for API is hitting the correct URLs.

Complex Structural Validation

ℹ️ If you start with thorough OpenAPI specs, you're less likely to run into this problem.

While we'll fairly quickly generate a random value that "must be an integer", it'll take quite a long time to generate a random value that, for example, "must be a JSON document with exactly one key, named 'page', whose value is an integer".

Stateful Validation

The trickiest validations are ones which are based on the current state of the service. For example, a value that "must be the unique id of an existing row in the 'pages' table" might take essentially forever to generate through pure randomness.

Mayhem for API has some tricks up its sleeves (such as trying values from the output of one request in the inputs of another) that help, but it's not perfect.

Whatever the reason, what follows are suggestions for diagnosing and fixing problems in successful coverage.

Diagnosing Problems in Successful Coverage

If you're still here, that means you have at least one endpoint that Mayhem for API can't yet successfully cover. So, how do you begin to figure out why that's happening?

It's worth noting that this may be extremely hard unless you either are (or working closely with someone who is) deeply familiar with the service being tested.

Step One: Focus

We strongly recommend that you dig into one endpoint (or a very small set of interrelated endpoints) at a time, when attempting to get successful coverage, and to keep the runs fairly short (60 seconds or less).

Filter your runs down via the --include-endpoint <endpoint> flag to the mapi run, or other filtering methods documented in more detail here.

Step Two: Logs

When looking at logs, what you're looking for are patterns: what kinds of requests is Mayhem for API making, and why are they failing? What kinds of requests is Mayhem for API not making, and how can it be coerced into making them?

Start by looking at your own service logs, since hopefully you're familiar with those!

If your own logs aren't illuminating enough, you can also capture the full request/response payloads of an entire fuzzing run using the --har <file> flag to mapi run. The output is in JSON HAR format.

Step Three: Reach Out!

Don't hesitate to reach out to us on Discord or by emailing support@forallsecure.com. Between your expertise with your service, and our expertise with Mayhem for API, we should be able to figure it out!

How To Improve Successful Coverage

Correct the URLs

Mayhem for API always constructs URLs by concatenating each endpoint's path with a common server URL.

The paths are taken directly from the API specification.

The server URL is given via the --url <url> argument to mapi run. (There are other ways, but we'll focus on this one, as it is the most explicit!)

Sometimes the specification elides a common prefix from all endpoints. Consider a specification:

  /version:
    get:
     ...
  /user:
    get:
      ...

If the complete URLs are supposed to be https://localhost/api/v2/version and https://localhost/api/v2/user, the common prefix (/api/v2) will need to be included in the server URL given to Mayhem for API:

mapi run ... --url 'https://localhost/api/v2'

Improve The Spec

Often, Mayhem for API just needs a bit more... specificity... in the specification, in order to successfully cover an endpoint.

Provide Schemas

If Mayhem for API is consistently unable to generate structurally valid payloads, this can usually be resolved by adding or refining schemas in the spec.

OpenAPI allows for parameters and request body types to be declared, in whatever level of detail is needed, using Schema Objects.

For example, just listing the required properties for an endpoint's request body makes Mayhem for API massively more likely to successfully cover that endpoint:

  /example:
    post:
      summary: example
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
+             properties:
+               page:
+                 type: integer

Use Consistent Names and Types

Mayhem for API will do a better job generating statefully valid payloads if stateful identifiers that are passed between endpoints have the same name and type.

For example, if you have an endpoint that lists resources and another that fetches a resource by id, make sure the identifiers share a name and type, so that Mayhem for API can more easily connect the dots:

  /example:
    get:
      summary: list of examples
      responses:
        '200':
          description: all the examples
          content:
            application/json:
              schema:
                type: array
                items:
                  type: object
                  properties:
                    example_id:
                      type: integer
- /example/{id}:
+ /example/{example_id}:
    get:
      summary: details of a single example
      parameters:
        - in: path
-         name: id
+         name: example_id
          schema:
-           type: string
+           type: integer

ℹ️ The spec changes you make to help Mayhem for API exercise your API will also improve anything else—code, documentation, and so on—that you derive from your specs! Win-win!

Provide Examples in your Specification

There are situations where Mayhem for API really just needs a hint. For example, consider endpoints that include the username of the currently-authenticated user, like '/user/{username}/settings'.

ℹ️ Examples are often needed when the service being tested has some pre-loaded state (like the user, for authentication!) that's difficult to naturally "discover" through fuzzing.

In this case, OpenAPI allows "example" values, and Mayhem for API will take the examples into account when generating request payloads. For the "{username}" parameterized endpoint, this might be as simple as:

  /user/{username}/settings:
    get:
      summary: settings
      parameters:
        - in: path
          name: username
          schema:
            type: string
+           example: mayhem4api-user

Resource Hints

Mayhem for API may need additional hints to make a successful requests to endpoints. Perhaps every request requires that a constant value is applied to every request body or path parameter. It is possible to provide hints for all generated requests by passing one or more --resource-hint options to the mapi run command.

In order to understand which resources are applicable to the --resource-hint option, you should first list the available resources for your API specification with the mapi describe specfication command.

For example, let's list the resources of the petstore demo API:

mapi describe specification \
  "https://demo-api.mayhem4api.forallsecure.com/api/v3/openapi.json"

Spec path is https://demo-api.mayhem4api.forallsecure.com/api/v3/openapi.json
PUT /pet BODY category/id
PUT /pet BODY category/name
PUT /pet BODY id
PUT /pet BODY name
...
PUT /user/{username} BODY userStatus
PUT /user/{username} BODY username
DELETE /user/{username} PATH username

Every resource for the specification is flattened and listed out in the response. For example, the following represents a field value that is passed as part of a request body when generating a PUT request against /pet.

Method
|
|   Path
|   |
|   |    Request part (can be QUERY, PATH, HEADER or BODY)
v   v    v
PUT /pet BODY category/id
              ^
              Fully qualified path to request body

Another example is a path parameter:

Method
|
|       Path
|       |
|       |               Request part
v       v               v
DELETE /user/{username} PATH username
                             ^
                             Path parameter

The --resource-hint option to mapi run takes a tuple of resource path and value, separated by a colon.

For example, say we wanted to always use the same username value of foo for the DELETE method noted above:

mapi run \
     petstore auto "https://demo-api.mayhem4api.forallsecure.com/api/v3/openapi.json" \
     --url "https://demo-api.mayhem4api.forallsecure.com/api/v3/" \
     --resource-hint "DELETE /user/\{username\} PATH username:foo"

The --resource-hint specifies that any request generated for this DELETE resource will now use foo for the username parameter:

                 Resource Path                          split with ':'
                 v                                      v
--resource-hint "DELETE /user/\{username\} PATH username:foo"
                              ^                          |
                              must escape '{' for        |
                              valid regex                |
                                                         ^ Value to use

--resource-hint will accept a regular expression for matching resource path as well. For instance, say you wanted to substitute foo for EVERY username value:

                         Match ANY request that requires a 'username' 
                         v
--resource-hint "username$:foo"

When multiple --resource-hint are specified that match the same resource path, Mayhem for API will randomly select one of the provided hints whenever it generates a request.

                Choose between 'foo'  ...       or 'bar'
                v                               v
--resource-hint "username$:foo" --resource-hint "username$:bar"

Ignore Endpoints

If, for some reason, your endpoint cannot be covered successfully, consider skipping it during fuzzing (using the --ignore-endpoint <endpoint> argument to mapi run), because we have low confidence in lack of findings.


At this point, Mayhem for API is able to exercise your API quite thoroughly, and probably uncover all sorts of issues... and, more importantly in the long run, give confidence that your API is free of the issues we don't find!

The next section describes optional setup steps to maximize the specificity and detail of the issues Mayhem for API reports. You can do that now, or you can come back to it later.

If you decide to skip that for now, here are some other docs you might be interested in jumping to:

How long should I run?

Unlike other testing methods, like unit and integration tests, the definition for being 'done' for a fuzz test is, well, fuzzy. In general, the longer you run mapi, the more of your endpoints, and the code behind them, will be covered.

Your run duration depends on a few properties of your target API. For example:

  • Latency between CLI and API - local APIs will typically have lower latency
  • API Configuration - is your API configured with multiple threads/workers to handle high concurrency?
  • Number of endpoints - is your API a few or hundreds of endpoints?

The following will help guide you toward the right choice for testing your target API.

Concurrent Requests

Mayhem for API will automatically run in concurrent execution mode as long as you are running on a machine with more than one CPU core. The higher the concurrency, the more requests can be sent in parallel to your target API. Mayhem for API defaults its concurrency level to "4", or the number of CPU cores -- whichever is less.

You can increase the number of parallel fuzzers by passing the --concurrency flag to mapi run command.

For example, to run with 8 parallel threads, pass --concurrency 8 to mapi run:

mapi run \
     <target_name> \
     auto \
     <path_to_api_specification> \
     --url "https://example.com" \
     --concurrency 8

Automatic run duration

We recommend running Mayhem for API with automatic duration if it is your first time using the tool, or aren't sure how long to run your scans. When running with auto duration, mapi will attempt to make a request to every endpoint at least 100 times.

You can run with automatic duration by passing the auto string to the mapi run command. For example:

mapi run \
     <target_name> \
     auto \
     <path_to_api_specification> \
     --url "https://example.com"

When should I use an automatic run duration?

  • It is your first time trying Mayhem for API
  • You want mapi to automatically adjust to your API as it changes size

Tweaking the minimum request count

When running with auto duration, mapi will attempt to make a request to every endpoint at least 100 times.

When running in auto mode, you can change the threshold of the minimum number of endpoint request with the --min-request-count flag. For instance, you may wish to increase the minimum request threshold in order to exercise your API more than the default of 100.

For example:

mapi run \
     <target_name> \
     auto \
     <path_to_api_specification> \
     --min-request-count 500
     --url "https://example.com"

Fixed Time run duration

mapi will continuously scan your target API until the duration has been reached when running with a fixed duration. The duration time is specified as part of the mapi run command.

For example:

mapi run \
     <target_name> \
     5min \
     <path_to_api_specification> \
     --url "https://example.com"

The duration syntax is fairly flexible, allowing callers to specify time in seconds, minutes or hours.

 - Run for 30 seconds                 [30s or 30sec]
 - Run for 90 minutes                 [90m or 90min]
 - Run for 1 hour                     [1h  or 1hr]
 - Run for 2 hours and 20 minutes     [2h20m]

When should I use a fixed time run duration?

A fixed duration scan is ideal if you want to run Mayhem for API, but are limited in your computing time/resources. For instance, if you are running mapi as part of your continuous build pipeline, you may not want a step running that takes an unpredictable amount of time. By setting a fixed duration, you can ensure that mapi you are explicit about how long the mapi step will take.

Configuring Your API For More Detailed Issues

In this section we will examine how to configure specific frameworks to help Mayhem For API find problems, such as possible SQL Injection, that might not be as discoverable with default application settings. These configuration changes are optional and are rarely safe for production; however, they can provide more value out of Mayhem for API when enabled in a safe environment such as part of your CI/CD pipeline.

These changes typically involve adjusting your API configuration to return additional information in the response whenever an internal error occurs, such as the stack trace.

This will help you get to the root cause of buggy endpoints faster and can help the fuzzer identify security threats more accurately such as error-based SQL Injection attacks:

💡 If you do not see your API framework here yet, please let us know: support@forallsecure.com

Spring Boot (Java)

Various properties can be specified inside your application.properties file, inside your application.yml file, or as command line switches. please visit https://docs.spring.io/spring-boot/docs/current/reference/html/application-properties.html for a comprehensive list of properties.

For the following configurations, assume we are working with application.properties file.

⚠️ These settings are appropriate for development testing, but should not be used in production.

Include Stack Trace on Server Error

Update your application.properties file with the following line:

server.error.include-stacktrace=always

Whenever the fuzzer triggers an error, Spring Boot will include a stack trace in the response.

Flask (Python)

When developing a Flask-based API, the built-in Werkzeug development server provides a debugger which shows an interactive traceback in the browser when an unhandled error occurs during a request.

For more information, visit https://flask.palletsprojects.com/en/latest/debugging/.

⚠️ These settings are appropriate for development testing, but should not be used in production.

Include Traceback on Server Error

Either set the FLASK_ENV=development environment variable prior to running your app:

$ export FLASK_ENV=development
$ flask run

Or, include the debug=true flag when running from Python code

from flask import Flask
app = Flask(__name__)

if __name__ == '__main__':    
    app.run(debug=True)

Whenever the fuzzer triggers an error, Flask will include the traceback in the response.

Identifying Buggy Endpoints

Mayhem for API reports any issues it finds as well as the input that triggers them. In this chapter, we describe the different solutions you can use to see the issues Mayhem for API finds.

Web UI

Mayhem for API uploads findings from each fuzzing run to the ForAllSecure cloud service. Sign in to the UI at https://mayhem4api.forallsecure.com, where you can dig into the details of each run. This includes the details of any issues found, as well as per-endpoint statistics such as response code distribution and latency percentiles. Here's what the web UI looks like:

Web UI

Detailed Report in the Web UI

HTML Reports

Mayhem for API can generate static, local HTML reports using the --html argument to mapi run. These reports contain essentially the same information as their web UI counterparts. They can be saved as "build artifacts" in a CI system. Here's what an HTML report looks like:

HTML Report

Detailed Static HTML Report

JUnit Reports

Mayhem for API can generate industry standard, CI-friendly JUnit reports with the --junit argument to mapi run. These reports contain all of the detailed issues found—but not the additional statistical details from web and HTML reports!—in a machine-readable form, which most CI software can consume and render natively.

By default, only Issues with Error level severity will be included in the junit report. If you wish to include Warning level Issues in your report, you must specify the --warnaserror flag when calling mapi run ... --warnaserror --junit <test-output.xml>

SARIF Reports

Mayhem for API can also generate SARIF files with the --sarif argument to mapi run. SARIF files are a great way to upload issues to GitHub, if you happen to use GitHub. See our “GitHub Integration" section for more details.

GitHub

Mayhem for API Issues in GitHub

SARIF files can also be loaded in VSCode using a SARIF extension from Microsoft:

VSCode

Mayhem for API Issues in VSCode

ℹ️ To give accurate file and line information for each issue, Mayhem for API requires your API to include stacktraces in the body of "500 Internal Server Error" responses. Only do that in dev & testing -- never in production!

Debugging tools

Of course, having Mayhem for API find an issue is, in some ways, just the start. Isolating, reproducing and ultimately fixing issues is hard work!

Here are a few approaches we hope will make it easier. (These are all completely independent of Mayhem for API.)

API Logs

The simplest solution is to look through your API logs when a 500 is triggered. Logs often include stack traces pinpointing which line of code triggered the issue. If you have centralized logging infrastructure, with ElasticSearch or Splunk for instance, you'll likely be able to filter the logs by response code to only see 500 responses.

The fuzzer-generated requests will contain the following headers, which can help you look through your logs and correlate log events with specific fuzzing inputs:

  • User-agent: mapi/X.X.X, where X.X.X represents the mapi CLI version.
  • X-Mapi-Program-UUID: <UUID>, where UUID corresponds to a fuzing input. This header allows you to correlate buggy requests from the mapi reports to your API logs.

Error Monitoring Tools

We believe that installing an error monitoring tool in your API is the best way to dig into bugs. They can monitor not only your running APIs, but also any other backend systems like other microservices, background workers, and so on. Those backend systems might error out without the API returning a 500 responses, and error monitoring tools can help you detect those issues triggered but undetected by the fuzzer.

Error monitoring tools detect errors, gather as much context as possible, group similar errors together, and make them available to you through their UI, which looks something like this for sentry.io:

Sentry UI

Plenty of third-party companies offer great error monitoring tools:

Odds are your organization is already using such a tool in production. You can leverage it for fuzzing to get deep insights into the bugs that the fuzzer triggered. Your error monitoring tool will now alert you when the fuzzer finds an issue. It will give you plenty of context like the stack trace to understand the issues. You will be alerted of those issues perhaps even before the bug gets deployed in production!

Issue Rules (Checkers)

Issues discovered by Mayhem for API are categorized by Rule. Each Rule corresponds to one or more checkers that are executed as part of a fuzzing run.

The following is a list of supported Rules. Click a link below to view the details of a specific rule.

Suppressing Issues

Not all issues reported by Mayhem for API may be relevant or of interest to an API maintainer. Mayhem for API offers a mechanism to suppress issues discovered during fuzzing jobs.

The .mapi configuration file

A .mapi configuration file specifies rules for issue suppression:

#
# Example .mapi configuration file
#
version: "1.0"

# An array of rules for suppressing issues.
#
# If an issue matches any rule, it will be suppressed
# from reporting.
#
# Any issue will only match a rule if ALL statements for
# a rule match the issue.

suppressed:
- reason: "Suppress Internal API issues"
  path: /internal/.*
- reason: "Ignore flask debug exceptions"
  summary: flask\.debughelpers\.*

An issue suppress rule is composed of regular expressions which match on different issue fields:

suppressed:
- reason: "(required string) Description of why issues matches this rule are suppressed"
  method: "(optional regex) GET|PUT|POST|DELETE|PATCH... etc. Matches HTTP Method"
  path: "(optional regex) /foo|/tools/.* Matches the HTTP request path"
  rule_id: "(optional regex) internal-server-error|auth-bypass|server-crash... Matches mapi rule id"
  summary: "(optional regex) matches additional issue summary - such as stacktrace message"
  response: "(optional regex) Matches against the entire response, headers and payload. *"

Response matching

If a suppression rule has a response regular expression, that regular expression will be applied against the entire response from the server. This is the raw response seen over the TCP connection which starts with the headers and then proceeds to the payload, if a payload is sent.

You can see something close to the raw output from your server by accessing it with curl -v.

e.g. curl -v https://google.com:

[...]
< location: https://www.google.com/
< content-type: text/html; charset=UTF-8
< date: Thu, 24 Mar 1970 21:04:22 GMT
< expires: Sat, 23 Apr 1970 21:04:22 GMT
< cache-control: public, max-age=2592000
< server: gws
< content-length: 220
< x-xss-protection: 0
< x-frame-options: SAMEORIGIN
< alt-svc: h3=":443"; ma=2592000,h3-29=":443";
<
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/">here</A>.
</BODY></HTML>

To match against a header in the response you would add a response field like:

server: gws

To match against part of the body, you would add a response field like:

The document has moved

Where to put .mapi file?

💡 We recommended checking the .mapi file into source control so that you may track changes to suppressed rules with your SCM tool.

The mapi CLI will resolve the .mapi file with the following order of precedence:

  1. Set with the --config <path-to-file> when calling mapi run
  2. Set with the MAPI_CONFIG_FILE environment variable
  3. Located in the working directory where mapi run is called
  4. Located in the project root (adjacent to the .git folder, if one exists)

How do I see suppressed issues?

mapi will still upload suppressed issues during a run, but they will be hidden by default. In order to see suppressed issues from the CLI, for instance, you will need to supply an extra flag --include-suppressed:

▶ mapi issue list <job-id>> --include-suppressed

 ID     RULE_ID                SUMMARY  METHOD  PATH                 SUPPRESSED  SUPPRESSED_REASON
 35717  internal-server-error           GET     pet/findByTags       true        Ignore all GETs
 35716  internal-server-error           GET     pet/{petId}          true        Ignore all GETs
 35715  internal-server-error           POST    user                 false
 35714  internal-server-error           POST    pet                  false

Selective Route Testing

By default, Mayhem for API will fuzz every endpoint it finds in the provided spec.

If you need to filter down from this full set of endpoints, for example if:

  • you need to exclude a "logout" endpoint because it breaks all subsequent authenticated requests
  • you need to divide your spec into subsets with different authentication requirements
  • you want to incrementally add routes to mapi as the initial findings in each route are addressed

... or whatever other reason, the mapi command-line tool has you covered with a number of options.

Using OpenAPI Tags

If you already have an OpenAPI spec with tags at the right granularity to describe the subsets you want to filter on, you're in luck.

Use the --include-endpoints-by-tag <tag> option to specify that only endpoints with the given tag should be included (implicitly excluding all endpoints which don't have it.) If you specify multiple tags (by including the option more than once), they are logically "OR"ed together.

Use the --ignore-endpoints-by-tag <tag> option to specify that endpoints with the given tag should be explicitly excluded. If you specify multiple tags, they are logically "OR"ed together.

If you combine the two options, --include is applied first, followed by --ignore.

Using Endpoint Names

If you don't have tags at the right granularity, you can use powerful (but somewhat more complicated!) endpoint name matching to choose which endpoints to test.

ℹ️ This feature computes a unique "endpoint name" for each endpoint, which is just the HTTP verb and the path, joined with a space, like GET /ping or POST /users/{id}!

Use the --include-endpoint <pattern> option to specify that only endpoints whose computed name matches the given pattern should be included (implicitly excluding all endpoints which don't match it.) If you specify multiple patterns (by including the option more than once), they are logically "OR"ed together.

Use the --ignore-endpoint <pattern> option to specify that endpoints whose computed name matches the given pattern should be explicitly excluded. If you specify multiple patterns, they are logically "OR"ed together.

If you combine the two options, --include is applied first, followed by --ignore.

ℹ️ The "pattern" used to match endpoint names is a full-blown regular expression, if you need that kind of power.

Zed Attack Proxy (ZAP) Integration

The OWASP Zed Attack Proxy, or ZAP, is an open-source web app scanner, and the de-facto industry standard for automated security scanning of web applications.

ZAP's priorities and approach are quite different from Mayhem for API. We believe they're complementary: if you already run ZAP against your API, you'll get substantially deeper coverage using Mayhem for API, and if you already use Mayhem for API, ZAP is likely to report lots of potential problems, especially configuration and operational problems.

The tools complement each other so well, in fact, that we built a super-easy way to run ZAP's API Scan tool alongside Mayhem for API. Just add a --zap flag to your existing invocation:

mapi run [...] --zap

And you should see ZAP issues reported alongside Mayhem for API issues, like so:

ZAP Results

Read on if you want some of the technical details, or more advanced options, but really: that's it. Go try it!

The Fine Print

ZAP vs. ZAP API Scan

It's worth pointing out that the tool we've integrated with, ZAP API Scan, isn't the full ZAP suite: it is a subset of the ZAP suite which, like Mayhem for API, is driven by an API specification. The full ZAP suite looks at web applications as a whole; ZAP API Scan and Mayhem for API are more narrowly focused on API back-ends.

Docker Required!

Our integration with ZAP relies on a working Docker environment (it's just the easiest way, by far, for us to manage all of the transitive dependencies of the ZAP toolchain.) This dependency is specific to the --zap flag, i.e. it's not a requirement for running Mayhem for API in general.

No OpenAPI 3.1 Support

ZAP hasn't yet implemented OpenAPI 3.1 support.

Any other spec format accepted by Mayhem for API (OpenAPI 2.x, OpenAPI 3.0, Postman and HAR) should work, but if your spec is using OAS 3.1 features, ZAP won't work with it until they update their support.

Only Header Auth Supported

While Mayhem for API supports several authentication methods, only --header-auth is currently propagated correctly to ZAP.

Transformations of the Raw ZAP Results

In order to fit the Mayhem for API data model, we transform and filter the raw ZAP results in a couple of ways.

First, ZAP issues are organized differently: there are alerts (roughly analogous to Mayhem for API "rules") and there are one or more "instances" of each alert. Mayhem for API, being API-oriented, likes to organize issues by endpoint. To map ZAP results into our model, therefore, we create an endpoint-associated issue for each distinct "instance" of each ZAP alert.

Second, in our experience, the lowest-risk ZAP alerts had a terrible signal-to-noise ratio when applied specifically to an API. As a result, by default, we filter these out! You can use the --zap_min_risk_code flag to adjust the default, if it's not what you want.

Why isn't --zap the default?

We're committed to making Mayhem for API, out of the box, a tool API developers are happy to use every day, and happy to add as a gating step in their CI/CD pipelines. This means we work very hard only to report issues which we are certain are genuine and resolveable.

ZAP, on the other hand, reports on everything it finds that might be an issue. Their results—while interesting and useful!—tend to require developer triage, which (usually) makes them unusable as a CI/CD gate.

So: we don't include ZAP results by default, but we make it easy for you to get them periodically.

Ingest your own

If you're already running ZAP, and just want to unify your ZAP and Mayhem for API issue reporting, you can also ingest ZAP result json into a run, instead of having Mayhem for API invoke ZAP for you. This is done with the --zap-import-json-results command-line flag.

Organizations

Organizations are used to share your API Targets and Run results with other users. You may belong to any number of Organizations. As an Organization owner, you can invite others to join you and create service accounts to integrate Mayhem for API into your DevOps pipeline.

When you sign up you will be prompted to create a new Organization. You will be the Owner of this Organization.

Authorization

Mayhem for API implements a form of Role Based Access Control (RBAC).

All resources are scoped to an Organization. API Targets, for example, are assigned to one, and only one Organization. In order to view an API Target, you must belong to the same Organization as that API Target (unless the Target has been marked as "public").

Access Model

Access to Organization resources are controlled with roles.

Owner: Organization Owners have full control over Organizations, Members and API Targets

Member: Organization Members may create API Targets and view Job results, but may not manage other members.

ActionOwnerMember
Organization Settings--
Create organization invitation
Remove member( ✔ can remove self )
Create Service Accounts
Change Member Roles
API Target Settings--
Create Target
Update Target
Delete Target
Change Target Settings
Change Target access
Jobs--
Scan an API
View Jobs
Delete Jobs

You can change the roles of Organization members with the mapi organization set-role command or from your Organization Settings page.

Inviting others to join your Organization

From your Browser

As an organization owner, you can send an email invitation to your organization directly from your browser. Navigate to the Members tab on your Organization page and hit the Invite button to send an invitation to any email recipient that you wish to join your Organization.

Invite Email

From the mapi CLI

Organization Owners can create a new invitation with the mapi organization invite command using the name of their Organization.

Invitations are only valid for one claim and will expire after two days by default. You can extend the number of claims and duration with the --max-claims and expiration-min options respectively. For example:

mapi organization invite --max-claims 5 --expiration-min 2880 my-organization

This will present you a message that you can send to whomever you would like to invite to your Organization via Email/Slack/SMS/etc:

A new invitation has been created for organization <your-organization>>!

Up to 1 user(s) can join the organization by going to 'https://mayhem4api.forallsecure.com/join/uGb79...'.

The invitation link is valid for 1 day.

Anyone that accepts your invite will join with the Member role.

Service Accounts

Organization owners may create service accounts for accessing the API as part of automation to avoid using API Tokens from individual accounts. You may create as many service accounts as you require in your organization.

Service accounts do not count toward developer count for the purpose of billing.

To create a new service account, use the mapi CLI:

mapi organization service-account create <my-organization> <service-account-name>

The service account will be added to your organization - and an API token will be presented once. You must record this token as it will not be shown again.

New Service Account named '<service-account-name>' created! A new API token has been
created.

           SWOJg...

You can use this token in the environment variable, MAPI_TOKEN, to access
the API Fuzzer API as the Service Account. We will not display this value
again.

You may now use the API token presented in the response to access targets and run fuzzing jobs in your Organization.

Removing a Service Account

To delete a service account (and invalidate its API Token) you must remove the account from your organization. Find the ID of your service account by listing organization members:

mapi organization list-members <my-organization>

 ID  Name               Role
  1  foo                Owner
  2  bar                Member
  3  <service-account>  Member   <<<<<

Now remove the service account from your organization:

mapi organization remove-member <my-organization> <service-account-id>

Successfully removed Organization '<my-organization>>' member '3'!

Target Access Control

By default, Targets are only visible to members of their Organization. Organization owners can make targets publicly visible to unauthenticated visitors and members of other organizations by changing Access Control setting of a Target from "Private" to "Public".

Access Control

CI Integration

We recommend running Mayhem for API as part of your CI pipelines. Adding the fuzzer should take only a few minutes. Running the fuzzer as part of your continuous build will automatically check your API with your latest changes before they are merged and deployed.

Downloading the CLI

The first step is to download the CLI. We recommend using the latest url to always get the latest release. We aim to retain backwards compatibility for command line arguments so that your build always works.

Assuming your build is running in Docker, we recommend adding the following lines to add the CLI to your Docker image:

ADD https://mayhem4api.forallsecure.com/downloads/cli/latest/linux-musl/mapi /usr/bin/mapi
RUN chmod 0755 /usr/bin/mapi

This ensures that you get the latest release every time, without affecting the Docker build cache for follow on layers if there hasn't been a new release. This will download the CLI (~25MB) every time you build the image. We recommend adding the line near the end of your build, shortly before you add your code, to minimize the number of uncached layers to rebuild when there is a new CLI release.

Optionally, to avoid re-downloading the CLI every time, we recommend using Docker's BuildKit. BuildKit checks the URL to see if a new version is available, and only downloads it if necessary. This ensures you get the latest release at all times without slowing down your builds with unnecessary downloads. Enable BuildKit with a simple environment variable:

$ DOCKER_BUILDKIT=1 docker build .

Finally, if you happen to use Docker's experimental syntax, you can download the CLI in one command:

ADD --chmod=755 https://mayhem4api.forallsecure.com/downloads/cli/latest/macos/mapi /usr/bin/mapi

Authentication

The CLI requires a token to function. We recommend storing your mapi token in your CI secret stores, and loading it in the MAPI_TOKEN environment variable. The CLI will use the value in that environment variable automatically.

Running the fuzzer

To fuzz your API, you'll have to first start your API locally following our best practices. Once it's up and running, invoke the fuzzer with:

mapi run <target> <duration> <spec-file> --url <url> --junit results.xml
  • <target> is the name of your api target that you created with mapi target create. The target name is often the name of your project.
  • <duration> depends on how long you want your build to take. We recommend auto, but you may specify a fixed duration such as 60sec instead. To maximize fuzzing duration without slowing down the build, we recommend fuzzing in parallel to other build steps, for the duration of those build steps.
  • If your API requires authentication for some endpoints, consider passing it to the fuzzer for best results.

Note that --url and --junit options are optional. If --url is unspecified, it will come from the URL you specified in the API target. --junit is only necessary if you want your CI to parse a JUnit XML to extract the tests and potential failures (see below).

Gathering the results

The CLI will return an error code if any 500 responses are observed. This can be used to fail the build if you so choose. In addition, with the --junit option, the fuzzer will output a JUnit XML file that your CI can parse.

The JUnit will contain a sample request/response for any endpoint that returned a 500, which can be helpful to debug the issue (especially if the API is run in debug mode and the response contains debugging output like stacktraces).

To get a human-friendly report that you can save as a build artifact, use the --html option to get a report like the following:

Api Fuzzer

Ignoring Errors

You may filter out certain types of response errors that you do not want to appear in your results using the CLI. For example to prevent 500 Internal Server Error responses from being marked as issues, include the --ignore-rule InternalServerError flag in your call to mapi run.

The --ignore-rule flag may be used multiple times to ignore more than one type of error. By default, ALL detected response errors will be classified as issues.

mapi run <target> <duration> <url> --ignore-rule InternalServerError

💡 To see a full list of --ignore-rule options, see the output of mapi run --help.

git Configuration

The Mayhem for API CLI tool, mapi, leverages git if ran within a git repository. It detects the author, branch, commit SHA, commit message, and the parent commit SHA. This information is attached to a job and helps Mayhem display useful information about a job.

Default Branch

The default branch is considered the baseline for comparisons and overall target health. Typical default branches are main, master or develop. The default branch is set on the target's settings page.

Jobs ran on the default branch will have a default tag shown on the job's row.

Issues found on the default branch will be replayed on other branches.

Diffs

Mayhem supports two different scenarios for computing the difference between jobs with git context.

Note: To auto-detect diffs, Mayhem needs a non-shallow history.

Default Branch Job

Each default branch job is compared to the last job of the previous commit.

If the previous commit of the default branch does not have a job, because of a pipeline failure or CI configuration, a reason explaining the cause of the missing diff will be shown in place of a number.

For example, a git command to find the previous commit SHA is git rev-parse HEAD~1.

Jenkins

Jenkins can group commits together. Default branch jobs on Jenkins will use the value of the environment variable GIT_PREVIOUS_SUCCESSFUL_COMMIT if it is available. Jenkins publishes the available environment variables at your-jenkins.host/env-vars.html. If GIT_PREVIOUS_SUCCESSFUL_COMMIT is not available, Mayhem will fall back to the logic above.

Branch Job

A job ran on any branch other than the default branch is compared to the last job of the commits merge base with the default branch.

If a merge base commit cannot be found, or a job does not exist for it because of a pipeline failure or CI configuration, a reason explaining the cause of the missing diff will be shown in place of a number.

The example of a git command is git merge-base HEAD origin/main.

Git History

Some CI providers like GitHub Actions fetch a shallow clone of the repository by default. We recommend cloning a full repository if possible. If it is not desirable or possible to clone the full repository, a partial clone of a depth that will capture the merge base is recommended.

If Mayhem cannot detect the parent SHA with the context it is given, it will try to deepen the repository by 50 commits and retry the SHA detection a single time. This requires that fetch access to the remote repository is available to git in the CI step.

Identifying the Previous Job

The previous job that was used as a baseline for a diff can be seen by hovering over the diff stats on the job row.

Previous Job

Overriding the Parent SHA

If a project's git workflow is not compatible with the default behaviour, override the parent SHA by supplying it:

mapi run \
     <target_name> \
     auto \
     <path_to_api_specification> \
     --url "https://example.com" \
     --parent-sha <your_sha>

Troubleshooting

No git branch or commit information

To enable git support for mapi, ensure that it is:

  • Ran within the path of a git repository
  • git is available on the Path

No diffs

Try adding the shell scripts to your CI provider before executing mapi and verify that the output is a SHA:

# this should return the SHA when the default branch is checked out
git rev-parse HEAD~1

# this should return a SHA when 
git merge-base HEAD origin/main

Azure DevOps Pipeline Integration

The Mayhem for API CLI is built to run on any Continuous Integration platform. This guide will describe how to configure an Azure DevOps Pipeline to run Mayhem for API against your API.

Configure Secrets

You will need an API token to run Mayhem for API in your Azure DevOps pipeline:

  1. Create a Service Account token for your organization
  2. Add the newly created token as a "Secret variable" in the pipeline's variables named MAPI_TOKEN

Configure your pipeline

This will demonstrate how to run mAPI against an API that built and run in a Azure DevOps pipeline.

steps:
  - checkout: self
    clean: "true"
    # Fetch a deep enough history to compute the merge base of branches with the
    # default branch.  Alternatively, set to "0" to fetch the entire history.
    fetchDepth: "50"

  # 1. Build and test your API
  
  # 2. Run the API in debug mode. Output stack traces for better error reporting.
  
  # 3. Run Mayhem for API
  - script: |
      curl -Lo mapi https://mayhem4api.forallsecure.com/downloads/cli/latest/linux-musl/mapi \
        && chmod +x mapi

      ./mapi run azure-pipelines-example auto http://localhost:8000/openapi.json \
        --url 'http://localhost:8000' \
        --junit TEST-mapi.xml \
        --html mapi.html
    env:
      MAPI_TOKEN: $(MAPI_TOKEN)
    continueOnError: "true"
    displayName: Run Mayhem for API to check for vulnerabilities

  - task: PublishPipelineArtifact@1
    displayName: 'Publish Pipeline Artifact'
    inputs:
      path: 'mapi.html'
      artifact: mapi-html-report
    continueOnError: "true"

  - task: PublishTestResults@2

GitHub Example

A full working example is available on GitHub.

Circle CI

Add the Mayhem for API CircleCI Orb to your .circleci/config.yml file.

version: 2.1
orbs:
  mapi: forallsecure/mapi@1.0.0

Pipeline Hosted Service

Create a new job to scan your API and start your service. Then call the mapi/scan command to run Mayhem against your service.

jobs:
  mayhem-for-api:
    machine:
      image: ubuntu-2204:2022.07.1
    steps:
      # Start your service
      - run:
          command: start-service.sh &
      # Scan your API with Mayhem for API
      - mapi/scan:
          api-url: "http://localhost:8000"
          api-spec: "https://demo-api.mayhem4api.forallsecure.com/api/v3/openapi.json"
      - store_artifacts:
          path: /tmp/mapi
      - store_test_results:
          path: /tmp/mapi/junit.xml

Then add the new job to your workflow.

workflows:
  tests-and-security:
    jobs:
      - mayhem-for-api

Deployed Service

To test a service that is already deployed and not running as part of the build pipeline, use the mapi/scan job in your workflow. Mayhem should never be executed against production environments.

workflows:
  tests-and-security:
    jobs:
      - mapi/scan:
          api-url: "https://your.staging.com/"
          api-spec: "https://your.staging.com/openapi.json"

GitHub Integration

Mayhem for API comes with a GitHub Action and a GitHub App to help you check every change to your API with Mayhem for API. Our CLI can also upload Mayhem for API results to GitHub Code Scanning from any CI.

GitHub Action

With our GiHub Action, you'll get Mayhem for API testing every API change in no time.

To integrate Mayhem for API into GitHub Actions workflows:

  1. Create a Service Account token for your organization
  2. Add the newly created token to your GitHub secrets.
  3. Create a workflow that starts your API & invoke the Mayhem for API action.

Configure your workflow

At the base directory of your code repository, add a .github/workflows/mapi.yml file to configure GitHub Actions to run Mayhem for API. Your file should look like this.

name: Mayhem for API
on:
  push:
    branches: [ main, master ]
  pull_request:
jobs:
  security:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
        with:
          # fetch a greater number of commits for computing diffs between jobs
          fetch-depth: 50

      - name: Start your API
        run: ./run_your_api.sh &
        #           ^-------------- update this, but keep the '&' to run
        #                           the API in the background.

      - name: Run Mayhem for API to check for vulnerabilities
        uses: ForAllSecure/mapi-action@v1
        with:
          mapi-token: ${{ secrets.MAPI_TOKEN }}
          api-url: http://localhost:8000 # <- update this
          api-spec: your-openapi-spec-or-postman-collection.json # <- update this

This configuration tells GitHub Actions to run a workflow which checks out your code, starts your API, and then runs Mayhem for API against your API. We pass the Mayhem for API token from the GitHub secret, MAPI_TOKEN.

Here's a fully working example repository showing you an ideal integration that you can use as a template. For more details on Mayhem for API Action, visit the Mayem for API Action in the GitHub Actions marketplace.

GitHub Code Scanning

If you are testing a public repository, or a private repository on a GitHub Enterprise plan, we recommend generating a SARIF report. Uploading SARIF reports to GitHub allows you to see any issue found by Mayhem for API in the "Security" tab of your repository. In addition, if your API is sending back stacktraces as part of the error response body, Mayhem for API will show you exactly where in your code the issue happened, right in your PR diffs:

GitHub

Using GitHub action

To do so with our GitHub Action, after starting your API in the workflow, run the Mayhem for API action followed by github/codeql-action/upload-sarif@v1 to upload the report. Note continue-on-error needs to be set to true in the Mayhem for API action in order to run the next step, even if Mayhem for API finds issues.

- name: Run Mayhem for API to check for vulnerabilities
  uses: ForAllSecure/mapi-action@v1
  continue-on-error: true
  with:
    mapi-token: ${{ secrets.MAPI_TOKEN }}
    api-url: http://localhost:8000 # <- update this
    api-spec: your-openapi-spec-or-postman-collection.json # <- update this
    sarif-report: mapi.sarif

# Upload SARIF file (only available on public repos or github enterprise)
- name: Upload SARIF file
  uses: github/codeql-action/upload-sarif@v1
  with:
    sarif_file: mapi.sarif

Using our CLI

To upload our results to GitHub code scanning from anywhere, locally or from a CI, you can use our CLI. You'll need to pass a GitHub token with the security_event scope so that Mayhem for API can upload findings. See GitHub documentation on creating access tokens for more information. Once you have your token, simply run:

mapi run "<target>" "<duration>" "<spec>" --sarif "mapi.sarif" --github-token "<github-token>"

Mayhem for API will attempt to infer the git information from the environment set by the CI, and the git repository on the filesystem. If those are not available, Mayhem for API will ask you to pass the missing information so that it can upload the findings to the right place.

In addition, Mayhem for API also supports GitHub Enterprise on premise! Just pass --github-api-url <your-github-api-url> to mapi run so that Mayhem for API knows where to upload the results. Your GitHub instance needs to be accessible from the host running our CLI, but it does not need to be accessible by our cloud infrastructure.

GitHub Application

Installing the Mayhem for API GitHub App into your Personal or Organization GitHub account provides additional benefits by automatically updating the checks for your builds and pull requests with the result of your API Fuzzer jobs.

Install the GitHub app into your Personal or Organization repository by following the direct link:

https://github.com/apps/mayhem-for-api

You must select which repositories you would like Mayhem for API to have access to. Once you have decided and authorized the app, we will attempt to automatically detect GitHub builds whenever you run the CLI as part of your build. We do this by matching your git repository commit hash/branch/remote with repositories which you have given our app access.

If we cannot determine your repository settings, you may set them manually when calling mapi run with additional options such as --branch, --remote and --sha. See mapi run --help for a full listing.

To see an example of the GitHub App in action, please visit our example repository:

https://github.com/ForAllSecure/api-fuzzer-swagger-petstore-demo

Jenkins Integration

The Mayhem for API CLI or Docker Image is built to run on any Continuous Integration platform. "Jenkins pipelines" are the most common way to build and test your software on Jenkins today. This guide will describe how to configure a Jenkins Pipeline to run Mayhem for API against your API.

Configure Secrets

You will need an API token to run Mayhem for API in your Jenkins pipeline:

  1. Create a Service Account token for your organization
  2. Add the newly created token as a "Secret text" entry in Jenkins Credentials named MAPI_TOKEN

Configure your pipeline

This will demonstrate how to run mAPI against an API that built and run in a Jenkins scripted pipeline.

Run with Docker

In this scripted pipeline, after building and testing your API, a local instance is started and scanned by "Mayhem for API", using the Docker Hub mapi image.

// Run the build on a node with the 'docker' label
node("docker") {
  checkout scm

  // MAPI_TOKEN - The API Token secret text added to Credentials 
  withCredentials([
      string(credentialsId: "${MAPI_TOKEN}", variable: "MAPI_TOKEN")
      ]) {
      
    //  
    // 1. BUILD AND TEST YOUR API HERE
    //
    
    stage("Run Mayhem for API") {
        //
        // 2. Start your API
        //    eg. http://localhost:8080/api
        //

        
        //
        // 3. Run Mayhem for API
        //
        sh '''
          docker run -t --rm \
            --network=host \
            -e MAPI_TOKEN=${MAPI_TOKEN} \
            -e NO_COLOR=true \
            forallsecure/mapi:latest \
            run my-api auto <path_to_openapi_spec> \
            --url 'http://localhost:8080/api' \
            --junit results.xml
        '''

        //
        // 4.  Collect junit results
        //
        junit testResults: 'results.xml'
        
    }
  }
}

Run with CLI

This example is nearly identical as the one above, except we will use the compiled mapi CLI, rather than a Docker container to run the scan.

It is recommended to take advantage of Jenkins Tool Installer support for the most robust setup, but for simplicity we will download the CLI as part of our pipeline.

// Run the build on a node with the 'docker' label
node("docker") {
  checkout scm

  // MAPI_TOKEN - The API Token secret text added to Credentials 
  withCredentials([
      string(credentialsId: "${MAPI_TOKEN}", variable: "MAPI_TOKEN")
      ]) {
      
    //  
    // 1. BUILD AND TEST YOUR API HERE
    //  
    
    stage("Run Mayhem for API") {
        //
        // 2. Start your API
        //    eg. http://localhost:8080/api
        //

        //
        // 3. Download the CLI (or use Jenkins Tools)
        //
        sh '''
         curl -Lo mapi https://mayhem4api.forallsecure.com/downloads/cli/latest/linux-musl/mapi \
           && chmod +x mapi
        '''
        
        //
        // 4. Run Mayhem for API
        //
        sh '''
          mapi run my-api auto <path_to_openapi_spec> \
            --url 'http://localhost:8080/api' \
            --junit results.xml
        '''

        //
        // 5.  Collect junit results
        //
        junit testResults: 'results.xml'
        
    }
  }
}

Best Practices & Troubleshooting

⚠️ Please do not attempt to fuzz any applications that are not owned or managed by you. The fuzz testing will look like an attack at worst and a probe at best to any organization that's not aware of what's happening. Please make sure to only fuzz your API assets. Thanks!

Here are some recommendations to maximize the value of fuzzing for your API:

  • Avoid running fuzzing against production services, as it can be HIGHLY disruptive. It generates a fair amount of load, and might delete resources you do not want deleted. We recommend using it on a QA or staging instance.

  • Populating the service with some realistic data will help the fuzzer uncover more issues.

  • Since the fuzzer may create many new entities in the service under state, resetting the state between runs will prevent the API from being bogged down by previous fuzzing state.

Troubleshooting

CLI Logging

The mapi CLI will print info level logs by default. You can control the logging verbosity in order to get more details about what the CLI os doing with the --verbosity flag.

For example, to log debug output, include the --verbosity debug flag before your subcommand:

     Must come BEFORE subcommand
     v
mapi --verbosity debug run ...
                       ^
                       subcommand

You can also use the MAPI_LOG environment variable for the same effect.

MAPI_LOG=debug mapi run ...

Logging levels (not case sensitive) include:

off
critical
error
warn
info
debug
trace

Logging with trace level includes trace, debug, info, and higher.
Logging with debug level includes debug, info, and higher.
... etc

Jobs

Looking at the job status using the CLI can help you understand what the fuzzer is doing, and if it's running into any issue.

  • The job logs will contain errors, if any. We aim to provide actionable feedback, making it obvious how to resolve the issue. For instance, the job log might contain messages about not being able to connect to the target API.

  • The endpoint coverage table can tell you a lot about what the fuzzer is able to exercise. Ideally you'd see all endpoints having 2xx response codes. If you don't, this means that the fuzzer could not generate a valid request. If you only see 401 unauthorized responses, then the fuzzer is having trouble with authentication. If you only see 400 response codes, that means the fuzzer could not generate a valid input for this endpoint, maybe because it was too complex to be generated by our fuzzer, or maybe because the specification did not match the behavior. If you're not sure why the fuzzer fails to generate 2xx responses, please reach out and we'll look into it and make the fuzzer better.

  • Sometimes, the job log will contain an error due to something on our end. Please feel free to reach out to support@forallsecure.com if you run into one.

  • If you have a Swagger 2.0 specification, we will attempt to convert it to OpenAPI 3 for you. If this fails, you can also try using the swagger2openapi command line tool:

    npm install -g swagger2openapi
    swagger2openapi -p -y swagger-spec.json > openapi-spec.yaml
    
  • Consider specifying a Postman API key and collection ID instead of an exported Postman file for better fuzzing and more features. If you are using an exported Postman 2.x specification, we will attempt to convert it to OpenAPI 3 for you. If this fails, you can also try using postman-to-openapi.

Tracing Issues

Issues can be traced back to the specification. OpenAPI operationId, summary and tags are attached to issues. Likewise, Postman request ID, request name and folder are attached to issues.

issue-specification-details

In addition, please feel free to reach out to us on Discord or to support@forallsecure.com if you have any issues, feedback of feature requests. We are happy to help!

HTTP Proxy Configuration

If your target is behind a proxy, or you require a proxy to reach the API for Mayhem for API, you can provide proxy information for the mapi CLI. Support for common proxy environment variables follows:

Environment VariableDescription
HTTP_PROXYThe address to use when initiating HTTP connection(s)
HTTPS_PROXYThe address to use when initiating HTTPS connection(s)
NO_PROXYNetwork address(es)/range(s) and domains to exclude from proxying

Examples

Say you wish to capture all the communication between the fuzzer and your API. You can place a proxy between Mayhem for API and your API such as Charles Proxy (which listens on localhost:8888 by default) to intercept and record all traffic.

HTTP_PROXY=http://localhost:8888 mapi run <target> ... 

If your target is an https, you can use the HTTPS_PROXY variable:

HTTPS_PROXY=https://localhost:8888 mapi run <target> ... 

If your proxy server requires authentication with username and password, you must include the credentials:

https_proxy=http://username:password@localhost:8888 mapi run <target> ...

For SOCKS5 proxy servers, you must specify the socks5 protocol in the URL:

https_proxy=socks5://socksproxy.example.com

Proxying https traffic may result in some certificate errors when the CLI attempts to communicate with the API for Mayhem for API. This can be resolved by ignoring certain addresses from being sent to the proxy with the NO_PROXY variable. For example:

HTTPS_PROXY=https://localhost:8888 \
 NO_PROXY=mayhem4api.forallsecure.com \
 mapi run <target> ... 

The 'Mayhem for API' API

The Mayhem for API is available in OpenApi 3.0 format and can be interacted with using the online Swagger Editor.

Please note: this is an unstable, pre-release API under active development. If you want to know more, or if you want to build against it, we'd like to hear from you! Please get in touch by emailing us at support@forallsecure.com.

Authentication

An API Token is required to access the 'Mayhem for API' API. The token must be presented as a Bearer token in the Authorization header with every request in order to access your resources.

For example, to fetch a list of API Targets directly from the API with curl with your API Token <YOUR_TOKEN>:

curl --request GET 'https://mayhem4api.forallsecure.com/api/v1/api-target' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer <YOUR_TOKEN>'

gRPC Fuzzing

While Mayhem for API does not currently support native fuzzing of gRPC servers, it does support gRPC servers fuzzing through the use of a gRPC gateway. The gateway comes with a protoc plugin. It reads a gRPC service definition and generates a reverse-proxy server which translates a RESTful JSON API into gRPC.

gRPC Proxy diagram

This guide will walk you through the steps to fuzz a gRPC server. While this method can be used with any language, the guide assumes you're using go. If you're not and are having trouble following the steps for another language, let us know, we'll be happy to help out.

Live Example

A live example that follows this process is available in GitHub:

https://github.com/ForAllSecure/mapi-grpc-example

Installing gRPC-Gateway

Follow the instructions in the README. You'll need to have go and protoc installed.

Generate Server Stub

The following command generates the proxy stubs from your .proto file:

protoc -I . --grpc-gateway_out . \
    --grpc-gateway_opt logtostderr=true \
    --grpc-gateway_opt paths=source_relative \
    --grpc-gateway_opt generate_unbound_methods=true \
    your/service/v1/your_service.proto

The command above generates the gateway stub at your/service/v1/your_service.pb.gw.go. We will import the stubs to write the proxy.

Write an entrypoint for the proxy

We will now write the proxy in rest-proxy.go. The proxy will listen on port 8081 and forward requests to your gRPC server.

package main

import (
  "context"
  "flag"
  "net/http"

  "github.com/golang/glog"
  "github.com/grpc-ecosystem/grpc-gateway/v2/runtime"
  "google.golang.org/grpc"

  gw "github.com/your-org/your-project/your/service/v1"
)

var (
  // command-line options:
  // gRPC server endpoint
  grpcServerEndpoint = flag.String("grpc-server-endpoint",  "localhost:10990", "gRPC server endpoint")
)

func run() error {
  ctx := context.Background()
  ctx, cancel := context.WithCancel(ctx)
  defer cancel()

  // Register gRPC server endpoint
  // Note: Make sure the gRPC server is running properly and accessible
  mux := runtime.NewServeMux()
  opts := []grpc.DialOption{grpc.WithInsecure()}
  err := gw.RegisterYourServiceHandlerFromEndpoint(ctx, mux,  *grpcServerEndpoint, opts)
  if err != nil {
    return err
  }

  // Start HTTP server (and proxy calls to gRPC server endpoint)
  return http.ListenAndServe(":8081", mux)
}

func main() {
  flag.Parse()
  defer glog.Flush()

  if err := run(); err != nil {
    glog.Fatal(err)
  }
}

You will need to change three things:

  • The import line "github.com/your-org/your-project/your/service/v1" to match your project and the path to the stub you generated in the previous step.
  • The url to your gRPC server. The code above assumes your gRPC server is listening on localhost:10990.
  • The call to gw.RegisterYourServiceHandlerFromEndpoint to match your service name. The function should be present in the gateway stub you generated earlier if you're not sure what the name should be.

Generate the OpenAPI Specification

The following command generates the OpenAPI specification:

protoc -I . --openapiv2_out . \
    --openapiv2_opt logtostderr=true \
    --openapiv2_opt generate_unbound_methods=true \
    your/service/v1/your_service.proto

Fuzz

  1. Start your gRPC server
  2. Start the proxy with go run rest-proxy.go
  3. Create a target: mapi target create target-name http://localhost:8081
  4. Fuzz: mapi run target-name 120 your/service/v1/your_service.openapi.json

Request Rewrite Plugins

Mayhem for API has support for request rewriting via plugins, which are standalone programs that implement a protobuf interface and expose it via gRPC.

The interface details are below. But first...

Do I Need This?

We've tried to make Mayhem for API work out of the box for as many use-cases as we can, and we'll keep adding more. The request rewrite plugin is intended as a kind of escape-hatch for situations where an API has requirements that we can't currently meet out of the box.

Here are the cases we had in mind when building the feature:

Just In Time Authentication

In any situation where the authentication needs to change during a fuzzing job, we don't currently have a solution directly in Mayhem for API.

A request rewrite plugin can inject the appropriate credentials at whatever cadence your API requires.

Request Digesting/Signing

If your API uses any kind of digesting or signing as authentication measures, Mayhem for API will be unable to generate valid requests.

A request rewrite plugin can compute and inject these per-request checksums or signatures.

Something else?

The request plugin interface is extremely open-ended, and can rewrite just about any part of any request.

If you're thinking about using a rewrite plugin for any reason, please get in touch. This is a very young, very raw feature, and we'd like to know how people are using it, and how we can make it better!

Interface

Invocation

To add a request rewrite plugin to a fuzzing run, use the --rewrite-plugin <path-to-plugin-executable> argument to mapi run.

Executable gRPC Server

mapi will run the executable with no command-line arguments, and expects it to be a long-lived process that:

  • listens for gRPC traffic on a localhost port
  • prints the port number, followed by a newline, to stdout as the first line

Beyond that, it can do whatever it wants. Additional stdout/stderr output from the plugin are forwarded to the mapi debug logs.

At the end of the fuzzing run, mapi will attempt to kill the plugin process.

Protobuf

Request rewrite plugins implement the following protobuf interface:

syntax = "proto3";

package mapi.rewrite;

service RewritePlugin {
   rpc Rewrite(Request) returns (Request);
}

message Request {
   message Header {
      bytes name = 1;
      bytes value = 2;
   }
   string url = 1;
   repeated Header headers = 2;
   bytes body = 3;
   string http_method = 4;
}

Example

Here's an example request rewrite plugin, which does... absolutely nothing, just returning the original request unchanged!

First, generate all of the gRPC and protobuf boilerplate from the .proto above:

$ pip install grpcio-tools
$ python -m grpc_tools.protoc --python_out=. --grpc_python_out=. -I. request-rewrite-plugin.proto

Then create a python program which implements the RewritePluginServicer, stands up a gRPC server, and lets mapi know what port it's listening on:

#!/usr/bin/env python

import sys
from concurrent import futures
import grpc
import request_rewrite_plugin_pb2
import request_rewrite_plugin_pb2_grpc

## implement the RewritePlugin interface
class RewritePluginServicer(request_rewrite_plugin_pb2_grpc.RewritePluginServicer):
    def Rewrite(self, request, context):
        ## well, this isn't very interesting :)
        return request

if __name__ == '__main__':
    ## boot up the gRPC server
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
    request_rewrite_plugin_pb2_grpc.add_RewritePluginServicer_to_server(
        RewritePluginServicer(), server)
    server.add_insecure_port('127.0.0.1:50051')
    server.start()
    
    ## inform mapi of the port we're listening on
    print("50051")
    sys.stdout.flush()

    server.wait_for_termination()

Make sure your .py file is executable, then pass it to mapi via --rewrite-plugin. That's it!

More Examples

Please visit our GitHub Repository for more examples that you can try yourself!

Caveats and Cautions

bytes, Not string!

Mayhem for API requests are not always representable as strings. Be careful!

Subject to Change

All of this is volatile and subject to change. If you're using this feature, please get in touch, so that we can keep you in the loop as we consider refinements and changes.

Response Classify Plugins

Mayhem for API has support for response classification via plugins, which are standalone programs that implement a protobuf interface and expose it via gRPC.

The interface details are below. But first...

Do I Need This?

We've tried to make Mayhem for API work out of the box for as many use-cases as we can, and we'll keep adding more. These plugins are intended as a kind of escape-hatch for situations where an API has requirements that we can't currently meet out of the box.

Here are the cases we had in mind when building the feature:

Sensitive Data Leaking

If there are certain keys, values or patterns that you want to assert should never show up in your API output, regardless of inputs, Mayhem for API doesn't currently have a direct solution.

A response classify plugin can make these assertions.

Nonstandard Response Code Assertions

The rules Mayhem for API follows based on each endpoint's response codes are baked in, and based on standards. There may be specific endpoints where you want to apply more strict constraints—for example, an endpoint whose contract is that it never returns a 400 error.

A response classify plugin can be used to check for this kind of constraint.

Something else?

The classifier plugin interface is extremely open-ended, and can be used to check just about any aspect of a response.

If you're thinking about using a classify plugin for any reason, please get in touch. This is a very young, very raw feature, and we'd like to know how people are using it, and how we can make it better!

Interface

Invocation

To add a response classify plugin to a fuzzing run, use the --classify-plugin <path-to-plugin-executable> argument to mapi run.

Executable gRPC Server

mapi will run the executable with no command-line arguments, and expects it to be a long-lived process that:

  • listens for gRPC traffic on a localhost port
  • prints the port number, followed by a newline, to stdout as the first line

Beyond that, it can do whatever it wants. Additional stdout/stderr output from the plugin are forwarded to the mapi debug logs.

At the end of the fuzzing run, mapi will attempt to kill the plugin process.

Protobuf

Response classify plugins implement the following protobuf interface:

syntax = "proto3";

package mapi.classify;

service ClassifyPlugin {
   rpc Classify(Response) returns (Issues);
}

message Response {
   message Header {
      bytes name = 1;
      bytes value = 2;
   }
   string request_url = 1;
   string final_url = 2;
   uint32 status = 3;
   repeated Header headers = 4;
   bytes body = 5;
}

message Issues {
   message Issue {
      string summary = 1;
   }
   repeated Issue issues = 1;
}

Example

Here's an example classifier plugin, which does... absolutely nothing, just acknowledging each response without flagging any issues.

First, generate all of the gRPC and protobuf boilerplate from the .proto above:

$ pip install grpcio-tools
$ python -m grpc_tools.protoc --python_out=. --grpc_python_out=. -I. response-classify-plugin.proto

Then create a python program which implements the ClassifyPluginServicer, stands up a gRPC server, and lets mapi know what port it's listening on:

#!/usr/bin/env python

import sys
from concurrent import futures
import grpc
import response_classify_plugin_pb2
import response_classify_plugin_pb2_grpc

## implement the ClassifyPlugin interface
class ClassifyPluginServicer(response_classify_plugin_pb2_grpc.ClassifyPluginServicer):
    def Classify(self, response, context):
        issues = response_classify_plugin_pb2.Issues()
        if something_went_wrong():
            issue = response_classify_plugin_pb2.Issues.Issue()
            issue.summary = "something went wrong!"
            issues.issues.append(issue)
        return issues

if __name__ == '__main__':
    ## boot up the gRPC server
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
    response_classify_plugin_pb2_grpc.add_ClassifyPluginServicer_to_server(
        ClassifyPluginServicer(), server)
    server.add_insecure_port('127.0.0.1:50051')
    server.start()

    ## inform mapi of the port we're listening on
    print("50051")
    sys.stdout.flush()

    server.wait_for_termination()

Make sure your .py file is executable, then pass it to mapi via --classify-plugin. That's it!

Caveats and Cautions

bytes, Not string!

API responses are not always representable as strings. Be careful!

Subject to Change

All of this is volatile and subject to change. If you're using this feature, please get in touch, so that we can keep you in the loop as we consider refinements and changes.

Open Source Notices

Mayhem For API ❤️ open-source software! We rely on a number of open-source dependencies, and we are grateful for the work all the maintainers are putting into them. To give back, we regularly upstream any improvements we make.

Here are the copyright notices of our CLI dependencies:

Here are the copyright notices of our Frontend dependencies:

Third-Party Tool Support

OWASP ZAP

Mayhem for API includes support for running and importing results from OWASP ZAP via ZAP - API Scan

The OWASP® Foundation works to improve the security of software through its community-led open source software projects, hundreds of chapters worldwide, tens of thousands of members, and by hosting local and global conferences.

OWASP ZAP is licensed under the Apache 2 License

Release Notes

Supported Versions

2.19.30 (Sep 18, 2023)

  • CLI - Fixed an issue during a run which could fail to pull issues from a
    previous run when an Organization name does not match its slug.

2.19.29 (Sep 7, 2023)

  • CLI - Addressed some false positives with verb tampering checker when following redirects (301 or 302) where the verb would change to GET regardless of the initial request.

2.19.28 (August 30, 2023)

  • CLI - Address an issue where too many targets or issues could be fetched when paging.

2.19.27 (August 8, 2023)

  • CLI - Update docker image to use new https://app.mayhem.security backend.

2.19.26 (July 25, 2023)

  • CLI - Fixes an issue where Zap was crashing. Updates the Zap docker image to w2023-07-24.

2.19.25 (July 20, 2023)

  • CLI - Fixes an issue where invalid query parameters were not reported to the platform - such as command injection strings that contain a &.

2.19.24 (July 14, 2023)

  • CLI - More permissive HAR parsing.

2.19.23 (July 12, 2023)

  • CLI - insecure-mayhem flag applies in more situations.

2.19.22 (July 7, 2023)

  • CLI - Introduced a flag insecure-mayhem to the mapi CLI that permits access to Mayhem APIs over https that are running on premises without valid SSL/TLS certificates in place.

2.19.21 (June 29, 2023)

  • CLI - Improved support for more Python Traceback variants

2.19.20 (June 22, 2023)

  • CLI - Dependency updates

2.19.19 (June 22, 2023)

  • CLI - Fixed an issue when running mapi convert postman which would print debug output to stdout.

2.19.18 (June 14, 2023)

  • CLI - Suppress a false error message.

2.19.17 (June 12, 2023)

  • CLI - Resolved an issue where a certain sequence of mapi login commands with an existing pre-2.19.16 configuration file could generate a malformed user config file.

2.19.16 (June 2, 2023)

  • CLI - Backend interaction fixes.

2.19.15 (May 24, 2023)

  • CLI - Resolved an issue where issues would sometimes be replayed from unrleated public targets on the first scan of an API.

2.19.14 (May 19, 2023)

  • CLI - Resolved a false positive issue with the 'Invalid Response Spec' classifier.

2.19.13 (May 10, 2023)

  • CLI - Increase error message detail.

2.19.12 (May 8, 2023)

  • CLI - Backend interaction fixes.

2.19.11 (May 5, 2023)

  • CLI - Backend interaction fixes.

2.19.9 (May 1, 2023)

  • CLI - Fixed an issue where a stackoverflow could occur when scanning an API with circular references ($ref) in the OpenAPI schema.

2.19.8 (Apr 26, 2023)

  • CLI - Fixed authentication issue when run inside Github Action.

2.19.5 (Apr 25, 2023)

  • CLI - Improved logic used to generate non-unicode strings for regex values.

2.19.4 (Apr 20, 2023)

  • CLI - Security fixes

2.19.3 (Apr 20, 2023)

  • CLI - Backend interaction fixes.

2.19.2 (Apr 6, 2023)

  • CLI - Improvements to how mapi run inserts GitHub comment.

2.19.1 (Apr 2, 2023)

  • CLI - Fixed an issue with job status display if connection is lost to the Mayhem API.

2.19.0 (Mar 30, 2023)

  • CLI - Pass HTTP method (GET/POST/...) on the Request Rewrite Plugin interface to enable users to view and manipulate the HTTP method of a generated request.

2.18.5 (Mar 22, 2023)

  • CLI - Updated documentation in mapi --help

2.18.4 (Mar 15, 2023)

  • CLI - Resolved an issue where resource ID reuse was not working for some OpenAPI specifications that included multiple possible schemas in response content specification.
    • This was preventing the fuzzer from accurately covering endpoints that use IDs (or other values) returned by other requests.
    • An example of endpoints affected by this includes those that make use of anyOf or a combination of a schema AND example in the response content specification.

2.18.3 (Mar 15, 2023)

  • CLI - Performance improvements

2.18.2 (Mar 14, 2023)

  • CLI - Ensure Linux musl builds are running the latest CLI.

2.18.1 (Mar 14, 2023)

  • CLI - Fixed an issue where Results link is not always displayed at the end of a run.

2.18.0 (Mar 7, 2023)

  • CLI - Include more rule checkers with run option --experimental-rules

2.17.3 (Mar 6, 2023)

  • CLI - New option for mapi run, --max-response-size. APIs with endpoints that return very large documents can lead to degraded performance and out of memory issues while fuzzing. This setting configures the threshold for which responses will be ignored if they exceed this maximum size. Setting this too high can result in out of memory issues, while setting it too low may lead to less test coverage against your API. The default value is 100KB. Values can be configured as 100B, 500KB.

2.17.2 (Feb 28, 2023)

  • CLI - Issues from previous runs will no longer be replayed if they were suppressed. See Suppressing Issues for more details on issue suppression.

2.17.1 (Feb 17, 2023)

  • CLI - Cancelling a mapi run in progress (CTRL+C) will now cancel issue replay at the end of the run as well.

2.17.0 (Feb 15, 2023)

  • CLI - Support for limiting mapi memory usage with option --max-memory-usage

2.16.5 (Feb 2, 2023)

  • CLI - Improved support for $ref JSON References when working with OpenAPI specifications. This should reduce the number of warnings displayed when a $ref is made to a resource that is not under #/components/* (such as #/paths/*.

2.16.4 (Jan 30, 2023)

  • CLI - fix issue matching content-type header value against Open API specification

2.16.3 (Jan 19, 2023)

  • CLI - fixed an issue where issues from multiple previous jobs would replay at the end of a run, rather than one previous job.

2.16.2 (Jan 16, 2023)

  • Authentication Bypass checker will no longer be triggered when Authorization header is set using a Request Rewrite plugin.

2.16.1 (Jan 3, 2023)

  • CLI - Added support to keep run results local with --local flag passed to mapi run (mapi run --local ...). Local results are limited to Organizations with an active 'Enterprise' plan.

2.15.11 (Dec 12, 2022)

  • CLI - Response values used for request generation are no longer case-sensitive. For example, a response containing a field named id will now be considered for a request field named ID.

2.15.10 (Dec 6, 2022)

  • CLI
    • Issue replay now replays issues from the default branch of the target.
      • If a parent commit sha is available, issues found in the last job matching the sha are used.
      • If no parent commit sha is available, issues found in last job for the default branch are used.
    • Git commit author, author email, and subject are now detected and used for better identification of jobs.

2.15.9 (Nov 23, 2022)

  • CI/CD support for job diffs based on git in a number of CI systems. Default branches have recently been added to target settings and are required to populate diffs in new jobs.

2.15.8 (Nov 14, 2022)

2.15.7 (Nov 10, 2022)

  • New CI/CD documentation along with native support for Circle CI with our Circle CI Orb!

  • We are rolling out changes to help you understand the health of your APIs over time. With new support for "Default Branch" (accessible from your Target Settings page), you will be able to see how your changes are impacting your API against your default branch when you run Mayhem for API in continuous build and deploy pipelines (CI/CD).

  • We are making lots of changes to our website to improve the user experience. Our goal is to make it easier to find and understand your Mayhem for API results.

  • CLI - Added support to override the timeout (default of 5 seconds) to connect and communicate with your API targets with mapi run --timeout...

  • CLI - Support for fuzzing endpoints with content type application/octet-stream.

  • CLI - We are increasing the variety of data we send to your APIs by complimenting fully random inputs with locale-specific fake data with the help of the fake-rs library. The motivation here is to make more successful requests against your APIs so that we can go deeper and find harder-to-reach bugs and security issues.

2.15.6 (Oct 27, 2022)

  • CLI - Improved support for more regular expression formats in your specification.

2.15.5 (Oct 26, 2022)

  • CLI - Support for Postman environment files. Added checksums on CLI download page.

2.15.4 (Oct 13, 2022)

  • CLI - Upgraded to the latest JSON Schema document for validating OpenAPI specifications: https://spec.openapis.org/oas/3.1/schema/2022-10-07

2.15.3 (Oct 13, 2022)

  • CLI - Parsing improvements for java stacktraces - get more issue details for APIs that include Java stacktraces in the response.

2.15.2 (Oct 7, 2022)

  • CLI - Fixed an issue where statistics stop updating when running mapi run with the --interactive flag.

2.15.1 (Oct 5, 2022)

2.15.0 (Oct 3, 2022)

2.14.2 (Sep 29, 2022)

  • CLI - Fixed some problems with issue replay and display.

2.14.0 (Sep 5, 2022)

  • The OWASP Zed Attack Proxy, or ZAP, is an open-source web app scanner, and the de-facto industry standard for automated security scanning of web applications. While ZAP takes a different approach to API security testing, we believe that it is complementary to Mayhem for API. We have built a super-easy way to run ZAP's API Scan tool alongside Mayhem for API. Just add a --zap flag to your existing mapi run invocation to get started!

    See Zed Attack Proxy (ZAP) Integration for more details.

2.13.4 (Aug 8, 2022)

  • CLI - Fixed Server Crash false positives seen primarily when servers close connections quickly after the first response, but before a second request is sent. This false positive is seen often when running against FastAPI based APIs where this behavior occurs whenever
    an unhandled exception is triggered. Note that this can happen on with other API frameworks as well and is not limited to FastAPI.

2.13.2 (Aug 2, 2022)

2.13.1 (Jul 26, 2022)

  • CLI - Improvements to SARIF report
    • run.invocations will now be included in SARIF output
    • version property will now appear in the correct order in SARIF output as per the SARIF specification:

    Although the order in which properties appear in a JSON object value is not semantically significant, the version property SHOULD appear first.

    https://docs.oasis-open.org/sarif/sarif/v2.0/csprd02/sarif-v2.0-csprd02.html#_Toc10127671).

2.13.0 (Jul 26, 2022)

  • New Issue: Invalid Request Spec
    • If Mayhem is able to elicit a successful response from an endpoint using a payload that's invalid according to that endpoint's spec, it will now raise an issue. Much like Invalid Response Spec and Verb Tampering, these may indicate simple specification problems, but can sometimes indicate serious underlying problems.
  • New Issue: PII Disclosure
    • If Mayhem detects anything that appears to be PII in response payloads, it will raise an issue. For now, this is limited to credit card numbers.

2.12.0 (Jul 7, 2022)

  • New Issue: Verb Tampering
    • Mayhem will make requests to endpoints with HTTP Verbs (ie: GET, POST, PATCH...) that are invalid, or not part of the specification. If the API responds successfully, then a Verb Tampering issue will be raised. This may simply be an issue with the specification, or it could be indicative of a security-related misconfiguration on the API server.

2.11.3 (Jun 28, 2022)

  • CLI - Reduced logging noise significantly when running mapi run at info level logging (the default -- configurable with the MAPI_LOG environment variable).
  • Updated fuzzer so that checkers will run more frequently when running with more than 2 workers with mapi run and --concurrency argument of 2 or higher.

2.11.1 (Jun 21, 2022)

  • CLI - Fix a regression in OpenAPI 3.0 support

2.11.0 (Jun 20, 2022)

  • CLI - Enable HAR uploads up to 200 MB in size

2.10.0 (Jun 14, 2022)

  • CLI -Support for OpenAPI 3.1 specifications
  • CLI -auto run duration support to mapi run. mapi will automatically determine how long to fuzz a target. It is still possible to specify a duration.
  • CLI - Improved support for multipart/form-data endpoints

2.9.9 (Jun 6, 2022)

  • CLI -Fix service account authentication.

2.9.8 (Jun 3, 2022)

  • CLI -Performance enhancements for CLI talking to MAPI API.

2.9.7 (May 17, 2022)

  • CLI -mapi-action posts a run report to the pull request commment thread.

2.9.6 (May 16, 2022)

  • CLI -mapi issue detail will now include issue stacktrace, if one was captured during a run
  • CLI -Added --verbosity flag to mapi CLI for improved control of logging verbosity
  • CLI -Improved validation when converting specifications (ie: Postman -> OpenAPI)

2.9.5 (May 5, 2022)

  • CLI -mapi status now includes account details if you are logged in
  • CLI issue replay will only replay issues from the last "Completed" run by default

2.9.3 (Mar 22, 2022)

  • CLI - Release preview for resource hint support in mapi run. This option allows callers to specify values that should be used by mapi whenever it generates API requests.

2.9.2 (Mar 9, 2022)

2.9.0 (Feb 25, 2022)

  • CLI - BREAKING CHANGE - mapi login will only support logging with an API token going forward. Support for interactive login with GitHub or username/password will no longer be supported as we are reworking our authentication services.
    • API Tokens can be managed on the website from the following link: https://mayhem4api.forallsecure.com/settings/tokens
  • CLI - Handle OpenAPI/Swagger JSON specification with unsupported regex patterns. Patterns with arbitrary look-ahead and back-references are not supported at this time.

2.8.2 (Feb 9, 2022)

  • CLI - Automatically disable the InvalidResponseSpec rule when working with converted specifications.

2.8.0 (Feb 3, 2022)

  • New Issue: Timeout
    • When an API server doesn't respond in a reasonable amount of time, the fuzzer will give up on its request. This issue can lead to an efficient Denial-of-Service attack. An attacker can leverage problematic requests to consume server resources disproportionate to their own effort.

2.7.21 (Feb 1, 2022)

  • CLI - Fixed an issue where responses specified with a content-type containing a suffix, such as image/svg+xml, would be reported as Invalid Response Spec issue.

2.7.19 (Jan 25, 2022)

  • CLI - Fixed replay logic to respect the endpoint and rule filtering command-line arguments.
  • CLI - Fine-tuned some fuzzing parameters to more predictably generate payloads checking for known vulnerabilities.
  • CLI - Fixed issues with checking certain responses for issues they could not exhibit, wasting effort and conceivably causing false positives.
  • CLI - Fixed an issue when minimizing multi-request programs.

2.7.17 (Dec 22, 2021)

  • CLI - mapi run duration now accepts human-friendly strings such as mapi run <target> 30sec ...
  • CLI - Improve console output when running mapi issue replay

2.7.16 (Dec 17, 2021)

  • CLI - Fixed an issue where job logs may be truncated when running mapi run with --interactive flag.
  • CLI - Issues replayed at the end of mapi run will track issues from the same SCM branch, if one is detected.

2.7.15 (Nov 30, 2021)

  • CLI - Added new flag to mapi run, --warnaserror, that will treat Issues with Severity of 'Warning' as errors. The junit report will include warnings when --warnaserror flag is specified and mapi will exit with an exit code of 1 if any 'Warning' level Issues are discovered during a run.

2.7.13 (Nov 23, 2021)

  • CLI - Fixed an issue where the junit report produced by mapi run --junit may be missing issues

2.7.11 (Nov 5, 2021)

  • CLI - Added a number of prompts for further configuration steps users can take to deepen testing.
  • CLI - Fixed a case-sensitivity issue in token redaction.

2.7.10 (Nov 1, 2021)

  • CLI - Handle OpenAPI/Swagger JSON specification with regex patterns containing escaped forward slashes (for example \\w+:(\\/?\\/?)[^\\s]+")

2.7.9 (Oct 15, 2021)

  • CLI - Added mapi job delete subcommand

2.7.8 (Oct 6, 2021)

  • CLI - Fixed generated SARIF when parsed stacktraces reference "line 0"
  • CLI - Added ability to filter fuzzed endpoints by their OpenAPI tags

2.7.7 (Sep 30, 2021)

  • CLI - Expanded response spec validation warnings
  • CLI - Fixed an issue with failing to stop a run against an unreachable API

2.7.6 (Sep 21, 2021)

  • CLI - Added experimental support for response classifying plugins, which can be used to implement arbitrary logic for identifying issues

2.7.3 (Aug 11, 2021)

  • CLI - If present, OpenAPI tags will be used to organize and group issues
  • CLI - Fixed an issue with authentication redaction in issue details

2.7.2 (Aug 3, 2021)

  • Invalid specification issues will now be reported with "warning" severity if API responses do not conform with the API specification.
  • SARIF/JUnit Report - Fixed an issue where an extra '/' prefix may appear in path display

2.7.1 (Jul 27, 2021)

  • CLI - Support suppressing issues - see Suppressing Issues for more details.
  • Checkers - Added support for Error-based NoSQL Injection for MongoDB

2.7.0 (Jul 13, 2021)

  • CLI - mapi will report warnings when responses are incompatible with the specification used to fuzz your target
  • CLI - Fixed some issues with C# stacktrace parsing
  • CLI/SARIF - It is now possible to report multiple issues for the same path, method and rule id. This is most effective when fuzzing a target that is running in debug mode and will respond with a stacktrace/traceback on error.
    • For example, a 500 Internal Server Error triggered on GET /foo, resulting in different stacktraces will be reported as two separate issues:
      • Issue 1: GET /foo?bar=1 -> Internal Server Error (NullPointerException)
      • Issue 2: GET /foo?bar=0 -> Internal Server Error (DivideByZeroException)

2.6.18 (Jul 2, 2021)

  • CLI - Save fuzzing run output to an HTTP Archive (.har file) with mapi run ... --har out.har

2.6.16 (Jun 24, 2021)

  • CLI - Added beta support for fuzzing with an HTTP Archive (.har file) for the specification.

    • mapi run will now accept a .har file for the specification argument
    • It is recommended to convert a .har to an OpenAPI specification with mapi convert har... to inspect the converted specification before running a job
  • CLI - Added mapi convert command. This runs automatically when calling mapi run with a document which supports conversion.

    The convert subcommand provides an opportunity to inspect the converted OpenAPI 3.0 document prior to fuzzing an API.

    • Convert HTTP Archive .har file to an OpenAPI spec with mapi convert har <recording.har>
    • Convert Swagger 2.0 json or yaml file to an OpenAPI spec with mapi convert swagger2 <spec.json|yaml>
    • Convert Postman 2.x collection json or yaml file to an OpenAPI spec with mapi convert postman <collection.json|yaml>
    • Run mapi convert --help for more details

2.6.14 (Jun 18, 2021)

  • CLI - Issues from the most recent previous job will now be replayed at the end of a run. The intent of this change is to reduce issues fluctuating between jobs.
    • To opt-out of this behavior, pass the --no-replay flag to mapi run.

2.6.13 (Jun 14, 2021)

  • The CLI now handles specifications with missing path parameters. If a path parameter is not in the specification, Mayhem for API will now infer it automatically and fuzz it. Previously, the CLI would exit early with an error.

2.6.12 (Jun 14, 2021)

  • CLI - Resolve an issue where the CLI may panic when generating some invalid URLs during a fuzzing run
  • Fixed an issue where calling mapi run with a path to the specification on Windows (C:\path\to\spec.yaml) would fail to read specification

2.6.11 (Jun 10, 2021)

  • CLI - Resolved an issue that could cause duplicate reported issues in certain corner conditions
    • Issue replay now supports providing a modified OpenAPI/Swagger/Postman specification
    • MAPI_LOG environment variable will now correctly output trace and debug level logs

2.6.10 (Jun 3, 2021)

  • CLI -Support for issue replay. Replay issues using mapi issue replay <job_id>
    • Try mapi issue replay --help to see more details
    • Note that some issues depend on the state of your API and may not always reproduce

2.6.9 (Jun 1, 2021)

  • CLI -Show incomplete request counts in interactive job display
  • CLI -Add markdown display support to SARIF output
  • Header parameters listed in API specifications will now be fuzzed

2.6.7 (May 27, 2021)

  • New Issue: Server Crash
    • Server crashes are now detected and reported in Job Issues.
    • A server that can be forced to crash may be vulnerable to denial-of-service attacks.
  • CLI -mapi issue list print a list of issues for a job

2.6.5 (May 11, 2021)

  • Fix parsing of stacktraces generated on windows host.

2.6.4 (May 10, 2021)

  • Authentication data is now redacted from buggy requests before being uploaded to our API. Note that in some cases, slightly mutated auth tokens might still be updated to our API (since they might be necessary to demonstrate authentication bypasses)
  • Add support for C# stacktraces with async code and nested exceptions

2.6.3 (May 5, 2021)

  • Legacy github token (which are not prefixed by ghp_) are now accepted.

2.6.0 (Apr 29, 2021)

  • Enable new security checkers by default: Command injection, Blind command injection, Path traversals, SSRF (assumes CLI can be reached through localhost from the API)
  • Improve authentication bypass checker by differentiating between -H (for regular headers) and --header-auth (for authentication header).
  • Redirect loops will not terminate a run anymore
  • Improvements in the fuzzing engine to increase coverage

2.5.16

  • CLI: Ignore errors with the --ignore-rule option to mapi run. Callers may ignore specific errors such as InternalServerError or AuthenticationBypass when including this flag.

2.5.13

  • CLI: Upload code scanning results to GitHub from anywhere by passing --github-token with a token to mapi run.
  • CLI: Upload code scanning results to your on-premise GitHub instance by passing --github-api-url to mapi run.
  • CLI: Auto-detect build information from the environment in Drone CI
  • CLI: Golang stacktrace parsing, in addition to the existing parsing for Ruby, JS, Python and Java.

2.5.10

  • CLI: Add SARIF support. mapi run now accepts a --sarif <output-file> option which outputs a SARIF file that you can upload to supported systems, like GitHub if you have enterprise plan.

2.5.7

  • The CLI now has beta support for postman 2.x collections 🚀! You can pass your collection to mapi run instead of an OpenAPI specifications.

2.5.6

  • CLI: mapi run can now automatically create a new API Target if the --url option is supplied.

2.5.5

  • Security: Added support for detecting authentication configuration issues. The fuzzer will now periodically remove or mangle any authentication parameters used for a job. Any endpoints that are specified as with security configuration will be marked as buggy if they can be accessed successfully with missing or altered authentication parameters.

2.5.4

  • Security: Added support for detecting SQL Injection errors. See the new documentation on configuring your API for tips on improving detection of SQL Injection problems.

  • CLI: Change the logging level of the mapi CLI with the MAPI_LOG environment variable. Supported values include trace,debug,info,warn,error. The default is set to info.

2.4.1

  • Renamed cohorts to organizations

2.3.28

  • Added experimental path traversal detection, enabled by setting the MAPI_FEATURE_PATH_TRAVERSAL environment variable.

2.3.19

  • Added support to CLI for rending email verification with mapi signup resend-verify <email-address>

2.3.18

  • Creating a new account when joining a Cohort now requires email verification
  • Added support to CLI for completing email verification with mapi signup verify

2.3.16

  • Improve parsing of URLs given through --url
  • Fix memory leak in type generation during fuzzing

2.3.15

  • Renamed the product to 'Mayhem for API' and the CLI to 'mapi'. Please re-download the CLI.

2.3.8

  • Parallel fuzzing is now available using -j when invoking mapi run (defaults to 1). This will allow you to potentially find concurrency issues, and load test your APIs.

2.3.7

  • bug fix: better support for reference loops in specifications

2.3.5

  • mapi run will now generate a human-friendly html report when --html is given.

2.3.4

  • CLI prints warnings on unexpected properties in the specification instead of aborting.

2.3.3

  • API Targets will now include their owner when presented in the CLI or when referenced to start a new run.

    You may continue to refer to any existing API Targets without the owner; however, shared API Targets will require the owner to be specified.

2.3.2

  • Windows support: Our CLI can now run natively on 64-bit windows.

2.3.0

  • Users can specify cookies for the fuzzer to authenticate to the target API.

2.2.4

  • EULA agreement

2.2.3

  • GitHub Cloud Integration support is now available. Please reach out to us at support@forallsecure.com so that you can get connected with GitHub. Once you have been approved, you can connect with the CLI with the command:

    mapi github connect
    
  • Our GitHub app is also live. During Limited Availability it will not be accessible via the marketplace, but can be installed into your Personal or Organization account by following this link:

    https://github.com/apps/mayhem-for-api

2.2.2

  • Preparation for GitHub support. The CLI will attempt to gather CI details from GitHub Actions, Jenkins, or simply invocations of git to synchronize Job Status with GitHub checks. If details cannot be inferred from the environment, then they may be provided manually with new optional arguments passed to mapi run.
  • GitHub integration support will be fully supported once the Mayhem for API app is available in the GitHub marketplace.

2.2.1

  • CLI will auto-update across patch and minor versions

2.2.0

  • CLI supports API pagination for job and target API resources

2.1.10

  • Fix unresponsive arrow keys in interactive job status

2.1.9

  • JUnit output contains percentile response times per endpoint.
  • Fuzzer better uses examples in the API specification, which allows users to give a hint to the fuzzer when it cannot generate 200 responses on its own.
  • Fuzzer warns if the API closes a connection early before sending a full response. This will eventually be classified as a bug.

2.1.8

  • For accounts registered via GitHub, a New Token may now be requested when using mapi github connect --new-token.

2.1.7

  • Added interactive mode to mapi run with the new --interactive flag. This will now open the interactive UI that was previously only accessible with the mapi job status sub-command.
  • Job status UI now detects when a tty is not present. In instances where a tty is not present, the job log and coverage table will be printed to standard output.
  • All covered endpoints and statistics will now print to the console at the end of a run

2.1.6

  • Bug fixes for GitHub connect

2.1.5

  • Fuzzer will not consider HTTP response code 501 as an error
  • CLI will exit with return code 2 on fuzzer error
  • CLI will now automatically detect Swagger 2.0 specifications and attempt to convert them to OpenAPI 3.0 automatically with the help of the openapi-webconverter

2.1.4

  • CLI can now override API Target authentication parameters when calling mapi run

2.1.3

  • CLI support for handling upcoming pagination changes to job and target API resources

Breaking Changes

As Mayhem for API is continuously evolving, we make an effort to avoid breaking changes. Not breaking your workflow and CI builds is of utmost important to us. We want it to just work, always.

There are times due to security constraints or technical limitations that require us to make a breaking change. We aim to make this as painless as possible, usually requiring no more than downloading a newer CLI. If breaking changes are required, we will try to make it as painless as possible:

  • We will give you a heads up if we notice you are using a version that will become incompatible.
  • We aim to provide command-line arguments backwards compatibility, so that you can always use the latest release without having to change your invocation.
  • We give you tips on how to integrate the fuzzer into your builds so that you automatically use new releases.
  • When necessary, we will slowly deprecate features or workflows, giving you enough time to upgrade.

Unsupported versions

2.1.2

  • CLI includes request and response in junit failures

2.1.1

  • CLI support for blacklisting endpoints when calling mapi run

2.1.0

  • CLI improved logging, feedback and error handling
  • Fixed cascade delete for API Targets

FAQ

(Frequently Asked Questions)

What platforms are supported?

The Mayhem for API CLI (command line interface), mapi, will run on the following platforms:

  • Linux (64-bit)
  • MacOS
  • Windows (64-bit)

What languages are supported?

Mayhem for API scans Web APIs. It does not matter what language the API is written in, as long as it is accessible from the same network as the mapi CLI.

For more detailed issues, Mayhem for API can collect stacktraces returned by the API. It is common to enable returning stacktraces when running an API in a debug mode, but not in production.

Stacktrace parsing enables Mayhem for API to achieve tighter integrations with tools like GitHub / Visual Studio Code with the help of SARIF results export.

The supported languages for Stacktrace parsing include:

  • python
  • ruby
  • java
  • javascript
  • go
  • C#

What kind of information do you collect?

Personal Information

When you sign up for Mayhem for API, we collect your email address and any other optional information that you provide such as Name and Phone Number.

When you sign up using a social login, such as Atlassian, GitLab, Github or Google, we will automatically collect your email address and Name.

If you choose to delete your account, are personal information will remove all personal information at, or before, 28 days in accordance with GDPR compliance.

Application Data

During a Mayhem scan, the mapi CLI can make thousands, or even millions, of requests to your API. We do not store the contents of every request. Only when an issue is discovered, will we collect the request that was sent to your API, and the response that it returned to the CLI.

Other information that is considered 'secret', such as Bearer tokens used for Authentication, will be redacted prior to uploading issue details.

ℹ️ Organizations with Enterprise plans can run in 'local' mode. This mode prevents the mapi CLI from uploading all run details to the mAPI API. Results will be kept local to the computer where mapi was run.

See Keeping Results Local for more details.

Current Terms and Policies