Mayhem for API
Develop reliable APIs faster
What is Mayhem for API?
Mayhem for API automates testing REST APIs by bringing the full might of fuzzing methodology to API testing. With the guidance of an API specification, Mayhem for API provides accurate and informative test coverage tailored to any REST API.
How does it work?
Mayhem for API uses a fuzzing engine to automatically generate a comprehensive suite of inputs used to test function and robustness of an application’s API infrastructure. By using fuzzing techniques to generate inputs and observing the response from the application, Mayhem for API can quickly iterate through multitudes of test cases to find weakness in an API’s functionality or security.
Why should I use it?
It is no secret that web APIs have become increasingly important to the operation of modern business. Many business models for new products and services are constructed based on APIs such as billing and identity providers. Trust has become a necessity for APIs. APIs that perform consistently and with high quality earn trust, and those that fail are abandoned. Mayhem for API helps you build trust in your APIs in a number of ways:
- Resilience - Mayhem for API is great for discovering troublesome
500 Internal Server Errorresponses and server crashes automatically, before your clients do. - Security - With a growing list of Security Checkers, Mayhem for API can discover issues such as Server Side Request Forgery (SSRF) and SQL Injection.
- Quality - Mayhem for API validates all the responses returned from your API with your specification to identify endpoints which are inconsistent with the spec, such as missing fields or incorrect response codes. Keeping your API synchronized with your specification ensures that API consumers are not caught of guard with unexpected behavior.
- Performance - Latency statistics of every endpoint are recorded on every run to provide a clear picture of what endpoints have inconsistent or degraded performance.
The Mayhem for API website provides reproduction steps so that you can
replay any issues either with curl or with the mapi CLI.
Mayhem for API is not a replacement for existing automated tests. It is complementary! With the help of fuzzing, Mayhem for API identifies blind spots in your existing testing, without the bias or the tedium of manually written tests.
Point and Shoot
Setting up the fuzzer is a breeze. Once you have signed up for a
free account all you need to do is
download the mapi CLI and start testing your API with
a compatible specification such as OpenAPI
or a Postman Collection.
Reach out!
We aim to provide a great experience. Please reach out to us on Discord or by emailing support@forallsecure.com if you have any suggestions or run into any issues. We're more than happy to help!
Let's get started!
Getting Started
Let’s start your API fuzzing journey! There’s a lot to learn, but every journey starts somewhere. In this chapter, we’ll discuss:
- Installing the Mayhem for API command-line client on Linux, macOS, or Windows.
- Using the CLI to fuzz our first API
- Monitoring progress
Signing up
If you haven't done so yet, the fastest way to get started is to sign up for a free plan at https://mayhem4api.forallsecure.com/signup. If you already have an account, then you are ready to go for the next steps!
Installation
The Mayhem for API CLI is available to download for various common platforms.
ℹ️ The CLI will automatically keep itself updated when used as we make fixes and bug improvements.
MacOS
curl -Lo mapi https://mayhem4api.forallsecure.com/downloads/cli/latest/macos/mapi \
&& chmod +x mapi
Here's an easy way to add the mapi executable to your path:
sudo mv mapi /usr/local/bin
You may download this SHA256 checksum or this SHA512 checksum to verify the integrity of the file you downloaded.
Linux (64-bit)
curl -Lo mapi https://mayhem4api.forallsecure.com/downloads/cli/latest/linux-musl/mapi \
&& chmod +x mapi
Here's an easy way to add the mapi executable to your path:
sudo mkdir -p /usr/local/bin/
sudo install mapi /usr/local/bin/
You may download this SHA256 checksum or this SHA512 checksum to verify the integrity of the file you downloaded.
Windows (64-bit)
From a Windows 10 terminal (PowerShell or cmd):
curl.exe -Lo mapi.exe https://mayhem4api.forallsecure.com/downloads/cli/latest/windows-amd64/mapi.exe
or download :
https://mayhem4api.forallsecure.com/downloads/cli/latest/windows-amd64/mapi.exe
You may download this SHA256 checksum or this SHA512 checksum to verify the integrity of the file you downloaded.
Test it out!
Make sure the CLI works by running:
mapi --help
Authentication
The mapi CLI communicates with our API using
OAuth 2.0 Bearer Tokens. The token
will be read by the environment variable, MAPI_TOKEN, if available.
To get a new token, visit the "Manage API Tokens page"
to create a new token, <NEW_TOKEN>, and login:
$ mapi login <NEW_TOKEN>
Welcome to Mayhem for API! We have saved a new API token in
your local settings at '/Users/mapi_fuzzer/Library/Preferences/rs.mapi/mapi.toml':
3BzW...
Setting the displayed API Token to the environment variable, MAPI_TOKEN, will
allow you to run the CLI on other computers, such as part of your Continuous
Integration build.
Now you can try contacting the API. Let's get the list of targets to which you have access:
mapi target list
You should see an (empty) list of API targets. Let's add our first target so that list won't stay empty for long.
Hello, Fuzz!
We have prepared an instance of the Swagger Petstore API for you to try fuzzing. The Swagger Petstore is a stand-alone REST API server which implements the OpenAPI 3 Specification.
Fuzz the Swagger Pestore API by running the following in your terminal...
mapi run \
petstore auto "https://demo-api.mayhem4api.forallsecure.com/api/v3/openapi.json" \
--url "https://demo-api.mayhem4api.forallsecure.com/api/v3/" \
--interactive
...once the run completes, visit your dashboard to see the results!
What just happened?
Let's break that command down, to see what happened:
mapi run
Use the CLI to run a fuzzing job.
ℹ️ Check out
mapi helpif you want to explore some other things the CLI can do!
petstore
Use (and create, if it doesn't already exist) a "target" named petstore for this run. We'll come back to targets in more detail later. For now, a short name that uniquely identifies the API being tested is perfect.
ℹ️ Targets are used to group related fuzzing jobs.
auto
Runs the job until it is decided that enough endpoints have been exercised.
You can specify a specific time period here if you wish, such as 30sec to run
the job for 30 seconds. The longer the job is run, the more edge cases will be
uncovered.
See How long to run for more information.
https://demo-api.mayhem4api.forallsecure.com/api/v3/openapi.json
Tells the fuzzer to base its testing on the OpenAPI 3 specification of the petstore demo API, which is hosted on our site for demo purposes.
--url "https://demo-api.mayhem4api.forallsecure.com/api/v3/"
Tells the fuzzer, which is running locally on your computer, to address its tests to the petstore demo API on our network.
--interactive
Runs the fuzzing job with an interactive, terminal-based user interface. This option should be omitted in a continuous integration environment.
You will see the following UI in interactive mode:
The status UI contains a fair amount of information:
- A progress bar at the top, based on the duration of the job and elapsed time.
- The job logs. If the job doesn't start properly, or runs into trouble, this is where you should see information on how to resolve the issue.
- The status codes that the fuzzer observed per endpoint. This will help you determine if the fuzzer is covering the endpoints and status code you expected. For instance, if you see only 401 responses, then that means the fuzzer probably needs help with authentication. We cover this in more details in the “API Authentication chapter".
The UI has a few navigation keys that might be handy:
q,^Cor<Esc>to quit<Tab>to switch focus between widgets<Space>to make the current widget full screen- Arrow keys for horizontal and vertical scrolling
In your run you should see a number of red endpoints which means that at least one 500 response code is observed. That means we just found some bugs! You should also see a number of green endpoints. These mean that at least one 2xx response has been observed.
You're already off to a great start on your API fuzzing journey! Hopefully, you're thinking to yourself: "this is neat! I wonder how I can use it to test my own APIs?"... because that's the next topic!
Fuzz Your Own API
You've seen what Mayhem for API can do in a demo. Now it's time to fuzz your own!
To start testing an API, you only need to provide two things: a specification describing the API, and a URL where it can be reached. You'll be running something like:
mapi run my-api 30 <specification> --url <url>
API Specification
ℹ️ Specifications can be passed to
mapias either local files or URLs.
Mayhem for API is built around OpenAPI 3 specifications. If you have an OpenAPI
spec describing your API, you're all set. Pass either a URL or file system path directly to mapi as the <specification> argument!
If you don't have an OpenAPI spec, we support a few alternatives, described below.
OpenAPI 2.x (sometimes a.k.a. "Swagger")
mapi run will automatically detect and work with older OpenAPI/Swagger specs.
Alternatively, you can do a one-time conversion into OpenAPI 3 by running
your old spec through the mapi convert swagger2 command (see mapi help convert swagger2 for details.)
Postman
Mayhem supports Postman collections. Read more about our Postman Collection support here.
HAR
If you don't have any of the above, you'll need to do a little bit more work to generate a spec for Mayhem for API to use. The process is documented in detail over here.
API URL
Mayhem for API needs to know the URL of your API server. Because the requests come
directly from the mapi CLI tool running locally on your computer, this will
work for APIs that are:
- Behind a corporate firewall
- On localhost
- In a private network
- On the public internet
If the machine running the CLI can access the API - then we can fuzz it!
Proximity and Latency
Although it'll work in just about any configuration, Mayhem for API works better the "closer" it is (in network terms) to the API server. For the best results, this means pointing the fuzzer at a locally-running instance of your server.
Never, Ever, Ever Production
Mayhem for API has one job: finding ways to break an API! You should absolutely not give Mayhem for API the URL of your production services to test against.
Keeping Results Local
Organizations with Enterprise plans can run the mapi CLI in 'local' mode by passing
the --local flag to calls to mapi run. This will fuzz your Target API, as usual,
with some caveats:
What is required to run Mayhem for API in local mode
- An active 'Enterprise' plan for the Organization member running the
mapiCLI.
What is uploaded to Mayhem for API servers for local runs?
- Anonymized contributor seat counts
- API Target placeholder (if a matching target does not exist)
What is NOT uploaded to Mayhem for API servers for local runs?
- Job details (latency, covered endpoints)
- Issues (requests sent, responses received)
- Specifications for conversion (ie: HAR files will not be automatically uploaded and convert to a specification)
- Custom issue rules from third party integrations, such as OWASP ZAP API Scan rules
How do I get the results of a local run?
All report formats supported by the mapi CLI will continue to work as described
in the the chapter about reporting issues, except for 'Web UI'
reporting -- because no results are uploaded the Mayhem for API service.
It is the responsibility of the caller to persist the results of local runs.
Files for conversion (no swagger2 → openapi)
All right, hopefully you've got Mayhem for API configured to test your API, congrats!
ℹ️ If we're already uncovering some issues in your API, you might want to take a detour into the chapter about issues, where we'll go into detail about those results.
For most real-world APIs, the next step is going to be telling Mayhem for API how to authenticate, which is what we'll cover next!
HTTP Archive (HAR)
Not all APIs have an OpenAPI/Postman/Swagger specification available. The only specification is the code itself. It is still possible to fuzz an API without a specification by recording transactions with the API as an HTTP Archive (.har file) and converting the recording into an OpenAPI specification. While this enables fuzzing of APIs without specifications, there is a trade-off in the accuracy of the converted specification compared with one generated from the code or maintained by hand.
HAR Files may contain sensitive information
⚠️ HAR files contain everything that was sent between your client/browser and the
target API. This may include Authorization headers, cookies, etc. It is not
advisable to persist or convert a .har file unless it has been stripped of sensitive
information beforehand.
Converting a .HAR to an OpenAPI 3 specification
ℹ️ Requires
mapi 2.6.16or newer
The mapi convert command is used to convert from other formats to an OpenAPI 3 specification.
For example, the following command converts a recording, recording.har into an OpenAPI spec, openapi.yaml:
mapi convert har recording.har --host "127.0.0.1:8080" --base-path "api/v1" --out openapi.yaml
Let"s examine the arguments:
--host "127.0.0.1:8080"The conversion will attempt to infer the URL of your API from the recording. If the recording contains requests to multiple URLs, you can specify the--hostoption to restrict conversion to requests that are only from the specified host.--base-path "api/v1"If your API has a common base path this option will strip the common prefix from all request URLs. This can significantly improve the accuracy of the converted spec.--out opennapi.yamlThis will overwrite any existing file,openapi.yaml, with the converted specification. Without this argument, the conversion will be printed to standard out in your terminal.
ℹ️ See all available options with the
mapi convert --helpcommand
Upon successful conversion you may inspect the converted file and make any modifications you feel are necessary or start a new run immediately.
Create a .HAR recording with your Browser
If you have a web frontend that interacts with your API, you can record an HTTP Archive by navigating your website and running through various use cases to build a recording of API interactions based on which features you access from the frontend.
Most modern browsers support recording your interactions and exporting a HAR file. See https://toolbox.googleapps.com/apps/har_analyzer/ for browser-specific details.
Create a .HAR recording with a proxy
A web frontend may not make sufficient requests to your API to generate a useful OpenAPI
specification. You may have other tools such as a CLI, automated test suites, or a collection
of manual curl invocations that can do this. In this case, you can use a proxy to record
all of your traffic to produce a HAR file.
The following tools provide support for this capability:
💡 There are many other tools that can help you record your traffic to export as a HAR file. If you see any that should be included here, please let us know: support@forallsecure.com
Postman Collections
Specifying a postman api key and collection id enables Mayhem to read postman collections. You can optionally also specify a postman environment id or file.
mapi run my-api 30 <postman collection id> \
--url <url> \
--postman-api-key <postman api key> \
--postman-environment <postman environment id or file>
Requests that use many of the Postman dynamic variables as path variables, headers, query parameters, or within a request body are handled by Mayhem and mapped to appropriate types during fuzzing.
Exported Postman File
mapi run will also automatically detect if it's given a Postman 2.x collection in
place of an OpenAPI spec, and work with that.
Alternatively, you can do a one-time conversion from Postman into an OpenAPI 3
spec by running the collection through the mapi convert postman command (see
mapi help convert postman for details.)
Postman Variables and Scripts
Mayhem supports Postman collection and environment variables.
Postman has a scripting engine that is not supported by Mahyem. Typically, these scripts are used to seed requests with values that are returned from another API's response. Mayhem uses API responses automatically during fuzzing.
It may be useful to fork a Postman collection that makes use of scripts, replace scripts with variables where possible, and target the forked collection with Mayhem.
Authenticating With Your API
For most APIs, the next step is setting up authentication. After all, without successfully authenticating, Mayhem for API can only test for very superficial problems! Giving the fuzzer a way to authenticate to the target API will enable it to exercise more endpoints and maximize coverage.
Mayhem for API has built-in support for basic authentication, header-based authentication (such as bearer tokens) and cookie-based authentication. If none of these are sufficient, our rewrite plugin system gives you a powerful option to implement whatever you need for your specific authentication scheme. All of these are described in more detail below. But first, a common gotcha...
Accidental Credential Invalidation
If the credentials you use in fuzzing can be invalidated through a logout
endpoint, you will almost certainly need to prevent the fuzzer from issuing
requests to that endpoint, using the --ignore-endpoint flag to mapi run,
something like this:
mapi run [...] --ignore-endpoint "/api/logout"
ℹ️ For more details on this and other mechanisms for choosing which endpoints to fuzz, see Selective Route Testing.
Basic Authentication
Basic access
authentication
is a simple technique for enforcing access control to web resources,
which is supported by most web servers. You can specify basic
authentication credentials when you fuzz your target with mapi run
with a command line option:
mapi run --basic-auth "username:password" <target> <duration> <specification>
... or as an environment variable:
export MAPI_BASIC_AUTH="username:password"
mapi run <target> <duration> <spec>
Note that basic authentication does not protect the username and password by itself. They are simply encoded with base64 in transit, but not encrypted or hashed. Basic authentication should be used in conjunction with HTTPS to protect the credentials, or on a trusted network.
Header Authentication (e.g. Bearer Tokens)
Bearer authentication (also called token authentication) is an HTTP authentication scheme that involves security tokens called bearer tokens The client must send this token in the Authorization header when making requests to protected resources:
mapi run --header-auth "Authorization: bearer <token>" <target> <duration> <spec>
In Mayhem for API, the same mechanism is generalized to work with any header-based authentication, for instance:
mapi run --header-auth "X-Custom: auth <token>" <target> <duration> <spec>
You can also use an environment variable to pass such headers:
export MAPI_HEADER_AUTH="Authorization:Bearer <token>"
mapi run <target> <duration> <spec>
Note that the authorization header does not protect the token by itself. It is not encrypted or hashed before sending it to the server. As with basic authentication, this authentication method should be used in conjunction with HTTPS to protect the credentials, or on a trusted network.
ℹ️ To specify custom headers that do not contain credentials, use
--headerinstead of--header-auth. Mayhem for API treats--header-authdifferently when probing for issues, and when redacting potentially sensitive data.
Cookie Authentication
Cookie authentication uses HTTP cookies to authenticate client requests
and maintain session information. Cookies are generally returned by the
server after a successful login, and sent by the clients in subsequent
requests. You can specify cookies when you fuzz your target with mapi run with a command line option:
mapi run --cookie-auth "PHPSESSID=abe67cd" <target> <duration> <spec>
... or as an environment variable:
export MAPI_COOKIE_AUTH="PHPSESSID=abe67cd"
mapi run <target> <duration> <spec>
Note that cookies are not encrypted or hashed before being sent to the server. As with basic and header authentication, this authentication method should be used in conjunction with HTTPS to protect the credentials, or on a trusted network.
Authentication Using Rewrite Plugins
For cases where the above built-in methods are insufficient (e.g. if the authentication is dynamic over the course of a fuzzing job), you can use our rewrite plugin system and code your own.
Rewrite plugins aren't authentication-specific and have lots of capabilities documented over here.
Postman Authentication
If the specification provided is a postman collection id, the Postman collection's authentication is used.
The authentication methods currently supported are
- API Key
- Bearer Token
- Basic Auth
- OAuth 2.0 (Token must be synced)
Authentication on folders or requests is not currently supported. Specifying authentication with arguments to Mayhem overrides Postman's authentication configuration.
More to come!
We are planning on adding built-in support for more authentication methods soon, and we'd love to hear from you. Please reach out to us on Discord or by emailing support@forallsecure.com to let us know what you'd like to see!
At this point, Mayhem for API should be able to exercise even the authenticated endpoints of your API. Excellent!
ℹ️ If we're already uncovering some issues in your API, you might want to take a detour into the chapter about issues, where we'll go into detail about those results.
We're not quite done, though. Most real-world APIs have other, non-authentication input requirements that are too specific to be reliably found through random fuzzing alone. The next section covers the why and the how of getting Mayhem for API to successfully exercise your entire API.
Successful Coverage
Once authentication is configured, the next step is to ensure that Mayhem for API is able to successfully cover as much of your API as possible.
What is Successful Coverage?
Mayhem for API considers any 2xx response from the API server to indicate success, and a single successful response from a given endpoint is enough for us to consider it successfully covered.
If your API has some other criteria for success: we don't currently support it, but we are interested in this use case, so please reach out to us on Discord or by emailing support@forallsecure.com, and let's see what we can figure out!
Why is Successful Coverage Important?
The absence of findings, for an endpoint which is not successfully covered, is meaningless.
Mayhem for API fuzzing is designed around this idea: that the most interesting test results come from requests that are almost correct. Some of the payloads we generate are purely random—that's important, too!—but once we find a successful payload for a given endpoint, we use that as a starting point for many future requests.
Consider a single POST endpoint which executes an INSERT statement against a relational database on success. If every single request issued by Mayhem for API is invalid, the INSERT statement has never been executed; if that INSERT statement has a SQL injection vulnerability, it won't be found.
If, on the other hand, we pass a valid payload (causing the INSERT statement to be executed) and get a 200 from this endpoint, we'll then use the successful payload as the basis for many future tests. If that INSERT statement has a SQL injection bug, we'll eventually find it.
That's why we emphasize "successful coverage": successfully covering an endpoint massively increases our confidence that the absence of findings is meaningful.
Reasons For Lack of Successful Coverage
Authentication, which is documented separately, is often the first barrier to successful coverage.
Mayhem for API (and fuzzing in general) usually does just fine, without intervention, generating values that satisfy simple data validations—things like "the 'page' parameter must be an integer".
What are other reasons Mayhem for API might be failing to successfully cover an endpoint?
Incorrect URLs
Sometimes the simplest solution is hard to see! If you're getting unexpected failures on endpoints that should be easy to exercise successfully—especially if the failures are 404s!—double-check that Mayhem for API is hitting the correct URLs.
Complex Structural Validation
ℹ️ If you start with thorough OpenAPI specs, you're less likely to run into this problem.
While we'll fairly quickly generate a random value that "must be an integer", it'll take quite a long time to generate a random value that, for example, "must be a JSON document with exactly one key, named 'page', whose value is an integer".
Stateful Validation
The trickiest validations are ones which are based on the current state of the service. For example, a value that "must be the unique id of an existing row in the 'pages' table" might take essentially forever to generate through pure randomness.
Mayhem for API has some tricks up its sleeves (such as trying values from the output of one request in the inputs of another) that help, but it's not perfect.
Whatever the reason, what follows are suggestions for diagnosing and fixing problems in successful coverage.
Diagnosing Problems in Successful Coverage
If you're still here, that means you have at least one endpoint that Mayhem for API can't yet successfully cover. So, how do you begin to figure out why that's happening?
It's worth noting that this may be extremely hard unless you either are (or working closely with someone who is) deeply familiar with the service being tested.
Step One: Focus
We strongly recommend that you dig into one endpoint (or a very small set of interrelated endpoints) at a time, when attempting to get successful coverage, and to keep the runs fairly short (60 seconds or less).
Filter your runs down via the --include-endpoint <endpoint> flag to the mapi run, or other filtering methods documented in more detail here.
Step Two: Logs
When looking at logs, what you're looking for are patterns: what kinds of requests is Mayhem for API making, and why are they failing? What kinds of requests is Mayhem for API not making, and how can it be coerced into making them?
Start by looking at your own service logs, since hopefully you're familiar with those!
If your own logs aren't illuminating enough, you can also capture the full request/response payloads of an entire fuzzing run using the --har <file> flag to mapi run. The output is in JSON HAR format.
Step Three: Reach Out!
Don't hesitate to reach out to us on Discord or by emailing support@forallsecure.com. Between your expertise with your service, and our expertise with Mayhem for API, we should be able to figure it out!
How To Improve Successful Coverage
Correct the URLs
Mayhem for API always constructs URLs by concatenating each endpoint's path with a common server URL.
The paths are taken directly from the API specification.
The server URL is given via the --url <url> argument to mapi run. (There are other ways, but we'll focus on this one, as it is the most explicit!)
Sometimes the specification elides a common prefix from all endpoints. Consider a specification:
/version:
get:
...
/user:
get:
...
If the complete URLs are supposed to be https://localhost/api/v2/version and https://localhost/api/v2/user, the common prefix (/api/v2) will need to be included in the server URL given to Mayhem for API:
mapi run ... --url 'https://localhost/api/v2'
Improve The Spec
Often, Mayhem for API just needs a bit more... specificity... in the specification, in order to successfully cover an endpoint.
Provide Schemas
If Mayhem for API is consistently unable to generate structurally valid payloads, this can usually be resolved by adding or refining schemas in the spec.
OpenAPI allows for parameters and request body types to be declared, in whatever level of detail is needed, using Schema Objects.
For example, just listing the required properties for an endpoint's request body makes Mayhem for API massively more likely to successfully cover that endpoint:
/example:
post:
summary: example
requestBody:
required: true
content:
application/json:
schema:
type: object
+ properties:
+ page:
+ type: integer
Use Consistent Names and Types
Mayhem for API will do a better job generating statefully valid payloads if stateful identifiers that are passed between endpoints have the same name and type.
For example, if you have an endpoint that lists resources and another that fetches a resource by id, make sure the identifiers share a name and type, so that Mayhem for API can more easily connect the dots:
/example:
get:
summary: list of examples
responses:
'200':
description: all the examples
content:
application/json:
schema:
type: array
items:
type: object
properties:
example_id:
type: integer
- /example/{id}:
+ /example/{example_id}:
get:
summary: details of a single example
parameters:
- in: path
- name: id
+ name: example_id
schema:
- type: string
+ type: integer
ℹ️ The spec changes you make to help Mayhem for API exercise your API will also improve anything else—code, documentation, and so on—that you derive from your specs! Win-win!
Provide Examples in your Specification
There are situations where Mayhem for API really just needs a hint. For example, consider endpoints that include the username of the currently-authenticated user, like '/user/{username}/settings'.
ℹ️ Examples are often needed when the service being tested has some pre-loaded state (like the user, for authentication!) that's difficult to naturally "discover" through fuzzing.
In this case, OpenAPI allows "example" values, and Mayhem for API will take the examples into account when generating request payloads. For the "{username}" parameterized endpoint, this might be as simple as:
/user/{username}/settings:
get:
summary: settings
parameters:
- in: path
name: username
schema:
type: string
+ example: mayhem4api-user
Resource Hints
Mayhem for API may need additional hints to make a successful requests to endpoints. Perhaps every request requires
that a constant value is applied to every request body or path parameter. It is possible to provide hints for
all generated requests by passing one or more --resource-hint options to the mapi run command.
In order to understand which resources are applicable to the --resource-hint option, you should first
list the available resources for your API specification with the mapi describe specfication command.
For example, let's list the resources of the petstore demo API:
mapi describe specification \
"https://demo-api.mayhem4api.forallsecure.com/api/v3/openapi.json"
Spec path is https://demo-api.mayhem4api.forallsecure.com/api/v3/openapi.json
PUT /pet BODY category/id
PUT /pet BODY category/name
PUT /pet BODY id
PUT /pet BODY name
...
PUT /user/{username} BODY userStatus
PUT /user/{username} BODY username
DELETE /user/{username} PATH username
Every resource for the specification is flattened and listed out in the response. For example, the
following represents a field value that is passed as part of a request body when generating a PUT
request against /pet.
Method
|
| Path
| |
| | Request part (can be QUERY, PATH, HEADER or BODY)
v v v
PUT /pet BODY category/id
^
Fully qualified path to request body
Another example is a path parameter:
Method
|
| Path
| |
| | Request part
v v v
DELETE /user/{username} PATH username
^
Path parameter
The --resource-hint option to mapi run takes a tuple of resource path and value, separated by a colon.
For example, say we wanted to always use the same username value of foo for the DELETE method noted
above:
mapi run \
petstore auto "https://demo-api.mayhem4api.forallsecure.com/api/v3/openapi.json" \
--url "https://demo-api.mayhem4api.forallsecure.com/api/v3/" \
--resource-hint "DELETE /user/\{username\} PATH username:foo"
The --resource-hint specifies that any request generated for this DELETE resource will now
use foo for the username parameter:
Resource Path split with ':'
v v
--resource-hint "DELETE /user/\{username\} PATH username:foo"
^ |
must escape '{' for |
valid regex |
^ Value to use
--resource-hint will accept a regular expression for matching resource path as well. For instance,
say you wanted to substitute foo for EVERY username value:
Match ANY request that requires a 'username'
v
--resource-hint "username$:foo"
When multiple --resource-hint are specified that match the same resource path, Mayhem for API
will randomly select one of the provided hints whenever it generates a request.
Choose between 'foo' ... or 'bar'
v v
--resource-hint "username$:foo" --resource-hint "username$:bar"
Ignore Endpoints
If, for some reason, your endpoint cannot be covered successfully, consider skipping it during fuzzing (using the --ignore-endpoint <endpoint> argument to mapi run), because we have low confidence in lack of findings.
At this point, Mayhem for API is able to exercise your API quite thoroughly, and probably uncover all sorts of issues... and, more importantly in the long run, give confidence that your API is free of the issues we don't find!
The next section describes optional setup steps to maximize the specificity and detail of the issues Mayhem for API reports. You can do that now, or you can come back to it later.
If you decide to skip that for now, here are some other docs you might be interested in jumping to:
- The chapter about issues, for details about working with findings.
- CI/CD Integration, for how to get Mayhem for API into your build pipeline.
- Organizations to learn about how Mayhem for API works for teams.
How long should I run?
Unlike other testing methods, like unit and integration tests, the definition for
being 'done' for a fuzz test is, well, fuzzy. In general, the longer you run mapi,
the more of your endpoints, and the code behind them, will be covered.
Your run duration depends on a few properties of your target API. For example:
- Latency between CLI and API - local APIs will typically have lower latency
- API Configuration - is your API configured with multiple threads/workers to handle high concurrency?
- Number of endpoints - is your API a few or hundreds of endpoints?
The following will help guide you toward the right choice for testing your target API.
Concurrent Requests
Mayhem for API will automatically run in concurrent execution mode as long as you are running on a machine with more than one CPU core. The higher the concurrency, the more requests can be sent in parallel to your target API. Mayhem for API defaults its concurrency level to "4", or the number of CPU cores -- whichever is less.
You can increase the number of parallel fuzzers by passing the --concurrency flag
to mapi run command.
For example, to run with 8 parallel threads, pass --concurrency 8 to mapi run:
mapi run \
<target_name> \
auto \
<path_to_api_specification> \
--url "https://example.com" \
--concurrency 8
Automatic run duration
We recommend running Mayhem for API with automatic duration if it is your first time
using the tool, or aren't sure how long to run your scans. When running with auto
duration, mapi will attempt to make a request to every endpoint at least 100 times.
You can run with automatic duration by passing the auto string to the mapi run
command. For example:
mapi run \
<target_name> \
auto \
<path_to_api_specification> \
--url "https://example.com"
When should I use an automatic run duration?
- It is your first time trying Mayhem for API
- You want
mapito automatically adjust to your API as it changes size
Tweaking the minimum request count
When running with
autoduration,mapiwill attempt to make a request to every endpoint at least 100 times.
When running in auto mode, you can change the threshold of the minimum
number of endpoint request with the --min-request-count flag. For instance,
you may wish to increase the minimum request threshold in order to exercise
your API more than the default of 100.
For example:
mapi run \
<target_name> \
auto \
<path_to_api_specification> \
--min-request-count 500
--url "https://example.com"
Fixed Time run duration
mapi will continuously scan your target API until the duration has been reached
when running with a fixed duration. The duration time is specified as part of the
mapi run command.
For example:
mapi run \
<target_name> \
5min \
<path_to_api_specification> \
--url "https://example.com"
The duration syntax is fairly flexible, allowing callers to specify time in seconds, minutes or hours.
- Run for 30 seconds [30s or 30sec]
- Run for 90 minutes [90m or 90min]
- Run for 1 hour [1h or 1hr]
- Run for 2 hours and 20 minutes [2h20m]
When should I use a fixed time run duration?
A fixed duration scan is ideal if you want to run Mayhem for API, but are limited
in your computing time/resources. For instance, if you are running mapi as part of
your continuous build pipeline, you may not want a step running that takes an
unpredictable amount of time. By setting a fixed duration, you can ensure that mapi
you are explicit about how long the mapi step will take.
Configuring Your API For More Detailed Issues
In this section we will examine how to configure specific frameworks to help Mayhem For API find problems, such as possible SQL Injection, that might not be as discoverable with default application settings. These configuration changes are optional and are rarely safe for production; however, they can provide more value out of Mayhem for API when enabled in a safe environment such as part of your CI/CD pipeline.
These changes typically involve adjusting your API configuration to return additional information in the response whenever an internal error occurs, such as the stack trace.
This will help you get to the root cause of buggy endpoints faster and can help the fuzzer identify security threats more accurately such as error-based SQL Injection attacks:
💡 If you do not see your API framework here yet, please let us know: support@forallsecure.com
Spring Boot (Java)
Various properties can be specified inside your application.properties file, inside your application.yml file,
or as command line switches. please visit https://docs.spring.io/spring-boot/docs/current/reference/html/application-properties.html
for a comprehensive list of properties.
For the following configurations, assume we are working with application.properties file.
⚠️ These settings are appropriate for development testing, but should not be used in production.
Include Stack Trace on Server Error
Update your application.properties file with the following line:
server.error.include-stacktrace=always
Whenever the fuzzer triggers an error, Spring Boot will include a stack trace in the response.
Flask (Python)
When developing a Flask-based API, the built-in Werkzeug development server provides a debugger which shows an interactive traceback in the browser when an unhandled error occurs during a request.
For more information, visit https://flask.palletsprojects.com/en/latest/debugging/.
⚠️ These settings are appropriate for development testing, but should not be used in production.
Include Traceback on Server Error
Either set the FLASK_ENV=development environment variable prior to running your app:
$ export FLASK_ENV=development
$ flask run
Or, include the debug=true flag when running from Python code
from flask import Flask
app = Flask(__name__)
if __name__ == '__main__':
app.run(debug=True)
Whenever the fuzzer triggers an error, Flask will include the traceback in the response.
Identifying Buggy Endpoints
Mayhem for API reports any issues it finds as well as the input that triggers them. In this chapter, we describe the different solutions you can use to see the issues Mayhem for API finds.
Web UI
Mayhem for API uploads findings from each fuzzing run to the ForAllSecure cloud service. Sign in to the UI at https://mayhem4api.forallsecure.com, where you can dig into the details of each run. This includes the details of any issues found, as well as per-endpoint statistics such as response code distribution and latency percentiles. Here's what the web UI looks like:
Detailed Report in the Web UI
HTML Reports
Mayhem for API can generate static, local HTML reports using the --html argument to
mapi run. These reports contain essentially the same information
as their web UI counterparts. They can be saved as "build artifacts" in
a CI system. Here's what an HTML report looks like:
Detailed Static HTML Report
JUnit Reports
Mayhem for API can generate industry standard, CI-friendly JUnit
reports with the --junit argument to mapi run. These reports contain all of
the detailed issues found—but not the additional statistical details from web
and HTML reports!—in a machine-readable form, which most CI software can consume
and render natively.
By default, only Issues with Error level severity will be included in the junit
report. If you wish to include Warning level Issues in your report, you must
specify the --warnaserror flag when calling mapi run ... --warnaserror --junit <test-output.xml>
SARIF Reports
Mayhem for API can also generate
SARIF files with the --sarif argument to mapi run. SARIF files are a great way to upload issues to
GitHub, if you happen to use GitHub. See our “GitHub
Integration" section for more details.
Mayhem for API Issues in GitHub
SARIF files can also be loaded in VSCode using a SARIF extension from Microsoft:
Mayhem for API Issues in VSCode
ℹ️ To give accurate file and line information for each issue, Mayhem for API requires your API to include stacktraces in the body of "500 Internal Server Error" responses. Only do that in dev & testing -- never in production!
Debugging tools
Of course, having Mayhem for API find an issue is, in some ways, just the start. Isolating, reproducing and ultimately fixing issues is hard work!
Here are a few approaches we hope will make it easier. (These are all completely independent of Mayhem for API.)
API Logs
The simplest solution is to look through your API logs when a 500 is triggered. Logs often include stack traces pinpointing which line of code triggered the issue. If you have centralized logging infrastructure, with ElasticSearch or Splunk for instance, you'll likely be able to filter the logs by response code to only see 500 responses.
The fuzzer-generated requests will contain the following headers, which can help you look through your logs and correlate log events with specific fuzzing inputs:
User-agent: mapi/X.X.X, whereX.X.Xrepresents the mapi CLI version.X-Mapi-Program-UUID: <UUID>, whereUUIDcorresponds to a fuzing input. This header allows you to correlate buggy requests from the mapi reports to your API logs.
Error Monitoring Tools
We believe that installing an error monitoring tool in your API is the best way to dig into bugs. They can monitor not only your running APIs, but also any other backend systems like other microservices, background workers, and so on. Those backend systems might error out without the API returning a 500 responses, and error monitoring tools can help you detect those issues triggered but undetected by the fuzzer.
Error monitoring tools detect errors, gather as much context as possible, group similar errors together, and make them available to you through their UI, which looks something like this for sentry.io:
Plenty of third-party companies offer great error monitoring tools:
Odds are your organization is already using such a tool in production. You can leverage it for fuzzing to get deep insights into the bugs that the fuzzer triggered. Your error monitoring tool will now alert you when the fuzzer finds an issue. It will give you plenty of context like the stack trace to understand the issues. You will be alerted of those issues perhaps even before the bug gets deployed in production!
Issue Rules (Checkers)
Issues discovered by Mayhem for API are categorized by Rule. Each Rule corresponds to one or more checkers that are executed as part of a fuzzing run.
The following is a list of supported Rules. Click a link below to view the details of a specific rule.
- Authentication Bypass (
auth-bypass) - Command Injection (
command-injection) - Internal Server Error (
internal-server-error) - Invalid Request Spec (
invalid-request-spec) - Invalid Response Spec (
invalid-response-spec) - NoSQL Injection (
nosql-injection) - PII Disclosure (
pii-disclosure) - Path Traversal (
path-traversal) - Reported by a custom error-classifying plugin. (
plugin) - SQL Injection (
sql-injection) - Server Crash (
server-crash) - Server Side Request Forgery (SSRF) (
ssrf) - Timeout (
timeout) - Verb Tampering (
verb-tampering)
Suppressing Issues
Not all issues reported by Mayhem for API may be relevant or of interest to an API maintainer. Mayhem for API offers a mechanism to suppress issues discovered during fuzzing jobs.
The .mapi configuration file
A .mapi configuration file specifies rules for issue suppression:
#
# Example .mapi configuration file
#
version: "1.0"
# An array of rules for suppressing issues.
#
# If an issue matches any rule, it will be suppressed
# from reporting.
#
# Any issue will only match a rule if ALL statements for
# a rule match the issue.
suppressed:
- reason: "Suppress Internal API issues"
path: /internal/.*
- reason: "Ignore flask debug exceptions"
summary: flask\.debughelpers\.*
An issue suppress rule is composed of regular expressions which match on different issue fields:
suppressed:
- reason: "(required string) Description of why issues matches this rule are suppressed"
method: "(optional regex) GET|PUT|POST|DELETE|PATCH... etc. Matches HTTP Method"
path: "(optional regex) /foo|/tools/.* Matches the HTTP request path"
rule_id: "(optional regex) internal-server-error|auth-bypass|server-crash... Matches mapi rule id"
summary: "(optional regex) matches additional issue summary - such as stacktrace message"
response: "(optional regex) Matches against the entire response, headers and payload. *"
Response matching
If a suppression rule has a response regular expression, that regular
expression will be applied against the entire response from the server.
This is the raw response seen over the TCP connection which starts with
the headers and then proceeds to the payload, if a payload is sent.
You can see something close to the raw output from your server by
accessing it with curl -v.
e.g. curl -v https://google.com:
[...]
< location: https://www.google.com/
< content-type: text/html; charset=UTF-8
< date: Thu, 24 Mar 1970 21:04:22 GMT
< expires: Sat, 23 Apr 1970 21:04:22 GMT
< cache-control: public, max-age=2592000
< server: gws
< content-length: 220
< x-xss-protection: 0
< x-frame-options: SAMEORIGIN
< alt-svc: h3=":443"; ma=2592000,h3-29=":443";
<
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/">here</A>.
</BODY></HTML>
To match against a header in the response you would add a response field like:
server: gws
To match against part of the body, you would add a response field like:
The document has moved
Where to put .mapi file?
💡 We recommended checking the
.mapifile into source control so that you may track changes to suppressed rules with your SCM tool.
The mapi CLI will resolve the .mapi file with the following order of
precedence:
- Set with the
--config <path-to-file>when callingmapi run - Set with the
MAPI_CONFIG_FILEenvironment variable - Located in the working directory where
mapi runis called - Located in the project root (adjacent to the
.gitfolder, if one exists)
How do I see suppressed issues?
mapi will still upload suppressed issues during a run, but they will be
hidden by default. In order to see suppressed issues from the CLI, for instance,
you will need to supply an extra flag --include-suppressed:
▶ mapi issue list <job-id>> --include-suppressed
ID RULE_ID SUMMARY METHOD PATH SUPPRESSED SUPPRESSED_REASON
35717 internal-server-error GET pet/findByTags true Ignore all GETs
35716 internal-server-error GET pet/{petId} true Ignore all GETs
35715 internal-server-error POST user false
35714 internal-server-error POST pet false
Selective Route Testing
By default, Mayhem for API will fuzz every endpoint it finds in the provided spec.
If you need to filter down from this full set of endpoints, for example if:
- you need to exclude a "logout" endpoint because it breaks all subsequent authenticated requests
- you need to divide your spec into subsets with different authentication requirements
- you want to incrementally add routes to mapi as the initial findings in each route are addressed
... or whatever other reason, the mapi command-line tool has you covered with
a number of options.
Using OpenAPI Tags
If you already have an OpenAPI spec with tags at the right granularity to describe the subsets you want to filter on, you're in luck.
Use the --include-endpoints-by-tag <tag> option to specify that only
endpoints with the given tag should be included (implicitly excluding all endpoints which don't have it.)
If you specify multiple tags (by including the option more than
once), they are logically "OR"ed together.
Use the --ignore-endpoints-by-tag <tag> option to specify that endpoints
with the given tag should be explicitly excluded. If you specify multiple
tags, they are logically "OR"ed together.
If you combine the two options, --include is applied first, followed
by --ignore.
Using Endpoint Names
If you don't have tags at the right granularity, you can use powerful (but somewhat more complicated!) endpoint name matching to choose which endpoints to test.
ℹ️ This feature computes a unique "endpoint name" for each endpoint, which is just the HTTP verb and the path, joined with a space, like
GET /pingorPOST /users/{id}!
Use the --include-endpoint <pattern> option to specify that only endpoints whose computed name matches the given pattern should be included (implicitly excluding all endpoints which don't match it.) If you specify
multiple patterns (by including the option more than once), they are logically
"OR"ed together.
Use the --ignore-endpoint <pattern> option to specify that endpoints whose computed
name matches the given pattern should be explicitly excluded. If you specify multiple
patterns, they are logically "OR"ed together.
If you combine the two options, --include is applied first, followed by
--ignore.
ℹ️ The "pattern" used to match endpoint names is a full-blown regular expression, if you need that kind of power.
Zed Attack Proxy (ZAP) Integration
The OWASP Zed Attack Proxy, or ZAP, is an open-source web app scanner, and the de-facto industry standard for automated security scanning of web applications.
ZAP's priorities and approach are quite different from Mayhem for API. We believe they're complementary: if you already run ZAP against your API, you'll get substantially deeper coverage using Mayhem for API, and if you already use Mayhem for API, ZAP is likely to report lots of potential problems, especially configuration and operational problems.
The tools complement each other so well, in fact, that we built a super-easy way to run ZAP's API Scan tool alongside Mayhem for API. Just add a --zap flag to your existing invocation:
mapi run [...] --zap
And you should see ZAP issues reported alongside Mayhem for API issues, like so:
Read on if you want some of the technical details, or more advanced options, but really: that's it. Go try it!
The Fine Print
ZAP vs. ZAP API Scan
It's worth pointing out that the tool we've integrated with, ZAP API Scan, isn't the full ZAP suite: it is a subset of the ZAP suite which, like Mayhem for API, is driven by an API specification. The full ZAP suite looks at web applications as a whole; ZAP API Scan and Mayhem for API are more narrowly focused on API back-ends.
Docker Required!
Our integration with ZAP relies on a working Docker environment (it's just the easiest way, by far, for us to manage all of the transitive dependencies of the ZAP toolchain.) This dependency is specific to the --zap flag, i.e. it's not a requirement for running Mayhem for API in general.
No OpenAPI 3.1 Support
ZAP hasn't yet implemented OpenAPI 3.1 support.
Any other spec format accepted by Mayhem for API (OpenAPI 2.x, OpenAPI 3.0, Postman and HAR) should work, but if your spec is using OAS 3.1 features, ZAP won't work with it until they update their support.
Only Header Auth Supported
While Mayhem for API supports several authentication methods, only --header-auth is currently propagated correctly to ZAP.
Transformations of the Raw ZAP Results
In order to fit the Mayhem for API data model, we transform and filter the raw ZAP results in a couple of ways.
First, ZAP issues are organized differently: there are alerts (roughly analogous to Mayhem for API "rules") and there are one or more "instances" of each alert. Mayhem for API, being API-oriented, likes to organize issues by endpoint. To map ZAP results into our model, therefore, we create an endpoint-associated issue for each distinct "instance" of each ZAP alert.
Second, in our experience, the lowest-risk ZAP alerts had a terrible signal-to-noise ratio when applied specifically to an API. As a result, by default, we filter these out! You can use the --zap_min_risk_code flag to adjust the default, if it's not what you want.
Why isn't --zap the default?
We're committed to making Mayhem for API, out of the box, a tool API developers are happy to use every day, and happy to add as a gating step in their CI/CD pipelines. This means we work very hard only to report issues which we are certain are genuine and resolveable.
ZAP, on the other hand, reports on everything it finds that might be an issue. Their results—while interesting and useful!—tend to require developer triage, which (usually) makes them unusable as a CI/CD gate.
So: we don't include ZAP results by default, but we make it easy for you to get them periodically.
Ingest your own
If you're already running ZAP, and just want to unify your ZAP and Mayhem for API issue reporting, you can also ingest ZAP result json into a run, instead of having Mayhem for API invoke ZAP for you. This is done with the --zap-import-json-results command-line flag.
Organizations
Organizations are used to share your API Targets and Run results with other users. You may belong to any number of Organizations. As an Organization owner, you can invite others to join you and create service accounts to integrate Mayhem for API into your DevOps pipeline.
When you sign up you will be prompted to create a new Organization. You will be the Owner of this Organization.
Authorization
Mayhem for API implements a form of Role Based Access Control (RBAC).
All resources are scoped to an Organization. API Targets, for example, are assigned to one, and only one Organization. In order to view an API Target, you must belong to the same Organization as that API Target (unless the Target has been marked as "public").
Access to Organization resources are controlled with roles.
Owner: Organization Owners have full control over Organizations, Members and API Targets
Member: Organization Members may create API Targets and view Job results, but may not manage other members.
| Action | Owner | Member |
|---|---|---|
| Organization Settings | - | - |
| Create organization invitation | ✔ | |
| Remove member | ✔ | ( ✔ can remove self ) |
| Create Service Accounts | ✔ | |
| Change Member Roles | ✔ | |
| API Target Settings | - | - |
| Create Target | ✔ | ✔ |
| Update Target | ✔ | |
| Delete Target | ✔ | |
| Change Target Settings | ✔ | |
| Change Target access | ✔ | |
| Jobs | - | - |
| Scan an API | ✔ | ✔ |
| View Jobs | ✔ | ✔ |
| Delete Jobs | ✔ | ✔ |
You can change the roles of Organization members with the mapi organization set-role command or from your Organization
Settings page.
Inviting others to join your Organization
From your Browser
As an organization owner, you can send an email invitation to your organization directly
from your browser. Navigate to the Members tab on your Organization page and hit the Invite
button to send an invitation to any email recipient that you wish to join your
Organization.
From the mapi CLI
Organization Owners can create a new invitation with the mapi organization invite command
using the name of their Organization.
Invitations are only valid for one claim and will expire after two days by
default. You can extend the number of claims and duration with the --max-claims and
expiration-min options respectively. For example:
mapi organization invite --max-claims 5 --expiration-min 2880 my-organization
This will present you a message that you can send to whomever you would like to invite to your Organization via Email/Slack/SMS/etc:
A new invitation has been created for organization <your-organization>>!
Up to 1 user(s) can join the organization by going to 'https://mayhem4api.forallsecure.com/join/uGb79...'.
The invitation link is valid for 1 day.
Anyone that accepts your invite will join with the Member role.
Service Accounts
Organization owners may create service accounts for accessing the API as part of automation to avoid using API Tokens from individual accounts. You may create as many service accounts as you require in your organization.
Service accounts do not count toward developer count for the purpose of billing.
To create a new service account, use the mapi CLI:
mapi organization service-account create <my-organization> <service-account-name>
The service account will be added to your organization - and an API token will be presented once. You must record this token as it will not be shown again.
New Service Account named '<service-account-name>' created! A new API token has been
created.
SWOJg...
You can use this token in the environment variable, MAPI_TOKEN, to access
the API Fuzzer API as the Service Account. We will not display this value
again.
You may now use the API token presented in the response to access targets and run fuzzing jobs in your Organization.
Removing a Service Account
To delete a service account (and invalidate its API Token) you must remove the account from your organization. Find the ID of your service account by listing organization members:
mapi organization list-members <my-organization>
ID Name Role
1 foo Owner
2 bar Member
3 <service-account> Member <<<<<
Now remove the service account from your organization:
mapi organization remove-member <my-organization> <service-account-id>
Successfully removed Organization '<my-organization>>' member '3'!
Target Access Control
By default, Targets are only visible to members of their Organization. Organization owners can make targets publicly visible to unauthenticated visitors and members of other organizations by changing Access Control setting of a Target from "Private" to "Public".
CI Integration
We recommend running Mayhem for API as part of your CI pipelines. Adding the fuzzer should take only a few minutes. Running the fuzzer as part of your continuous build will automatically check your API with your latest changes before they are merged and deployed.
Downloading the CLI
The first step is to download the CLI. We recommend using the latest url to
always get the latest release. We aim to retain backwards compatibility for
command line arguments so that your build always works.
Assuming your build is running in Docker, we recommend adding the following lines to add the CLI to your Docker image:
ADD https://mayhem4api.forallsecure.com/downloads/cli/latest/linux-musl/mapi /usr/bin/mapi
RUN chmod 0755 /usr/bin/mapi
This ensures that you get the latest release every time, without affecting the Docker build cache for follow on layers if there hasn't been a new release. This will download the CLI (~25MB) every time you build the image. We recommend adding the line near the end of your build, shortly before you add your code, to minimize the number of uncached layers to rebuild when there is a new CLI release.
Optionally, to avoid re-downloading the CLI every time, we recommend using Docker's BuildKit. BuildKit checks the URL to see if a new version is available, and only downloads it if necessary. This ensures you get the latest release at all times without slowing down your builds with unnecessary downloads. Enable BuildKit with a simple environment variable:
$ DOCKER_BUILDKIT=1 docker build .
Finally, if you happen to use Docker's experimental syntax, you can download the CLI in one command:
ADD --chmod=755 https://mayhem4api.forallsecure.com/downloads/cli/latest/macos/mapi /usr/bin/mapi
Authentication
The CLI requires a token to function. We recommend storing your mapi token in
your CI secret stores, and loading it in the MAPI_TOKEN environment
variable. The CLI will use the value in that environment variable
automatically.
Running the fuzzer
To fuzz your API, you'll have to first start your API locally following our best practices. Once it's up and running, invoke the fuzzer with:
mapi run <target> <duration> <spec-file> --url <url> --junit results.xml
<target>is the name of your api target that you created withmapi target create. The target name is often the name of your project.<duration>depends on how long you want your build to take. We recommendauto, but you may specify a fixed duration such as60secinstead. To maximize fuzzing duration without slowing down the build, we recommend fuzzing in parallel to other build steps, for the duration of those build steps.- If your API requires authentication for some endpoints, consider passing it to the fuzzer for best results.
Note that --url and --junit options are optional. If --url is
unspecified, it will come from the URL you specified in the API target.
--junit is only necessary if you want your CI to parse a JUnit XML to extract
the tests and potential failures (see below).
Gathering the results
The CLI will return an error code if any 500 responses are observed. This can
be used to fail the build if you so choose. In addition, with the --junit
option, the fuzzer will output a JUnit XML file that your CI can parse.
The JUnit will contain a sample request/response for any endpoint that returned a 500, which can be helpful to debug the issue (especially if the API is run in debug mode and the response contains debugging output like stacktraces).
To get a human-friendly report that you can save as a build artifact, use the --html option
to get a report like the following:
Ignoring Errors
You may filter out certain types of response errors that you do not want to appear in
your results using the CLI. For example to prevent 500 Internal Server Error responses
from being marked as issues, include the --ignore-rule InternalServerError flag in your
call to mapi run.
The --ignore-rule flag may be used multiple times to ignore more than one type
of error. By default, ALL detected response errors will be classified as issues.
mapi run <target> <duration> <url> --ignore-rule InternalServerError
💡 To see a full list of
--ignore-ruleoptions, see the output ofmapi run --help.
git Configuration
The Mayhem for API CLI tool, mapi, leverages git if ran within a git repository.
It detects the author, branch, commit SHA, commit message, and the parent commit SHA.
This information is attached to a job and helps Mayhem display useful information about
a job.
Default Branch
The default branch is considered the baseline for comparisons and overall target health.
Typical default branches are main, master or develop. The default branch is set
on the target's settings page.
Jobs ran on the default branch will have a default tag shown on the job's row.
Issues found on the default branch will be replayed on other branches.
Diffs
Mayhem supports two different scenarios for computing the difference between jobs with git context.
Note: To auto-detect diffs, Mayhem needs a non-shallow history.
Default Branch Job
Each default branch job is compared to the last job of the previous commit.
If the previous commit of the default branch does not have a job, because of a pipeline failure or CI configuration, a reason explaining the cause of the missing diff will be shown in place of a number.
For example, a git command to find the previous commit SHA is git rev-parse HEAD~1.
Jenkins
Jenkins can group commits together. Default branch jobs on Jenkins will use
the value of the environment variable GIT_PREVIOUS_SUCCESSFUL_COMMIT if it is available.
Jenkins publishes the available environment variables at your-jenkins.host/env-vars.html.
If GIT_PREVIOUS_SUCCESSFUL_COMMIT is not available, Mayhem will fall back to the logic above.
Branch Job
A job ran on any branch other than the default branch is compared to the last job of the commits merge base with the default branch.
If a merge base commit cannot be found, or a job does not exist for it because of a pipeline failure or CI configuration, a reason explaining the cause of the missing diff will be shown in place of a number.
The example of a git command is git merge-base HEAD origin/main.
Git History
Some CI providers like GitHub Actions fetch a shallow clone of the repository by default. We recommend cloning a full repository if possible. If it is not desirable or possible to clone the full repository, a partial clone of a depth that will capture the merge base is recommended.
If Mayhem cannot detect the parent SHA with the context it is given, it will try to deepen
the repository by 50 commits and retry the SHA detection a single time. This requires that
fetch access to the remote repository is available to git in the CI step.
Identifying the Previous Job
The previous job that was used as a baseline for a diff can be seen by hovering over the diff stats on the job row.
Overriding the Parent SHA
If a project's git workflow is not compatible with the default behaviour, override the parent SHA by supplying it:
mapi run \
<target_name> \
auto \
<path_to_api_specification> \
--url "https://example.com" \
--parent-sha <your_sha>
Troubleshooting
No git branch or commit information
To enable git support for mapi, ensure that it is:
- Ran within the path of a git repository
gitis available on the Path
No diffs
Try adding the shell scripts to your CI provider before executing mapi and verify
that the output is a SHA:
# this should return the SHA when the default branch is checked out
git rev-parse HEAD~1
# this should return a SHA when
git merge-base HEAD origin/main
Azure DevOps Pipeline Integration
The Mayhem for API CLI is built to run on any Continuous Integration platform. This guide will describe how to configure an Azure DevOps Pipeline to run Mayhem for API against your API.
Configure Secrets
You will need an API token to run Mayhem for API in your Azure DevOps pipeline:
- Create a Service Account token for your organization
- Add the newly created token as a "Secret variable" in the pipeline's variables
named
MAPI_TOKEN
Configure your pipeline
This will demonstrate how to run mAPI against an API that built and run in a Azure DevOps pipeline.
steps:
- checkout: self
clean: "true"
# Fetch a deep enough history to compute the merge base of branches with the
# default branch. Alternatively, set to "0" to fetch the entire history.
fetchDepth: "50"
# 1. Build and test your API
# 2. Run the API in debug mode. Output stack traces for better error reporting.
# 3. Run Mayhem for API
- script: |
curl -Lo mapi https://mayhem4api.forallsecure.com/downloads/cli/latest/linux-musl/mapi \
&& chmod +x mapi
./mapi run azure-pipelines-example auto http://localhost:8000/openapi.json \
--url 'http://localhost:8000' \
--junit TEST-mapi.xml \
--html mapi.html
env:
MAPI_TOKEN: $(MAPI_TOKEN)
continueOnError: "true"
displayName: Run Mayhem for API to check for vulnerabilities
- task: PublishPipelineArtifact@1
displayName: 'Publish Pipeline Artifact'
inputs:
path: 'mapi.html'
artifact: mapi-html-report
continueOnError: "true"
- task: PublishTestResults@2
GitHub Example
A full working example is available on GitHub.
Circle CI
Add the Mayhem for API CircleCI Orb to
your .circleci/config.yml file.
version: 2.1
orbs:
mapi: forallsecure/mapi@1.0.0
Pipeline Hosted Service
Create a new job to scan your API and start your service. Then call the mapi/scan command to
run Mayhem against your service.
jobs:
mayhem-for-api:
machine:
image: ubuntu-2204:2022.07.1
steps:
# Start your service
- run:
command: start-service.sh &
# Scan your API with Mayhem for API
- mapi/scan:
api-url: "http://localhost:8000"
api-spec: "https://demo-api.mayhem4api.forallsecure.com/api/v3/openapi.json"
- store_artifacts:
path: /tmp/mapi
- store_test_results:
path: /tmp/mapi/junit.xml
Then add the new job to your workflow.
workflows:
tests-and-security:
jobs:
- mayhem-for-api
Deployed Service
To test a service that is already deployed and not running as part of the build pipeline,
use the mapi/scan job in your workflow. Mayhem should never be executed against production environments.
workflows:
tests-and-security:
jobs:
- mapi/scan:
api-url: "https://your.staging.com/"
api-spec: "https://your.staging.com/openapi.json"
GitHub Integration
Mayhem for API comes with a GitHub Action and a GitHub App to help you check every change to your API with Mayhem for API. Our CLI can also upload Mayhem for API results to GitHub Code Scanning from any CI.
GitHub Action
With our GiHub Action, you'll get Mayhem for API testing every API change in no time.
To integrate Mayhem for API into GitHub Actions workflows:
- Create a Service Account token for your organization
- Add the newly created token to your GitHub secrets.
- Create a workflow that starts your API & invoke the Mayhem for API action.
Configure your workflow
At the base directory of your code repository, add a .github/workflows/mapi.yml file to configure GitHub Actions to run Mayhem for API. Your file should look like this.
name: Mayhem for API
on:
push:
branches: [ main, master ]
pull_request:
jobs:
security:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
with:
# fetch a greater number of commits for computing diffs between jobs
fetch-depth: 50
- name: Start your API
run: ./run_your_api.sh &
# ^-------------- update this, but keep the '&' to run
# the API in the background.
- name: Run Mayhem for API to check for vulnerabilities
uses: ForAllSecure/mapi-action@v1
with:
mapi-token: ${{ secrets.MAPI_TOKEN }}
api-url: http://localhost:8000 # <- update this
api-spec: your-openapi-spec-or-postman-collection.json # <- update this
This configuration tells GitHub Actions to run a workflow which checks
out your code, starts your API, and then runs Mayhem for API against
your API. We pass the Mayhem for API token from the GitHub secret,
MAPI_TOKEN.
Here's a fully working example repository showing you an ideal integration that you can use as a template. For more details on Mayhem for API Action, visit the Mayem for API Action in the GitHub Actions marketplace.
GitHub Code Scanning
If you are testing a public repository, or a private repository on a GitHub Enterprise plan, we recommend generating a SARIF report. Uploading SARIF reports to GitHub allows you to see any issue found by Mayhem for API in the "Security" tab of your repository. In addition, if your API is sending back stacktraces as part of the error response body, Mayhem for API will show you exactly where in your code the issue happened, right in your PR diffs:
Using GitHub action
To do so with our GitHub Action, after starting your API in the
workflow, run the Mayhem for API action followed by
github/codeql-action/upload-sarif@v1 to upload the report. Note
continue-on-error needs to be set to true in the Mayhem for API action
in order to run the next step, even if Mayhem for API finds issues.
- name: Run Mayhem for API to check for vulnerabilities
uses: ForAllSecure/mapi-action@v1
continue-on-error: true
with:
mapi-token: ${{ secrets.MAPI_TOKEN }}
api-url: http://localhost:8000 # <- update this
api-spec: your-openapi-spec-or-postman-collection.json # <- update this
sarif-report: mapi.sarif
# Upload SARIF file (only available on public repos or github enterprise)
- name: Upload SARIF file
uses: github/codeql-action/upload-sarif@v1
with:
sarif_file: mapi.sarif
Using our CLI
To upload our results to GitHub code scanning from anywhere, locally or
from a CI, you can use our CLI. You'll need to pass a GitHub token with
the security_event scope so that Mayhem for API can upload findings.
See GitHub documentation on creating access
tokens
for more information. Once you have your token, simply run:
mapi run "<target>" "<duration>" "<spec>" --sarif "mapi.sarif" --github-token "<github-token>"
Mayhem for API will attempt to infer the git information from the environment set by the CI, and the git repository on the filesystem. If those are not available, Mayhem for API will ask you to pass the missing information so that it can upload the findings to the right place.
In addition, Mayhem for API also supports GitHub Enterprise on premise!
Just pass --github-api-url <your-github-api-url> to mapi run so that
Mayhem for API knows where to upload the results. Your GitHub instance
needs to be accessible from the host running our CLI, but it does not
need to be accessible by our cloud infrastructure.
GitHub Application
Installing the Mayhem for API GitHub App into your Personal or Organization GitHub account provides additional benefits by automatically updating the checks for your builds and pull requests with the result of your API Fuzzer jobs.
Install the GitHub app into your Personal or Organization repository by following the direct link:
https://github.com/apps/mayhem-for-api
You must select which repositories you would like Mayhem for API to have access to. Once you have decided and authorized the app, we will attempt to automatically detect GitHub builds whenever you run the CLI as part of your build. We do this by matching your git repository commit hash/branch/remote with repositories which you have given our app access.
If we cannot determine your repository settings, you may set them manually
when calling mapi run with additional options such as --branch,
--remote and --sha. See mapi run --help for a full listing.
To see an example of the GitHub App in action, please visit our example repository:
https://github.com/ForAllSecure/api-fuzzer-swagger-petstore-demo
Jenkins Integration
The Mayhem for API CLI or Docker Image is built to run on any Continuous Integration platform. "Jenkins pipelines" are the most common way to build and test your software on Jenkins today. This guide will describe how to configure a Jenkins Pipeline to run Mayhem for API against your API.
Configure Secrets
You will need an API token to run Mayhem for API in your Jenkins pipeline:
- Create a Service Account token for your organization
- Add the newly created token as a "Secret text" entry in Jenkins Credentials
named
MAPI_TOKEN
Configure your pipeline
This will demonstrate how to run mAPI against an API that built and run in a Jenkins scripted pipeline.
Run with Docker
In this scripted pipeline, after building and testing your API,
a local instance is started and scanned by "Mayhem for API",
using the Docker Hub mapi
image.
// Run the build on a node with the 'docker' label
node("docker") {
checkout scm
// MAPI_TOKEN - The API Token secret text added to Credentials
withCredentials([
string(credentialsId: "${MAPI_TOKEN}", variable: "MAPI_TOKEN")
]) {
//
// 1. BUILD AND TEST YOUR API HERE
//
stage("Run Mayhem for API") {
//
// 2. Start your API
// eg. http://localhost:8080/api
//
//
// 3. Run Mayhem for API
//
sh '''
docker run -t --rm \
--network=host \
-e MAPI_TOKEN=${MAPI_TOKEN} \
-e NO_COLOR=true \
forallsecure/mapi:latest \
run my-api auto <path_to_openapi_spec> \
--url 'http://localhost:8080/api' \
--junit results.xml
'''
//
// 4. Collect junit results
//
junit testResults: 'results.xml'
}
}
}
Run with CLI
This example is nearly identical as the one above, except we
will use the compiled mapi CLI, rather than a Docker container
to run the scan.
It is recommended to take advantage of Jenkins Tool Installer support for the most robust setup, but for simplicity we will download the CLI as part of our pipeline.
// Run the build on a node with the 'docker' label
node("docker") {
checkout scm
// MAPI_TOKEN - The API Token secret text added to Credentials
withCredentials([
string(credentialsId: "${MAPI_TOKEN}", variable: "MAPI_TOKEN")
]) {
//
// 1. BUILD AND TEST YOUR API HERE
//
stage("Run Mayhem for API") {
//
// 2. Start your API
// eg. http://localhost:8080/api
//
//
// 3. Download the CLI (or use Jenkins Tools)
//
sh '''
curl -Lo mapi https://mayhem4api.forallsecure.com/downloads/cli/latest/linux-musl/mapi \
&& chmod +x mapi
'''
//
// 4. Run Mayhem for API
//
sh '''
mapi run my-api auto <path_to_openapi_spec> \
--url 'http://localhost:8080/api' \
--junit results.xml
'''
//
// 5. Collect junit results
//
junit testResults: 'results.xml'
}
}
}
Best Practices & Troubleshooting
⚠️ Please do not attempt to fuzz any applications that are not owned or managed by you. The fuzz testing will look like an attack at worst and a probe at best to any organization that's not aware of what's happening. Please make sure to only fuzz your API assets. Thanks!
Here are some recommendations to maximize the value of fuzzing for your API:
-
Avoid running fuzzing against production services, as it can be HIGHLY disruptive. It generates a fair amount of load, and might delete resources you do not want deleted. We recommend using it on a QA or staging instance.
-
Populating the service with some realistic data will help the fuzzer uncover more issues.
-
Since the fuzzer may create many new entities in the service under state, resetting the state between runs will prevent the API from being bogged down by previous fuzzing state.
Troubleshooting
CLI Logging
The mapi CLI will print info level logs by default. You can control the
logging verbosity in order to get more details about what the CLI os doing
with the --verbosity flag.
For example, to log debug output, include the --verbosity debug flag before
your subcommand:
Must come BEFORE subcommand
v
mapi --verbosity debug run ...
^
subcommand
You can also use the MAPI_LOG environment variable for the same effect.
MAPI_LOG=debug mapi run ...
Logging levels (not case sensitive) include:
off
critical
error
warn
info
debug
trace
Logging with trace level includes trace, debug, info, and higher.
Logging with debug level includes debug, info, and higher.
... etc
Jobs
Looking at the job status using the CLI can help you understand what the fuzzer is doing, and if it's running into any issue.
-
The job logs will contain errors, if any. We aim to provide actionable feedback, making it obvious how to resolve the issue. For instance, the job log might contain messages about not being able to connect to the target API.
-
The endpoint coverage table can tell you a lot about what the fuzzer is able to exercise. Ideally you'd see all endpoints having 2xx response codes. If you don't, this means that the fuzzer could not generate a valid request. If you only see 401 unauthorized responses, then the fuzzer is having trouble with authentication. If you only see 400 response codes, that means the fuzzer could not generate a valid input for this endpoint, maybe because it was too complex to be generated by our fuzzer, or maybe because the specification did not match the behavior. If you're not sure why the fuzzer fails to generate 2xx responses, please reach out and we'll look into it and make the fuzzer better.
-
Sometimes, the job log will contain an error due to something on our end. Please feel free to reach out to support@forallsecure.com if you run into one.
-
If you have a Swagger 2.0 specification, we will attempt to convert it to OpenAPI 3 for you. If this fails, you can also try using the
swagger2openapicommand line tool:npm install -g swagger2openapi swagger2openapi -p -y swagger-spec.json > openapi-spec.yaml -
Consider specifying a Postman API key and collection ID instead of an exported Postman file for better fuzzing and more features. If you are using an exported Postman 2.x specification, we will attempt to convert it to OpenAPI 3 for you. If this fails, you can also try using postman-to-openapi.
Tracing Issues
Issues can be traced back to the specification. OpenAPI operationId, summary and tags are attached to issues. Likewise, Postman request ID, request name and folder are attached to issues.
In addition, please feel free to reach out to us on Discord or to support@forallsecure.com if you have any issues, feedback of feature requests. We are happy to help!
HTTP Proxy Configuration
If your target is behind a proxy, or you require a proxy to reach the
API for Mayhem for API, you can provide proxy information for the
mapi CLI. Support for common proxy environment variables follows:
| Environment Variable | Description |
|---|---|
HTTP_PROXY | The address to use when initiating HTTP connection(s) |
HTTPS_PROXY | The address to use when initiating HTTPS connection(s) |
NO_PROXY | Network address(es)/range(s) and domains to exclude from proxying |
Examples
Say you wish to capture all the communication between the fuzzer and your API. You
can place a proxy between Mayhem for API and your API such as
Charles Proxy (which listens on localhost:8888 by
default) to intercept and record all traffic.
HTTP_PROXY=http://localhost:8888 mapi run <target> ...
If your target is an https, you can use the HTTPS_PROXY variable:
HTTPS_PROXY=https://localhost:8888 mapi run <target> ...
If your proxy server requires authentication with username and password, you must include the credentials:
https_proxy=http://username:password@localhost:8888 mapi run <target> ...
For SOCKS5 proxy servers, you must specify the socks5 protocol in the
URL:
https_proxy=socks5://socksproxy.example.com
Proxying https traffic may result in some certificate errors when the
CLI attempts to communicate with the API for Mayhem for API. This can be
resolved by ignoring certain addresses from being sent to the proxy with
the NO_PROXY variable. For example:
HTTPS_PROXY=https://localhost:8888 \
NO_PROXY=mayhem4api.forallsecure.com \
mapi run <target> ...
The 'Mayhem for API' API
The Mayhem for API is available in OpenApi 3.0 format and can be interacted with using the online Swagger Editor.
Please note: this is an unstable, pre-release API under active development. If you want to know more, or if you want to build against it, we'd like to hear from you! Please get in touch by emailing us at support@forallsecure.com.
Authentication
An API Token is required to access the 'Mayhem for API' API. The token must be
presented as a Bearer token in the Authorization header with every request
in order to access your resources.
For example, to fetch a list of API Targets directly from the API with curl
with your API Token <YOUR_TOKEN>:
curl --request GET 'https://mayhem4api.forallsecure.com/api/v1/api-target' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer <YOUR_TOKEN>'
gRPC Fuzzing
While Mayhem for API does not currently support native fuzzing of gRPC servers, it does support gRPC servers fuzzing through the use of a gRPC gateway. The gateway comes with a protoc plugin. It reads a gRPC service definition and generates a reverse-proxy server which translates a RESTful JSON API into gRPC.
This guide will walk you through the steps to fuzz a gRPC server. While this method can be used with any language, the guide assumes you're using go. If you're not and are having trouble following the steps for another language, let us know, we'll be happy to help out.
Live Example
A live example that follows this process is available in GitHub:
https://github.com/ForAllSecure/mapi-grpc-example
Installing gRPC-Gateway
Follow the instructions in the README. You'll need to have go and protoc installed.
Generate Server Stub
The following command generates the proxy stubs from your .proto file:
protoc -I . --grpc-gateway_out . \
--grpc-gateway_opt logtostderr=true \
--grpc-gateway_opt paths=source_relative \
--grpc-gateway_opt generate_unbound_methods=true \
your/service/v1/your_service.proto
The command above generates the gateway stub at
your/service/v1/your_service.pb.gw.go. We will import the stubs to write the
proxy.
Write an entrypoint for the proxy
We will now write the proxy in rest-proxy.go. The proxy will listen on port
8081 and forward requests to your gRPC server.
package main
import (
"context"
"flag"
"net/http"
"github.com/golang/glog"
"github.com/grpc-ecosystem/grpc-gateway/v2/runtime"
"google.golang.org/grpc"
gw "github.com/your-org/your-project/your/service/v1"
)
var (
// command-line options:
// gRPC server endpoint
grpcServerEndpoint = flag.String("grpc-server-endpoint", "localhost:10990", "gRPC server endpoint")
)
func run() error {
ctx := context.Background()
ctx, cancel := context.WithCancel(ctx)
defer cancel()
// Register gRPC server endpoint
// Note: Make sure the gRPC server is running properly and accessible
mux := runtime.NewServeMux()
opts := []grpc.DialOption{grpc.WithInsecure()}
err := gw.RegisterYourServiceHandlerFromEndpoint(ctx, mux, *grpcServerEndpoint, opts)
if err != nil {
return err
}
// Start HTTP server (and proxy calls to gRPC server endpoint)
return http.ListenAndServe(":8081", mux)
}
func main() {
flag.Parse()
defer glog.Flush()
if err := run(); err != nil {
glog.Fatal(err)
}
}
You will need to change three things:
- The import line
"github.com/your-org/your-project/your/service/v1"to match your project and the path to the stub you generated in the previous step. - The url to your gRPC server. The code above assumes your gRPC server is
listening on
localhost:10990. - The call to
gw.RegisterYourServiceHandlerFromEndpointto match your service name. The function should be present in the gateway stub you generated earlier if you're not sure what the name should be.
Generate the OpenAPI Specification
The following command generates the OpenAPI specification:
protoc -I . --openapiv2_out . \
--openapiv2_opt logtostderr=true \
--openapiv2_opt generate_unbound_methods=true \
your/service/v1/your_service.proto
Fuzz
- Start your gRPC server
- Start the proxy with
go run rest-proxy.go - Create a target:
mapi target create target-name http://localhost:8081 - Fuzz:
mapi run target-name 120 your/service/v1/your_service.openapi.json
Request Rewrite Plugins
Mayhem for API has support for request rewriting via plugins, which are standalone programs that implement a protobuf interface and expose it via gRPC.
The interface details are below. But first...
Do I Need This?
We've tried to make Mayhem for API work out of the box for as many use-cases as we can, and we'll keep adding more. The request rewrite plugin is intended as a kind of escape-hatch for situations where an API has requirements that we can't currently meet out of the box.
Here are the cases we had in mind when building the feature:
Just In Time Authentication
In any situation where the authentication needs to change during a fuzzing job, we don't currently have a solution directly in Mayhem for API.
A request rewrite plugin can inject the appropriate credentials at whatever cadence your API requires.
Request Digesting/Signing
If your API uses any kind of digesting or signing as authentication measures, Mayhem for API will be unable to generate valid requests.
A request rewrite plugin can compute and inject these per-request checksums or signatures.
Something else?
The request plugin interface is extremely open-ended, and can rewrite just about any part of any request.
If you're thinking about using a rewrite plugin for any reason, please get in touch. This is a very young, very raw feature, and we'd like to know how people are using it, and how we can make it better!
Interface
Invocation
To add a request rewrite plugin to a fuzzing run, use the
--rewrite-plugin <path-to-plugin-executable> argument to mapi run.
Executable gRPC Server
mapi will run the executable with no command-line arguments, and expects it to be a long-lived process that:
- listens for gRPC traffic on a localhost port
- prints the port number, followed by a newline, to stdout as the first line
Beyond that, it can do whatever it wants. Additional stdout/stderr output from the plugin are forwarded to the mapi debug logs.
At the end of the fuzzing run, mapi will attempt to kill the plugin process.
Protobuf
Request rewrite plugins implement the following protobuf interface:
syntax = "proto3";
package mapi.rewrite;
service RewritePlugin {
rpc Rewrite(Request) returns (Request);
}
message Request {
message Header {
bytes name = 1;
bytes value = 2;
}
string url = 1;
repeated Header headers = 2;
bytes body = 3;
}
Example
Here's an example request rewrite plugin, which does... absolutely nothing, just returning the original request unchanged!
First, generate all of the gRPC and protobuf boilerplate from the .proto above:
$ pip install grpcio-tools
$ python -m grpc_tools.protoc --python_out=. --grpc_python_out=. -I. request-rewrite-plugin.proto
Then create a python program which implements the RewritePluginServicer, stands
up a gRPC server, and lets mapi know what port it's listening on:
#!/usr/bin/env python
import sys
from concurrent import futures
import grpc
import request_rewrite_plugin_pb2
import request_rewrite_plugin_pb2_grpc
## implement the RewritePlugin interface
class RewritePluginServicer(request_rewrite_plugin_pb2_grpc.RewritePluginServicer):
def Rewrite(self, request, context):
## well, this isn't very interesting :)
return request
if __name__ == '__main__':
## boot up the gRPC server
server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
request_rewrite_plugin_pb2_grpc.add_RewritePluginServicer_to_server(
RewritePluginServicer(), server)
server.add_insecure_port('127.0.0.1:50051')
server.start()
## inform mapi of the port we're listening on
print("50051")
sys.stdout.flush()
server.wait_for_termination()
Make sure your .py file is executable, then pass it to mapi via
--rewrite-plugin. That's it!
More Examples
Please visit our GitHub Repository for more examples that you can try yourself!
Caveats and Cautions
bytes, Not string!
Mayhem for API requests are not always representable as strings. Be careful!
Subject to Change
All of this is volatile and subject to change. If you're using this feature, please get in touch, so that we can keep you in the loop as we consider refinements and changes.
Response Classify Plugins
Mayhem for API has support for response classification via plugins, which are standalone programs that implement a protobuf interface and expose it via gRPC.
The interface details are below. But first...
Do I Need This?
We've tried to make Mayhem for API work out of the box for as many use-cases as we can, and we'll keep adding more. These plugins are intended as a kind of escape-hatch for situations where an API has requirements that we can't currently meet out of the box.
Here are the cases we had in mind when building the feature:
Sensitive Data Leaking
If there are certain keys, values or patterns that you want to assert should never show up in your API output, regardless of inputs, Mayhem for API doesn't currently have a direct solution.
A response classify plugin can make these assertions.
Nonstandard Response Code Assertions
The rules Mayhem for API follows based on each endpoint's response codes are baked in, and based on standards. There may be specific endpoints where you want to apply more strict constraints—for example, an endpoint whose contract is that it never returns a 400 error.
A response classify plugin can be used to check for this kind of constraint.
Something else?
The classifier plugin interface is extremely open-ended, and can be used to check just about any aspect of a response.
If you're thinking about using a classify plugin for any reason, please get in touch. This is a very young, very raw feature, and we'd like to know how people are using it, and how we can make it better!
Interface
Invocation
To add a response classify plugin to a fuzzing run, use the
--classify-plugin <path-to-plugin-executable> argument to
mapi run.
Executable gRPC Server
mapi will run the executable with no command-line arguments, and expects it to be a long-lived process that:
- listens for gRPC traffic on a localhost port
- prints the port number, followed by a newline, to stdout as the first line
Beyond that, it can do whatever it wants. Additional stdout/stderr output from the plugin are forwarded to the mapi debug logs.
At the end of the fuzzing run, mapi will attempt to kill the plugin process.
Protobuf
Response classify plugins implement the following protobuf interface:
syntax = "proto3";
package mapi.classify;
service ClassifyPlugin {
rpc Classify(Response) returns (Issues);
}
message Response {
message Header {
bytes name = 1;
bytes value = 2;
}
string request_url = 1;
string final_url = 2;
uint32 status = 3;
repeated Header headers = 4;
bytes body = 5;
}
message Issues {
message Issue {
string summary = 1;
}
repeated Issue issues = 1;
}
Example
Here's an example classifier plugin, which does... absolutely nothing, just acknowledging each response without flagging any issues.
First, generate all of the gRPC and protobuf boilerplate from the .proto above:
$ pip install grpcio-tools
$ python -m grpc_tools.protoc --python_out=. --grpc_python_out=. -I. response-classify-plugin.proto
Then create a python program which implements the
ClassifyPluginServicer, stands up a gRPC server, and lets mapi know what
port it's listening on:
#!/usr/bin/env python
import sys
from concurrent import futures
import grpc
import response_classify_plugin_pb2
import response_classify_plugin_pb2_grpc
## implement the ClassifyPlugin interface
class ClassifyPluginServicer(response_classify_plugin_pb2_grpc.ClassifyPluginServicer):
def Classify(self, response, context):
issues = response_classify_plugin_pb2.Issues()
if something_went_wrong():
issue = response_classify_plugin_pb2.Issues.Issue()
issue.summary = "something went wrong!"
issues.issues.append(issue)
return issues
if __name__ == '__main__':
## boot up the gRPC server
server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
response_classify_plugin_pb2_grpc.add_ClassifyPluginServicer_to_server(
ClassifyPluginServicer(), server)
server.add_insecure_port('127.0.0.1:50051')
server.start()
## inform mapi of the port we're listening on
print("50051")
sys.stdout.flush()
server.wait_for_termination()
Make sure your .py file is executable, then pass it to mapi via
--classify-plugin. That's it!
Caveats and Cautions
bytes, Not string!
API responses are not always representable as strings. Be careful!
Subject to Change
All of this is volatile and subject to change. If you're using this feature, please get in touch, so that we can keep you in the loop as we consider refinements and changes.
Open Source Notices
Mayhem For API ❤️ open-source software! We rely on a number of open-source dependencies, and we are grateful for the work all the maintainers are putting into them. To give back, we regularly upstream any improvements we make.
Here are the copyright notices of our CLI dependencies:
- addr2line: MIT
- adler: MIT
- adler32: Zlib
- ahash: MIT
- ahash: MIT
- android_system_properties: MIT
- ansi_term: MIT
- anyhow: MIT
- apollo-encoder: MIT
- apollo-parser: MIT
- arc-swap: MIT
- arrayvec: MIT
- askama: MIT
- askama_derive: MIT
- askama_escape: MIT
- askama_shared: MIT
- async-compression: MIT
- async-stream: MIT
- async-stream-impl: MIT
- async-trait: MIT
- atty: MIT
- backtrace: MIT
- base64: MIT
- base64: MIT
- bit-set: MIT
- bit-vec: MIT
- bitflags: MIT
- bitvec: MIT
- block-buffer: MIT
- bstr: MIT
- bumpalo: MIT
- bytecount: MIT
- bytes: MIT
- card-validate: MIT
- cassowary: MIT
- cesu8: MIT
- cfg-if: MIT
- cfg-if: MIT
- chrono: MIT
- chrono-humanize: MIT
- clap: MIT
- cli-table: MIT
- cli-table-derive: MIT
- combine: MIT
- confy: MIT
- console: MIT
- convert_case: MIT
- convert_case: MIT
- cookie: MIT
- cookie_store: MIT
- core-foundation: MIT
- core-foundation-sys: MIT
- countme: MIT
- cpufeatures: MIT
- crc32fast: MIT
- crossbeam-channel: MIT
- crossbeam-deque: MIT
- crossbeam-epoch: MIT
- crossbeam-utils: MIT
- crossterm: MIT
- crossterm_winapi: MIT
- ctrlc: MIT
- cxx: MIT
- cxxbridge-macro: MIT
- darling: MIT
- darling: MIT
- darling_core: MIT
- darling_core: MIT
- darling_macro: MIT
- darling_macro: MIT
- debugid: Apache-2.0
- deflate: MIT
- derive-getters: MIT
- derive_more: MIT
- deunicode: BSD-3-Clause
- dialoguer: MIT
- digest: MIT
- directories: MIT
- dirs-next: MIT
- dirs-sys: MIT
- dirs-sys-next: MIT
- dns-lookup: MIT
- dyn-clone: MIT
- either: MIT
- encode_unicode: MIT
- encoding_rs: Apache-2.0
- erased-serde: MIT
- errno: MIT
- errno-dragonfly: MIT
- error-chain: MIT
- faccess: MIT
- fake: MIT
- fancy-regex: MIT
- fastrand: MIT
- flashtext: MIT
- flate2: MIT
- fnv: MIT
- foreign-types: MIT
- foreign-types-shared: MIT
- form_urlencoded: MIT
- fraction: MIT
- funty: MIT
- futures-channel: MIT
- futures-core: MIT
- futures-io: MIT
- futures-macro: MIT
- futures-sink: MIT
- futures-task: MIT
- futures-util: MIT
- generic-array: MIT
- gethostname: Apache-2.0
- getrandom: MIT
- gimli: MIT
- git2: MIT
- gzip-header: MIT
- h2: MIT
- har: MIT
- hashbrown: MIT
- hashconsing: MIT
- heck: MIT
- hermit-abi: MIT
- hermit-abi: MIT
- hex: MIT
- hostname: MIT
- http: MIT
- http-body: MIT
- httparse: MIT
- httpdate: MIT
- humansize: MIT
- humantime: MIT
- hyper: MIT
- hyper-rustls: MIT
- hyper-timeout: MIT
- hyper-tls: MIT
- iana-time-zone: MIT
- iana-time-zone-haiku: MIT
- ident_case: MIT
- idna: MIT
- idna: MIT
- idna: MIT
- indexmap: MIT
- indicatif: MIT
- instant: BSD-3-Clause
- io-lifetimes: MIT
- ipnet: MIT
- iso8601: MIT
- itertools: MIT
- itoa: MIT
- itoa: MIT
- jni: MIT
- jni-sys: MIT
- js-sys: MIT
- jsonschema: MIT
- junit-report: MIT
- lazy_static: MIT
- lexical-core: MIT
- libc: MIT
- libgit2-sys: MIT
- libm: MIT
- libz-sys: MIT
- link-cplusplus: MIT
- linked-hash-map: MIT
- linux-raw-sys: MIT
- lock_api: MIT
- log: MIT
- match_cfg: MIT
- matchers: MIT
- matches: MIT
- memoffset: MIT
- memoffset: MIT
- mime: MIT
- mime_guess: MIT
- minidom: MPL-2.0
- minimal-lexical: MIT
- miniz_oxide: MIT
- mio: MIT
- mio: MIT
- miow: MIT
- native-tls: MIT
- ndk: MIT
- ndk-context: MIT
- ndk-glue: MIT
- ndk-macro: MIT
- ndk-sys: MIT
- nix: MIT
- nix: MIT
- nom: MIT
- nom: MIT
- ntapi: MIT
- ntapi: MIT
- nu-ansi-term: MIT
- num: MIT
- num-bigint: MIT
- num-cmp: MIT
- num-complex: MIT
- num-integer: MIT
- num-iter: MIT
- num-rational: MIT
- num-traits: MIT
- num_cpus: MIT
- num_enum: MIT
- num_enum_derive: MIT
- num_threads: MIT
- number_prefix: MIT
- object: MIT
- once_cell: MIT
- opaque-debug: MIT
- openapiv3: MIT
- openssl: Apache-2.0
- openssl-macros: MIT
- openssl-probe: MIT
- openssl-sys: MIT
- overload: MIT
- parking_lot: MIT
- parking_lot: MIT
- parking_lot_core: MIT
- parking_lot_core: MIT
- parse-size: MIT
- percent-encoding: MIT
- percent-encoding: MIT
- pest: MIT
- pin-project: MIT
- pin-project-internal: MIT
- pin-project-lite: MIT
- pin-utils: MIT
- postman2openapi: Apache-2.0
- postman_collection: MIT
- ppv-lite86: MIT
- proc-macro-crate: MIT
- proc-macro-error: MIT
- proc-macro-error-attr: MIT
- proc-macro-hack: MIT
- proc-macro2: MIT
- proptest: MIT
- prost: Apache-2.0
- prost-derive: Apache-2.0
- psl-types: MIT
- publicsuffix: MIT
- querystring: MIT
- quick-error: MIT
- quick-xml: MIT
- quote: MIT
- radium: MIT
- rand: MIT
- rand_chacha: MIT
- rand_core: MIT
- rand_distr: MIT
- rand_hc: MIT
- rand_regex: MIT
- rand_xorshift: MIT
- rayon: MIT
- rayon-core: MIT
- redox_syscall: MIT
- redox_users: MIT
- regex: MIT
- regex-syntax: MIT
- reqwest: MIT
- rowan: MIT
- rustc-demangle: MIT
- rustc-hash: MIT
- rustix: MIT
- rustls: MIT
- rustls-pemfile: MIT
- rustversion: MIT
- ryu: Apache-2.0
- schannel: MIT
- schemars: MIT
- schemars_derive: MIT
- scopeguard: MIT
- sct: MIT
- security-framework: MIT
- security-framework-sys: MIT
- self_update: MIT
- semver: MIT
- semver: MIT
- semver-parser: MIT
- sentry: Apache-2.0
- sentry-backtrace: Apache-2.0
- sentry-contexts: Apache-2.0
- sentry-core: Apache-2.0
- sentry-panic: Apache-2.0
- sentry-tracing: Apache-2.0
- sentry-types: Apache-2.0
- serde: MIT
- serde_derive: MIT
- serde_derive_internals: MIT
- serde_json: MIT
- serde_regex: MIT
- serde_urlencoded: MIT
- serde_with: MIT
- serde_with: MIT
- serde_with_macros: MIT
- serde_with_macros: MIT
- serde_yaml: MIT
- serde_yaml: MIT
- sha2: MIT
- sharded-slab: MIT
- shell-words: MIT
- signal-hook: MIT
- signal-hook-registry: MIT
- slab: MIT
- slog: MIT
- slog-async: MIT
- slog-envlogger: MIT
- slog-scope: MIT
- slog-stdlog: MIT
- slog-term: MIT
- smallvec: MIT
- socket2: MIT
- spin: MIT
- static_assertions: MIT
- strip-ansi-escapes: MIT
- strsim: MIT
- strsim: MIT
- structopt: MIT
- structopt-derive: MIT
- strum: MIT
- strum_macros: MIT
- syn: MIT
- sysinfo: MIT
- take_mut: MIT
- tap: MIT
- tempfile: MIT
- term: MIT
- term_size: MIT
- text-size: MIT
- textwrap: MIT
- thiserror: MIT
- thiserror-impl: MIT
- thread_local: MIT
- time: MIT
- time: MIT
- time-core: MIT
- time-macros: MIT
- tinyvec: MIT
- tinyvec_macros: MIT
- tokio: MIT
- tokio-io-timeout: MIT
- tokio-macros: MIT
- tokio-native-tls: MIT
- tokio-rustls: MIT
- tokio-socks: MIT
- tokio-stream: MIT
- tokio-util: MIT
- tokio-util: MIT
- toml: MIT
- tonic: MIT
- tower: MIT
- tower-layer: MIT
- tower-service: MIT
- tracing: MIT
- tracing-attributes: MIT
- tracing-core: MIT
- tracing-futures: MIT
- tracing-log: MIT
- tracing-subscriber: MIT
- try-lock: MIT
- tui: MIT
- typenum: MIT
- ucd-trie: MIT
- uname: MIT
- unicase: MIT
- unicode-bidi: MIT
- unicode-ident: MIT
- unicode-normalization: MIT
- unicode-segmentation: MIT
- unicode-width: MIT
- unsafe-libyaml: MIT
- untrusted: ISC
- url: MIT
- url: MIT
- url_serde: MIT
- urlencoding: MIT
- utf8parse: MIT
- uuid: MIT
- uuid: MIT
- valuable: MIT
- vec_map: MIT
- vte: MIT
- vte_generate_state_changes: MIT
- want: MIT
- wasi: MIT
- wasi: MIT
- wasm-bindgen: MIT
- wasm-bindgen-backend: MIT
- wasm-bindgen-futures: MIT
- wasm-bindgen-macro: MIT
- wasm-bindgen-macro-support: MIT
- wasm-bindgen-shared: MIT
- web-sys: MIT
- webbrowser: MIT
- webpki-roots: MPL-2.0
- which: MIT
- widestring: MIT
- winapi: MIT
- winapi-i686-pc-windows-gnu: MIT
- winapi-x86_64-pc-windows-gnu: MIT
- windows-sys: MIT
- windows-sys: MIT
- windows-targets: MIT
- windows_aarch64_gnullvm: MIT
- windows_aarch64_msvc: MIT
- windows_i686_gnu: MIT
- windows_i686_msvc: MIT
- windows_x86_64_gnu: MIT
- windows_x86_64_gnullvm: MIT
- windows_x86_64_msvc: MIT
- winreg: MIT
- wyz: MIT
- xml-rs: MIT
- yaml-rust: MIT
- zeroize: MIT
- zip: MIT
Here are the copyright notices of our Frontend dependencies:
- @chakra-ui/react: MIT
- @emotion/react: MIT
- @emotion/styled: MIT
- @godaddy/terminus: MIT
- @heroicons/react: MIT
- @oclif/command: MIT
- @reduxjs/toolkit: MIT
- @sentry/node: BSD-3-Clause
- @sentry/react: MIT
- @sentry/tracing: MIT
- @tailwindcss/forms: MIT
- @tailwindcss/typography: MIT
- @types/js-md5: MIT
- @types/lodash.debounce: MIT
- @types/lodash.flatmap: MIT
- @types/lodash.sortby: MIT
- autoprefixer: MIT
- avantation: MIT
- axios-retry: Apache-2.0
- axios: MIT
- classnames: MIT
- compression: MIT
- core-js: MIT
- date-fns: MIT
- dompurify: Apache-2.0
- express: MIT
- framer-motion: MIT
- git-url-parse: MIT
- graphql: MIT
- highlight.js: BSD-3-Clause
- http-proxy-middleware: MIT
- immer: MIT
- js-md5: MIT
- keycloak-js: Apache-2.0
- lodash.debounce: MIT
- lodash.flatmap: MIT
- lodash.sortby: MIT
- marked: MIT
- minimist: MIT
- morgan: MIT
- multer: MIT
- npm: Artistic-2.0
- pako: MIT
- parse-url: MIT
- postcss: MIT
- prettier: MIT
- react-collapsible: MIT
- react-copy-to-clipboard: MIT
- react-dom: MIT
- react-error-boundary: MIT
- react-gtm-module: MIT
- react-icons: MIT
- react-notification-system: MIT
- react-password-strength-bar: MIT
- react-redux: MIT
- react-router-dom: MIT
- react-syntax-highlighter: MIT
- react: MIT
- reactjs-popup: MIT
- redux-persist: MIT
- shell-escape: MIT
- styled-components: MIT
- swagger2openapi: BSD-3-Clause
- tailwindcss: MIT
- tmp: MIT
- typescript: Apache-2.0
- url-parse: MIT
- url-search-params-polyfill: MIT
- url: MIT
- web-vitals: Apache-2.0
Third-Party Tool Support
OWASP ZAP
Mayhem for API includes support for running and importing results from OWASP ZAP via ZAP - API Scan
The OWASP® Foundation works to improve the security of software through its community-led open source software projects, hundreds of chapters worldwide, tens of thousands of members, and by hosting local and global conferences.
OWASP ZAP is licensed under the Apache 2 License
Release Notes
Supported Versions
2.18.5 (Mar 22, 2023)
- CLI - Updated documentation in
mapi --help
2.18.4 (Mar 15, 2023)
- CLI - Resolved an issue where resource ID reuse was not working for some
OpenAPI specifications that included multiple possible schemas in response
contentspecification.- This was preventing the fuzzer from accurately covering endpoints that use IDs (or other values) returned by other requests.
- An example of endpoints affected by this includes those
that make use of
anyOfor a combination of aschemaANDexamplein the responsecontentspecification.
2.18.3 (Mar 15, 2023)
- CLI - Performance improvements
2.18.2 (Mar 14, 2023)
- CLI - Ensure Linux musl builds are running the latest CLI.
2.18.1 (Mar 14, 2023)
- CLI - Fixed an issue where Results link is not always displayed at the end of a run.
2.18.0 (Mar 7, 2023)
- CLI - Include more rule checkers with run option --experimental-rules
2.17.3 (Mar 6, 2023)
- CLI - New option for
mapi run,--max-response-size. APIs with endpoints that return very large documents can lead to degraded performance and out of memory issues while fuzzing. This setting configures the threshold for which responses will be ignored if they exceed this maximum size. Setting this too high can result in out of memory issues, while setting it too low may lead to less test coverage against your API. The default value is 100KB. Values can be configured as 100B, 500KB.
2.17.2 (Feb 28, 2023)
- CLI - Issues from previous runs will no longer be replayed if they were suppressed. See Suppressing Issues for more details on issue suppression.
2.17.1 (Feb 17, 2023)
- CLI - Cancelling a
mapi runin progress (CTRL+C) will now cancel issue replay at the end of the run as well.
2.17.0 (Feb 15, 2023)
- CLI - Support for limiting mapi memory usage with option
--max-memory-usage
2.16.5 (Feb 2, 2023)
- CLI - Improved support for
$refJSON References when working with OpenAPI specifications. This should reduce the number of warnings displayed when a$refis made to a resource that is not under#/components/*(such as#/paths/*.
2.16.4 (Jan 30, 2023)
- CLI - fix issue matching content-type header value against Open API specification
2.16.3 (Jan 19, 2023)
- CLI - fixed an issue where issues from multiple previous jobs would replay at the end of a run, rather than one previous job.
2.16.2 (Jan 16, 2023)
- Authentication Bypass checker will no longer be triggered when
Authorizationheader is set using a Request Rewrite plugin.
2.16.1 (Jan 3, 2023)
- CLI - Added support to keep run results local with
--localflag passed tomapi run(mapi run --local ...). Local results are limited to Organizations with an active 'Enterprise' plan.
2.15.11 (Dec 12, 2022)
- CLI - Response values used for request generation are no longer case-sensitive. For example,
a response containing a field named
idwill now be considered for a request field namedID.
2.15.10 (Dec 6, 2022)
- CLI
- Issue replay now replays issues from the default branch of the target.
- If a parent commit sha is available, issues found in the last job matching the sha are used.
- If no parent commit sha is available, issues found in last job for the default branch are used.
- Git commit author, author email, and subject are now detected and used for better identification of jobs.
- Issue replay now replays issues from the default branch of the target.
2.15.9 (Nov 23, 2022)
- CI/CD support for job diffs based on git in a number of CI systems. Default branches have recently been added to target settings and are required to populate diffs in new jobs.
2.15.8 (Nov 14, 2022)
- Updated default Docker tag for ZAP's API Scan to
tag owasp/zap2docker-weekly:w2022-07-25
for runs using the
--zapflag.
2.15.7 (Nov 10, 2022)
-
New CI/CD documentation along with native support for Circle CI with our Circle CI Orb!
-
We are rolling out changes to help you understand the health of your APIs over time. With new support for "Default Branch" (accessible from your Target Settings page), you will be able to see how your changes are impacting your API against your default branch when you run Mayhem for API in continuous build and deploy pipelines (CI/CD).
-
We are making lots of changes to our website to improve the user experience. Our goal is to make it easier to find and understand your Mayhem for API results.
-
CLI - Added support to override the timeout (default of 5 seconds) to connect and communicate with your API targets with
mapi run --timeout... -
CLI - Support for fuzzing endpoints with content type
application/octet-stream. -
CLI - We are increasing the variety of data we send to your APIs by complimenting fully random inputs with locale-specific fake data with the help of the fake-rs library. The motivation here is to make more successful requests against your APIs so that we can go deeper and find harder-to-reach bugs and security issues.
2.15.6 (Oct 27, 2022)
- CLI - Improved support for more regular expression formats in your specification.
2.15.5 (Oct 26, 2022)
- CLI - Support for Postman environment files. Added checksums on CLI download page.
2.15.4 (Oct 13, 2022)
- CLI - Upgraded to the latest JSON Schema document for validating OpenAPI specifications: https://spec.openapis.org/oas/3.1/schema/2022-10-07
2.15.3 (Oct 13, 2022)
- CLI - Parsing improvements for java stacktraces - get more issue details for APIs that include Java stacktraces in the response.
2.15.2 (Oct 7, 2022)
- CLI - Fixed an issue where statistics stop updating when running
mapi runwith the--interactiveflag.
2.15.1 (Oct 5, 2022)
- CLI - Support for Postman dynamic variables in request headers, paths, query parameters and request bodies.
2.15.0 (Oct 3, 2022)
- CLI - Improved integration for Postman Collections,
Postman Environments
and Postman Authorization.
Mayhem for API now supports reading collections directly from Postman. Supply a
--postman-api-keyand set the target to a collection id. See Fuzzing Postman for more detail.
2.14.2 (Sep 29, 2022)
- CLI - Fixed some problems with issue replay and display.
2.14.0 (Sep 5, 2022)
-
The OWASP Zed Attack Proxy, or ZAP, is an open-source web app scanner, and the de-facto industry standard for automated security scanning of web applications. While ZAP takes a different approach to API security testing, we believe that it is complementary to Mayhem for API. We have built a super-easy way to run ZAP's API Scan tool alongside Mayhem for API. Just add a
--zapflag to your existingmapi runinvocation to get started!See Zed Attack Proxy (ZAP) Integration for more details.
2.13.4 (Aug 8, 2022)
- CLI - Fixed
Server Crashfalse positives seen primarily when servers close connections quickly after the first response, but before a second request is sent. This false positive is seen often when running against FastAPI based APIs where this behavior occurs whenever
an unhandled exception is triggered. Note that this can happen on with other API frameworks as well and is not limited to FastAPI.
2.13.2 (Aug 2, 2022)
- CLI - Added SOCKS5 proxy support. See HTTP Proxy Configuration for more details.
2.13.1 (Jul 26, 2022)
- CLI - Improvements to SARIF report
run.invocationswill now be included in SARIF outputversionproperty will now appear in the correct order in SARIF output as per the SARIF specification:
Although the order in which properties appear in a JSON object value is not semantically significant, the
versionproperty SHOULD appear first.https://docs.oasis-open.org/sarif/sarif/v2.0/csprd02/sarif-v2.0-csprd02.html#_Toc10127671).
2.13.0 (Jul 26, 2022)
- New Issue:
Invalid Request Spec- If Mayhem is able to elicit a successful response from an endpoint using a payload that's invalid according to that endpoint's spec, it will now raise an issue. Much like Invalid Response Spec and Verb Tampering, these may indicate simple specification problems, but can sometimes indicate serious underlying problems.
- New Issue:
PII Disclosure- If Mayhem detects anything that appears to be PII in response payloads, it will raise an issue. For now, this is limited to credit card numbers.
2.12.0 (Jul 7, 2022)
- New Issue:
Verb Tampering- Mayhem will make requests to endpoints with HTTP Verbs (ie:
GET, POST, PATCH...) that are invalid, or not part of the specification. If the API responds successfully, then a Verb Tampering issue will be raised. This may simply be an issue with the specification, or it could be indicative of a security-related misconfiguration on the API server.
- Mayhem will make requests to endpoints with HTTP Verbs (ie:
2.11.3 (Jun 28, 2022)
- CLI - Reduced logging noise significantly when running
mapi runatinfolevel logging (the default -- configurable with theMAPI_LOGenvironment variable). - Updated fuzzer so that checkers will run more frequently when running with more
than 2 workers with
mapi runand--concurrencyargument of 2 or higher.
2.11.1 (Jun 21, 2022)
- CLI - Fix a regression in OpenAPI 3.0 support
2.11.0 (Jun 20, 2022)
- CLI - Enable HAR uploads up to 200 MB in size
2.10.0 (Jun 14, 2022)
- CLI -Support for OpenAPI 3.1 specifications
- CLI -
autorun duration support tomapi run.mapiwill automatically determine how long to fuzz a target. It is still possible to specify a duration. - CLI - Improved support for multipart/form-data endpoints
2.9.9 (Jun 6, 2022)
- CLI -Fix service account authentication.
2.9.8 (Jun 3, 2022)
- CLI -Performance enhancements for CLI talking to MAPI API.
2.9.7 (May 17, 2022)
- CLI -mapi-action posts a run report to the pull request commment thread.
2.9.6 (May 16, 2022)
- CLI -
mapi issue detailwill now include issue stacktrace, if one was captured during a run - CLI -Added
--verbosityflag tomapiCLI for improved control of logging verbosity - CLI -Improved validation when converting specifications (ie: Postman -> OpenAPI)
2.9.5 (May 5, 2022)
- CLI -
mapi statusnow includes account details if you are logged in - CLI issue replay will only replay issues from the last "Completed" run by default
2.9.3 (Mar 22, 2022)
- CLI - Release preview for resource hint support in
mapi run. This option allows callers to specify values that should be used bymapiwhenever it generates API requests.
2.9.2 (Mar 9, 2022)
- CLI - Security update for CVE-2022-24713
2.9.0 (Feb 25, 2022)
- CLI - BREAKING CHANGE -
mapi loginwill only support logging with an API token going forward. Support for interactive login with GitHub or username/password will no longer be supported as we are reworking our authentication services.- API Tokens can be managed on the website from the following link: https://mayhem4api.forallsecure.com/settings/tokens
- CLI - Handle OpenAPI/Swagger JSON specification with unsupported regex patterns. Patterns with arbitrary look-ahead and back-references are not supported at this time.
2.8.2 (Feb 9, 2022)
- CLI - Automatically disable the InvalidResponseSpec rule when working with converted specifications.
2.8.0 (Feb 3, 2022)
- New Issue:
Timeout- When an API server doesn't respond in a reasonable amount of time, the fuzzer will give up on its request. This issue can lead to an efficient Denial-of-Service attack. An attacker can leverage problematic requests to consume server resources disproportionate to their own effort.
2.7.21 (Feb 1, 2022)
- CLI - Fixed an issue where responses specified with a content-type containing a suffix, such as
image/svg+xml, would be reported as Invalid Response Spec issue.
2.7.19 (Jan 25, 2022)
- CLI - Fixed replay logic to respect the endpoint and rule filtering command-line arguments.
- CLI - Fine-tuned some fuzzing parameters to more predictably generate payloads checking for known vulnerabilities.
- CLI - Fixed issues with checking certain responses for issues they could not exhibit, wasting effort and conceivably causing false positives.
- CLI - Fixed an issue when minimizing multi-request programs.
2.7.17 (Dec 22, 2021)
- CLI -
mapi runduration now accepts human-friendly strings such asmapi run <target> 30sec ... - CLI - Improve console output when running
mapi issue replay
2.7.16 (Dec 17, 2021)
- CLI - Fixed an issue where job logs may be truncated when running
mapi runwith--interactiveflag. - CLI - Issues replayed at the end of
mapi runwill track issues from the same SCM branch, if one is detected.
2.7.15 (Nov 30, 2021)
- CLI - Added new flag to
mapi run,--warnaserror, that will treat Issues with Severity of 'Warning' as errors. The junit report will include warnings when--warnaserrorflag is specified andmapiwill exit with an exit code of1if any 'Warning' level Issues are discovered during a run.
2.7.13 (Nov 23, 2021)
- CLI - Fixed an issue where the junit report produced by
mapi run --junitmay be missing issues
2.7.11 (Nov 5, 2021)
- CLI - Added a number of prompts for further configuration steps users can take to deepen testing.
- CLI - Fixed a case-sensitivity issue in token redaction.
2.7.10 (Nov 1, 2021)
- CLI - Handle OpenAPI/Swagger JSON specification with
regex patterns
containing escaped forward slashes (for example
\\w+:(\\/?\\/?)[^\\s]+")
2.7.9 (Oct 15, 2021)
- CLI - Added
mapi job deletesubcommand
2.7.8 (Oct 6, 2021)
- CLI - Fixed generated SARIF when parsed stacktraces reference "line 0"
- CLI - Added ability to filter fuzzed endpoints by their OpenAPI tags
2.7.7 (Sep 30, 2021)
- CLI - Expanded response spec validation warnings
- CLI - Fixed an issue with failing to stop a run against an unreachable API
2.7.6 (Sep 21, 2021)
- CLI - Added experimental support for response classifying plugins, which can be used to implement arbitrary logic for identifying issues
2.7.3 (Aug 11, 2021)
- CLI - If present, OpenAPI tags will be used to organize and group issues
- CLI - Fixed an issue with authentication redaction in issue details
2.7.2 (Aug 3, 2021)
- Invalid specification issues will now be reported with "warning" severity if API responses do not conform with the API specification.
- SARIF/JUnit Report - Fixed an issue where an extra '/' prefix may appear in path display
2.7.1 (Jul 27, 2021)
- CLI - Support suppressing issues - see Suppressing Issues for more details.
- Checkers - Added support for Error-based NoSQL Injection for MongoDB
2.7.0 (Jul 13, 2021)
- CLI - mapi will report warnings when responses are incompatible with the specification used to fuzz your target
- CLI - Fixed some issues with C# stacktrace parsing
- CLI/SARIF - It is now possible to report multiple issues for the same path, method and rule id.
This is most effective when fuzzing a target that is running in debug mode and will respond with
a stacktrace/traceback on error.
- For example, a 500 Internal Server Error triggered on
GET /foo, resulting in different stacktraces will be reported as two separate issues:- Issue 1:
GET /foo?bar=1->Internal Server Error (NullPointerException) - Issue 2:
GET /foo?bar=0->Internal Server Error (DivideByZeroException)
- Issue 1:
- For example, a 500 Internal Server Error triggered on
2.6.18 (Jul 2, 2021)
- CLI - Save fuzzing run output to an HTTP Archive (
.harfile) withmapi run ... --har out.har
2.6.16 (Jun 24, 2021)
-
CLI - Added beta support for fuzzing with an HTTP Archive (
.harfile) for the specification.mapi runwill now accept a.harfile for the specification argument- It is recommended to convert a
.harto an OpenAPI specification withmapi convert har...to inspect the converted specification before running a job
-
CLI - Added
mapi convertcommand. This runs automatically when callingmapi runwith a document which supports conversion.The
convertsubcommand provides an opportunity to inspect the converted OpenAPI 3.0 document prior to fuzzing an API.- Convert HTTP Archive
.harfile to an OpenAPI spec withmapi convert har <recording.har> - Convert Swagger 2.0 json or yaml file to an OpenAPI spec with
mapi convert swagger2 <spec.json|yaml> - Convert Postman 2.x collection json or yaml file to an OpenAPI spec with
mapi convert postman <collection.json|yaml> - Run
mapi convert --helpfor more details
- Convert HTTP Archive
2.6.14 (Jun 18, 2021)
- CLI - Issues from the most recent previous job will now be replayed at
the end of a run. The intent of this change is to reduce issues fluctuating
between jobs.
- To opt-out of this behavior, pass the
--no-replayflag tomapi run.
- To opt-out of this behavior, pass the
2.6.13 (Jun 14, 2021)
- The CLI now handles specifications with missing path parameters. If a path parameter is not in the specification, Mayhem for API will now infer it automatically and fuzz it. Previously, the CLI would exit early with an error.
2.6.12 (Jun 14, 2021)
- CLI - Resolve an issue where the CLI may panic when generating some invalid URLs during a fuzzing run
- Fixed an issue where calling
mapi runwith a path to the specification on Windows (C:\path\to\spec.yaml) would fail to read specification
2.6.11 (Jun 10, 2021)
- CLI - Resolved an issue that could cause duplicate reported issues in certain corner conditions
- Issue replay now supports providing a modified OpenAPI/Swagger/Postman specification
MAPI_LOGenvironment variable will now correctly outputtraceanddebuglevel logs
2.6.10 (Jun 3, 2021)
- CLI -Support for issue replay. Replay issues using
mapi issue replay <job_id>- Try
mapi issue replay --helpto see more details - Note that some issues depend on the state of your API and may not always reproduce
- Try
2.6.9 (Jun 1, 2021)
- CLI -Show incomplete request counts in interactive job display
- CLI -Add markdown display support to SARIF output
- Header parameters listed in API specifications will now be fuzzed
2.6.7 (May 27, 2021)
- New Issue:
Server Crash- Server crashes are now detected and reported in Job Issues.
- A server that can be forced to crash may be vulnerable to denial-of-service attacks.
- CLI -
mapi issue listprint a list of issues for a job
2.6.5 (May 11, 2021)
- Fix parsing of stacktraces generated on windows host.
2.6.4 (May 10, 2021)
- Authentication data is now redacted from buggy requests before being uploaded to our API. Note that in some cases, slightly mutated auth tokens might still be updated to our API (since they might be necessary to demonstrate authentication bypasses)
- Add support for C# stacktraces with async code and nested exceptions
2.6.3 (May 5, 2021)
- Legacy github token (which are not prefixed by ghp_) are now accepted.
2.6.0 (Apr 29, 2021)
- Enable new security checkers by default: Command injection, Blind command injection, Path traversals, SSRF (assumes CLI can be reached through localhost from the API)
- Improve authentication bypass checker by differentiating between
-H(for regular headers) and--header-auth(for authentication header). - Redirect loops will not terminate a run anymore
- Improvements in the fuzzing engine to increase coverage
2.5.16
- CLI: Ignore errors with the
--ignore-ruleoption tomapi run. Callers may ignore specific errors such asInternalServerErrororAuthenticationBypasswhen including this flag.
2.5.13
- CLI: Upload code scanning results to GitHub from anywhere by passing
--github-tokenwith a token tomapi run. - CLI: Upload code scanning results to your on-premise GitHub instance
by passing
--github-api-urltomapi run. - CLI: Auto-detect build information from the environment in Drone CI
- CLI: Golang stacktrace parsing, in addition to the existing parsing for Ruby, JS, Python and Java.
2.5.10
- CLI: Add
SARIF
support.
mapi runnow accepts a--sarif <output-file>option which outputs a SARIF file that you can upload to supported systems, like GitHub if you have enterprise plan.
2.5.7
- The CLI now has beta support for postman 2.x collections 🚀! You can pass your collection
to
mapi runinstead of an OpenAPI specifications.
2.5.6
- CLI:
mapi runcan now automatically create a new API Target if the--urloption is supplied.
2.5.5
- Security: Added support for detecting authentication configuration issues. The fuzzer will
now periodically remove or mangle any authentication parameters used for a job. Any endpoints
that are specified as with
securityconfiguration will be marked as buggy if they can be accessed successfully with missing or altered authentication parameters.
2.5.4
-
Security: Added support for detecting SQL Injection errors. See the new documentation on configuring your API for tips on improving detection of SQL Injection problems.
-
CLI: Change the logging level of the
mapiCLI with theMAPI_LOGenvironment variable. Supported values includetrace,debug,info,warn,error. The default is set toinfo.
2.4.1
- Renamed cohorts to organizations
2.3.28
- Added experimental path traversal detection, enabled by setting the
MAPI_FEATURE_PATH_TRAVERSALenvironment variable.
2.3.19
- Added support to CLI for rending email verification with
mapi signup resend-verify <email-address>
2.3.18
- Creating a new account when joining a Cohort now requires email verification
- Added support to CLI for completing email verification with
mapi signup verify
2.3.16
- Improve parsing of URLs given through --url
- Fix memory leak in type generation during fuzzing
2.3.15
- Renamed the product to 'Mayhem for API' and the CLI to 'mapi'. Please re-download the CLI.
2.3.8
- Parallel fuzzing is now available using
-jwhen invokingmapi run(defaults to 1). This will allow you to potentially find concurrency issues, and load test your APIs.
2.3.7
- bug fix: better support for reference loops in specifications
2.3.5
mapi runwill now generate a human-friendly html report when--htmlis given.
2.3.4
- CLI prints warnings on unexpected properties in the specification instead of aborting.
2.3.3
-
API Targets will now include their owner when presented in the CLI or when referenced to start a new run.
You may continue to refer to any existing API Targets without the owner; however, shared API Targets will require the owner to be specified.
2.3.2
- Windows support: Our CLI can now run natively on 64-bit windows.
2.3.0
- Users can specify cookies for the fuzzer to authenticate to the target API.
2.2.4
- EULA agreement
2.2.3
-
GitHub Cloud Integration support is now available. Please reach out to us at support@forallsecure.com so that you can get connected with GitHub. Once you have been approved, you can connect with the CLI with the command:
mapi github connect -
Our GitHub app is also live. During Limited Availability it will not be accessible via the marketplace, but can be installed into your Personal or Organization account by following this link:
https://github.com/apps/mayhem-for-api
2.2.2
- Preparation for GitHub support. The CLI will attempt to gather CI details
from GitHub Actions, Jenkins, or simply invocations of
gitto synchronize Job Status with GitHub checks. If details cannot be inferred from the environment, then they may be provided manually with new optional arguments passed tomapi run. - GitHub integration support will be fully supported once the Mayhem for API app is available in the GitHub marketplace.
2.2.1
- CLI will auto-update across patch and minor versions
2.2.0
- CLI supports API pagination for
jobandtargetAPI resources
2.1.10
- Fix unresponsive arrow keys in interactive job status
2.1.9
- JUnit output contains percentile response times per endpoint.
- Fuzzer better uses examples in the API specification, which allows users to give a hint to the fuzzer when it cannot generate 200 responses on its own.
- Fuzzer warns if the API closes a connection early before sending a full response. This will eventually be classified as a bug.
2.1.8
- For accounts registered via GitHub, a New Token may now be requested when using
mapi github connect --new-token.
2.1.7
- Added interactive mode to
mapi runwith the new--interactiveflag. This will now open the interactive UI that was previously only accessible with themapi job statussub-command. - Job status UI now detects when a tty is not present. In instances where a tty is not present, the job log and coverage table will be printed to standard output.
- All covered endpoints and statistics will now print to the console at the end of a run
2.1.6
- Bug fixes for GitHub connect
2.1.5
- Fuzzer will not consider HTTP response code
501as an error - CLI will exit with return code
2on fuzzer error - CLI will now automatically detect Swagger 2.0 specifications and attempt to convert them to OpenAPI 3.0 automatically with the help of the openapi-webconverter
2.1.4
- CLI can now override API Target authentication parameters when calling
mapi run
2.1.3
- CLI support for handling upcoming pagination changes to
jobandtargetAPI resources
Breaking Changes
As Mayhem for API is continuously evolving, we make an effort to avoid breaking changes. Not breaking your workflow and CI builds is of utmost important to us. We want it to just work, always.
There are times due to security constraints or technical limitations that require us to make a breaking change. We aim to make this as painless as possible, usually requiring no more than downloading a newer CLI. If breaking changes are required, we will try to make it as painless as possible:
- We will give you a heads up if we notice you are using a version that will become incompatible.
- We aim to provide command-line arguments backwards compatibility, so that you can always use the latest release without having to change your invocation.
- We give you tips on how to integrate the fuzzer into your builds so that you automatically use new releases.
- When necessary, we will slowly deprecate features or workflows, giving you enough time to upgrade.
Unsupported versions
2.1.2
- CLI includes request and response in junit failures
2.1.1
- CLI support for blacklisting endpoints when calling
mapi run
2.1.0
- CLI improved logging, feedback and error handling
- Fixed cascade delete for API Targets
FAQ
(Frequently Asked Questions)
What platforms are supported?
The Mayhem for API CLI (command line interface), mapi, will run on the following platforms:
- Linux (64-bit)
- MacOS
- Windows (64-bit)
What languages are supported?
Mayhem for API scans Web APIs. It does not matter what language the API is written in,
as long as it is accessible from the same network as the mapi CLI.
For more detailed issues, Mayhem for API can collect stacktraces returned by the API. It is common to enable returning stacktraces when running an API in a debug mode, but not in production.
Stacktrace parsing enables Mayhem for API to achieve tighter integrations with tools like GitHub / Visual Studio Code with the help of SARIF results export.
The supported languages for Stacktrace parsing include:
- python
- ruby
- java
- javascript
- go
- C#
What kind of information do you collect?
Personal Information
When you sign up for Mayhem for API, we collect your email address and any other optional information that you provide such as Name and Phone Number.
When you sign up using a social login, such as Atlassian, GitLab, Github or Google, we will automatically collect your email address and Name.
If you choose to delete your account, are personal information will remove all personal information at, or before, 28 days in accordance with GDPR compliance.
Application Data
During a Mayhem scan, the mapi CLI can make thousands, or even millions, of
requests to your API. We do not store the contents of every request. Only when
an issue is discovered, will we collect the request that was sent to your API,
and the response that it returned to the CLI.
Other information that is considered 'secret', such as Bearer tokens used for Authentication, will be redacted prior to uploading issue details.
ℹ️ Organizations with Enterprise plans can run in 'local' mode. This mode prevents the
mapiCLI from uploading all run details to the mAPI API. Results will be kept local to the computer wheremapiwas run.See Keeping Results Local for more details.