Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions assets/contributors.csv
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,8 @@ Karthik Gopalakrishnan,Tinkerblox,,,,
David Torres,ThirdAI,,,,
Sammy Sahnine,,,,,
Matthew Sloyan,Arm,matthewsloyanARM,,,
Austin Hill,,,,,
Andreas Mullen,Arm,,,,
Liam Randall,Cosmonic,,hectaman,,https://cosmonic.com/
Austin Hill,,,,,
Andreas Mullen,Arm,,,,
Michalis Spyrou,Arm,,,,
Michael Koslov,Arm,,,,
Liam Randall,Cosmonic,,hectaman,,https://cosmonic.com/
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
---
title: Understand the Metis review workflow
description: Learn how Metis gathers evidence, runs security reviews, and presents findings for human triage.
weight: 2
layout: "learningpathall"
---

## What is Metis?

Arm Metis is an open-source, agentic security code-review framework. It combines language-aware source tooling with a large language model (LLM) to investigate security-relevant behavior and return structured review candidates. Metis supports multiple programming and hardware description languages through language plugins. This Learning Path uses C examples, but the same concepts can be applied to the other supported languages.

You first run Metis on a small C program with known vulnerabilities. This gives you a quick look at the command flow, JSON output, SARIF output, and reported findings. You then prepare a libpng source tree from the [Magma](https://hexhive.epfl.ch/magma/) benchmark and triage a captured Metis finding against the source diff.

The exercises show how to use Metis during a security review. They do not benchmark Metis accuracy. Model choice, provider behavior, Metis version, and source changes can all affect a run. Use published benchmark results for comparative performance data.

{{% notice Note %}}
Model-backed review can consume substantial API tokens. The initial exercise is less expensive than the optional live libpng run. A pre-captured libpng report candidate is included so you can complete the triage exercise without running the larger review.
{{% /notice %}}

## How Metis reviews code

A Metis review starts with source files and project context. Metis collects language-aware evidence, asks your configured model to investigate possible vulnerabilities, and returns structured findings for a human reviewer.

Metis offers two review approaches:

- **Simple LLM review (default):** the model examines source code or a patch using language-specific security prompts and project guidance. It returns structured findings without requiring a code graph. This is also the review path for languages without C/C++ reachability support.
- **Optional reachability review:** for C/C++, Tree-sitter builds a CodeGraph of functions, calls, and source facts. When enabled, `reachability` reviews selected functions with this evidence and surrounding caller/callee context. Graph evidence does not by itself prove attacker control, build reachability, or exploitability.

In Metis 2.0.0, the packaged Review stage runs `simple_llm_review`, `finding_dedup`, and `result`. `review_code` and `review_file` use simple LLM review by default. Metis builds a CodeGraph for triage, but runs reachability review only when you enable it.

## Follow a review workflow with Metis

A security review with Metis typically follows these steps:

1. Choose a codebase, directory, file, or patch to review.
2. Add project-specific security context in `.metis.md` when needed.
3. Run a review and save the findings as JSON or SARIF.
4. Optionally use automated triage to check the findings against more source code.
5. Verify the findings against the source code, build configuration, and tests.

## Assess findings with automated triage

Automated triage checks a reported finding against more source code and marks it `valid`, `invalid`, or `inconclusive`. Metis checks the source lines cited for a `valid` or `invalid` verdict, but the verdict does not prove that the finding is correct or that an exploit works.

## How this Learning Path uses Metis

You will use two review commands on C code:

- In the first exercise, you use `review_code` to review the small C codebase.
- In the second exercise, you use `review_file` to focus on `pngrutil.c` with context from the rest of the libpng codebase.

The exercises start Metis with `--interactive` and run these commands at its prompt. Each command initializes the target automatically, so you do not need to run `init` first. You will inspect the findings yourself without enabling automated triage.

## Interpret Metis findings

A Metis finding normally includes the issue, source location, reasoning, suggested mitigation, Common Weakness Enumeration (CWE) identifier, severity, and confidence.

{{% notice Note %}}
CWE identifiers, such as `CWE-190` for integer overflow, provide standard names for common classes of software weakness.
{{% /notice %}}

You still decide whether a candidate is reachable, exploitable, already mitigated, or worth fixing.

## What you've learned and what's next

You have seen how Metis reports possible issues and why each finding needs a source check. Next, you will install the tested release and add your provider API key.
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
---
title: Install and configure Metis
description: Install a tested Metis release and configure a model provider with an API key.
weight: 3
layout: "learningpathall"
---

## Install system tools

The commands in this Learning Path use a Unix-like shell. Use Terminal on macOS, a Linux terminal, or an existing Ubuntu/Debian terminal running under WSL2 on Windows. Native Windows PowerShell and Command Prompt are not covered.

Select your platform and install the compiler, Git, and command-line utilities used to fetch code, apply patches, and inspect reports:

{{< tabpane code=true >}}
{{< tab header="macOS" language="bash">}}
xcode-select --install
{{< /tab >}}
{{< tab header="Ubuntu Linux or WSL2" language="bash">}}
sudo apt-get update
sudo apt-get install -y build-essential ca-certificates curl git grep patch sed
{{< /tab >}}
{{< /tabpane >}}

If the Xcode Command Line Tools are already installed, macOS reports that no installation is needed. On WSL2, use the Linux filesystem for this exercise and avoid creating the workspace under `/mnt/c`.

## Install uv and Python

Metis requires Python 3.12 or later. This Learning Path uses `uv` to install Python 3.12 and create an isolated Metis environment:

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```

Load the shell environment created by the installer and confirm that `uv` is available:

```bash
source "$HOME/.local/bin/env"
uv --version
```

Install Python 3.12 using `uv`:

```bash
uv python install 3.12
uv python find 3.12
```

The second command prints the path to the managed Python 3.12 executable.

Check that all command-line tools used by the exercises are available:

```bash
for tool in curl git grep patch sed uv; do
command -v "$tool" >/dev/null || echo "Missing: $tool"
done
```

The command prints nothing when all required tools are available. Install any tools that are missing.

## Create the exercise workspace

Create one workspace in your home directory for Metis, the review targets, and generated reports. On WSL2, `$HOME` is inside the Linux filesystem:

```bash
mkdir -p "$HOME/metis-security-review/results"
cd "$HOME/metis-security-review"
```

## Clone and install Metis

Clone the Metis repository at the tested [2.0.0 release](https://github.com/arm/metis/releases/tag/metis-v2.0.0):

```bash
git clone --branch metis-v2.0.0 --depth 1 \
https://github.com/arm/metis.git \
"$HOME/metis-security-review/metis"

cd "$HOME/metis-security-review/metis"

uv sync --no-dev --python 3.12
source "$HOME/metis-security-review/metis/.venv/bin/activate"

command -v metis
metis --version
git rev-parse HEAD

cd "$HOME/metis-security-review"
```

`uv sync --no-dev --python 3.12` installs only the runtime dependencies needed to run the Metis CLI, using Python 3.12. It creates `metis/.venv` and installs Metis there. Activating the environment makes the `metis` command and the matching Python interpreter available in the current terminal.

The expected output from `metis --version` for this tag is:

```output
Metis 2.0.0
```

The `metis-v2.0.0` tag points to commit [`742b20d685b0aeb3216b36ddfc44bc5e05ace82c`](https://github.com/arm/metis/commit/742b20d685b0aeb3216b36ddfc44bc5e05ace82c). The output of `git rev-parse HEAD` should match this commit. Record the version and commit when comparing runs.

If you open a new terminal, return to the exercise workspace and reactivate the Metis environment before continuing:

```bash
cd "$HOME/metis-security-review"
source "$HOME/metis-security-review/metis/.venv/bin/activate"
command -v metis
```

The path printed by `command -v` should end with `metis-security-review/metis/.venv/bin/metis`. If it points elsewhere, open a new terminal or run `deactivate`, then run the activation command again.

## Configure the model provider

Metis supports several hosted and local model providers. These examples use an OpenAI-compatible endpoint that serves the model ID `gpt-6.1-sol`. Export your API key in the same terminal that will run Metis:

```bash
export OPENAI_API_KEY="your-api-key"
```

Do not add secrets to `.metis.md`, `metis-review.yaml`, shell history, source control, or report files.

Create a provider configuration outside the review targets. Replace `https://your-provider.example/v1` below with the trusted base URL of the provider that serves `gpt-6.1-sol`, including any required API prefix. The example address is a placeholder, not a service endpoint:

```bash
cat > "$HOME/metis-security-review/metis-review.yaml" <<'EOF'
llm_provider:
name: openai
model: gpt-6.1-sol
base_url: https://your-provider.example/v1
EOF
```

The review commands load this file with `--config` and read the API key from `OPENAI_API_KEY`. Your endpoint must support the OpenAI Responses API and the model named in the file. If you choose a different model from the captured `gpt-6.1-sol` runs, your findings might differ. For other providers, see the [Metis 2.0.0 provider setup instructions](https://github.com/arm/metis/blob/metis-v2.0.0/README.md#2-set-up-llm-provider).

## Confirm that the command starts

Display the command help before starting a model-backed review. Metis 2.0.0 runs its full workflow by default, including automated triage. The exercises use `--interactive` to run review-only commands; Metis initializes each target automatically.

```bash
metis --help
```

The help output should include `--codebase-path`, `--config`, `--interactive`, and `--output-file`. Repeat `--output-file` to request more than one format.

## What you've accomplished and what's next

You installed the tested Metis 2.0.0 release and set it up with your model provider. Next, you will review a small C program and compare the report with defects that are intentionally present in its source.
Original file line number Diff line number Diff line change
@@ -0,0 +1,163 @@
---
title: Compare a Metis report with vulnerable C code
description: Run a Metis review, inspect JSON and SARIF output, and compare the report with known defects in a small C program.
weight: 4
layout: "learningpathall"
---

## Prepare the example program

The [Damn Vulnerable C Program](https://github.com/hardik05/Damn_Vulnerable_C_Program) contains vulnerabilities intentionally placed and labeled in one C file. Use it first to learn the Metis review process: prepare a small target, run `review_code`, save JSON and SARIF, summarize findings, and compare source locations.

In your terminal, clone the repository at the tested commit and copy `dvcp.c` into a separate review directory. The copy keeps the review target small and avoids sending the rest of the repository to Metis:

```bash
git clone https://github.com/hardik05/Damn_Vulnerable_C_Program.git
git -C Damn_Vulnerable_C_Program checkout 8dcbd202887f587783952a320a2301de701be529
mkdir -p dvcp-review
cp Damn_Vulnerable_C_Program/dvcp.c dvcp-review/
```

Add a short threat-model note for the review. Metis reads `.metis.md` as project context:

```bash
cat > dvcp-review/.metis.md <<'EOF'
# Project security context

This command-line C program parses an image file supplied by an untrusted user.
Treat file contents, dimensions, offsets, and derived allocation sizes as
attacker-controlled. Review memory safety, integer arithmetic, resource
exhaustion, input validation, and cleanup on error paths.
EOF
```

The note tells Metis how to treat inputs and security boundaries. The source file still contains teaching comments, which is acceptable here because this step teaches the review workflow. You will use libpng for a more realistic triage example.

## Run the review

Change into the review directory and start the Metis interactive prompt with the provider configuration you created. Save JSON for detailed inspection and SARIF for code-scanning tools:

```bash
cd "$HOME/metis-security-review/dvcp-review"
metis \
--interactive \
--codebase-path . \
--config "$HOME/metis-security-review/metis-review.yaml" \
--output-file ../results/dvcp.json \
--output-file ../results/dvcp.sarif
```

At the Metis prompt, run the review, then exit after it completes. Metis initializes the project automatically, then runs the default simple LLM review and deduplicates the findings:

```console
review_code
exit
```

Metis prints review progress and detailed findings under `File:`, with an `Identified issue` heading. Finding fields include `Snippet`, `Line number`, `CWE`, `Severity`, `Why`, `Mitigation`, and `Confidence`. After the findings, the export and token-usage output is similar to:

```output
Results saved to ../results/dvcp.json
SARIF report saved to ../results/dvcp.sarif
Token usage (review_code)
Current (input: 8,420 · output: 3,349 · total: 11,769)
```

Back in your shell, return to the exercise workspace:

```bash
cd "$HOME/metis-security-review"
```

The run sends source-derived evidence to your configured model provider. Runtime, token use, and findings depend on the model, provider, and Metis version.

It can take a couple of minutes to complete.

## Summarize the JSON report

The JSON report stores findings in `reviews[].reviews`. Source locations can appear in an `anchor`, in `primary_file` and `line_number`, or in the enclosing review group's `file` field. Use the Python interpreter from the active Metis environment to count the findings and print their issue types and source locations:

```bash
python - <<'PY'
import json
from pathlib import Path

data = json.loads(Path("results/dvcp.json").read_text())
review_groups = data["reviews"]
count = sum(len(group["reviews"]) for group in review_groups)
print(f"Finding count: {count}")
for review_group in review_groups:
for finding in review_group["reviews"]:
anchor = finding.get("anchor") or {}
file_path = (
anchor.get("file_path")
or finding.get("primary_file")
or finding.get("file")
or review_group.get("file")
)
line = anchor.get("start_line") or finding.get("line_number")
print(
f"- {finding['cwe']} {finding['issue']} "
f"({file_path}:{line})"
)
PY
```

This example was captured on November 2, 2026 using the `metis-v2.0.0` tag ([`742b20d`](https://github.com/arm/metis/commit/742b20d685b0aeb3216b36ddfc44bc5e05ace82c)), with `gpt-6.1-sol` through an OpenAI-compatible endpoint and DVCP commit [`8dcbd20`](https://github.com/hardik05/Damn_Vulnerable_C_Program/commit/8dcbd202887f587783952a320a2301de701be529). The provider reported the model ID as `gpt-6.1-sol`. The run used the packaged simple LLM review and deduplication nodes; reachability review and automated triage were not enabled.

The output is similar to:

```output
Finding count: 10
- CWE-125 Unterminated image fields cause out-of-bounds string reads (dvcp.c:48)
- CWE-122 Unchecked dimension-derived allocations permit heap overflows and NULL dereferences (dvcp.c:54)
- CWE-415 An even dimension sum causes a double free (dvcp.c:59)
- CWE-416 An odd dimension sum divisible by three causes a use-after-free write (dvcp.c:64)
- CWE-369 Unvalidated image dimensions cause invalid integer division (dvcp.c:82)
- CWE-787 File-controlled quotient indexes outside a stack buffer (dvcp.c:90)
- CWE-787 The heap buffer is accessed one byte past its allocation (dvcp.c:91)
- CWE-674 Recursive operation has no termination condition (dvcp.c:22)
- CWE-400 Allocation loop intentionally retains memory until exhaustion (dvcp.c:109)
- CWE-401 Large-quotient branch discards an allocated buffer without freeing it (dvcp.c:97)
```

Your exact count and wording can differ. A valid report can contain zero findings. An empty report means that this run did not report any findings; it does not mean the program is safe. If you expected findings, check your provider connection and any Metis log output before judging the result.

## Compare the report with known defects

Display four source regions that the program identifies as deliberately vulnerable:

```bash
sed -n '14,28p' dvcp-review/dvcp.c
sed -n '42,69p' dvcp-review/dvcp.c
sed -n '72,87p' dvcp-review/dvcp.c
sed -n '89,115p' dvcp-review/dvcp.c
```

Check whether your report covers these concrete cases:

- Signed image dimensions are combined and passed to `malloc()` without overflow or range checks.
- `buff1` is freed, then freed again when `size1` is even or written after free when it is divisible by three.
- `img.height` is used as a divisor without a zero check.
- `size3` is used as an index into fixed-size and heap buffers without bounds checks.
- Attacker-controlled dimensions can select recursive stack exhaustion or repeated heap allocation.

The captured Metis 2.0.0 JSON and SARIF reports contain ten findings. Source anchors can span multiple lines, and one finding can discuss additional sites in its reasoning. The table maps those reported regions and explanations to the known source defects:

| Source issue | Tested-run outcome |
| --- | --- |
| Non-NUL-terminated image fields passed to `%s` | `CWE-125` at `dvcp.c:48` |
| Dimension-derived allocations followed by fixed-size copies | `CWE-122` anchored at `dvcp.c:54–58`; its reasoning also covers `dvcp.c:74–79` and `dvcp.c:82–87` |
| `buff1` double free and use-after-free | `CWE-415` at `dvcp.c:59–63`; `CWE-416` at `dvcp.c:64–69` |
| Integer division using unvalidated image dimensions | `CWE-369` at `dvcp.c:82` |
| Indexed stack and heap accesses | Separate `CWE-787` findings at `dvcp.c:90–94` and `dvcp.c:91–95` |
| `buff4` allocation lost without being freed | `CWE-401` at `dvcp.c:97–100` |
| Unbounded recursion and repeated heap allocation | `CWE-674` at `dvcp.c:22–27`; `CWE-400` at `dvcp.c:109–115` |

For each reported finding, verify the source location, attacker control, and operation. Also record known source defects that the run omitted. The goal is to practice reviewing Metis output, not to turn the finding count into a coverage score.

The same review is also available in `results/dvcp.sarif`. SARIF allows code-hosting services and security tools to associate findings with source locations. Keep JSON when you want the complete Metis reasoning and SARIF when you want to integrate results into an existing code-scanning workflow.

## What you've accomplished and what's next

You ran a complete Metis review, exported two report formats, and compared the reported findings with known defects. Next, you will prepare a realistic libpng target without leaving benchmark canaries or fixes in its source.
Loading
Loading