diff --git a/admin/creating-api-key.mdx b/admin/creating-api-key.mdx index ab076399..d9308f1a 100644 --- a/admin/creating-api-key.mdx +++ b/admin/creating-api-key.mdx @@ -9,7 +9,7 @@ The Checkly public API and CLI use API keys to authenticate requests. API keys c ## User API keys -User API keys are tied to one specific user in your account and inherit the access level of that user, i.e. "read only" or "admin". +User API keys are tied to one specific user in your account and inherit that user's [account role and permissions](/admin/team-management/overview#permissions-and-roles), including [custom roles](/admin/team-management/custom-roles) (beta) when enabled for your account. When a user is removed from your account, that user's API keys are no longer able to access your account through the API. **Steps to create an user API key:** @@ -49,15 +49,21 @@ Service API keys allow you to set a role access level on the key itself. Availab - **Read & Run** - Can view all resources and trigger checks and tests, but cannot create, edit, or delete. Ideal for CI/CD pipelines that only need to run tests. - **Read Only** - View-only access to all resources. +If your account has [custom roles](/admin/team-management/custom-roles) (beta) enabled, you can also select one for a service API key. + Prime use cases for service API keys are: - Background services like CI pipelines invoke the [Checkly CLI](/cli/overview). - Any custom integrations that needs to create, update or delete resources through our public API. - Replacing user API keys for customers using our SSO integration and cannot create "service users" in their user directories due to compliance reasons. -**Steps to create a service API key:** + +You need [**Create API keys** (`api-keys:create`)](/admin/team-management/overview#permissions-and-roles) permission and a plan that includes service API keys. Owner and Admin have this permission by default. If access is missing, ask your account Owner or Admin to assign a suitable role. -Only users with the Admin and Owner role can create service API keys. +You can only give a key a role whose permissions you hold yourself. The role picker disables roles you cannot give. + + +**Steps to create a service API key:** 1. Go to the API keys section in the [Account Settings](https://app.checklyhq.com/settings/account/api-keys). 2. Click **Create service API key**. @@ -74,4 +80,4 @@ curl -H "Authorization: Bearer my_service_api_key" -H "X-Checkly-Account: my_acc ### Deleting a service API key -Only Admin and Owner users can delete an API key. To delete an API key click on the **Delete** icon in the Service Api Keys section of the [Account Settings](https://app.checklyhq.com/settings/account/api-keys). \ No newline at end of file +You need [**Delete API keys** (`api-keys:delete`)](/admin/team-management/overview#permissions-and-roles) permission to delete a service API key. Owner and Admin have it by default; a [custom role](/admin/team-management/custom-roles) (beta) can grant it when enabled for your account. Click the **Delete** icon in the Service API Keys section of [Account Settings](https://app.checklyhq.com/settings/account/api-keys). You can revoke an existing key even if your plan no longer includes service API keys. diff --git a/admin/team-management/custom-roles.mdx b/admin/team-management/custom-roles.mdx index ff1116b4..4aaddee4 100644 --- a/admin/team-management/custom-roles.mdx +++ b/admin/team-management/custom-roles.mdx @@ -1,20 +1,20 @@ --- title: "Custom roles" -description: "Create custom roles with the permissions you pick and give them to members and invitations." +description: "Beta documentation for creating custom roles and assigning them to members, invitations, and service API keys." sidebarTitle: "Custom Roles" tag: "Beta" canonical: 'https://www.checklyhq.com/docs/admin/team-management/custom-roles/' --- -Custom roles are in beta for [Enterprise plans](https://www.checklyhq.com/pricing/). They aren't generally available yet. To take part in the beta, contact your customer success team. +Custom roles are in beta and are not generally available. This documentation applies only to accounts where custom roles are enabled. -A custom role is a named set of permissions that you pick one by one. Custom roles sit next to the [built-in roles](/admin/team-management/overview#team-member-roles). You give them to members and invitations the same way. +A custom role is a named set of permissions that you pick one by one. Custom roles sit next to the [built-in roles](/admin/team-management/overview#team-member-roles). You can assign them to members, invitations, and [service API keys](/admin/creating-api-key#service-api-keys). ## The Roles page -**Account settings > Roles** lists your account's custom roles. It shows once custom roles are turned on for your account. Each row has the role's name and description, how many permissions it grants, and how many members and pending invitations hold it. Click a role's name to open it. +**Account settings > Roles** lists your account's custom roles. It shows once custom roles are turned on for your account. Each row has the role's name and description, how many permissions it grants, and how many members, pending invitations, and service API keys hold it. Click a role's name to open it. The Roles page with the New role button and a table of custom roles, each showing its name and description, its number of permissions, how many members and invites hold it, and a delete button @@ -26,7 +26,7 @@ A custom role is a named set of permissions that you pick one by one. Custom rol | View roles | See the Roles page and open a role to see its permissions | All roles | | Create roles | Create custom roles | Owner, Admin | | Edit roles | Change a role's name, description and permissions | Owner, Admin | -| Delete roles | Delete custom roles that no member or invitation holds | Owner, Admin | +| Delete roles | Delete custom roles and choose a replacement for affected holders | Owner, Admin | A custom role can hold these permissions too. They're listed under **Roles**. @@ -44,6 +44,8 @@ A custom role can hold these permissions too. They're listed under **Roles**. ## The permission matrix +The matrix uses permission names such as **Run checks**. If another guide refers to an ID such as `checks:run`, see [Permission names and IDs](/admin/team-management/overview#permissions-and-roles) to find its name, resource row, and how to get access. + The Permissions card of the On-call responder role, with its incident, on-call and alert permissions ticked The Permissions card of the On-call responder role, with its incident, on-call and alert permissions ticked @@ -58,16 +60,16 @@ A custom role can hold these permissions too. They're listed under **Roles**. ## Edit a role -Click the role's name on the Roles page, make your changes and click **Save role**. The page says how many members and pending invitations hold the role. Changes apply to them right away. +Click the role's name on the Roles page, make your changes and click **Save role**. The page says how many members, pending invitations, and service API keys hold the role. Changes apply to them right away. - You can only edit a role when you hold all of its permissions yourself. - Without **Edit roles**, the role opens read-only. ## Delete a role -Click the trash button in the role's row and confirm. Deleting a role can't be undone. +Click the trash button in the role's row. If the role is assigned, review the affected members, pending invitations, and service API keys, then choose their **Replacement role** before confirming. Their permissions change immediately. Deleting a role can't be undone. -You can only delete a role that no member or pending invitation holds. Until then, the trash button is disabled. Hover it to see how many hold the role. To free the role, [change those members' roles](/admin/team-management/managing-team-members#change-members-roles) and [remove the invitations](/admin/team-management/managing-team-members#pending-invites). An invitation keeps the role it was sent with. +You must be allowed to assign the replacement to every affected holder. If no suitable replacement is available to you, ask your account owner or an admin to make the change. ## Give a custom role @@ -87,4 +89,4 @@ The same rules apply as for built-in roles: ## API keys -[User API keys](/admin/creating-api-key#user-api-keys) act with their member's role. A member with a custom role has the same permissions through the API and the CLI. Changes to the role apply to their keys right away. [Service API keys](/admin/creating-api-key#service-api-keys) take a built-in role. +[User API keys](/admin/creating-api-key#user-api-keys) act with their member's role. A member with a custom role has the same permissions through the API and the CLI. Changes to the role apply to their keys right away. When the beta is enabled, [service API keys](/admin/creating-api-key#service-api-keys) can also use a custom role whose permissions you hold. diff --git a/admin/team-management/managing-team-members.mdx b/admin/team-management/managing-team-members.mdx index 08d73f27..136bd813 100644 --- a/admin/team-management/managing-team-members.mdx +++ b/admin/team-management/managing-team-members.mdx @@ -50,7 +50,7 @@ If your account is managed through the [Vercel native integration](/integrations ## Change members' roles 1. Select the members and click **Change role**. -2. Pick a role: the built-in roles, then any [custom roles](/admin/team-management/custom-roles) (beta). You can type to search the picker. It only offers roles you can give to every member you selected, and shows the description of the one you pick. +2. Pick a role: the built-in roles, then any [custom roles](/admin/team-management/custom-roles) (beta) enabled for your account. You can type to search the picker. It only offers roles you can give to every member you selected, and shows the description of the one you pick. 3. Click **Change role**. @@ -141,4 +141,4 @@ Every role can see the members list. Acting on it needs these permissions: | Invite members | Invite people, and resend or remove any invitation | Owner, Admin | | Lower your own role | Move yourself to a role whose permissions you already hold | Owner, Admin, Read & Write | -The owner holds **Lower your own role**. The owner's role only changes by transferring ownership. [Custom roles](/admin/team-management/custom-roles) (beta) can grant these permissions one by one. See [Team member roles](/admin/team-management/overview#team-member-roles) for what each role can do. +The owner holds **Lower your own role**. The owner's role only changes by transferring ownership. If enabled for your account, [custom roles](/admin/team-management/custom-roles) (beta) can grant these permissions one by one. See [Team member roles](/admin/team-management/overview#team-member-roles) for what each role can do. diff --git a/admin/team-management/overview.mdx b/admin/team-management/overview.mdx index db0e7960..f7ef504f 100644 --- a/admin/team-management/overview.mdx +++ b/admin/team-management/overview.mdx @@ -9,7 +9,7 @@ This section covers all aspects of administering your Checkly account, from mana ## Team Member Roles -Checkly has five built-in roles: **Owner**, **Admin**, **Read & Write**, **Read & Run**, and **Read Only**. Each role inherits all permissions from the roles below it. On Enterprise plans, you can also create [custom roles](/admin/team-management/custom-roles) (beta) with the permissions you pick. +Checkly has five built-in roles: **Owner**, **Admin**, **Read & Write**, **Read & Run**, and **Read Only**. Each role inherits all permissions from the roles below it. If [custom roles](/admin/team-management/custom-roles) (beta) are enabled for your account, you can also define roles with the permissions you pick. Each account has one Owner. The account creator is the Owner until they [transfer ownership](#transferring-ownership). You can invite teammates with any other role. You can [change a member's role](/admin/team-management/managing-team-members#change-members-roles) after they join. @@ -28,6 +28,30 @@ Each account has one Owner. The account creator is the Owner until they [transfe | Create service API keys | ✓ | ✓ | ✗ | ✗ | ✗ | | Transfer ownership | ✓ | ✗ | ✗ | ✗ | ✗ | +### Permissions and roles + +Checkly defines permissions for actions on resources. Your account role grants those permissions. In API and MCP documentation, an ID such as `checks:run` identifies the **Run checks** permission; the format is `resource:action`. + +The permissions referenced in these guides use the following names. If [custom roles](/admin/team-management/custom-roles#the-permission-matrix) (beta) are enabled, you can find these names under **Account settings > Roles**, in the listed group and resource row. The editor shows permission names, rather than IDs. + +| Permission ID | Name in the role editor | Group and resource | +| --- | --- | --- | +| `api-keys:create` | Create API keys | Account > API keys | +| `api-keys:delete` | Delete API keys | Account > API keys | +| `members:create` | Invite members | Account > Members | +| `checks:run` | Run checks | Monitoring > Checks and monitors | +| `checks:update` | Edit checks | Monitoring > Checks and monitors | +| `check-repairs:create` | Repair checks with Rocky | Rocky AI > Check repairs | +| `root-cause-analyses:run` | Start root cause analyses | Rocky AI > Root cause analyses | +| `environment-variables:create` | Create environment variables | Check resources > Environment variables | +| `environment-variables:update` | Edit environment variables | Check resources > Environment variables | +| `incidents:create` | Open incidents | Communicate > Incidents | +| `incidents:update` | Update incidents | Communicate > Incidents | +| `status-pages:read` | View status pages | Communicate > Status pages | +| `usage:read` | View usage | Billing > Usage | + +To get access, ask your account owner or an admin to [assign a role](/admin/team-management/managing-team-members#change-members-roles) with the required permissions. Use the built-in role table above to choose a role. On accounts with custom roles enabled, someone who can edit roles can adjust a role's permissions, within the [limits on granting access](/admin/team-management/custom-roles#the-permission-matrix). Custom roles are not generally available, so the Roles page may not appear in your account. + ### Choosing the Right Role | Role | Best for | diff --git a/ai/ai-assistant.mdx b/ai/ai-assistant.mdx index 0f5b713b..ce6dec55 100644 --- a/ai/ai-assistant.mdx +++ b/ai/ai-assistant.mdx @@ -21,10 +21,22 @@ Every check it creates includes an intent: the goal and required outcomes the ch 1. Open **AI Assistant** in the Checkly app and select **New**. 2. Describe the site, app, or API you want to monitor. Include the starting point, the journey, and the observable outcome that should pass. -3. Review the proposed monitoring plan. Ask the assistant to adjust the flow, assertions, frequency, locations, or alerting before you approve it. +3. Review the proposed monitoring plan and linked Test Session results. Ask the assistant to adjust the flow, assertions, frequency, locations, or alerting before you approve it. 4. Select the checks you want and choose **Create checks**. -5. Open each new check, run it, and review its behavior and settings. For Browser and Multistep Checks, also review **Check intent** under **Automatic Repair**. +5. Open each new check, run it, and review its behavior and settings. For Browser and Multistep Checks, also review the **Check intent** section in the check settings. + +While preparing your plan, the Assistant tests Browser, Multistep, and API Checks in recorded Test Sessions. URL and Heartbeat Monitors are outside this testing step. For a browser journey, ask for a Browser Check. For an API endpoint, ask for an API Check; for several related requests, ask for a Multistep Check. Use a URL Monitor for site availability or a Heartbeat Monitor for a scheduled job. If a journey has important negative cases or guardrails, describe them in your request and confirm they appear in the generated intent. The AI Assistant does not change a check's automatic repair setting when it creates the check. A Browser or Multistep Check inherits the account default until you [set automatic repair for that check](../resolve/automatic-check-repair#enable-automatic-repair). Review its intent and choose **On** if you want to enable repair explicitly. + +## Checks that need project files + +Standalone creation cannot attach a custom Playwright configuration, visual snapshots, or setup/teardown script files and their helper files. If your check needs these, use a [local Checkly CLI project](../cli/overview) and deploy it. + +## Resume a conversation + +Open **Conversations** to revisit previous conversations. Choose **All conversations** to search the history. You can continue a conversation you started while its workspace is still available and it has remaining conversation usage. + +A conversation becomes read-only when another member owns it, its workspace expires, or it reaches its conversation usage limit. You can still read its history. Only the owner can continue their conversation. For an expired workspace or a conversation usage limit, choose **Start a new conversation** and describe what you want to continue. Its first run uses one invocation from your [account's AI allowance](./rocky-ai#consumption-and-invocations). diff --git a/ai/mcp-server.mdx b/ai/mcp-server.mdx index 9632f7f9..84499ca4 100644 --- a/ai/mcp-server.mdx +++ b/ai/mcp-server.mdx @@ -16,8 +16,7 @@ https://api.checklyhq.com/mcp Use it when your agent needs live Checkly account context, check status, check results, test sessions, root cause analyses, result assets, status pages, incidents, account environment variables, or when it needs to trigger existing checks. -Use [Checkly Skills](/ai/skills) and the [Checkly CLI](/cli/overview) when your agent needs to create, edit, test, bundle, -or deploy check code. The MCP server runs remotely and cannot access your local filesystem. +You can create and update standalone Browser and Multistep Checks through MCP using inline scripts. Use [Checkly Skills](/ai/skills) and the [Checkly CLI](/cli/overview) when your agent needs to work with local project files, bundle code, or deploy a project. The MCP server runs remotely and cannot access your local filesystem. OAuth is the recommended way to connect. The OAuth flow only supports Checkly-approved clients listed in setup. User and service API keys are available for clients that can send custom headers when OAuth is unavailable or a workflow needs a stable service identity. @@ -27,7 +26,7 @@ or deploy check code. The MCP server runs remotely and cannot access your local ## What you can do - Inspect account membership, plan, and feature entitlements. -- Create,read update and delete synthetic & uptime monitors and groups. +- Create, read, update, and delete synthetic checks, uptime monitors, and groups. - Check which checks are passing, failing, muted, or deactivated. - Read compact check results and test session results. - Fetch normalized result asset manifests and bounded text assets. diff --git a/ai/mcp-server/security-and-permissions.mdx b/ai/mcp-server/security-and-permissions.mdx index 039369a6..d66e210a 100644 --- a/ai/mcp-server/security-and-permissions.mdx +++ b/ai/mcp-server/security-and-permissions.mdx @@ -5,7 +5,7 @@ sidebarTitle: 'Security & permissions' canonical: 'https://www.checklyhq.com/docs/ai/mcp-server/security-and-permissions/' --- -The Checkly MCP Server uses bearer-token authentication and Checkly account authorization together. Session permissions decide which tools are visible to your MCP client. Checkly account membership, role, and feature entitlements decide whether a specific tool call can operate on an account. +The Checkly MCP Server uses bearer-token authentication and Checkly account authorization together. Session permissions decide which tools are visible to your MCP client. Checkly account membership, [role and permissions](/admin/team-management/overview#permissions-and-roles), and feature entitlements decide whether a specific tool call can operate on an account. ## Authentication @@ -58,16 +58,19 @@ Blocked accounts can be inspected with read-only tools, but write, run, and admi ## Role checks -Some tools require both an MCP session permission and a Checkly account role: +MCP session permissions and account permissions are separate. A session permission makes a tool available; your account role must also grant the permissions needed to use it. This applies to built-in roles and, when enabled for your account, [custom roles](/admin/team-management/custom-roles) (beta). See [permissions and roles](/admin/team-management/overview#permissions-and-roles) for permission names and how to get access: | Tool or action | Additional account access required | | --- | --- | -| `invite-account-member` | Owner or Admin | -| `create-account-environment-variable` | Write access | -| `update-account-environment-variable` | Write access | -| `trigger-checks` | Run access | -| `trigger-root-cause-analysis` | Run access | -| Status page incident writes | Access required by the underlying incident action | +| `invite-account-member` | [`members:create`](/admin/team-management/overview#permissions-and-roles), plus all permissions of the role you give | +| `create-account-environment-variable` | [`environment-variables:create`](/admin/team-management/overview#permissions-and-roles) | +| `update-account-environment-variable` | [`environment-variables:update`](/admin/team-management/overview#permissions-and-roles) | +| `trigger-checks` | [`checks:run`](/admin/team-management/overview#permissions-and-roles) | +| `trigger-root-cause-analysis` | [`root-cause-analyses:run`](/admin/team-management/overview#permissions-and-roles) | +| `create-status-page-incident` | [`incidents:create`](/admin/team-management/overview#permissions-and-roles) and [`status-pages:read`](/admin/team-management/overview#permissions-and-roles) | +| `update-status-page-incident`, `resolve-status-page-incident` | [`incidents:update`](/admin/team-management/overview#permissions-and-roles) | + +`invite-account-member` supports Admin, Read & Write, Read & Run, and Read Only invitations. You cannot invite an Owner or assign a [custom role](/admin/team-management/custom-roles) (beta) through this tool. ## Write-action safety diff --git a/ai/mcp-server/tools.mdx b/ai/mcp-server/tools.mdx index 694ff478..9ee1522c 100644 --- a/ai/mcp-server/tools.mdx +++ b/ai/mcp-server/tools.mdx @@ -7,7 +7,7 @@ canonical: 'https://www.checklyhq.com/docs/ai/mcp-server/tools/' This reference reflects the current MCP Server v1 tool surface. It is not a roadmap or a commitment to future tools. -Each tool is shown only when the MCP session has the required permission. Some write tools also require a matching Checkly account role, such as Admin or Read & Write. +Each tool is shown only when the MCP session has the required permission. Your [account role and permissions](/admin/team-management/overview#permissions-and-roles) must also allow the action. See [security and permissions](/ai/mcp-server/security-and-permissions#role-checks). ## Account tools @@ -16,7 +16,7 @@ Each tool is shown only when the MCP session has the required permission. Some w | `whoami` | Read | Return the authenticated user and account memberships visible to the MCP session. | | `get-account-entitlements` | Read | Return the resolved account plan and feature entitlement state. | | `list-account-members` | Read | List account members and pending invites. | -| `invite-account-member` | Write | Invite a user to the resolved account. OAuth-only. Requires Owner or Admin access, sends an invite email, and is not idempotent. | +| `invite-account-member` | Write | Invite a user to the resolved account. OAuth-only. Requires [`members:create`](/admin/team-management/overview#permissions-and-roles) and all permissions of the built-in role you give. Owner and [custom role](/admin/team-management/custom-roles) (beta) invitations are unsupported. Sends an invite email and is not idempotent. | | `prepare-local-check-authoring` | Read | Return an account-aware runbook for creating, testing, and deploying check code with the local Checkly CLI. | Example prompts: @@ -236,7 +236,7 @@ Draft a major status page incident for the API outage and wait for confirmation | `get-usage-summary` | Read | Return organization usage totals for a date range plus credit projections, including the `CHECK_RUN` and `AI_INVOCATION` meters. | | `list-usage-series` | Read | Return organization usage per period, grouped by account and/or check type, as a zero-filled, cursor-paginated series. Supports `day`, `week`, `month`, and `total` intervals. | -Usage tools report on the whole organization the account belongs to. Use `accountIds` and `checkTypes` to narrow a report. These tools require an organization usage contract and Owner or Admin access. +Usage tools report on the whole organization the account belongs to. Use `accountIds` and `checkTypes` to narrow a report. These tools require an organization usage contract and [`usage:read`](/admin/team-management/overview#permissions-and-roles) account permission, which Owner and Admin have by default. Example prompt: diff --git a/ai/mcp-server/troubleshooting.mdx b/ai/mcp-server/troubleshooting.mdx index 03dffd04..386c6fae 100644 --- a/ai/mcp-server/troubleshooting.mdx +++ b/ai/mcp-server/troubleshooting.mdx @@ -99,11 +99,11 @@ Service API keys are already scoped to one account. If a service-key session rep ## Local check authoring -### The MCP server cannot create or deploy check code +### The MCP server cannot access or deploy my local project -The MCP server runs remotely and cannot access your local project files. It cannot author, bundle, test, or deploy check code. +You can create and update standalone Browser and Multistep Checks through MCP using inline scripts. The MCP server runs remotely and cannot access your local project files, bundle code, or deploy a project. -Ask for a CLI handoff instead: +For local project authoring, ask for a CLI handoff: ```text title="Prompt" wrap Use Checkly to prepare the local check authoring runbook for a browser check. diff --git a/cli/monitoring-pull-requests.mdx b/cli/monitoring-pull-requests.mdx index c62d6943..4dc16b82 100644 --- a/cli/monitoring-pull-requests.mdx +++ b/cli/monitoring-pull-requests.mdx @@ -10,6 +10,7 @@ Review monitoring changes in GitHub, test them, and deploy the merged code to Ch - Access to edit the resource and use your Checkly account's GitHub integration. +- If GitHub is not connected yet, the GitHub personal account owner or an active organization owner must connect it. - A local checkout of the repository and the [CLI prerequisites](/cli/installation), including a Node.js version supported by your Checkly version. - The [GitHub CLI](https://cli.github.com/manual/gh_pr_checkout) for the checkout command below. You can also use GitHub's PR checkout instructions. @@ -17,7 +18,7 @@ Review monitoring changes in GitHub, test them, and deploy the merged code to Ch ## 1. Connect GitHub and choose a repository -Open the resource's **Export to code** dialog and select **Code**. If prompted, connect GitHub from Checkly and grant the App access to your repository. See [GitHub integration setup](/integrations/ci-cd/github/deployments#set-up-github-deployment-hooks) for installation help; you don't need a deployment hook for this workflow. +Open the resource's **Export to code** dialog and select **Code**. If prompted, authorize GitHub from Checkly, choose an installation you own, and click **Connect**. Grant the App access to your repository. See [GitHub integration setup](/integrations/ci-cd/github/deployments#set-up-github-deployment-hooks) for installation help; you don't need a deployment hook for this workflow. For a UI-created resource, choose the repository to export into. If it has multiple Checkly configs, select the folder for your project. Checkly can add the resource to a project already deployed to this account or create a config for a new project. @@ -148,7 +149,7 @@ Deploying manual import code before applying its plan creates duplicates. You ca | Problem | Next step | | --- | --- | -| Repository missing or GitHub error | Check the App's access to this account and repository. Look for an existing PR before retrying: it may have been created despite the error. The reservation stays protected; use `checkly import cancel` to cancel its uncommitted plan if you're abandoning the export. | +| Repository missing or GitHub error | Check the App's connection to this Checkly account and its repository access. For a missing account or organization, follow [GitHub connection troubleshooting](/integrations/ci-cd/github/deployments#github-account-or-organization-missing). Look for an existing PR before retrying: it may have been created despite the error. The reservation stays protected; use `checkly import cancel` to cancel its uncommitted plan if you're abandoning the export. | | Checkly can't select the project | Choose the config folder and check its `logicalId`, `repoUrl`, and account. An existing project must already be deployed to this account. Use [manual import](/cli/importing) if automatic export can't use it. | | No source file, several candidates, or unsupported sync | Confirm the file that declares the resource. If Checkly can't patch it, edit the code directly. For a resource still managed in the UI, try manual import if the CLI supports its type. A project that only **References** a resource can't sync it. | | Dependencies, sign-in, secrets, or tests fail | Install dependencies in the config's package, use a supported Node.js version, check `whoami`, and supply the required secrets. Fix the reported error and rerun the test or preview. | diff --git a/constructs/browser-check.mdx b/constructs/browser-check.mdx index 4f20457e..e0a9ccdc 100644 --- a/constructs/browser-check.mdx +++ b/constructs/browser-check.mdx @@ -84,6 +84,7 @@ The Browser Check configuration consists of specific Browser Check options and i | `name` | `string` | ✅ | - | Friendly name for your check | | `description` | `string` | ❌ | `null` | A description of the check. Supports markdown. Max 500 characters | | `intent` | `CheckIntent` | ❌ | - | Structured guidance for root cause analysis and check repair. See [General Check Options](#general-check-options). | +| `aiAutoRepairEnabled` | `boolean \| null` | ❌ | - | Enable or disable automatic repair, or inherit the account setting. Requires CLI 9.4.0 or later. See [General Check Options](#general-check-options). | | `activated` | `boolean` | ❌ | `true` | Whether the check is enabled | | `alertChannels` | `AlertChannel[]` | ❌ | `[]` | Array of AlertChannel objects for notifications | | `alertEscalationPolicy` | `AlertEscalationPolicy` | ❌ | - | Advanced alert settings | @@ -442,6 +443,12 @@ new BrowserCheck("feature-test", { + +Requires Checkly CLI 9.4.0 or later. Set `true` to enable automatic repair, `false` to disable it, or `null` to follow your account default. Omit the property to preserve an existing setting when you deploy. + +You also need a valid check intent. See [Configure automatic repair](../resolve/automatic-check-repair#define-intent-with-monitoring-as-code) for a complete example and repair prerequisites. + + ## Examples diff --git a/constructs/multistep-check.mdx b/constructs/multistep-check.mdx index a701e75e..cd8ba17d 100644 --- a/constructs/multistep-check.mdx +++ b/constructs/multistep-check.mdx @@ -82,6 +82,7 @@ new MultiStepCheck("complex-multistep-check", { | `name` | `string` | ✅ | - | Friendly name for your check | | `description` | `string` | ❌ | `null` | A description of the check. Supports markdown. Max 500 characters | | `intent` | `CheckIntent` | ❌ | - | Structured guidance for root cause analysis and check repair. See [General Check Options](#general-check-options). | +| `aiAutoRepairEnabled` | `boolean \| null` | ❌ | - | Enable or disable automatic repair, or inherit the account setting. Requires CLI 9.4.0 or later. See [General Check Options](#general-check-options). | | `activated` | `boolean` | ❌ | `true` | Whether the check is enabled | | `alertChannels` | `AlertChannel[]` | ❌ | `[]` | Array of AlertChannel objects for notifications | | `alertEscalationPolicy` | `AlertEscalationPolicy` | ❌ | - | Advanced alert settings | @@ -367,3 +368,9 @@ Multistep checks are only supported on runtime `2023.09` or later. Make sure to + + +Requires Checkly CLI 9.4.0 or later. Set `true` to enable automatic repair, `false` to disable it, or `null` to follow your account default. Omit the property to preserve an existing setting when you deploy. + +You also need a valid check intent. See [Configure automatic repair](../resolve/automatic-check-repair#define-intent-with-monitoring-as-code) for a complete example and repair prerequisites. + diff --git a/detect/synthetic-monitoring/agentic-checks/migrate.mdx b/detect/synthetic-monitoring/agentic-checks/migrate.mdx index a5c057f5..2cda536e 100644 --- a/detect/synthetic-monitoring/agentic-checks/migrate.mdx +++ b/detect/synthetic-monitoring/agentic-checks/migrate.mdx @@ -56,7 +56,7 @@ Available on Trial, Starter, Team, and Enterprise plans. ## Verify and switch over 1. Run the replacement check and confirm it reports the expected result. Correct the script or settings if needed. Match the original check's frequency, locations, and alerting before relying on the replacement. -2. In the replacement check's **Automatic Repair** settings, review its intent, select **On** under **Automatic repair behavior**, and save. If the check has no intent yet, select **Initialize from saved check** first. [Learn how automatic repair works](../../../resolve/automatic-check-repair). +2. In the replacement check's **Check intent** settings, review and save its intent. If it has no intent yet, select **Initialize from saved check** first. Then expand **Automatic Repair**, select **On** under **Automatic repair behavior**, and save. [Learn how automatic repair works](../../../resolve/automatic-check-repair). 3. Once the replacement is running as expected, deactivate the Agentic Check. Keep it for its existing configuration and history, or remove it later according to your retention needs. @@ -65,7 +65,7 @@ Migrate one journey at a time. If the original prompt covered several unrelated ## Add automatic repair to other checks -You can enable automatic repair on Browser and Multistep Checks you already use. Open a check's **Automatic Repair** settings and select **Initialize from saved check** to get an editable intent suggestion, or enter the intent manually. Review and save the intent, then set **Automatic repair behavior** to **On** and save the check. See [Add intent to an existing check](../../../resolve/automatic-check-repair#add-intent-to-an-existing-check). +You can enable automatic repair on Browser and Multistep Checks you already use. Open a check's **Check intent** settings and select **Initialize from saved check** to get an editable intent suggestion, or enter the intent manually. Review and save the intent, then expand **Automatic Repair** and set **Automatic repair behavior** to **On** and save the check. See [Add intent to an existing check](../../../resolve/automatic-check-repair#add-intent-to-an-existing-check). ## AI invocation usage diff --git a/integrations/ci-cd/github/deployments.mdx b/integrations/ci-cd/github/deployments.mdx index c3e6fb97..9dfdb73f 100644 --- a/integrations/ci-cd/github/deployments.mdx +++ b/integrations/ci-cd/github/deployments.mdx @@ -8,13 +8,14 @@ canonical: 'https://www.checklyhq.com/docs/integrations/ci-cd/github/deployments Use GitHub deployment hooks when you want Checkly to listen for GitHub `deployment_status` events and trigger checks linked to a repository. This is separate from the [Checkly GitHub Action](/integrations/ci-cd/github/actions), which starts test sessions from a GitHub Actions workflow. - - The Checkly GitHub integration installed from the [Account settings > Integrations](https://app.checklyhq.com/settings/account/integrations) page of the Checkly account that owns the checks. + - To connect the integration, you must be the GitHub personal account owner or an active organization owner. Repository access or ordinary organization membership is not enough. + - The Checkly GitHub integration connected from the [Account settings > Integrations](https://app.checklyhq.com/settings/account/integrations) page of the Checkly account that owns the checks. - A deployment provider that sends successful GitHub `deployment_status` events, such as Vercel for GitHub or Heroku Pipelines. - One or more Checkly checks linked to the GitHub repository from the check's **CI/CD** settings. - Start the installation from [Account settings > Integrations](https://app.checklyhq.com/settings/account/integrations) in Checkly and click **Integrate with GitHub**. Grant the installation access to the repository you want to link. Installing the app directly from GitHub does not connect it to the Checkly account, so deployment hooks and GitHub Check reporting will not work correctly. + Connect the installation from [Account settings > Integrations](https://app.checklyhq.com/settings/account/integrations) in Checkly and grant the App access to the repository you want to link. Installing the app directly from GitHub does not connect it to the Checkly account, so deployment hooks and GitHub Check reporting will not work correctly. GitHub `deployment_status` webhook events are triggered whenever a deployment service reports a successful deployment. This works with services that report successful deployment statuses through the GitHub Deployments API, including: @@ -29,10 +30,11 @@ GitHub `deployment_status` webhook events are triggered whenever a deployment se ## Set up GitHub deployment hooks -1. Go to [Account settings > Integrations](https://app.checklyhq.com/settings/account/integrations) and click **Integrate with GitHub**. -2. In GitHub, choose the account where you want to install the Checkly GitHub App and grant it access to the repositories you want to connect. After installation, GitHub redirects you to Checkly. -3. Open the check or check group you want to trigger and go to its **CI/CD** settings. -4. Click **Link GitHub repo** and select the repository that reports your deployment events. +1. Go to [Account settings > Integrations](https://app.checklyhq.com/settings/account/integrations), click **Integrate with GitHub**, then **Continue with GitHub**. +2. Authorize GitHub, choose a Checkly App installation you own, and click **Connect**. You can connect the same installation to multiple Checkly accounts. +3. If you need a new installation or different repository access, use **Install on another GitHub account** or **Manage repository access**. Return to Checkly, click **Refresh**, then **Connect**. +4. Open the check or check group you want to trigger and go to its **CI/CD** settings. +5. Click **Link GitHub repo** and select the repository that reports your deployment events. After you link the repository, Checkly listens for successful deployment events and runs the linked check or check group. You can also choose a run location, filter deployments by environment or environment URL, and include screenshots in the pull request comment. @@ -40,6 +42,12 @@ After you link the repository, Checkly listens for successful deployment events You can link multiple checks to the same repository. Checkly runs the linked checks as a test suite when GitHub reports a successful deployment. +### GitHub account or organization missing + +Only installations you own appear in the chooser. Click **Use a different GitHub account** to sign in as the personal account owner or an organization owner. + +If organization ownership cannot be verified, ask an organization owner to approve **Members: read** in the Checkly App's GitHub installation settings and authorize organization SSO, then authorize GitHub again. + ## Manage deployment hooks with the API Use the Checkly Public API to create GitHub deployment hooks programmatically. A deployment trigger represents the link between one GitHub repository and either a check or a check group. diff --git a/platform/views.mdx b/platform/views.mdx index 40f457ee..22b54344 100644 --- a/platform/views.mdx +++ b/platform/views.mdx @@ -103,7 +103,7 @@ Your own private views and the order of your bar need no permission. Shared view | Edit shared views | Rename shared views and change their filters and counter | Owner, Admin, Read & Write | | Delete shared views | Delete shared views, or make one private again | Owner, Admin, Read & Write | -Custom roles can grant these permissions one by one. They're listed under **Shared views**. See [Team member roles](/admin/team-management/overview#team-member-roles) for the built-in roles. +[Custom roles](/admin/team-management/custom-roles) (beta) can grant these permissions one by one when enabled for your account. They're listed under **Shared views**. See [Team member roles](/admin/team-management/overview#team-member-roles) for the built-in roles. ## Arrange your bar diff --git a/resolve/ai-root-cause-analysis/overview.mdx b/resolve/ai-root-cause-analysis/overview.mdx index e153bb40..8a593159 100644 --- a/resolve/ai-root-cause-analysis/overview.mdx +++ b/resolve/ai-root-cause-analysis/overview.mdx @@ -105,6 +105,12 @@ Root cause: The check is calling a ... +## Check intent and repair + +For API, Browser, Multistep, and Playwright Checks, the **Check intent** card beside the analysis lets you select **Define intent** or **Edit intent** to open the check's settings. Use it to record the outcomes the check must prove and behavior a repair must preserve. Intent informs root cause analysis independently of whether you enable automatic repair. + +For Browser and Multistep Checks, select **Repair with AI** on a completed analysis to attempt a script fix. See [Start a manual repair](../automatic-check-repair#start-a-manual-repair) for prerequisites, repair history, and restoring the original script. + ## Key Capabilities - **Categorize**: Rocky AI determines if an error is of a specific category. For example: is it a typo in your code or is it an infrastructure error? diff --git a/resolve/automatic-check-repair.mdx b/resolve/automatic-check-repair.mdx index e585c10c..a6e7bdf0 100644 --- a/resolve/automatic-check-repair.mdx +++ b/resolve/automatic-check-repair.mdx @@ -1,29 +1,32 @@ --- title: 'Automatic check repair' sidebarTitle: 'Automatic Repair' -description: 'Set a check intent and enable automatic repair for Browser and Multistep Checks.' +description: 'Configure automatic repair, start a manual attempt, and restore a Browser or Multistep Check script.' canonical: 'https://www.checklyhq.com/docs/resolve/automatic-check-repair/' --- -For a check managed in Checkly, automatic repair can attempt to fix a Browser or Multistep Check when its script fails because of a check issue. A check intent states the goal, required outcomes, and guardrails that a repair must preserve. A failed check is not automatically repaired just because this setting is on; Rocky AI first needs a repairable failure and a valid intent. +Automatic repair can attempt to fix a Browser or Multistep Check when its script fails because of a check issue. A check intent states the goal, required outcomes, and guardrails that a repair must preserve. A failed check is not automatically repaired just because this setting is on; Rocky AI first needs a repairable failure and a valid intent. -Browser and Multistep Checks linked to a GitHub source file can also receive a repair proposal through a pull request. Merge and deploy that change to update the running check. +Repair tests the proposed script before saving it to the running check. For a check managed as code, repair can also create a GitHub pull request to keep its source file in sync. Review, merge, and deploy that change so a later deployment does not overwrite the repair. -- A saved Browser or Multistep Check -- Permission to edit the check -- Rocky AI enabled for automatic root cause analysis, with AI invocations available for analysis and repair +- A saved Browser or Multistep Check with at least one public execution location +- [Edit checks and Repair checks with Rocky permissions](/admin/team-management/overview#permissions-and-roles) (`checks:update` and `check-repairs:create`) +- Rocky AI enabled, with access to check repair and AI invocations available for analysis and repair +- For Monitoring as Code configuration: Checkly CLI 9.4.0 or later ## Enable automatic repair -1. Open the check editor and expand **Automatic Repair** in the check settings. -2. Review the **Check intent**. It needs a goal and at least one required outcome or guardrail. Include the outcomes the check must still prove, such as a successful sign-in or an order confirmation. Add guardrails for behavior a repair must not weaken, such as authentication. -3. Under **Automatic repair behavior**, select **On** to enable repair for this check. You can also select **Account default** to follow the account setting. +1. Open the check editor and expand **Check intent** in the check settings. +2. Review the intent. It needs a goal and at least one required outcome or guardrail. Include the outcomes the check must still prove, such as a successful sign-in or an order confirmation. Add guardrails for behavior a repair must not weaken, such as authentication. +3. Expand **Automatic Repair**. Under **Automatic repair behavior**, select **On** to enable repair for this check. You can also select **Account default** to follow the account setting. 4. Save the check. +Enable automatic root cause analysis in the [Rocky AI settings](https://app.checklyhq.com/accounts/rocky-ai) so new failures can trigger repair. + An AI-generated check already has an intent. Review it before enabling repair. Repair cannot start without a valid intent, even if the check or account setting is on. ## Add intent to an existing check @@ -34,7 +37,7 @@ You can also write the intent yourself. Use this option when the script does not ## Define intent with Monitoring as Code -You can also define `intent` in your [Monitoring as Code](/cli/overview) configuration using the Checkly CLI. Keep the goal, required outcomes, and guardrails alongside your check definition. See the General Check Options in the [Browser Check](/constructs/browser-check#general-check-options) and [Multistep Check](/constructs/multistep-check#general-check-options) construct references for the `intent` schema and a complete configuration example. +You can also define `intent` and enable repair with `aiAutoRepairEnabled` in your [Monitoring as Code](../cli/overview) configuration using the Checkly CLI. Keep the goal, required outcomes, and guardrails alongside your check definition. See the General Check Options in the [Browser Check](../constructs/browser-check#general-check-options) and [Multistep Check](../constructs/multistep-check#general-check-options) construct references for the `intent` schema and a complete configuration example. Use your existing Browser or Multistep Check test file in these examples. @@ -46,6 +49,8 @@ import { BrowserCheck } from "checkly/constructs" new BrowserCheck("dashboard-browser", { name: "Authenticated dashboard", + locations: ["eu-west-1"], + aiAutoRepairEnabled: true, code: { entrypoint: "./dashboard.spec.ts", }, @@ -73,6 +78,8 @@ import { MultiStepCheck } from "checkly/constructs" new MultiStepCheck("authenticated-api", { name: "Authenticated API flow", + locations: ["eu-west-1"], + aiAutoRepairEnabled: true, code: { entrypoint: "./authenticated-api.spec.ts", }, @@ -95,7 +102,29 @@ new MultiStepCheck("authenticated-api", { -Defining `intent` does not by itself enable automatic repair or change which check types support Checkly-managed repair. +Set `aiAutoRepairEnabled` to `true` to enable automatic repair or `false` to disable it for the check. Set it to `null` to follow your account default. Omit the property to preserve an existing setting when you deploy. Defining `intent` alone does not enable repair. + +## Start a manual repair + +1. Open a failed check's completed [root cause analysis](./ai-root-cause-analysis/overview). +2. Select **Repair with AI** to start an attempt. You need a saved, valid check intent, but automatic repair does not need to be enabled. +3. Select **View repair** to review the attempt, its test results, and any script change or pull request. + +To revisit earlier attempts, open the menu beside **View repair** labeled **Repair history and actions**. Select **Start another repair** to retry after an attempt finishes. Each new attempt uses another repair invocation. You can also open an attempt from its repair marker on the check timeline. + + +Repair tests the candidate in the check's first configured public location. Checks configured only on private locations are not supported by this workflow. Its test does not reproduce a custom per-check Playwright configuration; review the results and verify the check with your normal configuration. + + +## Restore the original script + +In the repair conversation, find **Script saved before this repair**, select **View script**, then **Restore this script**. You need [Edit checks permission](/admin/team-management/overview#permissions-and-roles), and any pending or running repair of that check must finish first. + +You can restore scripts for checks managed in the Checkly UI. Restore repository-managed checks through your repository. This action restores only the saved script, not dependencies or other check settings. + +## Repair notifications + +Your alert emails can link to a repair attempt. A recovery email credits an in-place repair after a passing check result confirms recovery. A successful repair test or proposed pull request alone does not confirm recovery. ## AI invocation usage