feat(gateway): support Google Public CA with external account binding

docs/certbot-dns-persist-01.md

@@ -25,7 +25,8 @@ Cloudflare account, no API token, no provider integration.
 > Let's Encrypt has said it will not deploy the challenge to production until
 > that is resolved. It is live on Let's Encrypt **staging** and in
 > [Pebble][pebble]. Treat the record format as unstable, and expect to
-> republish when the draft settles.
+> republish when the draft settles. Google Trust Services also supports draft-01
+> in production; see [Google Public CA](#google-public-ca) below.
 
 [draft]: https://datatracker.ietf.org/doc/html/draft-ietf-acme-dns-persist-01
 [pebble]: https://github.com/letsencrypt/pebble
@@ -181,6 +182,76 @@ script, an older SDK — cannot downgrade a `dns-persist-01` domain to `dns-01`
 editing something else, which its hand-published CAA would then refuse. Send the
 field explicitly to switch a domain over.
 
+### Google Public CA
+
+Google Public CA can issue certificates for gateways hosted outside Google Cloud.
+A Google Cloud project is required to register the ACME account, not to host the
+service. GTS supports `dns-persist-01` draft-01 in production as of September 28,
+2026; the protocol is still experimental and a final standard may require DNS
+record changes.
+
+1. Create or select a Google Cloud project, enable `publicca.googleapis.com`, and
+   obtain the `roles/publicca.externalAccountKeyCreator` role. Follow the
+   [Public CA tutorial][gts-tutorial] to create an external account binding (EAB)
+   key. Treat its HMAC value as a secret: do not paste it into logs, shell history,
+   or issue reports.
+2. Call `Admin.SetCertbotConfig` with:
+
+   ```json
+   {
+     "acme_url": "https://dv.acme-v02.api.pki.goog/directory",
+     "issuer_domain_name": "pki.goog"
+   }
+   ```
+
+   For testing, use `https://dv.acme-v02.test-api.pki.goog/directory` instead.
+   Staging certificates are not publicly trusted; use an EAB key created for the
+   corresponding environment as described in the tutorial.
+3. Before adding domains, call the authenticated admin RPC
+   `Admin.RotateAcmeCredentials` with a request body containing:
+
+   ```json
+   {
+     "eab_kid": "<EAB key ID>",
+     "eab_hmac_key": "<base64url-encoded EAB HMAC key>"
+   }
+   ```
+
+   Send the real values through a protected request body over a trusted channel,
+   not as command-line arguments. Both fields are required when using EAB;
+   padded and unpadded base64url are accepted. The gateway uses them only for this
+   registration, never stores them in KV, and never returns them. Existing
+   clients and CAs without EAB can still send `{}`.
+4. Add the ZT domain with `challenge: "dns-persist-01"`, then publish the exact
+   `required_dns_records` returned by the gateway. For example:
+
+   ```dns
+   _validation-persist.app.example.com. IN TXT "pki.goog; accounturi=<returned account URI>; policy=wildcard"
+   ```
+
+   Update existing CAA restrictions as well: they must allow `pki.goog`, the new
+   account URI, and `dns-persist-01`, rather than pinning issuance to the previous
+   CA or account. Renew the domain certificate once DNS has propagated.
+
+Each EAB key can register only one account. Renewals reuse the stored ACME
+account and need neither EAB nor Google Cloud credentials. A later account
+rotation requires a fresh EAB key and republishing persistent TXT and CAA
+records. Every rotation creates a new account; do not repeat it merely because
+manual DNS updates or CAA re-pinning are still outstanding.
+
+The ACME directory and issuer are global to the cluster: switching an existing
+cluster from Let's Encrypt affects every ZT domain, not just newly added ones.
+For `dns-01`, the same EAB registration flow works, but domains still need their
+DNS provider credentials. The dashboard's register-account button sends no EAB;
+use the admin RPC above for Google account registration and rotation.
+
+Google's default `newOrder` quota is 100 requests/hour per project, shared by all
+ACME accounts in that project. Stagger bulk issuance and renewals, and respect
+`Retry-After` on rate-limited responses. See [Public CA quotas][gts-quotas].
+
+[gts-tutorial]: https://docs.cloud.google.com/certificate-manager/docs/public-ca-tutorial
+[gts-quotas]: https://docs.cloud.google.com/certificate-manager/docs/quotas#public_ca_request_quotas
+
 ### Naming the CA
 
 For a non-production ACME server, set `issuer_domain_name` in the global certbot

dstack/certbot/Cargo.toml

@@ -31,6 +31,7 @@ tracing.workspace = true
 x509-parser.workspace = true
 
 [dev-dependencies]
+base64.workspace = true
 rand.workspace = true
 tempfile.workspace = true
 tokio = { workspace = true, features = ["full"] }

dstack/certbot/src/acme_client.rs

@@ -10,7 +10,7 @@ use hickory_resolver::proto::rr::RData;
 use hickory_resolver::TokioResolver;
 use instant_acme::{
     Account, AccountCredentials, AuthorizationStatus, AuthorizedIdentifier, ChallengeType,
-    Identifier, NewAccount, NewOrder, Order, OrderStatus, Problem,
+    ExternalAccountKey, Identifier, NewAccount, NewOrder, Order, OrderStatus, Problem,
 };
 use rcgen::{CertificateParams, DistinguishedName, KeyPair};
 use serde::{Deserialize, Serialize};
@@ -418,7 +418,16 @@ impl AcmeClient {
     /// is forced under `dns-persist-01`: the record an operator publishes names
     /// the account, so the account has to exist first.
     pub async fn register_account(acme_url: &str) -> Result<AcmeAccount> {
-        let (_, credentials) = Self::register(acme_url).await?;
+        Self::register_account_with_eab(acme_url, None).await
+    }
+
+    /// Register an account using optional RFC 8555 external account binding.
+    /// The binding key is used only for registration and is not persisted.
+    pub async fn register_account_with_eab(
+        acme_url: &str,
+        external_account: Option<&ExternalAccountKey>,
+    ) -> Result<AcmeAccount> {
+        let (_, credentials) = Self::register(acme_url, external_account).await?;
         Ok(AcmeAccount {
             account_uri: credentials.account_id.clone(),
             credentials: serde_json::to_string(&credentials)?,
@@ -436,7 +445,7 @@ impl AcmeClient {
         // transient failure there would drop an account that exists at the CA
         // and counts against its registration limit, on a path whose caller has
         // not persisted anything yet.
-        let (account, credentials) = Self::register(acme_url).await?;
+        let (account, credentials) = Self::register(acme_url, None).await?;
         Ok(Self {
             account,
             validation,
@@ -447,7 +456,10 @@ impl AcmeClient {
 
     /// Register an account and keep both halves: the live account and the
     /// credentials that reconstruct it.
-    async fn register(acme_url: &str) -> Result<(Account, Credentials)> {
+    async fn register(
+        acme_url: &str,
+        external_account: Option<&ExternalAccountKey>,
+    ) -> Result<(Account, Credentials)> {
         let http_client = Box::new(ReqwestHttpClient::new()?);
         let (account, credentials) = Account::builder_with_http(http_client)
             .create(
@@ -457,7 +469,7 @@ impl AcmeClient {
                     only_return_existing: false,
                 },
                 acme_url.to_string(),
-                None,
+                external_account,
             )
             .await
             .with_context(|| format!("failed to create ACME account for {acme_url}"))?;
@@ -2090,3 +2102,121 @@ mod publish_tests {
         );
     }
 }
+
+#[cfg(test)]
+mod eab_tests {
+    use super::*;
+    use base64::{engine::general_purpose::URL_SAFE_NO_PAD, Engine};
+    use tokio::io::{AsyncReadExt, AsyncWriteExt};
+    use tokio::net::TcpListener;
+
+    #[tokio::test]
+    async fn registration_sends_optional_eab_without_persisting_it() {
+        for with_eab in [false, true] {
+            let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
+            let base = format!("http://{}", listener.local_addr().unwrap());
+            let server_base = base.clone();
+            let server = tokio::spawn(async move {
+                loop {
+                    let (mut stream, _) = listener.accept().await.unwrap();
+                    let mut data = Vec::new();
+                    let header_end = loop {
+                        let mut buf = [0; 4096];
+                        let n = stream.read(&mut buf).await.unwrap();
+                        assert_ne!(n, 0);
+                        data.extend_from_slice(&buf[..n]);
+                        if let Some(end) = data.windows(4).position(|w| w == b"\r\n\r\n") {
+                            break end + 4;
+                        }
+                    };
+                    let headers = String::from_utf8(data[..header_end].to_vec()).unwrap();
+                    let content_length: usize = headers
+                        .lines()
+                        .find_map(|line| {
+                            let (name, value) = line.split_once(':')?;
+                            name.eq_ignore_ascii_case("content-length")
+                                .then(|| value.trim().parse().unwrap())
+                        })
+                        .unwrap_or(0);
+                    while data.len() < header_end + content_length {
+                        let mut buf = [0; 4096];
+                        let n = stream.read(&mut buf).await.unwrap();
+                        assert_ne!(n, 0);
+                        data.extend_from_slice(&buf[..n]);
+                    }
+                    let is_account = headers.starts_with("POST /account ");
+                    let body = if headers.starts_with("GET /directory ") {
+                        serde_json::json!({
+                            "newNonce": format!("{server_base}/nonce"),
+                            "newAccount": format!("{server_base}/account"),
+                            "newOrder": format!("{server_base}/order")
+                        })
+                        .to_string()
+                    } else if is_account {
+                        let jws: serde_json::Value =
+                            serde_json::from_slice(&data[header_end..]).unwrap();
+                        let payload: serde_json::Value = serde_json::from_slice(
+                            &URL_SAFE_NO_PAD
+                                .decode(jws["payload"].as_str().unwrap())
+                                .unwrap(),
+                        )
+                        .unwrap();
+                        let binding = &payload["externalAccountBinding"];
+                        assert_eq!(!binding.is_null(), with_eab);
+                        if with_eab {
+                            let protected: serde_json::Value = serde_json::from_slice(
+                                &URL_SAFE_NO_PAD
+                                    .decode(binding["protected"].as_str().unwrap())
+                                    .unwrap(),
+                            )
+                            .unwrap();
+                            assert_eq!(protected["kid"], "test-eab-id");
+                            assert_eq!(protected["alg"], "HS256");
+                            assert_eq!(protected["url"], format!("{server_base}/account"));
+                            let outer: serde_json::Value = serde_json::from_slice(
+                                &URL_SAFE_NO_PAD
+                                    .decode(jws["protected"].as_str().unwrap())
+                                    .unwrap(),
+                            )
+                            .unwrap();
+                            let bound_key: serde_json::Value = serde_json::from_slice(
+                                &URL_SAFE_NO_PAD
+                                    .decode(binding["payload"].as_str().unwrap())
+                                    .unwrap(),
+                            )
+                            .unwrap();
+                            assert_eq!(bound_key, outer["jwk"]);
+                            assert!(!binding["signature"].as_str().unwrap().is_empty());
+                        }
+                        "{}".to_string()
+                    } else {
+                        assert!(headers.starts_with("HEAD /nonce "));
+                        String::new()
+                    };
+                    let response = format!(
+                        "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nReplay-Nonce: test-nonce\r\nLocation: {server_base}/acct/1\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", body.len()
+                    );
+                    stream.write_all(response.as_bytes()).await.unwrap();
+                    if is_account {
+                        break;
+                    }
+                }
+            });
+            let key = ExternalAccountKey::new("test-eab-id".into(), b"synthetic-test-hmac-key");
+            let account = tokio::time::timeout(
+                Duration::from_secs(5),
+                AcmeClient::register_account_with_eab(
+                    &format!("{base}/directory"),
+                    with_eab.then_some(&key),
+                ),
+            )
+            .await
+            .unwrap()
+            .unwrap();
+            server.await.unwrap();
+            assert_eq!(account.account_uri, format!("{base}/acct/1"));
+            assert!(!account.credentials.contains("test-eab-id"));
+            assert!(!account.credentials.contains("synthetic-test-hmac-key"));
+        }
+    }
+}

dstack/certbot/src/lib.rs

@@ -24,6 +24,7 @@ pub use acme_client::{
 pub use bot::{read_pubkey, CertBot, CertBotConfig};
 pub use dns01_client::Dns01Client;
 pub use dns_persist::{resolve_issuer_domain_name, LETS_ENCRYPT_ISSUER_DOMAIN_NAME};
+pub use instant_acme::ExternalAccountKey;
 pub use workdir::WorkDir;
 
 mod acme_client;

dstack/gateway/README.md

@@ -1,7 +1,7 @@
 # dstack-gateway
 
 The reverse proxy that gives deployed dstack apps zero-trust network access. It
-terminates TLS, provisions certificates over ACME DNS-01, and routes traffic to
+terminates TLS, provisions certificates over ACME DNS-01 or DNS-PERSIST-01, and routes traffic to
 CVMs across a WireGuard mesh it manages itself.
 
 ## Ingress mapping
@@ -37,6 +37,7 @@ The gateway needs `CAP_NET_ADMIN` to manage its WireGuard interface, hence
 
 ## Further reading
 
+- [Certificate provisioning](../../docs/certbot-dns-persist-01.md) — DNS-PERSIST-01 and Google Public CA
 - [Cluster deployment](docs/cluster-deployment.md) — running a replicated gateway
 - [Builder image](dstack-app/builder/README.md) — reproducible container image
 - [Deployment guide](../../docs/deployment.md)

dstack/gateway/rpc/proto/gateway_rpc.proto

@@ -214,6 +214,14 @@ message AcmeInfoResponse {
   string account_attestation = 5;
 }
 
+// Optional one-time external account binding for account registration.
+// Omit both fields for CAs that do not require EAB. Never persisted or returned.
+message RotateAcmeCredentialsRequest {
+  string eab_kid = 1;
+  // Base64url-encoded HMAC key, with or without padding.
+  string eab_hmac_key = 2;
+}
+
 // Result of replacing the shared ACME account credentials.
 message RotateAcmeCredentialsResponse {
   // URI of the newly-created ACME account. The private credentials are never returned.
@@ -579,7 +587,8 @@ service Admin {
   // nodes by a best-effort lock (WaveKV has no compare-and-swap), so still
   // avoid rotating from multiple gateways concurrently. This re-pins issuance
   // to the new account; it does not deactivate the old ACME account at the CA.
-  rpc RotateAcmeCredentials(google.protobuf.Empty) returns (RotateAcmeCredentialsResponse) {}
+  // Optional EAB credentials are used only to register the new account.
+  rpc RotateAcmeCredentials(RotateAcmeCredentialsRequest) returns (RotateAcmeCredentialsResponse) {}
 
   // ==================== Tombstone GC Configuration ====================
   // Get the effective tombstone GC pacing on this node