diff --git a/crates/tinycomputer-browser/README.md b/crates/tinycomputer-browser/README.md index 4ee4cbb59..f89545f40 100644 --- a/crates/tinycomputer-browser/README.md +++ b/crates/tinycomputer-browser/README.md @@ -42,6 +42,11 @@ contents remain opaque to the proxy and require TinySecurity's explicit [`configuration.md`](../../docs/crates/tinycomputer/configuration.md#browser) for the host contract and its privacy behavior. +A host may also name upstream HTTP proxies for some destination domains +(`EnforcedAgentBrowser::with_upstream_proxies`, from the module's +`browser.egress.upstream_proxies`). Those destinations are still authorized +first, and the upstream proxy's own dial goes through the same host grant. + The proxy internals and request-framing constraints are described in [`src/egress/README.md`](src/egress/README.md). diff --git a/crates/tinycomputer-browser/src/egress/README.md b/crates/tinycomputer-browser/src/egress/README.md index ecdb599b0..f284e87a9 100644 --- a/crates/tinycomputer-browser/src/egress/README.md +++ b/crates/tinycomputer-browser/src/egress/README.md @@ -17,6 +17,8 @@ grant before each dial, and revalidates active leases. response data while checking active leases. - `serve.rs` sequences authorization, grant consumption, exact-address connection, request forwarding, and response handling for one connection. +- `upstream.rs` holds `UpstreamProxy`, the optional host-configured upstream + HTTP proxy for a list of destination domains, and its `CONNECT` handshake. ## Operational constraints @@ -33,3 +35,34 @@ once per second while uploading, tunneling, or receiving a response, and closes the connection when validation is denied or fails. Do not add a direct-dial fallback or locally resolve a host here: destination policy belongs to the authenticated host callback. + +## Upstream proxies + +A host may route some destination domains (each entry matches itself and its +subdomains) through an upstream HTTP proxy, for sites that refuse the host's +own address (`EgressProxy::bind_with_upstreams`). Routing never weakens the +policy: + +1. The destination is authorized exactly as for a direct dial. Its grant is + consumed for one approved address (IPv4 first), and that address is the + tunnel's target: the upstream proxy resolves nothing, so split DNS or a + rebinding answer at the proxy cannot move the connection elsewhere. +2. The upstream proxy's host and port are authorized through the same host + callback as any other dial (`scheme` `http`), and the proxy socket opens + only at an address that grant names, after consuming it. Nothing is + resolved locally and nothing is dialed outside a grant. +3. The proxy is asked for `CONNECT ` with the configured + `Proxy-Authorization: Basic` credentials. The browser's `Host` header and + TLS SNI travel inside the tunnel unchanged. Plain `http` and WebSocket + requests to a routed host travel the same kind of tunnel (to the URL's + port, 80 when it names none), with the usual + origin-form request written into it, so no request reaches a routed host + other than through a tunnel. +4. Every lease check validates both the destination lease and the proxy + lease; either one failing closes the connection. + +A refused upstream dial answers the browser `403`; an upstream proxy that +answers but will not open the tunnel (a `407` for bad credentials, a `502`, +a timeout) answers `502` and logs only the upstream status. Only `http://` +upstream proxies are supported. The credentials stay inside `UpstreamProxy`: +its `Debug` output, its errors, and the log lines never carry them. diff --git a/crates/tinycomputer-browser/src/egress/egress_tests.rs b/crates/tinycomputer-browser/src/egress/egress_tests.rs index f864482ee..409fc1e50 100644 --- a/crates/tinycomputer-browser/src/egress/egress_tests.rs +++ b/crates/tinycomputer-browser/src/egress/egress_tests.rs @@ -14,6 +14,7 @@ mod lease_tests; mod lifecycle_tests; mod retry_tests; mod simulator; +mod upstream_tests; mod websocket_tests; #[tokio::test] diff --git a/crates/tinycomputer-browser/src/egress/egress_tests/upstream_tests.rs b/crates/tinycomputer-browser/src/egress/egress_tests/upstream_tests.rs new file mode 100644 index 000000000..f6328b1ea --- /dev/null +++ b/crates/tinycomputer-browser/src/egress/egress_tests/upstream_tests.rs @@ -0,0 +1,385 @@ +//! Destinations routed through an upstream proxy, end to end through the +//! egress proxy: policy still runs for the destination, the upstream proxy's +//! own dial is granted by the host, and other hosts are unaffected. + +use super::simulator::*; +use crate::{BrowserDialTarget, DialGrant, EgressAuthorizer, EgressProxy, UpstreamProxy}; +use std::{ + collections::HashMap, + io, + net::SocketAddr, + sync::{Arc, Mutex}, +}; +use tokio::{ + io::{AsyncReadExt, AsyncWriteExt}, + net::{TcpListener, TcpStream}, +}; + +const PROXY_HOST: &str = "gate.proxy.test"; + +/// Grants each known hostname its one address and records every consume and +/// validation, so a test can see which dials the host was asked for. +#[derive(Default)] +struct RoutingAuthorizer { + hosts: HashMap>, + authorized: Mutex>, + consumed: Mutex>, + validated: Mutex>, +} + +impl EgressAuthorizer for RoutingAuthorizer { + fn authorize( + &self, + target: BrowserDialTarget, + ) -> std::pin::Pin> + Send>> { + self.authorized.lock().unwrap().push(format!( + "{}://{}:{}", + target.scheme, target.hostname, target.port + )); + let grant = self.hosts.get(&target.hostname).map(|addresses| DialGrant { + lease: format!("lease-{}", target.hostname), + addresses: addresses.clone(), + }); + Box::pin(async move { + grant.ok_or_else(|| io::Error::new(io::ErrorKind::PermissionDenied, "denied")) + }) + } + + fn consume( + &self, + target: BrowserDialTarget, + grant: DialGrant, + address: SocketAddr, + ) -> std::pin::Pin> + Send>> { + let allowed = grant.addresses.contains(&address); + if allowed { + self.consumed + .lock() + .unwrap() + .push((target.hostname, address)); + } + Box::pin(async move { + if allowed { + Ok(()) + } else { + Err(io::Error::new( + io::ErrorKind::PermissionDenied, + "wrong address", + )) + } + }) + } + + fn validate( + &self, + lease: String, + ) -> std::pin::Pin> + Send>> { + self.validated.lock().unwrap().push(lease); + Box::pin(async { Ok(true) }) + } +} + +fn upstream_proxy(port: u16) -> UpstreamProxy { + UpstreamProxy::new( + &format!("http://resi-user:resi%20pass@{PROXY_HOST}:{port}"), + &["reddit.com".to_owned()], + ) + .unwrap() +} + +fn expected_proxy_authorization() -> String { + use base64::Engine as _; + format!( + "Basic {}", + base64::engine::general_purpose::STANDARD.encode("resi-user:resi pass") + ) +} + +async fn read_head(stream: &mut TcpStream) -> String { + let mut head = Vec::new(); + let mut byte = [0u8; 1]; + while !head.ends_with(b"\r\n\r\n") { + stream.read_exact(&mut byte).await.unwrap(); + head.push(byte[0]); + } + String::from_utf8(head).unwrap() +} + +/// A destination listener that must never be dialed. +fn untouched_destination() -> (std::net::TcpListener, SocketAddr) { + let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + listener.set_nonblocking(true).unwrap(); + let address = listener.local_addr().unwrap(); + (listener, address) +} + +async fn bind(authorizer: Arc, upstream: UpstreamProxy) -> EgressProxy { + EgressProxy::bind_with_upstreams( + authorizer, + "routed-session".into(), + FIXTURE_PASSWORD.into(), + Arc::from([upstream]), + ) + .await + .unwrap() +} + +async fn send(proxy: &EgressProxy, request: &str) -> TcpStream { + let mut client = TcpStream::connect(proxy.url().replace("http://", "")) + .await + .unwrap(); + client.write_all(request.as_bytes()).await.unwrap(); + client +} + +fn connect_request(authority: &str) -> String { + format!( + "CONNECT {authority} HTTP/1.1\r\nHost: {authority}\r\nProxy-Authorization: {}\r\n\r\n", + proxy_authorization() + ) +} + +#[tokio::test] +async fn a_routed_connect_tunnels_through_the_upstream_proxy_with_its_credentials() { + let (destination, destination_address) = untouched_destination(); + let upstream = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let upstream_address = upstream.local_addr().unwrap(); + let authorizer = Arc::new(RoutingAuthorizer { + hosts: HashMap::from([ + ("www.reddit.com".to_owned(), vec![destination_address]), + (PROXY_HOST.to_owned(), vec![upstream_address]), + ]), + ..RoutingAuthorizer::default() + }); + let proxy = bind(authorizer.clone(), upstream_proxy(upstream_address.port())).await; + + let mut client = send(&proxy, &connect_request("www.reddit.com:443")).await; + let (mut tunnel, _) = upstream.accept().await.unwrap(); + let head = read_head(&mut tunnel).await; + assert_eq!( + head, + format!( + "CONNECT {destination_address} HTTP/1.1\r\nHost: {destination_address}\r\nProxy-Authorization: {}\r\n\r\n", + expected_proxy_authorization() + ) + ); + tunnel + .write_all(b"HTTP/1.1 200 Connection established\r\n\r\n") + .await + .unwrap(); + let response = read_head(&mut client).await; + assert!( + response.starts_with("HTTP/1.1 200 Connection Established"), + "{response}" + ); + client.write_all(b"ping").await.unwrap(); + let mut payload = [0u8; 4]; + tunnel.read_exact(&mut payload).await.unwrap(); + assert_eq!(&payload, b"ping"); + tunnel.write_all(b"pong").await.unwrap(); + client.read_exact(&mut payload).await.unwrap(); + assert_eq!(&payload, b"pong"); + + assert_eq!( + *authorizer.authorized.lock().unwrap(), + vec![ + "https://www.reddit.com:443".to_owned(), + format!("http://{PROXY_HOST}:{}", upstream_address.port()), + ] + ); + assert_eq!( + *authorizer.consumed.lock().unwrap(), + vec![ + ("www.reddit.com".to_owned(), destination_address), + (PROXY_HOST.to_owned(), upstream_address), + ] + ); + let validated = authorizer.validated.lock().unwrap().clone(); + assert!(validated.contains(&"lease-www.reddit.com".to_owned())); + assert!(validated.contains(&format!("lease-{PROXY_HOST}"))); + assert_eq!( + destination.accept().unwrap_err().kind(), + io::ErrorKind::WouldBlock + ); +} + +#[tokio::test] +async fn a_plain_http_request_to_a_routed_host_also_goes_through_a_tunnel() { + let (_destination, destination_address) = untouched_destination(); + let upstream = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let upstream_address = upstream.local_addr().unwrap(); + let authorizer = Arc::new(RoutingAuthorizer { + hosts: HashMap::from([ + ("reddit.com".to_owned(), vec![destination_address]), + (PROXY_HOST.to_owned(), vec![upstream_address]), + ]), + ..RoutingAuthorizer::default() + }); + let proxy = bind(authorizer, upstream_proxy(upstream_address.port())).await; + + let mut client = send( + &proxy, + &format!( + "GET http://reddit.com/r/x?y=1 HTTP/1.1\r\nHost: reddit.com\r\nProxy-Authorization: {}\r\n\r\n", + proxy_authorization() + ), + ) + .await; + let (mut tunnel, _) = upstream.accept().await.unwrap(); + let head = read_head(&mut tunnel).await; + assert!( + head.starts_with(&format!("CONNECT {destination_address} HTTP/1.1\r\n")), + "{head}" + ); + tunnel + .write_all(b"HTTP/1.1 200 Connection established\r\n\r\n") + .await + .unwrap(); + let forwarded = read_head(&mut tunnel).await; + assert!( + forwarded.starts_with("GET /r/x?y=1 HTTP/1.1\r\n"), + "{forwarded}" + ); + assert!( + !forwarded + .to_ascii_lowercase() + .contains("proxy-authorization") + ); + tunnel + .write_all(b"HTTP/1.1 204 No Content\r\nContent-Length: 0\r\n\r\n") + .await + .unwrap(); + drop(tunnel); + let mut response = Vec::new(); + client.read_to_end(&mut response).await.unwrap(); + assert!(String::from_utf8_lossy(&response).starts_with("HTTP/1.1 204")); +} + +#[tokio::test] +async fn a_host_outside_the_routed_domains_is_dialed_directly() { + let destination = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let destination_address = destination.local_addr().unwrap(); + let (upstream, upstream_address) = untouched_destination(); + let authorizer = Arc::new(RoutingAuthorizer { + hosts: HashMap::from([ + ("example.test".to_owned(), vec![destination_address]), + (PROXY_HOST.to_owned(), vec![upstream_address]), + ]), + ..RoutingAuthorizer::default() + }); + let proxy = bind(authorizer.clone(), upstream_proxy(upstream_address.port())).await; + + let mut client = send( + &proxy, + &connect_request(&format!("example.test:{}", destination_address.port())), + ) + .await; + let (mut direct, _) = destination.accept().await.unwrap(); + let response = read_head(&mut client).await; + assert!( + response.starts_with("HTTP/1.1 200 Connection Established"), + "{response}" + ); + client.write_all(b"ping").await.unwrap(); + let mut payload = [0u8; 4]; + direct.read_exact(&mut payload).await.unwrap(); + assert_eq!(&payload, b"ping"); + assert_eq!(authorizer.authorized.lock().unwrap().len(), 1); + assert_eq!( + upstream.accept().unwrap_err().kind(), + io::ErrorKind::WouldBlock + ); +} + +#[tokio::test] +async fn rejected_proxy_credentials_answer_bad_gateway() { + let (_destination, destination_address) = untouched_destination(); + let upstream = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let upstream_address = upstream.local_addr().unwrap(); + let authorizer = Arc::new(RoutingAuthorizer { + hosts: HashMap::from([ + ("reddit.com".to_owned(), vec![destination_address]), + (PROXY_HOST.to_owned(), vec![upstream_address]), + ]), + ..RoutingAuthorizer::default() + }); + let proxy = bind(authorizer, upstream_proxy(upstream_address.port())).await; + + let mut client = send(&proxy, &connect_request("reddit.com:443")).await; + let (mut tunnel, _) = upstream.accept().await.unwrap(); + read_head(&mut tunnel).await; + tunnel + .write_all( + b"HTTP/1.1 407 Proxy Authentication Required\r\nProxy-Authenticate: Basic\r\n\r\n", + ) + .await + .unwrap(); + let mut response = Vec::new(); + client.read_to_end(&mut response).await.unwrap(); + let response = String::from_utf8(response).unwrap(); + assert!( + response.starts_with("HTTP/1.1 502 Bad Gateway"), + "{response}" + ); + assert!(!response.contains("Proxy-Authenticate")); +} + +#[tokio::test] +async fn a_routed_destination_is_refused_when_the_host_denies_it_or_the_proxy() { + let (upstream, upstream_address) = untouched_destination(); + let (_destination, destination_address) = untouched_destination(); + // The destination is denied: the upstream proxy is never asked for. + let denies_destination = Arc::new(RoutingAuthorizer { + hosts: HashMap::from([(PROXY_HOST.to_owned(), vec![upstream_address])]), + ..RoutingAuthorizer::default() + }); + // The proxy's own dial is denied: nothing is consumed or dialed. + let denies_proxy = Arc::new(RoutingAuthorizer { + hosts: HashMap::from([("reddit.com".to_owned(), vec![destination_address])]), + ..RoutingAuthorizer::default() + }); + for (authorizer, authorizations) in [(denies_destination, 1), (denies_proxy, 2)] { + let proxy = bind(authorizer.clone(), upstream_proxy(upstream_address.port())).await; + let mut client = send(&proxy, &connect_request("reddit.com:443")).await; + let mut response = Vec::new(); + client.read_to_end(&mut response).await.unwrap(); + assert!(String::from_utf8_lossy(&response).contains("403 Forbidden")); + assert_eq!(authorizer.authorized.lock().unwrap().len(), authorizations); + assert!(authorizer.consumed.lock().unwrap().is_empty()); + } + assert_eq!( + upstream.accept().unwrap_err().kind(), + io::ErrorKind::WouldBlock + ); +} + +#[tokio::test] +async fn the_tunnel_is_pinned_to_an_approved_ipv4_address_before_ipv6() { + let (_destination, destination_address) = untouched_destination(); + let ipv6 = SocketAddr::from(( + [0x2001, 0xdb8, 0, 0, 0, 0, 0, 7], + destination_address.port(), + )); + let upstream = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let upstream_address = upstream.local_addr().unwrap(); + let authorizer = Arc::new(RoutingAuthorizer { + hosts: HashMap::from([ + ("reddit.com".to_owned(), vec![ipv6, destination_address]), + (PROXY_HOST.to_owned(), vec![upstream_address]), + ]), + ..RoutingAuthorizer::default() + }); + let proxy = bind(authorizer.clone(), upstream_proxy(upstream_address.port())).await; + + let _client = send(&proxy, &connect_request("reddit.com:443")).await; + let (mut tunnel, _) = upstream.accept().await.unwrap(); + let head = read_head(&mut tunnel).await; + assert!( + head.starts_with(&format!("CONNECT {destination_address} HTTP/1.1\r\n")), + "{head}" + ); + assert_eq!( + authorizer.consumed.lock().unwrap()[0], + ("reddit.com".to_owned(), destination_address) + ); +} diff --git a/crates/tinycomputer-browser/src/egress/forward.rs b/crates/tinycomputer-browser/src/egress/forward.rs index 841810492..45d985bc3 100644 --- a/crates/tinycomputer-browser/src/egress/forward.rs +++ b/crates/tinycomputer-browser/src/egress/forward.rs @@ -243,6 +243,16 @@ pub(super) async fn refuse(client: &mut TcpStream) -> io::Result<()> { .await } +/// Answer a routed request whose upstream proxy would not open a tunnel. +pub(super) async fn bad_gateway(client: &mut TcpStream) -> io::Result<()> { + write_all_bounded( + client, + b"HTTP/1.1 502 Bad Gateway\r\nConnection: close\r\nContent-Length: 0\r\n\r\n", + "browser gateway response timed out", + ) + .await +} + pub(super) async fn write_all_bounded( stream: &mut W, bytes: &[u8], diff --git a/crates/tinycomputer-browser/src/egress/mod.rs b/crates/tinycomputer-browser/src/egress/mod.rs index fc7760fb3..91501bf45 100644 --- a/crates/tinycomputer-browser/src/egress/mod.rs +++ b/crates/tinycomputer-browser/src/egress/mod.rs @@ -14,6 +14,9 @@ use tokio::{ mod forward; mod request; mod serve; +mod upstream; + +pub use upstream::UpstreamProxy; #[cfg(test)] pub(crate) use serve::connect_approved_addresses; @@ -94,6 +97,26 @@ impl EgressProxy { authorizer: Arc, session_id: String, proxy_password: String, + ) -> io::Result { + Self::bind_with_upstreams(authorizer, session_id, proxy_password, Arc::from([])).await + } + + /// Bind like [`EgressProxy::bind`], routing each destination an + /// [`UpstreamProxy`] covers through that proxy. The destination is + /// authorized exactly as for a direct dial; the upstream proxy's own + /// address is then authorized, consumed and dialed through the same + /// `authorizer`, and the connection stays open only while both leases + /// validate. + /// + /// # Errors + /// + /// Returns an I/O error when the listener cannot bind or its address cannot + /// be read. + pub async fn bind_with_upstreams( + authorizer: Arc, + session_id: String, + proxy_password: String, + upstreams: Arc<[UpstreamProxy]>, ) -> io::Result { let listener = TcpListener::bind(("127.0.0.1", 0)).await?; let address = listener.local_addr()?; @@ -106,8 +129,9 @@ impl EgressProxy { let auth = authorizer.clone(); let session = session_id.clone(); let password = proxy_password.clone(); + let routes = upstreams.clone(); connections.spawn(async move { - let _ = Box::pin(serve::serve(stream, auth, session, password)).await; + let _ = Box::pin(serve::serve(stream, auth, session, password, routes)).await; }); } _ = connections.join_next(), if !connections.is_empty() => {} diff --git a/crates/tinycomputer-browser/src/egress/serve.rs b/crates/tinycomputer-browser/src/egress/serve.rs index 21c718cac..c6cc0f8be 100644 --- a/crates/tinycomputer-browser/src/egress/serve.rs +++ b/crates/tinycomputer-browser/src/egress/serve.rs @@ -7,10 +7,11 @@ use tokio::net::TcpStream; use super::{ BrowserDialTarget, DialGrant, EgressAuthorizer, forward::{ - forward_request_body_with_lease, forward_response_with_lease, initialize_upstream, refuse, - tunnel_with_lease, validate_lease, validation_interval, + bad_gateway, forward_request_body_with_lease, forward_response_with_lease, + initialize_upstream, refuse, tunnel_with_lease, validate_lease, validation_interval, }, request::{has_valid_buffered_body, read_proxy_request}, + upstream::{RoutedLease, UpstreamProxy, open_tunnel}, }; pub(super) async fn serve( @@ -18,12 +19,14 @@ pub(super) async fn serve( auth: Arc, session_id: String, proxy_password: String, + upstreams: Arc<[UpstreamProxy]>, ) -> io::Result<()> { Box::pin(serve_with_connector( client, auth, session_id, proxy_password, + &upstreams, TcpStream::connect, )) .await @@ -34,6 +37,7 @@ pub(super) async fn serve_with_connector( auth: Arc, session_id: String, proxy_password: String, + upstreams: &[UpstreamProxy], mut connect: C, ) -> io::Result<()> where @@ -49,10 +53,24 @@ where let Some(first_grant) = authorize(&auth, &request.target).await else { return refuse(&mut client).await; }; - let Some((mut upstream, lease)) = - connect_approved_addresses(&auth, &request.target, first_grant, &mut connect).await - else { - return refuse(&mut client).await; + let route = upstreams + .iter() + .find(|proxy| proxy.routes(&request.target.hostname)); + let (mut upstream, lease, auth) = if let Some(proxy) = route { + match connect_through_upstream(&auth, &request.target, first_grant, proxy, &mut connect) + .await + { + Routed::Open(upstream, lease, routed) => (upstream, lease, routed), + Routed::Refused => return refuse(&mut client).await, + Routed::Gateway => return bad_gateway(&mut client).await, + } + } else { + let Some((upstream, lease)) = + connect_approved_addresses(&auth, &request.target, first_grant, &mut connect).await + else { + return refuse(&mut client).await; + }; + (upstream, lease, auth) }; let validations = validation_interval(); if !validate_lease(&auth, &lease).await? { @@ -145,6 +163,80 @@ where connected } +/// How a routed connection ended up. +enum Routed { + /// A tunnel to the destination, its upstream proxy lease, and the + /// authorizer that validates both leases. + Open(TcpStream, String, Arc), + /// The host refused the upstream proxy dial, or it could not be made. + Refused, + /// The upstream proxy answered but would not open the tunnel. + Gateway, +} + +/// Reach an authorized destination through its upstream proxy. The +/// destination grant was issued by the host's policy check; it is spent here +/// for this connection on the one approved address the tunnel is pinned to, +/// and the only socket opened locally is to an address the host granted for +/// the upstream proxy itself. +async fn connect_through_upstream( + auth: &Arc, + target: &BrowserDialTarget, + destination_grant: DialGrant, + proxy: &UpstreamProxy, + connect: &mut C, +) -> Routed +where + C: FnMut(SocketAddr) -> F, + F: Future>, +{ + let proxy_target = proxy.dial_target(&target.session_id); + let Some(proxy_grant) = authorize(auth, &proxy_target).await else { + log::warn!("browser upstream proxy dial was not authorized by the host"); + return Routed::Refused; + }; + // The tunnel is pinned to one address the host approved for the + // destination, so the upstream proxy never resolves the name itself (it + // could resolve it elsewhere: split DNS, rebinding). IPv4 first, since + // many residential exits have no IPv6. The browser's own `Host` header + // and TLS SNI travel inside the tunnel unchanged. + let Some(destination_address) = destination_grant + .addresses + .iter() + .find(|address| address.is_ipv4()) + .or_else(|| destination_grant.addresses.first()) + .copied() + else { + return Routed::Refused; + }; + let destination_lease = destination_grant.lease.clone(); + if !matches!( + tokio::time::timeout( + Duration::from_secs(5), + auth.consume(target.clone(), destination_grant, destination_address), + ) + .await, + Ok(Ok(())) + ) { + return Routed::Refused; + } + let Some((mut upstream, proxy_lease)) = + connect_approved_addresses(auth, &proxy_target, proxy_grant, connect).await + else { + log::warn!("browser upstream proxy could not be reached"); + return Routed::Refused; + }; + if let Err(error) = open_tunnel(&mut upstream, proxy, destination_address).await { + log::warn!("browser upstream proxy did not open a tunnel: {error}"); + return Routed::Gateway; + } + let routed: Arc = Arc::new(RoutedLease { + inner: auth.clone(), + destination_lease, + }); + Routed::Open(upstream, proxy_lease, routed) +} + async fn authorize( auth: &Arc, target: &BrowserDialTarget, diff --git a/crates/tinycomputer-browser/src/egress/upstream.rs b/crates/tinycomputer-browser/src/egress/upstream.rs new file mode 100644 index 000000000..49871fb36 --- /dev/null +++ b/crates/tinycomputer-browser/src/egress/upstream.rs @@ -0,0 +1,314 @@ +//! Routes chosen destinations through a host-configured upstream HTTP proxy. +//! +//! Some sites refuse a datacenter address outright. A host can name an +//! upstream proxy (a residential or mobile exit, say) for a list of domains; +//! a browser request to one of them still passes the full destination +//! authorization, and then reaches one of the destination's approved +//! addresses through a `CONNECT` tunnel on the upstream proxy instead of a +//! direct dial. The upstream proxy's own address is authorized, granted and consumed through the same +//! host callback as any other dial, so this adds no unchecked socket and no +//! local name resolution. +//! +//! The proxy URL carries credentials. They live only in +//! [`UpstreamProxy::authorization`], are written only into the `CONNECT` +//! request, and never appear in `Debug` output, logs or errors. + +use std::{fmt, future::Future, io, net::SocketAddr, pin::Pin, sync::Arc, time::Duration}; + +use base64::Engine as _; +use tokio::io::{AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt}; + +use super::{BrowserDialTarget, DialGrant, EgressAuthorizer}; + +/// The longest upstream `CONNECT` response head accepted. +const MAX_RESPONSE_HEAD: usize = 8 * 1024; +/// How long the upstream proxy has to answer a `CONNECT`. +const HANDSHAKE_TIMEOUT: Duration = Duration::from_secs(15); + +/// An upstream HTTP proxy and the destination domains routed through it. +#[derive(Clone, PartialEq, Eq)] +pub struct UpstreamProxy { + domains: Vec, + host: String, + port: u16, + authorization: Option, +} + +impl fmt::Debug for UpstreamProxy { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str("UpstreamProxy([REDACTED])") + } +} + +/// Why an upstream proxy setting was refused: a static message that never +/// repeats the URL, since it may carry credentials. +#[derive(Debug, Clone, Copy)] +struct Refusal(&'static str); + +impl UpstreamProxy { + /// Route `domains` (each matching itself and every subdomain) through the + /// proxy at `url`, an `http://[user:password@]host[:port]` URL whose + /// percent-encoded credentials are sent as `Proxy-Authorization: Basic`. + /// + /// # Errors + /// + /// Returns [`crate::Error::InvalidInput`], whose message never quotes the + /// URL, when the URL is not a plain `http` proxy URL or a domain is not a + /// DNS name. + pub fn new(url: &str, domains: &[String]) -> crate::Result { + Self::parse(url, domains).map_err(|Refusal(reason)| crate::Error::invalid_input(reason)) + } + + fn parse(url: &str, domains: &[String]) -> Result { + let domains = normalize_domains(domains)?; + let parsed = + url::Url::parse(url).map_err(|_| Refusal("upstream proxy url is not a valid URL"))?; + if parsed.scheme() != "http" { + return Err(Refusal("upstream proxy url must be an http:// proxy URL")); + } + if !matches!(parsed.path(), "" | "/") + || parsed.query().is_some() + || parsed.fragment().is_some() + { + return Err(Refusal( + "upstream proxy url must name only credentials, a host and a port", + )); + } + let host = match parsed.host() { + Some(url::Host::Domain(domain)) => domain.to_ascii_lowercase(), + Some(url::Host::Ipv4(ip)) => ip.to_string(), + Some(url::Host::Ipv6(ip)) => ip.to_string(), + None => return Err(Refusal("upstream proxy url has no host")), + }; + let port = parsed + .port_or_known_default() + .ok_or(Refusal("upstream proxy url has no port"))?; + let user = percent_decode(parsed.username())?; + let password = percent_decode(parsed.password().unwrap_or_default())?; + if user.contains(&b':') { + return Err(Refusal("upstream proxy user name must not contain ':'")); + } + let authorization = (!user.is_empty() || !password.is_empty()).then(|| { + let mut credentials = user; + credentials.push(b':'); + credentials.extend_from_slice(&password); + format!( + "Basic {}", + base64::engine::general_purpose::STANDARD.encode(credentials) + ) + }); + Ok(Self { + domains, + host, + port, + authorization, + }) + } + + /// Whether a request for `hostname` goes through this proxy: `hostname` + /// equals one of its domains or is a subdomain of one. + #[must_use] + pub fn routes(&self, hostname: &str) -> bool { + let hostname = hostname.trim_end_matches('.').to_ascii_lowercase(); + self.domains.iter().any(|domain| { + hostname == *domain + || hostname + .strip_suffix(domain.as_str()) + .is_some_and(|prefix| prefix.ends_with('.')) + }) + } + + /// The dial the host authorizes for the proxy itself. + pub(super) fn dial_target(&self, session_id: &str) -> BrowserDialTarget { + BrowserDialTarget { + scheme: "http".to_owned(), + hostname: self.host.clone(), + port: self.port, + session_id: session_id.to_owned(), + } + } +} + +fn normalize_domains(domains: &[String]) -> Result, Refusal> { + if domains.is_empty() { + return Err(Refusal("upstream proxy needs at least one domain")); + } + domains + .iter() + .map(|domain| { + let domain = domain.trim().trim_matches('.').to_ascii_lowercase(); + let valid = !domain.is_empty() + && domain.len() <= 253 + && domain.split('.').all(|label| { + !label.is_empty() + && label.len() <= 63 + && label + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || byte == b'-') + }); + if valid { + Ok(domain) + } else { + Err(Refusal( + "upstream proxy domain must be a DNS name such as example.com", + )) + } + }) + .collect() +} + +fn percent_decode(value: &str) -> Result, Refusal> { + let invalid = Refusal("upstream proxy url has malformed percent-encoded credentials"); + let bytes = value.as_bytes(); + let mut decoded = Vec::with_capacity(bytes.len()); + let mut index = 0; + while index < bytes.len() { + if bytes[index] == b'%' { + let hex = bytes.get(index + 1..index + 3).ok_or(invalid)?; + let hex = std::str::from_utf8(hex).map_err(|_| invalid)?; + decoded.push(u8::from_str_radix(hex, 16).map_err(|_| invalid)?); + index += 3; + } else { + decoded.push(bytes[index]); + index += 1; + } + } + Ok(decoded) +} + +/// Ask the upstream proxy on `stream` for a tunnel to `destination`, an +/// address the host approved, so the proxy resolves nothing. On success the +/// stream carries bytes to and from the destination, exactly as a direct +/// socket to it would. +/// +/// # Errors +/// +/// Returns an error naming only the upstream status when the proxy refuses +/// (`PermissionDenied` for 407, `ConnectionRefused` otherwise), or an I/O, +/// timeout or framing error. No error carries the credentials. +pub(super) async fn open_tunnel( + stream: &mut S, + proxy: &UpstreamProxy, + destination: SocketAddr, +) -> io::Result<()> +where + S: AsyncRead + AsyncWrite + Unpin, +{ + // `SocketAddr` displays an IPv6 address in brackets, as an authority. + let authority = destination.to_string(); + let mut request = format!("CONNECT {authority} HTTP/1.1\r\nHost: {authority}\r\n"); + if let Some(authorization) = &proxy.authorization { + request.push_str("Proxy-Authorization: "); + request.push_str(authorization); + request.push_str("\r\n"); + } + request.push_str("\r\n"); + tokio::time::timeout(HANDSHAKE_TIMEOUT, async { + stream.write_all(request.as_bytes()).await?; + drop(request); + let head = read_response_head(stream).await?; + let status = response_status(&head)?; + if (200..300).contains(&status) { + Ok(()) + } else if status == 407 { + Err(io::Error::new( + io::ErrorKind::PermissionDenied, + "upstream proxy refused the credentials (407)", + )) + } else { + Err(io::Error::new( + io::ErrorKind::ConnectionRefused, + format!("upstream proxy refused the tunnel ({status})"), + )) + } + }) + .await + .map_err(|_| io::Error::new(io::ErrorKind::TimedOut, "upstream proxy did not answer"))? +} + +/// Read the response head one byte at a time, so nothing the destination +/// sends after it is consumed here. +async fn read_response_head(stream: &mut S) -> io::Result> { + let mut head = Vec::with_capacity(256); + let mut byte = [0u8; 1]; + while !head.ends_with(b"\r\n\r\n") { + if head.len() >= MAX_RESPONSE_HEAD { + return Err(io::Error::new( + io::ErrorKind::InvalidData, + "upstream proxy response head is too long", + )); + } + if stream.read(&mut byte).await? == 0 { + return Err(io::Error::new( + io::ErrorKind::UnexpectedEof, + "upstream proxy closed before answering", + )); + } + head.push(byte[0]); + } + Ok(head) +} + +fn response_status(head: &[u8]) -> io::Result { + let invalid = || { + io::Error::new( + io::ErrorKind::InvalidData, + "upstream proxy sent no HTTP status", + ) + }; + let line = head + .split(|byte| *byte == b'\r') + .next() + .ok_or_else(invalid)?; + let line = std::str::from_utf8(line).map_err(|_| invalid())?; + let mut parts = line.split_ascii_whitespace(); + if !parts + .next() + .is_some_and(|version| version.starts_with("HTTP/1.")) + { + return Err(invalid()); + } + parts + .next() + .filter(|code| code.len() == 3) + .and_then(|code| code.parse().ok()) + .ok_or_else(invalid) +} + +/// The authorizer a routed connection forwards under: every lease check +/// requires both the destination's lease and the upstream proxy's, so a +/// revoked destination closes the tunnel just as a revoked proxy dial does. +pub(super) struct RoutedLease { + pub(super) inner: Arc, + pub(super) destination_lease: String, +} + +impl EgressAuthorizer for RoutedLease { + fn authorize( + &self, + target: BrowserDialTarget, + ) -> Pin> + Send>> { + self.inner.authorize(target) + } + + fn consume( + &self, + target: BrowserDialTarget, + grant: DialGrant, + address: SocketAddr, + ) -> Pin> + Send>> { + self.inner.consume(target, grant, address) + } + + fn validate(&self, lease: String) -> Pin> + Send>> { + let inner = self.inner.clone(); + let destination = self.destination_lease.clone(); + Box::pin( + async move { Ok(inner.validate(destination).await? && inner.validate(lease).await?) }, + ) + } +} + +#[cfg(test)] +#[path = "upstream_tests.rs"] +mod tests; diff --git a/crates/tinycomputer-browser/src/egress/upstream_tests.rs b/crates/tinycomputer-browser/src/egress/upstream_tests.rs new file mode 100644 index 000000000..f613b00e5 --- /dev/null +++ b/crates/tinycomputer-browser/src/egress/upstream_tests.rs @@ -0,0 +1,203 @@ +//! Upstream proxy settings, matching, redaction and the `CONNECT` handshake. + +#![allow(clippy::expect_used, clippy::unwrap_used)] + +use super::*; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; + +const SECRET_URL: &str = "http://res%40user:p%3Ass-word@gate.proxy.test:7000"; + +fn domains(names: &[&str]) -> Vec { + names.iter().map(|name| (*name).to_owned()).collect() +} + +fn target(address: &str) -> SocketAddr { + address.parse().unwrap() +} + +#[test] +fn a_domain_routes_itself_and_its_subdomains_only() { + let proxy = UpstreamProxy::new(SECRET_URL, &domains(&["Reddit.com", ".redd.it."])).unwrap(); + for routed in [ + "reddit.com", + "www.reddit.com", + "old.reddit.com.", + "i.redd.it", + "WWW.REDDIT.COM", + ] { + assert!(proxy.routes(routed), "{routed}"); + } + for direct in [ + "notreddit.com", + "reddit.com.evil.test", + "example.com", + "it", + "redd.it.test", + ] { + assert!(!proxy.routes(direct), "{direct}"); + } +} + +#[test] +fn credentials_are_decoded_into_basic_authorization_and_never_debugged() { + let proxy = UpstreamProxy::new(SECRET_URL, &domains(&["reddit.com"])).unwrap(); + let expected = format!( + "Basic {}", + base64::engine::general_purpose::STANDARD.encode("res@user:p:ss-word") + ); + assert_eq!(proxy.authorization.as_deref(), Some(expected.as_str())); + let dial = proxy.dial_target("session"); + assert_eq!(dial.scheme, "http"); + assert_eq!(dial.hostname, "gate.proxy.test"); + assert_eq!(dial.port, 7000); + let debug = format!("{proxy:?}"); + assert_eq!(debug, "UpstreamProxy([REDACTED])"); + for secret in ["res", "ss-word", "gate.proxy.test", &expected] { + assert!(!debug.contains(secret)); + } +} + +#[test] +fn a_proxy_without_credentials_sends_no_authorization() { + let proxy = UpstreamProxy::new("http://10.0.0.5", &domains(&["example.com"])).unwrap(); + assert!(proxy.authorization.is_none()); + assert_eq!(proxy.dial_target("s").port, 80); + let v6 = UpstreamProxy::new("http://[2001:db8::1]:3128", &domains(&["example.com"])).unwrap(); + assert_eq!(v6.dial_target("s").hostname, "2001:db8::1"); +} + +#[test] +fn invalid_settings_are_refused_without_quoting_the_url() { + let routed = domains(&["reddit.com"]); + for url in [ + "https://user:hunter2@gate.proxy.test:7000", + "socks5://user:hunter2@gate.proxy.test:7000", + "http://user:hunter2@gate.proxy.test:7000/path", + "http://user:hunter2@gate.proxy.test:7000/?q=1", + "http://user:hunter2%zz@gate.proxy.test:7000", + "http://us%3Aer:hunter2@gate.proxy.test:7000", + "not a url hunter2", + ] { + let error = UpstreamProxy::new(url, &routed).unwrap_err(); + assert!(matches!(error, crate::Error::InvalidInput { .. })); + assert!(!error.to_string().contains("hunter2"), "{url}: {error}"); + assert!(!format!("{error:?}").contains("hunter2")); + } + for bad in [ + vec![], + domains(&["*.reddit.com"]), + domains(&[""]), + domains(&["a..b"]), + domains(&["reddit.com/x"]), + ] { + assert!(UpstreamProxy::new(SECRET_URL, &bad).is_err(), "{bad:?}"); + } +} + +async fn handshake(reply: &'static [u8], target: SocketAddr) -> (io::Result<()>, String) { + let proxy = UpstreamProxy::new(SECRET_URL, &domains(&["reddit.com"])).unwrap(); + let (mut ours, mut theirs) = tokio::io::duplex(4096); + let server = tokio::spawn(async move { + let mut seen = Vec::new(); + let mut byte = [0u8; 1]; + while !seen.ends_with(b"\r\n\r\n") { + theirs.read_exact(&mut byte).await.unwrap(); + seen.push(byte[0]); + } + theirs.write_all(reply).await.unwrap(); + // The destination's first bytes must stay unread by the handshake. + theirs.write_all(b"after").await.unwrap(); + (String::from_utf8(seen).unwrap(), theirs) + }); + let result = open_tunnel(&mut ours, &proxy, target).await; + let (request, _theirs) = server.await.unwrap(); + if result.is_ok() { + let mut after = [0u8; 5]; + ours.read_exact(&mut after).await.unwrap(); + assert_eq!(&after, b"after"); + } + (result, request) +} + +#[tokio::test] +async fn the_tunnel_asks_for_the_approved_address_with_the_proxy_credentials() { + let (result, request) = handshake( + b"HTTP/1.1 200 Connection established\r\n\r\n", + target("151.101.1.140:443"), + ) + .await; + result.unwrap(); + let expected = base64::engine::general_purpose::STANDARD.encode("res@user:p:ss-word"); + assert_eq!( + request, + format!( + "CONNECT 151.101.1.140:443 HTTP/1.1\r\nHost: 151.101.1.140:443\r\nProxy-Authorization: Basic {expected}\r\n\r\n" + ) + ); + let (result, request) = handshake( + b"HTTP/1.0 200 OK\r\nVia: x\r\n\r\n", + target("[2001:db8::7]:80"), + ) + .await; + result.unwrap(); + assert!(request.starts_with("CONNECT [2001:db8::7]:80 HTTP/1.1\r\n")); +} + +#[tokio::test] +async fn a_refused_tunnel_names_only_the_status() { + let (result, _) = handshake( + b"HTTP/1.1 407 Proxy Authentication Required\r\nProxy-Authenticate: Basic\r\n\r\n", + target("151.101.1.140:443"), + ) + .await; + let error = result.unwrap_err(); + assert_eq!(error.kind(), io::ErrorKind::PermissionDenied); + assert!(error.to_string().contains("407")); + assert!(!error.to_string().contains("ss-word")); + + let (result, _) = handshake( + b"HTTP/1.1 502 Bad Gateway\r\n\r\n", + target("151.101.1.140:443"), + ) + .await; + let error = result.unwrap_err(); + assert_eq!(error.kind(), io::ErrorKind::ConnectionRefused); + assert!(error.to_string().contains("502")); + + let (result, _) = handshake(b"SSH-2.0-nope\r\n\r\n", target("151.101.1.140:443")).await; + assert_eq!(result.unwrap_err().kind(), io::ErrorKind::InvalidData); +} + +#[tokio::test] +async fn a_proxy_that_closes_or_floods_the_head_is_refused() { + let proxy = UpstreamProxy::new(SECRET_URL, &domains(&["reddit.com"])).unwrap(); + let (mut ours, theirs) = tokio::io::duplex(64); + drop(theirs); + assert!( + open_tunnel(&mut ours, &proxy, target("151.101.1.140:443")) + .await + .is_err() + ); + + let (mut ours, mut theirs) = tokio::io::duplex(1 << 16); + tokio::spawn(async move { + let mut sink = [0u8; 512]; + let _ = theirs.read(&mut sink).await; + let _ = theirs.write_all(&vec![b'a'; MAX_RESPONSE_HEAD + 16]).await; + tokio::time::sleep(Duration::from_secs(1)).await; + }); + let error = open_tunnel(&mut ours, &proxy, target("151.101.1.140:443")) + .await + .unwrap_err(); + assert_eq!(error.kind(), io::ErrorKind::InvalidData); +} + +#[tokio::test(start_paused = true)] +async fn a_silent_proxy_times_out() { + let proxy = UpstreamProxy::new(SECRET_URL, &domains(&["reddit.com"])).unwrap(); + let (mut ours, _theirs) = tokio::io::duplex(4096); + let error = open_tunnel(&mut ours, &proxy, target("151.101.1.140:443")) + .await + .unwrap_err(); + assert_eq!(error.kind(), io::ErrorKind::TimedOut); +} diff --git a/crates/tinycomputer-browser/src/lib.rs b/crates/tinycomputer-browser/src/lib.rs index 4786da69a..496f4fa95 100644 --- a/crates/tinycomputer-browser/src/lib.rs +++ b/crates/tinycomputer-browser/src/lib.rs @@ -57,7 +57,7 @@ pub use action_gate::{ ActionContext, ActionGate, ActionRequest, ActionTarget, ActionVerdict, AllowAllActions, ObservationKind, }; -pub use egress::{BrowserDialTarget, DialGrant, EgressAuthorizer, EgressProxy}; +pub use egress::{BrowserDialTarget, DialGrant, EgressAuthorizer, EgressProxy, UpstreamProxy}; pub use engine::{Engine, Launcher, Reply}; pub use error::{Error, Result}; #[cfg(feature = "agent-browser")] diff --git a/crates/tinycomputer-browser/src/linked/linked_tests/launch_tests.rs b/crates/tinycomputer-browser/src/linked/linked_tests/launch_tests.rs index 07cae4a32..1604d3265 100644 --- a/crates/tinycomputer-browser/src/linked/linked_tests/launch_tests.rs +++ b/crates/tinycomputer-browser/src/linked/linked_tests/launch_tests.rs @@ -141,6 +141,7 @@ fn a_request_cannot_name_an_executable_the_host_did_not_configure() { &HostBrowser { executable: host.map(str::to_owned), profile: None, + upstreams: Vec::new(), }, ) .is_err(), @@ -164,6 +165,7 @@ fn host_profile() -> HostBrowser { HostBrowser { executable: Some("/opt/host/chrome".into()), profile: Some("/srv/state/browser/profile".into()), + upstreams: Vec::new(), } } diff --git a/crates/tinycomputer-browser/src/linked/mod.rs b/crates/tinycomputer-browser/src/linked/mod.rs index 85becf837..b0d282a6a 100644 --- a/crates/tinycomputer-browser/src/linked/mod.rs +++ b/crates/tinycomputer-browser/src/linked/mod.rs @@ -11,7 +11,7 @@ //! by the session lock. use crate::{ - egress::{EgressAuthorizer, EgressProxy}, + egress::{EgressAuthorizer, EgressProxy, UpstreamProxy}, engine::{Engine, Launcher, Reply}, }; use agent_browser::{DaemonState, StateOptions, execute_command}; @@ -39,6 +39,9 @@ pub struct EnforcedAgentBrowser { pub(crate) struct HostBrowser { pub(crate) executable: Option, pub(crate) profile: Option, + /// Destination domains the egress proxy reaches through an upstream + /// proxy, after the same destination authorization as any dial. + pub(crate) upstreams: Vec, } impl HostBrowser { @@ -47,6 +50,7 @@ impl HostBrowser { Self { executable: Some(executable.to_owned()), profile: None, + upstreams: Vec::new(), } } } @@ -85,6 +89,16 @@ impl EnforcedAgentBrowser { self.host.profile = profile; self } + + /// Route the destinations each [`UpstreamProxy`] covers through it. It + /// is host control, like the proxy itself: a browser call cannot name + /// one, and every routed dial is still authorized by the host callback + /// (see the egress module's README). + #[must_use] + pub fn with_upstream_proxies(mut self, upstreams: Vec) -> Self { + self.host.upstreams = upstreams; + self + } } impl Launcher for AgentBrowser { @@ -153,10 +167,11 @@ impl Engine for Linked { return failure_reply(&command, "network policy has no host authorizer"); }; if self.proxy.is_none() { - match EgressProxy::bind( + match EgressProxy::bind_with_upstreams( authorizer, self.proxy_session_id.clone(), self.proxy_password.clone(), + self.host.upstreams.clone().into(), ) .await { diff --git a/crates/tinycomputer/src/tinybus_module/browser_egress/browser_egress_tests.rs b/crates/tinycomputer/src/tinybus_module/browser_egress/browser_egress_tests.rs index 5e385e679..9e389fa91 100644 --- a/crates/tinycomputer/src/tinybus_module/browser_egress/browser_egress_tests.rs +++ b/crates/tinycomputer/src/tinybus_module/browser_egress/browser_egress_tests.rs @@ -184,3 +184,5 @@ async fn bus_adapter_refuses_host_denials_and_malformed_grants() -> tinybus::Res } Ok(()) } + +mod upstream_tests; diff --git a/crates/tinycomputer/src/tinybus_module/browser_egress/browser_egress_tests/upstream_tests.rs b/crates/tinycomputer/src/tinybus_module/browser_egress/browser_egress_tests/upstream_tests.rs new file mode 100644 index 000000000..d00619158 --- /dev/null +++ b/crates/tinycomputer/src/tinybus_module/browser_egress/browser_egress_tests/upstream_tests.rs @@ -0,0 +1,225 @@ +//! Upstream proxy routing through the module's real host boundary: the +//! routes come from the module config (`browser.egress.upstream_proxies`), +//! and every authorization, consume and lease check crosses `TinyBus` to the +//! host's `BrowserEgressHost` callback through `BusAuthorizer`. + +use super::*; +use crate::tinybus_module::config::BrowserDefaults; +use std::collections::HashMap; +use tinycomputer_browser::EgressProxy; +use tokio::{ + io::{AsyncReadExt, AsyncWriteExt}, + net::{TcpListener, TcpStream}, +}; + +const PROXY_HOST: &str = "gate.proxy.test"; +const LOCAL_PASSWORD: &str = "local-proxy-password"; +/// The approved destination address. It is never dialed: it only names the +/// tunnel's target at the upstream proxy. +const REDDIT: &str = "151.101.1.140:443"; + +/// A host that grants each known hostname its addresses and records every +/// callback, as `BrowserEgressHost` sees them. +#[derive(Clone)] +struct RoutingHost { + hosts: HashMap>, + seen: Arc>>, +} + +#[tinybus::interface(name = "ai.tinyhumans.tinycomputer.BrowserEgressHost")] +impl RoutingHost { + async fn authorize(&self, request: BrowserEgressTarget) -> tinybus::Result { + tokio::task::yield_now().await; + self.seen.lock().unwrap().push(format!( + "authorize:{}://{}:{}", + request.scheme, request.hostname, request.port + )); + let addresses = self + .hosts + .get(&request.hostname) + .cloned() + .ok_or_else(|| tinybus::Error::failed("refused"))?; + Ok(BrowserEgressGrant { + generation: 7, + authorization: format!("lease:{}", request.hostname), + addresses, + }) + } + + async fn consume( + &self, + request: BrowserEgressConsumeRequest, + ) -> tinybus::Result { + tokio::task::yield_now().await; + self.seen.lock().unwrap().push(format!( + "consume:{}:{}", + request.target.hostname, request.address + )); + Ok(BrowserEgressDecision { allowed: true }) + } + + async fn validate( + &self, + request: BrowserEgressValidateRequest, + ) -> tinybus::Result { + tokio::task::yield_now().await; + self.seen + .lock() + .unwrap() + .push(format!("validate:{}", request.authorization)); + Ok(BrowserEgressDecision { allowed: true }) + } +} + +async fn routed_proxy( + hosts: HashMap>, + upstream_port: u16, +) -> (EgressProxy, Arc>>, Connection) { + let seen = Arc::new(Mutex::new(Vec::new())); + let bus = MemoryBus::new(); + Broker::new().spawn(bus.clone()); + let host = Connection::connect(bus.connect().await.unwrap()) + .await + .unwrap(); + host.serve_at( + contract::OBJECT_PATH.try_into().unwrap(), + RoutingHost { + hosts, + seen: seen.clone(), + }, + ) + .await + .unwrap(); + host.request_name(contract::INTERFACE).await.unwrap(); + let module = Connection::connect(bus.connect().await.unwrap()) + .await + .unwrap(); + let egress = BrowserDefaults::egress_from_raw(&serde_json::json!({"browser": {"egress": { + "policy_id": "policy-private", + "generation": 7, + "upstream_proxies": [{ + "url": format!("http://resi:pass@{PROXY_HOST}:{upstream_port}"), + "domains": ["reddit.com"], + }], + }}})) + .unwrap() + .unwrap(); + let proxy = EgressProxy::bind_with_upstreams( + Arc::new(BusAuthorizer::new( + module, + egress.policy_id.clone(), + egress.generation, + )), + "session-private".into(), + LOCAL_PASSWORD.into(), + egress.upstream_proxies.clone().into(), + ) + .await + .unwrap(); + (proxy, seen, host) +} + +async fn connect_through(proxy: &EgressProxy, authority: &str) -> TcpStream { + // Basic base64 of `tinycomputer:` + LOCAL_PASSWORD. + let local = "dGlueWNvbXB1dGVyOmxvY2FsLXByb3h5LXBhc3N3b3Jk"; + let mut client = TcpStream::connect(proxy.url().replace("http://", "")) + .await + .unwrap(); + client + .write_all( + format!( + "CONNECT {authority} HTTP/1.1\r\nHost: {authority}\r\nProxy-Authorization: Basic {local}\r\n\r\n" + ) + .as_bytes(), + ) + .await + .unwrap(); + client +} + +async fn read_head(stream: &mut TcpStream) -> String { + let mut head = Vec::new(); + let mut byte = [0u8; 1]; + while !head.ends_with(b"\r\n\r\n") { + stream.read_exact(&mut byte).await.unwrap(); + head.push(byte[0]); + } + String::from_utf8(head).unwrap() +} + +#[tokio::test] +async fn a_configured_route_is_authorized_by_the_host_for_destination_and_proxy() { + let upstream = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let upstream_address = upstream.local_addr().unwrap(); + let (proxy, seen, _host) = routed_proxy( + HashMap::from([ + ("www.reddit.com".to_owned(), vec![REDDIT.to_owned()]), + (PROXY_HOST.to_owned(), vec![upstream_address.to_string()]), + ]), + upstream_address.port(), + ) + .await; + + let mut client = connect_through(&proxy, "www.reddit.com:443").await; + let (mut tunnel, _) = upstream.accept().await.unwrap(); + assert_eq!( + read_head(&mut tunnel).await, + format!( + "CONNECT {REDDIT} HTTP/1.1\r\nHost: {REDDIT}\r\nProxy-Authorization: Basic cmVzaTpwYXNz\r\n\r\n" + ) + ); + tunnel + .write_all(b"HTTP/1.1 200 Connection established\r\n\r\n") + .await + .unwrap(); + let response = read_head(&mut client).await; + assert!( + response.starts_with("HTTP/1.1 200 Connection Established"), + "{response}" + ); + client.write_all(b"ping").await.unwrap(); + let mut payload = [0u8; 4]; + tunnel.read_exact(&mut payload).await.unwrap(); + assert_eq!(&payload, b"ping"); + + let calls = seen.lock().unwrap().clone(); + assert_eq!( + calls[..4], + [ + "authorize:https://www.reddit.com:443".to_owned(), + format!("authorize:http://{PROXY_HOST}:{}", upstream_address.port()), + format!("consume:www.reddit.com:{REDDIT}"), + format!("consume:{PROXY_HOST}:{upstream_address}"), + ] + ); + assert!(calls.contains(&"validate:lease:www.reddit.com".to_owned())); + assert!(calls.contains(&format!("validate:lease:{PROXY_HOST}"))); +} + +#[tokio::test] +async fn a_host_that_refuses_the_proxy_dial_leaves_the_route_closed() { + let upstream = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + upstream.set_nonblocking(true).unwrap(); + let upstream_address = upstream.local_addr().unwrap(); + let (proxy, seen, _host) = routed_proxy( + HashMap::from([("reddit.com".to_owned(), vec![REDDIT.to_owned()])]), + upstream_address.port(), + ) + .await; + + let mut client = connect_through(&proxy, "reddit.com:443").await; + let mut response = Vec::new(); + client.read_to_end(&mut response).await.unwrap(); + assert!(String::from_utf8_lossy(&response).contains("403 Forbidden")); + assert_eq!( + *seen.lock().unwrap(), + vec![ + "authorize:https://reddit.com:443".to_owned(), + format!("authorize:http://{PROXY_HOST}:{}", upstream_address.port()), + ] + ); + assert_eq!( + upstream.accept().unwrap_err().kind(), + std::io::ErrorKind::WouldBlock + ); +} diff --git a/crates/tinycomputer/src/tinybus_module/config.rs b/crates/tinycomputer/src/tinybus_module/config.rs index bd91c1cce..35e4ebe6d 100644 --- a/crates/tinycomputer/src/tinybus_module/config.rs +++ b/crates/tinycomputer/src/tinybus_module/config.rs @@ -6,7 +6,9 @@ //! a desktop `user_agent` and launch `args` once here rather than on every //! request. -use tinycomputer_browser::{CursorPace, Perception, ScreenCursor, SessionOptions, Settle}; +use tinycomputer_browser::{ + CursorPace, Perception, ScreenCursor, SessionOptions, Settle, UpstreamProxy, +}; use tinycomputer_bus::config::{BrowserConfig, BrowserPerception, BrowserSettle}; use crate::Result; @@ -39,8 +41,15 @@ pub(crate) struct BrowserDefaults { pub(crate) struct BrowserEgressConfig { pub(crate) policy_id: String, pub(crate) generation: u64, + /// Destination domains the browser reaches through an upstream HTTP + /// proxy, still under the policy above. Host-private: the URLs carry + /// credentials. + pub(crate) upstream_proxies: Vec, } +/// The most upstream proxies one egress scope may name. +const MAX_UPSTREAM_PROXIES: usize = 16; + impl std::fmt::Debug for BrowserEgressConfig { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { f.write_str("BrowserEgressConfig([REDACTED])") @@ -114,10 +123,19 @@ impl BrowserDefaults { .and_then(serde_json::Value::as_u64) .filter(|generation| *generation > 0) .ok_or_else(invalid)?; + let upstream_proxies = match object.get("upstream_proxies") { + None => Vec::new(), + Some(value) => upstream_proxies(value)?, + }; + let expected_keys = if object.contains_key("upstream_proxies") { + 3 + } else { + 2 + }; // The host's own `executable` stays allowed: the enforced launcher // runs it with the proxy and resolver flags. Extra `args` could undo // them, so they are refused. - if object.len() != 2 + if object.len() != expected_keys || browser .as_object() .is_some_and(|browser| browser.contains_key("args")) @@ -127,6 +145,7 @@ impl BrowserDefaults { Ok(Some(BrowserEgressConfig { policy_id: policy_id.to_owned(), generation, + upstream_proxies, })) } @@ -149,6 +168,47 @@ impl BrowserDefaults { } } +/// Reads `browser.egress.upstream_proxies`: a list of +/// `{"url": "http://user:pass@host:port", "domains": ["example.com"]}`. A +/// refusal names the field, never the URL, which carries credentials. +fn upstream_proxies(value: &serde_json::Value) -> Result> { + let invalid = || crate::Error::ConfigFieldType { + field: "browser.egress.upstream_proxies", + expected: "a list of objects with an http:// proxy `url` and its `domains`", + }; + let entries = value.as_array().ok_or_else(invalid)?; + if entries.len() > MAX_UPSTREAM_PROXIES { + return Err(invalid()); + } + entries + .iter() + .map(|entry| { + let entry = entry + .as_object() + .filter(|entry| entry.len() == 2) + .ok_or_else(invalid)?; + let url = entry + .get("url") + .and_then(serde_json::Value::as_str) + .ok_or_else(invalid)?; + let domains = entry + .get("domains") + .and_then(serde_json::Value::as_array) + .ok_or_else(invalid)? + .iter() + .map(|domain| domain.as_str().map(str::to_owned).ok_or_else(invalid)) + .collect::>>()?; + // The browser crate's message never quotes the URL either; this + // one names the field and the accepted shape. + UpstreamProxy::new(url, &domains).map_err(|_| crate::Error::ConfigFieldType { + field: "browser.egress.upstream_proxies", + expected: "an http:// proxy `url` naming only credentials, a host and a port, \ + and `domains` that are DNS names such as example.com", + }) + }) + .collect() +} + /// Whether the host explicitly selected the trace/report callback. pub(crate) fn trace_sink_destination( config: &serde_json::Value, diff --git a/crates/tinycomputer/src/tinybus_module/dispatch/service.rs b/crates/tinycomputer/src/tinybus_module/dispatch/service.rs index b243c10dc..17764b3dc 100644 --- a/crates/tinycomputer/src/tinybus_module/dispatch/service.rs +++ b/crates/tinycomputer/src/tinybus_module/dispatch/service.rs @@ -133,7 +133,8 @@ impl DesktopService { egress.generation, ))) .with_executable(host.executable) - .with_profile(host.profile), + .with_profile(host.profile) + .with_upstream_proxies(egress.upstream_proxies.clone()), ))) } else { Arc::new(Browser::new(Arc::new(AgentBrowser))) diff --git a/crates/tinycomputer/src/tinybus_module/tinybus_module_tests/config_tests.rs b/crates/tinycomputer/src/tinybus_module/tinybus_module_tests/config_tests.rs index 620808a8e..29c1ccab5 100644 --- a/crates/tinycomputer/src/tinybus_module/tinybus_module_tests/config_tests.rs +++ b/crates/tinycomputer/src/tinybus_module/tinybus_module_tests/config_tests.rs @@ -49,6 +49,56 @@ fn browser_egress_is_private_strict_and_requires_the_host_connection() { } } +#[test] +fn browser_egress_reads_upstream_proxies_without_echoing_their_credentials() { + use crate::tinybus_module::config::BrowserDefaults; + let scope = |upstream: serde_json::Value| { + BrowserDefaults::egress_from_raw(&json!({"browser":{"egress":{ + "policy_id":"p","generation":7,"upstream_proxies":upstream + }}})) + }; + let parsed = scope(json!([{ + "url": "http://resi:hunter2@gate.proxy.test:7000", + "domains": ["reddit.com", "redd.it"] + }])) + .expect("valid upstream proxy") + .expect("scope is present"); + assert_eq!(parsed.upstream_proxies.len(), 1); + assert!(parsed.upstream_proxies[0].routes("www.reddit.com")); + assert!(!parsed.upstream_proxies[0].routes("example.com")); + assert!(!format!("{parsed:?}").contains("hunter2")); + assert_eq!(scope(json!([])).unwrap().unwrap().upstream_proxies.len(), 0); + // Without the key nothing is routed. + let plain = BrowserDefaults::egress_from_raw( + &json!({"browser":{"egress":{"policy_id":"p","generation":7}}}), + ) + .unwrap() + .unwrap(); + assert_eq!(plain.upstream_proxies.len(), 0); + + for upstream in [ + json!({"url": "http://resi:hunter2@gate.proxy.test:7000", "domains": ["reddit.com"]}), + json!([{"url": "https://resi:hunter2@gate.proxy.test:7000", "domains": ["reddit.com"]}]), + json!([{"url": "http://resi:hunter2@gate.proxy.test:7000", "domains": []}]), + json!([{"url": "http://resi:hunter2@gate.proxy.test:7000"}]), + json!([{"url": "http://resi:hunter2@gate.proxy.test:7000", "domains": ["reddit.com"], "extra": 1}]), + json!([{"url": 7, "domains": ["reddit.com"]}]), + json!([{"url": "http://resi:hunter2@gate.proxy.test:7000", "domains": [7]}]), + json!( + (0..17) + .map(|_| json!({"url": "http://h:1", "domains": ["a.test"]})) + .collect::>() + ), + ] { + let error = scope(upstream.clone()).expect_err("invalid upstream proxies are refused"); + assert!( + !error.to_string().contains("hunter2"), + "{upstream}: {error}" + ); + assert!(error.to_string().contains("upstream_proxies"), "{error}"); + } +} + #[test] fn the_desktop_is_available_only_with_accessibility() { use crate::tinybus_module::dispatch::desktop_availability; diff --git a/docs/crates/tinycomputer/configuration.md b/docs/crates/tinycomputer/configuration.md index 9cdf66a96..b54b8105b 100644 --- a/docs/crates/tinycomputer/configuration.md +++ b/docs/crates/tinycomputer/configuration.md @@ -285,7 +285,8 @@ optional: | `perception` | how a task reads a page: `sight` (the default) reads the rendered page as a person sees it, `tree` the accessibility tree alone ([`browser-sight.md`](../../technical/specs/browser-sight.md)) | | `settle` | how a task lets a page settle after an action before reading it again: `prompt` (the default) counts the network's 500 ms of quiet from the start, counting only requests that can change the page and for at most 1 s, and then waits only while the page is still changing (at most 400 ms), so an idle page is read again after about 0.6 s; `steady` waits for the network to go idle, at least about 1.1 s, then 400 ms more | | `prelaunch` | `true` (the default) opens a browser-only task's browser while `StartTask` plans it (a `task` with no `flow`), so the first step does not wait for the launch, and loads in it the one web address the task's text names, if it names one, so a first step that browses there does not wait for the page either; `false` opens it at the first step | -| `egress` | Host-private TinySecurity scope. When present, `policy_id` and `generation` are required; browser `args`, attached DevTools endpoints, caller-supplied launch flags, and any executable or profile other than this section's `executable` and `profile` are refused. The scope is never a browser or task argument. | +| `egress` | Host-private TinySecurity scope. When present, `policy_id` and `generation` are required, and `upstream_proxies` is optional (below); browser `args`, attached DevTools endpoints, caller-supplied launch flags, and any executable or profile other than this section's `executable` and `profile` are refused. The scope is never a browser or task argument. | +| `egress.upstream_proxies` | Host-private list of `{"url": "http://user:pass@host:port", "domains": ["example.com"]}`, at most 16. A destination equal to a listed domain or under it is still authorized as usual, then reached through a `CONNECT` tunnel on that upstream HTTP proxy to one of the addresses the host approved for it (the proxy resolves nothing), and the proxy's own host and port go through the same host authorization and one-use grant. Plain `http` to such a host uses the tunnel too. Only `http://` proxies; credentials are percent-decoded into `Proxy-Authorization: Basic` and never logged. For sites that refuse the host's own address | The launch and attach options above apply only when `egress` is absent, except `executable` and `profile`. With `egress` set, TinyComputer owns the launch arguments and diff --git a/docs/technical/architecture.md b/docs/technical/architecture.md index f9e221991..4ec53e035 100644 --- a/docs/technical/architecture.md +++ b/docs/technical/architecture.md @@ -228,6 +228,7 @@ configuration validates. | `planner` | object | `api_key`, `provider` (`open_router` or `tiny_humans`), approved `endpoint_url`, `sdk_name`, and optional `model` for the planner, optional `rescue_route` giving the rescuer its own route and key, optional `rescue_model` for rescuing failed steps (`docs/technical/specs/task-rescue.md`), and optional `output_model` for shaping a finished task's answer (`docs/technical/specs/task-output.md`); absent means no planner, no rescues, and no output shapes | | `browser.executable` | string | the Chrome or Chromium binary to launch, when discovery would not find one | | `browser.profile` | string | the profile folder kept between sessions; under egress the only profile a launch may name, and it launches with extensions off | +| `browser.egress.upstream_proxies` | array | host-private `{url, domains}` entries: destinations under `domains` are still authorized, then tunnelled through that `http://` upstream proxy, whose own dial the host also authorizes ([`configuration.md`](../crates/tinycomputer/configuration.md#browser)) | | `cursor` | string or object | the agent's on-screen cursor for desktop and browser: a pace (`off`, `brisk`, `natural` (default), `calm`) or `{pace, overlay}` with the overlay helper's path ([spec](specs/virtual-cursor.md)) | Without `jev`, `ResolveIntent`, `RunGoal`, `RunFlow`, and task runs answer