Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -21,4 +21,6 @@ GEMINI.md
ds-bundle/
.design-sync/.cache/
.design-sync/learnings/
.design-sync/node_modules
.design-sync/node_modules
# Chrome generates this when it loads the unpacked extension
extension/_metadata/
30 changes: 22 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,23 @@

---

> **Working against the new Google Flow API.** Flow moved to `flow.google.com`
> in September 2026 and stopped minting the `Bearer ya29.…` that the old
> `aisandbox-pa.googleapis.com` REST API needed. The `batchexecute` transport
> that replaces it is in place and verified end to end against the live API —
> image generation, 2K image export, and image-to-video all run green. Upgrading
> from an older Flow Kit: reload the extension (v0.3.2+) and pin
> `FLOW_PROJECT_ID`; Flow Kit can no longer create the project for you.
>
> Four capabilities are still unported because their payloads were never
> captured off the new UI — **video upscale**, **reference-to-video (r2v)**,
> **start+end-frame chaining**, and **Omni Flash frame/reference**. They fail
> loudly with `UNSUPPORTED_ON_BATCH_API` instead of quietly producing the wrong
> thing. `FLOW_ALLOW_DEGRADED=1` drops chaining and r2v to plain i2v; video
> upscale has no fallback. To restore one properly see [`docs/CAPTURE.md`](docs/CAPTURE.md).

---

### ☕ Sponsor this project

<table align="center">
Expand Down Expand Up @@ -188,9 +205,9 @@ One signed-in Flow tab has to stay open; nothing here works headless.
> **September 2026 — Flow moved.** It now lives at `flow.google.com` and the old
> `aisandbox-pa.googleapis.com` REST API has no caller: the `Bearer ya29.…` it
> needed stopped being minted. If you are upgrading from an older Flow Kit,
> reload the extension (v0.3.0+) and pin `FLOW_PROJECT_ID` — see
> [Configuration](#configuration). The legacy path is still there behind
> `USE_BATCH_RPC=0`, but it is a post-mortem tool, not a fallback.
> reload the extension (v0.3.2+) and pin `FLOW_PROJECT_ID` — see
> [Configuration](#configuration). The REST path has been removed; `git log`
> has it if a payload is ever needed for reference.

## Quick Start

Expand Down Expand Up @@ -240,7 +257,6 @@ You can also pass `flow_project_id` per project on `POST /api/projects`.
| Env var | Default | What it does |
|---------|---------|--------------|
| `FLOW_PROJECT_ID` | — | The Flow project every RPC is scoped to. Required. |
| `USE_BATCH_RPC` | `1` | `0` falls back to the pre-migration REST path (dead auth). |
| `FLOW_ALLOW_DEGRADED` | `0` | `1` lets scene chaining and r2v fall back to plain i2v instead of failing. |
| `DEFAULT_PAYGATE_TIER` | `PAYGATE_TIER_TWO` | Carried for the DB and dashboard; no longer selects a model. |

Expand Down Expand Up @@ -690,7 +706,6 @@ agent/
│ # flow, models, providers, reviews, materials, music, tts)
├── services/
│ ├── flow_client.py # WS bridge to extension
│ ├── headers.py # Randomized browser headers
│ ├── tts.py # OmniVoice TTS (subprocess-based)
│ ├── scene_chain.py # Continuation scene logic
│ ├── video_reviewer.py # AI vision review — contact sheet + claude/agy/codex CLI dispatch
Expand Down Expand Up @@ -805,13 +820,13 @@ These arrive in the response body as `data.error.details[].reason`. The worker a
| Status | Source | Meaning | Handling |
|--------|--------|---------|----------|
| **400** | Flow API | Invalid payload, UNSAFE_GENERATION, entity not found (sometimes) | Route by `details.reason` — some are auto-recoverable, others terminal |
| **401** | Flow API (legacy path only) | Bearer expired — on a post-migration profile it was never minted | Switch to the batch path (`USE_BATCH_RPC=1`) |
| **401** | Flow API | Should not occur — batchexecute authenticates in the page, not with a bearer | Check the Flow tab is signed in; see `NO_AT_TOKEN` |
| **403** | Extension (`background.js:432`) | `CAPTCHA_FAILED`, `NO_FLOW_TAB`, or `MODEL_ACCESS_DENIED` | CAPTCHA → retry loop; NO_FLOW_TAB → fail (user must open Flow); tier → fail |
| **404** | Flow API | `media_id` not found (expired upload) | Same as "Requested entity was not found" — auto re-upload |
| **429** | Flow API | Rate limited / quota | Back off + retry; if `USER_QUOTA_REACHED` appears, fail |
| **500** | Flow backend **or** extension fetch exception (`background.js:504`) | Transient server error OR network drop during fetch | Retry with exponential backoff |
| **502** | FastAPI default (`agent/api/flow.py:80,92`) | Extension returned error without explicit status | Retry; check extension health |
| **503** | FastAPI (`api/flow.py`) | "Extension not connected" or `NO_FLOW_KEY` | Worker waits for reconnect — status set to PENDING, not FAILED |
| **503** | FastAPI (`api/flow.py`) | "Extension not connected" | Worker waits for reconnect — status set to PENDING, not FAILED |
| **504** | Agent | 60s timeout waiting for extension WS response | Treated as transient; re-queue PENDING |

Status-code detection logic lives in `agent/worker/_parsing.py:_is_error` — a result is an error if `result.error` is set, `status >= 400`, **or** `data.error` is present.
Expand All @@ -825,7 +840,6 @@ String patterns in `error_message` that the worker recognizes:
| `Extension not connected` | Chrome extension offline or WS dropped | 503 returned; worker re-queues PENDING and waits |
| `extension reconnected` / `extension disconnected` | WS bounce mid-request | Re-queue PENDING without incrementing `retry_count` |
| `extension_switched` | User switched Flow tabs mid-generation | Re-queue PENDING |
| `NO_FLOW_KEY` | Extension has no captured bearer token — **legacy path only**, expected on the batch path | Only meaningful with `USE_BATCH_RPC=0` |
| `NO_AT_TOKEN` | Flow tab is signed out, on an interstitial, or still booting | Open `flow.google.com`, sign in, let the app load |
| `NO_FLOW_PROJECT` | No Flow project to scope the RPC to | Pin `FLOW_PROJECT_ID` — **terminal, not retried** |
| `UNSUPPORTED_ON_BATCH_API` | Upscale / r2v / chaining — payload never captured | See `docs/CAPTURE.md` — **terminal, not retried** |
Expand Down
12 changes: 7 additions & 5 deletions agent/api/flow.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
from pydantic import BaseModel, Field
from typing import Literal, Optional

from agent.config import USE_BATCH_RPC, FLOW_PROJECT_ID, FLOW_ALLOW_DEGRADED
from agent.config import FLOW_PROJECT_ID, FLOW_ALLOW_DEGRADED
from agent.services.flow_client import get_flow_client
from agent.services.omni_flash import (
check_omni_flash_status,
Expand Down Expand Up @@ -122,15 +122,17 @@ class UpscaleImageRequest(BaseModel):

@router.get("/status")
async def extension_status():
"""Extension health, and which transport it is being asked to speak.
"""Extension health.

`flow_key_present` is a legacy-path signal: the batchexecute path has no
bearer token at all, so false is expected there rather than a fault.
`flow_key_present: false` is expected, not a fault: batchexecute
authenticates in the page and there is no bearer token to capture.
"""
client = get_flow_client()
return {
"connected": client.connected,
"transport": "batch" if USE_BATCH_RPC else "legacy_rest",
# One transport now. The key stays so the documented pre-flight check
# (CLAUDE.md) keeps reading {"transport": "batch", ...}.
"transport": "batch",
"flow_project_id": FLOW_PROJECT_ID or None,
"allow_degraded": FLOW_ALLOW_DEGRADED,
"flow_key_present": client._flow_key is not None,
Expand Down
8 changes: 4 additions & 4 deletions agent/api/projects.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
from fastapi import APIRouter, HTTPException
from pydantic import BaseModel

from agent.config import BASE_DIR, USE_BATCH_RPC
from agent.config import BASE_DIR
from agent.models.project import Project, ProjectCreate, ProjectUpdate
from agent.models.character import Character
from agent.sdk.persistence.sqlite_repository import SQLiteRepository
Expand Down Expand Up @@ -172,9 +172,9 @@ async def create(body: ProjectCreate):

detected_tier = await _detect_user_tier(client)

# On the batch path Flow no longer creates projects for us — the uuid comes
# from the request or from FLOW_PROJECT_ID. The legacy path still mints one.
flow_project_id = client.flow_project_id(body.flow_project_id) if USE_BATCH_RPC else None
# Flow no longer creates projects for us — the uuid comes from the request
# or from FLOW_PROJECT_ID.
flow_project_id = client.flow_project_id(body.flow_project_id)
if flow_project_id:
logger.info("Flow project reused: %s", flow_project_id)
else:
Expand Down
57 changes: 5 additions & 52 deletions agent/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,19 +15,12 @@
WS_HOST = os.environ.get("WS_HOST", "127.0.0.1")
WS_PORT = int(os.environ.get("WS_PORT", "9222"))

# ─── Google Flow API ────────────────────────────────────────
# Legacy REST host. Flow moved to flow.google.com in September 2026 and stopped
# minting the `Bearer ya29.…` this host needs, so these are only reachable with
# USE_BATCH_RPC=0 on a browser profile that still has an old token.
GOOGLE_FLOW_API = "https://aisandbox-pa.googleapis.com"
GOOGLE_API_KEY = os.environ.get("GOOGLE_API_KEY", "AIzaSyBtrm0o5ab1c-Ec8ZuLcGt3oJAA5VWt3pY")
RECAPTCHA_SITE_KEY = os.environ.get("RECAPTCHA_SITE_KEY", "6LdsFiUsAAAAAIjVDZcuLhaHiDn5nnHVXVRQGeMV")

# ─── Flow batchexecute (the current path) ───────────────────

# ─── Flow batchexecute ──────────────────────────────────────
# Every call is signed in the page with the session cookie plus a per-page `at`
# token, so the extension runs it inside a signed-in flow.google.com tab. Set
# USE_BATCH_RPC=0 only to fall back to the dead REST path for a post-mortem.
USE_BATCH_RPC = os.environ.get("USE_BATCH_RPC", "1") == "1"
# token, so the extension runs it inside a signed-in flow.google.com tab. This
# is the only transport; the REST path it replaced was removed once Flow stopped
# minting the bearer it needed.

# The Flow project every RPC is scoped to. Project creation went with the old
# labs.google tRPC endpoint, so a project is made once in the Flow UI and its
Expand Down Expand Up @@ -66,20 +59,6 @@
# newly introduced image models do not require a Flow Kit release.
DEFAULT_IMAGE_MODEL = _MODELS.get("default_image_model", "NANO_BANANA_PRO")

# ─── API Endpoints ───────────────────────────────────────────
ENDPOINTS = {
"generate_images": "/v1/projects/{project_id}/flowMedia:batchGenerateImages",
"generate_video": "/v1/video:batchAsyncGenerateVideoStartImage",
"generate_video_start_end": "/v1/video:batchAsyncGenerateVideoStartAndEndImage",
"generate_video_references": "/v1/video:batchAsyncGenerateVideoReferenceImages",
"upscale_video": "/v1/video:batchAsyncGenerateVideoUpsampleVideo",
"upscale_image": "/v1/flow/upsampleImage",
"upload_image": "/v1/flow/uploadImage",
"check_video_status": "/v1/video:batchCheckAsyncVideoGenerationStatus",
"get_credits": "/v1/credits",
"get_media": "/v1/media/{media_id}",
}

# ─── Output Directories ─────────────────────────────────────
OUTPUT_DIR = BASE_DIR / "output"
SHARED_OUTPUT_DIR = OUTPUT_DIR / "_shared"
Expand Down Expand Up @@ -131,29 +110,3 @@ def _load_suno_key() -> str:
SUNO_POLL_INTERVAL = int(os.environ.get("SUNO_POLL_INTERVAL", "5"))
SUNO_POLL_TIMEOUT = int(os.environ.get("SUNO_POLL_TIMEOUT", "600"))

# ─── Header Randomization Pools ─────────────────────────────
USER_AGENTS = [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36",
]

CHROME_VERSIONS = [
'"Google Chrome";v="109", "Chromium";v="109"',
'"Google Chrome";v="110", "Chromium";v="110"',
'"Google Chrome";v="111", "Chromium";v="111"',
'"Google Chrome";v="113", "Not-A.Brand";v="24"',
'"Google Chrome";v="120", "Not-A.Brand";v="24"',
'"Google Chrome";v="141", "Not?A_Brand";v="8", "Chromium";v="141"',
]

BROWSER_VALIDATIONS = [
"SgDQo8mvrGRdD61Pwo8wyWVgYgs=",
]

CLIENT_DATA = [
"CKi1yQEIh7bJAQiktskBCKmdygEIvorLAQiUocsBCIagzQEYv6nKARjRp88BGKqwzwE=",
]
21 changes: 6 additions & 15 deletions agent/sdk/services/operations.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ def _char_matches(c: dict, name_set: set) -> bool:
import aiohttp

from agent.db import crud
from agent.config import USE_BATCH_RPC, VIDEO_POLL_INTERVAL, VIDEO_POLL_TIMEOUT
from agent.config import VIDEO_POLL_INTERVAL, VIDEO_POLL_TIMEOUT
from agent.utils.paths import scene_4k_path
from agent.utils.slugify import slugify
from agent.worker._parsing import (
Expand Down Expand Up @@ -447,28 +447,19 @@ async def generate_scene_video(self, scene: dict, orientation: str,
base_prompt = scene.get("video_prompt") or scene.get("prompt", "")
prompt = await _build_video_prompt(base_prompt, scene, pid)

# Check if already submitted (op_name saved from previous attempt)
# OLD schema (Lite/Fast/Ultra): op_name is "models/.../operations/..." → re-poll via check_video_status
# NEW schema (Low Priority workflow): op_name is bare UUID → cannot recover (need primary_media_id
# which isn't persisted yet); fall through and resubmit (Low Priority is free, duplicate is OK)
# Already submitted on a previous attempt? Re-poll it.
existing_op = None
if request_id:
req_row = await crud.get_request(request_id)
existing_op = req_row.get("request_id") if req_row else None

# A bare uuid means different things on the two transports. On the
# legacy path it is a Low Priority workflow name that cannot be
# re-polled, so the retry resubmits. On the batch path it is the
# operation id, and looking it up in the project listing is exactly
# what the status poll does — resubmitting there would abandon a
# running render and pay for a second one.
bare_uuid = bool(existing_op and len(existing_op) == 36 and existing_op.count("-") == 4)
looks_like_workflow_uuid = bare_uuid and not USE_BATCH_RPC
if existing_op and not looks_like_workflow_uuid:
# A bare uuid is the operation id, and looking it up in the project
# listing is exactly what the status poll does — resubmitting instead
# would abandon a running render and pay for a second one.
if existing_op:
logger.info("Video gen already submitted (op=%s), re-polling", existing_op[:30])
operations = [{"operation": {"name": existing_op}, "status": "MEDIA_GENERATION_STATUS_PENDING"}]
return await _poll_operations(self._client, operations)
# else: workflow UUID — fall through and resubmit fresh

submit_result = await self._client.generate_video(
start_image_media_id=image_media_id,
Expand Down
Loading