diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 368baf9e..a416a89f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -43,6 +43,16 @@ jobs: - name: Validate observability IaC run: bash deploy/observability/scripts/validate-iac.sh + - name: Verify local log pipeline and trace correlation + timeout-minutes: 10 + run: python3 deploy/observability/scripts/verify-log-pipeline.py + + - name: Upload log pipeline verification + uses: actions/upload-artifact@v4 + with: + name: observability-log-pipeline + path: build/reports/observability/log-pipeline.json + backend-build-test: name: Backend Build & Test (Unit + Integration) runs-on: ubuntu-latest diff --git a/deploy/docker-compose.ec2.yml.example b/deploy/docker-compose.ec2.yml.example index 9a9b1449..5f4f04be 100644 --- a/deploy/docker-compose.ec2.yml.example +++ b/deploy/docker-compose.ec2.yml.example @@ -3,6 +3,15 @@ services: image: ${IMAGE_REPOSITORY}:${IMAGE_TAG} container_name: beach-complex-backend restart: unless-stopped + labels: + beach.logs: backend + logging: + driver: json-file + options: + max-size: "50m" + max-file: "5" + compress: "false" + labels: "beach.logs" env_file: - ${APP_RUNTIME_ENV_FILE:-.env} ports: diff --git a/deploy/observability/README.md b/deploy/observability/README.md new file mode 100644 index 00000000..dd2ff9cb --- /dev/null +++ b/deploy/observability/README.md @@ -0,0 +1,182 @@ +# 관측 로그 파이프라인 + +앱 서버의 Docker `json-file` 로그를 Alloy가 읽어 관측 서버 Loki로 보내고, Grafana에서 로그와 Tempo Trace를 연결한다. +Trace는 기존 경로대로 애플리케이션에서 Tempo로 직접 전송한다. + +```text +앱 EC2: Backend stdout → Docker json-file → Alloy ── TCP 3100 ──→ 관측 EC2: Loki + └────────────────── OTLP 4317/4318 ──────────────────→ Tempo + ↑ + Grafana 조회 +``` + +구현 범위는 [이슈 #233](https://github.com/Beach-complex/Beach_complex/issues/233)의 PR4다. 서버 적용과 실제 앱 검증은 아직 수행하지 않았다. + +- 검증 명령만 필요하면 [verify-local.txt](verify-local.txt)를 사용한다. +- 서버에 적용할 때는 아래의 서버 적용 절차와 운영 제약을 함께 확인한다. + +## 로컬 검증 + +필요 도구: Docker Engine + Compose v2 이상, Python 3, Terraform 1.15.8, cloud-init. +Terraform 테스트는 mock AWS provider를 사용하므로 AWS 자격증명이나 실제 리소스가 필요하지 않다. + +기존 `beach.logs=backend` 컨테이너가 있으면 중지된 컨테이너도 포함하여 통합 테스트가 실행을 거부한다. +이 경우 앱 컨테이너를 건드리지 않고 별도의 테스트용 Docker 데몬에서 실행한다. + +저장소 루트에서 다음 명령을 실행한다. + +```bash +bash deploy/observability/scripts/validate-iac.sh +python3 deploy/observability/scripts/verify-log-pipeline.py +``` + +### 설정 검사 + +`validate-iac.sh`는 Terraform fmt/init/validate와 mock plan 3건(dev/staging/prod), cloud-init schema 및 EC2 user data 16KB 제한, +Compose 병합, Loki/Alloy 실제 이미지의 설정 검증을 수행한다. + +### 통합 테스트 + +`verify-log-pipeline.py`는 임의의 localhost 포트, 전용 Compose 프로젝트와 볼륨을 만든다. 합성 JSON 로그와 OTLP span으로 다음을 확인하고 자체 리소스를 정리한다. + +- 실제 Docker json-file → Alloy → Loki 수집, 원본 JSON 보존 및 traceId 검색 +- 다른 컨테이너 제외, `/series` 기준 인덱스 라벨 4개만 존재 +- Grafana 데이터 소스의 변수·정규식·연결 상태, Grafana 프록시 API를 통한 로그 ↔ Trace 조회 +- Alloy 정상 재시작 후 저장된 파일 지문·읽기 위치 사용 및 중지 중 쌓인 로그 수집 +- 두 컨테이너의 stdout/stderr에서 조각이 교차하는 40KB JSON의 원문 보존과 traceId 검색 +- Alloy 중지 중 회전된 로그 110건과 파일 경계에 걸친 90KB JSON 복구, 재시작 후 중복 전송 검사 +- backend 컨테이너 교체 시 새 로그 경로 발견, 짧은 Loki 중단 후 전송 재시도 +- Loki 컨테이너 재생성 후 기존 데이터 조회 및 실제 적용된 보존 기간 설정 + +보고서는 `build/reports/observability/log-pipeline.json`에 생성하며 CI에서도 같은 검증을 실행하도록 구성했다. + +### 서버에서 별도로 확인할 항목 + +로컬 검증은 합성 데이터와 Docker 볼륨을 사용한다. 실제 EC2/EBS 부팅·재부팅, 브라우저 클릭, +운영 앱의 로그와 span 연결, 보존 기간 경과에 따른 실제 삭제는 서버 적용 후 확인할 항목이다. +24시간 운영 검증, 대시보드·알림·SLI/SLO 및 Terraform 원격 상태 저장소는 #233에서 분리한 후속 범위다. + +## 구성과 저장 정책 + +| 구성 | 파일 / 동작 | +|---|---| +| Loki | `compose/loki/local-config.yaml`, 단일 프로세스, TSDB v13 + 파일 시스템 저장소 | +| 저장소 | 관측 EBS의 `/opt/beach-observability/data/loki` → `/loki`; chunk/index/WAL/compactor 상태 포함 | +| 보존 기간 | Terraform `env`: dev 3일, staging 7일, prod 14일; compactor 활성화, 삭제 지연 2시간 | +| 접근 | 관측 SG TCP 3100은 앱 SG만 허용. 인증 없는 Loki를 인터넷에 직접 공개하지 않는다. | +| 수집 | `app-agent/alloy/config.alloy`, Docker 로그 디렉터리 읽기 전용; Docker socket 사용 없음 | +| 수집 대상 | Docker logging attribute `beach.logs=backend`인 컨테이너만 통과 | +| 재배포 | 기본 200ms 주기로 새 로그 파일을 탐색. 컨테이너 ID를 설정에 고정하지 않는다. | +| 읽기 위치 | 파일 지문과 읽기 위치를 앱 서버 `alloy-data` Docker 볼륨에 저장. `down -v`로 삭제하지 않는다. | +| 회전 | backend Docker 로그 파일당 50MB, 최대 5개, 압축하지 않음 | +| 인덱스 라벨 | `app=beach-complex`, `env`, `service=backend`, `host`만 유지 | +| 식별자 | traceId/spanId/requestId/userId/notificationId/outboxEventId는 JSON 본문에 유지; 검색 인덱스용 라벨로 만들지 않는다. | + +## 서버 적용 절차 + +### 1. 관측 서버의 변경 사항 확인 + +관측 서버 Terraform은 검증된 EBS 마운트 아래에 Loki 데이터 디렉터리(UID/GID 10001)를 만들고, +설정 및 환경별 `LOKI_RETENTION_PERIOD`를 배치한다. 기존 마운트 가드를 유지한다. +현재 모듈의 `user_data_replace_on_change = true` 때문에 이 변경의 apply는 관측 EC2 교체를 유발할 수 있다. +적용 전 실제 state 기준 plan의 교체 대상과 EBS 보존 여부를 확인해야 한다. + +### 2. 앱 서버의 환경 변수 설정 + +`app-agent/.env.example`을 참고해 `app-agent/.env`를 작성한다. + +```dotenv +LOKI_URL=http://<관측-서버-private-ip>:3100/loki/api/v1/push +APP_ENVIRONMENT=dev +LOG_HOST=beach-app-ec2 +DOCKER_LOG_ROOT=/var/lib/docker/containers +``` + +`LOG_HOST`에는 재배포 후에도 유지되는 서버 식별자를 사용한다. +`DOCKER_LOG_ROOT`는 Docker 데이터 경로 아래의 `containers` 디렉터리다. +앱 서버에서 `docker info --format '{{.DockerRootDir}}'`로 데이터 경로를 확인한 뒤 `/containers`를 붙인다. + +### 3. 앱 로그 설정 적용 및 Alloy 기동 + +실제 앱 Compose의 서비스가 `app`인지 확인한 뒤 `backend-logging.override.yml`을 함께 사용한다. +서비스명이 다르면 override의 서비스 키를 실제 이름으로 맞춘다. +로그 속성은 컨테이너 생성 때 적용되므로 기존 컨테이너에는 재생성이 필요하다. + +```bash +# 앱 서버, 실제 기존 Compose 파일·환경 파일을 그대로 지정해서 병합 결과를 먼저 확인한다. +docker compose -f <기존-compose.yml> -f <경로>/app-agent/backend-logging.override.yml config +# 이후 배포 단계에서만 app을 재생성하고 Alloy를 기동한다. +docker compose -f <기존-compose.yml> -f <경로>/app-agent/backend-logging.override.yml up -d --no-deps app +docker compose --env-file <경로>/app-agent/.env -f <경로>/app-agent/docker-compose.yml up -d +``` + +지속 배포에 사용하는 Compose에도 logging 설정이 남아 있어야 한다. 저장소의 `deploy/docker-compose.ec2.yml.example`에도 같은 설정을 추가했다. +`APP_ENVIRONMENT`는 Loki 라벨이며 Spring profile을 변경하지 않는다. 현재 Logback의 prod profile이 NDJSON을 출력한다. +JSON 검색을 확인할 때는 실제 앱 출력 형식을 먼저 확인하고, 수집 때문에 애플리케이션 profile을 임의로 바꾸지 않는다. + +### 4. Grafana에서 로그와 Trace 조회 + +Grafana Explore → Loki에서 다음을 조회한다. `env`는 적용한 환경에 맞춘다. + +```logql +{app="beach-complex", env="dev", service="backend"} | json +{app="beach-complex", env="dev", service="backend"} | json | traceId="<32자리 trace ID>" +``` + +로그의 `View trace` 링크는 Tempo를 가리킨다. JSON `traceId`와 현재 개발용 텍스트의 `traceId=...`를 지원한다. +Tempo의 Trace to logs는 span 시간 전후 5분에서 동일 traceId 문자열을 검색한다. +Trace가 샘플링되지 않았거나 Tempo 보존 기간이 지났으면 해당 로그의 링크 대상은 없을 수 있다. + +## 운영 제약 + +- **전송 보장:** 읽기 위치 저장과 Loki 전송 완료는 하나의 작업으로 처리되지 않는다. 짧은 장애에는 재시도하지만, 장시간 장애·강제 종료·회전 한도 초과 시 무손실이나 정확히 한 번의 전송을 보장하지 않는다. + +- **긴 로그 재조립:** 조각은 메모리에만 저장한다. 첫 조각을 읽은 뒤 5초 안에 한 줄이 완성되지 않거나 재조립 도중 Alloy가 종료되면 원문 보존을 보장하지 않는다. + +- **로그 크기와 회전:** 한 줄은 256KiB 이내로 유지한다. 수집 대상은 최대 5개 파일이므로 Docker의 보존 개수와 압축하지 않는 설정도 유지한다. + +- **읽기 위치 보존:** `alloy-data` 볼륨을 삭제하면 저장된 읽기 위치도 사라진다. `docker compose down -v`를 사용하지 않는다. + +- **이전 초안에서 전환:** 기존 `loki.source.file`의 positions 파일은 자동 이관되지 않는다. 새 수집기를 적용하면 남아 있는 로그를 다시 읽을 수 있다. + +- **Loki 보존 기간:** 보존 기간은 디스크 용량 상한이 아니다. 오래된 데이터는 compactor 실행 주기와 삭제 지연 이후에 삭제된다. + +## 구현 상세 + +
+Alloy 실행 권한, 파일 식별 방식과 정렬 우회 설정 + +### 실행 권한 + +Alloy는 root 소유 Docker 파일을 읽기 위해 UID 0으로 실행하지만, capability를 모두 제거하고 `no-new-privileges`를 사용한다. +빈 상태 볼륨에 이미지 내부 UID 소유권이 복사되지 않도록 `nocopy: true`를 지정한다. +이는 rootful Docker의 기본 `json-file` 배치를 기준으로 한다. rootless Docker나 user namespace remapping 환경은 별도 권한 검증이 필요하다. + +### 수집기 버전과 로그 처리 + +`otelcol.receiver.filelog`와 `otelcol.storage.file`은 고정한 Alloy v1.19.2에서 공개 미리보기 기능이다. +Compose와 설정 검사에 `--stability.level=public-preview`를 명시했다. 업그레이드할 때 같은 회귀 테스트를 실행한다. + +파일 지문은 파일 내용 일부를 비교해 같은 파일인지 식별하는 값이다. Docker 레코드 끝의 시각까지 비교하도록 크기를 128KiB로 설정했다. 긴 로그 조각의 앞부분이 같아도 다른 파일로 구분한다. +파일 지문으로 이름이 바뀐 파일을 추적하고 `.log.4`부터 `.log`까지 차례로 읽는다. +Docker가 나눈 한 줄은 컨테이너 ID와 stdout/stderr를 기준으로 묶어, 줄바꿈이 나올 때까지 구분자 없이 합친다. + +### 정렬 옵션의 우회 설정 + +이 버전의 Alloy 정렬 설정은 `lexicographic`을 받지만 내부 Stanza는 `alphabetical`을 요구한다. +이를 피하려고 `operators`의 `file_input`에 정렬을 직접 설정했다. 바깥쪽 입력은 사용하지 않는 경로를 가리킨다. +두 입력이 같은 파일을 읽지 않으며, 실제 입력은 같은 file storage에 읽기 위치를 저장한다. +이 우회 설정은 해당 버전의 매핑이 수정되면 제거할 수 있다. + +
+ +## 참고 자료 + +- [ADR-014: Loki 및 보존·라벨 정책](https://github.com/Beach-complex/beach_complex_docs/blob/main/adr/ADR-014-logging-storage-backend.md) +- [ADR-015: Alloy 및 file tail](https://github.com/Beach-complex/beach_complex_docs/blob/main/adr/ADR-015-log-collector.md) +- [Alloy filelog receiver](https://grafana.com/docs/alloy/latest/reference/components/otelcol/otelcol.receiver.filelog/) +- [Alloy file storage](https://grafana.com/docs/alloy/latest/reference/components/otelcol/otelcol.storage.file/) +- [Stanza 재조립 설정](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.158.0/pkg/stanza/docs/operators/recombine.md) +- [Alloy Loki write 및 재시도](https://grafana.com/docs/alloy/latest/reference/components/loki/loki.write/) +- [Loki retention](https://grafana.com/docs/loki/latest/operations/storage/retention/) +- [Grafana Tempo와 로그 연결](https://grafana.com/docs/grafana/latest/datasources/tempo/configure-tempo-data-source/) diff --git a/deploy/observability/app-agent/.env.example b/deploy/observability/app-agent/.env.example new file mode 100644 index 00000000..4a7c6ec7 --- /dev/null +++ b/deploy/observability/app-agent/.env.example @@ -0,0 +1,5 @@ +LOKI_URL=http://:3100/loki/api/v1/push +APP_ENVIRONMENT=dev +LOG_HOST=beach-app-ec2 +# Set to DockerRootDir/containers when Docker uses a non-default data root. +DOCKER_LOG_ROOT=/var/lib/docker/containers diff --git a/deploy/observability/app-agent/alloy/config.alloy b/deploy/observability/app-agent/alloy/config.alloy new file mode 100644 index 00000000..7bed5db9 --- /dev/null +++ b/deploy/observability/app-agent/alloy/config.alloy @@ -0,0 +1,114 @@ +logging { + level = "info" +} + +// Fingerprints identify files even after Docker renames them during rotation. +// Both components are public preview in the pinned Alloy v1.19.2 image. +otelcol.storage.file "docker" { + directory = "/var/lib/alloy/docker-offsets" + create_directory = true + fsync = true +} + +otelcol.receiver.filelog "docker" { + // v1.19.2's ordering wrapper accepts "lexicographic", whereas its bundled + // stanza reader requires "alphabetical". Configure the file_input operator + // directly until that upstream mapping is fixed. The wrapper input is idle. + include = ["/var/log/docker/__unused_wrapper_input__"] + storage = otelcol.storage.file.docker.handler + operators = [ + { + type = "json_parser", + id = "parse_docker", + output = "select_backend", + on_error = "drop", + timestamp = { + parse_from = "attributes.time", + layout_type = "gotime", + layout = "2006-01-02T15:04:05.999999999Z07:00", + }, + }, + { + type = "file_input", + id = "docker_files", + output = "parse_docker", + include = ["/var/log/docker/*/*-json.log", "/var/log/docker/*/*-json.log.[1-4]"], + start_at = "beginning", + // Include Docker timestamps after escaped 16 KiB chunks in the fingerprint. + // A 1 KiB prefix can be identical across containers with repetitive payloads. + fingerprint_size = "128KiB", + include_file_path = true, + include_file_name = false, + // Oldest file first, one reader at a time, across all five files. + max_concurrent_files = 1, + max_batches = 0, + ordering_criteria = { + regex = "/[^/]+/[^/]+-json\\.log(?P\\.[1-4])?$", + group_by = "/([^/]+)/[^/]+-json\\.log", + top_n = 5, + sort_by = [{sort_type = "alphabetical", regex_key = "rotation", ascending = false}], + }, + attributes = { + app = "beach-complex", + env = sys.env("APP_ENVIRONMENT"), + service = "backend", + host = sys.env("LOG_HOST"), + "loki.attribute.labels" = "app,env,service,host", + "loki.format" = "raw", + }, + }, + { + type = "filter", + id = "select_backend", + expr = "attributes.attrs == nil or attributes.attrs[\"beach.logs\"] != \"backend\"", + }, + { + type = "regex_parser", + parse_from = "attributes[\"log.file.path\"]", + regex = "/(?P[^/]+)/[^/]+-json.log", + on_error = "drop", + }, + { + type = "add", + field = "attributes.source", + value = "EXPR(attributes.container + \"/\" + attributes.stream)", + }, + { + type = "recombine", + combine_field = "attributes.log", + combine_with = "", + is_last_entry = "attributes.log endsWith \"\\n\"", + source_identifier = "attributes.source", + overwrite_with = "oldest", + force_flush_period = "5s", + max_log_size = 262144, + }, + { + type = "move", + from = "attributes.log", + to = "body", + }, + ] + + output { + logs = [otelcol.exporter.loki.backend.input] + } +} + +otelcol.exporter.loki "backend" { + forward_to = [loki.process.backend.receiver] +} + +loki.process "backend" { + forward_to = [loki.write.observability.receiver] + // Preserve app JSON; IDs and Docker paths never become indexed labels. + stage.label_keep { + values = ["app", "env", "service", "host"] + } +} + +loki.write "observability" { + endpoint { + url = sys.env("LOKI_URL") + } +} diff --git a/deploy/observability/app-agent/backend-logging.override.yml b/deploy/observability/app-agent/backend-logging.override.yml new file mode 100644 index 00000000..68e00e33 --- /dev/null +++ b/deploy/observability/app-agent/backend-logging.override.yml @@ -0,0 +1,12 @@ +# Merge with the app server's existing Compose file. Service name must be app. +services: + app: + labels: + beach.logs: backend + logging: + driver: json-file + options: + max-size: "50m" + max-file: "5" + compress: "false" + labels: "beach.logs" diff --git a/deploy/observability/app-agent/docker-compose.yml b/deploy/observability/app-agent/docker-compose.yml new file mode 100644 index 00000000..22547f11 --- /dev/null +++ b/deploy/observability/app-agent/docker-compose.yml @@ -0,0 +1,32 @@ +services: + alloy: + image: grafana/alloy:v1.19.2 + restart: unless-stopped + # Docker log files are root-owned; no Docker socket or host write access. + user: "0:0" + command: + - run + - --stability.level=public-preview + - --server.http.listen-addr=127.0.0.1:12345 + - --storage.path=/var/lib/alloy + - /etc/alloy/config.alloy + environment: + LOKI_URL: ${LOKI_URL:?Set the observability private Loki push URL} + APP_ENVIRONMENT: ${APP_ENVIRONMENT:?Set dev, staging or prod} + LOG_HOST: ${LOG_HOST:?Set a stable application host identifier} + volumes: + - ./alloy/config.alloy:/etc/alloy/config.alloy:ro + - ${DOCKER_LOG_ROOT:-/var/lib/docker/containers}:/var/log/docker:ro + # Keep the empty volume root-owned instead of copying image UID ownership. + - type: volume + source: alloy-data + target: /var/lib/alloy + volume: + nocopy: true + security_opt: + - no-new-privileges:true + cap_drop: + - ALL + +volumes: + alloy-data: diff --git a/deploy/observability/compose/docker-compose.yml b/deploy/observability/compose/docker-compose.yml index aae74802..11472548 100644 --- a/deploy/observability/compose/docker-compose.yml +++ b/deploy/observability/compose/docker-compose.yml @@ -1,4 +1,20 @@ services: + loki: + image: grafana/loki:3.7.8 + container_name: beach-loki + restart: unless-stopped + command: + - -config.file=/etc/loki/local-config.yaml + - -config.expand-env=true + environment: + LOKI_RETENTION_PERIOD: ${LOKI_RETENTION_PERIOD:-72h} + volumes: + - ./loki/local-config.yaml:/etc/loki/local-config.yaml:ro + - ../data/loki:/loki + # Terraform permits TCP 3100 only from the application server SG. + ports: + - 3100:3100 + prometheus: image: prom/prometheus:v3.13.2 container_name: beach-prometheus @@ -46,6 +62,7 @@ services: depends_on: - prometheus - tempo + - loki environment: GF_USERS_ALLOW_SIGN_UP: "false" volumes: diff --git a/deploy/observability/compose/grafana/provisioning/datasources/datasources.yml b/deploy/observability/compose/grafana/provisioning/datasources/datasources.yml index 777168da..e78c4dd7 100644 --- a/deploy/observability/compose/grafana/provisioning/datasources/datasources.yml +++ b/deploy/observability/compose/grafana/provisioning/datasources/datasources.yml @@ -11,6 +11,21 @@ datasources: isDefault: true editable: false + - name: Loki + uid: loki + type: loki + access: proxy + url: http://loki:3100 + isDefault: false + editable: false + jsonData: + derivedFields: + - name: TraceID + datasourceUid: tempo + matcherRegex: '(?:"traceId"\s*:\s*"|traceId=)([a-f0-9]{32})' + url: '$${__value.raw}' + urlDisplayLabel: View trace + - name: Tempo uid: tempo type: tempo @@ -18,3 +33,10 @@ datasources: url: http://tempo:3200 isDefault: false editable: false + jsonData: + tracesToLogsV2: + datasourceUid: loki + spanStartTimeShift: '-5m' + spanEndTimeShift: '5m' + customQuery: true + query: '{app="beach-complex", service="backend"} |= "$${__span.traceId}"' diff --git a/deploy/observability/compose/loki/local-config.yaml b/deploy/observability/compose/loki/local-config.yaml new file mode 100644 index 00000000..12920b3c --- /dev/null +++ b/deploy/observability/compose/loki/local-config.yaml @@ -0,0 +1,46 @@ +auth_enabled: false + +server: + http_listen_port: 3100 + +common: + instance_addr: 127.0.0.1 + path_prefix: /loki + storage: + filesystem: + chunks_directory: /loki/chunks + rules_directory: /loki/rules + replication_factor: 1 + ring: + kvstore: + store: inmemory + +schema_config: + configs: + - from: 2024-01-01 + store: tsdb + object_store: filesystem + schema: v13 + index: + prefix: index_ + period: 24h + +ingester: + wal: + enabled: true + dir: /loki/wal + +compactor: + working_directory: /loki/compactor + retention_enabled: true + retention_delete_delay: 2h + delete_request_store: filesystem + +limits_config: + # ADR-014: dev 3d, staging 7d, prod 14d; cloud-init supplies the environment. + retention_period: ${LOKI_RETENTION_PERIOD:-72h} + # Do not implicitly add service_name to the explicit four-label contract. + discover_service_name: [] + +analytics: + reporting_enabled: false diff --git a/deploy/observability/scripts/fixtures/log-emitter.py b/deploy/observability/scripts/fixtures/log-emitter.py new file mode 100644 index 00000000..06807c55 --- /dev/null +++ b/deploy/observability/scripts/fixtures/log-emitter.py @@ -0,0 +1,23 @@ +"""Write synthetic logs through PID 1; /partial holds back the newline.""" +import sys + +from http.server import BaseHTTPRequestHandler, HTTPServer + + +class Handler(BaseHTTPRequestHandler): + def do_POST(self): + line = self.rfile.read(int(self.headers['Content-Length'])).decode() + stream = sys.stderr if self.path.startswith('/stderr') else sys.stdout + print(line, end='' if self.path.endswith('/partial') else '\n', file=stream, flush=True) + self.send_response(204) + self.end_headers() + + def do_GET(self): + self.send_response(200) + self.end_headers() + + def log_message(self, *_): + pass + + +HTTPServer(('0.0.0.0', 8000), Handler).serve_forever() diff --git a/deploy/observability/scripts/validate-iac.sh b/deploy/observability/scripts/validate-iac.sh index 8a132f5b..2acbcdc1 100644 --- a/deploy/observability/scripts/validate-iac.sh +++ b/deploy/observability/scripts/validate-iac.sh @@ -42,6 +42,8 @@ validate_terraform() { -lockfile=readonly \ -no-color terraform -chdir="$terraform_environment" validate -no-color + # Mock provider: verifies SG and retention contracts without contacting AWS. + terraform -chdir="$terraform_environment" test -no-color } render_cloud_init() { @@ -59,6 +61,8 @@ base64gzip(templatefile("${path.module}/../../modules/observability_ec2/cloud-in ) ), tempo_config_base64gzip = base64gzip(file("${path.module}/../../../compose/tempo/tempo.yaml")), + loki_config_base64gzip = base64gzip(file("${path.module}/../../../compose/loki/local-config.yaml")), + loki_retention_period = lookup({ dev = "72h", staging = "168h", prod = "336h" }, var.env), grafana_datasources_base64gzip = base64gzip(file("${path.module}/../../../compose/grafana/provisioning/datasources/datasources.yml")) })) EOF @@ -96,7 +100,24 @@ EOF } validate_compose() { + local loki_image alloy_image docker compose -f "$compose_dir/docker-compose.yml" config --quiet + LOKI_URL=http://loki:3100/loki/api/v1/push APP_ENVIRONMENT=dev LOG_HOST=validation \ + docker compose -f deploy/observability/app-agent/docker-compose.yml config --quiet + IMAGE_REPOSITORY=local/backend IMAGE_TAG=validation APP_RUNTIME_ENV_FILE=/dev/null \ + docker compose -f deploy/docker-compose.ec2.yml.example \ + -f deploy/observability/app-agent/backend-logging.override.yml config --quiet + loki_image="$(docker compose -f "$compose_dir/docker-compose.yml" config --format json \ + | python3 -c 'import json,sys; print(json.load(sys.stdin)["services"]["loki"]["image"])')" + alloy_image="$(LOKI_URL=http://loki:3100/loki/api/v1/push APP_ENVIRONMENT=dev LOG_HOST=validation \ + docker compose -f deploy/observability/app-agent/docker-compose.yml config --images)" + docker run --rm \ + --volume "$compose_dir/loki/local-config.yaml:/etc/loki/local-config.yaml:ro" "$loki_image" \ + -config.file=/etc/loki/local-config.yaml -config.expand-env=true -verify-config=true + LOKI_URL=http://loki:3100/loki/api/v1/push APP_ENVIRONMENT=dev LOG_HOST=validation \ + docker run --rm --env LOKI_URL --env APP_ENVIRONMENT --env LOG_HOST \ + --volume "$PWD/deploy/observability/app-agent/alloy/config.alloy:/etc/alloy/config.alloy:ro" \ + "$alloy_image" validate --stability.level=public-preview /etc/alloy/config.alloy } main() { diff --git a/deploy/observability/scripts/verify-log-pipeline.py b/deploy/observability/scripts/verify-log-pipeline.py new file mode 100644 index 00000000..e49bc671 --- /dev/null +++ b/deploy/observability/scripts/verify-log-pipeline.py @@ -0,0 +1,310 @@ +#!/usr/bin/env python3 +"""Local Docker E2E: real json-file -> Alloy -> Loki -> Grafana/Tempo. + +Uses repository configs and isolated named volumes, network and random loopback +ports. Never calls AWS or deploys the backend. Only its own project is removed. +""" +import base64 +import copy +import datetime +import json +import os +from pathlib import Path +import re +import secrets +import subprocess +import tempfile +import time +import urllib.error +import urllib.parse +import urllib.request + +ROOT = Path(__file__).resolve().parents[3] +OBS = ROOT / 'deploy/observability' +PROJECT = 'beach-logs-test-' + secrets.token_hex(4) +REPORT = ROOT / 'build/reports/observability/log-pipeline.json' +ENV = dict(os.environ, LOKI_URL='http://loki:3100/loki/api/v1/push', + APP_ENVIRONMENT='dev', LOG_HOST=PROJECT, LOKI_RETENTION_PERIOD='72h') +PASSWORD = secrets.token_hex(20) +checks = [] + + +def run(*args, **kwargs): + return subprocess.check_output(args, text=True, env=ENV, **kwargs).strip() + + +def request(url, payload=None, headers=None): + req = urllib.request.Request(url, data=payload, headers=headers or {}) + with urllib.request.urlopen(req, timeout=8) as response: + return response.read().decode() + + +def wait_for(description, check, timeout=90): + deadline = time.monotonic() + timeout + last = None + while time.monotonic() < deadline: + try: + value = check() + if value: + return value + except (urllib.error.URLError, TimeoutError, ConnectionError, json.JSONDecodeError) as exc: + last = str(exc) + time.sleep(1) + raise AssertionError(f'Timed out: {description}; last error: {last}') + + +def passed(message): + checks.append(message) + print('PASS ' + message, flush=True) + + +def main(): + REPORT.unlink(missing_ok=True) + # Refuse to mix another backend's logs into the test Loki instance. + if run('docker', 'ps', '-aq', '--filter', 'label=beach.logs=backend'): + raise RuntimeError('A labelled backend exists; use an isolated Docker daemon for this test.') + docker_logs = run('docker', 'info', '--format', '{{.DockerRootDir}}') + '/containers' + base = json.loads(run('docker', 'compose', '-f', str(OBS / 'compose/docker-compose.yml'), + 'config', '--format', 'json')) + agent = json.loads(run('docker', 'compose', '-f', str(OBS / 'app-agent/docker-compose.yml'), + 'config', '--format', 'json')) + services = {n: copy.deepcopy(base['services'][n]) for n in ['loki', 'tempo', 'grafana']} + services['alloy'] = agent['services']['alloy'] + volumes = {} + # Keep actual command, image, read-only config mounts and env configuration. + # Replace only production storage, host ports and global container names. + for name, service in services.items(): + service.pop('container_name', None) + service.pop('depends_on', None) + service['restart'] = 'no' + for port in service.get('ports', []): + port.update(host_ip='127.0.0.1', published='0') + for mount in service.get('volumes', []): + if mount['target'] == '/var/log/docker': + mount['source'] = docker_logs + elif not mount.get('read_only', False): + volume = name + '-data' + volumes[volume] = {} + mount.clear() + mount.update(type='volume', source=volume, + target={'loki': '/loki', 'tempo': '/var/tempo', + 'grafana': '/var/lib/grafana', 'alloy': '/var/lib/alloy'}[name]) + if name == 'alloy': + mount['volume'] = {'nocopy': True} + if name == 'alloy': + service['environment']['LOG_HOST'] = PROJECT + # Expose collector metrics only on a random loopback port in this test. + service['command'] = [arg.replace('127.0.0.1:12345', '0.0.0.0:12345') + for arg in service['command']] + service['ports'] = [{'target': 12345, 'published': '0', 'host_ip': '127.0.0.1'}] + services['grafana']['environment'].update(GF_SECURITY_ADMIN_PASSWORD=PASSWORD, + GF_ANALYTICS_REPORTING_ENABLED='false') + logging_contract = json.loads(run('docker', 'compose', '-f', + str(OBS / 'app-agent/backend-logging.override.yml'), 'config', '--no-consistency', '--format', 'json'))['services']['app'] + for name, label in [('emitter', 'backend'), ('emitter2', 'backend'), ('ignored', 'unrelated')]: + services[name] = { + 'image': 'python:3.12-slim', 'command': ['python3', '-u', '/fixture.py'], + 'volumes': [{'type': 'bind', 'source': str(OBS / 'scripts/fixtures/log-emitter.py'), + 'target': '/fixture.py', 'read_only': True}], + 'ports': [{'target': 8000, 'published': '0', 'host_ip': '127.0.0.1'}], + 'labels': {'beach.logs': label}, 'logging': copy.deepcopy(logging_contract['logging']), + } + # Exercise real rotation quickly while retaining the production five-file policy. + for name in ['emitter', 'emitter2', 'ignored']: + services[name]['logging']['options']['max-size'] = '64k' + document = {'services': services, 'volumes': volumes} + with tempfile.TemporaryDirectory(prefix=PROJECT) as temp: + config = Path(temp) / 'compose.json' + config.write_text(json.dumps(document)) + os.chmod(config, 0o600) + + def compose(*args): + return run('docker', 'compose', '-p', PROJECT, '-f', str(config), *args) + + def address(service, port): + return 'http://' + compose('port', service, str(port)) + + try: + compose('up', '-d') + loki, tempo, grafana = address('loki', 3100), address('tempo', 3200), address('grafana', 3000) + emitter, ignored = address('emitter', 8000), address('ignored', 8000) + wait_for('Loki readiness', lambda: 'ready' in request(loki + '/ready')) + wait_for('Tempo readiness', lambda: 'ready' in request(tempo + '/ready')) + wait_for('Grafana readiness', lambda: json.loads(request(grafana + '/api/health'))['database'] == 'ok') + wait_for('log emitters', lambda: request(emitter) == '' and request(ignored) == '') + passed('Local Loki, Tempo, Grafana and Docker log emitters ready') + + def query(expression): + qs = urllib.parse.urlencode({'query': expression, 'limit': '1000'}) + return json.loads(request(loki + '/loki/api/v1/query_range?' + qs))['data']['result'] + + selector = '{app="beach-complex",env="dev",service="backend",host="' + PROJECT + '"}' + + def entries(marker): + return [v for stream in query(selector + ' |= "' + marker + '"') for v in stream['values']] + + trace_id, span_id = secrets.token_hex(16), secrets.token_hex(8) + record = {'message': 'pr4-first-' + PROJECT, 'traceId': trace_id, + 'spanId': span_id, 'requestId': 'synthetic-request', 'userId': 'synthetic-user', + 'notificationId': 'synthetic-notification', 'outboxEventId': 'synthetic-outbox'} + line = json.dumps(record, separators=(',', ':')) + request(emitter, line.encode()) + found = wait_for('Docker -> Alloy -> Loki', lambda: entries(record['message'])) + assert found[0][1].rstrip('\n') == line, 'Original app JSON was altered' + parsed = query(selector + ' | json | traceId="' + trace_id + '"') + assert parsed, 'traceId JSON filter did not work' + # Query results also contain structured metadata (e.g. detected_level). + # The series endpoint reports the actual indexed stream labels. + series_qs = urllib.parse.urlencode({'match[]': selector}) + series = json.loads(request(loki + '/loki/api/v1/series?' + series_qs))['data'] + assert series, 'No indexed series found' + for labels in series: + assert set(labels) == {'app', 'env', 'service', 'host'}, labels + request(ignored, json.dumps({'message': 'must-not-collect-' + PROJECT}).encode()) + # Send a backend barrier after the negative fixture, then let batching finish. + request(emitter, json.dumps({'message': 'barrier-' + PROJECT}).encode()) + wait_for('backend barrier', lambda: entries('barrier-' + PROJECT)) + time.sleep(6) + assert not entries('must-not-collect-' + PROJECT) + passed('Real Docker JSON is preserved; unrelated logs excluded; only four low-cardinality labels') + + now = time.time_ns() + span = {'resourceSpans': [{'resource': {'attributes': [ + {'key': 'service.name', 'value': {'stringValue': 'beach-complex'}}]}, + 'scopeSpans': [{'scope': {'name': 'pr4-local-smoke'}, 'spans': [{ + 'traceId': trace_id, 'spanId': span_id, 'name': 'synthetic-log-correlation', + 'kind': 2, 'startTimeUnixNano': str(now - 1000000), 'endTimeUnixNano': str(now), + }]}]}]} + request(address('tempo', 4318) + '/v1/traces', json.dumps(span).encode(), + {'Content-Type': 'application/json'}) + auth = {'Authorization': 'Basic ' + base64.b64encode(('admin:' + PASSWORD).encode()).decode()} + + def grafana_get(path): + return json.loads(request(grafana + path, headers=auth)) + + ds = grafana_get('/api/datasources/uid/loki') + field = ds['jsonData']['derivedFields'][0] + assert field['datasourceUid'] == 'tempo' + assert field['url'] == '${__value.raw}', field + assert re.search(field['matcherRegex'], line).group(1) == trace_id + assert re.search(field['matcherRegex'], '[traceId=' + trace_id + ',spanId=x]').group(1) == trace_id + tempo_ds = grafana_get('/api/datasources/uid/tempo') + reverse = tempo_ds['jsonData']['tracesToLogsV2'] + assert reverse['datasourceUid'] == 'loki' + assert '${__span.traceId}' in reverse['query'] + wait_for('trace through Grafana proxy', lambda: grafana_get('/api/datasources/proxy/uid/tempo/api/traces/' + trace_id)) + health = wait_for('Grafana Loki datasource', lambda: grafana_get('/api/datasources/uid/loki/health')) + assert health['status'] == 'OK', health + reverse_query = reverse['query'].replace('${__span.traceId}', trace_id) + proxy_query = urllib.parse.urlencode({'query': reverse_query}) + assert grafana_get('/api/datasources/proxy/uid/loki/loki/api/v1/query_range?' + proxy_query)['data']['result'] + passed('Grafana datasource health and provisioned log-to-trace / trace-to-log targets resolve') + + # Graceful stop persists offsets; logs produced while stopped must be read on restart. + compose('stop', 'alloy') + request(emitter, json.dumps({'message': 'while-stopped-' + PROJECT}).encode()) + compose('start', 'alloy') + wait_for('Alloy offset resume', lambda: entries('while-stopped-' + PROJECT)) + offsets = compose('exec', '-T', 'alloy', 'sh', '-c', + 'find /var/lib/alloy/docker-offsets -type f -size +0c') + assert offsets, 'No persistent file fingerprint/offset database' + assert len(entries(record['message'])) == len(found) + passed('Alloy persisted file fingerprints/offsets and resumed logs written during its stop') + + # #263: a 40 KB JSON record spans multiple Docker 16 KiB records. + emitter2 = address('emitter2', 8000) + wait_for('second backend', lambda: request(emitter2) == '') + long_lines = [] + for index, endpoint in enumerate([emitter, emitter + '/stderr', emitter2, emitter2 + '/stderr']): + payload = {'message': f'long-{index}-' + PROJECT, + 'padding': 'x' * 40000, 'traceId': secrets.token_hex(16)} + original = json.dumps(payload, separators=(',', ':')) + long_lines.append((endpoint, payload, original)) + request(endpoint + '/partial', original[:20000].encode()) + # Let the first fragment from every source reach Alloy before finishing. + time.sleep(1) + for endpoint, payload, original in long_lines: + request(endpoint, original[20000:].encode()) + for endpoint, payload, original in long_lines: + received = wait_for('complete long JSON', lambda: entries(payload['message'])) + assert len(received) == 1 and received[0][1] == original + '\n', [len(v[1]) for v in received] + assert query(selector + ' | json | traceId="' + payload['traceId'] + '"') + assert re.search(field['matcherRegex'], received[0][1]).group(1) == payload['traceId'] + for name in ['emitter', 'emitter2']: + container_id = compose('ps', '-q', name) + compose('exec', '-T', 'alloy', 'test', '-s', + f'/var/log/docker/{container_id}/{container_id}-json.log.1') + passed('40 KB JSON preserved during live rotation with interleaved stdout/stderr from two containers') + + # #263: all these records remain in Docker's five files while Alloy is stopped. + compose('stop', 'alloy') + rotation_marker = 'rotation-' + PROJECT + expected_rotation = [] + for number in range(110): + original = json.dumps({'message': rotation_marker, 'number': number, + 'padding': 'r' * 1000}, separators=(',', ':')) + expected_rotation.append(original + '\n') + request(emitter, original.encode()) + retained = compose('logs', '--no-log-prefix', 'emitter') + assert retained.count(rotation_marker) == 110, 'Fixture exceeded Docker retention' + # Force a long record across a rotation boundary while the reader is offline. + offline_long = json.dumps({'message': 'offline-long-' + PROJECT, + 'padding': 'z' * 90000, 'traceId': secrets.token_hex(16)}, + separators=(',', ':')) + request(emitter, offline_long.encode()) + assert compose('logs', '--no-log-prefix', 'emitter').count(rotation_marker) == 110 + compose('start', 'alloy') + recovered = wait_for('all 110 rotated records', + lambda: (rows if len(rows := entries(rotation_marker)) == 110 else None)) + assert sorted(v[1] for v in recovered) == sorted(expected_rotation) + complete = wait_for('long JSON across rotated files', lambda: entries('offline-long-' + PROJECT)) + assert len(complete) == 1 and complete[0][1] == offline_long + '\n', [len(v[1]) for v in complete] + # Recreate the collector using the same volume and check that completed files are not replayed. + compose('up', '-d', '--force-recreate', '--no-deps', 'alloy') + request(emitter, json.dumps({'message': 'rotation-resume-' + PROJECT}).encode()) + wait_for('resume after rotation', lambda: entries('rotation-resume-' + PROJECT)) + assert len(entries(rotation_marker)) == 110 + metrics = request(address('alloy', 12345) + '/metrics') + sent = sum(float(value) for value in re.findall( + r'^loki_write_sent_entries_total\{[^\n]*\} ([0-9.e+]+)$', metrics, re.MULTILINE)) + assert sent == 1, f'Restarted Alloy replayed previously read records: {sent} sends' + passed('All 110 retained rotated records and a 90 KB JSON spanning files recovered after Alloy restart') + + compose('up', '-d', '--force-recreate', '--no-deps', 'emitter') + emitter = address('emitter', 8000) + wait_for('recreated backend emitter', lambda: request(emitter) == '') + request(emitter, json.dumps({'message': 'new-container-' + PROJECT}).encode()) + wait_for('new Docker container ID discovery', lambda: entries('new-container-' + PROJECT)) + passed('Alloy discovers a recreated backend without a hardcoded container ID') + + compose('stop', 'loki') + request(emitter, json.dumps({'message': 'loki-outage-' + PROJECT}).encode()) + time.sleep(6) + compose('start', 'loki') + loki = address('loki', 3100) + wait_for('Loki outage recovery', lambda: entries('loki-outage-' + PROJECT)) + passed('Alloy retries delivery across a short Loki outage') + + compose('up', '-d', '--force-recreate', '--no-deps', 'loki') + loki = address('loki', 3100) + wait_for('recreated Loki readiness', lambda: 'ready' in request(loki + '/ready')) + wait_for('Loki persistent log data', lambda: entries(record['message'])) + config_text = request(loki + '/config') + assert re.search(r'retention_period: (72h|3d)', config_text), 'Dev retention was not applied' + assert 'retention_enabled: true' in config_text + passed('Loki data survives container recreation; compactor retention active at 72h') + REPORT.parent.mkdir(parents=True, exist_ok=True) + REPORT.write_text(json.dumps({'checked_at_utc': datetime.datetime.now(datetime.timezone.utc).isoformat(), + 'scope': 'isolated local Docker; synthetic app logs and OTLP trace; not EC2 deployment or browser UI', + 'checks': checks, 'images': {k: v['image'] for k, v in services.items()}, + 'production_deployed': False}, ensure_ascii=False, indent=2) + '\n') + print('Report: ' + str(REPORT), flush=True) + except Exception: + print(compose('logs', '--no-color', '--tail', '35', 'alloy', 'loki', 'grafana'), flush=True) + raise + finally: + compose('down', '--volumes', '--remove-orphans') + + +if __name__ == '__main__': + main() diff --git a/deploy/observability/terraform/environments/dev/tests/log-pipeline.tftest.hcl b/deploy/observability/terraform/environments/dev/tests/log-pipeline.tftest.hcl new file mode 100644 index 00000000..3b91d4b0 --- /dev/null +++ b/deploy/observability/terraform/environments/dev/tests/log-pipeline.tftest.hcl @@ -0,0 +1,76 @@ +mock_provider "aws" { + mock_data "aws_subnet" { + defaults = { + availability_zone = "us-east-1a" + vpc_id = "vpc-0123456789abcdef0" + } + } +} + +variables { + instance_name = "local-test" + project_name = "beach-complex" + app_server_security_group_id = "sg-0123456789abcdef0" + app_server_private_ip = "10.0.1.200" + subnet_id = "subnet-0123456789abcdef0" + ami_id = "ami-0123456789abcdef0" + instance_type = "t3.small" + volume_size_gb = 30 +} + +run "dev_logs" { + command = plan + module { + source = "../../modules/observability_ec2" + } + variables { + env = "dev" + } + assert { + condition = ( + aws_vpc_security_group_ingress_rule.loki.from_port == 3100 && + aws_vpc_security_group_ingress_rule.loki.to_port == 3100 && + aws_vpc_security_group_ingress_rule.loki.ip_protocol == "tcp" && + aws_vpc_security_group_ingress_rule.loki.referenced_security_group_id == var.app_server_security_group_id && + aws_vpc_security_group_ingress_rule.loki.cidr_ipv4 == null && + aws_vpc_security_group_ingress_rule.loki.cidr_ipv6 == null + ) + error_message = "Loki must accept TCP 3100 only from the app security group." + } + assert { + condition = local.loki_retention_period == "72h" && strcontains(local.cloud_init_plan_probe, "LOKI_RETENTION_PERIOD=72h") + error_message = "Dev bootstrap must set the ADR-014 three-day retention." + } + assert { + condition = local.cloud_init_plan_probe_bytes <= 16384 + error_message = "The new Loki asset must fit EC2 user data." + } +} + +run "staging_retention" { + command = plan + module { + source = "../../modules/observability_ec2" + } + variables { + env = "staging" + } + assert { + condition = strcontains(local.cloud_init_plan_probe, "LOKI_RETENTION_PERIOD=168h") + error_message = "Staging bootstrap must set seven-day retention." + } +} + +run "prod_retention" { + command = plan + module { + source = "../../modules/observability_ec2" + } + variables { + env = "prod" + } + assert { + condition = strcontains(local.cloud_init_plan_probe, "LOKI_RETENTION_PERIOD=336h") + error_message = "Prod bootstrap must set fourteen-day retention." + } +} diff --git a/deploy/observability/terraform/modules/observability_ec2/cloud-init.yml.tftpl b/deploy/observability/terraform/modules/observability_ec2/cloud-init.yml.tftpl index 0e67b7fa..677b83c8 100644 --- a/deploy/observability/terraform/modules/observability_ec2/cloud-init.yml.tftpl +++ b/deploy/observability/terraform/modules/observability_ec2/cloud-init.yml.tftpl @@ -50,6 +50,18 @@ write_files: encoding: gzip+base64 content: ${grafana_datasources_base64gzip} + - path: /run/beach-observability/loki.yaml + owner: root:root + permissions: "0644" + encoding: gzip+base64 + content: ${loki_config_base64gzip} + + - path: /run/beach-observability/observability.env + owner: root:root + permissions: "0644" + content: | + LOKI_RETENTION_PERIOD=${loki_retention_period} + - path: /etc/systemd/system/docker.service.d/observability-mount.conf owner: root:root permissions: "0644" @@ -217,14 +229,18 @@ write_files: install -d -m 0755 \ "$compose_directory/prometheus" \ "$compose_directory/tempo" \ + "$compose_directory/loki" \ "$compose_directory/grafana/provisioning/datasources" install -d -m 0755 -o 65534 -g 65534 "$data_directory/prometheus" install -d -m 0755 -o 10001 -g 10001 "$data_directory/tempo" + install -d -m 0755 -o 10001 -g 10001 "$data_directory/loki" install -d -m 0755 -o 472 -g 0 "$data_directory/grafana" install_asset "$staging_directory/docker-compose.yml" "$compose_directory/docker-compose.yml" install_asset "$staging_directory/prometheus.yml" "$compose_directory/prometheus/prometheus.yml" install_asset "$staging_directory/tempo.yaml" "$compose_directory/tempo/tempo.yaml" + install_asset "$staging_directory/loki.yaml" "$compose_directory/loki/local-config.yaml" + install_asset "$staging_directory/observability.env" "$compose_directory/.env" install_asset "$staging_directory/grafana-datasources.yml" "$compose_directory/grafana/provisioning/datasources/datasources.yml" docker compose --project-directory "$compose_directory" config --quiet diff --git a/deploy/observability/terraform/modules/observability_ec2/main.tf b/deploy/observability/terraform/modules/observability_ec2/main.tf index b79899da..5ba2ab3c 100644 --- a/deploy/observability/terraform/modules/observability_ec2/main.tf +++ b/deploy/observability/terraform/modules/observability_ec2/main.tf @@ -25,6 +25,10 @@ locals { tempo_config_base64gzip = base64gzip( file("${path.module}/../../../compose/tempo/tempo.yaml") ) + loki_config_base64gzip = base64gzip( + file("${path.module}/../../../compose/loki/local-config.yaml") + ) + loki_retention_period = lookup({ dev = "72h", staging = "168h", prod = "336h" }, var.env) grafana_datasources_base64gzip = base64gzip( file("${path.module}/../../../compose/grafana/provisioning/datasources/datasources.yml") ) @@ -39,6 +43,8 @@ locals { compose_base64gzip = local.compose_base64gzip prometheus_config_base64gzip = local.prometheus_config_base64gzip tempo_config_base64gzip = local.tempo_config_base64gzip + loki_config_base64gzip = local.loki_config_base64gzip + loki_retention_period = local.loki_retention_period grafana_datasources_base64gzip = local.grafana_datasources_base64gzip }) @@ -52,6 +58,8 @@ locals { compose_base64gzip = local.compose_base64gzip prometheus_config_base64gzip = local.prometheus_config_base64gzip tempo_config_base64gzip = local.tempo_config_base64gzip + loki_config_base64gzip = local.loki_config_base64gzip + loki_retention_period = local.loki_retention_period grafana_datasources_base64gzip = local.grafana_datasources_base64gzip }) @@ -115,6 +123,15 @@ resource "aws_vpc_security_group_ingress_rule" "otlp_http" { to_port = 4318 } +resource "aws_vpc_security_group_ingress_rule" "loki" { + security_group_id = aws_security_group.this.id + referenced_security_group_id = var.app_server_security_group_id + description = "Loki log ingestion from the application server Alloy" + from_port = 3100 + ip_protocol = "tcp" + to_port = 3100 +} + resource "aws_vpc_security_group_egress_rule" "all" { security_group_id = aws_security_group.this.id description = "Outbound access for package installation and service communication" diff --git a/deploy/observability/verify-local.txt b/deploy/observability/verify-local.txt new file mode 100644 index 00000000..3d20d971 --- /dev/null +++ b/deploy/observability/verify-local.txt @@ -0,0 +1,21 @@ +로그 파이프라인 로컬 검증 + +준비 +- 저장소 루트에서 실행한다. +- Docker Engine + Compose v2 이상, Python 3, Terraform 1.15.8, cloud-init이 필요하다. +- AWS 자격증명이나 실제 AWS 리소스는 필요하지 않다. +- 기존 beach.logs=backend 컨테이너가 있으면 중지 상태여도 통합 테스트가 중단된다. + 이 경우 별도의 테스트용 Docker 데몬에서 실행한다. + +1. 설정 검사: Terraform mock plan 3건, cloud-init, Compose, Loki/Alloy +bash deploy/observability/scripts/validate-iac.sh + +2. 통합 테스트: Docker 로그 수집, 회전 복구, 긴 JSON 재조립, Grafana/Tempo 연결 +python3 deploy/observability/scripts/verify-log-pipeline.py + +3. 결과 확인 +cat build/reports/observability/log-pipeline.json + +통합 테스트가 만든 컨테이너와 볼륨은 종료 시 자동으로 정리한다. +서버 적용과 실제 앱 검증은 별도로 수행한다. +자세한 검증 범위와 적용 절차: deploy/observability/README.md