- Problem
- Hand-written JSON Schema and OpenAPI contracts drift from the Pydantic models that actually define the data.
- Decision
- Generate the JSON Schema and OpenAPI contracts from the shared Pydantic models instead of maintaining them by hand.
Distributed health-data platform
CnesData
A contract-led platform for moving municipal source extracts from edge environments into a tenant-isolated central data layer.
- Python
- FastAPI
- Go
- PostgreSQL
- Parquet
- Docker
- React
- Problem
- Municipal source systems produce operational data at the edge, while reconciliation and downstream analysis need stable contracts, source provenance, and an explicit boundary between tenants.
- Contribution
- Built the edge-to-central path: Go extraction agent, manifest registration, central API and dashboard on shared generated contracts.
- Key decision
- Generate contracts from source models and keep tenant context at the storage boundary, not only in application filters.
- Repository
- CnesData public repository ↗
01 / Context
Context & contribution
Problem
Municipal source systems produce operational data at the edge, while reconciliation and downstream analysis need stable contracts, source provenance, and an explicit boundary between tenants.
Context
CnesData is developed as a public software platform. The repository demonstrates the application boundaries and local development path; this case study does not claim a municipal rollout, production scale, or processing results.
Contribution
- Defined canonical domain contracts and generated schemas shared by clients and services.
- Built the Go edge-agent path for source extraction, compressed Parquet files, and manifest registration.
- Implemented central API workflows for orchestration, device activation, access, and dashboard reads.
- Kept tenant context explicit across authentication, persistence, and row-level isolation.
- Developed the web dashboard around typed API contracts and operational state.
02 / Technical choices
Engineering decisions
- Problem
- Multi-source ingestion must distinguish a complete extraction from a partial or inconsistent set of uploaded files.
- Decision
- Give every file a raw manifest (source, competência, snapshot mode, SHA-256) and register the full set as one extraction unit.
- Problem
- A shared multi-tenant database cannot depend on every query remembering to filter by tenant.
- Decision
- Set the tenant as a transaction-local PostgreSQL setting and enforce it with row-level security policies.
03 / Current state
Current state
Outcomes
- The public implementation provides a coherent path from edge extraction to tenant-scoped landing records.
- Shared contracts make incomplete and planned portions visible instead of hiding them behind broad platform claims.
04 / Public record
Public evidence
CnesData public repository
Source code, contracts, application boundaries, tests, and current-status documentation.
Open public evidence ↗Architecture documentation
Public system model and service-boundary reference.
Open public evidence ↗05 / Optional walkthrough
A paused, synthetic lifecycle of the system. Press Play or step through the chapters to explore states, failures and limits.
Systems Atlas / LifecycleSynthetic · documented / planned target
01 / 06
See the starting state
CURRENT aponta para v0 enquanto uma nova competência é preparada.
Record
See the starting state
Source captures
CURRENT: v-demo-00. Candidate: none. User reads: none. Run: PLANNED.
Paused · operation 1 / 19
Alvo aprovado em develop; etapas centrais não são afirmadas como release concluído. Perfis local/AWS preservam semântica; fixture usa labels AWS. Sem dados reais, FTP, credenciais ou engine analítica carregada no navegador.
Read the complete lifecycle transcript
Calculated from the engine during the build. Available without JavaScript.
Da extração ao dataset publicado.
Versão anterior permanece ativa
CURRENT aponta para v0 enquanto uma nova competência é preparada.
CURRENT: v-demo-00. Candidate: none. User reads: none. Run: PLANNED.
Solicitar extração
Trabalho municipal e captura nacional são coordenados; ainda sem novos dados.
CURRENT: v-demo-00. Candidate: none. User reads: none. Run: WAITING INPUTS.
Claim autenticado
Gateway falso valida mTLS e o Job concede lease/fence. JSON não concede identidade.
CURRENT: v-demo-00. Candidate: none. User reads: none. Run: WAITING INPUTS.
Ler fonte municipal
Três registros fictícios mantêm a proveniência; Edge não faz reconciliação.
CURRENT: v-demo-00. Candidate: none. User reads: none. Run: WAITING INPUTS.
Source captures
Record Municipal h National h Comparison EST-A "040 " pending pending EST-B "020" pending pending EST-C "030" pending pending CNES_LOCAL + CNES_NATIONAL · 2026-07 · run-demo-01
Preparar Parquet
Encoding e tipos de transporte viram data.parquet com hash; ainda no Edge.
CURRENT: v-demo-00. Candidate: none. User reads: none. Run: WAITING INPUTS.
Publicar bytes raw
Primeiro PUT aceita os bytes imutáveis; isso não publica o dataset.
CURRENT: v-demo-00. Candidate: none. User reads: none. Run: WAITING INPUTS.
Replay seguro
Mesmo objeto e mesmos bytes: nenhuma cópia persistida a mais.
CURRENT: v-demo-00. Candidate: none. User reads: none. Run: WAITING INPUTS.
Aceitar manifesto local
Objeto e manifesto irmãos ficam rastreáveis; Job conclui; Run ainda espera a outra fonte.
CURRENT: v-demo-00. Candidate: none. User reads: none. Run: WAITING INPUTS.
Capturar fonte nacional
Adapter central valida PF mensal e competência antes do filtro municipal.
CURRENT: v-demo-00. Candidate: none. User reads: none. Run: WAITING INPUTS.
Source captures
Record Municipal h National h Comparison EST-A "040 " "20" pending EST-B "020" "20" pending EST-C "030" "30" pending CNES_LOCAL + CNES_NATIONAL · 2026-07 · run-demo-01
Aceitar raw nacional
A segunda origem converge no mesmo contrato raw, não no Edge municipal.
CURRENT: v-demo-00. Candidate: none. User reads: none. Run: WAITING INPUTS.
Iniciar processamento central
Inputs completos habilitam Run e unidades de processamento.
CURRENT: v-demo-00. Candidate: none. User reads: none. Run: PROCESSING.
Normalizar por fonte
Tipos/chaves são padronizados, mantendo origem e competência em cada artefato.
CURRENT: v-demo-00. Candidate: none. User reads: none. Run: PROCESSING.
Normalized records
Record Municipal h National h Comparison EST-A 40 20 pending EST-B 20 20 pending EST-C 30 30 pending CNES_LOCAL + CNES_NATIONAL · 2026-07 · run-demo-01
Reconciliar
Comparar fontes e registrar divergência ilustrativa; raw permanece intacto.
CURRENT: v-demo-00. Candidate: none. User reads: none. Run: PROCESSING.
Reconciliation · 3 comparisons
Record Municipal h National h Comparison EST-A 40 20 Different EST-B 20 20 Equal EST-C 30 30 Equal CNES_LOCAL + CNES_NATIONAL · 2026-07 · run-demo-01
Materializar a tela
JSON pequeno é derivado do mesmo run; browser não acessa raw.
CURRENT: v-demo-00. Candidate: none. User reads: none. Run: PROCESSING.
Reconciliation · 3 comparisons
Record Municipal h National h Comparison EST-A 40 20 Different EST-B 20 20 Equal EST-C 30 30 Equal CNES_LOCAL + CNES_NATIONAL · 2026-07 · run-demo-01
Verificar completude
Checar hashes, schemas e leitura dos artefatos antes de publicar.
CURRENT: v-demo-00. Candidate: none. User reads: none. Run: PUBLISHING.
Criar versão imutável
v1 existe, mas CURRENT ainda aponta para a versão anterior.
CURRENT: v-demo-00. Candidate: v-demo-01. User reads: none. Run: PUBLISHING.
Publicar por CAS
Somente com expectedVersion/fence válidos CURRENT muda de v0 para v1.
CURRENT: v-demo-01. Candidate: v-demo-01. User reads: none. Run: PUBLISHED.
Resolver acesso autorizado
Membership e tenant válidos; o request fixa uma única versão para toda a resposta.
CURRENT: v-demo-01. Candidate: v-demo-01. User reads: v-demo-01. Run: PUBLISHED.
Resultado pronto para uso
O usuário vê o JSON da versão publicada, com proveniência e run identificados.
CURRENT: v-demo-01. Candidate: v-demo-01. User reads: v-demo-01. Run: PUBLISHED.
Reconciliation · 3 comparisons
Record Municipal h National h Comparison EST-A 40 20 Different EST-B 20 20 Equal EST-C 30 30 Equal CNES_LOCAL + CNES_NATIONAL · 2026-07 · run-demo-01
Current state and operation
{
"type": "INIT"
}06 / Limits & details
Limitations & details
Limitations
- Full Parquet-to-Gold processing remains planned and is not described as complete.
- Kubernetes remains a target deployment model; this page makes no production availability or scale claim.
- No municipal records, health data, or quantitative operational results are published here.
Reliability
- Contract and OpenAPI artifacts can be regenerated from source models and checked for drift.
- Landing work has explicit states and ownership boundaries so incomplete processing is visible.
- Local development composes the API, object storage, identity provider, database, agent, and dashboard without implying production deployment.
- Keep audit-rule execution outside this repository and expose stable data contracts for downstream consumers.
System View
Architecture and component details
Conceptual architecture map for public discussion. Connections describe system responsibilities rather than a live deployment; processing and deployment boundaries remain illustrative.
System map
Central API is the branching hub between contracts, clients, and downstream boundaries.
Extraction payloads
API contracts
Registers manifests
Tenant context
Typed API
Delivery work
Configuration
Component details
Shared models and exported schemas establish one vocabulary for landing work, clients, and downstream services.
Sends to
- Defines extraction payloads
- Defines API contracts
Edge extraction writes compressed Parquet artifacts and registers multi-file manifests with the central platform.
Receives from
- Defines extraction payloads
Sends to
- Registers extraction manifests
A FastAPI service coordinates activation, extraction requests, manifest registration, and landing metadata.
Receives from
- Defines API contracts
- Registers extraction manifests
- Uses typed API contracts
- Shows the deployment shape
Sends to
- Carries tenant context
- Creates a processing-work boundary
Tenant context is carried through authorization and PostgreSQL row-level security boundaries.
Receives from
- Carries tenant context
The typed dashboard presents activation, agent state, access, and overview workflows.
Sends to
- Uses typed API contracts
Worker foundations exist, but the full landing-to-Gold processing path is not presented as complete.
Receives from
- Creates a processing-work boundary
Kubernetes is a target deployment shape rather than a published production claim.
Sends to
- Shows the deployment shape