gal/constructionbot
Watch
2
0
Fork
You've already forked constructionbot
 0
Code Issues 8 Pull requests Projects Releases Packages Wiki Activity Actions

Repoint RFI extraction off sage to a large world-knowledge model (Synthetic syn-large-vision) — conduit-in-slab RFI misrouted to electrical #9

New issue
Open
opened 2026-08-21 20:46:54 +00:00 by gal-bot · 1 comment
gal-bot commented 2026-08-21 20:46:54 +00:00
Owner
Copy link

Problem

Live demo with Jake Marovich (2026-08-21, ~17:12–18:01 in the call recording) surfaced a
discipline-classification failure that a bigger world-knowledge model would not make.

The bot assigned an RFI to Brian Nguyen — Lead Electrical, ACE MEP Group. The RFI actually
asked how large a conduit pipe may run through the lower mat slab. That is a structural
question (ZFA Structural Engineers), not an electrical one.

Jake's diagnosis, verbatim from the transcript:

Julian: Why would it assign that? … Oh, probably cause conduit.
Jake: I bet electrical conduit and lower mat slab. Yeah. That confused it.
Jake: Cause the electrical engineer won't really care and imagine this RFI is more geared
toward how big of a conduit pipe we can have running through the slab.
Jake: That'd be a structural engineer question.

Root cause

The model keyed on the noun ("conduit" → electrical) and missed the governing relationship
("conduit penetrating a structural mat slab" → the structural engineer owns the penetration,
because it affects rebar, cover, and slab capacity). Correct routing requires construction-domain
world knowledge, not keyword association. sage (Qwen3.6-27B) does not reliably carry it.

This is not a prompt bug. Discipline routing is the single highest-consequence field the extractor
produces — a misrouted RFI goes to someone who "won't really care," burns response-time SLA, and in
a legally-material workflow that delay is exactly what immunizes a subcontractor.

Requested change

Repoint RFI extraction from sage to syn-large-vision on Synthetic (operator directive,
2026-08-21).

Affected code

File What
config.toml:13 sage_base_url = "https://sage.symbiotrip.com"
config.toml:20 sage_model = "sage"
src/config.rs:52-53,108 sage_model field + default_sage_model()
src/sage.rs client module; CHAT_COMPLETIONS_PATH, bearer auth via SAGE_API_KEY

The client already speaks OpenAI-compatible /v1/chat/completions, so the transport needs no
rework — this is an endpoint, model-id, and credential change plus verification.

Blocking constraint — do not skip

The fleet's Synthetic provider is configured against a tailnet address
(http://100.64.0.1:8011/v1). constructionbot runs in the gal tenant container and must never
route through the tailnet — tenant traffic reaches external model backends over public HTTPS
only. That is why sage_base_url is https://sage.symbiotrip.com today and not a 100.64.x.x
address.

Implementation must therefore resolve a public HTTPS endpoint for Synthetic before any config
change lands. Pointing at 100.64.0.1:8011 will fail closed in the container.

Open questions (resolve before implementing)

  1. Exact model id. syn-large-vision is not one of the four models the fleet's Synthetic
    provider currently lists. Confirm the upstream model identifier Synthetic serves it under.
  2. Public endpoint. What is Synthetic's public API base for this tenant, and does egress need
    declaring for the gal tenant?
  3. Credential. New key into gal OpenBao (suggest secret/gal/synthetic/credentials); the
    client reads SAGE_API_KEY today — see naming below.
  4. Vision. -vision implies image input. RFI attachments are PDFs handled today via
    pdftotext. Should the extractor start sending page images for markup-heavy RFIs? That is a
    larger change than a model swap and probably belongs in its own ticket.
  5. Cost/latency. Per-extraction cost and p95 latency vs. sage, which is flat-cost on-prem.

Acceptance criteria

  • The conduit-in-mat-slab RFI routes to the structural discipline, not electrical.
  • The case is added to tests/fixtures/calibration.yaml as a regression fixture with expected
    discipline_guess = structural.
  • Full calibration corpus re-run; no regression against sage's current baseline on the other
    Morella samples.
  • No tailnet address appears in any config path reachable from the tenant container.
  • scripts/no_hardcoded_endpoints.sh still passes.

Follow-on (separate ticket, not this one)

Once the backend is provider-neutral, the sage_* config keys and src/sage.rs module name become
actively misleading. Suggest renaming to llm_* / src/llm.rs with sage as one selectable backend.
Kept out of scope here to keep the repoint small and revertible.

Filed from the 2026-08-21 demo call. Full transcript in the gal workspace at
meetings/2026-08-21-constructionbot-demo-wishlist-roadmap-review-jake/.

## Problem Live demo with Jake Marovich (2026-08-21, ~17:12–18:01 in the call recording) surfaced a discipline-classification failure that a bigger world-knowledge model would not make. The bot assigned an RFI to **Brian Nguyen — Lead Electrical, ACE MEP Group**. The RFI actually asked how large a conduit pipe may run through the lower mat slab. That is a **structural** question (ZFA Structural Engineers), not an electrical one. Jake's diagnosis, verbatim from the transcript: > **Julian:** Why would it assign that? … Oh, probably cause conduit. > **Jake:** I bet electrical conduit and lower mat slab. Yeah. That confused it. > **Jake:** Cause the electrical engineer won't really care and imagine this RFI is more geared > toward how big of a conduit pipe we can have running through the slab. > **Jake:** That'd be a structural engineer question. ### Root cause The model keyed on the noun ("conduit" → electrical) and missed the governing relationship ("conduit *penetrating a structural mat slab*" → the structural engineer owns the penetration, because it affects rebar, cover, and slab capacity). Correct routing requires construction-domain world knowledge, not keyword association. `sage` (Qwen3.6-27B) does not reliably carry it. This is not a prompt bug. Discipline routing is the single highest-consequence field the extractor produces — a misrouted RFI goes to someone who "won't really care," burns response-time SLA, and in a legally-material workflow that delay is exactly what immunizes a subcontractor. ## Requested change Repoint RFI extraction from sage to **`syn-large-vision` on Synthetic** (operator directive, 2026-08-21). ## Affected code | File | What | |---|---| | `config.toml:13` | `sage_base_url = "https://sage.symbiotrip.com"` | | `config.toml:20` | `sage_model = "sage"` | | `src/config.rs:52-53,108` | `sage_model` field + `default_sage_model()` | | `src/sage.rs` | client module; `CHAT_COMPLETIONS_PATH`, bearer auth via `SAGE_API_KEY` | The client already speaks OpenAI-compatible `/v1/chat/completions`, so the transport needs no rework — this is an endpoint, model-id, and credential change plus verification. ## Blocking constraint — do not skip The fleet's Synthetic provider is configured against a **tailnet** address (`http://100.64.0.1:8011/v1`). **constructionbot runs in the gal tenant container and must never route through the tailnet** — tenant traffic reaches external model backends over public HTTPS only. That is why `sage_base_url` is `https://sage.symbiotrip.com` today and not a `100.64.x.x` address. Implementation must therefore resolve a **public HTTPS endpoint** for Synthetic before any config change lands. Pointing at `100.64.0.1:8011` will fail closed in the container. ## Open questions (resolve before implementing) 1. **Exact model id.** `syn-large-vision` is not one of the four models the fleet's Synthetic provider currently lists. Confirm the upstream model identifier Synthetic serves it under. 2. **Public endpoint.** What is Synthetic's public API base for this tenant, and does egress need declaring for the gal tenant? 3. **Credential.** New key into gal OpenBao (suggest `secret/gal/synthetic/credentials`); the client reads `SAGE_API_KEY` today — see naming below. 4. **Vision.** `-vision` implies image input. RFI attachments are PDFs handled today via `pdftotext`. Should the extractor start sending page images for markup-heavy RFIs? That is a larger change than a model swap and probably belongs in its own ticket. 5. **Cost/latency.** Per-extraction cost and p95 latency vs. sage, which is flat-cost on-prem. ## Acceptance criteria - [ ] The conduit-in-mat-slab RFI routes to the structural discipline, not electrical. - [ ] The case is added to `tests/fixtures/calibration.yaml` as a regression fixture with expected `discipline_guess = structural`. - [ ] Full calibration corpus re-run; no regression against sage's current baseline on the other Morella samples. - [ ] No tailnet address appears in any config path reachable from the tenant container. - [ ] `scripts/no_hardcoded_endpoints.sh` still passes. ## Follow-on (separate ticket, not this one) Once the backend is provider-neutral, the `sage_*` config keys and `src/sage.rs` module name become actively misleading. Suggest renaming to `llm_*` / `src/llm.rs` with sage as one selectable backend. Kept out of scope here to keep the repoint small and revertible. --- *Filed from the 2026-08-21 demo call. Full transcript in the gal workspace at `meetings/2026-08-21-constructionbot-demo-wishlist-roadmap-review-jake/`.*
gal-bot commented 2026-08-21 20:59:47 +00:00
Author
Owner
Copy link

Open questions 1–3 resolved — verified live, 2026-08-21

The blocking constraint clears. synthshim already provides exactly the public HTTPS front end
this needs (spec 100, symbiotrip/synthshim).

Question Answer How verified
1. Exact model id syn:large:vision GET /v1/models through the shim; 11 models served, the syn:* family exposes large:text, large:vision, small:text, small:vision
2. Public endpoint https://synthetic.symbiotrip.com/v1 ACME cert active; synthetic-shim.service active on sage; unauthenticated GET /v1/models returns 401 (auth enforced, service reachable)
3. Credential synthshim shared client bearer — interim Sourced from /run/openbao/synthetic-shim-bearer on sage; needs writing into gal OpenBao for the tenant container to fetch

Why this satisfies the tenancy constraint

synthetic.symbiotrip.com is a public HTTPS endpoint, structurally identical to how the tenant
already reaches https://sage.symbiotrip.com. No tailnet involvement. The shim listens on
127.0.0.1:8010 on sage with nginx/ACME in front; the tenant container talks to the public name.

The shim is OpenAI-compatible and proxies /v1/chat/completions and /v1/models upstream to
https://api.synthetic.new/openai/v1, selecting whichever pooled key has the most remaining quota
headroom per request. It applies no model allowlist — model ids pass through — so syn:large:vision
needs no shim-side change.

Net effect on this ticket: the change is sage_base_url → the shim, sage_model → syn:large:vision,
plus a bearer in gal OpenBao. The client code needs no transport rework.

Two things the implementer must carry, not discover

1. Interim spend lands on DeepNet's pooled quota. The shim pools symbiotrip-side Synthetic keys.
Until Gallaher has their own key, every RFI extraction draws down the shared pool. Operator direction
(2026-08-21): acceptable as an interim — "we will need to set them up with their own api key, but for
now they can use synthshim." Treat it as a subsidised bridge with an explicit end, not a resting state.

2. The shim has ONE shared client bearer, fleet-wide. shim/src/auth.rs compares the presented
token against a single configured value (check_bearer(presented, &Option<String>)) — there is no
per-client bearer. Placing that bearer inside the gal tenant container means a compromise of that
container exposes access to the entire pooled key set, not just Gallaher's slice. Note that
constructionbot currently sends no auth header to sage at all, so this introduces a new secret with
fleet-wide blast radius into a client tenant.

Not a blocker — the operator has accepted the interim — but it should be a recorded decision rather
than a side effect. Bound it:

  • Bearer stored in gal OpenBao (suggest secret/gal/synthetic/credentials, field value), never in
    config.toml, git, or logs
  • Rotate the fleet bearer when Gallaher migrates off it, and on gal offboarding
  • Track the migration to a Gallaher-owned Synthetic key as a follow-on ticket with a date, not "someday"

A per-client bearer in synthshim would remove the tradeoff entirely, but that is a change in
symbiotrip/synthshim, out of scope for this repo.

Remaining open questions (4 and 5 stand)

  1. Vision. syn:large:vision accepts image input. RFI attachments go through pdftotext today.
    Sending page images for markup-heavy RFIs is plausibly a real accuracy win — Jake's misrouted RFI
    involved a marked-up drawing — but it is a bigger change than a model swap. Keep it a separate ticket.
  2. Cost and latency. Unmeasured against sage's flat-cost on-prem baseline. Capture p95 latency and
    per-extraction cost during the calibration re-run required by the acceptance criteria.

Reference

  • Runbook (adding pool keys): symbiotrip/stp-infra → docs/runbook-add-synthshim-key.md
  • Module: platform/synthetic-shim.nix · Shim source: symbiotrip/synthshim · Spec 100
## Open questions 1–3 resolved — verified live, 2026-08-21 The blocking constraint clears. **synthshim** already provides exactly the public HTTPS front end this needs (spec 100, `symbiotrip/synthshim`). | Question | Answer | How verified | |---|---|---| | **1. Exact model id** | **`syn:large:vision`** | `GET /v1/models` through the shim; 11 models served, the `syn:*` family exposes `large:text`, `large:vision`, `small:text`, `small:vision` | | **2. Public endpoint** | **`https://synthetic.symbiotrip.com/v1`** | ACME cert active; `synthetic-shim.service` active on sage; unauthenticated `GET /v1/models` returns **401** (auth enforced, service reachable) | | **3. Credential** | synthshim **shared client bearer** — interim | Sourced from `/run/openbao/synthetic-shim-bearer` on sage; needs writing into gal OpenBao for the tenant container to fetch | ### Why this satisfies the tenancy constraint `synthetic.symbiotrip.com` is a public HTTPS endpoint, structurally identical to how the tenant already reaches `https://sage.symbiotrip.com`. **No tailnet involvement.** The shim listens on `127.0.0.1:8010` on sage with nginx/ACME in front; the tenant container talks to the public name. The shim is OpenAI-compatible and proxies `/v1/chat/completions` and `/v1/models` upstream to `https://api.synthetic.new/openai/v1`, selecting whichever pooled key has the most remaining quota headroom per request. It applies **no model allowlist** — model ids pass through — so `syn:large:vision` needs no shim-side change. Net effect on this ticket: the change is `sage_base_url` → the shim, `sage_model` → `syn:large:vision`, plus a bearer in gal OpenBao. The client code needs no transport rework. ### Two things the implementer must carry, not discover **1. Interim spend lands on DeepNet's pooled quota.** The shim pools symbiotrip-side Synthetic keys. Until Gallaher has their own key, every RFI extraction draws down the shared pool. Operator direction (2026-08-21): acceptable as an interim — *"we will need to set them up with their own api key, but for now they can use synthshim."* Treat it as a subsidised bridge with an explicit end, not a resting state. **2. The shim has ONE shared client bearer, fleet-wide.** `shim/src/auth.rs` compares the presented token against a single configured value (`check_bearer(presented, &Option<String>)`) — there is no per-client bearer. Placing that bearer inside the gal tenant container means a compromise of that container exposes access to the **entire pooled key set**, not just Gallaher's slice. Note that constructionbot currently sends *no* auth header to sage at all, so this introduces a new secret with fleet-wide blast radius into a client tenant. Not a blocker — the operator has accepted the interim — but it should be a recorded decision rather than a side effect. Bound it: - [ ] Bearer stored in gal OpenBao (suggest `secret/gal/synthetic/credentials`, field `value`), never in config.toml, git, or logs - [ ] Rotate the fleet bearer when Gallaher migrates off it, and on gal offboarding - [ ] Track the migration to a Gallaher-owned Synthetic key as a follow-on ticket with a date, not "someday" A per-client bearer in synthshim would remove the tradeoff entirely, but that is a change in `symbiotrip/synthshim`, out of scope for this repo. ### Remaining open questions (4 and 5 stand) 4. **Vision.** `syn:large:vision` accepts image input. RFI attachments go through `pdftotext` today. Sending page images for markup-heavy RFIs is plausibly a real accuracy win — Jake's misrouted RFI involved a marked-up drawing — but it is a bigger change than a model swap. Keep it a separate ticket. 5. **Cost and latency.** Unmeasured against sage's flat-cost on-prem baseline. Capture p95 latency and per-extraction cost during the calibration re-run required by the acceptance criteria. ### Reference - Runbook (adding pool keys): `symbiotrip/stp-infra` → `docs/runbook-add-synthshim-key.md` - Module: `platform/synthetic-shim.nix` · Shim source: `symbiotrip/synthshim` · Spec 100
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
gal/constructionbot#9
No description provided.