Find similar failures

When a job fails, its error text is embedded and stored. similar_failures then answers “have we seen this before?” across every case on the server.

Turn it on

It needs the instrumentation on Postgres with pgvector, and an OpenAI-compatible embeddings endpoint:

MCOPENFOAM_INSTRUMENTATION_URL=postgresql+asyncpg://user:pass@db:5432/mcopenfoam
MCOPENFOAM_EMBEDDINGS_URL=https://<your-gateway>/v1
MCOPENFOAM_EMBEDDINGS_API_KEY=<token>

The compose stack in deploy/ sets the first for you. server_info reports instrumentation: "postgres + embeddings" when both are on. Failures are embedded from then on; earlier ones are not.

Ask

By job, which uses that job’s own error:

similar_failures({"job_id": "flight-e-10-a6.7-b0-e421fd1e", "k": 5})

Or by text, such as an error you saw elsewhere:

similar_failures({"text": "FOAM FATAL ERROR: Maximum number of iterations exceeded"})

Each match has the job id, the event, a score from 0 to 1 and the text that matched. A job’s own failure is left out of its matches.

What gets matched

A failure’s text starts at the first fatal line in the last 40 lines of the log, such as --> FOAM FATAL ERROR, and runs for 12 lines. Starting there matters: the lines above a fatal error are usually solver residuals, and when they were included the scores followed the residuals instead of the error. In a live check, two jobs with the same solver failure scored 0.986 against each other, and a mesh failure scored 0.761.