Findings and recommended improvements

While building the example site, I encountered four issues that are not covered in Thally’s documentation. Each finding includes steps to reproduce the issue, what happened, and a recommended resolution.

1. Invalid MDX can pass CI checks

Steps to reproduce: Create src/content/_broken.mdx with valid frontmatter and an unclosed tag. Run npm run check, followed by npm run runtime-sources:build.

Observed: The check reports two warnings (orphan page and short body) but exits successfully. The build then fails with an MDX compilation error identifying the unclosed tag.

The problem is that CI runs only thally check --ci when changes are limited to src/content/. As a result, a pull request can pass CI even though the page will fail to build later. Although the check already parses MDX on pages containing , it does not report the parsing errors.

Recommended resolution: Make thally check report MDX parsing errors, or add npm run runtime-sources:build to the content-only CI workflow.

2. Local and CI checks use different CLI versions

Steps to reproduce: Compare the check script in package.json with the version specified in .github/thally-tooling/package.json. Then run npm view @thallylabs/cli version to check the latest published version.

Observed: On October 24, 2026, the local check used @thallylabs/cli@latest (0.8.124), while CI used the pinned version 0.8.120. Different versions may produce different results, making local checks unreliable as a predictor of CI results.

Recommended resolution: Configure the local check script to use the same pinned CLI version as CI:

json "check": "npm ci --ignore-scripts --prefix .github/thally-tooling && .github/thally-tooling/node_modules/.bin/thally check"

This keeps local and CI checks aligned, with the weekly framework update responsible for bumping the shared version.

3. Unknown frontmatter fields pass validation silently

Steps to reproduce: Add sidebarTitle: Wrong label to a documentation page's frontmatter and run npm run check.

Observed: The check reports no issues, but the sidebar label does not change. The supported field is navTitle; sidebarTitle is used by another documentation tool and is already mapped by the migration package. The checker does not currently validate frontmatter keys against a list of supported fields.

Recommended resolution: Warn when a page uses an unsupported frontmatter key. Reuse the migration package's existing field mapping to identify likely mistakes and suggest the correct field.

4. The readiness score is not fully documented, and the starter does not score perfectly

Steps to reproduce: Create a site from the starter template and open the agent-readiness report. Search the docs.json reference for readiness settings, then add readiness.goldenQuestions and run the check again.

Observed: The starter site scores 89/100 (grade B), with openapi_examples failing. However, the docs.json reference does not document the readiness configuration, even though the starter includes it. The retrieval_eval check also carries the largest weight in the readiness score.

During testing, one golden question initially failed because generated /rates operation pages ranked above the authored page for questions containing the word “rate”.

Recommended resolution:

Include successful response examples for every operation in the starter OpenAPI specification. Document readiness.topK and readiness.goldenQuestions in the docs.json reference. Improve search ranking so authored documentation pages can rank above generated operation pages when they better answer the reader's question.