Essay 04 · Companion to a forthcoming chapter

When Should a Good Answer Fail?

A persuasive answer can still fail a consequential requirement. The harder task is to justify the rule, examine the judgment and make the finding matter.

Essay and CDFI evaluation methodology by Mark Julius Banasihan

SAICRED originator and project lead: Filip Ponulak, PhD

SAICRED lead engineer: Naveen Kumar Puppala

CDFI first repository record ·

Forthcoming chapter · SAICRED

Can You Trust AI with Your Faith?

Introducing SAICRED: A Standard for Assessing AI for Catholic Reliability and Doctrinal Fidelity

Visit the publisher’s book page

This editorial essay examines CDFI’s evaluation methodology. The forthcoming SAICRED chapter is a separate publication; its publication date and chapter number remain unconfirmed.

In the volume
AI and the Catholic Perspective
Edited by
Dr. Jane F. Adolphe
Dr. Łukasz Mirosław
Publisher
En Route Books and Media
Author sign-off

Filip Ponulak, PhD — Project Lead
SAICRED project
Evolving

A teacher preparing a lesson on Catholic teaching receives an AI-generated explanation. It is clear, considerate and includes a quotation attributed to a Church document. Before using it, the teacher opens the cited source and cannot find the quoted words.

The teacher now needs to establish what happened. The reference may be incomplete, the quotation may come from another document, or the answer may have invented it. Until that question is resolved, the passage cannot provide the support the answer claims.

This situation is hypothetical. It illustrates a problem addressed by the Catholic Doctrinal Fidelity Index: how should an evaluation treat an answer that reads well but may fail a requirement essential to its intended use?

The distinction matters to the people who will encounter the lesson. Clear writing helps them understand an explanation. It cannot establish that the explanation represents its sources accurately.

Good for what?

The Catholic Doctrinal Fidelity Index approaches that question in a specific setting: evaluating how AI responses represent Catholic teaching. The CDFI Framework supplies the evaluation governance layer of SAICRED, the wider benchmarking project. It translates the project’s evaluation criteria into scoring rules, authority-sensitive weights, failure gates and procedures for reviewing the automated judge. Its subject matter requires attention to the authority of a teaching, the distinction between settled and open questions, and the sources an answer invokes. Those requirements make the purpose of an evaluation unusually visible.

A comparative account of religious views and an explanation prepared for Catholic instruction can be accurate in different ways. A response that appropriately describes disagreement in the first setting may fail to answer the question asked in the second. Before evaluating either, a reviewer needs the actual request, its audience and the intended use. The institution must say which task it is assessing, rather than allow the score to imply a universal judgment about the answer.

This distinction matters beyond religious education. Whenever an institution evaluates AI for a particular purpose, it chooses which obligations the answer must satisfy. Those choices deserve examination alongside the output. A demanding standard can protect the intended audience; a badly specified standard can penalize a response for doing something it was never clearly instructed not to do.

What an average cannot forgive

CDFI separates graded qualities from failures that trigger a limit on the final score. It calls such a specified failure condition a gate. Its formula specification describes a hallucination gate for invented ecclesiastical documents or language falsely attributed to a real source. An incomplete reference is treated separately. That distinction matters: evidence of an imperfect citation is not automatically evidence of fabrication.

The second gate concerns how a response represents teaching that the relevant Catholic authority regards as settled. The framework calls it relativism resistance. Applying this rule requires a defensible classification of the question and its context. A reviewer must be able to distinguish an inaccurate representation of a tradition from an appropriate explanation of differences between traditions. The rule cannot settle that interpretive work simply by existing.

Under the stated gate rule, a failing response receives the lower of its raw score and 40. A hypothetical raw score of 86 becomes 40; a raw score of 32 remains 32. If neither gate fails, the raw score remains unchanged. These are arithmetic illustrations, not observed model results. They show why the failure status must be retained separately: a low final score alone does not reveal whether a gate fired.

The architecture expresses a substantive choice. Some defects should remain visible even when an answer performs well on other dimensions. But choosing 40 as the cap does not establish that 40 is the right boundary for institutional use. That number is a framework rule requiring justification and evaluation. A correct implementation of the calculation would establish how the rule operates, not whether the resulting recommendation is sound.

Nor does capping one response establish that a model’s overall average will fall below a deployment threshold. Stronger responses can still raise an aggregate. A reader assessing suitability needs to see the kinds of failure, their frequency and the circumstances in which they occur, as well as the mean. The institution must decide what those failures imply for the particular use it is considering.

Illustrating the cap rule

The score.
The condition.

  1. 86 → 86

    Neither gate fails. The raw score remains.

  2. 86 → 40

    A gate fails. The cap limits the final score.

  3. 32 → 32

    A gate fails. The cap never raises a lower score.

  4. 32 → 32

    Neither gate fails. The same final score can occur without a gate failure.

Keep the failure status alongside the score: a final score alone cannot tell you whether a gate failed. Invented inputs illustrate final = min(raw, 40) when either gate fails. These are not model results, and they do not establish that the threshold is valid for deployment.

Beyond Catholic institutions

CDFI was developed with Catholic institutions in mind, within the SAICRED benchmarking work. Its criteria address a particular responsibility: representing Catholic teaching accurately, with appropriate attention to sources and authority.

Other institutions face a related evaluation question: which defects make an answer unsuitable for its intended use, even when much of it is useful? A school might require that quoted sources be authentic. A company might require that instructions accurately reflect its approved procedures. Clear writing would not resolve a fabricated quotation or an instruction that contradicts the governing procedure.

Applying this approach elsewhere would require each institution to define its own requirements, identify who is qualified to judge them and test whether reviewers can apply them consistently. CDFI’s theological categories, weights and numerical cap cannot be assumed appropriate for another setting.

The broader question is how an institution makes an essential requirement count in practice. It needs a defensible rule, evidence supporting the judgment and someone authorized to act when the requirement is not met. Whether an adapted evaluation achieves that would require separate testing.

The rule must survive inspection

The evaluation method must accept the scrutiny it asks of the answer. The repository’s published scope and open questions preserve CDFI’s provisional status, with scoring-contract reconciliation still open. In the inspected version, the formula, configuration and calculator do not supply one consistent account of how every score is produced. The technical note below preserves the specific differences.

A scoring contract would make those choices explicit: the input scale, applicable weights, gate definitions, treatment of missing values and version of the calculation. A reproducible evidence bundle would then connect that contract to the prompts, responses and judgments behind a result. Until those connections are established, a precise-looking number can conceal an unresolved choice about what was measured.

Who can question the judge?

The calculation is only one part of an evaluation. Someone, or another model, must decide whether a response misstates a teaching, invents a source or expresses an appropriate degree of certainty. If that judgment is wrong, perfectly consistent arithmetic carries the error into the final score.

The limitations register reports that the SAICRED v2 metric scores were produced by Gemini 2.5 Flash without validation against human theological expert judgment. It also states that all prompts used one default authority category while classification by theological advisers remained pending. These are limits on what the reported scores can establish about the intended domain, not details that disappear after averaging.

The same register reports automated judge checks, but consistency and domain correctness are distinct questions. A judge may repeatedly apply the same mistaken interpretation. Expert review needs access to the original prompt, full response, source passages, rubric and automated rationale. Reviewers also need sufficient time, relevant expertise and a way to record disagreement without being required to endorse the automated result.

Authority enters twice here. Qualified people must be able to challenge an individual judgment, and those responsible for the evaluation must be able to revise the rule when the challenge exposes a recurring defect. A reviewer who can leave a comment but cannot trigger reconsideration has supplied information without necessarily affecting the outcome.

For the opening example, the inquiry should preserve what the reviewer found: which cited passage could not be located, what sources were checked, whether the attribution was corrected and whether the revised answer was reassessed. If the dispute concerns the rubric itself, the record should retain that disagreement. Treating every objection as a scoring mistake would close off the possibility that the standard needs revision.

A result needs its boundaries

Another disclosed limit concerns stability. The source reports that stability scores were fixed at 3.0 while repeated-response measurement was deferred. A constant inserted into a formula cannot establish how consistently a model answers. Readers need to know which components were observed, which were assumed and which remain unmeasured before interpreting the composite.

The preserved website review also states that the complete response and scoring files were unavailable in the supplied archive, and that the reported results were not independently reproduced. This essay therefore does not endorse a model ranking or a deployment tier. It examines the proposed architecture and the unresolved conditions for interpreting its results.

Research on statistical evaluation provides a further reason to examine headline comparisons. Scores depend on the sampled questions and variation in responses; related questions can provide less independent information than their count suggests. Uncertainty around a comparison matters when deciding whether a reported difference warrants a conclusion. These statistical principles do not validate CDFI’s particular weights, cap or use thresholds.

A method can draw on published research while making additional choices that the cited research did not test. Those choices should remain identifiable. Linking a paper makes the relationship inspectable; it does not turn a design judgment into an established finding. The same discipline applies to this essay’s links: the reader should be able to see both the source of a claim and the point at which the argument goes beyond it.

The responsibilities behind a failed answer

For leaders commissioning an evaluation, the first responsibility is to state the intended use and the requirements an answer must satisfy. Qualified reviewers need a shared basis for judging those requirements and a way to record uncertainty. A disputed quotation should trigger investigation before it is classified as fabricated.

The institution also needs to decide what follows from a failure. A numerical cap does not withhold a lesson, notify an editor or prevent an answer from appearing in a product. Someone must have authority to hold its use, arrange review and decide whether a correction is sufficient for release.

Disagreement needs a route to resolution. Reviewers should preserve the answer, the relevant sources and their reasons so another qualified person can examine the finding. If the problem lies in the evaluation rule, those responsible must be able to revise that rule and identify which earlier results need reconsideration.

These responsibilities would need to be tested in the institution’s own setting. Applying CDFI’s calculation correctly does not establish that its thresholds are suitable for deployment or that the required response occurred.

A failure that can change the outcome

A failed answer matters institutionally when the finding can affect what happens next. In the hypothetical lesson, a designated editor could hold publication, ask a qualified reviewer to examine the disputed source and require a corrected version before release. That arrangement would connect scrutiny to action. A red indicator in an evaluation report, on its own, would not establish that the lesson was withheld.

If the answer has already reached readers, correction requires further work. The institution needs to identify the published version, replace or withdraw it where appropriate and determine who should be informed. A record of the original finding, the authorized change and its delivery would show how far the correction reached. Any copies or downstream uses that remain unaddressed should stay visible as unresolved consequences.

These are proposed conditions for meaningful control, not outcomes demonstrated by the CDFI reference implementation. Their value is that they make the institutional question concrete. Who may stop the answer from being used? What evidence do they need? Where can they change the result? What would show that their judgment reached the people who otherwise would have relied on it?

The teacher’s disputed quotation may ultimately be corrected, rejected or found to have been misunderstood. Careful evaluation must leave those possibilities open. A good answer should fail when a defensible requirement for its intended use has not been met, and the finding should itself remain open to challenge. What matters is whether the institution can explain the requirement, examine the evidence and carry the resulting correction into practice.

AI use & disclosure

Disclosure updated November 3, 2026

CDFI’s disclosure records Claude (Anthropic) assisting with methodology development, drafting, scoring architecture and repository construction during March–May 2026. Mark Julius Banasihan remains responsible for the methodology he adopted. SAICRED’s Gemini 2.5 Flash scoring judge is a separate system component; project leadership, engineering and theological authority retain their distinct credits. Reported technical reliability checks do not establish human theological validation, which remains open alongside scoring-contract reconciliation. The inspected record does not provide a complete task-level AI-use log or independently establish completion of every human review.

Codex (OpenAI) also assisted this website essay’s drafting, revision and source-record inspection. That editorial work does not establish independent human verification of every underlying source.

Project AI-use and verification record ↗

Program AI-use statement

Technical evidence and version boundaries

These observations concern the inspected CDFI snapshot. They do not establish the status of subsequent releases.

A concrete difference appears in the scale. The formula document describes inputs on a zero-to-ten scale, while the reference calculator describes zero-to-five inputs and multiplies the weighted sum by twenty. The weights also differ. For the ordinary-magisterium category, the formula document assigns doctrinal precision a weight of 0.25; the configuration and calculator use 0.26. Other weights diverge too.

These differences are not resolved by choosing whichever file is easiest to run. A developer needs to know which specification governs, which inputs a published result used and whether the result can be reproduced from that combination. Otherwise, two people can follow different public materials conscientiously and obtain scores whose apparent comparability hides a different calculation.

The repository’s website-alignment statement preserves this boundary. It describes the source as reference implementation v1.5 and says that repository changes do not automatically close the website’s open questions. Making materials public allows a reader to find disagreements. Resolving those disagreements requires a separate, recorded review.