Home 01 · Vector DB 02 · LLM Frameworks 03 · AI Systems 04 · Prompt & Tuning 05 · AI Testing 06 · AI Architect 07 · AI Integration

Why AI testing is different from traditional software testing

Traditional software testing has deterministic pass/fail criteria: the function returns the right value or it doesn't. LLM systems are probabilistic — the same input can produce different outputs across runs, "correct" is often a matter of degree rather than binary, and failures are frequently silent (the system returns something, just not something accurate or useful). This demands a different testing philosophy and a different toolset.

Unit
Single prompt or chain step
Integration
Full RAG or agent pipeline
Regression
Did a change make things worse?
Safety / Red-team
Can it be misused?
Production monitoring
Is it drifting over time?

Tool map — which tool owns which layer

ToolPrimary layerCore strength
RagasRAG evaluationAutomated retrieval + generation quality metrics without human labels
DeepEvalUnit + regression testingpytest-style LLM test suite with 14+ built-in metrics
LangSmithObservability + tracingFull trace visibility across every LLM call and chain step
PromptfooPrompt testing + red-teamSide-by-side prompt comparison, automated adversarial probes
GarakSecurity / safety red-teamingSystematic LLM vulnerability scanning — jailbreaks, injections, leaks
Evidently AIProduction ML monitoringData + model drift detection, embedding drift, production dashboards
Postman / NewmanAPI & contract testingFunctional correctness of AI service endpoints — schema, auth, latency, CI/CD gating
Ragas
RAG quality metrics
DeepEval
pytest for LLMs
LangSmith
trace & observe
Promptfoo
prompt CI/CD
Garak
safety scanner
Evidently
drift monitoring
Postman
API test suites

These tools are fast-moving. Treat specifics as directionally accurate and verify current API and metric names against each tool's official docs before an interview.

Ragas

RAG Evaluation Reference-free scoring

Ragas evaluates RAG pipelines by scoring both the retrieval and generation stages using LLM-as-a-judge techniques — without requiring a large set of human-labeled ground-truth answers. It computes metrics that together give a full picture of whether retrieval found the right content and whether generation used it faithfully.

Core Ragas metrics

MetricWhat it measuresWhat a low score means
FaithfulnessIs every claim in the answer grounded in the retrieved context?Model is hallucinating beyond what the context supports
Answer RelevancyDoes the answer actually address the question asked?Answer is off-topic or incomplete relative to the query
Context PrecisionOf the retrieved chunks, how many were actually useful?Retrieval is returning noise alongside signal
Context RecallDid the retrieved context contain all information needed to answer?Retrieval missed relevant content — index gap or chunking issue
Context RelevancyHow relevant are the retrieved chunks to the question?Retrieval is finding topically adjacent but not useful chunks

What you can build with it

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall

from datasets import Dataset

data = Dataset.from_dict({
    "question":  ["What is the 277CA transaction?"],
    "answer":    ["The 277CA is the Health Care Claim Acknowledgement..."],
    "contexts":  [["277CA is sent by payer to acknowledge receipt of claims..."]],
    "ground_truth": ["The 277CA acknowledges receipt and validity of 837 claims"]
})

results = evaluate(data, metrics=[faithfulness, answer_relevancy, context_recall])
print(results)
# {'faithfulness': 0.94, 'answer_relevancy': 0.89, 'context_recall': 0.91}
Advantages
  • Reference-free for most metrics — no large human-labeled dataset required to get started
  • Covers both retrieval and generation quality in one framework
  • Works well as a regression gate: run before/after a prompt or index change to confirm quality didn't drop
  • Integrates with LangSmith for tracing alongside evaluation
Disadvantages
  • LLM-as-judge means evaluation itself has latency and API cost
  • Judge LLM bias — scores can vary across judge model versions
  • Context Recall requires ground truth answers, removing the "reference-free" advantage for that metric
  • Best for RAG pipelines specifically; not a general-purpose LLM test framework

DeepEval

Unit Testing Regression CI/CD

DeepEval is a pytest-compatible LLM testing framework with 14+ built-in metrics. It brings traditional software testing discipline (test cases, assertion logic, CI integration) to LLM outputs — letting you write, run, and track test suites the same way you'd manage unit tests in a software project.

Key built-in metrics

AnswerRelevancyMetric

Did the response stay on topic with the input?

FaithfulnessMetric

Is every claim supported by the retrieved context?

HallucinationMetric

Does the output contradict known context or facts?

ToxicityMetric

Does the output contain harmful or offensive content?

BiasMetric

Does the output exhibit demographic or factual bias?

GEval (custom)

Define any arbitrary criteria in natural language — flexible for domain-specific quality checks

from deepeval import evaluate
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
from deepeval.test_case import LLMTestCase

test_case = LLMTestCase(
    input="What does a 277CA acknowledge?",
    actual_output="The 277CA is sent by the payer to acknowledge 837 claim receipt.",
    retrieval_context=["277CA acknowledges receipt and syntactic validity of 837 submissions."]
)

relevancy = AnswerRelevancyMetric(threshold=0.7)
faithfulness = FaithfulnessMetric(threshold=0.8)

evaluate([test_case], [relevancy, faithfulness])
Advantages
  • pytest-compatible — slots into existing CI/CD pipelines naturally
  • 14+ built-in metrics covering quality, safety, bias, and hallucination
  • GEval allows custom natural-language criteria for domain-specific quality checks
  • Test history and regression tracking built in via DeepEval's platform
Disadvantages
  • LLM-as-judge evaluation incurs latency and cost per test run
  • Requires representative test cases upfront — getting those right takes real effort
  • Scores are probabilistic — a test can pass one run and narrowly fail another for the same input
  • Advanced features (regression dashboard, CI integration) are behind the paid platform tier

LangSmith

Observability Tracing

LangSmith is an observability and evaluation platform from LangChain. It traces every step of a LangChain or LangGraph execution — each LLM call, retrieval, tool invocation, and chain branch — giving you full visibility into what happened, how long it took, what it cost, and whether the output was correct. Think of it as the APM (Application Performance Monitoring) layer for LLM applications.

What LangSmith gives you

import os
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langsmith import traceable

os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_API_KEY"] = "your-api-key"

@traceable   # auto-traces this function in LangSmith
def run_rag_pipeline(question: str, context: str) -> str:
    prompt = ChatPromptTemplate.from_template(
        "Answer from context only:\n{context}\n\nQ: {question}"
    )
    chain = prompt | ChatOpenAI(model="gpt-4o-mini")
    return chain.invoke({"question": question, "context": context})
Advantages
  • Best-in-class trace visibility for LangChain/LangGraph applications
  • Converts production traffic into evaluation datasets automatically
  • Prompt versioning and A/B testing built in
  • Closes the feedback loop: trace → debug → evaluate → improve in one platform
Disadvantages
  • Tightly coupled to the LangChain ecosystem — less useful if you're not using LangChain
  • Production monitoring and advanced features require a paid plan
  • Adds a data egress dependency — traces leave your infrastructure to LangSmith's servers (consider for HIPAA contexts)
  • Overkill for simple single-step LLM calls with no chain complexity

Promptfoo

Prompt Testing Red-teaming

Promptfoo is a CLI and CI-integrated tool for systematically testing and comparing prompts. It runs your prompt variants across a test suite, scores outputs against assertions, and generates comparison reports — making prompt changes as reviewable and measurable as code changes. It also includes a built-in automated red-team module that probes prompts for safety and security vulnerabilities.

What you can build with it

# promptfooconfig.yaml
prompts:
  - "Answer using only this context:\n{{context}}\n\nQ: {{question}}"
  - "You are a healthcare assistant. Context: {{context}}\nQuestion: {{question}}"

providers:
  - openai:gpt-4o-mini
  - openai:gpt-4o

tests:
  - vars:
      question: "What are the 277CA status codes?"
      context: "277CA uses status codes CA, CD, and CE to indicate..."
    assert:
      - type: contains
        value: "CA"
      - type: llm-rubric
        value: "Answer must reference HIPAA 5010 standards"
      - type: not-contains
        value: "I don't know"
Advantages
  • Provider-agnostic — works with OpenAI, Anthropic, local Ollama models, and more
  • CI/CD native — runs in GitHub Actions, returns pass/fail exit codes
  • Built-in red-team module covers 40+ attack categories
  • Side-by-side HTML comparison reports for prompt A/B decisions
  • Open source core with no mandatory cloud dependency
Disadvantages
  • YAML-first config can get verbose for complex test suites
  • LLM-as-judge assertions still carry evaluation cost
  • Red-team module breadth is wide but depth on any single attack vector is shallower than Garak
  • Primarily a testing tool, not an observability/monitoring platform for production traffic

Garak

Security Scanner Red-teaming

Garak is an open-source LLM vulnerability scanner specifically focused on security and safety red-teaming. Where Promptfoo's red-team module is a broad sweep, Garak goes deep: it systematically probes a model or application across dozens of attack categories with hundreds of individual probes, and generates structured vulnerability reports.

What Garak tests for

CategoryWhat it probes
JailbreaksPrompt patterns that override system instructions (DAN, roleplay, encoding tricks)
Prompt injectionMalicious instructions hidden in user input or tool results that hijack the model
Data exfiltrationAttempts to extract system prompt contents or internal context the model shouldn't reveal
Hallucination probesQueries designed to elicit confident false statements
Toxicity / harmful contentProbes for harmful, violent, or discriminatory output generation
Package / code risksLLM-generated code that references malicious or non-existent packages
Continuation attacksPrompts designed to make the model continue harmful completions
# run Garak against an OpenAI endpoint
garak --model_type openai \
      --model_name gpt-4o-mini \
      --probes jailbreak,promptinject,leakreplay \
      --report_prefix my_system_audit

# Garak outputs a structured JSONL report
# listing each probe, outcome, and severity
Advantages
  • Most comprehensive open-source LLM security scanner available
  • Structured vulnerability reports suitable for compliance review or security audit
  • Continually updated with new probe categories as new attack types emerge
  • Works against any LLM endpoint — not framework-dependent
  • Critical for HIPAA or any compliance context where demonstrating security due diligence matters
Disadvantages
  • Primarily a research/audit tool — not designed as a CI gate for every commit
  • Can generate a high volume of output requiring interpretation; not plug-and-play for non-security teams
  • A passing Garak scan is not a guarantee of safety — it tests known attack patterns, not unknown ones
  • Probe results depend on the model and system prompt configuration being tested — not portable results

For healthcare or regulated environments: Garak scans are a strong artifact to produce as evidence of security due diligence, even if the full output requires expert interpretation.

Evidently AI

Drift Monitoring Production ML

Evidently AI is an open-source ML observability platform focused on monitoring models and data in production. For LLM systems it tracks text quality, embedding drift, data distribution shifts, and output quality degradation over time — answering the question traditional testing can't: "Is the system still performing the same way it was when we shipped it?"

What Evidently monitors for LLM systems

MonitorWhat it detectsWhy it matters
Embedding driftQuery embeddings shifting away from the distribution the system was tested onUsers are asking different things than anticipated — retrieval may be degrading silently
Text quality metricsOutput length, readability, OOV word rate, sentiment driftModel behavior or output style changing post-deployment
Data drift (input)Statistical distribution shift in incoming queries or documentsData upstream of the pipeline changed — index may be stale
LLM quality scoresTracks Ragas-style metrics (faithfulness, relevancy) sampled from production traffic over timeCatches gradual quality degradation without waiting for user complaints
Custom metricsAny numeric metric you compute on outputsDomain-specific KPIs (e.g. % of answers that cite a source)
from evidently.report import Report
from evidently.metric_preset import TextEvals
from evidently import ColumnMapping
import pandas as pd

# production sample: questions + answers logged from live traffic
current_data = pd.DataFrame({
    "question": [...],
    "answer":   [...],
    "context":  [...]
})

report = Report(metrics=[TextEvals()])
report.run(reference_data=baseline_data, current_data=current_data)
report.save_html("drift_report.html")
Advantages
  • Catches quality degradation that pre-deployment testing misses — the "silent drift" problem
  • Open source with a rich dashboard for non-technical stakeholders
  • Works on both classic ML and LLM systems — useful across a full ML platform
  • Embedding drift detection is especially valuable for RAG pipelines with evolving query patterns
Disadvantages
  • Requires production traffic logging infrastructure to feed it — not zero-setup
  • Statistical drift detection generates alerts that need human interpretation to act on
  • LLM-specific features are newer than its traditional ML monitoring capabilities; some are still maturing
  • Self-hosted deployment means you own the infra; managed cloud tier adds cost

API Testing (Postman & Newman)

API & Contract Testing CI/CD Integration

Every AI system — a RAG pipeline, an agent, a fine-tuned model endpoint — is ultimately served behind an API: a REST or GraphQL endpoint that takes a request and returns a response. Ragas, DeepEval, and Promptfoo test whether the content of that response is good. Postman and Newman test whether the service itself is correct — status codes, schema, authentication, latency, and error handling. Skipping this layer means shipping an LLM system that scores well on quality metrics but breaks under a malformed payload, a slow downstream call, or an expired token. Your EDI background maps onto this directly: a 277CA has to validate against its schema and envelope structure before anyone asks whether its content is meaningful, and an AI API is no different.

What Postman/Newman covers that AI-quality tools don't

CheckWhat it verifiesWhy it matters for AI systems
Schema validationResponse body matches the expected JSON schemaAn LLM API returning malformed or partial JSON breaks every downstream consumer silently
Status code / error handlingCorrect codes for auth failure, rate limits, timeouts, bad inputModel providers rate-limit and time out constantly; the wrapping API must fail predictably
Auth & headersAPI keys, OAuth tokens, and required headers are enforcedSame compliance discipline as HIPAA EDI transactions — access control is non-negotiable
Latency thresholdsResponse time stays under an agreed SLALLM calls are slow and variable; a regression here is a real production incident
Contract testingRequest/response shape stays stable across versionsPrevents a backend model swap or prompt change from silently breaking client integrations
Chained requestsMulti-step flows — e.g. auth → retrieve → generate → log — run and pass in sequenceMirrors real agent/RAG call chains, not just a single isolated endpoint

What you can build with it

// Postman test script — attached to a "Generate Answer" request
pm.test("Status code is 200", function () {
    pm.response.to.have.status(200);
});

pm.test("Response matches schema", function () {
    const body = pm.response.json();
    pm.expect(body).to.have.property("answer");
    pm.expect(body).to.have.property("sources").that.is.an("array");
    pm.expect(body.sources.length).to.be.above(0);
});

pm.test("Latency under SLA", function () {
    pm.expect(pm.response.responseTime).to.be.below(3000);
});

pm.test("Auth header enforced on 401 case", function () {
    if (!pm.request.headers.has("Authorization")) {
        pm.response.to.have.status(401);
    }
});
# Newman — run the collection headlessly in CI/CD
newman run ai-api-collection.json \
  -e staging.postman_environment.json \
  --reporters cli,junit \
  --reporter-junit-export results/api-test-report.xml

# Exit code is non-zero on any failed assertion,
# which fails the pipeline build step automatically
Advantages
  • Catches structural and contract failures that content-quality tools like Ragas or DeepEval aren't designed to catch
  • Newman's JUnit output plugs directly into Jenkins, GitHub Actions, or any CI/CD dashboard your team already uses
  • Environment variables make one collection reusable across dev, staging, and production
  • Low learning curve relative to AI-specific tools — leverages skills already common on integration teams
Disadvantages
  • Has no concept of answer quality, faithfulness, or hallucination — it only validates the envelope, not the content
  • Test scripts are hand-written JavaScript inside each request — collections can get hard to maintain at scale
  • Chained multi-step flows require careful variable-passing between requests, which adds setup overhead
  • Best treated as one layer in a stack — it complements Ragas/DeepEval/Promptfoo rather than replacing any of them

Positioning for an interview: API testing is the layer that makes sure the AI system is a reliable service. Content-quality tools make sure it's a good service. Both are required — a system can pass every Ragas metric and still fail production if the endpoint times out under load or returns an unvalidated schema to a downstream consumer.

Production-Level Interview Q&A

Click a question to expand the answer.

How would you design a complete testing strategy for a RAG pipeline before shipping to production?
Think in layers. Start with a labeled evaluation set of 50–100 representative queries with known-good answers, built before any other testing work. Use Ragas to score faithfulness, answer relevancy, context precision, and context recall against that set — this gives you a retrieval and generation quality baseline. Use DeepEval to write pytest-style test cases for specific edge cases and format requirements, integrated into CI so a prompt or index change that breaks a case fails the build. Run Promptfoo for prompt A/B comparison when you're iterating on the system prompt. Before release, run Garak for a security and safety audit, especially if the system will be customer-facing. After release, set up Evidently AI or LangSmith monitoring to detect silent quality drift in production traffic. The key principle from your ETL background applies directly: test each stage independently (retrieval, generation, output format) rather than only judging the final answer.
What's the difference between Ragas and DeepEval — they both measure faithfulness and relevancy?
They have overlapping metrics but different primary purposes. Ragas is purpose-built for RAG pipeline evaluation — it's optimized for measuring the relationship between retrieved context, query, and generated answer, and it's designed to produce a diagnostic picture of whether retrieval or generation is the weak link. DeepEval is a general-purpose LLM test framework that happens to include faithfulness and relevancy metrics; its primary value is the pytest integration, test case management, and CI/CD regression tracking. In practice, many teams use both: Ragas for deep RAG quality diagnostics, DeepEval for the test suite discipline and CI gate.
Why does LangSmith matter if you're already running Ragas evaluations?
Ragas tells you what the quality score is. LangSmith tells you why a specific call produced the output it did. When a Ragas faithfulness score drops, LangSmith traces let you inspect exactly which retrieval step returned weak context, which prompt version was in effect, how long each step took, and what the token cost was per call. Evaluation without observability leaves you knowing something is wrong with no way to locate it. Ragas and LangSmith are complementary: evaluation tells you there's a problem, tracing tells you where it is.
How would you use Promptfoo in a CI/CD pipeline for prompt changes?
Store prompt templates in version control. When a prompt change is proposed, Promptfoo runs the candidate and the current-production prompt against the same test suite (your labeled evaluation set plus edge cases) and returns pass/fail exit codes, blocking the merge if quality drops below threshold on any assertion. The HTML comparison report is attached as a PR artifact so reviewers can see side-by-side output diffs, not just a pass/fail signal. This makes prompt changes as reviewable as code changes — which is the discipline most teams lack when they start and pay for later in unexplained production regressions.
When would you run Garak, and what would you do with the output?
Garak is best positioned as a pre-production security audit tool run against any customer-facing LLM application, and re-run after significant changes to the system prompt or model. The output is a structured JSONL vulnerability report listing which probes succeeded (meaning the attack worked) and which failed (the model was robust). Act on the results by hardening the system prompt against successful attack categories, adding output validation to catch elicited harmful content, and scoping tool permissions more tightly where injection risks were found. For a HIPAA-regulated application, the Garak report itself is valuable compliance documentation — evidence that systematic security testing was performed.
What is 'silent quality drift' and which tool is specifically designed to catch it?
Silent quality drift is when a production LLM system's output quality degrades gradually over time without any error being thrown — the system keeps returning responses, they're just increasingly less accurate, less relevant, or less faithful to context. Causes include embedding model changes, upstream data going stale, query distribution shifting away from what the system was optimized for, or a model provider silently changing model behavior. Evidently AI is specifically designed to catch this: it monitors production traffic for statistical distribution shifts in inputs, embedding drift, and quality metric trends over time, alerting before users notice and before a complaint-driven post-mortem is necessary.
How would you explain LLM-as-judge evaluation to a stakeholder, and what are its limits?
LLM-as-judge uses a separate LLM call to score the output of your main LLM call — asking a judge model 'is this answer faithful to the context?' rather than comparing against a human-labeled ground truth. The appeal: it scales to thousands of test cases without human annotation cost and handles natural language quality criteria that rule-based checks can't. The limits: the judge model itself has biases and can be wrong, scores can drift as the judge model is updated, it adds latency and token cost per evaluation, and a sufficiently clever model can produce outputs that score well on judge metrics while still being subtly wrong in domain-specific ways a general judge won't catch. The practical implication: use LLM-as-judge as a scalable first signal, but calibrate it against a smaller set of human-labeled examples to verify the judge's scores correlate with actual quality in your domain.
A production RAG system is showing declining answer quality over three weeks but no code changes were deployed. How do you investigate?
This is a data/infrastructure drift problem, not a code problem. Investigate in order: check whether the source data feeding the index has changed — new document formats, schema changes, deleted or updated records that made previously valid chunks stale. Use Evidently AI or LangSmith to check if the distribution of incoming queries has shifted (users started asking a different type of question the current chunking strategy doesn't serve well). Check the embedding model — did the provider push an update? If vectors in the index were generated with a prior version and queries are now embedded with a new version, they're no longer in the same semantic space. Check retrieval metrics specifically (context precision, context recall via Ragas) to isolate whether the problem is in retrieval or generation — if retrieval scores held steady but faithfulness dropped, the model changed; if retrieval scores fell, the index is the culprit.
How does API testing with Postman/Newman fit alongside AI-quality tools like Ragas or DeepEval — isn't that redundant?
They test different failure modes, not the same one. Ragas and DeepEval answer "is the content of this response good?" — faithful, relevant, non-toxic. Postman/Newman answer "is the service itself correct?" — right status code, valid schema, auth enforced, latency within SLA. A model can produce a perfectly faithful, relevant answer and the API can still fail production by returning it with a 500 on retry, dropping a required field under load, or exceeding a timeout the client doesn't handle gracefully. In a mature testing strategy both run in the same CI/CD pipeline: Newman gates on contract and functional correctness, DeepEval or Promptfoo gates on content quality, and neither is a substitute for the other.
How would you design a Postman/Newman suite for an AI API, and what would you assert on?
Structure the collection around the real call chain, not isolated endpoints: authentication, the core inference or retrieval call, and any feedback/logging call that follows it, chained with variables passed between requests. Assert on four things per request — status code correctness for both success and expected failure cases (bad auth, rate limit, malformed input), schema validation on the response body so a partial or malformed payload fails loudly instead of breaking a downstream consumer silently, a latency threshold tied to the agreed SLA since LLM calls are inherently slower and more variable than typical REST calls, and negative-path coverage confirming the API degrades predictably — a clear error, not a silent empty response — when a required field is missing or the upstream model call times out. Wire the collection into CI via Newman with a JUnit reporter so a broken contract fails the build before it reaches staging.
How would you validate that a data pipeline feeding a RAG index only reprocesses changed records, rather than doing a full reload each time?
This is a CDC (change data capture) and incremental-load validation problem, and it maps directly onto SCD Type 2 discipline from traditional ETL. First, confirm the source extraction is only pulling records with a changed timestamp, version flag, or hash since the last successful run — not the full table. Second, validate that the embedding/indexing step only re-embeds those changed records and correctly retires or supersedes the prior vector entries for updated documents, rather than leaving stale duplicate vectors in the index. Third, add a reconciliation check comparing source record counts against index record counts after each run, plus a spot-check that a known recently-updated document returns its new content on retrieval, not the old version. This prevents two silent failure modes at once: wasted compute from unnecessary full reloads, and stale answers from an index that never picked up the change.