Healthcare chatbots: semantic search, RAG, or both?

1. Three choices, not three competing products

RAG is a pattern: retrieve evidence, then give that evidence to a model to help generate a response. Semantic search helps find material by meaning. They are not opposites. A RAG application always needs a retrieval step, but that step can be keyword search, vector retrieval, a curated lookup, or a combination.

Azure AI Search also has a specific feature called semantic ranking. It reranks an initial text or hybrid result set. Its captions and answers are extracted from source text, not newly composed chatbot answers. In this comparison, "RAG without semantic reranking" means a useful baseline, not generation without retrieval.

An exact policy number may be better served by ordinary keyword lookup. Do not add a semantic or generative step just because the interface looks like a chatbot.

Documentation: Microsoft Foundry: retrieval-augmented generation; Azure AI Search: semantic ranking; Azure AI Search: hybrid retrieval

2. Start with the inquiry, especially when it concerns care

Separate finding a source, explaining a process, and making a patient-specific decision. A polished response can hide that distinction. Make the route visible: "approved document," "draft explanation," or "patient record lookup." A source citation supports checking; it does not certify that a response is clinically correct.

For a care-related workflow, keep the approved source, owner, effective date, location, and audience together. A retired instruction sheet and a current one may be nearly identical to an embedding model. Relevance alone cannot decide which is safe to use.

This is software architecture guidance, not medical advice. Any patient-facing or clinical decision-support use requires separate clinical safety evaluation and appropriate professional oversight.

3. Small office, regional practice, large health system

A small office may have little IT capacity but still handle highly sensitive information. A regional practice may have only a modest library, yet complex location rules. A health system may have a huge archive, while a particular chatbot needs only one specialty's approved documents. Size helps plan ownership and capacity; it does not pick the answer mode for you.

• Small office: name a document owner and a support owner. If a maintained knowledge page solves the task, that may be the better first investment.
• Regional practice: distinguish organization-wide guidance from local exceptions. A float nurse may have several site memberships; a local receptionist may have one.
• Large organization: budget for ingestion failures, permission changes, isolation requirements, observability, review queues, and operational ownership, not just model tokens.

These document ranges are fictional planning examples. Ten long scanned manuals can be harder to retrieve from than a thousand clean, short pages. Location count does not determine the required Azure tier.

4. Count chunks and difficult questions, not just documents

Inventory pages, usable text, chunk counts, duplicates, languages, tables, and update frequency. For example, 5,000 documents averaging 12 chunks produce roughly 60,000 indexed chunks before accounting for overlap and version retention. That is arithmetic for a capacity conversation, not a benchmark or a sizing recommendation.

Measure retrieval using questions people actually ask: an exact form code, a colloquial description, a question spanning two sources, a site-specific exception, and something the library cannot answer. Preserve table headings and section context during extraction. Treat OCR quality and conflicting versions as source problems before changing models.

Compare the three modes on the same eligible sources and access rules. Record source recall, citation support, unsupported claims, useful refusals, end-to-end latency, and cost per successfully reviewed task. Measure retrieval and generation separately so a fluent answer does not mask poor evidence.

Price the deployed Search capacity, ingestion and embedding updates, optional semantic ranking, Foundry model use, storage, monitoring, and ongoing support against current commercial terms. Search without generation avoids chat-model calls for answers; it does not make indexing or the retrieval service free. Add a layer only when its benefit is worth its latency and operating cost.

5. SSO gets you through the door, not into every document

Federated sign-in tells the application who authenticated. Authorization decides what that person may read now. For an Entra-backed API, validate the access token with a supported authentication library, including signature, issuer, audience, expiry, and allowed tenant. Do not treat a decoded token, an email address, or a browser-supplied group list as proof.

In this design, an application role such as Knowledge.Reader permits entry to the retrieval feature. Source-derived user and group grants determine document scope. These are AND conditions: being allowed to use the chatbot does not bypass document permissions. Application roles are not the same as tenant directory administrator roles.

For Entra identities, use tenant and object IDs rather than display names. When another identity provider federates into Entra, map to the validated Entra identity; for a separate federation model, maintain an explicit issuer-and-subject mapping to canonical principals. Never assume equal email addresses mean equal access.

Resolve group membership through a trusted path. A missing groups claim can indicate group overage rather than no membership. If required membership cannot be resolved, fail closed. Test guest identities and the direct or transitive membership semantics your policy actually needs; a renamed group should not silently become a new permission.

Documentation: Microsoft Entra: application roles; Configure group claims and application roles; Group claims in federated applications

6. Put a trusted retrieval boundary in front of Foundry

The proposed path is: staff sign-in, trusted backend authorization, permission-scoped Search query, current document access check, then a Foundry model call if the task needs generation. Keep the raw Search endpoint and its credentials out of the browser. The model may propose a question; it must not choose the tenant, groups, or access filter.

A Foundry project role or a managed-identity connection to Search controls service access. It is not proof that the person chatting can read every indexed document. Verify end-user identity propagation for the exact agent tool and API version you use. Where it cannot enforce your policy, use a controlled retrieval service rather than handing an agent an unscoped index.

Keep access context per request. Do not rewrite a shared agent's filter when each person signs in: concurrent users could inherit each other's scope. If a tool accepts a filter, compose it from trusted authorization context and intersect any user-selected refinements with that scope.

Documentation: Connect Azure AI Search to Foundry agents; Microsoft Foundry role-based access

7. Store permission metadata beside every vector

The example below is for an approved staff knowledge library, not a patient-record index. Each chunk contains text, an embedding, a document identifier, version information, and an effective allowlist. Tenant, readiness, and permission fields are filterable metadata. Do not embed an access rule into prose and expect vector similarity to enforce it.

At ingestion, resolve the source's effective read policy and copy it to every child chunk. The u: and g: prefixes distinguish normalized user and group IDs. This example supports effective allow grants; raw deny rules, nested conditions, or "site AND specialty" rules must be resolved by an authoritative policy service or modeled explicitly. Putting two groups in one allowlist means either group can match, not that both are required.

An empty allowlist grants nothing. Shared documents need an explicit approved all-staff principal, not a wildcard fallback. Chunk keys must be unique across the index, including across tenants. aclState and aclVersion are application-managed fields; Azure does not make them current automatically. Mark unverified or partially refreshed content unavailable. The 1536 dimensions must match the embedding deployment; they are an example, not a universal setting.

REST schema target: 2025-09-01. This is the application-managed security-filter pattern, not the separate native preview ACL schema. Hiding metadata with retrievable: false is data minimization, not authorization.

{
  "name": "governed-chatbot-chunks",
  "fields": [
    { "name": "chunkId", "type": "Edm.String", "key": true },
    { "name": "documentId", "type": "Edm.String", "filterable": true },
    { "name": "tenantId", "type": "Edm.String", "filterable": true, "retrievable": false },
    { "name": "title", "type": "Edm.String", "searchable": true },
    { "name": "content", "type": "Edm.String", "searchable": true },
    { "name": "sourcePath", "type": "Edm.String" },
    { "name": "sourceVersion", "type": "Edm.String", "filterable": true },
    { "name": "updatedAt", "type": "Edm.DateTimeOffset", "filterable": true },
    { "name": "aclVersion", "type": "Edm.String", "filterable": true },
    { "name": "aclState", "type": "Edm.String", "filterable": true, "retrievable": false },
    { "name": "isActive", "type": "Edm.Boolean", "filterable": true, "retrievable": false },
    {
      "name": "allowedPrincipals",
      "type": "Collection(Edm.String)",
      "searchable": false,
      "filterable": true,
      "retrievable": false
    },
    {
      "name": "contentVector",
      "type": "Collection(Edm.Single)",
      "searchable": true,
      "retrievable": false,
      "dimensions": 1536,
      "vectorSearchProfile": "content-profile"
    }
  ],
  "vectorSearch": {
    "algorithms": [
      { "name": "content-hnsw", "kind": "hnsw", "hnswParameters": { "metric": "cosine" } }
    ],
    "profiles": [
      { "name": "content-profile", "algorithm": "content-hnsw" }
    ]
  },
  "semantic": {
    "configurations": [
      {
        "name": "content-semantic",
        "prioritizedFields": {
          "titleField": { "fieldName": "title" },
          "prioritizedContentFields": [{ "fieldName": "content" }]
        }
      }
    ]
  }
}

Documentation: Azure AI Search: create a vector index; Create Index REST reference: 2025-09-01; Azure AI Search: security filter pattern

8. Build the access filter on the server

The query builder assumes that a trusted authentication and policy layer has already verified the caller, checked the tenant, resolved complete group membership, and supplied the application roles. None of those values may come directly from form fields, request JSON, or model tool arguments. The function deliberately refuses incomplete authorization context.

It combines the tenant boundary, active content, ACL readiness, and principal membership before returning candidates. Hybrid retrieval uses both text and vectors; semantic ranking then helps order the evidence. preFilter constrains vector candidate search. Search-side postFilter is not the same as hiding a forbidden passage after it has already entered a model prompt.

The filter is a first boundary, not proof that the index reflects a permission change made a second ago. Before passing text to Foundry, reauthorize each returned document against a current trusted policy snapshot and its ACL version. Drop stale or unauthorized evidence; if authorization is unavailable, do not generate from it. Source links need access checks too.

This is a teaching example, not a complete authentication service or a deployed Azure integration. For a controlled baseline comparison, retain the same access filter and retrieval inputs, remove semantic query settings, and evaluate the resulting evidence before generation.

/**
 * Illustrative query builder, not an authentication implementation.
 * POST /indexes/governed-chatbot-chunks/docs/search?api-version=2025-09-01
 *
 * Call only from a trusted backend after token validation, tenant checks,
 * and complete group resolution. Never populate access directly from
 * request JSON or model-generated tool arguments. Types are not proof
 * of authorization; the trusted backend must establish these values.
 */
export type TrustedAccessContext = {
  tenantId: string;
  userOid: string;
  groupOids: readonly string[] | null;
  groupsComplete: boolean;
  appRoles: readonly string[];
};

function normalizeId(value: string): string {
  const uuid = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
  if (typeof value !== "string" || !uuid.test(value)) {
    throw new Error("Expected a canonical UUID");
  }
  return value.toLowerCase();
}

export function buildQuery(
  question: string,
  vector: readonly number[],
  access: TrustedAccessContext,
) {
  if (!access || !Array.isArray(access.appRoles) ||
      !access.appRoles.includes("Knowledge.Reader")) {
    throw new Error("Knowledge retrieval is not permitted");
  }
  if (access.groupsComplete !== true || !Array.isArray(access.groupOids)) {
    throw new Error("Access scope could not be resolved");
  }
  const tenant = normalizeId(access.tenantId);
  const principals = ["u:" + normalizeId(access.userOid)];
  for (const group of access.groupOids) {
    principals.push("g:" + normalizeId(group));
  }
  if (typeof question !== "string" || !question.trim()) {
    throw new Error("A question is required");
  }
  if (!Array.isArray(vector) || vector.length !== 1536 ||
      !Array.from(vector).every(value => Number.isFinite(value))) {
    throw new Error("Expected 1536 finite embedding values");
  }

  // UUID validation excludes OData quotes and list delimiters.
  const allowed = [...new Set(principals)].sort().join(",");
  const securityFilter =
    "tenantId eq '" + tenant + "' and isActive eq true " +
    "and aclState eq 'ready' and " +
    "allowedPrincipals/any(p: search.in(p, '" + allowed + "', ','))";

  return {
    search: question,
    filter: securityFilter,
    vectorFilterMode: "preFilter",
    vectorQueries: [{
      kind: "vector", fields: "contentVector", vector: [...vector], k: 50,
    }],
    queryType: "semantic",
    semanticConfiguration: "content-semantic",
    select: "chunkId,documentId,title,content,sourcePath," +
      "sourceVersion,updatedAt,aclVersion",
    top: 5,
  };
}

// Not shown: token verification, authoritative group resolution,
// embedding generation, managed-identity Search authentication,
// current document authorization checks, or Foundry generation.
// Before retrieved text enters model context, recheck access and
// ACL freshness for its documentId. Fail closed on uncertainty.
// This example is for staff knowledge, not patient-record access.

Documentation: Azure AI Search: hybrid retrieval; Azure AI Search: security filter pattern; Azure AI Search: vector query filters

9. Patient-specific care needs a separate authorization path

Approved education material is not a substitute for the current chart. A patient-specific question may require recent results, encounter context, consent, and restrictions that change independently of a staff group. Retrieve current structured data through an authorized EHR or FHIR integration with the system's relevant access controls. A broad service credential is not an end-user permission model.

Keep the general knowledge route separate from the patient-data route. Do not silently copy charts into the staff library or let a prompt select any patient identifier. Patient-facing access and staff access need separate policies. Break-glass access, if supported, belongs in an explicit controlled and audited workflow, never in a chatbot's reasoning.

A clinician-facing assistant can help locate approved guidance or prepare a source-backed draft for review. It must not imply that matching a guideline to a chart establishes a diagnosis, suitability, or treatment plan. Evaluate intended clinical use with qualified stakeholders. An Azure service name, SSO, or this schema does not establish regulatory compliance.

Documentation: Azure Health Data Services: authentication and authorization

10. Permissions must survive updates, memory, and caching

When a document is retired, replaced, or restricted, remove or quarantine every affected chunk. Track source and ACL versions together; avoid a refresh that mixes new text with old permissions. Keep revocation handling and its failure mode explicit. If the current policy cannot be checked, use a safe unavailable response rather than last-known broad access.

Do not share answer caches across users just because their questions match. Any cache needs tenant, effective permission scope, source version, and policy freshness in its validity rules. Recheck access before reusing evidence in a later chat turn; earlier permission does not last for the whole conversation.

Treat retrieved documents as untrusted inputs. A passage saying "ignore access rules" is content, not an instruction to the tool layer. Minimize sensitive prompt and response logging, restrict audit access, and verify retention and processing arrangements before introducing protected data.

Documentation: OWASP: prompt injection

11. Native permission features are a separate implementation choice

Azure AI Search also documents native document-level permission features, with preview constraints and source-specific support. Those features use a different schema contract, including permissionFilterOption and permissionFilter fields, rather than the application-owned allowedPrincipals convention above.

Where supported, a query can carry the end-user identity through x-ms-query-source-authorization. Confirm the exact API, identity propagation, supported principals, and missing-identity behavior. The agentic retrieval documentation warns that indexed knowledge-source requests without the required user identity can return unfiltered results. Reject that condition at your trusted boundary; never treat a missing token as public access.

Check the current documentation and release status before choosing native ACL features. Do not combine preview schema examples with the stable example here and assume they have the same security behavior.

Documentation: Azure AI Search: document-level access overview; Index ACLs using the push REST API; Query-time ACL and RBAC enforcement; Agentic retrieval: permissions at query time

12. A pilot should prove both usefulness and refusal

For a small office, choose one repeated question and a small approved library. For a 5-10-location practice, include at least one shared policy and two conflicting local versions. For a large organization, add cross-source and cross-tenant attempts, concurrent users, and permission-update failures. Use synthetic records and authorized test identities rather than real patient data in a casual prototype.

The useful outcome is not always a longer answer. It may be a direct source link, a clear conflict, a request for context, or a safe handoff. Choose the least complex design that passes the real task and access tests.