Posted by Mahdi
Posted by Mahdi
Umbraco AI Governance: Guardrails, Permissions and Audit Logs
Learn how Umbraco AI guardrails, permissions, audit logs and observability work together—and what Australian digital teams should configure before launch.
Adding AI to a content management system changes more than the editing experience. It creates new paths for information to leave the CMS, new actions software can take on behalf of users, and new records that may contain prompts, responses and personal information.
Umbraco.AI provides four useful control layers: guardrails, agent permissions, audit logs and OpenTelemetry-based observability. They are capable controls, but they solve different problems. A guardrail cannot replace access control. A permission cannot prove what happened. A detailed log can itself become sensitive data.
This guide explains how the layers fit together, where their boundaries sit, and how an Australian marketing or digital team can configure a sensible first release. It is implementation guidance, not legal advice.
The Four Control Layers
Good governance starts by assigning each control a job. Trying to make one feature carry the whole risk model produces impressive settings screens and surprisingly fragile operations.
| Layer | Primary question | Practical role |
|---|---|---|
| Guardrails | Should this input or output be allowed? | Detect, block, warn about or redact defined content. |
| Permissions | Which tools may this agent use for this user? | Restrict read, write, media, search, navigation and web capabilities. |
| Audit logs | What operation occurred and how did it finish? | Record user, provider, model, tokens, duration, outcome and optional content snapshots. |
| Observability | How is AI behaving across the application? | Trace latency, token use, failures and the CMS context around an operation. |
Together they create prevention, authorisation, evidence and monitoring. Human ownership, vendor assessment, privacy review, content approval and incident response still sit outside the product configuration.
1. Use Guardrails for Defined Input and Output Risks
Umbraco guardrails can run before a request reaches the AI provider and after a response returns. A rule can Block, Warn or Redact. Built-in evaluators include simple contains checks, regular expressions and an LLM-based safety judge.
This makes guardrails useful for risks that can be expressed clearly: stopping known prohibited terms, redacting email-like patterns before generation, flagging a response that does not match brand criteria, or blocking output that fails a defined safety check.
Choose deterministic rules where possible
Contains and regex rules are predictable and can support redaction. Model-based judgement is more flexible but less deterministic. Umbraco documents that its model-based evaluator cannot identify exact text positions for redaction, so a Redact action degrades to Warn. During streaming, post-generation redaction also degrades to Warn because content already sent to the caller cannot be pulled back.
That limitation matters. If a data pattern must never leave your boundary, detect and remove it before generation or prevent the source data from entering the workflow. Do not rely on a post-generation rule as a privacy force field.
Version the policy, then test it
Guardrails support version history, comparison and rollback. Treat each version as a policy change: record the owner, intended risk, examples that should pass, examples that should fail, false-positive tolerance and approval date. Test both ordinary use and evasive variations before assigning it to a production profile.
See Umbraco's guardrails documentation for evaluator and streaming behaviour.
2. Give Agents the Smallest Useful Permission Set
For standard agents, Umbraco supports three permission levels: tool scopes, explicit tools and user-group overrides. Built-in scopes distinguish read operations from destructive content-write and media-write operations. The runtime excludes tools that are not in the resolved allowed list.
Begin with a read-only role. Add one capability when a tested use case genuinely requires it. A content assistant that finds pages and suggests edits may need content-read and search; it does not automatically need permission to publish, delete media or fetch arbitrary web pages.
Build permissions around tasks, not seniority
Administrators often receive broad access because of their job title, but an AI agent should receive access according to the task it performs. A senior user running a research assistant may still need read-only tools. An editor running an approved publishing workflow may require a narrow write capability with a human confirmation step.
User-group overrides can add or deny tools, and deny rules take precedence over ordinary allows. This is useful for subtracting destructive scopes from groups that inherit broader access. System tools cannot be denied, so teams should document what those tools do and include them in testing.
Separate the environments
Test permission resolution with representative users in a non-production environment. Check direct requests, agent workflows and edge cases such as membership in multiple groups. Review permissions when an agent's instructions, tools or business purpose changes.
Umbraco's agent permissions documentation describes the resolution order and destructive scopes.
3. Configure Audit Logs as Sensitive Business Records
Umbraco AI audit records can show when an operation ran, its status, capability, profile, provider, model, user, token use and duration. That evidence helps investigate failures, blocked requests, unexpected costs and disputed actions.
The difficult choice is content capture. The documented settings show prompt and response persistence enabled by default. That can improve debugging, but it also means the audit store may receive the very customer information, draft content or confidential material the organisation is trying to govern.
Decide deliberately what to retain
- Document why prompt or response snapshots are needed.
- Disable snapshots when metadata is enough for the operational purpose.
- Apply redaction patterns to known sensitive formats.
- Restrict who can view, export and delete audit records.
- Set retention from business and compliance needs rather than accepting a default unexamined.
- Test cleanup and incident-preservation procedures.
The documented configuration shows a 14-day default retention period, but that is a software default—not a recommendation for every organisation. A suitable period depends on investigation needs, contracts, privacy obligations and the sensitivity of captured content.
Review the current audit-log settings before deployment.
4. Monitor Behaviour, Cost and Reliability
Audit logs answer questions about individual operations. Observability helps teams see patterns across the system. Umbraco.AI can emit OpenTelemetry traces and metrics using standard gen_ai.* conventions, enriched with the profile, entity, feature, audit record and initiating user.
Available metrics include input and output token use, operation duration, time to first streamed chunk and time per output chunk. These support practical dashboards and alerts:
- token consumption by feature or profile
- failure and blocked-operation rates
- latency changes after a provider or model update
- unexpected use by an entity, agent or user group
- audit records associated with a slow or failed trace
Do not collect every attribute merely because it exists. Decide which tags and traces help operate the service, who can access them, and how long the monitoring platform retains them. Observability data can reveal user behaviour and content relationships even when prompt text is absent.
See Umbraco's observability documentation for the emitted metrics and trace correlation.
A Practical Umbraco AI Launch Checklist
Before configuration
- Name the business owner and technical owner.
- Define the approved task, users, data sources, outputs and prohibited uses.
- Map what reaches the model provider, audit store and monitoring platform.
- Decide which outcomes require human review or approval.
During configuration
- Start with read-only tool scopes and add individual write tools only when required.
- Create deterministic pre-generation rules for data that must be blocked or redacted.
- Add post-generation checks for quality, policy and brand risks.
- Set audit snapshot, redaction, access and retention options deliberately.
- Enable the minimum traces and metrics needed to operate the service.
Before launch
- Test normal tasks, prohibited requests, prompt injection, group overrides and destructive actions.
- Confirm blocked, warned and redacted events appear correctly in logs.
- Verify alerts, rollback, shutdown and incident escalation.
- Train editors on capabilities, limitations and the reporting path.
- Record the approved configuration and next review date.
Begin with one bounded workflow. A small, observable use case teaches more than a sprawling pilot with unclear ownership and a heroic permissions list.
What These Controls Do Not Solve
Product controls are part of governance, not the whole of it. They do not determine whether the organisation has an appropriate purpose for using information, whether provider terms are acceptable, whether generated content is accurate, or whether an automated decision is fair.
They also do not remove the need for content ownership. Someone must decide which claims need evidence, who approves publication, when an agent may change a record, and how affected people can challenge an outcome.
The most useful operating principle is simple: prevent what you can define, restrict what can act, record what matters and monitor what changes. Then place accountable people around the gaps that software cannot resolve.
Frequently Asked Questions
Short answers for teams planning a governed Umbraco AI rollout.
Design the Controls Before the AI Workflow Grows
VaniTech can help scope Umbraco AI use cases, permissions, guardrails, logging and monitoring around your team's real content workflow.