Building an AI management system before the requirements tightened

Why the question came up

The AI Act is clear about what must be achieved and far less clear about how. For a small company without a legal department, that is where the work sits.

At Lidodevelopment we build systems with a lot of AI in them. AI agents assist in development, and Civisto uses model inference against municipal data. Governance existed, but it was spread across three operational documents. No coherent AI policy, no register of which AI systems actually existed, no defined process for classifying a system or reclassifying it when it changed.

Three things made it impossible to postpone any longer:

The AI Act is phasing in. Transparency requirements already apply at limited risk, and every Civisto function that moves toward municipal decision support gets closer to the Annex III boundary.

Municipalities are already asking. AI-specific questions show up more often in procurement questionnaires: the model’s jurisdiction, training on customer data, what human control looks like, how bias is handled. Our data-residency policy answered where the data sat. It said nothing about how the AI was governed.

ISO 27001 does not go all the way. An information security management system shows that data is protected. It says nothing about whether an AI system is fit for purpose, whether operations are monitored, or whether anyone has assessed how it affects residents.

ISO/IEC 42001:2023 is the certifiable standard for AI management systems. It uses the same harmonised structure, clauses 4–10, as ISO 27001 and ISO 9001. That structure was already in place here, which made the choice simpler than it otherwise would have been.

The choice: a layer, not a system beside

We placed the AI management system (AIMS) as a layer on top of the existing ISMS and QMS, not as a parallel system. For a company of our size that is the only sustainable setup we can see. Whether it holds is still a hypothesis that has not been tested in an audit.

In practice:

• Lidodevelopment owns the management system. Civisto and the AI agent chain are the first stated scope.
• The shared governance routines in clauses 9 and 10 are reused unchanged: internal audit, management review, corrective action, document control. No AIMS copies exist.
• Risk assessment, risk appetite and the register are inherited from the existing risk policy. AI risks are tagged ai-risk in the same register.
• Only genuinely AI-specific artefacts are produced: AI policy, statement of applicability against Annex A, AI system register, AI risk template, impact assessment template, lifecycle process and data-governance guidelines.
• We declare ourselves as a provider and deployer of AI systems, expressly not as a model producer. We do not train or fine-tune models.
• No certification is sought now. The layer is built so it can be certified later and so it can be used immediately as evidence in procurement.

The basic rule is simple. A control that exists in several standards is documented once and referenced from the other statement of applicability. Only AI requirements with no counterpart in ISO 27001 become new documents: impact assessment, data provenance, transparency toward affected individuals, and an AI inventory.

Alternatives that were considered

Five other paths were on the table before the layer was chosen.

Rely on the AI documents that already existed. They were operational rules, not a management system. They would not hold up in an AI-specific procurement questionnaire.

Build a standalone AIMS with its own audit and its own routines. That doubles the administrative load. In a small company the risk is high that two sets of records slowly drift apart.

Informally stretch ISO 27001 to cover AI. Annex A has no counterpart to impact assessment and transparency. Stretching the standard creates claims that cannot be evidenced.

Wait for harmonised standards for the AI Act. The timeline is unknown, while procurement requirements exist now. The hope is that the artefacts can be remapped when the standards arrive.

Certify immediately. Too early. There is no operational evidence to audit.

How the AI Act maps to the management system

No harmonised standard for the AI Act has been published yet. ISO 42001 conformity is therefore treated as supporting evidence, not as a presumption of regulatory compliance.

AI Act mapped to the management system

Classification in the AI Act	Does it apply to us?	Coverage in the management system
Prohibited AI, Article 5	No	Prohibition list in AI policy section 7
High-risk AI, Annex III	Conditional	Impact assessment (A.5), lifecycle (A.6), data governance (A.7), event logging, human approval points
Limited risk and transparency, Article 50	Yes	AI policy section 7 and user-information requirements in the lifecycle process
Minimal risk	Yes	Current normal case. Still covered by the management system
GPAI model obligations	No	We are a downstream deployer. Responsibility sits with the model provider and is traced via architecture decisions and DPA

Prohibited AI, Article 5. Does not apply to us. The prohibition list still sits in AI policy §7, so the boundary is written down somewhere.

High-risk AI, Annex III. Conditional. This is where impact assessment (A.5), lifecycle management (A.6), data governance (A.7), event logging and the human approval points sit.

Limited risk and transparency, Article 50. Applies to us. Covered by AI policy §7 and by the user-information requirement in the lifecycle process.

Minimal risk. Applies to us, and is the current normal case. It still sits inside the management system.

GPAI model obligations. Do not apply to us, because we are a downstream deployer. Responsibility sits with the model provider and is traced through architecture decisions and the DPA.

Civisto and Annex III

Civisto supports Swedish municipalities in managing invasive species. Two things drive the classification.

Annex III point 5(a) covers AI that public authorities use to assess eligibility for essential public benefits. Reporting and case handling around invasive species is not a benefits decision. The current feature set is therefore assessed as not high-risk.

Municipal supervision sits closer to the boundary: confidential information, decisions that affect a property owner. A function that produces or substantially influences an administrative decision — public authority — requires a new classification before development starts.

The classification lives in the AI system register and is reconfirmed at every impact assessment. If it changes to high-risk, that is a material change that requires a CEO decision.

This should be said plainly. The assessment “not high-risk under Annex III” is an assessment, not a legal opinion. We do not have legal competence in-house to do anything else. The reasoning is documented, and the recommendation is legal review before the first municipal production go-live. Interpretations will likely need to be revisited as practice develops.

Twelve commitments and the mechanisms behind them

A commitment without a mechanism is an ambition. Each of these therefore points to what makes it auditable:

1. Human accountability. An AI system does not merge, deploy, release or invoice. A named person approves.
2. Correct jurisdiction. Production inference on personal data or municipal confidentiality runs only on models that have passed our approval process.
3. Sensitive data stays in the EU. Protected information does not go to tools outside the EU. Perplexity and Cursor are used for open, internal or synthetic material.
4. Transparency. The user is told when they meet an AI system or receive an AI-generated result.
5. Fitness for purpose. No AI result reaches a customer without a verification step proportionate to the risk.
6. Impact assessment before go-live. No processing of personal data or confidential data until the assessment is approved.
7. Data provenance. Source, licence and classification are recorded for all data in RAG context, evaluation sets and prompts.
8. Fairness. Systems that affect individuals are reviewed for disparate outcomes in a Swedish municipal context, including language and geography.
9. Traceability. AI-assisted changes can be traced to a work package, commit and CI run.
10. Contestability. Where AI contributes to a decision about a person, the municipality retains the ability to review, explain and change it.
11. Proportionate resource use. Model size is chosen for the task, not for what happens to be available.
12. Lifecycle discipline. Systems, models and skills have owners, versions and review dates, and are decommissioned deliberately.

How well the commitments are followed over time we will know only after the first internal audit.

The model approval process

Point two deserves a detour, because it is the question municipalities ask most sharply.

There is no official list of approved models to lean on. No authority publishes one, and the landscape changes too fast for a list to hold. What is needed is therefore not a list but a process, and criteria that can be applied to a model that did not exist when the process was written.

Our criteria for a model to process personal data or municipal confidentiality in production:

• Jurisdiction. The provider is established under EU law and inference runs on EU infrastructure. An EU data centre is not enough if the parent company is subject to the US CLOUD Act.
• Data processing agreement. A signed DPA with a legal basis under GDPR Article 28.
• No training on customer data. Explicitly confirmed by the provider, not assumed.
• Sufficient capability for the task. The model must be able to do what it is actually supposed to do. That is a requirement, not an afterthought.
• Documented decision. Approval is recorded in an architecture decision with reasoning and date, so it can be reviewed later.

Every AI skill allowed to run in production declares its data classification and approved models in its frontmatter, and approval is required before it is set to active.

Development and acceptance environments are a different matter. There any provider may be used, because no real personal or confidential data is present. That boundary is more important than which model happens to be primary at the moment.

For a procurer the distinction is decisive. A list goes stale within a quarter. A process with criteria and documented decisions can be audited, and it still gives an answer about a model none of us had heard of when the policy was written.

The lifecycle: six steps, five decision gates

The principle is that a system does not advance because it works, but because a named person has confirmed that the criteria are met, and because the evidence sits on the right work package.

The lifecycle

1. 01Design
2. G1
3. 02Development
4. G2
5. 03Verification and validation
6. G3
7. 04Deployment
8. G4
9. 05Operation and monitoring
10. G5
11. 06Decommission

A system does not advance because it works, but because a named person has confirmed that the criteria are met.

The step that changed most in practice is the third: verification and validation. Unit tests are not enough. For systems with an impact assessment the following are required:

• Evaluation sets. Realistic cases, including hard and ambiguous ones, versioned in git with the system.
• Accuracy against a threshold. Measured against the threshold set at G1. A threshold set after seeing the result is not a threshold.
• Hallucination check. How often the system answers with certainty and wrongly where the right answer would have been “don’t know”.
• Refusal behaviour. The system prefers to abstain rather than guess when input sits outside its scope.
• Fairness slices. Accuracy reported per dimension, not only in aggregate.
• Robustness. Behaviour under malformed input, prompt injection and provider failure.
• Human spot checks. A qualified person reviews a sample of real outcomes and records a judgement, regardless of what the automated tests show.
• Transparency check. That the information to the end user exists and is understandable.

A failed threshold stops at G3. It is not decided in a pull-request comment.

In operation a change table governs when revalidation is required. A prompt adjustment within the same purpose is a minor change. A new model version from the same provider goes back to step three. A provider change, a new data class or a new user group goes back to step one.

One thing has turned out more valuable than expected: recording the occasions when the system was wrong but a human caught it. That log is both evidence that the control works and data on how often the system actually misses. Without it we would only have a gut feeling.

What it has cost

This part is probably the most useful for anyone facing the same choice, so it deserves space.

The impact assessment became a hard gate before Civisto may process personal data or confidential data. That is a real schedule dependency, and it is outstanding right now. The hard trade-off is to make it light enough to run often and still solid enough to serve as evidence. If it becomes too heavy it stops being done in time, which is exactly the behaviour it is meant to prevent. Our next iteration is to scale the assessment’s scope by risk class instead of running the same template for everything.

Verification and validation draw more resources than a test suite. Versioned evaluation sets, fairness slices and manual spot checks cost time every time something changes. Automating that quality control is the most interesting technical challenge in the whole effort, and the one we put most energy into right now. Parts of it can be built away with automated evaluation runs in CI. The human spot check should not be automated away, because it is the one that catches what the automated tests do not see. The question is where the line sits, and there we would gladly compare notes with others who have built similar pipelines.

Two statements of applicability have to be kept in sync. Annex A numbering also needs to be checked line by line against the purchased standard before any certification audit becomes relevant. The control numbers in our current statement of applicability were written from working knowledge of the standard and are flagged for reconciliation. Themes and implementation references we judge as correct regardless.

Data provenance had to be backfilled. Frontmatter metadata had to be added to all brain repositories after the fact, a time-consuming manual job that should have been planned from the start.

The administrative weight is noticeable. In a small company bureaucracy is felt immediately, and the trade-off between governance and mobility is ongoing. We do not yet know whether we have set the level right.

This is a first attempt. It is entirely possible that someone else has solved parts of it considerably more smoothly.

The minimum from the start

If only one thing is done: do the inventory. Everything else depends on knowing what actually exists.

Nine things make up the minimum.

• AI inventory. Every AI system and agent listed with a named owner.
• Classification. AI Act class and data class recorded before production.
• Impact assessment. Complete before personal or confidential data is processed.
• Human control. A human approves merge, deploy and invoice, without exception.
• Model choice. A documented approval process with criteria for jurisdiction, DPA and training on customer data.
• Data provenance. Source, licence and classification for all RAG and evaluation data.
• Transparency. The end user is told when they meet AI.
• Logging. AI-assisted actions traceable to a work package and CI run.
• Incident handling. AI failures are handled in the ordinary incident process.

The summary for a procurer

Lidodevelopment operates an AI management system adapted to ISO/IEC 42001, integrated with the existing ISO 27001 ISMS and ISO 9001 QMS. AI systems are inventoried, impact-assessed, risk-rated and lifecycle-managed, with mandatory human approval points. Models that process personal data or municipal confidentiality in production pass a documented approval process requiring EU jurisdiction, a signed DPA and no training on customer data. Civisto and the AI agent chain are the first stated scope.

The point is not the certificate. The point is that when the question comes, from a municipality, an auditor or a supervisory authority, there is a written, versioned line of reasoning tied to evidence. Whether it holds we will know only when it is tested.