Three legal AI tools each answered three Norwegian-law questions in three separate runs; Alternative 1 ran two of them again with web search. Every run is scored on which core law sources it cited, with penalties for hallucinated or outdated citations, and on nine quality aspects. Every citation was checked against its source text.
The raw data behind this analysis is available on request: all 29 answer files (3 tools × 3 questions × 3 runs, plus Alternative 1's 2 web-search runs), screen recordings of the Alternative 1 and Alternative 2 sessions, links to the Lovkart + Claude conversations, and the check of every citation.
Lovkart + Claude Opus 5.5
— / 5
mean aspect score
—%
net coverage: core sources cited, minus citation penalties
Before penalties —% · penalty points per run —
Retrieval (RAG) over legal sources + Claude Opus 5.5
— / 5
mean aspect score
—%
net coverage: core sources cited, minus citation penalties (web-search runs excluded)
Before penalties —% · penalty points per run —
Likely an index of current Norwegian law plus web search; model unknown
— / 5
mean aspect score
—%
net coverage: core sources cited, minus citation penalties
Before penalties —% · penalty points per run —
Asked word for word, in English, in three separate runs per tool.
Setup: Alternative 1 serves several jurisdictions, so its Q1 question began with “I am based in Norway:” to name the jurisdiction, and its EU law database was switched on for Q1, since EU rules matter for a software product. Lovkart + Claude and Alternative 2 are built for Norwegian law and ran on their default sources, with no choice of databases made.
Each aspect is scored 1–5 per question (5 best). Hover over or tab to a cell to read the reason. Definitions are under Method.
Each dot is one run (R1–R3) after citation penalties; a hollow dot is an Alternative 1 web-search run, and a grey tick marks a run's coverage before penalties where they differ. The bar spans R1–R3: a short bar means consistent runs.
One symbol per run, in order R1, R2, R3. Core sources count toward coverage; useful ones earn no points.
Every cited section, article and preparatory work in every run, checked against the source text. Bars show each tool's citations for the selected question by verdict; the table lists every citation that is not fully correct, leaving out those a tool itself flags as not on point.
Coverage: each core source counts 1 when a run cites it with its provision or verifies it, ½ when the run makes the point without the provision or marks it unverified, and 0 when it is absent, wrong or outdated. Penalties then apply to every citation in the run, core or not: −1 for a hallucinated citation (the section is about something else, the content is misstated, or it does not exist), −½ for an outdated or mislabelled one; off-point citations and small imprecisions cost nothing. A citation a tool shows but itself marks as not on point, to let the user know it looked, is counted as neutral and not listed as a problem. Net coverage = (points − penalties) ÷ the question's core list, so it can fall below zero. The tiles average R1–R3 per question, then the three questions.
Citation check: every cited section, article, preparatory work, act and decision in all 29 runs (567 run-citations) was checked against its text in Lovdata's consolidated text (through the Lovkart corpus), Stortinget, regjeringen.no and EUR-Lex, applying the same standard to each tool.
Aspect scores are the assessor's judgment on a 1–5 scale, with the reason for each cell in the scorecard. Completeness follows mean coverage; the thresholds are in the aspect table.
Web search: Alternative 1 was also run once with web search on Q2 and Q3, as a check. It added no core source the standard runs missed and scored lower than their mean on both questions, partly because it pulled in outdated sources, so web search was not repeated and the scores use the standard runs. Those two runs still show as hollow dots in the run-by-run chart and in the source and citation tables.
Setup: Alternative 1 serves several jurisdictions, so its Q1 question named Norway as the jurisdiction, and its EU law database was switched on for Q1. The other two tools ran on their default sources. This gave Alternative 1 extra EU sources on Q1, which bear on the Cyber Resilience Act and GDPR points; the scores take its answers as given.
Alternative 2 required a company profile for the session. Advice that came from that profile (sole-proprietorship liability) is scored neutral.
Only legal sources are compared. Non-legal web material, such as job adverts and threat assessments, is left out, since not every tool searches the web by default.
Exported files: Lovkart + Claude's PDFs link each provision to Lovdata. Alternative 1's Word exports and Alternative 2's PDFs carry no links, though their apps may show source links on screen.
Assessor: Claude. Lovkart + Claude and Alternative 1 both run on Claude Opus 5.5, so they differ mainly in their legal sources and retrieval, and two of the three tools share the assessor's model family. Scores were set against statute and preparatory-work text, not against any tool's answer. Statute text came from Lovdata's consolidated text (through the Lovkart corpus), cross-checked on stortinget.no and regjeringen.no.