Putting that logic behind a small, well tested web service solves this. There is one implementation, one set of tests, and every consumer (dashboards, notebooks, rating tools, even Excel) calls the same thing. This post shows how to build such a service with FastAPI, using a layer loss cost calculation as the example.
FastAPI is a good fit for analytics teams already working in Python: it uses type hints for validation through Pydantic, generates interactive documentation automatically, and is fast enough for almost any internal workload.
The most important design decision is to keep the business logic in plain functions with no knowledge of HTTP. That makes it easy to test, reuse in notebooks, and review.
Here is the expected loss to an excess of loss layer when individual loss sizes follow a lognormal distribution. It uses the closed form limited expected value of the lognormal (see Klugman, Panjer and Willmot, Loss Models):
# pricing/layers.py
import math
from scipy.stats import norm
def lognormal_lev(limit: float, mu: float, sigma: float) -> float:
"""Limited expected value E[min(X, limit)] for X ~ Lognormal(mu, sigma)."""
if limit <= 0:
return 0.0
log_l = math.log(limit)
return (
math.exp(mu + sigma**2 / 2) * norm.cdf((log_l - mu - sigma**2) / sigma)
+ limit * (1 - norm.cdf((log_l - mu) / sigma))
)
def layer_expected_loss(
attachment: float, limit: float, frequency: float, mu: float, sigma: float
) -> float:
"""Expected annual loss to a per-risk layer `limit` xs `attachment`."""
per_loss = lognormal_lev(attachment + limit, mu, sigma) - lognormal_lev(attachment, mu, sigma)
return frequency * per_loss
Request and response models define the contract. Pydantic rejects bad input with a clear error before it reaches the maths, which is exactly where validation belongs:
# api/schemas.py
from pydantic import BaseModel, Field
class LayerRequest(BaseModel):
attachment: float = Field(ge=0, description="Layer attachment point")
limit: float = Field(gt=0, description="Layer limit")
frequency: float = Field(ge=0, description="Expected ground-up losses per year")
mu: float = Field(description="Lognormal mu (log scale)")
sigma: float = Field(gt=0, description="Lognormal sigma (log scale)")
class LayerResponse(BaseModel):
expected_loss: float
rate_on_line: float
model_version: str
The endpoint does nothing except translate between HTTP and the pure function:
# api/main.py
from fastapi import FastAPI
from api.schemas import LayerRequest, LayerResponse
from pricing.layers import layer_expected_loss
MODEL_VERSION = "2026.11.0"
app = FastAPI(title="Pricing Analytics API", version=MODEL_VERSION)
@app.post("/v1/layers/expected-loss", response_model=LayerResponse)
def expected_loss(req: LayerRequest) -> LayerResponse:
el = layer_expected_loss(req.attachment, req.limit, req.frequency, req.mu, req.sigma)
return LayerResponse(
expected_loss=el,
rate_on_line=el / req.limit,
model_version=MODEL_VERSION,
)
@app.get("/health")
def health() -> dict:
return {"status": "ok", "model_version": MODEL_VERSION}
Run it with uvicorn api.main:app --reload and open /docs for interactive documentation that underwriters and analysts can try directly.
Two details are worth copying:
/v1/...). When the contract has to change, add /v2 and retire /v1 gracefully instead of breaking every consumer at once.Test the maths against known values and properties, and test the API contract separately:
# tests/test_layers.py
import math
import pytest
from fastapi.testclient import TestClient
from api.main import app
from pricing.layers import lognormal_lev, layer_expected_loss
def test_lev_tends_to_mean_for_large_limits():
mu, sigma = 13.0, 1.2
assert lognormal_lev(1e15, mu, sigma) == pytest.approx(math.exp(mu + sigma**2 / 2), rel=1e-6)
def test_stacked_layers_add_up():
args = dict(frequency=10, mu=13.0, sigma=1.2)
lower = layer_expected_loss(1e6, 4e6, **args)
upper = layer_expected_loss(5e6, 5e6, **args)
combined = layer_expected_loss(1e6, 9e6, **args)
assert lower + upper == pytest.approx(combined)
client = TestClient(app)
def test_rejects_negative_attachment():
r = client.post("/v1/layers/expected-loss", json={
"attachment": -1, "limit": 1e6, "frequency": 5, "mu": 13, "sigma": 1.2,
})
assert r.status_code == 422
def test_returns_version():
r = client.post("/v1/layers/expected-loss", json={
"attachment": 1e6, "limit": 4e6, "frequency": 5, "mu": 13, "sigma": 1.2,
})
assert r.status_code == 200
assert r.json()["model_version"]
Property tests like “stacked layers add up to the combined layer” catch subtle errors that a single hard coded expected value would miss. Run the suite in CI on every pull request.
However good a web app is, many underwriters and actuaries live in Excel. Office Add-ins support custom functions written in JavaScript, which can call your API and return the result into a cell like any built in function:
/**
* Expected annual loss to a layer.
* @customfunction
* @param {number} attachment
* @param {number} limit
* @param {number} frequency
* @param {number} mu
* @param {number} sigma
* @returns {Promise<number>}
*/
async function LAYERLOSS(attachment, limit, frequency, mu, sigma) {
const response = await fetch("https://pricing-api.internal/v1/layers/expected-loss", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ attachment, limit, frequency, mu, sigma }),
});
if (!response.ok) {
throw new CustomFunctions.Error(
CustomFunctions.ErrorCode.invalidValue,
`Pricing API error ${response.status}`
);
}
return (await response.json()).expected_loss;
}
Now =LAYERLOSS(1000000, 4000000, 5, 13, 1.2) in a cell calls the same tested logic as every other tool. In production you would add authentication (for example Microsoft Entra ID) so only authorised users can call the API.
A service is not about technology for its own sake. It is about making sure that a pricing or reserving calculation exists once, is tested, is versioned, and gives the same answer wherever it is used. FastAPI and Pydantic make that achievable for any analytics team comfortable with Python.
This post covers the decisions and habits that have mattered most when moving actuarial pipelines, such as reserving and loss triangle data, from locally run scripts onto scheduled Fabric pipelines.
Both store Delta tables in OneLake, so this is mainly a question of how your team works:
| If your team mostly… | Lean towards |
|---|---|
| writes Python and PySpark, works with files and semi structured data | Lakehouse |
| writes T-SQL, wants stored procedures and multi table transactions | Warehouse |
| needs both | Lakehouse for ingestion and transformation, Warehouse or SQL endpoint for serving |
For actuarial teams with a mix of Python and SQL skills, a lakehouse for ingestion and transformation with SQL access for analysts has worked well.
Microsoft’s guidance recommends a medallion architecture: bronze for raw data as received, silver for cleaned and conformed data, gold for business ready models. The pattern is only useful if each layer has a clear contract:
Keeping bronze immutable means that when a transformation bug is found, you can fix it and replay history rather than asking source teams for files again.
Scheduled pipelines fail, get rerun, and receive corrected files. Every load from bronze to silver should be safe to run twice. Delta Lake’s MERGE makes this straightforward in a Fabric notebook:
from pyspark.sql import functions as F
from delta.tables import DeltaTable
valuation_date = "2026-09-30" # passed in as a pipeline parameter in practice
incoming = (
spark.read.option("header", True)
.csv(f"Files/bronze/claims/{valuation_date}/*.csv")
.withColumn("paid_to_date", F.col("paid_to_date").cast("decimal(18,2)"))
.withColumn("outstanding", F.col("outstanding").cast("decimal(18,2)"))
.withColumn("valuation_date", F.lit(valuation_date).cast("date"))
.withColumn("_loaded_at", F.current_timestamp())
.dropDuplicates(["claim_id", "valuation_date"])
)
target = DeltaTable.forName(spark, "silver_claim_snapshot")
(
target.alias("t")
.merge(
incoming.alias("s"),
"t.claim_id = s.claim_id AND t.valuation_date = s.valuation_date",
)
.whenMatchedUpdateAll()
.whenNotMatchedInsertAll()
.execute()
)
Rerunning for the same valuation date updates rows rather than duplicating them. The same pattern, keyed on the documented grain, works for premium and exposure data.
A merge will happily write bad data. Add checks between reading and merging, and fail the notebook with a clear message so the pipeline run shows as failed:
problems = []
if incoming.filter(F.col("claim_id").isNull()).limit(1).count() > 0:
problems.append("null claim_id")
if incoming.filter(F.col("paid_to_date") < 0).limit(1).count() > 0:
problems.append("negative paid_to_date")
if problems:
raise ValueError(f"Claims load {valuation_date} rejected: {', '.join(problems)}")
Delta tables need some housekeeping to stay fast:
OPTIMIZE periodically to compact them.VACUUM removes files no longer needed beyond your retention period. Keep retention long enough for any “as at” queries you rely on.The Delta Lake paper by Armbrust et al. (2020) explains why these operations matter: the transaction log and file layout are what give Delta its reliability and performance.
Fabric supports Git integration (Azure DevOps or GitHub) for workspace items and deployment pipelines to promote content between development, test and production workspaces. Use them:
Keeping business logic in version controlled notebooks or Python modules, rather than in manual edits in production, is what makes a pipeline auditable.
Fabric runs on capacities, and every workload (Spark, SQL, Power BI refreshes) draws from the same pool. A heavy notebook scheduled at the same time as the morning dashboard refresh can slow everything down. The Fabric Capacity Metrics app shows where capacity goes. Stagger schedules, right size Spark sessions, and avoid rerunning full history when an incremental load would do.
Once gold tables are in good shape, a Direct Lake semantic model lets Power BI read them straight from OneLake. Dashboards see fresh data without a separate import refresh, and measures are defined once in the semantic model rather than in every report. Gold tables designed as a clean star schema (see the data modelling post) make Direct Lake models far simpler.
MERGE on the documented grain.OPTIMIZE and a considered VACUUM policy.This post walks through a dimensional model for premium and claims analytics, following the approach in Kimball and Ross’s The Data Warehouse Toolkit, and adapted for specialty insurance.
Before drawing any tables, list the questions the model must answer. For a specialty portfolio they usually include:
Each question tells you something about the grain, the dimensions and the history you need.
A dimensional model separates facts (measurable events, usually numeric) from dimensions (the descriptive context you slice by).
A sensible core for specialty insurance:
Facts
fact_premium_transaction: one row per premium movement, per policy section, per layer, per currency.fact_claim_movement: one row per claim payment or reserve change, per claim, per currency.fact_exposure_snapshot: one row per risk, per reporting date, with limits and sums insured at our share.Dimensions
dim_policy (with section and layer attributes), dim_class (line of business hierarchy), dim_broker, dim_insured, dim_geography, dim_currency, and a dim_date used in several roles (transaction date, inception date, loss date). dim_date (as transaction / inception / loss date)
|
dim_broker --- fact_premium_transaction --- dim_policy --- dim_class
|
dim_currency
It is tempting to store one row per policy with “total premium” and “total incurred” columns. It feels simpler, but it throws away the information you need most:
Storing movements (each payment, each reserve change, each additional premium) keeps all of that. Totals at any date are then just a sum of movements up to that date:
-- Incurred to date, as at any chosen valuation date
SELECT
c.claim_id,
SUM(m.paid_amount) AS paid_to_date,
SUM(m.reserve_change) AS outstanding,
SUM(m.paid_amount + m.reserve_change) AS incurred
FROM fact_claim_movement AS m
JOIN dim_claim AS c ON c.claim_key = m.claim_key
WHERE m.transaction_date <= @valuation_date
GROUP BY c.claim_id;
With claim movements in place, a reserving triangle is a straightforward aggregation. Development period is the gap between the start of the underwriting year and the transaction date:
WITH movements AS (
SELECT
p.underwriting_year,
-- development quarter, 1 = first quarter of the underwriting year
(YEAR(m.transaction_date) - p.underwriting_year) * 4
+ DATEPART(QUARTER, m.transaction_date) AS dev_quarter,
m.paid_amount + m.reserve_change AS incurred_movement
FROM fact_claim_movement AS m
JOIN dim_policy AS p ON p.policy_key = m.policy_key
WHERE m.transaction_date <= @valuation_date
)
SELECT
underwriting_year,
dev_quarter,
SUM(SUM(incurred_movement)) OVER (
PARTITION BY underwriting_year ORDER BY dev_quarter
ROWS UNBOUNDED PRECEDING
) AS cumulative_incurred
FROM movements
GROUP BY underwriting_year, dev_quarter
ORDER BY underwriting_year, dev_quarter;
Pivot the result and you have the cumulative incurred triangle that chain ladder methods, including Mack’s (1993) distribution free model for reserve uncertainty, start from. Because the triangle is derived from movements rather than stored, you can rebuild it at any past valuation date simply by changing @valuation_date.
(The date functions above are T-SQL; Spark SQL uses year() and quarter() but the logic is identical.)
Dimension attributes change: a broker is acquired, a policy is reassigned to a different class, a country’s risk rating changes. The question is whether reports should show the attribute as it is now or as it was at the time.
Kimball’s slowly changing dimension types give you the vocabulary:
valid_from, valid_to and is_current. Reports can use the value at the time of each fact.-- Type 2 lookup: the broker attributes that applied on each transaction date
SELECT f.*, b.broker_group
FROM fact_premium_transaction AS f
JOIN dim_broker AS b
ON b.broker_id = f.broker_id
AND f.transaction_date >= b.valid_from
AND f.transaction_date < b.valid_to;
For most insurance hierarchies (class of business, broker group, territory), Type 2 is the safer default, because restating history without warning is exactly what makes figures fail to reconcile between quarters.
Insurance data has two independent timelines:
Most of the time, storing transaction dates on movements is enough. But some questions, like “what did we believe the exposure was on 31 December, before late bordereaux arrived?”, need both. Snodgrass’s work on time oriented databases describes this bitemporal approach in detail. You do not need it everywhere, but it is worth recognising when a question needs it.
A dimension is conformed when every fact table uses the same version of it. If premium and claims both join to the same dim_class, a loss ratio by class is just two aggregations side by side. If premium uses one class mapping and claims another, every loss ratio becomes a reconciliation exercise. Conformed dimensions are the cheapest way to make figures agree across reports.
The London market, centred on Lloyd’s and the company market around it, is built around subscription. A large risk is rarely written by a single insurer. A broker places it with a lead underwriter, who sets the terms, and then with following markets who each take a share. The same risk might be split into layers (for example the first 10m, then 15m excess of 10m), each with its own panel of insurers.
That structure has direct consequences for data:
Business comes in through two broad routes, and the data looks quite different for each.
Open market business is placed risk by risk through brokers. Data usually comes from your own policy administration system and from market messaging.
Delegated authority business is written by coverholders or managing general agents under a binding authority. You typically learn about individual risks and claims through bordereaux: periodic spreadsheets listing the risks bound and claims handled. Bordereaux quality varies enormously, which is why the Lloyd’s Coverholder Reporting Standards exist to define the expected fields. Mapping every incoming bordereau into that standard, even approximately, gives you a stable internal shape to build on.
For years, much of the market’s data moved as documents and spreadsheets. Lloyd’s Blueprint Two programme aims to change that, with the Core Data Record (CDR) defining a standard set of structured data captured at placement, so that downstream processes like accounting, settlement and regulatory reporting can use the same data. The ACORD data standards have long been the reference for insurance messaging and are a useful vocabulary for internal models too.
For analytics teams, the direction of travel is clear: more structured data at the point of placement, and more expectation that internal models line up with market standards. Designing your internal data model with those standards in mind will save painful remapping later.
London market business is written in many currencies. A single portfolio report might include USD, GBP, EUR, CAD and JPY premium. Decide, and document, three things:
Mixing rate bases is a classic source of “the numbers don’t match finance”. Store original amounts, apply conversion in one tested place, and label every report with its rate basis.
A single claim or policy has several dates that answer different questions:
| Date | Question it answers |
|---|---|
| Inception / expiry | When is the policy on risk? |
| Underwriting year (year of account) | Which year’s book does it belong to? |
| Loss date | When did the event happen? |
| Reported date | When did we hear about it? |
| Transaction / booked date | When was the movement recorded? |
| As at date | What did the data look like at a point in time? |
Reserving triangles, for example, are built from underwriting or accident year against development period, using data as at successive valuation dates. If you cannot reconstruct what the data looked like at a past date, you cannot rebuild last quarter’s triangle to check it. That is why keeping history explicitly (snapshots or effective dated records) is not optional.
Premium and claims arrive as a stream of transactions: original premium, additional and return premiums, adjustments, payments, reserve movements and recoveries. Agree a sign convention (for example, inwards premium positive, return premium negative) and enforce it at ingestion. A surprising number of reconciliation breaks come from one source treating returns as negative and another as positive with a type flag.
London market data rewards curiosity. Every odd value usually reflects something real about how a risk was placed, written or settled. The analysts who do best are the ones who learn the business behind the fields, and then build pipelines that make those rules explicit and repeatable.
GLMs became the industry standard for personal lines pricing because they fit the problem well:
Goldburd, Khare, Tevet and Guller’s CAS monograph is the standard practical reference.
Tree based ensembles such as gradient boosting machines (Chen and Guestrin’s XGBoost being the best known) are good at finding interactions and non-linear effects that are tedious to specify by hand in a GLM. Henckaerts et al. (2021) showed that boosted trees can produce competitive insurance tariffs and offered ways to extract interpretable insights from them.
Useful applications include:
The comparison only means something if both models are trained on the same target, with the same exposure handling and the same validation scheme. A minimal example with scikit-learn:
import numpy as np
from sklearn.linear_model import TweedieRegressor
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.metrics import mean_tweedie_deviance
from sklearn.model_selection import train_test_split
# X: rating factors (numeric or one-hot encoded), y: incurred loss, exposure: years at risk
X_train, X_test, y_train, y_test, e_train, e_test = train_test_split(
X, y, exposure, test_size=0.25, random_state=0
)
# Model pure premium (loss per unit exposure), weighting each row by its exposure
glm = TweedieRegressor(power=1.5, link="log", alpha=1e-4, max_iter=1000)
glm.fit(X_train, y_train / e_train, sample_weight=e_train)
gbm = HistGradientBoostingRegressor(loss="poisson", max_iter=500, learning_rate=0.05)
gbm.fit(X_train, y_train / e_train, sample_weight=e_train)
for name, model in [("GLM", glm), ("GBM", gbm)]:
pred = model.predict(X_test)
dev = mean_tweedie_deviance(y_test / e_test, pred, sample_weight=e_test, power=1.5)
print(f"{name}: Tweedie deviance {dev:.4f}")
Two things to note. First, use a deviance that matches the data (Tweedie or Poisson), not mean squared error, which is dominated by a few large claims. Second, look beyond one summary metric: lift charts and double lift charts comparing the two models by predicted decile are what pricing teams actually use to decide whether a model is better.
Pricing decisions affect customers and are scrutinised by regulators. In the UK, the FCA’s general insurance pricing practices rules (PS21/5) are a reminder that how prices are set matters, not just whether they are accurate. Any model used in pricing needs to be explainable to underwriters, compliance and, where relevant, customers.
Tools that help:
Most published work on machine learning pricing uses personal lines data with hundreds of thousands of policies. Specialty and London market business looks very different:
In this setting, the most valuable “machine learning” is often modest: well engineered features, credibility weighting between portfolio experience and market benchmarks, GLMs with sensible constraints, and good tooling that lets underwriters see the drivers behind a technical price. Wüthrich and Merz (2023) cover the statistical foundations that link classical actuarial models with modern machine learning, and it is a good reminder that the two traditions have more in common than the hype suggests.
These are the principles I have found matter most.
The grain of a table is what one row represents. “One row per policy” sounds clear until you meet layered placements, endorsements, multiple sections, multiple currencies and several participating syndicates on the same risk. Many reporting errors come from joining two tables at different grains and silently duplicating premium.
Write the grain down for every table, for example: one row per policy section, per layer, per transaction, per currency. Then enforce it with a uniqueness test on those columns.
An idempotent pipeline produces the same result whether it runs once or five times. This matters because pipelines fail halfway, get rerun by hand, and receive corrected files for a period that has already been loaded.
Practical patterns:
MERGE on a natural key instead of blind inserts.Bad data is cheapest to catch the moment it arrives. A bordereau that is missing its inception dates, or has premium in the wrong currency, should fail loudly before it reaches a dashboard.
A lightweight validation step in pandas goes a long way:
import pandas as pd
REQUIRED = ["policy_ref", "inception_date", "expiry_date", "currency", "gross_premium"]
VALID_CCY = {"GBP", "USD", "EUR", "CAD", "AUD", "JPY"}
def validate_bordereau(df: pd.DataFrame, source: str) -> pd.DataFrame:
errors = []
missing = [c for c in REQUIRED if c not in df.columns]
if missing:
raise ValueError(f"{source}: missing columns {missing}")
if df["policy_ref"].isna().any():
errors.append("blank policy_ref")
dupes = df.duplicated(subset=["policy_ref", "section"], keep=False)
if dupes.any():
errors.append(f"{dupes.sum()} duplicate policy/section rows")
bad_dates = df["expiry_date"] <= df["inception_date"]
if bad_dates.any():
errors.append(f"{bad_dates.sum()} rows with expiry on or before inception")
bad_ccy = ~df["currency"].isin(VALID_CCY)
if bad_ccy.any():
errors.append(f"unknown currencies: {sorted(df.loc[bad_ccy, 'currency'].unique())}")
if errors:
raise ValueError(f"{source}: " + "; ".join(errors))
return df
Libraries such as pandera or Great Expectations formalise this with declarative schemas and readable reports, and are worth adopting once the number of checks grows. The principle is the same: reject or quarantine bad input with a clear message, never silently coerce it.
For delegated authority business, the Lloyd’s Coverholder Reporting Standards define the expected fields for risk and claims bordereaux. Even where a coverholder does not follow them exactly, mapping incoming files to that standard gives you a stable internal shape.
Validation checks that data is internally consistent. Reconciliation checks that it agrees with the outside world. Every important dataset should reconcile to a known control total: written premium to the finance ledger, paid claims to the claims system, policy counts to the underwriting system.
Automate the reconciliation, store the result alongside the data, and show it on the report. A dashboard that says “reconciles to ledger within 0.1%” earns far more trust than one that does not mention it.
Actuarial work constantly asks “what did we know at the time?”. Reserving uses data as at each quarter end; pricing reviews compare this year’s assumptions with last year’s. If your tables only hold the current state, those questions become impossible to answer.
Two common approaches:
valid_from and valid_to dates. More compact, but queries need care.The data modelling post goes deeper on this, including the difference between when something happened and when we recorded it.
Rules such as “how to convert to settlement currency”, “which rate of exchange to use” or “how to allocate premium across sections” should live in one well tested place, not be repeated in five SQL scripts and three notebooks. Put them in pure functions or in a single curated layer, write unit tests for them, and have everything else call that.
When an underwriter asks why a number changed, you need to be able to trace it back: which file, which load, which version of the transformation. At minimum, stamp every row with the source file, the load timestamp and the pipeline version. Platform tools can add richer lineage on top, but these three columns answer most questions.
Version control, code review, automated tests and CI/CD are not just for application developers. A pricing dataset that feeds underwriting decisions deserves the same discipline as a production web service. Reis and Housley (2022) describe this as treating data as a product with owners, contracts and service levels, which is a useful framing for actuarial data.
Linear (Pearson) correlation is a single number, and many very different joint behaviours share the same correlation. Embrechts, McNeil and Straumann (2002) set out the problems clearly:
Sklar’s theorem (1959) says that any joint distribution can be split into its marginal distributions and a copula that describes how they move together. In practice that gives a clean workflow:
Rank based measures such as Kendall’s tau and Spearman’s rho depend only on the copula, not the marginals, which makes them the natural tools for calibration.
The two copulas you meet most often are the Gaussian and the Student t. With the same correlation parameter they look similar in the middle of the distribution, but they behave very differently in the extremes.
The coefficient of upper tail dependence measures the probability that one variable is extreme given that the other is extreme, in the limit. For the Gaussian copula it is zero for any correlation below 1: extremes eventually become independent. For the t copula it is positive and grows as the degrees of freedom fall. McNeil, Frey and Embrechts (2015) give the formulas and derivations.
That single fact explains why a Gaussian copula can understate the chance of two lines having a bad year together.
import numpy as np
from scipy import stats
rng = np.random.default_rng(seed=7)
n = 200_000
rho = 0.5
corr = np.array([[1.0, rho], [rho, 1.0]])
chol = np.linalg.cholesky(corr)
# Gaussian copula: correlated normals mapped to uniforms
z = rng.standard_normal((n, 2)) @ chol.T
u_gauss = stats.norm.cdf(z)
# Student t copula with 4 degrees of freedom:
# same correlated normals, divided by a shared chi-square mixing variable
nu = 4
w = rng.chisquare(nu, size=(n, 1)) / nu
t_samples = z / np.sqrt(w)
u_t = stats.t.cdf(t_samples, df=nu)
def joint_exceedance(u, q):
"""Probability that both variables exceed their q-quantile."""
return np.mean((u[:, 0] > q) & (u[:, 1] > q))
for q in (0.9, 0.99, 0.995):
print(
f"q={q}: independent {(1-q)**2:.5f} "
f"gaussian {joint_exceedance(u_gauss, q):.5f} "
f"t(4) {joint_exceedance(u_t, q):.5f}"
)
Run it and you will see the three columns agree reasonably at the 90th percentile, then diverge sharply at 99.5%, with the t copula producing several times more joint extremes than the Gaussian. Same correlation, very different capital.
Once you have dependent uniforms, apply each line’s fitted marginal:
# Marginals fitted separately for each line (illustrative parameters)
property_cat = stats.lognorm(s=1.4, scale=8e6)
marine_cargo = stats.genpareto(c=0.3, scale=2e6)
losses_property = property_cat.ppf(u_t[:, 0])
losses_marine = marine_cargo.ppf(u_t[:, 1])
portfolio = losses_property + losses_marine
print(f"1-in-200 portfolio loss: {np.quantile(portfolio, 0.995)/1e6:.1f}m")
Swap u_t for u_gauss and compare. The difference is a direct measure of how much your choice of copula is driving the tail.
For elliptical copulas there is a convenient link between Kendall’s tau and the correlation parameter: rho = sin(pi * tau / 2). Calibrating via rank correlation is more robust than using Pearson correlation on raw losses:
def rho_from_kendall(x, y):
tau, _ = stats.kendalltau(x, y)
return np.sin(np.pi * tau / 2)
The degrees of freedom of a t copula are harder to estimate and usually need maximum likelihood on pseudo-observations (ranks scaled into the unit interval). With the short histories typical of specialty lines, expect the estimate to be uncertain and treat it as a judgement to be documented and stress tested, not a fact.
Copulas are a useful tool for one specific job: combining well understood marginals with a defensible dependence structure. They are not a substitute for thinking about why lines are dependent. In practice the most valuable output is often the comparison itself: showing underwriters and management how much the tail moves when dependence assumptions change.
The standard starting point is the collective risk model. Annual aggregate loss is
S = X1 + X2 + … + XN
where N is the number of losses in the year and X1, X2, … are individual loss sizes, assumed independent of each other and of N. This is covered in depth in Klugman, Panjer and Willmot’s Loss Models, and it is the right mental model even when the production version is far more elaborate.
Two modelling choices matter most:
The code below simulates 100,000 years. It avoids a Python loop over years by drawing every loss at once and then summing them back into their years with np.add.at.
import numpy as np
rng = np.random.default_rng(seed=2024) # always seed, so results are reproducible
n_years = 100_000
# Frequency: negative binomial with mean 12 and variance 24 (overdispersed)
mean_n, var_n = 12.0, 24.0
p = mean_n / var_n
r = mean_n * p / (1 - p)
counts = rng.negative_binomial(r, p, size=n_years)
# Severity: lognormal body with a mean of about 1m
mu, sigma = np.log(1_000_000) - 0.5 * 1.2**2, 1.2
losses = rng.lognormal(mu, sigma, size=counts.sum())
# Map every loss back to the year it belongs to
year_index = np.repeat(np.arange(n_years), counts)
# Gross annual aggregate loss
gross = np.zeros(n_years)
np.add.at(gross, year_index, losses)
A per-risk excess of loss layer “5m xs 2m” pays the part of each loss between 2m and 7m. Applying it per loss before aggregating is the step people most often get wrong; applying it to the annual total gives an aggregate stop loss instead, which is a different contract.
attachment, limit = 2_000_000, 5_000_000
ceded_per_loss = np.clip(losses - attachment, 0, limit)
ceded = np.zeros(n_years)
np.add.at(ceded, year_index, ceded_per_loss)
net = gross - ceded
If the layer has an annual aggregate limit or reinstatements, apply those to ceded per year after this step.
Two exceedance curves come out of a simulation like this, and they answer different questions:
# Largest single loss per year (years with no losses have a maximum of 0)
max_loss = np.zeros(n_years)
np.maximum.at(max_loss, year_index, losses)
def return_period_loss(annual_values, return_period):
"""Loss level exceeded on average once every `return_period` years."""
return np.quantile(annual_values, 1 - 1 / return_period)
def tvar(annual_values, level):
"""Average of outcomes beyond the `level` quantile (Tail Value at Risk)."""
threshold = np.quantile(annual_values, level)
return annual_values[annual_values >= threshold].mean()
for rp in (10, 50, 100, 200):
print(
f"1-in-{rp:<3} AEP gross {return_period_loss(gross, rp)/1e6:6.1f}m"
f" OEP gross {return_period_loss(max_loss, rp)/1e6:6.1f}m"
f" AEP net {return_period_loss(net, rp)/1e6:6.1f}m"
)
print(f"Average annual loss (gross): {gross.mean()/1e6:.2f}m")
print(f"99.5% TVaR (net): {tvar(net, 0.995)/1e6:.2f}m")
The 1 in 200 AEP is the 99.5th percentile of annual loss, which is the confidence level used in Solvency II style capital calculations. TVaR averages the whole tail beyond that point and is a coherent risk measure in the sense of Artzner et al. (1999), which VaR is not.
Simulation error is often ignored, but it is easy to estimate. For a mean, the standard error falls with the square root of the number of simulations:
se_aal = gross.std(ddof=1) / np.sqrt(n_years)
print(f"AAL = {gross.mean()/1e6:.3f}m +/- {1.96*se_aal/1e6:.3f}m (95% interval)")
Tail quantiles converge much more slowly than the mean because only a handful of simulations land beyond them. With 100,000 years, only 500 sit beyond the 1 in 200 point. A quick, honest check is to rerun with several seeds and look at the spread of the quantile you care about. If a 1 in 200 figure moves by more than your materiality threshold between seeds, you need more simulations or a variance reduction technique; Glasserman (2003) covers importance sampling and control variates in detail.
np.random.default_rng(seed) and record the seed.np.random.seed and global state make parallel runs hard to reproduce. The Generator API is the current recommendation in the NumPy documentation.This simple model already supports useful work: comparing reinsurance structures, sizing retentions and explaining tail risk to underwriters. Extensions worth adding, roughly in order of value, are parameter uncertainty (simulate the parameters as well as the losses), inflation and exposure trends, reinstatement terms, and dependence between lines.
Most actuarial techniques assume the past is a reasonable guide to the future once you adjust for inflation, exposure and mix. Emerging risks break that assumption in three ways:
The practical consequence is that exposure data, scenario analysis and clear documentation of assumptions become more valuable than another decimal place of precision on a fitted distribution.
Cyber is the textbook accumulation risk: a single vulnerability in widely used software or a cloud outage can trigger losses across thousands of insureds in different territories and classes. The market response has been as much about wording as pricing. Lloyd’s Market Bulletin Y5381 (August 2022) required managing agents to include clauses excluding losses from state-backed cyber attacks in standalone cyber policies from March 2023, and the Lloyd’s Market Association published model clauses (LMA5564 to LMA5567) to support that.
For an analyst, this has concrete implications:
The aircraft lessor claims arising from aircraft stranded in Russia after 2022 were a reminder that political risk and aviation war exposures can crystallise suddenly and at very large scale, and that the question of which policy responds (all risks or war risks, contingent or possessed) can matter as much as the size of the loss.
What this means for data:
Peak perils like US hurricane and Japanese earthquake dominate catastrophe model output, but much of the insured loss in recent years has come from so-called secondary perils: severe convective storms, flooding and wildfire. The Swiss Re Institute’s annual sigma reports on natural catastrophes have repeatedly highlighted this trend, alongside the effect of rising exposure values in hazard-prone areas.
For a specialty book this matters in a few ways:
Marine cargo, contingent business interruption and trade credit share a common feature: the loss often occurs far from the insured. A port closure, a canal blockage or a regional conflict can generate claims across cargo, hull, war and political risk policies simultaneously. Capturing where goods are and which routes they take is hard, and most portfolios rely on declared values, voyage-level bordereaux or storage location schedules of mixed quality.
Across all four themes, the same habits make a portfolio easier to understand:
| Habit | Why it matters |
|---|---|
| Policy-level wording and exclusion flags | Makes year-on-year exposure comparisons valid |
| Clean entity and geography reference data | Prevents double counting and missed aggregations |
| Versioned scenario definitions | Makes committee figures reproducible |
| Separate “exposure growth” from “hazard change” | Avoids reading inflation and growth as trend |
| Documented model adjustments | Keeps expert judgement visible and reviewable |
None of these are glamorous, and none require machine learning. They do require good data engineering and a close working relationship between analysts, exposure managers and underwriters.
The temptation with emerging risks is to reach for a more sophisticated model. In my experience, the bigger gains come from knowing exactly what you have written, where it is, and which wording applies. Sophisticated models built on uncertain exposure data produce confident looking numbers that are not worth very much.