Why a 99% maritime anomaly score may tell you very little

In brief

• Metrics from different AIS anomaly papers are often not comparable because label sources and study regions differ.
• Rule-derived, pseudo-labelled and synthetic anomalies can make classification easier than real operational detection.
• A useful benchmark must document label provenance, independent evidence, temporal/region splits and uncertainty.

A maritime anomaly detector can report high classification accuracy while remaining weakly validated for operational use.

The central issue is not the metric itself, but what is treated as ground truth.

Four label regimes I keep separate

Independent observationexternal operational/event record that did not come from the detector definition.

Expert annotationa human interprets the same trajectory; useful, but subjective and context-dependent.

Rule / pseudo-labelthresholds or clustering generate labels, then a model learns them.

Synthetic anomalyconstructed trajectories/events inserted into real background traffic.

These regimes answer different questions. They should not be placed in one ranking table as if the F1 values measure the same thing.

What the literature pack reveals

The literature review behind this note found only one shared test set across several compared loitering methods: 137 US West Coast vessels, with labels from one expert. On that set, geometrical trajectory features achieved 0.99 accuracy / 0.96 F-score, while the earlier spatiotemporal score achieved 0.97 / 0.92. Useful comparison—but still one region, one period and one labeller.

Other papers report stronger-looking numbers under different evidence regimes. Liu et al. report around 99.8% aggregate accuracy/F1 on anomaly classes created from the authors’ own rules plus annotation. A 2026 LSTM–Transformer study reports strong loitering F1 on synthetic circular anomalies inserted into real Danish trajectories. A 2025 Transformer reconstruction study reports classification against pseudo-labels derived from spectral clustering.

The key limitation is that a high score may primarily measure agreement with the label-generation process rather than recovery of the underlying maritime event.

A benchmark I propose to trust

I propose to require every result table to include:

• data source and geographic region;
• train/test time periods;
• vessel-type coverage;
• label source and number of independent annotators;
• whether anomalies are natural, rule-derived or synthetic;
• whether the same vessel appears in train/test;
• whether data gaps are excluded;
• event-level as well as point/segment metrics;
• calibration and abstention performance;
• external evidence available for a subset.

For behaviour models, I propose to also report performance separately in ports/anchorages, approaches, open sea and high-density traffic. The same motion can mean different things in each context.

The dark-data problem

Some methods discard trajectories with large AIS gaps before detection. That makes modelling easier, but it systematically removes one of the behaviours a maritime-intelligence system may care about most.

A benchmark for anomaly/dark-shipping research should therefore include a missingness axis: regular reception, sparse reporting, abrupt gaps, reconstructed gaps and genuinely unmatched non-cooperative detections.

Causal validity before architectural novelty

A benchmark should test whether the model recovers the underlying event rather than reproducing the rule that generated the label. Where possible, I propose to include variables that explain competing causes—port state, receiver coverage, vessel type, weather/current proxies and independent observations—then evaluate whether model conclusions remain stable under controlled counterfactual changes.

This is a stricter criterion than classification accuracy: does the representation preserve the distinction between cause, context and observation?

Reproducibility before architectural novelty

The literature pack also highlights a useful opportunity: Danish Maritime Authority and Marine Cadastre data recur across papers. That makes them ideal for a reproducible benchmark where multiple rule, classical ML, deep-learning and world-model approaches are rerun under the same event definitions and evaluation split.

That contribution may be more scientifically valuable than another architecture with a higher headline score.

References

• Wijaya & Nakamura (2023) — loitering detection from AIS characteristics.
• Wijaya & Nakamura (2024) — unexpected trajectory detection from geometrical features.
• Liu, Li & Liu (2024) — kinematic anomaly classification.
• Tao et al. (2025) — Transformer reconstruction and loitering-pattern classification.