How to Read a Fraud Rate: Denominators, Samples, Flagged Shares

Someone hands you a number. A vendor deck puts a share on applications carrying an altered document. A survey says fraud is rising. An industry report puts a percentage on edited bank statements. You are about to put that number in a board paper, a business case or a risk-appetite memo. Before you do, find out what the number is a fraction of.

This guide is for fraud operations and risk leads at lenders and fintechs, and for analysts and journalists who cite fraud figures. It also applies to our own monthly integrity reports, and we use those as the worked examples, because we know exactly what went into them.

A rate is a ratio

Short answer: every fraud rate is a numerator divided by a denominator. Before you cite one, ask five things. What was counted (the denominator)? Who supplied the items (the sample)? What does the label in the numerator actually mean? Over what period, with what method in force? And how large is the base? If any of the five is missing or has changed, the figure cannot carry the weight you want to put on it.

None of this makes a statistic useless. The five questions tell you what the statistic is about, which is usually narrower than the headline suggests.

What is in the denominator

The same pipeline can produce different denominators depending on what you choose to count:

• Checks – every analysis that ran, including the same file submitted twice.
• Unique documents – each distinct file counted once.
• Users or applications – each person or case counted once, however many files they sent.

These answer different questions. ‘What share of applications included an altered file’ is not the same question as ‘what share of analyses returned a flag’, and the two can differ a lot.

Our own reports show how much the denominator matters. The February and March reports covered checks submitted through the website only. February reported 418 checks, of which 169 (40.4%) were flagged as modified. March reported 866 checks and 420 flagged (48.5%). From May onwards, the reports covered checks from both the website and the API, so the base itself changed shape.

From July the reports state the denominator up front, in a section that comes before any number. It is all processed checks – completed analyses, not unique documents or unique users – and it includes our own internal and automated runs, plus retries. It is not deduplicated. The same file submitted twice counts twice.

April is the extreme case. An internal test batch entered the dataset that month, and it was large enough to swamp the organic submissions. The report published no headline file count at all, because the total would have described mostly our own testing. Its shares were computed on the same mixed base, so they carry the same caveat: dropping a count does not repair the ratio built on it.

Who is in the sample

A denominator can be defined perfectly and still describe a skewed group. Selection effects decide who ends up in the sample, and in document fraud they are strong.

People rarely run a check on a file they have no doubts about. They check the bank statement that looks slightly off, the invoice from a new supplier, the offer letter with an odd font. Developers testing an integration often send files they already know are edited, because that is how you confirm the integration returns a flag. So the files that reach any tamper-detection tool lean towards the doubtful before a single analysis runs.

This is not peculiar to us. The same logic applies to every source:

• A vendor’s figure describes the files its customers chose to send it, from the segments those customers serve.
• An insurer’s figure describes the claims that reached its investigation unit, which were often referred because something looked wrong.
• A survey describes the people who answered it, and respondents who have had a fraud problem may be keener to reply.

Every sample is selected somehow, so whether one is biased is the wrong question. Ask which way the selection pushes, and whether the source tells you.

What a ‘flagged’ share is and is not

In our reports, a flagged share is the share of checks that HTPBE? returned as structurally modified – the file shows evidence that it was changed after it was created. That is all it is.

It is not:

• a fraud rate or prevalence – it does not tell you how often documents in circulation are altered, only what share of the checks run through HTPBE? were returned as modified;
• accuracy, precision, recall or a false-positive rate – those can only be measured against files whose true status is already known, and a production share has no such labels;
• a detection rate – you cannot say what share of altered files were found without knowing how many altered files there were.

A higher or lower share says nothing on its own about how good a detector is. A tool that flagged every file would top any share chart and be worthless.

Two more distinctions matter. First, modified is not the same as fraudulent. A modification is a fact about the file; it does not establish intent, and many edits are harmless – an issuer correcting a typo, a redaction, several documents combined into one file. Second, the other outcomes are not ‘pass’ results. intact means no modification was detected, not that the document is genuine. inconclusive is a separate outcome: the evidence in the file cannot rule modification in or out. It is not a mark against the document, and it is not a softer version of modified. Our guide to what inconclusive means covers it in more detail.

Watch for buckets that merge outcomes. The February report’s figure for files not returned as modified, 249, combined 211 intact and 38 inconclusive. Read as one number, it looks like 249 files with nothing to report. It was two different results.

Keep prevalence and conditional shares apart as well. The July report gave two figures for a single structural property: how many of all processed checks carried it (about one in ten) and how many of those were flagged (almost all). ‘Almost all of these were flagged’ and ‘about one in ten checks carried this property’ answer different questions. Quoting the first as if it were the second turns a narrow observation into a sweeping one.

Small bases need the same care. In February only 11 files were digitally signed, and 3 of them had been changed after signing. The report itself treats that as an observation, not a rate. On a base that small, a single file changing category moves the percentage sharply.

Comparing months or sources

When a share moves between two periods, two things can have changed: the population and the ruler.

The population is what came in. The ruler is the method that classified it. In our case the ruler moves often – between eight and twenty-nine algorithm versions shipped each month from March to September, with May’s twenty-nine the heaviest. The March report said plainly that its month-to-month changes mix the two effects and that it does not separate them. The July report described its figures as a blend of ‘a moving population and a moving ruler’. The label set itself changed as well: in March, the three-outcome model was in force only from 8 March, and earlier checks that month were reported only as modified or not flagged.

The June report is the clearest example. May’s flagged share was roughly seven in ten; June’s was just under half. The June report attributes the difference almost entirely to traffic composition – about four in five June checks came through the website, while May had leaned heavily towards the API. In its own words: ‘Same engine, same thresholds, a very different headline number.’ That is the report’s reading, not a controlled test: the ruler did not stand still either, with sixteen algorithm versions shipping in June.

The August report marks one of its comparisons as directional only, because no significance test was run. That caution applies widely: a small difference between two periods may simply be noise.

The September report goes further and says it is not directly comparable with August, because the intake differed and parts of the measurement and classification changed within the month.

So when two figures from different months or different sources disagree, the first question is not ‘what happened in the world?’ It is ‘what changed in the intake, and what changed in the method?’

Reading industry figures

Most document-fraud figures in circulation come from one of three types of source. Each measures something real, and none measures ‘how much fraud exists’.

Complaint and incident registries count reported cases. They depend on victims knowing they were defrauded, knowing where to report it and bothering to do so. A rise can mean more fraud, more awareness, or an easier reporting form. Ask: who reports here, and what changed about reporting?

Vendor customer samples describe what reached that vendor’s customers. They reflect the vendor’s market segments, its customers’ intake and its own definitions of ‘fraud’ or ‘suspect’. Ask: what is the denominator, which customers are included, and what exactly does the label mean?

Market-research estimates and forecasts are modelled. They combine assumptions, survey inputs and extrapolation. Ask: what are the inputs, what is assumed, and is this a measurement or a projection?

For the external loss figures themselves, with sources, see our document fraud statistics roundup. Read each one with the questions above.

Five questions to ask before citing any rate

• What is the denominator? Checks, documents, users or applications – and are retries, duplicates and internal runs included?
• Who is in the sample? How did items get there, and which way does that selection push the figure?
• What does the label mean? ‘Fraud’, ‘suspect’, ‘flagged’ and ‘modified’ are different claims. Does the source define its terms?
• What period and what method? Did the population or the method change during the period, or between the periods being compared?
• How large is the base, and was anything tested? A share on eleven items, or a difference with no significance test, is an observation, not a finding.

If a source cannot answer these, quote it with that caveat attached, or do not quote it.

Where our figures live

Our running aggregate is on the statistics page, which shows the date it was last updated; each monthly integrity report describes one month’s dataset. Both cover what reached HTPBE? and how HTPBE? classified it, not rates of fraud in the world. Each report sets out its own sample and its own limits – from July onwards, with the denominator stated before any number. If you cite them, cite the update date or the month, and carry the caveat with the number.

If what you need is a structural signal on your own documents rather than a statistic about other people’s, the API returns intact, modified or inconclusive for each PDF you send. It reports whether the file shows structural evidence of change after it was created. It does not tell you whether the content is true.