Why Regex Alone Misses Sensitive Data

A regex pattern matching a PAN or an Aadhaar number is either present in a document or it is not. That binary judgment is what most scanning tools sell as "PII detection": a library of shapes, searched for across a file. The shape is cheap to write and fast to run. It is also a poor stand-in for the two questions that actually matter — whether a string that looks like an identifier really is one, and whether the sensitive information a scan is supposed to find has any fixed shape to begin with.

AIQ, the proprietary AI engine behind DiscoverIQ, treats regex as one layer in a pipeline rather than the whole answer. Each additional layer exists because pattern matching on its own under-fires on data with no fixed format and over-fires on strings that merely resemble one.

What a pattern alone gets wrong

The failure runs in both directions. An internal order number, a ticket ID, or a test record can carry the same digit count and grouping as a GSTIN or a PAN. A scan built purely on shape has no way to tell them apart, so it flags all of them.

The opposite failure is quieter and harder to notice. A resignation letter that describes an employee's medical condition in prose, a spreadsheet column of salary figures with no header naming it as such, an email that references a deal's terms before they are public — none of this has a pattern regex can search for. The sensitive part is the meaning carried by the surrounding words. A tool that only knows how to look for shapes has nothing to catch here at all.

A matching shape is not proof

Regex confirms that a string has the right count of digits and letters in the right positions. It says nothing about whether the string is a real, valid instance of that format. DiscoverIQ's regex layer is paired with format-specific checksum and structural validators — for identifiers such as GSTIN, PAN and Aadhaar — that recompute each format's checksum or structural rule against the candidate before it is treated as a match. A string that merely has the right shape but fails that check is set aside rather than logged as sensitive data. The validator does not replace the pattern; it stops the pattern from being taken at face value.

Five layers, each catching what the one before missed

Layer	What it does	What it catches that the layer before it misses
1. Regex + checksum validators	Matches known identifier formats and recomputes each format's checksum or structural rule (GSTIN, PAN, Aadhaar, and other structured identifiers)	Confirms a shape is a genuine identifier before it is treated as sensitive, and sets aside strings that only resemble one
2. Dictionary matching	Matches content against curated term lists using a multi-pattern string-matching approach built to run efficiently at volume	Sensitive terms that carry no numeric pattern at all
3. NER models	A trained named-entity-recognition model tags people, organizations and locations by their role in a sentence, not by a fixed format	Names and places written in free text, which have no shape for regex to search for
4. Local AI language model	A locally run language model is designed to read the language around a candidate match for its meaning in context	Cases where the earlier layers agree on a candidate but the surrounding text changes the answer
5. UPSI keyword-context layer	Reads the context around specific keywords to identify unpublished price-sensitive information — financial or deal terms not yet made public	Sensitivity that lives entirely in surrounding language, not in any single token

The layers are ordered by cost as much as by capability. Regex and checksum validation run first because they are fast and deterministic, and they resolve the majority of structured identifiers on their own. Dictionary matching and NER pick up terms and names the first layer was never built to see. The local LLM layer is the most expensive step, so it is reserved for the candidates the earlier layers cannot resolve alone — a number that could be a genuine identifier in one document and an unrelated string in another, disambiguated by reading what surrounds it. Because the model runs locally rather than through an external API, that reasoning happens on the content being scanned without sending it outside the environment.

From a detected value to a compliance-ready record

Every layer resolves to one of a shared catalog of roughly 60 attribute types, defined once in a common reference (pii_attributes.json) and carrying a risk score and a mapping to the regulations and standards it falls under. That catalog is what DiscoverIQ's detection and ClassifyIQ's classification share: DiscoverIQ's pipeline is what finds and identifies the attribute in a data source, and ClassifyIQ applies rule-based classification on top of what DiscoverIQ finds, turning a detected attribute into a labeled record governed by policy. Detection and classification stay separate steps, run by separate engines, on a common set of definitions.

Why the layering matters more than any single number

A pattern-only scanner forces a constant trade-off: tighten the regex and prose-level sensitive data slips through unseen; loosen it and lookalike strings pile up for someone to triage by hand. Validating structured identifiers against their checksum rules removes one source of false matches. Dictionary matching and NER add coverage for content that never had a numeric shape. A context-reading layer resolves the cases where the value alone is genuinely ambiguous. None of these layers is a claim about a measured accuracy figure — they are five different mechanisms, each built to catch a specific kind of miss the others leave behind.

DiscoverIQ runs this pipeline across a growing connector framework spanning databases, cloud storage, email, CRM and developer tools, with detection focused on English-language content today. See the five-layer pipeline work against your own data: book a DiscoverIQ demo.