Transforming Data into Opportunities: Metric in Motion – AI-Powered Duplicate Detection

High-quality data is more than a benchmark – it is a strategic necessity for global trust, compliance and interoperability. In this blog, Zornitsa Manolova, Head of Data Quality Management and Data Science at GLEIF, explores how AI is further enhancing duplicate detection to promote unique and reliable entity identification across the Global LEI System.

---

---

Every Legal Entity Identifier (LEI) is unique and can represent only one entity. Each entity can hold only one LEI code. These principles underpin the Global LEI System, enabling anyone, anywhere in the world, to identify legal entities with confidence.

Preventing duplicate LEI records is therefore essential to preserving trust in the Global LEI System and is a key part of GLEIF’s proactive data quality management program. A well-established duplicate detection and remediation process – comprising preventive controls, LEI issuer commitment and review, and clear operational procedures – already means that potential duplicates account for less than 0.2% of all records in the Global LEI System.

Strengthening Duplicate Detection with AI

As data volumes and complexity continue to increase, GLEIF is committed to further improvements to duplication detection processes. In particular, AI presents new opportunities to promote a more accurate, scalable, and consistent approach.

One key control already being enhanced by AI is GLEIF’s 'Check for Duplicates' facility.

Check for Duplicates is a feature that enables LEI issuers to assess whether the proposed LEI and the associated reference data may already exist in the Global LEI Index before a new LEI is published. During the issuance process, new records are compared against both the full LEI Index and records that have not yet been issued by other LEI issuers. This helps ensure that, even if the same legal entity approaches multiple LEI issuers, potential duplicates can be identified and resolved before publication.

With AI support, the facility now goes beyond the previous algorithm, which relied mainly on fuzzy name matching. It enables earlier detection, more effective comparison of potentially related records, and coordinated resolution before publication.

How It Works

The enhanced duplicate detection process follows a structured workflow that consists of three main steps:

• Pre-processing:
  The submitted record is cleaned and standardized. This includes removing punctuation, normalizing spaces, parsing the record to extract relevant reference data, and generating vector embeddings for later comparison.
• Filtering:
  An AI-enhanced backend then identifies potential matches. The process first checks the LEI code, then compares registration authorities and registration identifiers. Selected reference data elements are also converted into vector embeddings and matched against existing records in the Global LEI Index. Records with potential matches are then passed to the scoring stage.
• Scoring:
  This step further evaluates these potential matches to reduce false positives. It considers elements such as the legal name, legal form, address, jurisdiction, and entity creation date, with additional handling for specific categories such as funds and branches.

Faster, Scalable, and More Consistent Duplicate Detection

Together, this process strengthens duplicate detection by enabling:

• Faster and earlier detection of potential duplicate registrations before a new LEI is published in the Global LEI Index.
• Improved scalability as the Global LEI System continues to grow, as the use of vector embeddings enables records to be compared against large volumes of existing LEI reference data.
• Consistent and standardized approach independent of the nature and location of the LEI issuer.
• Stronger data quality controls by checking multiple data elements, including LEI code, registration authority, identifiers, legal name, address, jurisdiction, and legal form.

Harnessing the Potential of AI for Duplicate Detection

Through AI-supported pre-processing, filtering, and scoring, it is clear how AI is enabling the Check for Duplicates feature to become more scalable, efficient, and context-aware.

For AI to deliver reliable results, it needs to be built on complete and trustworthy data and supported by clear governance. At GLEIF, this governance is reinforced through proactive data quality management, established validation processes, and continuous monitoring of the Global LEI System. AI recommendations are combined with transparent decision-making, continuous refinement of the detection models, and human expertise and oversight to ensure accurate and consistent results.

Together, these elements support GLEIF’s continued commitment to proactive data quality management and maintaining trust across the Global LEI System.

---