Fraud Blocker Data Cleansing: Dictionaries » Match Data Pro

Data Cleansing: Dictionaries

Working with Dictionaries

Overview

A dictionary is a simple lookup list: each entry pairs a word or phrase found in your data with what you want to do about it. Match Data Pro scans a column for every entry in the dictionary and, wherever it finds one, replaces it with a standardized value, deletes it, or records it in a new column. It is the fastest way to standardize the same handful of variations across thousands of records.

Typical uses:

  • Standardizing company suffixes: “Incorporated”, “Inc”, and “INC.” all become “Inc”.
  • Expanding or abbreviating states, provinces, street types, and directions.
  • Fixing recurring misspellings and abbreviations of product names, titles, or departments.
  • Stripping noise words such as “N/A”, “UNKNOWN”, or “TEST” from a field.
  • Flagging records that contain certain terms without changing the data.

Dictionaries live in the Cleansing & Standardization module under the Dictionaries tab. There are two ways to work with them: apply a dictionary you already have as a data source, or build one directly from the values in your data.

Option 1: Apply an Existing Dictionary

Use this option when your dictionary already exists as a data source in the project, for example a spreadsheet you imported with one column of original values and one column of standardized values.

1. Select Dictionary

  • Select Datasource for Dictionary – the data source that holds your lookup list.
  • Source Text – the column in that data source containing the words to look for.

2. Select Data to Apply Dictionary

  • Select Datasource – the data source you want to cleanse.
  • Select Column to Apply Dictionary – the column that will be scanned.

3. Output Options

Choose what happens when a source text is found (see Output Options Explained below). For Replace with Standardized Text, also pick the dictionary column that holds the replacement text. Then choose where the result goes:

  • Change existing data – the cleansed values overwrite the selected column.
  • New Column Name – the original column is kept and the result is written to a new column. Pick a column from the dictionary data source to supply the new column name, or tick Customize column name and type one.
  • Create Flag Column – leaves your data untouched and only writes the new column, so you can review what would have changed before committing to it.

Click Save and Add to Task List. The rule joins the module’s task list and runs with the rest of your cleansing tasks. Matching is case-insensitive.

Option 2: Create and Apply a Dictionary

Use this option when you don’t have a dictionary yet. Match Data Pro reads the distinct values from a column, shows you how often each occurs, and lets you decide what to do with each one, all on one screen.

Build the word list

  1. Select the Datasource and the Column to Extract Dictionary from.
  2. Choose how values are listed. Show entire cell value makes each distinct cell value one row. Show delimited words splits cells into individual words using the delimiters you enter (the default is space, semicolon, and comma), so you can standardize single words inside longer text.
  3. Click Display Data. The table lists every distinct Word with its Count (how many records contain it) and Character Count.

Decide what to do with each word

Each row has three controls:

  • Replacement – type the standardized value that should replace the word.
  • New Column Name – to write the result to a new column instead of changing the original, type the column name here.
  • Delete – tick to remove the word from the data wherever it appears.

Use Bulk Actions to apply one setting to many rows at once: tick the rows, then choose Replace With, New Column For, or Mark Delete and enter the value.

Search and matching mode

The search box filters the word list. The two badges next to it control both the search and how the rule matches when it runs:

  • AND / OR – with several search terms, show rows that contain all of them or any of them.
  • Whole / Partial – Whole matches only when the word is the complete cell value (or a complete delimited word when you’re using delimiters). Partial matches the word anywhere inside the text, so “Inc” would also match “Incorporated”. Whole is the default and the safer choice.

Add as Flag Column works the same way as in Option 1: your original data is left alone and only the new columns are written. When it is on, every row that has a Replacement must also have a New Column Name.

Import, Export, and Clear

  • Export saves the table as a new data source in the project, with the name you enter, so you can reuse the dictionary in other projects or edit it in a spreadsheet. The exported columns are word, replacement, new_column, and is_delete.
  • Import loads a previously exported dictionary data source back into the table. Rows whose word is already in the table are skipped, and you’re told how many.
  • Clear Table empties the list so you can start over.

Click Save and Add to Task List when you’re done. Only rows that have a Replacement, a New Column Name, or Delete ticked are saved.

Output Options Explained

These options appear in Option 1 and decide what happens to a cell when one of the dictionary words is found in it.

OptionWhat it doesExample with dictionary word “Street” and standardized text “St”
Replace with Standardized TextReplaces the word with the value from the dictionary’s standardized column.“12 Main Street” becomes “12 Main St”
Delete Values OnlyRemoves the word and leaves the rest of the cell.“12 Main Street” becomes “12 Main ” (note the trailing space)
Delete Values (leading trailing)Removes the word together with the surrounding spaces, so no stray spaces are left behind.“12 Main Street” becomes “12 Main”
Delete Entire Cell ContentsEmpties any cell that contains the word.“12 Main Street” becomes blank

In Option 2 the same outcomes are set per row: fill in Replacement to replace, tick Delete to remove, and fill in New Column Name to write to a new column.

Tips

  • Start with a profile. Run the Data Profiler on the column first. Its value counts show you which variations are worth standardizing.
  • Use Show delimited words for multi-word fields. Address lines, company names, and product descriptions usually need word-level dictionaries, while status codes and categories work best as whole cell values.
  • Prefer Whole matching. Partial matching is powerful but can change words you didn’t intend, such as replacing “St” inside “Stuart”. Switch to Partial only when you need it.
  • Use a flag column first. With Create Flag Column or Add as Flag Column, you can preview exactly which records a dictionary would touch before you change anything.
  • Export the dictionaries you build. An exported dictionary is an ordinary data source, so you can share it across projects, edit it in Excel, and import it back.
  • Sort by Count. Fixing the most frequent variations first gives you the biggest improvement for the least work.

FAQs

Option 1 applies a dictionary that already exists as a data source in your project. Option 2 builds a dictionary from the values in your data and applies it in one step. Anything you build in Option 2 can be exported as a data source and used later with Option 1.

No. “street”, “Street”, and “STREET” are all matched by the dictionary word “Street”.

Whole matches only complete values, or complete words when you use delimiters. Partial matches the text anywhere it appears, including inside other words. Whole is the default.

Only if you choose Change existing data, or set a Replacement or Delete without a flag or new column. Choose New Column Name, Create Flag Column, or Add as Flag Column to keep the original column intact.

Yes. Export it from Option 2 as a data source, then import that data source into the other project and apply it with Option 1, or use Import in Option 2.

An exported dictionary is a data source in the project, so anyone with access to the project can use it. Team members can also import it into their own projects.

No. Dictionary rules run with the rest of your cleansing tasks in the background, and you can keep working in Match Data Pro while they process.

Start Your First Project

To begin, click the New Project button from the dashboard.