Fraud Blocker Data Cleansing: Filter » Match Data Pro

Data Cleansing: Filter

Filtering Records

Overview

Filter keeps or removes records based on rules you define, so only the records you want move on to the next module. Use it to drop test records, keep one country or state, exclude records with blank key fields, remove records outside a date range, or split a data source into the records that meet a condition and those that don’t.

A filter works on whole records. Every rule is evaluated against each record, the rules are combined with AND or OR, and the record is kept or removed accordingly. Columns are never changed.

Building a Filter

  1. Select Datasource.
  2. Action – Keep Matching Records or Remove Matching Records.
  3. Match Type – Match All Criteria (AND) requires every rule to be true; Match Any Criteria (OR) requires at least one.
  4. Case Sensitive / Case Insensitive – applies to all text rules. Case Insensitive is the default.
  5. Click Add Rule for each condition. A rule has a Column, an Operator, and a Value. The operators offered depend on the column’s data type, and the value box adapts too: a date picker for date columns, a number box for numeric columns, True/False for boolean columns. Between and Not Between show a second value box. Is Blank and Is Not Blank need no value.
  6. Remove a rule with the trash icon. Click Save and Add to Task List when done.

Operators by Column Type

Column typeOperators
TextContains, Not Contains, Equals, Not Equals, Begins With, Not Begins With, Ends With, Not Ends With, Is Blank, Is Not Blank, RegEx Matches, RegEx Not Matches
NumberEquals, Not Equals, Greater Than, Less Than, Greater Than or Equal To, Less Than or Equal To, Between, Not Between, Is Blank, Is Not Blank
DateBefore, After, Equals, Not Equals, Between, Not Between, Is Blank, Is Not Blank
True/FalseEquals, Not Equals, Is Blank, Is Not Blank
  • Is Blank treats empty cells, cells containing only spaces, and missing values all as blank.
  • For text operators, special characters in your value are taken literally. Only RegEx Matches and RegEx Not Matches treat the value as a regular expression.
  • Number rules on a text column ignore values that aren’t numbers; they neither match nor cause an error.

Tips

  • Remove test data first. A single rule, Email Contains “test” with Remove Matching Records, is the fastest way to clear test accounts before matching.
  • Require the fields you match on. Keep Matching Records where the match columns are Is Not Blank, so records with no usable data never reach the matcher.
  • Combine AND rules for precision, OR rules for breadth. Use AND to narrow to one segment; use OR to catch several spellings or codes in one pass.
  • Use RegEx for patterns. A rule such as Postal Code RegEx Not Matches ^\d{5}(-\d{4})?$ removes every record whose ZIP is malformed.
  • Keep and Remove are mirror images. Keep Matching with rule X gives you exactly the records that Remove Matching with rule X would drop, so you can build two filters to split a file in two.

FAQs

No. Like every cleansing task, it writes the result to the cleansed output data source. The imported original is untouched.

As many as you need. They are all combined with the Match Type you chose.

No. One filter uses one Match Type. To express A and (B or C), run two filters in sequence.

Only if you choose Case Sensitive. The default compares without regard to case.

Use the date picker; dates are compared as calendar dates, so the time of day in your data is ignored.

They are simply not included in the output. To keep them, build a second filter with the opposite Action.

Start Your First Project

To begin, click the New Project button from the dashboard.