Data & AI

Innovation is not just a word – it’s a necessity to surviving and achieving in the new normal. How are others How can you? Don’t miss the conversation!

Entity Resolution (ER) is an important tool used to address data issues, from the most serious – stopping international fraud – to the most basic – finding duplicate names in your customer list. Now, advancements in artificial intelligence and automation for ER make it more useful and valuable than ever.

What is Entity Resolution

Entity resolution is the analysis of multiple “entities,” which includes people, places, events and objects/assets, to determine how those items are – or are not – linked. The power of ER is to look at non-obvious connections within a data set and make those connections obvious. Jeff Jonas, CEO of Senzing, an artificial intelligence (AI) company focused on ER, puts it in simpler terms: “ER determines who is who and who is related to who in data.”

As a simple example from Senzing, here are different entity profiles housed in a company.

In this scenario, there could be anywhere from one to five separate entities. Entity resolution is important because it tries to create a single version of the “truth” for any given entity that the business deals with. Senzing Entity Resolution software uses “entity-centric learning,” a form of artificial intelligence, to make new judgments – and change previous resolutions – in real time, based on new information coming in or leaving.

Three major uses for ER

Fraud – Bad guys are continually trying to get away with things, and they’re becoming more and more sophisticated with multiple false or misleading identities and the type of transactions they try to carry out. Most larger companies have fraud alerts, but what they don’t have is the time or manpower to analyze every alert, let alone find otherwise non-obvious connections with other data. ER software can be programmed for automatic entity resolution based on the number and types of attributes linked. The ER system then runs mock scenarios to see how outcomes would change under the resolution – helping to indicate if and where the fraud exists.

Data governance – Companies want clean, understandable data for a host of analytical purposes. For many types or organizations, the largest amount of data relates to customers. Customer data is often duplicated across and within many systems: transaction history, credit card info, loyalty programs and so on. ER gives companies a big step toward clean data and greater insight when it comes to “know your customer” (KYC). This makes for much more efficient and successful sales and marketing campaigns as well as generating a better overall customer experience. ER has the added benefit of saving money on data storage costs by eliminating multiple duplicate records.

CCPA/GDPR – California’s Consumer Privacy Act (CCPA) and EU’s General Data Protection Regulation (GDPR) and laws like them give individuals a level of control over how companies use their collected data. This includes what’s generally referred to as the “right to be forgotten” – having a company delete the requestor’s information as mandated by the applicable law. Hefty fines can be imposed for non-compliance. ER takes the request form and compares it against the company’s databases to connect related information into one record, which can then be deleted, all while providing an audit trail.

Other uses include ensuring regulatory compliance, applicant and vendor screening, and modernizing record and registration systems.

How Prolifics can help

Prolifics has joined forces with Senzing to further Jeff Jonas’ goal of making high quality ER available to mainstream companies. Senzing offers a plug-and-play, real-time AI for ER desktop app and a more advanced API version for developers. Senzing’s ER is a type of middleware. Prolifics can provide the data solutions and services to prepare your data for ER, and the services and solutions to get the desired operational outcomes implemented after entity resolution. You don’t need a million-dollar-plus budget, expensive ER experts, or a large number of IT resources to deploy Senzing ER. Connect with Prolifics at solutions@prolifics.com.

For more information, watch Jeff Jonas’ visit to the Innovation Sandbox. And don’t miss his story about running a marathon with his mom!

Data Fabric and Data Lineage are two terms we see a lot – on websites, in blogs and technical papers – but what’s the relevance? What do we really need to understand about our data and its power?

Manta and Talend send their data experts to visit the Innovation Sandbox. Ernie Ostic (M) and Rolf Heimes (T) answer questions about data and explore its significance in today’s new normal – all delivered with a dose of their personal style and passion for the topic.

Watch now.

National Geographic named him the Wizard of Big Data. He works on innovation, national security and privacy with government leaders, think tanks and executives all over the world. And now, Jeff Jonas is talking with you from inside the Innovation Sandbox!

“The fabric of our lives.” We all know this as a phrase the cotton industry has used for years. But given the exponential flood of information we devour today, there’s really a new fabric that all organizations need to understand and live with – data fabric. And as that data fabric is weaved, it’s important to know where the data originated and how it flows – the data lineage.

Gartner ranked data fabric as a top 10 data and analytics trend for 2019.

What is data fabric all about?

The term “data fabric” has been around for a few years but has only recently gained more attention and traction. Leading research and advisory company Gartner had it as one of their “Top 10 Data and Analytics Technology Trends for 2019.”

Prolifics partner Talend, a leader in data integration and data integrity, knows a thing or two about data fabric. Last month, independent research firm Forrester named Talend “a leader in enterprise data fabric evaluation.” Talend describes data fabric this way:

• Think of data fabric as a weave that is stretched over a large space that connects multiple locations, types, and sources of data, with methods for accessing that data. The data can be processed, managed, and stored as it moves within the data fabric.
• In simplest terms, a data fabric is a single environment consisting of a unified architecture, and services or technologies running on that architecture, that helps organizations manage their data. The ultimate goal of data fabric is to maximize the value of your data and accelerate digital transformation…

Where does data lineage fit in?

Data lineage is describing where data came from (how it entered your organization), how it’s moved and spread through the organization, the data’s characteristics and how it’s changed, and its overall quality. Prolifics partner MANTA succinctly describes the importance of data lineage:

• What makes a decision bulletproof? Solid data. What makes data solid? Knowing its source, its journey, and all its transformations from the moment it entered the database to the moment you saw it in the report you are using to make your decision.

For MANTA, data lineage is more important than ever given today’s environment. When dealing with data privacy regulations, like the EU’s General Data Protection Regulation (GDPR) and California’s Consumer Privacy Act (CCPA), MANTA notes that “data governance (is) made possible with powerful data lineage…full lineage is a must when proving compliance with any data-oriented regulation.” When dealing with the unexpected, like remote workers in a pandemic lockdown, “…understanding of how your data flows and its transformations, you will easily come up with strategies to overcome the data-related obstacles that the current pandemic has caused for your organization.”

Learn more

“Innovation Sandbox – Powered by Prolifics,” is a new, six-week YouTube Live series with today’s brightest minds, latest tech and most creative ideas.

Episode 2: “Take a Byte Out of Data,” airs live on Thursday, July 16, 10-10:30 a.m. ET on our YouTube channel, Prolifics TV. Rolf Heimes from Talend and Ernie Ostic from MANTA will discuss how companies that adopt new technologies, architectures, and methodologies for data will succeed – while companies that don’t will risk becoming obsolete. Learn more here Data Fabric and Data Lineage – The Basics.

The term “garbage in, garbage out” is as relevant to your data today as it was 20 years ago. If your company’s data is in chaos – incomplete, fragmented, trapped in siloes, hidden in legacy systems, poorly identifiable, duplicated – anything you try to do with that data will be, unfortunately, garbage.

Four Ways Your Data Chaos Will Hurt You – Badly

The pandemic shutdown exposed or exacerbated problems with cost optimization, cost efficiency, business continuity and disaster recovery. As companies reopen and retool post-lockdown, it’s more important than ever that you have clean and complete data – everything starts with having a handle on your data. If it’s in chaos, it will hurt you.

1) The drip, drip, drip of legacy systems

It’s kind of a “Catch-22” scenario – your data situation is in chaos because of your legacy systems, but you don’t think you can move from the legacy systems because of how deeply ingrained your data is. But legacy systems cost more to maintain, things take longer, and opportunities are missed. Data is growing exponentially at a time when you need speed and efficiency – but that new data is going into chaos. Whether moving to the cloud or staying on-prem, it’s time for application modernization and integration. You don’t need to do it all at once – but you better start soon, or the drips will keep on coming.

2) The data privacy movement

The EU’s General Data Protection Regulation (GDPR) and California’s Consumer Privacy Act (CCPA) and laws like it give individuals a level of control over how companies use their collected data. This includes the “right to be forgotten” – a company must delete a requestor’s information in its systems as mandated. Fines can add up – in CCPA it’s $2,500 per compliance violation, going up to $7,500 if the violation is deemed intentional. If you don’t know what data matches to which customers, or all the nooks and crannies where that data is and how to get to it, the penalties could add up fast.

3) Entity resolution

“Wizard of Data” Jeff Jonas defines entity resolution as “who is who and who is related to who” in data. The common terms identity resolution; record linking, matching and deduplication; and “merge-purge” are forms of ER. Entity resolution means you have a single version of something that your business must deal with, like a customer or a vendor, or you understand the relationship between somethings, like which salesman belongs with which vendor. If your data is garbage, and entity resolution is difficult, you’re setting yourself up for problems from bad customer experiences and cost inefficiencies to outright fraud.

4) Advanced and predictive analytics

Companies today are using artificial intelligence and machine learning (AI/ML) tools to automatically detect sophisticated data patterns to predict potential outcomes, giving them better decision-making insight and abilities. For example, a financial organization using AI/ML can analyze its entire historical loan portfolio on an ongoing basis, learning and recognizing patterns of factors in real time that lead to a much higher rate of default. If a new loan application has those factors, the AI/ML program would flag it for denial.  AI/ML is for “what’s next” questions, like which current customers will likely buy your new product or when will a piece of equipment break down. These insights can have you crushing your competition – unless your data is in chaos. Then the competition will be crushing you.  

Get a handle on your data

Prolifics experts can help you with your data, systems modernization and integration, and cloud and hybrid cloud platforms. Visit www.prolifics.com or email solutions@prolifics.com.

Coffee with Talend

Join data experts Brian Kordelski (Prolifics) and Rolf Heimes (Talend) for a discussion on how companies are striving to be leaner and more efficient when it comes to housing and managing data. They’ll also look at how migrating to an opensource framework can be cost-effective and quickly deployed. More information and registration here 4 Ways Your Data Chaos Will Hurt You.

We’re in the era of data – tsunamis of data, in fact, that are growing exponentially. With it comes concerns about what we can learn from the data – separating the melody from the noise – as well as overcoming worries surrounding privacy and fraud. Entity resolution (ER) is an important tool used to address these data issues.

On the cutting edge of ER technology is Jeff Jonas, CEO and Chief Scientist for Senzing, Inc., an artificial intelligence‐based (AI) software company focused on ER.

What Is Entity Resolution (ER)?

In the book “Entity Resolution and Information Quality,” writer Terry Talley states:

Entity resolution is the process of probabilistically identifying some real thing based upon a set of possibly ambiguous clues. Humans have been performing entity resolution throughout history. Early humans looked at footprints and tried to match that clue to the animals that made the tracks.

In the same book, writer John R. Talburt gives it a more formal, scientific view:

Entity resolution is about determining when references to real-world entities are equivalent (refer to the same entity) or not equivalent (refer to different entities). Linking is appending a common identifier to reference instances to denote the decision that they are equivalent.

Other related terms include identity resolution, record linking, record matching, record deduplication, merge-purge, and entity analytics, all of which represent particular forms or aspects of ER.

Jeff Jonas gives ER a simpler and more understandable definition:

ER determines “who is who and who is related to who” in data.

Why Is Entity Resolution Important?

Entity resolution is important to a business because it helps create a single version of the “truth” for any given entity that the business deals with.

The Single Customer View Example

Many companies can relate to the idea of a single customer view. Multiple systems may store separate pieces of information — purchase history, demographics, credit info, loyalty programs, and more — for the same person.

Example:
If your systems send an email blast to K. Coggs, KT Coggs, Katie Coggs, and Kathryn Coggs — are they one annoyed customer receiving four emails or four separate people?

Benefits of Clean Data

Having a single, unified customer view improves data accuracy for advanced and predictive analytics. Without it, you might send the wrong offer or communication — for example, sending Katie Coggs’ product recommendation to her mother, Kathryn.

Detecting Fraud and Ensuring Privacy

ER also plays a vital role in fraud detection — such as when multiple people use the same assets as loan collateral at different branches.

In addition, ER supports compliance with data privacy laws like the EU’s GDPR and California’s CCPA, which grant individuals control over their data — including the “right to be forgotten.”

For example, if KT Coggs requests deletion of her data, failure to recognize that K. Coggs and KT Coggs are the same person could lead to non-compliance fines.

Jeff Jonas – “Wizard of Big Data”

Jeff Jonas addresses all these ER challenges and more as CEO and Chief Scientist at Senzing, Inc. His work ranges from identifying duplicate contacts to uncovering criminal identities in real time across thousands of data sources.

The nickname “Wizard of Big Data” originated from a 2014 National Geographic article that profiled his innovative ER work. Some of his high-profile cases include:

• Identifying potential terrorists
• Detecting fraudulent behavior in casinos
• Connecting loved ones after natural disasters
• Modernizing voter registration systems

Privacy and Legal Data Use

As National Geographic described, Jonas’ systems use only legally obtained data and include privacy safeguards. They connect data points — such as addresses, job applications, and loyalty program details — to detect hidden links between people.

Making ER Accessible

Jonas’ goal is to make high-quality ER technology available to mainstream businesses. Senzing offers both a plug-and-play real-time ER desktop app and an advanced API for developers — removing the need for million-dollar budgets or large IT teams.

Jonas continues to collaborate with government leaders, think tanks, and global executives on innovation, national security, and privacy. He holds over a dozen patents and is one of only five people in the world who have completed every Ironman triathlon currently on the global circuit.

See Jeff Jonas live on “Innovation Sandbox, Powered by Prolifics”

Jeff Jonas will be the first guest for the inaugural season of “Innovation Sandbox, Powered by Prolifics,” a new YouTube Live series featuring today’s brightest minds, latest technologies, and creative ideas.

📅 Date: Thursday, July 9
🕙 Time: 10–10:30 a.m. ET
📺 Where: Prolifics TV

The episode, “We’re LIVE to see the Wizard!”, will be hosted by Greg Hodgkinson, Prolifics’ CTO and Innovation Center leader.

Join us for a dynamic discussion on how entity resolution impacts everyday life, plus insights from Jonas on innovation, AI, and even Ironman training during a pandemic.

Around the country, sweeping data privacy regulations are compelling businesses to rethink every aspect of their data governance capabilities.

If your organization complied with General Data Protection Regulation (GDPR) standards set by the European Union in 2016, you may have a head start. However, compliance with the GDPR will not be sufficient for compliance with the new state regulations. If your data lifecycle management process hasn’t changed in recent years—or if you never had a holistic, company-wide process to begin with — it’s time to stop kicking that can down the road.

Our new infographic shows you how three states are changing their data privacy landscape. You’ll also see steps your company can take to avoid unexpected penalties in the future.

California Consumer Privacy Act (CCPA)

• Enacted in 2018, the California Consumer Privacy Act (CCPA) requires transparency concerning the hosting and exchange of personal data, which includes a range of individual, or household, identifiers.
• As of July 2020, companies have 30 days to comply once notified of a violation; otherwise they are subject to civil penalties of up to $2,500 per violation and $7,500 per “intentional violation.”

Act to Protect the Privacy of Online Consumer Information (Maine)

• Maine’s Act to Protect the Privacy of Online Consumer Information requires broadband internet service providers (ISPs) to obtain customer consent before selling or sharing their data with a third party.
• While the CCPA gives customers the right to opt-out, Maine’s law prohibits ISPs from using customer data unless the customer opts in.

Stop Hacks and Improve Electronic Data Security Act (New York)

• The Stop Hacks and Improve Electronic Data Security Act (SHIELD Act) requires any businesses that maintain New York residents’ private information to take concrete steps intended to prevent data breaches.
• To comply, businesses must take reasonable protective measures, including:
  • risk assessments
  • workforce training
  • incident response planning and testing
• Failure to implement a compliant program is enforced by the New York State Attorney General and may result in injunctive relief and civil penalties.

Federal Privacy Laws

The United States does not yet have a comprehensive federal-level privacy law that reaches the scope of the GDPR in the European Union. Still, there are several vertically focused federal laws that businesses must be aware of if they deal with private consumer data. These include:

• Privacy Act of 1974: Gives individuals a way to access and correct their records, and sets forth various agency record-keeping requirements.
• Health Insurance Portability and Accountability Act (HIPAA): Regulates the collection and disclosure of patient health information and requires health care providers to protect data from unauthorized use.
• Children’s Online Privacy Protection Act (COPPA): Enforced by the Federal Trade Commission, outlines the appropriate use of information of children under the age of 13.
• Gramm-Leach-Bliley Act (GLBA): Requires financial institutions to protect customers’ data by limiting how information is shared with third parties.

Steps Toward Compliance

Your organization’s data quality and data governance team should discuss your approach tomanaging data assets. A methodical approach to making sure your organization is compliant will involve the following process:

• Determine how much data exposure is present in your environment and which state/federal rules apply to your business.
• Determine who in your organization needs to be involved and to what extent.
• Identify the utilities and software solutions needed to facilitate compliance.
• Identify the processes required to move toward compliance and the options you have to maximize your investments.
• Build a roadmap toward compliance with minimal requirements and timeline.