12
Wed, Jun
4 New Articles

For AI, Synthetic Data Is Anything but Fake News

Typography
  • Smaller Small Medium Big Bigger
  • Default Helvetica Segoe Georgia Times

AI apps must be trained to evaluate data. However, some data uses are restricted by law or the logistics of gathering it. These difficulties created a need for synthetic data, and the market is providing it.

To most people, “synthetic data” sounds like a paradox, or at least a contradiction in terms. Synthetic is something made up, and data is presumed to be a collection of facts. In the realm of AI, though, synthetic data is not only real, it has become a critical tool. In fact, it is becoming so important that it may actually surpass the use of real data for training AI apps by the end of this decade, according to at least one prediction. Any enterprise that plans to use AI at some point will have to embrace this concept with the confusing name, if not become an outright consumer of synthetic data.

Machine learning (ML), the process by which the neural networks that run AI apps learn how to do what they do, requires large amounts of data to both train an AI and to help ensure that the training an app does receive isn't biased in some subtle way. Essentially, an AI app is presented known data and works on it until it produces an expected value or output, at which point the owner of the app can consider it trained enough to proceed to the next training step. Although the process sounds simple, it can be rather complicated in execution.

Without getting too much into the weeds here, at a basic level, an AI app isn't learning the data itself, but rather how to classify it. The app turns that classification into a math function for subsequent processing. Part of this classification process also involves labeling the data, a function that currently is mostly done by humans to shortcut the trial-and-error process an AI app would have to go through to do that itself. Labeling provides a context for the data, and as the ML process continues, more labels are added to the data to give it more context. Generally speaking, the more data in a dataset and the more accurate the labels a dataset is given, the more efficient the ML process can be.

Finding Enough of the Right Kind of Data

Ah, but where does that data come from? What's the guarantee for its accuracy? And how much data of the kind needed can one get?

If you were trying to figure out something like whether people would be more or less likely to use an antacid tablet for heartburn depending on the tablet's flavor, asking every person in the nation that question would produce a large dataset, but the time and expense of doing it would be prohibitive. If instead you tried to get data from hospitals on how often their patients needed antacid tablets based on what they had consumed from hospital room service, you'd run up against privacy protections embedded in such laws as the Health Insurance Portability and Accountability Act (HIPAA) that don't let hospitals release that kind of data because the data could be traced back to the individuals from which it came, potentially violating their privacy.

Finally, even if somehow there was data available by either method, how should that data be labeled? How would you prevent human error in the labeling process? Synthetic data solves these problems.

It used to be that the privacy dilemma could be worked around by anonymizing data. For example, a large amount of data would be processed about epidemic victims by jumbling or substituting other letters for their names so specific records couldn't be traced back to an individual. However, in 2019, researchers from Imperial College London and Belgium's Université Catholique de Louvain proved that the actual people from an anonymized dataset could be identified via as few as 15 demographic attributes.

While there had been similar proofs with smaller datasets earlier, this was a tipping point at which Europe's General Data Protection Regulation (GDPR) would be violated. Simple anonymizing is no longer enough protection, particularly for text data. Now the standard must be that datasets used for research on medical problems, or those dealing with financial records, must be particularly careful about being true to the structure of a dataset without violating any person's privacy.

The Value of Synthetic Data

Synthetic data is generated by custom-built algorithms that can generate artificial datasets that replicate the structure of any kind of actual dataset. Because these algorithms model the data in a real dataset without replicating the actual data within it, the synthetic dataset retains the form, probability distribution, and other characteristics of a dataset of any particular kind that might be used for AI app training. However, the synthetic dataset has no actual information that can be tied to any individual, so it avoids the privacy risk. Synthetic data simply represents actual data and lets an AI app set baselines just as it could with real data. This enables ML training to retain the ability to help Ai apps learn to recognize statistical patterns and other properties in a dataset that can't be traced back to anything because it's replacement data rather than actual data. And it's well-known that ML training activities can generate more-accurate AI app models if the training data is more diverse.

In keeping with the idea of making medical-related data available for AI training and other purposes without violating privacy rules, health-insurance provider Anthem announced in May it would partner with Google Cloud to generate synthetic data that will include "medical histories, health care claims and other medical data."

Synthetic data has an economic benefit that counteracts the problem of prohibitive data-gathering costs as well. The generated datasets can be scaled to any size without affecting validity. This lets enterprises looking for a very large amount of data find some without having to research or generate it themselves and lets smaller companies access this aid to ML activity at a reasonable cost. It also provides enough data diversity to represent the reality-based models that ML training tries to teach and opens possibilities for enterprises to simply experiment with ML training relatively inexpensively. Such experimentation can act as a confidence-building measure for organizations choosing to sample AI technology before launching any full-bore project.

In addition, synthetic datasets come with labels already assigned. The entity using such datasets gets to skip that step in preparing AI training data. Making tweaks to the synthetic datasets also helps AI support staff adjust data-generation parameters and observe those results. The technology has the potential, for example, to help spot and counteract lending biases in the financial services industry and to help retailers analyze consumer purchasing behavior, all without exposing personal information about individuals.

Synthetic Data's Types and Categories

Generally speaking, synthetic data is divided into three major types and three categories. The three major types are synthetic text, synthetic images, and tabular synthetic data. Text is data in a natural language format. Images, primarily used in computer vision applications, include still and video graphics that are used to train AI apps to correctly "see" and discriminate between images of different objects. Tabular synthetic data is data of many types that appears in field-like structures for analysis by algorithms.

The three categories are fully synthetic data, partially synthetic data, and hybrid synthetic data. Fully synthetic data, as one might expect, is generated independently of real data and therefore has no relationship to actual data except that its format is similar. Partially synthetic data uses real data as a structure but substitutes for actual data values if there is a potential privacy issue. Hybrid synthetic data uses randomly selected items of real data that is then combined with similarly structured synthetic records to create an independent synthetic data item.

There are two major ways to create synthetic data. The first is to take a statistical distribution of a set of real-world data and then substitute numbers for the original data to create a dataset with different actual values. The second is to generate a model of an observed statistical distribution of an actual dataset and then substitute random data using the model as a structure. The models primarily used are generative, which means they include a model of the data distribution itself and can tell users the probability that a particular data item is representative of the overall dataset. (This is opposed to discriminative models, which use labels and tell users how likely the label is to be correct.)

(As a side note, synthetic data shouldn't be confused with a somewhat similar term, surrogate data. Surrogate data is specific to a group of data points in a sequence that have been taken and displayed at equally spaced moments in time, which can subsequently be analyzed via moving-average models.)

Obtaining Synthetic Data

If these definitions aren't entirely clear, the good news is that rather than building an algorithm to generate synthetic data, it's easier to find enterprises that provide synthetic data for training AI apps. For most situations, synthetic data produced by established companies using their own tested generation algorithms both produces more accurate synthetic data without violating privacy rules and is cheaper than an enterprise inventing a synthetic data-generating algorithm on their own.

On the one hand, no one can guarantee total accuracy with today's means of generating synthetic data. However, the general consensus among many enterprises seeking ML training data for AI apps is that the accuracy is high enough to offset the expense, inconvenience, and necessity of generating and labeling some version of synthetic data in-house.

There are currently scores of vendors offering synthetic data of various kinds and tools for building in-house synthetic datasets, should anyone's needs make traveling that route more practical in the long term. Because of space considerations, presented here is simply a series of links to associated providers. Readers interested in obtaining synthetic data or related tools for building it independently are encouraged to research which vendors provide aid best suited to any particular AI training needs.

Synthetic Data and Data Generation Apps Vendors

Accelario

Aindo

Avo Automation

Betterdata

Broadcom

Bulian AI

Chatterbox

Clearbox AI

Curiosity Software

DataCebo

DataGen

DatProf

Datomize

Diveplane

Esito

ExactData

Exploristics

FinCrime Dynamics

GenRocket

Gretel

Hazy

Intelligent Delivery Solutions

Informatica

Instill AI

IvySys

MD Clone

Mirry.ai

Mostly AI

Neutigers

Octopize MD

Particle Health

Replica Analytics

Sarus Technologies

Sky Engine

Sogeti

Statice

Syndata AB

Syntegra

Synthesis.ai

Synthesized

Syntho

Tonic

Validata

Veil.ai

Ydata

John Ghrist

John Ghrist has been a journalist, programmer, and systems manager in the computer industry since 1982. He has covered the market for IBM i servers and their predecessor platforms for more than a quarter century and has attended more than 25 COMMON conferences. A former editor-in-chief with Defense Computing and a senior editor with SystemiNEWS, John has written and edited hundreds of articles and blogs for more than a dozen print and electronic publications. You can reach him at This email address is being protected from spambots. You need JavaScript enabled to view it..

BLOG COMMENTS POWERED BY DISQUS

LATEST COMMENTS

Support MC Press Online

$0.00 Raised:
$

Book Reviews

Resource Center

  • SB Profound WC 5536 Have you been wondering about Node.js? Our free Node.js Webinar Series takes you from total beginner to creating a fully-functional IBM i Node.js business application. You can find Part 1 here. In Part 2 of our free Node.js Webinar Series, Brian May teaches you the different tooling options available for writing code, debugging, and using Git for version control. Brian will briefly discuss the different tools available, and demonstrate his preferred setup for Node development on IBM i or any platform. Attend this webinar to learn:

  • SB Profound WP 5539More than ever, there is a demand for IT to deliver innovation. Your IBM i has been an essential part of your business operations for years. However, your organization may struggle to maintain the current system and implement new projects. The thousands of customers we've worked with and surveyed state that expectations regarding the digital footprint and vision of the company are not aligned with the current IT environment.

  • SB HelpSystems ROBOT Generic IBM announced the E1080 servers using the latest Power10 processor in September 2021. The most powerful processor from IBM to date, Power10 is designed to handle the demands of doing business in today’s high-tech atmosphere, including running cloud applications, supporting big data, and managing AI workloads. But what does Power10 mean for your data center? In this recorded webinar, IBMers Dan Sundt and Dylan Boday join IBM Power Champion Tom Huntington for a discussion on why Power10 technology is the right strategic investment if you run IBM i, AIX, or Linux. In this action-packed hour, Tom will share trends from the IBM i and AIX user communities while Dan and Dylan dive into the tech specs for key hardware, including:

  • Magic MarkTRY the one package that solves all your document design and printing challenges on all your platforms. Produce bar code labels, electronic forms, ad hoc reports, and RFID tags – without programming! MarkMagic is the only document design and print solution that combines report writing, WYSIWYG label and forms design, and conditional printing in one integrated product. Make sure your data survives when catastrophe hits. Request your trial now!  Request Now.

  • SB HelpSystems ROBOT GenericForms of ransomware has been around for over 30 years, and with more and more organizations suffering attacks each year, it continues to endure. What has made ransomware such a durable threat and what is the best way to combat it? In order to prevent ransomware, organizations must first understand how it works.

  • SB HelpSystems ROBOT GenericIT security is a top priority for businesses around the world, but most IBM i pros don’t know where to begin—and most cybersecurity experts don’t know IBM i. In this session, Robin Tatam explores the business impact of lax IBM i security, the top vulnerabilities putting IBM i at risk, and the steps you can take to protect your organization. If you’re looking to avoid unexpected downtime or corrupted data, you don’t want to miss this session.

  • SB HelpSystems ROBOT GenericCan you trust all of your users all of the time? A typical end user receives 16 malicious emails each month, but only 17 percent of these phishing campaigns are reported to IT. Once an attack is underway, most organizations won’t discover the breach until six months later. A staggering amount of damage can occur in that time. Despite these risks, 93 percent of organizations are leaving their IBM i systems vulnerable to cybercrime. In this on-demand webinar, IBM i security experts Robin Tatam and Sandi Moore will reveal:

  • FORTRA Disaster protection is vital to every business. Yet, it often consists of patched together procedures that are prone to error. From automatic backups to data encryption to media management, Robot automates the routine (yet often complex) tasks of iSeries backup and recovery, saving you time and money and making the process safer and more reliable. Automate your backups with the Robot Backup and Recovery Solution. Key features include:

  • FORTRAManaging messages on your IBM i can be more than a full-time job if you have to do it manually. Messages need a response and resources must be monitored—often over multiple systems and across platforms. How can you be sure you won’t miss important system events? Automate your message center with the Robot Message Management Solution. Key features include:

  • FORTRAThe thought of printing, distributing, and storing iSeries reports manually may reduce you to tears. Paper and labor costs associated with report generation can spiral out of control. Mountains of paper threaten to swamp your files. Robot automates report bursting, distribution, bundling, and archiving, and offers secure, selective online report viewing. Manage your reports with the Robot Report Management Solution. Key features include:

  • FORTRAFor over 30 years, Robot has been a leader in systems management for IBM i. With batch job creation and scheduling at its core, the Robot Job Scheduling Solution reduces the opportunity for human error and helps you maintain service levels, automating even the biggest, most complex runbooks. Manage your job schedule with the Robot Job Scheduling Solution. Key features include:

  • LANSA Business users want new applications now. Market and regulatory pressures require faster application updates and delivery into production. Your IBM i developers may be approaching retirement, and you see no sure way to fill their positions with experienced developers. In addition, you may be caught between maintaining your existing applications and the uncertainty of moving to something new.

  • LANSAWhen it comes to creating your business applications, there are hundreds of coding platforms and programming languages to choose from. These options range from very complex traditional programming languages to Low-Code platforms where sometimes no traditional coding experience is needed. Download our whitepaper, The Power of Writing Code in a Low-Code Solution, and:

  • LANSASupply Chain is becoming increasingly complex and unpredictable. From raw materials for manufacturing to food supply chains, the journey from source to production to delivery to consumers is marred with inefficiencies, manual processes, shortages, recalls, counterfeits, and scandals. In this webinar, we discuss how:

  • The MC Resource Centers bring you the widest selection of white papers, trial software, and on-demand webcasts for you to choose from. >> Review the list of White Papers, Trial Software or On-Demand Webcast at the MC Press Resource Center. >> Add the items to yru Cart and complet he checkout process and submit

  • Profound Logic Have you been wondering about Node.js? Our free Node.js Webinar Series takes you from total beginner to creating a fully-functional IBM i Node.js business application.

  • SB Profound WC 5536Join us for this hour-long webcast that will explore:

  • Fortra IT managers hoping to find new IBM i talent are discovering that the pool of experienced RPG programmers and operators or administrators with intimate knowledge of the operating system and the applications that run on it is small. This begs the question: How will you manage the platform that supports such a big part of your business? This guide offers strategies and software suggestions to help you plan IT staffing and resources and smooth the transition after your AS/400 talent retires. Read on to learn: