Generative AI Poses Unique Risks to Data Security and Privacy, NIST Warns

The National Institute of Standards and Technology (“NIST”) recently released a draft report highlighting the risks to data security and data privacy that generative AI poses.  The report, entitled “Artificial Intelligence Risk Management Framework:  Generative Artificial Intelligence Profile,” details the data security risks and proposes actions for generative AI risk management.

The report defines the data security risk as “[l]eakage and unauthorized disclosure or de-anonymization of biometric, health, location, personally identifiable [information], or other sensitive data.”  This leakage or disclosure occurs because training generative AI requires an enormous amount of data culled from the internet and other publicly available sources.  ChatGPT4 was trained with 570 gigabytes from books, web texts, articles, and other writing on the internet, which amounts to about 300 billion words residing in a generative AI database, and much of this is personal, confidential, or sensitive information.

Generative AI systems have been known to disclose such information upon request.  During adversarial attacks, large language models have revealed private or sensitive information within their training data, including phone numbers, code, and conversations.  The New York Times has sued ChatGPT’s creator, OpenAI, alleging in part that ChatGPT will furnish articles behind the Times paywall.  This disclosure risk poses obvious data security issues.

 Less obvious are the data security issues posed by generative AI’s capacity for inference.  AI generally, and generative AI specifically, can correctly infer personal or sensitive information, including a person’s race, location, gender, or political leanings – even if that information is not within the AI’s training data.  NIST warns that these AI models, or individuals using the models, might use these predictive inferences to undermine privacy or make discriminatory decisions about the individual.  Already, we have seen a company settle an EEOC lawsuit alleging that it used AI to make discriminatory employment decisions.  Generative AI threatens to increase this kind of legal exposure. 

From an AI governance perspective, NIST suggests several broad principles to mitigate the data privacy risk.  Among other things, NIST recommends:

• Aligning generative AI use with applicable laws, including those related to data privacy and the use, publication or distribution of intellectual property;

• Categorizing different types of generative AI content with associated data privacy risks

• Develop an incident response plan specifically tailored to address breaches, and regularly test and update the incident response plan with feedback from external and third-party stakeholders

• Establish incident response plans for third-party generative AI technologies deemed high-risk.  As with all incident response plans, this incident response plan should include

  • Communicating third-party generative AI incident response plans to all relevant AI actors

  • Defining ownership of the incident response functions;

  • Rehearsing (or “table topping”) the incident response plans regularly

• Conduct periodic audits and monitor AI-generated content for privacy risks

• Regular review of incident response plans for alignment with relevant breach reporting, data protection, data privacy, or other laws

• Update and integrate due diligence processes for generative AI acquisition and procurement vendor assessments to include data privacy, security, and other risks

These actions will involve more than simply adding a reference to artificial intelligence to existing cybersecurity plans. They will involve carefully analyzing a company’s legal obligations, its contract obligations, and the company culture to design an AI governance plan that keeps confidential information out of the public domain and away from bad actors.