Navigating AI Privacy Risks in the Age of Data Analytics

As technology advances, so do the associated risks. Tools that enhance data collection and analysis also increase the likelihood of personal and sensitive information appearing where it shouldn’t. This risk, particularly privacy risk, is significant in the era of artificial intelligence (AI). Sensitive information is collected and used to develop and refine AI and machine learning systems. As policymakers introduce privacy regulations around AI, businesses face new compliance challenges. 

Despite privacy and compliance concerns, companies continue to deploy AI models to boost productivity and unlock value. Let's explore the AI privacy risks and safeguards impacting society and commerce today. 

What is AI Privacy? 

AI privacy involves protecting personal or sensitive information collected, used, shared, or stored by AI systems. It is closely linked to data privacy, which is the principle that individuals should control their personal data, including how organizations collect, store, and use it. The concept of data privacy has evolved with the advent of AI. Jennifer King, a fellow at the Stanford University Institute for Human-Centered Artificial Intelligence, explains, “Ten years ago, most people thought about data privacy in terms of online shopping. But now, we've seen companies shift to ubiquitous data collection that trains AI systems, which can significantly impact society, especially our civil rights.” 

Understanding AI Privacy Risks 

AI privacy concerns often stem from issues related to data collection, cybersecurity, model design, and governance. One major risk is the collection of sensitive data. AI systems often handle vast amounts of data, including healthcare records, social media data, financial information, and biometric data. The more sensitive data collected, the higher the risk of privacy infringements. Another concern is the collection of data without consent. Controversies arise when data is collected for AI development without explicit consent. Users expect more control and transparency over their data, as seen in recent backlash against LinkedIn for automatically opting users into data collection for AI training. 

Even with consent, privacy risks persist if data is used beyond the initially disclosed purposes. For example, a patient’s medical photos used in AI training without explicit consent highlight this issue. AI can also exacerbate privacy concerns related to surveillance and bias. AI-powered decision-making in law enforcement has led to wrongful arrests, particularly affecting people of color. Additionally, AI models are attractive targets for attackers. Hackers can exploit vulnerabilities to steal sensitive data, as seen in prompt injection attacks. Accidental exposure of sensitive data is another significant risk. Instances like ChatGPT showing users’ conversation histories illustrate the potential for data breaches. 

Privacy Protection Laws 

Policymakers have long sought to protect individual privacy from technological advancements. The rapid growth of AI has intensified the need for data privacy laws, such as the European Union’s General Data Protection Regulation (GDPR). GDPR mandates that companies have a specific, lawful purpose for data collection, inform users, and collect only the necessary data. Data must be used fairly and deleted once its purpose is fulfilled. The EU Artificial Intelligence (AI) Act, the world’s first comprehensive AI regulatory framework, prohibits certain AI uses and imposes strict governance, risk management, and transparency requirements. 

Regulatory Landscape and Best Practices for AI Privacy 

Though the EU AI Act doesn't specifically outline separate prohibited practices on AI privacy, it enforces limitations on data usage. Prohibited AI practices include untargeted scraping of facial images from the internet or CCTV for facial recognition databases, and law enforcement use of real-time remote biometric identification systems in public, unless an exception applies and pre-authorization by a judicial or independent administrative authority is required. High-risk AI systems must comply with specific requirements, such as adopting rigorous data governance practices to ensure that training, validation, and testing data meet specific quality criteria. 

US Privacy Regulations 

In recent years, data privacy laws have taken effect in multiple American jurisdictions. Examples include the California Consumer Privacy Act and the Texas Data Privacy and Security Act. In March 2024, Utah enacted the Artificial Intelligence and Policy Act, considered the first major state statute to specifically govern AI use. At the federal level, the US government has yet to implement new nationwide AI and data privacy laws. However, in 2022, the White House Office of Science and Technology Policy (OSTP) released its “Blueprint for an AI Bill of Rights.” This nonbinding framework delineates five principles to guide AI development, including a section dedicated to data privacy, encouraging AI professionals to seek individuals’ consent on data use. 

China's Interim Measures for Generative AI Services 

China is among the first countries to enact AI regulations. In 2023, China issued its Interim Measures for the Administration of Generative Artificial Intelligence Services. Under this law, the provision and use of generative AI services must respect the legitimate rights and interests of others and must not endanger the physical and mental health of others, nor infringe upon others' portrait rights, reputation rights, honor rights, privacy rights, and personal information rights. 

AI Privacy Best Practices 

Organizations can devise AI privacy approaches to help comply with regulations and build trust with their stakeholders. Recommendations from the OSTP include conducting risk assessments, limiting data collection, seeking and confirming consent, following security best practices, providing more protection for data from sensitive domains, and reporting on data collection and storage. 

• Conducting Risk Assessments: Privacy risks should be assessed and addressed throughout the development lifecycle of an AI system. These risks may include possible harm to those who aren’t users of the system but whose personal information might be inferred through advanced data analysis. 
• Limiting Data Collection: Organizations should limit the collection of training data to what can be collected lawfully and used consistent with the expectations of the people whose data is collected. In addition to data minimization, companies should establish timelines for data retention, with the goal of deleting data as soon as possible. 
• Seeking Explicit Consent: Organizations should provide the public with mechanisms for consent, access, and control over their data. Consent should be reacquired if the use case that prompted the data collection changes. 
• Following Security Best Practices: Organizations that use AI should follow security best practices to avoid the leakage of data and metadata. Such practices might include using cryptography, anonymization, and access-control mechanisms. 
• Providing More Protection for Data from Sensitive Domains: Data from certain domains should be subject to extra protection and used only in narrowly defined contexts. These sensitive domains include health, employment, education, criminal justice, and personal finance. Data generated by or about children is also considered sensitive, even if it doesn’t fall under one of the listed domains. 
• Reporting on Data Collection and Storage: Organizations should respond to individuals’ requests to learn which of their data is being used in an AI system. Organizations should also proactively provide general summary reports to the public about how people’s data is used, accessed, and stored. Regarding data from sensitive domains, organizations should also report security lapses or breaches that caused data leaks. 

Data governance tools and programs can help businesses follow OSTP recommendations and other AI privacy best practices. Companies can deploy software tools to conduct privacy risk assessments on the models they use, create dashboards with information on data assets and the status of privacy assessments, enable privacy issue management, including collaboration between privacy owners and data owners, and enhance data privacy through approaches such as anonymizing training data, encrypting data, and minimizing the data used by machine learning algorithms. 

As AI and data privacy laws evolve, emerging technology solutions can enable businesses to keep up with regulatory changes and be prepared if regulators request audits. Cutting-edge solutions automate the identification of regulatory changes and conversion into enforceable policies.