“Delete Means Delete?”: The Problem of AI Memory and the Right to Be Forgotten in Canada

Most people assume that when they click “delete”, their information is gone. The expectation makes sense for traditional digital services. Whether deleting a social media account, history on an AI platform, or a conversation on an online platform, users generally expect that the organization will no longer retain or use their personal information.

Generative artificial intelligence (AI) challenges that assumption. Traditional databases simply store information where someone can easily erase it later. However, large language models (LLMs) learn statistical patterns from large datasets during training. These models can learn from information without retaining it as an identifiable record. Many generative AI systems now offer memory and personalization features that retain users’ preferences and interactions from previous conversations. These developments raise an important question: if an AI system learns from a person's information rather than simply storing it, what does it mean to "delete" that information?

Unlike traditional databases, AI models do not respond to user prompts by locating and reproducing information from stored records. During training, an AI model analyzes large amounts of information and adjusts its internal parameters in response to patterns, relationships, and associations it identifies within that data. The resulting model does not contain a searchable copy of each individual record, but the information used in training can influence how the model responds to future user prompts. This distinction creates the central challenge for deletion rights: removing the original data does not necessarily remove the influence that information had on the model’s behaviour.

While Canadian privacy law is built around the idea that individuals retain control over their personal information, AI blurs the line between stored data and learned behaviour. The result is growing uncertainty about whether existing deletion rights remain meaningful in the age of machine learning.

Deleting stored personal information does not restore meaningful individual control over that information where it has shaped an AI model's learned behaviour. Privacy law protects more than records stored in databases. It protects an individual’s ability to control how organisations use personal information in ways that affect their autonomy, dignity and participation in society. If Canadian privacy law is to preserve meaningful deletion rights, it should address not only stored data but also the continuing influence that personal information has on AI systems.

Where Personal Information Lives in AI Systems

Whether deletion is effective depends on where an AI system retains personal information. Traditional computers generally store information in identifiable locations where someone can easily locate and delete the records. AI systems operate differently; information can influence their outputs without remaining stored as an identifiable record. AI systems hold personal information in at least three different forms. First, they may retain the original records used to train a model. Second, they may retain user-specific information such as chat histories, saved memories, or account data that support personalization features. Third, the model itself may reflect patterns learned from training data through its internal parameters, even though it does not retain searchable copies of individual records.

Each form of information creates different privacy challenges. Organizations can generally delete training records, chat histories, or account data using ordinary account management or data deletion processes. The more difficult question arises when personal information contributes to a model’s learned behaviour through training. At that stage, deleting the original records or a user's account does not necessarily remove the influence that information continues to have on the model's responses. Canadian privacy law traditionally assumes that organizations can identify, manage, and dispose of personal information. AI systems challenge that assumption because information continues affecting a model even after organizations delete the records from which it learned.

The Right to Erasure in Comparative Perspective

The European Union has developed one of the strongest legal approaches to data deletion through Article 17 of the General Data Protection Regulation (GDPR), commonly known as the "right to erasure". Under certain circumstances, individuals may request the deletion of their personal information if they withdraw consent to its further processing, if the processing is unlawful, or if the information is no longer necessary for the purpose for which it was collected. Although the GDPR includes several exceptions, Article 17 reflects a broader commitment to meaningful user control over personal information.

Canada's approach is more limited. The Personal Information Protection and Electronic Documents Act, SC 2000 c 5 (PIPEDA) requires organizations to retain personal information only as long as necessary for the fulfilment of identified purposes and to develop policies governing retention and disposal. Unlike the GDPR, PIPEDA does not provide a standalone right to erasure. Instead, it relies on principles of consent, limited use, and retention minimization to protect personal information.

Canadian regulators have begun exploring issues arising from the absence of a data deletion right. The Office of the Privacy Commissioner of Canada has recognized that de-indexing and the removal of search results may, in some circumstances, fall within the scope of privacy protection. The Commissioner has also repeatedly emphasized the need for privacy-protective approaches to generative AI and has advocated for recognizing privacy as a fundamental right. The challenges have also begun to emerge in practice. Recent investigations by the Office of the Privacy Commissioner of Canada into generative AI systems demonstrate that these questions have moved beyond theory and now form part of Canada's evolving privacy landscape.

PIPEDA and the Limits of Data-Centric Deletion

Canada’s privacy framework focuses primarily on how organizations collect, use, retain, and dispose of identifiable personal information. PIPEDA requires organizations to remain accountable for personal information under their control, limit its use to identified purposes, and retain it only as long as necessary. However, AI systems challenge this approach because personal information can influence a model’s learned behaviour even after an organization deletes the original record.

This creates tension between legal compliance and meaningful individual control. Meaningful control protects individuals’ ability to decide how organizations use their personal information in decisions that affect them. Even after an organization deletes the underlying record, an AI system continues relying on patterns derived from an individual's personal information. As a result, the individual remains affected by the use of that information. In those circumstances, deleting the original record does not necessarily restore the level of control that privacy law seeks to protect. The legal question is not simply whether organizations have deleted personal information. It is whether they should continue benefiting from the influence that information has on their AI systems after an individual has requested its deletion.

Technical Impossibility Versus Legal Obligation

One of the central challenges in this area is the tension between technical feasibility and legal obligation. Current machine learning systems do not reliably support selective forgetting. Where personal information has contributed to what a model has learned, removing its influence may require retraining parts of the model or other technical interventions.

Researchers have begun developing techniques to reduce or remove the influence of specific data from trained models without retraining them from scratch. “Machine unlearning” addresses this problem by modifying a trained model so that it reduces or removes the influence of specific training data. Rather than deleting a stored record, machine unlearning alters the model’s internal parameters so that information derived from data no longer affects its outputs in the same way. Although these techniques remain experimental, they demonstrate that the law’s understanding of deletion may need to evolve beyond removing identifiable records and consider whether organizations can reduce the continuing effects of personal information within trained models.

Machine unlearning remains imperfect; it can be costly, difficult to verify, and may reduce model performance. Yet technical difficulty alone should not determine the scope of an individual's privacy rights. If organizations could limit their privacy obligations simply because compliance is difficult, technology would begin to define the law instead of the other way around. Privacy law has historically required organizations to adapt when new technologies challenged existing protections, and AI should be no exception.

The issue is therefore not whether AI systems can achieve perfect forgetting, but whether Canadian privacy law should require meaningful forgetting – a standard that recognizes technical limitations while still requiring organizations to reduce the continuing influence of personal information when reasonably possible.

When Deletion Doesn’t End the Story

The central challenge for privacy law is determining what meaningful deletion should require in the age of AI. Traditional deletion rights focus on removing identifiable personal information held by organizations, but AI systems require a broader understanding of control.

If AI systems continue relying on information derived from personal data after organizations delete the underlying records, deleting the original record does not fully protect the individual's privacy interests. Privacy law protects those interests by preserving an individual's dignity and autonomy, including the ability to control consequential uses of their personal information. Regulators, courts, and organizations must reconsider whether existing concepts of deletion adequately protect the values that privacy law is intended to serve.

Privacy Must Mean More Than Data Removal

Modern AI systems require Canadian privacy law to reconsider what meaningful forgetting should require. Where organizations cannot eliminate every influence of an individual's personal information from a trained model, privacy law should not simply ask whether the original records have been deleted, but rather ask whether organizations have taken reasonable steps to reduce that influence and whether they have been transparent about the limits of those efforts. A standard of meaningful forgetting should recognize both the technical realities of AI systems and the dignity and autonomy interests that privacy law seeks to protect.

Parliament has begun the process of modernizing Canada’s privacy framework by tabling Bill C-36, the Protecting Privacy and Consumer Data Act. While the bill would strengthen individual privacy rights and expands deletion rights, it leaves unresolved an important question: what should meaningful deletion require once personal information has already influenced a trained AI model? As Canada continues to develop its approach to AI governance, addressing that question will be essential to ensuring that privacy rights remain effective in the age of machine learning.

Canadian privacy law should not require perfect forgetting. Current AI systems do not always permit organizations to eliminate every influence of personal information from trained models. Instead, privacy law should require meaningful forgetting. At a minimum, organizations should take reasonable steps to reduce the continuing influence of personal information where technology permits, stop using the original information for future training once a valid deletion request is made, and be transparent about the limits of any remaining influence. A standard of meaningful forgetting would recognize current technical limitations while preserving the dignity and autonomy interests that privacy law exists to protect. The way Canadian privacy law defines meaningful forgetting may ultimately determine whether "delete" continues to mean what users expect it to mean.

The opinion is the author's, and does not necessarily reflect CIPPIC's policy position.