Shubhank Suman
TMT, IP and AI Counsel
What This Report Covers
This report examines the collision between artificial intelligence and data protection law. It is written for lawyers, compliance officers, and business leaders who need to understand the practical real-world implications of this fast-developing area of law.
Topics Covered:
- Why AI and privacy law are structurally in conflict, and why that conflict cannot be managed with superficial fixes
- How India's Digital Personal Data Protection Act (DPDPA) compares with the EU's General Data Protection Regulation (GDPR), focusing on their key structural and practical differences, particularly in how each framework regulates the processing of personal data by AI systems
- Privacy by Design and Privacy by Default: what these principles require in practice at every stage of an AI project
- The highest-risk phase of any AI system: data collection and model training, and the specific legal pitfalls organizations routinely fall into
- How privacy obligations continue after deployment, including automated decisions, Shadow AI, cross-border transfers, and transparency duties
- The EU AI Act and its four-tier risk framework: what it means for organizations operating at scale
- A stage-by-stage compliance framework covering the full AI lifecycle, from data sourcing to ongoing vendor management
- Real-world compliance gaps: the specific failures organizations are most likely to overlook, drawn from regulatory enforcement trends in India and Europe
Data Privacy & AI Regulation | Shubhank Suman, TMT, IP & AI Counsel
Introduction:
In March 2026, India's National Human Rights Commission (NHRC) made an announcement that should have sent a clear signal across every boardroom, legal department, and product team handling personal data in the country. The Commission took formal cognizance of alleged violations of the Digital Personal Data Protection Act. Specifically, the NHRC identified the failure of major digital platforms to maintain systems for tracking the transfer of children's data, and the absence of functioning grievance redressal mechanisms. This was not a theoretical complaint raised by privacy advocates. It was a human rights body stepping into the space of data governance and treating the mishandling of personal data as a matter of fundamental rights.
That development is a useful lens through which to read everything in this report. For years, organisations in India and around the world treated data protection law as a box-ticking exercise, something to be managed with a privacy policy, a consent banner, and a periodic review. Artificial intelligence has changed that calculation entirely. AI systems collect data at unprecedented scale, process it in ways that are often invisible to the people involved, and make decisions that can affect access to credit, employment, healthcare, and public services. The stakes are no longer merely administrative. They are human.
This report examines the meeting point between AI and data protection law in depth. Its purpose is not to catalogue rules for their own sake. It is to help legal, compliance, and technology professionals understand why those rules exist, where they create genuine difficulty in practice, and what organisations must do to stay on the right side of them. The analysis draws on two of the world's most consequential data protection frameworks: India's Digital Personal Data Protection Act (DPDPA) and the European Union's General Data Protection Regulation (GDPR). It also covers the newer EU AI Act, which treats AI not merely as a data issue but as a distinct category of technological risk.
Why This Matters Right Now
The NHRC's intervention on children's data under the DPDP Act illustrates a fundamental shift: data protection is increasingly being treated not as a commercial compliance obligation but as a matter of fundamental rights enforcement. Organisations that approach AI governance as a legal checkbox exercise face a serious new risk -not just from regulators, but from human rights bodies, civil society, and affected individuals who have strengthening legal tools to push back.
Case Study: The School App That Shared Too Much
During the COVID-19 pandemic, millions of Indian schoolchildren were enrolled on edtech platforms. Parents provided their children's names, ages, school grades, learning progress, and sometimes health conditions, to platforms purely so that classes could continue. Several years later, some of those platforms began using the same data to build AI-powered 'personalised learning engines', and in some cases shared the data with advertising partners. Parents were never told this was happening. They had
Page 2 of 31 Shubhank Suman, TMT, IP & AI Counsel. All Rights Reserved.
Data Privacy & AI Regulation | Shubhank Suman, TMT, IP & AI Counsel signed up their children for classes. They had not consented to their children's learning profiles being used to train commercial AI models or to target advertising.
Page 3 of 31 Shubhank Suman, TMT, IP & AI Counsel. All Rights Reserved.
Data Privacy & AI Regulation | Shubhank Suman, TMT, IP & AI Counsel
01 The Structural Conflict: AI Vs. Privacy Law
There is a foundational tension at the heart of AI and data protection law, and understanding it is the starting point for everything that follows. The two systems do not merely differ in emphasis. They are, in a structural sense, pulling in opposite directions.
Two Systems, Two Logics
Artificial intelligence improves by learning from data. The more varied and substantial the dataset, the more reliably the model spots patterns, avoids errors, and produces useful outputs. A fraud detection system trained on ten years of transaction history will outperform one trained on ten months of data. A medical AI trained on millions of annotated patient records will be more accurate than one built from a small clinical sample. The technical and commercial logic of AI therefore points consistently in one direction: collect more, store longer, reuse broadly.
Data protection law starts from precisely the opposite position. Its foundational principles require organisations to collect only what is genuinely necessary for a specific and declared purpose, to keep it only for as long as that purpose remains active, and to delete it when the purpose is complete. These are not aspirational guidelines. Under both the GDPR and the DPDPA, they are binding legal duties whose violation can attract significant financial penalties and reputational consequences.
The Core Tension - Stated Simply
AI development asks: how much data can we get, and how long can we keep it? Privacy law asks: do you genuinely need this data, and when will you delete it? A compliance strategy that treats these as compatible by default -rather than as competing pressures requiring active management -is already on the wrong track.
Case Study: Zomato Knows More About You Than You Think
When you create a Zomato or Swiggy account, you hand over your name, phone number, address, and payment details so that food can be delivered to you. That is the purpose for which you shared the data. Over time, the platform also learns your food preferences, your order frequency, your average spend, the times of day you order, and even health-related signals (for example, if you consistently order diabetic-friendly meals). Now suppose the platform's AI team decides to feed all of this behavioural data -accumulated over three years -into a new model that predicts whether a user is likely to have a chronic health condition. That prediction engine is then licensed to insurance companies.
What the law says: The person who ordered biryani on Friday evenings never consented to their food habits being used to profile their health risks for insurance pricing. The original purpose was
Page 4 of 31 Shubhank Suman, TMT, IP & AI Counsel. All Rights Reserved.
Data Privacy & AI Regulation | Shubhank Suman, TMT, IP & AI Counsel food delivery. The new use -health risk profiling for insurers -is a completely different purpose and requires a separate legal basis under both the DPDPA and the GDPR. Without that basis, the reuse is unlawful regardless of how useful the model might be commercially.
Where The Conflict Becomes Concrete
This structural tension is not abstract. It surfaces in specific, recurring decisions that organisations face during AI development. A company may want to use customer data gathered for one service to train a model intended for another. It may want to hold onto historical records because they might improve accuracy later. It may want to combine data from multiple sources -support logs, usage records, payment histories -to build a richer model. At each of these points, privacy law asks a straightforward but demanding question: is that collection and reuse actually necessary, and is it fair to the people whose data is involved?
Consider a health technology company that wants to build an AI system to predict patient deterioration risk in hospitals. The model would be more accurate if trained on a decade of patient records from multiple hospital systems. But those patients may have consented only to the use of their data for their own treatment. They may never have been told that their records would be used to train an algorithm. The very data that makes the model powerful may also be data the organisation has no legal right to use in that way.
The Compliance Mistake Organisations Keep Making
The most common error organisations make is treating privacy law as a pre-launch checklist rather than an ongoing obligation. They may conduct a privacy review before a model goes live, add a sentence to their privacy notice, and consider the matter closed. This misunderstands both the nature of the legal duty and the nature of the risk. Privacy obligations apply throughout the entire AI lifecycle -from the moment training data is first identified to the day the model is retired. A system built on weak legal foundations does not become lawful simply because it is technically impressive or commercially valuable.
Case Study - HR Software
A mid-sized Indian company used an HR software platform to manage employee attendance, performance reviews, and leave records. When the platform launched a new AI modal that predicted which employees were likely to resign in the next six months, the company switched it on immediately. The AI was using three years of employee performance and attendance data to make its predictions -the same data that had been collected purely for payroll and HR administration. Employees had no idea this was happening. No fresh consent was taken. No new legal basis was identified. No privacy impact assessment was conducted.
What the law says: Both the DPDPA and the GDPR require that when personal data is used for a new and materially different purpose, the organisation must identify a fresh legal basis and provide updated notice to the individuals concerned. Using performance data collected for HR administration to power an attrition prediction model is a new purpose. It is not covered by the original consent or the original privacy notice. The company had built a useful tool on a legally deficient foundation.
Page 5 of 31 Shubhank Suman, TMT, IP & AI Counsel. All Rights Reserved.
Data Privacy & AI Regulation | Shubhank Suman, TMT, IP & AI Counsel
02 Privacy By Design And Privacy By Default
Two principles sit at the heart of good practice for any organisation building or using AI systems: Privacy by Design and Privacy by Default. They are different ideas, but they work together. One concerns how a system is built. The other concerns how it behaves once it is running.
Privacy By Design: Building Protection In
Privacy by Design requires that data protection is incorporated into a system from the earliest stages of development. It is not a patch to be applied once the product is finished. The principle is codified in Article 25 of the GDPR and is reflected in the guidance of data protection authorities across jurisdictions. In practical terms, it means that when engineers, architects, and product managers make decisions about how a system will handle data, privacy is a design criterion from day one.
The best analogy is structural engineering. A building can have locks fitted after it is built, but the strength of the locks depends on what the doors and walls are made of. A building designed from the start with security in mind will always offer better protection than one that had security bolted on later. The same is true of AI systems. A model designed from the outset to minimise data collection, enforce purpose boundaries, and enable deletion is far more legally robust than one that treats these requirements as problems to solve after launch.
Case Study - Whatsapp End-To-End Encryption
When WhatsApp introduced end-to-end encryption, it did not add it as an optional feature or a premium upgrade. It built the encryption directly into the architecture of how every single message is sent. The result is that even WhatsApp itself cannot read your conversations. The privacy protection is not layered on top of the product. It is baked into the product's fundamental design. A message that is intercepted in transit is meaningless without the private key held only on the recipient's device.
What the law says: This is exactly what Privacy by Design looks like in practice. The protection does not depend on a user changing a setting, reading a privacy policy, or trusting the company not to look at their messages. It is a structural guarantee built into the system from the start. The GDPR's Article 25 requires precisely this approach: technical measures that implement data protection principles in an effective manner and integrate the necessary safeguards into the processing itself.
Key Requirements In Practice
Before any data is collected, development teams must confirm that there is a valid legal basis for using it, that the purpose is clearly defined, and that a Data Protection Impact Assessment (DPIA) has been carried out where the processing is likely to pose a high risk to individuals. A DPIA is a structured risk review: it
Page 6 of 31 Shubhank Suman, TMT, IP & AI Counsel. All Rights Reserved.
Data Privacy & AI Regulation | Shubhank Suman, TMT, IP & AI Counsel asks what personal data is being used, what could go wrong, and what controls should be in place. If the legal basis is weak, data should not be used regardless of how useful it might be for the model.
During training, purpose controls must be enforced technically, not just stated in policy documents. If data was collected for customer support, it must not be silently diverted into model training. Minimum-data approaches -using anonymised datasets, synthetic data, or differential privacy -should reduce the amount of identifiable information in training sets wherever possible. Systems must also be designed with deletion in mind, so that erasure requests can be handled at the model level and not just at the data store level.
Privacy By Default: Safe Operation From Day One
Privacy by Default addresses a different but equally important problem. It concerns what happens after the system is switched on. The principle requires that the most privacy-protective settings apply automatically, without users having to take any action to obtain basic protection. A system should not require individuals to navigate menus before their data is handled with care.
In organisational terms, this means that AI tools deployed internally should not by default collect more data than the task requires, share data with unapproved third parties, or retain inputs beyond the purpose that justified their collection. Employees should not have to actively opt out of data collection to get the minimum required protection. The safest setting is the starting point.
Case Study - The New Phone
Think about the last time you set up a new Android or iOS smartphone. Before you even downloaded a single app, the phone asked you whether you wanted to share diagnostic data with the manufacturer, whether you wanted personalised ads, and whether you wanted location services enabled. A device built with Privacy by Default would start with all of these settings turned off. You would have to actively choose to turn them on. Some manufacturers do exactly this. Others, however, come with location sharing already enabled, diagnostic reporting switched on, and ad personalisation active from the first boot -requiring the user to hunt through several menus to turn them off.
What the law says: The second approach -where data collection is on by default and users must opt out -is precisely what Privacy by Default prohibits under the GDPR. The most privacy-protective setting must apply automatically. This principle applies equally to enterprise AI tools deployed within organisations: if the tool collects usage data, retains conversation logs, or shares inputs with the vendor by default, the organisation has a legal duty to audit and adjust those settings before deploying the tool to its staff.
Real-World Gap: Default Settings Are Often Business-Optimised, Not Privacy-Optimised
In practice, most commercial AI platforms ship with default settings that favour data collection over data minimisation. Usage data is retained, conversation logs are kept, and in many cases inputs are used for model improvement by the vendor unless the customer actively opts out. Organisations that
Page 7 of 31 Shubhank Suman, TMT, IP & AI Counsel. All Rights Reserved.
Data Privacy & AI Regulation | Shubhank Suman, TMT, IP & AI Counsel deploy these tools without auditing and adjusting the default configurations -and most do not -may be inadvertently creating a continuous stream of unlawful data transfers. This is one of the most overlooked compliance failures in enterprise AI deployment today.
Together, these principles create an integrated set of obligations that span the full lifecycle of an AI system. The table below summarises the key practical requirements at each stage.
AI builders must design systems that only use necessary data, hide identities early, start with privacy-safe settings, can forget data later, and never quietly reuse data for new purposes.
Privacy By Design & Privacy By Default – Obligations At The AI Builder / Programmer End
Step | What the AI Builder Must Do | What This Means | Examples |
|---|---|---|---|
1 | Fix the purpose before building | Clearly decide why the AI is being built before any data is collected. | This AI helps customer-support agents draft replies, not “let’s collect data first and decide later.” |
2 | Collect only necessary data | Do not take extra data just because it exists or may be useful someday. | A typing assistant needs text, not names, phone numbers, or home addresses. |
3 | Remove identities early | Names and direct identifiers must be removed as soon as data enters the system. | Replace “Anita Sharma - HR file” with “Employee ID 3245.” |
4 | Separate training data from live data | User data generated today must not automatically become training data tomorrow. | Customer chats with a bank chatbot are not reused for training unless separately approved. |
5 | Build the AI to forget data later | The system must allow data to be deleted if consent is withdrawn. | When an employee leaves, their data should not remain permanently embedded in the AI. |
6 | Start with privacy-safe default settings | The AI should begin in minimum-data mode, not maximum-collection mode. | Chat history storage is OFF by default unless deliberately switched ON. |
7 | Control who can see the data | Not every developer or engineer should have access to real personal data. | Developers test the AI using masked or dummy data, not actual salary records. |
8 | Test if the AI remembers too much | Check whether the AI can accidentally repeat personal or sensitive information. | The AI must never output “Rahul’s salary was ₹8 lakh” when prompted cleverly. |
9 | Build fairness and bias checks | The system must be testable for unfair or biased outcomes. | A hiring AI is checked to ensure it does not reject candidates from certain regions more often. |
Page 8 of 31 Shubhank Suman, TMT, IP & AI Counsel. All Rights Reserved.
Data Privacy & AI Regulation | Shubhank Suman, TMT, IP & AI Counsel
10 | Clearly document what the AI uses | Write simple, clear explanations of what data the AI does and does not rely on. | This AI uses skills and experience. It does not use age, religion, caste, or health data. |
|---|
Privacy By Design & Privacy By Default - Obligations At The Deployer / Organisation End
No. Obligation | What It Requires in Practice | Practical Example | |
1 | Confirm Legal Basis Before Activation | Organisations must have a documented legal basis before switching an AI system on, not after rollout. | An HR team completes a consent assessment before enabling an attrition-prediction module using employee data. |
2 | DPIA as a Design Gate | High-risk AI systems must not be deployed until a DPIA is completed and risks are addressed. | A bank delays deployment of a credit-scoring AI until bias and explainability risks identified in the DPIA are mitigated. |
3 | Purpose Compatibility Checks | Data collected for one purpose must not be reused for AI purposes without fresh analysis and, where needed, fresh consent. | Attendance data collected for payroll is not reused to train performance-ranking AI without notifying employees and updating the legal basis. |
4 | Privacy-Protective Configuration on Day One | Default vendor settings must be audited and adjusted before employee or customer use begins. | The IT team disables “inputs used for model improvement” before rolling out a public chatbot to customer care staff. |
5 | Minimum Collection in Use | The AI tool must not collect more data than is required for the task during live operation. | A customer chatbot is configured not to request address or date of birth unless strictly required for issue resolution. |
6 | Human Review for High-Impact Decisions | Where AI influences employment, credit, healthcare, or similar outcomes, human review must be real and meaningful. | Loan rejections generated by AI are reviewed by a credit officer before final decision, with authority to override the model. |
7 | Transparency to Affected Individuals | Individuals must be clearly informed that AI is used, how | Job applicants are told that an AI tool assists CV screening and are given a contact channel for human review. |
Page 9 of 31 Shubhank Suman, TMT, IP & AI Counsel. All Rights Reserved.
Data Privacy & AI Regulation | Shubhank Suman, TMT, IP & AI Counsel
it affects them, and how they can challenge outcomes. | |||
8 | Vendor Data Controls & DPAs | Deployers must execute DPAs that prohibit vendors from using inputs for unrelated purposes like model training. | A telecom company amends a chatbot vendor contract to ban reuse of customer complaints for vendor model training. |
9 | Ongoing Purpose Drift Monitoring | Organisations must monitor whether AI use expands beyond the purpose originally assessed. | A compliance review flags that a sales-forecasting model is being repurposed for employee performance scoring without approval. |
10 | Continuous Review & Retirement Planning | AI systems must be periodically reviewed and retired when the original purpose expires. | An AI built for a pandemic-era triage system is decommissioned once emergency use ends instead of being quietly repurposed. |
Page 10 of 31 Shubhank Suman, TMT, IP & AI Counsel. All Rights Reserved.
Data Privacy & AI Regulation | Shubhank Suman, TMT, IP & AI Counsel
03 DPDPA Vs. GDPR: Two Frameworks, One Compliance Problem
For any organisation operating across India and Europe -or building AI products for both markets -the differences between the DPDPA and the GDPR are not academic. They create real, practical divergences in what is legally permissible. Understanding those differences is essential for designing a compliance strategy that actually works.
The Core Philosophical Difference
India's DPDPA is primarily built around consent. The default legal basis for processing personal data under the Act is a clear, specific, informed, and unambiguous agreement from the individual concerned. The Act does provide for processing without consent in a defined set of circumstances called 'certain legitimate uses', but these exceptions are narrow and specifically listed. Crucially, the DPDPA does not give private companies a general 'legitimate interests' ground -the kind of broad, balancing-test mechanism that allows a company to process data if its interests are sufficiently compelling and do not unduly harm the individual's rights.
The GDPR takes a more pluralistic approach. Article 6 provides six potential lawful bases, including consent, contractual necessity, legal obligation, vital interests, public task, and legitimate interests. This gives European organisations significantly more flexibility in how they structure the legal basis for AI-related data processing. However, and this point is frequently misunderstood, that flexibility does not mean freedom. Each basis comes with its own conditions, and legitimate interests in particular requires a rigorous three-part balancing test: the interest must be real and lawful, the processing must be necessary, and it must not be outweighed by the individual's rights.
Case Study - Two Countries, Two Rules: The Recruitment AI Problem
Imagine a global staffing company that has built an AI tool to screen job applications. It wants to use three years of historical recruitment data -thousands of CVs, cover letters, and interview scores -to train the model. In its European operations, the company relies on 'legitimate interests' as its legal basis, arguing that improving hiring efficiency is a genuine business need and the impact on applicants is proportionate. It conducts a balancing test, documents its reasoning, and proceeds. In its Indian operations, the same company attempts to use the same approach. It cannot. The DPDPA does not recognise legitimate interests as a general ground for private-sector processing. The company must instead obtain fresh, specific, and informed consent from each former applicant whose data it wants to use -or it must redesign the project entirely.
What the law says: This case illustrates why a single global compliance policy often fails. What is a valid legal basis in Europe may simply not exist in India. Organisations must run jurisdiction-specific legal basis assessments for every AI project that processes personal data, rather than applying a single one-size-fits-all analysis.
Page 11 of 31 Shubhank Suman, TMT, IP & AI Counsel. All Rights Reserved.
Data Privacy & AI Regulation | Shubhank Suman, TMT, IP & AI Counsel
What This Means In Practice For AI
The practical consequences of this difference are significant, particularly for AI training and model development. An Indian company that wants to use customer data to train a recommendation engine will generally need fresh, specific consent from each customer for that purpose. It cannot rely on an argument that the training serves a legitimate business interest and the impact on individuals is low. That reasoning simply does not provide a valid legal basis under the DPDPA.
A European company in the same situation has more options. It may be able to rely on legitimate interests, but only after conducting a genuine and documented balancing test, and only if it has implemented meaningful safeguards and given individuals a clear way to object. The European Data Protection Board has been explicit that AI training is not automatically compatible with the original purpose of data collection, and that 'we find it useful' is not sufficient justification.
Real-World Gap: Many Global Organisations Run A GDPR-First Strategy That Fails In India
Organisations operating across both jurisdictions often design their data processing architecture around GDPR compliance and assume it satisfies Indian law as well. This assumption is wrong in a critical respect: legitimate interests, which the GDPR allows as a basis for many AI-related activities, does not exist as a general ground under the DPDPA. A compliance programme that relies on legitimate interests as its primary basis for AI training in India is built on a legal foundation that does not exist under Indian law. Organisations must run jurisdiction-specific legal basis assessments.
DPDPA Vs. GDPR – Key Differences In Regulating AI-Driven Processing Of Personal Data
Feature | DPDPA | GDPR |
|---|---|---|
Primary legal basis for AI processing | Consent-centric framework. In most private-sector AI use cases (training, profiling, prediction), organisations must rely on clear, specific, informed consent from individuals, unless a narrow statutory exception applies. This significantly limits large-scale AI training on existing datasets. | Multiple lawful bases available. AI systems may be trained or deployed using consent, contract, legal obligation, or legitimate interests, giving organisations greater flexibility to justify AI processing without fresh consent in some cases. |
Legitimate interests as a basis for AI | Not recognised as a general ground for private entities. AI training or model improvement cannot generally be justified on “business interest” grounds alone. This makes broad AI experimentation on historical customer or employee data legally risky in India. | Explicitly available, subject to a three-part balancing test (legitimate interest, necessity, rights impact). This has become a commonly used basis for AI training and analytics in Europe, though heavily scrutinised by regulators. |
Page 12 of 31 Shubhank Suman, TMT, IP & AI Counsel. All Rights Reserved.
Data Privacy & AI Regulation | Shubhank Suman, TMT, IP & AI Counsel
Purpose limitation in AI training and reuse | Strict purpose binding. Data collected for one purpose (e.g., food delivery, HR administration) cannot be reused for AI training or prediction unless the new purpose was specifically disclosed and consented to. Purpose drift in AI models is high-risk. | Compatibility analysis permitted. New AI uses may be allowed if they are considered compatible with the original purpose, taking into account expectations, safeguards, and impact on individuals. This enables broader internal AI reuse, though not automatically. |
|---|---|---|
Use of customer / employee data for AI training | Highly constrained. Using existing customer or employee data to train AI models usually requires fresh, explicit consent, even if the data was lawfully collected earlier. Retrospective AI training is often unlawful. | More permissive but conditional. Historical data may sometimes be reused for AI training under legitimate interests or compatibility analysis, provided safeguards, transparency, and objection rights are ensured. |
Children’s data and AI systems | Parental consent mandatory. Processing children’s data for AI systems must avoid any activity that is harmful to a child’s wellbeing, including profiling or behavioural inference for commercial gain. AI systems involving children are high-risk. | Stricter baseline protections. Parental consent generally required under 16 (or lower age set by Member State). AI profiling of children is heavily discouraged and subject to strict scrutiny by regulators. |
Automated decision-making by AI | Principles-based obligations. Emphasis on fairness, transparency, and grievance redressal. Detailed rules for AI-driven automated decisions are still evolving through subordinate legislation and guidance. | Article 22 provides explicit rights. Individuals have rights relating to decisions based solely on automated processing with legal or similarly significant effects, including human intervention, explanation, and the right to contest decisions. |
AI explainability and transparency | Emerging but implicit. Transparency obligations exist, but specific AI explainability standards are not yet detailed. Organisations must still explain AI use where it affects individual rights. | Well-developed jurisprudence. GDPR requires “meaningful information about the logic involved” in automated decisions, driving explainability, documentation, and model interpretability requirements for AI systems. |
Right to erasure vs. AI model training | Right to withdraw consent and request deletion. Creates serious challenges for AI models trained on personal data, especially where data influence cannot easily be removed from trained models. | Broader erasure right (Article 17). Includes the “right to be forgotten”. Regulators have recognised that AI models may still constitute personal data if they can memorise or reproduce training data. |
Page 13 of 31 Shubhank Suman, TMT, IP & AI Counsel. All Rights Reserved.
Data Privacy & AI Regulation | Shubhank Suman, TMT, IP & AI Counsel
Cross-Border Data Transfers For AI Platforms
Government-controlled approach. The government may notify restricted countries. AI vendors hosting or training models outside India must monitor evolving transfer rules closely.
Established transfer mechanisms. Transfers allowed to adequate countries or via Standard Contractual Clauses (SCCs), which are widely used by AI vendors and cloud providers.
The Grey Areas: Where Both Frameworks Struggle
Web Scraping For AI Training
Large-scale scraping of publicly available internet data is one of the most contested areas in AI regulation. The instinct of many developers is that if data is publicly visible, it is freely usable. This is wrong under both frameworks. Under the DPDPA, the absence of a legitimate interests ground makes it very difficult to justify mass scraping of personal data without consent from the individuals concerned. The fact that someone's employer is visible on a professional networking site does not mean they have consented to that information being copied into an AI training set at scale.
Under the GDPR, scraping may sometimes be defensible through legitimate interests, but regulators have been sceptical. Italy's data protection authority took enforcement action against a major AI company over exactly this issue, finding that scraping personal data without a valid legal basis and without meaningful notice to individuals constituted unlawful processing.
Case Study Your Linkedin Profile And The AI Training Set
You update your LinkedIn profile with your current job title, employer, city, skills, and educational background. You are networking with colleagues and potential employers. You are not signing up to have your information scraped, aggregated with millions of other profiles, and fed into an AI model that a company then sells commercially. But that is exactly what several AI companies have done with publicly available professional profile data. The reasoning was simple: the data is public, so it must be free to use.
What the law says: Both Indian and European regulators have rejected this logic. Under the DPDPA, there is no general legitimate interests ground that would cover this kind of mass scraping of personal data for commercial AI training. Under the GDPR, even the legitimate interests basis requires a genuine balancing test, and regulators have consistently found that the individual's reasonable expectation of privacy outweighs the commercial interest of the scraper. Visibility online is not the same as consent to be used.
Indefinite Data Retention
Both frameworks require data to be kept only for as long as is necessary for the purpose for which it was collected. 'We might need it for AI development later' is not a purpose. It is an indefinite reservation with no defined end point. Organisations that keep personal data in reserve for potential future AI projects -Page 14 of 31 Shubhank Suman, TMT, IP & AI Counsel. All Rights Reserved.
Data Privacy & AI Regulation | Shubhank Suman, TMT, IP & AI Counsel without a defined retention schedule and a plan for deletion -are in breach of the storage limitation principle under both frameworks. This is one of the easiest failures for regulators to identify, because the absence of a retention policy is visible on its face.
Page 15 of 31 Shubhank Suman, TMT, IP & AI Counsel. All Rights Reserved.
Data Privacy & AI Regulation | Shubhank Suman, TMT, IP & AI Counsel
04 Privacy Issues During AI Training
The training phase is where the most serious and most durable privacy risks are created. Unlike deployment-phase failures, which can often be corrected through policy changes or system adjustments, failures during training become embedded in the model itself. Understanding why this is so -and what the law requires to prevent it -is essential for anyone involved in AI development.
Why Training Is The Highest-Risk Stage
When personal data is used to train a machine learning model, it does not simply pass through the system and leave. It shapes the model's internal parameters -the mathematical weights that determine how the model responds to inputs. Once this process is complete, deleting the original dataset does not remove the influence of that data on the model. The model has already learned from it.
The legal consequences of this are significant. A compliance failure at the training stage does not disappear when the training run is finished. It travels with the model into every subsequent use. This is what regulators refer to as the 'cascade effect' of unlawful training: a tainted foundation contaminates everything built on top of it.
The EDPB's Position On Model Anonymity
A common misconception is that once training is complete, the personal data disappears into the model and ceases to be personal data. The European Data Protection Board has directly rejected this view. A model trained on personal data may still be capable of reproducing, revealing, or inferring information about individuals when queried in the right way -a phenomenon known as memorisation. Such models may still involve the processing of personal data, and the lawfulness of the training phase therefore remains legally relevant after deployment.
The Five Core Training-Phase Risks
1. Unlawful Data Sourcing
Training datasets are frequently assembled from scraped websites, purchased third-party datasets, or large collections of records whose legal provenance has not been adequately verified. A dataset can look professionally compiled on the surface while containing serious underlying legal problems: data collected without valid consent, data obtained in breach of platform terms of service, or data taken from a context where individuals had no reasonable expectation of AI use.
Case Study The Hospital That Fed Its Records Into An AI Without Asking Patients
A large private hospital group in India had a decade of patient records: diagnoses, prescriptions, lab results, and treatment histories. A technology company approached the hospital with a proposal to
Page 16 of 31 Shubhank Suman, TMT, IP & AI Counsel. All Rights Reserved.
Data Privacy & AI Regulation | Shubhank Suman, TMT, IP & AI Counsel use this data to build an AI model that could predict early disease onset. The hospital agreed and transferred ten years of records to the company's servers. Patients had consented, when they first registered at the hospital, to their data being used for their treatment and for the hospital's internal quality improvement. They had not consented to their records being handed to a third-party technology company to train a commercial AI model.
What the law says: Under the DPDPA, processing personal health data requires not just general consent but specific, informed consent for each distinct purpose. Using patient records for treatment and using them to build a commercial AI product are two distinct purposes. The hospital could not point to the original registration consent as justification for the transfer. A fresh consent exercise was required -and because many patients would have been difficult to contact years later, much of the historical data could not lawfully be used at all.
2. Memorisation Risk
Large language models and generative AI systems can, in certain conditions, memorise verbatim fragments of their training data and reproduce them when prompted appropriately. This has been demonstrated empirically across multiple commercially deployed systems, where targeted prompting has caused models to repeat names, addresses, identification numbers, and other sensitive personal details drawn from training data.
From a legal perspective, memorisation risk means that the model itself may constitute a storage medium for personal data. If an individual exercises a right to erasure, deleting the source record may not be sufficient if the information has been absorbed into the model's parameters. Technical solutions -notably machine unlearning -are developing to address this, but the legal obligation to address the risk does not wait for the technology to mature.
3. Purpose Drift
Purpose drift occurs when data collected for one clear reason is later reused for a materially different purpose, often without any fresh legal basis or new notice to individuals. It is among the most common compliance failures in AI development, and it is easy to overlook internally because the reuse often feels natural to the team involved.
Case Study The Bank That Trained A Biased Loan Model
A major bank decided to build an AI system to automate loan approval decisions. The team pulled together ten years of historical loan application data, including applicant demographics, employment history, credit scores, and -crucially -the bank's own past approval and rejection decisions. The model was trained to predict future default risk based on what had happened historically. What the team did not adequately account for was that the historical decisions themselves reflected the biases of the human loan officers who made them. During a period when the bank had informal policies that made it harder for applicants from certain postcodes, certain professions, and certain income brackets to get loans, those patterns were baked into the training data. The AI learned that people from those
Page 17 of 31 Shubhank Suman, TMT, IP & AI Counsel. All Rights Reserved.
Data Privacy & AI Regulation | Shubhank Suman, TMT, IP & AI Counsel categories were 'higher risk' -not because they actually were, but because the bank had historically treated them that way.
What the law says: Under GDPR Article 5(1)(a), processing must be fair. A model that reproduces historical discrimination does not produce fair outcomes, even if the model itself is technically neutral. The EU AI Act also requires bias testing for high-risk systems including credit scoring. Using biased historical data without conducting a bias audit and correcting for documented unfairness is a legal failure, not just an ethical one. The bank would have been obliged to identify the bias in the training data, adjust for it, and document the steps taken before deployment.
4. Invisible Processing
The vast majority of individuals whose data is used to train AI models have no knowledge that it has happened. There is typically no notice, no opportunity to object, and no subsequent explanation of how their data influenced the model. This strikes at a foundational principle of both the GDPR and the DPDPA: that individuals are entitled to know how their personal information is being used. Adding a retrospective notice after training is complete does not cure this failure. The processing has already occurred, and the individual's right to be informed cannot be restored after the fact.
5. The Right To Erasure Problem
Under GDPR Article 17 and the DPDPA's erasure provisions, individuals can ask for their personal data to be deleted. For AI-trained models, this right creates a genuine technical and legal challenge. If the data has already been used to shape the model's parameters, deleting the original record does not remove what the model has learned from it. Organisations may need technical tools such as machine unlearning to address this. Another practical option is to reduce risk at the outset by limiting how much personally identifiable data is included in training sets in the first place.
Training-Phase Risk Summary
Risk | What It Means | Applicable Legal Duty | Most Common Real-World Failure |
|---|---|---|---|
Unlawful data sourcing | Training data collected without valid legal basis | Lawful basis: GDPR Art. 6 / DPDPA S.4 | Datasets purchased from third parties without legal provenance check |
Memorisation | Model retains and can reproduce personal details | Erasure rights: GDPR Art. 17 / DPDPA erasure provisions | No machine unlearning capability built into model architecture |
Page 18 of 31 Shubhank Suman, TMT, IP & AI Counsel. All Rights Reserved.
Data Privacy & AI Regulation | Shubhank Suman, TMT, IP & AI Counsel
Risk | What It Means | Applicable Legal Duty | Most Common Real-World Failure |
|---|---|---|---|
Invisible processing | Individuals not informed their data is being used | Transparency: GDPR Art. 13-14 / DPDPA S.5 | No notice given at or after data collection for AI purposes |
Purpose drift | Data reused for materially different AI purpose | Purpose limitation: GDPR Art. 5(1)(b) / DPDPA S.4 | Customer or employee data reused for AI training without fresh legal basis |
Biased training data | Model learns and reproduces historical unfairness | Fairness: GDPR Art. 5(1)(a) / EU AI Act bias requirements | No bias audit conducted before or after training |
Page 19 of 31 Shubhank Suman, TMT, IP & AI Counsel. All Rights Reserved.
Data Privacy & AI Regulation | Shubhank Suman, TMT, IP & AI Counsel
05 Privacy Issues When Organisations Use AI
A common misconception is that the main legal risk in AI sits with the companies that build and train models. In reality, organisations that deploy and use AI systems -even commercially available, off-the-shelf products -carry their own substantial layer of legal obligations. Compliance does not end at the vendor's door.
Automated Decision-Making And Individual Rights
When an AI system makes or significantly influences a decision that affects an individual -approving or rejecting a loan, screening a job application, calculating an insurance premium, or restricting access to a public service -specific legal duties arise. Under GDPR Article 22, individuals have the right not to be subject to a decision based solely on automated processing if that decision produces significant legal or similarly significant effects on them. They are entitled to request human review, to receive a meaningful explanation of the logic behind the decision, and to contest the outcome.
Case Study Rejected By An Algorithm With No Explanation
Priya applies for a personal loan through a bank's mobile app. Within thirty seconds, she receives an automated message: 'Your application has not been approved at this time.' There is no phone number to call. There is no explanation of which factor led to the rejection. There is no information about what she could do differently. A human loan officer never looked at her file. An AI model assessed her credit score, her employer, her age, her postcode, and her past transaction patterns, assigned her a risk score below the threshold, and rejected her -all without any human involvement at any stage.
What the law says: Under GDPR Article 22, Priya has the right to request human review of this decision, to understand the main factors that led to it, and to challenge it if she believes it is incorrect. The bank cannot tell her 'the algorithm decided' and consider the matter closed. It must be able to provide a meaningful explanation in plain language and must have a functioning human review process. In India, while Article 22's exact equivalent is still developing in DPDPA secondary legislation, the principles of fairness and grievance redressal under the Act create analogous practical duties. Organisations that build automated decision-making systems without these mechanisms are taking a legal risk that is increasing as regulation tightens.
Shadow AI: The Internal Threat
Shadow AI refers to the use of AI tools by employees outside the knowledge or approval of the organisation's IT and compliance functions. It typically begins with small, well-intentioned shortcuts. A lawyer pastes a client's contractual terms into a commercial summarisation tool to save time. An HR manager uploads a performance review to an AI writing assistant to improve the language. A finance analyst feeds confidential projections into a public chatbot to get a quick draft.
Page 20 of 31 Shubhank Suman, TMT, IP & AI Counsel. All Rights Reserved.
Data Privacy & AI Regulation | Shubhank Suman, TMT, IP & AI Counsel
Each of these actions may feel trivial in the moment. In legal terms, each may constitute a personal data breach. The information has been transmitted to an external processor with which the organisation has no Data Processing Agreement, whose data handling practices have not been assessed, and whose default settings may involve using inputs for model improvement. A single employee action can create a compliance failure that the organisation never intended and cannot easily undo.
Case Study The HR Manager And The Chatbot
Neha works in the HR department of a large retail company. She is writing a termination letter for an employee named Rahul, who is being let go for performance reasons. She has his full name, employee ID, salary details, performance review scores, and some personal background notes that his manager wrote. Neha finds the letter difficult to phrase sensitively, so she copies all of this information into a free online AI writing tool to help her draft it better. The tool suggests a cleaner version, she copies it into her email, and sends it. What Neha did not know is that the free AI tool retains all inputs for model improvement purposes and its servers are located in a country outside India with no data transfer agreement in place.
What the law says: Rahul's personal employment data -his name, salary, performance records, and reasons for dismissal -has now been transmitted to an unvetted third-party processor outside India without a Data Processing Agreement, without a valid cross-border transfer mechanism, and without the company's knowledge or approval. This is a data breach under both the DPDPA and the GDPR, even though Neha had no malicious intent whatsoever. Shadow AI typically begins with helpfulness. It ends with a compliance crisis.
Cross-Border Data Transfers
The majority of commercial AI platforms are operated from the United States or other countries outside India and the EU. Every time personal data is sent to such a platform for processing, an international data transfer occurs. This is not a background administrative formality. It is a legally significant act that requires a valid transfer mechanism.
Under the GDPR, transfers to third countries require either an adequacy decision from the European Commission, Standard Contractual Clauses incorporated into the relevant contract, or another approved safeguard mechanism. Under the DPDPA, the Indian government has the power to restrict transfers to certain countries, and organisations must track developments in this area as secondary legislation continues to evolve. Signing up to a vendor's standard terms and conditions does not satisfy these requirements.
Vendor Relationships And The Controller-Processor Problem
When an organisation uses a third-party AI tool to process personal data, it remains the data controller. It is still the main party responsible for what happens to that data in the vendor's hands. This relationship must be formalised in a Data Processing Agreement specifying what the vendor may do with the data,
Page 21 of 31 Shubhank Suman, TMT, IP & AI Counsel. All Rights Reserved.
Data Privacy & AI Regulation | Shubhank Suman, TMT, IP & AI Counsel how it will be protected, which sub-processors are involved, how long data will be retained, and what happens in the event of a breach.
One of the most important provisions in any such agreement concerns model training. Many commercial AI platforms, by default, use customer inputs to improve their own models. An employee who submits personal data to such a platform may inadvertently authorise its use for a purpose that the organisation's customers never consented to. Organisations must either contractually prohibit this use or confirm that a separate and valid legal basis exists for it.
Case Study The Customer Service Bot That Was Learning From Your Complaints
A telecom company deploys a commercial AI chatbot to handle customer complaints. The chatbot vendor's standard terms of service state, in clause 14.3 of a forty-page document, that conversation data may be used to improve the vendor's AI models. Customer complaints about billing errors, service disruptions, and account details are now being used to train the vendor's commercial product. The telecom company's customers never agreed to this. The telecom company itself never noticed clause 14.3 before signing the agreement.
What the law says: The telecom company is the data controller. It chose to use the vendor, it presented the chatbot to its customers, and it is legally responsible for what the vendor does with the data. A vendor's privacy policy or standard terms are not a substitute for a proper Data Processing Agreement. The DPA must specify exactly what the vendor is and is not permitted to do with personal data -and permitting the vendor to use customer conversations to train its own commercial models almost certainly requires separate consent from those customers, which was never obtained.
Page 22 of 31 Shubhank Suman, TMT, IP & AI Counsel. All Rights Reserved.
Data Privacy & AI Regulation | Shubhank Suman, TMT, IP & AI Counsel
06 The EU AI Act: A New Layer Of Regulation
The EU AI Act, which entered into force in 2024 with phased implementation through 2027, represents a qualitative shift in how AI is regulated. Where the GDPR addresses AI through the lens of data protection, the AI Act addresses it as a distinct category of technological risk. The two frameworks overlap substantially -a high-risk AI system is almost invariably a significant data processing activity -but they are not the same thing. Compliance with one does not imply compliance with the other.
The easiest way to understand the AI Act is as a risk classification system -similar to the safety ratings used for medicines, electrical appliances, or food products. The more serious the potential harm a system can cause, the more stringent the obligations on those who build and deploy it.
The Four-Tier Risk Framework
Risk Tier | Definition | Examples | Legal Consequence |
|---|---|---|---|
Unacceptable Risk | Systems that pose a threat to fundamental rights or safety that cannot be mitigated | Social scoring by public authorities; AI exploiting psychological vulnerabilities; real-time biometric surveillance in public spaces | Outright prohibition. These systems may not be deployed in the EU under any circumstances. |
High Risk | Systems used in contexts where errors can cause significant harm to individuals' rights or opportunities | CV screening tools; credit scoring; medical diagnosis support; law enforcement prediction; educational assessment; critical infrastructure | Mandatory conformity assessment, technical documentation, human oversight, bias testing, transparency to affected individuals, registration in EU database |
Limited Risk | Systems that interact with humans without being clearly identified as AI | Customer-facing chatbots; AI-generated content; deepfakes | Disclosure obligation: users must be clearly informed they are interacting with an AI system |
Minimal Risk | Systems with low potential for harm | Spam filters, AI-powered games, basic recommendation engines | No mandatory obligations under the AI Act, but GDPR still applies if personal data is involved |
Page 23 of 31 Shubhank Suman, TMT, IP & AI Counsel. All Rights Reserved.
Data Privacy & AI Regulation | Shubhank Suman, TMT, IP & AI Counsel
Case Study The Cv Screening Tool And The High-Risk Classification
A logistics company with 5,000 employees receives around 800 job applications every month. It deploys an AI-powered CV screening tool that reads each application and assigns a score from 1 to 100. Only applicants who score above 70 are passed to a human recruiter for review. The rest are rejected automatically. No human looks at the applications that score below 70. The AI considers factors including the applicant's previous employers, educational institutions, years of experience, and the language and formatting of the CV.
What the law says: Under the EU AI Act, an AI system used in employment recruitment that makes or significantly influences decisions about whether an individual is considered for a role is classified as a high-risk system. This means the company must conduct a conformity assessment before deployment, maintain detailed technical documentation, implement human oversight mechanisms (a score below 70 cannot alone be the final word), test the system for accuracy and bias against different demographic groups, and provide affected applicants with meaningful information about the role of AI in the process. The fact that the tool is commercially purchased rather than built in-house does not reduce these obligations. The deployer -the logistics company -is responsible for ensuring compliance in the specific context of its own use.
What High-Risk Classification Means In Practice
For organisations using AI in recruitment, lending, healthcare, or public services -which covers a large proportion of enterprise AI deployment -the high-risk classification is the most consequential. A system classified as high risk must, before deployment, undergo a conformity assessment demonstrating that it meets the Act's technical requirements. It must have human oversight mechanisms built in -meaningful oversight, not a rubber stamp. It must be tested for accuracy, robustness, and bias. And affected individuals must be able to obtain a meaningful explanation of how the system affected any decision that concerned them.
Case Study The Chatbot That Did Not Say It Was A Bot
A financial services company deploys an AI chatbot on its website to handle customer queries about savings accounts and investment products. The chatbot is designed to sound conversational and uses a human name ('Hi, I'm Ananya, how can I help you today?'). Customers who ask complex questions about suitability of products receive recommendations from 'Ananya' without knowing they are interacting with an AI. Some customers make financial decisions based on those recommendations.
What the law says: Under the EU AI Act's limited risk provisions, a chatbot that interacts with humans must clearly disclose that it is an AI system. Using a human name and a conversational tone without any indication that the user is speaking with a machine is a direct violation of this disclosure requirement. The obligation exists regardless of how sophisticated or helpful the chatbot is. Users have a right to know when they are not speaking with a human -particularly when the conversation involves financial advice that could influence their decisions.
Page 24 of 31 Shubhank Suman, TMT, IP & AI Counsel. All Rights Reserved.
Data Privacy & AI Regulation | Shubhank Suman, TMT, IP & AI Counsel
The Cascade Effect Of Unlawful Training
One of the most important ideas that emerges from reading the GDPR and the EU AI Act together is the cascade effect of unlawful training. If a model was trained on data that was collected or processed unlawfully, that legal defect does not stay at the training stage. It travels with the model into every subsequent deployment. A model built on improperly sourced data may itself be considered the product of unlawful processing -and every decision it makes may be tainted by that original defect. Good operational controls after deployment cannot cure a legal failure at the training stage. The legal foundation must be sound from the beginning.
Page 25 of 31 Shubhank Suman, TMT, IP & AI Counsel. All Rights Reserved.
Data Privacy & AI Regulation | Shubhank Suman, TMT, IP & AI Counsel
07 Compliance Framework: The Full AI Lifecycle
AI compliance is not a project with a defined end date. It is an ongoing discipline that must be embedded into the full lifecycle of every AI system that handles personal data. The framework below maps the key legal obligations and practical requirements to each phase of that lifecycle.
Stage 1: Data Collection And Sourcing
Before any data is identified for AI purposes, the organisation must establish a valid legal basis for its collection and use. Under the GDPR, this means identifying the appropriate Article 6 ground and, for sensitive data, an additional condition under Article 9. Under the DPDPA, it typically means obtaining valid, specific, and informed consent unless a specific statutory exception applies. This legal basis should be documented in writing and tied to the specific AI purpose, not expressed in vague or general terms.
A Data Protection Impact Assessment must be completed where the processing is likely to create a high risk to individuals. For AI training on large datasets, this will almost invariably be required. The DPIA is not bureaucratic overhead. It is a substantive analysis that forces the organisation to think clearly about what could go wrong, who could be harmed, and what controls are necessary.
Case Study The Healthtech Startup That Got It Right
A Bengaluru-based healthtech startup wanted to build an AI model to help general practitioners identify early signs of Type 2 diabetes from routine blood test results. Before writing a single line of training code, the team brought in their legal counsel and data protection officer. Together, they identified the lawful basis for collecting data (patient consent, obtained separately for AI training purposes), drafted a clear and simple consent notice explaining exactly what the AI would do and who would see its outputs, conducted a full DPIA identifying the risks to patient privacy, put in place pseudonymisation protocols so that no patient's name or Aadhaar number would appear in the training set, and agreed on a data retention schedule -training data would be deleted eighteen months after the model's first production deployment. The entire process took six weeks before a single patient record was collected.
What the law says: This is what the legal framework requires. The startup's approach was not unusual for its diligence -it was simply what the law asks organisations to do. The six-week investment at the start eliminated months of potential legal exposure later. The consent notices they drafted also became a competitive advantage: patients trusted the startup's transparent approach and were more willing to participate.
Stage 2: Model Training
During training, the organisation must enforce the purpose controls established at the collection stage. Data gathered for one purpose must not be diverted into AI training without a separately assessed and
Page 26 of 31 Shubhank Suman, TMT, IP & AI Counsel. All Rights Reserved.
Data Privacy & AI Regulation | Shubhank Suman, TMT, IP & AI Counsel documented legal basis. Where the training dataset contains directly identifiable personal data, the organisation should apply anonymisation or pseudonymisation to the greatest extent technically feasible.
Bias audits should be conducted before training begins, not as an afterthought after the model is built. The audit should examine the training data for historical imbalances, discriminatory proxies, and underrepresentation that could cause the model to produce unfair outputs. The results must be acted upon, not merely recorded. If significant bias is identified in the training data, the data should be corrected before training proceeds.
Stage 3: Testing And Validation
Before deployment, the organisation must test the model for accuracy, consistency, and fairness. Privacy-specific testing should cover memorisation risk (can the model be prompted to reproduce training data?), purpose drift (is the model being used in ways that were not contemplated at the design stage?), and data quality (is the model making decisions based on inaccurate or outdated information?). If testing reveals that the model produces biased, unfair, or inaccurate outputs, those issues must be fixed before deployment. Launching a system that is known to perform badly is not acceptable when the system affects individual rights.
Stage 4: Deployment
When the system goes live, Privacy by Default must be in place. Access controls must be implemented so that only authorised personnel and systems can interact with personal data through the AI. AI use policies must be written clearly and communicated specifically to the people who need to follow them. Where the system makes or significantly influences decisions about individuals, transparency mechanisms must be available -in plain language, at the point of the decision, explaining that AI was used and how it can be challenged. Vendor agreements must be finalised before personal data is shared with any third-party AI platform.
Stage 5: Ongoing Use And Monitoring
The most important thing to understand about AI compliance is that it does not end at deployment. The system must be monitored throughout its active life for continued accuracy, fairness, and consistency with the legal basis on which it was deployed. If the context changes -new data sources, a wider user base, a new business application -a fresh compliance assessment is required. A model that was lawful in a small pilot may not be lawful when expanded to full production scale.
Lifecycle Stage | Key Legal Obligations | Critical Documents Required | Most Common Failure |
|---|---|---|---|
Data Collection | Valid legal basis; DPIA; data minimisation; | Legal basis assessment; DPIA report; privacy notice; consent | Legal basis not documented before collection begins; DPIA skipped |
Page 27 of 31 Shubhank Suman, TMT, IP & AI Counsel. All Rights Reserved.
Data Privacy & AI Regulation | Shubhank Suman, TMT, IP & AI Counsel
Lifecycle Stage | Key Legal Obligations | Critical Documents Required | Most Common Failure |
|---|---|---|---|
transparency to individuals | records where applicable | ||
Model Training | Purpose control; bias audit; pseudonymisation where possible; deletion-ready architecture | Purpose compatibility analysis; bias audit report; technical specification; data processing records | Training data reused without fresh legal basis; no bias audit conducted |
Testing and Validation | Accuracy testing; fairness testing; memorisation risk assessment; EU AI Act conformity for high-risk systems | Test results; privacy impact review; conformity assessment documentation where applicable | Testing focused on technical performance, not legal or privacy compliance |
Deployment | Privacy by default; access controls; transparency mechanisms; DPAs with vendors; AI use policy | AI use policy; vendor DPAs; transparency notices; access control records | Default settings not reviewed; vendor DPA not executed before data sharing begins |
Ongoing Use | Continuous monitoring; erasure rights management; transfer safeguard review; incident response readiness | Monitoring logs; erasure request records; transfer mechanism documentation; incident response plan | System treated as done after launch; monitoring and reviews not resourced or scheduled |
Page 28 of 31 Shubhank Suman, TMT, IP & AI Counsel. All Rights Reserved.
Data Privacy & AI Regulation | Shubhank Suman, TMT, IP & AI Counsel
Key Takeaways And Conclusion
The Seven Foundational Obligations
The legal requirements examined in this report can be distilled into seven foundational obligations that apply to any organisation that uses AI to process personal data. These are not policy aspirations. They are enforceable legal duties.
Obligation | What It Requires | The Most Common Way It Is Violated |
|---|---|---|
Lawful Collection | A documented, specific, and valid legal basis must exist before any data is collected for AI purposes. The basis must be able to withstand regulatory scrutiny. | Work begins before legal basis is confirmed. The basis is stated in vague or general terms that do not cover the specific AI use contemplated. |
Purpose- Specific Use | Data may only be used for the purpose for which it was collected, unless a separately assessed compatible use exists. Real technical controls must enforce this, not just policy statements. | Customer, employee, or user data is silently redirected to AI training without a fresh legal basis or any notice to individuals. |
Data Minimisation | Only the smallest amount of personal data necessary for the defined task should be collected, processed, and retained. This applies at every stage: collection, training, testing, and deployment. | Business teams insist on collecting everything that might be useful. No minimum-data analysis is conducted before training. Unnecessary identifiers are retained in training sets. |
Retention Control | Data must be deleted when the purpose for which it was collected is complete. Retention schedules must be defined, automated where possible, and actually enforced. | Data is kept indefinitely in case it is useful for future AI development. No retention schedule exists. Automated deletion is not implemented. |
Transparency | Individuals must be told, in plain language, that AI is being used, what data it relies on, how automated decisions are reached, and what rights they have to challenge outcomes. | AI is not disclosed to affected individuals. Explanations are buried in privacy policies. No human review mechanism is available for automated decisions. |
Page 29 of 31 Shubhank Suman, TMT, IP & AI Counsel. All Rights Reserved.
Data Privacy & AI Regulation | Shubhank Suman, TMT, IP & AI Counsel
Obligation | What It Requires | The Most Common Way It Is Violated |
|---|---|---|
Compliant Transfers | International transfers of personal data to AI vendors must be covered by a valid legal mechanism -adequacy, SCCs, or equivalent -not just the vendor's standard terms. | Organisations assume that signing up to a major vendor's terms satisfies transfer requirements. No SCCs or other transfer mechanisms are in place. |
Accountability | Organisations must be able to demonstrate, at any time, that their AI-related data processing is lawful, fair, transparent, and proportionate. Documentation is not optional -it is evidence. | Decisions are made verbally or informally. Legal basis, DPIA results, vendor assessments, and compliance checks are not recorded. Regulators find nothing to review. |
The Real-World Gaps: What Most Organisations Are Missing
This report has identified a consistent set of compliance gaps that appear across organisations of all sizes and sectors. They are worth restating plainly, because acknowledging them is the first step toward closing them.
Most organisations do not know the legal provenance of their training data. They purchase datasets, use existing records, or scrape publicly available content without conducting a genuine legal basis assessment. The assumption that 'if we have it, we can use it' is not a legal basis.
Most commercial AI deployments do not have properly executed Data Processing Agreements with vendors. Standard terms of service are not DPAs. Organisations that process personal data through third-party AI platforms without a DPA are in breach of both the GDPR and the DPDPA.
Automated decision-making is frequently deployed without adequate transparency or human review mechanisms. The ability to tell an individual why an AI made a decision about them -and how they can challenge it -is not optional. It is a legal requirement.
Cross-border transfer safeguards are routinely overlooked for AI platforms. The fact that a vendor is large and reputable does not make the transfer lawful. The legal mechanism must be documented separately.
Privacy by Design and Privacy by Default are treated as aspirational principles rather than practical requirements. Most organisations have no mechanism for enforcing purpose controls in their AI infrastructure, and most commercial AI tools are deployed with default settings that are not privacy-optimised.
Page 30 of 31 Shubhank Suman, TMT, IP & AI Counsel. All Rights Reserved.
Data Privacy & AI Regulation | Shubhank Suman, TMT, IP & AI Counsel
Conclusion: Legality Must Be Built In
The intersection of artificial intelligence and data protection law is one of the defining legal challenges of the present decade. The frameworks are evolving: the DPDPA's secondary legislation is still being developed, the EU AI Act is in phased implementation, and regulators on both sides of the world are still working out how their powers apply to novel AI architectures. But the core obligations are not new or uncertain. They have been present in data protection law for years. What is new is the scale, the speed, and the stakes.
The fundamental lesson of this report is simple: legality must be built into AI systems from the beginning, not added at the end. A compliance problem at the training stage travels with the model into every subsequent use. A tainted legal foundation cannot be repaired by good operational practice in production. And a regulatory framework that is treated as a post-launch problem will always arrive too late.
Organisations that build privacy compliance into the design of their AI systems -treating it as an engineering and legal discipline rather than a box-ticking exercise -will be better positioned to meet current legal duties, adapt as the rules continue to develop, and earn the trust of the individuals whose data they depend upon. Those that do not will find that the regulators, courts, and human rights bodies are increasingly well-equipped to hold them to account.
Shubhank Suman TMT, IP and AI Counsel
Page 31 of 31 Shubhank Suman, TMT, IP & AI Counsel. All Rights Reserved.