India's AI Mission has ₹10,371.92 crore but zero money for victims when it goes wrong

As India aims to ramp up its AI mission, several discrepancies in the dispensing of welfare benefits have come to the fore. Consider the case of Dhuli Chand, a 102-year-old resident of Gandhra village in Rohtak district, Haryana.

His old-age pension was stopped in March 2022 after Haryana's Parivar Pehchan Patra (PPP) database recorded him as dead. Six months later, on September 8, 2022, Chand rode a wedding chariot to the sound of a brass band, wearing a garland of currency notes, in a procession from Mansarovar Park to the Canal Rest House in Rohtak city, unabashedly announcing that he remains alive. At the end of the procession, he met a notable leader and former MLA from the ruling dispensation, presenting his Aadhaar card, family ID and bank statements to demand restoration of his pension.

The department administering the scheme has never explained how the PPP database concluded that he was dead. This is what is meant by a black box: a system where the data that goes in and the verdict that comes out are both visible, but the reasoning connecting the two is not disclosed. Unfortunately, when State functionaries deploy AI or other allied technological tools, the disruptive influence of the black-box phenomenon leads to immense hardship.

Recently, the government approved ₹10,371.92 crore for its AI Mission. The Mission aims to augment the AI infrastructure and reinvigorate State capacity to that end. But, at the same time, there are episodes involving ordinary citizens like Chand who lost 6 months of pension for an error that nobody explained or compensated him for. If the State keeps building these unaccountable systems, the brunt of which is borne by citizens, one question has to be answered first: who is liable to pay them back? 

In light of such challenges associated with the use of technology, the Indian government has issued the India AI Governance Guidelines, a framework meant to anchor "safe, trusted, and inclusive" AI innovation across sectors. The document rests on seven guiding principles one of which is accountability. AI developers and users should remain visible and answerable in their mode of functioning. Responsibility would also be imputed based on the function: “the function performed, risk of harm, and due diligence conditions imposed."

One can say that it is a reasonable expectation to interrogate AI tools and ostensibly technocratic solutions. It is necessary to ask how governments can be more responsive to AI systems getting it wrong and assume responsibility after identifying the error. However, a major drawback with the guidelines is that they are silent about how the State can be held responsible while deploying AI technologies. There is nothing on attributing liability, compensation for losses suffered or its timely payment. Dhuli Chand still doesn't have an answer.

The Aadhaar (Targeted Delivery of Financial and Other Subsidies, Benefits, and Services) Act of 2016 made Aadhaar linkage a condition for accessing subsidies that the State provides as a part of the constitutional’s welfare goals. By 2017, that role had been undermined by the advent of technology to the extent that instances of deprivation have increased significantly. The ration shop too has been one such site of denial. In a village in Jharkhand, a family's ration card was struck off the rolls because it had not been linked to an Aadhaar number. The dealer stopped handing out grain. 11-year-old Santoshi Kumari, from that household, died of starvation soon after.

She was not the only one. Jharkhand alone recorded a string of similar starvation deaths between 2017 and 2019, each traced to the same failure point: a verification system rejecting a real, hungry person because a database entry didn't match. For a household with no margin for a missed week of grain, a software bug culminates in the right to life under Article 21 being diluted.

The irony is that India's highest court was, at that very moment, deep in the Aadhaar debate. Justice KS Puttaswamy v. Union of India ran through these years and eventually split into two judgments — Puttaswamy I, which made privacy a fundamental right, and Puttaswamy II, which upheld the Aadhaar Act's core validity while trimming some of its provisions. Both are landmark rulings on what the State can do with a citizen's data. Neither one answers the much narrower, much harder question Santoshi Kumari's death actually raises: when a government verification system gets it wrong and someone starves as a result, what is that family owed and by whom? Indian law still lacks a dedicated statutory framework allocating liability and compensation for harms caused by government algorithmic decision-making.

India's problem is not a shortage of technology or even a shortage of legal rules, but how to remould or leverage these rules with a sense of purpose. In the absence of a concrete policy direction that underscores how accountability will translate to identification of liability and compensation, the ongoing challenges will only increase. Currently, perfunctory debates about exercising human oversight over technology are the norm while families are denied ration cards, or a pensioner erased from the government database.

In a different context, the Supreme Court confronted a similar question more than four decades ago. In Rudul Shah v. State of Bihar (1983), the Supreme Court held that compensation is not a matter of generosity. When a State violates a fundamental right, compensation in itself is a constitutional remedy. Restoring a right without compensating for its violation, the Court said, is “mere lip-service” to the Constitution. Yet, that principle is missing from India's AI governance framework, where citizens may recover their entitlements but remain uncompensated for the harm caused.

Other countries have shown what it looks like to build the legal apparatus before the system goes live, not after. Kenya shows a useful contrast: when the government introduced the Huduma Namba digital identity project, it was challenged before the Kenyan High Court as the project lacked adequate safeguards for citizens’ privacy. Faced with the Huduma Namba rollout, the Kenyan High Court held that the State cannot deploy the system without an adequate legal foundation protecting the rights of the citizens; the project was halted until a data protection framework was put in place. The technology has to wait for the law, not the other way around; India has done precisely the opposite.

The European Union offers a different model. Rather than leaving the liability to be worked out after the damage has been done, it has written the consequences directly into statute. Its AI Act classifies welfare-related systems as high-risk and non-compliance with it will lead to the penalty of €35 million or seven per cent of worldwide annual turnover. Its revised Product Liability Directive lets a citizen claim damages for algorithmic harm directly from the government or company that deployed the system, without first proving negligence. Although these instruments do not create a comprehensive compensation regime for every government algorithm, they reflect a legislative effort to allocate responsibility before harm occurs, something India's framework has yet to do.

India's framework suffers from the very same omission, where the State has committed about ₹10,371.92 crore on technological infrastructure for an AI-driven welfare State, but it has neglected to build the legal architecture that will determine who answers when the system causes harm.  The investment is in AI-driven technology, not in accountability and the right carries no remedy when it is violated.

The deepest irony is that the people most likely to be failed by these systems are the least likely to know that the law may entitle them to more than a correction. Dhuli Chand got his pension back eventually, but he was never compensated for the 6 months of pension he was wrongfully denied. Until India's AI governance framework recognises compensation as an integral part of accountability, every algorithmic error that harms a citizen will remain only partially remedied.

If India intends its AI Mission to be more than an investment in the technological infrastructure, a share of that money should have gone toward the people who have been harmed by the algorithmic error. The development cannot be measured by the sophistication of technology, but it can be measured by the State’s willingness to assume responsibility when these AI systems go wrong.

What the law still has not proved is that it knows how to compensate a citizen when its own technology wrongly declares otherwise. Parliament cannot longer afford to leave this question unanswered. Until a statutory framework assigns liability and guarantees compensation the way the EU AI Act does for algorithmic harm, every new AI system developed by the State will carry the same deficit and the absence of the remedy AI systems fail.

Nandini Agrawal is a student at UPES, Dehradun.