AI-Driven Healthcare In India - The Need For A Unified Framework Balancing Safety And Innovation
Author: LawSphereX
Category: General Law
Published:
Updated:
AI-DRIVEN HEALTHCARE IN INDIA: THE NEED FOR A
UNIFIED FRAMEWORK BALANCING SAFETY AND
INNOVATION
Nallam Aditya Sri Ram, Sastra Deemed University, Thanjavur Achanta Rama Chandra, Sastra Deemed University, Thanjavur Atchuta Sai Gowtham, Sastra Deemed University, Thanjavur
ABSTRACT
Traditionally, healthcare relied on human knowledge, manual recordkeeping, and traditional diagnostic approaches, which frequently resulted in inefficiencies, delays, and an increased risk of human error. The healthcare industry functions with the objective of patient care which in recent times has been boosted with technological advancements. With the rapid phase of globalization, technological acceleration has reshaped the global health industry. However, AI-driven innovations such as predictive analytics, robotic procedures, and automated administrative processes have greatly improved operational efficiency and patient outcomes. Artificial Intelligence being the brainchild of technology is the most discussed contemporary concept in the 21st century.
The Covid-19 pandemic has significantly impacted the global health industry, raising concerns about technology usage, innovative diagnosis, treatment, and disease prevention. The incorporation of Artificial Intelligence (AI) into healthcare is transforming medical practice by providing unprecedented prospects for enhanced diagnoses, personalized therapy, and healthcare delivery. The basic or salient feature of AI in the medical field is treatment management as well as its diagnosis. However, this technological breakthrough creates enormous issues for health laws, which must adapt to address the ethical, legal, and regulatory consequences of artificial intelligence in medicine. The adoption of AI technologies in healthcare, including machine learning algorithms, natural language processing, and robotics, offers significant potential benefits.
However, these advancements also bring forth critical challenges, particularly concerning data privacy, ethical considerations like patient autonomy, biased algorithmic functions and legal liabilities when AI systems fail or cause harm. The future of AI in global healthcare promises transformative advancements, bridging gaps in medical accessibility, efficiency and personalized treatment. Currently, there is no separate legislation to regulate AI which creates gloominess on application of AI in the healthcare industry.
This paper explores the dual facets of AI in healthcare, analyzing both its advantages, such as improved diagnostic accuracy and operational efficiency, and its drawbacks, including risks like data privacy of the patients and potential legal challenges in cases of AI-induced harm. This article explores the integration of AI technology with traditional diagnostic methods like radiology and pathology, as well as how real-time data processing improves clinical decisions. A comprehensive review of the global scenario was conducted, focusing on studies that explore the implementation of AI in various healthcare settings, its impact on patient outcomes, and the ethical and legal frameworks surrounding its use. This paper examines these ethical ramifications, highlighting the necessity of transparent, equitable, and patient-focused AI applications in the medical field.
Keywords: Artificial Intelligence, Healthcare Industry, Consent, Autonomy, Data Privacy and Ethical considerations. Background of Study Artificial intelligence (AI) is rapidly revolutionizing the global healthcare environment by boosting diagnostic accuracy, treatment results, and clinical workflow efficiency. However, its integration poses important ethical and legal concerns about patient privacy, informed consent, algorithmic bias, and malpractice liability. In India, where healthcare is rapidly digitizing, the lack of a robust legislative framework to govern AI applications poses significant threats to patient rights and data security. While global standards such as the EU AI Act and WHO guidelines promote ethical governance, India's regulatory initiatives remain fragmented.
This study is significant because it attempts to bridge the gap between technological growth and legal safeguards, suggesting a balanced framework that encourages innovation while preserving openness, privacy, and patient protection in AI-driven healthcare. Literature Review India’s approach to regulating Artificial Intelligence (AI) in healthcare remains nascent compared to jurisdictions like the EU or the US. According to Parry and Aneja (2023), India lacks a dedicated AI law but relies on frameworks like the National Digital Health Mission (NDHM) and ICMR guidelines (2023), which emphasize ethical deployment and human oversight and lacking legal enforcement. Stephenson (2021) highlights the challenges of accountability in AI-driven diagnostics, where liability for errors remains unclear.
Singh and Saha (2022) note that existing product liability laws, including the Consumer Protection Act, 2019, are inadequate for autonomous AI systems, as they assume human agency. Moreover,
Kaur (2021) explains that AI in healthcare poses medico-legal risks because autonomous systems can act beyond the developer’s intent. Scholars argue that liability in India is largely fault-based, making it difficult to assign responsibility when AI operates independently, hence limiting its application in fully autonomous medical technologies. Across this literature, consensus emerges on the promise of AI to enhance diagnosis, efficiency, and accessibility. Yet, scholars uniformly caution that these benefits depend on establishing robust legal accountability, eliminating algorithmic bias, and safeguarding patient autonomy. The comparative analysis reveals that while Western jurisdictions are moving toward codified AI oversight, India’s regime remains fragmented necessitating comprehensive legislation integrating ethics, data protection, and product liability.
Statement of Research Problem AI is transforming healthcare in India but also exposing gaps in ethics and law. Weak regulations on consent, data privacy, bias, and liability leave patients’ autonomy, safety, and Article 21 rights at risk. This regulatory gap underscores the need for a future-ready framework that can address emerging risks while enabling responsible technological development. Research Objective • To analyse the impact of Artificial Intelligence on patient care, privacy, and legal accountability within the Indian healthcare system. • To compare India’s approach with global regulatory models such as the EU AI Act and U. S. FDA guidelines, highlighting best practices relevant to medical AI. • To suggest a comprehensive, transparent, and patient-centric legal-ethical framework for the safe and responsible integration of AI in healthcare.
Research Methodology This study adopts a doctrinal method using sources such as laws, case rulings, policies, and academic works. It compares India’s AI healthcare regulations with those of the U. S. , EU, and China, analyzing issues like consent, privacy, and liability. Drawing on WHO and ICMR guidelines, it synthesizes global insights to propose a suitable legal-ethical framework for
India’s AI-driven healthcare system. Research Questions • How does the lack of a comprehensive legal and ethical framework in India influence the regulation and accountability of artificial intelligence in healthcare? • To what extent do current legislation, such as the Digital Personal Data Protection Act of 2023 and the prospective Digital Information Security in Healthcare Act of 2018, properly protect patient privacy and consent in AI-powered medical systems? • What are the ethical consequences of algorithmic bias, data misuse, and self-decisionmaking in AI-powered healthcare technologies? • How can India implement and adapt global best practices, such as those outlined in the EU AI Act and WHO ethical guidelines, to create a strong national framework for AI in healthcare?
Scope and Limitation of Study This paper explores the legal, ethical, and regulatory challenges of AI in Indian healthcare, evaluating laws like the Data Protection Act (2023) and DISHA (2018) against global standards such as the EU AI Act and WHO guidelines. Using a doctrinal approach based on sources, it focuses on legal accountability and patient rights, acknowledging limits due to evolving AI technology and lack of empirical data. Introduction In India Medical profession is considered to be a divine profession in which ethics are paramount to be followed by every medical practitioner. The recent shifts in technological advancement have been playing a key role in reshaping the healthcare industry by combating various challenges which resulted in effective diagnosis and treatment.
AI has been utilized in healthcare for a range of purposes, including machine learning, natural language processing and computer vision1. 1 Dr. Rituja Sharma (2023), Legal and Ethical Concerns in AI Driven Healthcare- A Study of Legal Approaches,
Machine Learning: Machine learning allows devices to analyze large data sets and detect patterns without requiring explicit programming. In healthcare, it helps to forecast illness outcomes, detect outbreaks, and optimize therapies. For example, AI can analyze medical data to discover early indicators of diseases such as tumours, allowing for earlier intervention. Natural Language Processing (NLP): NLP helps computers understand and respond to human speech. In healthcare, it helps with speech recognition, sentiment analysis, and diagnostic reporting. It can digitally document patient interactions for better record-keeping and decision- making. Computer Vision: Computer vision enables computers to evaluate images for medical diagnosis, remote monitoring, and surgical assistance. It examines X-rays and MRIs to identify abnormalities and guide complex surgeries.
With its integration, AI has provided enormous advantages to the healthcare industry but it also possesses moral implications and possible peril by raising concern about privacy, ethical considerations, lack of transparency and algorithmic bias.
WHAT IS AI:
Artificial intelligence is a discipline of computer science that focuses on constructing intelligent computers that function and react in the same way as people do. In some sectors, artificial intelligence is considered to be capable of replacing humans. In common language, the ability of a computer system to make judgments on its own is known as artificial intelligence. Mr. John McCarthy, a computer scientist, invented the phrase "artificial intelligence" during a conference in 1956. The term Artificial Intelligence is not limited to single edition but a compilation of various sub technologies.
The evolution of AI technology and its usage can be categorized into two different categories as narrow usage and broader usage Artificial Intelligence represents a number of subfields (such as machine learning and deep learning) that, alone or in combination, enhance the intelligence of applications. Machine learning (ML) is the study of techniques that enable computer programs to improve automatically via experience. Deep learning (DL) is a type of algorithm that learns by exposing connected processors to a huge number of examples. DL has emerged as the primary method in AI today, generating advancements in fields such as image and speech recognition. Educational Administration: Theory and Practice, 29(3), 466-473, Doi: 10. 53555/kuey. v29i3. 5026
After recognizing the existence of artificial intelligence (AI), the World Intellectual Property Organization (WIPO) proposed three types of AI: expert systems, perception systems, and systems that use natural language. Expert systems are computer programs that handle issues in certain domains of knowledge, such identifying geological situations, suggesting treatments, and detecting medical ailments, to mention a few.
APPLICATION OF AI IN THE HEALTH CARE
AI is revolutionizing areas like transportation, agriculture, and healthcare, as demonstrated by self- driving cars (Tesla) and robotic aides in hospitals and hotels, particularly during the Covid-
19 outbreak. An AI virtual chatbot called Clare&Me provides conversational phone and
WhatsApp support and guidance for mental health in Germany. Ora is a Singapore-based telehealth platform that focuses on providing direct-to-patient healthcare for Southeast Asians on topics related to skin care, female reproductive healthcare, and men's health. The drug discovery and development process can be accelerated and made more economical and efficient with the use of artificial intelligence (AI). AI was previously used to find possible Ebola virus medications, but as with any drug study, finding a lead molecule does not ensure the creation of a safe and effective treatment2.
According to the recently published draft National Strategy for Artificial Intelligence in India, rising technological advancements, as well as the interest and activity of innovators, offer India the chance to address some of its long- standing issues with providing adequate healthcare to a sizable portion of its population. The Maharashtra state government has also signed a memorandum of understanding with NITI Aayog and the Wadhwani AI group to launch the International Centre for Transformational Artificial Intelligence (ICTAI), focusing on rural healthcare. Similarly, the Telangana state government has adopted the Microsoft Intelligent Network for Eyecare, which was created in collaboration with Hyderabad-based LV Prasad Eye Institute.
In August 2019, the Central TB Division under the Ministry of Health and Family Welfare (MoHFW) signed a Memorandum of Understanding (MoU) with Wadhwani Institute for Artificial Intelligence to research the application of Artificial Intelligence technology in the fight against Tuberculosis. Wadhwani AI would assist the National TB program in becoming AI-ready as part of the partnership, which would entail creating, testing, and implementing AI- 2 Stephenson J. Who offers guidance on use of Artificial Intelligence in medicine. JAMA Health Forum. (2021) 2:e212467. doi: 10. 1001/jamahealthforum. 2021. 2467
based solutions. In addition to helping the RNTCP embrace additional AI technologies, it would support the program in vulnerability and hotspot mapping, modelling innovative screening and diagnostic techniques, and offering decision support for careers3.
CHALLENGES OF AI IN THE HEALTHCARE
Although the integration of Artificial Intelligence with the medical industry provides bear fruit results but also possesses an uphill battle with its algorithmic bias, ethical and legal issues which predominantly affects the existing doctor-patient relationship, privacy and liability concerns. India requires a robust legislation for AI in healthcare to overcome the uncertainty. While Indian Judiciary bring forth guiding principles, but gaps remain when compared to more structured legislations like GDPR in Europe.
1. Ethical Consideration: Ethics are considered to be bedrock which need to be followed by
every professional in their respective fields. It is even more an important duty for a doctor or medical practitioner to follow the ethics for patient safety and betterment. To protect the dignity, rights, safety, and well-being of the community and participants, all health and biomedical practitioners whether using AI-based or traditional methods should first and foremost follow the fundamental ethical principles of respect for persons (autonomy), do good (beneficence), do no harm (non-malfeasance), and distributive justice. 1. 1. Autonomy: The most important ethical principle to be adhered in the field of medical sciences is Autonomy. According to the principle of Autonomy Patients are entitled to information prior to treatment, including the dangers of screening, imaging, data privacy, and AI errors.
They may refuse treatment and must understand who is responsible for robotic medical equipment malfunctions. Ensuring accountability is critical for both patient rights and the medical staff. The patient has a legal right to autonomy and selfdetermination under Article 21 of the Indian Constitution. He has the right to decline treatment unless it is an emergency in which case the doctor does not need his consent. The consent granted should be legally valid. AI technology in healthcare has the potential 3 Claire Munoz Parry, Dr. Urvashi Aneja, Artificial Intelligence for Healthcare: Insights from India, CHATHAM HOUSE [(20 July 2023)], https://www. chathamhouse. org/2020/07/artificial-intelligence-healthcare-insightsindia/3-ai-healthcare-india-applications
to erode human autonomy by allowing it to operate independently. AI technology in healthcare could potentially automate decision-making processes. Patients should be thoroughly educated about the advantages of using AI technology as well as the risks physical, psychological, and social that come with them. The decision to accept or reject AI technologies must be entirely up to the patients.
1.1.1. Consent:
The ethical and legal dimensions of autonomy are reflected in healthcare consent. Criminal and civil (tort) penalties result from treating a patient without getting their consent. Alternative therapies are exempt from the Medical Council of India's (MCI) requirement for written consent for procedures. In India, medical consent is standardized by requiring written approval for complex therapies, oral consent for specialized surgeries, and implied consent for routine treatments. Citing Article 214 of the Constitution, the Supreme Court underlined the significance of consent in its 2008 decision in Samira Kohli vs. Dr. Prabha Manchanda5. Section 136 of the Indian Contract Act of 1872, which calls for explicit, precise, and informed consent, is aligned with data privacy by the Digital Personal Data Protection (DPDP) Act of 2023.
According to Section 67 of the DPDP Act Data Principal's consent must be free, explicit, informed, unconditional, unambiguous, and clearly affirmative. This consent must be limited to the necessary information and pertinent to the stated purpose. A required notification under Section 5 must be sent to the Data Principal
explaining the requested data and its intended use prior to obtaining consent. Additionally, Section 128 creates the "right to be forgotten," which enables people to request that their data be deleted if they no longer need it or if their consent is no longer valid. Unless required by law or for legal claims, patients in healthcare have the right to request that hospitals or clinicians delete their Personal Health Information (PHI). Unless there is an exception, healthcare providers are required to abide by such requirements. Comparatively, GDPR
4 INDIA CONST. art. 21
5 Samira Kohli v. Prabha Manchanda Dr. & ANR 1 (2008) CPJ 56
6 THE INDIAN CONTRACT ACT 1872, § 13 NO. 9, Acts of parliament, 1872 (India).
7 THE DIGITAL PERSONAL DATA PROTECTION ACT 2023, § 6 (NO. 22, Acts of parliament, 1872 (India)
8 THE DIGITAL PERSONAL DATA PROTECTION ACT 2023, § 12 (NO. 22, Acts of parliament, 1872 (India)
directs explicit consent for sensitive health data (Art 9) governed by processing conditions. UK Courts and regulators (ICO) confirm strict enforcement, unlike India where enforcement remains fragile.
1.1.2. Data Privacy and Security
In today's technological world, where data is considered to be the wealth, it is paramount necessary to safeguard the privacy of an individual. The Indian Supreme Court in the case of K. Puttaswamy vs Union of India9 held that Privacy is considered to be a fundamental right under Article-21 (Right to life and personal liberty). It is also noteworthy to mention that one of the most important duties of the medical practitioner is to maintain the confidentiality of the patient regarding his/her health by not infringing the privacy of such individuals. The application of AI in healthcare requires huge amounts of patient data raising worries about data privacy and security. It is critical to protect patient data from unauthorized access and give patients the ability to regulate how their information is used. In the field of digital health, privacy is extremely complicated.
This is due to the fact that details regarding an individual's health, such as whether they have had an abortion or are HIV positive, can be very personal. Furthermore, those whose personal information is gathered may include some of the most vulnerable members of society, including teenagers, transgender and gender- variant people, and those with HIV. Such individuals may be significantly more vulnerable to harm from an invasion of their privacy, which may necessitate even more protection. The Digital Information Security Healthcare Act10 (DISHA) of 2018 provided a new approach for data protection. DISHA's primary goal is to establish standards for confidentiality, privacy, and security for digital health information.
It states that information about a person's physical, physiological, mental, and sexual orientation, as well as their medical records, medical history, and biometric data, is proprietary to that individual.
9 K. Puttaswamy vs Union of India, 2017 (10) SCC 1.
10 Digital Information Security in Healthcare Act 2018, Acts of parliament, 2018 (India).
The Apex court of India in K. S. Puttaswamy v. Union of India (2017) strongly established Privacy as a Fundamental right under Article 21 of the Constitution. Primarily, the Court established the Proportionality test: any restriction on privacy must satisfy legality, necessity and proportionality. All the AI driven health apps must satisfy these three conditions to ensure that privacy is not affected. During Covid 19, Aarogya Setu app has collected immense data which raised questions about statutory backing and legality. According to Puttaswamy, without clear legal backing such collection of data amounts to unconstitutional. Contact tracing may serve the public health objectives but data retention or non-anonymized storage goes beyond necessity. GDPR’s principle of data minimization (Art. 5(1)(c)) contrasts with India’s broad collection practices.
For example, UK’s NHS COVID-19 contact tracing app incorporated decentralized data storage, minimizing proportionality risks, unlike India’s centralized Aarogya Setu. Thus, while GDPR and UK law stress consent, limited purpose, and accountability, Indian health-tech apps often fail to meet proportionality, reflecting weaker patient safeguards.
1.1.3. Balancing Patient Privacy and AI-Driven Apps
Due to the requirement for large amounts of patient data, AI applications in healthcare pose security vulnerabilities, making hacks and breaches a possible hazard. The existing approach puts Electronic Health Records (EHRs), which hold Personal Health Information (PHI) such as names, contacts, and IDs, at risk. Concerns regarding patient privacy and data security are raised in India by the absence of strong guidelines for medical apps. For example, in 2020, a ransomware assault on the All-India Institute of Medical Sciences caused the loss of research and outpatient data from both primary and backup servers11 Due to the fact that unencrypted data might reveal victims' identities, bank accounts, and other sensitive information, medical data can potentially be used 11 Bhardwaj D.
(2023) AIIMS ransomware attack led to new SOP on cyber breaches: Ex: cybersecurity chief pant, Hindustan Times, Available at: https://www. hindustantimes. com/india-news/aiims-ransomware-attackledto-new-sop-on- cyber-breaches-ex-cybersecurity-chief-pant-101688321198625. html (Last visited on Feb 11,2025).
for fraudulent purposes, including insurance fraud. Chinese hackers recently gained access to more than 6.8 million medical records on an Indian healthcare website, including information about patients and doctors. Traditional healthcare has been revolutionized by the move to a digital health sector, which has been sped up by AI applications in medical equipment. The Indian government launched the Aarogya Setu contact tracing app during the COVID-
19 pandemic. However, this app's substantial data collecting and privacy
concerns drew criticism. Concerns regarding the absence of user data protection procedures were voiced by experts, including Justice BN Sri Krishna12.
2. Patient-Provider Relationship: The use of artificial intelligence in healthcare may alter the
patient-provider interaction. While AI can speed up administrative tasks and improve clinical judgments, it may potentially reduce personalized care. If patients feel that AI makes key decisions, they could become skeptical or suspicious of their treatment if no human assistance is provided. AI technology has the potential to transform medical communication with patients. Over reliance on AI recommendations may hinder clinical decision-making and the relationship between doctors and patients. Balancing AI benefits with maintaining compassionate, patientcentered care is an ethical dilemma in this scenario.
3. Bias in the Data: AI systems can be biased if the data used to train them is not representative
of the population they will be serving. This can lead to erroneous or discriminatory outcomes, especially for marginalized communities. Algorithmic bias, where AI systems exhibit preconceptions based on their training data, might result in unequal treatment outcomes for distinct demographic groups. To ensure fair and equitable patient treatment, it's important to build transparent algorithms and check them continuously. One of the most common medical treatments is oxygen administration, which is detected in the blood using a pulse oximeter. The pulse oximeter detects oxygen saturation by transmitting infrared light via the skin. Pulse oximeter measurements are known to be impacted by the patient's skin colour, as the equipment consistently overestimates oxygen saturation levels in non-white patients.
As a result, black patients are three times more likely than white patients to have occult hypoxemia that is
12 Rukmini Rao, who will be held accountable for data breach?’, Justice Srikrishna on Aarogya Setu
app,BUSINESSTODAY.IN(Last visited on Feb 11,2025), https://www.businesstoday.in/technology/news/story/who-will- be-held-accountable-for-data-breach-justicesrikrishna-on-aarogya-setu-app-258161-2020-05-13 .
undetectable by pulse oximeters13. Algorithmic bias in AI healthcare like Pulse Oximeters excessively harms marginalized communities. In India, such bias could raise claims under Article 14 (equality before law) Article 21 (Right to health) of the constitution and Consumer Protection Ac, 2019 for defective Service claims. The EU AI Act explicitly condemns discriminatory algorithms as high-risk, demands supervision. India is in need of such authoritative mandates, instead of relying on ICMR’s Ethical Guidelines, 2023.
4. Liability of AI in Medical Malpractice: Universally, rights and duties are only attributed
to legal persons (humans and corporations) by the law. Therefore, AI needs to be given legal personhood in order to be held accountable. Although there is currently no national or international legislation that aims to give AI legal personhood, efforts have been started to investigate the potential and balance the risks, difficulties, and opportunities associated with doing so. It is indeed impossible to make a liability shift towards the AI. The inclusion of Artificial Intelligence in the medical sector raises a significant question of liability. The usage of AI in the medical sector can be categorized into two approaches:- narrower and broader approach. The narrower approach includes the usage of AI applicant devices, AI functioning chat boxes and AI based tools for diagnosis. Several scholars have opined the liability of AI should be on product liability and vicarious liability.
According to Abramoff et al. , manufacturers of autonomous AI products ought to be held accountable for any harm caused by the device if it was used appropriately and in accordance with its label, leaving the providers in charge of ensuring proper use. In the case of assistive AI, however, where clinicians are able to independently assess the AI's outputs, the providers would still be held accountable. Three criteria can be used to classify medical robots under criminal law. First, robots are only tools when they are operated by humans to conduct crimes, and the human operator is the only one responsible. Second, a medical robot's legal position determines liability if it malfunctions and causes harm. The humanoid robot Sophia, who was awarded Saudi citizenship, may be held negligent, including for medical malpractice, if it were anthropomorphized.
Otherwise, medical facilities are liable. Third, culpability depends on the legal personhood of highly sophisticated medical robots in the event that they conduct crimes on their own. They would be held liable if they were acknowledged; otherwise, it is unclear who is criminally responsible14. These 13 Bickler P. E. , Feiner J. R. , Severinghaus J. W. Effects of skin pigmentation on pulse oximeter accuracy at low saturation. Anesthesiology. 2005;102:715–719. Doi: 10. 1097/00000542-200504000-00004. [DOI] [PubMed] [Google Scholar] 14 Xiaoli Shentu, A review on legal issues of medical robots Medicine (Baltimore). 2024 May 24;103(21):e38330. Doi: 10. 1097/MD. 0000000000038330.
complications must be addressed by AI healthcare law in order to strike a balance between innovation and responsibility in malpractice and injury situations. Devices like Pulse Oximeters highlight the hazards of biased algorithms that hyperbolize the oxygen saturation in darkerskinned patients, threatening misdiagnosis. In India, rules such as Consumer Protection Act,
2019 and Tort law establish accountability for defective products that cause injury by holding
manufacturers accountable for anticipated harm. Drugs and Cosmetics Act, 1940 governs medical devices, but does not specifically address AI liability. However, the EU Act (2021) places strict standards for high-risk AI (like medical devices) such transparency, risk assessment and human oversight. India falls behind in designating obligations and liabilities for AI developers and hospitals.
5. Medical Negligence: In order to make Medical Professional liable for Medical Negligence,
Medical Professional must have failed to uphold the standard duty of care to the patient which resulted in harm or damages. On the other hand, if a doctor uses AI and it causes harm, Court must follow two approaches to allocate and determine the liability i. e. , Assistive AI and Autonomous AI. In Assistive AI, where doctors can countermand outputs, Negligence rests with the doctor, since he relied on AI without clinical judgement breaches the standard duty of care according to Jacob Mathew v State of Punjab. In Autonomous AI, where the machines can act independently, liability questions are very complex. Scholars suggests Manufacturer liability for defective algorithms, aligning with the law of product liability.
But UK law provides detailed standards through the Consumer Protection Act 1987 (implementing Product Liability Directive), holding manufacturers strictly liable for defective products including Medical AI devices.
EXISTING GLOBAL FRAMEWORK OF AI IN HEALTHCARE
framework for Modifications to AI/ML based SaMD making developers responsible for performance of their AI systems and report updates affecting system performance and inputs. In 2021, FDAs based SaMD Action Plan outlined five actions under a Total Product Life cycle (TPLC) approach European Union Yes.
The EU AI act, 2024 has established rules for AI across the EU aiming for a risk-based approach categorizing as minimal, risk, transparency risk, high risk and unacceptable risk.
AI systems used or embedded in medical devices classified as high-risk system which are classified as Highrisk AI systems which requires stricter obligations.
In June 2019, the Technical Guideline on AI-aided Software covered deep learning technology, software data quality, algorithm validation, and clinical risk assessment.
The Guidelines for the Classification and Definition of AI- Based Software as a Medical Device were released on July 8,
2021. They focused on
AI-MD classification, algorithm safety and effectiveness, and their role in clinical decision-making.
GLOBAL TO LOCAL REGULATION OF AI IN HEALTHCARE INDUSTRY
The World Health Organization (WHO) urges caution while employing artificial intelligence (AI)- generated large language model tools (LLMs) to protect and promote human well-being, safety, and autonomy, as well as to preserve public health. WHO emphasizes the need of designing, developing, and deploying AI for health in accordance with ethical standards and adequate governance, as outlined in the WHO guidance on the ethics and governance of AI for health? The WHO has identified six core principles: (1) protect autonomy; (2) promote human well-being, human safety, and the public interest; (3) ensure transparency, explainability, and intelligibility; (4) foster responsibility and accountability; (5) ensure inclusiveness and equity; and (6) promote responsive and sustainable AI15. India's healthcare AI regulations are still in their infancy.
The first steps towards thorough AI governance in healthcare were taken in 2020 when MeitY published a draft National AI Strategy that included ethical standards and a regulatory framework for AI-powered medical products. The Indian Council of Medical Research (ICMR) recently issued the country's first
15 WORLD HEALTH ORGANIZATION,WHO calls for safe and ethical AI for health(Last visited on Feb
10,2025)
Ethical Guidelines for the Application of Artificial Intelligence in Biomedical Research and Healthcare, with the goal of developing "an ethics framework which can assist in the development, deployment, and adoption of AI-based solutions" in the fields specified. They seek to make "AI-assisted platforms available for the benefit of the largest section of common people with safety and highest precision possible," while also tackling developing ethical problems related to AI in biomedical research and healthcare delivery16.
OECD Principles on Artificial Intelligence The international approach to trustworthy AI is further strengthened by the OECD Principles on Artificial Intelligence, which outline five globally accepted standards for responsible AI development: (1) AI should benefit people and the planet; (2) AI systems must respect the rule of law, human rights, democracy, and include safeguards such as human oversight; (3) AI should be transparent and explainable; (4) AI actors must be accountable for outcomes; and (5) AI systems must be robust, secure, and safe throughout their lifecycle. These principles directly resonate with the challenges faced in India’s healthcare sector particularly the concerns around privacy, informed consent, algorithmic bias, explain ability, and legal accountability.
SUGGESTIONS
1. To safeguard patient data, use strong encryption and anonymization methods. Enforcement
of provisions like Right to be forgotten under DPDP Act, 2023 ensures patient can demand deletion. Regulate Medical data confidentiality by adopting 2018 DISHA draft.
2. To avoid biased results, make sure AI models are continuously audited and have a variety of
datasets.
3. Create healthcare laws tailored to AI that guarantee safety and accountability. Categorize AI
in healthcare as high risk, mandating safety certifications.
4. Extenuating circumstances pertaining to patient care and diagnosis should be taken into
account by DPDPA, and healthcare professionals engaged should be granted exemptions. Consideration should also be given to how healthcare professionals store and exchange medical 16 LINKEDIN,(10) AI Regulation in Healthcare: India, USA, Europe, and Singapore | LinkedIn (last visited on Feb 10,2025)
data for patient care and research.
5. To avoid data breaches, organizations that store private medical information should put
strong cybersecurity safeguards in place.
6. To protect the privacy of the data principals, the data fiduciary should pseudonymized any
personal medical information that was first obtained for the purpose of treating the data principal's patients and is subsequently needed for analysis or archiving for research purposes.
7. Make use of interpretable AI models to help medical professionals comprehend and have
faith in AI judgments.
8. Patient should be informed about usage of AI in treatment, ensuring GDPR obligations.
Enforce DPDP Provisions like explicit, Informed consent for AI-driven medical apps.
9. Patients should be given exclusive right over the data which has been given to service
providers. Patient should update and monitor the application of his data. 10.Vicarious liability should be adopted by Hospitals where negligence occurs from usage of AI. Legislate clear rules on Product liability for AI-driven medical devices, aligned with CP, 2019.
CONCLUSION
The integration of technology and Artificial intelligence into the healthcare industry is inevitable and functions as a double-edged sword. The integration of AI and ML technologies has enormous potential for revolutionizing healthcare delivery, increasing patient outcomes, and advancing medical research. However, this revolutionary potential is accompanied with numerous ethical concerns that necessitate careful evaluation and proactive mitigating techniques. Healthcare stakeholders can navigate the ethical complexities of AI and machine learning integration in health care by addressing issues such as data privacy and security, algorithmic bias, transparency, clinical validation, and professional responsibility, all while protecting patient welfare and upholding the principles of beneficence, nonmaleficence, autonomy, and justice.
By embracing ethical best practices and encouraging collaboration among interdisciplinary teams, the healthcare community can fully realize the potential of AI and ML technologies, ushering in a new era of personalized data-driven health care that
emphasizes patient well- being and equity. Although there are ethical concerns with AI use, it is likely to supplement, coexist with, or replace existing systems, ushering in the era of artificial intelligence in healthcare. Additionally, it may be unethical and unscientific to not use AI. Ultimately, AI’s success in healthcare depends not just on technical accuracy, but on constitutional compliance, patient autonomy, and legal accountability. It is evident from the current scenario that starting with the global health organizations like WHO to the governments of the various countries were rebooting the existing healthcare legislations to compile and combat with the unprecedented Artificial Intelligence aiming at the balancing of innovation and ethical considerations.