The Use Of Artificial Intelligence In Bank Lending Decisions - Bias,Fairness, And Regulation

Author: LawSphereX

Category: Technology, Cyber & Privacy

Published: 2026-08-23T21:33:43.446Z

Updated: 2026-08-30T07:10:44.173Z

THE USE OF ARTIFICIAL INTELLIGENCE IN BANK

LENDING DECISIONS: BIAS, FAIRNESS, AND REGULATION

Vashita Ahuja, OP Jindal Global University

ABSTRACT

This paper will provide an overview of how AI is used in bank lending decisions and how it has benefited the sector. There would be discussion on how AI can have a bias such as algorithmic bias and also violate several anti-discrimination laws in the same way. We will be looking at various case studies focusing on India and the US and how there have been instances where the existing legal and ethical standards are infringed (focus on bias in microfinance lending- Zest Money/Rupifi; Apple card gender bias controversy-Goldman Sachs in US).We will also look into the specific regulatory norms that have been proposed in United State and European Union to deal with this along with analyzing the case studies with them. The paper will also explain what these countries are doing to help find solutions in this aspect and how the regulatory bodies in these countries have addressed these issues.

The advancement of technology has rapidly caught hold of all the sectors in the world, especially in banking, and it can be said that soon it will be on the verge of replacing the traditional/conventional ways of human- made decision making. We can see that Artificial Intelligence (AI) is applied all over the banking sector to carry out in various activities such as resolution of queries, personal service, lending services, perceiving/detecting frauds in this industry. In India, majority of the banks use AI to generate new business ideas and getting to know customer preferences with the help of which they can accordingly address to all the demands of the customers. AI is still advancing in major parts of the world, seeing India it is developing and is being used by the banking sector for their benefits in the midst of all the decision making.

1 The banks generate the results on the basis of the use of algorithms which use data input to help generate better opportunities and functions by the AI. One can come to the point of thinking what exactly AI is, so to define it is ‘a set of encoded procedures where the input data provides the desired output after certain calculations, it is step-by-step algorithmic procedure where the role of human work is almost negligible’. 2 In one of the research works done, Jewandah S talked about How Artificial Intelligence is changing the Banking Sector with the comparative analysis of the top four commercial banks. She spoke about Machine Intelligence along with how AI is applied in the banks where they are adopting these technologies to improve their working.

She also discussed that even though this development and improvement in different parts of the sector was helping in several ways, it was exposed to the risk of vulnerabilities and cyber threats. 3 AI also causes social and ethical challenges in the sector leading to biases and discriminatory outcomes which further causes customer dissatisfaction and economic challenges in the industry. 4 The data produced by the banks is due to the algorithms and the inputs provided for it. The maxim ‘garbage in, garbage out’ described that an erroneous or biased input would cause 1 Shashikala K. G. , Ramesh A. P. & Gowri Shenoy, Role of Artificial Intelligence and its Applications in Indian Banking Sector, Int’l J. Res. Publication & Rev. 9148-9151 (2024), https://ijrpr. com/uploads/V5ISSUE5/IJRPR28384.

pdf 2 Vidushi Marda, Artificial Intelligence Policy in India: A Framework for Engaging the Limits of Data-Driven Decision-Making, 376 Phil. Trans. R. Soc'y A 20180087 (2018), https://doi. org/10. 1098/rsta. 2018. 0087. 3 Simran Jewandah, How Artificial Intelligence Is Changing the Banking Sector – A Case Study of Top Four Commercial Indian Banks, Int’l J. Mgmt. , Tech. & Eng’g 525-530 (2018), https://ijamtes. org/gallery/66. july%20ijmte%20-%20711. pdf

4 Sarita Kumari Singh, Jayanta Kumar Parida & Sameer Shekhar, The Landscape of AI in Indian Banking

Sector: A Theoretical Perspective, J. Informatics Educ. & Res. (2024), https://doi. org/10. 52783/jier. v4i1. 688. bias/errors in the output. The developer is responsible mostly for such instructional algorithms where bias is easily visible. 5 To have a better understanding of the challenges faced by the banking sector we would be looking at two major case studies and analyzing how biases thrive and how this can be prevented. Zest Finance and Rupifi are using algorithmic lending to raise loan production, but in the process as noticed by witnesses the AI used in the models show unconscious bias along the racial lines. One of the people who noticed this concluded that “African Americans were subjected to higher interest credit cards because the AI was able to infer their race.

” 6 This is because the highly regulated areas like lending also require powerful AI which leads to opaqueness in the system which makes even the creators question a particular conclusion. Zest strategy to improve the model involved the technique of ‘drop one’ wherein the lending model of a financial institution dops the variable of consideration and sees the output change by the model, this method is applied to all variables to see the best performance. 7 Zest also claims to use a technique of adversarial debiasing where two models are pit against each other and where one model is made to guess the creditworthiness and the other is for guessing the race/gender of the applicant. The unsaid competition between both the models leads them to be more efficient and thereby being more accurate. 8 Fairness has various interpretations considering in the AI models.

When we see it as a direct way of indicating discrimination in a society it contributes to value and equity in it. We come to distinguish between equity as the goal and fairness as uniformity across the globe. Fairness can be achieved in this digital lending world by legal frameworks that defend against discrimination and AI models that are intertwined with algorithms that support every society. 9 As in the case of Zest Money the access is broadened to exemplify the AI’s potential for

5 White & Case, Algorithms and Bias

What Lenders Need to Know (Jan. 20, 2017), https://www.lexology.com/library/detail.aspx?g=c806d996-45c5-4c87-9d8a-a5cce3f8b5ff 6 Neil Burnett, Artificial Intelligence in Lending: Bias, Model Regulation, and Liability, Banking Dive (Oct. 18, 2019), https://www.bankingdive.com/news/artificial-intelligence-lending-bias-model-regulationliability/561085/. 7 Jeremy Kahn, Artificial Intelligence Could Entrench Unfair Lending Practices, Fortune (Oct. 20, 2020), https://fortune.com/2020/10/20/artificial-intelligence-unfair-lending/. 8 Kyle Wiggers, Zest AI Raises 5 Million to Reduce Loan Algorithm Bias, VentureBeat (Oct. 20, 2020), https://venturebeat.com/ai/zest-raises-15-million-to-reduce-loan-algorithm-bias/.

9 Sundaraparipurnan Narayanan & Mahesh Hariharan, Beyond Algorithms

Navigating Fairness in India's Lending Landscape, NASSCOM (2024), https://nasscom. in/ai/beyond-algorithms-navigating-fairness-in-indialending-landscape/. financial inclusion but at the end this all depends on the fairness of the AI models being designed and how well they are regulated to prevent the discrimination in results. Another case study which we will use is ‘Apple Card Gender Bias Controversy’. It basically started out with the complaints that the algorithms were inherently biased against women. One of the customers noticed that the Apple card gave the guy 20 times the credit limit that what was given to the woman. Goldman Sachs is responsible for the Apple Card, therefore while defending itself they answered that these decisions were taken on the basis of the customer’s credit worthiness and not the age/gender of the customer.

10 One of the another incidents also showed similar bias against a customer’s wife and himself and when he started protesting to Apple Card’s treatment she suddenly got a VIP bump to match the same credit limit as her husband. This not only was reflective of the flawed algorithm but also the cover-up done not expose the company to any questions. 11 The problem isn’t with the intention of the Apple representatives, it begins with the algorithm on the basis of which the whole company operates which discriminates all their customers. Though the company continues, it bases its arguments and defends itself by saying that even though they use AI the foundation is built of simplicity, transparency and privacy. 12 This highlights the main issue in AI lending – ‘the black box’ problem which indicates the lack of transparency in decision making.

In this problem the creator/modern developer himself is unable to identify the process of AI decision making because of the density of the algorithms used by the AI. Goldman Sach’s customer business hasn’t indicated any machine learning/AI which suggests that they use the traditional logistic models which become the source of the biases. This can only be controlled if the lawmakers / policymakers set standards which are to be met by the banks and lending institutions while providing credit in a fair manner. 13 To battle the majority of the problems, the European Union has come up with the Artificial Intelligence Act (AIA) which came into force on August 1st, 2024, which provides guidelines

10 Apple Card Investigated after Gender Bias Complaints, BBC News (Nov. 11, 2019),

https://www.bbc.com/news/business-50365609.

11 Kang, Cecilia. "Apple Card’s Gender Bias Is Under Investigation." The New York Times,

November10,2019. https://www.nytimes.com/2019/11/10/business/Apple-credit-card-investigation.html. 12 Hsu, Tiffany. (2019). Apple Card algorithm sparks gender bias allegations against Goldman Sachs. The Washington Post. https://www.washingtonpost.com/business/2019/11/11/apple-card-algorithm-sparks-genderbias-allegations-against-goldman-sachs/

13 Wallace, Mack. "Inside Apple Card’s Use of the ‘Black Box’." Medium, October

2024. https://medium.com/@Mackswallace/inside-apple-cards-use-of-the-black-box-ac60e6bb5892.

and procedures on how AI can avoid the biases that are prohibited under law. 14 It is the first time that this challenge has been addressed and provides guidelines to counter the same. The AI Act fills up the major regulatory gap and ensures that accountability is taken up where it is due. It brings in provisions of human oversight which allow them to intervene, and override system codes in case they think that the AI isn’t coming to the right conclusions. Few of the conditions to prevent biases that the act has set up are that the sensitive date mustn’t be easily accessible and there must be strict access control so that the data subjects aren’t immediately identified which may lead to any biases. 15 As per Article 13(1) AIA, the AI systems are mandated to be designed in a particular way so that their operation is transparent, and the decisions can be easily monitored.

16 The AI Act has set out different penalties if the mandatory provisions are not followed such as if there is non-compliance with the prohibited AI practices the penalty would amount to be 7% of global annual turnover or 35 million euros; if there is non-compliance with other obligations the penalty would be up to 3% of global annual turnover or 15 million euros; and if there is incorrect information supplied then the penalty is up to 1% global annual turnover or 7. 5 million euros. 17 Now we can analyze the two case studies through the lens of the AI act to assess how the solutions might have been come up with differently. The AI Act categories the systems by risks and further imposes strict obligations to the high-risk systems here in case, lending.

The AI Act classifies the creditworthiness of the systems and if Zest Money’s algorithm were to be examined, they would be subject to strict requirements. They would have to be following transparency norms and bias mitigation. The act would also impose auditing on a periodic basis to assess fairness in the system and to check the compliance with the legislation. In the case of Apple Card Bias, if this were to be scrutinized under the AI Act it would compel both the companies to assess their machine models for the biases along with mandatory documentation of data sources to come to the conclusions and re-check the steps that led to the

14 Scandizzo, Sergio. "In Praise of AI Bias." LSE Europp Blog, 4 September

2024. https://blogs.lse.ac.uk/europpblog/2024/09/04/in-praise-of-ai-bias/.

15 Connectontech, EU AI Act

Diversity and Inclusion Prevails Over Data Protection, https://www.connectontech.com/eu-ai-act-diversity-and-inclusion-prevails-over-data-protection-2/

16 Daniel Eder "The AI Act and Bias

Effectiveness and Technical Feasibility of Countermeasures."

ResearchGate. https://www. researchgate. net/publication/381011736_The_AI_Act_and_Bias_Effectiveness_and _Technical_Feasibility_of_Countermeasures. 17 Lewis Silkin. "EU AI Act:101 – An In-depth Analysis of Europe’s AI Regulatory Framework. " Lewis Silkin, September 25, 2024. https://www. lewissilkin. com/en/insights/ed-eu-ai-act101-an-in-depth-analysis-of-europesai-regulatory-framework. result. Also under the act, the company would be liable to provide clear explanations as to the reason for the bias and the history of the data provided, the human verifiers would also be under the lens in such cases to prevent such outcomes and protect consumer interests. The AI Act just created a leveled playing field so as the use of AI is done under harmonized rules and further build trust and confidence amongst the consumers.

18 If we see United States, the President’s Executive order is governing the AI at the moment where it has laid down basic principles for guidance regarding the safety, security, and rights of the customers. This order makes it mandatory to make amends in the existing framework so as to monitor the use of the AI. 19 Similarly in India, the existing regulation of the Information Technology Act 2000 and the Information Technology (Reasonable Security Practices and Procedures and Sensitive Personal Data or Information) Rules, 2011 reflects on the practices with regards to data. Particularly Section 43A, similar to the AI Act, makes it an obligation for the companies to collect the data and adopt security measures for the collected data. However, this act could be further amended with the AI models and the financial analysis clauses.

20The Credit Information Companies (Regulations) Act,2005, requires the decisions regarding the credit facilities to be transparent as prescribed in the AI Act but this act also needs to be modified with respect to the AI usage in the banking sector. One of the most recent acts, Consumer Protection Act 2019, cares about the individual data protection and the issues regarding the liability of the services offered in the financial markets but it can achieve a more accurate data protection by including the AI model and also committing along the lines of fairness and transparency while collecting the same. There should be a balance between the liability and the services as the companies otherwise would be hampered to use the technology at all or the use of AI would be so high that it would defeat the purpose of the legislation.

Seeing the analysis above we clearly would be able to prove that there need to be more legal

18 Thomson Reuters, AI Act

The World’s First Comprehensive Laws to Regulate AI (Aug. 8, 2024), https://legalsolutions.thomsonreuters.co.uk/blog/2024/08/08/ai-act-the-worlds-first-comprehensive-lawsto-regulate-ai/. 19 Nikita. "Artificial Intelligence in Banking." Hindustan Times, . https://www.hindustantimes.com/htinsight/future-tech/artificial-intelligence-in-banking-101729594392837.html.

20 Guha, Shouvik Kumar, Savage-Mansary, Bash, and Samanta, Navyajyoti. "The Present and Future of AI

Usage in the Banking and Financial Decision-Making Processes within the Developing Indian Economy. " Indian Journal of Law and Technology, vol. 18, no. 1, 2022, article 5. https://repository. nls. ac. in/ijlt/vol18/iss1/5. provisions so as to address the concerns arising from the AI usage in the financial market more effectively. The AI Act being the first ever act to address major issues involving this technology and align it with ethical and fundamental rights leads to the conclusion that the market using the AI is highly subjective to discriminatory and biased processes which can only be brought into light and be solved via new legislation in all countries. In a country like India, there have been instances where the systemic discrimination exists and due to these reasons, the external observer may never be able to see the problem of bias.

There are various concerns when it comes to AI-based technology being adopted in sectors like banking, it just needs to go under ‘significant overhaul’ to check the quality of data being provided and the adherence of it to the AI regulation while giving the output. 21 “It’s true that AI can mimic the human brain, but it can also outperform us mere humans by discovering complex patterns that no human being could ever process and identify. ” ― Ronald M. Razmi22

21 Guha, Shouvik Kumar, Savage-Mansary, Bash, and Samanta, Navyajyoti. "The Present and Future of AI

Usage in the Banking and Financial Decision-Making Processes within the Developing Indian Economy." Indian Journal of Law and Technology, vol. 18, no. 1, 2022, article 5. https://repository.nls.ac.in/ijlt/vol18/iss1/5.

22 Goodreads, Quotes About Artificial

Intelligence, https://www.goodreads.com/quotes/search?q=artificial+intelligence