#  Contestability on the Margins: Implications for the Design of Algorithmic Decision-making in Public Services 

 



 More and more, algorithms are being used to decide who gets public benefits. One area receiving increasing attention is housing assistance. For example, public service agencies in the US and the World Bank are developing tools to allocate public housing, vouchers, or climate resilient homes. However, housing assistance is already challenging to navigate. And algorithmic decision-making tools add another layer of complexity. Prior work has found that it’s important to center client autonomy when introducing data-driven tools in this context. Ultimately, people trying to access housing assistance should be able to advocate for the resources they need. Policymakers across the world are starting to recognize this too. One recommended safeguard on algorithmic decision-making is contestability, or the ability to challenge a decision made by or with an algorithmic tool. But within policy documents, this recommendation remains high-level, and there is much to learn about contestability from how people already contest decisions, even when algorithms are not involved.

 [This project](/publications/understanding-contestability-margins-implications-design-algorithmic-decision), published at CHI 2024 with collaborators at Harvard and Georgetown, asked the following research questions:

 \* How do marginalized communities currently navigate denials of housing assistance? Again, we were looking at cases without algorithms, to understand existing pathways and barriers

 \* What factors mediate communities’ ability to achieve their desired outcomes?

 \* And then based on our findings, how might algorithmic decision-making processes support people in navigating adverse outcomes?

 The context for this research was in the urban Northeast US and rural South India. We wanted to understand contestation in different cultural and political contexts, for more robust and transferable insights. We interviewed 44 stakeholders in housing assistance across the US and India contexts. This included applicants accessing these services, staff who administer these services, and intermediaries such as lawyers, social workers, and non-governmental organization (NGO) workers. These are people who created links between the worlds of applicants and public services. We asked about the whole application process, from applying to appealing. We also attended to different power dynamics in each context, including race and immigration in the US, and caste and rurality in India.

 Our findings complicated a couple of assumptions in Human-Computer Interaction (HCI), Artificial Intelligence (AI), and policy about what is required to contest a decision. One assumption is that explanations help people contest decisions. We found that an explanation of why someone was denied, while necessary, was insufficient to actually contest. This was because there were numerous barriers to acting on explanations for marginalized communities. For example, there was fear of contestation, or lack of access to legal knowledge. Another assumption is that human review is sufficient to inject accountability in an algorithmic decision-making process. But for our participants, influencing human decision-makers was fraught with power dynamics. For example, they might not be able to speak in a way that is convincing to decision-makers. In the Indian context, where power differentials were especially high, applicants might simply be ignored.

 So then, what did enable participants to contest decisions? We found that it was the work of accompaniment done by intermediaries. Accompaniment, a concept developed by physician and medical anthropologist Paul Farmer, is the open ended work of supporting people as they navigate inaccessible social systems. Intermediaries helped address emotional barriers to contestation. They used culturally-situated strategies to help participants feel safer contesting decisions. They also helped applicants construct effective appeals and ensure responsiveness of public service administrators. Despite their importance, it was challenging for applicants to get in touch with intermediaries, especially in the US context. Many applicants did not know who they could call for support. This was less of an issue in the rural Indian context, where NGO workers did outreach door to door.

 Clearly contestation was costly, and applicants might not be successful without support. So we also highlight that there were mechanisms that helped avoid denials in the first place, and these could help inform algorithm design. In one case, administrators were required to request information about individual circumstances before a denial. For example, in the US context, applicants had the right to provide contextual information before they were denied on the basis of their criminal records. Or in the Indian context, applicants could be considered for alternative parcels of land if their original request couldn’t be fulfilled.

 So what do these findings on the work of contestation mean for designing algorithmic decision-making tools? First, we need to acknowledge that decision-making processes are already inequitable, and intermediaries fill the gap. This means we have to consider intermediaries as key stakeholders in designing algorithmic decision-making processes, and contestation mechanisms. Existing relationships between public services and intermediaries could be leveraged to make sure that intermediaries have input into algorithmic tools and understand their inner workings. This ensures that advocating for their clients does not become even more difficult. Relatedly, there are opportunities to strengthen connections between applicants and intermediaries. For example, systems could proactively ask applicants if they want to be connected to someone, rather than just providing contact information. This also implies that policy recommendations for equitable contestation mechanisms should also mean more resources for intermediaries to do outreach. Finally, we can infuse accompaniment into algorithmic decision-making processes. This could help avoid denials or make appeals more accessible. For example, algorithms can be designed to ask for more information if decisions are too close to the decision boundary. Or explanations can specifically point out information that needs to be reframed or contextualized for an appeal. Intermediaries' work is still essential to support the most vulnerable applicants, but this could go some distance in reducing the burden of contestation.