Abstract:
This paper focuses on learning index-based policies in rest- less multi-armed bandits (RMAB) with applications to public health concerns such as maternal health. Maternal health is a very important public health concern. It refers to the health of women during their pregnancy, childbirth, and the post- natal period. Although maternal health has received significant attention [World Health Organization, 2015], the number of maternal deaths remains unacceptably high, mainly because of the delay in obtaining adequate care [Thaddeus and Maine, 1994]. Most maternal deaths can be prevented by providing timely preventive care information. However, such information is not easily accessible by underprivileged and low-income communities. For ensuring timely information, a non-profit organization, called ARMMAN [2015], carries out a free call-based program called mMitra for spreading preventive care information among pregnant women. Enrollment in this program happens through hospitals and non-government organizations. Each enrolled woman receives around 140 automated voice calls, throughout their pregnancy period and up to 12 months after childbirth. Each call equips women with critical life-saving healthcare information. This program pro- vides support for around 80 weeks. To achieve the vision of improving the well-being of the enrolled women, it is important to ensure that they listen to most of the information sent to them via the automated calls. However, the organization observed that, for many women, their engagement (i.e., the overall time they spend listening to the automated calls) gradually decreases. One way to improve their engagement is by providing an intervention (that would involve a personal visit by health-care worker). These interventions require the dedicated time of the health workers, which is often limited. Thus, only a small fraction of the overall enrolled women can be provided with interventions during a time period. More- over, the extent to which the engagement improves upon intervention varies among individuals. Hence, it is important to carefully choose the beneficiaries who should be provided interventions at a particular time period. This is a challenging problem owing to multiple key reasons: (i) Engagement of the individual beneficiaries is un- certain and changes organically over time; (ii) Improvement in the engagement of a beneficiary post-intervention is un- certain; (iii) Decision making with respect to interventions (which beneficiaries should have intervention) is sequential, i.e., decisions at a step have an impact on the state of beneficiaries and decisions to be taken at the next step; (iv) Number of interventions are budgeted and are significantly smaller than the total number of beneficiaries. Due to the uncertainty, sequential nature of decision making, and weak dependency amongst patients through a budget, existing research [Lee et al., 2019; Mate et al., 2020; Bhattacharya, 2018] in health interventions has justifiably employed RMABs. However, existing research focuses on the planning problem assuming a priori knowledge of the underlying uncertainty model, which can be quite challenging to obtain. Thus, we focus on learning intervention decisions in absence of the knowledge of underlying uncertainty.
