Global Landscape Analysis of Quality within PBI Programs

Women waiting at a health clinic. Photo Credit: URC.

Research Overview

Globally, research has demonstrated that performance-based incentives (PBI) programs are effective in increasing the quantity of health services provided. In comparison, there is little evidence for the effect of PBI programs on quality of care (QoC) provided to patients. For programs that do have a quality component, providers are often scored based on structural indicators of quality over which they have little to no control, such as infrastructure, equipment and availability of pharmaceuticals. Program implementers recognize the need to move from these structural indicators of quality to process and outcome indicators, including the competence, knowledge application, and effort of clinicians, meetings organized, and hygiene and facility plans in place. Because of this TRAction aimed to build an evidence base on the effect of PBI programs on these dimensions of quality of care.

TRAction selected ThinkWell Global to carry out two landscape analyses focusing on quality and quantity indicators and develop several country case studies that demonstrated how and to what extent QoC had been incorporated into PBI programs globally. The landscape analyses included a compilation of information on quality and quantity indicators, data verification, and the revision process used to improve PBI programs. In addition, the landscape analyses examined different methods, frequency, and costs associated with score verification. The quantity and quality indicator analyses included logs of QoC indicators found in PBI program checklists, classified by health service level (primary, secondary, and tertiary), dimension (hygiene and sanitation, privacy, family planning counseling, etc.), indicator description, score and indicator type (structural, process, and output). From the quality landscape analysis, ThinkWell Global and TRAction identified four country PBI programs on which in-depth case studies were developed. The case studies examined how program planners and implementers incentivized quality of care within the unique context of each country’s PBI program.

Project Location

Global Landscape Analyses. Case studies have been developed on PBI programs in Mozambique, Nigeria, the Democratic Republic of the Congo, and Senegal.

Research Objectives

To conduct landscape analyses of all PBI programs implemented globally
To catalog and map PBI program indicators, checklists, and verification procedures by technical area and system level
To develop country case studies on the planning and implementation of PBI programs that focus on quality of care

Study Approach

In order to develop a global quality indicator landscape analysis, ThinkWell Global conducted a literature review on quality within PBI programs and produced an annotated bibliography. The annotated bibliography summarized peer-reviewed and grey literature on PBI interventions with a QoC component, cataloging which types of services were incentivized and described the QoC measurement tools utilized in each scheme. ThinkWell Global also produced a database of current PBI projects with specific QoC improvement objectives and furthermore, a database of current PBI projects with specific QoC quantity indicators. The matrices lists PBI projects by country and funder, describes each program, lists the types of services incentivized, and describes the QoC tool utilized by the scheme to measure either quality or quantity.

ThinkWell Global compiled a database of QoC indicators found in program quality checklists, which are used to measure quality within PBI schemes and another database of QoC indicators found in program quality checklists, which are used to measure quantity within PBI schemes. The databases classify each indicator by health service level (primary, secondary, tertiary), dimension (hygiene and sanitation, privacy, family planning counseling, etc.), indicator description, score, and indicator type (structural, process, output).

Findings

The quality indicator analysis identified 32 PBI implementations with QoC improvement objectives between 2008-2015 in 28 low and middle income countries (predominantly in Sub-Saharan Africa). These projects were funded by national governments, the World Bank, USAID, CDC and the German Development Bank KfW. The majority of programs were funded by the World Bank. To date, ThinkWell Global cataloged approximately 8,490 quality indicators in a database. On average, checklists contained 125 indicators; within the same program. Using the Donabedian framework, 80% of indicators were structure-type, 19% process-type, and 1% outcome-type. Furthermore, 57% of indicators related to availability of resources, 25% to managing the facility and 17% assess the competence and effort of clinicians. This dataset is the largest in existence providing robust information on how QoC is being incentivized within PBI.

The indicators were classified into 6 dimensions of QoC:

Resource Availability – The structural resources required to delivering quality care (structural)
Pharmaceutical Availability – The pharmaceutical and consumable resources required to delivering quality care (structural)
Facility Management – The management processes required to ensure that the health facility functions optimally to deliver quality care (process)
Competence of Clinician – The clinician has the clinical knowledge required to care for, or diagnose and treat, patients (process)
Effort of Clinician – The clinician correctly applies appropriate clinical knowledge based on the latest medical teaching to deliver quality care (process)
Other – A variety of types of indicator which don’t fit easily into these other dimensions

The study team is still cataloging the quantity indicators in a database.

Publications

Quality of Care in Performance-Based Financing: How It Is Incorporated in 32 Programs Across 28 Countries

Country Case Studies

From the quantity indicator analysis, ThinkWell Global identified country PBI programs on which to develop in-depth case studies that examined the how and to what extent countries were incentivizing quality of care. Case studies were developed on PBI programs in Mozambique, Nigeria, the Democratic Republic of the Congo, and Senegal. These case studies combined information from document reviews, in-depth interviews with key informants, and a review of available national facility-level on quality indicators.

These case studies exemplified that improving quality of care is complex in achieving and an iterative process. PBI programs in-country, found the need to initially amend quality of care instruments because not one instrument ‘fits all’ and due to their excessive length and therefore burden of completing. Furthermore, PBI programs realized the need to regularly amend instruments due to changes in standards of care. Interestingly, the quality score improved within the DRC PBI program (average score during first quarter of implementation ranged between 32-47%, and the average score ranged from 58-78% by the eighth quarter of implementation) whereas the quality score within the PBI program in Mozambique was inconsistent throughout implementation. Overall, findings proved that completing quality checklists and performing data verifications can be burdensome and insufficient, the need for baseline data and more transparent performance data, the need for initial quality health worker training regarding quality of care and instruments (for example, the definition of quality of care can be different for providers versus clients), and reliable timing of incentive payments.

Databases

Research Into Action

Donors and country implementers are increasingly interested in incorporating quality improvement objectives into PBI programs globally. These key stakeholders want to learn the best ways to incorporate quality into programs, including which indicators are most appropriate and feasible to measure, how to verify quality scores, and the effect of quality checklists on provider behavior and health outcomes. The case studies were presented to donors and country implementers at workshops and conferences throughout 2016, including a workshop co-hosted by TRAction and Harvard’s Maternal Task Force and another workshop co-hosted by the World Bank and USAID|TRAction. This study provided a rich analysis of how PBI programs incentivize different types of services, how checklists and indicators have changed over time, and how programs contain costs associated with score verification. Findings from this study offer a high-level, global comparison of paying for QoC within PBI programs. The outcomes of this research informed PBI program planners as they target low-performing service areas in the future.

Research Partners

ThinkWell Global

Principal Investigator

Jessica Gergen, ThinkWell Global

Project Status

Region:

Africa

Country / Countries:

Tweets by @TRActionProject