New York State DSRIP Evaluation Design

  • Slides is also available in Portable Document Format (PDF)

Informational Webinar
June 23, 2014

Tom Melnik and Mark Sharp
Office of Quality and Patient Safety

Evaluation Design Requirements

  • The New York State Department of Health will be responsible for developing the DSRIP Evaluation Design for CMS approval.
  • The Evaluation Design will meet the prevailing standards of scientific and academic rigor.
  • Public engagement in the development of the Evaluation Design.
  • An Independent Evaluator to be selected through a competitive RFP to conduct the evaluation.
  • Interim and summative evaluation reports will be submitted to CMS.

Introduction

  • To assess program effectiveness on a statewide level with respect to the MRT triple aim of improved care, better health, and reduced cost.
  • To obtain stakeholder feedback regarding the DSRIP program and the services provided.

DSRIP Logic Model

  Incentive Payments for Achieving Project Progress Milestones   Incentive Payments for Improvement Milestones  
Resources
  • DSRIP Funds
  • Design Grants
  • IAAF Funds
  • NYSDOH
  • Staff/Resources
  • CMS Resources
 → ↓   ↓  
DSRIP Activities

Performance Provider Systems Established:

Projects selected from 3 Domains:
  • System Transformation
  • Clinical Improvements
  • Population–Wide Strategies
 → Short–Term Outcomes

Health Care Safety Net Transformation
  • Improved integration of health care delivery system.
  • Improved care coordination
  • Improved connection between health care settings.
Clinical Improvements
  • Behavioral Health
  • CVD
  • Asthma
  • Diabetes
  • HIV
  • Perinatal/Palliative/Renal
Improved Population Health
  • Reduced mental illness/substance abuse.
  • Reduction in chronic disease.
  • Reduction in HIV/AIDS.
  • Improved health of women, infants, children.
 →





→		 Intermediate Outcomes

Reduced Avoidable Hospital Use
  • Potentially preventable emergency room visits.
  • Potentially preventable readmissions.
  • Prevention Quality Indicators.
  • Prevention Quality Indicators – Pediatric
↓
Long–Term Outcomes

Reduced Health Care Costs
  • Reduced Medicaid expenditures
  • Program cost effectiveness
|top of page|

Methodology

  • Interrupted time series design will be used to assess DSRIP´s effects on health care in New York State.
    • Involves calculating a summary measure of the outcome variable (e.g., average quarterly per person pharmacy costs) at equal time intervals prior to DSRIP's implementation, followed by a series of the same summary measure after DSRIP is implemented.
    • Look for a change in the pattern of the outcome measurement at the time of implementation.

Methodology Interrupted Time Series

Methodology Interrupted Time Series
  • A control group will be added to this design, if available and appropriate.
  • Segmented regression analysis will be used to evaluate changes in level and trend of the outcome before and after DSRIP implementation.
    • Allows for the control of serial autocorrelation (correlation of consecutive outcome values) and seasonal fluctuation in the outcome.
    • Since the unit of analysis is a summary measure (e.g., average per person pharmacy costs), individual-level predictors (e.g., sex) cannot be included in the model.
|top of page|

Methodology Measurement & Date Sources

  • Domain 2, 3 and 4 measures, used in the incentive payment process, will be used in the evaluation to the extent possible.
    • Most of the measures are used in Medicaid quality improvement (QARR/HEDIS), developed by known measures stewards such as NCQA and AHRQ.
  • These measures are based on a number of existing data sources:
    • Medicaid claims
    • SPARCS
    • BRFSS
    • Vital Statistics
    • CAHPS
    • US Census
    • HIV Surveillance
    • Uniform Assessment System
    • Medicare claims
|top of page|

DSRIP Evaluation Objectives

  1. Evaluate the extent to which performing provider systems achieve health care system transformation (Domain 2).
    • Expected changes:
      • Improved system integration.
      • Increased availability and use of primary care.
      • Greater access to health care.
      • Improved care transition protocols.
      • Increased Medicaid spending on primary care services.
      • Decreased Medicaid spending on ER and inpatient services.
  2. Evaluate the extent to which health care quality is improved on a statewide level through clinical improvement (Domain 3) in the treatment of selected diseases and conditions.
    • Improvements expected in:
      • Behavioral health care.
      • Care for cardiovascular disease.
      • Diabetes care.
      • Asthma treatment.
      • Palliative care.
      • HIV/AIDS Care.
      • Perinatal Care
      • Renal Care
  • For purposes of determining incentive payments, multiple existing quality measures for each condition are included in the set of metrics for this domain.
  • To reduce the number of individual pre- and post-DSRIP analyses, aggregation of measures for each condition, or selection of a key measure for each, will be considered.
  1. Evaluate the extent to which population health (Domain 4) is improved as a result of implementation of the DSRIP initiative.
    • Improvements expected in:
      • Population health status (e.g., reduction in premature deaths, increased percentage of adults with health insurance).
      • Chronic disease (e.g., reductions in obesity, cigarette smoking).
      • HIV/STD´s (e.g., reductions in newly diagnosed case rate of HIV, gonorrhea, syphilis, per 100,000).
      • Health of women, infants, & children (e.g., reduction in preterm births, reduced maternal mortality).
      • Mental health and substance abuse (e.g., reduction in suicide death rate, reduction in adult binge drinking).
  • Aggregation of measures or selecting a key measure under each population–wide outcome will again be considered.
  1. Assess the extent to which avoidable hospital use is reduced as a result of DSRIP using four measures:
    • Potentially preventable ER visits.
    • Potentially preventable hospital readmissions.
    • Prevention quality indicators – adult & pediatric.
  2. Evaluate the impact of DSRIP on health care costs.
    • It is expected that Medicaid expenditures will be reduced, or the growth slowed, with the implementation of DSRIP.
  3. Assess the degree of improvement in care quality for specific diseases and conditions under Domain 3.
    • Analyses will involve comparison of care quality for the Domain 3 diseases/conditions between PPS´s that select a particular disease/condition to address (e.g., diabetes) vs. the PPS´s that do not choose that condition.
    • It is anticipated that larger increases in care quality for a particular condition will be observed among PPS´s addressing that condition as compared to PPS´s that did not.
    • Comparisons to be made are contingent upon the final selection of PPS´s.
  4. Compare major program outcomes across Performing Provider Systems.
    • Since sustainability of program activities are an important consideration in the program´s development, comparison of strategies will address the question of which tend to be more effective.
    • It is anticipated that PPS´s will vary on characteristics such as the number of projects selected (a minimum of 5 and a maximum of 10) and the diseases/conditions that are chosen for care improvements.
    • PPS´s will be grouped on these characteristics, and differences in avoidable hospital use and care costs will be examined.
|top of page|

Qualitative Component

Qualitative data will be collected to obtain stakeholders´ experience and perceptions regarding DSRIP at both the implementation and operational stages of the program.

Questions that may be addressed:

  • What difficulties were encountered in developing a PPS?
  • How was rapid cycle evaluation used in developing PPS projects?
  • How did the learning collaboratives support system change?
  • How was DSRIP received by the community?
  • What care improvements have been most notable?

Qualitative data sources:

  • Key informant interviews.
  • Focus groups.
  • Web–based surveys.
  • Planning, implementation, and/or financial documents.

DSRIP Evaluation Timeline

  • Aug. 14, 2014: Submit draft of evaluation plan to CMS.
  • Sept. 14, 2014: Receive feedback from CMS on evaluation plan.
  • Oct. 14, 2014: Submit revised evaluation plan to CMS.
  • November 2014: Begin procurement process for independent evaluator.
  • Fall 2016: Independent evaluator begins work.
  • March 31, 2019: Interim evaluation report due to CMS.
  • June 30, 2020: Preliminary summative evaluation report due to CMS.
  • December 28, 2020: Final summative evaluation report due to CMS.
|top of page|

Contact Information

Tom Melnik
Phone: 518–486–9012
Email: tom.melnik@health.ny.gov

Mark Sharp
Phone: 518–486–9012
Email: mark.sharp@health.ny.gov

|top of page|