Introduction

A recent special issue of the Journal for General Internal Medicine, dedicated to obesity treatment in Veterans Health Administration’s (VHA), highlighted research and evaluation priorities, which include establishing structures to coordinate weight management at the health system level, implement large-scale measurement strategies to evaluate outcomes and support data-driven quality improvement, and test interventions to improve reach and weight outcomes. MOVE! continues to be the centerpiece program of VHA’s comprehensive lifestyle behavior change aimed at addressing the high prevalence of obesity among Veterans. Overweight/obesity prevalence continues to increase among Veterans while reach and participation in MOVE! continues to be low. Veterans who may most benefit from integrated obesity treatment options may not participate for an array of reasons including personal barriers and preferences, and site-level lack of access or availability to treatment options. Other novel programs may be needed to help increase access (e.g., telephone-based coaching as an option for rural Veterans who live far from an on-site MOVE! Program).

VHA’s National Center for Health Promotion and Disease Prevention (NCP) is charged with setting prevention policies and testing novel strategies to improve Veterans’ health. They provide oversight for MOVE!. They also work with facility staff (e.g., Health Promotion and Disease Prevention Coordinators) to increase appropriate referrals from Patient Aligned Care Teams (PACT) for prevention programs. Between 2011 and 2013, in partnership with our research teams, NCP evaluated implementation of the Telephone Lifestyle Coaching (TLC) intervention that was launched in 24 VA facilities. TLC addressed multiple lifestyle behaviors including diet and weight loss, stopping smoking, and improving physical activity. 5,321 Veterans were enrolled through referrals from their PACT and 4,110 participated in at least 3 calls. At 6-months, participants reported high rates of success: smoking quit rate was 40%, and 33% of participants reported losing >5% of initial body weight. However, the evaluation lacked controls and relied on self-report of behaviors and behavior change. Given NCP’s plans to disseminate TLC in collaboration with the Office of Rural Health, these preliminary findings need to be validated to better establish the evidence-base for TLC by: 1) confirming self-reported outcomes with weight outcomes from the Clinical Data Warehouse (CDW); 2) establishing a matched-control cohort for comparison; and 3) assessing other effects from TLC including improved disease outcomes, and healthcare costs.

More broadly, NCP, in collaboration with the Healthcare Analysis and Information Group (HAIG), recently fielded the most comprehensive survey of obesity treatment programming across VHA ever conducted. The HAIG survey provides an opportunity for more comprehensive and updated assessment of the association of program structures on weight outcomes across an array of programs including bariatric surgery, pharmacotherapy, and comprehensive lifestyle intervention programs, including integration of smartphone apps and other innovative approaches. Understanding program typologies that are associated with higher reach and weight outcomes will help identify priorities and guide policy aimed at improving current low rates of reach and highly variable weight outcomes.

Thus, the Data-driven Care Evaluation for Prevention (DCEP) project was designed to address two main aims,

  • Aim 1. Determine if the Telephone Lifestyle Coaching Pilot resulted in long term improvements in health outcomes, utilization and costs.
    • 1.a. Describe TLC participants and a control group of Veterans referred, but not enrolled in TLC along demographic and clinical parameters
    • 1.b. Determine electronic medical record weight information for TLC participants and compare that with patient self-report.
    • 1.c. Compare healthcare cost for the TLC pilot cohort to controls for three periods (year before the pilot, the period of the pilot, and up until June 1, 2017).
    • 1.d. Determine if TLC participants who were diagnosed with diabetes experienced improvements in HgA1C.
    • 1.e. Determine if TLC participants who were diagnosed with hypertension experienced improvements in blood pressure.
  • Aim 2. Explore association of obesity treatment program characteristics and structures across VA Medical Centers with reach and weight outcomes.
    • 2.a. Conduct a systematic review of methods that rely on VHA clinical data (CDW) to assess weight outcomes within Veteran populations.
      • 2.a.1. Document algorithms used to assess outcomes based on literature review and review of operational measures
      • 2.a.2. Recommend standard algorithms for valid assessment of weight outcomes based on CDW data.
    • 2.b. Using data from the recently fielded HAIG survey of obesity treatment programs in VHA, develop program typologies to help identify priorities for program development within VHA.
    • 2.c. Compare and contrast configurational comparative and regression analysis approaches, identify program characteristics associated with facility level weight outcomes.

The work described in this document pertains to Aim 2, specifically 2.a., 2.a.1, and 2.a.2. The original plan involves first, developing rules for identifying “valid” weights. Before any analyses of weight outcomes can be established, algorithms for determining valid weights must be established. Currently, there are no standard algorithms for calculating weight change over time at the patient- or site-level. Thus, our first step will be to develop recommendations to standardize these algorithms. Key challenges to resolve, include developing rules for identifying “valid” weights and defining “windows” of time from which to extract available weights for specific outcomes (e.g., 12-month weight change). Criteria guiding algorithm formulation will include minimizing missing data while maximizing precision and accuracy. We will conduct a systematic literature review of all published scientific studies with documented reliance on VHA weight data using administrative data and also contact authors to ensure we capture detailed algorithms already used. This information will be combined with internal operations rules (e.g., used in the VSSC outcomes reports), to guide statistical analyses of alternative approaches. We will extract weight data from the Vital Signs table housed in the VA Corporate Data Warehouse (CDW). The CDW is updated nightly by extracting data from patient EHRs at all VA sites and therefore has near real-time patient information. Based on our experience, issues related to the use of CDW data include measurement error, implausible values or data entry errors, and the lack of associated units of measurement for weight, which may be entered as kilograms or pounds, are common. Thus we will examine weight values for outliers and clear errors (e.g., extremely high or low measures), consistency of weight within patient over time (e.g. weight values of 95 and 210 within a month for the same patient). Careful cleaning and assessment of weight data will minimize missing data for the analysis timeframe.

In the following sections we will describe the sampling scheme, algorithms, and then a thorough examination of the statistical properties of the application of these algorithms to administrative data.

Published Documents

Literature Review

Abstract:

Objective: Administrative data are increasingly used in research and evaluation, yet lack standardized guidelines for constructing measures using these data. Body weight measures from administrative data serve critical functions of monitoring patient health, evaluating interventions, and informing research. We aimed to describe the algorithms used by researchers to construct and use weight measures.

Methods: A structured, systematic literature review of studies that constructed body weight measures from Veterans Health Administration was conducted. We collected key information regarding time frames and time windows of data collection, measure calculations, data cleaning, treatment of missing and outlier weight values, and validation processes.

Results: We identified 39 studies out of 492 nonduplicated records for inclusion. Studies parameterized weight outcomes as: change in weight from baseline to follow-up (62%), weight trajectory over time (21%), proportion of subjects meeting weight threshold (46%), or multiple methods (28%). Most (90%) reported total time in follow-up and number of time points. Fewer reported time windows (54%), outlier values (51%), missing values (34%), or validation strategies (15%).

Conclusions: We found high variability in the operationalization of weight measures. Improving methods to construct clinical measures will support transparency and replicability in approaches, guide interpretation of findings, and facilitate comparisons across studies.

Algorithms

NCP / DCEP Abstract based on HSRD Annual Meeting Submission

Development of Standard Algorithms for Weight Outcomes using (VHA) Administrative Data Wyndy L. Wiitala, Rich Evans, Ann Annis, Jennifer A. Burns, Michelle B. Freitag, Susan D. Raffa, Michael G. Goldstein, Laura J. Damschroder (Ann Arbor COIN and QUERI Program affiliations)

Objectives

Tracking measures of body weight in VHA data systems provides critical health information and is necessary for weight management program evaluations. However, there is conflicting documentation on constructing weight measures, presenting challenges for research and evaluation. We sought to describe and compare methods for extracting and cleaning weight data to develop guidelines for standardized approaches that promote reproducibility.

Methods

We conducted a systematic review of studies that used VHA electronic health record (EHR) weight data, published from 2008 – 2018, and documented the algorithms for constructing patient weight. We applied these algorithms to four cohorts of Veterans in 2008 and 2016 who had primary care visits or were enrolled in the MOVE! Weight Management Program for Veterans. Resulting weight measures were compared at the patient and site levels.

Results

We identified 492 studies and included 39 that utilized weight as outcome variables; 74% included a replicable algorithm. Algorithms varied from cut-offs of implausible weights to complex models using measures within patient over time. Using the MOVE! 2016 cohort, we found differences in the number of weight values after applying the algorithms (54% to 100% of raw data) and slight decreases in variance (SD=55 to 52) and average weights across methods (245 to 240 lbs). The site-level percent of patients with at least 5% weight loss over one year ranged from a minimum of 3 - 13% to a maximum of 22 - 28% across methods. In preliminary site-level analysis, we ranked sites by the percent of patients with at least 5% weight loss. The median rank difference for sites across methods was 46; differences ranged from 5 to 111 across the 129 sites.

Conclusions

Patient weight is an important outcome. Determining the best method to assess weight using EHR data can be computationally demanding. Our preliminary results suggest that for many studies, applying simple cut-offs that require fewer computing resources and are cognitively easier to understand may be sufficient for many studies (e.g., examining point estimates, one-year weight change). Other analyses (e.g., trajectories, facility-level comparisons) may require more nuanced approaches.

Impacts

EHR systems provide clinically rich data that can be used for research and evaluation to improve patient care and outcomes. Our work will help to inform the development of guidelines to facilitate standardization across projects and to promote reproducibility and replication of findings.