Publications

Our team has conducted research that has led to publications in several high quality professional and academic journals. Below are abstracts from our team’s featured publications; follow the links provided to read the articles.

Facility-Level Variation in Cardiac Stress Test Utilization among Patients with Diabetes Mellitus: Findings from the Veterans Affairs National Database. (Mahtta D, Ahmed ST, Shah NR, Ramsey DJ, Akeroyd JM, Nasir K, Hamzeh IR, Elgendy IY, Waldo SW, Al-Mallah MH, Jneid H, Ballantyne CM, Petersen LA, Virani SS, 2020). Cardiac stress testing in patients with diabetes mellitus (DM) is a topic of much debate.  We demonstrated a 37% residual facility-level variation in stress test use. This level of unexplained variation across a nationwide health care system presents an opportunity for conglomeration of our methodology along with identification of provider-level variables, patient outcomes data, and appropriateness of stress testing in order to improve quality of stress testing.

Risk of Cirrhosis and Hepatocellular Cancer in Patients with Non-Alcoholic Fatty Liver Disease and Normal Liver Enzymes. (Natarajan Y, Kramer JR, Yu X, Li L, Thrift AP, El-Serag HB, Kanwal F, 2020). The long-term risk of disease for patients with nonalcoholic fatty liver disease (NAFLD) in the absence of elevated enzymes is unclear. We conducted a retrospective cohort study using the Corporate Data Warehouse of the Veterans Health Administration. Patients with hepatic steatosis with persistently normal ALT are at lower risk for cirrhosis compared to those with steatosis and elevated ALT and not different from the risk in a clinical cohort without hepatic steatosis.

A Roadmap to Advance Patient Safety in Ambulatory Care. (Singh H, Carayon P, 2020). In the two decades since “To Err is Human: Building a Safer Health System” was issued by the Institute of Medicine (now the National Academy of Medicine-NAM), advances in patient safety have focused mainly on inpatient settings whereas outpatient settings have been overlooked. However, accumulated evidence leaves little justification to continue neglecting ambulatory safety.  The time to accelerate initiatives to reduce preventable harm in the outpatient setting has arrived. Key milestones related to scientific advances, practice improvements, policy changes, and strategies to partner with patients and families can accelerate meaningful advances to reduce patient harm in ambulatory care.   

A Sociotechnical Framework for Safety-related EHR Research and Reporting: The SAFER Reporting Framework. (Singh H, Sittig DF, 2020). Electronic health record (EHR)-based interventions to improve patient safety are complex and sensitive to who, what, where, why, when, and how they are delivered. Success or failure depends not only on the characteristics and behaviors of individuals who are targeted by an intervention, but also on the technical characteristics of the intervention and the culture and environment of the health system that implements it. Current reporting guidelines do not capture the complexity of sociotechnical factors (technical and nontechnical factors, such as workflow and organizational issues) that confound or influence these interventions. This article proposes a methodological reporting framework for EHR interventions targeting patient safety and builds on an 8-dimension sociotechnical model previously developed by the authors for design, development, implementation, use, and evaluation of health information technology. The Safety-related EHR Research (SAFER) Reporting Framework enables reporting of patient safety-focused EHR-based interventions while accounting for the multifaceted, dynamic sociotechnical context affecting intervention implementation, effectiveness, and generalizability. As an example, an EHR-based intervention to improve communication and timely follow-up of subcritical abnormal test results to operationalize the framework is presented. For each dimension, reporting should include what sociotechnical changes were made to implement an EHR-related intervention to improve patient safety, why the intervention did or did not lead to safety improvements, and how this intervention can be applied or exported to other health care organizations. A foundational list of research and reporting recommendations to address implementation, effectiveness, and generalizability of EHR-based interventions needed to effectively reduce preventable patient harm is provided. The SAFER Reporting Framework is not meant to replace previous research reporting guidelines, but rather provides a sociotechnical adjunct that complements their use.

COVID-19 and the Need for a National Health Information Technology Infrastructure. (Sittig DF, Singh H, 2020). The need for timely, accurate, and reliable data about the health of the US population has never been greater. Critical questions include the following: (1) how many individuals test positive for severe acute respiratory syndrome coronavirus (SARS-CoV-2) and how many are affected by the disease it causes—novel coronavirus disease 2019 (COVID-19) in a given geographic area; (2) what are the age and race of these individuals; (3) how many people sought care at a health care facility; (4)  how many were hospitalized;(5)within individual hospitals, how many patients required intensive care, received  ventilator support, or died; and (6) what was the length of stay in the hospital and in the intensive care unit for patients who survived and for those who died. The privacy, legal, and ethical trade-offs warrant further consideration, even though in an era of eroded trust, some discussions will be difficult. Recently, HHS issued limited waivers to facilitate the nation’s ability to care for patients during the COVID-19 pandemic. These changes show how regulations can be modified during extraordinary times. With a sharp focus on maximizing benefits of scarce resources, treating everyone equally, and prioritizing efforts to save lives while maintaining trust and confidentiality, a national health IT infrastructure could meet the highest ethical standards. It is time to make some difficult decisions and exploit and enhance existing technical capability to build and deploy these solutions. Given the severity and immediacy of theCOVID-19 pandemic, the US should no longer rely on outdated laws, social norms, or potentially inaccurate modalities to obtain timely, accurate, and reliable health information essential to save lives.

Operational Measurement of Diagnostic Safety: State of the Science. (Singh H, Bradford A, Goeschel C, 2020) This article discusses the evidence based on operational measurement of diagnostic safety and how health care organizations can measure and learn from diagnostic errors to improve patient safety.

The Path to Diagnostic Excellence Includes Feedback to Calibrate how Clinicians Think. (Meyer AND, Singh H, 2019) Improving diagnosis in health care is considered the next imperative for patient safety. Rapidly evolving diagnostic tests and treatments and competing priorities and pressures encountered by clinicians to deliver high-quality, low-cost health care make this a major challenge. Clinicians frequently balance undertesting, possibly missing a diagnosis, with pursuing overzealous diagnostic testing, which could be harmful and costly. Rigorous multidisciplinary research and innovation from cognitive psychology, human factors, informatics, and social sciences are needed to stimulate previous efforts to reduce diagnostic errors.

Application of Electronic Trigger Tools to Identify Targets for Improving Diagnostic Safety. (Murphy DR, Meyer AND, Sittig DF, et. al., 2018) Electronic trigger (e-trigger) tools, which mine vast amounts of patient data to identify signals indicative of a likely error or adverse event, offer a promising method to efficiently identify diagnostic errors. The authors present a knowledge discovery framework, the Safer Dx Trigger Tools Framework, that enables health systems to develop and implement e-trigger tools to identify and measure diagnostic errors using comprehensive electronic health record (EHR) data. The authors outline key future research, including advances in natural language processing and machine learning, needed to improve effectiveness of e-triggers. Integrating diagnostic safety e-triggers in institutional patient safety strategies can accelerate progress in reducing preventable harm from diagnostic errors.

Beyond Dr. Google: The Evidence about Consumer-Facing, Digital Tools for Diagnosis. (Millenson ML, Baldwin JL, Zipperer L, et. al., 2018) Direct-to-consumer (DTC), interactive, diagnostic apps with information personalization capabilities beyond those of static search engines are rapidly proliferating. While these apps promise faster, more convenient and more accurate information to improve diagnosis, little is known about the state of the evidence on their performance or the methods used to evaluate them. The authors conducted a scoping review of the peer-reviewed and gray literature and found that apps vary widely in functionality, accuracy, safety and effectiveness, although the usefulness of this evidence was limited by a frequent failure to provide results by named individual app.

Methods for Patient-Centered Interface Design of Test Result Display in Online Portals. (Nystrom DT, Singh H, Baldwin J, et. al., 2018) Patients have unique information needs to help them interpret and make decisions about laboratory test results they receive on web-based portals. However, current portals are not designed in a patient-centered way and little is known on how best to harness patients’ information needs to inform user-centered interface design of portals. The authors designed a patient-facing laboratory test result interface prototype based on requirement elicitation research and used a mixed-methods approach to evaluate this interface. They found that requirement elicitation studies can inform the design of a patient-facing test result interface, but considerable user-centered design efforts are necessary to create an interface that patients find useful. To promote patient engagement, health information technology designers and developers can use similar approaches to enhance user-centered software design in patient portals.

Adherence to recommended electronic health record safety practices across eight health care organizations. (Sittig DF, Salimi M, Aiyagari R, et. al., 2018) The Safety Assurance Factors for EHR Resilience (SAFER) guides were released in 2014 to help health systems conduct proactive risk assessment of electronic health record (EHR)- safety related policies, processes, procedures, and configurations. The extent to which SAFER recommendations are followed is unknown. The authors conducted risk assessments of 8 organizations of varying size, complexity, EHR, and EHR adoption maturity. They found that, despite availability of recommendations on how to improve use of EHRs, most recommendations were not fully implemented. They suggest new national policy initiatives are needed to stimulate implementation of these best practices.

Evaluating a mobile application for improving clinical laboratory test ordering and diagnosis. (Meyer AND, Thompson PJ, Khanna A, et. al., 2018) Mobile applications for improving diagnostic decision making often lack clinical evaluation. The authors evaluated if a mobile application improves generalist physicians' appropriate laboratory test ordering and diagnosis decisions and assessed if physicians perceive it as useful for learning. They found that a mobile app, PTT Advisor, may contribute to better test ordering and diagnosis, serve as a learning tool for diagnostic evaluation of certain clinical disorders, and improve patient outcomes. Similar methods could be useful for evaluating apps aimed at improving testing and diagnosis for other conditions.

Electronic health record reviews to measure diagnostic uncertainty in primary care. (Bhise V, Rajan SS, Sittig DF, et al., 2018) Diagnostic uncertainty is common in primary care. Because it is challenging to measure, there is inadequate scientific understanding of diagnostic decision-making during uncertainty. The authors' study found that, while current diagnosis coding mechanisms (ICD-9 and ICD-10) are unable to capture uncertainty, review of EHR documentation can help identify diagnostic uncertainty with moderate reliability.

Patient perceptions of receiving test results via online portals: a mixed-methods study (Giardina TD, Baldwin J, Nystrom DT, et al., 2017) Online portals provide patients with access to their test results, but it is unknown how patients use these tools to manage results and what information is available to promote understanding. A mixed-methods study was conducted to explore patients’ experiences and preferences when accessing their test results via portals. The author's findings suggest that online portals are not currently designed to present test results to patients in a meaningful way. They found that simply providing access via portals is insufficient, and additional strategies are needed to help patients interpret and manage their online test results.

Electronic Triggers to Identify Delays in Follow-Up of Mammography: Harnessing the Power of Big Data in Health Care (Murphy DR, Meyer AN, Vaghani V, et al., 2017) The authors conducted a study to evaluate the effectiveness of an electronic trigger to flag delayed follow-up on mammography. Flagged records were reviewed to determine the trigger's performance characteristics. The frequency of delays and patient communication related to abnormal results, reasons for lack of follow-up, and whether patients were subsequently diagnosed with breast cancer were also assessed. The authors concluded that clinical application of mammography-related triggers could help detect delays in follow-up of results.

An electronic trigger based on care escalation to identify preventable adverse events in hospitalised patients (Bhise V, Sittig DF, Vaghani V, et al., 2017). The authors refined the methods of Institute of Healthcare Improvement’s Global Trigger Tool (GTT) application and leveraged electronic health record (EHR) data to improve detection of preventable adverse events. The EHR data-based trigger and modified review process were able to efficiently identify hospitalized patients with preventable adverse events, including diagnostic errors. Such e-triggers can help overcome limitations of currently available methods to detect preventable harm in hospitalized patients.

The Burden of Inbox Notifications in Commercial Electronic Health Records. (Murphy DR, Meyer AND, Russo E, et. al., 2016) With wider use of electronic health records (EHRs), physicians increasingly receive notifications via EHR-based inboxes (e.g., Epic’s “In-Basket” and GE Centricity’s “Documents”). Types of notifications include test results, referral responses, medication refill requests, and messages from clinicians, among others. Information overload is of emerging concern because new types of EHR-based notifications and ‘FYI’ messages can be easily created (versus in paper-systems) and this additional workload remains uncompensated despite reimbursement reductions. Moreover, EHRs make it easier to measure information load. The authors quantified notifications physicians received via inboxes of commercial EHRs to estimate their burden.

Challenges in Patient Safety Improvement Research in the Era of Electronic Health Records. (Russo E, Sittig DF, Murphy DR, et. al., 2016) Electronic health record (EHR) data repositories contain large volumes of aggregated, longitudinal clinical data that could allow patient safety researchers to identify important safety issues and conduct comprehensive evaluations of health care delivery outcomes. However, few health systems have successfully converted this abundance of data into useful information or knowledge for safety improvement. In this paper, the authors use a case study involving a project on missed/delayed follow-up of test results to discuss real-world challenges in using EHR data for patient safety research. To leverage EHRs and their abundant data for patient safety improvement research, many current data access and security policies and procedures must be rewritten and standardized across health care organizations. These efforts are essential to help make EHRs and EHR data useful for progress in our journey to safer health care.

Computerized Triggers of Big Data to Detect Delays in Follow-up of Chest Imaging Results (Murphy DR, Meyer AN, Bhise V et al., 2016). A “trigger” algorithm was used to identify delays in follow-up of abnormal chest imaging results in a large national clinical data warehouse of electronic health record (EHR) data. The report found that the application of triggers on “big” EHR data may aid in identifying patients experiencing delays in diagnostic evaluation of chest imaging results suspicious for malignancy.

The global burden of diagnostic errors in primary care (Singh H, Schiff GD, Graber ML, 2016). The authors discuss the global significance, burden and contributory factors related to diagnostic errors in primary care. They also summarize interventions based on available data and suggest next steps to reduce the global burden of diagnostic errors. They recommend that the World Health Organization (WHO) consider bringing together primary care leaders, practicing frontline clinicians, safety experts, policymakers, the health IT community, medical education and accreditation organizations, researchers from multiple disciplines, patient advocates, and funding bodies among others, to address the many common challenges and opportunities to reduce diagnostic error.

Measuring and improving patient safety through health information technology: The Health IT Safety Framework (Singh H, Sittig DF, 2015). In response to the fundamental conceptual and methodological gaps related to both defining and measuring health IT-related patient safety, the authors propose a new framework, the Health IT Safety (HITS) measurement framework, to provide a conceptual foundation for health IT related patient safety measurement, monitoring and improvement. The framework proposes to integrate both retrospective and prospective measurement of HIT safety with an organization's existing clinical risk management and safety programs. It aims to facilitate organizational learning, comprehensive 360-degree assessment of HIT safety that includes vendor involvement, refinement of measurement tools and strategies, and shared responsibility to identify problems and implement solutions.

Improving Diagnosis in Health Care – The Next Imperative for Patient Safety (Singh H, Graber M, 2015). Based on a 1999 Institute of Medicine report, the authors discuss the history of diagnostic error, previous work and research in the area, current policies and recommendations regarding the reduction diagnostic error and future actions that can be taken to further improve diagnosis and reduce patient harm from diagnostic error.

Editorial: Helping Health Care Organizations to Define Diagnostic Errors as Missed Opportunities in Diagnosis (Singh H, 2014). An editorial discussing an article from Graber et al. In the editorial, Dr. Singh touches on challenges that healthcare organizations face regarding diagnostic errors and missed opportunities, providing a conceptual model of missed opportunities in diagnosis.

Ebola US Patient Zero: lessons on misdiagnosis and effective use of electronic health records (Upadhyay DK, Sittig DF, Singh H, 2014). The authors discuss the first travel-associated case of U.S. Ebola on Sept. 30, 2014, to highlight the public health challenge of diagnostic errors and discuss the effective use of EHRs in the diagnostic process. We analyze the case to discuss several missed opportunities and outline key challenges and opportunities facing diagnostic decision-making in EHR-enabled healthcare.

The frequency of diagnostic error in outpatient care: estimations from three large observational studies involving US adult populations (Singh H, Meyer AN, Thomas, EJ, 2014). The authors estimated the frequency of diagnostic errors in the U.S. adult population by synthesizing data from three previous studies of clinic-based populations that used conceptually similar definitions of diagnostic error. Their estimate suggests that diagnostic errors affect at least 1 in 20 U.S. adults. This foundational evidence should encourage policymakers, healthcare organizations and researchers to start measuring and reducing diagnostic errors.

Types and Origins of Diagnostic Errors in Primary Care Settings (Singh H, Giardina TD, Meyer AN et al., 2013). The authors reviewed medical records of diagnostic errors detected at two sites through EHR–based triggers. It was found that most errors were related to process breakdowns in the patient-practitioner clinical encounter. It was found that the diagnostic errors identified in the study involved a large variety of common diseases and had significant potential for harm. Most errors were related to process breakdowns in the patient-practitioner clinical encounter.

Electronic health record-based surveillance of diagnostic errors in primary care (Singh H, Giardina TD, Forjuoh SN et al., 2012). Diagnostic errors in primary care are harmful but difficult to detect. The authors tested an electronic health record (EHR)-based method to detect diagnostic errors in routine primary care practice. The report found that while physician agreement on diagnostic error remains low, an EHR-facilitated surveillance methodology could be useful for gaining insight into the origin of these errors.