Kaiser Permanente researchers #stress need to test how prediction models perform in all #racial, #ethnic, groups
SEATTLE, April 28, 2021 /PRNewswire/ — Models that can successfully predict suicides in a general population sample can perform poorly in some #racial or #ethnic groups, according to a study by #KaiserPermanente researchers published April 28 in JAMA Psychiatry.
The new findings show that 2 #suicide prediction models are less accurate for #Black, #AmericanIndian, and Alaska Native people and demonstrate how all prediction models should be evaluated before they are used. The study is believed to be the first to look at how the latest statistical methods to assess #suicide risk perform when tested specifically in different #ethnic and #racial groups.
More than 47,500 people died from #suicide in the #UnitedStates in 2019, an increase of 33% since 1999. Health care leaders hope to reduce #suicide by using records of #mentalhealth visits, diagnoses, and other factors to identify #patients at highest risk of #suicide and intervene. In recent years, the #VeteransHealthAdministration, HealthPartners, and several #KaiserPermanente regions have started using #suicide prediction models to guide care.
“With enthusiasm growing for #suicide prediction modeling, we must be sure that such efforts consider health equity,” said Yates Coley, PhD, the study’s first author and an assistant investigator at #KaiserPermanente Washington Health Research Institute. “Current methods maximize predictive performance across the entire population, with predictive accuracy in the largest subgroup — white #patients — eclipsing the performance for less-prevalent #racial and #ethnic subgroups.”
The JAMA Psychiatry article follows several studies that have uncovered similar concerns with #racial/ethnic bias in algorithms in domains ranging from criminal justice and policing to health care.
In the new study, Coley and colleagues gathered electronic health records for nearly 14 million outpatient #mentalhealth visits over a 7-year period from 7 health care systems.
Using these #health records, the research team developed 2 different models — a standard statistical logistic regression approach and a random forest machine learning algorithm — to predict #suicide deaths within 90 days of a #mentalhealth visit. The models used demographic characteristics, comorbidities, #mentalhealth and substance use diagnoses, dispensed psychiatric medications, prior suicide attempts, prior #mentalhealth encounters, and responses to #Patient Health Questionnaire 9, which is filled out routinely at #mentalhealth visits.
The models had a high degree of precision in identifying #suicides and avoiding false positives across the entire sample, and for white, #Hispanic, and #Asian #patients. The models fared much worse with #Black, #AmericanIndian, and Alaska Native #patients and #patients without #race/ethnicity recorded.
Welcome to the “next chapter” of my life… being a voice and an advocate for #mentalhealthawarenessandsuicideprevention, especially pertaining to our younger generation of students and student-athletes.
Getting men to speak up and reach out for help and assistance is one of my passions. Us men need to not suffer in silence or drown our sorrows in alcohol, hang out at bars and strip joints, or get involved with drug use.
Having gone through a recent bout of #depression and #suicidalthoughts myself, I realize now, that I can make a huge difference in the lives of so many by sharing my story, and by sharing various resources I come across as I work in this space. #http://bit.ly/JamesMentalHealthArticle
For example, after one of the models was used to predict the 5% of visits with the highest risk of #suicide, the researchers zeroed in on white patient visits in that risk group and found that nearly half of eventual #suicides were identified. By contrast, that risk group only included about 7% of #suicide deaths by #AmericanIndian, Alaska Native, and Black patients. This means that people in these populations who die by #suicide would be much less likely to be identified by the model, indicating that this particular tool is not useful for guiding their care.
- Researchers cited several possible reasons for shortcomings in prediction accuracy that could be the result of embedded biases in the data. #Black, #AmericanIndian, and Alaska Native patients face barriers to accessing #mentalhealthservices. This means that there’s less data on #suicide risk factors so it may be more difficult to make accurate forecasts.
- Even when these populations do have access to #mentalhealthservices, research shows that they are less likely to be diagnosed and treated for #mentalhealthconditions. The clinical data may not accurately reflect risk, affecting the models’ #suicide predictions.
- Suicides in these populations may be incorrectly identified as unintentional or accidental, contributing to the challenge in predicting suicides in these populations.
The 2 prediction models in the study are not the same as the ones now being implemented in health systems. This study examined models that predict #suicide deaths, while models being used in clinical care at #KaiserPermanente predict self-harm, or #suicide attempts. An audit, like the one in this study, of the #suicide-attempt prediction model in use at #KaiserPermanente in Washington did not find #racial or #ethnic disparities in prediction performance. Such audits may be needed at other health organizations using #suicide prediction models, and the study lays out the steps that should be taken when implementing a prediction model to ensure inequities are not perpetuated.
“Before we implement any prediction model in our clinics, we must test for disparities in accuracy and think about possible negative consequences,” said Gregory Simon, MD, MPH, a study co-author and #KaiserPermanente Washington Health Research Institute senior investigator. “Some prediction models pass that test and some don’t, but we only know by doing research like this.”
In addition to Coley and Simon, the other co-authors are Eric Johnson, MS; Maricela Cruz, PhD; and Susan M. Shortreed, PhD. Research reported in this work was supported by the #MentalHealth Research Network with funding from the #NationalInstituteofMentalHealth.
About Kaiser Permanente
#KaiserPermanente is committed to helping shape the future of #healthcare. We are recognized as one of America’s leading #healthcareproviders and not-for-profit health plans. Founded in 1945, #KaiserPermanente has a mission to provide high-quality, affordable health care services and to improve the health of our members and the communities we serve. We currently serve 12.4 million members in 8 states and the District of Columbia. Care for members and #patients is focused on their total #health and guided by their personal Permanente Medical Group physicians, specialists, and team of caregivers. Our expert and caring medical teams are empowered and supported by industry-leading technology advances and tools for health promotion, disease prevention, state-of-the-art care delivery, and world-class chronic disease management. #KaiserPermanente is dedicated to care innovations, clinical research, health education, and the support of community health.