Behind the Scenes of U.S. News & World Report Best Hospitals Rankings

August 01, 2023 - For more than 30 years, the U.S. News & World Report Best Hospitals rankings have sought to help patients make the best decisions about where they can access care.

But for the 2023-2024 ratings year, the publication’s team of journalists and data scientists is taking a new approach with methodology updates. In a June announcement, U.S. News & World Reports unveiled 18 revisions to the publication’s methodology that range from updates to how the Best Hospitals rankings risk adjust to efforts to weave in elements of health equity.

And that’s just skimming the surface, according to Ben Harder, managing editor and chief of Health Analysis at U.S. News & World Reports.

Harder, who co-authored a post announcing the updates, said that the methodology overhaul was a wide-ranging effort that builds off the publication’s history of making updates where applicable.

“We do this work year-round,” Harder said in a Teams call with PatientEngagementHIT. “As soon as we published the results of last year, my team was already hard at work on these changes.”

Those changes are reflected in the 2023-2024 Best Hospitals rankings recently released by the publication. The rankings list out 484 Best Regional Hospitals, the 22 best of which are added to the U.S. News & World Reports “Honor Roll.”

Using Ratings as a Patient Engagement Tool

Fundamentally, these rankings—and the methodology behind them—are intended to help patients make the best decisions they can about their care. Harder emphasized that healthcare consumers and their care teams, which include family caregivers, are the rankings’ target audience.

“Consumers, when they are patients, want a good hospital,” according to Harder. “That's one of several criteria—they want to make sure their insurance covers it, that it's conveniently located, and so on—but they want a good hospital.”

“And they don't just want a generic good hospital,” he added. “They want a good hospital for what they need.”

Over the years, U.S. News & World Reports has built out its Best Hospitals rankings to get more granular than simply listing which facility is the best. Over the course of the rankings’ 34-year history, the publication has expanded the number and type of services it evaluates and become more intentional about how it measures quality.

For example, this year, the rankings list the best hospitals for leukemia, lymphoma, and myeloma, which is the 36th category in which U.S. News & World Report assesses general acute care hospitals. Overall, the publication has 15 different rankings, which should help the patient better assess which hospital they visit for care.

“Our goal is to build the best approach to identifying which hospitals are strong in each of those services and to gradually expand the number of services that we'll be able to provide that decision support for patients,” Harder explained.

But how does Harder and his team get to those ratings?

Whenever possible, it starts with outcome measures because those typically mean more to patients and healthcare professionals alike. There are some hiccups around risk adjustment (this year’s ratings reflect an update in the tool the publication uses to calculate risk adjustment), so Harder said his team has to be intentional about also including other key clinical quality measures.

“To complement those outcome measures, we also look at a number of structural characteristics of hospitals,” he noted.

Those might include nurse staffing ratios or the sufficiency of nurse staffing, the clinical technologies available to patients, and even patient experience measures like HCAHPS scores.

Patients aren’t responsible for being able to decipher each of those measures and how they work together to illustrate hospital quality. That’s the whole premise of the Best Hospital rankings, Harder said. The composite score for each service line of the hospital is how patients can get the big picture of hospital quality, and if they choose, they can look at the granular data that went into getting to that score.

Methodology Innovations

Like Harder mentioned, these quality ratings haven’t stayed static for the more than 30 years they’ve been around. Ensuring the Best Hospital rankings continue to deliver accurate and actionable information to patient end-users, it’s incumbent upon his team to make sure they are constantly reassessing the ranking methodology.

Particularly, the team works to ensure the rankings look at the measures that matter the most to patients.

For example, Harder’s team has started looking at home time or days alive at home in recognition that most patients want to spend time at home with their loved ones as opposed to in a hospital or skilled nursing facility.

The team has also taken into account trends in healthcare access, like where patients get surgeries done. In alignment with more site-neutral payments, Harder and his team observed more searches for outpatient surgeries, meaning that the Best Hospital ratings need to start displaying information beyond just the inpatient setting.

And it’s not just patient feedback that’s informing methodology shifts.

“We would never be able to do the job that we do just with the smart people in my team,” Harder noted. “It is because we also have smart people in the healthcare professions who are providing feedback to us as we go, and we try to listen very carefully to that feedback.”

Take, for instance, feedback on patient safety indicators (PSIs) that were once used as part of the Best Hospital rankings. Harder was alerted to new data uncovering flaws in those PSI from some of his colleagues in the healthcare industry. Once the U.S. News & World Report team was able to further investigate, they dropped them from their methodology.

Interacting With Healthcare Professionals

The innovation in quality measurement and quality ratings is just the tip of the iceberg for industry collaboration, Harder added. While he stressed that the Best Hospital rankings are patient-facing and are intended for consumer use, he acknowledged that many organizations have their own uses for the ratings.

It’s not uncommon for hospitals to use the scorecards for their own quality improvement initiatives, Harder indicated, so his team has experts tasked specifically with helping hospitals understand the data.

But the tension between the rankers and the rankees might be more complex than that; it is for nearly every measure of hospital quality. In summer 2023, around the same time Harder’s team unveiled the 2023-2024 methodology updates, the University of Pennsylvania Health System said it was going to pull out of the ratings.

Some of Penn Med’s hospitals, including the University of Pennsylvania-Penn Presbyterian were longtime leaders in the Best Hospitals rating, ranking in the top 20 Honor Roll for 16 consecutive years. But still, Penn said it was pulling out due to what it suggested were methodology issues.

“Healthcare is evolving at an unprecedented pace, and the ways performance is measured must also change. The ‘Best Hospitals’ rankings don’t account for all of the elements essential to improving patient outcomes, such as research, innovation, or value-based care,” Kevin B. Mahoney, CEO of the University of Pennsylvania Health System, said of the organization’s decision.

“Transparent metrics are an important tool for health systems to track and strengthen their efforts, but they should measure the full scope of operations dedicated to care delivery.”

Other industry leaders have also questioned hospital quality ratings. In 2019, a group of researchers published a rubric in the New England Journal of Medicine Catalyst to “rate the raters” and get to the bottom of differences in hospital rating systems.

The researchers said each of the rating systems they assessed had some room for improvement. For the record, U.S. News & World Report outranked other types of hospital rating systems.

Harder indicated that the dialogue between U.S. News & World Report and the hospitals it ranks has, in many ways, been productive.

“That also leads to a very fruitful exchange of ideas with healthcare professionals about the methodology,” he explained. “Every year we get memoranda, some of them, we've solicited, some of them unsolicited.”

Still, there is some incongruence in terms of the healthcare industry’s push toward more health equity. Just a few weeks before U.S. News & World Report dropped its 2023-2024 Best Hospitals list, the Lown Institute, a Boston-based think tank, issued its list of the most socially responsible hospitals.

None of the hospitals included in the U.S. News Best Hospitals list from 2022-2023 ranked very high for the Lown Institute. Said otherwise, the hospitals deemed best by U.S. News & World Report do much to deliver on health equity. The Lown Institute list was published before this most recent round of Best Hospital rankings was made public and therefore doesn’t take any updates into account.

Harder said some health equity measures are on their way to the U.S. News rankings, but that it is difficult to measure for such a thing.

“Developing measures that matter is hard work, and it takes time, and it takes a lot of input and collaboration,” he pointed out.

Right now, Harder and his team are publishing data about how representative a hospital’s patient population is of the community in which it’s embedded.

Moving forward, U.S. News & World Report said it plans to keep getting stakeholder feedback about embedding health equity measures into its Best Hospital rankings.

“We've been getting feedback from hospitals as we've been going on that, and we're making some changes to how we're on health equity in part because of the very helpful feedback that we got from healthcare professionals,” Harder concluded.