New AI tool helps scientists see how cells work together inside diseased tissue

Doctors and scientists have long relied on microscopes to study human tissue and diagnose disease. But today’s medical research produces far more information than the human eye alone can handle, including detailed maps of genes and proteins inside cells.

A new study from Yale University researchers shows how artificial intelligence can bring these different kinds of data together, offering a clearer picture of what is happening inside the body and how diseases develop. The study is published in Nature Biomedical Engineering.

In the study, researchers introduce a new computer system called spEMO, short for “spatial multi-modal embeddings.” The system uses artificial intelligence to combine images of tissue slides with information about gene and protein activity, allowing scientists to analyze biological data in a more complete and meaningful way.

“Each type of data tells part of the story, but on its own it’s incomplete,” said Tianyu Liu, the study’s lead author and a PhD candidate working in the field of computational biology and biomedical informatics at Yale. “Our goal was to design a method that could integrate all of these signals so we can better understand how cells behave in real tissue.”

Traditionally, pathologists examine stained tissue samples under a microscope to identify signs of disease. At the same time, modern technologies can measure which genes are turned on or off at precise locations in those tissues. The challenge is that these different data types—images, gene activity, protein levels, and text-based biological knowledge—are difficult to analyze together.

The spEMO system tackles this problem by using powerful “Pathology Foundation Models,” a type of AI trained on massive datasets. Some models specialize in reading images, while others understand language or biological descriptions. spEMO combines what these models learn into a shared framework that can be used to answer important biological and medical questions.

Using spEMO, the researchers showed they could more accurately identify distinct regions within tissues, predict disease states from tissue samples, uncover communication patterns between cells, and even help generate draft medical reports. In many tests, the system performed better than methods that relied on only one type of data.

“This approach moves us closer to a more holistic view of disease,” said Dr. Hongyu Zhao, PhD, the study’s senior author and a professor of biostatistics at the Yale School of Public Health. “By bringing together molecular data and tissue structure, we can gain insights that would otherwise be missed.”

The researchers provided a compelling example of spEMO’s capabilities using cancer research. By combining tissue images with predicted gene activity, the system identified potential interactions between immune cells inside tumors. These biological signals could help researchers better understand how cancers grow or respond to treatment.

The researchers found that spEMO also could generate detailed medical reports that included both visual and genetic information. When evaluated by expert pathologists, these AI-generated reports were determined to be more complete and accurate than reports using images alone.

While the system is still being tested and refined, the authors say it could eventually help speed up research, support doctors in making diagnoses, and improve personalized medicine.

“spEMO isn’t meant to replace doctors or scientists,” Liu said. “It’s a tool that helps them see patterns and connections that may otherwise be too complex to immediately spot.”

Researchers with the Department of Pathology at Yale; Broad Institute of MIT and Harvard University; the Department of Computer Science at Yale and Harvard; and the Department of Biomedical Data Science at Stanford University contributed to this study.