Scientists use brain imaging to reveal the movies in our mind

UC Berkeley researchers decode and reconstruct dynamic visual experiences, in this case Hollywood movie trailers

 

Berkeley, CA, USA (September 22, 2011) – Imagine tapping into the mind of a coma patient, or watching one’s own dream on YouTube. With a cutting-edge blend of brain imaging and computer simulation, scientists at the University of California, Berkeley, are bringing these futuristic scenarios within reach.

 

Using functional Magnetic Resonance Imaging (fMRI) and computational models, UC Berkeley researchers have succeeded in decoding and reconstructing people’s dynamic visual experiences – in this case, watching Hollywood movie trailers.

 

As yet, the technology can only reconstruct movie clips people have already viewed. However, the breakthrough paves the way for reproducing the movies inside our heads that no one else sees, such as dreams and memories, according to researchers.

 

 

"This is a major leap toward reconstructing internal imagery," said Professor Jack Gallant, a UC Berkeley neuroscientist and coauthor of the study to be published online Sept. 22 in the journal Current Biology. "We are opening a window into the movies in our minds."

 

Eventually, practical applications of the technology could include a better understanding of what goes on in the minds of people who cannot communicate verbally, such as stroke victims, coma patients and people with neurodegenerative diseases.

 

It may also lay the groundwork for brain-machine interface so that people with cerebral palsy or paralysis, for example, can guide computers with their minds.

 

However, researchers point out that the technology is decades from allowing users to read others’ thoughts and intentions, as portrayed in such sci-fi classics as "Brainstorm," in which scientists recorded a person’s sensations so that others could experience them.

 

Previously, Gallant and fellow researchers recorded brain activity in the visual cortex while a subject viewed black-and-white photographs. They then built a computational model that enabled them to predict with overwhelming accuracy which picture the subject was looking at.

 

In their latest experiment, researchers say they have solved a much more difficult problem by actually decoding brain signals generated by moving pictures.

 

"Our natural visual experience is like watching a movie," said Shinji Nishimoto, lead author of the study and a post-doctoral researcher in Gallant’s lab. "In order for this technology to have wide applicability, we must understand how the brain processes these dynamic visual experiences."

 

Nishimoto and two other research team members served as subjects for the experiment, because the procedure requires volunteers to remain still inside the MRI scanner for hours at a time.

 

They watched two separate sets of Hollywood movie trailers, while fMRI was used to measure blood flow through the visual cortex, the part of the brain that processes visual information. On the computer, the brain was divided into small, three-dimensional cubes known as volumetric pixels, or "voxels."

 

"We built a model for each voxel that describes how shape and motion information in the movie is mapped into brain activity," Nishimoto said.

 

The brain activity recorded while subjects viewed the first set of clips was fed into a computer program that learned, second by second, to associate visual patterns in the movie with the corresponding brain activity.

 

Brain activity evoked by the second set of clips was used to test the movie reconstruction algorithm. This was done by feeding 18 million seconds of random YouTube videos into the computer program so that it could predict the brain activity that each film clip would most likely evoke in each subject.

 

Finally, the 100 clips that the computer program decided were most similar to the clip that the subject had probably seen were merged to produce a blurry yet continuous reconstruction of the original movie.

 

Reconstructing movies using brain scans has been challenging because the blood flow signals measured using fMRI change much more slowly than the neural signals that encode dynamic information in movies, researchers said. For this reason, most previous attempts to decode brain activity have focused on static images.

 

"We addressed this problem by developing a two-stage model that separately describes the underlying neural population and blood flow signals," Nishimoto said.

 

Ultimately, Nishimoto said, scientists need to understand how the brain processes dynamic visual events that we experience in everyday life.

 

"We need to know how the brain works in naturalistic conditions," he said. "For that, we need to first understand how the brain works while we are watching movies."

 

 

Other coauthors of the study are Thomas Naselaris with UC Berkeley’s Helen Wills Neuroscience Institute; An T. Vu with UC Berkeley’s Joint Graduate Group in Bioengineering; and Yuval Benjamini and Professor Bin Yu with the UC Berkeley Department of Statistics.

 

 


University of California Berkeley, 22.09.2011 (tB).

MEDICAL NEWS

Inadequate sequencing of SARS-CoV-2 variants impedes global response to COVID-19
New meta-analysis finds cannabis may be linked to development of…
New guidance on how to diagnosis and manage osteoporosis in…
Starting the day off with chocolate could have unexpected benefits
Better mental health supports for nurses needed, study finds

SCHMERZ PAINCARE

Versorgung verbessern: Deutsche Gesellschaft für Schmerzmedizin fordert die Einführung des…
Pflegeexpertise im Fokus: Schmerzmanagement nach Operationen
Versorgung verbessern: Bundesweite Initiative der Deutschen Gesellschaft für Schmerzmedizin zu…
Jedes vierte Kind wünscht bessere Schmerzbehandlung
Lebensqualität von Patienten in der dauerhaften Schmerztherapie mit Opioiden verbessern

DIABETES

Bundestag berät über DMP Adipositas: DDG begrüßt dies als Teil…
Mit der Smartwatch Insulinbildung steuern
Verbände fordern bessere Ausbildung und Honorierung von Pflegekräften für Menschen…
Minimalinvasive Geräte warnen ungenügend vor Unterzuckerung
Typ-1-Diabetes und Hashimoto-Thyreoiditis treten häufig gemeinsam auf

ERNÄHRUNG

Wie eine Diät die Darmflora beeinflusst: Krankenhauskeim spielt wichtige Rolle…
DGEM plädiert für Screening und frühzeitige Aufbautherapie: Stationäre COVID-19-Patienten oft…
Führt eine vegane Ernährungsweise zu einer geringeren Knochengesundheit?
Regelmässiger Koffeinkonsum verändert Hirnstrukturen
Corona-Erkrankung: Fehl- und Mangelernährung sind unterschätze Risikofaktoren

ONKOLOGIE

Anti-Myelom-Therapie mit zusätzlich Daratumumab noch effektiver
Positive Ergebnisse beim fortgeschrittenen Prostatakarzinom: Phase-III-Studie zur Radioligandentherapie mit 177Lu-PSMA-617
Lymphom-News vom EHA2021 Virtual. Alle Berichte sind nun online verfügbar!
Deutsch-dänisches Interreg-Projekt: Grenzübergreifende Fortbildungskurse in der onkologischen Pflege
Sotorasib: Neues Medikament macht Lungenkrebs-Patienten Hoffnung

MULTIPLE SKLEROSE

NMOSD-Erkrankungen: Zulassung von Satralizumab zur Behandlung von Jugendlichen und Erwachsenen
Verzögerte Verfügbarkeit von Ofatumumab (Kesimpta®)
Neuer Biomarker bei Multipler Sklerose ermöglicht frühe Risikoeinschätzung und gezielte…
Multiple Sklerose beginnt oft lange vor der Diagnose
Goldstandard für Versorgung bei Multipler Sklerose

PARKINSON

Meilenstein in der Parkinson-Frühdiagnose
Parkinson-Erkrankte besonders stark von Covid-19 betroffen
Gangstörungen durch Kleinhirnschädigung beim atypischen Parkinson-Syndrom
Parkinson-Agenda 2030: Die kommenden 10 Jahre sind für die therapeutische…
Gemeinsam gegen Parkinson: bessere Therapie durch multidisziplinäre Versorgung