| Paper: | PS-2B.13 |
| Session: | Poster Session 2B |
| Location: | Symphony/Overture |
| Session Time: | Friday, September 7, 19:30 - 21:30 |
| Presentation Time: | Friday, September 7, 19:30 - 21:30 |
| Presentation: |
Poster
|
| Publication: |
2018 Conference on Cognitive Computational Neuroscience, 5-8 September 2018, Philadelphia, Pennsylvania |
| Paper Title: |
Reconstructing faces from fMRI patterns using Generative Adversarial Networks |
| Manuscript: |
Click here to view manuscript |
| DOI: |
https://doi.org/10.32470/CCN.2018.1031-0 |
| Authors: |
Rufin VanRullen, Leila Reddy, CNRS, France |
| Abstract: |
Recent years have seen steady improvements in our ability to read out sensory inputs from fMRI brain response patterns –so-called “fMRI mind-reading”. While visually distinct inputs, such as objects from different categories, can be accurately decoded and partially reconstructed from fMRI patterns, it has proved more difficult to distinguish visually similar inputs, such as different instances of the same category. Here, we apply a recently developed deep learning system to the reconstruction of face images from human fMRI patterns. We trained a variational auto-encoder (VAE) neural network using a GAN (Generative Adversarial Network) unsupervised training procedure over a dataset of > 200K celebrity faces. The auto-encoder latent space provided a meaningful (topologically organized) 1024-dimensional description of each image. We then presented > 4000 face images to a human subject in the scanner, and learned a simple linear mapping between the multi-voxel fMRI activation patterns and the 1024 latent dimensions. Then we applied this mapping to novel test images, turning the obtained fMRI patterns into VAE latent codes, and ultimately the codes into face reconstructions. Qualitative and quantitative evaluation of the reconstructions reveal robust pairwise decoding (>90% correct), and a strong improvement relative to a baseline model relying on PCA decomposition. |
