2018 Conference on Cognitive Computational Neuroscience

5-8 September 2018    Philadelphia, Pennsylvania

Technical Program

Paper Detail

Paper: PS-1B.34
Session: Poster Session 1B
Location: Symphony/Overture
Session Time: Thursday, September 6, 18:45 - 20:45
Presentation Time:Thursday, September 6, 18:45 - 20:45
Presentation: Poster
Publication: 2018 Conference on Cognitive Computational Neuroscience, 5-8 September 2018, Philadelphia, Pennsylvania
Paper Title: Short-term Sequence Memory: Compressive effects of Recurrent Network Dynamics
Manuscript:  Click here to view manuscript
DOI: https://doi.org/10.32470/CCN.2018.1086-0
Authors: Adam Charles, Princeton University, United States; Han Lun Yap, DSO National Laboratories, Singapore; Dong Yin, University of California, Berkeley, United States; Christopher Rozell, Georgia Institute of Technology, United States
Abstract: Neural networks have become a ubiquitous as cognitive models in neuroscience and as machine learning systems. Deep neural networks in particular are achieving near-human performance in many applications. More recently, recurrent neural networks (RNNs) are being increasingly utilized, both as stand-alone structures and as layers of deep networks. RNNs are especially interesting as cortical networks are recurrent, indicating that recurrent connections are important in human-level processing. Despite their growing use, theory on the computational properties of RNNs is sparse. As many applications hinge on RNNs accumulating information dynamically, the ability of RNNs to iteratively compress information into the network is particularly critical. We thus present here non-asymptotic bounds on the network’s short-term memory (STM; the number of inputs that can be compressed into and recovered from a network state). Previous bounds on a random RNN’s STM limit the number of recoverable inputs by the number of network nodes. We show that when the input sequences are sparse in a basis or the matrix inputs is low-rank, the number of network nodes needed to achieve an STM grows as the overall information rate. Thus RNNs can efficiently store information embedded in longer input streams, shedding light on their computational capabilities.