A fundamental function of the visual system is to encode the building blocks of natural scenes-edges, textures and shapes-that subserve visual tasks such as object recognition and scene understanding. Essential to this process is the formation of abstract representations that generalize from specific instances of visual input. A common view holds that neurons in the early visual system signal conjunctions of image features, but how these produce invariant representations is poorly understood. Here we propose that to generalize over similar images, higher-level visual neurons encode statistical variations that characterize local image regions. We present a model in which neural activity encodes the probability distribution most consistent with a given image. Trained on natural images, the model generalizes by learning a compact set of dictionary elements for image distributions typically encountered in natural scenes. Model neurons show a diverse range of properties observed in cortical cells. These results provide a new functional explanation for nonlinear effects in complex cells and offer insight into coding strategies in primary visual cortex (V1) and higher visual areas.

This paper presents a statistical data-driven method for learning intrinsic structures of impact sounds. The method applies principal and independent component analysis to learn low-dimensional representations that model the distribution of both the time-varying spectral and amplitude structure. As a result, the method is able to decompose sounds into a small number of underlying features that characterize acoustic properties such as ringing, resonance, sustain, decay, and onsets. The method is highly flexible and makes no a priori assumptions about the physics, acoustics, or dynamics of the objects. In addition, by modeling the underlying distribution, the method can capture the natural variability of ensembles of related impact sounds.

We address the problem of robust coding in which the signal information should be preserved in spite of intrinsic noise in the representation. We present a theoretical analysis for one- and two-dimensional cases and characterize the optimal linear encoder and decoder in the mean squared error sense. Our analysis allows for an arbitrary number of coding units, thus including both under- and over-complete representations, and provides insights into optimal coding strategies. In particular, we show how the form of the code adapts to the number of coding units and to different data and noise conditions in order to achieve robustness. We also present numerical solutions of robust coding for high-dimensional image data, demonstrating that these codes are substantially more robust than other linear image coding methods such as PCA, ICA, and wavelets.

The auditory neural code must serve a wide range of auditory tasks that require great sensitivity in time and frequency and be effective over the diverse array of sounds present in natural acoustic environments. It has been suggested that sensory systems might have evolved highly efficient coding strategies to maximize the information conveyed to the brain while minimizing the required energy and neural resources. Here we show that, for natural sounds, the complete acoustic waveform can be represented efficiently with a nonlinear model based on a population spike code. In this model, idealized spikes encode the precise temporal positions and magnitudes of underlying acoustic features. We find that when the features are optimized for coding either natural sounds or speech, they show striking similarities to time-domain cochlear filter estimates, have a frequency-bandwidth dependence similar to that of auditory nerve fibres, and yield significantly greater coding efficiency than conventional signal representations. These results indicate that the auditory code might approach an information theoretic optimum and that the acoustic structure of speech might be adapted to the coding capacity of the mammalian auditory system.

Capturing statistical regularities in complex, high-dimensional data is an important problem in machine learning and signal processing. Models such as PCA and ICA make few assumptions about the structure in the data, have good scaling properties, but are limited to representing linear statistical regularities and assume that the distribution of the data is stationary. For many natural, complex signals, the latent variables often exhibit residual dependencies as well as non-stationary statistics. Here we present a hierarchical Bayesian model that is able to capture higher-order non-linear structure and represent non-stationary data distributions. The model is a generalization of ICA in which the basis function coefficients are no longer assumed to be independent; instead, the dependencies in their magnitudes are captured by a set of density components. Each density component describes a common pattern of deviation from the marginal density of the pattern ensemble; in different combinations, they can describe non-stationary distributions. Adapting the model to image or audio data yields a non-linear, distributed code for higher-order statistical regularities that reflect more abstract, invariant properties of the signal.

Non-stationary acoustic features provide essential cues for many auditory tasks including sound localization, auditory stream analysis, and speech recognition. These features can be best characterized relative to a precise point in time such as the onset of a sound or the beginning of a harmonic periodicity. Extracting these types of features is a difficult problem. Part of the difficulty is that with standard block-based signal analysis methods the representation is sensitive to the arbitrary alignment of the blocks with respect to the signal. Convolutional techniques such as shift-invariant transformations can reduce this sensitivity, but these do not yield a code that is efficient, i.e. one that forms a non-redundant representation of the underlying structure. Here, we develop a non-block based method for signal representation that is both time-relative and efficient. Signals are represented using a linear superposition of time-shiftable kernel functions each with an associated magnitude and temporal position. Signal decomposition in this method is a nonlinear process that consists of optimizing the kernel function scaling coefficients and temporal positions to form an efficient, shift-invariant representation. We demonstrate the properties of this representation for the purpose of characterizing structure in various types of non-stationary acoustic signals. The computational problem investigated here has direct relevance to the neural coding at the auditory nerve and the more general issue of how to encode complex, time-varying signals with a population of spiking neurons.

The theoretical principles that underlie the representation and computation of higher-order structure in natural images are poorly understood. Recently, there has been considerable interest in using information theoretic techniques, such as independent component analysis, to derive representations for natural images that are optimal in the sense of coding efficiency. Although these approaches have been successful in explaining properties of neural representations in the early visual pathway and visual cortex, because they are based on a linear model, the types of image structure that can be represented are very limited. Here, we present a hierarchical probabilistic model for learning higher-order statistical regularities in natural images. This non-linear model learns an efficient code that describes variations in the underlying probabilistic density. When applied to natural images the algorithm yields coarse-coded, sparsedistributed representations of abstract image properties such as object location, scale and texture. This model offers a novel description of higher-order image structure and could provide theoretical insight into the response properties and computational functions of lower level cortical visual areas.

An unsupervised classification algorithm is derived by modeling observed data as a mixture of several mutually exclusive classes that are each described by linear combinations of independent, non-Gaussian densities. The algorithm estimates the data density in each class by using parametric nonlinear functions that fit to the non-Gaussian structure of the data. This improves classification accuracy compared with standard Gaussian mixture models. When applied to images, the algorithm can learn efficient codes (basis functions) for images that capture the statistically significant structure intrinsic in the images. We apply this technique to the problem of unsupervised classification, segmentation, and denoising of images. We demonstrate that this method was effective in classifying complex image textures such as natural scenes and text. It was also useful for denoising and filling in missing pixels in images with complex structures. The advantage of this model is that image codes can be learned with increasing numbers of classes thus providing greater flexibility in modeling structure and in finding more image features than in either Gaussian mixture models or standard independent component analysis (ICA) algorithms.

The auditory system encodes sound by decomposing the amplitude signal arriving at the ear into multiple frequency bands whose center frequencies and bandwidths are approximately exponential functions of the distance from the stapes. This organization is thought to result from the adaptation of cochlear mechanisms to the animal's auditory environment. Here we report that several basic auditory nerve fiber tuning properties can be accounted for by adapting a population of filter shapes to encode natural sounds efficiently. The form of the code depends on sound class, resembling a Fourier transformation when optimized for animal vocalizations and a wavelet transformation when optimized for non-biological environmental sounds. Only for the combined set does the optimal code follow scaling characteristics of physiological data. These results suggest that auditory nerve fibers encode a broad set of natural sounds in a manner consistent with information theoretic principles.

In an overcomplete basis, the number of basis vectors is greater than the dimensionality of the input, and the representation of an input is not a unique combination of basis vectors. Overcomplete representations have been advocated because they have greater robustness in the presence of noise, can be more sparse, and can have greater flexibility in matching structure in the data. Overcomplete codes have also been proposed as a model of some of the response properties of neurons in primary visual cortex. Previous work has focused on finding the best representation of a signal using a fixed overcomplete basis (or dictionary). We present an algorithm for learning an overcomplete basis by viewing it as probabilistic model of the observed data. We show that overcomplete bases can yield a better approximation of the underlying statistical distribution of the data and can thus lead to greater coding efficiency. This can be viewed as a generalization of the technique of independent component analysis and provides a method for identification when there are more sources than mixtures.

An unsupervised classification algorithm is derived from an ICA mixture model assuming that the observed data can be categorized into several mutually exclusive data classes whose components are generated by linear mixtures of independent non-Gaussian sources. The algorithm finds the independent sources, the mixing matrix for each class and also computes the class membership probability for each data point. The new algorithm can improve classification accuracy compared with standard Gaussian mixture models. When applied to blind source separation in nonstationary environments, the method can switch automatically between learned mixing matrices. The algorithm can learn efficient codes to represent images containing both natural scenes and text. This method shows promise for modeling structure in high-dimensional data and has many potential applications.

We apply a Bayesian method for inferring an optimal basis to the problem of finding efficient image codes for natural scenes. The basis functions learned by the algorithm are oriented and localized in both space and frequency, bearing a resemblance to Gabor functions, and increasing the number of basis functions results in a greater sampling density in position, orientation, and scale. These properties also resemble the spatial receptive fields of neurons in the primary visual cortex of mammals, suggesting that the receptive field structure of these neurons can be accounted for by a general efficient coding principle. The probabilistic framework provides a method for comparing the coding efficiency of different bases objectively by calculating their probability given the observed data or by measuring the entropy of the basis function coefficients. The learned bases are shown to have better coding efficiency compared to traditional Fourier and wavelet bases. This framework also provides a Bayesian solution to the problems of image denoising and filling-in of missing pixels. We demonstrate that the results obtained by applying the learned bases to these problems are improved over those obtained with traditional techniques.

Empirical results were obtained for the blind source separation of more sources than mixtures using a recently proposed framework for learning overcomplete representations. This technique assumes a linear mixing model with additive noise and involves two steps: (1) learning an overcomplete representation for the observed data and (2) inferring sources given a sparse prior on the coefficients. We demonstrate that three speech signals can be separated with good fidelity given only two mixtures of the three signals. Similar results were obtained with mixtures of two speech signals and one music signal.

The detection of neural spike activity is a technical challenge that is a prerequisite for studying many types of brain function. Measuring the activity of individual neurons accurately can be difficult due to large amounts of background noise and the difficulty in distinguishing the action potentials of one neuron from those of others in the local area. This article reviews algorithms and methods for detecting and classifying action potentials, a problem commonly referred to as spike sorting. The article first discusses the challenges of measuring neural activity and the basic issues of signal detection and classification. It reviews and illustrates algorithms and techniques that have been applied to many of the problems in spike sorting and discusses the advantages and limitations of each and the applicability of these methods for different types of experimental demands. The article is written both for the physiologist wanting to use simple methods that will improve experimental yield and minimize the selection biases of traditional techniques and for those who want to apply or extend more sophisticated algorithms to meet new experimental challenges.

Auditory neurons in the forebrain nucleus HVc (hyperstriatum ventrale pars caudale) are highly sensitive to the temporal structure of the bird's own song. These ``song-specific'' neurons respond strongly to forward song, weakly to the song with the order of the syllables reversed, and little or not at all to reversed song. To investigate the cellular mechanisms underlying these responses, in vivo intracellular recordings were made from adult HVc neurons. Song-specific cells could be divided into those that responded strongly throughout autogenous song (tonic cells) and those that responded with bursts of action potentials at specific points during the song (phasic cells). Phasic cells were hyperpolarized during autogenous song even though this stimulus also elicited the strongest response. Less hyperpolarization was seen to the same song with the syllables in reverse order, and none was seen to reversed song. The responses of both types of song-specific cells contained high frequency bursts of action potentials. The bursts of the phasic cells showed attenuation of the action potential height and lack of full repolarization following each spike. This type of bursting was significantly correlated with the amount of hyperpolarization prior to each burst in phasic cells and non-auditory cells that generated such bursts spontaneously. These data suggest that song-specific neurons use long-lasting hyperpolarization as a mechanism to integrate auditory context, an important component of temporal order selectivity. Hyperpolarization may also increase the precision of spike timing, which could be important for the neural code subserving song learning and production.

Some of the most complex auditory neurons known are contained in the songbird forebrain nucleus HVc. These neurons are highly sensitive to auditory temporal context: they respond strongly to the bird's own song, but respond weakly or not at all when the sequence of the song syllables is altered. It is not known whether this property arises de novo in HVc or if it is relayed from the properties of neurons in afferent nuclei. To address this issue, we recorded from neurons in both HVc and its afferent nuclei, collectively called field L. Experimental tests were designed to determine the degree of auditory context sensitivity in field L and HVc. Tests were also performed to compare the responses to individual syllables and syllable combinations in order to see if these responses could account for the response seen to the whole song.

Our results show a substantial increase in the auditory temporal context sensitivity between field L and HVc. Most field L neurons respond equally well to both normal song and to temporally manipulated versions of the same song. A few field L neurons show sensitivity to local temporal structure, such as the sequence of syllable pairs. In contrast, HVc neurons are highly dependent upon both the song's local and global temporal structure. This shows that HVc neurons can integrate auditory context over periods much longer than neurons in field L and suggests that additional mechanisms are required to explain the marked sensitivity of HVc neurons to the temporal structure of the bird's own song.

Neurons in the songbird forebrain area HVc are sensitive to the temporal structure of the bird's own song and are capable of integrating auditory information over a period of several hundred milliseconds. Extracellular studies have shown that the responses of some HVc neurons depend on the combination and temporal order of syllables from the bird's own song, but little is known about the mechanisms underlying these response properties. To investigate these mechanisms, we recorded intracellular responses to a set of auditory stimuli designed to assess the degree of dependence of the responses on temporal context. This report provides evidence that HVc neurons encode information about temporal structure using a variety of mechanisms including syllable-specific inhibition, excitatory post-synaptic potentials with a range of different time courses, and burst-firing non-linearity. The data suggest that the sensitivity of HVc neurons to temporal combinations of syllables results from the interactions of several cells and does not arise in a single step from afferent inputs alone.

Identifying and classifying action potential shapes in extracellular neural waveforms has long been the subject of research, and although several algorithms for this purpose have been successfully applied, their use has been limited by some outstanding problems. The first is how to determine shapes of the action potentials in the waveform and, second, how to decide how many shapes are distinct. A harder problem is that action potentials frequently overlap making difficult both the determination of the shapes and the classification of the spikes. In this report, a solution to each of these problems is obtained by applying Bayesian probability theory. By defining a probabilistic model of the waveform, the probability of both the form and number of spike shapes can be quantified. In addition, this framework is used to obtain an efficient algorithm for the decomposition of arbitrarily complex overlap sequences. This algorithm can extract many times more information than previous methods and facilitates the extracellular investigation of neuronal classes and of interactions within neuronal circuits.

Efficient coding models predict that the optimal code for natural images is a population of oriented Gabor receptive fields. These results match response properties of neurons in primary visual cortex, but not those in the retina. Does the retina use an optimal code, and if so, what is it optimized for? Previous theories of retinal coding have assumed that the goal is to encode the maximal amount of information about the sensory signal. However, the image sampled by retinal photoreceptors is degraded both by the optics of the eye and by the photoreceptor noise. Therefore, de-blurring and de-noising of the retinal signal should be important aspects of retinal coding. Furthermore, the ideal retinal code should be robust to neural noise and make optimal use of all available neurons. Here we present a theoretical framework to derive codes that simultaneously satisfy all of these desiderata. When optimized for natural images, the model yields filters that show strong similarities to retinal ganglion cell (RGC) receptive fields. Importantly, the characteristics of receptive fields vary with retinal eccentricities where the optical blur and the number of RGCs are significantly different. The proposed model provides a unified account of retinal coding, and more generally, it may be viewed as an extension of the Wiener filter with an arbitrary number of noisy units.

The visual system must encode a visual stimulus reliably in spite of several sources of noise and biological constraints. First, the neural code has limited precision (only 1~3 bits/spike). Second, photoreceptors do not output a clean image signal, but one that has been distorted by optical blur and noise in photon transduction. Finally, the biological limitations on synaptic strength and dendritic field size also constrain the neural code. Here we address the problem of how a noisy neural population represents the underlying, undistorted image signal reliably subject to the limited neural resources. We derived theoretically optimal filters (receptive fields) by training on a large set of natural images to which the visual system is assumed to be adapted. This yielded a wide range of theoretical predictions for the retinal ganglion cells. The optimal filters show center-surround organization, whose characteristic changes from band-pass to low-pass as sensory noise increases. Accordingly, the receptive fields become larger and more overlapped with each other (Fig. 1). Our model can be applied to any retinal eccentricity and predicts different characteristics. Fig. 1 shows the results for the central fovea, while Fig. 2 for the 40 degrees retinal eccentricity as in some physiological experiments (Frechette et al., 2005). The proposed model is more comprehensive than preceding ones (Srinivasan, Laughlin and Dubs, 1982; Atick, 1992; Vincent and Baddeley, 2003) because it more accurately incorporates sources of noise and biological constraints, and predicts the theoretically optimal retinal population codes for these conditions.

Smith and Lewicki (2005a, 2005b, 2006) demonstrated that mammalian hearing follows an efficient coding principle (Barlow, 1961; Atick, 1992; Simoncelli & Olshausen, 2001; Laughlin & Sejnowski, 2003). Auditory neurons efficiently code for natural sounds in the environment, maximizing information rate while minimizing coding cost (Shannon, 1948). Applying the same analysis to speech coding suggests that speech acoustics are optimally adapted to the mammalian auditory code (Smith and Lewicki, 2005b, 2006). The present work applies this efficient coding theory to the problem of speech perception in individuals using cochlear implants (CI), for which there exist vast individual differences in speech perception and spectral resolution (Zeng et al., 2004). We present a machine-learning method for CI filterbank design based on the efficient-coding hypothesis. Further, we describe a pair of experiments which evaluate this approach using noise-excited vocoder speech (Shannon et al., 1995). Participants' recognition of continuous speech and isolated syllables is significantly more accurate for speech filtered through the theoretically-motivated efficient-coding filterbank relative to the standard cochleotopic filterbank, particularly for speech transients. These findings offer insight in CI design and provide behavioral evidence for efficient coding in human perception.

Cochlear implants are neuroprosthetic devices that use direct, electrical stimulation of auditory nerve fibers along the tonotopic axis of the cochlea to restore some degree of hearing to individuals with profound peripheral hearing loss (Zeng et al., 2004). Despite twenty years of research and wide clinical application, speech perception in cochlear implant users is highly variable and often quite degraded. Although a variety of reasons have been proposed for the poor performance of cochlear implant users, one clear factor is their limited spectral resolution. Normal human hearing has a rich spectral representation of the auditory world (30,000 spiral ganglion cells lie along the tonotopic axis of the cochlea), but cochlear implants carry very few frequency channels, often fewer than eight. The design of the filterbank, which determines the content of these channels, has a potent effect on performance. Here we present a theoretically motivated method for filterbanks design and a pair of experiments demonstrating a substantial increase in speech intelligibility when using these filters compared with standard filters. These finding represent some of the first behavioral evidence of efficient coding in human perception.

Biological sensory systems are faced with the problem of encoding a high-fidelity sensory signal with a population of noisy, low-fidelity neurons. This problem can be expressed in information theoretic terms as coding and transmitting a multi-dimensional, analog signal over a set of noisy channels. Previously, we have shown that robust, overcomplete codes can be learned by minimizing the reconstruction error with a constraint on the channel capacity. Here, we present a theoretical analysis that characterizes the optimal linear coder and decoder for one- and two-dimensional data. The analysis allows for an arbitrary number of coding units, thus including both under- and over-complete representations, and provides a number of important insights into optimal coding strategies. In particular, we show how the form of the code adapts to the number of coding units and to different data and noise conditions to achieve robustness. We also report numerical solutions for robust coding of high-dimensional image data and show that these codes are substantially more robust compared against other image codes such as ICA and wavelets.

Linear implementations of the efficient coding hypothesis, such as independent component analysis (ICA) and sparse coding models, have provided functional explanations for properties of simple cells in V1. These models, however, ignore the non-linear behavior of neurons and fail to match individual and population properties of neural receptive fields in subtle but important ways. Hierarchical models, including Gaussian Scale Mixtures and other generative statistical models, can capture higher-order regularities in natural images and explain non-linear aspects of neural processing such as normalization and context effects. Previously, it had been assumed that the lower level representation is independent of the hierarchy, and had been fixed when training these models. Here we examine the optimal lower-level representations derived in the context of a hierarchical model and find that the resulting representations are strikingly different from those based on linear models. Unlike the the basis functions and filters learned by ICA or sparse coding, these functions individually more closely resemble simple cell receptive fields and collectively span a broad range of spatial scales. Our work unifies several related approaches and observations about natural image structure and suggests that hierarchical models might yield better representations of image structure throughout the hierarchy.

Natural images are not random; instead, they exhibit statistical regularities. Assuming that our vision is designed for tasks on natural images, computation in the visual system should be optimized for such regularities. Recent theoretical investigations along this line have provided many insights into the visual response properties in the early visual system. In this article we review both the known statistical regularities of natural images, the extent to which low-level vision might be adapted to them, and the recent development in theoretical models to explain this relationship.

It has been suggested that the primary goal of the sensory system is to represent input in such a way as to reduce the high degree of redundancy. Given a noisy neural representation, however, solely reducing redundancy is not desirable, since redundancy is the only clue to reduce the effects of noise. Here we propose a model that best balances redundancy reduction and redundant representation. Like previous models, our model accounts for the localized and oriented structure of simple cells, but it also predicts a different organization for the population. With noisy, limited-capacity units, the optimal representation becomes an overcomplete, multi-scale representation, which, compared to previous models, is in closer agreement with physiological data. These results offer a new perspective on the expansion of the number of neurons from retina to V1 and provide a theoretical model of incorporating useful redundancy into efficient neural representations.

The representation of acoustic signals at the cochlear nerve must serve a wide range of auditory tasks that require exquisite sensitivity in both time and frequency. Lewicki (2002) demonstrated that many of the filtering properties of the cochlea could be explained in terms of efficient coding of natural sounds. This model, however, did not account for properties such as phase-locking or how sound could be encoded in terms of action potentials. Here, we extend this theoretical approach with algorithm for learning efficient auditory codes using a spiking population code. Here, we propose an algorithm for learning efficient auditory codes using a theoretical model for coding sound in terms of spikes. In this model, each spike encodes the precise time position and magnitude of a localized, time varying kernel function. By adapting the kernel functions to the statistics natural sounds, we show that, compared to conventional signal representations, the spike code achieves far greater coding efficiency. Furthermore, the inferred kernels show both striking similarities to measured cochlear filters and a similar bandwidth versus frequency dependence.

We present a hierarchical Bayesian model for learning efficient codes of higher-order structure in natural images. The model, a non-linear generalization of independent component analysis, replaces the standard assumption of independence for the joint distribution of coefficients with a distribution that is adapted to the variance structure of the coefficients of an efficient image basis. This offers a novel description of higherorder image structure and provides a way to learn coarse-coded, sparsedistributed representations of abstract image properties such as object location, scale, and texture.

Multilayer architectures such as those used in Bayesian belief networks and Helmholtz machines provide a powerful framework for representing and learning higher order statistical relations among inputs. Because exact probability calculations with these models are often intractable, there is much interest in finding approximate algorithms. We present an algorithm that efficiently discovers higher order structure using EM and Gibbs sampling. The model can be interpreted as a stochastic recurrent network in which ambiguity in lower-level states is resolved through feedback from higher levels. We demonstrate the performance of the algorithm on benchmark problems.

Exponential power distributions (Box and Tiao, 1973) provide a general method for modeling non-Gaussian statistical structure of univariate distributions that have the form p(x) ~ exp(-|x|^q). By inferring q, a wide class of statisical distributions can be characterized including uniform, Gaussian, Laplacian, and other so-called sub- and super-Gaussian densities. Using this distribution in independent component analysis (ICA), we show that the exponential power distribution can be used to infer the optimal degree of non-Gaussian statistical structure for multivariate densities. We also show that this class of distributions provides a near exact fit to the source distributions of natural images and demonstrate that this leads to better estimated coding efficiency.

Multilayer architectures such as those used in Bayesian belief networks and Helmholtz machines provide a framework for representing and learning higher order statistical relations among nputs Because exact probability calculations with these models are often intractable there is much interest in finding approximate algorithms. We present an algorithm that efficiently discovers higher order structure using an EM algorithm and approximate Gibbs sampling. The model can be interpreted as a stochastic recurrent network in which ambiguity in lower level units is resolved through feedback from higher levels. Recognition is viewed as an inference process. Unlike networks that compute the internal state using feedforward computations feed back allows the network to find the most probable internal state for each input pattern. We demonstrate the performance of the algorithm on benchmark problems.

An unsupervised algorithm is proposed for learning distributed representations of stochastic binary features. The representation is also hierarchical and higher level units encode higher order combinations of lower level features.