 Research
 Open access
 Published:
Sparse identification of contrast gain control in the fruit fly photoreceptor and amacrine cell layer
The Journal of Mathematical Neuroscience volume 10, Article number: 3 (2020)
Abstract
The fruit fly’s natural visual environment is often characterized by light intensities ranging across several orders of magnitude and by rapidly varying contrast across space and time. Fruit fly photoreceptors robustly transduce and, in conjunction with amacrine cells, process visual scenes and provide the resulting signal to downstream targets. Here, we model the first step of visual processing in the photoreceptoramacrine cell layer. We propose a novel divisive normalization processor (DNP) for modeling the computation taking place in the photoreceptoramacrine cell layer. The DNP explicitly models the photoreceptor feedforward and temporal feedback processing paths and the spatiotemporal feedback path of the amacrine cells. We then formally characterize the contrast gain control of the DNP and provide sparse identification algorithms that can efficiently identify each the feedforward and feedback DNP components. The algorithms presented here are the first demonstration of tractable and robust identification of the components of a divisive normalization processor. The sparse identification algorithms can be readily employed in experimental settings, and their effectiveness is demonstrated with several examples.
1 Introduction
Sensory processing systems in the brain extract relevant information from stimuli whose amplitude can vary orders of magnitude [1–4]. Consequently, at each layer of processing, starting right from sensory transduction, neurons need to map their output into a range that can be effectively processed by subsequent neural circuits. As an example, photoreceptors [5–8] and olfactory receptor neurons [9, 10] in both vertebrates and invertebrates respectively adapt to a large range of intensity/temporal contrast values of visual and odorant stimuli. Adaptation to mean and variance of the stimuli has been observed in the auditory system [11, 12] as well. Further down the visual pathway, motion sensitive neurons in vertebrates and invertebrates have been shown to be robust at various brightness and contrast levels [13–15].
Early visual circuits, such as the photoreceptor/amacrine cell layer of the fruit fly brain, are believed to perform spatiotemporal intensity and contrast gain control for dynamic adaptation to visual stimuli whose intensity and contrast levels vary orders of magnitude both in space and time. The mechanism underlying temporal and spatiotemporal contrast gain control in the vertebrate retina [16–20] has often been characterized as a change of the receptive fields in response to changing stimulus statistics [21]. However, while linear receptive fields can be estimated at each transition between different gain values, characterizing the full dynamics of contrast gain control received little attention. Current theoretical methods for describing spatiotemporal gain control lack a systematic framework for characterizing its dynamics, and identification algorithms to estimate circuit components are generally not available.
One exception is the use of theory of dynamical systems for modeling nonlinear gain control. Examples include the Volterra series expansion [22, 23] and the nonlinear autoregressive moving average model with exogenous inputs (NARMAX) [24, 25]. Both formalisms exhibit extensive rigorous tools of functional identification. However, the former typically requires highorder Volterra kernels to model the highly nonlinear gain control, and its identification suffers from the curse of dimensionality. In practice, using the second or thirdorder Volterra kernels is computationally not tractable with commodity hardware, while at the same time not fully capturing the dynamics of gain control. For the latter, an extension to the spatiotemporal domain is often out of reach.
Divisive normalization provides an alternative nonlinear operator to model gain control. Divisive normalization [26] has been proposed as a canonical circuit model of computation for many sensory processing circuits underlying adaptation and attention [4, 27–31]. Divisive normalization models in sensory systems are often associated with a population of neurons where each receives inputs from the pool [26, 32]. Recent studies have shown that divisive normalization is a suitable candidate for describing the contrast gain control and its dynamics [33].
Existing modeling studies of divisive normalization in sensory systems mostly focus on establishing a connection between gain control and the statistical properties of natural sensory stimuli [28, 34–36]. There is a lack of general mathematical framework for identifying divisive normalization circuits from recorded data.
Early models of divisive normalization [27] and their derivatives [26, 28, 29, 37–40] only consider feedforward divisive normalization circuits. The modeled neural circuits often exhibit, however, extensive feedback circuits [22, 41].
In this paper, we address the above issues by modeling the photoreceptor/amacrine cells layer of the fruit fly as a multiinput multioutput (MIMO) feedforward and feedback temporal and spatiotemporal divisive normalization processor (DNP). The MIMO DNPs are built upon temporal and spatiotemporal feedforward and feedback of divisive normalization operators constructed from lowdimensional Volterra operators. Combining with sparse identification methods [42], we provide efficient algorithms for identifying all the components of the temporal as well as spatiotemporal divisive normalization processors.
This manuscript is organized as follows. In Sect. 2 the overall architecture of the divisive normalization processor (DNP) is introduced and its power of modeling contrast gain control is demonstrated. We first describe in Sect. 2.1 the biological model of photoreceptoramacrine cell layer. In Sect. 2.2, we introduce a general model for divisive normalization in the time domain. The temporal DNP consists of the ratio of two nonlinear functionals acting on the input stimulus. In Sect. 2.3 we then extend the model to spacetime domain to include models of lateral feedback from amacrine cells and demonstrate its processing power. In Sects. 3 and 4, we provide identification algorithms and show that the temporal and spatiotemporal DNPs can be efficiently identified. We demonstrate the effectiveness of the algorithms with several examples. We conclude the paper with a discussion in Sect. 5.
2 The architecture of divisive normalization processors
In Sect. 2.1 we start by motivating the present work. We then introduce the architecture of divisive normalization processors in the time domain (Sect. 2.2) and spacetime domain (Sect. 2.3). Finally, in Appendix 1 we provide examples that characterize the I/O mapping of the class of temporal and spatiotemporal divisive normalization processors previously described.
2.1 Modeling the photoreceptors and amacrine cells layer
In what follows, we anchor the model description around the photoreceptoramacrine cell layer of the fruit fly. The fly retina consists of ∼800 ommatidia, each of which hosts photoreceptors whose axons terminate in a secondary neuropil called lamina. There, they provide inputs to columnar large monopolar cells (LMCs) that project to the third visual neuropil, and to amacrine cells [43]. Amacrine cells are interneurons that innervate axon terminals of multiple photoreceptors. The photoreceptors, in turn, receive lateral feedback from the amacrine cells as well as feedback from LMCs such as L2 neurons [41, 44, 45].
A circuit diagram of the photoreceptoramacrine cell layer is shown in Fig. 1. For the sake of clarity, we assume here that an ommatidium consists of a single photoreceptor. It has been shown that the outputs of photoreceptors exhibit rapid gain control through both the phototransduction process and the interaction with such feedback loops [25, 46–49].
In what follows we propose a model comprising nonlinear transformations combined with divisive normalization that can model such gain control and can account for diverse dynamics. We will also show that the model we propose can be systematically identified from observed input output pairs.
2.2 Divisive normalization processors in the time domain
In this section we present the modeling of temporal stimuli in 2.2.1 and introduce a class of temporal divisive normalization processors for modeling photoreceptors in Sect. 2.2.2.
2.2.1 Modeling temporal stimuli
We model the temporal varying stimuli \(u_{1}=u_{1}(t)\), \(t \in \mathbb{D} \subseteq {\mathbb {R}}\), to be realvalued elements of the space of trigonometric polynomials [50]. The choice of the space of the trigonometric polynomials has, as we will see, substantial computational advantages. A temporal stimulus models the visual field arising at the input of a single photoreceptor.
Definition 1
The space of trigonometric polynomials \({\mathcal {H}}_{1}\) is the Hilbert space of complexvalued functions
over the domain \(\mathbb{D} = [0, S]\), where
Here, Ω denotes the bandwidth and L the order of the space. Stimuli \(u_{1}\in {\mathcal {H}}_{1}\) are extended to be periodic over \({\mathbb {R}}\) with period \(S = \frac{2\pi L}{\varOmega }\).
\({\mathcal {H}}_{1}\) is a reproducing kernel Hilbert space (RKHS) [51] with reproducing kernel (RK)
RKHSs have been previously employed for modeling neural encoding of temporal, auditory, and visual stimuli [52–55].
We denote the dimension of \({\mathcal {H}}_{1}\) by \(\dim ({\mathcal {H}}_{1})\) and \(\dim ({\mathcal {H}}_{1}) = 2L+1\).
Definition 2
The tensor product space \({\mathcal {H}}_{2}={\mathcal {H}}_{1} \otimes {\mathcal {H}}_{1}\) is an RKHS with reproducing kernel
Note that \(\dim ({\mathcal {H}}_{2}) = \dim ({\mathcal {H}}_{1})^{2} = (2 L +1)^{2}\).
2.2.2 Temporal divisive normalization processors
We first consider single photoreceptors without feedback from the amacrine cells shown in Fig. 1. A schematic of the temporal divisive normalization processor (DNP) modeling the photoreceptor is shown in Fig. 2. For notational simplicity, we consider a single photoreceptor here. The input visual stimulus to the photoreceptor is denoted by \(u=u(t), t\in \mathbb{D}\), and the output electric current by \(v=v(t), t\in \mathbb{D}\).
Remark 1
Note that, in a single photoreceptor, photons are first absorbed by a large number of microvilli [47] (not shown). Microvilli generate “quantum bumps” in response to photons; the photoreceptor aggregates the bumps and in the process creates the transduction current. Calcium ion influx and calcium diffusion into the photoreceptor cell body may change the sensitivity of the transduction cascade. A high concentration of calcium (buffer) can result in a photon to be ineffective and may also affect the duration and the magnitude of quantum bumps [56].
The DNP consists of (1) a feedforward Volterra processor (VP) \(\mathcal{T}^{1}\), (2) a feedforward normalization VP \(\mathcal{T}^{2}\), and (3) a feedback normalization VP \(\mathcal{T}^{3}\). The output of the photoreceptor amounts to
where
and
Here, \(b^{l}\), \(l=1,2,3\), are the zerothorder Volterra kernels (constants), \(h_{1}^{l}(t)\), \(l=1,2,3\), are firstorder Volterra kernels (impulse responses of linear filters), and \(h_{2}^{l}(t,s)\), \(l=1,2,3\), are secondorder Volterra kernels. As before, \(\mathbb{D}\) denotes the domain of the input space, and \(\mathbb{D}^{2}=\mathbb{D}\times \mathbb{D}\).
Remark 2
For the sake of tractability, we limit each nonlinear functional in (6) and (7) to be composed of only first and secondorder Volterra kernels. The division in (5) allows us, however, to model nonlinear processing of much higher orders.
We note that v in (5) is invariant under scaling by the same factor of the numerator and denominator. Hence, without loss of generality, we will assume \(b^{2}+b^{3}=1\). We will also assume that the DNP is boundedinput boundedoutput [57].
2.2.3 Modeling temporal DNP feedback filters
Here, we define the filter kernels in equations (6) and (7).
Definition 3
Let \(h^{l}_{p} \in \mathbb{L}^{1}(\mathbb{D}^{p})\), \(l=1,2\), \(p=1,2\), where \(\mathbb{L}^{1}\) denotes the space of Lebesgue integrable functions. The operator \(\mathcal{P}_{1}: \mathbb{L}_{1}(\mathbb{D}) \rightarrow {\mathcal {H}}_{1}\) given by
is called the projection operator from \(\mathbb{L}^{1}(\mathbb{D})\) to \({\mathcal {H}}_{1}\). Similarly, the operator \(\mathcal{P}_{2}: \mathbb{L}_{1}(\mathbb{D}^{2}) \rightarrow {\mathcal {H}}_{2}\) given by
is called the projection operator from \(\mathbb{L}^{1}(\mathbb{D}^{2})\) to \({\mathcal {H}}_{2}\).
Note that, for \(u^{n} \in {\mathcal {H}}_{1}, \mathcal{P}_{1} u^{n} = u^{n}\). Moreover, with \(u_{2}^{n}(t_{1},t_{2}) = u^{n}(t_{1})u^{n}(t_{2})\), \(\mathcal{P}_{2} u_{2}^{n} = u_{2}^{n}\). Thus,
and by assuming that \(h^{l}_{1} \in {\mathcal {H}}_{1}\), \(l=1,2\), and \(h^{l}_{2} \in {\mathcal {H}}_{2}\) we recover the simple form of equation (6).
We model the output waveforms \(v^{o} = v^{o} (t)\), \(t \in \mathbb{D} \subseteq {\mathbb {R}}\), to be realvalued elements of the space of trigonometric polynomials [50].
Definition 4
The space of trigonometric polynomials \({\mathcal {H}}_{1}^{o}\) is the Hilbert space of complexvalued functions
over the domain \(\mathbb{D} = [0, S]\), where
Here, \(\varOmega ^{o}\) denotes the bandwidth and \(L^{o}\) the order of the space. The output waveforms \(v^{o}\in {\mathcal {H}}_{1}^{o}\) are extended to be periodic over \({\mathbb {R}}\) with period \(S^{o} = \frac{2\pi L^{o}}{\varOmega ^{o}}\).
\({\mathcal {H}}_{1}^{o}\) is a reproducing kernel Hilbert space (RKHS) [51] with reproducing kernel (RK)
We denote the dimension of \({\mathcal {H}}_{1}^{o}\) by \(\dim ({\mathcal {H}}_{1}^{o})\) and \(\dim ({\mathcal {H}}_{1}^{o}) = 2L^{o}+1\).
Definition 5
The tensor product space \({\mathcal {H}}_{2}^{o} = {\mathcal {H}}_{1}^{o} \otimes {\mathcal {H}}_{1}^{o}\) is an RKHS with reproducing kernel
Note that \(\dim ({\mathcal {H}}_{2}^{o}) = \dim ({\mathcal {H}}_{1}^{o})^{2} = (2 L^{o} +1)^{2}\).
Definition 6
Let \(h^{3}_{p} \in \mathbb{L}^{1}(\mathbb{D}^{p}), p=1,2\), where \(\mathbb{L}^{1}\) denotes the space of Lebesgue integrable functions. The operator \(\mathcal{P}_{1}^{o}: \mathbb{L}_{1}(\mathbb{D}) \rightarrow {\mathcal {H}}_{1}^{o}\) given by
is called the projection operator from \(\mathbb{L}^{1}(\mathbb{D})\) to \({\mathcal {H}}_{1}\). Similarly, the operator \(\mathcal{P}_{2}^{o}: \mathbb{L}_{1}(\mathbb{D}^{2}) \rightarrow {\mathcal {H}}^{o}_{2}\) given by
is called the projection operator from \(\mathbb{L}^{1}(\mathbb{D}^{2})\) to \({\mathcal {H}}_{2}^{o}\).
We note that
If we now assume that \(h^{3}_{1}\in {\mathcal {H}}^{o}_{1}\) and \(h^{3}_{2} \in {\mathcal {H}}^{o}_{2}\), then \(\mathcal{P}_{1}^{o} h_{1}^{3} = h_{1}^{3}\) and \(\mathcal{P}_{2}^{o} h_{2}^{3} = h_{2}^{3}\), respectively, and the above equation is identical with equation (7) above.
2.3 Divisive normalization processors in the spacetime domain
In Sect. 2.2, we described a temporal divisive normalization processor model. The normalization term was the sum of a processed version of the input and the output. However, many biological circuits are thought to exhibit lateral inhibition and gain control [26, 58–61]. An example is provided by the photoreceptoramacrine cell layer shown in Fig. 1. In Fig. 3, we provide a model of the schematic in Fig. 1. This model is a ‘circuit’ extension of the one shown in Fig. 2 and accounts explicitly for lateral inhibition. We anchor the extension in the visual system where the input domain is spatiotemporal.
In Sect. 2.3.1 we model spatiotemporal stimuli and in Sect. 2.3.2 the spatiotemporal divisive normalization processors.
2.3.1 Modeling spatiotemporal stimuli
In this section we provide a model of the interaction between a group of photoreceptors and an amacrine cell. In each photoreceptor, the phototransduction process converts light into current and excites the membrane of the photoreceptor. The voltage signal is then propagated through its axon to the lamina. While photoreceptors provide inputs to amacrine cells, their axon terminals also receive amacrine cell input. Since an amacrine cell innervates multiple lamina cartridges, it provides spatial feedback to several photoreceptors in a small neighborhood.
We extend the temporal divisive normalization processor depicted in Fig. 2 to process spatiotemporal stimuli as shown in Fig. 3. Each spatially sampled point, or pixel, denoted as \(u^{i}(t), i=1,2,\ldots ,N\), is first processed by a temporal DNP. For simplicity rather than picking different Volterra kernels for each branch, \(\mathcal{T}^{1}, \mathcal{T}^{2}\), and \(\mathcal{T}^{3}\) are shared across branches. In addition, we introduce a multiinput Volterra processor (MVP) to model the spatiotemporal feedback due to the amacrine cell. Each of the branches in Fig. 3, without the input of the MVP block, is equivalent to the model in Fig. 2.
2.3.2 Spatiotemporal divisive normalization processors
As shown in Fig. 3(b), MVPs are comprised of secondorder filters acting on DNP output pairs in addition to linear filters independently acting on each DNP output. Thus, the inputs to the MVP are the DNP outputs \(v^{i}(t), i = 1,2,\ldots ,N\), and the MVP output amounts to
where
\(b^{4}\) is the zerosthorder Volterra kernel (constant). Furthermore, \(h^{i4}_{1}\in {\mathcal {H}}^{o}_{1}\), \(i=1,2,\ldots ,N\), are the firstorder Volterra kernels whose inputs are \(v^{i}\), \(i=1,2,\ldots ,N\), respectively, and \(h^{ij4}_{2}\in {\mathcal {H}}^{o}_{2}\), \(i,j=1,2,\ldots ,N\), are the secondorder Volterra kernels whose inputs are the pairs \((v^{i}, v^{j})\), \(i,j=1,2,\ldots ,N\), respectively.
The full model in Fig. 3 thus consists of parallel channels as depicted in Fig. 2 with the added crosschannel feedback normalization/gain control provided by the MVP block.
The overall output of the spatiotemporal DNP can be expressed as
W.L.O.G., we assume that \(b^{2} + b^{3} + b^{4} = 1\).
2.3.3 Spatiotemporal DNPs and contrast gain control
The relationship and some intuition behind the modeling power of spatiotemporal DNPs are provided in several examples in Appendix 1. The I/O of three simple spatiotemporal DNPs stimulated with different inputs is briefly mentioned here. In the first example, we evaluated the response of a DNP with four photoreceptors under different background light intensity levels. In Fig. 9 one of the photoreceptors is subjected to an additional flash of different light intensity, while the inputs of the other three are kept at the same background level. The steady state response of the photoreceptor that receives the additional flash is shifted as a function of the backgroundintensity level. In the second example, contrast gain control exerted by the amacrine cells is demonstrated for the same DNP in Fig. 10. The effect of the MVP block on the RMS contrast can be clearly seen in Fig. 10 and is quantitatively evaluated in Fig. 11. Finally, an example of steady state I/O visualization of a natural image with resolution of \(1536\times 1024\) pixels at low, medium, and high luminance values is shown in Fig. 13. The image was divided into \(16\times 16\) spatiotemporal DNP blocks with a fourpixel overlap in each direction.
3 Sparse identification of temporal DNPs
In what follows we derive sparse identification algorithms for the components of spatiotemporal DNPs depicted in Fig. 2 and formally defined in equation (5). In what follows, we assume that during experimental trials, the single isolated photoreceptor n is presented with M test stimuli \(u^{nm} = u^{nm}(t)\), \(m=1, 2, \ldots ,M\), and for each trial the outputs \(v^{nm} = v^{nm}(t)\), \(m=1, 2, \ldots , M\), are recorded. The objective is to identify the model components \(b_{1}, h^{l}_{1}, h^{l}_{2}\), \(l = 1,2,3\), from the knowledge of the inputs and outputs.
3.1 Deriving the sparse identification algorithm for temporal DNPs
Lemma 1
WithMinput stimuli presented to a temporal DNP, let the inputs\(u^{nm}(t)\)and the outputs\(v^{nm}(t)\)be sampled at times\((t_{k})\), \(k = 1,2,\ldots ,T\). Then
where the sampling functions\(\phi _{11}^{nmk} \in {\mathcal {H}}_{1}\), \(\phi _{12}^{nmk}\in \mathcal{H}_{1}\), \(\phi _{13}^{nmk}\in \mathcal{H}_{1}^{o}\), \(\phi _{21}^{nmk}\in \mathcal{H}_{2}\), \(\phi _{22}^{nmk}\in \mathcal{H}_{2}\), and\(\phi _{23}^{nmk}\in \mathcal{H}_{2}^{o}\)are given by
and\(q^{nmk} = v^{nm}(t_{k})\)for all\(m=1,2,\ldots,M\)and\(k=1,2,\ldots,T\).
Proof
See Appendix 2. □
Remark 3
From (21), it can be seen that the identification of the divisive normalization processor has been reformulated as a generalized sampling problem [62] in \({\mathbb {R}}\oplus {\mathcal {H}}_{1}\oplus {\mathcal {H}}_{1}\oplus {\mathcal {H}}_{1}^{o}\oplus {\mathcal {H}}_{2}\oplus {\mathcal {H}}_{2}\oplus {\mathcal {H}}_{2}^{o}\). Subsequently, the divisive normalization model can be identified by solving a system of linear equations.
In order to solve the system of linear equations in (21), we rewrite them first in matrix form.
Lemma 2
Equation (21) can be expressed in matrix form as follows:
for all\(m=1,2,\ldots ,M\)and\(k=1,2,\ldots ,T\), where\(\operatorname{Tr}(\cdot )\)denotes the trace operator, and
Proof
See Appendix 3 for more notation and detailed proof. □
A necessary condition on the number of trials and the number of measurements required for identifying the divisive normalization processor for solving the system of equations in Theorem 1 is that the number of trials \(M \ge 3 + 2 \cdot \dim ({\mathcal {H}}_{1})\) and the number of total samples \(TM \ge 1 + 2 \cdot \dim ({\mathcal {H}}_{1}) + 2 \cdot \dim ({\mathcal {H}}_{1})^{2}\).
It is easy to see that solving the system of equations above suffers from the curse of dimensionality. As the dimension of \({\mathcal {H}}_{1}\) increases, the number of samples needed to identify components increases quadratically. Note that the secondorder Volterra kernels \(h_{2}^{l}\), \(l = 1,2,3\), have unique symmetric forms with orthogonal expansions as follows [57]:
where \(g_{1}^{kl} \in {\mathcal {H}}_{1}, k \in {\mathbb {N}}\), are orthogonal to each other. In what follows, we assume that the secondorder Volterra kernels are sparse, i.e., \(\lambda ^{kl}=0\) for \(k>r_{l}\), where \(r_{l} \ll \dim ({\mathcal {H}}_{1})\). Sparse kernels often arise in modeling sensory processing, e.g., in complex cells in the primary visual cortex [42]. By exploiting the sparse structure of the secondorder kernels, the identification problem can be made tractable.
The sparsity of the kernels can be translated into a lowrank condition on the matrix representation of \(h^{l}_{2}\), \(l=1,2,3\) (see also Appendix 3). Ideally, the optimization problem would be a rank minimization problem. But rank minimization being NPhard, we use the surrogate of nuclear norm minimization instead, which is the convex envelope of the rank operator [63].
To perform sparse identification of the divisive normalization processor, we devised Algorithm 1. By optimizing over \(\mathbf{c}_{1}\) and \(\mathbf{C}_{2}\) and subsequently assigning the corresponding block entries according to (25), Algorithm 1 identifies \(b_{1}\), \(\mathbf{h}^{i}_{1}\), \(i= 1,2,3\), and \(\mathbf{H}^{i}_{2}\), \(i =1,2,3\).
As a surrogate of rank minimization, Algorithm 1 minimizes a linear combination of the nuclear norm of \(\mathbf{C}_{2}\) and the Euclidean norm of \(\mathbf{c}_{1}\). The optimization constraints correspond (i) in (29) to the generalized sampling problem allowing certain amount of error, (ii) in (30) to zero mean slack variables, (iii) in (31) to the zeros in the two blocks in the topright of \(\mathbf{C}_{2}\), (iv) in (32) to the zeros in the block in the lowerleft of \(\mathbf{C}_{2}\), and (v) in (33), (34), and (35), respectively, \(\mathbf{H}^{1}_{2}\), \(\mathbf{H}^{2}_{2}\), and \(\mathbf{H}^{3}_{2}\) are Hermitian.
Algorithm 1
\(\widehat{\mathbf{c}}_{1}\) and \(\widehat{\mathbf{C}}_{2}\) are the solution to the following optimization problem:
where \(\\cdot \_{*}\) denotes the nuclear norm defined as \(\\mathbf{C}_{2}\_{*} = \operatorname{Tr} ( (\mathbf{C}_{2}^{\mathsf {H}}\mathbf{C}_{2} )^{\frac{1}{2}} )\), \(\lambda _{1}, \lambda _{2}\) are appropriately chosen hyperparameters, \(\varepsilon^{i} \in \mathbb{R}\), \(i = 1,2,\ldots,MT\), represent slack variables, 1 represents a vector of all ones, \(\mathbf{I}_{p}\) represents a \(p\times p\) identity matrix, \(\mathbf{0}_{p}\) represents a \(p\times p\) matrix of all zeros, \(\mathbf{0}_{p\times q}\) represents a \(p\times q\) matrix of all zeros, and 0 represents a matrix of all zeros with dimensions that make the equation consistent.
Theorem 1
The filters of the DNP are identified as \(\hat{b}^{1} = \hat{b}^{1}\) and
where
Remark 4
By exploiting the structure of lowrank secondorder Volterra kernels, Algorithm 1 provides a tractable solution to the identification of the components of the divisive normalization processor.
3.2 Examples of sparse identification of temporal DNPs
We provide here identification examples solved using Algorithm 1.
Example 1
Here, we identify a temporal divisive normalization processor in Fig. 2, where
We choose the input space \({\mathcal {H}}_{1}\) to have \(L = 10, \varOmega = 100\pi \). Thus \(S = 0.2s\), \(\dim ({\mathcal {H}}_{1}) = 21\), and \(\dim ({\mathcal {H}}_{2}) = 441\). Note that all three secondorder Volterra kernels exhibit lowrank structure. We presented the model with 25 stimuli from \({\mathcal {H}}_{1}\), whose coefficients were chosen to be i.i.d Gaussian variables. Then, a total of 425 measurements were used from the input and the observed output pairs to solve the identification problem using Algorithm 1. The results of the identification are shown in Fig. 4. As can be seen from the figure, Algorithm 1 was able to identify the model with high precision using only 425 measurements, much less than the 1387 measurements that would have been required to solve the generalized sampling problem directly. The factor of reduction in the required measurements is critical when the model needs to be identified in a much larger space, for example, a space of spatiotemporal stimuli as shown in the next example.
Example 2
Here, we identify a detailed biophysical model of Drosophila photoreceptors as a DNP with feedforward only (see Fig. 2). The detailed biophysical model consists of 30,000 microvilli [5, 64]. The photon absorption in each microvillus is described by a Poisson process whose rate is proportional to the number of photons per microvillus incident on the ommatidium. Photon absorption leads to a transduction process governed by a cascade of chemical reactions. The entire transduction process is described by 13 successive differential equations and is given in [64]. The total number of equations of the photoreceptor model is \(390,000\).
For identifying the DNP, we used natural images from the van Hateren database [65] and simulated movements of a fly superimposed on the natural scenes. Each visual scene was captured by a previously developed retina model [64], endowed with realistic optics and the geometry of the fly retina. A 200second stimulus was generated. We then divided the stimulus into several segments, each of which scaled to a different level of mean light intensity. The resulting stimulus is shown in Fig. 5(a). The bandwidth of the DNP input and output spaces was limited to 50 Hz.
The visual input was presented to the DNP photoreceptor model. We used 10 seconds of the stimulus for identifying the DNP filters. The identified filters are depicted in Fig. 6. The output of the photoreceptor DNP model when stimulated by the other 190 seconds of the input stimulus is shown in Fig. 5(b). We evaluated the identified photoreceptor DNP model by computing the SNR between the output of the detailed biophysical model and that of the identified DNP model. The SNR was 26.14 [dB].
We additionally trained a model without normalization (i.e., with only \(\mathcal{T}^{1}\)). As shown in Fig. 5(b) the identified model without normalization does not match the output well across different light intensities. Compared with the output of the DNP model, the SNR was only 14.48 [dB].
The DNP filters were identified using naturalistic stimuli. To test if the identified filters can also match the output of other types of stimuli, we presented a Gaussian noise stimulus with a bandwidth of 50 Hz to the photoreceptor model. The resulting output was compared with the output of the detailed biophysical model. As shown in Fig. 7, the output of the DNP model closely follows the actual photoreceptor output, and the SNR was 15.04 [dB]. We note that since the input space is defined as an RKHS, the statistics of the input stimuli do not affect the quality of the DNP output. We also evaluated in Fig. 7 the identification model without normalization. The SNR of the output is 4.55 [dB], a significant decrease from the output of the DNP model.
4 Sparse identification of spatiotemporal DNPs
In what follows we derive sparse identification algorithms for the components of spatiotemporal DNPs.
Given the spatiotemporal divisive normalization processor depicted in Fig. 3, we are interested in identifying all the filters from input and output observations. We formulate an optimization problem, which achieves such identification, with high fidelity and with a relatively small number of measurements.
4.1 Deriving the sparse identification algorithm for spatiotemporal DNPs
Here, we make the assumption that the filters \(h^{i4}(t), i = 1,2,\ldots ,N\), and \(h_{2}^{ij4}(t_{1}, t_{2})\), \(i,j = 1,2,\ldots ,N\), are followed by the LPF with bandwidth \(\varOmega ^{o}\) and that the output is sampled at a rate of \(2f_{\max }\), where \(f_{\max } = \frac{\varOmega }{2\pi }\) is the maximum of the bandwidth of the filters \(h^{i4}(t)\) and \(h_{2}^{ij4}(t_{1}, t_{2})\). By abuse of notation, we will use \(v^{i}(t)\) to denote the lowpassed version of the actual output. Note that, based upon the assumptions on the bandlimited nature of the feedback filters acting on the output, the responses of these filters to the lowpassed outputs will be the same as the responses to the actual outputs.
We present M trials where input stimuli \(u^{nm}\) are chosen to be elements of the space of trigonometric polynomials \({\mathcal {H}}_{1}\) for \(n=1, \ldots , N\), \(m=1, \ldots ,M\). We project the outputs \(v^{nm}(t)\) on the Hilbert space of trigonometric polynomials \({\mathcal {H}}_{1}^{o}\), i.e., \(\mathcal{P}_{1}^{o} v^{nm} (s)\). Note that \(\mathcal{P}_{1}^{o} v^{nm} (s)\) is approximately \(v^{nm} (s)\) for large values of \(\varOmega ^{o}\) [66]. Further we assume that \(v^{nm}, h^{1}_{1}, h^{2}_{1}, h^{3}_{1}, h^{i4}_{1}\in \mathbb{L}^{2}(\mathbb{D})\), \(i = 1,2,\ldots ,N\), the space of square integrable functions over domain \(\mathbb{D}\) and \(h_{2}^{1}, h_{2}^{2}, h_{2}^{3}, h_{2}^{ij4} \in \mathbb{L}^{2}(\mathbb{D}^{2})\), \(i,j = 1,2,\ldots ,N\), the space of square integrable functions over domain \(\mathbb{D}^{2}\).
We consider here the identification of the entire DNP circuit at once for two reasons. First, as all channels are connected in the spatial domain through the MVP, the inputs to the MVP are the outputs of the entire DNP circuit. Therefore, all outputs are required to identify the MVP. Second, since \(h^{i}_{1}, h^{i}_{2}\), \(i=1,2,3\), are shared across all channels, fewer trials are needed to identify these filters. We present the following.
Lemma 3
WithMtrials presented to the spatiotemporal DNP, let the inputs\(u^{nm}(t)\)and the outputs\(v^{nm}(t)\)be sampled at times\((t_{k})\), \(k = 1, \ldots , T\). Then, for\(i,j=1,2,\ldots ,N\), we have the following equations:
where the sampling functions\(\phi _{11}^{nmk} \in {\mathcal {H}}_{1}\), \(\phi _{12}^{nmk}\in \mathcal{H}_{1}\), \(\phi _{13}^{nmk}\in \mathcal{H}_{1}^{o}\), \(\phi _{14}^{inmk}\in \mathcal{H}_{1}^{o}\), \(\phi _{21}^{nmk}\in \mathcal{H}_{2}\), \(\phi _{22}^{nmk}\in \mathcal{H}_{2}\), \(\phi _{23}^{nmk}\in \mathcal{H}_{2}^{o}\), and\(\phi _{24}^{ijnmk}\in \mathcal{H}_{2}^{o}\)are given by
and\(q_{k}^{nm} = v^{nm}(t_{k})\)for all\(m=1,2,\ldots,M\), \(k=1,2,\ldots,T\), and\(i,j,n=1,2,\ldots,N\).
Proof
See Appendix 4. □
Remark 5
Theorem 3 suggests that identifying the lateral divisive normalization model is equivalent to solving a generalized sampling problem with noisy measurements. It also suggests that the output needs to be sampled at a high enough rate, and that the choice of the Hilbert space used to reconstruct the feedback filters is critical since incorrect choices for these parameters can negatively affect the identification by introducing ‘noise’ in the measurements.
We now present the following algorithm to identify the model that exploits the lowrank constraints imposed on the quadratic filters.
Lemma 4
Equation (43) can be expressed in matrix form as follows:
for all\(n=1,2,\ldots ,N\), \(m=1,2,\ldots ,M\), and\(k=1,2,\ldots ,T\), where
We provide the following algorithm to identify \(b_{1}\), \(\mathbf{h}^{i}_{1}\), \(i= 1,2,3\), \(\mathbf{h}^{i4}_{1}\), \(i= 1,2,\ldots ,N\), \(\mathbf{H}^{i}_{2}\), \(i =1,2,3\), and \(\mathbf{H}^{ij4}_{2}\), \(i,j =1,2,\ldots ,N\).
Again, we assume that all the secondorder filters have sparse structures akin to (27).
Algorithm 2
Let \(\widehat{\mathbf{c}}_{1}\) and \(\widehat{\mathbf{C}}_{2}\) be the solution to the following optimization problem:
where \(\lambda _{1}, \lambda _{2}\) are appropriately chosen hyperparameters and \(\varepsilon^{i} \in \mathbb{R}\), \(i=1,2,\ldots,MTN\), represent slack variables.
The constraints in Algorithm 2 are similar to those in Algorithm 1. Note that \(H^{ij4}\) are not constrained to be Hermitian. This follows from the assumption that the MVP block may perform asymmetric processing on any pair of inputs to the block.
Theorem 2
The identified spatiotemporal divisive normalization is specified as\(\hat{b}^{1} = \hat{b}^{1}\)and
where
and
Remark 6
Assuming that all the secondorder kernels in the lateral divisive normalization model have rank r, the expected number of measurements for Algorithm 2 to identify the model is of the order \(\mathcal{O} (r \cdot \dim ({\mathcal {H}}_{1}) + N^{2}r \cdot \dim ({\mathcal {H}}_{1}^{o}) )\). When N is large, the \(N^{2}\) factor may become prohibitive in identifying the model. Additional assumptions on \(h^{ij4}_{2}\) may help mitigate this and maintain tractability of solving the identification problem. For example, with the assumption that \(h^{ij4} = h^{14}_{1}\) if \(i=j\) and \(h^{ij4}_{2} = h^{24}_{1}\) otherwise, the expected number of measurements required will be \(\mathcal{O} (r \cdot \dim ({\mathcal {H}}_{1}) + Nr \cdot \dim ({\mathcal {H}}_{1}^{o}) )\).
4.2 An example of sparse identification of a spatiotemporal DNP
We now present an example of identification obtained by Algorithm 2. We demonstrate here that, in addition to the identification of the Volterra kernels operating within each channel, the MVP in the spatiotemporal DNP can be identified.
Example 3
Here, we choose the DNP in Fig. 3 with \(N=4\), and
and the other Volterra kernels are set to 0. Note that Yiyin, and \(h^{1}_{1}\), \(h^{2}_{1}\) are shared across all channels. In addition, \(h^{ij4} = h^{ji4}\) in this case. We assumed knowledge of this symmetric structure of the model and adapted the identification algorithm accordingly and only identified six linear filters and ten quadratic filters.
We performed the identification in the Hilbert space of bandlimited functions with \(\varOmega = \varOmega ^{o} = 40\pi \), and we chose the same space for the input stimuli and for both the feedforward filters and the feedback ones. We solved for the filters truncated to a period of 0.4 s, and thus there were 17 coefficients to be identified for the linear filters and \(17\times 17\) coefficients to be identified for each of the quadratic filters. A total of 1116 measurements (279 measurements from each cell) were used to perform the identification, and the results are depicted in Fig. 8. The average SNR of reconstruction across all filters was more than 150 [dB]. Note that solving the generalized sampling problem directly for the same problem would have required at least 3570 measurements.
5 Discussion
As already mentioned in the Introduction, the photoreceptor/amacrine cell layer of the early vision system of the fruit fly rapidly adapts to visual stimuli whose intensity and contrast vary orders of magnitude both in space and time.
In this paper we presented a spatiotemporal divisive normalization processor that models the transduction and the contrast gain control in the photoreceptor and amacrine cell layer of the fruit fly. It incorporates processing blocks that explicitly model the feedforward and the temporal feedback path of each photoreceptor and the spatiotemporal feedback from amacrine cells to photoreceptors. We demonstrated that with some simple choice of parameters, the DNP response maintains the contrast of the input visual field across a large range of average spatial luminance values.
We characterized the I/O of the spatiotemporal DNP and highlighted the highly nonlinear behavior of the DNP in contrast gain control. Despite the divisive nonlinearity, we provided an algorithm for the sparse identification of the entire DNP. We showed that the identification of the DNP components can be interpreted as a generalized sampling problem. More importantly, the sparse identification algorithm does not suffer from the curse of dimensionality that would otherwise require a large number of measurements that are quadratically related to the dimension of the input and output spaces.
Neural circuits are often noisy. Although the I/O characterization of MIMO DNPs provided in Sect. 2 is noise free, it can be easily extended to account for noise [52–54]. The identification of DNPs can be interpreted as a generalized sampling problem in an RKHS with noisy measurements. The sparse identification algorithms we provided in this paper include slack variables to account for these inaccurate measurements. Similar algorithms have been shown to be robust to noise [42].
Contrast gain control is an intrinsically nonlinear problem. Early approaches to modeling contrast gain control rely on analyzing the Volterra kernels, i.e., linear or nonlinear receptive fields identified when stimuli of different statistics are presented [19, 20, 23]. The objective of these studies is to observe and characterize the differences between the identified Volterra kernels. To fully capture the nonlinear effect in contrast gain control, these approaches would rely on identifying higherorder Volterra kernels that are too costly to compute and are simply ignored. The contrast gain control model advanced in this paper, however, results from the divisive normalization operation, albeit having Volterra kernels modeling internal filters in the DNP. The division largely expands upon DNP’s ability to model higherorder nonlinearities in contrast gain control.
Biologically, divisive normalization can take place in feedforward or feedback circuits, or both [26]. Most divisive normalization models only consider the feedforward normalization circuit. Although normalization can be achieved by a multiplicative feedback mechanism, it mainly serves as a transformation leading to an equivalent form of feedforward divisive normalization when the input is a constant [27, 67]. The MIMO DNP models described in this paper explicitly include divisive normalization with both feedforward and feedback paths that can be more readily employed in modeling underlying neuronal circuit mechanisms, such as those arising in the photoreceptor and amacrine cell layer.
Predictive coding often takes the form subtractive negative feedback [68]. The MIMO DNP models described in this paper represent an alternative form of predictive coding. Indeed, divisive input modulation has been proposed to implement predictive coding by calculating the residual errors using division rather than subtraction [69]. In flies, it has been suggested that inhibition in the lamina, where retina photoreceptors and amacrine cells interact, leads to predictive coding [70]. Our study advances a spatiotemporal MIMO model for predictive coding and provides an algorithm to identify the components of the MIMO DNP.
The MIMO DNP model opens a new avenue for exploring and quantifying the highly nonlinear nature of sensory processing. The MIMO DNP in Fig. 3 can be further extended to allow the different transformations \(\mathcal{T}^{i}\), \(i=1,2,3\), to incorporate spatiotemporal Volterra kernels, thereby making it more versatile for modeling other types of sensory processing, including (i) interactions between cones and horizontal cells in vertebrate retinas [58], (ii) channels/glomeruli in olfactory circuits and interactions between them through local neurons [9, 71], and (iii) crosssuppression and gain control in the auditory [12, 72], and visual cortices [73–75]. The sparse identification algorithms advanced here can be easily extended to identify the MIMO DNPs of these systems as well.
Abbreviations
 DNP:

Divisive normalization processor
 LMC:

Large monopolar cells
 LPF:

Lowpass filter
 MVP:

Multiinput Volterra processor
 RKHS:

Reproducing kernel Hilbert space
 RMS:

Root mean square
 SNR:

Signaltonoise ratio
References
Laughlin SB. Matching coding, circuits, cells, and molecules to signals: general principles of retinal design in the fly’s eye. Prog Retin Eye Res. 1994;13(1):165–96. Available from http://www.sciencedirect.com/science/article/pii/1350946294900094.
Rodieck RW. The first steps in seeing. London: Oxford University Press; 1998.
de Bruyne M, Clyne PJ, Carlson JR. Odor coding in a model olfactory organ: The Drosophila maxillary palp. J Neurosci. 1999;19(11):4520–32. Available from http://www.jneurosci.org/content/19/11/4520.
Olsen SR, Bhandawat V, Wilson RI. Divisive normalization in olfactory population codes. Neuron. 2010;66(2):287–99. Available from http://www.sciencedirect.com/science/article/pii/S0896627310002497.
Song Z, Postma M, Billings S, Coca D, Hardie RC, Juusola M. Stochastic adaptive sampling of information by microvilli in fly photoreceptors. Curr Biol. 2012;22:1371–80.
Meister M, Berry MJ II. The neural code of the retina. Neuron. 1999;22(3):435–50. https://doi.org/10.1016/S08966273(00)80700X.
Scholl B, Latimer KW, Priebe NJ. A retinal source of spatial contrast gain control. J Neurosci. 2012;32(29):9824–30. Available from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3432019/.
Baccus SA, Meister M. Fast and slow contrast adaptation in retinal circuitry. Neuron. 2002;36:909–19.
Wilson RI. Early olfactory processing in Drosophila: mechanisms and principles. Annu Rev Neurosci. 2013;36:217–41.
Firestein S. How the olfactory system makes sense of scents. Nature. 2001;413:211. https://doi.org/10.1038/35093026.
Clemens J, OzeriEngelhard N, Murthy M. Fast intensity adaptation enhances the encoding of sound in Drosophila. Nat Commun. 2018;9:134. Available from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC5760620/.
Rabinowitz NC, Willmore BD, Schnupp JW, King AJ. Contrast gain control in auditory cortex. Neuron. 2011;70(6):1178–91. Available from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3133688/.
Kohn A, Movshon JA. Neuronal adaptation to visual motion in area MT of the macaque. Neuron. 2003;39(4):681–91.
Rien D, Kern R, Kurtz R. Octopaminergic modulation of contrast gain adaptation in fly visual motionsensitive neurons. Eur J Neurosci. 2012;36(8):3030–9.
Heuer HW, Britten KH. Contrast dependence of response normalization in area MT of the rhesus macaque. J Neurophysiol. 2002;88(6):3398–408.
Shapley RM, Victor JD. The effect of contrast on the transfer properties of cat retinal ganglion cells. J Physiol. 1978;285(1):275–98.
Shapley RM, Victor JD. How the contrast gain control modifies the frequency responses of cat retinal ganglion cells. J Physiol. 1981;318(1):161–79. Available from https://physoc.onlinelibrary.wiley.com/doi/abs/10.1113/jphysiol.1981.sp013856.
Shapley R, EnrothCugell C. Chapter 9 visual adaptation and retinal gain controls. Prog Retin Eye Res. 1984;3:263–346. Available from http://www.sciencedirect.com/science/article/pii/0278432784900117.
Chander D, Chichilnisky EJ. Adaptation to temporal contrast in primate and salamander retina. J Neurosci. 2001;21(24):9904–16.
Kim KJ, Rieke F. Temporal contrast adaptation in the input and output signals of salamander retinal ganglion cells. J Neurosci. 2001;21(1):287–99. Available from https://www.jneurosci.org/content/21/1/287.
Weber AI, Krishnamurthy K, Fairhall AL. Coding principles in adaptation. Annu Re Vis Sci. 2019;5(1):427–49. https://doi.org/10.1146/annurevvision091718014818.
Shapley R. The contrast gain control and feedback within the retina. IFAC Proc Vol. 1982;15(4):489–94. 6th IFAC Symposium on Identification and System Parameter Estimation, Washington USA, 7–11 June. Available from http://www.sciencedirect.com/science/article/pii/S1474667017630362.
Yu Y, Lee TS. Dynamical mechanisms underlying contrast gain control in single neurons. Phys Rev E. 2003;68:011901. Available from https://link.aps.org/doi/10.1103/PhysRevE.68.011901.
Friederich U, Coca D, Billings S, Juusola M. Data modelling for analysis of adaptive changes in fly photoreceptors. In: Neural information processing, lecture notes in computer science. vol. 5863. 2009. p. 34–48.
Friederich U, Billings SA, Hardie RC, Juusola M, Coca D. Fly photoreceptors encode phase congruency. PLoS ONE. 2016;11(6):1–21. https://doi.org/10.1371/journal.pone.0157993.
Carandini M, Heeger DJ. Normalization as a canonical neural computation. Nat Rev Neurosci. 2012;13:51–62.
Heeger DJ. Normalization of cell responses in cat striate cortex. Vis Neurosci. 1992;9(2):181–97.
Schwartz O, Simoncelli EP. Natural sound statistics and divisive normalization in the auditory system. Advances in Neural Information Processing Systems. vol. 13. 2000.
Ohshiro T, Angelaki DE, DeAngelis GC. A normalization model of multisensory integration. Nat Neurosci. 2011;14(6):775–82. https://doi.org/10.1038/nn.2815.
Reynolds JH, Heeger DJ. The normalization model of attention. Neuron. 2009;61(2):168–85. https://doi.org/10.1016/j.neuron.2009.01.002.
Montijn J, Klink P, van Wezel RJA. Divisive normalization and neuronal oscillations in a single hierarchical framework of selective visual attention. Front Neural Circuits. 2012;6:22. Available from https://www.frontiersin.org/article/10.3389/fncir.2012.00022.
Sawada T, Petrov AA. The divisive normalization model of V1 neurons: a comprehensive comparison of physiological data and model predictions. J Neurophysiol. 2017;118(6):3051–91. https://doi.org/10.1152/jn.00821.2016. PMID: 28835531. Available from.
Cui Y, Wang YV, Park SJH, Demb JB, Butts DA. Divisive suppression explains highprecision firing and contrast adaptation in retinal ganglion cells. eLife. 2016;5:e19460. https://doi.org/10.7554/eLife.19460.
Wainwright MJ, Schwartz O, Simoncelli EP. In: Rao R, Olshausen B, Lewicki M, editors. Natural image statistics and divisive normalization: modeling nonlinearity and adaptation in cortical neurons. Cambridge: MIT Press; 2002. p. 203–22.
Lyu S. Divisive normalization: justification and effectiveness as efficient coding transform. In: Lafferty JD, Williams CKI, ShaweTaylor J, Zemel RS, Culotta A, editors. Advances in neural information processing systems. vol. 23. Red Hook: Curran Associates; 2010. p. 1522–30. Available from http://papers.nips.cc/paper/4065divisivenormalizationjustificationandeffectivenessasefficientcodingtransform.pdf.
CoenCagli R, Schwartz O. The impact on midlevel vision of statistically optimal divisive normalization in V1. J Vis. 2013;13(8):13. https://doi.org/10.1167/13.8.13.
Lyu S, Simoncelli EP. Nonlinear image representation using divisive normalization. In: Proceedings IEEE computer society conference on computer vision and pattern recognition. 2008. p. 1–8. Available from https://www.ncbi.nlm.nih.gov/pubmed/25346590.
Beck JM, Latham PE, Pouget A. Marginalization in neural circuits with divisive normalization. J Neurosci. 2011;31(43):15310–9. Available from https://www.jneurosci.org/content/31/43/15310.
Rabinowitz NC, Willmore BDB, Schnupp JWH, King AJ. Spectrotemporal contrast kernels for neurons in primary auditory cortex. J Neurosci. 2012;32(33):11271–84. Available from https://www.jneurosci.org/content/32/33/11271.
Malo J, Bertalmio M. Appropriate kernels for divisive normalization explained by Wilson–Cowan equations. 2018. Preprint. arXiv:1804.05964.
RiveraAlba M, Vitaladevuni SN, Mishchenko Y, Lu Z, Takemura Sy, Scheffer L et al.. Wiring economy and volume exclusion determine neuronal placement in the Drosophila brain. Curr Biol. 2011;21(23):2000–5. http://www.sciencedirect.com/science/article/pii/S0960982211011468.
Lazar AA, Ukani NH, Zhou Y. Sparse functional identification of complex cells from spike times and the decoding of visual stimuli. J Math Neurosci. 2018;8(1):2.
Fischbach KF, Dittrich APM. The optic lobe of Drosophila melanogaster. I. A Golgi analysis of wildtype structure. Cell Tissue Res. 1989;258:441–75.
Nikolaev A, Zheng L, Wardill TJ, O’Kane CJ, de Polavieja GG, Juusola M. Network adaptation improves temporal representation of naturalistic stimuli in drosophila eye: II mechanisms. PLoS ONE. 2009;4(1):e4306. https://doi.org/10.1371/journal.pone.0004306.
Sterling P, Laughlin S. Principles of neural design. Cambridge: The MIT Press; 2015.
Matić T, Laughlin SB. Changes in the intensityresponse function of an insect’s photoreceptors due to light adaptation. J Comp Physiol. 1981;145(2):169–77. https://doi.org/10.1007/BF00605031.
Juusola M, Hardie RC. Light adaptation in Drosophila photoreceptors: I. Response dynamics and signaling efficiency at 25^{∘}C. J Gen Physiol. 2001;117:3–25.
Zheng L, de Polavieja GG, Wolfram V, Asyali MH, Hardie RC, Juusola M. Feedback network controls photoreceptor output at the layer of first visual synapses in Drosophila. J Gen Physiol. 2006;127(5):495–510. Available from http://jgp.rupress.org/content/127/5/495.
Laughlin SB. Fly optic lamina as a guide to neural circuit design. 2nd ed. In: Shepard GM, Grillner S, editors. Handbook of brain microcircuits. London: Oxford University Press; 2017.
Lazar AA, Pnevmatikakis EA, Zhou Y. Encoding natural scenes with neural circuits with random thresholds. Vis Res. 2010;50(22):2200–12. Special Issue on Mathematical Models of Visual Coding.
Berlinet A, ThomasAgnan C. Reproducing kernel Hilbert spaces in probability and statistics. Norwell: Kluwer Academic; 2004.
Lazar AA, Pnevmatikakis EA. Reconstruction of sensory stimuli encoded with integrateandfire neurons with random thresholds. EURASIP J Adv Signal Process. 2009;2009(1):682930. https://doi.org/10.1155/2009/682930.
Lazar AA, Zhou Y. Reconstructing natural visual scenes from spike times. In: Proceedings of the IEEE. vol. 102. 2014. p. 1500–19.
Lazar AA, Zhou Y. Volterra dendritic stimulus processors and biophysical spike generators with intrinsic noise sources. Front Comput Neurosci. 2014;8:95.
Lazar AA, Slutskiy YB. Channel identification machines for multidimensional receptive fields. Front Comput Neurosci. 2014;8:117.
Pumir A, Graves J, Ranganathan R, Shraiman BI. Systems analysis of the single photon response in invertebrate photoreceptors. Proc Natl Acad Sci. 2008;105(30):10354–9. Available from https://www.pnas.org/content/105/30/10354.
Rugh WJ. Nonlinear system theory: Volterra/Wiener approach. Baltimore: Johns Hopkins University Press; 1981.
VanLeeuwen M, Fahrenfort I, Sjoerdsma T, Numan R, Kamermans M. Lateral gain control in the outer retina leads to potentiation of center responses of retinal neurons. J Neurosci. 2009;29(19):6358–66. Available from http://www.jneurosci.org/content/29/19/6358.
Blakemore C, Carpenter RHS, Georgeson MA. Lateral inhibition between orientation detectors in the human visual system. Nature. 1970;228:37–9.
Olsen SR, Wilson RI. Lateral presynaptic inhibition mediates gain control in an olfactory circuit. Nature. 2008;452(7190):956–60. Available from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2824883/.
Polat U, Sagi D. Lateral interactions between spatial channels: suppression and facilitation revealed by lateral masking experiments. Vis Res. 1993;33(7):993–9.
Christensen O. Frames and bases: an introductory course. Basel: Birkhäuser; 2008.
Fazel M, Hindi H, Boyd S. Rank minimization and applications in system theory. In: Proceedings American control conference. 2004. p. 3273–8.
Lazar AA, Psychas K, Ukani NH, Zhou Y. A parallel processing model of the Drosophila retina. Neurokernel Request for Comments. Neurokernel RFC #3. 2015 Aug. https://doi.org/10.5281/zenodo.30036.
van Hateren JH, van der Schaaf A. Independent component filters of natural images compared with simple cells in primary visual cortex. Proc R Soc Lond B, Biol Sci. 1998;265(1394):359–66.
Martin R. Approximation of Ωbandlimited functions by Ωbandlimited trigonometric polynomials. Sampl Theory Signal Image Process. 2007;6(3):273–96.
Ohshiro T, Angelaki DE, DeAngelis GC. A neural signature of divisive normalization at the level of multisensory integration in primate cortex. Neuron. 2017;95(2):399–411.e8. Available from http://www.sciencedirect.com/science/article/pii/S0896627317305950.
Spratling MW. A review of predictive coding algorithms. Brain Cogn. 2017;112:92–7. Available from http://www.sciencedirect.com/science/article/pii/S027826261530035X.
Spratling MW, De Meyer K, Kompass R. Unsupervised learning of overlapping image components using divisive input modulation. Comput Intell Neurosci. 2009;2009:381457.
Srinivasan MV, Laughlin SB, Dubs A. Predictive coding: a fresh view of inhibition in the retina. Proc R Soc Lond B, Biol Sci. 1982;216(1205):427–59.
Lazar AA, Yeh CH. A Parallel Processing Model of Drosophila Olfactory Sensory Neurons and Its Biological Validation. Neurokernel Request for Comments Neurokernel RFC #10. 2017 Dec.
Natan RG, Carruthers IM, MwilambweTshilobo L, Geffen MN. Gain control in the auditory cortex evoked by changing temporal correlation of sounds. Cereb Cortex. 2017;27(3):2385–402. https://doi.org/10.1093/cercor/bhw083.
Allison JD, Smith KR, Bonds AB. Temporalfrequency tuning of crossorientation suppression in the cat striate cortex. Vis Neurosci. 2001;18(6):941–8.
Geisler WS, Albrecht DG. Cortical neurons: isolation of contrast gain control. Vis Res. 1992;32(8):1409–10. Available from http://www.sciencedirect.com/science/article/pii/004269899290196P.
Priebe NJ, Ferster D. Mechanisms underlying crossorientation suppression in cat visual cortex. Nat Neurosci. 2006;9(4):552. https://doi.org/10.1038/nn1660.
Acknowledgements
Not applicable.
Availability of data and materials
Data sharing is not applicable to this article as no datasets were generated or analyzed during the current study.
Authors’ information
The authors’ names are listed in alphabetical order.
Funding
The research reported here was supported by AFOSR under grant #FA95501610410.
Author information
Authors and Affiliations
Contributions
AAL, NHU, and YZ conceptualized and designed the study, performed analysis, and wrote the manuscript. NHU and YZ performed the simulations. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
Not applicable.
Competing interests
The authors declare that they have no competing interests.
Consent for publication
Not applicable.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
Appendix 1: Spatiotemporal DNPs and contrast gain control
Here, we characterize the I/O of simple DNPs stimulated with three different inputs. In the first example, we evaluate the response of a \(1 \times 4\) DNP under different background light intensity levels. In the second example, contrast gain control exerted by the amacrine cells is demonstrated with the same DNP. In the third example, a DNP consisting of \(16 \times 16\) DNPs tiling a \(1536 \times 1024\) visual field is stimulated with a natural image taken at low, medium, and high luminance values. The DNP output is evaluated with and without the MVP block.
Example 4
Here, we consider a simple \(1 \times 4\) DNP consisting of four photoreceptors and a single amacrine cell receiving inputs from and providing feedback to all four photoreceptors. The choices of the component filters of the DNP are as follows:
for the transformations \(\mathcal{T}^{1}\) and \(\mathcal{T}^{2}\), the kernels are
for the transformation \(\mathcal{T}^{3}\), the kernels are
and for the transformation \(\mathcal{L}^{4}\), the kernels are
Here, the bandwidth of \({\mathcal {H}}^{o}_{1}\) was chosen to be \(\varOmega ^{o} = 10\cdot 2\pi \) rad/s. The I/O of the fly’s individual photoreceptors in steady state is described by a saturating nonlinearity similar in appearance to a sigmoid when plotted on a logarithmic scale [1]. Photoreceptors can deal with the orders of magnitude of the input stimuli while maintaining their output in a suitable range. In addition, the photoreceptors exhibit adaptation so that the sigmoidal curves shift to adjust to the mean local luminance value [26].
We examine here the steady state response of the above DNP under four different background light intensity levels. At the start of each trial, all four photoreceptors are first adapted to the same background light intensity level. One of the photoreceptors is then subjected to an additional flash of different light intensity with a duration of two seconds, while the inputs to the other three are kept at the same background level. We observe the value of the steady state response of the photoreceptor that receives the additional flash. Figure 9 depicts the relationship between the observed steady state response and the light intensity of the flash at the four background intensity levels. It demonstrates that the response of the DNP is backgrounddependent, and the overall slope and range are similar across different background levels. With the MVP block, the spatiotemporal DNP can reproduce responses of photoreceptors observed in experimental settings [26, 46].
Since all the photoreceptors exhibit a sigmoid like nonlinearity, the output of the retina will be constrained in a suitable current range that can be processed by postsynaptic neurons in the lamina. However, without adaptation to mean local luminance, the saturating nature of the nonlinearities leads to a loss of spatial contrast. The spatial contrast is preserved by spatial gain control or adaptation modeled here with the MVP block.
Example 5
Here, the I/O of DNPs with and without the MVP block is evaluated. Using the same DNP as in the example above, we stimulated the DNP with 25 “images” with a resolution of \(1\times 4\) pixels. Each image has a different average luminance and root mean square (RMS) contrast. The RMS contrast is defined as the standard deviation of pixel intensities normalized by the mean, i.e.,
where
and \(N=4\). These images are shown in the “input” block in Fig. 10, with each bar representing the input intensity to one photoreceptor. Note that the pixels are extended along the yaxis for a quick visual examination.
In the “Photoreceptors” block in Fig. 10, the steady state responses of the DNP without the MVP block to the respective inputs are shown. Here, pure black and white represent a response of 0 and 1, respectively. This can be interpreted as a circuit in which the reciprocal connections between photoreceptors and amacrine cells are blocked.
In the “Photoreceptors + Am” block in Fig. 10, the steady state responses of the full DNP to their respective inputs are shown. Comparing the “Photoreceptors” and “Photoreceptors + Am” blocks, the responses of the DNP without feedback are washed out, or exhibit low contrast, particularly at three of the four corners of the \(5\times 5\) image array, i.e., when either the luminance or contrast is too high or too low. By contrast, the individual bars in the response of the full DNP model are more readily discernible across several orders of magnitude of luminance.
We note that a saturating nonlinearity can maintain its output in a constrained range even when the input varies by orders of magnitude. However, the nonlinearity may lead to a loss of spatial contrast. In contrast, the DNP constrains its output to a suitable range while maintaining the spatial contrast. This is demonstrated in Fig. 11, where the RMS contrast of the input images in Fig. 10 is plotted against the RMS contrasts of the responses for both cases—with and without MVP in the DNP. Spatial contrast, arguably, is an important feature of the image that should be preserved or even enhanced for subsequent stages of extraction of ethologically relevant information from the visual stimulus.
Example 6
Here, we apply a fullscale DNP model to a natural image. The image is taken in raw format so that its pixel values are proportional to the light intensity each pixel is exposed to [65]. The resolution of the image is \(1536\times 1024\). The image is first divided into \(16\times 16\) blocks with a fourpixel overlap in each direction. A DNP is assigned to each block, and the filters in the DNP are designed as follows:
for the transformations \(\mathcal{T}^{1}\) and \(\mathcal{T}^{2}\), the kernels are
for the transformation \(\mathcal{T}^{3}\), the kernels are
and for the transformation \(\mathcal{L}^{4}\), the kernels are
where \((x^{i}, y^{i})\) is the coordinate of pixel i, and \((x^{0}, y^{0})\) is the coordinate of the center pixel in the \(16\times 16\) block. Here, the bandwidth of \({\mathcal {H}}^{o}_{1}\) was chosen to be \(\varOmega ^{o} = 40\cdot 2\pi \) rad/s.
Figure 12 provides a graphical visualization of the feedback processing of the DNP. We note that the RHS of both equalities (83) and (84) can be separated into a temporal term and a spatial term. The temporal terms correspond to temporal Volterra kernels processing the output at each pixel (see cylinders in Fig. 12). The feedback term is then the weighted sum with a weight according to the spatial term. Each weight is the value of a Gaussian function evaluated at the distance between the location of the pixel and the center of the block. Due to the overlaps between blocks, there are pixels that belong to multiple blocks. The feedback term for such a pixel is the average of the feedback terms from all the blocks the pixel belongs to.
We tested the DNP using low, medium, and high luminance (1×, 10×, and 100×) of the original image measured in the number of photons and represented by arbitrary units. They are shown on the top row of Fig. 13. The logarithms of the inputs are shown on the second row. On the third row, we show the response of the DNP without the MVP block, i.e., each photoreceptor processes independently a single pixel. The responses of the full DNP model are shown on the bottom row.
As can be seen from Fig. 13, the response of the DNP with MVP is robust across the different levels of luminance, showing the best quality for all three luminance levels. By contrast, the response of the DNP without MVP is either too dark for low contrast or saturated at high luminance. The contrast within each response is also significantly worse. It is also not ideal to use only a logarithmic nonlinearity to process the images (second row), as the contrast of the images is not sharp enough.
To conclude we note that, with a simple choice of filters, the spatiotemporal DNP proposed in Sect. 2.3 operates in a range of luminance values spanning several orders of magnitude, much like the photoreceptoramacrine cell layer. The contrast of the output of the DNP is not compromised while its range remains strongly bounded.
Appendix 2: Proof of Lemma 1
For the mth trial and the kth time sample, multiplying out the denominator in (5), we get
where \(u_{2}^{nm} (t_{1}, t_{2}) \in \mathcal{H}_{2}\) and \(u_{2}^{nm} (t_{1}, t_{2}) = u^{nm} (t_{1})u^{nm} (t_{2})\).
With the notation in (22) and after rearranging terms in equation (85) above, we obtain
With \(q^{nmk} = v^{nm}(t_{k})\), equation (86) becomes
for all \(m=1,2,\ldots,M\) and \(k=1,2,\ldots,T\).
Appendix 3: Proof of Lemma 2
Let
and the \((2L+1)\times 1\) vectors
the \((2L^{o}+1)\times 1\) vector
the \((2L+1)\times (2L+1)\) Hermitian matrices
and the \((2L^{o}+1)\times (2L^{o}+1)\) Hermitian matrix
Further, let
where \(a^{nm}_{l}\) represents the coefficient of \(u^{nm}(t)\) w.r.t. basis element \(e_{l}\) in \({\mathcal {H}}_{1}\) such that \(u^{nm} = \sum_{l=L}^{L} a^{nm}_{l} e_{l}\), and \((\cdot )^{\mathsf {H}}\) represents conjugate transpose, and let
where \(d^{nm}_{l}\) represents the coefficient of \((\mathcal{P}_{1}^{o} v^{nm})\) w.r.t. basis element \(e^{o}_{l}\) in \({\mathcal {H}}^{o}_{1}\) such that \((\mathcal{P}_{1}^{o} v^{nm}) = \sum_{l=L^{o}}^{L^{o}} d^{nm}_{l} e^{o}_{l}\).
Finally, we define the \((4L+2L^{o}+4)\times 1\) vector
and \((4L+2L^{o}+3)\times (2L+2L^{o}+2)\) matrix
Finally, we define the \((4L+2L^{o}+4)\times 1\) vector
and the \((4L+2L^{o}+3)\times (2L+2L^{o}+2)\) matrix
The proof is based on two simple observations. First,
and therefore
Second,
and, therefore,
Appendix 4: Proof of Lemma 3
For the mth trial, the output of the nth channel, by multiplying by the denominator of (20) with \(v^{nm}\) sampled at times \(t_{k}\), we obtain
Finally, with \(v^{nm}(t_{k}) = q^{nmk}\) and the notation in (44), we get
and, therefore,
for all \(m=1,2,\ldots,M\), \(k+1,2,\ldots,T\), and \(i,j,n=1,2,\ldots,N\).
Appendix 5: Proof of Lemma 4
and the \((2L^{o}+1)\times 1\) vector
and the \((2L^{o}+1)\times (2L^{o}+1)\) matrix
Note that \(\mathbf{H}^{ij4}_{2}, i,j = 1,2,\ldots ,N\), are not necessarily Hermitian matrices. Further, by abuse of notation, let the \((2L+1)\times 1\) vector
where \(a^{nm}_{l}\) represents the coefficient of \(u^{nm}(t)\) w.r.t. basis element \(e_{l}\) in \({\mathcal {H}}_{1}\) such that \(u^{nm}(t) = \sum_{l=L}^{L} a^{nm}_{l} e_{l}\), and let the \((2L^{o}+1)\times 1\) vector
where \(d^{nm}_{l}\) represents the coefficient of \((\mathcal{P}_{1}^{o} v^{nm})\) w.r.t. basis element \(e^{o}_{l}\) in \({\mathcal {H}}^{o}_{1}\) such that \((\mathcal{P}_{1}^{o} v^{nm}) = \sum_{l=L^{o}}^{L^{o}} d^{nm}_{l} e^{o}_{l}\).
Additionally, we define the \((1+2(2L+1)+(N+1)(2L^{o}+1) )\times 1\) vector
and the \((2(2L+1)+(N^{2}+1)(2L^{o}+1) )\times (2L+1+2L^{o}+1 )\) matrix
Finally, we define the \((1+2(2L+1)+(N+1)(2L^{o}+1) )\times 1\) vector
and the \((2(2L+1)+(N^{2}+1)(2L^{o}+1) )\times (2L+1+2L^{o}+1 )\) matrix
With the notation above, the proof follows the same steps as the proof of Lemma 2.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Lazar, A.A., Ukani, N.H. & Zhou, Y. Sparse identification of contrast gain control in the fruit fly photoreceptor and amacrine cell layer. J. Math. Neurosc. 10, 3 (2020). https://doi.org/10.1186/s1340802000805
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s1340802000805