 Research
 Open Access
 Published:
Mesoscopic population equations for spiking neural networks with synaptic shortterm plasticity
The Journal of Mathematical Neuroscience volume 10, Article number: 5 (2020)
Abstract
Coarsegraining microscopic models of biological neural networks to obtain mesoscopic models of neural activities is an essential step towards multiscale models of the brain. Here, we extend a recent theory for mesoscopic population dynamics with static synapses to the case of dynamic synapses exhibiting shortterm plasticity (STP). The extended theory offers an approximate meanfield dynamics for the synaptic input currents arising from populations of spiking neurons and synapses undergoing Tsodyks–Markram STP. The approximate meanfield dynamics accounts for both finite number of synapses and correlation between the two synaptic variables of the model (utilization and available resources) and its numerical implementation is simple. Comparisons with Monte Carlo simulations of the microscopic model show that in both feedforward and recurrent networks, the mesoscopic meanfield model accurately reproduces the first and secondorder statistics of the total synaptic input into a postsynaptic neuron and accounts for stochastic switches between Up and Down states and for population spikes. The extended mesoscopic population theory of spiking neural networks with STP may be useful for a systematic reduction of detailed biophysical models of cortical microcircuits to numerically efficient and mathematically tractable meanfield models.
Introduction
One of the primary goals in computational neuroscience is to understand how brain functions arise from the interactions of billions of nerve cells and their underlying biophysical processes at the microscopic scale. Towards that goal, a crucial step is to develop a theoretical framework that links biophysically detailed networks of spiking neurons at the microscopic scale with simplified firingrate or neuralmass models [1] for neuronal populations at the coarsegrained mesoscopic or macroscopic scale. Firingrate models are mathematically tractable and thus permit a theoretical understanding of neuronal population dynamics implicated in various neural computations [2–6]. However, firing rate models are heuristic models that lack a clear link to the underlying microscopic properties. On the other hand, highly detailed biophysical models of cortical microcircuits [7] and simplified networks of spiking point neurons [8–10] are closely linked to biophysical properties but lack mathematical tractability and do not provide a mechanistic understanding of emergent functional behavior. However, if we were able to systematically reduce biophysically detailed models to simplified networks of spiking point neurons [11] and further to coarsegrained firingrate models [12], we might be able to understand neural computations at the population level in terms of biophysical parameters. One common strategy for coarsegraining a cortical network is to identify approximately homogeneous neuronal populations that consist of cells with similar properties and inputs; e.g., neurons may be grouped with respect to their cell type and cortical location [12, 13]. Biological data suggests that the size of such neuronal populations are typically small containing only on the order of hundred to thousand of neurons [14]. Recently, a mesoscopic meanfield theory that accounts for finitesize noise has been proposed [12] based on the refractory density equations for macroscopic homogeneous populations of neurons (where the number of neurons tends to infinity) [15–18]. However, in [12], synapses are assumed to be static in the sense that the effective synaptic coupling between two populations of neurons is constant over time.
A ubiquitous feature of cortical dynamics is synaptic shortterm plasticity (STP) [19–22], i.e. dynamic changes of synaptic strength on time scales of 100 ms to 1000 ms induced by presynaptic neural activity. Theoretical studies have shown that STP exerts profound effects on network activity [23–25] and information processing capabilities [19, 26–29]. In particular, in a recent biophysically detailed microcircuit model [7], STP has been a critical factor for reproducing experimentally observed activity patterns. Therefore, a faithful reduction to population rate models should incorporate the effect of STP. Meanfield descriptions for populations of spiking neurons with dynamic synapses are central for such a reduction. Although meanfield theories for STP have been developed for the case of macroscopic populations [4, 30–32], a mesoscopic meanfield theory that would account for finitesize fluctuations is still lacking.
In this work, we extend the mesoscopic theory of [12] for static synapses to the case of dynamic synapses exhibiting Tsodyks–Markram STP [30]. In Sect. 2 we expose our theory by considering a feedforward setup [27, 33]. We use an assumption (loosely speaking a Poissonian assumption on the spike statistics), Assumption 1, which enables the derivation of mesoscopic meanfield dynamics for the effective input. We then compare numerically the effective input given by the mesoscopic meanfield dynamics with simulations of the full microscopic population, in the case where the presynaptic population consists of N Poisson neurons. The Poisson case is of special interest because Assumption 1 is satisfied. In Sect. 3, we first explain how the theory of Sect. 2 can be applied to general mesoscopic circuits. We then illustrate how the mesoscopic STP model accurately replicates population spikes and switches between Up and Down states exhibited by a recurrent network of timeinhomogeneous Poisson neurons. Finally, we incorporate the mesoscopic STP model into our previous mesoscopic population model [12] for generalized integrate and fire (GIF) neurons. We show that the resulting extension faithfully reproduces population spikes observed in a microscopic simulation. In Sect. 4, we discuss the limitations of our mesoscopic model for GIF neurons and mention possible theoretical extensions. The numerical implementation of the mesoscopic equations are detailed in the Appendix.
Feedforward network with a finite number of dynamic synapses
Network setting and theoretical approach
To derive the mesoscopic theory, let us consider a feedforward setup: N neurons from a presynaptic population are connected to a given postsynaptic neuron via N synapses (Fig. 1(A)). This setup is important because under the approximations we will use, the theory obtained for this simple case can be directly applied to general mesoscopic circuits. In addition, the setup is important for biological modeling because feedforward pathways exhibiting STP are prominent in the nervous system. Examples include visual [19], auditory [34], somatosensory [35] and periform [36] cortices.
Before we start with the treatment of dynamic synapses, it is instructive to first recall the simpler case of static synapses [12]. In this case, the synaptic input current (in a currentbased model) \(I_{\mathrm{syn}}^{N}(t)\), \(t>t_{0}\), is modeled microscopically as the sum of synapticallyfiltered spike trains:
Here, J is the synaptic weight (in units of electrical charge) assumed to be identical for all synapses onto the postsynaptic neuron, \(\epsilon (t)\) is a synaptic filtering kernel (defined as the postsynaptic current normalized by the total charge elicited by a single spike) and
is the Diracdelta spike train of neuron j with spike times \(\{t_{k}^{j}\}_{k\in \mathbb{Z}^{+}}\), \(t_{0}< t_{1}^{j}< t_{2}^{j}<\cdots <\infty \). Here and in the following, a superscript N (as in \(I_{\mathrm{syn}}^{N}\)) denotes a functional of the N presynaptic spike trains \(s_{1}(t), \ldots , s_{N}(t)\), and hence a mesoscopic quantity.
In mesoscopic models of homogeneous neuronal populations, the central mesoscopic variable is the population activity \(A^{N}(t)\) defined as the sum of spike trains of all neurons in a population divided by the number of neurons. In our case, the mesoscopic activity of the presynaptic population is thus given by
Using the mesoscopic activity of the presynaptic population, the synaptic input current, Eq. (1), can be rewritten as
Thus, for static synapses, the synaptic input current is completely determined by the past population activities \(\{A^{N}(t')\}_{t'< t}\). This property is crucial for mesoscopic population models that are formulated in terms of mesoscopic population activities [12, 37]. In particular, in simulations of such mesoscopic models forward in time, the information about past population activities is available at each point in time and can thus be used to compute the input current at present time. In our approach, we thus aim at finding a dynamics of the synaptic input conditioned on the history of the population activity \(A^{N}(t)\). We would like to stress that this aim is markedly different from wellknown diffusion approximations [38, 39] (see also [40, 41] for examples in neuroscience), where a jump process is approximated, when jumps become frequent and small, by a diffusion process. In particular, a diffusion approximation would yield a stochastic dynamics if conditioned on the past population activities, in stark contrast to the deterministic conditional dynamics, Eq. (4).
Finding a deterministic relationship between the synaptic input and a given realization \(A^{N}(t)\), as in Eq. (4), is no longer possible in the case of dynamic synapses. In this case, the input spike trains \(s_{j}(t)\) are modulated by a dynamic factor \(R_{j}(t)\) modeling the effect of STP. In contrast to static synapses, Eq. (1), the synaptic current for dynamic synapses reads
Inverting the sum and the integral we get
where we introduced the total postsynaptic input (TPSI) \(y^{N}(t)\). Equation (6) shows that determining the synaptic input \(I_{\mathrm{syn}}^{N}(t)\) from the knowledge of the population activity \(A^{N}(t)\) is an underconstrained problem because \(y^{N}(t)\) is a weighted average of spike trains whereas \(A^{N}(t)\) is an unweighted average. Thus, in mesoscopic simulations, we expect that the synaptic input is strongly but not fully constrained by the knowledge of past population activities. To capture STP in a mesoscopic model, our approach is to find an approximate meanfield dynamics for \(y^{N}(t)\) by introducing additional mesoscopic variables that are driven by the population activity and noise that accounts for the inevitable uncertainty of \(y^{N}(t)\) given the history of the population activity. Importantly, such meanfield dynamics would be mesoscopic in the sense that its simulation does not require simulating all the individual presynaptic neurons and synapses but only a few mesoscopic variables. In addition, for such an approximation to be useful for coarsegraining, its numerical implementation should be computationally economical compared with the simulation of the full microscopic model.
Now we fully specify the dynamics of \(y^{N}(t)\) as defined Eq. (6) (the microscopic model) in Sect. 2.2 and derive its mesoscopic approximation in Sect. 2.3.
Microscopic model
In order to fully specify the dynamics of \(y^{N}(t)\) defined in Eq. (6) given the collection of spike trains \(\{s_{j}\}_{j=1, \ldots , N}\), we need to define the dynamics of \(\{R_{j}\}_{j=1, \ldots , N}\), which modulates the amplitude of the spikes in \(s_{j}(t)\). For each synapse j, the time evolution of \(R_{j}\) is deterministic given \(s_{j}\) and follows the Tsodyks–Markram model for STP [30]. The modulation factor \(R_{j}(t)\) can be seen as the amount of neurotransmitter that would be released if a spike occurs at time t. It is given by the product of two synaptic variables, \(R_{j}(t)=u_{j}(t^{}) x_{j}(t^{})\), where \(x_{j}(t)\) is the amount of available synaptic resources and \(u_{j}(t)\) is the utilization of available resources (i.e. the fraction of these available resources that would be released if a spike occurs) at synapse j. Given the presynaptic spike trains \(s_{j}(t)\), these variables obey the dynamics
where \(\tau _{\mathrm {F}}\) and \(\tau _{\mathrm {D}}\) are the facilitation and depression time constants, respectively, \(U_{0}\) is the baseline utilization of synaptic resources, U determines the increase in the utilization of synaptic resources by a spike. Here, \(u_{j}(t^{})\) is a shorthand for the left limit at time t. Note that the presence of the product \(u_{j} x_{j}\) in Eq. (7b) introduces a nonlinear coupling between the dynamics of \(u_{j}\) and \(x_{j}\). Having specified the STP dynamics, we can rewrite the TPSI, Eq. (6), as follows:
In the next section, we present our main result, which provides a meanfield approximation \(y(t)\) that is determined by the history of the mesoscopic presynaptic population activity \(A^{N}(t)\) rather than individual presynaptic spike trains \(s_{j}(t)\) or synaptic variables \(u_{j}\) and \(x_{j}\).
Mesoscopic approximation
To relate the TPSI, Eq. (8), to the presynaptic population activity we consider the collection \(\{t_{k}\}_{k\in \mathbb{Z}^{+}}\) of all spike times of the superposition spike train \(S^{N}(t):=NA^{N}(t)=\sum_{j} s_{j}\). At the mesoscopic scale, we only know that there is a presynaptic spike arriving at time \(t_{k}\) but we do not know at which synapse. That is, we do not know the mapping \(j(k)\) that maps spike arrival times \(t_{k}\) to specific synapses j. In terms of the spike times \(t_{k}\) and the mapping \(j(k)\), the population activity and the TPSI can be rewritten as
and
respectively. Equation (10) still contains the microscopic variables \(u_{j}\) and \(x_{j}\) and thus the precise spike arrival times \(t_{k}^{j}\) at synapse j. To derive a meanfield approximation \(y(t)\) that only depends on the spike times \(\{t_{k}\}_{k\in \mathbb{Z}_{+}}\) of the superposition spike train \(S^{N}(t)\), we make two approximation steps: (i) a randomization of synapse indices \(j(k)\) at each spike time \(t_{k}\) and (ii) a Gaussian approximation concerning the variables \(u_{j(k)}(t)\) and \(x_{j(k)}(t)\) determining the “amplitudes” of the delta spikes in Eq. (10). The purpose of the first approximation step is a probabilistic description of the values \(u_{j(k)}(t_{k}^{})\) and \(x_{j(k)}(t_{k}^{})\) upon spiking that accounts for the lack of knowledge about synapse identities \(j(k)\) at spike times \(t_{k}\). This step rests on an assumption on the law of \(\{s_{j}\}_{j=1, \ldots , N}\). In feedforward models with biologically realistic spike train statistics or in recurrent networks of spiking neurons, this assumption has to be understood as a useful approximation as it is not satisfied in general.
Assumption 1
The set of spike trains \(\{s_{j}\}_{j=1, \ldots , N}\) has the same law as a set of spike trains \(\{s^{*}_{j^{*}}\}_{j^{*}=1, \ldots , N}\) that is constructed by picking independently and uniformly, for each spike time \(\{t_{k}\}_{k \in \mathbb{Z}^{+}}\) of the superposition spike train \(S^{N}(t)\), an integer \(j^{*}(k)\) between 1 and N and assigning a spike to neuron \(j^{*}(k)\) at time \(t_{k}\).
An important case for which Assumption 1 is satisfied is the case where \(\{s_{j}\}_{j=1, \ldots , N}\) is given by N independent Poisson processes with the same (timevarying) intensity. This special case will be extensively used in numerical simulations. Note that Assumption 1 is strictly weaker than assuming that the spike trains of the neurons are given by independent Poisson processes. Here is a simple counterexample: consider the process where at each time \(t_{k}=t_{0}+k\hat{\tau }\) for \(k \in \mathbb{Z}_{+}\) and \(\hat{\tau }>0\), a spike occurs at neuron \(j_{k}\), \(j_{k}\) being chosen randomly, uniformly and independently at each time \(t_{k}\). This process satisfies Assumption 1 but is clearly not Poisson.
Assumption 1 will be useful in the following because it will enable us to replace the unknown synapse \(j(k)\) at some spike time \(t_{k}\) by a randomly chosen synapse \(j^{*}(k)\). This randomization of synapse indices yields a new set of presynaptic spike trains \(\{s^{*}_{j}\}_{j=1, \ldots , N}\) whose population activity \(A^{*}(t):=\frac{1}{N}\sum_{j=1}^{N}s_{j}^{*}(t)\) is identical to the population activity \(A^{N}(t)\) and which by assumption is statistically equivalent to the original spike trains \(\{s_{j}\}_{j=1, \ldots , N}\). Furthermore, the spike trains \(\{s^{*}_{j}\}_{j=1, \ldots , N}\) induce new STP variables \(u_{j}^{*}\) and \(x_{j}^{*}\) that obey, analog to Eqs. (7a)–(7b), the dynamics
with initial conditions \(u_{j}^{*}(t_{0})=u_{j}(t_{0})\), \(x_{j}^{*}(t_{0})=x_{j}(t_{0})\). Assumption 1 implies that the jump processes \([u_{j}^{*}(t),x_{j}^{*}(t)]\), \(j=1,\ldots ,N\), have the same law as the original STP variables \([u_{j}(t),x_{j}(t)]\), \(j=1,\ldots ,N\).
Using the random indices \(j^{*}(k)\), we define a twodimensional càdlàg jump process \(\{(\hat{u}^{*}(t), \hat{x}^{*}(t))\}_{t >t_{0}}\): For all \(k \in \mathbb{Z}^{+}\), if \(t \in [t_{k}, t_{k+1}[\) we set
i.e. \(\hat{u}^{*}(t)\) and \(\hat{u}^{*}(t)\) are equal to \(u_{j}^{*}(t)\) and \(x_{j}^{*}(t)\) of that neuron j which had the most recent spike prior or equal to time t. In the following, the process \(\{(\hat{u}^{*}(t), \hat{x}^{*}(t))\}_{t>t_{0}}\) will appear in the STP dynamics only as prefactors of Diracdelta spikes. Therefore, only the values \((\hat{u}^{*}(t_{k}), \hat{x}^{*}(t_{k}))\) at times \(t_{k}\) will enter the dynamics. The formulation at all time \(t>t_{0}\), Eq. (12), will allow us to factorize modulated spike trains into modulation factor and population activity. For example, using the process \((\hat{u}^{*}(t), \hat{x}^{*}(t))\) we can define the new quantity
where the second equality in Eq. (13) follows from the properties of the Diracdelta function. In the last equation, we have introduced the shorthand notation \(\hat{u}^{*}:=\hat{u}^{*}(t)\) and \(\hat{x}^{*}:=\hat{x}^{*}(t)\). By construction of the processes \(\hat{u}^{*}(t)\) and \(\hat{x}^{*}(t)\), we can state an important fact.
Fact 1
Under Assumption 1, the processes\(y^{N}(t)\)and\(y^{*}(t)\)have the same law. Moreover, conditioned on the population activity\(A^{N}(t)\), both processes have the same spike times.
Therefore, we expect that \(y^{*}(t)\) yields an accurate approximation of the TPSI \(y^{N}(t)\) that is not only statistically equivalent but also captures the precise realization of spike times of the mesoscopic activity \(A^{N}(t)\). However, \(y^{*}(t)\) is not yet useful for a mesoscopic simulation because the process \((\hat{u}^{*}(t), \hat{x}^{*}(t))\) is constructed from the microscopic variables \(u_{j}^{*}(t)\) and \(x_{j}^{*}(t)\), \(j=1,\ldots ,N\), which still need to be simulated individually.
In order to obtain an approximation that does not rely on microscopic simulations, we introduce new mesoscopic variables given by the empirical means and covariances of \(u_{j}(t)\) and \(x_{j}(t)\):
Similar to \(y^{N}\), we can find an accurate approximation for the new mesoscopic variables using analog definitions based on the asterisk (^{∗}) variables:
Because \([u_{j},x_{j}]\) and \([u_{j}^{*},x_{j}^{*}]\) have the same law, we can state the following.
Fact 2
The processes\([u^{*}, x^{*}, P^{*}, Q^{*}, R^{*}]\)and\([u^{N},x^{N},P^{N},Q^{N},R^{N}]\)are multidimensional jump processes that have the same law. Moreover, conditioned on the population activity\(A^{N}(t)\), the two jump processes have the same jump times.
Lemma 2.1
The jump process\([u^{*}, x^{*}, P^{*}, Q^{*}, R^{*}]\)obeys the dynamics
with initial conditions\(u^{*}(t_{0})=u^{N}(t_{0})\), \(x^{*}(t_{0})=x^{N}(t_{0})\), \(P^{*}(t_{0})=P^{N}(t_{0})\), \(Q^{*}(t_{0})=Q^{N}(t_{0})\), \(R^{*}(t_{0})=R^{N}(t_{0})\).
The proof is presented in Appendix A.
So far, we obtained approximations \(y^{*}\) and \([u^{*},x^{*},P^{*},Q^{*},R^{*}]\), in which \(\hat{u}^{*}\) and \(\hat{x}^{*}\) are chosen randomly at spike times so as to account for the missing information about synapse identities at the microscopic scale. However, the process \([u^{*}(t),x^{*}(t),P^{*}(t),Q^{*}(t),R^{*}(t)]\) is not yet useful for a mesoscopic simulation because the processes \(\hat{u}^{*}(t)\) and \(\hat{x}^{*}(t)\) require the simulation of the microscopic variables \(u_{j}^{*}(t)\) and \(x_{j}^{*}(t)\), \(j=1,\ldots ,N\). To resolve this problem, we make a Gaussian approximation of the twodimensional random variables \((\hat{u}^{*}(t_{k}), \hat{x}^{*}(t_{k}))\) that is solely based on mesoscopic variables. We note again that only the values \((\hat{u}^{*}(t_{k}), \hat{x}^{*}(t_{k}))\) at spike times matter for the STP dynamics marked with asterisks, Eqs. (14) and (17a)–(17e), because they only appear as prefactors of Dirac delta spikes. For a Gaussian approximation of \([\hat{u}^{*}(t_{k}), \hat{x}^{*}(t_{k})]\), we need its first two empirical moments given by Eqs. (16a)–(16c). This means that at each spike \(t_{k}\) of \(A^{N}(t)\), a sample \([\hat{u}(t_{k}), \hat{x}(t_{k})]\) is drawn from a bivariate Gaussian distribution parametrized by \([u^{*}(t_{k}^{}),x^{*}(t_{k}^{}),P^{*}(t_{k}^{}),Q^{*}(t_{k}^{}),R^{*}(t_{k}^{})]\). To close the system, we also need to approximate \([u^{*},x^{*},P^{*},Q^{*},R^{*}]\) by new variables \([u,x,P,Q,R]\) that obey Eqs. (17a)–(17e) but in which \([\hat{u}^{*}(t_{k}), \hat{x}^{*}(t_{k})]\) is replaced by its Gaussian approximation \([\hat{u}(t_{k}), \hat{x}(t_{k})]\). Likewise, we use \([\hat{u}(t_{k}), \hat{x}(t_{k})]\) in Eq. (14) to approximate the TPSI \(y^{*}\) by a variable y. Thus, we obtain our main result.
Secondorder meanfield approximation
Let\(\{t_{k}\}_{k \in \mathbb{Z}^{+}}\)be the spike times of superposition spike train\(S=NA^{N}\). Consider the process\((u(t), x(t), P(t), Q(t), R(t), \hat{u}(t), \hat{x}(t))\)defined by the system of equations
where at each spike time\(\{t_{k}\}_{k \in \mathbb{Z}^{+}}\), a random sample\([\hat{u}(t_{k}), \hat{x}(t_{k})]\)is drawn from the twodimensional Gaussian distribution parametrized by\([u,x,P,Q,R]\):
and for the irrelevant values between spikes we arbitrarily set\((\hat{u}(t), \hat{x}(t)) = (\hat{u}(t_{k}), \hat{x}(t_{k}))\)if\(t \in [t_{k}, t_{k+1}[\)for all\(k \in \mathbb{Z}^{+}\). The mesoscopic meanfield approximation of the synaptic input is
Remarks on the approximation

(1)
Various initial conditions are possible for the mesoscopic dynamics. One reasonable choice is \(u(t_{0})=0\), \(x(t_{0})=1\), \(P(t_{0})=Q(t_{0})=R(t_{0})=0\).

(2)
The approximation is mesoscopic because the process \((u, x, P, Q, R, \hat{u}, \hat{x})\) defined by Eqs. (18a)–(18e) and (18f) does not involve any microscopic simulations. The process is solely driven by the mesoscopic population activity \(A^{N}(t)\). The key heuristic we use is the Gaussian approximation of the random variables \(u_{j^{*}(k)}(t_{k}^{})\) and \(x_{j^{*}(k)}(t_{k}^{})\) at spike times \(t_{k}\).

(3)
In principle, the jump process, Eqs. (18a)–(18g), can be simulated by a discretetime forward scheme, where at each spike time \(t_{k}\) a bivariate Gaussian random number is drawn. In practice, however, such a simulation would be inefficient because for large N the rate of spike times \(t_{k}\) is large and thus the discretization time step needs to be chosen extremely small in order to resolve each spike time. In Appendix B, we present an efficient simulation scheme that allows many spikes per time step.

(4)
The process \((u, x, P, Q, R, \hat{u}, \hat{x})\) associated with the mesoscopic approximation is well defined. First, as the number of neurons N is finite, we can safely assume that the number of spike in the population is finite on finite time intervals \(a.s\). (all spiking neuron model have a finite number of spikes on finite time interval \(a.s\).). Between spikes, the evolution of the process is deterministic and easy to see. The more tricky increments are at spike times \(\{t_{k}\}_{k \in \mathbb{Z}^{+}}\). At spike time \(t_{k}\), we first draw a random sample \((\hat{u}(t_{k}), \hat{x}(t_{k}))\) according to Eq. (18f). Then we update û and x̂. Finally, we update u, x, P, Q and R according to Eqs. (18a)–(18g). Thus, \((u, x, P, Q, R, \hat{u}, \hat{x})\) is a jump process with random jump sizes, even when conditioned on \(A^{N}\).

(5)
It is plausible that the second approximation step, in which the process \([y^{*},u^{*},x^{*},P^{*},Q^{*},R^{*},\hat{u}^{*},\hat{x}^{*}]\) is approximated by Eqs. (18a)–(18g), becomes valid for sufficiently large N. Here is a heuristic argument: \(I_{\mathrm{syn}}^{N}(t)\) is a convolution of \(y_{N}(t)\) with \(\epsilon (t)\) (Eq. (6)). Assuming that ϵ is smooth and bounded, \(I_{\mathrm{syn}}^{N}(t)\) is determined by the small time step integrals \(\int _{t'}^{t'+\Delta t} y_{N}(t'')\,dt''\), for all \(t'< t\) and \(\Delta t > 0\) arbitrarily small. If the number of spikes in the interval \([t', t' + \Delta t]\) grows linearly with N, and if Δt is small enough, under Assumption 1, we can see (heuristically) \(\int _{t'}^{t'+\Delta t} y_{N}(t'')\,dt''\) as the sum of αN (α is some constant) \(i.i.d\) random jumps scaled by \(1/N\). Hence, by a central limit theorem type of argument, \(\int _{t'}^{t'+\Delta t} y_{N}(t'')\,dt''\) only depends on the mean and the variance of the random jumps. In other words, even if the \([\hat{u}^{*}, \hat{x}^{*}]\) do not become Gaussian when N is large, Eqs. (18a)–(18g) should become accurate when N is sufficiently large because what matters (according to the above heuristic argument) is that \([\hat{u}, \hat{x}]\) has the right mean and variance. A rigorous proof of this heuristic argument would be of mathematical interest but goes beyond the scope of the present work.

(6)
By construction, the process \(\{y(t)\}_{t>t_{0}}\) can be easily conditioned on the process \(\{A^{N}(t)\}_{t>t_{0}}\): for any realization of the process \(A^{N}(t)\), there is a well defined conditioned process \(y_{A^{N}}(t)\). This is a very practical feature because it means that we have a well defined approximation for any \(A^{N}(t)\), stochastic or deterministic. Note that the process \(A^{N}(t)\) does not need Assumption 1 to be satisfied to be well defined. Hence, \(y_{A^{N}}(t)\) is well defined even if Assumption 1 is not satisfied. Furthermore, this conditioning feature allows us to generalize the current mesoscopic approximation for feedforward networks to general mesoscopic networks as it will be explained in Sect. 3.
Instead of the Gaussian approximation, Eq. (18f), it is tempting to consider a firstorder approximation, where \(\hat{u}^{*}\) and \(\hat{x}^{*}\) are approximated by the empirical means \(u^{*}\) and \(x^{*}\) neglecting their variance and covariance. Setting the covariance matrix in Eq. (18f) to zero yields the following.
Firstorder meanfield approximation
We have
This approximation is very similar to the classic meanfield equations derived for \(N\rightarrow \infty \) by [30] except that it is driven by a sum of spike trains \(A^{N}(t)=\frac{1}{N}\sum_{j=1}^{N} s_{j}(t)\) and not a continuous rate \(r(t)\).
In this paper, we name Eqs. (18a)–(18g) the secondorder meanfield theory (abbreviated secondorder MF) and Eqs. (19a)–(19c) the firstorder meanfield theory (abbreviated firstorder MF).
In the rest of this work, we compare numerically the more sophisticated secondorder MF to the simpler firstorder MF. In this section, we focus on the case where the presynaptic spike trains \(\{s_{j}\}_{j=1,\ldots ,N}\) are given by N independent Poisson processes with constant rate because in this case, Assumption 1 is satisfied. In Appendix B, we provide an efficient simulation algorithm for Eqs. (18a)–(18g).
Trajectories of \(u_{j}\), \(x_{j}\) and \(R_{j}\) as well as u, x and R obtained from a microscopic simulation are shown in Fig. 2. The mesoscopic variable R is tracked by the secondorder MF dynamics with high accuracy (Fig. 2(b4)). The firstorder MF also yields reasonable results, although a small deviation in the mean R over time is apparent (Fig. 2(b4)). This is consistent with previous findings [30], where it has been shown analytically that in the stationary case, relative correlations are small but significant. Note that the secondorder MF distinguishes two sources of finitesize noise: noise that comes from the finitesize fluctuations of \(A^{N}\) and second, noise that comes from the sampling of û and x̂ at each spike. This second source of noise is absent in the firstorder MF. In a numerical simulation with time step Δt, it is possible to isolate this second source of noise: if \(N \cdot \Delta t \cdot r\) (where r is the rate of the Poisson process) is a strictly positive integer α, we can choose, independently at each time step, α neurons uniformly across the N neurons and make them spike. This procedure generates N discretized Poisson spike trains of rate r with a “constant” population activity \(A^{N}\) over time. Here, “constant” means that the timediscretized population activity \(A^{N}(t)=n(t)/(N\Delta t)\) in the numerical simulation with time step Δt is constant, i.e. the total number of spikes in \([t,t+\Delta t)\) is fixed. Note that in this case, the spike trains are not independent of each other but this does not affect our derivation. This procedure is followed in Fig. 2(c1–4): the firstorder MF predicts noiseless STP dynamics whereas the secondorder MF accurately reproduces the residual finitesize fluctuations.
The deviations of the firstorder MF become more pronounced during nonstationary transients caused by stepwise increases of the rate of the Poisson process (Fig. 3). The response to step increases accurately traced by the secondorder MF, but not by the firstorder MF which neglects the correlations between \(u_{j}\) and \(x_{j}\).
Statistics of the total postsynaptic input
To compare microscopic and mesoscopic descriptions more systematically, we measured the first and secondorder statistics from simulations for varying parameters. At first, we computed the mean of the modulation factor R for the stationary process (\(\langle s_{j}(t) \rangle = r = \text{const.}\)) using the microscopic dynamics,
where \(\langle \cdot \rangle \) denotes the ensemble (trial) average, i.e. the average over realizations of the Poisson processes \(s_{j}(t)\). The mean TPSI is proportional to the mean modulation factor because \(\langle y\rangle =\frac{1}{N}\sum_{j}\langle u_{j}x_{j}\rangle \langle s_{j}\rangle =\langle R\rangle r\). This simple proportionality follows from the fact that \(u_{j}(t)x_{j}(t)\) at time t is uncorrelated with \(s_{j}(t)\) at the same time because of the Poisson statistic of the spike train and the update of variables after a spike; cf. Eqs. (7a) and (7b). As known from previous work [30], the mean modulation \(\langle R \rangle \) of a facilitating synapse increases with increasing firing rate of presynaptic neurons when firing rates are small, and decreases again at high rates due to depression (Fig. 4(C)). The firstorder MF shows small deviations in the mean modulation \(\langle R\rangle \), which are removed by the secondorder MF (Fig. 4(C)). A closer inspection of the full parameter regime reveals that the deviation of firstorder MF never exceeds 5% (Fig. 4(A)). Therefore the stationary mean TPSI is sufficiently well explained by the firstorder MF.
Also, we compared the statistics of fluctuations of the TPSI by measuring the respective power spectral densities (PSD). The PSD can be computed as
where \(\tilde{y}(f)=\int _{0}^{T}dt \exp (2\pi ift)y(t)\) denotes the Fourier transform of \(y(t)\) for a finite but large enough time window T. We found that the secondorder MF significantly better captured the variance (Fig. 4(D)) and the PSD (Fig. 5) of the stationary fluctuations than the firstorder MF. A closer inspection of the coefficient of variation of the fluctuations, \(\sqrt{\operatorname{Var} (y )}/\langle y\rangle \), over the full parameter space revealed that the firstorder MF deviated up to 30% (especially for slow synaptic dynamics or high rates, Fig. 5(ii), (iv)), whereas the secondorder model performed well in the whole parameter space (Fig. 4(B)). We should specify that the error of the firstorder MF is negative, i.e. the firstorder MF underestimates the coefficient of variation up to 30%. This comes from the fact that in the mesoscopic equations for the firstorder MF equations (19a)–(19c), finitesize fluctuations of the mesoscopic variables u and x are ignored.
In conclusion, while mean responses for stationary cases are well captured by the firstorder MF, the secondorder MF gives a significantly better description of transient responses (Fig. 3) and fluctuations (Figs. 4(B), (D) and 5).
Recurrent network with STP
Microscopic model
As shown in our previous work [12], networks of multiple interacting homogeneous populations of spiking neurons, with static synapses, can be accurately predicted with a mesoscopic model. Incorporating the effect of STP in this general model using the mesoscopic approximations of Sect. 2.3 for the feedforward case is actually easy. This is due to the fact the mesoscopic approximation \(y(t)\) (both the firstorder MF and the secondorder MF) can be conditioned on the population activity \(A^{N}(t)\), as mentioned in Sect. 2.3. In order to illustrate this, we consider for simplicity the special case of a single population with recurrent connectivity. The network architecture is random with fixed indegree \(C=pN\), where N is the number of neurons in the population and p is the connection probability. The synaptic strength is constant with magnitude w (in mV). The TPSI \(y_{i}(t)\) and the synaptic current \(I_{\mathrm{syn},i}(t)\) driving the postsynaptic neuron i are related by Eq. (1) with \(J=\frac{\tau _{\mathrm {m}}}{R_{\mathrm{m}}}Npw\). In this paper, we use a synaptic filtering kernel with instantaneous rise and exponential decay corresponding to the firstorder kinetics
where \(\tau _{s}\) is the synaptic filtering time constant. Importantly, the effect of STP is contained in the TPSI \(y_{i}(t)=\frac{1}{C}\sum_{j\in \varGamma _{i}}u_{j}(t)x_{j}(t)s_{j}(t)\) via the synaptic variables \(u_{j}\) and \(x_{j}\) given by the dynamics Eq. (7a) and (7b). Here, \(\varGamma _{i}\) denotes the index set of presynaptic neurons that connect to neuron i.
As our derivation of the mesoscopic theory of STP uses the assumption that neurons have Poisson statistics, we first apply our theory to Poisson rate neurons, which do not exhibit a dependence on spike history. Then, using the same setup, the theory will be applied to a network of generalized integrateandfire (GIF) neurons with pronounced spikehistory effects.
Poisson rate model
The Poisson rate model is defined by a continuous variable \(h_{i}(t)\), called input potential. The input potential of neuron i obeys the dynamics
where \(\tau _{\mathrm {m}}\) represents the membrane time constant and \(\mu (t)=V_{\mathrm{rest}}+R_{\mathrm{m}} I_{\mathrm{ext}}(t)\) is the total drive in the absence of synaptic input (constant resting potential \(V_{\mathrm{rest}}\) and common external stimulus \(I_{\mathrm{ext}}(t)\)). In the fullyconnected network, the synaptic current \(I_{\mathrm{syn},i}(t)\) is the same for all neurons i and is given by Eq. (22).
In each time interval \([t,t+dt)\), spikes are drawn with probability \(\lambda _{i}(t)\,dt\), where the firing rate \(\lambda _{i}(t)\) depends on the input potential as follows:
Here, we choose a sigmoidal transfer function of the form \(\varPhi (h)=r_{\mathrm{max}}/ [1+\exp (\beta (hh_{0})) ]\) and \(h_{0} = 0\mbox{ mV}\).
GIF model
The GIF model for the postsynaptic neuron dynamics is determined by the membrane potential \(V_{i}(t)\), the dynamic threshold \(\vartheta _{i}(t)\) and a conditional intensity \(\lambda _{i}(t)\) for the stochastic spike generating mechanism. Here, the index \(i=1,\ldots ,N\) represents the neuron label. Between the spikes, the membrane potential satisfies the dynamics
where the quantities \(\tau _{\mathrm {m}}\), \(\mu (t)\), \(R_{\mathrm{m}}\) and \(I_{\mathrm{syn},i}(t)\) are the same as in the rate model above.
After a spike, the voltage is immediately reset to the potential \(V_{\mathrm {r}}\), where it is clamped for an absolute refractory period \(t_{\mathrm {ref}}=4\text{ ms}\).
Spikes are generated by a conditional intensity (hazard rate) of an exponential form:
where \(V_{\mathrm {th}}\) is a constant. This means that the conditional intensity, and hence the probability \(\lambda _{i}(t)\,dt\) to fire in the interval \([t,t+dt)\), depends on the momentary distance between the membrane potential and threshold. This completes the definition of the microscopic model.
Mesoscopic meanfield model
As explained in [12], the random connectivity can be well approximated by a fully connected network (\(C=N\)), with rescaled synaptic weights pw, corresponding to a meanfield approximation. In the following, we shall therefore choose \(p=1\) unless stated otherwise. In the meanfield approximation, the TPSI \(y(t)\) and the synaptic current \(I_{\mathrm{syn}}(t)\) do not depend on the identities j of the postsynaptic neurons and are related by Eq. (1) with \(J=\frac{\tau _{\mathrm {m}}}{R_{\mathrm{m}}}Npw\). For the case of exponential synapses, the synaptic current reads
where û and x̂ obey the meanfield equations (18a)–(18g).
As shown in [12], the population activity \(A^{N}(t)\) can be determined by a single mesoscopic variable, the instantaneous rate \(r(t)\), whose dynamics depends on the microscopic model and the history of the population activity \(\mathcal{H}_{t}=\{A(t')t'< t\}\). Specifically, \(A^{N}(t)\) is given by the normalized spike train
where \(n^{N}(t)\) is a counting process with (conditional) intensity \(\hat{\lambda }(t)=Nr(t)\) representing the total number of spikes in the population up to time t. The second equality means that \(A^{N}(t)\) is proportional to a spike train with spike times \(t_{k}\) generated with rate \(Nr(t)\). This is similar to the superposition of Poisson spike train in the feedforward case, Sect. 2, where the pooled spike train also exhibits the rate \(Nr(t)\).
Poisson rate model
In the Poisson rate model, the rate \(r(t)\) is given by
Importantly, the coupling to the STP dynamics is contained in the synaptic current \(I_{\mathrm{syn}}\) governed by Eq. (27) and the synaptic meanfield dynamics given by Eqs. (18a)–(18g).
GIF population model
For the model with spikehistory dependence, the rate \(r(t)\) is obtained from an integral over refractory states. A possible representation of the neuronal refractory state is given by the time τ since the last spike (“age of the neurons”; an alternative representation in terms of the last spike times \(\hat {t}=t\tau \) is given in Appendix D). In the following, we assume that the process is initialized at \(t_{0}=\infty \). Given the distribution of ages in the population, \(q(\tau ,t)\), the rate at time t results from [12],
where the density \(q(\tau ,t)\) evolves according to the firstorder partial differential equation with timedependent boundary condition at \(\tau =0\):
Here, \(A^{N}(t)\) is given by Eq. (28). In Eqs. (31) and (32), the functions λ and Λ are given by
where V and W are dynamical variables that obey the following dynamics: The agedependent membrane potential \(V(\tau ,t)\) and variance function \(W(\tau ,t)\) follow the firstorder partial differential equations
with boundary conditions \(V(t,0)=V_{\mathrm {r}}\) and \(W(t,0)=0\). The coupling to the STP meanfield dynamics, Eqs. (18a)–(18g), is contained in the synaptic current \(I_{\mathrm{syn}}\) governed by Eq. (27), which influences the voltage \(V(t, \tau )\) (Eq. (34)), and hence changes \(\lambda (t, \tau )\) and \(\varLambda (t)\).
The population equations (31)–(35) have been efficiently integrated numerically by the algorithm presented in [12]. The numerical integration of the STP meanfield dynamics is given in Appendix B.
Recurrent network of Poisson rate neurons—microscopic vs. mesoscopic simulations
Finitesize noise induced population spikes
An interesting example of collective neural dynamics, potentially linked to synaptic depression, is the phenomenon of population spikes in cultured neural networks [37]. We asked whether population spikes, a brief period of high average population activity, can be explained by our finitesize population theory with STP. As in previous work [37], we considered a single excitatory population endowed with STP. The mesoscopic meanfield equations allowed us to choose parameters of this model such that the macroscopic meanfield dynamics (\(N \rightarrow \infty \), see Eqs. (56a)–(56e)) is in an excitable regime for the secondorder MF but not for the firstorder MF. Here excitable regime means that the macroscopic dynamics converges to an equilibrium point if the total drive remains below a certain threshold. However, if the threshold is exceeded (e.g. by a brief excitable stimulus or an increase of recurrent synaptic excitation), the activity rises rapidly to large values due to the positive feedback of recurrent excitation. The explosive rise of the activity is terminated by the beginning of synaptic depression, which acts as negative feedback and ultimately wins over recurrent excitation. As a result of the initial excitation, the population activity may show population spikes similar to action potential in other excitable systems such as single neurons.
As expected for an excitable system driven by noise [42], the population activity exhibits irregular population spikes if the population size is small (here \(N=100\)), i.e. if finitesize noise is sufficiently strong (Fig. 6(A)(i)–(iii)). In our case, the drive that causes population spikes originates from finitesize fluctuations as expressed by the stochastic terms in Eq. (28). For \(N = 100\), the secondorder MF accurately predicts the mean activity (Fig. 6(B)) and power spectrum (Fig. 6(C)) of the full microscopic simulation whereas the firstorder MF deviates quantitatively. Importantly, in the limit of large population size, population spikes vanish in the secondorder MF theory consistent with microscopic simulations (Fig. 6(A)(vi), (iv), \(N=5000\)). In marked contrast to microscopic simulations, highly regular population spikes persist even for \(N=5000\) in the firstorder MF approximation corresponding to a deterministic limitcycle dynamics (Fig. 6(A)(v)).
In summary, the secondorder MF approximation accurately reproduces the qualitative behavior and the mean and the power spectrum of excitatory networks of Poisson neurons with synaptic STP. The statistical properties of the firstorder MF dynamics exhibit quantitative deviations of statistical properties and in some cases fails to reproduce the qualitative behavior if the system is poised near a bifurcation. The large discrepancies between the firstorder MF and the microscopic model in the example we show (Fig. 6) are mainly caused by the error in the mean modulation factor R. Indeed, for our choice of \(\tau _{D} = \tau _{F} = 1\text{ s}\), correlations between \(u_{j}\) and \(x_{j}\) are relatively strong but are neglected by the firstorder MF. We note that this error appears already for the deterministic (i.e. \(N\rightarrow \infty \)) dynamics. Inaccuracies in the correct description of finitesize noise in the firstorder MF model may yield additional sources of errors.
Furthermore, the secondorder MF remains accurate when the connection probability p of the network is smaller than 1 (given that the number pN of incoming synapses per neuron is roughly greater than 100). When p is smaller than 1, we argue in Sect. 3.2 that the microscopic network can be approximated by a fullyconnected network with rescaled synaptic weights (meanfield approximation). We test this claim numerically in Fig. 7: using the same setup as in Fig. 6, for a population size of \(N=1000\) neurons, we show that the secondorder MF yields more accurate prediction than the firstorder MF even when the connection probability is reduced down to 0.1.
Bistable switching between Up and Down states induced by finitesize fluctuations
Another collective phenomenon in neural networks is multistability. In the presence of finitesize noise, systems with multistable behavior exhibit switches between different attractor states [12, 43, 44]. In particular, bistable neural systems driven by noise support stochastic switches between high and low population activity (“Up and Down states”) [12, 31, 45, 46]. As a starting point of our simulations of Up and Down states, following [31], we use an excitatory population with synaptic depression in the bistable regime. The qualitative behavior of the microscopic model exhibiting Up and Down states is captured by both first and secondorder MF (Fig. 8). A closer look at the mean firing rate, which is mainly determined by the ratio of the time spent in the Up or Down state, reveals that the firstorder MF dynamics predicts significantly longer residence times in the Up state (Fig. 8(B)). In contrast, the secondorder MF approximation accurately matches the simulation of the microscopic model. In this example we have chosen \(\tau _{D} \gg \tau _{F}\) such that the correlations between \(u_{j}\) and \(x_{j}\) are negligible. As the consequence, the mean modulation factors R predicted by the first and secondorder MF theories, and hence the mean TPSI \(\langle y\rangle \), are almost equal (cf. Fig. 4(A)). The error made by the firstorder MF approximation mainly results from an incorrect description of finitesize fluctuations: at high firing rate (Up state), finitesize fluctuations are largely underestimated in the firstorder MF dynamics as mentioned in Sect. 2.4. The weaker noise implies longer residence times in the Up state. This example highlights the relevance of the fluctuation statistics provided by the secondorder MF approximation.
Recurrent network of GIF neurons—microscopic vs. mesoscopic simulations
As a final demonstration of the mesoscopic MF theory with STP, we consider an excitable regime generating population spikes as in Sect. 3.3.1 but with more realistic neurons described by a GIF spiking neuron model (Sects. 3.1.2 and 3.2.2). Because of spikehistory dependencies such as refractoriness, spike arrivals at synapses are no longer Poisson processes. Hence, the Poisson assumption of first and secondorder MF theories is not fulfilled anymore for recurrent GIF networks. Nevertheless, it is interesting to see whether population spikes can still be captured by the mesoscopic MF equations. To this end, we simulated the recently developed mesoscopic population equations for populations of GIF neurons [12] given by Eqs. (22), (31)–(35) extended by the MF equations for STP, Eqs. (18a)–(18g). The full MF theory qualitatively reproduces population spikes at small population sizes (Fig. 9(A)) and their extinction for large populations (Fig. 9(B)). Both the mean (Fig. 9(C)) and fluctuation statistics (Fig. 9(D)) are roughly captured by the MF equations albeit with small deviations from the microscopic simulation. However, a clear advantage of 2nd vs. firstorder approximation is not apparent. This indicates that the secondorder approximation does not necessarily yield a better approximation for networks of nonPoisson neurons and that the computationally simpler firstorder MF model might be preferable in spiking neural networks with strong spikehistory effects.
Discussion
We have derived stochastic meanfield (MF) equations that capture the effect of synaptic shortterm plasticity (STP) at the level of populations. These equations generalize previous MF theories for deterministic population rates [4, 30, 31] to the case of finitesize populations with stochastic population rates (mesoscopic description). The mesoscopic STP dynamics is compatible with a recent mesoscopic population model [12], which has been originally derived for static synapses. The mesoscopic MF dynamics of STP can thus be easily included into existing mesoscopic models. We find that a firstorder meanfield approximation that accounts for stochastic rates but neglects correlations between facilitation and depression variables (as in [47]) approximates well the mean stationary input. This mean input is slightly improved by a secondorder approximation, which accounts for correlations but neglects third and higherorder cumulants. The main strength of the secondorder MF theory lies in the prediction of fluctuations and transient responses of the STP variables. We have shown that population spikes and UP and Down state switches in a onepopulation model with synaptic depression can be well described by the extended mesoscopic model. In particular, the secondorder MF equations accurately replicate simulations of a network of Poisson neurons coupled via dynamic synapses. For networks of GIF spiking neurons the agreement is less accurate but still captures the qualitative collective dynamics.
In simulations of neuronal populations with STP, our mesoscopic meanfield model yields a considerable reduction of computational complexity. Compared to a network with static synapses, each neuron is endowed with two additional variables \(u_{j}\) and \(x_{j}\) that capture the effect of dynamic synapses onto its postsynaptic target neurons. In a single population of N neurons and connection probability p, a microscopic simulation thus requires the numerical integration of \(2pN\) additional equations. By contrast, a simulation of the mesoscopic model only needs 4 additional equations per population. Thus we expect that our extended mesoscopic dynamics offers a significant speed up of largescale simulations of cortical circuits with dynamic synapses [7].
An interesting question that has been studied theoretically [27–29] is how STP affects information transmission through a large ensemble of dynamic synapses. Our reduction of a synaptic ensemble to a fourdimensional nonlinear meanfield dynamics offers a mathematical framework to derive approximate analytical expressions for measures of information transmission. Analysing information processing capabilities of STP in the context of our meanfield theory is an interesting topic for future studies.
We have employed the deterministic STP model of Tsodyks and Markram [30]. While the resulting meanfield equations hold for this specific model, the same approach can be applied straightforwardly to other deterministic models of STP (e.g. [21, 48]). It is less obvious how to treat stochastic models of STP. Biological synapses are highly stochastic owing to the small number of synaptic vesicles that are randomly released upon spike arrival. This includes a finite probability of transmission failure. Using stochastic models of STP, it has been shown that synaptic stochasticity has a strong impact on information transmission [28] and postsynaptic neural responses [49]. On the population level, it seems to be feasible to treat this source of randomness in a similar manner as we did in Sect. 2.3. A generalization to a mesoscopic STP model that is applicable for stochastic synapses, will be an important subject for further studies.
The meanfield equations for the STP dynamics have been derived under the assumption that presynaptic spike trains are, loosely speaking, Poisson (Assumption 1). We have tested the meanfield equations in a feedforward setup and a recurrent network of Poisson rate units, where Assumption 1 holds true, and we found excellent agreements with microscopic simulations. For the application to recurrent networks of generalized integrateandfire (GIF) neurons in Sect. 3.4, Assumption 1 is not fulfilled because of refractoriness and other spikehistory dependencies of single neurons [50, 51]. Despite the nonPoisson (colorednoise) statistics of spike trains in integrateandfire networks, a Poisson (whitenoise) assumption is commonly used in meanfield theories as a “firstorder approximation” [52]. In a similar spirit, we here simply assumed that synaptic input can be treated as a Poisson process so as to apply our MF theory for STP to networks of GIF neurons. For a simple onepopulation model with excitatory synaptic connections and STP that exhibits nontrivial dynamics in the form of population spikes, we have shown that the MF equations reproduce the qualitative behavior. This indicates that the MF theory may be valid beyond networks of Poisson neurons. A theoretical analysis of the effect of nonPoisson inputs and the region of validity of the present MF model is beyond the scope of the present paper and remains to be studied. To this end, theoretical approaches to treat dynamic synapses driven by renewal processes [53–55] might be a promising starting point.
Note that although the derivation of the mesoscopic meanfield approximation is systematic we do not obtain any mathematical guarantee that the process \(y(t)\) is close (in any sense) to the process \(y^{N}(t)\) we want to approximate. In the case where the spike trains \(\{s_{j}\}_{j=1, \ldots , N}\) are N independent Poisson processes of rate \(\lambda (t)\), it might be possible to prove that the processes \(y(t)\) and \(y^{N}(t)\) converge to the same diffusion approximation as N tends to infinity. Obtaining such a proof would be, however, challenging because in our case, we do not know around which mean value the process fluctuates, i.e. we do not know the deterministic \(N \to \infty \) limit. This is due to the product \(u_{j} x_{j}\) in Eq. (7b) and the fact that we do not have a closed form expression for the limit of \(\frac{1}{N}\sum_{j=1}^{N} u_{j} x_{j}\) when \(N \to \infty \). Note that if one considers purely facilitating (or purely depressing) synapses, the deterministic \(N \to \infty \) limit can be computed (see [32]). Even in the case of purely facilitating or depressing synapses, deriving the diffusion approximation for the evolution of the mesoscopic variables \(u(t)\) or \(x(t)\) would be nontrivial because the jump sizes are modulated by the microscopic \(u_{j}\) and \(x_{j}\) (see Eqs. (38a)–(38b)). These are two independent reasons why standard techniques from the fluid limit literature (see [38–40]) cannot be applied directly. Also, one clear advantage of our current approach over the diffusion approximation approach is that our mesoscopic approximation can be conditioned on the process \(A^{N}(t)\), i.e. it can be conditioned on any sequence of spike times \(\{t_{k}\}_{k \in \mathbb{Z}^{+}}\).
In our previous work [12], we have developed a meanfield theory of neuronal populations that incorporates spikehistory dependencies, such as refractoriness and adaptation, and finitesize fluctuations in a consistent manner. By adding another important feature—synaptic shortterm plasticity—we have here made a further step towards a microscopicallygrounded mesoscopic population model of a cortical circuit.
Notes
Here we assume that different spikes in \([t,t+\Delta t)\) belong to different neurons, which is justified if \(r(t)\Delta t\ll 1\).
References
Wilson HR, Cowan JD. Excitatory and inhibitory interactions in localized populations of model neurons. Biophys J. 1972;12(1):1.
BenYishai R, BarOr RL, Sompolinsky H. Theory of orientation tuning in visual cortex. Proc Natl Acad Sci USA. 1995;92(9):3844–8.
Wong KF, Wang XJ. A recurrent network mechanism of time integration in perceptual decisions. J Neurosci. 2006;26(4):1314–28.
Barak O, Tsodyks M. Persistent activity in neural networks with dynamic synapses. PLoS Comput Biol. 2007;3(2):e35.
Shpiro A, MorenoBote R, Rubin N, Rinzel J. Balance between noise and adaptation in competition models of perceptual bistability. J Comput Neurosci. 2009;27(1):37–54.
Rubin DB, Van Hooser SD, Miller KD. The stabilized supralinear network: a unifying circuit motif underlying multiinput integration in sensory cortex. Neuron. 2015;85(2):402–17.
Markram H, Muller E, Ramaswamy S, Reimann MW, Abdellah M, Sanchez CA et al.. Reconstruction and simulation of neocortical microcircuitry. Cell. 2015;163(2):456–92.
Izhikevich EM, Edelman GM. Largescale model of mammalian thalamocortical systems. Proc Natl Acad Sci USA. 2008;105(9):3593–8.
Potjans TC, Diesmann M. The celltype specific cortical microcircuit: relating structure and activity in a fullscale spiking network model. Cereb Cortex. 2014;24(3):785–806.
Fiebig F, Lansner A. A spiking working memory model based on Hebbian shortterm potentiation. J Neurosci. 2017;37(1):83–96.
Rössert C, Pozzorini C, Chindemi G, Davison AP, Eroe C, King J, et al. Automated pointneuron simplification of datadriven microcircuit models. arXiv eprints. 2016.
Schwalger T, Deger M, Gerstner W. Towards a theory of cortical columns: from spiking neurons to interacting neural populations of finite size. PLoS Comput Biol. 2017;13(4):e1005507.
Harris KD, Shepherd GMG. The neocortical circuit: themes and variations. Nat Neurosci. 2015;18(2):170–81.
Lefort S, Tomm C, Sarria JCF, Petersen CCH. The excitatory neuronal network of the C2 barrel column in mouse primary somatosensory cortex. Neuron. 2009;61(2):301–16.
Gerstner W. Time structure of the activity in neural network models. Phys Rev E. 1995;51:738.
Gerstner W. Population dynamics of spiking neurons: fast transients, asynchronous states, and locking. Neural Comput. 2000;12:43.
Naud R, Gerstner W. Coding and decoding with adapting neurons: a population approach to the peristimulus time histogram. PLoS Comput Biol. 2012;8(10):e1002711.
Schwalger T, Chizhov AV. Mind the last spike—firing rate models for mesoscopic populations of spiking neurons. Curr Opin Neurobiol. 2019;58:155–66.
Abbott LF, Varela JA, Sen K, Nelson SB. Synaptic depression and cortical gain control. Science. 1997;275:220.
Markram H, Wang Y, Tsodyks M. Differential signaling via the same axon of neocortical pyramidal neurons. Proc Natl Acad Sci USA. 1998;95(9):5323–8.
Dittman JS, Kreitzer AC, Regehr WG. Interplay between facilitation, depression, and residual calcium at three presynaptic terminals. J Neurosci. 2000;20:1374.
Zucker RS, Regehr WG. Shortterm synaptic plasticity. Annu Rev Physiol. 2002;64:355–405.
Levina A, Herrmann JM, Geisel T. Dynamical synapses causing selforganized criticality in neural networks. Nat Phys. 2007;3(12):857.
Pittorino F, IbáñezBerganza M, di Volo M, Vezzani A, Burioni R. Chaos and correlated avalanches in excitatory neural networks with synaptic plasticity. Phys Rev Lett. 2017;118(9):098102.
Seeholzer A, Deger M, Gerstner W. Stability of working memory in continuous attractor networks under the control of shortterm plasticity. PLoS Comput Biol. 2019;15(4):e1006928.
Fortune ES, Rose GJ. Shortterm synaptic plasticity as a temporal filter. Trends Neurosci. 2001;24:381.
Merkel M, Lindner B. Synaptic filtering of ratecoded information. Phys Rev E. 2010;81(4 Pt 1):041921.
Rosenbaum R, Rubin J, Doiron B. Short term synaptic depression imposes a frequency dependent filter on synaptic information transfer. PLoS Comput Biol. 2012;8(6):e1002557.
Droste F, Schwalger T, Lindner B. Interplay of two signals in a neuron with heterogeneous synaptic shortterm plasticity. Front Comput Neurosci. 2013;7:86.
Tsodyks M, Pawelzik K, Markram H. Neural networks with dynamic synapses. Neural Comput. 1998;10(4):821–35.
Holcman D, Tsodyks M. The emergence of up and down states in cortical networks. PLoS Comput Biol. 2006;2(3):e23.
Galves A, Löcherbach E, Pouzat C, Presutti E. A system of interacting neurons with short term plasticity. arXiv preprint. 2019. arXiv:1903.01270.
Lindner B, Gangloff D, Longtin A, Lewis JE. Broadband coding with dynamic synapses. J Neurosci. 2009;29(7):2076–88.
Cook DL, Schwindt PC, Grande LA, Spain WJ. Synaptic depression in the localization of sound. Nature. 2003;421(6918):66.
Higley MJ, Contreras D. Balanced excitation and inhibition determine spike timing during frequency adaptation. J Neurosci. 2006;26(2):448–57.
Oswald AM, Urban NN. Interactions between behaviorally relevant rhythms and synaptic plasticity alter coding in the piriform cortex. J Neurosci. 2012;32(18):6092–104.
Gigante G, Deco G, Marom S, Del Giudice P. Network events on multiple space and time scales in cultured neural networks and in a stochastic rate model. PLoS Comput Biol. 2015;11(11):e1004547.
Kurtz TG. Limit theorems for sequences of jump Markov processes approximating ordinary differential processes. J Appl Probab. 1971;8(2):344–56.
Kurtz TG et al.. Strong approximation theorems for density dependent Markov chains. Stoch Process Appl. 1978;6(3):223–40.
Pakdaman K, Thieullen M, Wainrib G. Fluid limit theorems for stochastic hybrid systems with application to neuron models. Adv Appl Probab. 2010;42(3):761–94.
Ditlevsen S, Löcherbach E. Multiclass oscillating systems of interacting neurons. Stoch Process Appl. 2017;127(6):1840–69.
Lindner B, GarcíaOjalvo J, Neiman A, SchimanskyGeier L. Effects of noise in excitable systems. Phys Rep. 2004;392:321.
LitwinKumar A, Doiron B. Slow dynamics and high variability in balanced cortical networks with clustered connections. Nat Neurosci. 2012;15(11):1498–505.
Mazzucato L, Fontanini A, La Camera G. Dynamics of multistable states during ongoing and evoked cortical activity. J Neurosci. 2015;35(21):8214–31.
MorenoBote R, Rinzel J, Rubin N. Noiseinduced alternations in an attractor network model of perceptual bistability. J Neurophysiol. 2007;98(3):1125–39.
Jercog D, Roxin A, Barthó P, Luczak A, Compte A, de la Rocha J. UPDOWN cortical dynamics reflect state transitions in a bistable network. eLife. 2017;6:e22425.
Tsodyks MV, Markram H. The neural code between neocortical pyramidal neurons depends on neurotransmitter release probability. Proc Natl Acad Sci USA. 1997;94:719.
Dayan P, Abbott LF. Theoretical neuroscience: computational and mathematical modeling of neural systems. 1st ed. Cambridge: MIT Press; 2005.
Bird A, Richardson M. Longterm plasticity determines the postsynaptic response to correlated afferents with multivesicular shortterm synaptic depression. Front Comput Neurosci. 2014;8:2.
Deger M, Schwalger T, Naud R, Gerstner W. Fluctuations and information filtering in coupled populations of spiking neurons with adaptation. Phys Rev E. 2014;90(6):062704.
Schwalger T, Droste F, Lindner B. Statistical structure of neural spiking under nonPoissonian or other nonwhite stimulation. J Comput Neurosci. 2015;39(1):29–51.
Brunel N. Sparsely connected networks of spiking neurons. J Comput Neurosci. 2000;8:183.
Romani S, Amit DJ, Mongillo G. Meanfield analysis of selective persistent activity in presence of shortterm synaptic depression. J Comput Neurosci. 2006;20(2):201.
Mongillo G, Hansel D, van Vreeswijk C. Bistability and spatiotemporal irregularity in neuronal networks with nonlinear synaptic transmission. Phys Rev Lett. 2012;108:158101.
Bird AD, Richardson MJE. Transmission of temporally correlated spike trains through synapses with shortterm depression. PLoS Comput Biol. 2018;14(6):1–25.
Acknowledgements
We would like to thank Eva Löcherbach for her comments on the manuscript and Alexander Seeholzer for discussions at the initial stage of this work.
Availability of data and materials
Not applicable.
Funding
This project received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 720270.
Author information
Authors and Affiliations
Contributions
VS developed the theory, performed the numerical simulations and was a major contributor in writing the manuscript. WG was a contributor in writing the manuscript. TS developed the theory and was a major contributor in writing the manuscript. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
Not applicable.
Competing interests
The authors declare that they have no competing interests.
Consent for publication
Not applicable.
Appendices
Appendix 1: Proof of Lemma 2.1
To derive the system of equations (16a)–(16c), it is useful to rewrite the microscopic synaptic dynamics Eqs. (11a)–(11b) in differential form:
Here, \(dn_{j}(t)\) denotes the increment of the counting process \(n_{j}(t)=\int ^{t}_{0}s_{j}(t') \,dt'\) at the jth synapse and \(u^{}_{j}\) is a shorthand for the left limit. The stochastic differential equations for the mesoscopic quantities \(u^{N}(t)\) and \(x^{N}(t)\) are then
With some calculations, we can write similar equations for \(P^{N}(t)\), \(Q^{N}(t)\) and \(R^{N}(t)\). For instance, to compute \(dP^{N}\) we need to evaluate \(d(u_{j}^{2})\). Using Itô formula for jump processes, we get
Similarly, we write
We now compute the jump terms:
Summing the differentials \(d(u_{j}^{2})\), \(d(x_{j}^{2})\) and \(d(u_{j}x_{j})\) over all N synapses, we obtain
For notational simplicity, we have derived Eqs. (38a)–(38b) and (42a)–(42c) for the variables \([u^{N},x^{N},P^{N},Q^{N},R^{N}]\) defined by Eqs. (15a)–(15c) starting from microscopic variables \(u_{j}\) and \(x_{j}\) defined by Eqs. (7a)–(7b). A completely analogous derivation can be performed for the variables \([u^{*},x^{*},P^{*},Q^{*},R^{*}]\) defined by Eqs. (16a)–(16c) starting from the corresponding microscopic variables \(u_{j}^{*}\) and \(x_{j}^{*}\) defined by Eqs. (11a)–(11b). For the variables marked with asterisk, we thus have
where \(u_{j}^{*}\) and \(x_{j}^{*}\) are shorthands for \(u_{j}^{*}(t^{})\) and \(x_{j}^{*}(t^{})\), respectively.
The sums over the weighted spike counts \(dn_{j}^{*}\) cannot be expressed as a deterministic function of the mesoscopic spike count \(dn^{N}=\sum_{j}n_{j}^{*}(t)\). However, we can make use of the fact that in an infinitesimally small time interval dt at most one spike can occur and contribute to the sums, which thereby simplify considerably: \(dn_{j}^{*}\) is either zero for all j (i.e. \(dn^{N}=0\)), or there exists one and only one j for which \(dn_{j}^{*}=1\) (in this case \(dn^{N}=1\)). This implies that we can write for any function \(g(u_{j}^{*},x_{j}^{*})\) multiplying \(dn_{j}^{*}\)
where \(j^{*}(k)\) is the random synapse index at time \(t_{k}\) and \((\hat{u}^{*}(t), \hat{x}^{*}(t))\) is given by Eq. (12). Applying relation (44) to Eqs. (43a)–(43e), we obtain
Using \(A^{N}(t)=\frac{1}{N}\frac{dn^{N}(t)}{dt}\) we recover Eqs. (17a)–(17e) of the main text.
Appendix 2: Efficient numerical implementation of the mesoscopic approximation
The mesoscopic STP dynamics given by Eqs. (18a)–(18g) are driven by a point process \(A^{N}(t)\) or increments \(dn(t)\) that are multiplied by a stochastic factor of the form \(f(\hat{u}(t),\hat{x}(t))\). In simulations, these stochastic amplitudes require some care. A straightforward discretization of Eqs. (18a)–(18g) would be to draw \(\hat{u}(t)\) and \(\hat{x}(t)\) from their joint Gaussian distribution (18f) in each time step independently, compute \(f(\hat{u}(t),\hat{x}(t))\) and multiply by the number of spikes
that occur in the discretization interval \([t,t+\Delta t)\). However, this approach is only correct if the discretization time step Δt is small enough such that Δn contains at most one spike. Because spikes result from a population of many neurons, this condition would require an extremely small time step (such that \(Nr(t)\Delta t\ll 1\)), and would thus yield a highly inefficient simulation algorithm. Luckily, the independence of the factors \(f(\hat{u}(t_{k}),\hat{x}(t_{k}))\) across spikes in the interval \([t,t+\Delta t)\) allows us to use larger time steps that may contain multiple spikes:^{Footnote 1} due to the independence, the integration of the stochastic term in Eqs. (18a)–(18g) simplifies to a sum of Δni.i.d. random variables:
The \((\hat{u}_{j},\hat{x}_{j})\) are identically distributed because the Gaussian (18f) is held fixed in the time interval \([t,t+\Delta t)\). If the random variables \(f(\hat{u}_{j},\hat{x}_{j})\) have mean \(\mu _{f}\) and variance \(v_{f}\), the sum in Eq. (47) has mean \(\mu _{f}\Delta n\) and variance \(v_{f}\Delta n\). Using a Gaussian approximation, we can thus approximate the integral, Eq. (47), by
where \(\varepsilon (t)\) is a centered Gaussian random variable with variance \(v_{f}\) that is independent in each time step.
For example, the integration of the mesoscopic equation for u, Eq. (18a), involves the integration of the stochastic term \(N^{1}U(1\hat{u})\,dn\). Using \(\langle \hat{u}\rangle =\mu _{u}=u\) and \(\operatorname{var}(\hat{u})=v_{u}=Pu^{2}\), Eq. (48) yields
where \(\varepsilon _{u}(t)\) is a centered Gaussian random variable with variance \(v_{u}\). Similarly, if \((\varepsilon _{u}(t), \varepsilon _{\varepsilon }(t))\) is a pair of correlated centered Gaussian variable with covariance matrix
the integration of the mesoscopic equation for x, Eq. (18b), yields
Here, we approximated \(\varepsilon _{u}\varepsilon _{x}\) by its mean \(\langle \varepsilon _{u}\varepsilon _{x}\rangle =Rux\), and neglected its fluctuations.
When we integrate Eqs. (18c)–(18e), we encounter the terms \(\varepsilon _{u}^{2} \varepsilon _{x}\), \(\varepsilon _{u} \varepsilon _{x}^{2}\) and \(\varepsilon _{u}^{2} \varepsilon _{x}^{2}\). As we cannot calculate their mean, we perform a moment closure approximation, neglecting all cumulants of order higher than two:
symmetrically, \(\langle \varepsilon _{u} \varepsilon _{x}^{2} \rangle \approx 0\); and finally,
These approximations allow for the completion of the derivation.
In summary, the Euler scheme corresponding to Eqs. (18a)–(18g) is
with increments given by
Here, we abbreviated
The generation of correlated centered Gaussian random variables \(\varepsilon _{u}(t)\) and \(\varepsilon _{x}(t)\) with covariance matrix (49) can be implemented by standard methods. For instance, one may compute in each time step the correlation coefficient
In the initial warmup phase, it may happen that the numerical values of the variances \(Pu^{2}\) and \(Qx^{2}\) are nonpositive or the absolute value of the correlation coefficient \(\rho \) exceeds unity. One simple practical solution to this problem is to set \(\varepsilon _{u}\) and \(\varepsilon _{x}\) to zero in these cases. Otherwise, we generate random variables by the formula
where \(z_{1}\) and \(z_{2}\) are independent standard normal random numbers.
Finally, we note that in numerical simulations it is convenient to operate on the spike counts \(\Delta n(t)\) rather than on the population activity \(A^{N}(t)\). The discretized population activity can be easily obtained from the spike counts via the formula
Appendix 3: Equations for infinitesize populations and steadystate formulas
A macroscopic theory of STP for infinitesize populations for what we call the firstorder MF has been presented in [30]. We detail here the adaptation of our secondorder MF to the case if infinitesize populations.
In the infinitesize case, the stochastic populations activity \(A^{N}(t)\) becomes a deterministic rate \(r(t)\), which simplifies our mesoscopic equations (18a)–(18g): in the stochastic ODEs, \(r(t)\)—being deterministic—is not multiplied by a random term but by the mean of this term, transforming the stochastic ODEs into deterministic ODEs. These means have already been computed in Appendix B. Hence, in the infinitesize case, our secondorder MF equations are
Again, we abbreviated
Given a fixed rate r, we can compute the steadystate values of u, x, P, Q and R:
We used the abbreviation
Note that these steadystates formulas are very useful for carrying out precise phaseplane analyses.
Appendix 4: Mesoscopic population equations for network of GIF neurons—ODE representation
The equation for the population rate \(r(t)\) for GIF neurons involves the integral, Eq. (31), over all possible refractory states and a set of partial differential equations (socalled quasilinear equations) for the quantities \(q(\tau ,t)\), \(V(\tau ,t)\) and \(W(\tau ,t)\). Instead of the age of the neuron (i.e. the time since its last spike), the refractory state can be equivalently specified by the last spike time \(\hat {t}=t\tau \). This variable transformation turns the partial differential equations into ordinary differential equations (ODEs), which yields an alternative formulation of the mesoscopic population equations.
If the refractory state is specified by the last spike time \(\hat {t}=t\tau \), we need to consider the density of last spike times \(Q(\hat {t},t)\equiv q(t\hat {t},t)\). Instead of Q, it is slightly more convenient to write \(Q(\hat {t},t)=S(t\hat {t})A^{N}(\hat {t})\), where we introduced the survivor function \(S(t\hat {t})\). With this notation the population rate, Eq. (31), becomes [12]
Using the method of characteristics, the quasilinear equation (32) for \(q(\tau ,t)\) has an equivalent ODE representation for the characteristic curves \(\{Q(\hat {t},t)\}_{t>\hat {t}}\): \(dQ/dt=\lambda Q\) with initial condition \(Q(\hat {t},\hat {t})=A^{N}(\hat {t})\). This corresponds to an ODE for the survivor function:
The functions \(\lambda (t\hat {t})\equiv \lambda (t\hat {t},t)\) and \(\varLambda (t)\) follow from Eq. (33) as
The dynamics for the variables \(V(t\hat {t})\) and \(W(t\hat {t})\) follow from those of the quasilinear equations (34) and (35):
with initial conditions \(V(\hat {t}\hat {t})=V_{\mathrm {r}}\) and \(W(\hat {t}\hat {t})=0\). Finally, the threshold function \(\vartheta (t\hat {t})\) reads
Appendix 5: Recurrent network parameters
Parameters of Fig. 6, Fig. 7 and Fig. 8 are detailed in Table 1.
In Fig. 9, we used the following parameters: \(\tau _{D} = 0.5\mbox{ s}\), \(\tau _{F} = 0.01\mbox{ s}\), \(U = 0.1\), \(U_{0} = 0.1\), \(\tau _{\mathrm {m}}= 0.015\mbox{ s}\), \(\tau _{s} = 0.01\mbox{ s}\), \(c = 15\text{ Hz}\), \(V_{\mathrm{reset}} = 0\mbox{ mV}\), \(V_{\mathrm{th}} = 15\mbox{ mV}\), \(\mu = 4\mbox{ mV}\), \(R_{\mathrm{m}} = 1~\Omega\), \(\Delta _{u} = 5\mbox{ mV}\), \(t_{\mathrm{ref}} = 0.004\mbox{ s}\), \(J\cdot N = 850\).
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Schmutz, V., Gerstner, W. & Schwalger, T. Mesoscopic population equations for spiking neural networks with synaptic shortterm plasticity. J. Math. Neurosc. 10, 5 (2020). https://doi.org/10.1186/s1340802000082z
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s1340802000082z
Keywords
 Shortterm plasticity
 Multiscale modeling
 Mesoscopic population dynamics