Models

Both fESN and wESN extend a standard Echo State Network with a second, dedicated memory reservoir. The vanilla reservoir captures short-term nonlinear dynamics; the memory reservoir sustains long-range dependence. Their states are concatenated and mapped to the output by a single trained ridge readout.

Background: the Echo State Network

With univariate input \(u(t)\in\mathbb{R}\) and reservoir size \(p\), the reservoir state \(\mathbf{x}(t)\in\mathbb{R}^p\) evolves as

\[ \mathbf{x}(t) = \tanh\!\big(\mathbf{W}_{xx}\,\mathbf{x}(t-1) + \mathbf{W}_{xu}\,u(t) + \boldsymbol{\eta}_x(t)\big), \qquad \boldsymbol{\eta}_x(t)\sim\mathcal{N}(\mathbf{0},\sigma_x^2\mathbf{I}_p), \]

where \(\mathbf{W}_{xx}\) and \(\mathbf{W}_{xu}\) are fixed after random initialization and \(\mathbf{W}_{xx}\) is sparsified and rescaled so that its spectral radius \(\rho(\mathbf{W}_{xx}) < 1\) (a sufficient design rule for the echo-state property). Only the linear readout is trained.

fESN, Fractional Echo State Network

The memory reservoir does not receive \(u(t)\) directly. Instead it receives a pure-past fractional-difference filter of it,

\[ f(t) = \big((1-B)^d - 1\big)\,u(t) = \sum_{k\ge 1}\omega_k\,u(t-k), \qquad \omega_k = \frac{\Gamma(k-d)}{k!\,\Gamma(-d)},\quad d\in(0,\tfrac12), \]

where \(B\) is the backshift operator. The “\(-1\)” removes the present term (\(\omega_0\)), so the memory input depends only on the past; in practice the sum is truncated at \(K\) lags. The weights decay polynomially as \(k^{-d-1}\), a soft attention over the distant past, in contrast to the exponential forgetting of the reservoir. The memory reservoir state \(\mathbf{m}(t)\in\mathbb{R}^q\) evolves as

\[ \mathbf{m}(t) = \tanh\!\big(\mathbf{W}_{mm}\,\mathbf{m}(t-1) + \mathbf{W}_{mf}\,f(t) + \boldsymbol{\eta}_m(t)\big). \]

In code

fESN(n_reservoir=(p, q), d=d, K=K). Passing a list d=[...] stacks several orders as a multivariate memory input.

wESN, Wavelet Echo State Network

wESN is fESN with one extra step: the fractional filter is applied to a wavelet-smoothed signal rather than the raw input. A shift-invariant MODWT first extracts the low-frequency trend

\[ s(t) = \operatorname{MODWT}_{\text{smooth}}(u(t)), \]

and the memory input becomes

\[ f(t) = \big((1-B)^d - 1\big)\,s(t). \]

Estimating long-range structure from low-frequency dynamics suppresses high-frequency noise and seasonal fluctuation. Any discrete PyWavelets family is available (haar, db, sym, coif, bior, rbio, dmey), and you may keep the smooth component, any detail level(s), or a stacked combination.

In code

wESN(n_reservoir=(p, q), d=d, K=K, wavelet="db4", wavelet_level=2, wavelet_components="smooth"). wavelet_norm switches between the standard MODWT normalization and the classic DWT filters (model-equivalent).

Readout and training

At each step the model forms the extended state

\[ \boldsymbol{\phi}(t) = \big[\,\mathbf{x}(t);\ \mathbf{m}(t);\ u(t);\ f(t)\,\big] \in \mathbb{R}^{p+q+2}, \]

and, after discarding a washout of \(T_0=\zeta T\) transient steps, stacks the rows into a design matrix \(\mathbf{Z}\). For each horizon \(h\in\{1,\dots,H\}\) the readout is a ridge regression

\[ \widehat{\mathbf{W}}_{\text{out}}^{(h)} = \arg\min_{\mathbf{W}}\ \big\lVert \mathbf{u}^{(h)} - \mathbf{Z}\mathbf{W}\big\rVert_2^2 + \lambda\lVert\mathbf{W}\rVert_2^2 . \]

Because \(\mathbf{Z}\) (and hence its Cholesky factor) does not depend on \(h\), the same factorization is reused across horizons, only the right-hand side changes. RidgeCV selects \(\lambda\) by closed-form leave-one-out, avoiding any refit loop.

Paper ↔ code

Symbol	Meaning	Argument
\(p,\ q\)	vanilla / memory reservoir sizes	n_reservoir=(p, q)
\(\rho_x,\ \rho_m\)	spectral radii	spectral_radius
\(\varphi_x,\ \varphi_m\)	sparsification proportion	sparsity
\(\psi_x,\ \psi_m\)	input scalings	input_scaling
\(\sigma_x,\ \sigma_m\)	state-noise std-dev	noise
\(d,\ K\)	fractional order, filter lags	d, K
\(\zeta\)	washout ratio \(T_0/T\)	washout=int(zeta*T)