(2)Forecasting
2.1
Embed each input
The forecaster is a Temporal Fusion Transformer (Lim et al., 2020). It starts by turning each of today's feature values into a vector, with a learned linear map per feature:
\[ e^{(k)}_t = x^{(k)}_t\, w^{(k)} + b^{(k)} \]
The real model uses 32 to 64 dimensions. The figure uses 4 so every number fits, and its weights are random stand-ins for trained ones, but every operation from here on is computed exactly.
2.2
Gated residual network: the dense layer
The gated residual network (GRN) is the TFT's building block. Its first layer takes the input \(a\) together with a context vector \(c\), here the asset's static embedding, and applies an ELU nonlinearity:
\[ \begin{gathered} \eta_2 = \operatorname{ELU}\!\big(W_2 a + W_3 c + b_2\big) \\ \operatorname{ELU}(z) = \begin{cases} z & z > 0 \\ e^{z} - 1 & z \le 0 \end{cases} \end{gathered} \]
Stacking \(a\) and \(c\) turns the two products into one: \([W_2\;W_3]\) times \([a;\,c]\).
2.3
The gate
A second layer doubles the width, and the result splits in half. One half is squashed by a sigmoid into gates between 0 and 1, and the gates scale the other half element by element. This gated linear unit (GLU) lets the network switch a whole branch off when it doesn't help.
\[ \eta_1 = W_1 \eta_2 + b_1 = \begin{bmatrix} \gamma_1 \\ \gamma_2 \end{bmatrix}, \qquad \operatorname{GLU}(\eta_1) = \gamma_1 \odot \sigma(\gamma_2) \]
2.4
Skip connection and LayerNorm
The input is added back, so the block learns a correction to \(a\) rather than a replacement, and LayerNorm rescales the result to mean 0 and standard deviation 1 across its entries:
\[ \begin{aligned} \operatorname{GRN}(a, c) &= \operatorname{LayerNorm}\!\big(a + \operatorname{GLU}(\eta_1)\big) \\ \operatorname{LayerNorm}(s) &= g \odot \frac{s - \bar s}{\sigma_s} + \beta \end{aligned} \]
The learned gain \(g\) and shift \(\beta\) start at 1 and 0, as in the figure.
2.5
Variable selection
Every feature gets its own GRN. A separate GRN scores the features, and a softmax turns the scores into weights that are positive and sum to one. The weighted sum is the day's input to the sequence model, and the weights themselves show which features the model leaned on.
\[ \begin{gathered} v = \operatorname{softmax}(z), \qquad v_k = \frac{e^{z_k}}{\sum_j e^{z_j}} \\ \xi_t = \sum_k v_k\, \operatorname{GRN}_k\big(e^{(k)}_t\big) \end{gathered} \]
2.6
LSTM encoder
An LSTM encoder-decoder reads the 60-day sequence of these vectors, initialized from the static context. At each day, gates decide how much of the running cell state to forget, how much new information to write, and how much to expose as the hidden state:
\[ \begin{aligned} f_t,\; i_t,\; o_t &= \sigma\big(W_{f,i,o}\,[h_{t-1}, \xi_t] + b_{f,i,o}\big) \\ c_t &= f_t \odot c_{t-1} + i_t \odot \tanh\big(W_c\,[h_{t-1}, \xi_t] + b_c\big) \\ h_t &= o_t \odot \tanh(c_t) \end{aligned} \]
The figure runs a one-unit LSTM over the window for real, so its gate readings are genuine.
2.7
Attention
Each forecast day issues a query, every past day offers a key, and their scaled dot products become scores. A softmax over the scores gives attention weights, and the output is the weighted sum of the past days' values:
\[ \operatorname{Attention}(Q, K, V) = \operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right) V \]
The TFT's version shares the values across heads, so the averaged weights can be read directly as which past days each forecast relied on. The pattern in the figure is illustrative.
2.8
The quantile head
The final layer outputs three numbers for each of the next five days: the 10th, 50th, and 90th percentiles. Training minimizes the pinball loss, which charges under-forecasts at rate \(q\) and over-forecasts at rate \(1-q\), so each output settles on its own quantile:
\[ \rho_q(u) = \max\big(q\,u,\; (q-1)\,u\big), \qquad u = y - \hat y \]
At \(q = 0.5\) that's mean absolute error, and the 10% to 90% band should hold about 80% of outcomes.
2.9
Quantiles become \(\mu\)
The optimizer needs one expected return per asset. The code takes the median forecasts for the five days ahead and adds them up, because log returns add across days:
\[ \mu_i = \sum_{h=1}^{5} \hat q_{0.5}^{(i)}(h) \]
The figure then annualizes, multiplying by 252/5, to put \(\mu\) on the same scale as the annual covariance it uses. The band around the median is far wider than the median is far from zero: return forecasts are noisy, which is why the next step damps the noise in the risk estimate.
(4)Optimization
4.1
\(\hat\Sigma w\), row by row
Every portfolio statistic is matrix arithmetic. Start from equal weights. Multiplying \(\hat\Sigma\) by \(w\) takes each row against the weight vector: the vector turns to lie along the row, the pairs multiply, and the products add into one entry of \(\hat\Sigma w\).
\[ (\hat\Sigma w)_i = \sum_k \hat\Sigma_{ik}\, w_k \]
4.2
Risk, return, Sharpe
Then \(w^\top(\hat\Sigma w)\) is the portfolio variance and \(w^\top\mu\) its expected return. With the risk-free rate they give the Sharpe ratio, the quantity being maximized:
\[ \operatorname{SR}(w) = \frac{w^\top \mu - r_f}{\sqrt{w^\top \hat\Sigma\, w}} \]
4.3
Climb to the top
The engine solves this with SLSQP under the long-only constraints \(\sum_i w_i = 1\) and \(w_i \ge 0\). The figure solves the same problem the simplest way: step along the gradient of the Sharpe ratio, then project back onto the allowed weights. The arrow is the direction of the next step, and it turns as the climb levels out.
\[ \begin{aligned} \nabla \operatorname{SR}(w) &= \frac{\mu - r_f \mathbf{1}}{\sigma_p} - \frac{(w^\top\mu - r_f)\,\hat\Sigma w}{\sigma_p^{3}} \\ w &\leftarrow \Pi_{\Delta}\big(w + \eta\, \nabla \operatorname{SR}(w)\big) \end{aligned} \]
4.4
Weights out
The result is the max-Sharpe weight vector, held for the next five trading days in the backtest. Minimizing variance at each target return instead, across 80 targets, traces the efficient frontier at the top of the page.