Abstract

The Continuous Thought Machine (CTM) by Sakana AI unfolds neural activity along an internal time axis, treating neuron-level temporal processing and neural synchronization as representations. This article combines two complementary analyses — a bottom-up optimization proposal and a top-down theory-engineering audit — into a single comprehensive review.

Part 1 draws on predictive coding and dual-pathway visual cognition theory, together with my engineering experience from the dual-pathway predictive coding vision system (v_predictive / v_dual / v_engine), to propose four concrete optimization directions for CTM: prediction-error-driven inner loops, ventral-dorsal dual-pathway architecture, training stability (micro-parameterization), and certainty-based early stopping.

Part 2 takes the opposite perspective: given a top-down theoretical proposal mapping the v_predictive / v_dual / v_engine architectural paradigm onto CTM across six directions, it audits every assumption against CTM’s actual source code. It identifies precisely what connects directly and what requires correction, producing a corrected implementation roadmap.

Every direction is grounded in mechanism principles, mathematical formalization, exact source-code modification points in models/ctm.py (lines 1–605) and models/modules.py (lines 1–693), design rationale, expected benefits and risks, and phased roadmaps with experimental protocols.

Part 1: Four Optimization Directions — Bottom-Up from CTM Code Facts

1. CTM Recap: A Machine That Unfolds Thought in Time

1.1 Three Core Ideas

Internal time axis: the model has internal ticks (denoted (T) in the paper) decoupled from input data, allowing “thinking” to unfold as a process. Data is fed forward once; subsequent iterations run purely on internal state.
Neuron-Level Temporal Processing (NLM): each neuron has independent weights processing its own past (M)-step input history (trace), enabling fine-grained temporal dynamics. In the code this is SuperLinear: weight shape ((M, H, D)) (history length (\times) output dim (\times) neuron count), executing (D) independent linear maps in parallel via einsum('BDM,MHD->BDH') (models/modules.py lines 146–236).
Synchronization as representation: the degree to which neuron-pair activity synchronizes over time directly serves as output and action representations. Synchronization is an exponentially decaying temporal accumulation of pairwise activation products, implemented in (O(1)) recurrent form.

1.2 Forward Pass in Detail

The following formalization corresponds to models/ctm.py forward (lines 527–603). Tensor shapes:

Input (x): ((B, C, H, W)) or task-dependent shape.
Features (kv): ((B, S, d_{\text{input}})), where (S) is the number of tokens (backbone feature map flattened then projected via kv_proj).
History state_trace: ((B, D, M)), where (D = d_{\text{model}}) is neuron count and (M = \text{memory_length}).
Activation activated_state: ((B, D)), i.e. post-activation (z_t).
Action sync synchronisation_action: ((B, n_{\text{synch_action}})).
Output sync synchronisation_out: ((B, n_{\text{synch_out}})).
Predictions: ((B, \text{out_dims}, T)); certainties: ((B, 2, T)).

Feature extraction (once, outside loop):

[ kv = W_{kv} \big( \text{flatten}( \text{Backbone}(x) + \text{PE}(\text{Backbone}(x)) ) \big) ]

(O(1)) synchronization recurrence (compute_synchronisation, lines 202–267): let (p_t = z_{t-1}^L \odot z_{t-1}^R) be the pairwise product of selected neuron pairs (elementwise for random-pairing, upper triangle of outer product for first-last/random). Then:

[ \alpha_t = r \alpha_{t-1} + p_t, \qquad \beta_t = r \beta_{t-1} + 1, \qquad S_t = \frac{\alpha_t}{\sqrt{\beta_t}} ]

where (r = \exp(-\text{decay_params})) with decay_params clamped to ([0, 15]) (lines 551–552), so (r \in [e^{-15}, 1]). This recurrence is equivalent to an exponentially weighted accumulation of historical pairwise products normalized by effective sample count — a continuous-time measure of “degree of synchronization.” On the first call decay_alpha/decay_beta are None; they are initialized with the current pairwise product and an all-ones vector (lines 259–261). Subsequent steps use (O(1)) recurrence without recomputing all historical dot products.

Loop body (each step stepi, lines 560–590):

[ q_t = W_q S_t^{\text{action}}, \quad o_t = \text{MultiheadAttention}(q_t, kv, kv) ]

[ a_t = \text{Synapses}([o_t; z_{t-1}]), \quad A_t = [a_{t-M+1}, \dots, a_t] ]

[ z_t = \text{NLM}(A_t), \quad S_t^{\text{out}} = \frac{\alpha’_t}{\sqrt{\beta’_t}}, \quad \hat{y}_t = W_o S_t^{\text{out}}, \quad c_t = 1 - H_n(\text{softmax}(\hat{y}_t)) ]

Temporal note: at loop entry activated_state = (z_{t-1}) (from the previous NLM step or initial value), updated to (z_t) after Synapses + NLM. Thus compute_synchronisation(action) uses (z_{t-1}) and compute_synchronisation(out) uses (z_t).

1.3 Key Submodule Source-Level Analysis

1.3.1 SynapseUNET: Multi-Level Down/Upsampling with Skip Connections (`modules.py` lines 45–143)

SynapseUNET accepts ([o_t; z_{t-1}]) (dimension d_input + D) and outputs (a_t \in \mathbb{R}^D). Structure:

Initial projection: LazyLinear(widths[0]) → LayerNorm → SiLU, mapping input to the first width level.
Downsampling path: num_blocks = depth - 1 blocks of Dropout → Linear(w[i] → w[i+1]) → LayerNorm → SiLU, width decaying exponentially from out_dims (i.e., (D)) to minimum_width (default 16). Each block output is saved for skip connections.
Upsampling path: in reverse order (deepest → shallowest), each block Dropout → Linear(w[i+1] → w[i]) → LayerNorm → SiLU, then add the corresponding downsampling output, then pass through that level’s LayerNorm.

Why U-Net instead of MLP: deep U-Net provides multi-level information mixing (coarse cross-population interaction vs. fine-grained local interaction), and skip connections preserve gradient path integrity at depth. Paper depth = 1 (single GLU) degrades to a simple mixing layer; depth > 1 works better (get_synapses lines 415–435 document this).

Entry point for MuPC: the weight scaling factor (1/\sqrt{l+1}) at each level’s Linear in the down/up paths can be directly injected (see Direction 3).

1.3.2 SuperLinear: Per-Neuron Independent Mapping via einsum (`modules.py` lines 146–236)

Essentially a linear transform over the last dimension (history dimension (M)), but each neuron ((D) dim) has its own distinct weights:

Weight w1: ((M, H, D)) — mapping from (M) history values to (H) outputs, (D) independent versions.
Bias b1: ((1, D, H)) — per-neuron, per-output bias.
Computation: einsum('BDM,MHD->BDH', x, w1) + b1 → squeeze(-1) / T.
Learnable temperature (T) (modules.py line 209): dividing by (T) controls output scale; initialized to 1.0. When deep_nlms=True, two SuperLinear blocks are chained through GLU (ctm.py lines 393–404).

The “neuron is a scalar” constraint: NLM input ((B, D, M)) means each neuron is merely a scalar time series — a prerequisite for understanding why Direction 1’s error signal must operate per-neuron.

1.3.3 Synchronization Strategies: Three `neuron_select_type` Variants (`ctm.py` lines 450–486)

Strategy	Left set	Right set	Sync shape	Use case
`first-last`	first (N_{\text{out}}) / last (N_{\text{action}})	same as left	(N(N+1)/2) (outer product upper triangle)	early paper experiments, fixed structure
`random`	randomly selected (N)	independently randomly selected (N)	(N(N+1)/2) (outer product upper triangle)	randomness provides diversity, ~2x bottleneck
`random-pairing` (default)	randomly selected (N)	first `n_self` same as left + rest random	(N) (elementwise product)	controllable bottleneck, output dim = (N)

random-pairing is the paper’s final strategy (lines 72–74 comment). n_random_pairing_self controls the number of i-i self-pairings — at 0, self-pairings are rare, making “snapshot representation” recovery difficult (lines 76–78). For Direction 1’s error definition, design on random-pairing since sync dim = n_synch rather than growing quadratically.

1.3.4 Certainty: Normalized Entropy Numerics and Hidden Pitfalls (`models/utils.py` lines 42–72)

compute_normalized_entropy softmaxes logits, computes entropy, and divides by the upper bound (\ln C) to normalize to ([0,1]). For multi-dimensional logits (e.g., multi-token sequences), it does flatten(1).mean(-1) over non-batch dims. Pitfall: when the class count (C) is large (e.g., ImageNet 1000 classes), most class probabilities are near 0, and initial certainty is artificially high — because normalized entropy (H_n = 1 - H/H_{\max}) is 0 for uniform and 1 for one-hot, but uniform in high-dimensional space has extremely low probability density; in practice even “uncertain” states tend to concentrate on a few classes, inflating certainty. Evaluate certainty jointly with random baselines and task difficulty; never use the raw ([0,1]) range in isolation.

1.4 Structural Deficiencies of Open-Loop Prediction

CTM’s loop is an open-loop extrapolator: each step (z_t) is determined purely by history through a fixed mapping, with no mechanism to measure the consistency between “the model’s own dynamical prediction” and “the model’s actual activity.” Four problems arise:

No internal supervision: the only supervision comes from the final task loss, backpropagated through (T) time steps (BPTT). Gradients are repeatedly compressed by U-Net depth and NLM nonlinearity — effective gradients at deep time steps are extremely weak.
No adaptive computation: regardless of input difficulty, the model always runs a fixed (T) steps; easy samples waste computation, hard samples may need more.
No internal convergence signal: certainty exists on the output side, but the dynamics side (whether state is stable, whether predictions are self-consistent) has no metric — the model cannot “know” whether it has formed a stable internal representation.
Dynamics are uninterpretable: synchronization is an implicit statistic of activity; researchers can only post-hoc visualize, not directly read “at which step, about what, was the model surprised.”

These four points respectively motivate the four optimization directions in this article.

2. Theoretical Background: Predictive Coding and Dual Pathways

2.1 Predictive Coding

Neuroscience foundation (Rao & Ballard, 1999; Friston, 2005): each cortical layer maintains a prediction of the next layer’s activity; only prediction errors propagate upward; learning minimizes prediction error (equivalent to maximizing generative-model evidence lower bound, i.e., the free-energy principle). Inference itself is a “predict-correct” iteration: state is corrected by error signals until error vanishes — at which point the internal representation is self-consistent with the generative model.

Three core components from engineering implementation (v_predictive):

Predictive coding layer: (z = \text{ReLU}(\text{LN}(W_e x))), prediction (\hat{x} = W_p z), error (e = x - \hat{x}).
Error-driven update (UpdateRule): (h \leftarrow h + \alpha \cdot \text{MLP}(e)). Key design: MLP uses bias=False. This guarantees (e = 0 \Rightarrow \text{MLP}(0) = 0 \Rightarrow h_{\text{new}} = h) — i.e., zero error yields a strict fixed point; convergence equals prediction self-consistency. (\alpha) is constrained positive via Softplus.
Iterative inference (InferenceLoop): state iterates per the above rule; stops early when error norm falls below threshold; BN statistics are frozen during the inference loop.

2.2 Dual Pathways

Neuroscience foundation (Ungerleider & Mishkin, 1982): the ventral pathway (“What”) processes object identity and semantics; the dorsal pathway (“Where/How”) processes spatial location and motor control.

Engineering implementation (v_dual):

Role separation: the ventral stream receives semantic input with frozen weights; the dorsal stream receives spatial input (residuals + 2D coordinates) and never receives semantic symbol IDs (enforced by runtime assertion).
Direction-aware bias (RelativeDirectionalBias2D): FH region tokens are irregularly shaped with no fixed grid, making traditional grid biases unusable. Solution: pairwise token distance (logarithmic bucketing) + direction (angular bucketing) jointly quantized → lookup in a learnable bias table, preserving directional information.
Cross-stream fusion (DualPathwayFusion): bidirectional cross-attention + learnable gating (fused = \sigma(\alpha) v + (1 - \sigma(\alpha)) d), gate initialized at 0.5 (equal weight). none mode physically removes the fusion module for ablation.

2.3 Training Stability

Micro-parameterized MuPC: in deep predictive networks, gradient variance diverges exponentially with depth. Layer-(l) weight scaling (1/\sqrt{l+1}) (suppresses shallow gradient explosion), error scaling (1/\sqrt{L-l+1}) (alleviates deep gradient vanishing); per-layer gradient variance converges within 10x of each other.
BN freezing: during predictive-coding iterative inference, freeze BN running statistics to prevent inference trajectory drift. Note: CTM’s BN only exists in the backbone feed-forward (once, outside the loop), so this mechanism does not apply in CTM (see Direction 3’s honest assessment).

3. Direction 1: Prediction-Error-Driven Inner Loop (Recommended First)

3.1 Problem Formalization

CTM’s current loop: (z_t = \text{NLM}(\text{Shift}(A_{t-1}) \cup {a_t})), purely open-loop. No measure of deviation between “expected” and “actual.” Predictive coding requires introducing error (e_t) and using it to drive state correction and learning.

3.2 Precise Temporal Analysis of Error Definition

In CTM’s loop, determine the timing of the “predict-observe” pair:

At each step t, the loop body (in execution order):
  synchronisation_action = f(z_{t-1})       # based on previous state
  o_t = Attention(q(S_action_t), kv)        # acquire context from data
  a_t = Synapses([o_t; z_{t-1}])            # pre-activation
  z_t = NLM([a_{t-M+1}, ..., a_t])          # post-activation (observation)
  S_out_t = f(z_t), y_t = W_o(S_out_t)      # output

Prediction timing: after obtaining (o_t) and before computing (a_t), the model can predict (z_t) using (z_{t-1}) and (o_t). This defines:

[ \hat{z}t = P(z{t-1}, o_t) \quad (\text{prediction}), \qquad e_t = z_t - \hat{z}_t \quad (\text{error}), \qquad z_t^{\text{corr}} = z_t + \alpha \cdot \text{MLP}_e(e_t) \quad (\text{correction}) ]

Intuition: given the previous internal state (z_{t-1}) and the current external context (o_t), predictor (P) predicts “the next state that dynamics + context would produce.” If the actual (z_t) differs from the prediction, a “surprise” has occurred — this is the predictive-coding surprise signal.

(P)‘s computation happens after attention and before synapses, not on the critical path (the correction term MLP_e is off the critical path and can run in parallel with synch_out).

3.3 Variant Designs

Variant A: Lagged Error with Zero New Parameters (ablation baseline)

[ \hat{z}t = \text{NLM}(A{t-1}), \qquad e_t = z_t - \text{sg}(\hat{z}_t) ]

At each step, run an extra NLM forward pass: truncate history to state_trace[:, :, :-1] (or a smoothed version), obtaining a one-step-lagged NLM prediction. No new weights needed, but the extra NLM forward doubles NLM computation at that step. When state converges, (e_t \to 0) (adjacent NLM outputs converge).

Variant B: Independent Predictor + Error Correction (primary proposal)

Add two lightweight modules:

Predictor (P): Linear(D + d_input → 2H) → GLU → Linear(H → D) (suggest (H) match memory_hidden_dims).
Corrector (\text{MLP}_e): Linear(D → 2H) → GLU → Linear(H → D), bias=False, ensuring (e=0) yields zero correction.
Learnable correction step (\alpha): Softplus-parameterized, initial ~0.1.

The corrected activation enters synchronization computation and the next loop iteration; the error signal genuinely influences dynamics (not a bypass).

Variant C: Auxiliary Loss Edition (stackable with B)

[ \mathcal{L} = \mathcal{L}{\text{task}} + \lambda \cdot \frac{1}{T} \sum{t} | e_t |_2^2 ]

Suggest starting (\lambda) at 0.005. Directly minimizes internal prediction error, making the model indirectly learn self-consistent dynamics while completing the task. Training-only.

3.4 Training-Inference Decoupling Strategy

This is the most critical engineering decision in Direction 1. If the correction term is active during both training and inference, three problems arise:

BPTT truncation: the correction term changes (z_t)‘s value but not its gradient path (if (e) is detached), causing training-inference path misalignment.
Predictor-state collusion: if (e)‘s gradient flows into both (P) and NLM/Synapses, they may mutually accommodate, hollowing out the error signal.
Distribution shift: during training, weights are constantly updating and (P)‘s prediction ability is unstable; during inference, weights are frozen and (P) operates on converged dynamics.

Recommended decoupling scheme:

Phase	Correction term	Predictor (P)	Auxiliary loss
Training	disabled	trained via (\mathcal{L}_{\text{pc}})	enabled (Variant C, small (\lambda))
Inference	enabled (Variant B)	frozen (no param updates)	disabled

During training, (P) learns dynamics solely through the auxiliary loss without participating in state correction in the main loop (avoiding interference with BPTT and task performance). During inference, (P) parameters are frozen and the correction term fine-tunes the state in a pure feed-forward manner, aiding convergence.

Implementation mechanism: self.training flag controls whether the correction term takes effect. Additionally, an independent use_error_correction parameter enables ablation experiments where correction is enabled during training.

3.5 Detailed Gradient Flow

Assuming both correction term and auxiliary loss are enabled (Variant B+C + training), the complete gradient flow is as follows. Training-time correction is recommended to be off by default, but the following analysis provides theoretical grounding for ablation scenarios.

Path 1: task loss → z_t → … → parameters (core path) Task loss backpropagates through predictions → synch_out → z_t → NLM/Synapses. If the correction term uses e.detach() (stop-gradient), this path is unaffected — but the relationship between corrected and uncorrected (z) is severed. If detach is not used, (e)‘s gradient on (z) (via the correction term (\alpha \cdot \text{MLP}e)) adds to the task gradient, potentially pushing (z) toward “satisfying prediction” rather than “completing the task.” Solution: the correction term is always based on (\text{sg}(e)) (stopping gradient from correction to (P)), but (P)‘s training receives gradients from (z_t) (not detached) through the auxiliary loss (\mathcal{L}{\text{pc}} = |z_t - P(z_{t-1}, o_t)|^2), so (P) learns the true dynamics.

Path 2: auxiliary loss → P → parameters (training the predictor) (\mathcal{L}{\text{pc}})‘s gradient flows into (P) (encouraging accurate prediction), not into (z) (since (z) is not a parameter of (P), but (z) as the target in (\mathcal{L}{\text{pc}})‘s MSE — the gradient w.r.t. (z) should also not be backpropagated: (z) is an observation target and should not be altered to accommodate (P). Thus (z_t) in (\mathcal{L}_{\text{pc}}) should also be detached). End result: (P) is trained by comparing “true (z) (detached)” vs. “predicted (z),” neither distorting NLM nor allowing (P) to degenerate.

Path 3: correction term → MLP_e → parameters (training the corrector, ablation-only) The corrector MLP_e is trained end-to-end via task loss gradients (through corrected (z) influencing outputs). Since MLP_e’s input is (\text{sg}(e)), gradients do not flow into (P) or any module of the previous state. MLP_e purely learns “what correction best aids the downstream task.”

Simultaneous training of all three paths is not recommended. Standard protocol: training uses only path 2 (auxiliary loss trains (P)); inference activates the path-3 direction (correction term, pure feed-forward).

3.6 Fixed Points and Convergence Cascade: Theoretical Bridge Between Direction 1 and Direction 4

When internal dynamics converge, (z_t \to z^*), (e_t = z_t - P(z_{t-1}, o_t) \to 0) ((P)‘s prediction matches reality), the correction term vanishes, and state enters a fixed point. State fixedness implies the synchronization recurrence stabilizes ((S_t \to S_{t-1})), output predictions become consistent, and certainty rises monotonically.

[ |e_t| \to 0 \iff z_t \to z^, \quad S_t \to S^, \quad c_t \to c^* ]

Thus error convergence and certainty convergence are theoretically isomorphic — the dual-signal stopping condition (“internally self-consistent and output-certain”) is the natural joint interface between Direction 1 and Direction 4.

3.7 Source Locations and Implementation Checklist

Change	Location	Notes
Insert prediction/correction into loop body	`models/ctm.py` `forward` lines 560–590	After attention, before synapse: `z_hat = P([z_{t-1}; o_t])`; after NLM: error correction
New predictor (P) module	end of `models/modules.py`	`nn.Sequential`, style consistent with NLM’s SuperLinear+GLU
New corrector MLP_e	end of `models/modules.py`	`bias=False` (fixed-point guarantee), Softplus-ized alpha
Auxiliary loss parameter lambda_pc	`__init__` lines 81–101	float, default 0.0 (off)
(P) registration and state_dict compatibility	`__init__` and `_from_pretrained` (lines 153–198)	New modules must be registered in `__init__` for HuggingFace Hub loading compatibility
Auxiliary loss injection	each task’s `train.py`	Add `lambda_pc * pc_loss` to existing loss

Pseudocode (core loop-body modification):

# After attention at line 567
z_prev = activated_state  # z_{t-1}
attn_out = attn_out.squeeze(1)

# Predict next state
if hasattr(self, 'predictor'):
    z_hat = self.predictor(torch.cat((z_prev, attn_out), dim=-1))
    pc_loss = torch.tensor(0.0, device=z_prev.device)
else:
    z_hat = None
    pc_loss = None

# Existing synapse + NLM path
pre_synapse_input = torch.cat((attn_out, z_prev), dim=-1)
state = self.synapses(pre_synapse_input)
state_trace = torch.cat((state_trace[:, :, 1:], state.unsqueeze(-1)), dim=-1)
activated_state = self.trace_processor(state_trace)

# Error correction
if z_hat is not None and self.use_error_correction_inference:
    e = activated_state - z_hat.detach()
    activated_state = activated_state + self.alpha_correction * self.mlp_e(e)
    if self.training:
        # Auxiliary loss: prediction vs. reality (each side detaches the other's branch)
        pc_loss = ((activated_state.detach() - z_hat) ** 2).mean()
else:
    if self.training and z_hat is not None:
        pc_loss = ((activated_state.detach() - z_hat) ** 2).mean()

3.8 Expected Benefits and Risks

Benefits: obtain a “surprise” signal (|e_t|) (interpretable, visualizable); dynamics self-consistency explicitly optimized; linkage with certainty enables adaptive computation.
Risks: the correction term alters synchronization dynamics, potentially affecting converged tasks (requires ablation); predictor training needs extra hyperparameter tuning ((\lambda_{\text{pc}}), independent learning rate for (P)); training/inference decoupling increases implementation complexity.
Minimum-risk path: implement Variant A first (pure signal), verify correlation between error and convergence, then decide whether to upgrade to Variant B.

4. Direction 2: Dual-Pathway Architecture (Ventral Semantics + Dorsal Spatial)

4.1 Problem Diagnosis

CTM processes all information in a single stream: backbone features (semantic + implicit spatial) are mixed with internal state through attention in a single state space. For strongly spatial tasks (mazes, RL), there is no independent spatial representation channel; positional encoding is merely injected information (compute_features does kv + pos_emb directly, lines 276–277) and does not constitute a reason-able spatial state.

4.2 Target Architecture

                Input x
                  │
        ┌─────────┴──────────┐
        │                    │
   Ventral (semantic)    Dorsal (spatial)
   Backbone(x)           Coord(x) / PE(x)
        │                    │
     kv_proj              W_spatial
        │                    │
   kv: (B,S,d_in)     z^s: (B,D_s) ← independent sync/state
        │                    │
        └──────┬─────────────┘
               │
        Cross-attention fusion
        fused = σ(α)·v + (1-σ(α))·d
               │
         Synapses / NLM / sync / output

4.3 Phased Landing

Phase A: Coordinate Channel Injection (minimal change, exploratory experiment)

In compute_features (lines 269–278), explicitly concatenate normalized coordinates to the backbone feature map:

Reuse add_coord_dim (models/utils.py lines 18–40, generates ((B, 1, H, W, 2)) normalized coordinates).
Concatenate feature map ((B, C, H, W)) with ((B, 2, H, W)) along the channel dimension → flatten → token.
~10 lines changed; validates the value of “explicit spatial information injection” itself.

Phase B: Full Dual-Stream (architecture-level)

Dorsal state (z_t^{\text{spatial}}): initialized from coordinate/position features via projection, independent sync computation (reuse compute_synchronisation, independent neuron_select_type configuration). Dorsal stream never receives semantic information.
Direction-aware bias: for regular grid features from the backbone, can degrade to traditional relative position bias (grid-based); for irregular spatial tasks (mazes, etc.), upgrade to distance + angle jointly quantized bucket bias (v_dual RelativeDirectionalBias2D), bucket size recommended 8x8=64.
Fusion module: bidirectional cross-attention + gating (fused = \sigma(\alpha)v + (1-\sigma(\alpha))d) at the end of each loop step. (\alpha) initialized to 0 (sigmoid=0.5 equal weight); sequence length alignment via adaptive average pooling.
Fusion output enters the next Synapses input, maintaining compatibility with the existing interface.

4.4 Expected Benefits and Risks

Benefits: spatial tasks gain an analyzable “internal map” representation; gating weights provide inter-stream weight interpretability; compatible with v_dual’s ventral-freezing strategy (train CTM, then freeze semantic stream for incremental learning).
Risks: dual states + fusion + bias table add significant parameters; Phase B architectural changes must be validated after Direction 1.

5. Direction 3: Training Stability Optimization

5.1 The BPTT Gradient Dilemma

CTM training is equivalent to (T)-step BPTT. Gradient propagation equals the product of per-step Jacobians, each going through SynapseUNET’s multiple layers + NLM nonlinearity — a double compression in depth and time:

[ \frac{\partial \mathcal{L}}{\partial \theta} = \sum_{t} \frac{\partial \mathcal{L}}{\partial z_T} \prod_{k=t}^{T-1} \underbrace{\frac{\partial z_{k+1}}{\partial z_k}}_{\text{Synapses(N layers)+NLM}} \cdot \frac{\partial z_t}{\partial \theta} ]

Two specific gradient degradation points:

Depth dimension: SynapseUNET’s down_projection Linear(int(w[i]), int(w[i+1])) width decay can cause deep gradient explosion/vanishing. Skip connections mitigate but don’t cure.
Time dimension: the synchronization recurrence’s (r < 1) gives higher weight to recent activity forward, but backward propagation has no temporal gradient regularization — gradients at farther time steps naturally decay, determined by the synchronization signal itself (earlier steps’ activity has less influence).

5.2 MuPC Landing Design

Inject micro-parameterization into SynapseUNET’s per-level projections:

Method A (initialization scaling, recommended first try): in SynapseUNET.__init__ (modules.py lines 86–110), multiply per down/up projection weight initialization by (1/\sqrt{l+1}), where (l) is the zero-based level index. Forward behavior unchanged; only gradient magnitude changes.
Method B (gradient scaling hooks): register backward_hook, zero per-level gradients after backward and rescale. More precise but more invasive.
MuPC error-signal scaling ((\text{error_scale}[l] = 1/\sqrt{L-l+1})): in SynapseUNET, this can be applied to scale the gradient on the skip-connection side — gradients from deeper layers on the Up path need amplification to match shallow-layer gradients.

5.3 BN Freezing: An Honest Conclusion

CTM does not need BN freezing. Three code facts:

BatchNorm exists only in the backbone (ShallowWide’s BatchNorm2d, ResNet internals); compute_features (ctm.py lines 273–274) runs once outside the loop.
Inside the loop (Synapses uses LayerNorm + SiLU + Dropout; NLM optionally uses LayerNorm), no BN running statistics are updated.
Therefore the premise “BN statistics drift during iterative inference” does not hold.

Only edge case: if backbone fine-tuning is enabled (PretrainedResNetWrapper with fine_tune=True), training/inference statistical discrepancy still exists, but this is a general BN problem, not CTM-specific.

5.4 NLM LayerNorm Experiment

do_layernorm_nlm=True applies LayerNorm to SuperLinear’s input (history dimension) (modules.py line 195) — equivalent to per-neuron history whitening. The authors note it “may encourage more periodic dynamics” but never used it in the paper. Low-cost experiment: compare on/off on the parity task; visualize synchronization trajectories.

6. Direction 4: Certainty Early Stopping and Adaptive Inference

6.1 Existing Evidence

The official analysis script (tasks/image_classification/analysis/run_imagenet_analysis.py lines 200–256) compares: instantaneous prediction, probability averaging, most-certain prediction, and certainty-weighted prediction. Results show certainty-guided strategies outperform instantaneous — the model’s decision quality is higher at its “most certain” moment. Additionally, thresholded (0.5/0.8/0.9) correctness/error statistics (lines 265–279) directly demonstrate that certainty is positively correlated with accuracy. Yet the training/inference loop has never utilized this signal.

6.2 Early-Stop Algorithm Design

New parameters (all off by default, preserving existing behavior):

early_stop_certainty: threshold (\tau); None = off
early_stop_patience: consecutive-satisfaction count (k); default 2
early_stop_fill: fill strategy 'last' / 'best' / 'none'

Algorithm:

running = torch.zeros(B, dtype=int)
best_step = torch.zeros(B, dtype=int)
for t in range(iterations):
    if active_mask.sum() == 0: break
    # ... only execute loop body for active samples ...
    c = certainties[active, 1, t]
    running[active] = torch.where(c > tau, running[active] + 1, 0)
    best_step[active] = torch.where(c > c_best[active], t, best_step[active])
    stopped = (running >= k)  # (B,)
    active_mask = active_mask & ~stopped
# Fill
if fill == 'last':
    for done_t in range(t+1, iterations):
        predictions[:, :, done_t] = predictions[:, :, t_break]

Key details:

In track=True mode, stopped samples have no subsequent records (the caller must handle varying lengths).
Unconditionally off during training (BPTT path integrity is mandatory).
Compatible with _from_pretrained: new parameters have defaults; state_dict lacks new buffers; old weights load normally.
Actual speedup depends on sparse computation: skip synapse/NLM/attention for active_mask=0 samples, filling only — if you just fill without skipping on already-stopped samples, no computation is saved.

6.3 Taxonomy of Certainty Temporal Evolution

Experimentally, certainty trajectories typically exhibit three types (distribution across CTM tasks needs confirmation):

Type	Trajectory	Example task	Early-stop strategy
Monotonic rise	from low certainty, monotonically converging to a high plateau	simple classification samples	low threshold + low patience
Oscillatory convergence	rises then fluctuates slightly at a high level	blurry/noisy inputs	moderate patience (need stability)
Late jump	long low-certainty period, sudden spike at some step	logical reasoning (parity/maze)	high patience + high threshold
Persistent low certainty	perpetually uncertain (hard sample)	noisy/anomalous inputs	no early stop; use as detection signal

The optimal threshold (\tau) and patience (k) should be tuned against the target task’s certainty distribution, not a single fixed value. Experimentally, first compute certainty percentiles on the validation set, then set (\tau) (e.g., P95).

6.4 Adaptive Threshold Strategy

Beyond a fixed threshold, one can learn a dynamic threshold. One lightweight scheme: compute per-step moving-average certainty (\bar{c}_t); when (c_t - \bar{c}_t > \delta), treat it as a “significant boost” and start counting patience. This is more robust than a fixed absolute threshold, adapting to varying input difficulty.

6.5 Computational Benefit Estimation

Per-step loop cost = sync ((O(D \cdot n_{\text{synch}}))) + attention ((O(S \cdot d_{\text{input}}))) + SynapseUNET ((\approx O(D^2)) level) + NLM ((O(D \cdot M \cdot H))). Benefit depends on the loop’s share of total computation in the task:

parity / sort / qamnist: backbone is tiny; loop dominates. A 40% step saving ≈ 40% total inference time.
ImageNet: backbone (ResNet) feed-forward accounts for 60–80% of total; loop accounts for 20–40%; early-stop benefit is relatively smaller but still quantifiable (e.g., for large images with n=224, more tokens mean higher loop cost).
RL: per-decision loop determines latency; early stopping directly reduces environment-interaction delay.

Evaluation method: follow the analysis script’s method (lines 265–279); plot “threshold–accuracy” trade-off curves on the validation set to determine the maximum early-stop ratio without accuracy loss.

6.6 Joint Stopping with Direction 1

Dual-signal stopping condition:

[ \text{stop}_t = (|e_t| < \varepsilon) \wedge (c_t > \tau) \wedge (\text{running_count} \ge k) ]

“Internally self-consistent AND output-certain” before stopping — high certainty but unstable state (early accidental high confidence) or stable state but uncertain output (sync not converged) will both avoid false stops. (\varepsilon) and (\tau) are jointly searched on the validation set.

7. Theory Assumptions vs. Code Facts

During landing, the following theoretical intuitions deviate from CTM’s implementation and have been corrected in the proposals:

Theoretical assumption	CTM code fact	Corrected landing approach
Neurons have vector states	Neurons are scalars; state ((B, D)) is a vector; NLM input ((B, D, M)); `SuperLinear` weights ((M, H, D))	All error signals designed as per-neuron scalar operations
Synchronization is a learnable coupling matrix	Synchronization is a fixed-structure exponentially decayed recurrence of pairwise activation products (`compute_synchronisation`)	Don’t directly modify sync; inject directional/topological bias into synapse or attention (Direction 2)
Need a new error norm as convergence signal	Already have `certainty = 1 - H_n` (`compute_certainty`)	Directly reuse certainty for early stopping; Direction 1’s error signal as supplement
BN freezing solves inference drift	BN only in backbone feed-forward (once, outside loop); no BN inside loop	Abandon BN freezing; pivot to MuPC + NLM LayerNorm experiments
Error signal usable for real-time correction during training	BPTT path must be intact; training-time correction truncates gradients	Training-inference decoupling: training has no correction (auxiliary loss only); inference has pure correction
Certainty range ([0,1]) is cross-sample comparable	`compute_certainty`’s normalized entropy denominator = (\ln C); artificially inflated for high class counts (Section 1.3.4)	Evaluate jointly with random baselines and task difficulty; never use absolute values in isolation

8. Potential Pitfalls and Debugging Guide

8.1 Predictor Degeneration (Direction 1)

Symptom: after some training steps, error (|e_t|) rapidly tends to 0 but task performance does not improve. Cause: predictor (P) degenerates — learns identity mapping (P(z_{t-1}, o_t) \approx z_t), or outputs a constant during inference. Diagnosis: compare (P)‘s prediction variance across different batches (near-zero variance → degenerated to constant); compare (P)‘s error distribution between train and validation sets (significantly larger on validation → overfit to training dynamics). Fix: (1) add dropout to (P) (already recommended); (2) increase (\lambda_{\text{pc}}) (encourage (P) to genuinely learn); (3) add L2 regularization to (P) to prevent degeneration.

8.2 Error Term Dominating Task Gradient (Direction 1)

Symptom: task accuracy drops after enabling auxiliary loss. Cause: (\lambda_{\text{pc}}) is too large; internal self-consistency loss overwhelms the task signal; the model prioritizes “perfect internal prediction” (degenerating to constant) over “completing the task.” Diagnosis: compare the relative decline rates of (\mathcal{L}{\text{task}}) and (\mathcal{L}{\text{pc}}) under different (\lambda_{\text{pc}}) values. Fix: (1) start from tiny values (0.001, 0.005, 0.01); (2) cosine-decay (\lambda_{\text{pc}}) (strengthen internal constraints only late in training after task loss stabilizes); (3) use a 10x lower learning rate for (P) than the main model to avoid hijacking the optimization direction.

8.3 Correction Term Inference Misalignment (Direction 1)

Symptom: enabling correction during inference degrades performance. Cause: during training, (P) sees “(z) from training dynamics”; during inference, it sees “(z) after convergence” — distribution shift. Diagnosis: compare the mean and variance of error on the same batch in train vs. eval modes (use model.eval() + torch.no_grad() vs. normal training snapshot). Fix: enable the correction term during the last 10–20% of training steps and freeze (P) (inference-aligned fine-tuning), letting MLP_e adapt to the inference distribution.

8.4 Certainty Threshold Mis-Setting (Direction 4)

Symptom: early stop too early (accuracy drops) or too late (no benefit). Diagnosis: (1) plot dual curves: “threshold vs. early-stop ratio” and “threshold vs. accuracy”; (2) compute the distribution of certainty on correct vs. incorrect samples (continuing the analysis script’s method at lines 265–279). Guidance: set the threshold such that among early-stopped samples at that threshold, the error rate does not exceed the allowed ceiling (e.g., 1%). Set patience (k) to at least cover certainty’s normal jitter cycle (infer by visualizing the autocorrelation of certainty[1, :]).

8.5 MuPC Suppressing Normal Learning (Direction 3)

Symptom: SynapseUNET converges slower or final performance drops after applying MuPC scaling. Cause: the scaling factor is too aggressive, equivalently lowering the effective learning rate of certain layers. Fix: (1) start from very mild scaling (e.g., (1/\sqrt{l+2}) instead of (1/\sqrt{l+1})); (2) only scale the shallowest/deepest two levels (keep intermediate levels unchanged); (3) make the scaling learnable per-layer parameters rather than fixed scaling.

9. Roadmap and Experimental Design

9.1 Priority Ordering (low-to-high risk)

Phase	Content	Timeline	Risk	Dependencies
P0	Certainty early stop (Direction 4)	1–2 days	zero architectural risk, quantifiable benefit	none
P1a	Prediction error signal (Direction 1, Variant A)	3–4 days	low (zero parameters)	none
P1b	Prediction error correction (Direction 1, Variant B/C)	1 week	medium, needs ablation	P1a
P2	MuPC + NLM LayerNorm experiment (Direction 3)	1 week	low–medium	none
P3	Dual-pathway Phase A → Phase B (Direction 2)	2–4 weeks	high	P1

9.2 Experimental Protocol

Benchmark tasks and metrics:

Task	Script path	Core metric	Theory direction verification
parity	`tasks/parity/`	accuracy, error convergence curve	Direction 1 (internal dynamics precisely testable)
mazes	`tasks/mazes/`	routing success rate	Direction 2 (spatial reasoning)
CIFAR/ImageNet	`tasks/image_classification/`	top-1/5, stopping-step distribution	Direction 4 (perceptual early-stop benefit)
CartPole	`tasks/rl/`	episode reward, decision latency	Direction 1+4 (policy learning)

Ablation matrix (each dimension independently switchable):

Configuration	Error correction	Auxiliary loss	MuPC	Early stop	Dual-pathway
Baseline	off	off	off	off	off
+Error	off (inference: on)	off	off	off	off
+Error+Loss	off (inference: on)	on	off	off	off
+Stability	off (inference: on)	on	on	off	off
+EarlyStop	off (inference: on)	on	on	on	off
All-on	off (inference: on)	on	on	on	Phase B

Uniform experimental discipline:

Fix seed, data split, training steps, and scheduler (utils/schedulers.py), consistent with the paper’s reproduction configuration.
At least 3 seeds per configuration; report mean and variance.
Visualization: use track=True to record per-step activations, sync, and attention (ctm.py lines 593–598); reference the GIF-generation scripts in tasks/parity/analysis/ and tasks/qamnist/analysis/.
All changes preserve default behavior (new parameters default off), compliant with the repo’s PR policy (README lines 143–155: additive features, quantitative evidence, no default-behavior changes).

10. Part 1 Conclusion

CTM’s iterative prediction structure has a natural mapping relationship with predictive coding theory: each internal tick is a prediction, but the explicit error-feedback closed loop is missing. Among the four directions, certainty early stopping (P0) and prediction-error-driven inner loops (P1) are the two paths with the highest theory-value-to-engineering-cost ratio: the former directly leverages the model’s existing convergence signal for adaptive inference; the latter injects the core predictive coding mechanisms (error-driven update, stable fixed point) into the model’s dynamical core. The two theoretically form a convergence cascade of “internal self-consistency → output certainty.” The third and fourth directions respectively guarantee numerical training stability and architectural support for spatial tasks.

Anchoring in code facts and verifying in phases is the key to truly landing theory on CTM.

Part 2: Six-Direction CTM Compatibility Audit — Top-Down from Theory to Code

11. Preface to Part 2

In Part 1, I approached from the perspective of “from code facts to theory,” proposing four optimization directions by analyzing CTM’s internal deficiencies. Part 2 adopts the opposite perspective: assuming a top-down theoretical proposal has already taken shape — systematically mapping the v_predictive / v_dual / v_engine architectural paradigm onto CTM — what happens to each assumption of this proposal when it meets CTM’s actual code? Which parts connect directly, and which require correction?

The proposal’s six directions are as follows:

The Proposal's Core Mapping
─────────────────────────────────────────────────────────────
Ventral pathway (What, frozen)        →   Semantic neuron group (frozen bottom NLM)
Dorsal pathway (Where/How, trainable) →   Spatial neuron group (trainable top NLM)
Predictive coding layer (error = x - pred) → Mutual-prediction NLM (error = actual input - neighbor prediction)
Hierarchical anchors (three-tier freeze)   → Neuron group hierarchical organization (general→domain→task)
Direction-aware 2D bias                   → Topology-aware sync coupling (spatial proximity→easy sync)
InferenceLoop (error convergence)         → Sync pattern convergence criterion (dSync/dt < epsilon)
Generative replay                          → Frozen-group sampling replay (data-free continual learning)

Methodology of Part 2: for each direction, first present the minimal viable implementation (MVP) code-change checklist, then correct each misalignment with CTM code facts, and finally give a verification plan. All source locations reference models/ctm.py (lines 1–605) and models/modules.py (lines 1–693).

12. Direction 1 (of Part 2): Dual-Pathway CTM — From “Homogeneous Neuron Soup” to “Ventral-Dorsal Separation”

Proposal core: explicitly split CTM’s neuron pool into a semantic group (NLM wide-shallow) and a spatial group (NLM narrow-deep); within-group sync is fully connected; cross-group sync uses a sparse cross matrix.

What I Agree With

The ventral-dorsal separation design philosophy is correct. CTM’s activated_state is a flat ((B, D)) vector; all neurons are functionally homogeneous. The “looking around” behavior in ImageNet experiments is a statistical emergent byproduct, not an architectural guarantee. For strongly spatial tasks (maze/RL), lacking an independent spatial representation channel is a clear bottleneck.

MVP Implementation Checklist

Change	Location	Notes
Neuron pool split	`__init__` lines 81–101	New `split_ratio` parameter; split `semantic_indices` and `spatial_indices` buffers
Dual NLM modules	replace existing `trace_processor`	Semantic and spatial groups each get an independent `trace_processor`, independently configurable `memory_hidden_dims` and `deep_nlms`
Coordinate channel separation	`compute_features` lines 269–278	Split `kv + pos_emb` into `kv_semantic` and `kv_spatial` two paths
Dual attention	`forward` loop lines 560–590	Semantic group attends to `kv_semantic`; spatial group attends to `kv_spatial` (or share kv parameters to reduce param count)
Cross-group modulation	`compute_synchronisation` lines 202–267	See Key Correction 1.2 below

Key Correction 1.1: Coordinate Channel Separation Replaces VSS Input Routing

Misalignment: the proposal’s input routing scheme (image → VSS → symbol_ids + residuals + coords) relies on v_predictive’s FH segmentation + Lab quantization frontend, which CTM does not possess. Introducing VSS would be an external independent encoder, with changes far exceeding “Direction 1” scope.

Correction: don’t split the input; only split the neuron pool + separate attention channels. CTM’s compute_features line 276 is already kv + pos_emb directly added — just change it to two independent paths:

# Existing code (ctm.py lines 274–278)
pos_emb = self.positional_embedding(self.kv_features)
combined_features = (self.kv_features + pos_emb).flatten(2).transpose(1, 2)
kv = self.kv_proj(combined_features)

# Modified
pos_emb = self.positional_embedding(self.kv_features)
# Semantic channel: raw features dominate
kv_semantic = self.kv_proj_semantic(
    (self.kv_features + 0.3 * pos_emb).flatten(2).transpose(1, 2)
)
# Spatial channel: position encoding dominates
kv_spatial = self.kv_proj_spatial(
    (0.3 * self.kv_features + pos_emb).flatten(2).transpose(1, 2)
)

~15 lines changed; no new encoder introduced; fully compatible with the existing training pipeline. The semantic/spatial weight ratio (0.3) is a configurable parameter for ablation.

Key Correction 1.2: Cross-Group Modulation Injects Pairwise Product, Not a Separate Matrix

Misalignment: the proposal’s cross_sync = nn.Parameter(torch.randn(n_semantic, n_spatial)) implies synchronization is a learnable matrix, but CTM’s synchronization is a recurrent process (compute_synchronisation, lines 202–267) and does not maintain an explicit coupling matrix.

Correction: superimpose a cross-group modulation scalar on the pairwise product (p_t). CTM’s recurrence formula:

[ \alpha_t = r \alpha_{t-1} + p_t, \quad \beta_t = r \beta_{t-1} + 1, \quad S_t = \frac{\alpha_t}{\sqrt{\beta_t}} ]

where (p_t = z_L \odot z_R) (elementwise product for random-pairing; flattened upper triangle of outer product for first-last/random). Inject cross-group modulation:

[ p_t^{\text{mod}} = z_L \odot z_R \odot m_{ij}, \quad m_{ij} = \text{Softplus}(w_{ij}) > 0 ]

(m_{ij}) is produced by a learnable parameter (w_{ij}) through Softplus (constrained positive), ensuring modulation never reverses the synchronization direction. When a neuron pair crosses groups (semantic-spatial), (m_{ij}) takes effect; within-group pairs get (m_{ij} = 1) (or independently learnable).

Code modification location: ctm.py line 252 pairwise_product = left * right, changed to:

if self.use_cross_group_modulation:
    # Query whether this neuron pair crosses groups
    left_group = self.neuron_group[left_indices]   # 0=semantic, 1=spatial
    right_group = self.neuron_group[right_indices]
    is_cross = (left_group != right_group).float()
    # Modulation factor: cross-group pairs use learnable weight; within-group pairs stay 1.0
    modulation = is_cross * self.cross_mod_weights + (1 - is_cross) * 1.0
    pairwise_product = left * right * F.softplus(modulation)
else:
    pairwise_product = left * right

The recurrence structure (\alpha_t = r * \alpha_{t-1} + p_t^{\text{mod}}) and normalization remain unchanged. ~10 lines changed; can share the same modification point with Direction 4’s topology modulation (merged into a pairwise_modulation parameter).

Key Correction 1.3: NLM Structural Differences Are Experimental Hypotheses, Not Inevitable Design

“Semantic-group NLM wide-shallow” and “spatial-group NLM narrow-deep” are intuition-driven, not data-driven, design choices. Recommend making them configurable parameters — CTM’s existing get_neuron_level_models (ctm.py lines 383–413) already has deep_nlms and memory_hidden_dims parameters; per-group independence requires ~20 lines:

self.trace_processor_semantic = self.get_neuron_level_models(
    deep_nlms=self.deep_nlms_semantic, ...)
self.trace_processor_spatial = self.get_neuron_level_models(
    deep_nlms=self.deep_nlms_spatial, ...)

Let ablation experiments determine the optimal configuration.

13. Direction 2 (of Part 2): Predictive Coding NLM — Let Neurons “Predict Each Other’s Firing”

Proposal core: upgrade NLM from auto-regression to mutual prediction — each neuron not only encodes its own history but also predicts neighboring neurons’ input; error = neighbor’s true input - neighbor’s prediction; error drives state updates.

What I Agree With

This is the most core architectural innovation among the six directions. It directly injects the UpdateRule mechanism you validated in v_predictive ((h_{\text{new}} = h + \alpha \cdot \text{MLP}(error)), bias=False guaranteeing fixed point) into CTM’s dynamical core.

MVP Implementation Checklist

Change	Location	Notes
Neighbor index buffer	`__init__` around lines 131–148	Pre-select (k) neighbors per neuron, store as ((D, k)) index buffer
Neighbor predictor P_neighbor	`modules.py` new	`Linear(D * k → 2H) → GLU → Linear(H → D)`
Error corrector MLP_e	`modules.py` new	`Linear(D → 2H) → GLU → Linear(H → D)`, all `bias=False`
Loop body insertion	`forward` around lines 572–577	After `state = synapses(...)`, gather neighbor pre-activation, predict, compute error, correct
Auxiliary loss parameter	`__init__` lines 81–101	`lambda_pc_neighbor`, default 0.0

Key Correction 2.1: Pre-Activation Neighbors as Observation Signal

Misalignment: CTM has no native mechanism for “neuron A directly observing neuron B’s input.” The proposal’s neighbor_inputs needs an explicit quantitative definition.

Plan X (pre-activation neighbors, recommended baseline): after state = synapses(...) outputs (a_t) (ctm.py line 572), gather neighbors’ pre-activation using the pre-stored neighbor index:

# Insert after line 572
state = self.synapses(pre_synapse_input)  # (B, D)

if self.use_predictive_nlm:
    # Gather neighbor pre-activation
    neighbor_pre_activation = state[:, self.neighbor_indices]  # (B, D, k)
    # Predictor: from neighbor input, predict self state
    neighbor_flat = neighbor_pre_activation.reshape(B, D * k)
    pred_self = self.neighbor_predictor(neighbor_flat)         # (B, D)
    # Prediction error (detach target to prevent gradient backflow into the predictor distorting state)
    e_neighbor = state.detach() - pred_self                    # (B, D)
    # Correct self state
    state = state + self.alpha_neighbor * self.mlp_e_neighbor(e_neighbor)
    # Store prediction error for auxiliary loss (training only)
    if self.training:
        pc_neighbor_loss = (e_neighbor ** 2).mean()

Key design decisions:

The prediction target is self pre-activation (not neighbor pre-activation) — a clean implementation of mutual prediction: use neighbors’ pre-activation to predict “what my own pre-activation should be”; the error measures “the inconsistency between me and my neighbors’ predictions.”
state.detach() ensures the predictor (P) does not backpropagate gradients into the state (respecting the Section 3.5 gradient flow analysis).
bias=False in mlp_e_neighbor is a hard requirement (see 2.3).

Plan Y (post-activation neighbors, ablation alternative): use activated_state (previous timestep’s post-activation) to gather neighbor firing. The information is more “processed” (has passed through NLM nonlinearity) but lags one step temporally. Use as an ablation comparison.

Key Correction 2.2: Neighbor Selection Annealing

Early training uses random neighbors (matching CTM random-pairing’s training inertia); after training stabilizes, reselect based on similarity.

Similarity metrics (by priority):

NLM weight cosine distance: cos_sim(SuperLinear.w1[:, :, i], SuperLinear.w1[:, :, j]) — measures two neurons’ “processing preference” similarity. Weight shape ((M, H, D)), compared channel-by-channel.
Synchronization correlation: during training, track the mapping from synchronisation_out vector dimensions (exponentially decayed accumulation of pairwise products) to neuron pairs, identifying frequently synchronizing neuron pairs.
If Direction 4 is online, use Euclidean distance of topology coordinates.

Annealing schedule:

Epochs 0–20%: random neighbors (k=16), reselect each epoch.
Epochs 20–80%: sort by the above metric, take top-k most-similar neighbors, reselect every 5 epochs.
Epochs 80–100%: fixed neighbors, fine-tune.
If Direction 4 is online, final neighbors are determined by topology coordinates (spatial proximity = neighbors).

Key Correction 2.3: Structural Constraints on the Error Corrector

bias=False is the mandatory condition for the fixed-point guarantee. Verification reasoning:

Suppose state converges with (e_{\text{neighbor}} = \text{state} - \text{Pred}(\text{neighbors}) \to 0). Then:

If the corrector has bias (b \neq 0): MLP_e(0) = g(W * 0 + b) = g(b) \neq 0; the correction term is non-zero; state continuously drifts; no convergence.
If the corrector has no bias (b = 0): MLP_e(0) = g(0) = 0 (for activations like ReLU/GLU, zero input yields zero output); the correction term is strictly zero; state reaches a fixed point.

Therefore all nn.Linear layers in mlp_e_neighbor must explicitly use bias=False. Batch normalization layers should not appear in the corrector (their running statistics would drift during inference). The learnable step size alpha_neighbor is recommended to be Softplus-parameterized, initialized at 0.1.

Key Correction 2.4: Gradient Flow of Spatial-Dimension Error

Three gradient paths for spatial-dimension mutual prediction error (supplementing Section 3.5):

Path 1 (training (P)): auxiliary loss (\mathcal{L}_{\text{pc_neighbor}} = \text{MSE}(\text{state_detach}, \text{pred})) → gradient flows into neighbor_predictor. Since the target is detached, (P) purely learns the conditional mapping “infer self from neighbors.”
Path 2 (training corrector MLP_e): task loss (\mathcal{L}{\text{task}}) goes through corrected state → … → output; gradient flows through the correction term into mlp_e_neighbor. Since input (e{\text{neighbor}}) is based on detached state, gradient does not flow into (P) or the original state. MLP_e learns “what correction aids the downstream task.”
Path 3 (core BPTT): task gradient through state → NLM → synapses → … is unaffected by mutual prediction (state’s gradient path is not truncated).

Simultaneous training of all three paths is not recommended. Recommended Phase 1: only path 1 (pure auxiliary loss trains (P)) and path 3 (core BPTT) active; at inference, activate path 2 (correction term). Joint activation is left for ablation experiments.

14. Direction 3 (of Part 2): Hierarchical Neuron-Group Organization — Implement “Anchor Freezing” at CTM’s Hardware Level

Proposal core: bottom layer (universal primitives) frozen / middle layer (domain-specific) frozen after training / top layer (task-specific) always plastic. Gradient mask zeros frozen-layer gradients before optimizer.step().

Assessment

Highest CTM compatibility and lowest implementation cost of all directions. Gradient masking touches zero architecture — only behavior before optimizer.step().

MVP Implementation Checklist

Change	Location	Notes
Tier partition buffer	`__init__` around lines 131–148	`tier_mask`: ((D,)) int buffer (0=bottom/1=middle/2=top)
Feature importance scoring	new `compute_neuron_importance()`	Called after pre-training; see 3.1
Gradient mask hook	replaces `AnchorFreezer` style	After `backward()`, before `optimizer.step()`, zero frozen-layer gradients
Elastic freeze parameters	`__init__` lines 81–101	`elastic_freeze_lr_ratio` (default 0.001), `elastic_l2_lambda` (default 1e-4)

Key Supplement 3.1: Feature Importance Ranking Algorithm

Fixed-index partitioning (first 2048 = bottom layer) is invalid in CTM — initialize_left_right_neurons (ctm.py lines 450–486) uses random-pairing initialization, making neuron indices functionally meaningless.

Algorithm (executed once after pre-training):

def compute_neuron_importance(self, val_loader, n_steps=100):
    """Rank neurons by contribution to task loss"""
    importance = torch.zeros(self.d_model)

    for batch in val_loader:
        # Compute activation variance per neuron (metric: "dynamic range")
        _, _, _, _, post_acts, _ = self(batch, track=True)
        # post_acts: (B, D, T) → temporal-mean variance
        importance += post_acts.var(dim=(0, 2)).cpu()  # (D,)

    # Sort descending by variance → high variance (active) = bottom, low variance (inert) = top
    sorted_indices = importance.argsort(descending=True)
    self.tier_mask[sorted_indices[:n_primitive]] = 0        # bottom
    self.tier_mask[sorted_indices[n_primitive:n_primitive+n_domain]] = 1  # middle
    self.tier_mask[sorted_indices[n_primitive+n_domain:]] = 2  # top

Alternative metrics: Frobenius norm of NLM weights (SuperLinear.w1[:, :, i].norm()), gradient norm, or integrated gradient of the neuron’s contribution to loss. Variance is the lightest first attempt.

Key Supplement 3.2: Elastic Freezing Replaces Hard Zeroing

Hard gradient zeroing easily creates gradient vacuums — frozen layers learn nothing, and the representations trainable layers build on top of them may brittlely collapse. Three elastic strategies, light to heavy:

Extremely low learning rate (recommended for Phase 3 first launch): frozen layers use an independent tiny learning rate (e.g., elastic_freeze_lr_ratio = 0.001 × trainable-layer learning rate). Don’t zero gradients; just hugely reduce the update magnitude.
L2 regularization pulling toward frozen values: apply (\lambda_{\text{elastic}} \cdot | \theta - \theta_{\text{frozen}} |_2^2) loss to frozen layers. Compared to EWC’s Fisher-weighted approach, this uses uniform weights (simpler). Parameter elastic_l2_lambda = 1e-4.
EWC-style (future extension): compute the diagonal of the Fisher information matrix for each frozen parameter; use Fisher-weighted elastic penalty. Higher engineering complexity than 1/2; recommend introducing only at Phase 4 verification stage.

Verification Plan

After freezing, evaluate three metrics:

Forward stability: cosine similarity of activated_state on the same batch before vs. after freezing (should be > 0.99, proving representations haven’t drifted).
Task retention rate: old-task accuracy after freezing / old-task accuracy before freezing (should be > 0.95).
New-task plasticity: loss decline rate over 100 steps of new-task training (should be comparable to the unfrozen baseline).

15. Direction 4 (of Part 2): Topology-Aware Synchronization Coupling — Turn “Direction-Aware 2D Bias” into Neurons’ “Spatial Affinity”

Proposal core: assign each neuron a 2D topological coordinate; sync coupling strength (c_{ij}) is jointly determined by spatial distance (logarithmic decay) and direction (angular bucketing).

Assessment

Theoretically elegant; directly maps to v_dual’s RelativeDirectionalBias2D.

MVP Implementation Checklist

Change	Location	Notes
Neuron coordinates	`__init__` around lines 131–148	`neuron_coords`: ((D, 2)) learnable parameter, or initialized from external semantic/spatial embeddings
Direction bias table	`__init__` around lines 131–148	`direction_table`: ((8,)) parameter (8 direction buckets), Softplus-constrained positive
Temperature parameter	`__init__` around lines 131–148	`topology_temperature`: Softplus-ized scalar
Pairwise product modulation	`compute_synchronisation` line 252	Replace `left * right` with `left * right * c_ij`; see 4.1

Key Correction 4.1: (c_{ij}) Injects into Pairwise Product, Not Sync Matrix

Misalignment: CTM doesn’t maintain an explicit sync coupling matrix; synchronization is implemented via recurrent accumulation of pairwise products. Topological bias must be injected at the recurrence’s input end (p_t).

Full pseudocode (random-pairing mode):

def compute_topology_coupling(self, left_indices, right_indices):
    """Compute topological coupling coefficient c_ij for neuron pairs"""
    # left_indices, right_indices: (n_synch,) — already paired
    coords_left = self.neuron_coords[left_indices]    # (n_synch, 2)
    coords_right = self.neuron_coords[right_indices]  # (n_synch, 2)

    # Distance
    delta = coords_right - coords_left                # (n_synch, 2)
    dist = torch.norm(delta, dim=-1)                  # (n_synch,)
    log_dist = torch.log(1 + dist)                    # logarithmic compression

    # Distance decay
    tau = F.softplus(self.topology_temperature)       # positive
    distance_decay = torch.exp(-log_dist / tau)       # (n_synch,)

    # Direction bucketing
    angle = torch.atan2(delta[:, 1], delta[:, 0])     # [-pi, pi]
    angle_01 = (angle + math.pi) / (2 * math.pi)      # [0, 1]
    bucket = (angle_01 * self.n_angle_buckets).long().clamp(0, self.n_angle_buckets - 1)
    direction_bias = self.direction_table[bucket]     # (n_synch,)

    c_ij = distance_decay * direction_bias            # (n_synch,)
    return c_ij

Then in compute_synchronisation:

# Line 252 replacement
pairwise_product = left * right
if self.use_topology_coupling:
    c_ij = self.compute_topology_coupling(neuron_indices_left, neuron_indices_right)
    pairwise_product = pairwise_product * c_ij.unsqueeze(0)  # (B, n_synch)

first-last / random modes work analogously, but c_ij’s shape is the flattened 1D vector of the outer product.

Key Correction 4.2: Angular Bucket Boundary Handling

atan2 has a discontinuity at (\pm \pi); bucket 0 and bucket n-1 are adjacent in angle space (cyclic). The bias table should enable cyclic padding (bucket 0 and bucket n-1’s biases share a smooth transition). Simple implementation: make the bucket count even (e.g., 8), so directionally opposite pairs (differing by (\pi)) fall into different buckets, avoiding the forced same-bias for opposite directions.

Key Correction 4.3: Neuron Coordinate Initialization

Three strategies (increasing information):

Random initialization (zero prior): torch.randn(D, 2) * 0.1, learnable. The model discovers meaningful topology on its own. Use: first experiment, validating the value of topological bias itself.
Semantic embedding initialization (Direction 1 semantic group): use pretrained word/image embeddings (e.g., CLIP or SBERT) to map semantic-group neurons to a 2D UMAP projection of the embedding space. Spatial group still random.
Grid initialization (Direction 1 spatial group): arrange spatial-group neurons on a regular 2D grid (e.g., meshgrid), analogous to retinotopic mapping.

Recommend random initialization for the first experiment (verify mechanism), then upgrade to hierarchical initialization later.

16. Direction 5 (of Part 2): Sync Pattern Convergence Criterion — Replace Fixed Steps with Error Norms

Proposal core: compute the per-step change rate of the synchronization matrix, jointly with prediction error norm; terminate early when below threshold.

Assessment

High value, low risk, quantifiable benefit. Complements Part 1’s Direction 4 (certainty early stop).

MVP Implementation Checklist

Change	Location	Notes
Previous-step sync cache	`forward` loop lines 560–590	`prev_sync_out` variable, storing `synchronisation_out_{t-1}`
Change-rate computation	inside `forward` loop	`sync_change =
Early-stop logic	end of `forward` loop	Joint decision with Direction 2’s `pred_error`
New parameters	`__init__` lines 81–101	`sync_convergence_threshold` (default None=off), `pred_error_threshold` (default None=off)

Key Correction 5.1: Use Sync Vector Change Rate Instead of Full Matrix Norm

Misalignment: computing the norm of an (N \times N) full sync matrix is prohibitive ((N = d_{\text{model}}) can reach 2048–4096).

Correction: use the already-available synchronisation_out vector ((B, n_{\text{synch_out}}))‘s per-step L2 change. Zero extra computation in the loop — this vector is already computed at line 583 and drives the output projection (line 586).

Pseudocode (inside forward loop):

synchronisation_out, decay_alpha_out, decay_beta_out = self.compute_synchronisation(
    activated_state, decay_alpha_out, decay_beta_out, r_out, synch_type='out')

if self.sync_convergence_threshold is not None and stepi > 0:
    sync_change = torch.norm(synchronisation_out - prev_sync_out, dim=-1)  # (B,)
    # Joint criterion
    if self.pred_error_threshold is not None:
        pred_error_norm = torch.norm(pc_neighbor_loss_per_sample, dim=-1)  # from Direction 2
        converged = (sync_change < self.sync_convergence_threshold) & \
                    (pred_error_norm < self.pred_error_threshold)
    else:
        converged = sync_change < self.sync_convergence_threshold
    # Reference Direction 4's patience mechanism
    running_stable = torch.where(converged, running_stable + 1, 0)
    # ... early stop and fill ...

prev_sync_out = synchronisation_out.detach()

Key Correction 5.2: Joint Threshold Search

The dual thresholds ((\varepsilon_{\text{sync}}, \varepsilon_{\text{pred}})) must be searched jointly because they interact (stable state tends to stabilize sync too, but the correlation is not 1:1).

Parity search protocol:

On the validation set, record the full (T)-step trajectory per sample with track=True; post-hoc determine the “true convergence step” (definition: all subsequent steps’ accuracy = final accuracy).
Grid-search (\varepsilon_{\text{sync}} \in [10^{-4}, 10^{-1}]) (log scale, 10 points) and (\varepsilon_{\text{pred}} \in [10^{-3}, 10^0]) (log scale, 10 points) — a 10x10 grid.
Select the ((\varepsilon_{\text{sync}}, \varepsilon_{\text{pred}})) pair with the highest Pearson correlation between early-stop step and true convergence step.
Report the early-stop ratio and accuracy on the test set under that threshold.

17. Direction 6 (of Part 2): Frozen-Group Sampling Replay — Give CTM a “Memory Palace”

Proposal core: don’t store raw data; only store typical synchronization patterns of frozen neuron groups as “concept prototypes”; generate synthetic inputs through a reverse decoder.

Assessment

Theoretically correct, but engineering complexity is underestimated. Reconstructing high-dimensional pixels from a 128-dim sync pattern is an ill-posed problem.

MVP Implementation Checklist (Path A: kv Feature-Level Replay)

Change	Location	Notes
Feature cache	training callback	Per-class cache of kv feature mean vectors & covariance matrices
Sampling logic	`forward` entry (before line 527)	Sample kv tokens from a Gaussian mixture → skip backbone feed-forward and inject directly into the loop
Gaussian noise	after sampling	`kv_synthetic += torch.randn_like(kv_synthetic) * noise_std`

Path A Detailed Pseudocode

class KVReplayBuffer:
    def __init__(self, n_classes, d_input, n_tokens):
        self.means = torch.zeros(n_classes, n_tokens, d_input)   # per-class kv means
        self.covs = torch.zeros(n_classes, n_tokens, d_input)    # per-class kv variances

    def update(self, kv_features, labels):
        """Gradual EMA update during training"""
        for cls in range(n_classes):
            mask = (labels == cls)
            if mask.sum() == 0: continue
            cls_kv = kv_features[mask]  # (n_samples, S, d_input)
            self.means[cls] = 0.99 * self.means[cls] + 0.01 * cls_kv.mean(0)
            self.covs[cls] = 0.99 * self.covs[cls] + 0.01 * cls_kv.var(0)

    def sample(self, class_ids, noise_std=0.1):
        """Sample synthetic kv tokens from the cache"""
        # class_ids: (B,)
        mu = self.means[class_ids]           # (B, S, d_input)
        sigma = self.covs[class_ids].sqrt()  # (B, S, d_input)
        kv_synthetic = mu + sigma * torch.randn_like(mu) * noise_std
        return kv_synthetic

# In training loop
if use_replay and step > warmup_steps:
    # 50% batch from real data, 50% from replay
    replay_kv = replay_buffer.sample(old_class_ids)
    # Skip backbone; directly inject synthetic kv into the loop
    predictions, certainties, _ = ctm.forward_with_kv(replay_kv)

Key design:

forward_with_kv is a variant of forward that skips compute_features and directly accepts pre-computed kv. ~10 new lines (or reuse an internal path in forward).
Storage cost: n_classes * S * d_input * 2 (mean + variance). Compared to raw images n_samples * H * W * 3, for ImageNet (1000 classes, S=196, d_input=256) this is ~100M floats = 400MB — acceptable.
Privacy: after backbone information compression, kv features cannot be inverted back to raw pixels (downsampling + nonlinear projection in the backbone form an information bottleneck).
Synergy with Direction 3: bottom-layer neuron freezing guarantees that the mapping from kv to activated_state is stable (same kv produces the same initial activation), ensuring replay validity.

18. Corrected Roadmap

Phase	Content	Code-level corrections	Verification task	Core metric
Phase 1	Direction 2 (mutual-prediction NLM) + Direction 5 (convergence criterion)	Plan X pre-activation neighbors + `bias=False` corrector; sync vector change rate replaces matrix norm	parity (minimum testable)	convergence step reduction % + no accuracy loss
Phase 2	Direction 1 (dual-pathway) + Direction 4 (topology sync)	Neuron pool dual-split + coordinate channel separation (no VSS); pairwise product modulation on `random-pairing`	mazes (spatial reasoning)	routing success rate + spatial-group sync interpretability
Phase 3	Direction 3 (hierarchical freezing) + Direction 6 (replay)	Variance-sorted tier partitioning + elastic freeze (small lr); kv feature-level replay (no pixel generation)	sort (sequential continual learning)	old-task retention rate + new-task plasticity
Phase 4	Six-direction joint verification + ablation	Ablation matrix with six independent switches	parity + mazes + CIFAR	joint benefit vs. individual direction benefit vs. baseline

Phase 1 quick verification cycle: parity task, single GPU, ~30 minutes per seed. 3 seeds × 4 configurations (baseline / +mutual-prediction / +convergence-criterion / joint) = 12 runs ≈ 6 hours to complete Phase 1 ablation.

19. Core Divergence: The Dimensionality of the Error Signal

Your proposal and Part 1 differ in the dimensionality of predictive coding’s landing direction — not a matter of right or wrong, but complementary:

Dimension	Proposal scheme (spatial-dim mutual prediction)	Part 1 scheme (temporal-dim self-prediction)
Prediction target	self state (based on neighbor input)	self next-timestep state (based on previous state + attention)
Error definition	(e = \text{state_detach} - P_{\text{nbr}}(\text{neighbors}))	(e = z_{t+1} - P(z_t, o_{t+1}))
Theoretical analogy	cortical lateral prediction (between columns)	top-down temporal prediction (between hierarchy levels)
Input signal	neighbor pre-activation	self history + current data
Change magnitude	medium (new neighbor predictor + corrector)	smaller (new predictor (P))
Fixed point	neighbor-prediction self-consistency	temporal-stability self-consistency
Gradient path	only auxiliary loss trains (P) (stable)	auxiliary loss + optional inference correction
Best verification task	spatial reasoning (maze)	temporal reasoning (parity)

Complementarity analysis:

Temporal-dim error measures “is my next step predictable” — internal dynamics self-consistency.
Spatial-dim error measures “do my neighbors agree with my state” — cross-neuron consistency.
Both coexist in the same loop: each step computes both errors, jointly driving correction ((state += \alpha_{\text{time}} * mlp_e_{\text{time}}(e_{\text{time}}) + \alpha_{\text{space}} * mlp_e_{\text{space}}(e_{\text{space}}))).

Recommended experimental path:

First run temporal self-prediction on parity (smallest change), validating correlation between error signal and convergence trajectory.
Run spatial mutual prediction on mazes (richer semantics), validating spatial reasoning gains.
Superimpose both signals on dual-task parity + mazes; ablate single vs. superimposed benefit.

20. Summary

This six-direction proposal is a solid theory-engineering bridge document. Every direction has a clear neuroscience/engineering theoretical root and attempts a one-to-one mapping with CTM’s core mechanisms.

Part 2’s core contribution is a code-fact compatibility audit for each direction, flagging six misalignment points requiring correction:

Input routing cannot rely on VSS (Direction 1) — use coordinate channel separation instead (compute_features line 276 split, ~15 lines).
“Neighbor input” is not a native signal in CTM (Direction 2) — define it as the pre-activation slice from Synapses output (Plan X), store neighbor index buffer.
Tier partitioning strategy should be learned from training outcomes (Direction 3) — use activation-variance ranking instead of fixed-index partitioning, overlay elastic freeze (small lr).
Topological bias injection point is pairwise product, not sync matrix (Direction 4) — modify ctm.py line 252: pairwise_product = left * right * c_ij, ~3 lines.
Convergence criterion uses sync vector change rate, not full matrix norm (Direction 5) — reuse synchronisation_out’s per-step L2 change, zero extra computation.
Generative replay targets kv features, not raw pixels (Direction 6) — Path A (feature-level replay), synthesize kv tokens from mean + covariance, bypassing ill-posed pixel generation.

The corrected proposal and Part 1’s four-direction scheme are complementary — Part 1’s bottom-up analysis (finding theory interfaces from CTM’s internal deficiencies) and Part 2’s top-down mapping (finding CTM contact points from existing theoretical paradigms) converge on the same goal from opposite ends: to truly inject predictive coding’s “predict-error-correct” closed loop into CTM’s dynamical core.

Abstract

Part 1: Four Optimization Directions — Bottom-Up from CTM Code Facts

1. CTM Recap: A Machine That Unfolds Thought in Time

1.1 Three Core Ideas

1.2 Forward Pass in Detail

1.3 Key Submodule Source-Level Analysis

1.3.1 SynapseUNET: Multi-Level Down/Upsampling with Skip Connections (modules.py lines 45–143)

1.3.2 SuperLinear: Per-Neuron Independent Mapping via einsum (modules.py lines 146–236)

1.3.3 Synchronization Strategies: Three neuron_select_type Variants (ctm.py lines 450–486)

1.3.4 Certainty: Normalized Entropy Numerics and Hidden Pitfalls (models/utils.py lines 42–72)

1.4 Structural Deficiencies of Open-Loop Prediction

2. Theoretical Background: Predictive Coding and Dual Pathways

2.1 Predictive Coding

2.2 Dual Pathways

2.3 Training Stability

3. Direction 1: Prediction-Error-Driven Inner Loop (Recommended First)

3.1 Problem Formalization

3.2 Precise Temporal Analysis of Error Definition

3.3 Variant Designs

3.4 Training-Inference Decoupling Strategy

3.5 Detailed Gradient Flow

3.6 Fixed Points and Convergence Cascade: Theoretical Bridge Between Direction 1 and Direction 4

3.7 Source Locations and Implementation Checklist

3.8 Expected Benefits and Risks

4. Direction 2: Dual-Pathway Architecture (Ventral Semantics + Dorsal Spatial)

4.1 Problem Diagnosis

4.2 Target Architecture

4.3 Phased Landing

4.4 Expected Benefits and Risks

5. Direction 3: Training Stability Optimization

5.1 The BPTT Gradient Dilemma

5.2 MuPC Landing Design

5.3 BN Freezing: An Honest Conclusion

5.4 NLM LayerNorm Experiment

6. Direction 4: Certainty Early Stopping and Adaptive Inference

6.1 Existing Evidence

6.2 Early-Stop Algorithm Design

6.3 Taxonomy of Certainty Temporal Evolution

6.4 Adaptive Threshold Strategy

6.5 Computational Benefit Estimation

6.6 Joint Stopping with Direction 1

7. Theory Assumptions vs. Code Facts

8. Potential Pitfalls and Debugging Guide

8.1 Predictor Degeneration (Direction 1)

8.2 Error Term Dominating Task Gradient (Direction 1)

8.3 Correction Term Inference Misalignment (Direction 1)

8.4 Certainty Threshold Mis-Setting (Direction 4)

8.5 MuPC Suppressing Normal Learning (Direction 3)

9. Roadmap and Experimental Design

9.1 Priority Ordering (low-to-high risk)

9.2 Experimental Protocol

10. Part 1 Conclusion

Part 2: Six-Direction CTM Compatibility Audit — Top-Down from Theory to Code

11. Preface to Part 2

12. Direction 1 (of Part 2): Dual-Pathway CTM — From “Homogeneous Neuron Soup” to “Ventral-Dorsal Separation”

What I Agree With

MVP Implementation Checklist

Key Correction 1.1: Coordinate Channel Separation Replaces VSS Input Routing

Key Correction 1.2: Cross-Group Modulation Injects Pairwise Product, Not a Separate Matrix

Key Correction 1.3: NLM Structural Differences Are Experimental Hypotheses, Not Inevitable Design

13. Direction 2 (of Part 2): Predictive Coding NLM — Let Neurons “Predict Each Other’s Firing”

What I Agree With

MVP Implementation Checklist

Key Correction 2.1: Pre-Activation Neighbors as Observation Signal

Key Correction 2.2: Neighbor Selection Annealing

Key Correction 2.3: Structural Constraints on the Error Corrector

Key Correction 2.4: Gradient Flow of Spatial-Dimension Error

14. Direction 3 (of Part 2): Hierarchical Neuron-Group Organization — Implement “Anchor Freezing” at CTM’s Hardware Level

Assessment

MVP Implementation Checklist

Key Supplement 3.1: Feature Importance Ranking Algorithm

Key Supplement 3.2: Elastic Freezing Replaces Hard Zeroing

Verification Plan

15. Direction 4 (of Part 2): Topology-Aware Synchronization Coupling — Turn “Direction-Aware 2D Bias” into Neurons’ “Spatial Affinity”

Assessment

MVP Implementation Checklist

Key Correction 4.1: (c_{ij}) Injects into Pairwise Product, Not Sync Matrix

Key Correction 4.2: Angular Bucket Boundary Handling

Key Correction 4.3: Neuron Coordinate Initialization

16. Direction 5 (of Part 2): Sync Pattern Convergence Criterion — Replace Fixed Steps with Error Norms

1.3.1 SynapseUNET: Multi-Level Down/Upsampling with Skip Connections (`modules.py` lines 45–143)

1.3.2 SuperLinear: Per-Neuron Independent Mapping via einsum (`modules.py` lines 146–236)

1.3.3 Synchronization Strategies: Three `neuron_select_type` Variants (`ctm.py` lines 450–486)

1.3.4 Certainty: Normalized Entropy Numerics and Hidden Pitfalls (`models/utils.py` lines 42–72)