This article does not have a corresponding language version

DETAILED DATA FLOW SCHEMATIC - STEP-BY-STEP DESCRIPTION

(Figure 2: TorCons-MoE System Architecture)

Stage 1: Input Reception and Preprocessing

Step 1.1: Token Embedding Input

  • The system receives input token embeddings X ∈ ℝ^(T×d), where T is the sequence length and d is the hidden dimension
  • Position indices m ∈ [0, T-1] are generated or retrieved for each token position in the sequence
  • Input data is loaded into GPU/TPU memory buffers for parallel processing

Step 1.2: Input Buffer Allocation

  • Double-buffering mechanism is initialized to support asynchronous stream processing
  • Memory regions are allocated for:
    • Attention stream output buffer
    • Expert stream output buffer
    • Routing decision metadata buffer

Stage 2: Dynamic Router Processing with Double Log Z-Loss Regularization

Step 2.1: Gating Logit Computation

  • Input tokens X are processed through the router network
  • Gating logits are computed: l = W_g · X + b_g, where l ∈ ℝ^(T×E) and E is the number of experts
  • Each logit value represents the raw relevance score for routing a token to a specific expert

Step 2.2: Temperature-Scaled Softmax Application

  • Temperature-scaled softmax function is applied to generate routing probabilities: p_{t,i} = exp(l_{t,i}/τ) / Σ_{j=1}^{E} exp(l_{t,j}/τ)
  • Temperature parameter τ controls the sharpness of the probability distribution
  • Output: Routing probability matrix P ∈ ℝ^(T×E)

Step 2.3: Composite Loss Function Calculation (Training Phase) The router is optimized using a composite loss function:

a) Task Loss (ℒ_task):

  • Standard cross-entropy loss for the primary task (e.g., next-token prediction)
  • Ensures routing decisions contribute positively to model performance

b) Entropy Regularization (ℒ_entropy):

  • Calculated as: ℒ_entropy = -1/T Σ_{t=1}^{T} Σ_{i=1}^{E} p_{t,i} log(p_{t,i})
  • Encourages diversity in expert selection
  • Prevents premature expert collapse

c) Double Log Z-Loss Regularization (ℒ_z) - Core Innovation:

  • Log-sum-exp computation: Z_{lse} = log(Σ_{i=1}^{E} exp(l_i))
  • Double-logarithmic penalty: ℒ_z = ||log(Z_{lse} + ε)||²
  • Where ε is a small constant (e.g., 10⁻⁸) for numerical stability
  • Stabilizes gating logit magnitudes to prevent extreme values
  • Gradient signal proportional to log(Z_{lse})/Z_{lse} provides adaptive stabilization

Step 2.4: Top-K Expert Selection

  • For each token t, select the K experts with highest probabilities
  • Generate routing decisions: (K_t, g_i) where K_t is the set of selected expert indices and g_i are the corresponding weights
  • Create micro-batch instructions: B_e = {x_t | t ∈ tokens routed to expert e}

Step 2.5: Telemetry Signal Generation

  • Logit variance metrics are computed and sent to MHEP Controller
  • Expert load distribution statistics are calculated
  • These signals enable adaptive scheduling in the execution engine

Stage 3: Asynchronous Stream Fork (MHEP Controller)

Step 3.1: Stream Initialization The MHEP Execution Controller forks processing into two independent parallel streams:

Stream S_A (Attention Pathway):

  • Assigned to first subset of hardware resources (e.g., GPUs 0-3)
  • Configured for self-attention matrix computations

Stream S_E (Expert Pathway):

  • Assigned to second subset of hardware resources (e.g., GPUs 4-7)
  • Configured for token dispatch and expert feed-forward computations

Step 3.2: Event Registration

  • Initialize synchronization primitives:
    • Event_A: To be triggered upon Attention stream completion
    • Event_E: To be triggered upon Expert stream completion
  • Configure dependency tracker for asynchronous coordination

Step 3.3: Input Data Branching

  • Input tensor X is duplicated or aliased for parallel consumption
  • Routing decisions (K_t, g_i) are transmitted to Expert Stream
  • Position indices m are made available to both streams

Stage 4: Parallel Stream Execution

4A. Stream S_A: Self-Attention Pathway

Step 4A.1: Query/Key/Value Projection

  • Compute linear projections:
    • Q = W_q · X (Query matrix)
    • K = W_k · X (Key matrix)
    • V = W_v · X (Value matrix)
  • Operations executed on local GPU memory without inter-device communication

Step 4A.2: Standard RoPE Application

  • Apply Rotary Position Embeddings to Q and K vectors:
    • Q’ = RoPE(Q, m)
    • K’ = RoPE(K, m)
  • RoPE encodes pairwise relative positional relationships between tokens
  • Rotation angles based on position indices m and pre-defined frequencies θ_i

Step 4A.3: Attention Matrix Computation

  • Compute attention scores: Scores = Q’K’^T / √d
  • Apply softmax: Attention Weights = Softmax(Scores)
  • Compute attention output: A(l) = Attention Weights · V
  • Apply output projection: A_out = W_o · A(l)

Step 4A.4: Attention Stream Completion

  • Store attention output A(l) in dedicated buffer
  • Trigger Event_A = COMPLETE signal
  • Release Attention Workers for next micro-batch

4B. Stream S_E: Expert Dispatch & RoPE-Integrated Computation

Step 4B.1: Token Dispatch (Top-K Selection)

  • Group tokens by target expert based on routing decisions
  • Construct micro-batches: B_e for each expert e
  • Each micro-batch contains all tokens assigned to a specific expert

Step 4B.2: All-to-All Communication (Overlap Zone)

  • CRITICAL OVERLAP MECHANISM:
    • Initiate asynchronous token dispatch via interconnect (NVLink/InfiniBand)
    • Transmit micro-batches B_e to expert-hosting devices
    • This communication occurs concurrently with Stream S_A attention matrix computation
  • Communication latency is hidden behind attention computation time
  • By the time Attention completes, tokens have typically arrived at expert devices

Step 4B.3: RoPE-Integrated Expert Module Processing For each expert E_e receiving micro-batch B_e:

a) Linear Projection:

  • Compute: h = W₁^(e) · x + b₁^(e)
  • Projects input to higher-dimensional hidden space (typically d_ff ≈ 4d)

b) RoPE Injector - Core Innovation:

  • Apply rotary transformation to intermediate hidden state: h’ = RoPE(h, m)
  • For each dimension pair (h_{2i}, h_{2i+1}):
    • h’{2i} = h{2i} · cos(mθ_i) - h_{2i+1} · sin(mθ_i)
    • h’{2i+1} = h{2i} · sin(mθ_i) + h_{2i+1} · cos(mθ_i)
  • Key Distinction: This encodes unary absolute position directly into the expert’s computational pathway
  • Unlike attention RoPE (pairwise relative), this ensures each token “knows” its absolute position regardless of which expert processes it

c) Non-Linear Activation:

  • Apply activation function: a = σ(h’) (typically GeLU or ReLU)
  • Activation operates on position-aware hidden states

d) Output Projection:

  • Compute expert output: o_e = W₂^(e) · a + b₂^(e)
  • Projects back to model dimension d

Step 4B.4: Expert Output Generation

  • Each expert produces position-aware output o_e
  • Outputs retain rotational transformation R(θ_m) encoding absolute position
  • Store expert outputs in dedicated buffer

Step 4B.5: Expert Stream Completion

  • Trigger Event_E = COMPLETE signal
  • Release Expert Workers for next micro-batch

Stage 5: Event-Driven Synchronization

Step 5.1: Dependency Wait

  • Aggregation Unit enters waiting state
  • Monitor synchronization condition: IF Event_A == COMPLETE AND Event_E == COMPLETE
  • No global barrier - lightweight event-based coordination
  • Faster stream waits for slower stream without releasing allocated memory

Step 5.2: Synchronization Trigger

  • Once both events are received, trigger LayerNorm execution
  • Reset event statuses for next micro-batch
  • Ensure data integrity without unnecessary serialization

Stage 6: Aggregation and Layer Normalization

Step 6.1: Weighted Expert Reduction

  • Retrieve routing weights g_i for each token
  • Combine expert outputs: E_agg(l) = Σ_{i ∈ K_t} g_i · o_i
  • Critical: Because all expert outputs o_i have undergone RoPE transformation with the same rotational rule R(θ_m), they exist in a unified positional coordinate system
  • This geometric compatibility ensures smooth transitions even when consecutive tokens route to different experts

Step 6.2: Residual Connection

  • Combine attention output, expert output, and original input: R = X + α · A(l) + β · E_agg(l)
  • Where α and β are learnable scaling parameters or fixed constants (typically 1.0)
  • Residual connection preserves information flow across layers

Step 6.3: Layer Normalization

  • Apply LayerNorm to stabilize activations: Y = LayerNorm(R)
  • Normalize across feature dimension
  • Final output Y ∈ ℝ^(T×d) contains:
    • Content information from attention and experts
    • Positional information from both attention RoPE and expert-integrated RoPE
    • Balanced expert utilization from Double Log Z-Loss stabilization

Stage 7: Output Propagation and System Telemetry

Step 7.1: Output Buffer Transfer

  • Transfer Y to next Transformer layer input buffer
  • Or route to final prediction head if last layer
  • Maintain double-buffering for pipeline safety

Step 7.2: Telemetry Update

  • Record performance metrics:
    • FLOPs utilization
    • Energy consumption
    • Expert Utilization Efficiency (EUE)
    • Load balance scores
  • Feed metrics back to MHEP Controller for adaptive optimization

Step 7.3: Next Layer Preparation

  • Update position indices if sequence length changes
  • Prepare input buffers for next layer or next training step
  • Release completed buffers for memory reuse

Key Integration Points Summary

Point Location Innovation System Benefit
① Double Log Z-Loss Application Router loss computation Double-logarithmic penalty ℒ_z = ||log(Z_{lse})||² Stabilizes logits → Predictable micro-batch sizes → Efficient MHEP scheduling
② Async Stream Fork Post-router, pre-computation Decoupled S_A and S_E streams Enables parallel execution → Masks communication latency
③ Communication-Compute Overlap All-to-All dispatch vs Attention matrix Temporal overlap of NVLink/InfiniBand transfer with QK^T computation Reduces communication overhead from 22.7% to 8.3%
④ RoPE in Expert FFN Between W₁ projection and activation σ(·) Unary absolute position encoding: h’ = RoPE(h, m) Positional coherence across sparse routing → Robust long-context modeling
⑤ Event-Based Sync Pre-LayerNorm aggregation Wait for Event_A ∧ Event_E Data integrity without global barriers → Minimal synchronization overhead

Technical Advantages of This Flow

✅ 31.4% Reduction in Training FLOPs - Achieved through stable routing (Double Log Z-Loss) + efficient scheduling (MHEP)

✅ 33.9% Reduction in Inference Energy - Enabled by overlapping communication + position-aware experts preventing re-computation

✅ Improved Long-Context Performance - Perplexity at 32K tokens: 13.6 vs 16.5 (baseline) due to dual-pathway positional encoding

✅ Expert Utilization Efficiency: 86.7% - vs 65.2% baseline through Double Log Z-Loss stabilization

✅ Near-Linear Scaling - Maintains efficiency to 32+ GPUs via asynchronous event-driven coordination

This detailed flow schematic demonstrates how the TorCons-MoE system achieves synergistic integration of routing stability, parallel execution efficiency, and positional coherence preservation.