On-chip communication on the ET-SoC-1: 20 ns per mesh hop, and a bandwidth cliff at the shire boundary

18 September 2026 · first measured on one card (aifoundry2, in the AI Foundry lab) in one session; the latencies, trees and barriers re-measured on three cards (aifoundry2, aifoundry3 and aifoundry1's card 1, three passes each) on 26 September, and the energies per byte on the same three cards, six passes each, also on 26 September; minion clock 600 MHz and mesh clock 400 MHz throughout · GPU values from published microbenchmarks (sources at the end) · code: workloads/nocbench, raw data: docs/reports/data/2026-09-18-nocbench-aifoundry2 · part of the ET-SoC-1 measurement reports

The ET-SoC-1's cores can pass data to each other directly, without going through memory, and quickly. Inside a shire a message takes 113–190 ns round trip, and each mesh hop adds 20 ns. A tree allreduce over all 1,024 minions takes 2.3 µs. Messaging costs 0.7–2.2 pJ per byte inside a shire with 1 KB messages, busy cores included (the energy manual's re-run on three cards at 600 MHz, 26 September). Crossing the mesh has a cost, though: there is 7–34× less message bandwidth between shires than inside them.

An A100's SMs can reach each other only through L2; Hopper adds direct shared-memory access, but only within a cluster of up to 16 SMs. The ET-SoC-1 has several direct paths. Hart 0 of any minion can send up to 127 × 32 B (about 4 KB, cycling through its 32 vector registers) straight into another minion's registers with TensorSend/TensorRecv, and the receiver can add, max or min the data into what it already holds. There are hardware reduction and broadcast trees, credit counters that let a core sleep until another core signals it, and barrier counters in every shire. This page measures each of these against its closest GPU equivalent and maps where the 32 shires sit on the chip's 2D mesh.

Where the shires are: a 6×6 mesh

Each square is one position on the mesh in marty1885's logical map (which appears to be the die turned a quarter), labelled with the logical shire ID the firmware uses. Colour is the measured TensorSend round-trip time from the selected shire (outlined), between minion 0 of each shire; stronger blue is slower. Card picks whose measurements: each of the three cards of the 26 September check (the median of its three passes) or the first session (aifoundry2, 18 September). Hover or focus a shire for its numbers; tap it, or press Enter on it, to measure from there. The four grey squares hold no compute shire: they are the master shire, the spare shire, and the PCIe and I/O shires, and they route traffic too. The datasheet's full 8 × 6 mesh adds eight memory shires, not shown: in this map's orientation the latency fit on aifoundry2 in Anatomy of a memory access (§4) places memory shires 0–3 above the top row and 4–7 below the bottom row, over the middle four columns (memory shire 2's own position is not pinned down by that fit).

Latency grows with distance, except through memory

Round-trip time between minion 0 of every pair of shires, against the number of mesh hops between them, for three ways to signal another core. TensorSend round trips rise by 20 ns per hop. Credit round trips rise by about 24 ns per hop on all three cards (234 + 24 ns per hop fitted, with a worst residual of 15 cycles; the first session's 246 + 20.4 ns fitted within 5), yet across shires they almost coincide with TensorSend's: the median pair differs by under a cycle, and 445 of the 496 by at most 15 cycles. A flag handoff through global atomics in memory, which is how a GPU passes a signal between SMs, lands on whichever shire's L3 slice holds the flag's line. It costs 610–1,150 ns (median 873 across shires on each card) whatever the distance: its slope against mesh hops is −2 cycles per hop, and at most +9 cycles (15 ns) per hop at 99% on each of the three cards. Points at 0 hops are pairs inside one shire.

Inside a shire, only the tree edges are fast

Every primitive, against a GPU

Round trips at 600 MHz on ET-SoC-1; the three cards agree within a few cycles, the chip-wide barriers within 15 (26 September). GPU numbers are published measurements, at 1.41 GHz for the A100 and 1.755 GHz for the H800 (Hopper). The GPU column lists the closest equivalent where one exists.

Drawn on one time axis, the ET-SoC-1's messages between cores land where an A100's handoff through L2 does, while each of its barriers is slower than the GPU's nearest equivalent:

Message size and bandwidth

Time per message for a one-way stream of messages from one minion to another, against message size. The receiver posts a new receive as soon as the last one finishes. A message is up to 127 registers of 32 B, reusing registers once past f31. Each path has a fixed cost per message (40, 86, 135 and 224 cycles) plus a cost per register (2.3, 4.0, 3.0 and 4.6 cycles), on all three cards. The chart is aifoundry2's (26 September); the other two cards agree within 0.3%. 1 KB round trips between all 496 shire pairs add 12 cycles per hop up to 5 hops and 27–36 per hop beyond (the 1 KB view of the distance chart), so the sender seems to have a limited number of packets in flight.

Reduction trees

Latency of one allreduce (TensorReduce up the tree, then TensorBroadcast back down) against the number of minions in the tree, for three sizes. Up to 32 minions the tree stays in one shire. Beyond that, each doubling adds a level between shires. Each level adds about one round trip between the minions it pairs: 68–71 cycles on the fast network (levels 0–2), 117 through the crossbar (levels 3–4), and 164–234 over the mesh (levels 5–9), where the slowest branch sets the pace (the first session's were 68, 114 and 156–235). The chart is aifoundry2's (26 September); the other two cards agree within 0.3%. The dashed line is the A100's grid.sync(), a barrier that moves no data.

Energy per byte moved

Why does a byte cost 6–26× more once it crosses the shire boundary? Hart 0 of all 1,024 minions sends 1 KB messages (128 B where marked) around rings inside a shire, or to shires a fixed number of IDs away; the rows are sorted by mean mesh distance. The three panels share the rows: bytes moved per second over the whole chip, board power above the idle just before and after each burst, and energy per byte. The last two are the energy manual's re-run of these rings (§5): each dot is the mean of six passes on each of three cards at 600 MHz (26 September), with the range the passes spanned. First measured on 18 September on aifoundry2 alone (two runs, sampled without the die temperature and so with no leakage correction, on a card cooling after another user's matmul): 0.8–2.3 pJ per byte inside a shire and 13–20 across the mesh, 2–13% above these means (4–16% above aifoundry2's own passes). Inside a shire messaging costs 0.7–2.2 pJ per byte with 1 KB messages. With 128 B messages a shire ring costs more: 3.7 against 2.2 pJ/B (3.6 against 2.1 on aifoundry2, 3.4 against 2.1 on aifoundry3, 3.9 against 2.5 on aifoundry1's card 1), a difference of +1.3 to +1.5 pJ/B, above zero at 99% on every card. The readout under the chart gives the cost's fit against mean hops across the mesh, per card and together. Grey rules are references: ET's own L2, remote-scratchpad (the mean over the zeros and random data its passes held) and DRAM reads at a steady 600 MHz (the energy manual, §4), and the A100's L2 and HBM accesses (SC'25). The last row is bulk data for comparison: every minion streaming TensorLoads from the scratchpad 16 shire IDs away (5.1 pJ/B when that scratchpad holds zeros, 11.8 when it holds random data).

What the numbers say

1. All three cards have marty1885's shire layout. He built the map from shire-to-shire bandwidth on another card. On aifoundry2, aifoundry3 and aifoundry1's card 1, TensorSend round trips fit a+b×(Manhattan distance on his map) within 1.5 cycles and scratchpad loads at a steady clock within 0.2, and credit round trips rise with it too, within 15 cycles (5 in the first session). Pure Manhattan distance means the routes are shortest paths through all 36 cells, the four non-compute ones included. Minion 31 of each shire gives the same numbers as minion 0, so each shire meets the mesh at a single point.
2. The mesh costs 20 ns per hop, in both directions together. That is 12 minion cycles at 600 MHz, or 4 mesh cycles each way at 400 MHz. In the one memory-hierarchy session, where aifoundry2's governor moved the clock mid-row, a hop still took 20 ns: remote scratchpad loads cost 66 minion cycles + (56 ns + 20 ns × hops), and a model fitted to two of the four rows fits the other two within a cycle at 600, 700 or 800 MHz, except one chase whose clock changed partway (5.4 cycles off). Only aifoundry2 can test this (a boot service holds aifoundry3 at 600 MHz by setting its TDP to 0 W at every boot).
3. Three speeds, not four. A message takes 68 cycles on a reduction-tree edge, 114 anywhere else in the shire, and 150 + 12 per hop between shires. Chains and trees should put partners on those edges.
4. Combining in the channel is free. A receive that adds or maxes into its registers takes the same time as a plain copy. That is the ⊕ of a (min,+) or (max,+) semiring, applied on arrival without a separate instruction.
5. The hardware trees are the fastest way to synchronise the whole chip. A 1,024-minion allreduce of 32 B takes 2.3 µs, against 8.3 µs for a barrier made from global atomics and credits, on each of the three cards. Inside a shire it takes 0.74 µs, and all 32 shires can reduce at once without slowing down. Reductions and barriers belong on the trees.
6. The shire boundary is a bandwidth cliff. Messages move 1.1–3.0 TB/s in aggregate inside shires and 0.09–0.16 TB/s between them, at 6–26× the energy per byte, mostly because the same ~2 W of busy cores moves 7–34× fewer bytes. Across shires, bulk data moves better as TensorLoads from the other shire's scratchpad (0.96 TB/s, at 5.1 pJ/B when that scratchpad holds zeros and 11.8 when it holds random data) than as messages; per byte it cost 3–5 pJ less than the s → s+8 ring on average on all three cards, but not in every pass (two of six on aifoundry3 and on aifoundry1's card 1): 5.8–8.9 pJ less in the passes that read zeros, −2.2 to +3.4 in those that read random data.
7. Latency in nanoseconds is GPU-like; the mechanisms and the energy are not. A message crossing 1–10 hops takes 135–225 ns one way. An A100 takes 166 ns to hand a flag from one SM to another through L2. The differences lie elsewhere. The data lands in the receiver's registers, with an optional combine. There are 1,024 addressable endpoints instead of 108 SMs, plus hardware trees and sleeping credit waits. Messaging costs 0.7–18 pJ per byte (1 KB messages), busy cores included, against about 38 pJ per byte for an A100 L2 access.

A trap: one ready flag per minion

Which TensorSend layouts hang? One shire's 32 minions, one neighbourhood per row, and the messages of one step of four layouts. Solid edges are the neighbourhood's tree edges (68-cycle round trips), dashed ones go through the shire crossbar (114 cycles); a minion that would take readies from two partners in the same phase is outlined and marked. Costs are this page's measured round trips.

What it means for systolic and wavefront designs

• 1D chains at minion grain inside a shire, 2D arrays at shire grain over the mesh. A one-way message costs about 34 cycles on a tree edge, 57 cycles elsewhere in the shire, and 75 + 6 cycles per hop between shires. To keep communication under a tenth of the time, a cell needs about 10× that much work per step: roughly 350–600 cycles inside a shire and 800–1,350 between shires one to ten hops apart. That sets the grain k ≈ 10·t_msg/t_cell cells per message for wavefront DP or lattice updates.
• In a 2D array, give each neighbour link its own minion. Over TensorSend a minion may have only one partner's ready outstanding (see the trap above). At shire grain that is easy to respect: a shire has 32 minions, so no minion needs to send to or receive from two partners. Bulk tiles can go through the scratchpads instead (Hand it to the next shire). Inside a shire the layouts are, as drawn above:
  The four layouts, safe or not

  • Safe: 1D rings (partners never change).
  • Safe by the RTL rule, not run here: 2D in alternating phases (west pass, barrier, north pass, barrier; a 32-minion FLB plus credit barrier is 233 cycles against 68–114 per hop).
  • Safe: the hardware trees (one ready bit per level; 444 cycles for a 32-minion reduce and broadcast).
  • Hangs by the RTL rule, not run here (the one hang seen had two pairs through one minion): a naive 2D array whose cells take readies from two partners, until the chip is reset; sys_emu tracks partners separately and will not show it.
• Order chains by the map, not by shire ID. Stepping through shires 0, 1, 2, … 31 averages 3.3 hops per boundary and up to 8 (3.5 and up to 10 if the ring closes from 31 back to 0). This ring through all 32 compute shires moves one hop at every step: 0 24 9 25 2 11 19 27 18 10 17 14 22 26 15 23 31 7 6 30 29 5 28 20 12 21 13 1 16 4 3 8, and back to 0.
• Send boundaries, load bulk. Register-to-register messages suit small, latency-critical exchanges and in-network combines. Tiles that move between shires go faster through the scratchpads and L3: handing each stage's output to the next shire's scratchpad, instead of a round trip through DRAM, runs about 12× faster at about a thirteenth of the energy per byte (Hand it to the next shire).

The first rule as a calculator: pick a link, a message size and the work a cell does per step. A message's time is half the measured round trip at that size.

How it was measured

Caveats

Reproduce

# build on the lab machine (sources only, nice -j4)
scripts/deploy-lab.sh aifoundry2 workloads/nocbench

# latency, size, trees and barriers: one timeout-10 process per probe, waits for an idle card
ssh aifoundry2 'cd ~/nekko && bash workloads/nocbench/run_lab.sh build/nocbench/host/nocbench_host build/nocbench-data'

# energy (quit et-powertop first; every configuration runs under timeout 10)
ssh aifoundry2 'cd ~/nekko && python3 workloads/nocbench/run_energy.py --host-bin build/nocbench/host/nocbench_host --out build/nocbench-data/energy-a'
# the same in the reverse order
ssh aifoundry2 'cd ~/nekko && python3 workloads/nocbench/run_energy.py --host-bin build/nocbench/host/nocbench_host --out build/nocbench-data/energy-b --only xshire1-c4,shire-c4,xshire6,xshire4,xshire2,xshire8,xshire16,xshire1,shire,neigh,pair,spin'

# the 26 September check on three cards: tools/claims-v3/lat/block.sh (its nocbench unit runs the same commands)

# numbers, layout check (12 search restarts, about 30 s) and chart data for this page; the energy
# re-runs come from docs/reports/data/2026-09-23-energy-manual/reruns.json (the rings' watts over idle from the passes it
# names), the three cards' latency passes from --v3
python3 workloads/nocbench/analyze.py docs/reports/data/2026-09-18-nocbench-aifoundry2 --memhier docs/reports/data/2026-09-18-memhier-aifoundry2 --search --v3 docs/reports/data/2026-09-25-claims-v3 --embed docs/reports/2026-09-18-et-soc1-on-chip-communication.html

Related reports

• The energy manual, §5 — these rings re-measured with bars on three cards; and §6, the energy of atomics and barriers.
• Heat per millimetre — a hop's wire energy with the bits on the links chosen, and 3.72 mm per hop measured from the die plot.
• Memory hierarchy — the scratchpad and L3 rates, and the pointer-chase rows used in item 2 of What the numbers say.
• Anatomy of a memory access — the same shire map; an L3 hit costs 110 cycles plus 12 per hop, and where the memory shires sit.
• Hand it to the next shire — bulk data handed between shires through the scratchpads, measured against a round trip through DRAM.
• One hot line stops a shire — what happens when many cores hammer one global-atomic line, such as a chip-wide flag: the shire that holds the line loses its own memory traffic.
• Ridge points — how much reuse each level of memory, and each kind of message on this page, demands before the tensor unit is the limit.

Sources

← All ET-SoC-1 measurement reports