Transformers or CNNs for MRI segmentation?

In a resource-constrained comparison of 3D medical image segmentation, that is what happened: a conventional 3D U-Net slightly outperformed Swin UNETR while using fewer computational resources.

That result is not an indictment of transformer models for medical image segmentation. It is a correction to a common assumption. Global attention does not automatically yield better anatomical boundaries. A larger receptive field does not compensate for weak data efficiency. In MRI, where acquisition protocols, field strength, coil geometry, motion, contrast weighting, and pathology distribution all constrain the signal, architecture is only one part of the error budget.

The practical question in CNN vs vision transformer MRI segmentation is therefore not which model is newer. It is which inductive bias, training regime, memory footprint, and failure mode fit the imaging task.

Inductive bias and the data-efficiency gap

CNNs begin with a structural advantage. Convolution assumes locality. The same kernel is applied across spatial positions. Translation invariance is built into the operation. The model does not need to learn from scratch that a lesion edge, gland boundary, or ventricular contour can appear at different coordinates while retaining similar local morphology.

That assumption is not glamorous. It is useful.

MRI segmentation is dominated by local intensity transitions, texture gradients, anatomical adjacency, and repeated spatial motifs. A convolutional encoder extracts these features with bounded receptive fields, then expands contextual coverage through depth, pooling, dilation, and skip connections. The original U-Net architecture, introduced in 2015, formalized the arrangement that remains difficult to displace: a contracting path for semantic abstraction, an expanding path for spatial recovery, and skip connections that preserve fine structure.

The model does not need to infer every relationship globally. It needs to retain the boundary evidence that determines whether a voxel belongs to the target structure.

Vision Transformers take a different route. An image or volume is divided into patches or tokens. The network estimates relationships among them through attention. A token can interact with distant tokens without passing through a long chain of convolutional operations. This is valuable when the segmentation decision depends on broad anatomical context rather than local appearance alone.

It is also expensive in data and memory.

A pure ViT trained from scratch on a modest medical dataset lacks the built-in spatial assumptions that make CNNs efficient. It must learn locality, translation tolerance, texture sensitivity, and anatomical regularity from examples. Clinical datasets rarely provide enough independent variation to make that a safe bet. A dataset may contain many volumes but far fewer genuinely independent subjects. Scanner vendor, protocol, site, and pathology composition can create correlations that inflate apparent training performance while weakening external generalization.

This is the central CNN vs ViT MRI segmentation accuracy issue. A transformer may possess a broader representational ceiling, but the ceiling is irrelevant if the training distribution cannot support the route to it.

In smaller or domain-specific medical datasets, CNNs commonly perform better than Vision Transformers when both are trained from scratch. Their locality and translation invariance reduce the hypothesis space. They constrain the model before optimization begins. That constraint is not a limitation in the pejorative sense. It is a prior aligned with the geometry of the image.

The comparison changes when pre-training enters the pipeline. Self-supervised learning can expose the transformer encoder to anatomical structure without requiring dense manual labels. Contrastive objectives, masked reconstruction, and related pre-training schemes allow the model to establish a representation before supervised segmentation begins. The benefit is measurable. In the cited comparative work, self-supervised pre-training improved Swin UNETR’s Dice Similarity Coefficient by 1.6%. Adding residual functionality to the pre-trained model increased the DSC gain to 3.6%.

Those improvements are not free. They move cost upstream. The model may need more unlabeled volumes, longer pre-training, more careful augmentation, and a validation design that distinguishes genuine anatomical transfer from scanner-specific memorization.

A transformer does not remove the data requirement. It relocates it into pre-training, architecture design, and validation.

Dice remains the most visible metric in these comparisons, but it compresses several clinically distinct errors into one overlap score. A model can achieve a strong DSC while producing a poor boundary on a small structure. It can miss a thin extension while preserving the dominant lesion volume. It can over-segment a region in a way that barely changes aggregate overlap but materially affects downstream radiomics or surgical planning.

For that reason, DSC should be read with HD95 or another boundary-sensitive measure. Hausdorff Distance at the 95th percentile exposes contour deviations that average overlap can conceal. In MRI, this distinction is not cosmetic. Partial-volume effects, low SNR, intensity nonuniformity, and susceptibility artifacts often concentrate error at boundaries rather than in the interior of a large homogeneous structure.

A model that yields a marginally higher Dice score but degrades boundary stability is not automatically the better segmentation system.

Global context versus local precision

The architectural difference becomes clearer in three dimensions.

A 3D CNN processes local neighborhoods through volumetric kernels. The receptive field grows across layers, but information remains filtered through successive transformations. This creates a strong hierarchy. Fine texture and edge information enter early. Broader shape and context emerge later. Skip connections return selected high-resolution features to the decoder.

This design is computationally predictable. Memory use is driven primarily by feature-map dimensions, channel count, kernel configuration, and the number of active resolution levels. Inference can be tiled. Operations are highly regular. GPU acceleration is mature.

The cost is that long-range relationships are indirect. If two distant regions of the brain influence the segmentation decision, the signal must travel through the network hierarchy. Deep CNNs can represent those relationships, but the path is not direct.

A Vision Transformer handles long-range dependencies more explicitly. Self-attention allows tokens in separate anatomical regions to exchange information. A distant structure can alter the interpretation of a local feature without requiring the network to construct the relationship through repeated convolutional stages.

That is the theoretical argument for transformer models in MRI segmentation. It is valid. It is also incomplete.

In volumetric MRI, attention operates on a large token space. The computational and memory burden rises with token count and attention strategy. Windowed attention reduces the immediate cost by restricting interactions to local windows. Shifted windows then allow information to cross window boundaries at subsequent stages. Swin UNETR uses this hierarchical strategy in a 3D setting, pairing a Swin Transformer encoder with a CNN-based decoder. The architecture reformulates 3D brain tumor semantic segmentation as a sequence-to-sequence prediction problem.

The result is not pure global attention. It is staged context aggregation under a computational constraint.

That distinction matters. The performance of Swin UNETR should not be used as evidence that an unrestricted pure ViT is the default solution for clinical MRI. Swin’s hierarchy, windowing, decoder design, skip connections, and training regime all contribute to the result. “Transformer” is not a single operating point.

The local-global trade-off can be summarized as follows:

The practical distinction is not local versus global in absolute terms. Every competent segmentation model needs both. The distinction is how much of each is encoded by the architecture and how much must be learned from the dataset.

Why local evidence remains decisive in MRI

MRI is not a uniform image-generation process. T1-weighted, T2-weighted, FLAIR, diffusion, susceptibility-weighted, and quantitative sequences expose different tissue contrasts. A boundary that is distinct in one sequence can be weak or absent in another. Bias fields alter intensity distributions. Motion introduces spatial inconsistency. Parallel imaging and accelerated acquisition modify noise behavior. Reconstruction can produce ringing, residual aliasing, or texture changes that are unrelated to anatomy.

These effects create a hostile environment for a model that depends on broad statistical regularities without sufficient domain coverage.

CNNs tolerate some of this variability because their early filters respond to local structures. A contrast shift can still preserve an edge. A small geometric displacement does not invalidate the kernel. This does not make CNNs scanner-invariant. It makes their starting assumptions less demanding.

Transformers can also learn robust features, particularly after self-supervised pre-training across heterogeneous data. But the burden is higher. The model must distinguish anatomy from acquisition-specific appearance while maintaining the global relationships that make attention valuable. If pre-training data are narrow, attention may encode the wrong regularities. The resulting model can appear context-aware while actually being protocol-aware.

That is a serious distinction for clinical deployment. A segmentation tool trained on one institution’s T2-weighted prostate MRI may perform well on that institution’s reader distribution and degrade when gland size, coil setup, sequence timing, or scanner vendor changes. The architecture does not eliminate this domain shift. It may amplify it if the model has enough capacity to memorize broad but non-causal correlations.

For prostate gland segmentation, the reported numbers illustrate why architecture comparisons must be read within their training setup. SwinUNETR achieved average Dice scores of 0.816 and 0.860 across two reader datasets in single-cohort training. Baseline U-Net reached 0.825 and 0.851 on the same respective datasets. Neither model dominates consistently. In a large gland-size cross-validated training setting, SwinUNETR reached up to 0.902.

The signal is not that the transformer wins. The signal is that performance depends on cohort composition, gland-size distribution, reader variability, and validation design. The same architecture can move from slightly behind to materially stronger when the training regime changes.

A segmentation benchmark that reports one DSC value without describing the acquisition distribution is incomplete.

The boundary problem

Local precision is especially important for structures whose clinical meaning is concentrated at the contour. Prostate capsule delineation, cortical thickness estimation, hippocampal segmentation, white-matter lesion boundaries, and tumor infiltration all punish coarse overlap metrics.

A transformer may correctly classify the broad region while softening or displacing the edge. A CNN may preserve a sharper boundary but miss a disconnected or atypical component. The preferred failure mode depends on the application.

The evaluation should therefore separate at least four questions:

• Does the model identify the correct anatomical region?
• Does it preserve small or thin structures?
• Does it maintain boundary accuracy under contrast variation?
• Does it fail systematically on particular scanners, protocols, or lesion sizes?

DSC addresses the first question imperfectly. HD95 addresses part of the third. Per-structure sensitivity, false-positive volume, calibration, and subgroup analysis are needed for the rest.

This is where software engineering meets clinical interpretation. The model output is not merely a mask. It is an input to registration, radiomics, volumetric measurement, treatment planning, or computer-aided diagnosis. A small contour error can propagate. Registration can align the wrong boundary. Radiomics features can shift when the mask changes by a narrow peripheral band. Volume trends can become protocol trends.

The segmentation architecture must be judged by the downstream quantity it is expected to preserve.

Self-supervised pre-training changes the transformer calculation

Without pre-training, the CNN advantage is often straightforward. With pre-training, the comparison becomes less stable.

Swin UNETR’s hierarchical encoder is designed to capture multiscale context. Shifted windows provide cross-window communication without imposing the full cost of global attention at every layer. The decoder then reconstructs voxel-level predictions. This is a rational compromise for 3D MRI. It does not pretend that every voxel must attend to every other voxel. It distributes context across stages.

Self-supervised pre-training strengthens the encoder before the segmentation labels are introduced. The improvement of 1.6% DSC from pre-training, and 3.6% after adding residual functionality to the pre-trained model, demonstrates the sensitivity of transformer performance to initialization and feature reuse.

The interpretation should remain disciplined. These are gains within a particular experimental framework. They do not establish a universal transformer premium. They establish that the architecture was under-supported without pre-training and became more competitive when representation learning was added.

That has operational consequences.

A CNN pipeline can often begin with supervised training on a smaller labeled cohort. A transformer pipeline may require an additional data layer:

1. Assemble unlabeled volumes that cover the intended scanner and protocol distribution.

2. Define a pre-training task that does not reward trivial acquisition shortcuts.

3. Monitor whether learned representations transfer across sequences and sites.

4. Fine-tune with strict separation between subject-level training and validation data.

5. Compare against a strong CNN baseline under matched preprocessing, augmentation, patch sampling, and compute.

6. Evaluate external data, not only held-out scans from the same acquisition process.

The fifth step is routinely mishandled. A transformer can receive more augmentation, a larger input context, longer training, or a better tuned optimizer and then be declared architecturally superior. That is not a clean comparison. If the question is CNN versus transformer, the experimental budget must be equalized. If the question is which system is best for deployment, then equalization is less important than total cost and reliability. These are different questions.

Self-supervised learning also introduces its own failure modes. Reconstruction-based pre-training may prioritize scanner texture or intensity statistics that are easy to predict but clinically irrelevant. Contrastive learning can create invariance to features that are actually needed for pathology discrimination. Masking strategies can favor large-scale structure while neglecting small lesions. The pre-training objective constrains what the encoder considers informative.

The encoder does not understand anatomy. It minimizes an objective.

That distinction should remain visible in every model card and validation report.

Pre-training can raise transformer Dice. It cannot certify that the learned representation is clinically causal.

Hybrid architectures: where the comparison becomes less clean

The most credible direction is not a forced choice between a classic CNN and a pure transformer. It is the hybrid architecture that assigns different work to different mechanisms.

Swin UNETR already follows this pattern. The encoder uses hierarchical shifted-window attention. The decoder is CNN-based. The system captures broader dependencies while retaining convolutional machinery for spatial reconstruction. TransUNet and related designs combine transformer representations with U-Net-style decoding and skip connections. These models acknowledge a simple engineering fact: global context and voxel-accurate recovery are not the same operation.

A convolutional front end can extract local texture and stabilize early features. A transformer block can integrate nonlocal relationships at reduced resolution. A convolutional decoder can restore spatial detail without asking attention to perform every reconstruction step. This division reduces the amount of high-resolution global interaction and therefore constrains memory use.

It also complicates attribution. If a hybrid model outperforms a U-Net, the gain may come from attention, a deeper encoder, additional pre-training, residual connections, improved normalization, a changed patch schedule, or a more favorable optimization path. Calling the result a transformer victory is technically weak.

For developers building MRI analysis software, the comparison should be decomposed by function:

• Feature extraction: Does the encoder preserve local contrast transitions under the target sequence and noise profile?
• Context integration: Does the model resolve ambiguous regions using anatomical relationships rather than acquisition artifacts?
• Decoder recovery: Are thin structures and boundaries reconstructed at the native or clinically relevant resolution?
• Calibration: Does confidence degrade in proportion to domain shift?
• Throughput: Does inference meet the workstation, PACS, cloud, or edge deployment constraint?
• Reproducibility: Can preprocessing, patch sampling, and post-processing be fixed and audited?

The hybrid model can yield the best aggregate score while creating the most complex deployment surface. More components mean more configuration points. Window size, depth, embedding dimension, patch size, normalization, decoder width, and resampling strategy all affect the result. A model that performs well in a research notebook may become difficult to validate when integrated with DICOM routing, scanner-specific preprocessing, and longitudinal storage.

This is not an argument for simplicity at any cost. It is an argument for measuring the cost.

The reported resource-constrained comparison is important precisely because it removes some of the rhetorical advantage of larger architectures. When 3D U-Net achieved a DSC approximately 1% higher than Swin UNETR, with lower GPU consumption and faster inference, the result exposed a deployment-relevant trade-off. The transformer may still be preferred for a task where long-range context materially reduces clinically significant errors. But that preference must be demonstrated at the task level.

A one-percent difference in DSC is not inherently meaningful. A one-percent difference accompanied by lower latency, lower memory demand, and simpler failure analysis can be decisive.

Clinical resource constraints and inference speed

The scanner is not the only constrained system. The software environment is constrained as well.

A research pipeline may have access to high-memory GPUs, batch scheduling, and unlimited preprocessing time. A clinical deployment may run on a workstation shared across applications, a hospital server with competing workloads, or a cloud service where latency and cost are explicit. The model may need to process a complete 3D study, multiple sequences, and repeated follow-up examinations without manual patch intervention.

Inference speed is not a secondary metric. It affects workflow integration, queue behavior, and whether the tool is used at all.

CNNs generally offer a more predictable latency profile. Their operations map well to optimized kernels. Tiled inference can handle volumes that exceed device memory. The trade-off is potential seam artifacts, redundant computation at tile overlaps, and loss of broad context when the tile is too small. These issues are manageable, but they must be tested on the target anatomy.

Transformer latency depends on tokenization, attention windowing, volume dimensions, and implementation. Shifted windows constrain the cost compared with unrestricted attention, but they do not make 3D processing free. Larger context can improve disambiguation while increasing memory pressure. More aggressive downsampling lowers the token count while degrading fine detail. The architecture therefore constrains the operating range before any clinical metric is calculated.

A deployment comparison should report more than peak GPU memory. It should include:

• Volume dimensions and voxel spacing used during inference.
• Resampling and cropping policy.
• Patch size and overlap, if tiled inference is used.
• Batch size and precision mode.
• End-to-end latency, including preprocessing and post-processing.
• Peak memory allocation.
• Failure behavior when the scan does not match the expected field of view.
• Variability across scanner vendors and sequence parameters.

These details determine whether a benchmark result survives contact with a hospital workflow.

There is also a question of determinism. Segmentation software used in clinical research must be reproducible. Randomized test-time augmentation, nondeterministic GPU operations, and version-sensitive preprocessing can produce small mask differences that become significant in longitudinal analyses. CNNs are not automatically deterministic, and transformers are not automatically unstable. But the larger and more configurable the system, the more aggressively its execution path must be pinned down.

The same principle applies to post-processing. Connected-component filtering, hole filling, morphological cleanup, and threshold selection can improve DSC while obscuring model failure. If a transformer requires extensive post-processing to produce an anatomically acceptable mask, that processing belongs in the evaluated system. The bare network score is not the clinical product.

Equal resources, unequal assumptions

The phrase “equal computational resources” should be treated carefully. Equal GPU memory is not equal engineering effort. Equal training time is not equal optimization maturity. Equal parameter counts do not imply equal representational capacity. A CNN and a transformer can consume the same nominal resources while receiving different benefits from those resources.

Still, controlled comparisons remain necessary. They prevent a model from winning through an unreported advantage in augmentation, resolution, pre-training, or hyperparameter search. The resource-constrained finding favoring 3D U-Net is valuable because it aligns performance with practical cost rather than theoretical novelty.

The correct selection process is blunt:

• If labels are limited, begin with a strong 3D CNN baseline.
• If the anatomy is locally defined and the deployment budget is narrow, do not add attention without evidence.
• If long-range structure is central to the task and sufficient data exist, test a hierarchical transformer or hybrid model.
• If self-supervised pre-training is used, validate its transfer across sites and protocols.
• If the output feeds radiomics or quantitative measurement, emphasize boundary metrics and downstream stability.
• If the system must run near real time, benchmark end-to-end latency on the actual hardware.

A transformer should earn its memory footprint.

What the evidence supports

The current evidence supports a conditional hierarchy, not a winner.

For small, domain-specific datasets trained from scratch, CNNs retain a clear practical advantage. Their inductive bias matches the local structure of MRI. They require less data to establish useful filters. They consume fewer resources in many 3D settings. A 3D U-Net can match or slightly exceed a transformer while yielding faster inference.

For larger datasets, broader acquisition diversity, and well-designed self-supervised pre-training, transformer and hybrid models become more competitive. Swin UNETR demonstrates why: hierarchical attention can model contextual relationships, while a CNN decoder preserves a path to voxel-level reconstruction. The gains are strongest when the training regime supports the architecture rather than merely attaching transformer blocks to a small labeled cohort.

For clinical software, the relevant comparison is not a leaderboard average. It is the complete operating profile:

• DSC for volumetric overlap.
• HD95 for boundary deviation.
• Sensitivity to lesion size and anatomical variation.
• Robustness across scanners and protocols.
• Calibration under distribution shift.
• Inference latency and memory demand.
• Reproducibility of the full pipeline.
• Stability of downstream measurements.

Pure Vision Transformers have not replaced CNNs in clinical MRI segmentation. That claim would exceed the evidence. Standard U-Net and related CNN systems remain competitive, often more efficient, and easier to deploy under constrained conditions. Hybrid models are promising because they make a narrower claim: attention can supply context where convolution is weakest, while convolution can retain precision where attention is expensive or diffuse.

The decision is therefore physical and statistical. MRI provides finite signal. Hardware provides finite memory. Datasets provide finite independent variation. The model must operate inside those constraints.

A transformer is not the default because it can connect distant tokens. A CNN is not the default because it is familiar. The correct architecture is the one that yields stable anatomy, tolerates protocol variation, constrains latency, and fails in a measurable way.

Anything less is architecture worship disguised as validation.