Annotated autism-specific facial dataset narrows a long-standing machine-learning gap
Facial emotion recognition in children with autism spectrum disorder has struggled for a simple reason: most available datasets were built for neurotypical faces, not for the facial movement patterns and expression variability that can accompany autism. This preprint introduces ASD-FER, an annotated benchmark built from 2,779 facial images of children with ASD and labeled across six emotion categories by three independent annotators using majority voting. The study is less about a single model winner than about building a more clinically relevant testing ground for a task that has been difficult to evaluate with any rigor.
The premise matters. If an algorithm is trained to read emotion from faces, its performance depends heavily on the quality and specificity of the data it sees during development. Generic facial expression datasets often assume clean, stereotyped emotion displays, yet the bedside reality in autism is more complex. Subtle eye movement, reduced facial expressivity, atypical mouth activation, and highly variable social signaling can all change what a model perceives. A benchmark built from ASD faces therefore does more than add another dataset; it changes the input distribution the field has been working with.
Transformer models outperformed classic CNNs, but the metric story was not uniform
Nine models were compared under stratified 10-fold cross-validation and a held-out 90/10 evaluation: CNN, SVM, AlexNet, ResNet34, ResNet50, MobileNetV2, and three backbone configurations of the proposed task-specific architecture EQFormer — ViT, Swin-T, and a compact Hybrid CNN-Transformer, or HybridT. Under cross-validation, ResNet50 led the conventional models with 84.89% mean accuracy. Among the transformer-based systems, ViT performed best at 88.27%, followed closely by Swin-T at 88.02% and HybridT at 86.58%.
The held-out split sharpened the comparison. ViT reached the highest overall accuracy at 91.01%, but Swin-T posted the strongest balanced accuracy at 74.47% and the best macro-F1 at 0.7078. HybridT stayed competitive at 89.57% accuracy while using far fewer parameters. In a field where raw accuracy can hide poor class-level performance, the balanced metrics carry real weight. A model that looks strong on paper can still fail on underrepresented emotions, and that failure would be especially consequential in a dataset where class imbalance already complicates interpretation.
The contrast between overall accuracy and balanced accuracy is the most clinically relevant fault line in the report. High aggregate performance can coexist with uneven behavior across emotion classes, a pattern that matters if the eventual goal is any kind of assistive or screening technology rather than a benchmark leaderboard.
Why the eye, eyebrow, and mouth regions kept rising to the top
Explainability analysis with Grad-CAM added an important layer to the comparison. The visual maps concentrated on emotion-relevant facial regions, especially the eyes, eyebrows, and mouth. Swin-T appeared the most spatially focused, while HybridT emerged as the most parameter-efficient model at 2.66 million parameters. The interpretability signal is useful because facial emotion recognition systems frequently fail in ways that are hard to audit; a heatmap that attends to the right facial zones does not prove clinical validity, but it does offer a plausible mechanistic link between model output and human emotion coding.
This biological logic is easy to follow. Human observers infer emotion from coordinated movement across the upper and lower face, and any architecture that localizes on those regions is closer to the underlying perceptual task than one that latches onto background noise or irrelevant facial texture. In autism, where expression can be subtle or atypical, that focus becomes even more important. The system has to learn not just which pixels change, but which facial configurations are informative when expressions do not follow the standard script.
Class imbalance and cross-dataset transfer remain the central limitations
The study does not claim the problem is solved. Class imbalance remains an unresolved obstacle, and cross-dataset generalization is still open. Those two issues are tightly linked. A model trained on a single curated benchmark can look strong inside that data environment and then lose performance when confronted with another site, another imaging protocol, another age mix, or another behavioral context. For a pediatric autism application, those shifts are not theoretical edge cases; they are the conditions under which any deployment would have to work.
A second limitation sits in the annotation process itself. Majority voting by three annotators strengthens label reliability, yet facial emotion is still a subjective construct, especially in children with ASD. Expressions may not map neatly onto canonical six-emotion categories, and the boundary between ambiguous affect, neutral presentation, and partial emotional display can be unstable. A benchmark can standardize comparison, but it cannot remove the underlying ambiguity of the task.
The most important near-term value of ASD-FER may therefore be methodological rather than clinical. It gives researchers a more realistic substrate on which to test recognition systems, compare architectures, and examine where models attend when decoding pediatric ASD faces. Without that substrate, performance claims drift easily toward inflated confidence. With it, the field can at least measure whether a model is reading the face or merely memorizing the dataset.
What the benchmark enables next
EQFormer’s three backbones point in different directions for future development. ViT delivered the best overall accuracy, Swin-T improved class-sensitive performance, and HybridT offered a compact design with strong efficiency. Each trade-off matters for eventual deployment in resource-limited environments, where memory footprint and compute demands may shape whether the tool is usable at all. A smaller model that preserves much of the performance of a larger one can be more practical than an accuracy leader that is too costly to run.
For now, the benchmark’s chief contribution is structural: it gives autism-specific FER research a clearer reference point. The next test will be whether models trained on ASD-FER can travel beyond this dataset without collapsing in performance. Cross-site validation, broader age diversity, and richer emotion definitions will decide whether these systems can move from careful benchmark comparisons to something closer to clinical relevance.