Hero image for Targets, Names, Drags: Which Mobile Accessibility Checks an AI Agent Can Actually Run

Targets, Names, Drags: Which Mobile Accessibility Checks an AI Agent Can Actually Run

One tree, two readers: what an operating agent can perceive

An operating agent perceives a mobile screen only through the accessibility tree, whose objects expose name, description, role and state (MDN accessibility tree).

A screen reader turns that same tree into speech, braille, or magnification for a person. An agent turns it into coordinates and actions. The channel is identical; the reader is not. Names are computed per the Accessible Name and Description Computation spec, which means what the agent “sees” is already an interpretation, not pixels.

That constraint defines the first question for this post: what an agent can perceive ends where the tree ends. Geometry may be attached by the platform, but intent, visual grouping, and affordance are absent unless encoded in those four properties. The mobile case is stricter than our earlier web accessibility agent baseline, because touch density and custom controls leave fewer DOM-like fallbacks.

Mobile accessibility guidance was written for humans, not agents

The W3C’s mobile accessibility page is guidance, not a normative specification (W3C mobile guidance), and it addresses people using assistive technology.

Human guidance can say “make targets comfortable” or “avoid aggregation.” An agent check must say which node, which property, and what pass value. Each rule must therefore be re-derived against name, description, role, state, plus bounding geometry. Where the source gives no threshold or node mapping, my reading is that the rule is not yet computable.

Target size is a geometry problem the agent can solve

WCAG 2.2 SC 2.5.8 Target Size (Minimum), Level AA requires pointer targets of at least 24 by 24 CSS pixels, with five exceptions named Spacing, Equivalent, Inline, User agent control, and Essential (W3C target-size-minimum).

The test is geometric: a solid, axis-aligned 24x24 CSS-px square must fit entirely inside the target. The Working Group explains the AA criterion is scoped to all pointer input, not only touch, while WCAG 2.2 SC 2.5.5 Target Size (Enhanced), Level AAA raises the same test to 44 by 44 CSS pixels. Unlike contrast, which did translate into an agent rule, size needs shape-aware geometry rather than tree properties alone.

Rounded corners make nominally large buttons undersized

A pill button can report a large bounding box and still fail, because rounded corners eat the area where the inscribed square would sit.

An agent that measures bounding-box width and height will pass such controls incorrectly; the computable version must test the rendered shape or its alpha mask. Without shape data, the agent should flag large-but-highly-rounded targets for review rather than assert a pass.

The Spacing exception is computable but neighbour-sensitive

Spacing passes an undersized target when a 24 CSS-px-diameter circle centred on each target’s bounding box does not intersect another target, or another undersized target’s circle.

That is computable from centres and radii, but only if the agent enumerates the correct neighbours, including overlapping layers, off-screen nodes, and occlusion. The other four exceptions need judgment rather than measurement: Equivalent depends on whether another control achieves the same function, Inline on sentence flow and line-height, User agent control on whether the author changed the default, and Essential on whether size and spacing are fundamental to the information conveyed, as with maps, dense data visualisations, or legally mandated paper-form replicas.

The platforms disagree about the floor, so the agent must choose one

Apple sets an iOS/iPadOS default of 44x44 pt and minimum of 28x28 pt, while Android guidance and WCAG AA set different floors, so an agent must explicitly select its threshold (Apple accessibility).

Apple’s full control-size table is iOS/iPadOS 44x44 pt default, 28x28 pt minimum; macOS 28x28 default, 20x20 minimum; tvOS 66x66 default, 56x56 minimum; visionOS 60x60 default, 28x28 minimum; watchOS 44x44 default, 28x28 minimum. Apple also says about 12 points of padding around elements with a bezel and about 24 points around elements without a bezel. Android developer guidance says each interactive UI element should have a touch target of at least 48dp x 48dp, adding “Larger is even better” (Android accessibility guide).

Comparison, not harmonisation, is the point. WCAG AA is the interoperability floor, AAA is stricter, and iOS and Android publish their own platform minima. My reading is that an agent should enforce the stricter of WCAG AA and the platform minimum after explicit pt/dp/CSS-px conversion, and log which standard it applied.

Dragging: the rule is computable, “essential” is not

WCAG 2.2 SC 2.5.7 Dragging Movements, Level AA requires functionality that uses a dragging movement be achievable with a single pointer without dragging, unless dragging is essential or the functionality is determined by the user agent and not modified by the author (W3C dragging-movements).

The first half translates: does a second node, button, or menu item offer the same outcome without a drag path? Roles and names answer that. The second half does not translate. Essential has no tree marker. A slider with arrow-key equivalents, a list with move-up buttons, and a freeform drawing canvas can read identically to the check, yet only human judgment decides whether drag is essential to the information or interaction.

Names: where label-in-name strains the translation

Accessible naming strains translation because aria-label supplies an accessible name when no visible label text exists in the DOM, and MDN recommends aria-labelledby instead wherever visible label text can be referenced (MDN aria-label). The match test itself comes from elsewhere: WCAG 2.2 SC 2.5.3 Label in Name requires that the words visually labelling a component match or be contained within its accessible name.

Android states the parallel rule: use the semantics modifier / contentDescription to describe a control’s purpose (Android accessibility guide). An agent can test presence of a non-empty computed name and, where visible text exists, test that the name contains that text. What it cannot test is whether the description is correct, concise, or localised. Form errors as a computable contract showed a similar limit: presence and association are checkable, correctness is not.

“Decorative” is an intent claim, not a tree fact

Android guidance says for decorative Image or Icon composables set contentDescription = null or hide them from accessibility, so they expose no name (Android accessibility principles).

From the tree, a decorative icon and a missing-label functional button look the same: an image role with empty name. The difference is author intent, which the tree does not preserve. An agent rule that fails every unnamed image will punish correct decorative exclusion; a rule that passes them will miss broken controls. My reading is that decorative status needs an explicit author flag outside the four properties to be safely automated.

Group descriptions go brittle

Android guidance warns to avoid building a group description by aggregating the text of its children, because such descriptions go brittle when a child’s text changes.

That brittleness is invisible in a single snapshot. The tree shows a plausible container description; only change over time reveals concatenation. An agent can flag the pattern — container description equals joined child strings — but cannot know without history whether the screen is static and harmless or dynamic and fragile. This parallels consistent help as a computable check, where order and repetition across screens matter more than one screen alone.

Four places the translation breaks

Translation breaks in four places: Spacing neighbour enumeration, decorative intent, label-in-name agreement, and drag-only essentiality.

Spacing breaks on scope: same-container siblings are easy, overlapping layers are not. Decorative breaks on intent: an empty name is both correct and incorrect. Label-in-name breaks on text reality: ellipsised or localised labels fail string-prefix tests despite being correct. Drag-only breaks on exemption: alternatives can be found, essential drags cannot be certified.

The evidence: agents are not reliable operators yet

AndroidWorld evidence shows agents are not yet reliable mobile operators, with the best baseline agent completing 30.6% of tasks across 116 programmatic tasks in 20 real Android apps, and a desktop web agent adapted to Android faring worse on mobile (arXiv paper).

The gap fits perception limits: if operation depends on the same semantics that make a screen reader work, missing names, undersized targets, and drag-only paths compound into task failure. This is a judgment call, but low task success suggests pre-flight accessibility checks would help agents more than extra action heuristics.

Size thresholds, side by side

WCAG and platform minima use different units and exception models, so side-by-side comparison is the only honest way to pick a floor.

Standard or platform Default or recommended size Minimum size and level Exceptions stated in the source Computable from tree plus geometry? Unit to convert
WCAG 2.2 SC 2.5.8 (AA) 24 by 24 CSS pixels 24 by 24 CSS pixels, Level AA Spacing, Equivalent, Inline, User agent control, Essential Yes, with shape and neighbour geometry CSS px
WCAG 2.2 SC 2.5.5 (AAA) 44 by 44 CSS pixels 44 by 44 CSS pixels, Level AAA Not detailed in the cited source Yes, same geometry test CSS px
WCAG 2.2 SC 2.5.7 (dragging — not a size criterion) Single-pointer alternative Level AA Essential dragging excepted Partial, essential is not marked None
iOS/iPadOS 44x44 pt default 28x28 pt minimum Not stated as WCAG exceptions Yes, after pt conversion pt
macOS 28x28 pt default 20x20 pt minimum Not stated as WCAG exceptions Yes, after pt conversion pt
tvOS 66x66 pt default 56x56 pt minimum Not stated as WCAG exceptions Yes, after pt conversion pt
visionOS 60x60 pt default 28x28 pt minimum Not stated as WCAG exceptions Yes, after pt conversion pt
watchOS 44x44 pt default 28x28 pt minimum Not stated as WCAG exceptions Yes, after pt conversion pt
Android guidance At least 48dp x 48dp, Larger is even better 48dp x 48dp guidance floor Not stated as WCAG exceptions Yes, after dp conversion dp

FAQ

Five questions cover the checks an implementer reaches for first, each self-contained.

What can a GUI agent actually perceive on a mobile screen? An agent perceives name, description, role and state from the accessibility tree (MDN tree), plus bounding geometry where exposed. It does not see intent, visual grouping, or whether an image is decorative. My reading: if a fact is not in those four properties or geometry, the agent cannot test it without guessing.

What is the minimum target size an agent should enforce? WCAG 2.2 SC 2.5.8 Target Size (Minimum), Level AA requires at least 24 by 24 CSS pixels, with five exceptions: Spacing, Equivalent, Inline, User agent control, Essential (W3C explanation). AAA under SC 2.5.5 raises the floor to 44 by 44 CSS pixels. This is a judgment call for agents to enforce AA by default.

How do Apple and Android size floors differ? On iOS/iPadOS Apple specifies 44x44 pt default and 28x28 pt minimum, while Android guidance asks for at least 48dp x 48dp (Apple) (Android accessibility guide). My reading: an agent should apply the stricter of WCAG 24 CSS px and the platform minimum after unit conversion.

When does dragging fail an agent check? SC 2.5.7 requires dragging functionality be achievable with a single pointer without dragging unless dragging is essential or the behaviour is the user agent’s own (W3C dragging). An agent can verify a named alternative path exists. This is a judgment call on essential: maps and drawing gestures resist a tree-only test and need human review.

How reliable are current screen-operating agents? AndroidWorld provides 116 programmatic tasks across 20 real Android apps and the best baseline agent completes 30.6% of tasks (paper). A desktop web agent adapted to Android performed worse on mobile, which suggests tree-based perception is not yet sufficient for reliable operation.

The Bottom Line

Geometry and name-presence checks survive translation into agent rules, while spacing scope, decorative intent, label agreement, and essential dragging do not. Treat the former as pre-flight gates and the latter as human review queues.

How This Guide Was Built

This guide is a document analysis built from the official sources cited inline; we did not run a device or audit an app. Sources were WCAG 2.2 understanding pages, Apple Human Interface Guidelines, Android accessibility guidance, MDN accessibility-tree documentation, and the AndroidWorld benchmark paper. Where inference was required, it is marked as “my reading” or “a judgment call,” and the computable rules below are proposals derived from cited guidance, not tested implementations.

Computable rules: what an agent can run today

The following eight rules are untested proposals derived from the guidance above, not validated implementations, and each needs explicit unit handling before use.

1. Target-size floor (SC 2.5.8 AA). Pass if an axis-aligned 24x24 CSS-px square fits inside each target’s actual shape. False-positive mode: measuring the bounding box instead of the inner shape passes rounded-corner buttons that are truly undersized; hidden or occluded nodes get measured.

2. Spacing exception. Pass if the 24px circle on an undersized target intersects neither another target nor another undersized target’s circle. False-positive mode: enumerating only same-container siblings misses overlapping layers.

3. Name presence. Pass if every interactive-role node exposes a non-empty computed name. False-positive mode: names arriving from async content, or attached at a different node depth.

4. Label-in-name agreement. Pass where a visible label exists and the computed name begins with it. False-positive mode: ellipsised or localised labels, case/punctuation differences, text split across styled spans.

5. Decorative exclusion. Pass if non-informational images and icons expose no name. False-positive mode: an unlabeled functional control looks identical to a decorative one.

6. Non-drag alternative (SC 2.5.7). Pass if every drag-reachable function also has a named node offering a single-pointer path. False-positive mode: alternatives exposed only via context menus, long-press or custom actions are missed; genuinely essential drags are flagged though exempt.

7. Platform threshold selection. Pass if every interactive target meets the stricter of WCAG 2.5.8 AA and the platform minimum. False-positive mode: pt/dp/CSS-px unit confusion across densities.

8. Group-description brittleness. Fail if a container’s description is a concatenation of its children’s text. False-positive mode: static screens whose child text never changes are flagged without harm.