The question “can an agent use software?” gets interesting after the polished demo disappears, when there is a trace of the work left on the table.
In CAD, a task is not just a chain of clicks. The system has to recognize a geometric intention, choose operations, handle constraints, produce an executable solid, and leave behind a model somebody can edit later. CADBench attacks that problem from the evaluation side: 18,000 samples, six benchmark families, five input modalities and six metrics for measuring editable CAD program generation.12 A separate paper on Task Model Induction starts from computer-use traces — screenshots, mouse events, keyboard events — and tries to recover what tasks were being pursued, in a form that can be audited.4 Microsoft Skill Recorder records a work session, reconstructs an intent and ordered steps, then turns the result into a reusable Skill or Automation for an agent.5
These are not the same announcement. Read together, though, they point to the same constraint: before agents can be trusted inside professional software, observed work has to become something people can test.
CAD exposes the shortcut
CADBench focuses on reconstructing editable CAD programs from images or 3D observations, a distinction that matters because producing a plausible mesh is not the same as producing CAD code that executes, creates a valid solid and remains compact enough to be useful.2
The benchmark combines families derived from DeepCAD, Fusion 360, ABC, MCB and Objaverse, with clean meshes, noisy meshes, single-view renders, multi-view renders and photorealistic renders. Its metrics cover volumetric IoU, Surface IoU, Chamfer distance, valid shape rate, token count and operation count.12 The public Hugging Face dataset exposes the same 18,000-row structure across six splits, with CAD files and image modalities attached to the samples.3
A visual score would hide too much: an agent can draw something that resembles the part while producing unusable CAD code, or get a good global overlap while missing surface details and wasting operations. CADBench makes those failures separable.
The paper’s results explain why this matters. Under idealized inputs, specialized mesh-to-CAD methods substantially outperform general vision-language models, but some of those specialized systems become fragile when the input modality changes. The authors identify three recurring failure modes: reconstruction degrades with geometric complexity, CAD-specialized models can be brittle under modality shift, and model rankings change depending on the metric used.2
So the useful question is not only “which model wins?” but also “which kind of mistake would make this result unusable in a workshop, an engineering office or a manufacturing pipeline?”
Traces replace the hand-written recipe
The other half of the signal comes from work traces. Inducing Task Models from Computer-Use Traces starts from a very ordinary situation: someone uses a computer, while screenshots and mouse or keyboard actions are passively recorded. The proposed TMI system tries to discover latent tasks, disentangle interleaved goals, and reconstruct a task model made of hierarchical objectives plus a procedure model for the observed execution.4
The abstract’s numbers are careful but worth noting. On controlled human and agent trajectories, TMI reaches 0.974 agreement with ground-truth task groupings, reconstructs 74.9% of the observed execution steps, and skills derived from its task models improve held-out task accuracy by 30.0% over the strongest baseline cited by the authors.4
Raw traces are not clean, and that is part of their value. A real session contains open windows, interruptions, dead ends, repeated checks and useless gestures; it shows work as it happens, not as a manual imagines it.
Skill Recorder takes a more product-shaped route: Microsoft’s repository describes a desktop app that records a screen session — clicks, app and window switches, visited pages and optional narration — then uses GitHub Copilot CLI to reconstruct an intent and ordered steps, from which the user can generate a Skill or an Automation.5 The README also makes the safety boundary explicit: recording and frame extraction happen locally, but analysis sends selected data to GitHub’s cloud, and users are warned not to capture secrets.5
What changes
AILANTA groups these publications under Computer-Use Workflow Learning: turning work sessions, traces and benchmarks into learning infrastructure for agents.6 The market framing is still early, but the grouping is useful.
For IRZ, the important shift is material. A lot of software automation has lived in two poor formats: the fragile macro that replays gestures, or the abstract documentation that explains what should happen. The work cited here moves toward a third format: a trace rich enough to audit, transform, test and reuse.
In CAD, that shift is concrete: an agent should not merely “see” a part, because it has to reconstruct editable intent, choose geometry that executes, and accept judgment from several metrics at once. In office or business software, the agent should not merely imitate a video; it has to extract the task, separate useful steps from noise, and produce a procedure a human can read.
The limits are just as clear. CADBench measures CAD program reconstruction from standardized inputs; it is not yet an autonomous agent spending a full day inside Fusion 360. TMI reconstructs task models in a research setting; that does not guarantee robustness in every professional application. Skill Recorder can turn one example into a reusable procedure, but the recording still has to be reviewed, especially when sensitive data might appear on screen.245
Still, the direction is visible. Software agents improve when work stops being a spectacular thirty-second demo and becomes something recorded, measured and criticized; for design tools, the stronger threshold may be an agent whose work can be inspected afterwards.