JoonHo Byun, Ph.D. student (Advisor: Prof. Unsang Park)

The paper “Training Refines Onset Placement but Erodes Event Correspondence: A Full-Trajectory Audit of Video-to-Audio Onset Alignment,” written by JoonHo Byun (Ph.D. student, first author), Tae Gon Kim (M.S. student), Juyeon Park (M.S. student), and Prof. Unsang Park (corresponding author) of the Computer Vision and Image Processing Lab (CVIP), has been accepted for publication at the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2027, a leading international conference in computer vision.

Recent video-to-audio (V2A) synthesis models can generate audio from silent video with fidelity and semantic relevance approaching that of recorded soundtracks. A persistent failure mode remains, however: temporal alignment. The generated sound often arrives tens to hundreds of milliseconds away from the moment the visible event occurs. Because human perception registers asynchrony as small as 25 ms as unnatural, onset precision largely governs the perceived quality of event-driven sounds such as impacts.

Prior work has approached this problem by adding temporal signals—a synchronization module, an onset condition, or an alignment loss—and has conventionally reported performance at a single checkpoint taken at the end of training. The research team questioned the reliability of this single-checkpoint evaluation and conducted a full-trajectory audit, measuring temporal alignment across all 90 checkpoints from five training runs over their entire training budgets.

The audit revealed that event-level onset correspondence peaks within the first 20–30K training steps and then erodes steadily over the following hundreds of thousands of steps, while the distributional quality of the generated audio (FD, IS, etc.) keeps improving over the same interval.

Using a tolerance-wise decomposition and a detector-free event-triggered average analysis, the team identified what is actually lost. As training proceeds, the onsets that survive are placed more tightly (F1 at 50 ms rises), but fewer reference events retain any generated counterpart (F1 at 300 ms falls). The mechanism is that the response at true events holds steady while the background response between events doubles, halving the event-to-background contrast. Controlled experiments further showed that a placement delay of roughly 65–86 ms is mediated by the synchronization pathway.

To test this diagnosis, the team compared four interventions under an identical backbone, seed, and batch configuration across full training trajectories: visual-explainability-based data curation, freezing the synchronization branch (sync-freeze), input-side onset injection, and an output-side auxiliary onset loss. Sync-freeze—stopping updates to the synchronization pathway (7.6M parameters) at 20K steps—was the only intervention whose effect exceeded the measured variation scale: it improved onset F1 by +0.012 at matched training budgets and shifted the timing–quality trade-off frontier outward. The output-side auxiliary loss consistently reduced the placement delay, whereas input-side onset injection was found to have no measurable effect on the onset structure of the generated audio at inference.

This work shows that sync-freeze, which simply fixes the synchronization module during training, retains higher onset alignment at the same training budget at no additional cost and pushes the timing–quality trade-off frontier outward. At the same time, it reveals that the standard practice of reporting temporal alignment at a single checkpoint can create an apparent gain, driven by training-budget mismatch alone, that is larger than the effect of the method itself. The full-trajectory, matched-budget comparison protocol and the public benchmark proposed by the team are expected to help reshape both the solutions and the evaluation standards of V2A temporal alignment research.

First author JoonHo Byun commented: “This work began by treating onset alignment in video-to-audio synthesis as a property of the entire training process rather than a number read from a single checkpoint. We showed through measurement that as training proceeds, audio quality improves while the correspondence between visible events and generated sounds erodes, and that the standard synchronization metric does not capture this. Starting from the diagnosis that the cause lies in late-training re-adaptation of the synchronization pathway, we proposed sync-freeze, which stops training only that pathway once its timing structure has formed, and showed that it retains higher alignment at the same training budget without any extra parameters or loss terms. What I found most meaningful was that this stood in contrast to the familiar approach of feeding the model more timing information, which turned out to be ineffective—it made us ask not ‘what should we add?’ but ‘what should we protect?’. Going forward, we plan to validate sync-freeze across multiple seeds and backbones and to extend this line of work toward video encoders specialized for temporal alignment. I am grateful to my co-authors Tae Gon Kim and Juyeon Park, and to Prof. Unsang Park for his guidance.”

The IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) is a premier international conference in computer vision co-sponsored by the IEEE Computer Society and the Computer Vision Foundation (CVF), and is classified as an excellent conference in the software-field conference list of the Korean Institute of Information Scientists and Engineers (KIISE). WACV 2027 will be held in Florida, USA, from January 5 to 9, 2027.

References:

• IEEE/CVF Winter Conference on Applications of Computer Vision 2027 (WACV 2027)

• Website: https://wacv.thecvf.com/