TorchCodec consolidates media decoding and encoding as other libraries focus on transformations
The media stack now separates decoding and encoding from transformations, with libraries able to follow their own release schedules.
AI illustrationMedia decoding and encoding have been consolidated in TorchCodec, while TorchVision and TorchAudio now focus on transformations. The reorganisation covers images, video and audio, replacing a previously fragmented collection of interfaces and backends.
For users, the division separates moving media into and out of tensors from the operations performed between those steps. TorchCodec handles decoding files or encoded bytes into tensors and encoding tensors back into media files. TorchVision supplies image and video transformations; TorchAudio handles audio transformations.
The changes also affect existing software. Decoding and encoding interfaces previously supplied by TorchVision and TorchAudio have been deprecated or removed. Models, datasets and pipelines in those libraries are no longer under active development, while alternatives elsewhere in the ecosystem cover those functions.
Nicolas Hug and Scott Schneider described maintenance and performance as reasons for concentrating media input and output in a single library. Their account says TorchCodec generally performs better than the earlier implementations, particularly when decoding video with CUDA. Consolidation also concentrates work involving FFmpeg, NVIDIA's codec SDK and libraries for individual image formats.
The authors said the transition was especially disruptive for TorchAudio, with numerous interfaces deprecated and ultimately removed. Community feedback led them to retain several popular interfaces that they had initially planned to withdraw.
All three libraries are now ABI stable, according to the authors. A library version is no longer tied to a single framework release and can continue working with subsequent versions. The libraries therefore do not require rebuilding for each framework release and no longer follow its release schedule. The consolidation took place over the past two years, rather than representing a single new media feature.