audiocraft_skill

Overview

This skill provides a PyTorch-based audio generation toolkit for creating music and sound from text using MusicGen and AudioGen, plus EnCodec for high-fidelity codec operations. It supports melody-conditioned generation, style transfer, stereo output, and multiple model sizes for tradeoffs between speed and quality. Use it to prototype music apps, generate sound effects, or integrate controllable audio synthesis into pipelines.

How this skill works

The models encode text (T5) to embeddings that a transformer decoder autoregressively converts into discrete audio tokens. EnCodec decodes tokens back to waveforms with configurable sampling rates and stereo support. Generation parameters (duration, temperature, top_k, cfg_coef) control creativity and adherence to prompts; specialized models add melody conditioning or style reference conditioning.

When to use it

• Generate music from natural-language prompts (MusicGen).
• Create short sound effects or environmental sounds (AudioGen).
• Condition music on an existing melody or reference style.
• Produce stereo audio or high-fidelity reconstructions with EnCodec.
• Batch-generate assets for games, films, or demos.

Best practices

• Choose model size based on resources: small for quick tests, medium/large for better quality.
• Set duration and sampling rate explicitly to control memory and output length.
• Use classifier-free guidance (cfg_coef) to balance fidelity to the prompt vs. diversity.
• Batch prompts to maximize GPU throughput and reduce overhead.
• Clear CUDA cache and use half precision on limited GPUs to lower memory usage.

Example use cases

• Build a web demo that turns user text into 8–30s music tracks with adjustable temperature and CFG.
• Generate libraries of sound effects for games (footsteps, explosions, weather) using AudioGen in batch mode.
• Create melody-conditioned arrangements by supplying an input melody and a genre prompt to MusicGen-melody.
• Apply style-transfer: feed a reference clip to MusicGen-style to produce new music matching a target timbre or production style.
• Compress and reconstruct audio assets using EnCodec to store audio as discrete codes and decode on demand.