debug-tensorflow_skill

Overview

This skill systematically debugs TensorFlow and Keras problems across training, inference, and data pipelines. It guides diagnosis and fixes for shape mismatches, GPU/CUDA detection, OOM errors, NaN/Inf loss, gradient problems, SavedModel loading, and input pipeline bottlenecks. Practical tools include tf.debugging assertions, TensorBoard profiling, eager-mode checks, and version compatibility advice.

How this skill works

The skill walks through a four-phase workflow: reproduce and isolate with minimal examples, analyze shapes/gradients/profiler traces, apply targeted fixes, and verify with tests. It leverages tf.debugging.assert_* and check_numerics, tf.print for graph-mode inspection, TensorBoard/TensorFlow Profiler and the Debugger V2. It also includes GPU/CUDA checks, memory growth and logical device configuration, and guidance for SavedModel and custom objects.

When to use it

• When training crashes with ResourceExhaustedError or CUDA OOM.
• When loss becomes NaN/Inf or gradients explode/vanish.
• When model.save/load raises unknown layer or signature errors.
• When tf.config.list_physical_devices('GPU') returns empty.
• When input pipeline causes slow training or InvalidArgumentError.

Best practices

• Create a minimal reproducible example and enable eager execution for step-by-step debugging.
• Add tf.debugging.assert_shapes and tf.debugging.check_numerics at key points in the pipeline.
• Use TensorBoard profiler and Debugger V2 to find hotspots and bad tensors.
• Enable GPU memory growth or set logical device memory limits before allocating GPU memory.
• Reduce batch size, use mixed precision, or apply gradient checkpointing for large models.

Example use cases

• Resolve a shape mismatch by printing shapes, using tf.ensure_shape, and reshaping inputs before model.forward.
• Stop training NaNs by enabling check_numerics, lowering the optimizer learning rate, and adding gradient clipping.
• Fix GPU detection by verifying TensorFlow build, matching CUDA/cuDNN versions, and running nvidia-smi in the environment.
• Diagnose slow data input by profiling the tf.data pipeline, adding cache/prefetch, and parallelizing map calls.
• Recover a model with Unknown layer errors by saving custom_objects or saving weights and rebuilding the architecture.