How an automated clip pipeline works, stage by stage

An automated clip pipeline takes one long recording and returns a set of short, captioned, correctly framed videos. Under the hood it is a batch job with seven or eight stages, and almost every production problem comes from treating it as one big script instead of a chain of small, restartable steps.

This guide walks through each stage, shows the commands that do the heavy lifting, and points out where pipelines usually break. The examples use ffmpeg and OpenAI Whisper because both are open source and widely deployed, but the shape is the same whatever tools you pick.

The shape of the job

Think of the pipeline as a directed list of stages, each of which reads artifacts from storage and writes new ones:

ingest -> probe -> extract audio -> transcribe -> detect highlights
       -> cut -> reframe -> caption -> encode/export -> store -> notify

Two rules make the rest easy. First, every stage writes its output to a deterministic location derived from the job ID, so a rerun can skip finished work. Second, no stage mutates the input of another. If the cutter needs a different audio file, it creates one, it does not overwrite the transcription input.

A minimal job record looks like this:

{
  "job_id": "ep-2026-10-12-a91f",
  "source": "s3://raw/ep-2026-10-12.mp4",
  "status": "transcribed",
  "stages": {
    "probe":      {"done": true, "output": "jobs/ep-2026-10-12-a91f/probe.json"},
    "audio":      {"done": true, "output": "jobs/ep-2026-10-12-a91f/audio.wav"},
    "transcribe": {"done": true, "output": "jobs/ep-2026-10-12-a91f/transcript.json"},
    "highlights": {"done": false, "attempts": 1, "last_error": "timeout"}
  },
  "outputs": []
}

1. Ingest and probe

Ingest means getting the source into storage you control: an upload from a browser, a file dropped in a bucket, or a download of a recording you have the rights to. Large files should be uploaded in parts or with a resumable protocol, because a single failed request on a multi gigabyte file is common.

Before doing anything expensive, probe the file. It is cheap, and it catches unsupported codecs, variable frame rate recordings and missing audio tracks before you spend compute on them.

ffprobe -v error \
  -show_entries format=duration:stream=codec_type,codec_name,width,height,r_frame_rate \
  -of json episode.mp4 > probe.json

Store the probe result next to the job. Later stages read width, height and frame rate from it rather than probing again.

2. Extract audio and transcribe

Speech recognition does not need your 4K video. Extract a mono, 16 kHz WAV once and feed that to the recognizer. It is small, fast to read and identical across reruns.

ffmpeg -i episode.mp4 -vn -ac 1 -ar 16000 -c:a pcm_s16le audio.wav

whisper audio.wav --model turbo --output_format json \
  --word_timestamps True --output_dir jobs/ep-2026-10-12-a91f/

The Whisper CLI defaults to the turbo model and accepts txt, vtt, srt, tsv, json, jsonl or all as output formats (see transcribe.py). Ask for JSON with word timestamps: segment level timing is too coarse to cut on, and you will need word timing again for captions.

3. Detect highlights

This is the stage that makes or breaks the product, and there is no single correct algorithm. Most systems combine several weak signals into a score per candidate window:

• Language signals from the transcript: a complete thought, a question and its answer, a strong opinion, a punchline, a quotable line.
• Audio energy: sudden loudness, laughter, raised voices.
• Visual change: scene cuts, motion, a face appearing.
• Structure: a clip should start at the beginning of a sentence and end after a pause, never mid word.

ffmpeg can produce the raw audio and visual signals for you. Momentary loudness per frame:

ffmpeg -i audio.wav \
  -af "ebur128=metadata=1,ametadata=mode=print:key=lavfi.r128.M:file=loudness.txt" \
  -f null -

Silences, which make good cut boundaries:

ffmpeg -i audio.wav -af silencedetect=noise=-35dB:d=0.4 -f null - 2> silence.log

Scene changes, written to a text file you can parse:

ffmpeg -i episode.mp4 \
  -vf "select='gt(scene,0.4)',metadata=print:file=scenes.txt" -f null -

The thresholds above are starting points, not recommendations. Tune them on your own footage and keep the values in config, not in code. Then snap every candidate window to the nearest sentence start and the nearest silence after the last word. That single step removes most of the "clip starts mid word" complaints.

The output of this stage should be a plain list of windows with a score and a reason, which makes debugging much easier than a black box:

[
  {"start": 754.20, "end": 812.04, "score": 0.81, "reason": "question + answer, laughter at 801s"},
  {"start": 1290.55, "end": 1331.10, "score": 0.74, "reason": "strong statement, loudness peak"}
]

4. Cut, reframe and caption

For a vertical clip from a 16:9 source you crop a 9:16 window and scale it to 1080x1920. Seeking before -i is fast and, when you re-encode, frame accurate.

ffmpeg -ss 754.2 -i episode.mp4 -t 57.84 \
  -vf "crop=ih*9/16:ih,scale=1080:1920,subtitles=clip-001.srt" \
  -c:v libx264 -preset medium -crf 20 -pix_fmt yuv420p \
  -c:a aac -b:a 128k -ar 48000 -movflags +faststart clip-001.mp4

A few details that matter:

• With input seeking, timestamps restart at zero, so clip-001.srt must be rebased to the clip, not to the full episode.
• The subtitles filter needs an ffmpeg build with libass. Check with ffmpeg -filters | grep subtitles.
• crop=ih*9/16:ih centers the crop. To follow a speaker, pass an x offset, for example crop=ih*9/16:ih:1200:0, computed per shot from face or facecam detection.
• Avoid -c copy for final clips. Stream copy can only start on a keyframe, so the clip may begin earlier than requested or open on a frozen frame.

Reframing is where a lot of engineering time goes. A fixed center crop is fine for a single speaker sitting in the middle. Two speakers, a facecam in a corner or gameplay with action at the edges each need tracking and smoothing so the crop does not jitter. Format choices for the export step are covered in detail in video format and codec for short clips.

5. Export, storage and naming

Write outputs to object storage with keys that encode the job and clip, for example clips/ep-2026-10-12-a91f/001-vertical.mp4. Keep a small manifest per job with the window, score, transcript excerpt and output keys. That manifest is what your CMS, your scheduler or your review UI reads. The guide on headless video tools shows how to push it into Drupal or WordPress.

Set lifecycle rules early. Intermediate WAV files and rejected candidates can expire after a few days. Final clips and manifests usually stay.

6. Retries, idempotency and failure modes

Every stage should be safe to run twice. In practice that means:

• Write to a temporary key, then rename or copy to the final key only on success.
• Before running a stage, check whether its output already exists and is valid (non zero size, probes correctly).
• Retry transient errors (network, timeouts, preempted workers) with exponential backoff and a cap.
• Do not retry deterministic errors (corrupt input, unsupported codec). Mark the job failed with a readable reason.
• Put a timeout on every external process. A hung ffmpeg process will hold a worker forever.
• Log the exact command line for each ffmpeg call, so you can reproduce a bad clip locally.

The classic failure modes are variable frame rate phone recordings that drift out of sync after cutting (normalize the frame rate during the cut with -r or the fps filter), sources with no audio stream (the probe catches these), and very long files that exceed a worker's disk. Stream the source from storage or split ingest into chunks when that happens.

7. Where the cost goes

Without inventing numbers, the cost structure is predictable:

• Transcription scales with audio length and model size. It is the main reason to cache transcripts.
• Highlight scoring is cheap if it is rule based, and can become the largest line item if every window goes through a large language model. Prefilter with cheap signals first.
• Encoding scales with output pixels, clip count and encoder preset. Encoding ten candidates when you will publish three is wasted compute: render previews at low resolution, final renders only for approved clips.
• Storage and egress grow quietly. Expire intermediates and serve finished clips from a CDN.

Measure cost per source minute on your own workload, then decide which stages are worth optimizing.

Build it or skip it

Running this pipeline yourself makes sense when clipping is core to your product or you need full control over every stage. If you only want the clips, a hosted tool does the same chain for you. ViceForge, for example, takes an uploaded MP4, MOV, AVI or WebM file, finds moments with AI, adds captions automatically and exports in 9:16, 16:9 or 4:3. Its AI Creator mode for talk and podcast videos covers the transcript-driven case described above. For a deeper comparison of both paths, read open source vs hosted video repurposing.

Pipeline checklist

• Every stage reads and writes deterministic keys derived from the job ID.
• Probe before you spend compute.
• Transcribe once from a mono 16 kHz WAV and keep word timestamps.
• Highlight windows snap to sentence starts and trailing silence.
• Final clips are re-encoded, not stream copied, with +faststart.
• Captions are rebased to clip time.
• Retries only for transient errors, with timeouts on every process.
• Previews are cheap, final renders only for approved clips.

Next: automating Shorts publishing with APIs and webhooks.