Lipsync

Sync lips in an image or a video to match an audio track. Provide a source image or video and an audio track, and Kolbo generates a video with matching lip movements.

Smart Select: Omit model (or send "auto" / "smart-select") and Kolbo picks a model from the family matching your source — see Two model families. Passing a specific identifier gives you deterministic behaviour, but you then have to match the family yourself. An identifier that matches no catalogued model is rejected with 400 MODEL_NOT_FOUND.

Model identifiers are Kolbo-specific. Never hardcode model identifiers — always fetch the current list from GET /api/v1/models?type=lipsync first. Models may be added, renamed, or retired at any time.

All generation endpoints accept an optional project_id body field that routes the output into a specific project. See Projects.

Two model families

Lipsync models are catalogued under two distinct model types, and they accept different sources:

Model type	Source it takes	Query
`lipsync-image`	A still portrait — the model animates the face	`GET /api/v1/models?type=lipsync-image`
`lipsync-video`	An existing video of a talking head — the model re-syncs the mouth	`GET /api/v1/models?type=lipsync-video`

GET /api/v1/models?type=lipsync returns both families in one list; read each model's types array to see which one it belongs to.

The family must match your source, and this is enforced at request time:

A lipsync-image model with no image source → 400 MISSING_IMAGE.
A lipsync-video model with no video source → 400 MISSING_VIDEO.
A model that belongs to neither family (for example a text-to-video identifier) is treated as an image model: it fails with 400 MISSING_IMAGE when you send a video, and is attempted on the image path when you send an image. Stick to identifiers from the lipsync lists.

A few models carry both types. For those the request decides: a video source takes the video path, otherwise the image path.

Smart Select (omitting model) sidesteps this: it looks at whether you supplied a video or an image and then picks a routable model from the matching family, so it can never mismatch.

Audio length is gated per model, and the limit is not published on the models endpoint. Before charging credits the API measures your audio and rejects it with 400 DURATION_BELOW_MODEL_MINIMUM or 400 DURATION_EXCEEDS_MODEL_LIMIT. The applicable bound and your actual audio length are stated in the error message of that response, so a rejected request tells you exactly what to trim to. The min_audio_duration / max_audio_duration / min_video_duration / max_video_duration / audio_max_follows_video_duration / supported_audio_formats fields on GET /api/v1/models are catalog metadata and are not what this endpoint checks against.

Dubbing models cannot be driven from this endpoint. Some lipsync-video models are dubbing models, which require a target language. This endpoint does not accept one, so those models always return 400 MISSING_TARGET_LANGUAGE. Pick a non-dubbing lipsync model.

Endpoint

POST /api/v1/generate/lipsync

Rate limit: 30 requests per minute per user, counted in a bucket shared with the other standard-limit /v1/generate/* endpoints. Image generation and transcription sit in their own buckets. Requests rejected with an error status do not count against the limit.

Request Body

Accepts multipart/form-data (for file uploads) or application/json (for URL-based inputs). The source slot and the audio slot are independent: you can pass the source as a URL and upload the audio, or the reverse. Within a single slot, a URL and a file are mutually exclusive — see the Callout below the table.

Field	Type	Required	Description
`source_url`	string	Conditional	URL of the source image or video. Required unless an `image` or `video` file is uploaded. The file extension decides the type: `.mp4`, `.mov`, `.webm`, `.mkv`, `.avi`, `.m4v` (optionally followed by a query string) = video, anything else = image.
`image`	file	Conditional	Source image file — multipart field name `image`, 1 file, 100 MB max.
`video`	file	Conditional	Source video file — multipart field name `video`, 1 file, 100 MB max.
`audio_url`	string	Conditional	URL of the audio track. Required unless an `audio` file is uploaded.
`audio`	file	Conditional	Audio file — multipart field name `audio`, 1 file, 100 MB max.
`prompt`	string	No	Performance / scene prompt for image models. Non-English text is translated to English first; when omitted, image models use a built-in neutral-expression prompt. On the video path (every `lipsync-video` model, Sync-3 included) the prompt is discarded — use `emotion` for Sync-3. Must be a string when present (`400 INVALID_PROMPT`).
`model`	string	No	Model identifier from `GET /api/v1/models?type=lipsync-image` or `?type=lipsync-video`. Must be a string, never an array (`400 INVALID_MODEL`). Omitted / `"auto"` / `"smart-select"` → Smart Select. An unknown identifier returns `400 MODEL_NOT_FOUND`.
`bounding_box_target`	array of 2 numbers	No	Primary-face position as `[x, y]`, normalized 0-1 (e.g. `[0.5, 0.4]` for a centred, slightly-high face). Used only by Hedra-provider image models such as Hedra Character 3 (`lipsync-v1`). Accepts an array or a JSON string (multipart). A malformed value is ignored rather than rejected.
`project_id`	string	No	Target project. Omit to use your auto-created "API Generations" project. Needs at least edit permission.
`session_id`	string	No	Add this generation to an existing lipsync session (the `session_id` returned by an earlier lipsync request). When present, `project_id` is ignored and the session's project is used.

Do not fill the same slot twice. Sending source_url together with an image upload returns 400 CONFLICTING_IMAGE_INPUT; with a video upload, 400 CONFLICTING_VIDEO_INPUT; sending audio_url together with an audio upload returns 400 CONFLICTING_AUDIO_INPUT. Mixing across slots (source as URL, audio as file) is fine.

Multipart field names matter. There is no source field — upload the source as image or video depending on what it is, and the audio as audio.

source_url type detection is purely filename-based. A video served from a URL without one of the recognised extensions (a signed URL ending in an id, for example) is classified as an image — which means a lipsync-video model then rejects the request with 400 MISSING_VIDEO, and Smart Select picks an image model that will treat your video as a still. Upload it as a video file instead, or use a URL that keeps the extension.

This endpoint does not accept aspect_ratio, resolution, duration, visual_dna_ids or a dubbing target language — the output follows the source, at each model's default resolution.

The source and the audio are always one item each — there are no array or multi-reference fields on this endpoint. To drive several characters from one track, use Sync-3 active speaker detection or the Speaker Director fields below.

Sync-3 advanced options

The Sync-3 model (sync-lipsync/v3, a lipsync-video model; the internal form fal-ai/sync-lipsync/v3 is accepted too) accepts these additional optional fields. All except sync_mode are ignored by other lipsync models.

Field	Type	Description
`sync_mode`	string	How to reconcile an audio/video length mismatch: `cut_off` (default), `loop`, `bounce`, `silence`, `remap`. Honoured by the whole sync-lipsync family, not just v3.
`model_mode`	string	Which region drives the sync: `lips`, `face`, `head`, `lipsync`, `emotion`, `talking_head`.
`emotion`	string	One of `neutral`, `happy`, `sad`, `angry`, `disgusted`, `surprised` (case-insensitive). The provider accepts only these values: anything else, including free text, is dropped silently rather than rejected. Sync-3 does not use `prompt`.
`temperature`	number	Expressiveness, `0` (subtle) to `1` (energetic). Numeric strings are accepted.
`occlusion_detection_enabled`	boolean	Handle objects (hands, mics) passing in front of the face. `true` or `"true"`.
`active_speaker_detection`	object	Choose which speaker to sync in a multi-person video (see below).

active_speaker_detection fields:

Field	Type	Description
`auto_detect`	boolean	`true` = auto-detect the active speaker. `false` = use your `coordinates`. `coordinates` and `frame_number` are sent only when `auto_detect` is explicitly `false` — otherwise they are ignored. `false` with no usable `coordinates` is sent as `true`.
`v3`	boolean	Use the Sync.so v3 detection engine.
`frame_number`	integer	Frame index the `coordinates` refer to.
`coordinates`	`[x, y]`	Pixel point on the speaker's face, in the resolution of the video as sent to the model. Rounded to integers, and clamped into the frame when the video's dimensions can be probed. Source videos can be normalized before dispatch (e.g. very large or rotated clips), so measure against a video no larger than 1920px on its long side.
`bounding_boxes`	array	Per-frame face boxes `[x1, y1, x2, y2]`.
`bounding_boxes_url`	string	URL to a JSON file with per-frame boxes.
`face_image`	string	Base64-encoded reference face image.

active_speaker_detection may be sent as an object (JSON body) or as a JSON string (multipart).

For multi-person videos, the simplest path is active_speaker_detection: { auto_detect: true }. To pin a specific person, send auto_detect: false with their face as a pixel coordinates point plus the frame_number it was taken from. If the provider rejects the point as not landing on a usable face, the generation is retried once with auto-detect.

Only sync_mode survives every execution path. model_mode, emotion, temperature, occlusion_detection_enabled and active_speaker_detection are attached to the request only when the generation runs on the fal execution path. Some sync-lipsync models are served by another provider first (with fal as the fallback), and that path forwards the source video, the audio and sync_mode only — the remaining options are dropped silently, not rejected. Treat them as best-effort refinements rather than guarantees.

curl -X POST https://api.kolbo.ai/api/v1/generate/lipsync \
  -H "X-API-Key: kolbo_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "source_url": "https://example.com/two-people.mp4",
    "audio_url": "https://example.com/speech.mp3",
    "model": "sync-lipsync/v3",
    "emotion": "happy",
    "active_speaker_detection": { "auto_detect": false, "coordinates": [640, 360], "frame_number": 30 }
  }'

Speaker Director (LTX audio-to-video)

The LTX audio-to-video image models (identifiers containing both ltx and audio-to-video, e.g. ltx-2.3-quality/audio-to-video or lightricks/ltx-2.5/audio-to-video/pro) accept a who-talks-when timeline. Kolbo combines it with a transcript of your audio into a multi-speaker prompt. All fields are optional and ignored by other models, and none of them has any effect unless who_talks_when is set. If composing the prompt fails, the generation continues with the plain prompt.

Field	Type	Description
`who_talks_when`	string	Comma-separated `speakerIndex:start-end` segments in seconds, e.g. `"0:0.00-1.80,1:1.80-3.60"`. Speaker indexes are 0-based.
`speakers`	array or JSON string	Speaker list, e.g. `[{ "index": 0, "position": "left", "label": "woman in red" }]`. When omitted, speakers are detected from the image.
`transcript`	object or JSON string	Optional word-level transcript `{ "text": "...", "words": [{ "text", "start", "end", "speaker" }] }`. When omitted, the audio is transcribed for you.
`character_directions`	array or JSON string	Up to 6 `{ "speaker": <integer 0-5>, "direction": "<up to 500 chars>" }` entries. When `speakers` is sent, entries whose `speaker` matches no `speakers[].index` are dropped.
`scene_prompt`	string	Global scene guidance, up to 1,500 characters. Used instead of `prompt` when set.

curl -X POST https://api.kolbo.ai/api/v1/generate/lipsync \
  -H "X-API-Key: kolbo_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "source_url": "https://example.com/two-people.jpg",
    "audio_url": "https://example.com/dialogue.mp3",
    "model": "ltx-2.3-quality/audio-to-video",
    "who_talks_when": "0:0.00-2.40,1:2.40-5.10",
    "character_directions": [{ "speaker": 1, "direction": "laughs before answering" }]
  }'

Examples

cURL with URLs (Smart Select)

curl -X POST https://api.kolbo.ai/api/v1/generate/lipsync \
  -H "X-API-Key: kolbo_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "source_url": "https://example.com/portrait.jpg",
    "audio_url": "https://example.com/speech.mp3"
  }'

cURL with File Uploads

curl -X POST https://api.kolbo.ai/api/v1/generate/lipsync \
  -H "X-API-Key: kolbo_live_YOUR_API_KEY" \
  -F "[email protected]" \
  -F "[email protected]"

cURL with Video Source

curl -X POST https://api.kolbo.ai/api/v1/generate/lipsync \
  -H "X-API-Key: kolbo_live_YOUR_API_KEY" \
  -F "[email protected]" \
  -F "[email protected]"

cURL with a mixed source (uploaded audio, source URL)

curl -X POST https://api.kolbo.ai/api/v1/generate/lipsync \
  -H "X-API-Key: kolbo_live_YOUR_API_KEY" \
  -F "source_url=https://example.com/portrait.jpg" \
  -F "[email protected]"

With Specific Model

To choose a specific model, first fetch identifiers from the family matching your source (?type=lipsync-image or ?type=lipsync-video), then pass the identifier value:

curl -X POST https://api.kolbo.ai/api/v1/generate/lipsync \
  -H "X-API-Key: kolbo_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "source_url": "https://example.com/portrait.jpg",
    "audio_url": "https://example.com/speech.mp3",
    "model": "your-model-identifier"
  }'

Model identifiers come from GET /api/v1/models?type=lipsync. Always fetch the latest list rather than hardcoding identifiers, as models may change over time.

JavaScript

const API_KEY = "kolbo_live_YOUR_API_KEY";

// Fetch available lipsync models
async function initModels() {
  const res = await fetch("https://api.kolbo.ai/api/v1/models?type=lipsync", {
    headers: { "X-API-Key": API_KEY },
  });
  const data = await res.json();
  console.log("Available models:", data.models.map((m) => m.identifier));
}

async function main() {
  await initModels();

  // Using URLs (Smart Select)
  const response = await fetch("https://api.kolbo.ai/api/v1/generate/lipsync", {
    method: "POST",
    headers: {
      "X-API-Key": API_KEY,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      source_url: "https://example.com/portrait.jpg",
      audio_url: "https://example.com/speech.mp3",
    }),
  });

  const started = await response.json();
  if (!started.success) throw new Error(started.error);

  const result = await pollUntilDone(started.generation_id, started.poll_interval_hint);
  if (result.state !== "completed") throw new Error(result.error || result.state);

  console.log("Video URL:", result.result.urls[0]); // urls is always an array
}

const TERMINAL = new Set(["completed", "failed", "cancelled"]);

// Minimal loop. A production client should also handle transient HTTP errors and
// an overall timeout — see /docs/developer-api/polling-and-cancellation.
async function pollUntilDone(generationId, intervalSeconds) {
  while (true) {
    await new Promise((r) => setTimeout(r, intervalSeconds * 1000));
    const status = await fetch(
      `https://api.kolbo.ai/api/v1/generate/${generationId}/status`,
      { headers: { "X-API-Key": API_KEY } }
    ).then((r) => r.json());
    if (TERMINAL.has(status.state)) return status;
  }
}

main();

Python

import requests
import time

API_KEY = "kolbo_live_YOUR_API_KEY"
BASE_URL = "https://api.kolbo.ai/api"
HEADERS = {"X-API-Key": API_KEY}

# Fetch available lipsync models
models_res = requests.get(
    f"{BASE_URL}/v1/models",
    headers=HEADERS,
    params={"type": "lipsync"},
)
print("Available models:", [m["identifier"] for m in models_res.json()["models"]])

# --- Option A: Using URLs (Smart Select) ---
response = requests.post(
    f"{BASE_URL}/v1/generate/lipsync",
    headers={**HEADERS, "Content-Type": "application/json"},
    json={
        "source_url": "https://example.com/portrait.jpg",
        "audio_url": "https://example.com/speech.mp3",
    },
)

data = response.json()
print("Generation ID:", data["generation_id"])

# --- Option B: Using file uploads ---
# with open("portrait.jpg", "rb") as img, open("speech.mp3", "rb") as aud:
#     response = requests.post(
#         f"{BASE_URL}/v1/generate/lipsync",
#         headers=HEADERS,
#         files={"image": img, "audio": aud},
#     )
#     data = response.json()

TERMINAL = {"completed", "failed", "cancelled"}

# Minimal loop. Add transient-error retries and an overall timeout for production
# — see /docs/developer-api/polling-and-cancellation.
while True:
    time.sleep(data["poll_interval_hint"])
    status = requests.get(
        f"{BASE_URL}/v1/generate/{data['generation_id']}/status",
        headers=HEADERS,
    ).json()
    if status["state"] in TERMINAL:
        break

if status["state"] != "completed":
    raise Exception(status.get("error", status["state"]))

print("Video URL:", status["result"]["urls"][0])  # urls is always a list

Response

This endpoint is asynchronous and fire-and-forget: the POST answers 202 Accepted as soon as the job is queued and never contains a video.

Polling is the only completion mechanism. Kolbo never calls you back: there is no webhook, no callback_url field on this request body, and no route delivers outbound notifications. Socket.IO events are the web app's internal transport and are not part of the API contract — API-key generations are registered so the shared progress emitter drops their events, and the few emit sites that bypass that check are undocumented, unversioned and unsafe to build on. After the POST, loop against GET /api/v1/generate/{generation_id}/status until state is completed, failed or cancelled. Full contract: Polling & Cancellation.

Generation Started

{
  "success": true,
  "generation_id": "6820f1c2a1b2c3d4e5f60718",
  "type": "lipsync",
  "model": "auto",
  "credits_charged": null,
  "poll_url": "/v1/generate/6820f1c2a1b2c3d4e5f60718/status",
  "poll_interval_hint": 8,
  "session_id": "6820f1c2a1b2c3d4e5f60700",
  "project_id": "6820f1c2a1b2c3d4e5f606ff"
}

Field	Type	Notes
`generation_id`	string	Mongo ObjectId. This is what you poll and cancel with.
`type`	string	Always `"lipsync"`
`model`	string	`"auto"` for Smart Select requests, otherwise the catalog identifier of the model you sent, in the public form listed by `GET /api/v1/models` (sending `fal-ai/sync-lipsync/v3` echoes `sync-lipsync/v3`)
`credits_charged`	null	Always `null` on this endpoint — the lipsync pipeline reports no estimate at submit time. Read `credits_used` from the completed status.
`poll_url`	string	The status path without the `/api` prefix. Prepend `https://api.kolbo.ai/api`.
`poll_interval_hint`	number	Suggested seconds between polls — `8` here
`session_id` / `project_id`	string	Where the generation lives in the Kolbo app

Completed Status

GET /api/v1/generate/{generation_id}/status

{
  "success": true,
  "generation_id": "6820f1c2a1b2c3d4e5f60718",
  "type": "lipsync",
  "state": "completed",
  "progress": 100,
  "result": {
    "urls": ["https://media.kolbo.ai/videos/.../lipsync.mp4"],
    "thumbnail_url": "https://media.kolbo.ai/videos/.../thumb.jpg",
    "duration": null,
    "aspect_ratio": "16:9",
    "model": null,
    "created_at": "2026-04-12T10:00:00.000Z"
  },
  "credits_used": 40,
  "credits_breakdown": [
    { "model": "MODEL_IDENTIFIER", "amount": 40, "base": 40, "final": 40, "duration_multiplier": 1, "pricing": null }
  ]
}

The output lives in result.urls — always an array of strings, even though lipsync produces exactly one video. Read urls[0].

result (and credits_used / credits_breakdown) appear only under state: "completed"; a processing body has exactly five keys and no result at all.

Lipsync results are sparser than the other video endpoints. The lipsync record does not persist the fields the shared status shaper reads, so on this endpoint specifically:

result.duration is always null — measure the output yourself, or use the length of the audio you supplied.
result.model is always null — the identifier that ran is reported in credits_breakdown[].model instead, or keep the one you sent.
result.prompt_used is omitted for video models (their prompt is discarded). For image models it holds the prompt actually used: your prompt translated to English, or the built-in default prompt when you sent none.
No creative-input keys (visual_dna, preset, cinematic_presets, moodboard) are ever returned.

urls, thumbnail_url, aspect_ratio and created_at are populated as normal.

Failure and cancellation

A failed generation is still an HTTP 200 with success: true — the failure is in state, with an error string and a best-effort failure object. cancelled carries neither result nor error, so treat all three terminal states explicitly. Failed generations are not charged. See Polling & Cancellation.

Errors

Status	Code	Cause
400	—	Neither `source_url` nor an `image`/`video` upload was supplied
400	—	Neither `audio_url` nor an `audio` upload was supplied
400	`INVALID_PROMPT`	`prompt` sent as a non-string
400	`INVALID_MODEL`	`model` sent as an array or object instead of a string
400	`MISSING_IMAGE`	A `lipsync-image` model (or a non-lipsync identifier) was chosen but the source is not an image
400	`MISSING_VIDEO`	A `lipsync-video` model was chosen but the source is not a video
400	`CONFLICTING_IMAGE_INPUT`	`source_url` (image) sent together with an `image` upload
400	`CONFLICTING_VIDEO_INPUT`	`source_url` (video) sent together with a `video` upload
400	`CONFLICTING_AUDIO_INPUT`	`audio_url` sent together with an `audio` upload
400	`DURATION_BELOW_MODEL_MINIMUM`	Audio shorter than the model's minimum — the bound is stated in the `error` message
400	`DURATION_EXCEEDS_MODEL_LIMIT`	Audio longer than the model's maximum — the bound is stated in the `error` message
400	`MISSING_TARGET_LANGUAGE`	A dubbing model was chosen; this endpoint cannot supply a target language
400	`URL_DOWNLOAD_FAILED`	A supplied `source_url` / `audio_url` could not be fetched
400	`SDK_PROJECT_INVALID_ID` / `SDK_SESSION_INVALID_ID`	`project_id` / `session_id` is not a valid id
403	`SDK_PROJECT_ACCESS_DENIED`	You have less than edit permission on the target project
403	`INSUFFICIENT_CREDITS`	Balance too low — checked before the generation starts. The body also carries `balance`, `required` and `pricing_url`
404	`SDK_PROJECT_NOT_FOUND` / `SDK_SESSION_NOT_FOUND`	Unknown or inaccessible project, or a `session_id` that is not a lipsync session
429	—	More than 30 accepted requests in one minute (shared bucket)

Tips

Lipsync generation typically takes 1-5 minutes depending on the source length.
The audio-duration check runs before credits are deducted, so a too-long or too-short track costs you nothing — retry with the bound reported in the error message.
For best results with lipsync-image models, use a clear, front-facing portrait with visible lips.
Uploads are limited to 100 MB per file.
To choose which face speaks: bounding_box_target on Hedra Character 3, active_speaker_detection on Sync-3, who_talks_when on LTX audio-to-video. Other models ignore these fields.
Credit cost depends on the model and the output length — see Models and Pricing.
Use poll_interval_hint from the initial response to set your polling interval.