Video Understanding API

The Gemini series supports video input and can be used for content summarization, scene recognition, subtitle extraction, time-based Q&A, and more. Video understanding is implemented through multimodal input via Gemini Native Calls.

Invocation Method

Use the native generateContent API and pass the video as inline_data (small files as base64):

curl "https://api.4allapi.com/v1beta/models/gemini-3.5-flash:generateContent" \
  -H "Authorization: Bearer $AIPROXY_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "contents": [{
      "parts": [
        {"inline_data": {"mime_type": "video/mp4", "data": "<base64 视频数据>"}},
        {"text": "Summarize the main content of this video and list the key time points"}
      ]
    }]
  }'

Capabilities and Scenarios

• Content summarization: compress long videos into a list of key points;
• Scene/object recognition: “What products appear in the video?”;
• Timeline Q&A: “At what second does the price information start appearing?”;
• Compliance spot checks: batch-check whether assets contain prohibited elements.

Key Usage Notes

• Videos are converted into tokens based on frames + audio, and input usage is much larger than text; for long videos, trim them first or extract key segments;
• Base64 inline is suitable for short clips (recommended ≤20MB); for longer videos, split them first;
• It is recommended to start with gemini-3.5-flash; use gemini-3.1-pro for more complex analysis;
• For image understanding (single frame) scenarios, Image Understanding (Vision) is more cost-effective.