Vision

Sending images to a model, where the tokens come from, and what is still rejected.

Models that declare vision accept image parts alongside text in the same message.

Sending an image

content becomes an array of parts instead of a string:

{
  "messages": [
    {
      "role": "user",
      "content": [
        { "type": "text", "text": "What is on this label?" },
        {
          "type": "image_url",
          "image_url": { "url": "https://example.com/label.jpg", "detail": "auto" }
        }
      ]
    }
  ]
}

detail is auto, low or high. It trades cost against how much the model can resolve: low sends a small fixed rendition, high sends tiles, auto lets the model decide.

A data URI works the same way, which is usually easier than hosting the file:

{ "type": "image_url", "image_url": { "url": "data:image/jpeg;base64,/9j/4AAQ…" } }

Images are counted as input tokens

There is no separate line for images in what you are billed. An image is converted to a token count by detail and dimensions, and that count lands in prompt_tokens alongside the text — so usage is still one number you can reconcile.

A high detail image of any size costs materially more than low. If you are processing many images and only need to know what kind of thing is in them, low is often enough and is several times cheaper.

Not every model can see

Vision is a model capability. Sending an image part to a model whose providers cannot serve it returns MEDIA_UNSUPPORTED — before the image is fetched or stored, so a mistaken request does not cost you a transfer.

Video and audio are not available yet

video_url and input_audio parts are defined in the API’s schema but are rejected today with MEDIA_UNSUPPORTED. They are in the schema so that the shape is fixed now and your code will not need to change when they open; they are rejected because they need an upload path and duration accounting that do not exist yet.

The schema describing a part is not a promise that the part is accepted. Where something is not available, this API says so with an error rather than dropping it — see Tool calling for the same principle.

Sizes

An image beyond the size limit returns 413 MEDIA_TOO_LARGE. A URL that cannot be fetched returns MEDIA_FETCH_FAILED, which is retryable — the network between us and your host is a thing that fails temporarily.

Next

Usage and billing

Where image tokens appear in what you are charged.

Chat completions

The full content-part schema.