ViT-GPT2 image captioning — in-browser image to text model (479 MB, Apache-2.0)

Image to text model for transformers.js (q8). Runs on WebGPU or WASM — 479 MB downloaded once from models.skillsafe.ai, then cached for every SkillSafe app that uses it. Inference happens on your device; nothing you enter is uploaded to load it, and it never costs a credit. Weights from Hugging Face · skillsafe-ai/vit-gpt2-image-captioning, pinned at 0ea8602f2c9f — also loadable straight from Hugging Face outside SkillSafe (how).

Use it in an app Browse all models Request a variant

Details

Catalogue idvit-gpt2-image-captioning-q8@215b4edc

Taskimage to text

Runtimetransformers.js ≥ 3.0.0

DeviceWebGPU, WASM

Variantq8

Download479 MB · 2 files

LicenceApache-2.0 · notice

SourceHugging Face · skillsafe-ai/vit-gpt2-image-captioning

Pinned at0ea8602f2c9f46c40f497298a7bf8dc4884735d0

ApprovedSep 22, 2026

Statusactive — every file is a live vetted hash

API/v1/browser-models/vit-gpt2-image-captioning-q8

Files

Each file is served at an immutable URL; the canonical form is the SHA-256 itself. Tokenizer and config JSON are not here by design — they ship in your app bundle.

Path	Format	Size	SHA-256
`onnx/decoder_model_merged_quantized.onnx`	onnx	151 MB	`1f3ec53b5fc3…dfa98f`
`onnx/encoder_model.onnx`	onnx	328 MB	`1e624fcd6a4a…090b29`

Signature

Graph inputs and outputs read from the ONNX bytes at vetting — the tensor names your session.run() call feeds and reads. Symbolic dimensions are shown by name.

`onnx/decoder_model_merged_quantized.onnx`

Inputs

input_ids int64 [batch_size, decoder_sequence_length]
encoder_hidden_states float32 [batch_size, encoder_sequence_length, 768]
past_key_values.0.key float32 [batch_size, 12, past_sequence_length, 64]
past_key_values.0.value float32 [batch_size, 12, past_sequence_length, 64]
past_key_values.1.key float32 [batch_size, 12, past_sequence_length, 64]
past_key_values.1.value float32 [batch_size, 12, past_sequence_length, 64]
past_key_values.2.key float32 [batch_size, 12, past_sequence_length, 64]
past_key_values.2.value float32 [batch_size, 12, past_sequence_length, 64]
past_key_values.3.key float32 [batch_size, 12, past_sequence_length, 64]
past_key_values.3.value float32 [batch_size, 12, past_sequence_length, 64]
past_key_values.4.key float32 [batch_size, 12, past_sequence_length, 64]
past_key_values.4.value float32 [batch_size, 12, past_sequence_length, 64]
past_key_values.5.key float32 [batch_size, 12, past_sequence_length, 64]
past_key_values.5.value float32 [batch_size, 12, past_sequence_length, 64]
past_key_values.6.key float32 [batch_size, 12, past_sequence_length, 64]
past_key_values.6.value float32 [batch_size, 12, past_sequence_length, 64]
past_key_values.7.key float32 [batch_size, 12, past_sequence_length, 64]
past_key_values.7.value float32 [batch_size, 12, past_sequence_length, 64]
past_key_values.8.key float32 [batch_size, 12, past_sequence_length, 64]
past_key_values.8.value float32 [batch_size, 12, past_sequence_length, 64]
past_key_values.9.key float32 [batch_size, 12, past_sequence_length, 64]
past_key_values.9.value float32 [batch_size, 12, past_sequence_length, 64]
past_key_values.10.key float32 [batch_size, 12, past_sequence_length, 64]
past_key_values.10.value float32 [batch_size, 12, past_sequence_length, 64]
past_key_values.11.key float32 [batch_size, 12, past_sequence_length, 64]
past_key_values.11.value float32 [batch_size, 12, past_sequence_length, 64]
use_cache_branch bool [1]

Outputs

logits float32 [batch_size, decoder_sequence_length, 50257]
present.0.key float32 [batch_size, 12, past_sequence_length + 1, 64]
present.0.value float32 [batch_size, 12, past_sequence_length + 1, 64]
present.1.key float32 [batch_size, 12, past_sequence_length + 1, 64]
present.1.value float32 [batch_size, 12, past_sequence_length + 1, 64]
present.2.key float32 [batch_size, 12, past_sequence_length + 1, 64]
present.2.value float32 [batch_size, 12, past_sequence_length + 1, 64]
present.3.key float32 [batch_size, 12, past_sequence_length + 1, 64]
present.3.value float32 [batch_size, 12, past_sequence_length + 1, 64]
present.4.key float32 [batch_size, 12, past_sequence_length + 1, 64]
present.4.value float32 [batch_size, 12, past_sequence_length + 1, 64]
present.5.key float32 [batch_size, 12, past_sequence_length + 1, 64]
present.5.value float32 [batch_size, 12, past_sequence_length + 1, 64]
present.6.key float32 [batch_size, 12, past_sequence_length + 1, 64]
present.6.value float32 [batch_size, 12, past_sequence_length + 1, 64]
present.7.key float32 [batch_size, 12, past_sequence_length + 1, 64]
present.7.value float32 [batch_size, 12, past_sequence_length + 1, 64]
present.8.key float32 [batch_size, 12, past_sequence_length + 1, 64]
present.8.value float32 [batch_size, 12, past_sequence_length + 1, 64]
present.9.key float32 [batch_size, 12, past_sequence_length + 1, 64]
present.9.value float32 [batch_size, 12, past_sequence_length + 1, 64]
present.10.key float32 [batch_size, 12, past_sequence_length + 1, 64]
present.10.value float32 [batch_size, 12, past_sequence_length + 1, 64]
present.11.key float32 [batch_size, 12, past_sequence_length + 1, 64]
present.11.value float32 [batch_size, 12, past_sequence_length + 1, 64]

`onnx/encoder_model.onnx`

Inputs

pixel_values float32 [batch_size, num_channels, height, width]

Outputs

last_hidden_state float32 [batch_size, encoder_sequence_length, 768]

Use it in an app declaration · SDK loader · transformers.js · URLs · Hugging Face — generated from this entry

Add to the body of POST /v1/apps/{slug}/releases (or a release session). An unknown or withdrawn model is refused with a 400 naming it; the app page then shows "downloads 479 MB · runs on your device" and /models.txt carries the attribution.

{
  "models": [
    {
      "id": "vit-gpt2-image-captioning-q8",
      "revision": "215b4edc"
    }
  ]
}

Streams with progress, verifies the SHA-256 against the catalogue, keeps a durable copy in the app's Cache API and reports whether the bytes came from cache or the network.

<script src="/sdk.js"></script>  <!-- vendored from https://skillsafe.ai/apps-sdk/v1.js -->

const ss = SkillSafe.init({ slug: "your-app" });
const caps = await ss.models.capabilities();        // { webgpu, wasm, wasmSimd, storage }
const files = await ss.models.loadAll("vit-gpt2-image-captioning-q8@215b4edc", {
  onProgress: (p) => (bar.style.width = Math.round(p.ratio * 100) + "%"),
});                                                   // { path: ArrayBuffer } — SHA-256 verified, cached
const decoder_model_merged_quantized = files["onnx/decoder_model_merged_quantized.onnx"];
const encoder_model = files["onnx/encoder_model.onnx"];

// Later visits: await ss.models.status("vit-gpt2-image-captioning-q8@215b4edc")  → { cached: [...], missing: [] }
// Background prefetch on a landing page: ss.models.warm("vit-gpt2-image-captioning-q8@215b4edc")

The runtime itself ships in your bundle as a vetted {path, sha256} reference — only the weights come from the registry. Pass the bytes in; do not re-fetch by URL.

import { env, pipeline } from "./transformers.min.js";
env.allowLocalModels = true;
env.localModelPath = "/models/";                   // ship config.json + tokenizer.json at /models/vit-gpt2-image-captioning-q8/
env.allowRemoteModels = true;                      // only the weights fall through to the registry
env.remoteHost = "https://models.skillsafe.ai/";
env.remotePathTemplate = "{model}@{revision}/";    // → https://models.skillsafe.ai/vit-gpt2-image-captioning-q8@215b4edc/onnx/decoder_model_merged_quantized.onnx

const pipe = await pipeline("image-to-text", "vit-gpt2-image-captioning-q8", {
  revision: "215b4edc",
  dtype: "q8",
  device: "webgpu",
});

Immutable, credential-free, Access-Control-Allow-Origin: *, range requests honoured. The host refuses requests carrying a foreign Origin; a plain curl is fine.

https://models.skillsafe.ai/vit-gpt2-image-captioning-q8@215b4edc/onnx/decoder_model_merged_quantized.onnx
https://models.skillsafe.ai/vit-gpt2-image-captioning-q8@215b4edc/onnx/encoder_model.onnx

curl -sSL -o decoder_model_merged_quantized.onnx "https://models.skillsafe.ai/vit-gpt2-image-captioning-q8@215b4edc/onnx/decoder_model_merged_quantized.onnx"
shasum -a 256 decoder_model_merged_quantized.onnx   # 1f3ec53b5fc3614c0b54ee786755a7ec3007841f57543f8e241499edfadfa98f

SkillSafe's copy on Hugging Face — the same bytes (the SHA-256 on each file page matches the hash above), plus manifest.json with the upstream commit, recipe and toolchain, the parity reference tensors and a model card. Use it directly for your own site or notebook. A SkillSafe app must load from the registry: its CSP allows models.skillsafe.ai only, and that copy is what the SDK verifies, the browser shares across apps and the app page discloses. SkillSafe keeps hosting its own copies and the API either way.

// Outside SkillSafe apps only: app hosts allow connect-src models.skillsafe.ai, not huggingface.co.
import { pipeline } from "@huggingface/transformers";
const pipe = await pipeline("image-to-text", "skillsafe-ai/vit-gpt2-image-captioning", { revision: "0ea8602f2c9f46c40f497298a7bf8dc4884735d0", device: "webgpu" });

https://huggingface.co/skillsafe-ai/vit-gpt2-image-captioning/resolve/0ea8602f2c9f46c40f497298a7bf8dc4884735d0/onnx/decoder_model_merged_quantized.onnx
https://huggingface.co/skillsafe-ai/vit-gpt2-image-captioning/resolve/0ea8602f2c9f46c40f497298a7bf8dc4884735d0/onnx/encoder_model.onnx

Evaluation

onnx.checker + onnxruntime CPU smoke run with zero-filled inputs at the declared shapes; per-file SHA-256 pinned to the source — imported as published upstream, then checked. Evaluated Sep 22, 2026.

Runs under onnxruntime

Zero-filled inputs at the declared shapes, CPU execution provider on the converter host; the check is that the graph loads, runs, and emits the declared output shapes.

onnx/decoder_model_merged.onnx: input_ids[1,4], encoder_hidden_states[1,197,768], past_key_values.0.key[1,12,1,64], past_key_values.0.value[1,12,1,64], past_key_values.1.key[1,12,1,64], past_key_values.1.value[1,12,1,64], past_key_values.2.key[1,12,1,64], past_key_values.2.value[1,12,1,64], past_key_values.3.key[1,12,1,64], past_key_values.3.value[1,12,1,64], past_key_values.4.key[1,12,1,64], past_key_values.4.value[1,12,1,64], past_key_values.5.key[1,12,1,64], past_key_values.5.value[1,12,1,64], past_key_values.6.key[1,12,1,64], past_key_values.6.value[1,12,1,64], past_key_values.7.key[1,12,1,64], past_key_values.7.value[1,12,1,64], past_key_values.8.key[1,12,1,64], past_key_values.8.value[1,12,1,64], past_key_values.9.key[1,12,1,64], past_key_values.9.value[1,12,1,64], past_key_values.10.key[1,12,1,64], past_key_values.10.value[1,12,1,64], past_key_values.11.key[1,12,1,64], past_key_values.11.value[1,12,1,64], use_cache_branch[1] → logits[1,4,50257], present.0.key[1,12,4,64], present.0.value[1,12,4,64], present.1.key[1,12,4,64], present.1.value[1,12,4,64], present.2.key[1,12,4,64], present.2.value[1,12,4,64], present.3.key[1,12,4,64], present.3.value[1,12,4,64], present.4.key[1,12,4,64], present.4.value[1,12,4,64], present.5.key[1,12,4,64], present.5.value[1,12,4,64], present.6.key[1,12,4,64], present.6.value[1,12,4,64], present.7.key[1,12,4,64], present.7.value[1,12,4,64], present.8.key[1,12,4,64], present.8.value[1,12,4,64], present.9.key[1,12,4,64], present.9.value[1,12,4,64], present.10.key[1,12,4,64], present.10.value[1,12,4,64], present.11.key[1,12,4,64], present.11.value[1,12,4,64] 17.3 ms
onnx/decoder_model_merged_quantized.onnx: input_ids[1,4], encoder_hidden_states[1,197,768], past_key_values.0.key[1,12,1,64], past_key_values.0.value[1,12,1,64], past_key_values.1.key[1,12,1,64], past_key_values.1.value[1,12,1,64], past_key_values.2.key[1,12,1,64], past_key_values.2.value[1,12,1,64], past_key_values.3.key[1,12,1,64], past_key_values.3.value[1,12,1,64], past_key_values.4.key[1,12,1,64], past_key_values.4.value[1,12,1,64], past_key_values.5.key[1,12,1,64], past_key_values.5.value[1,12,1,64], past_key_values.6.key[1,12,1,64], past_key_values.6.value[1,12,1,64], past_key_values.7.key[1,12,1,64], past_key_values.7.value[1,12,1,64], past_key_values.8.key[1,12,1,64], past_key_values.8.value[1,12,1,64], past_key_values.9.key[1,12,1,64], past_key_values.9.value[1,12,1,64], past_key_values.10.key[1,12,1,64], past_key_values.10.value[1,12,1,64], past_key_values.11.key[1,12,1,64], past_key_values.11.value[1,12,1,64], use_cache_branch[1] → logits[1,4,50257], present.0.key[1,12,4,64], present.0.value[1,12,4,64], present.1.key[1,12,4,64], present.1.value[1,12,4,64], present.2.key[1,12,4,64], present.2.value[1,12,4,64], present.3.key[1,12,4,64], present.3.value[1,12,4,64], present.4.key[1,12,4,64], present.4.value[1,12,4,64], present.5.key[1,12,4,64], present.5.value[1,12,4,64], present.6.key[1,12,4,64], present.6.value[1,12,4,64], present.7.key[1,12,4,64], present.7.value[1,12,4,64], present.8.key[1,12,4,64], present.8.value[1,12,4,64], present.9.key[1,12,4,64], present.9.value[1,12,4,64], present.10.key[1,12,4,64], present.10.value[1,12,4,64], present.11.key[1,12,4,64], present.11.value[1,12,4,64] 23.6 ms
onnx/encoder_model.onnx: pixel_values[1,3,224,224] → last_hidden_state[1,197,768] 25.4 ms

Toolchain: python 3.12.13 · platform Darwin 25.6.0 arm64 · torch 2.10.0 · onnx 1.23.0 · onnxruntime 1.30.0. Recipe models/recipes/vit-gpt2-image-captioning.yaml (2e3ec6f04880). Full manifest.json

Licence & attribution

Apache-2.0 · licence text · notice

vit-gpt2-image-captioning: nlpconnect (Ankur Kumar), Apache License 2.0 (https://huggingface.co/nlpconnect/vit-gpt2-image-captioning); ONNX export by Xenova.

Apps that declare this model get this text in their generated /models.txt, so a licence that requires a notice always carries one.

Every file here was approved by exact SHA-256 after a structural audit of the graph, fetched from a content-pinned source, and is served credential-free at an immutable URL. The SDK re-verifies the hash on your device before it caches or returns anything. Missing a variant? Request it — or read how the registry works.