API Reference

llamapi-server provides OpenAI-compatible chat, Embeddings, and model-query APIs, together with LlamaPi extensions for on-device model instance management and hardware platform discovery.

Overview

Item	Description
Base URL	`http://127.0.0.1:9265`
API prefix	OpenAI-compatible and management APIs use `/v1`; health does not
Request format	`POST` uses a JSON body; `GET` does not require a body
Response format	JSON for regular APIs, SSE for streaming chat, plain text for health
Request body limit	64 MiB
Authentication	None required
CORS	Cross-origin requests are allowed

The current llamapi-server has no API authentication and permits CORS. Add a firewall, reverse proxy, or other access control before exposing it to an untrusted network.

Quick Example

Load a model:

curl -s http://127.0.0.1:9265/v1/models/load \
  -H 'Content-Type: application/json' \
  -d '{
    "model_id": "Qwen3",
    "model_path": "/var/lib/llamapi/models/rkllm/rk3588/qwen3-4b"
  }'

Call the chat API:

curl -s http://127.0.0.1:9265/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen3",
    "messages": [
      { "role": "user", "content": "Hello" }
    ]
  }'

Conventions

Item	Description
Model ID	`model` or `model_id` must identify a loaded model
Streaming	Chat with `"stream": true` returns SSE and ends with `data: [DONE]`
Multimodal input	The protocol accepts `text`, `image_url`, and `input_audio`
Error response	Regular application errors use OpenAI-style `{ "error": ... }` JSON
SSE error	A mid-stream failure uses `event: error`

Discover models and platforms through /v1/models and /v1/platforms. Do not hard-code runtime model IDs, platform names, chip types, or model paths.

Value Conventions

Closed Values

Field	Values	Description
`messages[].role`	`system`, `user`, `assistant`, `tool`	Chat message roles
`messages[].content[].type`	`text`, `image_url`, `input_audio`	Supported content parts
Data URL image format	`png`, `jpeg`, `jpg`, `webp`	Does not apply to local paths
`input_audio.format`	`wav`, `mp3`	A model may support only a subset
`encoding_format`	`float`, `base64`	Embedding output format
`model_kind`	`chat`, `embedding`	Model capability
`error.type`	`invalid_request_error`, `rate_limit_exceeded`, `server_error`	Error category

Known but Extensible Response Values

Field	Known values	Client guidance
`choices[].finish_reason`	`stop`, `length`, `tool_calls`	Accept future string values
`object`	`chat.completion`, `chat.completion.chunk`, `list`, `embedding`, `model`	Process according to endpoint and structure
`error.code`	See Errors	Treat unknown values as generic errors

Open Values

Do not treat these fields as fixed enumerations:

Field	Description
`model`, `model_id`	Defined when a model is loaded
`id`	Response ID generated by the `llamapi-server`
`platform`	Discovered from the platform API
`owned_by`	Currently formatted as `llamapi/{platform}`; clients should not rely on it
`model_path`	`llamapi-server` filesystem path
`detected_chips[].chip_type`	Chip type detected on the current host
`message`	Human-readable text; do not parse it for program logic
`tool_call_id`, `tool_calls[].id`	Defined by model output or request context
`tools[].type`	`function` is recommended, but the `llamapi-server` parses a string
`tools[].function.name`	Defined by the client

Endpoints

Method	Path	Type	Description
`POST`	`/v1/chat/completions`	OpenAI-compatible	Chat completion with JSON or SSE
`POST`	`/v1/embeddings`	OpenAI-compatible	Text vectors for single or batched input
`GET`	`/v1/models`	OpenAI-compatible	List loaded models
`GET`	`/v1/models/{model_id}`	OpenAI-compatible	Get one loaded model
`POST`	`/v1/models/load`	LlamaPi extension	Load a model dynamically
`POST`	`/v1/models/resize`	LlamaPi extension	Resize a model group
`POST`	`/v1/models/unload`	LlamaPi extension	Unload a model
`GET`	`/v1/platforms`	LlamaPi extension	List platforms and detected chips
`GET`	`/health`	Health	Check whether HTTP is running

Errors

Application errors use an OpenAI-style structure:

{
  "error": {
    "message": "Model 'demo' not found",
    "type": "invalid_request_error",
    "param": "model",
    "code": "model_not_found"
  }
}

HTTP status	`type`	`code`	Condition
`400`	`invalid_request_error`	`unsupported_platform`	The model platform is unsupported
`400`	`invalid_request_error`	`platform_detection_failed`	The platform cannot be detected from the model directory
`400`	`invalid_request_error`	`wrong_model_type`	Chat uses an Embedding model, or Embeddings uses a chat model
`400`	`invalid_request_error`	`unsupported_content_part_type`	An unsupported content part is present
`400`	`invalid_request_error`	`invalid_content_part`	Image, audio, or another content part is malformed
`400`	`invalid_request_error`	`context_length_exceeded`	Input exceeds the model context limit
`400`	`invalid_request_error`	`invalid_instance_count`	Instance count is `0`
`404`	`invalid_request_error`	`model_not_found`	The model is not loaded
`409`	`invalid_request_error`	`model_already_exists`	The model ID already exists
`429`	`rate_limit_exceeded`	`queue_full`	All instances and queue slots are occupied
`500`	`server_error`	`null`	Engine, configuration, or internal error

A negative instance count fails during JSON parsing.

Chat Completions

POST /v1/chat/completions

The model must have model_kind=chat. stream defaults to false.

Request Fields

Field	Type	Required	Description
`model`	string	Yes	Loaded model ID
`messages`	array	Yes	Chat messages
`stream`	boolean	No	Return SSE when `true`
`temperature`	number	No	Override temperature
`top_p`	number	No	Override top-p
`top_k`	integer	No	Override top-k
`repeat_penalty`	number	No	Override repetition penalty
`frequency_penalty`	number	No	Override frequency penalty
`presence_penalty`	number	No	Override presence penalty
`max_tokens`	integer	No	Maximum generated tokens
`max_completion_tokens`	integer	No	Equivalent to `max_tokens`; takes priority when both are present
`stop`	string array	No	Stop sequences
`tools`	array	No	OpenAI-style function tools
`tool_choice`	string	No	Parsed, but not currently used by the `llamapi-server`
`enable_thinking`	boolean	No	Override model thinking mode

Message Fields

messages[] supports:

Field	Type	Required	Description
`role`	string	Yes	`system`, `user`, `assistant`, or `tool`
`content`	string or array	No	Plain text or content parts
`tool_call_id`	string	No	Tool call ID for a tool message
`tool_calls`	array	No	Tool calls in an assistant message

Multimodal Content

When content is an array, supported parts are:

Text:

{
  "type": "text",
  "text": "Describe this image."
}

Image:

{
  "type": "image_url",
  "image_url": {
    "url": "/path/to/image.jpg"
  }
}

Image URLs can be:

A local path on the llamapi-server.
A base64 data URL for png, jpeg, jpg, or webp.

Remote http:// and https:// image URLs are not supported.

Audio:

{
  "type": "input_audio",
  "input_audio": {
    "format": "wav",
    "data": "<base64>"
  }
}

The protocol accepts wav and mp3; a specific model may support only a subset.

video, file, and unknown types return unsupported_content_part_type.

Tool Definitions

tools[] uses the OpenAI function-tool shape:

Field	Type	Required
`type`	string	Yes; `function` recommended
`function.name`	string	Yes
`function.description`	string	Yes
`function.parameters`	JSON	Yes

Non-Streaming Request

curl -s http://127.0.0.1:9265/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen3",
    "messages": [
      { "role": "user", "content": "Introduce LlamaPi in one sentence." }
    ],
    "max_tokens": 128
  }'

Non-Streaming Response

{
  "id": "chatcmpl-...",
  "object": "chat.completion",
  "created": 1710000000,
  "model": "Qwen3",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "...",
        "tool_calls": []
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 13,
    "completion_tokens": 14,
    "total_tokens": 27
  }
}

Streaming Request

curl -N http://127.0.0.1:9265/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen3",
    "messages": [
      { "role": "user", "content": "Hello" }
    ],
    "stream": true
  }'

Streaming Response

Each SSE event contains a chat.completion.chunk JSON object. After the finish chunk, the llamapi-server sends a separate usage chunk and then [DONE]:

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":1710000000,"model":"Qwen3","choices":[{"index":0,"delta":{"role":"assistant","content":"Hel"}}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":1710000000,"model":"Qwen3","choices":[{"index":0,"delta":{"content":"lo"}}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":1710000000,"model":"Qwen3","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":1710000000,"model":"Qwen3","choices":[],"usage":{"prompt_tokens":13,"completion_tokens":14,"total_tokens":27}}

data: [DONE]

A mid-stream failure sends:

event: error
data: {"error":{...}}

The stream then ends without [DONE].

Embeddings

POST /v1/embeddings

The model must have model_kind=embedding.

Embeddings Request Fields

Field	Type	Required	Description
`model`	string	Yes	Loaded Embedding model ID
`input`	string or string array	Yes	Single or batched text
`encoding_format`	string	No	`float` or `base64`; default `float`
`dimensions`	integer	No	Parsed, but the `llamapi-server` does not truncate vectors
`user`	string	No	Parsed, but not currently used

base64 encodes raw little-endian f32 bytes.

Load the Embedding Model

Use the llamapi-cli:

llamapi load bge-m3
llamapi ps

Or call the management API:

curl -s http://127.0.0.1:9265/v1/models/load \
  -H 'Content-Type: application/json' \
  -d '{
    "model_id": "bge-m3",
    "model_path": "/var/lib/llamapi/models/rknn2/rk3588/bge-m3"
  }'

Embeddings Request Example

curl -s http://127.0.0.1:9265/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "bge-m3",
    "input": ["hello", "LlamaPi"],
    "encoding_format": "float"
  }'

Embeddings Response Example

{
  "object": "list",
  "data": [
    {
      "object": "embedding",
      "embedding": [0.1, 0.2],
      "index": 0
    }
  ],
  "model": "bge-m3",
  "usage": {
    "prompt_tokens": 2,
    "total_tokens": 2
  }
}

Query Models

List Models

GET /v1/models

curl -s http://127.0.0.1:9265/v1/models

Response:

{
  "object": "list",
  "data": [
    {
      "id": "Qwen3",
      "object": "model",
      "created": 0,
      "owned_by": "llamapi/rkllm",
      "platform": "rkllm",
      "instance_count": 1,
      "model_path": "/var/lib/llamapi/models/rkllm/rk3588/qwen3-4b",
      "model_kind": "chat"
    }
  ]
}

Get One Model

GET /v1/models/{model_id}

curl -s http://127.0.0.1:9265/v1/models/Qwen3

The response fields match an item in /v1/models data[]. A missing model returns 404 model_not_found.

Load a Model

POST /v1/models/load

Load Model Request Fields

Field	Type	Required	Description
`model_id`	string	Yes	Model ID exposed to clients
`model_path`	string	Yes	Model directory on the `llamapi-server` filesystem
`instance_count`	integer	No	Target count; default `1`, minimum `1`
`request_queue_size`	integer	No	Model queue capacity; uses the `llamapi-server` setting when omitted
`default_params`	object	No	Model generation defaults

default_params supports:

temperature
top_p
top_k
repeat_penalty
frequency_penalty
presence_penalty
max_tokens
max_context_len
stop
enable_thinking

Load Model Request Example

curl -s http://127.0.0.1:9265/v1/models/load \
  -H 'Content-Type: application/json' \
  -d '{
    "model_id": "Qwen3",
    "model_path": "/var/lib/llamapi/models/rkllm/rk3588/qwen3-4b",
    "instance_count": 2,
    "default_params": {
      "temperature": 0.7,
      "max_tokens": 512
    }
  }'

Load Model Response Example

{
  "success": true,
  "message": "model 'Qwen3' loaded",
  "requested_instance_count": 2,
  "actual_instance_count": 2
}

When at least one instance loads, the API returns 200 OK:

Complete success uses model '<id>' loaded.
Partial success uses model '<id>' partially loaded.
requested_instance_count is the target.
actual_instance_count is the actual count.

Coprocessor Instance Limitation

Requesting multiple model instances on a coprocessor through /v1/models/load or /v1/models/resize may cause loading failure, chip communication failure, and abnormal service state. Keep instance_count at 1 for coprocessor models. See Coprocessor Communication Failure After Loading Multiple Model Instances for recovery.

Resize a Model

POST /v1/models/resize

Field	Type	Required	Description
`model_id`	string	Yes	Loaded model ID
`instance_count`	integer	Yes	Target count, at least `1`

curl -s http://127.0.0.1:9265/v1/models/resize \
  -H 'Content-Type: application/json' \
  -d '{
    "model_id": "Qwen3",
    "instance_count": 1
  }'

Response:

{
  "success": true,
  "message": "model 'Qwen3' resized",
  "requested_instance_count": 1,
  "actual_instance_count": 1
}

A partial expansion can still return 200 OK with model '<id>' partially resized.

Unload a Model

POST /v1/models/unload

Field	Type	Required	Description
`model_id`	string	Yes	Loaded model ID

curl -s http://127.0.0.1:9265/v1/models/unload \
  -H 'Content-Type: application/json' \
  -d '{ "model_id": "Qwen3" }'

Response:

{
  "success": true,
  "message": "model 'Qwen3' unloaded"
}

Query Platforms

GET /v1/platforms

Returns registered chat and Embedding platforms and chips detected on the current host.

curl -s http://127.0.0.1:9265/v1/platforms

Example response:

{
  "platforms": [
    {
      "id": "rknn3",
      "display_name": "RKNN3",
      "available": true,
      "detected_chips": [
        {
          "chip_type": "RK1828",
          "count": 2
        }
      ]
    }
  ]
}

detected_chips groups devices by chip type. It is an empty array when no chip is detected.

Health

GET /health

curl -s http://127.0.0.1:9265/health

Response:

ok

Health only confirms that the HTTP service is running. Confirm model availability with /v1/models and an inference request.

Overview​

Quick Example​

Conventions​

Value Conventions​

Closed Values​

Known but Extensible Response Values​

Open Values​

Endpoints​

Errors​

Chat Completions​

Request Fields​

Message Fields​

Multimodal Content​

Tool Definitions​

Non-Streaming Request​

Non-Streaming Response​

Streaming Request​

Streaming Response​

Embeddings​

Embeddings Request Fields​

Load the Embedding Model​

Embeddings Request Example​

Embeddings Response Example​

Query Models​

List Models​

Get One Model​

Load a Model​

Load Model Request Fields​

Load Model Request Example​

Load Model Response Example​

Coprocessor Instance Limitation​

Resize a Model​

Unload a Model​

Query Platforms​

Health​

Overview

Quick Example

Conventions

Value Conventions

Closed Values

Known but Extensible Response Values

Open Values

Endpoints

Errors

Chat Completions

Request Fields

Message Fields

Multimodal Content

Tool Definitions

Non-Streaming Request

Non-Streaming Response

Streaming Request

Streaming Response

Embeddings

Embeddings Request Fields

Load the Embedding Model

Embeddings Request Example

Embeddings Response Example

Query Models

List Models

Get One Model

Load a Model

Load Model Request Fields

Load Model Request Example

Load Model Response Example

Coprocessor Instance Limitation

Resize a Model

Unload a Model

Query Platforms

Health