Attachments And Multimodal Input

KsADK normalizes text, image, file, and uploaded attachment inputs before they reach a framework runner. Local UI, Responses API, Chat Completions, and ADK-style parts all share one execution path.

Recommended Practices

• Route image analysis to a vision-capable model (see Default Multimodal Model).
• Do not paste large files directly into the prompt; extract a summary or structured content first.
• When a tool needs a file, use the workspace or an attachment reference; do not guess browser upload paths.
• Do not write attachment bytes into long-term memory.

Input Shapes

Common protocol shapes:

{"type": "input_text", "text": "..."}

{"type": "input_image", "image_url": "data:image/png;base64,..."}

{"type": "input_file", "filename": "note.txt", "file_data": "data:text/plain;base64,..."}

Business agents should handle missing, expired, or unsupported attachment references and return a clear error.

Normalization Flow

The runner receives both canonical content (input_content, input_messages) and structured attachment results (attachment_results, current_attachment_results).

Protocol Mapping

KsADK accepts several client representations, then maps them into the same internal attachment path:

Application code should use the normalized runner payload, not branch on which public protocol the client used.

Unified URI Resolution

Benefits of unified resolution:

• After a session switch or browser refresh, the runner and workspace preview can still read real file content through file_uri.
• Hosted attachments are materialized to the local files/ directory on first resolution; subsequent reads use the local cache instead of re-downloading.
• Application code does not need to distinguish local from hosted: hand the file_uri to the runtime for resolution.

Responses Examples

Text plus image:

{
  "model": "my-agent",
  "input": [
    {
      "role": "user",
      "content": [
        {"type": "input_text", "text": "Describe this image."},
        {
          "type": "input_image",
          "image_url": "data:image/png;base64,..."
        }
      ]
    }
  ],
  "stream": false
}

Text plus file:

{
  "model": "my-agent",
  "input": [
    {
      "role": "user",
      "content": [
        {"type": "input_text", "text": "Summarize this note."},
        {
          "type": "input_file",
          "filename": "note.txt",
          "file_data": "data:text/plain;base64,..."
        }
      ]
    }
  ]
}

Upload Then Reference

The local Web UI can upload a file and receive a URI:

{
  "FileData": {
    "fileUri": "ksadk-upload://abc123",
    "displayName": "contract.pdf",
    "mimeType": "application/pdf",
    "sizeBytes": 102400
  }
}

A later request can reference it:

{
  "type": "input_file",
  "filename": "contract.pdf",
  "file_url": "ksadk-upload://abc123"
}

Attachment Results

Each extracted attachment result can include:

Text and document content may be truncated. Business code should handle missing or partial extraction results.

Inline Data Versus References

KsADK supports two attachment transport styles:

Inline data is self-contained but increases request size. References keep the request small, but they are only meaningful to the runtime that created the upload reference. Public docs should not present ksadk-upload://... as a durable URL.

AttachmentContent Endpoint

The hosted runtime downloads ae-upload:// attachment bytes through the KOP Action endpoint:

GET /agentengine/api/v1/AttachmentContent?FileUri=<uri>

The response is a byte stream. Content-Type reflects the real MIME type and Content-Disposition carries the original file name.

Consume Structured Attachment Fields

Business agents should not guess upload paths or parse file_uri themselves. Consume the structured attachment fields from the runner payload instead:

def ksadk_prepare_state(payload: dict, session_context: dict) -> dict:
    return {
        "messages": payload.get("input_messages", []),
        # only files uploaded in the current turn
        "files": payload.get("current_attachment_results", []),
        # historical attachment context still reachable on follow-ups
        "file_context": payload.get("attachment_results", []),
    }

• Use current_attachment_results for workflows such as "process the file I just uploaded".
• Use attachment_results for follow-ups such as "now summarize the second section"; current_attachment_results is expected to be empty on a text-only follow-up.
• When a tool needs a file handle, take the file_uri from current_attachments / attachments and hand it to the runtime for unified resolution.

Display Content Versus Runner Content

The runtime keeps separate views of an uploaded file:

This separation avoids flooding the UI transcript with extracted document text while still giving the runner enough context to answer file-related questions. If your agent needs exact structured file metadata, read attachment_results instead of parsing the display text.

Supported Extraction Paths

ZIP handling is intentionally conservative: path traversal, nested archives, executable entries, large entries, and excessive total extracted size are blocked.

ZIP And Archive Boundaries

Archive handling is optimized for safe inspection, not arbitrary extraction. The runtime may list or extract text-like entries, but it can reject:

• path traversal entries.
• nested archives.
• executable files.
• very large entries.
• archives whose total expanded size exceeds local limits.

Treat archive results as partial unless status is ok and no warnings are present. Agents should ask the user for a narrower file or a direct text export when archive extraction is incomplete.

Runner Payload Fields

Framework adapters receive these file-related fields:

For follow-up questions, attachment_results can include the most recent session attachment context even when the current turn has no new file. Use current_attachment_results when the workflow must react only to newly uploaded files.

Framework Adapter Notes

Framework adapters receive the same normalized attachment payload, but they may feed it to the framework differently:

If an application uses a custom hook, include the attachment fields you actually need:

def ksadk_prepare_state(payload: dict, session_context: dict) -> dict:
    return {
        "messages": payload.get("input_messages", []),
        "files": payload.get("current_attachment_results", []),
        "file_context": payload.get("attachment_results", []),
    }

Prefer current_attachment_results for workflows such as "process the file I just uploaded". Prefer attachment_results for follow-up questions such as "now summarize the second section".

Session Carryover

If the current user turn includes attachments, those attachments become the effective context. If a later turn has no new attachment, the runtime can restore the most recent attachment context from the same session.

Use current_attachment_results when the user just uploaded a file. Use attachment_results when follow-up questions should keep working with the prior file.

Resume After Refresh

As long as the file_uri is still resolvable (local cache or a reachable ae-upload:// backend), the runner and workspace preview keep reading real file content after a refresh or session resume. Application code needs no special handling.

Default Multimodal Model

Precedence: explicit request-level model options > explicit environment variables > policy defaults.

Safety Rules

• Validate file type and size before business processing.
• Treat uploaded file content as untrusted user input.
• Do not publish uploaded customer files, OCR text, or local file paths.
• Prefer placeholder data in docs and tests.
• Keep large binary fixtures out of the public repository unless reviewed.
• Prefer file_data or ksadk-upload:// references for local processing; do not assume the runtime will fetch arbitrary remote file_url values.
• Avoid logging extracted attachment text in CI or public issue templates.