Profile
Back to NewsBack
GitHub Trending 8 min
Reader Mode
elevenlabs/elevenlabs-swift-sdk: ElevenLabs Conversational AI Swift SDK

elevenlabs/elevenlabs-swift-sdk: ElevenLabs Conversational AI Swift SDK

ElevenAgents Swift SDK

!SwiftSDK

A Swift SDK for integrating ElevenAgents capabilities into your iOS and macOS applications. Built on top of LiveKit WebRTC for real-time audio streaming and communication.


Why ElevenLabs Swift SDK?

  • Ultra-Low Latency: Built on LiveKit WebRTC for high-performance, real-time audio streaming.
  • Human-Like Interaction: Seamlessly handle interruptions and natural speech patterns.
  • Drop-In Widget: Ship a complete agent chat UI — voice calls, text input, live transcript, audio-reactive orb — with one view.
  • Dev-First API: Fully supports Swift Concurrency (Async/Await), SwiftUI observation, and Swift 6 strict concurrency.
  • Extensible: Native support for Client Tools and MCP (Model Context Protocol).
  • Native Performance: Optimized for iOS and macOS, ensuring buttery-smooth UI.

Quick Start

1. Installation

Add the package via Swift Package Manager:

dependencies: [
    .package(url: "https://github.com/elevenlabs/elevenlabs-swift-sdk.git", from: "4.0.0")
]

The package provides two libraries:

  • ElevenLabs — the core SDK: start conversations, observe state, build your own UI.
  • ElevenLabsWidget — an optional drop-in SwiftUI chat widget built on the core SDK (iOS 16+). Experimental, see below.
[!WARNING]
ElevenLabsWidget is experimental. Semantic versioning does not cover its API, and a minor or patch release can break it. To use it, import it as an experimental SPI: @_spi(Experimental) import ElevenLabsWidget. The core ElevenLabs library follows semantic versioning.

2. Requirements & Permissions

  • Platforms: iOS 13.0+ · macOS 10.15+ · macCatalyst 14.0+ · visionOS 1.0+ · tvOS 17.0+ (ChatWidget requires iOS 16+)
  • Tooling: Xcode 15.0+ · Swift 5.9+ (Swift 6 supported)
  • Privacy: Add NSMicrophoneUsageDescription to your Info.plist. WebRTC connection discovery may also trigger iOS Local Network permission; add NSLocalNetworkUsageDescription if needed, or use the .relayOnly network configuration to restrict ICE to relay candidates.

3. Fastest Path: the Drop-In Widget

Overlay ChatWidget on your UI and you're done — it renders a floating launcher that opens a chat drawer with a voice call, text composer, and live transcript:

@_spi(Experimental) import ElevenLabsWidget
import SwiftUI

struct ContentView: View { var body: some View { ZStack { YourAppContent() ChatWidget(mode: .voiceAndText(.publicAgent(id: "your-agent-id"))) } } }

Pick how users talk to the agent with WidgetConversationMode:

  • .voiceAndText(...) — a voice call the user can also type into.
  • .voiceOnly(...) — a voice call with no composer.
  • .textOnly(...) — typed messages only, no audio.
  • .voiceOrTextOnly(voice:textOnly:) — the user picks per session: the call button starts voice, typing starts text.
To observe or drive the widget from your app, pass a ChatWidgetController:
@StateObject private var chat = ChatWidgetController()

ChatWidget(mode: .voiceAndText(.publicAgent(id: "your-agent-id")), controller: chat)

// Anywhere in the host app: chat.open() try await chat.startConversation() await chat.endConversation()

Appearance and behavior are configurable via ChatWidgetConfig (theme, strings, backdrop, mic control), and you can replace the launcher with your own view. See Examples/DemoApp for a runnable showcase of every mode.

4. Build Your Own UI: ConversationClient

For full control, use the core SDK. Create one ConversationClient and hold it for the lifetime of your screen — it exposes conversation state as @Published properties and stays reusable across conversations, so your UI binds once:

import ElevenLabs
import SwiftUI

struct ChatView: View { @StateObject private var client = ConversationClient()

var body: some View { VStack(spacing: 20) { // Connection state Group { switch client.state { case .idle: Text("Status: idle") case .connecting: Text("Status: connecting…") case .connected(let info): Text("Connected to \(info.agentId)") case .ended: Text("Status: ended") case .error: Text("Status: error") } } .font(.caption).foregroundColor(.secondary)

// Live transcript — streamed agent replies update in place ScrollView { ForEach(client.chatHistory.compactMap(\.message)) { message in Text("\(message.role == .user ? "You" : "Agent"): \(message.content)") .frame(maxWidth: .infinity, alignment: .leading) } }

if client.state.isConnected { Button("End Conversation", role: .destructive) { Task { await client.endConversation() } } } else { Button("Start Voice Chat") { Task { do { _ = try await client.startVoiceConversation(.publicAgent(id: "your-agent-id")) } catch { print("Failed to start: \(error)") } } } .buttonStyle(.borderedProminent) } } .padding() } }

chatHistory contains messages and tool calls (item.message / item.toolCall). Starting a conversation returns a ConversationStartResult with the call info and startup timing metrics — ignore it with _ = if you don't need it.

Cancel connecting

If the user leaves the screen before the connection completes, cancel the task running the start — startup is torn down cleanly at any stage:

let startTask = Task {
    _ = try await client.startVoiceConversation(.publicAgent(id: "your-agent-id"))
}

// Later: startTask.cancel() // …or equivalently: await client.endConversation()


Authentication Modes

Voice and text-only conversations take different credentials, expressed as ConversationAuth.Voice and ConversationAuth.TextOnly.

Public Agents

Perfect for prototyping. Connect directly using your Agent ID from the ElevenLabs dashboard:

_ = try await client.startVoiceConversation(.publicAgent(id: "your-agent-id"))

Private Agents (Production Ready)

For private agents, your backend generates short-lived credentials using your API key: a conversation token for voice, or a signed WebSocket URL for text-only.

[!CAUTION]
Security First: Never store your ElevenLabs API Key directly in your mobile app. Always use a backend proxy.
// Voice: mint a conversation token on your backend.
// The closure is called once per start, so tokens are always fresh.
_ = try await client.startVoiceConversation(
    .conversationToken { try await myBackend.fetchConversationToken() }
)

// Text-only: mint a signed WebSocket URL on your backend _ = try await client.startTextOnlyConversation( .signedWebSocketURL { try await myBackend.fetchSignedWebSocketURL() } )

If you already have a credential in hand, pass it directly: .conversationToken(token) or .signedWebSocketURL(url).


Text-Only Conversations

Skip the microphone entirely and run a chat-style conversation over WebSocket. The client API is identical — sendMessage, chatHistory, endConversation all work the same:

_ = try await client.startTextOnlyConversation(.publicAgent(id: "your-agent-id"))
try await client.sendMessage("Hello!")

Empower Your Agent with Tools

You can allow your agent to perform actions in your app (like opening a screen or fetching local data) using Client Tools. Set a handler for each tool before you start a conversation. A handler runs once for each call of its tool, and the SDK sends what it returns to the agent:

let client = ConversationClient()
client.callbacks.clientTools = [
    "get_weather": { call in
        let city = try call.getParameters()["city"] as? String ?? ""
        return .result(await weather(for: city))
    },
    "get_order": { call in
        let id = try call.getParameters()["order_id"] as? String ?? ""
        return try .result(json: await orders.order(id: id)) // any Encodable
    },
    "confirm_cancellation": { _ in .pending } // the app answers it after the user decides
]

A thrown error reaches the agent as an error result. Throw ClientToolError to set the message and an errorType, such as .userRejected.

If a handler returns .pending, the call stays in client.pendingToolCalls until you answer it with client.sendToolResult, from anywhere in your app.

A call for any other tool name goes to client.callbacks.onUnhandledClientToolCall. It has the same type as a handler in clientTools, so it returns a ClientToolReply or throws. Without it, the call gets an error result, so the agent doesn't wait for it. To route every call yourself, leave clientTools empty and set only onUnhandledClientToolCall. For example, { _ in .pending } leaves every call in pendingToolCalls for you to answer.

If you use ChatWidget, pass a handler instead and the widget wires this up for you:

ChatWidget(
    mode: .voiceAndText(.publicAgent(id: "your-agent-id")),
    onClientToolCall: { call in
        let result = await myAppAction((try? call.getParameters()) ?? [:])
        return .init(toolCallId: call.toolCallId, result: result)
    }
)
[!TIP]
Check out the Advanced Usage Guide for full MCP tool integration and complex scenarios.

Configuration & Tuning

Callbacks

Want to handle events without Combine? Pass ConversationCallbacks when creating the client — they apply to every conversation it starts:

let callbacks = ConversationCallbacks(
    onDisconnect: { reason in print("Ended: \(reason)") },
    onAgentResponse: { text, _ in print("Agent said: \(text)") },
    onUserTranscript: { text, _ in print("User said: \(text)") },
    onVadScore: { score in print("Voice intensity: \(score)") }
)

let client = ConversationClient(callbacks: callbacks)

To change a callback later, set its field on client.callbacks, for example client.callbacks.onAgentResponse = { ... }. The change applies at once, also during a conversation.

Muting

try await client.setMicMuted(true) // mute the user's microphone
client.setAgentMuted(true)         // silence the agent's voice

Both states persist across conversations started by the same client.

Raw Audio Access

Tap decoded PCM audio for visualizers, recording, or analysis. Observers are durable — they re-attach to every conversation the client starts:

final class Visualizer: ConversationAudioObserver {
    func didReceive(_ buffer: AVAudioPCMBuffer) {
        // Called on the audio thread — copy what you need, then hop off.
    }
}

client.addAgentAudioObserver(visualizer) // agent's voice client.addMicAudioObserver(visualizer) // local microphone

Logging

Set verbosity per client:

let client = ConversationClient(logLevel: .info)

Custom Endpoints

Point the HTTP API base or either WebSocket endpoint at a proxy, regional host, or staging:

let config = ConversationConfig(
    endpoints: Endpoints(apiBase: URL(string: "https://your-proxy.example.com")!)
)

_ = try await client.startVoiceConversation(.publicAgent(id: "your-agent-id"), config: config)

ConversationConfig also covers agent/TTS overrides, dynamic variables, startup timeouts, microphone pipeline behavior, and WebRTC connection strategy.


Architecture at a Glance

The SDK handles all the heavy lifting of WebRTC coordination and protocol parsing, exposing a simple, thread-safe interface:

graph TD
    App[Your App] --> Widget[ChatWidget - optional drop-in UI]
    App --> Client[ConversationClient]
    Widget --> Client
    Client --> Session[Conversation session]
    Session --> WebRTC[WebRTC Connection Manager]
    Session --> WebSocket[WebSocket Connection Manager]
    WebRTC --> LiveKit[LiveKit SDK]

Example App

Examples/DemoApp is a minimal host app for ChatWidget: switch between all conversation modes, try public-agent and backend-credential auth, and drive the widget through ChatWidgetController.


Contributing

We love contributions!

  • Tests: Ensure all tests pass with swift test.
  • Patterns: Adhere to Swift Concurrency best practices (Actors/MainActor).
Explore our Usage Documentation for more depth.

Releasing

To release a new version of the SDK, use the Claude Code release command:

claude /release <version>

This will update version strings, lint, run checks, open a PR, and (after merge) tag the release to make it available via Swift Package Manager.

Chat with me