Profile
Back to NewsBack
GitHub Trending 14 min
Reader Mode
skye-harris/llm_intents: Exposes internet search tools for use by LLM-backed Assist in Home Assistant

skye-harris/llm_intents: Exposes internet search tools for use by LLM-backed Assist in Home Assistant

16 hours ago

Tools for Assist _(Custom Integration for Home Assistant)_

Additional tools for LLM-backed Assist for Home Assistant:
  • Web Search powered by your choice of _Brave_, _Tavily_, _Keenable_ or _SearXNG_
  • Location Search powered by Google Places
  • Routes & Travel Time powered by Google Routes
  • Wikipedia
  • Weather Forecast
  • YouTube Search and Playback
  • Basic Utilities — Calculator, Unit Converter, and Date Information
  • Entity History — Part of the Home control domain
Each tool is optional and configurable via the integrations UI. Some tools require API keys, but are usable on free tiers. A caching layer is utilised in order to reduce both API usage and latency on repeated requests for the same information within a 2-hour period. Additionally, a customisable clone of Home Assistants inbuilt Assist tooling API can be enabled and used with your Conversation Agents:
  • Edit the hidden prompt that the Assist API injects into your system prompt, if this is not to your liking, or conflicts with instructions that you have provided in your own prompt.
  • Disable any of the default Assist API tools that you don't want your Conversation Agents to have access to.
  • This is named Home Control in the list of tools.
---

Installation

Tools for Assist is available on the Home Assistant Marketplace: image</a>
Manual Install custom_components folder in your config directory.
  • Restart the Home Assistant.

Integration Configuration

After installation, configure the integration through Home Assistant's UI:
  1. Go to Settings → Devices & Services.
  2. Click Add Integration.
  3. Search for Tools for Assist.
  4. Follow the setup wizard to configure your desired services.

Conversation Agent Configuration

Once the integration is installed and configured, you will need to enable the desired services within your Conversation Agent entities. For the Ollama and OpenAI Conversation integrations, this can be found within your Conversation Agent configuration options, beneath the Control Home Assistant heading, and enabling the services desired for the Agent:
  • Search Services
  • Weather Forecast
  • Media Services
  • Basic Utilities

🔍 Brave Web Search

Uses the Brave Web Search API to return summarized, snippet-rich results. ##### Requirements * Brave provide $5 of free credit per month, equal to 1000 searches.

Configuration Steps

  1. Select "Brave" as the search provider during setup.
  2. Enter your Brave Search API key.
  3. Configure optional settings like number of results, location preferences.

Options

| Setting | Required | Default | Description | |---------------------------|----------|---------|----------------------------------------------------------------------| | API Key | ✅ | — | Brave Search API key | | Number of Results | ✅ | 2 | Number of results to provide to the LLM | | Max Snippets per Result | ✅ | 2 | Maximum number of content snippets to provide to the LLM, per result | | Country Code | ❌ | — | ISO country code to bias results | | Latitude | ❌ | — | Optional latitude for local result relevance (recommended) | | Longitude | ❌ | — | Optional longitude for local result relevance (recommended) | | Timezone | ❌ | — | Optional TZ timezone identifier for local result relevance | | Post Code | ❌ | — | Optional post code for local result relevance | ---

🔍 Brave LLM Context Search

Uses the Brave LLM Context Search API to return pre-extracted web context optimised for AI Agents. ##### Requirements * Brave provide $5 of free credit per month, equal to 1000 searches. * This does not work with the now-deprecated Data for AI API keys

Configuration Steps

  1. Select "Brave LLM Context" as the search provider during setup.
  2. Enter your Brave Search API key.
  3. Configure optional settings like number of results, location preferences.

Options

| Setting | Required | Default | Description | |---------------------------|----------|------------|----------------------------------------------------------------------| | API Key | ✅ | — | Brave Search API key | | Number of Results | ✅ | 2 | Number of results to provide to the LLM | | Max Snippets per Result | ✅ | 2 | Maximum number of content snippets to provide to the LLM, per result | | Max Tokens per Result | ✅ | 1024 | Set a target token limit for result content | | Context Threshold Mode | ❌ | Balanced | Relevance threshold for including content | | Country Code | ❌ | — | ISO country code to bias results | | Latitude | ❌ | — | Optional latitude for local result relevance (recommended) | | Longitude | ❌ | — | Optional longitude for local result relevance (recommended) | | Timezone | ❌ | — | Optional TZ timezone identifier for local result relevance | | Post Code | ❌ | — | Optional post code for local result relevance | ---

🔍 Tavily Web Search

Uses the Tavily Search API to return concise, relevant results, with an optional generated answer. ##### Requirements * Tavily provides 1,000 free API credits per month, no card required. * Basic, Fast and Ultra-fast searches cost one credit each, an Advanced search costs two.

Configuration Steps

  1. Select "Tavily" as the search provider during setup.
  2. Enter your Tavily API key.
  3. Configure the search depth, the number of results, chunks per result, whether to include a generated answer or full page content, a cap on the characters returned per result, and optionally a country to boost results from.

Options

| Setting | Required | Default | Description | |----------------------------|----------|---------|------------------------------------------------------------------------------------| | API Key | ✅ | — | Tavily API key | | Search Depth | ❌ | Basic | Costs are per search. Basic, Fast and Ultra-fast cost 1 credit, Advanced costs 2. Fast and Ultra-fast are beta options that trade relevance for speed | | Number of Results | ✅ | 2 | Number of results to provide to the LLM | | Chunks Per Result | ✅ | 1 | How many content chunks Tavily returns per result. Honored on Basic, Advanced and Fast. Ultra-fast returns a single summary per result | | Include Generated Answer | ❌ | false | Also return a short Tavily-generated answer alongside the search results | | Include Full Page Content| ❌ | false | Return the full extracted page content instead of Tavily's snippets. The content arrives as markdown, and the repeated whitespace is collapsed before it is returned. Full pages are large, so use Maximum Characters Per Result to cap them | | Maximum Characters Per Result | ❌ | 0 | Caps the characters returned per result. Tavily has no option for this, so the integration applies the cap after the search. 0 means no cap | | Country | ❌ | — | Supported country name, chosen from a list. Boosts results from it. Sent on a general web search only, so a news or finance search does not use it, and it is not sent with the fast or ultra-fast search depths |

LLM-provided arguments

| Argument | Required | Description | |--------------|----------|----------------------------------------------------------------------| | query | ✅ | The query to search for | | topic | ❌ | general, news or finance. news adds published dates | | time_range | ❌ | day, week, month or year recency filter | ---

🔍 SearXNG Web Search

Uses a self-hosted SearXNG search service to return summarized results. ##### Requirements
  • Requires a SearXNG server instance, with JSON responses enabled (https://github.com/searxng/searxng-docker).

Configuration Steps

  1. Select "SearXNG" as the search provider during setup.
  2. Configure your server URL and maximum search results to provide to the LLM.
1. Server should be in the format: protocol://host:port, eg: http://192.168.0.1:8080

Options

| Setting | Required | Default | Description | |---------------------|----------|---------|-----------------------------------------| | Number of Results | ✅ | 2 | Number of results to provide to the LLM | ---

🔍 Keenable Web Search

Uses the Keenable Search API to return titles, URLs and snippets. It works without an API key. ##### Requirements
  • No API key is needed. Without a key, requests go to Keenable's public endpoint, which has lower request limits.
* Keyless requests are logged by Keenable with the query and the caller's IP address. Add an API key if you don't want that.

Configuration Steps

  1. Select "Keenable" as the search provider during setup.
  2. Optionally enter a Keenable API key, and set the number of results to provide to the LLM.

Options

| Setting | Required | Default | Description | |---------------------|----------|---------|---------------------------------------------------------------------| | API Key | ❌ | — | Keenable API key. Leave empty to use the keyless public endpoint | | Number of Results | ✅ | 5 | Number of results to provide to the LLM | ---

📍 Google Places

Searches for locations, businesses, or points of interest using the Google Places API. Search results include the location name, address, rating score, current open state, and when it next opens/closes.

Requirements

Configuration Steps

  1. Select "Google Places" during setup.
  2. Enter your Google Places API key.
  3. Configure number of results to return.

Options

| Setting | Required | Default | Description | |---------------------|----------|------------|-----------------------------------------------------------------------------| | API Key | ✅ | — | Google Places API key | | Number of Results | ✅ | 2 | Number of location results to return | | Latitude | ❌ | — | Your locations latitude, if you wish to use location biasing (recommended) | | Longitude | ❌ | — | Your locations longitude, if you wish to use location biasing (recommended) | | Radius | ❌ | 5 | The radius around your location for location biased results (in kilometres) | | Rank Preference | ❌ | Distance | The ranking preference for search results from Google Places | ---

🗺️ Google Routes

Computes travel distance and duration from a configured home address to any destination, using the Google Routes API. The duration includes live traffic for driving and motorcycle/scooter trips, making it suitable for questions like:
  • "how far of a drive is it to the airport"
  • "how long would it take to drive to X"
  • "when should I leave to get to X by 6pm" — the LLM uses the returned duration to work backwards from the target arrival time.
When a future departure time is supplied, the API returns predicted traffic for that time window rather than current conditions.

Requirements

  • Requires a Google API key with both the Routes API and the Places API (New) enabled. The Places API is used to improve destination accuracy; you do not need to enable the Google Places tool in this integration.
  • The same Google API key can be shared with the Google Places and YouTube tools if those are configured.

Configuration Steps

  1. Enable "Google Routes" during setup.
  2. Enter your Google API key (the same key used for Google Places / YouTube if configured).
  3. Enter your home address — this is used as the starting point for all route lookups.
  4. Choose a default travel mode — used whenever the LLM doesn't specify one (e.g. select WALK if you don't drive).

LLM-provided arguments

| Argument | Required | Default | Description | |------------------|----------|--------------------|------------------------------------------------------------------------| | destination | ✅ | — | Destination address, place name, or business name in the user's words. | | departure_time | ❌ | now | ISO 8601 departure time. Omit for an immediate departure. | | mode | ❌ | configured default | One of DRIVE, WALK, BICYCLE, TRANSIT, TWO_WHEELER. |

Options

| Setting | Required | Default | Description | |------------------------|----------|---------|-------------------------------------------------------------------------| | API Key | ✅ | — | Google API key with both Routes API and Places API (New) enabled | | Home Address | ✅ | — | Starting point for route calculations (e.g. 123 Main St, Springfield) | | Default Travel Mode | ✅ | DRIVE | Travel mode used when the LLM doesn't specify one | Distance is reported in miles or kilometres based on your Home Assistant unit system. ---

📚 Wikipedia

Looks up Wikipedia articles and returns summaries of the top results.

Requirements

  • No API key required.
  • Uses the public Wikipedia search and summary APIs.

Configuration Steps

  1. Select "Wikipedia" during setup.
  2. Configure number of article summaries to return (no API key required).

Options

| Setting | Required | Default | Description | |---------------------|----------|---------|---------------------------------------| | Number of Results | ✅ | 1 | Number of article summaries to return | ---

⛅ Weather Forecast

Rather than accessing the internet directly for weather information, this tool utilises your existing Home Assistant weather integration and makes the forecast data accessible to your LLM in an intelligent manner. At a minimum, this tool requires a weather entity that provides either daily or twice-daily forecast data. It is recommended, though optional, to also specify a weather entity that provides hourly weather data. For cases where a specific days weather is requested (eg: today, tomorrow, wednesday), the hourly data will be provided if available. If data for the week is requested, no hourly forecast entity is set, or the hourly forecast does not contain data for the requested day, the daily weather data will be used instead.

Requirements

  • An existing weather forecast integration configured within Home Assistant.

Configuration Steps

  1. Select "Weather Forecast" during setup.
  2. Select the weather entity that provides daily forecast information.
  3. Optionally, select the weather entity that provides hourly forecast information.
  4. Optionally, select a local temperature sensor entity to display current temperature for today's hourly forecast.

Options

| Setting | Required | Description | |------------------------------|----------|-------------------------------------------------------------------------------------------------------------| | Daily Weather Entity | ✅ | The weather entity to use for daily weather forecast data | | Hourly Weather Entity | ❌ | The weather entity to use for hourly weather forecast data | | Current Temperature Sensor | ❌ | Optional local sensor entity to provide current temperature when requesting today's hourly weather forecast |

🎥 YouTube Search + Playback

Searches YouTube for videos and enables playback on compatible media players. The tool combines YouTube search capabilities with intelligent media player detection to provide a seamless video playback experience. Search results include video titles, URLs, channel names, descriptions, and publication dates. When a user requests to play a video, the search results are automatically used with the playback tool to start video on the appropriate device.

Requirements

  • Requires a Google API key with the YouTube Data API v3 enabled.
  • The same Google API key can be shared with the Google Places tool if both are configured.
  • Google provides a free tier with generous quotas for YouTube Data API v3.

Configuration Steps

  1. Select "YouTube" during setup.
  2. Enter your Google API key (the same key used for Google Places if configured).
  3. Configure the number of search results to return (default: 1).

Playback Compatibility

The YouTube tool works seamlessly with Home Assistant media players that support video playback. Video-capable devices are automatically detected based on their device_class attribute: Supported Device Classes:
  • tv - Television devices (e.g., smart TVs, Android TV boxes)
  • receiver - AV receivers with video output
Not Supported:
  • speaker - Audio-only devices are automatically excluded
  • Media players without a device_class set - These must be explicitly configured
Playback Targeting: Videos can be played by specifying:
  • Entity ID - Direct entity selection (e.g., media_player.living_room_tv)
  • Area - Play on all video-capable devices in an area (e.g., "Living Room")
  • Device ID - Target a specific device by its device registry ID
The tool automatically filters media players to only include video-capable devices when using area-based targeting, ensuring videos are only sent to devices that can display them.

How It Works

  1. Search: When a user requests a YouTube video, the search_youtube tool queries YouTube's API and returns matching videos with metadata.
  2. Playback: If the user wants to play a video, the play_video tool uses the video URL from search results and calls Home Assistant's media_player.play_media service on the target device(s).
  3. Caching: Search results are cached for 2 hours to reduce API usage and improve response times for repeated queries.

Options

| Setting | Required | Default | Description | |---------------------|----------|---------|--------------------------------------------------------------------------| | API Key | ✅ | — | Google API key with YouTube Data API v3 enabled | | Number of Results | ✅ | 1 | Number of video results to return (1-25). Use more for multiple options. | ---

🧮 Basic Utilities

A set of always-available utility tools.

🔢 Calculator

Evaluate mathematical expressions and return the result. Calculator supports mathematical expression evaluation, along with calculating the minimum, maximum, and average of a series of values.

📏 Unit Converter

Converts quantities between common volume, weight, length, and temperature units. Supports fractional amounts such as 1/8 or 1 1/2. Supported units:
  • Volume: cup, tablespoon, teaspoon, ml, dl, liter, gallon, fluid_ounce, pint, quart
  • Weight: gram, kilogram, milligram, pound, ounce, stone
  • Length: millimeter, centimeter, meter, kilometer, inch, foot, yard, mile
  • Temperature: celsius, fahrenheit
Parameters: | Parameter | Required | Description | |-------------|----------|--------------------------------------------------------------------------| | amount | ✅ | The quantity to convert (number or fraction, e.g. 1/8, 2.5, 1 1/2) | | from_unit | ✅ | Unit to convert from | | to_unit | ✅ | Unit to convert to |

📅 Calendar Day Information

Returns the day of the week and a formatted date string for a given day, month, and optional year. Useful for answering questions like "What day is March 15?" or planning events. Parameters: | Parameter | Required | Description | |-----------|----------|------------------------------------------------| | day | ✅ | Day of the month (1–31) | | month | ✅ | Month (1–12) | | year | ❌ | Year (1900–2100, defaults to the current year) | ---

🏡 Home Control API

The Home Control API is a direct clone of Home Assistants inbuilt Assist API, but with some customisation exposed. The Assist API, much like this and other tool API providers, injects some instructional directives into the system prompt on how to use the tools provided to them. In some situations, you may find that these conflict with instructions that you have provided your agent directly via your own system prompt. The Home Control API exposes Assists hidden prompt as a Jinja2 template that can be freely modified. Additionally, the inbuilt Assist tools can be disabled on a per-tool basis, in case there are tools that you do not wish your agent to have access to, but still see the exposed entities related to them. As the Home Control API is a direct clone of the Assist API, it is strongly recommended to only use either Assist OR Home Control in your Conversation Agents, and not both together.

Additional Features

  • Entity History Tool — Retrieves the past (historic) states of an entity from the Home Assistant recorder database.
This tool returns summary statistics (min, max, average for numeric entities) along with a downsampled set of significant state changes, making it efficient for the LLM to answer questions like _"What time did [name] get home last night?"_ ---

Contributors

I would like to thank the following people for their contributions.

JonahMMay

NickM-27

HenrikJohnson

QuadroKnoX

Thyraz

ianlunam

professor-sagittarius

balloob

Nick2253

KGKallasmaa
---

Support Development

If you find this integration useful and would like to support development, please consider buying me a coffee. " style="max-width:100%;">
Chat with me