apple-intelligence_skill

Overview

This skill integrates your app with iOS Visual Intelligence so users can find app content by pointing the camera at objects or using screenshots. It wires your searchable app entities to the system, supplies display representations, and enables deep links so users can open matched content in your app.

How this skill works

The integration implements an IntentValueQuery that receives a SemanticContentDescriptor containing classification labels and an optional pixel buffer. Your query uses labels and/or image data to search your catalog, returns AppEntity instances with DisplayRepresentation, and provides appLinkURL values for deep linking. The system shows results in the Visual Intelligence UI and can open your app for more results.

When to use it

• Add camera-based visual search to let users discover app content by pointing their device.
• Surface app content in system visual search results and let users open items directly.
• Handle visual intelligence queries for objects, screenshots, or other camera input.
• Provide quick results with a 'More Results' action to open your app for full lists.
• Support both label-based and image-based matching (ML models or tag lookups).

Best practices

• Return results quickly (aim < 1 second) and limit the initial set (10–20 items).
• Use labels first for fast matching; use pixelBuffer processing only when needed.
• Prioritize exact, high-confidence matches and filter low-confidence items.
• Provide clear DisplayRepresentation: concise title, useful subtitle, and thumbnail.
• Implement appLinkURL and handle onOpenURL to navigate directly to content.
• Localize titles/subtitles and cache search indexes for fast lookups.

Example use cases

• E-commerce app: point at a product to show matching catalog items and pricing.
• Recipe app: point at ingredients to surface relevant recipes with prep time and image.
• Retail app: scan store shelves to find matching SKUs, availability, and deep links to buy.
• Directory app: identify landmarks or storefronts and link to detail pages in-app.
• Media app: point at album art or posters to surface songs, movies, or related content.