# Meta Wearables Developer Platform v1.0 > Two product journeys available. Choose DAT SDK for native mobile Android Kotlin or iOS Swift integration with Bluetooth Low Energy sessions, MWDATCore MWDATDevice MWDATCamera MWDATMedia MWDATDisplay MWDATState MWDATLogger MWDATMock MWDATKey, device registration via Meta AI app deeplink, permissions, camera photo video streaming, microphone HFP audio, speaker playback, display rendering, IMU sensors, device discovery, telemetry, Mock Device Kit, Ray-Ban Meta Gen 1 Gen 2 Optics Display. Choose WebApps SDK for web-based HTML CSS JavaScript experiences hosted over HTTPS specifically for Meta Ray-Ban Display MRBD only, 600 by 600 fixed viewport additive waveguide display, dark backgrounds transparent light opaque, Neural Band arrow keys Enter Escape navigation, temple swipe, DeviceMotionEvent DeviceOrientationEvent navigator.geolocation localStorage sessionStorage, PNG icons, AI coding plugin, no camera microphone text input offline notifications back navigation. Filter via ?product=dat or ?product=webapps if needed, ?sdk= alias supported for backward compat. # DAT SDK v1.0 ## Section 1: DAT Guides ### Setup ## Overview The Wearables Device Access Toolkit supports iOS and Android mobile platforms, with the same OS version requirements as the Meta AI app (iOS 15.2+ and Android 10+). Xcode 14.0+ is supported for iOS. Android Studio Flamingo or newer is supported for Android. The recommended AI-assisted setup starts with two choices: your app platform and your coding tool. Choose iOS or Android, install the matching DAT coding guidance, and connect documentation search before scaffolding an integration. ## Set up AI coding tools The recommended workflow combines the [DAT GitHub plugin or repo-native skills](/docs/develop/dat/ai-assisted/) for your platform with the [Wearables MCP server](/docs/develop/dat/ai-assisted-mcp/). The plugin or skills provide platform-specific implementation and testing workflows. The MCP server provides current published documentation through `search_dat_docs`. Use the [DAT AI and MCP setup instructions](/docs/develop/dat/ai-assisted-mcp/) to configure Muse Code, Claude Code, Codex, or Cursor before asking the assistant to scaffold an integration. ## Hardware requirements Currently, the SDK supports the Ray-Ban Meta (Gen 1 and Gen 2), Ray-Ban Meta Optics, and Meta Ray-Ban Display glasses. You can test with a simulated device using [Mock Device Kit](/docs/develop/dat/mock-device-kit/), or directly with a device. Detailed version support of the Meta AI app and glasses firmware is located in the [Version Dependencies](/docs/develop/dat/version-dependencies/) page. ## Setting up your glasses 1. Ensure your versions of the Meta AI app and glasses software are in line with the version dependencies [outlined here](/docs/develop/dat/version-dependencies). Follow the instructions below to verify your current glasses version. 1. Connect your glasses to the Meta AI app. 1. Enable [developer mode](/docs/develop/dat/getting-started-toolkit#enable-developer-mode-in-the-meta-ai-app) in the Meta AI app. Developer mode allows your unpublished app to register and interact with your AI glasses without the need to submit it for publishing review. Your app appears under **Meta AI settings** > **App connections** > **Developer mode apps**. It also enables testing via invite-only [release channels](/docs/develop/dat/set-up-release-channels). ### Verify glasses software version 1. In the Meta AI app, go to the Devices tab (the glasses icon at the bottom of the app), and select your device. 2. Tap the gear icon to open **Device settings**. 3. Tap **General** > **About** > **Version**. 4. You should have the minimum supported version or above installed on your glasses, as outlined [here](/docs/develop/dat/version-dependencies/). 5. If your version is below minimum support requirements, update your glasses software. ### Enable developer mode in the Meta AI app 1. On your iOS or Android device, select **Settings** > **App Info**, and then tap the **App version** number five times to display the toggle for developer mode. 2. Select the toggle to enable **Developer Mode**. 3. Click **Enable** to confirm. **iOS** ![Image of enabling developer mode on an iOS device](/images/wearables-devmode-ios.png){: width="296"} **Android** ![Image of enabling developer mode on an android device](/images/wearables-devmode-android.png){: width="296"} ### Integration overview ## Overview The Wearables Device Access Toolkit lets your mobile app integrate with supported AI glasses. An integration establishes a session with the device so your app can access supported sensors on the user's glasses. Users start a session from your app, and then interact through their glasses. They can: * Speak to your app through the device's microphones * Send video or photos from the device's camera * Pause, resume, or stop the session by tapping the glasses, taking them off, or closing the hinges * Play audio to the user through the device's speakers ## Supported devices Detailed support for devices and versions of the Meta AI app and glasses firmware is available on the [Version Dependencies](/docs/develop/dat/version-dependencies/) page. ## Integration lifecycle 1. **Registration**: The user connects your app to their wearable device by tapping a call-to-action in your app. This is a one‑time flow. After registration, your app can identify and connect to the user's device when your app is open. The flow deeplinks the user to the Meta AI app for confirmation, then returns them to your app. 2. **Permissions**: The first time your app attempts to access the user's camera, you must request permission. The user can allow always, allow once, or deny. Your app deeplinks the user to the Meta AI app to confirm the requested permission, and then Meta AI returns them to your app. Microphone access uses the Hands‑Free Profile (HFP), so you request those permissions through iOS or Android platform dialogs. 3. **Session**: After registration and permissions, the user can start a session. During a session, the user engages with your app on their device. ## Sessions All integrations with AI glasses run as sessions. Only one session can run on a device at a time, and certain features are unavailable while your session is active. Users can pause, resume, or stop your session by closing the hinges, taking the glasses off (when wear detection is enabled), or tapping the glasses. Learn more in [Session lifecycle](/docs/develop/dat/lifecycle-events/). ## Key components `MWDATCore` is the foundation for your integration. It handles: - App registration with the user's device and registration state - Device discovery and management - Permission requests and state management - Telemetry `MWDATCamera` handles camera access and: - Resolution and frame rate selection - Starting a video stream and sending/listening for pause, resume, and stop signals - Receiving frames from devices - Capturing a single frame during a stream and delivering it to your app - Photo format `MWDATDisplay` powers visual experiences on Meta Ray-Ban Display glasses, with support for: - Content rendering with support for components like FlexBox, Text, Image, Button, and Icon - Video playback of MP4 clips on the glasses display For more, check out our **API reference documentation**: [iOS](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest), [Android](https://wearables.developer.meta.com/docs/reference/android/dat/latest). ### Microphones and speakers Use mobile platform functions to access the device over Bluetooth. To use the device's microphones for input, use HFP (Hands-Free Profile). Audio is streamed as 8 kHz mono from the device to your app. ### App management After registration, your app appears in the user's App Connections list in the Meta AI app, where permissions can be unregistered or managed. ## Next steps - See real-world integration concepts on [our blog](https://developers.meta.com/blog/introducing-meta-wearables-device-access-toolkit/). - Start building your first integration with our step‑by‑step guides for [iOS](/docs/develop/dat/build-integration-ios/) and [Android](/docs/develop/dat/build-integration-android/). ### Integrate Wearables Device Access Toolkit into your iOS app ## Overview This guide explains how to add Wearables Device Access Toolkit registration, streaming, and photo capture to an existing iOS app. For a complete working sample, compare with the [provided sample app](https://github.com/facebook/meta-wearables-dat-ios/tree/main/samples). ## Prerequisites Complete the environment and glasses steps in [Setup](/docs/develop/dat/getting-started-toolkit/). Your integration must use a registered bundle identifier. To register or manage bundle IDs, see Apple's [Register an App ID](https://developer.apple.com/help/account/identifiers/register-an-app-id/) and [Bundle IDs](https://developer.apple.com/documentation/appstoreconnectapi/bundle-ids) documentation. **App Store Submission Warning:** Publishing to the App Store is not currently supported, but we plan to do so in the future. In the meantime, you can share your integration with test users via our [release channels](/docs/develop/dat/set-up-release-channels/). As a result, since the SDK currently uses the `ExternalAccessory` framework, it will lead to App Store rejection due to Apple's MFi program and privacy manifest requirements. ## Step 1: Add info properties In your app's `Info.plist`, or using Xcode UI, provide the Wearables Device Access Toolkit with the required keys (`AppLinkURLScheme`, `MetaAppID`, `ClientToken`, `TeamID`). `AppLinkURLScheme` enables the Meta AI app to call back to your app during registration. The example below uses `myexampleapp` as a placeholder, but the real one should be a universal link registered with Apple. Adjust the scheme to match your project. These same identifiers are needed for **attestation** of your app, which ensures its authenticity. While app attestation is *not* used in Developer Mode, if you use incorrect identifiers or your app is misconfigured, it won't connect, and you will receive an error. **Note:** To find your `TeamID`, you must first [Sign in to your developer account](https://developer.apple.com/help/account/access/sign-in-to-your-developer-account). Meanwhile, `ClientToken` and `MetaAppID` are autogenerated in the Wearables Developer Center (see [Manage projects](/docs/develop/dat/manage-projects/)). **Note**: If you pre-process `Info.plist`, the `://` suffix will be stripped unless you add the `-traditional-cpp` flag. See [Apple Technical Note TN2175](https://developer.apple.com/library/archive/technotes/tn2175/_index.html#//apple_ref/doc/uid/DTS10004415-CH1-TNTAG3). ```xml CFBundleURLTypes CFBundleTypeRole Editor CFBundleURLName $(PRODUCT_BUNDLE_IDENTIFIER) CFBundleURLSchemes myexampleapp MWDAT AppLinkURLScheme myexampleapp:// MetaAppID $(META_APP_ID) ClientToken $(CLIENT_TOKEN) TeamID $(DEVELOPMENT_TEAM) UIBackgroundModes bluetooth-central NSBluetoothAlwaysUsageDescription Needed to connect to Meta Wearables NSLocalNetworkUsageDescription Needed to stream from Meta Wearables over Wi-Fi NSBonjourServices _bonjour._tcp NSCameraUsageDescription This app needs camera access to stream from your phone's camera as a mock device feed. ``` Bluetooth LE is the baseline connection for discovery, session setup, and control (permission via `NSBluetoothAlwaysUsageDescription`, with `bluetooth-central` added to the `UIBackgroundModes` key for background operation). Wi-Fi is added for high-bandwidth features like camera streaming (permission via `NSLocalNetworkUsageDescription`, with the device's Bonjour service declared in `NSBonjourServices` for discovery). **Note:** The usage-description strings appear in the system permission prompts. If a user denies Local Network access, your app integration will continue over Bluetooth LE, but without streaming. ## Step 2: Add the SDK Swift package Add the SDK through Swift Package Manager. 1. In Xcode, select **File** > **Add Package Dependencies...** 1. Search for `https://github.com/facebook/meta-wearables-dat-ios` in the top right corner. 1. Select `meta-wearables-dat-ios`. 1. Set the version to one of the [available versions](https://github.com/facebook/meta-wearables-dat-ios/tags). 1. Click **Add Package**. 1. Select the target to which you want to add the package. 1. Click **Add Package**. Import the required modules in any Swift files that use the SDK. ```swift import MWDATCamera import MWDATCore ``` ## Step 3: Initialize the SDK Call [`Wearables.configure()`](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest/mwdatcore_wearables#configure) once when your app launches. ```swift func configureWearables() { do { try Wearables.configure() } catch { assertionFailure("Failed to configure Wearables SDK: \(error)") } } ``` ## Step 4: Launch registration from your app Register your application with the Meta AI app either at startup or when the user wants to turn on your wearables integration. ```swift func startRegistration() throws { try Wearables.shared.startRegistration() } func startUnregistration() throws { try Wearables.shared.startUnregistration() } func handleWearablesCallback(url: URL) async throws { _ = try await Wearables.shared.handleUrl(url) } ``` Observe registration and device updates. ```swift let wearables = Wearables.shared Task { for await state in wearables.registrationStateStream() { // Update your registration UI or model } } Task { for await devices in wearables.devicesStream() { // Update the list of available glasses } } ``` ## Step 5: Manage camera permissions Check permission status before streaming and request access if necessary. ```swift var cameraStatus: PermissionStatus = .denied ... cameraStatus = try await wearables.checkPermissionStatus(.camera) ... cameraStatus = try await wearables.requestPermission(.camera) ``` ## Step 6: Create device session Use [`createSession`](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest/mwdatcore_wearablesinterface#createsession) to create a device session and access the capabilities of a Meta Wearable Device. You can also add a stream to a previously created session. You can use [`AutoDeviceSelector`](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest/mwdatcore_autodeviceselector) to make a smart decision for the user to select a device. Alternatively, you can use [`SpecificDeviceSelector`](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest/mwdatcore_specificdeviceselector) if you provide a UI for the user to manually select a specific device. ```swift let deviceSelector = AutoDeviceSelector(wearables: wearables) let session = try wearables.createSession(deviceSelector: deviceSelector) let stateStream = session.stateStream() try session.start() for await state in stateStream { if state == .started { ... } else if state == .stopped { ... } } ``` ## Step 7: Start a camera stream Create a stream by adding it to an existing [`DeviceSession`](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest/mwdatcore_devicesession), and observe its state and display frames. You can request resolution and frame rate control using [`StreamConfiguration`](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest/mwdatcamera_streamconfiguration). Valid `frameRate` values are `2`, `7`, `15`, `24`, or `30` FPS. `resolution` can be set to: - `high`: 720 x 1280 pixels - `medium`: 504 x 896 pixels - `low`: 360 x 640 pixels [`StreamState`](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest/mwdatcamera_streamstate) transitions through `stopping`, `stopped`, `waitingForDevice`, `starting`, `streaming`, and `paused`. Register callbacks to collect frames and state events. ```swift let config = StreamConfiguration( videoCodec: VideoCodec.raw, resolution: StreamingResolution.low, frameRate: 24) guard let camera = try session.addCamera(config: config) else { return } let stream = camera.stream let stateToken = stream.statePublisher.listen { state in Task { @MainActor in // Update your streaming UI state } } let frameToken = stream.videoFramePublisher.listen { frame in guard let image = frame.makeUIImage() else { return } Task { @MainActor in // Render the frame in your preview surface } } stream.start() ``` Resolution and frame rate are constrained by the Bluetooth Classic connection between the user's phone and their AI glasses. To manage limited bandwidth, an automatic ladder reduces quality as needed. It first lowers the resolution by one step (for example, from `high` to `medium`). If bandwidth remains constrained, it then reduces the frame rate (for example, 30 to 24), but never below 15 fps. The image delivered to your app may appear lower quality than expected, even when the resolution reports `high` or `medium`. This is due to per‑frame compression that adapts to available Bluetooth Classic bandwidth. Requesting a lower resolution, a lower frame rate, or both can yield higher visual quality with less compression loss. ## Step 8: Capture and share photos Listen for [`photoDataPublisher`](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest/mwdatcamera_stream#photodatapublisher) events and handle the returned [`PhotoData`](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest/mwdatcamera_photodata). Then, when a stream session is active, call [`capturePhoto`](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest/mwdatcamera_stream#capturephoto). ```swift _ = stream.photoDataPublisher.listen { photoData in let data = photoData.data // Convert to UIImage or hand off to your storage layer } stream.capturePhoto(format: .jpeg) ``` ## Next steps - See details on permission flows in [Permissions and registration](/docs/develop/dat/permissions-requests/). - See details on session lifecycles in [Session lifecycle](/docs/develop/dat/lifecycle-events/). - Test without a device with [Mock Device Kit](/docs/develop/dat/testing-mdk-ios/). - Compare against the [iOS sample app](https://github.com/facebook/meta-wearables-dat-ios/tree/main/samples). - Prepare for release with [Manage projects](/docs/develop/dat/manage-projects/) and [Set up release channels](/docs/develop/dat/set-up-release-channels/) in the Wearables Developer Center. ### Integrate Wearables Device Access Toolkit into your Android app ## Overview This guide explains how to add Wearables Device Access Toolkit registration, streaming, and photo capture to an existing Android app. For a complete working sample, compare with the [provided sample app](https://github.com/facebook/meta-wearables-dat-android/tree/main/samples). ## Prerequisites Complete the environment and glasses configuration steps in [Setup](/docs/develop/dat/getting-started-toolkit/). ## Step 1: Add manifest entries In your app's `AndroidManifest.xml`, add the permissions required to allow your app to communicate with the glasses through Bluetooth. The intent filter with the URI scheme is required so that the Meta AI app can call back to your application. The example below uses `myexampleapp` as a placeholder. Adjust the scheme to match your project. Provide the Wearables Device Access Toolkit with `APPLICATION_ID` and `CLIENT_TOKEN` metadata. Both are needed for **attestation** of your app, which ensures its authenticity, and they can be found in the Wearables Developer Center (see [Manage projects](/docs/develop/dat/manage-projects/)). While an App Signature is *not required* for attestation, the Meta AI app will use it to verify the authenticity of your app. If incorrect identifiers are used or your app is misconfigured, it won't connect, and you will receive an error. **Note:** App attestation is *not* used in Developer Mode, since these apps rely on local logic, rather than connecting to a release channel. If you are using Developer Mode, you can omit these values or simply use `0`. ```xml ``` ## Step 2: Add the SDK to Gradle The Wearables Device Access Toolkit is distributed through [Maven Central](https://central.sonatype.com/). Declare the Wearables Device Access Toolkit artifacts in `libs.versions.toml`, setting `mwdat` to one of the [available versions](https://central.sonatype.com/artifact/com.meta.wearable/mwdat-core/versions). Every artifact shares the same version reference. ```toml [versions] mwdat = "" [libraries] mwdat-core = { group = "com.meta.wearable", name = "mwdat-core", version.ref = "mwdat" } mwdat-camera = { group = "com.meta.wearable", name = "mwdat-camera", version.ref = "mwdat" } mwdat-display = { group = "com.meta.wearable", name = "mwdat-display", version.ref = "mwdat" } mwdat-mockdevice = { group = "com.meta.wearable", name = "mwdat-mockdevice", version.ref = "mwdat" } ``` Then, add them as dependencies in your app's `build.gradle.kts`. ```kotlin dependencies { implementation(libs.mwdat.core) implementation(libs.mwdat.camera) implementation(libs.mwdat.display) implementation(libs.mwdat.mockdevice) } ``` ## Step 3: Initialize the SDK Initialize the SDK once per process at start up. ```kotlin Wearables.initialize(context) ``` Invoking other Wearables Device Access Toolkit APIs before initialization yields [`WearablesError.NOT_INITIALIZED`](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_core_types_wearableserror). For lifecycle placement guidance, read [Session lifecycle](/docs/develop/dat/lifecycle-events/). ## Step 4: Launch registration from your app Register your application with the Meta AI app either at startup or when the user wants to turn on your wearables integration. ```kotlin fun requestWearablesRegistration(activity: Activity) { Wearables.startRegistration(activity) } fun requestWearablesUnregistration(activity: Activity) { Wearables.startUnregistration(activity) } ``` Observe registration and device updates. ```kotlin ... Wearables.registrationState.collect { state -> onState(state) } ... Wearables.devices.collect { devices -> onDevices(devices.toList()) } ``` ## Step 5: Manage camera permissions Before streaming, check the Wearables camera permission and launch the SDK contract if required. ```kotlin var permissionStatus = Wearables.checkPermissionStatus(Permission.CAMERA) if (permissionStatus == PermissionStatus.Granted) { // start streaming } permissionStatus = requestWearablesPermission(Permission.CAMERA) ... private var permissionContinuation: CancellableContinuation? = null private val permissionMutex = Mutex() // Requesting wearable device permissions via the Meta AI app private val permissionsResultLauncher = registerForActivityResult(Wearables.RequestPermissionContract()) { result -> permissionContinuation?.resume(result) permissionContinuation = null } // Convenience method to make a permission request in a sequential manner // Uses a Mutex to ensure requests are processed one at a time, preventing race conditions suspend fun requestWearablesPermission(permission: Permission): PermissionStatus { return permissionMutex.withLock { suspendCancellableCoroutine { continuation -> permissionContinuation = continuation continuation.invokeOnCancellation { permissionContinuation = null } permissionsResultLauncher.launch(permission) } } } ``` ## Step 6: Create device session Use [`createSession`](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_core_wearables#createsession) to create a device session and access the capabilities of a Meta Wearable Device. You can also add a stream to a previously created session. You can use [`AutoDeviceSelector`](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_core_selectors_autodeviceselector) to make a smart decision for the user to select a device. Alternatively, you can use [`SpecificDeviceSelector`](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_core_selectors_specificdeviceselector) if you provide a UI for the user to select a device. ```kotlin val session = Wearables.createSession(AutoDeviceSelector()).getOrElse { error -> showError(error.description) return } session.start() ``` ## Step 7: Start a camera stream Create a stream by adding it to an existing [`DeviceSession`](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_core_session_devicesession), and observe its state and display frames. You can request resolution and frame rate control using [`StreamConfiguration`](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_camera_types_streamconfiguration). Valid `frameRate` values are `2`, `7`, `15`, `24`, or `30` FPS. `videoQuality` can be set to: - `HIGH`: 720 x 1280 pixels - `MEDIUM`: 504 x 896 pixels - `LOW`: 360 x 640 pixels [`StreamState`](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_camera_types_streamstate) transitions through `STARTING`, `STARTED`, `STREAMING`, `PAUSED`, `STOPPING`, `STOPPED`, and `CLOSED`. Register callbacks to collect frames and state events. ```kotlin fun start(deviceId: DeviceIdentifier) { val config = StreamConfiguration(videoQuality = VideoQuality.MEDIUM, frameRate = 24) session.addCamera(config).fold( onSuccess = { camera -> val stream = camera.stream scope.launch { stream.videoStream.collect { frame -> displayFrame(frame) } } scope.launch { stream.state.collect { state -> updateStreamUi(state) if (state == StreamState.STOPPED) { stopStream() } } } stream.start() }, onFailure = { error, _ -> showError(error.description) }, ) } ``` Resolution and frame rate are constrained by the Bluetooth Classic connection between the user's phone and their AI glasses. To manage limited bandwidth, an automatic ladder reduces quality as needed. It first lowers the resolution by one step (for example, from `HIGH` to `MEDIUM`). If bandwidth remains constrained, it then reduces the frame rate (for example, 30 to 24), but never below 15 fps. The image delivered to your app may appear lower quality than expected, even when the resolution reports `HIGH` or `MEDIUM`. This is due to per‑frame compression that adapts to available Bluetooth Classic bandwidth. Requesting a lower resolution, a lower frame rate, or both can yield higher visual quality with less compression loss. ## Step 8: Capture and share photos When a stream session is active, call [`capturePhoto`](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_camera_stream#capturephoto) and handle the returned [`PhotoData`](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_camera_types_photodata). Add `app/src/main/res/xml/file_paths.xml` so that the FileProvider can expose cached images. ```kotlin stream.capturePhoto() .onSuccess { data -> ... } .onFailure(onError) ``` ## Next steps - See details on permission flows in [Permissions and registration](/docs/develop/dat/permissions-requests/). - See details on session lifecycles in [Session lifecycle](/docs/develop/dat/lifecycle-events/). - Test without a device with [Mock Device Kit](/docs/develop/dat/testing-mdk-android/). - Compare against the [Android sample app](https://github.com/facebook/meta-wearables-dat-android/tree/main/samples). - Prepare for release with [Manage projects](/docs/develop/dat/manage-projects/) and [Set up release channels](/docs/develop/dat/set-up-release-channels/) in the Wearables Developer Center. ### Display access overview ## Overview The Meta Wearables Device Access Toolkit gives you access to the display on Meta Ray-Ban Display glasses. By using a component-based layout API inspired by popular UI frameworks in the mobile development community, you can build layouts with familiar mobile UI patterns in Swift or Kotlin. The SDK then handles rendering on the glasses automatically. In addition to access to on-device sensors like camera, microphone, and speakers, the Meta Wearables Device Access Toolkit provides display functionality that enables you to build a variety of immersive experiences leveraging the display. Users can control display integrations through the Meta Ray-Ban Display glasses touchpad and Meta Neural Band. The system translates these inputs into actions delivered to your app. ## Requirements ### Hardware * Meta Ray-Ban Display glasses paired with a mobile device * A mobile device with Meta AI app installed and configured. **Note**: Refer to the [Version Dependencies](/docs/develop/dat/version-dependencies/) page for the latest Meta AI app and firmware versions supported. ### Software Download the Meta AI app from the [App Store (iOS)](https://apps.apple.com/us/app/meta-ai-assistant-glasses/id1558240027) or [Play Store (Android)](https://play.google.com/store/apps/details?id=com.facebook.stella) if you haven't done so already. The Device Access Toolkit needs to be installed on your glasses for the display functionality to work. To install DAT: 1. Update Meta AI app to v272 or later. 2. Update your glasses firmware to v125 or later via the Meta AI app. 3. Put on your glasses. This is required before proceeding. 4. Enable Dev Mode in the Meta AI app: * Go to Settings → App Info, then tap App Version 5 times. * If Dev Mode is already enabled, press the install button to install DAT on your glasses. * On iOS, a Wi-Fi prompt should appear after a few seconds. * App installation takes ~5–10 seconds. Please keep Meta AI app open during this time. ## Sessions Display access operates on a session-based model, where your mobile app creates a Meta Wearables Device Access Toolkit session, attaches a display capability to it, and sends UI content to the glasses over Bluetooth. Essentially, the SDK serializes your layouts, transmits them to these glasses, and routes user interactions back to your app as events. In this model: * Users must explicitly initiate a display session. * Your app has exclusive display control during active sessions. * Sessions must be explicitly terminated by user action or app logic. * The system may interrupt sessions for calls, notifications, or other priority events. ## Supported components | Component | Description | |-----------|-------------| | **FlexBox** | Flexible layout container with row/column direction, spacing, alignment, and wrapping | | **Text** | Styled text with heading, body, and meta presets | | **Image** | Images loaded from a URL, with size and corner radius options | | **Button** | Tappable buttons with label, icon, and style variants | | **Icon** | System icons. See [Icons](/develop/dat/display-icons) for the full list. | | **Video** | Video playback from a URL (MP4 format) | ## Examples of what you can build * Informational overlays (weather, navigation, notifications) * Step-by-step guides and tutorials * Media viewers with images and short video clips * Interactive menus with buttons and tap handlers * Status dashboards with icons and styled text ## Display lifecycle The display capability follows a defined lifecycle that your app must manage. Understanding these states is essential for building a reliable integration. ### States | State | Description | |-------|-------------| | `stopped` | Display capability is not active. This is the initial state. | | `starting` | The SDK is connecting to the glasses display service. | | `started` | The display is ready. You can now send content with `send()` (iOS) or `sendContent()` (Android). | | `stopping` | The display is shutting down. | ``` stopped ──▶ starting ──▶ started ──▶ stopping ──▶ stopped ``` The display transitions to `started` once the glasses accept the display capability request. If the glasses disconnect, the display transitions through `stopping` back to `stopped`. You can restart the display by calling `start()` again. ## Best practices ### Performance - **Keep layouts simple.** Simple views (text, small images) transition quickly. Complex views with large images may introduce lag due to Bluetooth bandwidth constraints. - **Optimize images.** Use appropriately sized images for the glasses display. The display resolution is 600x600, and there is no benefit to sending larger images. - **Limit video size.** Keep videos under 400px per side and 70,000 total pixels. Use short clips. ### User experience - **Always have a root view.** Designate one view as your L0 (root) view. Back from L0 ends the session, so make sure users can always navigate back to it. - **Handle disconnects gracefully.** Bluetooth connections can drop. Observe the display state and show appropriate UI in your mobile app when the connection is lost. - **Respect display sleep.** The display dims and then enters sleep mode after a period of inactivity. Design your content updates around this behavior, like sending fresh content when the display wakes. - **Use clear, readable text.** The glasses display is small. Use the `heading` style for key information, and keep text concise. - **Provide visual feedback.** Use button style changes and icons to indicate selection and activation, so interactive controls are obvious. ### Architecture - **One display per session.** Only one display capability can be attached to a session at a time. Remove the existing display before adding a new one. - **Send complete views.** Each [`send()`](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest/mwdatdisplay_display#send) (iOS) / [`sendContent()`](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_display_display#sendcontent) (Android) call replaces the entire display. There is no partial update mechanism, so always send the full layout. - **Manage state on the phone.** The glasses do not retain application state. Your app is the source of truth for navigation state, data, and user preferences. ## Sample app The SDK includes a **Display Access** sample app that demonstrates a complete display access integration. The sample shows: - Session setup and display lifecycle management - A scrollable list of tutorials with images and tap handlers - Detail views with styled text, images, and navigation buttons - Step-by-step navigation with back/next controls - Video playback of tutorial clips Find the sample apps in GitHub: - [DisplayAccess for iOS](https://github.com/facebook/meta-wearables-dat-ios/tree/main/samples/DisplayAccess) - [DisplayAccess for Android](https://github.com/facebook/meta-wearables-dat-android/tree/main/samples/DisplayAccess) ### Integrate Display capability into your iOS app ## Overview This guide explains how to integrate display functionality into your existing iOS app. For a complete guide on adding Wearables Device Access Toolkit capabilities to an existing iOS app, including registration, streaming, and photo capture, check out the [iOS integration guide](/docs/develop/dat/build-integration-ios/). ## Create a visual display experience for your app ### Step 1: Import modules ```swift import MWDATCore import MWDATDisplay ``` ### Step 2: Connect to the display Connect to the display by configuring the SDK, creating a session, and then initializing it. ```swift // Configure the SDK try Wearables.configure() // Create a session targeting display-capable glasses let wearables = Wearables.shared let session = try wearables.createSession() // Start the session and wait for it to connect. Then observe session.stateStream() for .started try session.start() // Attach the display capability let display = try session.addDisplay() // Start the display and observe state. Then observe display.statePublisher for .started display.start() // Send content once the display reaches .started let view = FlexBox(direction: .column, spacing: 12) { Text("Hello, glasses!", style: .heading) } try await display.send(view) ``` ### Step 3: Observe state changes ```swift // Using async stream for await state in display.statePublisher.stream { switch state { case .started: // Display is ready, send content case .stopped: // Display disconnected or stopped case .starting, .stopping: // Transitional states } } ``` ### Step 4: Stop display When your app is done using the display capabilities, clean up resources: ```swift display.stop() session.stop() ``` **Note:** The glasses display dims after 20 seconds of inactivity and enters sleep mode at 25 seconds. Display sleep does *not* end the Wearables Device Access Toolkit session. When the display wakes, your app can either show the previously displayed content or send a new view. ## Build content Wearables Device Access Toolkit uses a declarative component system to build UI layouts for display, so you compose views using a result builder in Swift. Each call to [`send()`](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest/mwdatdisplay_display#functions-public) replaces the entire display, so there are no incremental updates. Views are presented one at a time with vertical scrolling. Horizontal scrolling is not supported. ### FlexBox **FlexBox** is the primary layout container for Meta Ray-Ban Display apps. It helps you build declarative UI trees with text, images, icons, and buttons. This means that every layout starts with a root FlexBox. Inside, you arrange either these child components or nested FlexBoxes using specific layout properties. [FlexBox](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest/mwdatdisplay_flexbox) arranges children along a main axis (row or column) with control over spacing, alignment, and wrapping. ```swift FlexBox( direction: .column, // .column, .row, .columnReverse, .rowReverse spacing: 12, // Gap between children in dp alignment: .start, // Main axis: .start, .center, .end, .stretch crossAlignment: .center, // Cross axis: .start, .center, .end, .stretch wrap: false, // Whether children wrap to the next line padding: EdgeInsets(all: 16) ) { // Child components go here } ``` ### Text Displays styled text. Three style presets match the glasses design system. | [`TextStyle`](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest/mwdatdisplay_textstyle) | Description | |-------|-------------| | `heading` | Large, bold text for section titles | | `body` | Standard text for general content | | `meta` | Small text for captions and metadata | | [`TextColor`](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest/mwdatdisplay_textcolor) | Description | |-------|-------------| | `primary` | Default high-contrast text | | `secondary` | Lower-contrast supporting text | ```swift Text("Welcome", style: .heading) Text("Tap a button to continue", style: .body, color: .secondary) Text("Step 1 of 5", style: .meta, color: .secondary) ``` ### Images Display images loaded from URLs. | [`ImageSize`](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest/mwdatdisplay_imagesize) | Description | |-------------|-------------| | `icon` | Small, inline-sized image | | `fill` | Fills the available space | | [`CornerRadius`](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest/mwdatdisplay_cornerradius) | Description | |---------------|-------------| | `none` | Sharp corners | | `small` | Slightly rounded | | `medium` | Moderately rounded | ```swift Image( uri: "https://example.com/photo.png", sizePreset: .fill, cornerRadius: .medium ) ``` **Note:** The Meta Ray-Ban Display glasses render at 600x600 resolution. Use appropriately sized images — there is no benefit to sending larger images, and oversized assets introduce lag due to Bluetooth bandwidth constraints. ### Buttons Implement tappable buttons with corresponding labels, optional icons, and click handlers. | [`ButtonStyle`](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest/mwdatdisplay_buttonstyle) | Description | |-------|-------------| | `primary` | High-emphasis filled button | | `secondary` | Medium-emphasis filled button | | `outline` | Low-emphasis outlined button | ```swift Button(label: "Continue", style: .primary, iconName: .arrowRight) { // Handle tap } Button(label: "Go back", style: .outline, iconName: .arrowLeft) { // Handle tap } ``` ### Icons Display a system icon from a built-in catalog of 100+ glyphs, covering navigation, media, weather, social, and more. | [`IconStyle`](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest/mwdatdisplay_iconstyle) | Description | |-------|-------------| | `filled` | Solid icon | | `outline` | Hollow icon | ```swift Icon(name: .checkmarkCircle, style: .filled) Icon(name: .bell, style: .outline) ``` See the [icon catalog](/docs/develop/dat/display-icons) for all available `IconName` values. If you need an icon outside the built-in catalog, you can also provide your own [custom image](#images). ### View Modifiers On iOS, padding and tap handlers can be applied to any component. [`Flex modifiers`](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest/mwdatdisplay_flexbox) (`flexGrow`, `flexShrink`, `alignSelf`) only apply to `FlexBox` — wrap a non-`FlexBox` child in a `FlexBox` to opt into flex layout behavior. ```swift // Padding and tap handlers apply to any component. Text("Tappable text", style: .body) .padding(16) // Uniform padding .onTap { // Handle tap on this component } // Per-edge padding Image(uri: imageUrl, sizePreset: .fill) .padding(EdgeInsets(top: 8, bottom: 8, leading: 16, trailing: 16)) // Flex modifiers only apply to a FlexBox. // Wrap a non-FlexBox child in a FlexBox to apply flexGrow, flexShrink, or alignSelf. FlexBox { Text("Flexible text", style: .body) } .flexGrow(1) // Grow to fill available space .flexShrink(0) // Don't shrink below natural size .alignSelf(.center) // Override cross-axis alignment ``` ### Layouts #### Column layout Use [`direction`](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest/mwdatdisplay_direction) to determine the main axis along which children are arranged. This code sample illustrates a *vertical* layout with a header, body text, and action button. ```swift let view = FlexBox(direction: .column, spacing: 12, padding: EdgeInsets(all: 16)) { Text("Oil Change Guide", style: .heading) Text("Learn how to change your car's oil in 6 easy steps.", style: .body) Text("Duration: 30 min", style: .meta, color: .secondary) Button(label: "Start", style: .primary, iconName: .arrowRight) { showFirstStep() } } try await display.send(view) ``` #### Row layout This code sample illustrates a *horizontal* layout with an image thumbnail and text. ```swift let view = FlexBox(direction: .row, spacing: 12, crossAlignment: .center, padding: EdgeInsets(all: 16)) { FlexBox { Image(uri: thumbnailUrl, sizePreset: .fill, cornerRadius: .medium) } .flexGrow(1) FlexBox(direction: .column, spacing: 4) { Text("Brake Pad Replacement", style: .body) Text("45 min", style: .meta, color: .secondary) } .flexGrow(3) } try await display.send(view) ``` #### Clickable list items Create a list of tappable items, each with its own icon and label. ```swift let view = FlexBox(direction: .column, spacing: 8, padding: EdgeInsets(all: 16)) { Text("Tutorials", style: .heading) for tutorial in tutorials { FlexBox(direction: .row, spacing: 12, crossAlignment: .center) { Icon(name: .gear, style: .filled) Text(tutorial.title, style: .body) } .padding(12) .onTap { selectTutorial(tutorial) } } } try await display.send(view) ``` #### Navigation Create a step-by-step view with navigation buttons located at the bottom of the viewport. ```swift let view = FlexBox(direction: .column, spacing: 12, padding: EdgeInsets(all: 16)) { Text("Step \(stepIndex + 1) of \(totalSteps)", style: .meta, color: .secondary) Text(step.title, style: .heading) Text(step.description, style: .body) Image(uri: step.imageUrl, sizePreset: .fill, cornerRadius: .medium) FlexBox(direction: .row, spacing: 8, alignment: .center) { Button(label: "Back", style: .outline, iconName: .arrowLeft) { showPreviousStep() } Button(label: "Next", style: .primary, iconName: .arrowRight) { showNextStep() } } } try await display.send(view) ``` ## User inputs Users can interact with display content through the Meta Ray-Ban Display glasses touchpad. These interactions are abstracted as tap events and delivered to your app through callbacks. ### Button clicks Buttons receive tap events through a handler. ```swift Button(label: "Accept", style: .primary) { handleAccept() } ``` ### Tappable areas Make any component tappable by wrapping it in a tap handler. ```swift FlexBox(direction: .row, spacing: 8) { Icon(name: .heart, style: .filled) Text("Favorites", style: .body) } // Tap handler .onTap { openFavorites() } ``` ### Back Back (a two-finger tap on the Meta Ray-Ban Display glasses temple) ends the display session. ## Video Video playback allows you to play video content from a URL on the full-screen video player of Meta Ray-Ban Display glasses. ### Constraints - **Format:** MP4 only - **Maximum dimensions:** 400 pixels per side, 70,000 total pixels - **URL scheme:** `https` only - **Concurrency:** One video at a time per display session ### Play video Play video using [`VideoPlayer`](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest/mwdatdisplay_videoplayer). ```swift // Start video playback let video = VideoPlayer( provider: .uri("https://example.com/tutorial-clip.mp4"), codec: .mp4, onError: { error in print("Video error: \(error)") } ) try await display.send(video) // Stop video playback await display.sendVideoStop() ``` ### Observe video events ```swift display.onPlaybackEvent = { event in switch event.type { case .started: print("Video started playing") case .ended: print("Video finished") case .error: print("Video error: \(event.errorType)") case .stopped: print("Video was stopped") case .unknown: break } } ``` ## Error handling Display operations may fail. Handle errors gracefully to provide a better experience for your users. ### Display errors [`DisplayError`](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest/mwdatdisplay_displayerror) identifies errors for display operations on AI glasses. ```swift do { try await display.send(view) } catch let error as DisplayError { switch error { case .deviceDisconnected: // The glasses disconnected — prompt user to reconnect showReconnectPrompt() case .connectionNotAvailable: // No connection available to the glasses showConnectionError() case .deviceNotFound: // The target device was not found showDeviceNotFoundError() case .invalidVideoURL: // The video URL is blank or uses an unsupported scheme showInvalidUrlError() case .displayError(let message): // The glasses reported an error (e.g., capability not active) print("Display error: \(message)") } } ``` ### Session-level errors when adding a display ```swift do { let display = try session.addDisplay() } catch let error as DeviceSessionError { switch error { case .sessionIdle: // Session hasn't been started yet — call session.start() first case .sessionAlreadyStopped: // Session was already stopped — create a new session case .capabilityAlreadyActive: // A display capability is already attached to this session case .unexpectedError: // SDK initialization issue } } ``` ### Video errors [`VideoError`](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest/mwdatdisplay_videoerror) outlines errors for video playback on AI glasses. ```swift let video = VideoPlayer( provider: .uri(videoUrl), codec: .mp4, onError: { error in switch error { case .playbackFailed(let errorType): switch errorType { case .urlInvalid: print("Invalid video URL") case .alreadyPlaying: print("Another video is already playing") case .playbackFailed: print("Playback failed on the glasses") case .unknown: print("Unknown video error") } } } ) ``` ### Integrate Display capability into your Android app ## Overview This guide explains how to integrate display functionality into your existing Android app. For a complete guide on adding Wearables Device Access Toolkit capabilities to an existing Android app, including registration, streaming, and photo capture, check out the [Android integration guide](/docs/develop/dat/build-integration-android/). ## Create a visual display experience for your app ### Step 1: Add the Display dependency In your app-level `build.gradle.kts`: ```kotlin dependencies { implementation("com.meta.wearable:mwdat-core:") implementation("com.meta.wearable:mwdat-display:") } ``` ### Step 2: Import packages Import the required display packages. ```kotlin import com.meta.wearable.dat.core.* import com.meta.wearable.dat.display.* import com.meta.wearable.dat.display.types.* import com.meta.wearable.dat.display.views.* ``` ### Step 3: Connect to the display Connect to the display by initializing the SDK, creating a session, and attaching the display capability. ```kotlin // Initialize the SDK Wearables.initialize(context) // Create and start a session. Then observe session.state for DeviceSessionState.STARTED val session = Wearables.createSession(AutoDeviceSelector()).getOrThrow() session.start() session.state.first { it == DeviceSessionState.STARTED } // Attach the display capability. Then observe display.state for DisplayState.STARTED. val displayResult = session.addDisplay(DisplayConfiguration()) val display = displayResult.getOrThrow() // Send content once the display reaches STARTED display.sendContent { flexBox(direction = Direction.COLUMN, gap = 12) { text("Hello, glasses!", style = TextStyle.HEADING) } } ``` ### Step 4: Observe state changes ```kotlin // Using Flow display.state.collect { state -> when (state) { DisplayState.STARTED -> { // Display is ready, send content } DisplayState.STOPPED -> { // Display disconnected or stopped } DisplayState.STARTING, DisplayState.STOPPING -> { // Transitional states } DisplayState.CLOSED -> { // Terminal: capability has been removed from the session } } } ``` ### Step 5: Stop display When your app is done using the display capabilities, clean up resources. ```kotlin session.removeDisplay() session.stop() ``` **Note:** The glasses display dims after 20 seconds of inactivity and enters sleep mode at 25 seconds. Display sleep does *not* end a Wearables Device Access Toolkit session. When the display wakes, your app can either show the previously displayed content or send a new view. ## Build content Wearables Device Access Toolkit uses a declarative component system to build UI layouts for display, so you compose views using a result builder in Kotlin. Each call to [`sendContent()`](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_display_display#sendcontent) replaces the entire display, so there are no incremental updates. Views are presented one at a time with vertical scrolling. Horizontal scrolling is not supported. ### FlexBox **FlexBox** is the primary layout container for Meta Ray-Ban Display apps. It helps you build declarative UI trees with text, images, icons, and buttons. This means that every layout starts with a root FlexBox. Inside, you arrange either these child components or nested FlexBoxes using specific layout properties. [`flexBox`](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_display_views_flexboxscope) arranges children along a main axis (row or column) with control over spacing, alignment, and wrapping. ```kotlin flexBox( direction = Direction.COLUMN, // COLUMN, ROW, COLUMN_REVERSE, ROW_REVERSE gap = 12, // Gap between children in dp alignment = Alignment.START, // Main axis: START, CENTER, END, STRETCH crossAlignment = Alignment.CENTER, // Cross axis: START, CENTER, END, STRETCH wrap = true, // Whether children wrap paddingTop = 16, // Per-edge padding in dp paddingBottom = 16, paddingStart = 16, paddingEnd = 16 ) { // Child components go here } ``` ### Text Display styled text. Three style presets match the glasses design system. | [`TextStyle`](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_display_views_textstyle) | Description | |-------|-------------| | `HEADING` | Large, bold text for section titles | | `BODY` | Standard text for general content | | `META` | Small text for captions and metadata | | [`TextColor`](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_display_views_textcolor) | Description | |-------|-------------| | `PRIMARY` | Default high-contrast text | | `SECONDARY` | Lower-contrast supporting text | ```kotlin text("Welcome", style = TextStyle.HEADING) text("Tap a button to continue", style = TextStyle.BODY, color = TextColor.SECONDARY) text("Step 1 of 5", style = TextStyle.META, color = TextColor.SECONDARY) ``` ### Images Display images loaded from URLs. | [`ImageSize`](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_display_views_imagesize) | Description | |-------------|-------------| | `ICON` | Small, inline-sized image | | `FILL` | Fills the available space | | [`CornerRadius`](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_display_views_cornerradius) | Description | |---------------|-------------| | `NONE` | Sharp corners | | `SMALL` | Slightly rounded | | `MEDIUM` | Moderately rounded | ```kotlin image( uri = "https://example.com/photo.png", sizePreset = ImageSize.FILL, cornerRadius = CornerRadius.MEDIUM ) ``` **Note:** The Meta Ray-Ban Display glasses render at 600x600 resolution. Use appropriately sized images — there is no benefit to sending larger images, and oversized assets introduce lag due to Bluetooth bandwidth constraints. ### Buttons Implement tappable buttons with corresponding labels, optional icons, and click handlers. | [`ButtonStyle`](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_display_views_buttonstyle) | Description | |-------|-------------| | `PRIMARY` | High-emphasis filled button | | `SECONDARY` | Medium-emphasis filled button | | `OUTLINE` | Low-emphasis outlined button | ```kotlin button( label = "Continue", style = ButtonStyle.PRIMARY, iconName = IconName.ARROW_RIGHT, onClick = { /* Handle tap */ } ) button( label = "Go back", style = ButtonStyle.OUTLINE, iconName = IconName.ARROW_LEFT, onClick = { /* Handle tap */ } ) ``` ### Icons Display a system icon from a built-in catalog of 100+ glyphs, covering navigation, media, weather, social, and more. | [`IconStyle`](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_display_views_iconstyle) | Description | |-------|-------------| | `FILLED` | Solid icon | | `OUTLINE` | Hollow icon | ```kotlin icon(name = IconName.CHECKMARK_CIRCLE, style = IconStyle.FILLED) icon(name = IconName.BELL, style = IconStyle.OUTLINE) ``` See the [icon catalog](/develop/dat/display-icons) for all available `IconName` enum values. If you need an icon outside the built-in catalog, you can also provide your own [custom image](#images). ### Flex layout properties [Flex properties](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_display_views_flexboxscope) (`flexGrow`, `flexShrink`) are constructor parameters on `flexBox`. To apply flex behavior to a non-`flexBox` child, wrap it in a `flexBox`. ```kotlin flexBox(direction = Direction.ROW) { flexBox(flexGrow = 1f) { text("Label", style = TextStyle.BODY) } flexBox(flexShrink = 0f) { button("Action", style = ButtonStyle.PRIMARY) } } ``` FlexBox containers also support click handling and per-edge padding. ```kotlin flexBox(paddingAll = 16, onClick = { /* Handle tap */ }) { text("Tappable area", style = TextStyle.BODY) } ``` ### Layouts #### Column layout Use [`Direction`](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_display_views_direction) to determine the main axis along which children are arranged. This code sample illustrates a *vertical* layout with a header, body text, and action button. ```kotlin display.sendContent { flexBox(direction = Direction.COLUMN, gap = 12, paddingAll = 16) { text("Oil Change Guide", style = TextStyle.HEADING) text("Learn how to change your car's oil in 6 easy steps.", style = TextStyle.BODY) text("Duration: 30 min", style = TextStyle.META, color = TextColor.SECONDARY) button( label = "Start", style = ButtonStyle.PRIMARY, iconName = IconName.ARROW_RIGHT, onClick = { showFirstStep() } ) } } ``` #### Row layout This code sample illustrates a *horizontal* layout with an image thumbnail and text. ```kotlin display.sendContent { flexBox( direction = Direction.ROW, gap = 12, crossAlignment = Alignment.CENTER, paddingAll = 16, ) { flexBox(direction = Direction.COLUMN, flexGrow = 1f) { image( uri = thumbnailUrl, sizePreset = ImageSize.FILL, cornerRadius = CornerRadius.MEDIUM, ) } flexBox(direction = Direction.COLUMN, gap = 4, flexGrow = 3f) { text("Brake Pad Replacement", style = TextStyle.BODY) text("45 min", style = TextStyle.META, color = TextColor.SECONDARY) } } } ``` #### Clickable list items Create a list of tappable items, each with its own icon and label. ```kotlin display.sendContent { flexBox(direction = Direction.COLUMN, gap = 8, paddingAll = 16) { text("Tutorials", style = TextStyle.HEADING) for (tutorial in tutorials) { flexBox( direction = Direction.ROW, gap = 12, crossAlignment = Alignment.CENTER, paddingAll = 12, onClick = { selectTutorial(tutorial) }, ) { icon(name = IconName.GEAR, style = IconStyle.FILLED) text(tutorial.title, style = TextStyle.BODY) } } } } ``` #### Navigation Create a step-by-step view with navigation buttons located at the bottom of the viewport. ```kotlin display.sendContent { flexBox(direction = Direction.COLUMN, gap = 12, paddingAll = 16) { text( "Step ${stepIndex + 1} of $totalSteps", style = TextStyle.META, color = TextColor.SECONDARY, ) text(step.title, style = TextStyle.HEADING) text(step.description, style = TextStyle.BODY) image( uri = step.imageUrl, sizePreset = ImageSize.FILL, cornerRadius = CornerRadius.MEDIUM, ) flexBox(direction = Direction.ROW, gap = 8, alignment = Alignment.CENTER) { button( label = "Back", style = ButtonStyle.OUTLINE, iconName = IconName.ARROW_LEFT, onClick = { showPreviousStep() }, ) button( label = "Next", style = ButtonStyle.PRIMARY, iconName = IconName.ARROW_RIGHT, onClick = { showNextStep() }, ) } } } ``` ## User inputs Users can interact with display content through the Meta Ray-Ban Display glasses touchpad. These interactions are abstracted as tap events and delivered to your app through callbacks. ### Button clicks Buttons receive tap events through their `onClick` handler. ```kotlin button(label = "Accept", style = ButtonStyle.PRIMARY, onClick = { handleAccept() }) ``` ### Tappable areas Make any FlexBox tappable by passing an `onClick` handler. ```kotlin flexBox(direction = Direction.ROW, gap = 8, onClick = { openFavorites() }) { icon(name = IconName.HEART, style = IconStyle.FILLED) text("Favorites", style = TextStyle.BODY) } ``` ### Back Back (a two-finger tap on the Meta Ray-Ban Display glasses temple) ends the display session. ## Video Video playback allows you to play video content from a URL on the full-screen video player of Meta Ray-Ban Display glasses. ### Constraints - **Format:** MP4 only - **Maximum dimensions:** 400 pixels per side, 70,000 total pixels - **URL scheme:** `https` only - **Concurrency:** One video at a time per display session ### Play video Play video using [`VideoPlayer`](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_display_views_videoplayer). ```kotlin val player = VideoPlayer( source = VideoSource.Url("https://example.com/tutorial-clip.mp4"), codec = VideoCodec.MP4 ) // Send the video to the display display.sendContent { video(player = player) } // Start video playback player.play() // Observe playback state player.state.collect { state -> when (state) { VideoPlayerState.PLAYING -> { /* Video is playing */ } VideoPlayerState.ENDED -> { /* Video finished */ } VideoPlayerState.IDLE -> { /* Not started */ } VideoPlayerState.STARTING -> { /* Buffering */ } VideoPlayerState.PAUSE -> { /* Paused */ } } } // Stop playback player.close() ``` ## Error handling Display operations may fail. Handle errors gracefully to provide a better experience for your users. ### Display errors Display operations return [`DatResult`](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_core_types_datresult), which will contain a [`DisplayError`](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_display_types_displayerror) if an error occurred during that operation. ```kotlin display.sendContent { flexBox { text("Hello") } }.fold( onSuccess = { /* Content sent successfully */ }, onFailure = { error, _ -> when (error) { DisplayError.CAPABILITY_DENIED -> // Device denied display capability — check Wearables Developer Center config checkDeveloperCenterSetup() DisplayError.DEVICE_DISCONNECTED -> // Glasses disconnected — prompt user to reconnect showReconnectPrompt() DisplayError.INVALID_SESSION_STATE -> // Display not in .STARTED state — wait for state change waitForDisplayReady() DisplayError.RENDERING_FAILED -> // Content failed to render on the glasses showRenderError() DisplayError.UNEXPECTED_ERROR -> // Unknown error showGenericError() } } ) ``` ### Session-level errors when adding a display ```kotlin session.addDisplay().fold( onSuccess = { display -> /* Use the display */ }, onFailure = { error, _ -> showError(error.description) } ) ``` ### Video errors [`VideoPlayerError`](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_display_types_videoplayererror) outlines errors for video playback on Meta Ray-Ban Display glasses. ```kotlin player.error.collect { error -> when (error) { VideoPlayerError.NOT_BOUND -> // play() called before player was bound — send content first VideoPlayerError.INVALID_URL -> // URL is blank or uses unsupported scheme (only https) VideoPlayerError.INVALID_DIMENSIONS -> // Video exceeds max dimensions (400px per side, 70,000 total pixels) VideoPlayerError.ALREADY_PLAYING -> // Another video is active on the device VideoPlayerError.STREAM_REJECTED -> // Device rejected the video stream VideoPlayerError.PLAYBACK_FAILED -> // Codec error or surface unavailable on glasses VideoPlayerError.UNEXPECTED_ERROR -> // Unknown error null -> { /* No error */ } } } ``` ### Simulate Display content ## Overview Mock Device Kit can render the 600×600 pixel display surface of Meta Ray-Ban Display glasses in your app or in Chrome. Use either preview to test Display content and interactions without physical glasses. Both previews use the same Display content and interaction handlers as your app. Content updates appear in the preview, and selections are sent back to your app. ## Prerequisites Add Mock Device Kit to your project by following the setup guide for your platform: - [How to test with Mock Device Kit on iOS](/docs/develop/dat/testing-mdk-ios/) - [How to test with Mock Device Kit on Android](/docs/develop/dat/testing-mdk-android/) ## Simulate with the Chrome extension ### Install the Chrome extension 1. Open the [Meta Ray-Ban Display Simulator](https://chromewebstore.google.com/detail/jpjlmmodokemlepklkdbimceggpbjcll) page in the Chrome Web Store. 1. Select **Add to Chrome**. 1. Pin the extension to the Chrome toolbar for quick access. ### iOS Start a simulated display and the preview server: ```swift import MWDATMockDevice private let displayPreviewPort: UInt16 = 9000 @MainActor func startChromeDisplayPreview() async throws { let mockDeviceKit = MockDeviceKit.shared mockDeviceKit.enable() let mockGlasses = try mockDeviceKit.pairGlasses(model: .metaRayBanDisplay) mockGlasses.powerOn() mockGlasses.don() _ = try await mockDeviceKit.startTestServer(port: displayPreviewPort) } ``` Continue with the `DeviceSession` and content flow described in [Integrate Display capability into your iOS app](/docs/develop/dat/display-ios/). Content sent through the Display capability appears in the browser preview. Run the app in an iOS Simulator, open `http://127.0.0.1:9000/` in Chrome on your computer, and select the Meta Ray-Ban Display Simulator extension in the Chrome toolbar. Chrome preview is not available when your app runs on a physical iPhone. When the preview ends, stop the server and disable Mock Device Kit: ```swift await MockDeviceKit.shared.stopTestServer() await MockDeviceKit.shared.disable() ``` ### Android Start a simulated display and the preview server: ```kotlin import android.content.Context import com.meta.wearable.dat.mockdevice.MockDeviceKit import com.meta.wearable.dat.mockdevice.api.GlassesModel import com.meta.wearable.dat.mockdevice.api.MockDeviceKitError private const val DISPLAY_PREVIEW_PORT = 9000 fun startChromeDisplayPreview( context: Context, onFailure: (MockDeviceKitError) -> Unit, ) { val mockDeviceKit = MockDeviceKit.getInstance(context.applicationContext) mockDeviceKit.enable() mockDeviceKit .pairGlasses(GlassesModel.META_RAYBAN_DISPLAY) .onSuccess { mockGlasses -> mockGlasses.powerOn() mockGlasses.don() mockDeviceKit .startTestServer(DISPLAY_PREVIEW_PORT) .onFailure { error, _ -> onFailure(error) } } .onFailure { error, _ -> onFailure(error) } } ``` Continue with the `DeviceSession` and content flow described in [Integrate Display capability into your Android app](/docs/develop/dat/display-android/). Content sent through the Display capability appears in the browser preview. Run `adb forward tcp:9000 tcp:9000` on your computer, open `http://127.0.0.1:9000/` in Chrome, and select the Meta Ray-Ban Display Simulator extension in the Chrome toolbar. This works with an Android emulator or an Android device connected through Android Debug Bridge (ADB). When the preview ends, stop the server and disable Mock Device Kit: ```kotlin val mockDeviceKit = MockDeviceKit.getInstance(context.applicationContext) mockDeviceKit.stopTestServer() mockDeviceKit.disable() ``` ### Test the experience The Meta Ray-Ban Display Simulator extension provides: - **600×600 pixel display frame**: Preview the display resolution with an optional frame overlay and additive blending. - **Environment backgrounds**: Test content against built-in scenes, a custom image, an animated background, or a live webcam view. - **Directional input**: Use the on-screen directional controls and Select button to test focus and selection. Selections are routed to your app's Display interaction handlers. - **Display settings**: Adjust content brightness, background brightness, background blur, and auto-dimming. - **Viewport recorder**: Record the simulator viewport as a WebM video for demos or bug reports. ![Meta Ray-Ban Display Simulator](/images/display-simulator-chrome-extension-dat.png) ## Simulate in your app You can embed the simulated display in your app without starting the preview server or installing the Chrome extension. Pair and power on a Meta Ray-Ban Display mock device, then simulate wearing it as shown in the platform examples above. Add the mock device's preview view to a square container in your app. ### iOS Add the preview view returned by `createPreviewView()` to a `UIView` in your interface: ```swift import MWDATMockDevice import UIKit @MainActor func addDisplayPreview( from mockGlasses: any MockGlasses, to previewContainer: UIView ) { let previewView = mockGlasses.services.display.createPreviewView() previewContainer.addSubview(previewView) previewView.frame = previewContainer.bounds previewView.autoresizingMask = [.flexibleWidth, .flexibleHeight] } ``` ### Android Add the preview view returned by `createPreviewView()` to a `ViewGroup` in your interface: ```kotlin import android.view.ViewGroup import com.meta.wearable.dat.mockdevice.api.MockGlasses fun addDisplayPreview(mockGlasses: MockGlasses, previewContainer: ViewGroup) { val previewView = mockGlasses.services.display.createPreviewView(previewContainer.context) previewContainer.addView( previewView, ViewGroup.LayoutParams( ViewGroup.LayoutParams.MATCH_PARENT, ViewGroup.LayoutParams.MATCH_PARENT, ), ) } ``` ## Test your integration The preview view is transparent, so you can render your own background behind it. Display content updates appear in the view, and taps are sent to your Display interaction handlers. If you remove the preview view from its container, create a new preview view before attaching it again. Both simulation options are intended for development and visual testing. Do not include the preview view in production builds. Before releasing your integration, test it on physical glasses to verify device connectivity, performance, and how the content appears in the real world. ### Display icons ## Overview Use the `Icon` component to render a system icon from the built-in catalog. Reference an icon by its enum value on the platform you are targeting — the SDK serializes the icon name to the device, which renders it using the on-device icon set. ## Reference | Icon | iOS | Android | |------|-----|---------| | ![airplane](/images/icons/airplane.png){:width="32" style="filter: invert(1)"} | .airplane | IconName.AIRPLANE | | ![arrow-down-shallow-u](/images/icons/arrow-down-shallow-u.png){:width="32" style="filter: invert(1)"} | .arrowDownShallowU | IconName.ARROW_DOWN_SHALLOW_U | | ![arrow-left](/images/icons/arrow-left.png){:width="32" style="filter: invert(1)"} | .arrowLeft | IconName.ARROW_LEFT | | ![arrow-right](/images/icons/arrow-right.png){:width="32" style="filter: invert(1)"} | .arrowRight | IconName.ARROW_RIGHT | | ![arrow-u-left](/images/icons/arrow-u-left.png){:width="32" style="filter: invert(1)"} | .arrowULeft | IconName.ARROW_U_LEFT | | ![arrow-up-shallow-u](/images/icons/arrow-up-shallow-u.png){:width="32" style="filter: invert(1)"} | .arrowUpShallowU | IconName.ARROW_UP_SHALLOW_U | | ![avatar](/images/icons/avatar.png){:width="32" style="filter: invert(1)"} | .avatar | IconName.AVATAR | | ![avatar-off](/images/icons/avatar-off.png){:width="32" style="filter: invert(1)"} | .avatarOff | IconName.AVATAR_OFF | | ![bed-side](/images/icons/bed-side.png){:width="32" style="filter: invert(1)"} | .bedSide | IconName.BED_SIDE | | ![bell](/images/icons/bell.png){:width="32" style="filter: invert(1)"} | .bell | IconName.BELL | | ![bell-diagonal-right-dot](/images/icons/bell-diagonal-right-dot.png){:width="32" style="filter: invert(1)"} | .bellDiagonalRightDot | IconName.BELL_DIAGONAL_RIGHT_DOT | | ![bell-off](/images/icons/bell-off.png){:width="32" style="filter: invert(1)"} | .bellOff | IconName.BELL_OFF | | ![bike-share](/images/icons/bike-share.png){:width="32" style="filter: invert(1)"} | .bikeShare | IconName.BIKE_SHARE | | ![bug](/images/icons/bug.png){:width="32" style="filter: invert(1)"} | .bug | IconName.BUG | | ![bullhorn](/images/icons/bullhorn.png){:width="32" style="filter: invert(1)"} | .bullhorn | IconName.BULLHORN | | ![bus](/images/icons/bus.png){:width="32" style="filter: invert(1)"} | .bus | IconName.BUS | | ![calendar](/images/icons/calendar.png){:width="32" style="filter: invert(1)"} | .calendar | IconName.CALENDAR | | ![campfire](/images/icons/campfire.png){:width="32" style="filter: invert(1)"} | .campfire | IconName.CAMPFIRE | | ![car-front-view](/images/icons/car-front-view.png){:width="32" style="filter: invert(1)"} | .carFrontView | IconName.CAR_FRONT_VIEW | | ![caret-down](/images/icons/caret-down.png){:width="32" style="filter: invert(1)"} | .caretDown | IconName.CARET_DOWN | | ![caret-left](/images/icons/caret-left.png){:width="32" style="filter: invert(1)"} | .caretLeft | IconName.CARET_LEFT | | ![caret-right](/images/icons/caret-right.png){:width="32" style="filter: invert(1)"} | .caretRight | IconName.CARET_RIGHT | | ![caret-up](/images/icons/caret-up.png){:width="32" style="filter: invert(1)"} | .caretUp | IconName.CARET_UP | | ![cart](/images/icons/cart.png){:width="32" style="filter: invert(1)"} | .cart | IconName.CART | | ![checkmark](/images/icons/checkmark.png){:width="32" style="filter: invert(1)"} | .checkmark | IconName.CHECKMARK | | ![checkmark-circle](/images/icons/checkmark-circle.png){:width="32" style="filter: invert(1)"} | .checkmarkCircle | IconName.CHECKMARK_CIRCLE | | ![circle-8-rays-large](/images/icons/circle-8-rays-large.png){:width="32" style="filter: invert(1)"} | .circle8RaysLarge | IconName.CIRCLE_8_RAYS_LARGE | | ![circle-handle](/images/icons/circle-handle.png){:width="32" style="filter: invert(1)"} | .circleHandle | IconName.CIRCLE_HANDLE | | ![clock](/images/icons/clock.png){:width="32" style="filter: invert(1)"} | .clock | IconName.CLOCK | | ![cloud](/images/icons/cloud.png){:width="32" style="filter: invert(1)"} | .cloud | IconName.CLOUD | | ![cloud-crescent-moon](/images/icons/cloud-crescent-moon.png){:width="32" style="filter: invert(1)"} | .cloudCrescentMoon | IconName.CLOUD_CRESCENT_MOON | | ![cloud-dot-four-rays](/images/icons/cloud-dot-four-rays.png){:width="32" style="filter: invert(1)"} | .cloudDotFourRays | IconName.CLOUD_DOT_FOUR_RAYS | | ![cloud-five-dashes](/images/icons/cloud-five-dashes.png){:width="32" style="filter: invert(1)"} | .cloudFiveDashes | IconName.CLOUD_FIVE_DASHES | | ![cloud-hook-swirl](/images/icons/cloud-hook-swirl.png){:width="32" style="filter: invert(1)"} | .cloudHookSwirl | IconName.CLOUD_HOOK_SWIRL | | ![cloud-lightning](/images/icons/cloud-lightning.png){:width="32" style="filter: invert(1)"} | .cloudLightning | IconName.CLOUD_LIGHTNING | | ![cocktail-glass](/images/icons/cocktail-glass.png){:width="32" style="filter: invert(1)"} | .cocktailGlass | IconName.COCKTAIL_GLASS | | ![code](/images/icons/code.png){:width="32" style="filter: invert(1)"} | .code | IconName.CODE | | ![coffee-cup](/images/icons/coffee-cup.png){:width="32" style="filter: invert(1)"} | .coffeeCup | IconName.COFFEE_CUP | | ![compass-north-up-red](/images/icons/compass-north-up-red.png){:width="32" style="filter: invert(1) hue-rotate(180deg)"} | .compassNorthUpRed | IconName.COMPASS_NORTH_UP_RED | | ![container-with-lid](/images/icons/container-with-lid.png){:width="32" style="filter: invert(1)"} | .containerWithLid | IconName.CONTAINER_WITH_LID | | ![cross-briefcase](/images/icons/cross-briefcase.png){:width="32" style="filter: invert(1)"} | .crossBriefcase | IconName.CROSS_BRIEFCASE | | ![dropper](/images/icons/dropper.png){:width="32" style="filter: invert(1)"} | .dropper | IconName.DROPPER | | ![envelope-open](/images/icons/envelope-open.png){:width="32" style="filter: invert(1)"} | .envelopeOpen | IconName.ENVELOPE_OPEN | | ![exclamation-circle](/images/icons/exclamation-circle.png){:width="32" style="filter: invert(1)"} | .exclamationCircle | IconName.EXCLAMATION_CIRCLE | | ![exclamation-triangle](/images/icons/exclamation-triangle.png){:width="32" style="filter: invert(1)"} | .exclamationTriangle | IconName.EXCLAMATION_TRIANGLE | | ![eye](/images/icons/eye.png){:width="32" style="filter: invert(1)"} | .eye | IconName.EYE | | ![fork-knife](/images/icons/fork-knife.png){:width="32" style="filter: invert(1)"} | .forkKnife | IconName.FORK_KNIFE | | ![four-arcs-up-filled](/images/icons/four-arcs-up-filled.png){:width="32" style="filter: invert(1)"} | .fourArcsUpFilled | IconName.FOUR_ARCS_UP_FILLED | | ![four-arcs-up-grayscale](/images/icons/four-arcs-up-grayscale.png){:width="32" style="filter: invert(1)"} | .fourArcsUpGrayscale | IconName.FOUR_ARCS_UP_GRAYSCALE | | ![four-corner-frame](/images/icons/four-corner-frame.png){:width="32" style="filter: invert(1)"} | .fourCornerFrame | IconName.FOUR_CORNER_FRAME | | ![gear](/images/icons/gear.png){:width="32" style="filter: invert(1)"} | .gear | IconName.GEAR | | ![globe-western-hemisphere](/images/icons/globe-western-hemisphere.png){:width="32" style="filter: invert(1)"} | .globeWesternHemisphere | IconName.GLOBE_WESTERN_HEMISPHERE | | ![graduation-cap](/images/icons/graduation-cap.png){:width="32" style="filter: invert(1)"} | .graduationCap | IconName.GRADUATION_CAP | | ![hashtag](/images/icons/hashtag.png){:width="32" style="filter: invert(1)"} | .hashtag | IconName.HASHTAG | | ![headphones](/images/icons/headphones.png){:width="32" style="filter: invert(1)"} | .headphones | IconName.HEADPHONES | | ![heart](/images/icons/heart.png){:width="32" style="filter: invert(1)"} | .heart | IconName.HEART | | ![house](/images/icons/house.png){:width="32" style="filter: invert(1)"} | .house | IconName.HOUSE | | ![i-circle](/images/icons/i-circle.png){:width="32" style="filter: invert(1)"} | .iCircle | IconName.I_CIRCLE | | ![light-bulb](/images/icons/light-bulb.png){:width="32" style="filter: invert(1)"} | .lightBulb | IconName.LIGHT_BULB | | ![magic-wand](/images/icons/magic-wand.png){:width="32" style="filter: invert(1)"} | .magicWand | IconName.MAGIC_WAND | | ![meta-ai](/images/icons/meta-ai.png){:width="32" style="filter: invert(1)"} | .metaAi | IconName.META_AI | | ![mountain-square](/images/icons/mountain-square.png){:width="32" style="filter: invert(1)"} | .mountainSquare | IconName.MOUNTAIN_SQUARE | | ![mountain-square-stacked](/images/icons/mountain-square-stacked.png){:width="32" style="filter: invert(1)"} | .mountainSquareStacked | IconName.MOUNTAIN_SQUARE_STACKED | | ![museum-building](/images/icons/museum-building.png){:width="32" style="filter: invert(1)"} | .museumBuilding | IconName.MUSEUM_BUILDING | | ![music-note](/images/icons/music-note.png){:width="32" style="filter: invert(1)"} | .musicNote | IconName.MUSIC_NOTE | | ![nine-squares-grid](/images/icons/nine-squares-grid.png){:width="32" style="filter: invert(1)"} | .nineSquaresGrid | IconName.NINE_SQUARES_GRID | | ![padlock-closed](/images/icons/padlock-closed.png){:width="32" style="filter: invert(1)"} | .padlockClosed | IconName.PADLOCK_CLOSED | | ![padlock-open](/images/icons/padlock-open.png){:width="32" style="filter: invert(1)"} | .padlockOpen | IconName.PADLOCK_OPEN | | ![palette](/images/icons/palette.png){:width="32" style="filter: invert(1)"} | .palette | IconName.PALETTE | | ![paper-airplane](/images/icons/paper-airplane.png){:width="32" style="filter: invert(1)"} | .paperAirplane | IconName.PAPER_AIRPLANE | | ![pencil](/images/icons/pencil.png){:width="32" style="filter: invert(1)"} | .pencil | IconName.PENCIL | | ![pencil-square](/images/icons/pencil-square.png){:width="32" style="filter: invert(1)"} | .pencilSquare | IconName.PENCIL_SQUARE | | ![person](/images/icons/person.png){:width="32" style="filter: invert(1)"} | .person | IconName.PERSON | | ![person-circle](/images/icons/person-circle.png){:width="32" style="filter: invert(1)"} | .personCircle | IconName.PERSON_CIRCLE | | ![phone](/images/icons/phone.png){:width="32" style="filter: invert(1)"} | .phone | IconName.PHONE | | ![phone-handset-arrow-down-left](/images/icons/phone-handset-arrow-down-left.png){:width="32" style="filter: invert(1)"} | .phoneHandsetArrowDownLeft | IconName.PHONE_HANDSET_ARROW_DOWN_LEFT | | ![phone-handset-arrow-up-right](/images/icons/phone-handset-arrow-up-right.png){:width="32" style="filter: invert(1)"} | .phoneHandsetArrowUpRight | IconName.PHONE_HANDSET_ARROW_UP_RIGHT | | ![phone-slash](/images/icons/phone-slash.png){:width="32" style="filter: invert(1)"} | .phoneSlash | IconName.PHONE_SLASH | | ![pizza-slice](/images/icons/pizza-slice.png){:width="32" style="filter: invert(1)"} | .pizzaSlice | IconName.PIZZA_SLICE | | ![plus](/images/icons/plus.png){:width="32" style="filter: invert(1)"} | .plus | IconName.PLUS | | ![plus-circle](/images/icons/plus-circle.png){:width="32" style="filter: invert(1)"} | .plusCircle | IconName.PLUS_CIRCLE | | ![shopping-bag](/images/icons/shopping-bag.png){:width="32" style="filter: invert(1)"} | .shoppingBag | IconName.SHOPPING_BAG | | ![sliders-horizontal](/images/icons/sliders-horizontal.png){:width="32" style="filter: invert(1)"} | .slidersHorizontal | IconName.SLIDERS_HORIZONTAL | | ![smart-glasses](/images/icons/smart-glasses.png){:width="32" style="filter: invert(1)"} | .smartGlasses | IconName.SMART_GLASSES | | ![smiley-circle](/images/icons/smiley-circle.png){:width="32" style="filter: invert(1)"} | .smileyCircle | IconName.SMILEY_CIRCLE | | ![speaker-off](/images/icons/speaker-off.png){:width="32" style="filter: invert(1)"} | .speakerOff | IconName.SPEAKER_OFF | | ![speaker-with-one-arc](/images/icons/speaker-with-one-arc.png){:width="32" style="filter: invert(1)"} | .speakerWithOneArc | IconName.SPEAKER_WITH_ONE_ARC | | ![speaker-with-three-arcs](/images/icons/speaker-with-three-arcs.png){:width="32" style="filter: invert(1)"} | .speakerWithThreeArcs | IconName.SPEAKER_WITH_THREE_ARCS | | ![speaker-with-two-arcs](/images/icons/speaker-with-two-arcs.png){:width="32" style="filter: invert(1)"} | .speakerWithTwoArcs | IconName.SPEAKER_WITH_TWO_ARCS | | ![speech-bubble](/images/icons/speech-bubble.png){:width="32" style="filter: invert(1)"} | .speechBubble | IconName.SPEECH_BUBBLE | | ![speech-bubble-off](/images/icons/speech-bubble-off.png){:width="32" style="filter: invert(1)"} | .speechBubbleOff | IconName.SPEECH_BUBBLE_OFF | | ![stadium](/images/icons/stadium.png){:width="32" style="filter: invert(1)"} | .stadium | IconName.STADIUM | | ![star](/images/icons/star.png){:width="32" style="filter: invert(1)"} | .star | IconName.STAR | | ![star-circle-triangle-ai](/images/icons/star-circle-triangle-ai.png){:width="32" style="filter: invert(1)"} | .starCircleTriangleAi | IconName.STAR_CIRCLE_TRIANGLE_AI | | ![taxi](/images/icons/taxi.png){:width="32" style="filter: invert(1)"} | .taxi | IconName.TAXI | | ![three-dot-speech-bubble](/images/icons/three-dot-speech-bubble.png){:width="32" style="filter: invert(1)"} | .threeDotSpeechBubble | IconName.THREE_DOT_SPEECH_BUBBLE | | ![three-dots-horizontal](/images/icons/three-dots-horizontal.png){:width="32" style="filter: invert(1)"} | .threeDotsHorizontal | IconName.THREE_DOTS_HORIZONTAL | | ![three-horizontal-lines](/images/icons/three-horizontal-lines.png){:width="32" style="filter: invert(1)"} | .threeHorizontalLines | IconName.THREE_HORIZONTAL_LINES | | ![three-horizontal-lines-stacked-descending](/images/icons/three-horizontal-lines-stacked-descending.png){:width="32" style="filter: invert(1)"} | .threeHorizontalLinesStackedDescending | IconName.THREE_HORIZONTAL_LINES_STACKED_DESCENDING | | ![three-people-overlapping](/images/icons/three-people-overlapping.png){:width="32" style="filter: invert(1)"} | .threePeopleOverlapping | IconName.THREE_PEOPLE_OVERLAPPING | | ![train](/images/icons/train.png){:width="32" style="filter: invert(1)"} | .train | IconName.TRAIN | | ![tree](/images/icons/tree.png){:width="32" style="filter: invert(1)"} | .tree | IconName.TREE | | ![triangle-left-vertical-line](/images/icons/triangle-left-vertical-line.png){:width="32" style="filter: invert(1)"} | .triangleLeftVerticalLine | IconName.TRIANGLE_LEFT_VERTICAL_LINE | | ![triangle-right](/images/icons/triangle-right.png){:width="32" style="filter: invert(1)"} | .triangleRight | IconName.TRIANGLE_RIGHT | | ![triangle-right-circle](/images/icons/triangle-right-circle.png){:width="32" style="filter: invert(1)"} | .triangleRightCircle | IconName.TRIANGLE_RIGHT_CIRCLE | | ![triangle-right-vertical-line](/images/icons/triangle-right-vertical-line.png){:width="32" style="filter: invert(1)"} | .triangleRightVerticalLine | IconName.TRIANGLE_RIGHT_VERTICAL_LINE | | ![two-arrows-clockwise](/images/icons/two-arrows-clockwise.png){:width="32" style="filter: invert(1)"} | .twoArrowsClockwise | IconName.TWO_ARROWS_CLOCKWISE | | ![two-lines-parallel](/images/icons/two-lines-parallel.png){:width="32" style="filter: invert(1)"} | .twoLinesParallel | IconName.TWO_LINES_PARALLEL | | ![two-squares-stacked-right-down](/images/icons/two-squares-stacked-right-down.png){:width="32" style="filter: invert(1)"} | .twoSquaresStackedRightDown | IconName.TWO_SQUARES_STACKED_RIGHT_DOWN | | ![two-triangles-left](/images/icons/two-triangles-left.png){:width="32" style="filter: invert(1)"} | .twoTrianglesLeft | IconName.TWO_TRIANGLES_LEFT | | ![two-triangles-right](/images/icons/two-triangles-right.png){:width="32" style="filter: invert(1)"} | .twoTrianglesRight | IconName.TWO_TRIANGLES_RIGHT | | ![video-camera](/images/icons/video-camera.png){:width="32" style="filter: invert(1)"} | .videoCamera | IconName.VIDEO_CAMERA | | ![video-camera-off](/images/icons/video-camera-off.png){:width="32" style="filter: invert(1)"} | .videoCameraOff | IconName.VIDEO_CAMERA_OFF | | ![wristband](/images/icons/wristband.png){:width="32" style="filter: invert(1)"} | .wristband | IconName.WRISTBAND | | ![wristband-slash](/images/icons/wristband-slash.png){:width="32" style="filter: invert(1)"} | .wristbandSlash | IconName.WRISTBAND_SLASH | | ![x](/images/icons/x.png){:width="32" style="filter: invert(1)"} | .x | IconName.X | ### Session lifecycle ## Overview The Wearables Device Access Toolkit runs inside sessions. AI glasses expose two experience types: - **Device sessions** grant sustained access to device sensors and outputs. - **Transactions** are short, system-owned interactions (for example, notifications or "Hey Meta"). When your app requests a device session, the glasses grant or revoke access as needed, the app observes state, and the system decides when to change it. ## Device session states `DeviceSessionState` is device-driven and delivered asynchronously. On Android, observe the state using [`state`](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_core_session_devicesession#state). On iOS, use [`stateStream()`](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest/mwdatcore_devicesession#statestream). | State | Meaning | App expectation | |--------------------|--------------------------------------------|---------------------------------------| | `STOPPED` | Session is inactive and not reconnecting. | Free resources. Wait for user action. | | `STARTED` | Session is active and streaming data. | Perform live work. | | `PAUSED` | Session is temporarily suspended. | Hold work. Paths may resume. | **Note:**  `DeviceSessionState` does not expose the reason for a transition. Observe the `DeviceSessionState` and react without assuming the cause of a change. **Android** ```kotlin val session = Wearables.createSession(AutoDeviceSelector()).getOrThrow() session.state.collect { state -> when (state) { DeviceSessionState.STARTED -> onStarted() DeviceSessionState.PAUSED -> onPaused() DeviceSessionState.STOPPED -> onStopped() else -> {} } } ``` **iOS** ```swift let session = try wearables.createSession(deviceSelector: AutoDeviceSelector(wearables: wearables)) for await state in session.stateStream() { switch state { case .started: onStarted() case .paused: onPaused() case .stopped: onStopped() default: break } } ``` Recommended reactions: - On `STARTED`, confirm UI shows that the device session is live. - On `PAUSED`, keep the connection and wait for `STARTED` or `STOPPED`. - On `STOPPED`, release device resources and allow the user to restart. ## Common device session transitions The device can change `DeviceSessionState` when: - The user performs a system action that opens another experience. - Another app or system feature starts a device session. - The user removes or folds the glasses, disconnecting Bluetooth. - The user removes the app from the Meta AI app. - Connectivity between the Meta AI app and the glasses drops. Many events lead to `STOPPED`, while some input interactions pause a session and later resume it. ## Pause and resume When `DeviceSessionState` changes to `PAUSED`: - The device keeps the connection alive. - Streams stop delivering data while paused. - The device resumes streaming by returning to `STARTED`. Your app should not attempt to restart a device session while it is paused. ## Device availability Use device metadata to detect availability. Hinge position is not exposed, but it influences connectivity. Again, for an Android integration: **Android** ```kotlin Wearables.devicesMetadata[deviceId]?.collect { metadata -> if (metadata.available) { onDeviceAvailable() } else { onDeviceUnavailable() } } ``` **iOS** ```swift let token = Wearables.shared.deviceForIdentifier(deviceId).addLinkStateListener { linkState in if linkState == .connected { onDeviceAvailable() } else { onDeviceUnavailable() } } ``` Expected effects: - Closing the hinges disconnects Bluetooth, stops active streams, and forces `DeviceSessionState` to `STOPPED`. - Opening the hinges restores Bluetooth when the glasses are nearby, but does not restart the device session. Start a new session after the device becomes available/connected. ## Implementation checklist - Subscribe to `session.state`/`stateStream` and handle all `DeviceSessionState` values. - Monitor device availability before starting work. - Release resources only after receiving `STOPPED` or loss of availability. - Rely only on observable state rather than inferring transition causes. ### Permissions and registration ## Overview The Wearables Device Access Toolkit separates app registration and device permissions. All permission grants occur through the Meta AI app. Permissions work across multiple linked devices. Camera permissions are granted at the app level. However, each device will need to confirm permissions specifically, in turn allowing your app to support a set of devices with individual permissions. To create an integration, follow this guidance to build your first integration for [Android](/docs/develop/dat/build-integration-android/) or [iOS](/docs/develop/dat/build-integration-ios/). ## Registration Your app registers with the Meta AI app to be a permitted integration. This establishes the connection between your app and the glasses platform. Registration happens once through the Meta AI app with glasses connected. Users see your app name in the list of connected apps. They can unregister anytime through the Meta AI app. You can also implement an unregistration flow if desired. ## Device permissions After registration, request specific permissions (see possible values for [Android](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_core_types_permission#enumeration_constants) and [iOS](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest/mwdatcore_permission#enumeration_constants)). The Meta AI app runs the permission grant flow. Users choose **Allow once** (temporary) or **Allow always** (persistent). ### User experience flow ![Illustrating the user experience flow for permissions and using features.](/images/wearables-permissions-request-1.png) - Without registration, permission requests fail. - With registration but no permissions, your app connects but cannot access the camera. ## Multi-device permission behavior Users can link multiple glasses to Meta AI. The toolkit handles this transparently. ### How it works Users can have multiple pairs of glasses. Permission granted on any linked device allows your app to use that feature. When checking permissions, Wearables Device Access Toolkit queries all connected devices. If any device has the permission granted, your app receives "granted" status. ### Practical implications You don't track which specific device has permissions. Permission checks return granted if _any_ connected device has approved. If all devices disconnect, permission checks will indicate unavailability. Users manage permissions per device in the Meta AI app. ## Distribution and registration Testing vs. production have different permission requirements. When developer mode is activated, registration is always allowed. When a build is distributed, users must be in the proper release channel to get the app. This is controlled by the `MWDAT` application ID. **Note:** For security purposes, only one 3rd party app can remain registered at a time in Developer Mode. Registering a new app will automatically unregister any previously registered app. - For setting up developer mode, see [Getting started with the Wearables Device Access Toolkit](/docs/develop/dat/getting-started-toolkit/). - For details on creating release channels, see [Manage projects in Developer Center](/docs/develop/dat/manage-projects/). - This page also explains where to find the `APPLICATION_ID` that must be added to your production manifest/bundle configuration. ### Use device microphones and speakers ## Overview Device audio uses two Bluetooth profiles: | Profile | Direction | Quality | Use case | |---------|-----------|---------|----------| | **A2DP** (Advanced Audio Distribution Profile) | Output only | High quality (44.1/48 kHz stereo) | Music, media playback, TTS | | **HFP** (Hands-Free Profile) | Bidirectional | 8 kHz mono | Voice capture from the glasses microphone | In HFP mode, the wearable's microphones use beamforming to isolate the wearer's voice, which significantly reduces the volume of ambient sounds and other speakers. This is expected behavior, not a bug. A camera stream can also deliver audio from the glasses microphones in-band with video, without using either Bluetooth profile for capture. See [Capture audio while streaming from the camera](#capture-audio-while-streaming-from-the-camera). In this case, audio is not beamformed down to the wearer's voice, so ambient sounds and noise are present at their natural levels. ## Choose a profile Use **A2DP** when you only need to play audio to the glasses, since it provides significantly higher fidelity than HFP. Use **HFP** when you need microphone input from the wearer. The two profiles are mutually exclusive: activating HFP switches the glasses away from A2DP, and audio output quality drops to 8 kHz mono for the duration of the session. Wearables Device Access Toolkit sessions share microphone and speaker access with the system Bluetooth stack on the glasses. ## Capture audio while streaming from the camera **Beta:** In-stream audio capture is available in the development and beta [release channels](/docs/develop/dat/set-up-release-channels/) only. It is not supported in production release channels, and the API may change before general availability. A camera stream can deliver audio from the glasses microphones alongside video, without putting the glasses into HFP for capture. Because this path does not use HFP, output stays on A2DP, so you do not trade output quality for microphone input the way the two profiles otherwise force you to. Use it when you want audio that matches the video you are already receiving. Audio is off by default, and is enabled when you set an audio codec on the stream configuration. You can configure it with the following parameters, while bits per sample is set to 16. | Property | Value | |----------|-------| | Encoding delivered to your app | Uncompressed PCM | | Sample rates | 16,000 Hz, 44,100 Hz, or 48,000 Hz | | Channels | You choose the channel count when you configure the codec | To save bandwidth on the link to the phone and more battery on the glasses pick the lowest rate and channels that satisfy your feature. | Rate | Typical use | |------|-------------| | 16,000 Hz | Speech and voice, including passing audio to a transcription service. | | 44,100 Hz | General-purpose recording. | | 48,000 Hz | Highest fidelity. | The provided audio frames arrive with a timestamp that can be used for synchronization. ### Android Set an audio codec on the stream configuration, then collect `audioStream` alongside the video stream. ```kotlin import com.meta.wearable.dat.camera.types.AudioCodec import com.meta.wearable.dat.camera.types.AudioSampleRate import com.meta.wearable.dat.camera.types.StreamConfiguration import com.meta.wearable.dat.camera.types.VideoQuality val configuration = StreamConfiguration( audioCodec = AudioCodec.PCM( sampleRate = AudioSampleRate.RATE_16000, numberOfChannels = 1, ), videoQuality = VideoQuality.MEDIUM, frameRate = 24 ) ``` Leaving `audioCodec` unset, or setting it to `null`, streams video only. ```kotlin scope.launch { stream.audioStream.collect { frame -> // frame.buffer holds the PCM bytes; presentationTimeUs is in microseconds. handleAudio(frame.buffer, frame.presentationTimeUs) } } stream.start() ``` You can watch StreamState to know when audio data will be started or stopped. ### iOS Use the `StreamConfiguration` initializer that takes an audio codec, then listen on `audioFramePublisher`. ```swift import MWDATCamera let configuration = StreamConfiguration( videoCodec: .raw, audioCodec: .pcm(sampleRate: .rate16000, numberOfChannels: 1), resolution: .medium, frameRate: 24 ) ``` The initializers that do not take an audio codec will stream video only. ```swift let audioToken = stream.audioFramePublisher.listen { frame in // frame.pcmBuffer holds Float32 samples; presentationTimeStamp aligns with video. handleAudio(frame.pcmBuffer, frame.presentationTimeStamp) } stream.start() ``` Retain the token for as long as you want frames and release it during teardown. ## Android ### HFP (bidirectional) — Capture audio from the glasses microphone Select the glasses as the communication device, then switch the audio manager into communication mode. ```kotlin import android.content.Context import android.media.AudioDeviceInfo import android.media.AudioManager import android.os.Build private fun routeAudioToBluetooth(context: Context): Boolean { if (Build.VERSION.SDK_INT < Build.VERSION_CODES.S) { return false } val audioManager = context.getSystemService(AudioManager::class.java) // Get list of currently available devices val devices = audioManager.availableCommunicationDevices // User chooses one of the devices from the list. val userSelectedDeviceType = AudioDeviceInfo.TYPE_BLUETOOTH_SCO val selectedDevice = devices.firstOrNull { device -> device.type == userSelectedDeviceType } if (selectedDevice == null) { return false } audioManager.mode = AudioManager.MODE_IN_COMMUNICATION return audioManager.setCommunicationDevice(selectedDevice) } ``` ## iOS ### A2DP (output only) — Play audio to the glasses A2DP is the high-quality Bluetooth media route on iOS. Configure your app for playback, and the system can route output to paired glasses automatically. ```swift import AVFoundation let audioSession = AVAudioSession.sharedInstance() try audioSession.setCategory(.playback, mode: .default, options: []) try audioSession.setActive(true, options: .notifyOthersOnDeactivation) ``` Use `AVAudioPlayer`, `AVSpeechSynthesizer`, or any standard audio API to play audio, then verify the active route before assuming output is on the glasses. ```swift let player = try AVAudioPlayer(contentsOf: audioFileURL) player.play() ``` #### Verify the route ```swift let hasA2DPOutput = audioSession.currentRoute.outputs.contains { $0.portType == .bluetoothA2DP } ``` ### HFP (bidirectional) — Capture audio from the glasses microphone HFP requires more setup than A2DP. Use `.allowBluetoothHFP` for microphone capture; A2DP output options do not provide microphone access. If your SDK only exposes the older `.allowBluetooth` option, use that option for HFP. **Ordering constraint:** When using HFP with a DAT camera stream, the HFP microphone must be fully configured before the stream starts. The correct ordering is: 1. Add the DAT camera stream to the session. 2. Configure and start the HFP microphone. Wait for the route to settle. 3. Start the DAT camera stream. Starting the DAT stream before HFP is ready can cause the audio route to fail silently. #### Configure the audio session ```swift import AVFoundation // Request microphone permission let granted = await withCheckedContinuation { continuation in AVAudioApplication.requestRecordPermission { granted in continuation.resume(returning: granted) } } guard granted else { return } // Configure the session for HFP let audioSession = AVAudioSession.sharedInstance() try audioSession.setCategory(.playAndRecord, mode: .default, options: [.allowBluetoothHFP]) try audioSession.setActive(true, options: .notifyOthersOnDeactivation) ``` #### Select glasses as the preferred input The system may have multiple Bluetooth inputs available. Find the HFP port that corresponds to the glasses and set it as the preferred input. ```swift if let hfpInput = audioSession.availableInputs?.first(where: { $0.portType == .bluetoothHFP }) { try audioSession.setPreferredInput(hfpInput) } ``` #### Capture audio with `AVAudioEngine` Install a tap on the audio engine's input node to receive raw PCM buffers from the glasses microphone. ```swift let audioEngine = AVAudioEngine() let inputNode = audioEngine.inputNode let format = inputNode.inputFormat(forBus: 0) inputNode.installTap(onBus: 0, bufferSize: 1024, format: format) { buffer, _ in // Forward the buffer to your audio pipeline (e.g., LiveKit, a file writer, etc.) handleAudioBuffer(buffer) } audioEngine.prepare() try audioEngine.start() ``` #### Wait for the route to settle After starting the audio engine, the Bluetooth HFP route needs time to stabilize. After waiting, verify the route is active before starting the DAT stream. ```swift // Allow the Bluetooth HFP route to settle try await Task.sleep(nanoseconds: 2 * NSEC_PER_SEC) // Verify HFP is actually routed let hasHFPRoute = audioSession.currentRoute.inputs.contains { $0.portType == .bluetoothHFP } guard hasHFPRoute else { audioEngine.stop() try audioSession.setActive(false, options: .notifyOthersOnDeactivation) throw MyError.hfpRouteUnavailable } ``` #### Teardown When you're done capturing, call `removeTap(onBus:)` to stop receiving input buffers, then deactivate the session. If you need to return to A2DP playback, reconfigure the audio session with `.playback` category after deactivation. ```swift inputNode.removeTap(onBus: 0) audioEngine.stop() try audioSession.setActive(false, options: .notifyOthersOnDeactivation) ``` ## Next steps - Use [AVAudioSession](https://developer.apple.com/documentation/AVFAudio/AVAudioSession) for iOS audio routing. - Use [AudioManager](https://developer.android.com/reference/android/media/AudioManager) for Android audio routing. - Look up Wearables Device Access Toolkit types and members in the API reference for [Android](https://wearables.developer.meta.com/docs/reference/android/dat/latest) and [iOS](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest). ### Receive input events from glasses and Meta Neural Band ## Overview The Inputs capability delivers semantic input events from the wearer's AI glasses and band to your app. Use it to drive menu navigation, confirm and cancel actions, respond to the capture and action buttons, or track a continuous drag interaction. **Beta:** Inputs is a beta capability. It is available in the development and beta [release channels](/docs/develop/dat/set-up-release-channels/) only, and its API may change before general availability. Inputs reports interactions, not raw sensor signal. You receive a decoded event such as "navigate down" or "select" rather than touch coordinates or muscle activity. Type and value names below are given as `Android` / `iOS` where the two platforms spell them differently. Every event carries the surface that produced it and the time it occurred: | Field | Type | Notes | |-------|------|-------| | `source` | Enum | Which surface produced the event. See [Input sources](#input-sources). | | `timestampMs` | 64-bit integer | An on-device monotonic timestamp in milliseconds, recorded when the glasses forward the interaction. Use it for ordering and for measuring intervals within the same device boot, not as a wall-clock time, a raw hardware-event time, or a phone timestamp. | The capability delivers six event types: | Event (Android / iOS) | Extra fields | Meaning | |-------|--------------|---------| | `Nav` / `.nav` | `direction`: `UP`, `DOWN`, `LEFT`, `RIGHT` / `.up`, `.down`, `.left`, `.right` | Directional navigation. | | `Select` / `.select` | None | Confirm or activate the focused item. | | `Back` / `.back` | None | Dismiss or go back. Not currently delivered by real glasses. See [Current limitations](#current-limitations). | | `Button` / `.button` | `button`: `ACTION` / `.action` | The physical action button was pressed. | | `Capture` / `.capture` | `pressType`: `SHORT_PRESS`, `HOLD`, `DOUBLE_PRESS` / `.shortPress`, `.hold`, `.doublePress` | The physical capture button was pressed. | | `Drag` / `.drag` | `action`, `x`, `y`, `dx`, `dy` | One sample of a continuous drag stream. | A `Nav`, `Select`, or `Capture` interaction produces exactly one event, emitted when the press is released. Do not expect separate down and up events for those; only `Drag` reports a multi-sample interaction. The action button is also reported as a single `Button` event per press, but see [Current limitations](#current-limitations) for its hardware caveat. ### Drag coordinates A `Drag` event reports the phase of the interaction plus its position: | Field | Meaning | |-------|---------| | `action` | The phase of the drag: `DOWN` / `.down` begins the stream, `MOVE` / `.move` continues it, `UP` / `.up` ends it. A complete drag is one `DOWN`, zero or more `MOVE` samples, and one `UP`. | | `x`, `y` | The absolute position of this sample, in the device's input coordinate space. | | `dx`, `dy` | The change in position since the previous sample of the same drag, in the same units. | The SDK does not guarantee a particular origin, range, or axis direction for the coordinate space, so treat the values as relative to each other rather than as screen or pixel coordinates. ## Prerequisites - A working integration built from [iOS integrations](/docs/develop/dat/build-integration-ios/) or [Android integrations](/docs/develop/dat/build-integration-android/), including registration with the Meta AI app. - A device session in the started state. You cannot add Inputs to a session that has not started. Inputs does not require a user-facing permission request. There is no `Permission` value for input data, so you do not call `checkPermissionStatus` or `requestPermission` before using it. See [Permissions and registration](/docs/develop/dat/permissions-requests/) for the permissions that do require a user prompt. ## Input sources Each event names the surface it came from, so you can treat the same semantic action differently depending on where it originated. | Source (Android / iOS) | Surface | Events it produces | |--------|---------|--------------------| | `CAPTOUCH` / `.captouch` | The touchpad on the temple arm of the glasses. | `Nav`, `Select` | | `NEURAL_BAND` / `.neuralBand` | Discrete band inputs, such as a tap. | `Nav`, `Select` | | `CAPTURE_BUTTON` / `.captureButton` | The physical capture (camera) button. | `Capture` | | `ACTION_BUTTON` / `.actionButton` | The physical action button. | `Button` | | `NEURAL_BAND_DRAG` / `.neuralBandDrag` | The continuous band drag stream. | `Drag` | | `UNKNOWN` / `.unknown` | An unset or unrecognized surface. | Any | `UNKNOWN` is the SDK's decoding fallback, not a delivery guarantee: if an unset or unrecognized source reaches the SDK, it is represented as `UNKNOWN`. Because the default configuration requests the five named sources and does not include `UNKNOWN`, source filtering can discard an event from a future or unrecognized surface before the SDK decodes it. Handle the `UNKNOWN` case so your code stays exhaustive, but do not rely on it to receive events from surfaces this SDK version does not know about. Which physical input maps to which event is decided by the device, not by your app. A double tap on the touchpad arrives as `Select`, and a swipe arrives as `Nav` with the direction produced by the current device interaction mapping and settings. The wearer's device settings can invert horizontal and vertical swipe output, so a forward swipe does not always produce the same direction. Treat the exact mapping as device behavior that can differ across device models and settings. ### Interactions reserved by the device While a Meta Wearables Device Access Toolkit session is running and the AI glasses are not showing interactive display content, the AI glasses reserve two one-finger touchpad interactions as on-device session controls. Your app never receives them: | Interaction | Device behavior | |---------|-----------------| | Single tap with one finger | Pauses or resumes the device session. | | Tap and hold with one finger | Stops the device session. | Do not design an interaction around a single tap or tap and hold on the touchpad. Use a double tap to select, and use the capture button's `HOLD` / `.hold` press type if you need the button's hold action. Because tapping and holding with one finger stops the session, a user who taps and holds the temple arm ends your session rather than triggering an in-app action; handle the resulting session state change as you would any other external termination. See [Session lifecycle](/docs/develop/dat/lifecycle-events/). This reservation is scoped to that display state. When the Meta Ray-Ban Display glasses are showing interactive display content the interaction mapping is different, and mappings can differ across device models, so do not build a feature that depends on any particular touchpad interaction reaching your app. ## Choose which sources to capture Pass an `InputsConfiguration` to request specific sources. By default the capability requests every named source, so subscribe more narrowly when your feature only needs part of the surface area. | Setting | Default | Meaning | |---------|---------|---------| | `sources` | The five named sources; `UNKNOWN` is not included | The set of surfaces to capture. Events from a source you did not request are not delivered. An empty set (`emptySet()` / `[]`) is a wildcard meaning every available source. | | `consumeBack` | `true` | Whether a back interaction is delivered to your app instead of being handled by the system. Real glasses do not deliver `Back` yet. See [Current limitations](#current-limitations). | Requesting only what you need avoids consuming interactions that the system or on-glasses display would otherwise handle. ## Capability lifecycle Adding Inputs starts it: a successful call attaches Inputs and begins activation, and there is no separate `start()` call. Activation is asynchronous: an Android success result or an iOS return does not mean Inputs is already active. Wait for `ACTIVE` / `.active` before relying on event delivery or injecting Mock Device Kit events. | State (Android / iOS) | Meaning | |-------|---------| | `INACTIVE` / `.inactive` | Not running. This is the initial state and the resting state after the capability stops. | | `ACTIVATING` / `.activating` | Activation was requested but the device has not acknowledged it yet. | | `ACTIVE` / `.active` | Events are being delivered as they occur. | | `DEACTIVATING` / `.deactivating` | Teardown is in progress. | A clean removal winds down through `DEACTIVATING` to `INACTIVE`. An error, a device disconnect, or the wearer doffing the glasses also ends at `INACTIVE`. Because state alone does not tell you why the capability stopped, use the error stream to distinguish the cases. Removing the capability and doffing the glasses both leave the error value unset; a failure reports a reason. If activation is not acknowledged within 10 seconds, the capability reports an activation timeout and settles back to `INACTIVE`. The event stream buffers the most recent 64 events on both platforms and drops the oldest when a consumer falls behind. Keep per-event work small, and hand heavier processing off to another task or coroutine so you do not lose input. ## Error handling | Error | Meaning | |-------|---------| | Permission denied | Your app is not permitted to receive input events from the glasses. | | Connection closed | The connection delivering input events closed. | | Activation timeout | Activation was not acknowledged within 10 seconds. | | Activation failed | Activation was refused or is unsupported. Unlike a timeout, this is a definitive failure. | | Device disconnected | The glasses disconnected while Inputs was active. | | Communication error | A communication failure occurred on the input channel. | | Capability unavailable | iOS only. The capability is not available on this device. | For the enumerated values behind this table, see `InputsError` in the API reference for [Android](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_inputs_types_inputserror) and [iOS](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest/mwdatinputs_inputserror). Every error ends the current stream. There is no class of error you can recover from by immediately re-adding the capability, so repair the underlying condition first. A permission-denied error in particular will keep failing until the app is permitted to receive input events. Recovery then differs by platform: - **Android.** The SDK restores an attached Inputs instance itself after a session reconnect and clears the retained error, so you do not re-add it. Calling `addInputs` while an instance is still attached returns `CAPABILITY_ALREADY_ADDED`; call `removeInputs()` first if you deliberately want to replace the capability with a different configuration. - **iOS.** A published error stops delivery but leaves the capability attached to the session. Call `removeInputs()` before adding a replacement, or the next `addInputs` throws. ## Current limitations These are limitations of the current beta, not of the API shape. Write your event handling to cover every event type so your app keeps working as the gaps close. - **`Back` events are not currently delivered by real glasses.** The `Back` event type exists in the SDK, but the on-glasses delivery path is not wired yet, so no `Back` event reaches your app from real hardware. Mock Device Kit can inject `Back` and honors `consumeBack`, so you can write and test your back handling today. Until real glasses deliver it, also provide another way to dismiss or go back, such as a `Nav` target or a `Select` on an explicit back item. - **Volume interactions are not exposed.** The system owns volume, and volume changes are not delivered as input events. - **Action-button support is not validated on all hardware.** `Button` with `ACTION` / `.action` is the only button event, and physical action-button support has not been validated on every target device. Treat it as unconfirmed on hardware you have not tested, and do not assume a press-and-hold of the action button produces an event. ## Android ### Step 1: Add the dependency Add the inputs artifact alongside the core artifact in your version catalog, using the same version reference as the rest of the SDK. ```toml [libraries] mwdat-core = { group = "com.meta.wearable", name = "mwdat-core", version.ref = "mwdat" } mwdat-inputs = { group = "com.meta.wearable", name = "mwdat-inputs", version.ref = "mwdat" } ``` ```kotlin dependencies { implementation(libs.mwdat.core) implementation(libs.mwdat.inputs) } ``` ### Step 2: Add the capability to a started session Call `addInputs` on a started `DeviceSession`. It returns a `DatResult`, so handle the failure branch rather than assuming success. A successful call attaches Inputs and begins activation; wait for `ACTIVE` before relying on events. ```kotlin import com.meta.wearable.dat.inputs.Inputs import com.meta.wearable.dat.inputs.addInputs import com.meta.wearable.dat.inputs.types.InputSource import com.meta.wearable.dat.inputs.types.InputsConfiguration val configuration = InputsConfiguration( sources = setOf(InputSource.CAPTOUCH, InputSource.NEURAL_BAND), ) val result = session.addInputs(configuration) val inputs = result.getOrNull() if (inputs == null) { showError(result.errorOrNull()?.description.orEmpty()) return } ``` Unwrap through `getOrNull()` and `errorOrNull()`, or through `fold`/`onFailure`, to reach the typed `DeviceSessionError`. The single-argument `getOrElse` hands your lambda the underlying `Throwable`, which has no `description`. Omit the configuration to request every named source with back consumption enabled. ### Step 3: Collect events Collect `events`, `state`, and `errors` on a coroutine scope that you cancel during teardown. ```kotlin import com.meta.wearable.dat.inputs.types.InputEvent import com.meta.wearable.dat.inputs.types.NavDirection scope.launch { inputs.events.collect { event -> when (event) { is InputEvent.Nav -> when (event.direction) { NavDirection.UP -> moveFocusUp() NavDirection.DOWN -> moveFocusDown() NavDirection.LEFT -> moveFocusLeft() NavDirection.RIGHT -> moveFocusRight() } is InputEvent.Select -> activateFocusedItem() is InputEvent.Back -> dismiss() is InputEvent.Button -> onActionButton() is InputEvent.Capture -> onCapturePress(event.pressType) is InputEvent.Drag -> onDrag(event.action, event.x, event.y, event.dx, event.dy) } } } ``` `InputEvent` is a sealed class, so an exhaustive `when` gives you a compile error if a future SDK version adds an event type. Observe `state` and `errors` to know when events will flow and why they stopped. ```kotlin import com.meta.wearable.dat.inputs.types.InputsState scope.launch { inputs.state.collect { state -> if (state == InputsState.ACTIVE) onReceivingInput() else onNotReceivingInput() } } scope.launch { inputs.errors.collect { error -> if (error != null) showError(error.description) } } ``` `state` and `errors` are state flows, so a new collector immediately receives the current value. `errors` holds `null` until something goes wrong and retains the last error afterward, so treat a repeated value as the same error rather than a new one. After a session reconnect the SDK restarts Inputs and resets `errors` to `null`. ### Step 4: Remove the capability `removeInputs` is a separate extension function, so it needs its own import. ```kotlin import com.meta.wearable.dat.inputs.removeInputs session.removeInputs() ``` Removing the capability stops input streaming and releases the input surfaces on the glasses. Do it as soon as your feature no longer needs input, rather than waiting for the session to end. ### Test without hardware Mock Device Kit injects synthetic events over the same path a real device uses, so your event-handling code cannot tell them apart. Events are delivered only while Inputs is streaming and only for sources the session requested; other calls are dropped. ```kotlin val glasses = mockKit.pairGlasses(GlassesModel.RAYBAN_META).getOrThrow() // ...after your app has added the Inputs capability and it has reached ACTIVE... glasses.services.input.navDown() glasses.services.input.select() glasses.services.input.capture(CapturePressType.SHORT_PRESS) ``` Only the navigation, select, and back injections take a `source`, defaulting to `CAPTOUCH`, so you can simulate the same action arriving from the band. `capture()`, `button()`, and `drag()` take no source: their events always report the capture button, the action button, and the band drag stream respectively. See [Android testing](/docs/develop/dat/testing-mdk-android/) for how to create a mock device. ## iOS ### Step 1: Import the module The inputs module ships with the SDK Swift package you added in [iOS integrations](/docs/develop/dat/build-integration-ios/). Import it wherever you use the capability. ```swift import MWDATCore import MWDATInputs ``` ### Step 2: Add the capability to a started session `addInputs` returns `nil` if the session has not started, and throws if an Inputs capability is already attached. A successful return attaches Inputs and begins activation; wait for `.active` before relying on events. ```swift let configuration = InputsConfiguration(sources: [.captouch, .neuralBand]) guard let inputs = try session.addInputs(configuration: configuration) else { return } ``` ### Step 3: Consume events `addInputs` creates the `Inputs` object and starts activation before it returns, so register your listeners on the returned object immediately. Neither publisher replays, so an error published in the window before you register cannot be retrieved later; `inputs.state` always reports the current state. Retain the returned listener tokens for as long as you want callbacks. Releasing a token cancels its listener, so a token stored in a local variable inside a setup method stops observation as soon as that method returns. Keep them in a `ListenerTokenBag` held by the same object that owns the capability. ```swift private let listenerTokens = ListenerTokenBag() inputs.statePublisher.listen { state in Task { @MainActor in // Update your UI for .activating, .active, .deactivating, or .inactive } }.store(in: listenerTokens) inputs.errorPublisher.listen { error in Task { @MainActor in switch error { case .permissionDenied: showPermissionHelp() case .connectionClosed, .activationTimeout, .activationFailed, .capabilityUnavailable, .deviceDisconnected, .communicationError: showError(error) @unknown default: showError(error) } // Every error leaves the capability attached: remove it before adding a replacement. } } ``` `events` is an `AsyncStream`, so iterate it in a task that you retain and cancel during teardown. ```swift let events = inputs.events eventsTask = Task { for await event in events { switch event { case let .nav(direction, _, _): await moveFocus(direction) case .select: await activateFocusedItem() case .back: await dismiss() case .button: await onActionButton() case let .capture(pressType, _, _): await onCapturePress(pressType) case let .drag(action, x, y, dx, dy, _, _): await onDrag(action, x, y, dx, dy) @unknown default: break } } } ``` `InputEvent` and `InputsError` are public enums that are not `@frozen`, so add an `@unknown default` branch to every switch over them. Without it your code stops compiling when a future SDK version adds a case. Every case also exposes `source` and `timestampMs` directly on the event, so you can read them without unpacking the associated values. ### Step 4: Remove the capability ```swift eventsTask?.cancel() try session.removeInputs() ``` Call this after an error too: a failed capability stays attached until you remove it. ### Test without hardware Mock Device Kit exposes the same injection functions as Android through the mock glasses services. As on Android, only the navigation, select, and back injections take a source. ```swift mockGlasses.services.input.navDown() mockGlasses.services.input.select() mockGlasses.services.input.capture(pressType: .shortPress) ``` See [iOS testing](/docs/develop/dat/testing-mdk-ios/) for how to create a mock device. ## Mock Device Kit fidelity Mock Device Kit produces the same event types your app receives from real glasses and delivers them over the same path, so your event-handling code cannot tell them apart. It also applies the same source and `consumeBack` filters, so an injected event from a source you did not request, or a `Back` injection when `consumeBack` is `false`, is dropped exactly as it would be on device. It does not reproduce the device half of the pipeline. Input recognition, which physical surface a raw input is attributed to, which interactions the system reserves or passes through, duplicate-press handling, and per-model mappings are all device behavior that Mock Device Kit bypasses. Validate those on target hardware. Timestamps are the one place the two are visibly different: the mock kits do not share a clock domain with a real device, and the iOS mock kit uses Unix epoch milliseconds where the Android mock kit uses a boot-relative clock. Assert on event ordering and intervals in tests, never on absolute timestamp values. ## Platform differences Inputs behaves the same on both platforms except in these places: - **Adding the capability.** Android returns a `DatResult` you unwrap. iOS returns an optional that is `nil` when the session has not started, and throws when a capability is already attached. - **Event type.** Android models `InputEvent` as a sealed class with a data class per event, so an exhaustive `when` is checked at compile time. iOS models it as a non-frozen enum with associated values, so switches need an `@unknown default`. - **Error delivery.** Android exposes errors as a state flow that retains the most recent error, so a new collector sees a failure that already happened. iOS delivers errors as one-time events, so a listener attached after a failure does not see it. - **Error cases.** iOS adds a capability-unavailable error that has no Android equivalent. - **Recovery.** Android restores an attached instance after a session reconnect and clears the retained error. On iOS the capability stays attached after an error and must be removed before a replacement is added. ## Next steps - Understand how session state drives the capability in [Session lifecycle](/docs/develop/dat/lifecycle-events/). - Review what does and does not need a user prompt in [Permissions and registration](/docs/develop/dat/permissions-requests/). - Build a deterministic test setup with [Mock Device Kit basics](/docs/develop/dat/mock-device-kit/). - Ship to testers through [Manage versions and release channels](/docs/develop/dat/set-up-release-channels/). - Look up every type and member in the API reference for [Android](https://wearables.developer.meta.com/docs/reference/android/dat/latest) and [iOS](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest). ### Stream motion data from glasses sensors ## Overview The Motion capability streams inertial measurement unit (IMU) samples from the wearer's glasses to your app. Use it for head-orientation input, gesture and activity detection, image stabilization hints, or to align sensor data with a camera stream. **Beta:** Motion is a beta capability. It is available in the development and beta [release channels](/docs/develop/dat/set-up-release-channels/) only, and its API may change before general availability. The glasses emit a `MotionSample` each time a new accelerometer reading arrives. The sample carries that reading's timestamp together with the most recent gyroscope, magnetometer, and orientation values, which may be slightly older: | Field | Type | Unit | Notes | |-------|------|------|-------| | `timestampNs` | 64-bit integer | Nanoseconds | Device monotonic clock. Use it for ordering and for computing intervals, not as a wall-clock time. | | `accelerometer` | 3-component vector | m/s² | Includes gravity. A device at rest reads approximately 9.81 on the axis opposing gravity. Can be `null`. | | `gyroscope` | 3-component vector | rad/s | Angular velocity. Can be `null`. | | `magnetometer` | 3-component vector | µT | Not every device produces it, so this value can be `null`. | | `orientation` | Quaternion | — | Fused orientation, with `w` as the scalar component. Not every device produces it, so this value can be `null`. | | `source` | Enum | — | Which sensor package produced the sample: glasses, Meta Neural Band, or an unknown device. | `timestampNs` and `source` are always present. All four sensor readings are independently nullable: check each one you use, and do not discard a whole sample because a different reading is missing. A sample streamed from glasses carries an accelerometer reading in practice, since a new accelerometer reading is what triggers it. Treat that as an observation rather than a guarantee — the field is optional in the SDK type, a sample replayed from a Mock Device Kit feed can omit it, and other sources need not behave the same way. ## Prerequisites - A working integration built from [iOS integrations](/docs/develop/dat/build-integration-ios/) or [Android integrations](/docs/develop/dat/build-integration-android/), including registration with the Meta AI app. - A device session in the started state. You cannot add Motion to a session that has not started. Motion does not require a user-facing permission request. Unlike camera and microphone access, there is no `Permission` value to check or request for motion data, so you do not call `checkPermissionStatus` or `requestPermission` before using it. See [Permissions and registration](/docs/develop/dat/permissions-requests/) for the permissions that do require a user prompt. ## Choose a sampling rate Request a rate with `MotionConfiguration`. Higher rates cost more battery on the glasses and more bandwidth on the link to the phone, so choose the lowest rate that satisfies your feature. | Rate | Typical use | |------|-------------| | 5 Hz | Coarse activity or posture checks. Lowest power. | | 10 Hz | Default. Battery-conscious general use. | | 15 Hz | Smoother orientation tracking. | | 24 Hz | Matching a nominal 24 fps cadence. | | 30 Hz | Responsive head-motion input. | | 60 Hz | Highest fidelity. Highest power cost. | If you do not pass a configuration, the capability uses 10 Hz. The rate you request is a target, not a guarantee. Compute intervals from `timestampNs` instead of assuming a fixed cadence. ## Capability lifecycle Adding Motion to a session does not start it. Call `start()` explicitly, then observe state to know when samples will begin. | State | Meaning | |-------|---------| | `stopped` | Not streaming. This is the initial state and the resting state after `stop()`. | | `starting` | `start()` was called and the device is bringing the sensors up. | | `started` | Samples are flowing. | | `stopping` | Teardown is in progress. | | `paused` | Android only. The device session paused, so streaming is suspended. | On Android, if the device session pauses while Motion is `starting` or `started`, the capability reports `paused` and suspends streaming. When the session resumes, it restarts automatically and moves back through `starting` to `started`, so you do not call `start()` yourself. A `start()` call made while the capability is `starting`, `started`, or `stopping` is ignored, so the capability must be back at `stopped` before you start it again. The two platforms get there differently: on Android `stop()` completes synchronously and state is already `stopped` when the call returns, whereas on iOS `stop()` requests the transition and the capability passes through `stopping` asynchronously. Observe state rather than assuming the transition has finished if you intend to restart on iOS. On iOS, if the capability starts but no sample arrives within 10 seconds, it reports a sensor-unavailable error and stops. Android has no start timeout: the capability stays in `starting` until a sample arrives or an error ends the attempt. The sample stream buffers the most recent 256 samples and drops the oldest when a consumer falls behind. Keep per-sample work small, and hand heavy processing off to another worker so you do not lose data. ## Error handling Three errors can occur while Motion is active: | Error | Meaning | Recovery | |-------|---------|----------| | Sensor unavailable | The glasses could not bring the motion sensors up, or, on iOS, no sample arrived before the start timeout. | Ends the current start attempt, not the capability. Once state returns to `stopped`, you can call `start()` again. | | Capability closed | You used a capability that had already been removed, or whose session had ended. | Terminal for that instance. Add Motion again to a started session. | | Device disconnected | The glasses disconnected while the capability was starting or streaming. | On Android the capability restarts automatically when the session reconnects, so do not add it again yourself. On iOS, reattach once you have a started session. | For the enumerated values behind this table, see `MotionError` in the API reference for [Android](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_motion_types_motionerror) and [iOS](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest/mwdatmotion_motionerror). ## Android ### Step 1: Add the dependency Add the motion artifact alongside the core artifact in your version catalog, using the same version reference as the rest of the SDK. ```toml [libraries] mwdat-core = { group = "com.meta.wearable", name = "mwdat-core", version.ref = "mwdat" } mwdat-motion = { group = "com.meta.wearable", name = "mwdat-motion", version.ref = "mwdat" } ``` ```kotlin dependencies { implementation(libs.mwdat.core) implementation(libs.mwdat.motion) } ``` ### Step 2: Add the capability to a started session Call `addMotion` on a started `DeviceSession`. It returns a result, so handle the failure branch rather than assuming success. ```kotlin import com.meta.wearable.dat.motion.Motion import com.meta.wearable.dat.motion.addMotion import com.meta.wearable.dat.motion.types.MotionConfiguration import com.meta.wearable.dat.motion.types.MotionSamplingRate val configuration = MotionConfiguration(samplingRate = MotionSamplingRate.HZ_30) val motion: Motion = session .addMotion(configuration) .onFailure { error, _ -> showError(error.description) } .getOrNull() ?: return ``` Use `onFailure`, `fold`, or `errorOrNull()` to reach the typed `DeviceSessionError`. The single-argument `getOrElse` hands you the underlying `Throwable` instead, which has no `description`. ### Step 3: Start streaming and collect samples Collect `samples`, `state`, and `errors` on jobs that you keep references to and cancel during teardown. ```kotlin import com.meta.wearable.dat.motion.types.MotionState import kotlinx.coroutines.CoroutineStart val stateJob = scope.launch { motion.state.collect { state -> if (state == MotionState.STARTED) onStreaming() else onNotStreaming() } } // `samples` does not replay, so subscribe before `start()` rather than racing it. val samplesJob = scope.launch(start = CoroutineStart.UNDISPATCHED) { motion.samples.collect { sample -> sample.accelerometer?.let { onAcceleration(it.x, it.y, it.z) } sample.orientation?.let { onOrientation(it.x, it.y, it.z, it.w) } } } motion.start() ``` Handle each reading on its own. A sample with no `accelerometer` can still carry a usable `orientation`, so skipping the whole sample throws away data. `state` and `errors` are state flows, so a new collector immediately receives the current value. `samples` is a shared flow with no replay, which is why the example subscribes with `CoroutineStart.UNDISPATCHED` before calling `start()`; a plain `scope.launch` may not be collecting yet when the first samples arrive. `errors` holds `null` until something goes wrong, keeps the last error while the capability is stopped, and clears back to `null` when you call `start()` again. ```kotlin import com.meta.wearable.dat.motion.types.MotionError val errorsJob = scope.launch { motion.errors.collect { error -> when (error) { null -> Unit MotionError.SENSOR_UNAVAILABLE -> showError(error.description) MotionError.DEVICE_DISCONNECTED -> showReconnecting() MotionError.CAPABILITY_CLOSED -> scheduleReattach() } } } ``` Each value recovers differently. See [Error handling](#error-handling). ### Step 4: Stop and remove the capability ```kotlin import com.meta.wearable.dat.motion.removeMotion motion.stop() stateJob.cancel() samplesJob.cancel() errorsJob.cancel() session.removeMotion().onFailure { error, _ -> showError(error.description) } ``` Cancel the collector jobs you started in Step 3; the scope does not do it for you unless you cancel the scope itself. `removeMotion()` returns a `DatResult`, so handle its failure branch rather than assuming teardown succeeded. Cancelling the collectors immediately after `stop()` is safe here because Android's `stop()` is synchronous: state has already reached `stopped` by the time it returns, so nothing is lost by stopping observation at this point. This is a terminal teardown — if instead you are stopping with the intention of starting again, keep the state collector running and wait for `MotionState.STOPPED` before calling `start()`. Removing the capability releases the sensors on the glasses. Do it as soon as your feature no longer needs motion data, rather than waiting for the session to end. ### Test without hardware Mock Device Kit can replay a deterministic recording through the same path as a real device, either from an in-memory list of samples or from a CSV file. ```kotlin mockDevice.services.motion.setMotionFeed(samples = recordedSamples, loop = true) ``` `loop` is available only on the in-memory overload. The CSV overload, `setMotionFeed(fileUri)`, plays the recording through once. On Android you can set a feed before or during a session. Setting one while the capability is already streaming hot-swaps the recording and restarts replay from its first sample. Replay follows the sampling rate from `MotionConfiguration` rather than the intervals between recorded timestamps. A CSV recording has no header row and requires exactly 14 comma-separated numeric values per row, in this order: `timestampNs,accelX,accelY,accelZ,gyroX,gyroY,gyroZ,magX,magY,magZ,orientationW,orientationX,orientationY,orientationZ`. Blank lines and lines beginning with `#` are skipped. Because every column is required, a CSV recording cannot express an absent reading — use the in-memory overload to replay samples with `null` values. Replayed samples always report glasses as their `source`, whatever the recording contains, so you cannot exercise Meta Neural Band or unknown-source handling through a feed. See [Android testing](/docs/develop/dat/testing-mdk-android/) for how to create a mock device. ## iOS ### Step 1: Import the module The motion module ships in the SDK Swift package you added in [iOS integrations](/docs/develop/dat/build-integration-ios/), but adding a package reference does not link every product in it. Add the `MWDATMotion` product to your app target alongside `MWDATCore`, at the same SDK version, then import it wherever you use the capability. ```swift import MWDATCore import MWDATMotion ``` ### Step 2: Add the capability to a started session `addMotion` returns `nil` if the session has not started, and throws if a Motion capability is already attached. ```swift let configuration = MotionConfiguration(samplingRate: .hz30) guard let motion = try session.addMotion(configuration: configuration) else { return } ``` ### Step 3: Start streaming and consume samples `samples` is an `AsyncStream`, so iterate it in a task that you retain and cancel during teardown. Observe state and errors through their publishers. Store the listener tokens as properties, or keep them in a `ListenerTokenBag`. An `AnyListenerToken` cancels its subscription once nothing references it, so a token held only in a local constant stops observing as soon as setup returns. ```swift // Properties on the type that owns the capability, not local constants. private var stateToken: AnyListenerToken? private var errorToken: AnyListenerToken? private var samplesTask: Task? stateToken = motion.statePublisher.listen { state in Task { @MainActor in // Update your UI for .starting, .started, .stopping, or .stopped } } errorToken = motion.errorPublisher.listen { error in Task { @MainActor in switch error { case .sensorUnavailable: showError(error) case .capabilityClosed, .deviceDisconnected: scheduleReattach() @unknown default: showError(error) } } } let samples = motion.samples samplesTask = Task { for await sample in samples { if let acceleration = sample.accelerometer { await onAcceleration(acceleration.simd) } if let orientation = sample.orientation { await onOrientation(orientation.simd) } } } motion.start() ``` `MotionError` is a public, non-frozen enum, so include an `@unknown default` branch. Without it the compiler warns, and a switch that does not handle a case added in a later beta traps at runtime. Handle each reading on its own: a sample with no `accelerometer` can still carry a usable `orientation`. `Vector3` and `Quaternion` expose a `simd` property, so you can hand readings straight to `SIMD3` and `simd_quatf` math without converting component by component. Neither `statePublisher` nor `errorPublisher` replays to a new listener; both deliver only events that occur after you subscribe. Attach both listeners before calling `start()` so you do not miss the first transition or an early failure. ### Step 4: Stop and remove the capability ```swift samplesTask?.cancel() stateToken = nil errorToken = nil motion.stop() try session.removeMotion() ``` Releasing the tokens cancels the state and error subscriptions. ### Test without hardware Mock Device Kit replays a recording from an array of samples or from a CSV file using the same format as Android. Use `mockMotionGravityMetersPerSecondSquared` for the at-rest gravity reading when you synthesize samples. ```swift mockDevice.services.motion.setMotionFeed(recordedSamples) ``` Stage the feed before the consumer starts. Unlike Android, iOS ignores `setMotionFeed` while a Motion session is streaming, so a feed set mid-session has no effect. The array takes `MWDATMockDevice.MotionSample`, which is a distinct type from the `MWDATMotion.MotionSample` you receive from the live stream. The iOS in-memory overload has no `loop` parameter. As on Android, replay follows the configured sampling rate, a CSV recording has no header row and uses exactly 14 numeric columns per row, and replayed samples always report glasses as their `source`. See [iOS testing](/docs/develop/dat/testing-mdk-ios/) for how to create a mock device. ## Platform differences Key differences include: - **Paused state.** Android reports a `PAUSED` state when the device session pauses. iOS has no equivalent case, so on iOS a pause surfaces through the session state rather than through the capability. - **Start timeout.** iOS fails a start attempt with a sensor-unavailable error if no sample arrives within 10 seconds. Android has no start timeout. - **State and error delivery.** Android exposes `state` and `errors` as state flows, so a new collector immediately sees the current value, and `errors` clears to `null` on the next `start()`. On iOS both `statePublisher` and `errorPublisher` are non-replaying, so a listener attached after an event never sees it. - **Sample stream.** Android's `samples` is a shared flow that supports multiple collectors and does not replay. iOS's `samples` is a single-consumer `AsyncStream` that stays open across `stop()` and finishes only when the capability is removed or its session tears down. - **Automatic restart.** On Android a disconnect marks the capability for automatic restart when the session reconnects. On iOS you reattach it yourself. ## Next steps - Understand how session state drives the capability in [Session lifecycle](/docs/develop/dat/lifecycle-events/). - Review what does and does not need a user prompt in [Permissions and registration](/docs/develop/dat/permissions-requests/). - Build a deterministic test setup with [Mock Device Kit basics](/docs/develop/dat/mock-device-kit/). - Ship to testers through [Manage versions and release channels](/docs/develop/dat/set-up-release-channels/). - Look up every type and member in the API reference for [Android](https://wearables.developer.meta.com/docs/reference/android/dat/latest) and [iOS](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest). ### Respond to Hey Meta voice invocations ## Overview When the wearer says *"Hey Meta, start {YOUR APP NAME}"*, Meta AI launches or foregrounds your app and sends it a voice invocation. Your app receives the invocation, does the work, and acknowledges it back to the glasses. If your app is not already running, the command cold-launches it. Voice invocations are not a device-session capability. They are a **`Wearables`-level stream**: you do not add a capability to a session, and you do not need a running device session. This matches how [Session lifecycle](/docs/develop/dat/lifecycle-events/) describes "Hey Meta" — a short, system-owned transaction rather than a sustained device session. **Your app's identity cannot be spoofed.** Your package name (Android) or bundle identifier (iOS) is read from the initialized SDK configuration, not supplied by the caller, so an app cannot present itself as another developer's integration. ## Prerequisites - A working integration for [Android](/docs/develop/dat/build-integration-android/) or [iOS](/docs/develop/dat/build-integration-ios/), with the SDK initialized at app launch. - Your app registered with the Meta AI app. See [Permissions and registration](/docs/develop/dat/permissions-requests/). Then configure your app for voice invocations in the [Wearables Developer Center](https://wearables.developer.meta.com/devcenter/): - **Add mobile app configuration.** Provide your app's package name (Android) or bundle identifier (iOS). - **Request the Voice Invocation permission** and wait for approval. Meta AI uses it to identify and call your app. - **Configure your app name.** The name you provide acts as the keyword the voice service matches, so choose one that is phonetically clear and minimizes user confusion. See [Manage projects](/docs/develop/dat/manage-projects/). Invocation phrases are **not** declared in your Android manifest or `Info.plist`, and voice invocations do not require a `CAMERA` or `MICROPHONE` permission request. Whether invocations reach your app is determined entirely by your Developer Center configuration. ## How an invocation reaches your app 1. The wearer speaks to Meta AI on the glasses. 2. Meta AI resolves the phrase to your app and launches or foregrounds it. 3. Your app detects that it was launched by a voice invocation. 4. Your app starts a voice invocations stream and receives the invocation. 5. Your app does the work and answers through the invocation's response handle. **Answer every invocation exactly once.** Failing to answer leaves Meta AI waiting on the request and may surface a fallback UI to the wearer. The response handle tracks delivery status and rejects duplicate responses, so answering twice is harmless; never answering is the failure mode to avoid. ## Error handling Android exposes nine `VoiceInvocationError` values on `stream.errors`: | Error | Cause | |--------------------------------|---------------------------------------------------------| | `CHANNEL_NOT_CONNECTED` | The voice invocation channel is not connected. | | `CHANNEL_ERROR` | An unspecified channel error occurred. | | `FAIL_TO_SEND_INIT_REQUEST` | The init request could not be sent. | | `INIT_REQUEST_ERROR` | The init request returned an error. | | `FAIL_TO_SEND_MESSAGE` | A message could not be sent. | | `INVALID_ACTION_MESSAGE_PROTO` | A malformed action message was received. | | `UNKNOWN_MESSAGE_TYPE` | An unrecognized message type was received. | | `ACTION_FAILED` | An action failed. | | `INVALID_SESSION_STATE` | An operation was attempted in an invalid session state. | iOS has the same nine cases plus two more that come from resolving the device: `invalidWearablesInterface` and `deviceNotFound`. Three iOS cases are both thrown and publishable — `invalidWearablesInterface` from the initializer, and `deviceNotFound` and `channelNotConnected` from `start(deviceIdentifier:)`. On Android the equivalent conditions surface only on `errors`, because neither the stream factory nor its automatic start throws. A malformed invocation is dropped rather than delivered: the SDK reports `INVALID_ACTION_MESSAGE_PROTO` on the error stream and does not emit a bogus invocation. ## Android The voice invocation APIs live in `com.meta.wearable.dat.core.voiceinvocations`: ```kotlin import com.meta.wearable.dat.core.Wearables import com.meta.wearable.dat.core.selectors.AutoDeviceSelector import com.meta.wearable.dat.core.voiceinvocations.VoiceInvocationsStream import com.meta.wearable.dat.core.voiceinvocations.isVoiceInvocationsIntent import com.meta.wearable.dat.core.voiceinvocations.startVoiceInvocationsStream import com.meta.wearable.dat.core.voiceinvocations.types.actions.LaunchApp ``` ### Step 1: Detect the voice-triggered launch Meta AI launches your activity with an intent that carries a source marker. Check it in both `onCreate` and `onNewIntent` — an already-running app is foregrounded through `onNewIntent`, not `onCreate`. ```kotlin override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) if (isVoiceInvocationsIntent(intent)) { startYourExperience() } } override fun onNewIntent(intent: Intent) { super.onNewIntent(intent) setIntent(intent) if (isVoiceInvocationsIntent(intent)) { startYourExperience() } } ``` `isVoiceInvocationsIntent(intent)` returns `true` only for intents originating from the Meta AI app as a result of a voice command. Do not try to reconstruct the check yourself from intent extras. ### Step 2: Start the stream Hold the stream in a property. A local `val` inside `onCreate` goes out of scope before you can close it. ```kotlin private lateinit var stream: VoiceInvocationsStream override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) stream = Wearables.startVoiceInvocationsStream(AutoDeviceSelector()) } ``` The stream **starts itself** when it is created; there is no separate `start()` call. If the automatic start fails, the failure is logged and surfaces on `errors` and `state` rather than throwing. The optional second parameter is the `CoroutineDispatcher` used for background work, defaulting to the SDK's IO dispatcher. ### Step 3: Handle invocations ```kotlin lifecycleScope.launch { stream.invocations.collect { invocation -> when (invocation) { is LaunchApp -> { navigateToMainScreen() invocation.responseHandle.sendSuccess(null) } } } } ``` `sendSuccess` and `sendFailure` are `suspend` functions returning `Boolean`: `true` when the response was delivered. Both take an optional `actionOutput` string. ### Step 4: Observe state and errors ```kotlin lifecycleScope.launch { stream.state.collect { state -> updateUi(state) } } lifecycleScope.launch { stream.errors.collect { error -> log(error.description) } } ``` `SessionState` has three values: | State | Meaning | |------------|-------------------------------------------------| | `STARTING` | The stream is connecting. | | `STARTED` | The stream is listening for invocations. | | `STOPPED` | Not started, or no active devices are connected. | ### Step 5: Close the stream ```kotlin override fun onDestroy() { stream.close() super.onDestroy() } ``` `VoiceInvocationsStream` is `Closeable`. Close it when you stop caring about invocations; the stream's lifecycle is owned by you, not by the SDK. ### Test without hardware [Mock Device Kit](/docs/develop/dat/mock-device-kit/) can inject invocations through `MockGlasses.services.voiceInvocation`. ```kotlin val glasses = mockKit.pairGlasses(GlassesModel.RAYBAN_META).getOrThrow() val voiceInvocation = glasses.services.voiceInvocation if (voiceInvocation.hasConnectedApps()) { voiceInvocation.simulateLaunchAppAction() } ``` `simulateLaunchAppAction()` returns the request ID of the dispatched action, or `null` when no app is connected — so a `null` return means your app was not listening, not that the invocation failed. Use `simulateIncompleteAction()` to dispatch a deliberately malformed action and confirm your app surfaces the error rather than acting on a bogus invocation. See [Android testing](/docs/develop/dat/testing-mdk-android/) for how to create a mock device. ## iOS ### Step 1: Create and start the stream iOS separates construction from starting, and requires you to name the device. Keep the stream and both listener tokens as stored properties: if they go out of scope when the enclosing function returns, their subscriptions are released and invocations stop arriving. ```swift import MWDATCore @MainActor final class VoiceInvocationsController { private var stream: VoiceInvocationsStream? private var invocationToken: (any AnyListenerToken)? private var errorToken: (any AnyListenerToken)? func startListening(on deviceIdentifier: DeviceIdentifier) throws { let stream = try VoiceInvocationsStream(wearables: Wearables.shared) addListeners(to: stream) // Steps 2 and 3 try stream.start(deviceIdentifier: deviceIdentifier) self.stream = stream } } ``` Register your listeners before calling `start(deviceIdentifier:)` so an invocation that arrives immediately is not missed. The initializer throws `invalidWearablesInterface` if the SDK is not configured. `start(deviceIdentifier:)` throws `deviceNotFound` when no connected device matches, or `channelNotConnected` when the device has no active connection. Calling `start(deviceIdentifier:)` again tears down the previous stream first, so it is the way to switch to another device. To cover devices that connect after launch, call it for each device in `wearables.devices` and then for each update from `wearables.devicesStream()`. `VoiceInvocationsStream` is `@MainActor`-isolated. ### Step 2: Handle invocations Invocations arrive as `any VoiceInvocation`, so downcast to the concrete type. ```swift invocationToken = stream.invocationsPublisher.listen { invocation in guard let launchApp = invocation as? LaunchApp else { return } navigateToMainScreen() Task { _ = await launchApp.responseHandle.sendSuccess(actionOutput: nil) } } ``` On iOS, `LaunchApp` also carries the `deviceIdentifier` of the glasses that produced the invocation. ### Step 3: Observe errors ```swift errorToken = stream.errorPublisher.listen { error in log(error.description) } ``` iOS does not expose a session-state publisher. Track readiness from whether `start(deviceIdentifier:)` threw, plus the error publisher. ### Step 4: Stop the stream ```swift stream?.stop() ``` `stop()` is safe to call when no stream is active. After stopping, call `start(deviceIdentifier:)` again to listen once more. ### Test without hardware [Mock Device Kit](/docs/develop/dat/mock-device-kit/) can inject invocations through `MockGlasses.services.voiceInvocation`. ```swift let voiceInvocation = mockGlasses.services.voiceInvocation if voiceInvocation.hasConnectedClients { _ = voiceInvocation.sendLaunchAppAction() } ``` `sendLaunchAppAction()` returns the request ID of the dispatched action, or `nil` when no app is connected — so a `nil` return means your app was not listening, not that the invocation failed. Use `sendIncompleteAction()` to dispatch a deliberately malformed action and confirm your app surfaces the error rather than acting on a bogus invocation. See [iOS testing](/docs/develop/dat/testing-mdk-ios/) for how to create a mock device. ## Platform differences | Behavior | Android | iOS | |--------------------------------------|---------------------------------------------------------------------|-----------------------------------------------------| | Creating the stream | `Wearables.startVoiceInvocationsStream(deviceSelector, dispatcher)` | `VoiceInvocationsStream(wearables:)` | | Starting | Automatic on creation | Explicit `start(deviceIdentifier:)` | | Device targeting | `DeviceSelector` | A specific `DeviceIdentifier` | | Session state | `state: StateFlow` | **Not exposed** | | Failure reporting at start | Logged, then surfaced on `errors` | Thrown from `init` and `start(deviceIdentifier:)` | | Invocation type modeling | `sealed interface`; `when` is exhaustive | `protocol`; downcast with `as?` | | `deviceIdentifier` on the invocation | Not present | Present on `LaunchApp` | | Teardown | `close()` (`Closeable`) | `stop()`, restartable | | Detecting a voice launch | `isVoiceInvocationsIntent(intent)` | Not applicable | ## Best practices - **Avoid "AI" in your app name.** App names that contain "AI" are more easily confused with Meta AI and reduce voice command recognition. Choosing a name without "AI" improves the voice command success rate. - **Always acknowledge invocations.** Every emitted invocation must be answered through its response handle with success or failure. Unanswered invocations leave the glasses waiting on the request. - **Open the stream early.** Start the stream in an app-scoped or early lifecycle location so a voice command that cold-launches your app is captured before your UI appears. - **Handle disconnects gracefully.** Observe the stream's error and state channels and surface appropriate UI when a device disconnects. - **Release resources when done.** Close or stop the stream when you no longer need to receive invocations. ## Next steps - [Session lifecycle](/docs/develop/dat/lifecycle-events/) for how voice transactions differ from device sessions. - [Permissions and registration](/docs/develop/dat/permissions-requests/) for registering your app with Meta AI. - [Manage projects](/docs/develop/dat/manage-projects/) for configuring your app name and invocation phrases. - [Mock Device Kit basics](/docs/develop/dat/mock-device-kit/) for testing invocation handling without glasses. - Complete integrations in the [Android sample apps](https://github.com/facebook/meta-wearables-dat-android/tree/main/samples) and [iOS sample apps](https://github.com/facebook/meta-wearables-dat-ios/tree/main/samples). - Look up every type and member in the API reference for [Android](https://wearables.developer.meta.com/docs/reference/android/dat/latest) and [iOS](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest). ### Read battery, wear, and thermal state from glasses ## Overview Device state tells you the physical condition of the wearer's glasses: how much battery is left, whether the battery is charging, whether the glasses are being worn, whether the arms are folded closed, and how hot the device is. Use it to defer battery-intensive work, pause a feature when the wearer takes the glasses off, or back off before the device throttles. Unlike camera, microphone, motion, or inputs, device state is not a capability you add to a session. It is metadata carried on the device itself, alongside the device name and connection state. That has two consequences worth designing around: - There is nothing to start, stop, or remove, and no per-capability error stream. - State is available as soon as the device connects, **before** you start a session, and it keeps updating for as long as the device stays connected. | Field | Type | Unit | Notes | |-------|------|------|-------| | `batteryLevel` | Integer | Percent, 0 to 100 | Reported as unknown until the device sends a reading. See [Read battery level safely](#read-battery-level-safely) for the platform difference. | | `chargingState` | Enum | — | `UNKNOWN`, `CHARGING`, or `NOT_CHARGING`. | | `donState` | Enum | — | `UNKNOWN`, `DONNED` (being worn), or `DOFFED` (not being worn). | | `hingeState` | Enum | — | `UNKNOWN`, `OPEN`, or `CLOSED` (arms folded). | | `thermalLevel` | Enum | — | Eight escalating levels. See [Respond to thermal level](#respond-to-thermal-level). | For the type of each field, see `Device` in the API reference for [Android](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_core_device) and `DeviceState` for [iOS](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest/mwdatcore_devicestate). Every enum has an `UNKNOWN` case, and it is the initial value for all of them. `UNKNOWN` means no usable information is available—for example, the field has not been reported, is not applicable to the device, or the SDK did not recognise the value. It may not eventually become a real value. Device state is delivered only while the glasses are connected. When the device disconnects, updates stop; the last values you received are not refreshed and become stale. Check the device's connection state before you act on them. ## Prerequisites - A working integration built from [iOS integrations](/docs/develop/dat/build-integration-ios/) or [Android integrations](/docs/develop/dat/build-integration-android/), including registration with the Meta AI app. - A connected device. You do not need a started session. Device state does not require a user-facing permission request. It is device metadata rather than user content, so there is no `Permission` value to check or request, and you do not call `checkPermissionStatus` or `requestPermission` before reading it. See [Permissions and registration](/docs/develop/dat/permissions-requests/) for the permissions that do require a user prompt. ## Read battery level safely The platforms represent an unavailable battery reading differently: | Platform | Type | Unavailable value | |----------|------|-------------------| | Android | `Int` | `0` | | iOS | `Int?` | `nil` | On Android, the wire value `0` means the battery level is unknown or the battery is empty. The SDK exposes that value directly as `0`. On iOS, `nil` means no battery reading is available. ## Respond to thermal level Thermal level escalates as the device heats up. It is the same enum on both platforms. | Level | Meaning | What to do | |-------|---------|------------| | `UNKNOWN` | Not reported yet. | Nothing. Wait for a real value. | | `NONE` | Normal operating temperature. | Nothing. | | `LIGHT` | Slightly elevated. | Nothing, but stop increasing load. | | `MODERATE` | Moderately elevated. | Reduce optional work, such as a high sampling rate or a high-resolution stream. | | `SEVERE` | Severely elevated. Performance may be throttled. | Shed load. Expect the device to slow down regardless of what you do. | | `CRITICAL` | Critical. Device performance may be heavily restricted, and sessions may be denied or terminated. | Treat the device as potentially unavailable. | | `EMERGENCY` | Emergency. The device is preparing to shut down. | Treat the device as potentially unavailable. | | `SHUTDOWN` | The device is shutting down due to thermal conditions. | Treat the device as potentially unavailable. | At `CRITICAL` or above, the device may become unavailable. Stop device-dependent work, and handle session termination or disconnection through [Session lifecycle](/docs/develop/dat/lifecycle-events/). ## Update behavior The device pushes a new snapshot whenever any field changes; the SDK does not poll. Snapshots are compared by value, so you are only notified when something actually changed. A field you care about may be unchanged in a snapshot you receive, because a different field moved — always read the field you need from the snapshot rather than assuming the notification was about it. ## Android Device state is part of the `Device` snapshot, which you observe as a `StateFlow`. There are two entry points to the same underlying flow; pick whichever matches where you are in the session lifecycle. ### Before or without a session Get `deviceIdentifier` by collecting `Wearables.devices`, as shown in [Launch registration from your app](/docs/develop/dat/build-integration-android/#step-4-launch-registration-from-your-app). `Wearables.devicesMetadata` maps each known device identifier to a flow of its snapshot. Use it to show battery or wear state before the user starts anything. ```kotlin import com.meta.wearable.dat.core.Wearables import com.meta.wearable.dat.core.types.ChargingState import com.meta.wearable.dat.core.types.DonState val deviceFlow = Wearables.devicesMetadata[deviceIdentifier] ?: return scope.launch { deviceFlow.collect { device -> val battery = device.batteryLevel.takeIf { it != 0 } showBattery(battery, charging = device.chargingState == ChargingState.CHARGING) showWorn(device.donState == DonState.DONNED) } } ``` `devicesMetadata` returns `null` for a device the SDK does not know about, so handle the missing case rather than asserting. ### During a session `DeviceSession.deviceInfo` exposes the snapshot for the session's own device, backed by the same flow, so you do not need to hold on to the identifier. ```kotlin import com.meta.wearable.dat.core.types.ThermalLevel scope.launch { session.deviceInfo.collect { device -> if (device.thermalLevel >= ThermalLevel.SEVERE) { reduceWorkload() } } } ``` Comparing thermal levels with `>=` works because the enum is declared in escalating order, which keeps a threshold check from having to list every level above it. Both flows are state flows, so a new collector immediately receives the current snapshot. Cancel the collecting scope during teardown; there is nothing else to release. ### Test without hardware Mock Device Kit drives the same snapshot flow, so your production observation code runs unchanged. The setters work at the connection level, with or without a session. ```kotlin mockDevice.setBatteryLevel(42) mockDevice.setChargingState(ChargingState.CHARGING) mockDevice.setThermalLevel(ThermalLevel.SEVERE) mockDevice.don() mockDevice.doff() mockGlasses.fold() mockGlasses.unfold() ``` `setBatteryLevel` clamps its argument to 0 through 100. Wear state comes from `don()` and `doff()`, and hinge state from `fold()` and `unfold()` on the mock glasses. See [Android testing](/docs/develop/dat/testing-mdk-android/) for how to create a mock device. ## iOS On iOS device state is exposed both as properties on `Device` and as an immutable `DeviceState` snapshot delivered to a listener. ### Get a device and read its state Get `identifier` from `Wearables.shared.devicesStream()`, as shown in [Launch registration from your app](/docs/develop/dat/build-integration-ios/#step-4-launch-registration-from-your-app), then resolve the corresponding device. ```swift import MWDATCore guard let device = Wearables.shared.deviceForIdentifier(identifier) else { return } let battery = device.batteryLevel // Int?, nil when not reported let charging = device.chargingState // .unknown, .charging, .notCharging let worn = device.donState == .donned ``` Inside a session, `DeviceSession.device` gives you the same object without holding the identifier. ### Observe changes `addDeviceStateListener` fires immediately with the current state and again on every change. It returns a token you cancel to stop listening. ```swift let token = device.addDeviceStateListener { state in Task { @MainActor in showBattery(state.batteryLevel, charging: state.chargingState == .charging) showWorn(state.donState == .donned) if state.thermalLevel == .severe || state.thermalLevel == .critical { reduceWorkload() } } } ``` The `DeviceState` snapshot also carries `linkState` and `compatibility`, so a single callback tells you both the device's physical state and whether it is still connected and usable. Reading `linkState` from the same snapshot avoids acting on battery or thermal values from a device that has since disconnected. Cancel the token during teardown: ```swift token.cancel() ``` To observe raw connection transitions on their own, use `addLinkStateListener` instead. ### Test without hardware ```swift mockDevice.setBatteryLevel(42) mockDevice.setChargingState(.charging) mockDevice.setThermalLevel(.severe) mockDevice.don() mockDevice.doff() mockGlasses.fold() mockGlasses.unfold() ``` `setBatteryLevel` takes an optional, so pass `nil` to simulate a device that has not reported a battery reading. This is the case Android cannot represent directly, which makes it worth an explicit test. See [iOS testing](/docs/develop/dat/testing-mdk-ios/) for how to create a mock device. ## Platform differences The set of fields and every enum case are identical on both platforms. The surface around them differs: - **Unknown battery.** Android uses `Int` with `0` meaning unknown. iOS uses `Int?` with `nil` meaning unknown. - **Observation.** Android exposes a `StateFlow` of the whole `Device` snapshot. iOS delivers an immutable `DeviceState` snapshot to a listener and also exposes each field as a property on `Device`. - **Snapshot contents.** The iOS `DeviceState` snapshot includes `linkState` and `compatibility` alongside the physical state. On Android those fields live on the same `Device` object you are already collecting, so both platforms give you the same information in one place. - **Teardown.** On Android you cancel the coroutine scope collecting the flow. On iOS you cancel the listener token. - **Mock battery.** Only the iOS mock can simulate an unreported battery level, by passing `nil`. ## Next steps - React to session and connection changes in [Session lifecycle](/docs/develop/dat/lifecycle-events/). - Review what does and does not need a user prompt in [Permissions and registration](/docs/develop/dat/permissions-requests/). - Build a deterministic test setup with [Mock Device Kit basics](/docs/develop/dat/mock-device-kit/). - Ship to testers through [Manage versions and release channels](/docs/develop/dat/set-up-release-channels/). - Look up every type and member in the API reference for [Android](https://wearables.developer.meta.com/docs/reference/android/dat/latest) and [iOS](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest). ### Transcribe speech on the glasses ## Overview The Speech capability runs automatic speech recognition (ASR) on the glasses and delivers the recognized text to your app. **Beta:** Speech ships in the development and beta [release channels](/docs/develop/dat/set-up-release-channels/) only. Do not depend on it in a production build. Each recognition event arrives as a `TranscriptionResult`: | Field | Type | Meaning | |--------------|-----------|-------------------------------------------------------------------------| | `text` | `String` | The recognized text, either partial or final. | | `isFinal` | `Boolean` | `true` when no further updates will arrive for the current utterance. | | `confidence` | `Float` | Confidence in the inclusive range `[0.0, 1.0]`, or `-1.0` if unavailable. | Partial results arrive while the wearer is still speaking and are replaced by later results for the same utterance. Append text to your UI only when `isFinal` is `true`, and treat a `confidence` of `-1.0` as "no score reported" rather than as low confidence. ## Prerequisites - A working integration for [Android](/docs/develop/dat/build-integration-android/) or [iOS](/docs/develop/dat/build-integration-ios/). - A [device session](/docs/develop/dat/lifecycle-events/) in the `STARTED` state. - The `MICROPHONE` device permission, granted by the wearer. Speech **does** require a user-facing permission request. Request `MICROPHONE` through the Meta AI app flow described in [Permissions and registration](/docs/develop/dat/permissions-requests/) before calling `start()`. ## Capability lifecycle The Speech capability is attached to a session. You can start and stop it to listen for transcriptions, and can listen for SpeechState and SpeechError to properly handle changes in the capability. The events you will get from speech will contain the text, a confidence level and a flag indicating whether the transcription has completed. Run Speech only while your feature needs transcriptions to limit battery use. Observe `SpeechState` because the capability can stop after inactivity, a timeout, or another performance constraint. Observe `SpeechError` separately to distinguish failures from a normal stop. ## Error handling The capability reports failures as `SpeechError`. Handle every value, and observe `SpeechState` as well: the capability can also stop without an error, for example after a period of inactivity or a timeout. For the values on each platform, see the Speech API reference for [Android](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_speech_speech) and [iOS](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest/mwdatspeech_speech). ## Android ### Step 1: Add the dependency Add the speech artifact alongside the core artifact in your version catalog, using the same version reference as the rest of the SDK. ```toml [libraries] mwdat-core = { group = "com.meta.wearable", name = "mwdat-core", version.ref = "mwdat" } mwdat-speech = { group = "com.meta.wearable", name = "mwdat-speech", version.ref = "mwdat" } ``` ```kotlin dependencies { implementation(libs.mwdat.core) implementation(libs.mwdat.speech) } ``` For the full API surface, see the [Speech API reference](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_speech_speech). ### Test without hardware Drive Speech from [Mock Device Kit](/docs/develop/dat/mock-device-kit/) through `MockGlasses.services.speech`. See [Android testing](/docs/develop/dat/testing-mdk-android/) for how to create a mock device. ## iOS ### Step 1: Import the module The Speech functionality is provided by the `MWDATSpeech` module. ```swift import MWDATCore import MWDATSpeech ``` For the full API surface, see the [Speech API reference](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest/mwdatspeech_speech). ### Test without hardware Drive Speech from [Mock Device Kit](/docs/develop/dat/mock-device-kit/) through `MockGlasses.services.speech`. See [iOS testing](/docs/develop/dat/testing-mdk-ios/) for how to create a mock device. ## Next steps - Understand how session state drives the capability in [Session lifecycle](/docs/develop/dat/lifecycle-events/). - Review what does and does not need a user prompt in [Permissions and registration](/docs/develop/dat/permissions-requests/). - Build a deterministic test setup with [Mock Device Kit basics](/docs/develop/dat/mock-device-kit/). - Ship to testers through [Manage versions and release channels](/docs/develop/dat/set-up-release-channels/). - Look up every type and member in the API reference for [Android](https://wearables.developer.meta.com/docs/reference/android/dat/latest) and [iOS](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest). ### Capture high-quality photos ## Overview The Camera capability exposes two child features: - **Stream** for real-time video frames, including in-stream photo capture. - **Photo** for standalone, high-quality still capture at up to the native sensor resolution. This page covers **Photo**. Use it when you want a full-quality still image rather than a frame lifted out of a video stream. **Beta:** Standalone photo capture is available to every developer, but apps that use it cannot be published yet. Build and test against it; do not ship a production app that depends on it. ### Resolution | Value | Image size | |----------|---------------------------------------| | `SMALL` | 480p class | | `MEDIUM` | 720p class (default) | | `LARGE` | 1080p class | | `FULL` | Native sensor, 4032 × 3024 | Only the `FULL` has fixed dimensions. The glasses may resolve the other three values slightly differently, so treat them as size classes rather than exact pixel counts. ### Quality | Value | Trade-off | |----------|------------------------------------------| | `LOW` | Highest compression, smallest file | | `MEDIUM` | Balanced (default) | | `HIGH` | Lowest compression, largest file | Higher resolution and higher quality both increase on-device processing time and transfer time. `FULL` at `HIGH` produces the largest payload and takes the longest to arrive. ### Completed captures | Field | Android | iOS | Notes | |-------------|------------------------|-------------------|-----------------------------------------------------------------------| | Image bytes | `imageData: ByteArray` | `imageData: Data` | JPEG or HEIC, as produced by the glasses. Read EXIF from these bytes. | | Metadata | `metadata: ByteArray?` | `metadata: Data?` | Opaque capture-notification bytes. Not EXIF. **Nullable by API contract.** | | Timestamp | `timestamp: Long` | `timestamp: Date` | Phone-side time when the first transport half arrived, not the moment the shutter fired. Milliseconds on Android. | ## Prerequisites - Version `0.11.0` or newer of the Wearables Device Access Toolkit. Standalone photo capture is not available in earlier public releases. - A working integration for [Android](/docs/develop/dat/build-integration-android/) or [iOS](/docs/develop/dat/build-integration-ios/). - A [device session](/docs/develop/dat/lifecycle-events/) in the `STARTED` state. - The `CAMERA` device permission, granted by the wearer through the Meta AI app. See [Permissions and registration](/docs/develop/dat/permissions-requests/). ## Capability lifecycle **Stream and Photo compete for the camera hardware and cannot capture at the same time.** Stopping one before starting the other is your app's responsibility; the toolkit does not arbitrate between them. A capture request is asynchronous. You call `capturePhoto()`, the glasses take the picture, and the image transfers to the phone. Your app observes transfer progress and then receives the completed image. **Keep at most one capture in flight.** Completion and progress events carry no request identifier, and the photo stream is not a per-request response channel. On iOS, photos the wearer takes with the shutter button on the glasses arrive on the same publisher as the ones your app requests. If you issue a second `capturePhoto()` before the first completes, you cannot tell which result belongs to which call. Android activates the photo child implicitly and iOS requires an explicit `photo.start()`, so the startup sequence differs by platform. Each is covered in its own section below. ## Error handling In the current release, the error you receive can be less specific than the platform enumerations suggest. Both gaps are platform-specific. **Android reports device-side failures as `CAPTURE_FAILURE`.** A busy camera, an unavailable capture service, a permission denial, and a critical device-health state all arrive as `CAPTURE_FAILURE`, so `BUSY`, `SERVICE_UNAVAILABLE`, `PERMISSION_DENIED`, and `DEVICE_HEALTH_CRITICAL` never reach your app. Handle `CAPTURE_FAILURE` as a general capture failure and do not branch on those four values. iOS reports each of them as a distinct case. **iOS does not emit `deviceDisconnected`.** The case exists so the enumeration matches Android, but a disconnect during capture surfaces as `captureFailure`, and only once the 30-second silence timeout expires. Treat a `captureFailure` that arrives with no transfer progress as a possible disconnect and check the device session state. Android reports `DEVICE_DISCONNECTED` directly. For the enumerated values on each platform, see `PhotoError` in the API reference for [Android](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_camera_types_photoerror) and [iOS](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest/mwdatcamera_photoerror). For recovery guidance, see [Troubleshooting and known issues](#troubleshooting-and-known-issues). ## Android ### Step 1: Add the dependencies Add the camera artifact alongside the core artifact in your version catalog, using the same version reference as the rest of the SDK. ```toml [libraries] mwdat-core = { group = "com.meta.wearable", name = "mwdat-core", version.ref = "mwdat" } mwdat-camera = { group = "com.meta.wearable", name = "mwdat-camera", version.ref = "mwdat" } mwdat-mockdevice = { group = "com.meta.wearable", name = "mwdat-mockdevice", version.ref = "mwdat" } ``` ```kotlin dependencies { implementation(libs.mwdat.core) implementation(libs.mwdat.camera) testImplementation(libs.mwdat.mockdevice) } ``` ### Step 2: Check the camera permission `checkPermissionStatus()` is a suspend function and returns a `DatResult`, so call it from a coroutine and unwrap it before comparing. ```kotlin private suspend fun isCameraPermissionGranted(): Boolean { val status = Wearables.checkPermissionStatus(Permission.CAMERA).getOrThrow() return status == PermissionStatus.Granted } ``` When the permission is not granted, launch the grant flow with `Wearables.RequestPermissionContract()`. See [Permissions and registration](/docs/develop/dat/permissions-requests/). ### Step 3: Add a camera, subscribe, and wait for startup There is no `start()` on Android. The photo child is created and activated **implicitly on first access** to `camera.photo`, which means activation is already under way by the time you can subscribe to anything. That ordering matters because `photoStream`, `transferProgressStream`, and `errors` are `SharedFlow`s that do not replay. Subscribe to all three and wait for the subscriptions to be established before doing anything else. Then drive the startup decision from `state`, which is a `StateFlow` and therefore cannot miss the outcome. ```kotlin private suspend fun startPhotoCapture( session: DeviceSession, scope: CoroutineScope, ): Pair? { val camera = session.addCamera().getOrThrow() val photo = camera.photo val subscribed = CompletableDeferred() val remaining = AtomicInteger(3) fun markSubscribed() { if (remaining.decrementAndGet() == 0) subscribed.complete(Unit) } scope.launch { photo.errors .onSubscription { markSubscribed() } .collect { error -> showError(error.description) } } scope.launch { photo.transferProgressStream .onSubscription { markSubscribed() } .collect { progress -> updateProgress(progress.fraction) } } scope.launch { photo.photoStream .onSubscription { markSubscribed() } .collect { data -> saveImage(data.imageData) } } subscribed.await() var sawStarting = false val outcome = withTimeoutOrNull(PHOTO_STARTUP_TIMEOUT_MS) { photo.state.first { state -> when (state) { PhotoState.STARTING -> { sawStarting = true; false } PhotoState.STARTED -> true // A return to STOPPED after STARTING is a setup failure. PhotoState.STOPPED -> sawStarting PhotoState.STOPPING -> false } } } if (outcome != PhotoState.STARTED) { camera.stop() return null } return camera to photo } ``` A `SETUP_FAILED` emitted before your collectors attach is lost, which is why the timeout — not the error stream — is what bounds this wait. ### Step 4: Capture ```kotlin photo.capturePhoto( resolution = PhotoResolution.FULL, quality = PhotoQuality.HIGH, ) ``` Both parameters default to `MEDIUM`. Calling [`capturePhoto`](https://wearables.developer.meta.com/docs/reference/android/dat/latest/com_meta_wearable_dat_camera_photo#capturephoto) in any state other than `STARTED` reports `NOT_READY` instead of taking a picture. ### Step 5: Stop ```kotlin camera.stop() ``` `stop()` cascades to both children and detaches the capability from the session. No separate removal call is needed. ### Test without hardware Drive capture from [Mock Device Kit](/docs/develop/dat/mock-device-kit/) through `MockCameraCaptureKit`, reached as `glasses.services.cameraCapture`. `MockCameraCaptureKit` is the standalone-photo control surface. It is a different type from `MockCameraKit`, which controls the video stream. Confusing `setCapturedPhoto()` here with the stream kit's image controls is a common source of a mock that never delivers. Point it at a real file. A fabricated URI string cannot be opened by the mock. ```kotlin val fixture = File(context.cacheDir, "capture-fixture.jpg") fixture.writeBytes(testImageBytes) val mockKit = MockDeviceKit.getInstance(context) mockKit.enable() val glasses = mockKit.pairGlasses(GlassesModel.RAYBAN_META).getOrThrow() glasses.services.cameraCapture.setCapturedPhoto(Uri.fromFile(fixture)) ``` Without a configured image, the mock device reports that no capture is available. `simulateCaptureFailure()` makes the **next** capture respond with an error instead of an image; the flag resets after a single failure, which suits a retry test. Mock Device Kit exercises delivery and generic failure handling. It does not model Stream and Photo hardware contention, and its transfer progress is not representative of a real device. Validate both on hardware. ## iOS ### Step 1: Import the modules ```swift import MWDATCamera import MWDATCore ``` ### Step 2: Check the camera permission `Wearables.shared` requires an earlier `Wearables.configure()` call during app startup; see [Build an integration for iOS](/docs/develop/dat/build-integration-ios/). ```swift var status = try await Wearables.shared.checkPermissionStatus(.camera) if status != .granted { status = try await Wearables.shared.requestPermission(.camera) } guard status == .granted else { return } ``` ### Step 3: Register listeners, then start Unlike Android, iOS requires an explicit `photo.start()`. Register every listener **before** calling it: the publishers do not replay, and `start()` can resolve `.started` or a setup error before the next statement runs. Retain each listener token for as long as you want its callback to fire. Store them in a `ListenerTokenBag` so they are released together. ```swift @MainActor final class PhotoCaptureController { private var camera: Camera? private var photo: Photo? private let tokenBag = ListenerTokenBag() private var didCapture = false func start(on session: DeviceSession) throws { // `addCamera` returns nil until the session is `.started`. guard let camera = try session.addCamera() else { return } let photo = camera.photo self.camera = camera self.photo = photo photo.statePublisher.listen { [weak self] state in Task { @MainActor in self?.handle(state) } }.store(in: tokenBag) photo.photoDataPublisher.listen { [weak self] data in Task { @MainActor in self?.save(data.imageData) } }.store(in: tokenBag) photo.transferProgressPublisher.listen { [weak self] progress in Task { @MainActor in self?.updateProgress(progress.fraction) } }.store(in: tokenBag) photo.errorPublisher.listen { [weak self] error in Task { @MainActor in self?.show(error) } }.store(in: tokenBag) photo.start() } private func handle(_ state: PhotoState) { // `.started` can be published more than once. Capture on the first one only. guard state == .started, !didCapture else { return } didCapture = true photo?.capturePhoto(resolution: .full, quality: .high) } } ``` Calling `stop()` while the capability is `.starting` is recorded and honored as soon as startup resolves, rather than dropped. ### Step 4: Stop ```swift camera.stop() ``` `stop()` cascades to both children. iOS exposes no public camera-removal call. ### Test without hardware `MockCameraCaptureKit` mirrors the Android surface, taking a `URL` instead of a `Uri`. ```swift import MWDATMockDevice MockDeviceKit.shared.enable() let mockGlasses = try MockDeviceKit.shared.pairGlasses(model: .rayBanMeta) mockGlasses.services.cameraCapture.setCapturedPhoto(fileURL: fixtureURL) mockGlasses.services.cameraCapture.simulateCaptureFailure() ``` `pairGlasses(model:)` throws `MockDeviceKitError.notEnabled` if you call it before `enable()`. The same limits apply as on Android: delivery and generic failure handling are exercised, but Stream and Photo contention and realistic transfer progress are not. ## Platform differences Standalone photo capture behaves the same on both platforms except in these places: - **Startup.** Android has no `start()`: the photo child is created and activated implicitly on first access to `camera.photo`. iOS requires an explicit `photo.start()` after you register listeners. - **Teardown.** On Android, `camera.stop()` cascades to both children and detaches the capability from the session, so no separate removal call is needed. iOS exposes no public camera-removal call. - **Wearer-initiated photos.** On iOS, photos the wearer takes with the shutter button on the glasses arrive on the same publisher as the ones your app requests. Serialize your own captures so you can still tell the results apart. - **Error detail.** Android collapses four device-side failures into `CAPTURE_FAILURE`, and iOS reports them distinctly but never emits `deviceDisconnected`. See [Error handling](#error-handling). - **Mock control surface.** `MockCameraCaptureKit` takes a `Uri` on Android and a `URL` on iOS. ## Troubleshooting and known issues ### Capture reliability Standalone photo capture is not yet as reliable as the video stream. Treat a failed capture as an expected outcome and build a retry path rather than assuming success. **Startup is the most common failure point.** Sessions intermittently fail to start the photo capability, and the following attempt then succeeds. This surfaces as `SETUP_FAILED` on Android and `sessionSetupFailed` on iOS. Handle it with a bounded retry: 1. Wait for the photo child to reach `STARTED` under a timeout, and treat a return to `STOPPED` as a startup failure. 2. On failure, stop the camera, add a new one, and retry once. A stopped camera is invalidated, so the retry must go through a fresh `addCamera()` rather than re-reading the old instance. 3. Surface an error to the wearer only after the retry also fails. ### A capture never arrives - **Turn the glasses off and back on, then try again.** Restarting the glasses clears a stuck capture service on the device and resolves many cases on its own. Do this before you start debugging your app. - **Did you subscribe before the capability activated?** `photoStream`, `transferProgressStream`, and `errors` do not replay, and on Android activation begins the moment you read `camera.photo`. A capture that completes before your collectors attach is delivered to nobody. - **Did you register iOS listeners before `start()`?** `start()` can resolve before the following line runs. - **Was the state `STARTED`?** A `capturePhoto()` in any other state reports `NOT_READY` and takes no picture. - **Is the stream running?** Stream and Photo compete for the camera hardware. Stop the stream first. - **Is more than one capture in flight?** Results carry no request identifier. Serialize your captures. - **Did the session pause?** A paused device session suspends transports. Wait for the session to return to `STARTED` rather than retrying into a paused one. - **Is the payload very large?** `FULL` at `HIGH` is the slowest path. Watch `transferProgressStream` to confirm bytes are still arriving; 30 seconds of total silence fails the capture. ### Metadata is null on a successful capture `metadata` is nullable by API contract on both platforms, and a null value does not mean the capture failed. It carries opaque capture-notification bytes, not EXIF. Read EXIF from `imageData` instead. Code that parses `metadata` unconditionally will crash on an otherwise successful capture. See also [Known issues](/docs/develop/dat/knownissues/) for issues tracked across the toolkit. ## Next steps - [Session lifecycle](/docs/develop/dat/lifecycle-events/) for handling interruptions mid-capture. - [Permissions and registration](/docs/develop/dat/permissions-requests/) for the camera grant flow. - [Mock Device Kit basics](/docs/develop/dat/mock-device-kit/) for testing capture and failure paths without glasses. - The API reference for [Android](https://wearables.developer.meta.com/docs/reference/android/dat/latest) and [iOS](https://wearables.developer.meta.com/docs/reference/ios_swift/dat/latest) for every type and member used here. ### Mock Device Kit ## Overview Mock Device Kit is a component of the Device Access Toolkit that helps you build and test integrations for AI glasses, without the need to access the actual hardware. This kit provides a simulated device that mirrors the capabilities and behavior of AI glasses, including camera streaming, photo capture, permissions, and device state changes. You can use it to test your app integrations in a virtual environment. This is useful for rapid iteration, automated testing, and development workflows where physical devices may not be available or practical to use. When Mock Device Kit is enabled, it simulates the entire SDK stack -- including app connection (registration) and permission requests. Your app code works the same way regardless of whether it's talking to a real device or a mock device. **Note:** This page covers the debug UI in the CameraAccess sample app. For writing automated tests, see [Android testing with Mock Device Kit](/docs/develop/dat/testing-mdk-android/) or [iOS testing with Mock Device Kit](/docs/develop/dat/testing-mdk-ios/). ## Using Mock Device Kit in the CameraAccess sample The CameraAccess sample app includes a built-in debug UI for controlling Mock Device Kit, available in debug builds only. ### Enabling Mock Device Kit 1. Launch the CameraAccess sample app. You'll see the home screen prompting you to connect your glasses. 2. Tap the **debug icon** (ladybug) on the right side of the screen to open the Mock Device Kit sheet. 3. Tap **Enable MockDeviceKit**. This activates the mock device system, which simulates app connection and permission requests. The CameraAccess sample is configured to automatically complete registration when Mock Device Kit is enabled, so the app passes the connection step. 4. Close the sheet to see the main screen with a "Start streaming" button. ![Image showing the Mock Device Kit debug menu](/images/mock-device-kit-debug-menu.png){:width="60%"} You can tap **Disable MockDeviceKit** at any time to return to the real SDK stack. ### Pairing a mock device 1. Open the debug menu and tap **Pair RayBan Meta** to create a simulated device. A mock device card appears in the sheet. 2. Use the toggles on the device card to set the device state: - **Power** -- Simulates powering on the glasses. - **Don** -- Simulates wearing the glasses. This automatically unfolds the device as well. - **Unfold** -- Simulates unfolding the glasses. The device must be powered on and worn (donned) before streaming can start. You can pair up to three mock devices at a time. To remove a device, open the sheet and tap **Unpair** on the device card. ![Image showing mock device controls](/images/mock-device-kit-device-controls.png){:width="60%"} ## Simulating media streaming Once your mock device is powered on and worn, close the Mock Device Kit sheet. The app detects the active device and enables the **Start streaming** button. Before starting a stream, you can configure Mock Device Kit with either a live phone camera feed or a video file to simulate streaming from the glasses. ### Using phone camera as a mock feed You can use your phone's own camera as a live mock camera feed. This is useful for real-time testing and demos without needing pre-recorded video files. 1. Open the Mock Device Kit sheet and expand your paired device card. 2. Tap the **camera source picker** and select **Front Camera** or **Back Camera**. 3. If prompted, grant camera permission to the app. 4. Close the sheet and tap **Start streaming**. The phone's camera feed appears as if it were streaming from the glasses. When using a phone camera as the mock feed, photo capture takes a live still from the active camera session. **Requires phone camera permission.** This feed reads frames from the handset camera, so your app needs its own camera permission. On Android, declare `android.permission.CAMERA` and grant it at runtime. On iOS, add `NSCameraUsageDescription` to `Info.plist` and obtain camera authorization. This is separate from the Device Access Toolkit `CAMERA` permission, which Mock Device Kit always reports as granted. No real-glasses code path uses the handset camera, so an existing integration will not have requested it. Grant the permission before you select the camera source, as the CameraAccess sample does. On Android, Mock Device Kit otherwise rejects the stream start and names the missing permission. To skip camera access entirely, use a video file as the mock feed. ### Using a video file 1. Open the Mock Device Kit sheet and expand your paired device card. 2. Tap the **camera source picker** and select **Video File**. 3. Tap **Select video** and choose a video from your photo library. This video is used as the mock streaming feed. **Note**: Mock Device Kit requires video files in h265 format. The iOS CameraAccess sample app converts video automatically, but the Android sample does not. To transcode a video to h265 manually, you can use [FFmpeg](https://www.ffmpeg.org/). For example: ```bash ffmpeg -hwaccel videotoolbox -i input_video.mp4 -c:v hevc_videotoolbox -c:a aac_at -tag:v hvc1 -vf "scale=540:960" output_video.mov ``` ### Image capture To set a specific image as the mock capture result, tap **Select image** and choose a photo from your photo library. This image is returned when you trigger a photo capture during streaming. If you are using a phone camera as the mock feed, you don't need to set a separate captured image -- capture takes a live photo from the active camera instead. ## Simulating touchpad interactions The mock device card includes buttons for simulating touchpad interactions on the glasses temple: - **Tap** -- Simulates a single touchpad tap. During an active stream, this toggles between pause and resume. - **Tap and Hold** -- Simulates a tap and hold interaction. During an active stream, this stops the session. These inputs let you test how your app responds to user interactions on the glasses without needing physical hardware. ### How to test with Mock Device Kit on iOS ## Overview Use this guide when your iOS project already integrates the Wearables Device Access Toolkit and you need to test without physical glasses. ## Set up Mock Device Kit in XCTest Create a reusable base rule or test class that configures Mock Device Kit, grants permissions, and resets state. ```swift import XCTest import MetaWearablesDAT @MainActor class MockDeviceKitTestCase: XCTestCase { override func setUp() async throws { try await super.setUp() try? Wearables.configure() MockDeviceKit.shared.enable() } override func tearDown() async throws { MockDeviceKit.shared.disable() try await super.tearDown() } } ``` ## Configure camera feeds for streaming tests Mock camera feeds let you verify streaming and capture workflows without video hardware. ### Provide a mock video feed ```swift let device = try MockDeviceKit.shared.pairGlasses(model: .rayBanMeta) let camera = device.services.camera camera.setCameraFeed(fileURL: videoURL) ``` ### Provide a captured photo ```swift let device = try MockDeviceKit.shared.pairGlasses(model: .rayBanMeta) let camera = device.services.camera camera.setCapturedImage(fileURL: imageURL) ``` ## MockDevice test server for XCUITest The MockDevice test server enables UI automation testing by allowing XCUITest processes to control mock devices at runtime via HTTP. This eliminates the need to hardcode mock device setup in your app, enabling dynamic test scenarios where each test can configure its own device state. This approach enables UI automation without physical hardware (run XCUITests on CI simulators), dynamic test scenarios (each test pairs different devices and configures unique camera feeds), full flow testing (registration, pairing, streaming, photo capture), deterministic testing (control exact device state to test edge cases), and parallel test execution (each test manages its own mock device independently). > **Note:** For more examples of how to leverage the MockDevice Test Server to test your apps, see the Camera Access sample code at `ExternalSampleApps/CameraAccess/CameraAccessUITests/`. ### Architecture The test server runs in your app process and listens on localhost. The XCUITest client connects via HTTP to control mock devices. The server writes its port to a temp file that the client reads to discover the connection details. Both processes share the same `/tmp` directory on the simulator. ``` ┌─────────────────────────────────────┐ ┌──────────────────────────────────┐ │ App Process (DEBUG) │ HTTP (localhost) │ XCUITest Process │ │ │ │ │ │ MockDeviceKit │ POST /device/pair │ MockDeviceTestClient │ │ └─ startTestServer() │ POST /device/don │ │ │ │ POST /device/doff │ Port discovery: │ │ │ POST /device/power-on │ reads port from temp file │ │ │ GET /device/state │ written by server on start │ │ │ POST /camera/set-feed │ │ │ │ ... │ │ │ │◄────────────────────────│ │ └─────────────────────────────────────┘ └──────────────────────────────────┘ ``` ### Server setup (app process) Start the test server when the app launches with a `--ui-testing` flag. The server should only be enabled in DEBUG builds. ```swift import MWDATMockDevice if ProcessInfo.processInfo.arguments.contains("--ui-testing") { MockDeviceKit.shared.enable(config: MockDeviceKitConfig(initiallyRegistered: false)) let portFilePath = ProcessInfo.processInfo.environment["MWDAT_TEST_SERVER_PORT_FILE"] Task { try await MockDeviceKit.shared.startTestServer(portFilePath: portFilePath) // Server is now listening — tests drive mock device setup via HTTP } } ``` ### Client setup (XCUITest process) Link `MWDATMockDeviceTestClient` in your UI test target. The client reads the port from the temp file. ```swift import MWDATMockDeviceTestClient import XCTest final class MyUITests: XCTestCase { var portFilePath: String { NSTemporaryDirectory() + "mwdat_test_server_port.txt" } private let app = XCUIApplication() private var mockClient: MockDeviceTestClient! private var deviceId: String? override func setUpWithError() throws { // Remove any stale port file from a previous run try? FileManager.default.removeItem(atPath: portFilePath) app.launchArguments = ["--ui-testing"] app.launchEnvironment["MWDAT_TEST_SERVER_PORT_FILE"] = portFilePath app.launch() // Initialize the client after launch so the server has time to write the port file mockClient = MockDeviceTestClient(portFilePath: portFilePath) XCTAssertTrue(mockClient.waitForServer(timeout: 10), "Test server should be running") } override func tearDownWithError() throws { if let deviceId { mockClient.unpairDevice(deviceId: deviceId) } deviceId = nil } func testStreaming() { // Pair a device — pairDevice() powers it on and dons it automatically guard let id = mockClient.pairDevice() else { XCTFail("Failed to pair mock device") return } deviceId = id // Configure camera resources for this device mockClient.setCameraFeed(deviceId: id, resourceName: "plant", ext: "mp4") mockClient.setCapturedImage(deviceId: id, resourceName: "plant", ext: "png") // Device is now paired, powered on, and donned let startButton = app.buttons["Start streaming"] XCTAssertTrue(startButton.waitForExistence(timeout: 15)) // Control device state mid-test mockClient.doff(deviceId: id) // device becomes inactive mockClient.don(deviceId: id) // device becomes active again mockClient.fold(deviceId: id) // hinges close, streaming stops mockClient.unfold(deviceId: id) // hinges open // Query device state let state = mockClient.getDeviceState() XCTAssertEqual(state?["pairedDeviceCount"] as? Int, 1) } } ``` ### Available client methods | Method | Description | |--------|-------------| | `pairDevice()` | Pairs a mock Ray-Ban Meta, powers on, dons; returns the device ID (nil on failure) | | `unpairDevice(deviceId:)` | Unpairs the specified mock device | | `powerOn(deviceId:)` / `powerOff(deviceId:)` | Controls device power state | | `don(deviceId:)` / `doff(deviceId:)` | Simulates wearing / removing the device | | `fold(deviceId:)` / `unfold(deviceId:)` | Simulates folding / unfolding the glasses | | `captouchTap(deviceId:)` | Simulates a single touchpad tap (toggles pause/resume) | | `captouchTapAndHold(deviceId:)` | Simulates a touchpad tap and hold (stops active session) | | `setCameraFeed(deviceId:resourceName:ext:)` | Sets the camera feed video resource from the test bundle | | `setCapturedImage(deviceId:resourceName:ext:)` | Sets the captured image resource from the test bundle | | `getDeviceState()` | Returns `{"pairedDeviceCount": Int, "deviceIds": [String]}` | | `healthCheck()` | Checks if the server is reachable | | `waitForServer(timeout:)` | Polls until the server responds or timeout expires | ### Swift Package Manager setup Add `MWDATMockDeviceTestClient` as a dependency of your UI test target. The client has no dependency on `MWDATMockDevice` — it communicates purely over HTTP. ### How to test with Mock Device Kit on Android ## Overview Use this guide when your Android project already integrates the Wearables Device Access Toolkit and you need to test without physical glasses. ## Set up Mock Device Kit in instrumentation tests Create a reusable base rule or test class that configures Mock Device Kit, grants permissions, and resets state. ```kotlin import android.content.Context import androidx.test.ext.junit.rules.ActivityScenarioRule import androidx.test.platform.app.InstrumentationRegistry import com.meta.wearable.dat.mockdevice.MockDeviceKit import com.meta.wearable.dat.mockdevice.api.MockDeviceKitInterface import org.junit.After import org.junit.Before import org.junit.Rule open class MockDeviceKitTestCase( private val activityClass: Class ) { @get:Rule val scenarioRule = ActivityScenarioRule(activityClass) protected lateinit var mockDeviceKit: MockDeviceKitInterface protected lateinit var targetContext: Context @Before open fun setUp() { val instrumentation = InstrumentationRegistry.getInstrumentation() targetContext = instrumentation.targetContext mockDeviceKit = MockDeviceKit.getInstance(targetContext) mockDeviceKit.enable() grantRuntimePermissions() } @After open fun tearDown() { mockDeviceKit.disable() } private fun grantRuntimePermissions() { val packageName = targetContext.packageName val shell = InstrumentationRegistry.getInstrumentation().uiAutomation shell.executeShellCommand("pm grant $packageName android.permission.BLUETOOTH_CONNECT") shell.executeShellCommand("pm grant $packageName android.permission.CAMERA") } } ``` `android.permission.CAMERA` is needed only when the mock feed is the phone camera: `setCameraFeed(CameraFacing)` reads frames from the handset, and without the grant Mock Device Kit rejects the stream start. See [Mock Device Kit](/docs/develop/dat/mock-device-kit/). Tests that only use `setCameraFeed(Uri)` can drop that line. ## Configure camera data for streaming and capture Mock camera feeds let you test streaming logic without hardware. The examples below assume assets live under `androidTest/assets`. ### Provide a mock video feed ```kotlin @Test fun testCameraStreaming() { val device = mockDeviceKit.pairGlasses(GlassesModel.RAYBAN_META).getOrThrow() prepareForStreaming(device) val camera = device.services.camera camera.setCameraFeed(getAssetUri("test_video.mp4")) // Assert on streaming state in your UI } ``` ### Provide a captured photo ```kotlin @Test fun testPhotoCapture() { val device = mockDeviceKit.pairGlasses(GlassesModel.RAYBAN_META).getOrThrow() prepareForStreaming(device) val camera = device.services.camera camera.setCapturedImage(getAssetUri("test_image.png")) // Assert on capture results } ``` ### AI-Assisted Development ## Overview The Wearables Device Access Toolkit provides three levels of AI assistance for developers: 1. **Git-native tool integrations** — SDK knowledge delivered directly from the DAT GitHub repos through plugins for Muse Code, Claude Code or Codex, and through repo-native tool files. 2. **Live docs lookup** — The shared Wearables MCP server exposes `search_dat_docs` so your AI tool can search current DAT guides while it works. When your MCP client supports response formats, ask for Markdown output so headings, links, and code blocks stay intact. 3. **API reference endpoint** — The full API surface served via [llms.txt](https://llmstxt.org/) as a supplementary reference for on-demand queries. For implementation work, use the GitHub-hosted DAT plugin or repo-native skills for coding patterns, then use MCP for current docs lookup. The plugin teaches the assistant how DAT projects are structured; MCP keeps the assistant grounded in the latest published guides. Start by copying one of the prompts below into an AI coding tool. Each prompt includes the GitHub repo, setup guidance, MCP endpoint, MCP tool, and docs links so a fresh AI session knows where to start. ## Git-native tool integrations The SDK GitHub repos ship the same public DAT guidance in several formats. Each tool gets the same core knowledge — setup guides, streaming patterns, MockDeviceKit testing, session lifecycle, permissions, debugging, and sample app guidance — in whatever format it expects. | Tool | Public artifact | Recommended setup | |------|-----------------|-------------------| | Muse Code | `.muse-plugin/plugin.json` | Add the DAT GitHub repo as a marketplace, install the DAT plugin, then approve it | | [Claude Code](https://docs.anthropic.com/en/docs/claude-code) | `.claude-plugin/marketplace.json` + plugin manifests | Add the DAT GitHub repo as a marketplace, then install the DAT plugin | | Codex | `.codex-plugin/plugin.json` inside `plugins/` | Add the DAT GitHub repo as a marketplace, then add the DAT plugin | | [GitHub Copilot](https://github.com/features/copilot) | `.github/copilot-instructions.md` | Auto-loaded by Copilot in VS Code | | [Cursor](https://cursor.sh/) | `.cursor/rules/*.mdc` | Auto-loaded with glob-based triggers | | [AGENTS.md](https://agents.md) | `AGENTS.md` | Portable fallback for tools that read `AGENTS.md` | | Tool | Public artifact | Recommended setup | |------|-----------------|-------------------| | [Claude Code](https://docs.anthropic.com/en/docs/claude-code) | `.claude-plugin/marketplace.json` + plugin manifests | Add the DAT GitHub repo as a marketplace, then install the DAT plugin | | Codex | `.codex-plugin/plugin.json` inside `plugins/` | Add the DAT GitHub repo as a marketplace, then add the DAT plugin | | [GitHub Copilot](https://github.com/features/copilot) | `.github/copilot-instructions.md` | Auto-loaded by Copilot in VS Code | | [Cursor](https://cursor.sh/) | `.cursor/rules/*.mdc` | Auto-loaded with glob-based triggers | | [AGENTS.md](https://agents.md) | `AGENTS.md` | Portable fallback for tools that read `AGENTS.md` | See the dedicated setup guides for each tool: [Muse Code](/docs/develop/dat/ai-assisted-muse/), [Claude Code](/docs/develop/dat/ai-assisted-claude-code/), [Codex](/docs/develop/dat/ai-assisted-codex), [GitHub Copilot](/docs/develop/dat/ai-assisted-github-copilot/), [Cursor](/docs/develop/dat/ai-assisted-cursor/), [AGENTS.md](/docs/develop/dat/ai-assisted-agents-md/). See the dedicated setup guides for each tool: [Claude Code](/docs/develop/dat/ai-assisted-claude-code/), [Codex](/docs/develop/dat/ai-assisted-codex), [GitHub Copilot](/docs/develop/dat/ai-assisted-github-copilot/), [Cursor](/docs/develop/dat/ai-assisted-cursor/), [AGENTS.md](/docs/develop/dat/ai-assisted-agents-md/). ### Setup Start from the DAT GitHub repo for your platform when using Muse Code, Claude Code or Codex. Muse Code, iOS: ```bash export MUSE_EXPERIMENTAL_PLUGINS=1 muse plugins marketplace add mwdat-ios-marketplace https://github.com/facebook/meta-wearables-dat-ios muse plugins install mwdat-ios@mwdat-ios-marketplace muse plugins approve mwdat-ios ``` Muse Code, Android: ```bash export MUSE_EXPERIMENTAL_PLUGINS=1 muse plugins marketplace add mwdat-android-marketplace https://github.com/facebook/meta-wearables-dat-android muse plugins install mwdat-android@mwdat-android-marketplace muse plugins approve mwdat-android ``` Claude Code, iOS: ```bash claude plugin marketplace add facebook/meta-wearables-dat-ios claude plugin install mwdat-ios@mwdat-ios-marketplace ``` Claude Code, Android: ```bash claude plugin marketplace add facebook/meta-wearables-dat-android claude plugin install mwdat-android@mwdat-android-marketplace ``` Codex, iOS: ```bash codex plugin marketplace add facebook/meta-wearables-dat-ios codex plugin add mwdat-ios@mwdat-ios-marketplace ``` Codex, Android: ```bash codex plugin marketplace add facebook/meta-wearables-dat-android codex plugin add mwdat-android@mwdat-android-marketplace ``` Then follow the [DAT MCP setup instructions](/docs/develop/dat/ai-assisted-mcp/) to enable live docs lookup for your coding tool. For tools that use repo-native checked-in files instead of plugin manifests, use: ```bash ./install-skills.sh copilot ./install-skills.sh cursor ./install-skills.sh agents ``` ### What the config covers All surfaces receive the same SDK knowledge, covering: - **Getting started** — Dependencies, project setup, and first integration - **Session lifecycle** — Connecting, reconnecting, and managing device sessions - **Camera streaming** — Resolution selection, frame rates, and photo capture - **Permissions** — Requesting and handling device permissions - **MockDeviceKit** — Testing without physical hardware - **Debugging** — Common issues, Developer Mode, version compatibility - **Sample app guide** — Building a complete DAT app from scratch ## API reference endpoint (llms.txt) As a supplement to the GitHub-hosted integrations, the SDK provides an [llms.txt](https://llmstxt.org/) endpoint with the full API surface. The GitHub integrations cover patterns and best practices; the endpoint covers the API reference. ### Endpoints | Endpoint | Description | |----------|-------------| | [wearables.developer.meta.com/llms.txt](https://wearables.developer.meta.com/llms.txt) | Index of available documentation sections with links | | [wearables.developer.meta.com/llms.txt?full=true](https://wearables.developer.meta.com/llms.txt?full=true) | Full API reference content in a single document | The index endpoint follows the [llms.txt specification](https://llmstxt.org/) and returns a lightweight listing of documentation sections. The `?full=true` variant returns the complete API reference inline, which is what most AI tools need to write code. ### What's included The endpoint serves API reference documentation for both iOS (Swift) and Android (Kotlin) platforms, covering: - `MWDATCore` — App registration, device discovery, session management, and telemetry - `MWDATCamera` — Camera access, resolution and frame rate selection, and photo capture - `MWDATMockDevice` — Simulated device for testing without physical hardware ## Tips for effective use - **Start with the GitHub-hosted integration** — Use the Muse Code, Claude Code or Codex plugin when your tool supports it. Use the repo-native file installs for Copilot, Cursor, or `AGENTS.md` readers. This gives the assistant DAT-specific coding patterns, setup flow, and test guidance. - **Use MCP for live docs lookup** — Follow the [DAT MCP setup instructions](/docs/develop/dat/ai-assisted-mcp/) for your coding tool. Ask your AI tool to call `search_dat_docs` before choosing SDK APIs and request Markdown-formatted results when the client supports response formats. - **Add the API reference when you need specifics** — If your AI tool can't find a particular method signature or parameter type, point it at the llms.txt endpoint for the full API surface. - **Be specific in your prompts** — Mention the platform (iOS or Android) and the module you're working with (`MWDATCore`, `MWDATCamera`, or `MWDATMockDevice`). - **Combine with the guides** — For deeper integration patterns and lifecycle management, point your AI tool at the [integration overview](/docs/develop/dat/build-overview/) and platform-specific integration guides ([iOS](/docs/develop/dat/build-integration-ios/), [Android](/docs/develop/dat/build-integration-android/)). ### Muse Code ## Overview Install the DAT plugin from the SDK GitHub repo to give [Muse Code](https://developer.meta.com/ai/products/muse-code/) context on the SDK. Once installed and approved, Muse Code can help you: - Set up dependencies, app registration, and the first working integration - Manage session and stream lifecycle, pause and resume behavior, and device availability - Configure camera streaming, resolution, and frame rates - Request and handle device permissions - Test without physical glasses using MockDeviceKit - Look up live DAT documentation with `search_dat_docs` Muse Code also reads `AGENTS.md` from your project root with no setup at all. See [AGENTS.md](/docs/develop/dat/ai-assisted-agents-md/) if you want the portable fallback instead of the plugin. ## Setup Add the SDK repo as a marketplace, install the plugin, then approve it. Muse Code plugin support is experimental. Run `export MUSE_EXPERIMENTAL_PLUGINS=1` in your shell before using the plugin commands. iOS: ```bash muse plugins marketplace add mwdat-ios-marketplace https://github.com/facebook/meta-wearables-dat-ios muse plugins install mwdat-ios@mwdat-ios-marketplace muse plugins approve mwdat-ios ``` Android: ```bash muse plugins marketplace add mwdat-android-marketplace https://github.com/facebook/meta-wearables-dat-android muse plugins install mwdat-android@mwdat-android-marketplace muse plugins approve mwdat-android ``` The approve step is required. Muse Code holds a newly installed plugin's contributed capabilities for review, so the DAT guidance and live docs lookup stay inactive until you approve the plugin by name. Installing the plugin also registers the hosted DAT docs MCP server. You do not need to add the server separately or edit `~/.config/muse/settings.json`. If you cloned the SDK repo, you can run the bundled installer instead: ```bash ./install-skills.sh muse ``` Or install remotely: iOS: ```bash curl -sL https://raw.githubusercontent.com/facebook/meta-wearables-dat-ios/main/install-skills.sh | bash -s muse ``` Android: ```bash curl -sL https://raw.githubusercontent.com/facebook/meta-wearables-dat-android/main/install-skills.sh | bash -s muse ``` The installer requires `muse` on your `PATH` and runs the same three commands, including the approve step. ## Usage Ask SDK questions directly in a Muse Code session: ```text Set up a DAT session with camera streaming and handle lifecycle events. ``` ## Live docs lookup After approving the plugin, start a new session and confirm that the DAT guidance and `search_dat_docs` are available before editing code. The public docs MCP server does not require authentication. Example prompt: ```text Use the installed DAT plugin for platform-specific implementation patterns. Then ask search_dat_docs for current camera streaming and session lifecycle guidance. Inspect my project first, make the smallest safe change, and run the relevant local checks. ``` ## Adding the API reference The plugin covers integration patterns and best practices. To add the full API reference, point Muse Code at the [llms.txt](https://llmstxt.org/) endpoint in your prompt: ```text Read https://wearables.developer.meta.com/llms.txt?full=true for the full Wearables Device Access Toolkit API surface before choosing method signatures or parameter types. ``` ### Claude Code ## Overview [Claude Code](https://docs.anthropic.com/en/docs/claude-code) should use the DAT GitHub repository as a plugin marketplace. This keeps Claude on the same git-backed payload that the DAT repos publish, instead of relying on hand-copied `.claude/` files. Once loaded, Claude Code has full context on the SDK and can help you: - Set up your project and add DAT SDK dependencies - Implement camera streaming, session lifecycle, and permissions - Write tests with Mock Device Kit - Debug common integration issues ## Setup Add the DAT repository as a marketplace, then install the platform plugin. iOS: ```bash claude plugin marketplace add facebook/meta-wearables-dat-ios claude plugin install mwdat-ios@mwdat-ios-marketplace ``` Android: ```bash claude plugin marketplace add facebook/meta-wearables-dat-android claude plugin install mwdat-android@mwdat-android-marketplace ``` ## Usage Start Claude Code in your project directory: ```bash claude ``` You can then ask it about any DAT SDK topic, have it implement a specific app functionality, or make it write tests — the DAT plugin is already loaded: > How do I set up camera streaming in my Android app? > Write a test for my session lifecycle using MockDeviceKit ## Live docs lookup Installing the DAT plugin also registers the hosted DAT docs MCP server. No separate MCP configuration is required. The DAT plugin covers platform-specific implementation patterns and best practices. Add the Wearables MCP endpoint when you also want Claude Code to look up current DAT guides while it works: ```bash claude mcp add --transport http wearables https://mcp.developer.meta.com/wearables ``` Start a fresh Claude Code session if the running session cannot discover `search_dat_docs`. Confirm that both the DAT plugin and `search_dat_docs` are available before editing code. Example prompt: Use the installed DAT plugin for platform-specific implementation guidance. Then ask search_dat_docs for current camera streaming and session lifecycle guidance. Inspect my project first, make the smallest safe change, and run the relevant local checks. ## Adding the API reference The DAT plugin covers integration patterns and best practices. To also include the full API reference, paste the llms.txt URL directly into your Claude Code prompt: > Using https://wearables.developer.meta.com/llms.txt?full=true as reference, what parameters does Stream accept? Or add it to your local project guidance so it loads automatically: ## API Reference Fetch https://wearables.developer.meta.com/llms.txt?full=true for the Wearables Device Access Toolkit API reference. Claude Code will fetch and read the URL contents automatically when you include it in your prompt. ### Codex ## Overview Codex should use the DAT GitHub repository as a plugin marketplace. This keeps Codex on the same git-backed DAT guidance that is published for Claude Code and other tools, instead of relying on pasted prompt text. Once loaded, Codex has DAT-specific context and can help you: - Set up your project and add DAT SDK dependencies - Implement camera streaming, display, session lifecycle, and permissions - Write tests with Mock Device Kit - Debug common integration issues For the best results, combine the Codex plugin with the [DAT MCP server](/docs/develop/dat/ai-assisted-mcp/). The plugin provides coding patterns from the GitHub repo; MCP provides current published docs through `search_dat_docs`. ## Setup Add the DAT repository for your platform as a Codex plugin marketplace, then add the platform plugin. iOS: ```bash codex plugin marketplace add facebook/meta-wearables-dat-ios codex plugin add mwdat-ios@mwdat-ios-marketplace ``` Android: ```bash codex plugin marketplace add facebook/meta-wearables-dat-android codex plugin add mwdat-android@mwdat-android-marketplace ``` If you already have a checkout of the DAT GitHub repo, add it as a local marketplace from the repo root, then add the same plugin: ```bash codex plugin marketplace add . codex plugin add mwdat-ios@mwdat-ios-marketplace ``` Use `mwdat-android@mwdat-android-marketplace` for an Android checkout. ## Live docs lookup Installing the DAT plugin also registers the hosted DAT docs MCP server. No separate MCP configuration is required. The Codex plugin covers integration patterns and best practices. Add the Wearables MCP endpoint when you also want Codex to look up current DAT docs while it works: ```bash codex mcp add wearables --url https://mcp.developer.meta.com/wearables ``` Start a fresh Codex session if the running session cannot discover `search_dat_docs`, then confirm that both the plugin and tool are available before editing code. When Codex calls `search_dat_docs`, ask it to request Markdown-formatted results if the MCP client supports output formats. Example prompt: Use the installed DAT Codex plugin for implementation guidance. Then call search_dat_docs for current iOS camera streaming guidance, with Markdown-formatted results if available. Inspect my project first, make the smallest safe change, and run the relevant local checks. ## Usage Start Codex in your project directory and ask DAT-specific questions directly: How do I set up camera streaming in my Android app? Write a test for my session lifecycle using MockDeviceKit. Implement a DAT display session for iOS and explain the required permission flow. ## Adding the API reference The DAT plugin covers setup and integration guidance. If you need exact API signatures or full reference content, provide the llms.txt URL in your prompt: Use https://wearables.developer.meta.com/llms.txt?full=true as reference and show the current StreamSession configuration options. Or add it to your local project guidance so it loads automatically: ## API Reference Fetch https://wearables.developer.meta.com/llms.txt?full=true for the Wearables Device Access Toolkit API reference. Codex can use the plugin, MCP docs search, and llms.txt together: plugin for project patterns, MCP for current guides, and llms.txt for exact API reference details. ### GitHub Copilot ## Overview [GitHub Copilot](https://github.com/features/copilot) auto-loads `.github/copilot-instructions.md` when you open a project in VS Code. The SDK knowledge is available in Copilot Chat immediately. Copilot also provides inline completions as you write DAT SDK code — the project config gives it enough context to suggest correct API usage, parameter names, and patterns. ## Setup If you cloned the SDK repo, the `.github/copilot-instructions.md` file is already included. Otherwise, install with the CLI: ```bash ./install-skills.sh copilot ``` Or install remotely: iOS: ```bash curl -sL https://raw.githubusercontent.com/facebook/meta-wearables-dat-ios/main/install-skills.sh | bash -s copilot ``` Android: ```bash curl -sL https://raw.githubusercontent.com/facebook/meta-wearables-dat-android/main/install-skills.sh | bash -s copilot ``` ## Usage Use Copilot Chat to ask SDK questions directly: How do I request camera permissions in my iOS DAT app? ## Adding the API reference The project-level config covers integration patterns and best practices. To also pull in the full API reference during a chat session, paste the URL directly into your Copilot Chat prompt: Using https://wearables.developer.meta.com/llms.txt?full=true as reference, what parameters does MWDATCameraSession.startStreaming accept? Copilot Chat fetches and reads URL contents automatically when you include them in your prompt. ### Cursor ## Overview [Cursor](https://cursor.sh/) auto-loads `.cursor/rules/*.mdc` files with glob-based triggers. When you're editing a file that matches a rule's glob pattern, Cursor loads the relevant SDK knowledge automatically. ## Setup If you cloned the SDK repo, the `.cursor/` directory is already included. Otherwise, install with the CLI: ```bash ./install-skills.sh cursor ``` Or install remotely: iOS: ```bash curl -sL https://raw.githubusercontent.com/facebook/meta-wearables-dat-ios/main/install-skills.sh | bash -s cursor ``` Android: ```bash curl -sL https://raw.githubusercontent.com/facebook/meta-wearables-dat-android/main/install-skills.sh | bash -s cursor ``` ## Usage Use Cursor Chat or inline editing to ask SDK questions: Set up a DAT session with camera streaming and handle lifecycle events. ## Add live docs lookup The repo-native Cursor guidance covers platform-specific implementation patterns and best practices. Add the Wearables MCP endpoint when you also want Cursor to look up current DAT guides while it works: 1. Open **Settings** and go to **MCP**. 2. Add a server named `wearables` with the `HTTP` transport. 3. Set its URL to `https://mcp.developer.meta.com/wearables`. 4. Reconnect the server or run **Developer: Reload Window**. Confirm that the DAT guidance and `search_dat_docs` are available before editing code. Example prompt: Use the installed DAT Cursor guidance for platform-specific implementation patterns. Then ask search_dat_docs for current camera streaming and session lifecycle guidance. Inspect my project first, make the smallest safe change, and run the relevant local checks. ## Adding the API reference The project-level config covers integration patterns and best practices. To add the full API reference as a persistent documentation source: 1. Open **Settings** > **Features** > **Docs**. 2. Click **Add new doc**. 3. Enter `https://wearables.developer.meta.com/llms.txt?full=true` as the URL. 4. Name it "Wearables Device Access Toolkit". Once added, reference it in Cursor Chat with `@Docs`, and then select the Wearables Device Access Toolkit entry. ### DAT MCP ## Overview The recommended AI-assisted setup for the Wearables Device Access Toolkit combines the DAT GitHub plugin or repo-native skills for your platform with the shared public Wearables MCP server. Choose iOS or Android first. The plugin or skills provide platform-specific implementation and testing workflows; MCP provides current documentation lookup while the assistant works. Follow the client-specific setup below to make both the DAT guidance and live docs lookup available. The server exposes a DAT-specific tool: - `search_dat_docs` — semantic search over DAT guides, API reference, and code examples When your client exposes an output or response format option, choose Markdown. Markdown results preserve headings, links, tables, and code blocks, which makes it easier for the assistant to quote the docs accurately before editing code. The public endpoint is: ```text https://mcp.developer.meta.com/wearables ``` Use the direct MCP host above. Do not use the Wearables developer site URL as the MCP endpoint. This server complements the project-level AI config described in the [AI-Assisted Development overview](/docs/develop/dat/ai-assisted/). Use [llms.txt](https://wearables.developer.meta.com/llms.txt?full=true) as a supplementary source when you need exact API reference details. ## Muse Code Enable experimental plugins, add the DAT repository for your platform as a Muse Code plugin marketplace, install its plugin, then approve the plugin. iOS: ```bash export MUSE_EXPERIMENTAL_PLUGINS=1 muse plugins marketplace add mwdat-ios-marketplace https://github.com/facebook/meta-wearables-dat-ios muse plugins install mwdat-ios@mwdat-ios-marketplace muse plugins approve mwdat-ios ``` Android: ```bash export MUSE_EXPERIMENTAL_PLUGINS=1 muse plugins marketplace add mwdat-android-marketplace https://github.com/facebook/meta-wearables-dat-android muse plugins install mwdat-android@mwdat-android-marketplace muse plugins approve mwdat-android ``` Installing the plugin also registers the hosted DAT docs MCP server. You do not need to add it separately. Start a fresh Muse Code session if the running session does not discover the plugin or `search_dat_docs`, then confirm that both are available. Example prompt: Use the installed DAT plugin for platform-specific implementation guidance. Then ask search_dat_docs what I should install and check before building my first DAT app. ## Claude Code Add the DAT repository for your platform as a Claude Code plugin marketplace, then install its plugin. iOS: ```bash claude plugin marketplace add facebook/meta-wearables-dat-ios claude plugin install mwdat-ios@mwdat-ios-marketplace ``` Android: ```bash claude plugin marketplace add facebook/meta-wearables-dat-android claude plugin install mwdat-android@mwdat-android-marketplace ``` Installing the plugin also registers the hosted DAT docs MCP server. You do not need to add it separately. Add the Wearables MCP server: ```bash claude mcp add --transport http wearables https://mcp.developer.meta.com/wearables ``` Start a fresh Claude Code session if the running session does not discover the plugin or `search_dat_docs`, then confirm that both are available. Example prompt: Use the installed DAT plugin for platform-specific implementation guidance. Then ask search_dat_docs what I should install and check before building my first DAT app. ## Codex Add the DAT repository for your platform as a Codex plugin marketplace, then install its plugin. iOS: ```bash codex plugin marketplace add facebook/meta-wearables-dat-ios codex plugin add mwdat-ios@mwdat-ios-marketplace ``` Android: ```bash codex plugin marketplace add facebook/meta-wearables-dat-android codex plugin add mwdat-android@mwdat-android-marketplace ``` Installing the plugin also registers the hosted DAT docs MCP server. You do not need to add it separately. Add the Wearables MCP server: ```bash codex mcp add wearables --url https://mcp.developer.meta.com/wearables ``` Start a fresh Codex session if the running session does not discover the plugin or `search_dat_docs`, then confirm that both are available. Example prompt: Use the installed DAT plugin for platform-specific implementation guidance. Then ask search_dat_docs what I should install and check before building my first DAT app. ## Cursor Install the repo-native Cursor guidance for your platform. iOS: ```bash curl -sL https://raw.githubusercontent.com/facebook/meta-wearables-dat-ios/main/install-skills.sh | bash -s cursor ``` Android: ```bash curl -sL https://raw.githubusercontent.com/facebook/meta-wearables-dat-android/main/install-skills.sh | bash -s cursor ``` Restart Cursor or run **Developer: Reload Window** so it discovers the DAT guidance. Then add the endpoint as an HTTP MCP server: 1. Open **Settings** and go to **MCP**. 2. Add a new server. 3. Set the name to `wearables`. 4. Set the transport or server type to `HTTP`. 5. Use `https://mcp.developer.meta.com/wearables` as the URL. If your Cursor build uses JSON-backed MCP settings, the entry should look like this: ```json { "mcpServers": { "wearables": { "type": "http", "url": "https://mcp.developer.meta.com/wearables" } } } ``` After saving the config, reconnect the server or restart Cursor. Confirm that the DAT guidance and `search_dat_docs` are available, then ask Cursor Chat to search the DAT docs through the MCP tool. ## MCP Inspector Use MCP Inspector for a direct endpoint check: ```bash npx @modelcontextprotocol/inspector ``` In the browser UI: 1. Set **Transport Type** to `Streamable HTTP`. 2. Set **URL** to `https://mcp.developer.meta.com/wearables`. 3. Set **Connection Type** to `Direct`. 4. Click **Connect**. Run: 1. Initialize the session. 2. Open **Tools** and click **List Tools**. 3. Confirm that `search_dat_docs` is present. 4. Run `search_dat_docs` with a query such as `camera streaming`, and request Markdown output if your client exposes a format option. If your Inspector build labels the transport as `HTTP` instead of `Streamable HTTP`, use the HTTP option with the same URL. ## Example queries These are representative queries we tested against the DAT MCP: - `I want to build my first DAT app with an AI coding assistant. What should I install and check before I start?` Returns platform plugin or skills guidance, MCP setup, environment requirements, and relevant first-project documentation. - `How do I stream camera on Ray-Ban Meta glasses?` Returns the camera streaming guide, including `Session.addCamera(...)`, `StreamConfiguration`, valid frame rates, quality levels, and device-side caveats such as hinge-close and doff behavior. - `How do I initialize the DAT SDK?` Returns SDK startup and device session details such as `Wearables.initialize(context)`, `WearablesError.NOT_INITIALIZED`, `SessionState`, and the platform-specific session state listeners. - `How do I test DAT integrations without physical glasses?` Returns Mock Device Kit guidance, including the `Mock Device Kit` docs, simulated media streaming, and the Android and iOS MDK testing guides. - `Bluetooth connection lifecycle events` Returns `SessionState` transition behavior, device availability callbacks such as `devicesMetadata` on Android and `addLinkStateListener` on iOS, and the fact that reopening the hinges restores Bluetooth but does not restart the session. ## Troubleshooting | Symptom | Likely cause | Fix | |---------|--------------|-----| | The server does not connect | The client is still using the old DAT-specific URL or cached config | Re-enter `https://mcp.developer.meta.com/wearables` and reconnect | | No tools appear | The client did not finish initialization | Reconnect, then run initialize before `tools/list` | | The client asks for auth | The server was configured with stale custom headers | Remove custom auth headers and reconnect | | Search results are too broad | The query is too vague | Include the platform, module, or exact API name in the query | ### AGENTS.md ## Overview [AGENTS.md](https://agents.md) is the DAT SDK fallback path for AI coding agents that do not support the Claude marketplace or the Codex plugin format directly. It provides a universal, tool-agnostic way to give an AI assistant project context. ## What is AGENTS.md? `AGENTS.md` is a predictable, discoverable file at the repo root that any AI coding agent can find and use. Key properties: - **Standard Markdown** — no required fields or special syntax - **Closest file wins** — supports nested files for monorepos - **Open standard** — stewarded by the Linux Foundation's Agentic AI Foundation - **Widely supported** — works with Codex, Gemini CLI, Devin, Windsurf, Jules, Cursor, VS Code, Zed, Aider, and others It's complementary to `README.md`, but where README is for humans, AGENTS.md is for AI agents, providing additional context that might otherwise clutter a README. For DAT, prefer: 1. Claude marketplace install 2. Codex plugin add 3. `AGENTS.md` only when your tool does not support those native paths ## Which tools support it? AGENTS.md is supported by a growing ecosystem of AI coding tools, all of which have support for auto-discovery: * OpenAI Codex * Google Gemini CLI * Devin * Windsurf * Jules * Cursor * VS Code (GitHub Copilot) * Zed * Aider ## What's included The Device Access Toolkit SDK's `AGENTS.md` file contains the same knowledge as our other tool configs — SDK architecture, API patterns, setup instructions, testing guides, and debugging tips — but structured for universal agent consumption: - **Code style** — Architecture, naming conventions, error handling patterns - **Dev environment tips** — SDK setup, dependency management, initialization - **Testing instructions** — MockDeviceKit setup and test patterns - **Building and streaming** — Stream, VideoFrame, photo capture - **Session management** — Device session states, pause/resume behavior - **Permissions** — Registration and camera permission flows - **Debugging** — Common issues, Developer Mode, version compatibility - **Sample app** — Complete end-to-end integration example ## Setup ### Already included in the repo Both SDK repositories include `AGENTS.md` at the root: - [Android](https://github.com/facebook/meta-wearables-dat-android) — `AGENTS.md` - [iOS](https://github.com/facebook/meta-wearables-dat-ios) — `AGENTS.md` Once you clone the repo, any supported AI tool will auto-discover it. ### Adding to your own project Use the installer script when your tool needs `AGENTS.md` specifically: Android: ```bash curl -sL https://raw.githubusercontent.com/facebook/meta-wearables-dat-android/main/install-skills.sh | bash -s -- agents ``` iOS: ```bash curl -sL https://raw.githubusercontent.com/facebook/meta-wearables-dat-ios/main/install-skills.sh | bash -s -- agents ``` Or install another repo-native compatibility file explicitly: ```bash curl -sL https://raw.githubusercontent.com/facebook/meta-wearables-dat-android/main/install-skills.sh | bash -s -- cursor ``` ## Other tool configs The SDK also includes first-class Claude and Codex plugin payloads plus compatibility config for Copilot and Cursor. See the [AI-Assisted Development overview](/ai-assisted/) for the full list. ### Onboarding and organization management Wearables Developer Center manages the full lifecycle of wearables integrations, from development and testing to app sharing. It oversees integration projects, versions, and release channels. To manage your team and projects effectively, you need to understand organizational roles and account requirements. This guide explains how to set up and manage your organization, team, and members. ## One organization per company **Important:** Each company must have only **one** Managed Meta Account (MMA) organization in [Admin Center.](https://work.meta.com/admin/work_tools_overview) **Do not create a new MMA organization if one already exists for your company.** Check with your IT, engineering lead, or project manager before proceeding. ## Key terms | Term | Definition | | ------------------------------ | ------------------------------------------------------------------------------ | | **Managed Meta Account (MMA)** | A Meta account managed by an organization admin for secure access and control. | | **Admin Center** | A portal for managing IT tasks related to people management and security. | | **Organization** | Represents your company in Admin Center | | **Team** | A group within Wearables Developer Center representing your project team. | ## Set up your organization and team ### 1. Check for an existing MMA Organization - **Before you start:** Ask your company's IT, engineering lead, or project manager if an MMA organization already exists in Admin Center. ### 2. Designate an organization admin - Only one person (ideally IT, engineering lead, or project manager) should create the MMA organization for your company. - This person becomes the MMA organization admin and manages membership for all contributors. You can change [admin roles in Admin Center](https://work.meta.com/help/632623761283671) if needed. ### 3. Create the MMA organization in Admin Center - The admin should sign up to Wearables Developer Center. During this process they will be redirected to the MMA setup in [work.meta.com](https://work.meta.com/). - Use your company's official name for the organization. ### 4. Invite developers to the MMA organization - The admin invites all developers and contributors who need access to Wearables Developer Center. - Invited members will receive prompts to create or link their MMA. - **Note:** Only members of your company's MMA organization can join your Wearables Developer Center team. ### 5. Access Wearables Developer Center - Once your MMA and organization membership are set up, log into Wearables Developer Center with your MMA credentials. ### 6. Invite team members in Wearables Developer Center - You will have a default personal team when you first log into Wearables Developer Center with your MMA account. You can create a new team and/or add people to your existing team. You don't have to be admin in Admin Center to do this. - Only developers in the same MMA organization can join the Wearables Developer Center team. - Use the **Invite Member** process. If invitees lack an MMA, they will be prompted to create one and join the organization. ## Get started in Wearables Developer Center Use Wearables Developer Center to: - Create and manage AI glasses projects, including device permissions and connectivity. - Manage integration versions (Major, Minor, Patch). - Invite testers to [release channels](https://wearables.developer.meta.com/docs/set-up-release-channels). ## Invite team members: Admin rights - **If you're an admin in Admin Center:** - **Option 1:** Set up the new member's MMA first, then invite them via Wearables Developer Center. - **Option 2:** Invite directly from Wearables Developer Center; if the person lacks an MMA, they will receive an email to create one. - **If you are not an admin:** - You can only invite people who already have an MMA. Organizational setup remains the administrator's responsibility. ## Add, remove, or leave a team **Add a member:** 1. Select your team from the team selector in the header of Wearables Developer Center. 2. Click **Team** in the left menu. 3. Click **Invite member**. 4. Enter the member's email (must be linked to a Managed Meta account). 5. The invitee receives an invitation email. **Remove a member:** 1. Select your team from the team selector in the header. 2. Click **Team** in the left menu. 3. Find the member in the **Members** tab and click **Remove member**. 4. Confirm by clicking **Remove**. **Leave a team:** 1. Select your team from the team selector in the header. 2. Click **Team** in the left menu. 3. Next to your name in the **Members** tab, click **Remove member**. 4. Confirm by clicking **Remove Myself**. ### Manage projects Once you have [onboarded](https://wearables.developer.meta.com/docs/onboarding-and-organization-management), you can create a project or manage existing ones in [Meta Wearables Developer Center](https://wearables.developer.meta.com/). ## Projects You can create new projects or manage your existing ones directly in the Wearables Developer Center. ## Create a project 1. Click **New project**. 2. Give your project a name (what you want to call it) and a brief description (what it does). ## Configuration You can connect your own mobile apps with your Meta wearable device by defining the app details. 1. Click **Configuration** in the project sidebar. 1. Add the requested details for a mobile app you want to integrate with Meta wearable devices. ### Application ID integration To register your application successfully (without using Developer Mode), you must include the Wearables Application ID in your app's manifest and pass it in the registration call. Copy and paste the integration details into your iOS or Android application build to complete this step. If your bundle ID and package name are different for iOS and Android, respectively (for example, com.myapp.android vs. com.myapp.ios), you will need to create two separate apps — one that defines only the Android platform and another that defines only the iOS platform. > **Note:** A hyphen `-` is *not* supported for iOS bundle IDs. ## Product listing **App name and icon** - You need to provide your app's name and an icon. - The icon must be in PNG or JPEG format. - Separate icons for dark and light mode are supported. - The maximum supported dimensions for the icon are 200x200 pixels. These details will also be visible to other users in the Meta AI app when they [adjust permissions](https://wearables.developer.meta.com/docs/set-up-release-channels#manage-permissions-for-connected-apps). ## Permissions If your app or project needs access to device functionality like the camera or microphone, you must provide a justification in the **Permissions** tab. This justification is for Meta's internal review only and is not shown to end-users. Reviewers use your explanation to determine if the permission is necessary and appropriate for your app's functionality. > **Note:** Available permissions include camera, microphone, and voice invocation. New device capabilities may be added in future iterations. ## Distribute When you're ready for people to try your project, you need to [set up release channels](https://wearables.developer.meta.com/docs/set-up-release-channels). Device Access Toolkit automatically generates a corresponding application for each new version. Builds usually complete within 10 minutes, but may take up to 30 minutes. Use a version with a release channel only after its Device Access Toolkit app finishes building. Monitor build status in the Distribution dashboard: - **N/A:** Version is missing build artifact. - **In Progress:** Device Access Toolkit application is still building. - **Ready:** Device Access Toolkit application is ready to distribute. - **Failed:** Device Access Toolkit application build failed. Builds rarely fail. Create a new version to resolve build failures. ## Remove a project When you no longer need a project, you can remove it from the Meta Wearables Developer Center. Removed projects are kept for a recovery period before they are permanently deleted, so you can restore a project if you change your mind during this time. ### Before you remove a project Before you can remove a project, make sure it has no active version on the production release channel. If a version is currently distributed on the production channel, remove that version from the channel first. See [Manage versions and release channels](https://wearables.developer.meta.com/docs/set-up-release-channels) for details. > **Note:** Users who have already connected to your application will still have access until they manually disconnect their device. ### Remove a project 1. Open your project from the dashboard. 2. Go to **Overview** and click **Remove project**. 3. To confirm, type the project name exactly as it's shown. 4. Click **Remove**. After you remove a project, it no longer appears in your active projects list. You can find it under the **Deleted projects** view from your organization dashboard. ### Recovery window Removed projects are kept for **180 days**. During this period, you can restore the project and return it to development mode. After 180 days, the project and its configuration are permanently deleted and can no longer be recovered. ### Restore a removed project 1. From your organization dashboard, open the **Deleted projects** view. 2. Locate the project you want to restore. 3. Click **Restore**. The restored project returns to development mode. You can continue editing its configuration, create new versions, and assign them to release channels as usual. Production release channels and previously distributed versions are not automatically re-published; review your distribution setup before sharing the project with testers again. ### Manage versions and release channels in Meta Wearables Developer Center YOu can manage how you distribute and test Meta integrations by creating versions and configuring the private release channels that Meta Wearables Developer Center creates for your integration. Please note that app integration versions created *prior* to the release of Device Access Toolkit 1.0 will not work with builds that use this version of the SDK. ## Understand versions Wearables Developer Center uses a versioning system that helps track changes and maintain stability across your integrations. Each version details product specifics, including the name, icon, and any edits to permission requests or app configuration. After you add and save these details you can find them by going to **Distribute > Version details > Project data**. When you change any of these details, you need to create a new version of the integration so you can distribute it to testers on a release channel. When selecting the version to use, the type of change you are making determines the category you should choose: - **Major (for example, 2.3.4 to 3.0.0):** Choose this for significant changes or API revisions that are not guaranteed to maintain compatibility with previous versions. For example, select a major version if you change core app functionality in a way that breaks existing features. - **Minor (for example, 2.3.4 to 2.4.0):** Select a minor version when introducing new features while still maintaining backwards compatibility. For example, if you add a new button or feature. - **Patch (for example, 2.3.4 to 2.3.5):** Use a patch version for fixing bugs or delivering minor improvements that do not break compatibility, such as correcting a typo or a small bug fix. ## Create versions To create a new version of your integration: 1. Log in to the [Meta Wearables Developer Center](https://wearables.developer.meta.com/). 2. Select your project from the dashboard. 3. Go to the **Distribute** menu and choose **Versions**. 4. Click **Create new version**. 5. Select your version type (**Major, Minor, or Patch**). 6. Click **Create version**. ## About release channels Release channels let you distribute a version of your integration to invited testers before release. Meta Wearables Developer Center automatically creates one private, invite-only release channel named **Beta** when you create an integration. Each integration supports one private release channel. You can rename the channel, update its description, change the version it distributes, and manage its testers. A release channel distributes one version at a time. ### Release channel access - **User invitations by email:** Invite testers using the email address associated with their [Meta account](https://developers.meta.com/horizon/blog/introducing-meta-accounts-what-developers-need-to-know/). - **Tester autonomy:** Testers may accept or decline invitations and can remove themselves at any time. - **Developer control:** You can revoke tester access at any point. You can also reinvite users you have previously revoked. - **User limit:** Check the release channel's **Users** tab for its current tester limit. ## Configure your private release channel To configure the private release channel: 1. Log in to the [Meta Wearables Developer Center](https://wearables.developer.meta.com/). 2. Select your project. 3. Go to **Distribute > Release channels**. 4. Open the actions menu for the **Beta** release channel and select **View**. 5. Optional: On the **Overview** tab, update the channel's **Name** or **Description**, then select **Save changes**. 6. On the **Version** tab, select **Edit**, choose the version to distribute, then select **Save changes**. 7. On the **Users** tab, select **+ Invite users** and enter the email addresses associated with the testers' Meta accounts. 8. Select **Send invites**. Each tester must have an existing Meta account. A Managed Meta Account is not supported. If a tester needs a Meta account, they can [create one](https://auth.meta.com/). ## Manage test user access Each invited tester must accept the email invitation to join the integration's private release channel. Developers can remove testers, and testers can leave at any time. **Note:** Release channels control a user's ability to register an app integration. Removing a user from a channel after they've registered will not unregister the connected app for Meta AI and the wearable device. To manage test users, open the channel's actions menu, select **View**, then select the **Users** tab. You can change the distributed version from the **Version** tab. Test users can view the integrations they are testing at: [https://wearables.meta.com/invites](https://wearables.meta.com/invites) ## Manage permissions and confirm the release channel in the Meta AI app People testing your integration can manage app permissions and confirm the active release channel for devices and connected apps in the Meta AI app. ## Manage permissions for connected apps As a test user, managing permissions lets you control what each integration can access on your device. To manage permissions: 1. Open the Meta AI app. 2. Go to the device menu and tap **Settings**. 3. Select **App connections** to see a list of all apps linked to your Meta AI account. 4. Tap on an app to view its permissions. 5. Adjust specific permissions. For example, for the camera, you may see options like: - Always allow - Allow once - Don't allow 6. Click **Confirm** to save your changes. **Note:** Changes made to these settings will apply to all devices connected to your Meta AI app. ## Confirm the release channel After a tester accepts an invitation, the integration's private release channel is selected by default. To confirm the release channel: 1. Open the Meta AI app. 2. Go to the device menu and tap **Settings**. 3. Tap **Release Channel**. 4. Confirm that the integration's private release channel is selected. If it is not selected, select it. 5. Click **Confirm** to save your changes. ![switch release channel](/images/switch-release-channel.gif) Learn [How to disconnect apps from AI glasses](https://www.meta.com/help/ai-glasses/836668612353969/). ### Access your Meta Wearables Device Access Toolkit information You can request a copy of your information related to Wearables Device Access Toolkit integrations. You will get a file containing your profile information and other data. ### Telemetry ## Overview This page describes the telemetry data that the Device Access Toolkit SDK collects from Android and iOS apps that integrate with AI glasses. The SDK collects operational and diagnostic telemetry (for example, device discovery, registration flows, streaming session lifecycle, permission checks, attestation, and performance markers) to support reliability, performance monitoring, troubleshooting, and product improvement. In turn, telemetry helps Meta identify issues, measure SDK health, and improve the developer experience. Some of this collected data includes: * Device identifiers * Firmware versions * Session durations * Error types * Success/failure flags ## Opt Out Telemetry collection is enabled by default, but you can disable it at the app level. ### Android Add the following to your `AndroidManifest.xml` inside ``: ```xml ``` ### iOS Add the following to your `Info.plist`: ```xml MWDAT Analytics OptOut ``` ## Data Categories The following categories of telemetry data are collected via the SDK: ### Application and SDK identification Identifies which third-party app and SDK version generated an event, enabling usage attribution and version-specific issue diagnosis. **Examples:** * App bundle ID / package name, app name, and app version * Public-key SHA-256 * App session UUID and session start time, which resets on each app start * SDK version (for example, `0.7.0`) ### Device and platform context Describes the phone and glasses involved in an interaction. **Examples:** * Phone OS, OS version, and device model * Glasses device identifier, device type (for example, `RAYBAN_META`) * Glasses SoC firmware build version * Platform and link mode (`dynamic`, `static`, or `mixed`) ### Registration and authentication Tracks the one-time app linking flow between your app and the Meta AI app. **Examples:** * Registration step (for example, `started`, `completed`, `failed`) * App linking flow ID (UUID for end-to-end correlation) * Attestation session ID and event type ### Permissions Records when your app checks or requests device capabilities through the Meta AI app. **Examples:** * Permission name (for example, `CAMERA`, `MICROPHONE`) and whether it is granted * Success or failure of the permission check/request API, with error details if applicable * For bulk queries, a map of permission names to consent states ### Session lifecycle Measures the connection lifecycle between your app and the glasses. A session represents a sustained period of access to device sensors or outputs. **Examples:** * Session state transitions: `starting`, `started`, `paused`, `stopping`, `stopped` * Previous session state * Error details if a transition fails ### Camera streaming Captures usage and quality signals for video and photo streaming. **Examples:** * Stream event type (prepare started/completed, start, stop, error) * Configuration: video codec, audio codec, resolution tier (`LOW` 360×640, `MEDIUM` 504×896, `HIGH` 720×1280), frame delivery method * Duration in seconds ### Display content delivery Logs display session usage for glasses with display capabilities (Meta Ray-Ban Display glasses). **Examples:** * Display event type (start, stop, send content, video playback states) * Display session ID and duration * Video source type, video codec, payload size in bytes * Video transfer duration in milliseconds ### Device discovery Tracks glasses availability from the SDK's perspective. **Examples:** * Event type (`device_discovered`, `device_forgotten`, `error`) * Device identifier * Error details if applicable ### Error reporting and diagnostics Supports stability monitoring and crash triage. **Examples:** * Structured error name, description, and type category * Crash grouping hash, crash type (signal or exception class) * Breadcrumbs (last N analytics event names before the crash) ### Developer testing with Mock Device Kit When you use Mock Device Kit to test without physical hardware, separate events are logged. **Examples:** * Mock device event type, mock device ID, device type code * Mock service event type, service ID, success/error details ### Known issues ## Wearables Device Access Toolkit | Issue | Workaround | | --- | --- | |Device Access Toolkit Wearables App installation fails if the device battery is below 10%.|Charge your glasses until the battery level exceeds 10%, then try again.| |Device Access Toolkit Wearables App installation may fail intermittently while setting or resetting device connections.|No current fix available. Future versions of Meta AI and Glasses firmware will include a more robust installation process.| |Android Device Access Toolkit Wearables App transfer may fail silently if Wi-Fi is off.|Enable Wi-Fi and try again.| |Magnetometer readings are currently unavailable to the Motion capability on Ray-Ban Meta, so `MotionSample.magnetometer` is `null`.|This is expected platform behavior. Treat `MotionSample.magnetometer` as optional and handle a `null` value.| ## Wearables Developer Center | Issue | Workaround | | --- | --- | |Beta testing: Distribution functionality requires minimum firmware v125| Update glasses firmware to v125 or higher. | |Beta testing: Device Access Toolkit doesn't currently allow for an application to have both a unique package name and a different bundle ID.|In the Wearables Developer Center, developers should set up individual applications for the iOS and Android platforms.| |Beta testing: Device Access Toolkit currently doesn't support the '-' dash character in the iOS Bundle ID.|Don't use this character in the Bundle ID.| |Beta testing: For v272, if the Meta AI app is connected to multiple devices, registering an app will install the Device Access Toolkit Wearables App only on one device.|All other devices will require manual installation of the DAT Wearables App from the Meta AI app.| |Beta testing: New versions on Wearables Developer Center may not be compatible with apps that are not using the latest SDK and have the Device Access Toolkit Wearables App enabled.|Update your application to use the latest SDK.| |Beta testing: If your application is using the latest SDK and has the Device Access Toolkit Wearables App enabled it may not be compatible with old versions on Wearables Developer Center.|Create a new version on Wearables Developer Center and push to beta testers via a Release Channel.| |Email addresses of members invited to a release channel must already be associated with a Meta account. | Verify anyone you invite to a release channel has set up a Meta account at [meta.ai](https://www.meta.ai/). | |Users logged into [developers.meta.com](https://developers.meta.com/) (Meta Horizon) may face an error with links from the Wearables Developer Center because it uses a different domain ([developer.meta.com](https://developer.meta.com/)). | Logout from [developers.meta.com](https://developers.meta.com/) before signing up for the Wearables Developer Center. | | Issue | Workaround | | --- | --- | |Beta testing: Distribution functionality requires minimum firmware v125| Update glasses firmware to v125 or higher. | |Beta testing: For v272, if the Meta AI app is connected to multiple devices, registering an app will install the Device Access Toolkit Wearables App only on one device.|All other devices will require manual installation of the DAT Wearables App from the Meta AI app.| |Beta testing: New versions on Wearables Developer Center may not be compatible with apps that are not using the latest SDK and have the Device Access Toolkit Wearables App enabled.|Update your application to use the latest SDK.| |Beta testing: If your application is using the latest SDK and has the Device Access Toolkit Wearables App enabled it may not be compatible with old versions on Wearables Developer Center.|Create a new version on Wearables Developer Center and push to beta testers via a Release Channel.| |Email addresses of members invited to a release channel must already be associated with a Meta account. | Verify anyone you invite to a release channel has set up a Meta account at [meta.ai](https://www.meta.ai/). | |Users logged into [developers.meta.com](https://developers.meta.com/) (Meta Horizon) may face an error with links from the Wearables Developer Center because it uses a different domain ([developer.meta.com](https://developer.meta.com/)). | Logout from [developers.meta.com](https://developers.meta.com/) before signing up for the Wearables Developer Center. | ### Version Dependencies ## Overview This page outlines the supported versions of the Meta AI app and glasses firmware for each release of the Meta Wearables Device Access Toolkit. ## Handling version errors at runtime A session can fail or continue in a degraded state due to versioning. Handle each `DeviceSessionError` case separately because they are not equally recoverable. | Error | What it means | What to do | | --- | --- | --- | | `INSUFFICIENT_SDK_VERSION` | The app uses an SDK version the glasses no longer support. | Terminal. Release a newer version of your app built with the current SDK. There is no in-session recovery. | | `DWA_OUT_OF_STU_RANGE` | The app and glasses are on versions outside the recommended compatibility range. | Not blocking. The session can continue, but behavior may be unexpected. Prompt the user to update occasionally rather than block the session. | | `DAT_APP_ON_THE_GLASSES_UPDATE_REQUIRED` | The Device Access Toolkit app integration on the glasses needs an update before the session can start. | Direct the user to the App Connections page in the Meta AI app to update it. | | `DWA_UNAVAILABLE` | The Device Access Toolkit app integration on the glasses did not become reachable after session start, typically because it failed to launch or is not installed. | Verify the glasses are set up in the Meta AI app before retrying. | Do not collapse these errors into a single "incompatible versions" state. Only `INSUFFICIENT_SDK_VERSION` requires a new app build, and only `DWA_OUT_OF_STU_RANGE` lets the session proceed. ## Registration requests from Meta AI Starting with SDK 1.0.0, the Meta AI app can initiate registration and deliver it to your integration through an Android intent. For Android integrations, call `Wearables.handleIntent` from both `Activity.onCreate` and `Activity.onNewIntent` before you handle your own intent actions. It invokes your callback only for registration intents and ignores everything else, so it is safe to call unconditionally. ```kotlin class MainActivity : Activity() { override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) Wearables.handleIntent(intent, ::onRegistrationRequest) } override fun onNewIntent(intent: Intent) { super.onNewIntent(intent) Wearables.handleIntent(intent, ::onRegistrationRequest) } private fun onRegistrationRequest(request: MWARegistrationRequest) { // Optionally show confirmation or education first, then continue or cancel. request.continueRegistration(this) } } ``` An `MWARegistrationRequest` is single-use. `continueRegistration` resumes the existing Meta AI registration flow, and `cancelRegistration` returns control to Meta AI. Calling either a second time returns `MWARegistrationRequestError.ALREADY_HANDLED`. The other error cases are `INVALID_REQUEST`, `CONFIGURATION_INVALID`, `REGISTRATION_FAILED`, and `CANCELLATION_FAILED`. Apps on an SDK older than 1.0.0 do not have `handleIntent` and cannot participate in this flow. ## Versions alone do not guarantee a capability Some capabilities are also gated server-side per user, independent of SDK, Meta AI app, or firmware versions. Meeting every version requirement on this page does not by itself mean a gated capability is available on a given device. When you request a gated capability on a session, the device may decline it at runtime, returning `DeviceSessionError.CAPABILITY_DENIED`. Handle this decline gracefully as an expected runtime outcome, rather than a hard failure. ## 1.0.0 | App/Firmware | Support | | --- | --- | | Meta AI app (Android) | V290 | | Meta AI app (iOS) | V290 | | Ray-Ban Meta glasses | V128 | | Meta glasses | V128 | | Meta Ray-Ban Display glasses | V128 | | Oakley Meta HSTN glasses | V128 | | Oakley Meta Vanguard glasses | V128 | ## 0.9.0 | App/Firmware | Support | | --- | --- | | Meta AI app (Android) | V282 | | Meta AI app (iOS) | V282 | | Ray-Ban Meta glasses | V126 | | Meta glasses | V126 | | Meta Ray-Ban Display glasses | V125 | | Oakley Meta HSTN glasses | V126 | | Oakley Meta Vanguard glasses | V126 | ## 0.8.0 | App/Firmware | Support | | --- | --- | | Meta AI app (Android) | V275 | | Meta AI app (iOS) | V275 | | Ray-Ban Meta glasses | V125 | | Meta glasses | V125 | | Meta Ray-Ban Display glasses | V125 | | Oakley Meta HSTN glasses | V125 | | Oakley Meta Vanguard glasses | V125 | ## 0.7.0 | App/Firmware | Support | | --- | --- | | Meta AI app (Android) | V272 | | Meta AI app (iOS) | V272 | | Ray-Ban Meta glasses | V125 | | Meta Ray-Ban Display glasses | V125 | | Oakley Meta HSTN glasses | V125 | | Oakley Meta Vanguard glasses | V125 | ## 0.6.0 | App/Firmware | Support | | --- | --- | | Meta AI app (Android) | V254 | | Meta AI app (iOS) | V254 | | Ray-Ban Meta glasses | V22 | | Meta Ray-Ban Display glasses | V21 | | Oakley Meta HSTN glasses | V22 | | Oakley Meta Vanguard glasses | V22 | ## 0.5.0 | App/Firmware | Support | | --- | --- | | Meta AI app (Android) | V254 | | Meta AI app (iOS) | V254 | | Ray-Ban Meta glasses | V22 | | Meta Ray-Ban Display glasses | V21 | | Oakley Meta HSTN glasses | V22 | | Oakley Meta Vanguard glasses | V22 | ## 0.4.0 | App/Firmware | Support | | --- | --- | | Meta AI app (Android) | V254 | | Meta AI app (iOS) | V254 | | Ray-Ban Meta glasses | V20 | | Meta Ray-Ban Display glasses | V21 | | Oakley Meta HSTN glasses | V20 | | Oakley Meta Vanguard glasses | V22 | ## 0.3.0 | App/Firmware | Support | | --- | --- | | Meta AI app (Android) | V249 | | Meta AI app (iOS) | V249 | | Ray-Ban Meta glasses | V20 | | Oakley Meta HSTN glasses | V20 | ### Acceptable use ## Before you build Everything you build on this platform is bound by the [Acceptable Use Policy](https://wearables.developer.meta.com/acceptable-use-policy) and the [developer terms](https://wearables.developer.meta.com/terms). Both apply in full and are the binding instruments — read them before you start. The guidance below tells you what we are looking for; it does not narrow, replace, or exhaustively restate either document. ## What we encourage Experiences that are uniquely possible or better on glasses, for example: 1. **Hands-free capture and logging** — sport, fieldwork, inspection, inventory, repair. 2. **Assistive and accessibility** — scene description, captioning, live translation, navigation aid. 3. **Context on demand** — the wearer points at something and asks; you answer. Deliberate, not ambient. 4. **Motion and input** — gesture control, activity and form tracking, coaching. 5. **Enterprise workflows** — guided procedures, remote expert assistance, compliance checks, with clear on-site notice to everyone present. Design principle: **narrow purpose, short retention, obvious to bystanders.** ## We review and we enforce We review Device Access Toolkit apps, monitor them afterwards, and investigate reports. Where an app breaches the [Acceptable Use Policy](https://wearables.developer.meta.com/acceptable-use-policy) or the [developer terms](https://wearables.developer.meta.com/terms), we may act **without prior notice** — including revoking your app's access to any or all device capabilities, removing it from distribution, and suspending or terminating your developer account and associated accounts. ## Section 2: Android API Reference ### PhotoError (enum) Errors reported while starting, using, or stopping standalone photo capture. ### PhotoQuality (enum) Compression quality requested for a standalone photo capture. ### PhotoResolution (enum) Image resolution requested for a standalone photo capture. ### PhotoState (enum) Lifecycle state of the standalone photo capture child owned by a camera. ### PhotoCaptureData A completed standalone photo capture received from the connected glasses. ### PhotoConfiguration Configuration used when creating the standalone photo capture child. ### PhotoTransferProgress Progress for an image file being transferred from the glasses after a capture request. ### AudioSampleRate (enum) Supported PCM sample rates for audio captured from a Meta Wearables device. Select a rate when co... ### CameraState (enum) Represents the current lifecycle state of a [Camera](com_meta_wearable_dat_camera_camera) capabil... ### StreamError (enum) Represents all possible stream error types that can occur during camera streaming operations. ### StreamState (enum) Represents the current state of a media streaming session with a Meta Wearables device. ### VideoQuality (enum) Defines the quality levels available for video streaming from Meta Wearables devices. Higher qual... ### AudioFrame Represents one decoded audio frame from a Meta Wearables device. The frame contains PCM bytes tog... ### StreamConfiguration Configuration for a media streaming session with a Meta Wearables device. Defines the video quali... ### VideoFrame Represents a single frame of video data from a Meta Wearables device. Contains the raw video buff... ### RegistrationRequestError (enum) Errors reported while parsing or responding to a Meta AI-initiated registration request. Each val... ### RegistrationRequest A single-use registration request initiated by the Meta AI app and delivered through `Wearables.h... ### AutoDeviceSelector A device selector that automatically selects an optimal device from the provided device set. It u... ### DeviceSelectorBase Abstract base class for [DeviceSelector](com_meta_wearable_dat_core_selectors_deviceselector) imp... ### SpecificDeviceSelector A device selector that always selects a specific, predetermined device. Use this when you want to... ### DeviceSessionState (enum) Represents the current state of a Meta Wearables device session. ### DeviceSession Manages a session with a Meta Wearables device. Sessions are created via Wearables.createSession ... ### ChargingState (enum) Whether the glasses battery is currently charging. ### DeviceCompatibility (enum) Indicates the compatibility status between AI glasses and the Wearables Device Access Toolkit. ### DeviceSessionError (enum) Enum representing errors that can occur during session lifecycle operations in the DAT SDK. ### DeviceType (enum) Represents the types of Meta Wearables devices supported by the Wearables Device Access Toolkit. ### DonState (enum) Whether the glasses are currently being worn (donned) or not (doffed). ### HingeState (enum) Whether the glasses arms (hinge) are open or folded closed. ### LinkState (enum) Connection state between the host application and a wearable device. ### NavigationError (enum) Errors that can occur when navigating to a screen in the Meta AI app. ### Permission (enum) Represents the types of permissions that can be requested from Meta Wearables devices. ### PermissionError (enum) Represents the specific types of errors that can occur during permission operations with Meta Wea... ### RegistrationError (enum) Enum representing errors that can occur during the Meta Wearables registration process. ### RegistrationState (enum) Current availability and registration state for the Wearables Device Access Toolkit. ### ThermalLevel (enum) Thermal level reported by the device. ### WearablesError (enum) Enum representing errors that can occur in the Wearables Device Access Toolkit. ### DatException Base exception class for the DAT (Device Access Toolkit) SDK. ### DatResult A specialized Result wrapper for DAT SDK operations that provides type-safe error handling while ... ### Device Contains metadata information about a Meta Wearables device. ### DeviceIdentifier A unique identifier for a Meta Wearables device. This identifier is used to distinguish between d... ### WearablesException Base exception class for the DAT (Device Access Toolkit) SDK. ### SessionState (enum) Represents the current state of a voice invocations stream with a Meta Wearables device. ### VoiceInvocationError (enum) Errors that can occur while receiving or responding to voice invocations from Meta Wearables devi... ### LaunchApp Represents the action to launch the partner app triggered by a "Hey Meta" voice command. ### DisplayError (enum) Errors that can occur during display operations on a wearable device. ### DisplayState (enum) Represents the current lifecycle state of a display on a Meta Wearables device. ### VideoCodec (enum) Video codec format for streaming video data to the wearable device. ### VideoPlayerError (enum) Errors that can occur during [VideoPlayer](com_meta_wearable_dat_display_views_videoplayer) opera... ### VideoPlayerState (enum) Lifecycle state of a [VideoPlayer](com_meta_wearable_dat_display_views_videoplayer). ### DisplayConfiguration Configuration for a display capability on a Meta Wearables device. ### VideoSource Represents the source of video data for a [VideoPlayer](com_meta_wearable_dat_display_views_video... ### Alignment (enum) Alignment options for positioning children within a FlexBox layout. ### Direction (enum) Layout direction for FlexBox-backed layout components. ### FlexBoxBackground (enum) Background options for FlexBox layouts. ### ImageSize (enum) Size presets for image components displayed on the wearable. ### CornerRadius (enum) Corner radius presets for image components displayed on the wearable. ### ButtonStyle (enum) Visual style presets for button components on the wearable display. ### ActionRole (enum) Semantic role of a button action on the wearable display. ### ButtonGroupAlignment (enum) Positioning of the buttons within a [FlexBoxScope.buttonGroup](com_meta_wearable_dat_display_view... ### TextStyle (enum) Typography style presets for text components on the wearable display. ### TextColor (enum) Color presets for text components on the wearable display. ### IconStyle (enum) Visual style for icon components on the wearable display. ### IconName (enum) Supported icon names for display. ### ContentScope Scope for building the root display content sent to a wearable device. ### VideoPlayer Controls video playback on a wearable display. ### FlexBoxScope Scope for building children inside a FlexBox layout. ### ButtonGroupScope Scope for building the buttons inside a [FlexBoxScope.buttonGroup](com_meta_wearable_dat_display_... ### VideoScope Configuration scope for video content within [ContentScope.video](com_meta_wearable_dat_display_v... ### ButtonType (enum) Physical button for an [InputEvent.Button](com_meta_wearable_dat_inputs_types_inputevent#button).... ### CapturePressType (enum) The physical gesture performed on the capture button for an [InputEvent.Capture](com_meta_wearabl... ### DragAction (enum) Phase of a continuous pinch-and-drag stream for an [InputEvent.Drag](com_meta_wearable_dat_inputs... ### InputSource (enum) The input surface on the connected glasses that produced an [InputEvent](com_meta_wearable_dat_in... ### InputsError (enum) Errors that can occur while the Inputs capability is active. Observe via [Inputs.errors](com_meta... ### InputsState (enum) Represents the current lifecycle state of the Inputs capability. ### NavDirection (enum) Direction of a navigation [InputEvent.Nav](com_meta_wearable_dat_inputs_types_inputevent#nav). ### InputEvent A semantic input event delivered from the connected glasses to your app. ### InputsConfiguration Configuration for the Inputs capability. ### MockDeviceKit Main entry point for managing simulated Meta Wearables devices. ### GlassesModel (enum) Available glasses models that MockDeviceKit can simulate. ### MockDeviceKitConfig Configuration for MockDeviceKit behavior. ### CameraFacing (enum) Specifies which phone camera to use as the mock device camera source. ### MockSpeechSource (enum) Selects the transcription source used by a simulated Speech session. ### MotionError (enum) Errors that can occur while the Motion capability is active. Observe these via [Motion.errors](co... ### MotionSamplingRate (enum) Available sampling rates for motion data. ### MotionSource (enum) Which physical device produced a [MotionSample](com_meta_wearable_dat_motion_types_motionsample). ### MotionState (enum) Lifecycle states of the Motion capability. ### MotionConfiguration Configuration for a Motion capability. ### Vector3 3D vector representing sensor data. ### Quaternion Rotation represented as a quaternion. ### MotionSample A single motion measurement from the connected wearable. ### SpeechConfiguration Options for adding a [Speech](com_meta_wearable_dat_speech_speech) capability to a [DeviceSession... ### SpeechError (enum) Errors that can occur during speech recognition. ### SpeechState (enum) Lifecycle states of the Speech capability. ### TranscriptionResult Represents a transcription result produced by on-device speech recognition. ## Section 3: iOS Swift API Reference ### AudioCodec (enum) Specifies how decoded audio is delivered with a camera streaming session. ### Camera Consolidated camera capability for a Meta Wearables device. ### CameraState (enum) Represents the current lifecycle state of a [Camera](/reference/ios_swift/dat/1.0/mwdatcamera_cam... ### AudioSampleRate (enum) Supported audio sample rates for streaming from Meta Wearables devices. ### DATPhotoData A photo captured from the connected wearable device. ### Photo Captures high-quality still photos from a connected wearable device. ### PhotoCaptureData A photo captured from the connected wearable device. ### PhotoCaptureFormat (enum) Supported formats for capturing photos from Meta Wearables devices. ### PhotoData A photo captured from a Meta Wearables device. ### PhotoError (enum) Errors that can occur during the lifecycle of a [Photo](/reference/ios_swift/dat/1.0/mwdatcamera_... ### PhotoQuality (enum) The compression quality to use when capturing a photo from the glasses camera. ### PhotoResolution (enum) The resolution (size) to use when capturing a photo from the glasses camera. ### PhotoState (enum) The lifecycle state of a [Photo](/reference/ios_swift/dat/1.0/mwdatcamera_photo). ### PhotoTransferProgress Progress of a file transfer for a captured photo. ### Stream A class for managing media streaming capabilities with Meta Wearables devices. Handles video stre... ### StreamConfiguration Configuration for a media streaming session with a Meta Wearables device. Defines video codec, re... ### StreamError (enum) Errors that can occur during streaming sessions. ### StreamingResolution (enum) Valid Live Streaming resolutions. We are using a 9:16 aspect ratio. ### StreamState (enum) Represents the current state of a media streaming session with a Meta Wearables device. ### VideoFrame Represents a single frame of video data from a Meta Wearables device. Contains the raw video samp... ### VideoCodec (enum) Specifies the video codec to use for streaming. ### VideoFrameSize Represents the width and height of a video frame in pixels. ### Announcer A protocol for objects that can announce events to registered listeners. ### AnyListenerToken A token that can be used to cancel a listener subscription. When the token is no longer reference... ### AutoDeviceSelector A device selector that automatically selects the best available device. Selects the first connect... ### CapabilityState (enum) Represents the state of a capability attached to a [DeviceSession](/reference/ios_swift/dat/1.0/m... ### ChargingState (enum) Represents whether the glasses battery is charging. ### Compatibility (enum) Indicates the compatibility status between AI glasses and the Wearables Device Access Toolkit. ### DatError Base protocol for all DAT SDK error types. ### Device AI glasses accessible through the Wearables Device Access Toolkit. ### DeviceSelector Protocol for selecting which device should be used for operations. Device selectors determine whi... ### DeviceSession A session representing a connection to a specific wearable device. ### DeviceSessionError (enum) Errors that can occur during [DeviceSession](/reference/ios_swift/dat/1.0/mwdatcore_devicesession... ### DeviceState A snapshot of a connected device's state. ### DeviceSessionState (enum) Represents the current state of a [DeviceSession](/reference/ios_swift/dat/1.0/mwdatcore_devicese... ### DeviceType (enum) Represents the types of Meta Wearables devices supported by the Wearables Device Access Toolkit. ### DonState (enum) Represents whether the glasses are currently being worn (donned) or not (doffed). ### HingeState (enum) Represents whether the glasses arms are open or folded. ### LaunchApp Represents the action to launch the partner app triggered by a "Hey Meta" voice command. ### LinkState (enum) Represents the connection state between a device and the Wearables Device Access Toolkit. ### ListenerTokenBag Groups listener tokens that share a lifecycle. ### MWARegistrationRequest A pending registration request initiated by the Meta AI app. ### MWARegistrationRequestError (enum) Errors that can occur while responding to a Meta AI-initiated registration request. ### NavigationError (enum) Errors that can occur when navigating to a screen in the Meta AI app. ### Permission (enum) Represents the types of permissions that can be requested from AI glasses. ### PermissionError (enum) Errors that can occur during permission requests. ### RegistrationError (enum) Error conditions that can occur during the registration process. ### PermissionStatus (enum) Represents the status of a permission request. ### RegistrationState (enum) Represents the current state of user registration with the Meta Wearables platform. ### RegistrationRequestError (enum) Errors that can occur while responding to a Meta AI-initiated registration request. ### RegistrationRequest A pending registration request initiated by the Meta AI app. ### ResponseHandle Type-safe handle for sending responses to voice invocations. ### SpecificDeviceSelector A device selector that always selects a specific, predetermined device. Use this when you want to... ### ThermalLevel (enum) Represents the thermal level reported by the connected device. ### UnregistrationError (enum) Error conditions that can occur during the unregistration process. ### VoiceInvocation Base protocol for all voice invocations that can be received from Meta Wearables devices. ### VoiceInvocationError (enum) Errors that can occur while receiving or responding to voice invocations from Meta Wearables devi... ### VoiceInvocationsStream A stream that delivers voice invocations from Meta Wearables devices. ### Wearables (enum) The entry point for configuring and accessing the Wearables Device Access Toolkit. ### WearablesError (enum) Errors that can occur during Device Access Toolkit configuration. ### WearablesHandleURLError (enum) Errors that can occur during URL handling. ### WearablesInterface The primary interface for Wearables Device Access Toolkit. ### ActionRole (enum) Semantic role of a button action on the wearable display. ### Alignment (enum) Alignment options for positioning children within a [FlexBox](/reference/ios_swift/dat/1.0/mwdatd... ### Background (enum) Background-style options for [FlexBox](/reference/ios_swift/dat/1.0/mwdatdisplay_flexbox). ### ButtonGroup A horizontal group of [Button](/reference/ios_swift/dat/1.0/mwdatdisplay_button)s rendered as a s... ### Button A tappable button component displayed on the wearable. ### ButtonGroupAlignment (enum) Alignment of the buttons within a [ButtonGroup](/reference/ios_swift/dat/1.0/mwdatdisplay_buttong... ### ButtonGroupBuilder A result builder for composing [Button](/reference/ios_swift/dat/1.0/mwdatdisplay_button)s inside... ### ButtonStyle (enum) Visual style presets for button components on the wearable display. ### ComponentBuilder A result builder for composing view components inside a [FlexBox](/reference/ios_swift/dat/1.0/mw... ### CornerRadius (enum) Corner radius presets for image components on the wearable display. ### Direction (enum) Layout direction for [FlexBox](/reference/ios_swift/dat/1.0/mwdatdisplay_flexbox) and container c... ### Display Manages rendering content on a Meta Wearables display. ### DisplayableView A protocol for views that can be displayed on glasses. ### DisplayError (enum) Errors that can occur during display operations such as [Display.send(_:)](/reference/ios_swift/d... ### DisplayState (enum) The current lifecycle state of a [Display](/reference/ios_swift/dat/1.0/mwdatdisplay_display). ### Edge A set of edges for padding modifiers. ### EdgeInsets Edge inset values for padding, in density-independent pixels (dp). ### FlexBox A flex layout container that arranges children along a configurable axis. ### Icon An icon component displayed on the wearable. ### IconStyle (enum) Visual style for icon components on the wearable display. ### IconName (enum) Supported icon names for display. ### Image An image component displayed on the wearable. ### ImageSize (enum) Size presets for image components on the wearable display. ### Text A text component displayed on the wearable. ### TextColor (enum) Color presets for text components on the wearable display. ### TextStyle (enum) Typography style presets for text components on the wearable display. ### VideoError (enum) Errors that occur during video playback via [VideoPlayer](/reference/ios_swift/dat/1.0/mwdatdispl... ### VideoPlayer A video player configuration to be sent to the glasses via [Display.send(_:)](/reference/ios_swif... ### ViewComponent A protocol for components that can be arranged inside a [FlexBox](/reference/ios_swift/dat/1.0/mw... ### ButtonType (enum) A named hardware button. ### CapturePressType (enum) The physical capture-button gesture. ### DragAction (enum) The phase of a drag interaction. ### InputEvent (enum) A single input interaction delivered by an [Inputs](/reference/ios_swift/dat/1.0/mwdatinputs_inpu... ### Inputs Streams input interactions from the connected glasses over [events](/reference/ios_swift/dat/1.0/... ### InputsError (enum) Errors that can occur while an [Inputs](/reference/ios_swift/dat/1.0/mwdatinputs_inputs) capabili... ### InputsConfiguration Configuration for an [Inputs](/reference/ios_swift/dat/1.0/mwdatinputs_inputs) capability. ### InputSource (enum) The device surface that produced an input event. ### InputsState (enum) The lifecycle state of an [Inputs](/reference/ios_swift/dat/1.0/mwdatinputs_inputs) capability. ### NavDirection (enum) A directional navigation gesture. ### CameraFacing (enum) The camera to use for live streaming from the phone. ### GlassesModel (enum) Identifies a glasses model for use with [MockDeviceKitInterface.pairGlasses(model:)](/reference/i... ### MockCameraCaptureKit A suite for mocking photo capture from the glasses camera. ### MockCameraKit A suite for mocking camera functionality. ### MockCaptouchKit Public interface for simulating captouch gestures on a mock device. ### MockDevice ### MockDeviceKit (enum) The entry-point to the MockDeviceKit for managing simulated Meta Wearables devices. Use this in t... ### MockDeviceKitConfig Configuration options for MockDeviceKit. ### MockDeviceKitError (enum) Errors thrown by MockDeviceKit. ### MockDeviceKitInterface Interface for managing mock Meta Wearables devices for testing and development. ### MockDisplayKit Captures and interacts with display content sent to simulated display glasses. ### MockGlasses Protocol for simulating smart glasses behavior in testing and development. Provides functionality... ### MockInputKit Injects synthetic input interactions into a connected app during testing. ### MockGlassesServices Container for accessing mock device service kits. ### MockMotionKit Mocks the on-glasses Motion / IMU capability for a deterministic IMU stream without glasses. ### MockPermissions Interface for configuring mock permission behavior during testing. ### MockSpeechKit Scripted source for the mock on-glasses Speech/ASR capability; `simulate*` events are delivered o... ### MockSpeechSource (enum) Selects where a mock Speech session's transcriptions come from. ### MockVoiceInvocationKit Protocol for simulating Voice Invocation capabilities on a mock device. ### MockDeviceTestClient Client for communicating with the MockDeviceKit test server from XCUITest processes. ### TestCameraFacing (enum) Which phone camera the mock device should stream from. ### TestCapturePressType (enum) The physical capture-button gesture to simulate. ### TestDragAction (enum) The phase of a simulated drag interaction. ### TestInputSource (enum) The device surface a synthetic input event should appear to come from. ### TestNavDirection (enum) A directional navigation gesture to inject. ### TestPermission (enum) A wearable permission the mock device can report on. ### TestPermissionStatus (enum) The status the mock device should report for a permission. ### MotionConfiguration Configuration for a [Motion](/reference/ios_swift/dat/1.0/mwdatmotion_motion) capability. ### Motion A capability that streams motion samples from connected Meta smart glasses. ### MotionError (enum) Errors that can occur while a [Motion](/reference/ios_swift/dat/1.0/mwdatmotion_motion) capabilit... ### MotionSample A single motion measurement from the connected wearable. ### MotionSamplingRate (enum) Available sampling rates for motion data. ### MotionSource (enum) The device that produced a [MotionSample](/reference/ios_swift/dat/1.0/mwdatmotion_motionsample). ### MotionState (enum) The lifecycle state of a [Motion](/reference/ios_swift/dat/1.0/mwdatmotion_motion) capability. ### Quaternion A rotation as a unit quaternion; `w` is the scalar component. ### Vector3 A 3D vector of sensor data. ### Speech A capability that provides on-device speech recognition from connected Meta smart glasses. ### SpeechError (enum) Errors that can occur during speech recognition operations. ### SpeechState (enum) The lifecycle state of a [Speech](/reference/ios_swift/dat/1.0/mwdatspeech_speech) capability. ### TranscriptionResult The result of an on-device speech recognition event. # WebApps SDK > WebApps SDK for Meta Ray-Ban Display MRBD only, not other Ray-Ban models. HTML CSS JavaScript hosted over HTTPS with 600 by 600 fixed viewport additive waveguide display where dark backgrounds render transparent and light colors opaque, no scrolling overflow hidden, focus ring styling with 88px minimum interactive target. Input via Neural Band wristband or temple capacitive swipe translating to ArrowUp ArrowDown ArrowLeft ArrowRight Enter Escape keyboard events, no mouse touch or physical keyboard. No camera access, no microphone access, no text input, no offline support, no notifications, no back navigation yet. Supports DeviceMotionEvent accelerometer gyroscope, DeviceOrientationEvent heading tilt roll requiring user gesture permission, navigator.geolocation from paired phone with 5 to 50 meter accuracy, localStorage sessionStorage up to 5 MB each, PNG icons via link rel icon or web manifest start_url scope display standalone, SVG not supported. AI coding plugin available at github.com/facebookincubator/meta-wearables-webapp for Claude Code Cursor Copilot Codex with skills for Vercel deploy, MCP server at https://mcp.developer.meta.com/wearables provides search_webapps_docs tool. Requires Meta AI app developer mode enabled via tapping app version five times, HTTPS hosting on Vercel Netlify GitHub Pages Replit Lovable or similar, minimum glasses software v125 and Meta AI app v272. Skip this entire section if task mentions Android Kotlin, iOS Swift, DAT SDK, MWDATCore, MWDATCamera, MWDATDisplay, DeviceManager, SessionManager, native API reference, Bluetooth session, camera photo video streaming, microphone HFP audio, speaker playback, Mock Device Kit, Ray-Ban Meta Gen1 Gen2 Optics, or native mobile integration. ## Section 1: WebApps Guides ### Setup The recommended AI-assisted setup for Meta Ray-Ban Display Web Apps takes four steps: 1. **Set up AI coding tools.** Install the AI Coding plugin and connect the Wearables MCP server before scaffolding your first project. 2. **Check your hardware.** Confirm your glasses, paired phone, and (optionally) Meta Neural Band meet the minimum requirements. 3. **Check your Meta AI app and enable Developer Mode.** Verify the app version on your phone, then put the app into Developer Mode so it can load Web Apps onto the glasses. 4. **Prepare to host your Web App.** Pick a hosting platform that serves your app over HTTPS so the glasses can load it. ## Set up AI coding tools The recommended workflow combines the [AI Coding plugin](https://github.com/facebook/meta-wearables-webapp) with the [Wearables MCP server](/docs/develop/webapps/wearables-documentation-mcp/). The plugin provides Web Apps implementation and testing workflows. The MCP server provides current published documentation through `search_webapps_docs`. Use the [Web Apps AI and MCP setup instructions](/docs/develop/webapps/wearables-documentation-mcp/) to install both tools for Muse Code, Claude Code, Codex, or Cursor before asking the assistant to scaffold a project. ## Requirements ### Hardware Web Apps are supported only on Meta Ray-Ban Display (MRBD) glasses. Meta Neural Band is optional, though recommended for an optimal experience. ### Software Download the Meta AI app from the [App Store (iOS)](https://apps.apple.com/us/app/meta-ai-assistant-glasses/id1558240027) or [Play Store (Android)](https://play.google.com/store/apps/details?id=com.facebook.stella) if you haven't done so already. For the best experience, always update your glasses and Meta Neural Band with the latest software updates. **Glasses** The minimum glasses software version is `v125`+. To verify the glasses software version: 1. In the Meta AI app, tap **Devices** (the glasses icon), and select your device. 2. Tap the gear icon to open **Device settings**. 3. Tap **General** > **About** > **Release Version**. 4. Check for glasses software updates if your version is below `v125`. **Meta AI App** The minimum Meta AI app version is `v272`+. To verify Meta AI app version: 1. Open the **Meta AI app**. 2. Tap **Settings** > **App Info**. 3. Note the **App version** number. If it's older than `v272`, update the Meta AI app from the App Store (iOS) or Play Store (Android). ## Enabling Developer Mode in the Meta AI app Developer Mode in the Meta AI app unlocks the menu options used to load and reload Web Apps on MRBD. 1. Open the Meta AI app on your paired iOS or Android device. 2. Select **Settings** > **App Info**, and then tap the **App version** number five times to display a pop-up that enables Developer Mode. 3. Click **Enable** to confirm. **Note:** Developer Mode persists across sessions, so you don't need to re-enable it each time you open the Meta AI app. ## Hosting your Web App Your Web App must be hosted on a publicly accessible HTTPS URL so MRBD can load it. Common options include: Replit, Lovable, Vercel, GitHub Pages, Netlify, Cloudflare Pages, or any static site host that serves over HTTPS. You can also host from your own server, as long as it serves the app over HTTPS with a valid TLS certificate. While it's possible to point MRBD at any website, most sites are not configured to work well within the platform's display and input constraints. See the [Build guide](/docs/develop/webapps/build/) for the design and runtime constraints to keep in mind. **Note:** HTTP-only URLs are not supported. The glasses runtime requires HTTPS for every Web App URL it loads. ## Installing a Web App Wearers add a Web App to their glasses in one of two ways: by scanning a QR code, or by activating an install control on a page they already have open. Both routes end at the same on-glasses confirmation, and neither can add an app without the wearer agreeing to it. ### QR code A Web App is added by scanning a QR code that encodes an install link — a plain QR of your site's URL is not recognized. Rather than hand-crafting one, generate the install link and its QR code with a supported tool: the **View on Glasses QR** feature of the Meta Ray-Ban Display Simulator, or the MRBD Web App AI Coding plugin, both covered on the [Test page](/docs/develop/webapps/test/). Put the generated QR code somewhere a wearer can look at it — a page on your site, a slide, a printed card. The wearer looks at the code with their glasses, the glasses recognize it, and an add-to-glasses screen appears for them to confirm. What that screen shows depends on which surface handles it: the App Manager resolves an icon from your site's appearance, or shows a placeholder when it can't, while the legacy dialog shows your app's name, an optional description, and its URL. An app icon is not guaranteed. Because your app has not been reviewed, the confirmation screen marks it as unverified. Expect wearers to see that label and write your app name and description so they can tell it is yours. Serve the app over HTTPS. The QR parser rejects an HTTP inner URL, and HTTPS is required for Web Apps generally. ### `navigator.install()` From a page already open on the glasses, you can offer installation directly: ```javascript if (navigator.install) { installButton.hidden = false; } installButton.addEventListener('click', async () => { try { await navigator.install(); } catch (err) { // Installation is unavailable, or the URL was rejected. showManualInstructions(); } }); ``` `navigator.install` is `undefined` outside the glasses runtime — for example in a regular browser — so the check above keeps the control hidden where installation could never work. It does not, however, guarantee that a call will succeed: on a Web App page the property is defined even when installation is gated off, and `navigator.install()` then rejects. Treat rejection as an expected outcome and always keep the `catch`. **Arguments** Both arguments are optional: ```javascript navigator.install(installUrl, options); ``` | Argument | Notes | | -------- | ----- | | `installUrl` | Optional. The URL to install. Defaults to the current page. Relative URLs resolve against the document's base URL. Must be `https:` or `http:`. | | `options.name` | Optional. The name shown on the confirmation screen. Defaults to ``, then the document title, then the hostname, and finally the literal `"App"`. | Set `options.name`, or a `` tag, so the confirmation screen shows your app's name rather than whatever your `` happens to be on the current page. **A resolved promise does not mean the app was installed.** It means only that the install request was dispatched — accepted for launch, not confirmed as shown. The confirmation screen is not guaranteed to appear: the launched flow can still fail its own gate and finish immediately. And even when it does appear, the wearer may decline, with no callback telling you what they chose. Do not change your UI to an installed state on resolution, and do not count resolutions as installs. **Rejections** | Error | Cause | | ----- | ----- | | `DataError` | The `installUrl` could not be resolved, or is not `https:` or `http:`. | | `NotSupportedError` | Installation is not available on this device, or the URL was rejected. | | `AbortError` | The install request could not be dispatched. | Always catch. Rejection is a normal outcome, not a bug, because availability depends on the device. ### The `<install>` element An `<install>` element with an `installurl` or `href` attribute also triggers installation when activated: ```html <install installurl="https://example.com/myapp">Add to glasses</install> ``` Prefer `navigator.install()`. The element's failures are swallowed, so you cannot detect that installation was unavailable and cannot offer a fallback. `<install>` is also not a real element on other browsers, where it renders as an inert inline box and does nothing when clicked, so anything you put inside it needs to make sense as plain text. </gk> ### Build ## Overview Web Apps for Meta Ray-Ban Display glasses use standard web APIs. The easiest way to build Web Apps is with AI coding tools. Learn how to build optimized Meta Ray-Ban Display Web Apps by understanding: * How to Build with AI * Capabilities and Best Practices * Display * Input and navigation * Activation and continuous drag * Back navigation * Text composer * Sensors * Location * Local storage * Offline mode * App icons ## HTML metadata Add the following metadata to the `<head>` of your HTML file. This allows your web app to support upcoming discovery surfaces and enables us to message users when a website isn't compatible with Meta Ray-Ban Display glasses. ```html <head> <!-- A brief description of your app --> <meta name="description" content="Description of your web app"> <!-- Identify your web app as MRBD-compatible --> <meta name="mrbd-web-app-capable" content="yes"> </head> ``` ## Capabilities Web Apps on Meta Ray-Ban Display glasses **support** these capabilities: | Capability | Description and guidance | | ---------- | ------------------------ | | Display | Additive waveguide overlay. Use dark backgrounds/light, high-contrast UI colors. Fixed 600x600px viewport. Avoid scrolling. | | Input | Directional navigation and activation from the touchpad or band translate into standard browser behavior. Optional continuous drag uses standard pointer events. The on-glasses composer provides text input. No mouse, touchscreen, or physical keyboard is available. All interactive elements must be focusable. | | Sensors (IMU) | Standard `DeviceMotionEvent` (accelerometer, gyroscope) and `DeviceOrientationEvent` (heading, tilt, roll) W3C APIs. Requires user permission. | | Location (GPS) | Standard `navigator.geolocation` W3C API. Location is fetched from the paired mobile device. Requires user permission. | | Local storage | Standard Web Storage APIs (`localStorage`, `sessionStorage`). Best for lightweight data (preferences, small caches). Use JSON for structured data. | | Offline mode | Standard Service Workers and Cache API. Save a copy of your app’s files directly on a device, so apps can load and render without a network request. | | App icons | Use Unicode symbols or high-resolution PNG favicons (>= 52x52 px) via `<link>` tags or Web App Manifest. SVGs are not supported. | Web Apps do **not yet support**: * Camera * Microphone * Notifications <gk project="wdp_manual_web_apps_docs_connect_26" markdown="block"> Web Apps on Meta Ray-Ban Display glasses also **support**: | Capability | Description and guidance | | ---------- | ------------------------ | | Text to speech | Standard `window.speechSynthesis` Web Speech API. Audio plays through the glasses speaker. One English voice; no permission prompt. `volume` is ignored and `boundary` events never fire. | </gk> ### Display The display is an additive waveguide that overlays rendered pixels onto the wearer's real-world view. This has a direct impact on how your app looks. * A pixel rendered as pure black is fully transparent, since it contributes zero light. * Bright, vivid colors are the most visible, because they add light on top of the real-world scene. #### Color and typography Because the display is additive, color choices matter more than on a conventional screen: * Use dark backgrounds, since they effectively disappear. Bright backgrounds cause glare and reduce readability. * Use light, high-contrast colors for UI elements like text and interactive components. Also, use bright colors for accents. * Use large, readable fonts: a minimum of 16 px for body text and 20-24 px for primary content. #### Viewport All content should render within a fixed 600 x 600 pixel viewport and avoid scrolling. Include the following viewport `meta` tag to lock the scale and prevent unexpected zooming: ```html <!-- optional --> <meta name="viewport" content="width=600, height=600, initial-scale=1.0, user-scalable=no"> ``` Set `overflow: hidden` on the `<body>` element to ensure no content extends beyond the viewport boundary: ```css body { width: 600px; height: 600px; overflow: hidden; } ``` ### Input and navigation Meta Ray-Ban Display translates input from the touchpad and Meta Neural Band into directional navigation and activation. The glasses OS delivers those actions to your Web App as standard arrow key (`ArrowUp`, `ArrowDown`, `ArrowLeft`, `ArrowRight`) and `Enter` events. Since the Meta Ray-Ban Display glasses have no mouse, touchscreen, or physical keyboard, every interactive element in your Web App must support directional navigation and activation through standard web platform behavior. **JavaScript** ```javascript // — Input Constants — const DPAD = { UP: 'ArrowUp', DOWN: 'ArrowDown', LEFT: 'ArrowLeft', RIGHT: 'ArrowRight', SELECT: 'Enter', BACK: 'Escape', }; // — Focus Management — function moveFocus(direction) { var focusables = Array.from( document.querySelectorAll('.focusable:not([disabled]):not(.hidden)') ); if (!focusables.length) return; var idx = focusables.indexOf(document.activeElement); if (idx === -1) { focusables[0].focus(); return; } var next = (direction === 'up' || direction === 'left') ? (idx > 0 ? idx - 1 : focusables.length - 1) : (idx < focusables.length - 1 ? idx + 1 : 0); focusables[next].focus(); focusables[next].scrollIntoView({ block: 'nearest', behavior: 'smooth' }); } // — D-pad Listener — document.addEventListener('keydown', function(e) { switch (e.key) { case DPAD.UP: moveFocus('up'); break; case DPAD.DOWN: moveFocus('down'); break; case DPAD.LEFT: moveFocus('left'); break; case DPAD.RIGHT: moveFocus('right'); break; case DPAD.SELECT: if (document.activeElement.classList.contains('focusable')) { document.activeElement.click(); } break; case DPAD.BACK: history.back(); break; default: return; // don't preventDefault on unhandled keys } e.preventDefault(); }); ``` **HTML** ```html <!-- Mark interactive elements with the focusable class --> <button class="focusable" data-action="settings">Settings</button> <button class="focusable" data-action="start">Start</button> ``` **CSS** ```css .focusable { transition: all 150ms ease; border: 2px solid transparent; min-height: 88px; /* glasses minimum tap target */ } .focusable:focus { outline: none; border-color: #00d4ff; box-shadow: 0 0 20px rgba(0, 212, 255, 0.4); } ``` ### Activation and continuous drag Meta Ray-Ban Display provides standard web events for activation and optional continuous drag. No special SDK is required. Two interaction modes are available to your Web App: * **Activation (always on):** Activating the focused element is equivalent to pressing Enter. It works on any focusable element, including buttons, links, and custom interactive controls. No opt-in is needed. * **Drag (opt-in):** Continuous arm movement provides positional input delivered as *pointer events*. Use it for sliders, drawing, panning, or any other interaction that needs smooth, continuous control. Opt in at the page level. | Interaction | Web event | Action | | --- | --- | --- | | Directional navigation | `ArrowUp` / `ArrowDown` / `ArrowLeft` / `ArrowRight` keydown | Move focus between elements | | Activation | `Enter` keydown | Activate focused element | | Continuous drag | `pointerdown` / `pointermove` / `pointerup` | Update a continuous position (opt-in) | #### Activation Activation fires `Enter` on the currently focused element (`document.activeElement`). It is **not** a positioned click, so there is no cursor or screen coordinate. Activation targets the element that currently has focus, so build your UI around focusable elements, not click targets. This means your app must: 1. Make interactive elements *focusable* (use `<button>`, `<a>`, or add `tabindex="0"`). 2. Make them *keyboard-activatable* (respond to `Enter` key or `click` events). 3. Provide a *visible focus indicator* so the wearer knows which element is selected. For most apps, activation works automatically if you build a standard accessible, focusable UI. ```html <!-- These elements support activation by default --> <button data-focusable onclick="doAction()">Start</button> <a href="/next" data-focusable>Next Page</a> <!-- Make a custom element activatable --> <div tabindex="0" role="button" data-focusable onclick="doCustomAction()"> Custom Control </div> ``` Next, handle focus navigation with keyboard events. ```javascript var focusableElements = []; var focusIndex = 0; function updateFocusables() { focusableElements = Array.prototype.slice.call( document.querySelectorAll('[data-focusable]:not([disabled])') ); } function moveFocus(idx) { focusableElements.forEach(function(el) { el.classList.remove('focused'); }); focusIndex = Math.max(0, Math.min(idx, focusableElements.length - 1)); if (focusableElements[focusIndex]) { focusableElements[focusIndex].classList.add('focused'); focusableElements[focusIndex].focus(); } } // Keep `focusIndex` in sync if focus moves outside `moveFocus()` (for example, `programmatic .focus()`, native focus, or after `updateFocusables()`). document.addEventListener('focusin', function(e) { var idx = focusableElements.indexOf(e.target); if (idx !== -1) { focusableElements.forEach(function(el) { el.classList.remove('focused'); }); focusIndex = idx; e.target.classList.add('focused'); } }); document.addEventListener('keydown', function(e) { switch (e.key) { case 'ArrowUp': e.preventDefault(); moveFocus(focusIndex - 1); break; case 'ArrowDown': e.preventDefault(); moveFocus(focusIndex + 1); break; case 'Enter': e.preventDefault(); // Activation fires Enter on `document.activeElement`. // Activating `activeElement` (not a cached `focusIndex`) ensures the element the wearer sees as focused is the one that receives the click. if (document.activeElement && document.activeElement.matches('[data-focusable]')) { document.activeElement.click(); } break; case 'Backspace': e.preventDefault(); history.back(); break; case 'Escape': e.preventDefault(); history.back(); break; } }); // Initialize the focusable set once the DOM is ready and keep it in sync after any DOM updates that add or remove `data-focusable` elements document.addEventListener('DOMContentLoaded', function() { updateFocusables(); if (focusableElements.length) { moveFocus(0); } // The observer handles dynamic DOM changes automatically. Otherwise, call `updateFocusables()` directly from the code that mutates the DOM (for example, `container.appendChild(newItem); updateFocusables();`). var observer = new MutationObserver(function() { updateFocusables(); }); observer.observe(document.body, { childList: true, subtree: true }); }); ``` Provide a visible focus indicator. Since the code sample above toggles a `.focused` class and calls `.focus()`, you need to style both `:focus` and `.focused`. Without this CSS, the focus change has no visual effect. ```css /* Visible focus indicator — matches the earlier .focusable styles */ [data-focusable], .focusable { transition: all 150ms ease; border: 2px solid transparent; min-height: 88px; /* glasses minimum tap target */ } [data-focusable]:focus, [data-focusable].focused, .focusable:focus, .focusable.focused { outline: none; border-color: #00d4ff; box-shadow: 0 0 20px rgba(0, 212, 255, 0.4); } ``` You can use either convention: `class="focusable"` with `document.querySelectorAll('.focusable')`, or `data-focusable` with `document.querySelectorAll('[data-focusable]')`. Either way, keep the selector, the toggled class, and the CSS in sync, so the wearer always sees which element is focused. #### Drag (continuous input) When drag is enabled, the wearer can hold and move their arm to generate a continuous stream of positional data. This is delivered to your app as standard **pointer events** (`pointerdown`, `pointermove`, `pointerup`). Drag is *off* by default. The only interactions available without it are directional focus navigation and activation, meaning there is no free cursor or sliding motion unless you explicitly enable it. To enable drag, add `touch-action: none` to the `<body>` element in your initial stylesheet. ```css /* MUST be in your initial stylesheet, as it is not added dynamically via JavaScript */ body { touch-action: none; } ``` **Note:** `touch-action` is read on `<body>` only once at page load, so it must be in the corresponding CSS. It cannot be set on individual elements or included via JavaScript. Once drag is enabled, listen for pointer events to receive the continuous motion. ```javascript const slider = document.getElementById('slider'); let dragging = false; slider.addEventListener('pointerdown', function(e) { dragging = true; // Do not use requestPointerLock-- use standard pointer events instead slider.setPointerCapture(e.pointerId); }); slider.addEventListener('pointermove', function(e) { if (dragging) { // Use e.clientX and e.clientY for position updatePosition(e.clientX, e.clientY); } }); slider.addEventListener('pointerup', function() { dragging = false; }); ``` **Some drag use cases** - **Sliders:** Adjust volume, brightness, or any continuous value with arm movement - **Panning:** Navigate a map or large image by dragging in 2D - **Drawing:** Simple freehand drawing or annotation - **Games:** Control game elements with continuous positional input #### Considerations **Text input and activation** Activating a focused text input (such as `<input>` or `<textarea>`) opens the device text composer instead of dispatching a click to your page. **Verify on-device** The exact values in drag event fields (e.g., `movementX`, `movementY`, `clientX`, `clientY`) are generated device-side. Always **test on real glasses** by logging `pointermove` events before relying on specific coordinate behavior. Desktop emulation will not accurately represent the drag input. **Feature availability** Continuous drag relies on the band, which may not be available to every wearer. The APIs are always present (for example, pointer events exist in the browser), but drag input may not fire on some hardware. Always design your app to remain usable if drag events never arrive. Provide alternative directional navigation for critical actions. #### Best practices | Do | Don't | | --- | --- | | Use `<button>`, `<a>`, or `tabindex="0"` for activation targets | Rely on positioned click coordinates for activation | | Always show a visible focus indicator | Use hover-only styles for interactive elements | | Set `touch-action: none` on `<body>` in your initial stylesheet | Set `touch-action` per-element or via JavaScript at runtime | | Use `setPointerCapture()` for reliable drag tracking | Use `requestPointerLock()`, since it's not supported. | | Test drag behavior on real glasses hardware | Assume desktop pointer simulation matches device behavior | | Provide directional-navigation fallbacks for all critical actions | Make drag the only way to complete an essential task | | Call `e.preventDefault()` on all handled key events | Leave arrow key / Enter events unhandled (causes unexpected scrolling) | In addition to these best practices, verify the following before publishing your Web App with wearable input support: - Activation uses the currently focused element, so test every interactive control. - If using drag, test `pointermove` events directly on a device. - If using drag, the app still works through directional navigation and activation if drag events never fire. - All interactive elements are reachable from the home screen through directional navigation. ### Back navigation Back on Meta Ray-Ban Display glasses lets users move backward through a Web App's screens, panels, and other navigable states. For standard Web Apps, the native shell checks [`navigation.canGoBack`](https://developer.mozilla.org/en-US/docs/Web/API/Navigation/canGoBack) when the user invokes Back. If a previous navigation entry is available, the shell calls `history.back()`. Otherwise, it shows the native system menu overlay. Use the standard web [Navigation API](https://developer.mozilla.org/en-US/docs/Web/API/Navigation_API) and [History API](https://developer.mozilla.org/en-US/docs/Web/API/History_API) to manage navigation. No custom SDK is required. Your Web App is responsible for rendering the destination state and restoring app-managed state, including focus, scroll offsets, and page/carousel positions. | User action | System behavior | | --- | --- | | Back with a previous history entry | Calls `history.back()`. | | Back with no previous history entry | Shows the native system menu overlay. | | Back while the native menu is visible | Exits the Web App. | The native transient UI is handled before web history. For example, when the native text-input UI is open, Back dismisses this UI before traversing the page's history. #### Navigation history limit The shell guards entries created with `history.pushState()` using a limit of **five** total navigation entries, including the initial document, so a typical Web App can have its root entry plus four deeper nav entries. Once `navigation.entries().length` reaches 5, subsequent `history.pushState()` calls replace the current entry instead of adding another entry. **Note:** This guard applies *only* to `history.pushState()`. It is not a universal cap on entries created through other navigation mechanisms. Design your Web App's `pushState()`-based navigation to stay within this five-entry limit. #### Adding and restoring navigation history Create a history entry when the user moves to a destination they would expect back navigation to reverse, such as a detail screen or a substantial panel. Seed the initial entry with `history.replaceState()`, save the current view state, and restore the destination from a `popstate` handler. ```javascript const HOME_STATE = {screen: 'home'}; // Give the initial document an explicit state without adding another entry. history.replaceState(HOME_STATE, ''); function saveCurrentViewState() { history.replaceState( { ...history.state, focusId: document.activeElement?.id ?? null, scrollX: window.scrollX, scrollY: window.scrollY, // Store any app-specific position here, such as a pager index. }, '', ); } function navigateTo(state) { saveCurrentViewState(); history.pushState(state, ''); renderScreen(state); } function openSettings() { navigateTo({screen: 'settings'}); } function openDetail(itemId) { navigateTo({screen: 'detail', id: itemId}); } window.addEventListener('popstate', (event) => { const state = event.state ?? HOME_STATE; renderScreen(state); // Restore state after the destination DOM has been rendered. requestAnimationFrame(() => { window.scrollTo(state.scrollX ?? 0, state.scrollY ?? 0); if (state.focusId) { document.getElementById(state.focusId)?.focus(); } // Restore app-specific state, such as a pager or carousel index, here. }); }); ``` You can handle traversal with the standard web Navigation API's `navigate` event instead of `popstate`. Use one coordinated traversal strategy so the destination isn't restored twice. When no previous history entry remains, the next Back action shows the native system menu. This is the expected app boundary, so don't add synthetic entries or push a new entry during Back handling to prevent the user from reaching it. #### Best practices | Do | Don't | | --- | --- | | Push an entry for a user-recognizable destination that Back should reverse. | Push entries for hover, focus changes, tooltips, or other transient UI. | | Seed the initial document with `history.replaceState()`. | Push a duplicate initial entry that creates an artificial Back step. | | Store serializable identifiers, offsets, and indexes in history state. | Store DOM elements or other non-serializable objects in history state. | | Restore the destination screen, focus, scroll, and component positions after traversal. | Assume an arbitrary SPA UI state will be restored automatically. | | Design for five total `pushState()` navigation history entries, *including* the root document. | Assume five additional screens can be pushed *after* the root entry. | | Allow history to drain until the native system menu is reached. | Push a new entry while handling Back or otherwise trap the user. | #### Troubleshooting | Issue | Potential cause | Solution | | --- | --- | --- | | Back opens the native system menu instead of the previous app screen. | The current screen did not add a history entry, or no previous entry remains. | Add an entry when navigating to a user-recognizable destination. | | Back traverses history, but the displayed screen does not change. | The `popstate` or Navigation API traversal handler does not render the destination state. | Read the destination state and render the corresponding screen. | | A newly opened screen cannot be reached again with Back. | The five-entry limit was reached, so `pushState()` likely replaced the current entry. | Keep the root plus deeper navigation within five total entries. | | The app cannot reach the native system menu. | A traversal handler pushes new entries or repeatedly prevents history from draining. | Never push while handling Back, and allow traversal to reach the initial entry. | | Focus, scroll, pager, or carousel position is lost. | The app did not save or restore that state, or restored it before rendering completed. | Store stable IDs and numeric positions, then restore them after rendering the destination. | ### Input: On-glasses composer for text Meta Ray-Ban Display glasses provide text input for Web Apps through an on-glasses composer, a systems-level handwriting and dictation panel that appears when users focus and activate a standard HTML text field. Since the glasses have no touchscreen or physical keyboard, users must use this composer to enter text into your app's search boxes, forms, notes fields, and name prompts. No special integration is required. When the user focuses and activates a text field, an **on-glasses composer panel** appears, allowing them to enter text via handwriting or voice dictation. The composed text is committed back to your field via standard DOM `input` and `change` events. **Note:** Text input requires glasses firmware **v127+** and Meta AI app **v272+**. #### Composer lifecycle ```mermaid flowchart TD A["User navigates\ndirectionally"] --> B["Text field\nreceives focus"] B --> C{"User activates\nthe field?"} C -->|No| D["Field stays focused.\nComposer does not open"] C -->|Yes| E["Composer panel opens\n(handwriting \+ voice)"] E --> F["User enters text"] F --> G["Text committed\nto field"] G --> H["'input' event fires\non your element"] H --> I{"More input\nneeded?"} I -->|Yes| F I -->|No| J["User navigates away.\nComposer closes"] J --> K["'change' event fires"] ``` The composer opens on focus and activation, *not* on focus alone. Programmatic calls to `.focus()` will not surface the composer, so user-initiated activation is required. #### Input types **Supported input types** The following HTML input types **can** open the composer. * `<input type="text">` * `<input type="search">` * `<input type="email">` * `<input type="url">` * `<input type="tel">` * `<input type="number">` * `<textarea>` * `contenteditable` elements **Unsupported input types** The following HTML input types **can not** open the composer. * `<input type="password">` * `<input type="date">` * `<input type="checkbox">` * `<input type="radio">` <gk project="wdp_manual_web_apps_docs_connect_26" markdown="block"> #### On-screen keyboard In addition to handwriting and dictation, the composer offers an on-screen QWERTY keyboard, which is part of the composer. Text arrives in your field the same way it does for handwriting and dictation, and your app needs no integration to support it. **Note:** The on-screen keyboard requires glasses firmware **v129+**. **Opening the keyboard** The keyboard is reached from the composer, not from your text field. Users focus and activate a text field to open the composer, then swipe down on the composer to raise the keyboard. Users on TalkBack can activate the chevron control on the composer instead of swiping. **Note:** There is no way for your app to open the keyboard directly, and the keyboard cannot be raised while the composer is in dictation mode. **Typing** The keyboard is a four-row grid with a letters layer and a numbers and symbols layer. Directional input moves focus from key to key, and activation selects the focused key. Up and down move to the key physically above or below the current one, so navigation follows cardinal directions even though rows may have different key counts. | Key | Behavior | | --- | -------- | | Shift | Latches caps lock on and off. It is not a one-shot shift, so it stays on until the user turns it off. | | Backspace | Deletes the selection, or the character before the caret when there is no selection. | | `123` / `ABC` | Switches between the letters layer and the numbers/symbols layer. Focus stays on the toggle key across the switch. | | Space | Inserts a space. | | Done | Collapses the keyboard and leaves the composer open with the typed text intact.* | *Swiping up while the focus is on the top row also collapses the keyboard. **Available characters** | Layer | Characters | | ----- | ---------- | | Letters | `a`–`z`, space | | Numbers and symbols | `0`–`9` `@` `#` `$` `%` `&` `*` `-` `+` `(` `)` `=` `/` `:` `;` `,` `.` `?` `!` | The keyboard cannot type apostrophes, quotation marks, underscores, backslashes, angle brackets, square brackets, braces, pipes, tildes, carets, or backticks. Avoid requiring these characters in values entered with the on-screen keyboard, especially email addresses and URLs. Password fields cannot open the composer. See [**Input types**](https://wearables.developer.meta.com/docs/develop/webapps/build#input-types) below. **Events your app receives** Typed characters go into the composer's own text buffer, not into your element, so your field remains unchanged while the user types. Text reaches your element only when the user confirms the composer, at which point `input` fires and then `change` fires, following the same process for handwriting and dictation. Your app receives the finished string in one update. Because the keyboard is a focus-driven overlay rather than an Android input method, no per-character key events reach the page. Do not rely on `keydown`, `keypress`, or `keyup` to observe typing, and do not implement per-keystroke behavior, like inline autocomplete that depends on them. Validate and react in your `input` handler instead. </gk> #### Text field implementation Add a standard HTML text field with a descriptive placeholder attribute. The placeholder text should help the user understand what to enter in the composer panel. **HTML** ```html <!-- Search field --> <input type="search" id="search" class="focusable" placeholder="Activate to write or speak" /> <!-- Multi-line notes field --> <textarea id="note" class="focusable" placeholder="Activate to write or speak" ></textarea> ``` **JavaScript** ```javascript // Read the composed text using standard event listeners in JavaScript. const note = document.getElementById('note'); //The composer commits text via the 'input' event note.addEventListener('input', () => { console.log('Current text:', note.value); }); // 'change' fires when the field loses focus after editing note.addEventListener('change', () => { console.log('Final text:', note.value); }); ``` #### Best practices | Do | Don't | | :---- | :---- | | Provide a clear, descriptive placeholder or aria-label | Leave fields without hints | | Use supported input types (`text`, `search`, `email`, `url`, `tel`, `number`) | Use `type="password"` (since the composer won't open) | | Listen for `input` events to read composed text | Listen for `keydown` events (since no physical keyboard exists) | | Provide a fallback if the composer is unavailable | Assume the composer will always be available | | Make fields focusable and reachable through directional navigation | Rely on mouse-click or touch to open fields | #### Known limitations * The `inputmode` and `enterkeyhint` attributes do **not** affect the composer behavior. The `type` attribute controls only whether the field is eligible. * On some firmware builds, the composer may be unavailable. Design your app to remain functional without it (for example, you can provide alternative navigation or pre-filled options). #### Complete example ```html <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=600, height=600, initial-scale=1.0, user-scalable=no" /> <title>Search
``` #### Text input troubleshooting | Issue | Cause | Solution | | :---- | :---- | :---- | | Composer does not open when field is focused. | Composer requires **focus and activation**. | Ensure the user activates the field after focusing it. | | Composer does not open on a password field. | `type="password"` fields are excluded. | Use `type="text"` if composer input is needed. | | Programmatic `.focus()` does not open composer. | Only user-initiated activation opens the composer. | Let the user navigate to and activate the field. | | `inputmode` and `enterkeyhint` have no effect. | These attributes are not honored by the on-glasses composer. | Remove them or leave as progressive enhancement for other platforms. | | Text input features are simply not working. | Firmware version below v127. | Update glasses firmware to v127 or later. | ### Sensors #### Overview Meta Ray-Ban Display glasses expose access to accelerometer, gyroscope, and compass data through the standard `DeviceMotionEvent` and `DeviceOrientationEvent` web APIs. Simply add event listeners on `window` as you would in any mobile browser. #### Requesting permission Motion and orientation data require an explicit user permission grant. For cross-platform compatibility, check whether `DeviceOrientationEvent.requestPermission()` exists and call it before attaching listeners. ```javascript function startIMU() { window.addEventListener('deviceorientation', handleOrientation); window.addEventListener('devicemotion', handleMotion); } // Check whether requestPermission exists before calling it if (typeof DeviceOrientationEvent !== 'undefined' && typeof DeviceOrientationEvent.requestPermission === 'function') { // Platforms that require explicit permission (e.g., iOS Safari) DeviceOrientationEvent.requestPermission() .then(function(state) { if (state === 'granted') { startIMU(); } }); } else { // Browsers that do not expose the iOS-style requestPermission() API. startIMU(); } ``` Meta Ray-Ban Display glasses provide `requestPermission()`, so the first branch is the one that runs on them. The AI glasses prompt wearers the first time your app requests sensors and then remembers the decision for your domain. Sensor access is *never* granted automatically. Use either `DeviceOrientationEvent.requestPermission()` or `DeviceMotionEvent.requestPermission()` to access. You don't need to call both. **Note:** The permission request must be triggered by a user gesture (for example, a button press via Enter key). It cannot be called automatically. Do not trigger any browser history changes (such as `history.pushState()` or SPA routing) before the sensor permission request resolves. Doing so can leave the `DeviceMotionEvent.requestPermission()` promise pending indefinitely, causing the app to hang when sensors are requested. To avoid this, either: * Serve the Web App without any `history.pushState()` or SPA routing before the sensor permission completes. * Request permission from a user action and `await` it before changing browser history. #### DeviceMotionEvent `DeviceMotionEvent` provides real-time accelerometer and gyroscope readings. Use it to detect movement, measure G-forces, or track rotation speed. ```javascript window.addEventListener('devicemotion', function(e) { // Accelerometer (including gravity), in m/s² var acc = e.accelerationIncludingGravity; if (acc) { // Compute magnitude in G-force var g = Math.sqrt(acc.x * acc.x + acc.y * acc.y + acc.z * acc.z) / 9.81; document.getElementById('gforce').textContent = g.toFixed(2) + ' G'; } // Gyroscope (rotation rate in degrees/second). May be null — guard before use. if (e.rotationRate) { var yawRate = e.rotationRate.alpha; var pitchRate = e.rotationRate.beta; var rollRate = e.rotationRate.gamma; } }); ``` **Fields and nullability** Not every field is populated on glasses. Guard grouped fields before reading `x`, `y`, `z` or `alpha`, `beta`, `gamma` off them. Otherwise, your handler will throw on the first event. | Field | Type | Notes | | ----- | ---- | ----- | | `accelerationIncludingGravity.x/y/z` | number | Acceleration including gravity, in m/s². Populated. | | `acceleration.x/y/z` | number or `null` | Acceleration with gravity removed. **May be `null`.** | | `rotationRate.alpha/beta/gamma` | number or `null` | Rotation rate in **degrees per second**, not radians. **May be `null`.** | | `interval` | number | Milliseconds between readings. | `rotationRate` is reported in degrees per second, matching `DeviceMotionEvent` in mobile browsers. If you are porting code that used the Generic Sensor `Gyroscope` interface, note that it reports radians per second instead. #### DeviceOrientationEvent `DeviceOrientationEvent` provides the current orientation of the glasses relative to the Earth. Use it for compass heading, tilt detection, or spatial UI. ```javascript window.addEventListener('deviceorientation', function(e) { var heading = e.alpha; // Compass direction (rotation around z-axis): 0-360° var tilt = e.beta; // Forward/back tilt (rotation around x-axis): -180° to 180° var roll = e.gamma; // Left/right tilt (rotation around y-axis): -90° to 90° document.getElementById('heading').textContent = heading.toFixed(1) + '°'; }); ``` | Field | Type | Notes | | ----- | ---- | ----- | | `alpha` | number | Compass heading in degrees, 0–360, where 0 is north. | | `beta` | number | Front-to-back tilt in degrees, -180 to 180. | | `gamma` | number | Left-to-right tilt in degrees, -90 to 90. | | `absolute` | boolean | Whether the reading is relative to the Earth frame. | The glasses derive `alpha` from a dedicated compass sensor rather than from the rotation vector, so heading is usable for real navigation and orientation UI. `deviceorientationabsolute` is also available. It carries the same fields as `deviceorientation` but always reports Earth-relative values, so you can listen for it instead of checking `absolute` yourself. ### Losing sensor permission mid-session A wearer can revoke sensor permission while your app is running directly from the glasses permission management dialog. In turn, the runtime removes all sensor event listeners your app registered, and further readings will no longer arrive. Your listeners simply stop firing, and no event or callback tells you what happened. Design for sensor permission loss, rather than assuming permissions stay granted: * Do not treat the last reading you received as still current. If your UI shows a heading or tilt, drive it from a timestamp and show a neutral or stale state when readings stop arriving. * Do not block your app's core flow on sensor data. Wearer functionality should be reachable to their completion without sensors. * When permission is revoked, the runtime stops delivering sensor events, but retains your listener registrations. Reattaching listeners does not restore the stream. After the wearer re-enables sensors in the permission-management dialog, call either `DeviceOrientationEvent.requestPermission()` or `DeviceMotionEvent.requestPermission()` again. When it returns granted, the existing listeners resume. #### Best practices | Consider | Avoid | | -------- | ----- | | Requesting permission from a user gesture (for example, button press) | Calling `requestPermission()` automatically on page load | | Checking for API availability before attaching listeners | Assuming `DeviceOrientationEvent` is always defined | | Throttling or debouncing high-frequency sensor updates for UI rendering | Updating the DOM on every single sensor event without throttling | | Using `accelerationIncludingGravity` for tilt-based interactions | Relying on acceleration alone when gravity context is needed | | Removing event listeners when sensor data is no longer needed | Leaving listeners active in the background, which drains battery | ### Location #### Overview Meta Ray-Ban Display glasses implement the standard `navigator.geolocation` web API. Location data is fetched from the wearer's paired mobile device, since the glasses themselves do not have location-aware sensors. Use the API exactly as you would in any Web App. Like Sensor Data, Location also requires user permission. #### One-shot position Use `getCurrentPosition` to request a single location fix. ```javascript navigator.geolocation.getCurrentPosition( function(position) { var coords = position.coords; console.log('Latitude: ' + coords.latitude); // Decimal degrees console.log('Longitude: ' + coords.longitude); // Decimal degrees console.log('Accuracy: ' + coords.accuracy); // m console.log('Altitude: ' + coords.altitude); // m (may be null) console.log('Speed: ' + coords.speed); // m/s (may be null) console.log('Heading: ' + coords.heading); // Degrees from north (may be null) console.log('Timestamp: ' + position.timestamp); // ms since epoch (UTC) }, function(error) { // error.code: // 1 = PERMISSION_DENIED: wearer denied permission request // 2 = POSITION_UNAVAILABLE: location could not be retrieved (i.e., phone offline) // 3 = TIMEOUT: request exceeded the timeout // error.message - human-readable description console.error('Geolocation error:', error.code, error.message); }, { timeout: 15000 } ); ``` #### Continuous position tracking Use `watchPosition` to receive ongoing location updates as the wearer moves. ```javascript var watchId = navigator.geolocation.watchPosition( function(position) { // Called each time the position updates updateMap(position.coords.latitude, position.coords.longitude); }, function(error) { // error.code: // 1 = PERMISSION_DENIED: wearer denied permission request // 2 = POSITION_UNAVAILABLE: location could not be retrieved (i.e., phone offline) // 3 = TIMEOUT: request exceeded the timeout // error.message - human-readable description console.error('Watch error:', error.code, error.message); } ); ``` Call `clearWatch` when updates are no longer needed. ```javascript // Stop watching when done navigator.geolocation.clearWatch(watchId); ``` #### Position options Both `getCurrentPosition` and `watchPosition` accept an optional third argument to configure behavior: | Option | Type | Default | Description | | ------ | ---- | ------- | ----------- | | `enableHighAccuracy` | boolean | false | Request the most accurate position available. May take longer and use more power. | | `timeout` | number | Infinity | Maximum time (in milliseconds) to wait for a position. Use 10000-15000 ms as a practical default. | | `maximumAge` | number | 0 | Accept a cached position if it is no older than this value (in milliseconds). | ```javascript navigator.geolocation.getCurrentPosition(successCb, errorCb, { enableHighAccuracy: true, // Request most accurate position (boolean, default false) timeout: 15000, // Max wait time in ms (number, default Infinity) maximumAge: 5000 // Accept cached position if newer than this in ms (number, default 0) }); ``` #### Best practices | Consider | Avoid | | -------- | ----- | | Using a `timeout` of 10-15 seconds (10000-15000 ms), since the first request may take several seconds | Omitting using a `timeout` or setting it too low | | Handling `PERMISSION_DENIED` gracefully, since the wearer must grant permission | Assuming the wearer does not need to grant permissions | | Ensuring permissions requests are triggered by a user gesture | Assuming permission requests are triggered | #### Notes * Remember, location comes from the paired companion phone's GPS/network services. * Expect an accuracy of 5-50 meters, depending on signal quality. #### Location error handling Always provide an error callback to handle failure gracefully. Location may be unavailable if the following errors occur: | Description | Error code | Type | | ----------- | ---------- | ---- | | Wearer denies the permission prompt | 1 | `PERMISSION_DENIED` | | Location data could not be retrieved (for example, companion device is offline) | 2 | `POSITION_UNAVAILABLE` | | Request exceeded the specified timeout | 3 | `TIMEOUT` | ### Text to speech Meta Ray-Ban Display glasses implement the standard `window.speechSynthesis` Web Speech API. Audio plays through the glasses speaker. There is no permission prompt and nothing to enable — `speechSynthesis` is available to every Web App. Text to speech is useful on glasses in a way it is not on a phone: the display is small, additive, and often glanced at rather than read. Speaking a short result out loud is frequently a better answer than rendering more text. ```javascript var utterance = new SpeechSynthesisUtterance('Your train leaves in four minutes'); utterance.lang = 'en-US'; speechSynthesis.speak(utterance); ``` #### Voices The glasses expose exactly one voice. | Property | Value | | -------- | ----- | | `name` | `Default` | | `voiceURI` | `default` | | `lang` | `en-US` | | `localService` | `true` | | `default` | `true` | `speechSynthesis.getVoices()` returns this single-entry list synchronously, on the first call. You do not need to wait for voices to load, and the list never changes, so there is no reason to handle `voiceschanged`. Do not build a voice picker, and do not make speech conditional on finding a particular voice by name. #### Utterance properties | Property | Range | Default | Notes | | -------- | ----- | ------- | ----- | | `text` | — | — | The text to speak. | | `lang` | BCP 47 tag | — | Only English is synthesized today. Other tags are accepted but do not change the voice. | | `rate` | 0.1–10 | 1 | Values outside the range are clamped. | | `pitch` | 0.1–2 | 1 | Values outside the range are clamped. Note the upper bound is 2, not 10. | | `voice` | — | — | Optional. Leave unset to use the default voice. | | `volume` | — | 1 | **Ignored.** Setting it has no effect. Speech plays at the wearer's system volume. | `volume` is the one to watch. The property exists and is writable because it is part of the Web Speech API, but the glasses do not read it. If your app needs quieter output you cannot achieve that from the page. #### Events Attach handlers to the **utterance**, not to `speechSynthesis`. | Event | Fires | | ----- | ----- | | `start` | Speech begins. | | `end` | Speech finishes, or the utterance is cancelled. | | `error` | Synthesis fails. See error codes below. | | `pause` | `speechSynthesis.pause()` is called. | | `resume` | `speechSynthesis.resume()` is called. | `boundary` and `mark` are **never fired**. The underlying engine does not report word or SSML timing, so the properties exist and can be assigned but your handler will not run. Do not build word-level highlighting or karaoke-style captions — there is no timing signal to drive them. `speechSynthesis.addEventListener()` is a no-op on glasses. Registering a listener on the `speechSynthesis` object itself succeeds silently and never fires. Use the utterance's own handlers: ```javascript var utterance = new SpeechSynthesisUtterance('Directions ready'); utterance.onstart = function() { showSpeakingIndicator(); }; utterance.onend = function() { hideSpeakingIndicator(); }; utterance.onerror = function(e) { console.warn('TTS failed:', e.error); }; speechSynthesis.speak(utterance); ``` #### Pausing, resuming, and cancelling `speechSynthesis.pause()` and `resume()` work, but not the way they do in a desktop browser. The glasses engine cannot resume mid-utterance, so `resume()` **restarts the current utterance from the beginning** and then continues through the queue. For a short confirmation this is barely noticeable. For a long passage it means the wearer hears the opening again. Prefer splitting long content into several short utterances over offering a pause control. `speechSynthesis.cancel()` clears the queue and stops the active utterance. Every cancelled utterance receives `end`, not `error`, so an `onend` handler cannot by itself distinguish "finished speaking" from "was cancelled". Track that yourself if you need to know: ```javascript var cancelled = false; function stopSpeaking() { cancelled = true; speechSynthesis.cancel(); } utterance.onend = function() { if (!cancelled) advanceToNextStep(); }; ``` You can read `speechSynthesis.speaking`, `.pending`, and `.paused` to inspect current state. #### Error codes `error` events carry a standard Web Speech error code in `event.error`. | Code | Meaning | | ---- | ------- | | `canceled` | The utterance was removed from the queue before it started. | | `interrupted` | Speech was stopped after it had started. | | `audio-busy` | The audio output is unavailable, for example during a call. | | `audio-hardware` | No usable audio output. | | `network` | A network failure during synthesis. | | `synthesis-unavailable` | The synthesis engine could not be reached. | | `synthesis-failed` | The engine failed while producing audio. | | `language-unavailable` | No voice for the requested `lang`. | | `voice-unavailable` | The requested `voice` is not available. | | `text-too-long` | The `text` is too long to synthesize. | | `invalid-argument` | A property such as `rate` or `pitch` could not be used. | | `not-allowed` | Synthesis was not permitted. | Always attach an `onerror` handler. `audio-busy` in particular is routine rather than exceptional on glasses, because the wearer may be on a call, so treat it as a reason to fall back to on-screen text rather than as a bug. #### Best practices | Consider | Avoid | | -------- | ----- | | Speaking short, self-contained sentences | Reading long passages the wearer cannot pause or scrub | | Also showing the information on screen | Making speech the only way to receive something important | | Handling `error`, especially `audio-busy` | Assuming `speak()` always produces audio | | Cancelling before speaking something new | Queueing utterances until they pile up behind each other | | Letting the wearer turn speech off in your app | Speaking automatically on load, before any wearer action | ### Storage Web Apps on Meta Ray-Ban Display glasses have access to standard Web Storage APIs, including both `localStorage` and `sessionStorage`, to persist lightweight data on the glasses: * `localStorage` persists data across sessions, even after the app is closed and reopened. * `sessionStorage` persists data only for the current session, so values are cleared when the session ends. These work exactly as they do in any modern browser, and both APIs store data as key-value string pairs. #### Saving and reading data Use the standard `setItem`, `getItem`, and `removeItem` methods. ```javascript // Save a value localStorage.setItem('userPreference', 'dark'); // Read a value var preference = localStorage.getItem('userPreference'); // Returns 'dark', or null if the key does not exist // Remove a value localStorage.removeItem('userPreference'); // Clear all stored data localStorage.clear(); ``` #### Session storage `sessionStorage` has an identical API, but scopes data to the current session. Use it for temporary states that should not persist after the user exits your app. ```javascript // Track whether the user has seen the onboarding screen this session if (sessionStorage.getItem('onboardingSeen')) { showMainScreen(); } else { showOnboarding(); sessionStorage.setItem('onboardingSeen', 'true'); } ``` #### Storage limits The glasses runtime provides storage within the following limits: * `localStorage`: 5 MB * `sessionStorage`: 5 MB As a general practice, keep stored data lightweight - avoid storing large blobs, images, or multi-megabyte datasets. Web storage is best suited for user preferences, small caches, and application state. ### Offline mode Web Apps can continue to work even when AI glasses lose their internet connection. By implementing offline support, your Web App will load reliably on flaky Wi-Fi, display cached content when connectivity drops, and provide a seamless experience regardless of network conditions. Offline mode uses standard web platform APIs - **Service Workers** and the **Cache API** - to save a copy of your app's files directly on the device. Once cached, your app can load and render without a network request. Offline mode follows a three-stage lifecycle: 1. **Register:** Your app registers a service worker (`sw.js`) that runs in the background. 1. **Install and precache:** On first load, the service worker downloads and stores your app's core files (HTML, CSS, JS) in a local cache. 1. **Serve from cache:** On subsequent loads (including offline), the service worker intercepts network requests and serves cached files first, falling back to the network only when needed. This cache-first strategy means your app loads instantly from local storage, even with no connection. Requirements for Web App offline mode setup include the following: | Requirement | Details | | ----------- | ---------- | | HTTPS | Service workers only register in a secure context. Your app must be served over `https://`. Local `file://` URLs won't work.| | Single app shell | Your app's core files must be identifiable as a cacheable set. | | No permission prompt | Offline mode works in the background, so it does not require user permission. | #### Setup **Step 1: Register the service worker** In your main app file (usually `index.html` or `app.js`), register the `serviceWorker`: ```js if ('serviceWorker' in navigator) { navigator.serviceWorker.register('/sw.js') .then((registration) => { console.log('Service Worker registered:', registration.scope); }) .catch((error) => { console.log('Service Worker registration failed:', error); }); } ``` **Step 2: Create a service worker file** Create a file named `sw.js` at your app's root. This file handles caching and request interception. ```js // Define a versioned cache name and the files to precache. const CACHE_NAME = 'my-app-v1'; const APP_SHELL = [ '/', '/index.html', '/app.js', '/styles.css' ]; // Install: precache the app shell. self.addEventListener('install', (event) => { event.waitUntil( caches.open(CACHE_NAME).then((cache) => cache.addAll(APP_SHELL)) ); }); // Activate: clean up old caches on version bump. self.addEventListener('activate', (event) => { event.waitUntil( caches.keys().then((names) => Promise.all( names .filter((name) => name !== CACHE_NAME) .map((name) => caches.delete(name)) ) ) ); }); // Fetch serve from cache first, fall back to network. self.addEventListener('fetch', (event) => { event.respondWith( caches.match(event.request).then((cached) => cached || fetch(event.request)) ); }) ``` **Step 3: Show online/offline status in the UI** Let the wearer know when they are offline and when connectivity returns. ```js function updateConnectionStatus() { const status = document.getElementById('connection-status'); if (navigator.onLine) { status.textContent = '● Online'; status.style.color = '#00d4ff'; } else { status.textContent = '● Offline'; status.style.color = '#ffaa00'; } } window.addEventListener('online', updateConnectionStatus); window.addEventListener('offline', updateConnectionStatus); // Set initial state updateConnectionStatus(); ``` **HTML for status indicator** ```html
``` #### Updating your cached app When you release a new version of your app, update the `CACHE_NAME` (for example, change `my-app-v1` to `my-app-v2`). The new service worker will install alongside the old one, precache the updated files, and clean up the outdated cache on activation. ```js // Update this version string whenever you update your app const CACHE_NAME = 'my-app-v2'; ``` #### Clearing cached data There is no "clear cache" control for individual Web Apps and no JavaScript API your app can call to erase its own storage. A Web App's stored data can only be cleared by **removing** the Web App from the user's glasses altogether. When that cleanup runs, it erases: * Cookies * `localStorage` * IndexedDB * HTTP cache * Service Worker `CacheStorage` entries **The other Web Apps on your domain determine how much is erased.** The runtime will not wipe data belonging to a Web App that is still installed, so it narrows the scope when it has to. | Situation | What is erased | | --------- | -------------- | | No other installed Web App shares your registrable domain. | The whole site, including `HttpOnly` cookies and the HTTP cache. | | Another installed Web App shares your registrable domain. | Only your origin's data — `localStorage`, IndexedDB, `CacheStorage`, and cookies visible to JavaScript. `HttpOnly` cookies and the shared HTTP cache survive. | Essentially, if you host several Web Apps under a single domain, removing one of them is *not* a clean reset for that app. Server-set session cookies in particular can outlive any removal and just be picked up by the next install. **Do not build on either outcome.** The cleanup is behind a server-side configuration flag that is *off* by default, so on a given build, removal may clear nothing at all. Even if the server-side configuration flag is *on*, the scope narrows, and it runs in the background after removal, rather than at a point your app can observe. This means that your Web App is *not notified* and cannot run code during cleanup. What to do instead: * Treat stored data as something that may or may not be there on any given launch. Read defensively and rebuild from the server when what you find is missing or unrecognized. * Writing a schema version into your stored data and validating it on read is the only reliable way to detect data left behind by an older install. * Keep using versioned `CACHE_NAME` values as described above, since cache versioning is under your control. * Don't store anything you would be uncomfortable leaving on a device after the wearer removes your Web App. #### Best practices | Do | Don't | | ----------- | ---------- | | Precache all files your app needs to render its initial screen. | Assume every request will succeed. Always handle the offline case. | | Use a versioned cache name and clean old caches on activate. | Leave stale caches accumulating on the device. | | Show a clear offline indicator, so your wearers know what to expect. | Silently fail or show broken content when offline. | | Combine with `localStorage` for persisting user data between sessions. | Depend solely on the cache for user-generated data. | | Decide which content makes sense offline and show a friendly message for content that requires fresh data. | Try to cache everything, including API responses that change frequently. | #### Other considerations **Precache real sub-resource URLs:** Ensure the file paths in your `APP_SHELL` array exactly match the URLs your app actually requests. Mismatched paths will cause cache misses, so your app won't load offline. **Storage Is per-app:** Each Web App gets its own isolated cache. Combine offline caching with [localStorage](https://wearables.developer.meta.com/docs/develop/webapps/build#storage) to persist user settings, notes, or preferences between sessions. **Feature detection:** The Service Worker API is present on supported builds, but always handle the case where registration fails gracefully, as illustrated here. ```js if ('serviceWorker' in navigator) { // Safe to register } else { // Offline support not available — app still works, just requires a connection } ``` #### Content that requires internet Some features (live data feeds, API calls, real-time updates) cannot work offline. Design your app to show a friendly message for those sections. ```js async function fetchData(url) { try { const response = await fetch(url); return await response.json(); } catch (error) { // Network unavailable -- show cached or placeholder content showOfflineMessage('This content requires an internet connection.'); return null; } } ``` #### Checklist Before publishing your Web App with offline support, verify the following: * App is served over HTTPS. * `navigator.serviceWorker.register()` resolves successfully. * App shell files are precached on install (check DevTools → Application → Cache Storage). * App loads and renders correctly after going offline (for example, airplane mode or disconnect Wi-Fi). * Old caches are cleaned up when the cache version is bumped. * Online/offline UI indicator updates correctly when connectivity changes. * Content that requires a live connection shows a friendly fallback message. * Service Worker registration failure is handled gracefully (app still works online). ### App icons For app icons, use Unicode symbols or high-resolution PNG favicons (larger than 52x52 px). The system checks the Web App manifest and page source (not just `favicon.ico`) for this content. If no suitable icon is found, a default fallback icon is shown. SVGs are not supported. Icons can be implemented as HTML `` tags in your page's `` section. ```html ``` Icons can also be implemented by referencing the Web App JSON manifest. Each entry in the `icons` array must include a `src` attribute and ideal sizes. **HTML** ```html ``` **JSON** ```json { "icons": [ { "src": "/icons/icon-96.png", "sizes": "96x96" }, { "src": "/icons/icon-192.png", "sizes": "192x192" } ] } ``` ### Overview ## Overview Web Apps for Meta Ray-Ban Display glasses run in a Chromium-based browser. Build with semantic HTML, responsive CSS, and the documented web APIs. Feature-detect optional capabilities and provide a usable fallback when a permission, device, or network service is unavailable. Use these guides to build and test an app: - [Display and layout](/docs/develop/webapps/build/display-and-layout/) - [Input and navigation](/docs/develop/webapps/build/input-and-navigation/) - [Sensors and location](/docs/develop/webapps/build/sensors-and-location/) - [Text to speech](/docs/develop/webapps/build/text-to-speech/) - [Storage and offline support](/docs/develop/webapps/build/storage-and-offline-support/) - [App icons](/docs/develop/webapps/build/app-icons/) These pages describe platform behavior without requiring a particular framework or component library. ## Build with a coding assistant Coding assistants and app-building tools can help with Web Apps when they use the current platform documentation. Install the [Wearables Web App AI Toolkit](/docs/develop/webapps/wearables-web-app-ai-toolkit/) for implementation and testing workflows, and connect to the [Wearables Documentation MCP Server](/docs/develop/webapps/wearables-documentation-mcp/) at `https://mcp.developer.meta.com/wearables` for current platform guidance. The server's `search_webapps_docs` tool can look up current documentation during a task. Use these prompts as starting points: Read https://wearables.developer.meta.com/docs/develop/webapps/build/overview/, then use the Wearables Documentation MCP Server endpoint https://mcp.developer.meta.com/wearables to call search_webapps_docs for current Meta Ray-Ban Display Web Apps guidance. If the tool is unavailable, say so and use the linked documentation. Inspect my project before changing it, then build the smallest working version of this idea: [describe the app]. Use semantic controls, size the app from the available viewport, keep overflow intentional, preserve browser-managed directional navigation, and test the result at 600 × 600 and on the device. Read https://wearables.developer.meta.com/docs/develop/webapps/build/overview/, then use the Wearables Documentation MCP Server endpoint https://mcp.developer.meta.com/wearables to call search_webapps_docs for current guidance. Inspect my project first. Replace fixed-size layout assumptions with responsive sizing, make every interactive control semantic and reachable through browser-managed directional navigation, keep focus visible, and use one intentional scroll owner per axis when content exceeds the viewport. Avoid custom global focus handlers. Test the result at 600 × 600 and on the device. Read https://wearables.developer.meta.com/docs/develop/webapps/build/overview/, then use the Wearables Documentation MCP Server endpoint https://mcp.developer.meta.com/wearables to call search_webapps_docs for current constraints and troubleshooting guidance. If the tool is unavailable, say so and use the linked documentation. Inspect the app and identify whether the issue is layout, browser navigation, text input, storage, sensors, speech, or deployment. Make the smallest standards-based fix, preserve existing capabilities, and run the relevant checks. ## Preview in desktop Chrome Install the [Meta Ray-Ban Display Simulator](https://chromewebstore.google.com/detail/meta-ray-ban-display-simu/jpjlmmodokemlepklkdbimceggpbjcll) Chrome extension to preview the additive-display treatment and test a Web App in desktop Chrome. Use it while iterating on layout, readability, focus, and application flows without deploying every change to the device. The simulator is a development preview, not a substitute for device testing. It cannot reproduce the wearer's surroundings, wearable input hardware, system composer, sensor stream, paired-phone location, or speech output exactly. ## HTML metadata Add a description and identify the app as compatible with Meta Ray-Ban Display glasses: ```html ``` The compatibility tag is platform metadata. It does not replace runtime feature detection or permission handling. ## Capabilities | Capability | Description and guidance | | --- | --- | | Display and layout | The device has a 600 × 600 additive-light display. Treat that as a validation target, derive layout from the actual viewport, and make overflow intentional. | | Directional navigation | The browser spatially navigates semantic, visible controls and activates the selected control. Ordinary pages do not need a custom focus manager. | | Continuous drag | Apps can opt into pointer-event drag input for sliders, drawing, panning, games, and similar interactions. Keep a non-drag path for essential actions. | | Text input | Supported HTML text fields open the system composer for handwriting, dictation, and on-screen keyboard input. Read committed text through standard `input` and `change` events. | | Back navigation | The native shell integrates with standard browser history. Add history entries for destinations that Back should reverse and restore application state during traversal. | | Motion and orientation | Permission-gated `DeviceMotionEvent` and `DeviceOrientationEvent` data. Treat fields as nullable and readings as time-sensitive. | | Location | Permission-gated `navigator.geolocation`, normally sourced through the paired phone. Handle denied, unavailable, and timed-out requests. | | Storage | `localStorage` and `sessionStorage` for small string values. Writes can fail or data can be cleared, so handle both conditions. | | Offline support | Service Workers and Cache Storage can provide an offline app shell after an online installation and successful activation. | | Text to speech | A supported subset of `speechSynthesis` speaks concise content through the glasses speaker. Retain equivalent visible content so important information does not depend on speech. | | App icons | A Meta Wearables manifest supplies the app name, monochrome artwork, and background treatment. | Capabilities can vary by runtime version, device state, permission, origin policy, or hardware. Check for the API before use and handle an unavailable or denied result without blocking the app's core flow. ## Validate on the device After desktop and simulator testing, validate device-specific behavior on Meta Ray-Ban Display before publishing. Test the additive display in varied surroundings, navigation with the available wearable inputs, and every hardware- or permission-dependent capability the app uses. ### Wearables Web App AI Toolkit The [Wearables Web App AI Toolkit](https://github.com/facebook/meta-wearables-webapp) provides coding-agent workflows for building and testing Meta Ray-Ban Display Web Apps. Install it in the coding tool you use, then start a new session so the tool can discover its skills. For current platform guidance, connect the [Wearables Documentation MCP Server](/docs/develop/webapps/wearables-documentation-mcp/) alongside the toolkit. ## Choose a plugin Install `meta-wearables-webapp` for standard Web Apps. For games with real-time gameplay, a game loop, scoring, physics, collision, sprites, enemies, or levels, also install `meta-wearables-webapp-game`. The game plugin provides workflows for creating, iterating on, and validating Vite, TypeScript, and Three.js games. A leaderboard, scoreboard, or button-advanced quiz remains a standard Web App. Additional coding-agent skills are available for the [UI Toolkit for Meta Ray-Ban Display](/docs/develop/webapps/design/overview/). See [UI Toolkit for Meta Ray-Ban Display coding-agent skills](/docs/develop/webapps/design/getting-started/coding-agent-skills/) for installation and usage instructions. ## Muse Code Add the marketplace and install the base plugin: ```text /plugins marketplace add meta-wearables-webapp https://github.com/facebook/meta-wearables-webapp /plugins install meta-wearables-webapp@meta-wearables-webapp ``` For game development, also install the game plugin: ```text /plugins install meta-wearables-webapp-game@meta-wearables-webapp ``` Example game request for Muse Code: Use the installed meta-wearables-webapp-game plugin and run /create-webapp-game to create a Meta Ray-Ban Display game. Start by asking me about the game concept, controls, visual direction, and any starter art or audio. Then scaffold the smallest playable version, use the bundled game documentation, and run the plugin's validation and build workflows before giving me device playtest steps. ## Claude Code Add the plugin marketplace and install the base plugin: ```text /plugin marketplace add https://github.com/facebook/meta-wearables-webapp /plugin install meta-wearables-webapp@meta-wearables ``` For game development, also install the game plugin: ```text /plugin install meta-wearables-webapp-game@meta-wearables ``` Example game request for Claude Code: Use the installed meta-wearables-webapp-game plugin and run /create-webapp-game to create a Meta Ray-Ban Display game. Start by asking me about the game concept, controls, visual direction, and any starter art or audio. Then scaffold the smallest playable version, use the bundled game documentation, and run the plugin's validation and build workflows before giving me device playtest steps. ## Codex Add the toolkit marketplace: ```bash codex plugin marketplace add https://github.com/facebook/meta-wearables-webapp ``` Start Codex, open `/plugins`, and select **Meta Wearables**. Install `meta-wearables-webapp` for standard Web Apps. For game development, also install `meta-wearables-webapp-game`. Start a new session if Codex does not discover the installed skills. Example game request for Codex: Use the installed create-webapp-game skill from meta-wearables-webapp-game to create a Meta Ray-Ban Display game. Start by asking me about the game concept, controls, visual direction, and any starter art or audio. Then scaffold the smallest playable version, use the bundled game documentation, and run the plugin's validation and build workflows before giving me device playtest steps. ## Cursor Clone the public repository and run its Cursor installer: ```bash git clone https://github.com/facebook/meta-wearables-webapp.git cd meta-wearables-webapp ./install-skills.sh cursor ``` The installer adds both `meta-wearables-webapp` and `meta-wearables-webapp-game`. Restart Cursor or run **Developer: Reload Window** so it discovers the plugins. Example game request for Cursor: Use the installed create-webapp-game skill from meta-wearables-webapp-game to create a Meta Ray-Ban Display game. Start by asking me about the game concept, controls, visual direction, and any starter art or audio. Then scaffold the smallest playable version, use the bundled game documentation, and run the plugin's validation and build workflows before giving me device playtest steps. ## Verify the installation Ask your coding agent to list the available Wearables Web App AI Toolkit skills. For standard Web Apps, verify that it can find the base implementation and testing workflows. For games, verify that it can find `create-webapp-game`, `webapp-game-director`, `iterate-webapp-game`, and `validate-webapp-game`. If the skills do not appear, restart the coding tool and confirm that the plugins were installed for the active account or project. For documentation search, follow the [Wearables Documentation MCP Server setup instructions](/docs/develop/webapps/wearables-documentation-mcp/). ### Web Apps Game Plugin ## Overview The Web Apps game plugin is an AI coding plugin that scaffolds **games** as Web Apps for Meta Ray-Ban Display. Where the [AI Coding plugin](https://github.com/facebook/meta-wearables-webapp) generates vanilla HTML, CSS, and JavaScript apps, the game plugin sets up a modern game project: Vite, TypeScript, Three.js, and Vitest, with a static build and no backend. Three.js drives both 2D games (orthographic camera with sprites) and 3D games (perspective camera with meshes), inside the 600x600 additive display and the directional input model. **Required dependency**: The game plugin builds on the AI Coding plugin and declares it as a required dependency, so installing the game plugin installs the base plugin too. That gives you the base toolset as well, including `add-ui`, `connect-api`, `add-device-sensors`, and Vercel deployment. The plugin provides these skills: | Skill | Purpose | |--|--| | `/create-webapp-game` | Scaffolds a new Vite, TypeScript, Three.js, and Vitest game project from bundled templates. Point it at a folder of starter art or audio and it files those assets under `public/` and builds the game around them | | `/webapp-game-director` | Iterates on an existing game as a creative director. It surveys the game against a nine dimension quality rubric, agrees a ranked slate of issues with you, then works them one at a time and hands each back for an on-glasses playtest | | `/iterate-webapp-game` | Drives a real desktop Chrome instance so the assistant can load the game, take screenshots, send input, read game state, and collect console errors | | `/update-webapp-game-framework` | Pulls the latest bundled framework code into an existing game after you update the plugin | | `/read-webapp-game-docs` | Discovers and reads the bundled platform and game framework documentation | | `/validate-webapp-game` | Validates a game against the project guidelines, including input handling, localized text, asset loading, logging, the display shell, and the layer boundaries | | `/add-webapp-game-logging` | Adds a log ingest backend and a passcode gated `/logs` portal, so you can read logs from a game running on the glasses | ## Prerequisites Before you install the game plugin, make sure you have the following. - An AI coding tool that reads plugins or skill files, such as Claude Code, Codex CLI, Cursor, or GitHub Copilot. - The [AI Coding plugin](https://github.com/facebook/meta-wearables-webapp). You do not need to install it separately, because the game plugin declares it as a required dependency and installs it for you. Install it on its own first if you want the base Web App skills without the game scaffolding. - Node.js and `npm`, so you can run `npm install`, `npm run dev`, `npm test`, and `npm run build` in the scaffolded project. - A Vercel account, or another host that can serve a folder of static files over HTTPS. - Meta Ray-Ban Display glasses paired with the Meta AI app, for playtesting. See [Test](/docs/develop/webapps/test/) for how to load a Web App onto the glasses. - A desktop Chrome installation, if you want `/iterate-webapp-game` to inspect the game visually. The [Meta Ray-Ban Display Simulator](https://chromewebstore.google.com/detail/jpjlmmodokemlepklkdbimceggpbjcll) Chrome extension recreates the 600x600 display surface, including additive blending and directional input, so you can preview a build without the hardware. Two optional items improve the experience: - Starter art and audio. Gather whatever sprites, spritesheets, 3D models, and sound files you have into one folder and mention the path in your prompt. No assets is fine too, because the game is then built from procedural Three.js geometry. - [ImageMagick](https://imagemagick.org). It lets the assistant crop a single sprite out of a large sheet, zoom one frame at a time, and measure the sheet exactly. The skill checks for it, tells you the install command for your platform if it is missing, and carries on without it. Install ImageMagick with the command for your platform: **macOS**: ```bash brew install imagemagick ``` **Debian or Ubuntu**: ```bash sudo apt-get install imagemagick ``` **Fedora or RHEL**: ```bash sudo dnf install ImageMagick ``` ## Limitations Understand these constraints before you plan a game. - **No console on the device**. The glasses cannot be tethered, so once the game is on the glasses there is no browser console. Use the on-display options or remote logging described in [Debug on the glasses](#debug-on-the-glasses). - **The desktop browser is not the device**. `/iterate-webapp-game` and `npm run dev` verify logic and layout, but desktop input, the additive display, and desktop performance all differ from the glasses. Desktop checks complement a playtest, they do not replace one. - **Only you can playtest**. The assistant cannot wear the glasses. A change that nobody has played on the device is recorded as unverified, not done. The director skill will not declare a game finished on its own. - **Static build only**. Vite emits a folder of static assets and the game ships with no backend. Do not add a `server.js` file or a `start` script to `package.json`, because that makes Vercel run the app as a Node function and return 404 for every route. - **Deploy from the project root**. Because this is a Vite build tool app, deploy the project root and not `dist/`. Vercel detects Vite, runs the build, and serves the output. Use the base plugin's `/publish-to-vercel` and `/test-on-device` skills only for the account level steps, such as login, disabling Deployment Protection, and aliasing. - **Three.js renders the game world, the DOM renders the interface**. A WebGL canvas renders the game, and an HTML overlay renders the HUD, menus, and all text. Never render text in WebGL. - **Gameplay code stays renderer agnostic and input agnostic**. Gameplay talks to a `Renderer` interface and an `InputManager` interface, and never imports `three` or touches the DOM. This keeps gameplay unit testable and the renderer swappable. - **The framework directory is managed code**. Treat `src/framework/` as engine code that you re-pull rather than hand edit, and keep your own code in `src/models.ts`, `src/core/`, `src/hud/`, `src/config/`, and `src/main.ts`. - **Device budgets apply**. The scaffold respects the shared Meta Ray-Ban Display guidelines: a 600x600 viewport, HUD text of at least 16px, 30 fps, JavaScript under 500 KB gzipped, and runtime memory under 128 MB. - **Fill is the tightest constraint on the additive display**. Keep the page background black, because black is transparent on an additive display and anything you fill is light the app user sees over the real world. Dark gray is permitted on bounded surfaces, such as a card or a panel. The game plugin adds one carve-out of its own: an always-on HUD takes no fill at all, not even that dark gray, because it is on screen the whole time rather than only when the app user opens it. The plugin states this rule in its bundled `docs/core-contract.md`, and the shared display guidelines state the dark gray rule for bounded surfaces without the carve-out. ## Installation The game plugin ships in the [Meta Wearables Web App AI Toolkit](https://github.com/facebook/meta-wearables-webapp) repository, alongside the base AI Coding plugin. Both plugins live under `plugins/`, and the game plugin's skills are one directory per skill: ```text meta-wearables-webapp/ .claude-plugin/marketplace.json install-skills.sh plugins/ meta-wearables-webapp/ base AI Coding plugin meta-wearables-webapp-game/ game plugin .claude-plugin/plugin.json docs/ skills/ create-webapp-game/SKILL.md webapp-game-director/SKILL.md iterate-webapp-game/SKILL.md update-webapp-game-framework/SKILL.md read-webapp-game-docs/SKILL.md validate-webapp-game/SKILL.md add-webapp-game-logging/SKILL.md ``` 1. `install-skills.sh` installs the base plugin and the game plugin for every tool it supports. 1. `.claude-plugin/marketplace.json` registers the base `meta-wearables-webapp` plugin and the game plugin, so you can also install using Claude plugin marketplace. ### Muse Code Add the marketplace and install the game plugin: ```text /plugins marketplace add meta-wearables-webapp https://github.com/facebook/meta-wearables-webapp /plugins install meta-wearables-webapp-game@meta-wearables-webapp ``` The game plugin installs the base `meta-wearables-webapp` plugin as a required dependency. Start a new Muse Code session after installation so it discovers both plugins and their skills. ### Claude Code Add the marketplace once, then install the game plugin: ```bash /plugin marketplace add https://github.com/facebook/meta-wearables-webapp /plugin install meta-wearables-webapp-game@meta-wearables ``` Installing the game plugin also installs the base `meta-wearables-webapp` plugin, because the game plugin declares it as a required dependency. To update later: ```bash /plugin marketplace update meta-wearables && /plugin update meta-wearables-webapp-game@meta-wearables ``` ### Codex CLI Add the marketplace from your terminal: ```bash codex plugin marketplace add https://github.com/facebook/meta-wearables-webapp ``` Start Codex, type `/plugins`, tab to **Meta Wearables**, and install the game plugin. To refresh the marketplace source later: ```bash codex plugin marketplace upgrade meta-wearables ``` Then open `/plugins` inside Codex and select the update option if a newer version is available. ### Cursor, GitHub Copilot, and other tools Clone the repository and copy the game plugin into the location your tool reads. Cloning and copying does not change your app code. ```bash git clone https://github.com/facebook/meta-wearables-webapp.git cd meta-wearables-webapp ``` For Cursor, install the plugin for your user account: ```bash mkdir -p ~/.cursor/plugins/local cp -R plugins/meta-wearables-webapp-game ~/.cursor/plugins/local/ cp -R plugins/meta-wearables-webapp ~/.cursor/plugins/local/ ``` Restart Cursor, or run **Developer: Reload Window**, so it discovers the plugins. For any tool that reads project level skill files, copy both skill sets into your project: ```bash mkdir -p YOUR_PROJECT/.claude/skills cp -R plugins/meta-wearables-webapp/skills/. YOUR_PROJECT/.claude/skills/ cp -R plugins/meta-wearables-webapp-game/skills/. YOUR_PROJECT/.claude/skills/ ``` The repository's `install-skills.sh` script also configures GitHub Copilot through `.github/copilot-instructions.md`, and Gemini CLI, Windsurf, and Devin through `AGENTS.md`. Those paths cover the base Web App skills. Copy the game plugin skills as shown above to add the game workflows. ### Validate the installation Start a fresh session if a running session does not discover the plugin. Then copy this prompt into your AI tool. It only reads and reports, so it does not scaffold a project, install anything, or modify a file. Run a read-only check of my Meta Ray-Ban Display Web Apps game setup and report the results as a table. Do not create, modify, or delete any file, do not scaffold a project, and do not install anything. Check and report: (1) which of these skills you can see and invoke, by name: create-webapp-game, webapp-game-director, iterate-webapp-game, update-webapp-game-framework, read-webapp-game-docs, validate-webapp-game, add-webapp-game-logging; (2) whether the base Web Apps skills are also available, naming two of them; (3) the game plugin version you have loaded, and where you loaded the plugin from; (4) which of node, npm, and git are on my PATH, with versions, and whether Chrome and ImageMagick are present; (5) whether the Wearables MCP tool search_webapps_docs is configured, and if it is, the result of one query for "viewport size". Finish with a single line that says READY or NOT READY, and if NOT READY list only the missing items. A successful check reports all seven game skills, at least two base skills, a plugin version, and Node.js and `npm` on your `PATH`. Chrome, ImageMagick, and the documentation search tool are optional, so a report can be READY without them. ## How to use ### Scaffold a game 1. Gather your starter art and audio into one folder, if you have any. 2. Run `/create-webapp-game` and describe the game you want. Mention the asset folder path in your prompt. The skill files your assets under `public/`, moving them if they are already in the project and copying them if they are not, so your originals stay put. 3. Accept or decline the visual verification pass when the skill offers it during planning. If you accept, the skill proves a Chrome instance is reachable before the build starts, then sweeps the finished build. If you decline, the summary states plainly that the visual checks did not run. Or copy this prompt into your AI tool: Use the installed Web Apps game plugin to scaffold a new game for Meta Ray-Ban Display. Run /create-webapp-game, and file the starter art and audio in ASSET_FOLDER_PATH into the project. Before you build, read the bundled display and performance guidelines with /read-webapp-game-docs, and confirm the design fits a 600x600 viewport, 30 fps, and directional input. Include the visual verification pass. When the build is done, run the typecheck, test, validate, and build gates, then tell me exactly how to playtest it on my glasses. The result is a ready to run project: ```text my-game/ package.json tsconfig.json vite.config.ts README.md docs/design.md src/ index.html style.css main.ts log.ts models.ts core/ config/ hud/ framework/ ``` Run it locally: ```bash cd my-game npm install npm run dev npm run typecheck && npm test && npm run build ``` In the desktop browser, use the arrow keys, **Enter**, and a left mouse drag. ### Iterate on a game Scaffolding is the easy half. `/webapp-game-director` is the loop that follows. It plays the current build, scores it against a quality rubric covering the core loop, juice, clarity, onboarding, pacing, depth, cohesion, performance, and accessibility, then puts a ranked slate of issues in front of you rather than a verdict. You pick what to tackle. Each issue is worked to a change, gated by `typecheck`, `test`, `validate`, and `build`, then handed back to you to playtest on the glasses. Your playtest feedback closes the issue. When the slate is clear, the skill re-surveys the game and opens the next milestone. Underneath the director, `/iterate-webapp-game` is how the assistant sees the game without wearing the glasses. It drives a real desktop Chrome instance to load the build, take screenshots, send arrow key and **Enter** input, drag, read game state, and pull console errors. Run `npm run chrome` once and leave it running. The skill wires up that script, and the assistant starts Chrome itself where its sandbox permits. `/iterate-webapp-game` also carries a verification pass, which is a fixed sweep over a finished build. It checks whether the build boots cleanly, whether it draws anything, whether that would be visible on the additive display, whether every input does something, whether the HUD stays inside 600x600, and whether the build does what `docs/design.md` says. It fixes the clear cut failures and reports the rest. ### Validate a game Run `/validate-webapp-game` to check a game against the project guidelines. It verifies that input does not come from the DOM, that all text is localized, that assets are not loaded from the network at runtime, that no raw `console` calls remain, that the display shell is intact, and that the layer boundaries hold, so gameplay imports no `three`, no DOM, and no Web Audio. ### Update the framework `src/framework/` is managed engine code, shared across games and meant to be re-pulled rather than hand edited. When the plugin ships an improved framework, run `/update-webapp-game-framework` from inside a game project. It previews the diff, confirms before writing, re-runs the build gate, and leaves your game code untouched. The framework version matches the plugin version. It is recorded in `src/framework/VERSION` and in the `` tag in `src/index.html`, which is the attribution marker that survives into a released game. Updating re-stamps both, and inserts the tag in a game that was scaffolded before the tag existed. ### Read the bundled documentation Run `/read-webapp-game-docs` to discover and read the platform and game framework documentation bundled with the plugin, including the documentation index, the project structure and tooling conventions, what renders in Three.js and what renders in the DOM, the renderer and input agnostic layering, asset loading for 2D textures and 3D models, logging, and the shared display and performance guidelines. ### Debug on the glasses There is no console once the game is on the glasses. Three options are available, cheapest first: ```text ?stats ?log=debug&logview ?log=debug&logkey=YOUR_TOKEN ``` - `?stats` draws live frame rate, CPU, and draw call counts on the display. - `?log=debug&logview` draws the most recent log lines on the display, with no backend needed. - `?log=debug&logkey=YOUR_TOKEN` sends logs to your own deployment, where you read them at `/logs`. The first two options need nothing but a URL. The third is added by `/add-webapp-game-logging`. The game posts batched records to serverless endpoints in its own deployment, and you read them from a passcode gated portal on your laptop. The game shows a blocking consent screen before anything is transmitted, and every endpoint returns 404 unless you deliberately set a log token. ## Troubleshooting | Symptom | Likely cause | Fix | |--|--|--| | The game skills do not appear | The running session started before the install | Start a fresh session, then confirm that `/create-webapp-game` is available | | Every route returns 404 after deploying | The project has a `server.js` file or a `start` script, so Vercel runs it as a Node function | Remove both, and deploy the project root rather than `dist/` | | A redeploy is not picked up on the glasses | A stale cached asset | Reload the Web App, since content outside the hashed build output is revalidated on the next load | | `/iterate-webapp-game` cannot inspect the game | No Chrome instance is reachable | Run `npm run chrome` in the project and leave it running | | The assistant cannot read your spritesheet grid | ImageMagick is not installed | Install ImageMagick with the command for your platform, then retry | | Text looks blurry or is missing on the display | Text was rendered in WebGL | Move all text into the HTML overlay, and keep HUD text at 16px or larger | | The HUD washes out the real world | The HUD fills a background | Remove the fill from an always-on HUD, and keep dark gray for bounded surfaces only | ## Next steps - Set up the shared documentation search server so your assistant can query current Web Apps guides. See the [Wearables Documentation MCP Server](/docs/develop/webapps/wearables-documentation-mcp/). - Review the display and performance guidelines before your first playtest. - Load your build onto the glasses and playtest it, then bring the feedback back to `/webapp-game-director`. ### App icons For web apps launched directly by URL, Meta Ray-Ban Display glasses read the app's name and icon appearance from a Meta Wearables app manifest. The system combines this manifest information with the monochrome icon artwork to render an app icon that follows the system appearance. The device fetches the manifest directly from a well-known location relative to the launch URL. It does not inspect page markup or favicons for icon information. To render the same manifest-themed treatment inside a toolkit interface, use [`WebAppIcon`](/docs/develop/webapps/design/components/mrbd/web-app-icon/). The component accepts resolved artwork and appearance values; it does not fetch or parse the manifest. Use the manifest to provide: - the web app's display name; - an optional plain-text description; - transparent monochrome icon artwork; and - a theme color, custom gradient, or both. ## Add the manifest For a web app launched directly from a URL, place `meta-wearables-manifest.json` in a `.well-known` directory beside the launch directory. For a web app launched from: ```text https://example.com/ ``` the manifest is: ```text https://example.com/.well-known/meta-wearables-manifest.json ``` The location is relative to the launch URL's containing directory. This lets one host publish several independent web apps. | Web app launch URL | Manifest URL | | --- | --- | | `https://example.com/` | `https://example.com/.well-known/meta-wearables-manifest.json` | | `https://apps.example.com/drawing/` | `https://apps.example.com/drawing/.well-known/meta-wearables-manifest.json` | | `https://apps.example.com/music/index.html` | `https://apps.example.com/music/.well-known/meta-wearables-manifest.json` | | `https://games.example.com/chess/` | `https://games.example.com/chess/.well-known/meta-wearables-manifest.json` | Use a trailing slash for a directory-style launch URL. For example, `https://example.com/chess/` identifies the `chess` directory, while `https://example.com/chess` is treated like a filename and resolves from its containing directory. Subdomains and explicit ports are preserved. Production manifest and icon URLs must use `https://`. Plain `http://` URLs are accepted only for loopback development hosts such as `localhost` and `127.0.0.1`. Serve the manifest as UTF-8 JavaScript Object Notation with this content type: ```http Content-Type: application/json; charset=utf-8 ``` The manifest must be no larger than 64 KiB. ## Create a Version 1 manifest Start with a theme color and monochrome icon: ```json { "version": 1, "name": "Notes", "description": "Capture and organize quick notes.", "appearance": { "theme_color": "#3867D6", "icon": { "src": "../icons/notes.webp" } } } ``` The device derives the icon container's radial gradient from `theme_color` and uses the same color to tint the monochrome artwork. ### Top-level fields | Field | Required | Guidance | | --- | --- | --- | | `version` | Yes | Use the integer `1`. An unsupported version invalidates the manifest. | | `name` | Yes | A non-empty display name of at most 100 characters after trimming. | | `description` | No | Concise plain text with no markup. Keep it to 500 Unicode characters or fewer. It is reserved for future display and is not currently shown. | | `appearance` | Yes | Contains the icon and its background treatment. | Clients ignore unknown fields, so future additions do not break Version 1 manifests. ## Configure the appearance A valid `appearance` object requires: - an `icon` object with a valid `src`; and - at least one background source: `theme_color` or `icon_background.colors`. ```json { "appearance": { "theme_color": "#6750A4", "icon": { "src": "../assets/icon.webp", "type": "monochrome" }, "icon_background": { "colors": ["#8257FF", "#5A2AE6", "#3214A8", "#18065E"], "stops": [0.0, 0.32, 0.7, 1.0], "inner_glow_color": "#BCA7FF" } } } ``` ### Theme color `theme_color` is optional when a valid custom gradient is present. Use one of these hexadecimal forms: - `#RGB` - `#RGBA` - `#RRGGBB` - `#RRGGBBAA` When alpha is present, it is the final component. Opaque colors are recommended because their contrast is more predictable. The theme color has two roles: 1. Without a valid custom gradient, the device derives the container gradient from it. 2. It tints monochrome artwork, even when you supply a custom gradient. When `theme_color` is absent, the first valid gradient color tints the artwork. ### Icon The `icon` object supports: | Field | Required | Guidance | | --- | --- | --- | | `src` | Yes | A non-empty relative, origin-relative, or absolute icon URL. | | `type` | No | Omit it or use `monochrome`. Any other value invalidates the appearance. | The device resolves `src` against the manifest URL using standard URL resolution. For a manifest at `https://apps.example.com/drawing/.well-known/meta-wearables-manifest.json`: | `src` | Resolved icon URL | | --- | --- | | `icon.webp` | `https://apps.example.com/drawing/.well-known/icon.webp` | | `../icon.webp` | `https://apps.example.com/drawing/icon.webp` | | `/shared/icons/drawing.png` | `https://apps.example.com/shared/icons/drawing.png` | | `https://cdn.example.net/drawing.svg` | `https://cdn.example.net/drawing.svg` | The resolved icon URL follows the same secure-network requirements as the manifest URL. ### Icon background `icon_background` is optional when `theme_color` is present. Use it to define a custom radial gradient and focused inner glow. | Field | Required | Guidance | | --- | --- | --- | | `colors` | No | At least two valid color strings. If any color is invalid, the complete custom gradient is discarded. | | `stops` | No | At least two numbers from `0.0` through `1.0`, in non-decreasing order, with one stop per color. | | `inner_glow_color` | No | A color used for the focused inner glow. It does not change the artwork tint or container gradient. | Invalid or mismatched stops do not invalidate valid colors. The device keeps the custom gradient colors and selects standard stop positions instead. When `inner_glow_color` is absent or invalid, the device derives it from `theme_color`, or from the first gradient color when no theme color is present. ## Prepare the icon artwork Provide transparent, single-color mask artwork designed for a 64 × 64 icon area. The device centers and aspect-fits the artwork within that area. Supported source formats are: - Portable Network Graphics (`.png`) - WebP (`.webp`) - Scalable Vector Graphics (`.svg`) For monochrome artwork, transparency defines the mask. The source artwork's colors are not preserved. Leave enough transparent space around the visual bounds so the icon does not feel crowded after the device applies its highlight, blur, and material effects. The icon response must be no larger than 2 MiB. Raster images must decode successfully and cannot exceed 4,096 pixels in either dimension. A Scalable Vector Graphics file must be no larger than 512 KiB and cannot load external entities or files. Vector artwork is rendered to a transparent 64 × 64 bitmap before receiving the same treatment as raster artwork. ## Choose colors and contrast The icon container is an illuminated radial material, not a flat color fill. Evaluate the complete icon in its default, focused, and pressed states rather than reviewing individual color swatches alone. ### Use a theme color When you provide only `theme_color`, the device derives a four-stop palette that moves from bright at the center to dark at the edge. The approximate contrast targets against white are: | Radial stop | Position | Target contrast against white | | --- | ---: | ---: | | Brightest | `0.0` | `6.5:1` | | Midtone | `0.33` | `12.5:1` | | Dark | `0.66` | `18.8:1` | | Edge | `1.0` | `18.8:1` | The targets use the standard relative-luminance contrast ratio against white. The repeated dark target produces a stable edge around the material. Exact rendered values can vary slightly when converted to displayable colors. The original theme color remains the artwork tint and default focused inner-glow color. The device derives the lighter edge highlight automatically. ### Design a custom gradient Custom gradient colors are used as provided. The device does not reorder or correct them. For a four-stop gradient consistent with the system treatment, use this profile as a starting point: | Gradient stop | Recommended position | Target contrast against white | | --- | ---: | ---: | | First | `0.0` | `6.5:1` | | Second | `0.33` | `12.5:1` | | Third | `0.66` | `18.8:1` | | Fourth | `1.0` | `18.8:1` | These are recommended targets, not validation thresholds for custom colors. A brighter brand-led palette can use lower contrast against white when essential icon contours still maintain at least `3:1` contrast against every part of the rendered gradient. For a reliable custom gradient: - Use three or four colors ordered from brightest to darkest. - Keep relative brightness from increasing between neighboring stops. - Prefer neighboring hues over abrupt hue changes. - Keep at least `3:1` contrast between essential icon contours and every part of the rendered gradient. - If text appears over the material, use at least `4.5:1` contrast for standard text or `3:1` for large text. - Set `theme_color` independently when the artwork needs a different tint from the first gradient color. - Choose an inner-glow color from a related midtone that adds separation without overpowering the artwork. ### Appearance precedence | Manifest content | Container background | Monochrome artwork tint | | --- | --- | --- | | Valid gradient and theme color | Custom gradient | Theme color | | Valid gradient only | Custom gradient | First gradient color | | Theme color only | Derived gradient | Theme color | | Neither | Fallback treatment | Fallback treatment | ## Examples ### Custom gradient The first gradient color tints the artwork because no theme color is present. It also supplies the default inner-glow color. ```json { "version": 1, "name": "Weather", "appearance": { "icon": { "src": "../icons/weather.png", "type": "monochrome" }, "icon_background": { "colors": ["#72D9FF", "#3B82F6", "#2446A8", "#111B55"], "stops": [0.0, 0.28, 0.68, 1.0] } } } ``` ### Theme color, gradient, and inner glow The gradient controls the container, the theme color controls the artwork tint, and the inner-glow color controls focused lighting. ```json { "version": 1, "name": "Music Studio", "appearance": { "theme_color": "#FF4DA6", "icon": { "src": "../icons/music.svg" }, "icon_background": { "colors": ["#7C3AED", "#5B21B6", "#2E1065", "#13072E"], "stops": [0.0, 0.35, 0.72, 1.0], "inner_glow_color": "#70E1F5" } } } ``` ### Gradient without stops This is valid. The device chooses standard stop positions. ```json { "version": 1, "name": "Maps", "appearance": { "icon": { "src": "/shared/icons/maps.webp" }, "icon_background": { "colors": ["#34D399", "#059669", "#065F46", "#022C22"] } } } ``` ### Icon hosted on another origin The icon does not need to live beside the manifest. Absolute secure URLs are supported. ```json { "version": 1, "name": "Gallery", "appearance": { "theme_color": "#0EA5E9", "icon": { "src": "https://cdn.example.net/wearables/gallery-icon.webp" } } } ``` ## Fallback behavior The device uses a standard fallback icon when the manifest or appearance is missing or invalid. Common causes include: - the manifest or icon cannot be found; - the manifest contains malformed JavaScript Object Notation; - a required field is missing or invalid; - neither a theme color nor valid gradient colors are present; - `icon.type` has an unsupported value; or - the icon exceeds a size limit or cannot be decoded. A temporary network failure does not by itself replace a previously valid icon. The device revalidates published appearances and keeps the last valid appearance when a refresh fails temporarily. ## Publishing checklist - For a direct URL launch, put the manifest in the web app directory's `.well-known` subdirectory. - Serve both the manifest and icon over `https://` in production. - Keep the manifest under 64 KiB, the icon response under 2 MiB, and Scalable Vector Graphics artwork under 512 KiB. - Supply a non-empty name of at most 100 characters. - Optionally include a concise plain-text description for future use. - Supply transparent monochrome artwork designed for a 64 × 64 icon area. - Use Portable Network Graphics, WebP, or Scalable Vector Graphics artwork. - Supply `theme_color`, at least two gradient colors, or both. - When supplying stops, provide one ordered value from `0.0` through `1.0` per gradient color. - For a custom gradient, order colors from bright to dark and verify at least `3:1` contrast for essential icon contours across the rendered material. - For a direct URL launch, test the exact launch URL because its directory determines the manifest location. - Verify the default, focused, pressed, loading, and fallback states on the device. ### Display and layout ## How the display renders Meta Ray-Ban Display glasses use an additive-light display. An unlit black pixel adds no light, while brighter pixels add light over the wearer's view. The surroundings therefore remain part of the visual result. Use an unlit black page canvas. Add bounded surfaces where text or controls need separation from the surroundings instead of painting a bright full-screen background. Validate readability over both bright and dark real-world scenes. Communicate state with structure or text as well as color. ## Size from the viewport The device display is 600 × 600, but the page should not hardcode a 600-pixel layout. Treat 600 × 600 as a validation target and derive application dimensions from the browser's actual viewport. This keeps the page usable in desktop preview, text scaling, and future host configurations. Use the standard responsive viewport declaration: ```html ``` Keep browser scaling available by omitting `user-scalable=no`. A framework-neutral page shell can fill the available viewport without fixing its dimensions: ```css * { box-sizing: border-box; } html, body, #app { inline-size: 100%; block-size: 100%; } body { margin: 0; min-inline-size: 0; background: #000; } .app-shell { display: grid; grid-template-rows: minmax(0, 1fr) auto; inline-size: 100%; block-size: 100%; overflow: hidden; } .content { min-inline-size: 0; min-block-size: 0; } ``` Use percentages, flexible grid or flex tracks, intrinsic content sizes, and logical CSS properties. Reserve media queries for meaningful layout changes rather than forcing a 600-pixel width or height. ## Keep controls away from obstructed edges System UI and focus feedback can use space near the display edges. Keep important text and interactive controls comfortably inside the viewport. Full-bleed backgrounds and noninteractive artwork can extend farther than controls. Use a safe-region value only when the platform documents it for the supported runtime. Test every edge control on the device and verify that system UI, focus feedback, and pressed states remain visible. ## Make overflow intentional Keep document scrolling intentional and keep content reachable. When content exceeds the available area: - Prefer one scroll owner per axis. - Give the scroll region a flexible track such as `minmax(0, 1fr)` and set `min-block-size: 0` on shrinking ancestors. - Keep the selected control visible as directional navigation moves through the region. - Keep one scroller on each axis so focus and scroll ownership remain clear. - Use pagination or progressive disclosure when it makes the task clearer. - Provide a visible cue that more content is available. For a short, single-task screen, a static layout may be appropriate. For longer content, intentional scrolling is preferable to clipping or shrinking everything until it becomes unreadable. ## Use readable type and surfaces Use relative type sizes and adequate line height rather than fixed-height text boxes. Allow text to wrap and test the longest localized strings. Reserve large type for the information that needs the strongest hierarchy. Use sufficient luminance contrast, stroke weight, spacing, and bounded surfaces to separate content from the wearer's surroundings. Bright saturation by itself does not guarantee readability. Make focus and selection visible through more than color alone. ## Validate the complete layout Before publishing, verify: - the page at a 600 × 600 browser viewport without fixed-width assumptions; - the physical device in bright and dark surroundings; - browser text scaling and the longest supported localized strings; - directional traversal through every interactive control; - focus and pressed feedback near each edge; - the empty, loading, error, and maximum-content states; - intentional scrolling, clipping, and overflow behavior; and - that critical content remains reachable without a mouse, touchscreen, or continuous drag input. ### Input and navigation ## Browser-managed directional navigation Meta Ray-Ban Display glasses receive directional and activation input from the touchpad and, when available, Meta Neural Band. The browser handles ordinary navigation: it finds visible, semantic controls, moves spatial selection to the nearest control in the requested direction, and activates the selected control. When no control is available in a direction, the browser can scroll the page instead. For ordinary pages, let the browser own directional movement and activation. Global arrow-key listeners, manual focus indexes, and synthesized clicks compete with browser navigation, reduce a two-dimensional layout to document order, and can activate a control twice. ## Build a navigable interface Prefer native controls: ```html ``` Native buttons, links, inputs, selects, and text areas already provide semantics, focusability, and activation behavior. Handle ordinary `click`, `input`, and `change` events instead of translating device inputs yourself. Arrange the DOM and visual layout so nearby controls are sensible directional neighbors. Keep disabled or hidden controls out of navigation and preserve the native focus order instead of assigning positive `tabindex` values. Use `tabindex="0"` only for a custom widget that cannot be represented by a native element and that implements the complete corresponding semantics and interaction contract. Every critical action must remain reachable through browser-managed directional navigation and activation. ## Keep selection visible Keep the browser's focus indicator or provide an equivalent. A browser-neutral enhancement can use `:focus-visible`: ```css :where(button, a, input, select, textarea):focus-visible { outline: 3px solid currentColor; outline-offset: 4px; } ``` The wearable browser also provides spatial-selection feedback. Test custom styles alongside that feedback on the device, especially near viewport edges and inside scroll regions. Use browser focus and spatial selection as the source of truth rather than maintaining a second `.focused` state. ## Advanced custom interactions Intercept input only inside a widget whose internal behavior cannot be expressed with native controls, such as a game board or a specialized range editor. Scope the listener to the widget, handle only the input the widget owns, and call `preventDefault()` only when the custom behavior intentionally replaces the browser default and provides a clear way out. Wearable directional input is not a page-level `ArrowUp`, `ArrowDown`, `ArrowLeft`, `ArrowRight`, or `Enter` key-event contract. The browser can consume those inputs while performing spatial navigation and activation. Keyboard events remain useful for desktop testing and for hardware keyboards, but they are not the wearable navigation contract. ## Activation The activation action triggers the browser-selected control. Native controls therefore work with their normal `click` behavior: ```javascript const start = document.getElementById('start'); start.addEventListener('click', () => { beginSession(); }); ``` There is no free cursor in the default mode. Keep essential actions independent of hover and positioned click coordinates. ## Continuous drag Continuous drag is optional and useful for sliders, drawing, panning, and games. Opt in at page load by setting `touch-action: none` on `body` in the initial stylesheet: ```css body { touch-action: none; } ``` The host reads this page-level value once per load. Setting it later with JavaScript or only on an individual element does not opt the page into wearable drag input. Use standard pointer events and pointer capture: ```javascript const surface = document.getElementById('drawing-surface'); let dragging = false; surface.addEventListener('pointerdown', (event) => { dragging = true; surface.setPointerCapture(event.pointerId); updatePosition(event.clientX, event.clientY); }); surface.addEventListener('pointermove', (event) => { if (dragging) { updatePosition(event.clientX, event.clientY); } }); surface.addEventListener('pointerup', (event) => { dragging = false; surface.releasePointerCapture(event.pointerId); }); surface.addEventListener('pointercancel', () => { dragging = false; }); ``` Wearable drag availability depends on hardware and runtime configuration. Keep all essential actions usable through directional navigation and activation. Test device event coordinates before depending on a particular range or motion scale; desktop pointer simulation does not reproduce wearable input exactly. ## Back navigation The native shell handles Back. When browser history has a previous entry, it calls `history.back()`. When no previous entry remains, Back opens the native navigation overlay; pressing Back again exits to the native app boundary. Native transient UI, such as the text composer, is dismissed before page history is traversed. Prefer browser history to an on-screen Back button. Reaching a Back button can require the user to navigate through every control that precedes it, while the system Back action traverses the history stack in one step from anywhere on the page. Add an on-screen Back control only when the interface requires an additional visible navigation affordance; it should call the same history action rather than maintain separate navigation state. Add a history entry for a destination the user would expect Back to reverse, and restore application state during traversal: ```javascript const HOME_STATE = {screen: 'home'}; history.replaceState(HOME_STATE, ''); function openSettings() { history.pushState({screen: 'settings'}, ''); renderScreen({screen: 'settings'}); } window.addEventListener('popstate', (event) => { renderScreen(event.state ?? HOME_STATE); }); ``` The host limits `pushState()` history to five total entries, including the initial document. Keep that stack for meaningful destinations, not focus changes, tooltips, or other transient UI. Let history drain to the native app boundary instead of pushing a new entry while handling Back. Use either `popstate` or the Navigation API to coordinate traversal, not both for the same state transition. Restore serializable application state after the destination has rendered. ## Text input Activating a supported text field opens the system composer for handwriting, dictation, and on-screen keyboard input. Use semantic fields; no custom input SDK is required. Supported fields include: - text-like inputs: `text`, `search`, `email`, `url`, `tel`, and `number`; - `textarea`; and - `contenteditable` elements. Password, date, checkbox, and radio inputs do not open the composer. Use native checkbox and radio behavior for selection rather than treating them as text fields. ```html ``` Read committed text through `input` and `change`: ```javascript const search = document.getElementById('search'); const results = document.getElementById('results'); search.addEventListener('input', () => { results.textContent = `Searching: ${search.value}`; }); ``` The composer manages its own text buffer and commits the result through `input` and `change`. Per-character `keydown`, `keypress`, and `keyup` events are not available for handwriting, dictation, or the on-screen keyboard. The on-screen keyboard has a limited character set. Design device-entered values around the available characters or provide another input path. `inputmode` and `enterkeyhint` do not change the composer. Provide another path when text entry is unavailable or a required value cannot be entered through the supported fields. ## Test on the device Verify that: - every interactive control is reached in a sensible spatial order; - activation triggers each control once; - focus feedback remains visible near edges and while scrolling; - custom widgets do not trap directional navigation; - Back reverses meaningful destinations and can reach the native app boundary; - essential actions remain available without continuous drag; - text fields open the composer and process committed `input` values; and - the app remains usable when optional input hardware or composer features are unavailable. ### Sensors and location ## Sensors Meta Ray-Ban Display glasses expose motion and orientation through `DeviceMotionEvent` and `DeviceOrientationEvent`. These high-frequency readings are permission-gated, nullable, and time-sensitive. Feature-detect the APIs, request access in response to a clear user action, and keep the app usable when access is denied or readings stop. ### Request permission The device provides `requestPermission()` on both event constructors. They share one sensor permission, so call either available method once: ```javascript let sensorListenersAttached = false; function attachSensorListeners() { if (sensorListenersAttached) { return; } window.addEventListener('devicemotion', handleMotion); window.addEventListener('deviceorientation', handleOrientation); sensorListenersAttached = true; } async function requestSensorAccess() { const orientation = window.DeviceOrientationEvent; const motion = window.DeviceMotionEvent; if (!orientation && !motion) { showSensorUnavailable('Sensors are not supported.'); return; } const permissionOwner = typeof orientation?.requestPermission === 'function' ? orientation : typeof motion?.requestPermission === 'function' ? motion : null; try { const state = permissionOwner ? await permissionOwner.requestPermission() : 'granted'; if (state !== 'granted') { showSensorUnavailable('Sensor access was not granted.'); return; } attachSensorListeners(); } catch (error) { showSensorUnavailable('Sensor access is unavailable.'); } } document .getElementById('enable-sensors') .addEventListener('click', requestSensorAccess); ``` Requesting access from a button gives the user context and also works on browsers that require transient activation for sensor permission. Track listener state so repeated permission requests do not attach duplicate listeners. Remove listeners when the feature or page no longer needs them: ```javascript function stopSensors() { window.removeEventListener('devicemotion', handleMotion); window.removeEventListener('deviceorientation', handleOrientation); sensorListenersAttached = false; } ``` ### Motion data `DeviceMotionEvent` can provide acceleration, acceleration including gravity, rotation rate, and the interval between readings. Guard both each containing object and each numeric member: ```javascript function finite(value) { return typeof value === 'number' && Number.isFinite(value); } function handleMotion(event) { const acceleration = event.accelerationIncludingGravity; if ( acceleration && finite(acceleration.x) && finite(acceleration.y) && finite(acceleration.z) ) { updateAcceleration({ x: acceleration.x, y: acceleration.y, z: acceleration.z, interval: event.interval, }); } const rotation = event.rotationRate; if ( rotation && finite(rotation.alpha) && finite(rotation.beta) && finite(rotation.gamma) ) { updateRotation({ alpha: rotation.alpha, beta: rotation.beta, gamma: rotation.gamma, }); } } ``` | Field | Unit | Availability | | --- | --- | --- | | `accelerationIncludingGravity.x/y/z` | Meters per second squared | Values can be `null`; validate before use. | | `acceleration.x/y/z` | Meters per second squared | Gravity-removed readings can be unavailable. | | `rotationRate.alpha/beta/gamma` | Degrees per second | Values can be `null`. This differs from Generic Sensor `Gyroscope`, which uses radians per second. | | `interval` | Milliseconds | Treat as reported timing, not a guaranteed fixed frequency. | Use orientation or a suitable fusion method for sustained tilt. Dynamic acceleration alone does not provide a stable orientation reference. Throttle visual updates instead of writing to the DOM for every sensor event. ### Orientation data Orientation events expose `alpha`, `beta`, `gamma`, and `absolute`. Values can be missing or relative to different reference frames, so inspect `absolute` and validate every number before use: ```javascript function handleOrientation(event) { if ( !finite(event.alpha) || !finite(event.beta) || !finite(event.gamma) ) { showOrientationUnavailable(); return; } updateOrientation({ alpha: event.alpha, beta: event.beta, gamma: event.gamma, absolute: event.absolute === true, receivedAt: performance.now(), }); } ``` The runtime can improve heading when a higher-quality source is available. Treat Earth-relative or compass-grade heading as conditional on the event and source. Calibrate the experience for its intended use and retain a fallback when heading or accuracy is insufficient. `deviceorientationabsolute` can be available on supported builds; feature-detect it rather than assuming universal browser support. ### Permission loss and stale readings A user can revoke sensor access while the app is open. When access is revoked, the browser stops event delivery, but it does not dispatch a public revocation event to the page. The runtime retains the page's registered listeners, so a later successful `requestPermission()` call resumes forwarding without requiring the app to register them again. Track when the last reading arrived. When updates stop, move the interface to a neutral or stale state before the old reading can drive navigation, safety-sensitive decisions, or persistent state. Offer an explicit action to request access again. ## Location Meta Ray-Ban Display implements `navigator.geolocation`. One-shot positions are normally sourced through the paired phone because the glasses do not contain a standalone satellite-navigation receiver. Availability and accuracy depend on permission, the paired-device connection, environment, and location services. ### Request one position Always provide success and error callbacks: ```javascript function requestPosition() { if (!navigator.geolocation) { showLocationUnavailable('Location is not supported.'); return; } navigator.geolocation.getCurrentPosition( (position) => { const {coords} = position; showPosition({ latitude: coords.latitude, longitude: coords.longitude, accuracy: coords.accuracy, altitude: coords.altitude, altitudeAccuracy: coords.altitudeAccuracy, heading: coords.heading, speed: coords.speed, timestamp: position.timestamp, }); }, (error) => { showLocationUnavailable(locationErrorMessage(error)); }, {timeout: 15000}, ); } ``` | Error | Meaning | | --- | --- | | `PERMISSION_DENIED` | The user denied location access. | | `POSITION_UNAVAILABLE` | The runtime could not obtain a position. This can have several causes and does not prove that the phone is offline. | | `TIMEOUT` | A position was not available within the effective timeout. | Use `coords.accuracy` rather than promising a fixed accuracy range. Altitude, altitude accuracy, heading, and speed can be `null`. ### Watch position changes Use `watchPosition()` when a feature needs ongoing updates, and clear the watch as soon as it is no longer needed: ```javascript let watchId = null; function startLocationWatch() { watchId = navigator.geolocation.watchPosition( (position) => updateMap(position.coords), (error) => showLocationUnavailable(locationErrorMessage(error)), {enableHighAccuracy: true}, ); } function stopLocationWatch() { if (watchId !== null) { navigator.geolocation.clearWatch(watchId); watchId = null; } } ``` A watch is active only while the page and runtime are active; it is not a background-execution contract. Stop it during route teardown or when the feature is hidden. ### Position options on the device On Meta Ray-Ban Display glasses, the supported option set is narrower than in a general desktop browser: | Method | Supported options | | --- | --- | | `getCurrentPosition()` | `timeout`. The host applies its own default and upper bound. `enableHighAccuracy` and `maximumAge` are not used for one-shot requests. | | `watchPosition()` | `enableHighAccuracy` is a request for the best available watch source and can fall back. `timeout` and `maximumAge` are not used for watches. | Base feature behavior on reported accuracy rather than a particular source or precision tier. ## Privacy and resilience Motion and location can reveal sensitive user behavior. Explain why access is needed before requesting it. Collect for the shortest practical duration, keep logging and persistence to the minimum the feature needs, and transmit readings only when the user expects it. Provide a meaningful experience when permission is denied, the paired phone is unavailable, sensor delivery stops, or a location request fails. Test first use, denial, later grant, revocation, stale readings, route cleanup, and recovery on the device. ### Text to speech ## Text to speech Meta Ray-Ban Display glasses provide a supported subset of the Web Speech synthesis API through `window.speechSynthesis` and `SpeechSynthesisUtterance`. Speech plays through the device speaker without a web permission prompt. Feature-detect both objects and retain equivalent visible content: ```javascript function speakResult(text) { if ( !('speechSynthesis' in window) || !('SpeechSynthesisUtterance' in window) ) { showSpeechUnavailable(); return; } const utterance = new SpeechSynthesisUtterance(text); utterance.lang = 'en-US'; utterance.onerror = (event) => { console.warn('Speech failed:', event.error); showSpeechUnavailable(); }; window.speechSynthesis.speak(utterance); } ``` Use speech for concise results, confirmations, or guidance, with equivalent visible content for important information. ## Voice and utterance properties The runtime exposes one static voice entry: | Property | Value | | --- | --- | | `name` | `Default` | | `voiceURI` | `default` | | `lang` | `en-US` | | `localService` | `true` | | `default` | `true` | `speechSynthesis.getVoices()` returns the list synchronously. The runtime does not provide alternate selectable voices or fire `voiceschanged`, so a voice picker has no effect. | Property | Behavior | | --- | --- | | `text` | The string to speak. Keep it concise. | | `lang` | Forwarded to the native speech engine. Available locales can vary; test every language the app offers and retain visible fallback text. | | `rate` | Clamped to the Web Speech range from `0.1` through `10`. | | `pitch` | Clamped from `0.1` through `2`. | | `voice` | Accepted by the web object, but the runtime uses its default voice. | | `volume` | Accepted but ignored. Speech uses the user's system volume. | ## Events Attach event handlers to each utterance: ```javascript const utterance = new SpeechSynthesisUtterance('Directions ready'); utterance.onstart = () => showSpeakingIndicator(); utterance.onend = () => hideSpeakingIndicator(); utterance.onerror = (event) => { hideSpeakingIndicator(); showSpeechUnavailable(); console.warn('Speech failed:', event.error); }; speechSynthesis.speak(utterance); ``` The runtime supports utterance `start`, `end`, `error`, `pause`, and `resume` events. It does not provide word or markup timing, so `boundary` and `mark` do not fire. Use utterance-level events for status instead of word-level highlighting or synchronized captions. Listeners attached to the `speechSynthesis` object itself do not fire. Attach handlers to the utterance instead. ## Queue, pause, resume, and cancellation Calling `speak()` queues an utterance. Decide whether a new message should wait or replace current speech, and reserve `cancel()` for the replacement case. The native engine cannot resume in the middle of an utterance. `pause()` stops the active speech, and `resume()` speaks that utterance again from the beginning before continuing the queue. Prefer short utterances over long passages that need scrubbing. `cancel()` clears the queue and stops the active utterance. Cancelled utterances receive `end`, not `error`, so track whether a request was superseded: ```javascript let speechGeneration = 0; function replaceSpeech(text) { const generation = ++speechGeneration; speechSynthesis.cancel(); const utterance = new SpeechSynthesisUtterance(text); utterance.onend = () => { if (generation === speechGeneration) { hideSpeakingIndicator(); } }; utterance.onerror = () => { if (generation === speechGeneration) { hideSpeakingIndicator(); showSpeechUnavailable(); } }; showSpeakingIndicator(); speechSynthesis.speak(utterance); } ``` Read `speechSynthesis.speaking`, `.pending`, and `.paused` when the interface needs to reflect queue state. Cancel speech when its page is left or its content is no longer relevant. ## Errors and fallback Handle `error` and retain the visible result. The runtime reports these codes directly: | Code | Meaning | | --- | --- | | `not-allowed` | The call was not authorized by the page-scoped bridge. | | `invalid-argument` | The utterance could not be parsed or used. | | `synthesis-failed` | The native engine could not produce speech. | Future runtime versions can surface other Web Speech error strings. Treat an unknown error as a normal failure and retain the visible result rather than making behavior depend on an exhaustive error list. ## Responsible use - Speak short, self-contained content. - Show the same important information on screen. - Let the user enable, disable, replay, or stop optional speech. - Start optional speech from a clear user context rather than when a page opens. - Remember that nearby people can hear the speaker. Keep secrets and sensitive personal information out of speech unless a clear user action and expectation make it appropriate. - Test queueing, replacement, cancellation, route exit, errors, and every supported language on the device. ### Storage and offline support ## Web Storage Meta Ray-Ban Display glasses support `localStorage` and `sessionStorage` for small string values: - `localStorage` can persist across page sessions. - `sessionStorage` belongs to the current browsing-context session. Both APIs are synchronous. Keep values small, catch write failures, and treat stored values as recoverable rather than permanent. Browser data clearing, application removal, device reset, or storage pressure can remove data. ### Save and read values ```javascript function savePreferences(preferences) { try { localStorage.setItem('example.preferences', JSON.stringify(preferences)); return true; } catch (error) { console.warn('Could not save preferences:', error); return false; } } function loadPreferences() { const stored = localStorage.getItem('example.preferences'); if (stored === null) { return null; } try { return JSON.parse(stored); } catch (error) { localStorage.removeItem('example.preferences'); return null; } } ``` Use `removeItem()` to remove one key. Reserve `clear()` for an origin whose complete Web Storage namespace the app intentionally owns, because it deletes all values on that origin. Apps hosted on the same origin can share site data, so namespace keys. Use `sessionStorage` for temporary state that belongs only to the current page session: ```javascript sessionStorage.setItem('example.onboardingSeen', 'true'); ``` Handle `QuotaExceededError` instead of relying on a fixed quota. Use Web Storage for preferences and small application state. Choose a more appropriate storage and security design for images, large datasets, secrets, bearer tokens, or sensitive personal information. ## Offline support A Service Worker and Cache Storage can serve an installed app shell during an offline cold start. The first visit still requires a network connection, and a new worker may not control the page until a later navigation or reload. Service Workers require a secure context such as `https://`. They are event-driven and can be stopped between events rather than running continuously in the background. ### Register the worker Use a path relative to the app instead of assuming it is hosted at the origin root: ```javascript if ('serviceWorker' in navigator) { navigator.serviceWorker.register('./sw.js').catch((error) => { console.warn('Offline support is unavailable:', error); }); } ``` Registration success does not guarantee that installation, activation, or every cache write will succeed. Keep the app usable online when registration fails. ### Cache an app shell Use an app-specific cache prefix. During cleanup, delete only previous versions owned by this app: ```javascript const CACHE_PREFIX = 'example-app-'; const CACHE_NAME = `${CACHE_PREFIX}v1`; const APP_SHELL = [ new URL('./', self.registration.scope).href, new URL('./app.js', self.registration.scope).href, new URL('./styles.css', self.registration.scope).href, ]; self.addEventListener('install', (event) => { event.waitUntil( caches.open(CACHE_NAME).then((cache) => cache.addAll(APP_SHELL)), ); }); self.addEventListener('activate', (event) => { event.waitUntil( caches.keys().then((names) => Promise.all( names .filter( (name) => name.startsWith(CACHE_PREFIX) && name !== CACHE_NAME, ) .map((name) => caches.delete(name)), ), ), ); }); ``` `cache.addAll()` fails the installation if any requested resource fails. Keep the precache list small and verify every URL. Limit cleanup to the app's cache prefix because another application or library on the same origin may own other caches. ### Serve cached content Intercept only the requests the worker owns. This example uses cached responses first, goes to the network for a miss, and falls back to the cached app shell for a failed same-origin navigation: ```javascript self.addEventListener('fetch', (event) => { const request = event.request; const url = new URL(request.url); if (request.method !== 'GET' || url.origin !== self.location.origin) { return; } event.respondWith((async () => { const cache = await caches.open(CACHE_NAME); const cached = await cache.match(request); if (cached) { return cached; } try { return await fetch(request); } catch (error) { if (request.mode === 'navigate') { const shell = await cache.match(APP_SHELL[0]); if (shell) { return shell; } } throw error; } })()); }); ``` Choose a strategy per resource type. Cache-first works well for versioned static assets. Network-first or stale-while-revalidate can be more appropriate for data that changes. Cache authenticated or sensitive responses only when the app has a deliberate security and invalidation design. ## Updates and eviction Changing the Service Worker file triggers an update check. A newly installed worker commonly waits until pages controlled by the old worker close. Use versioned cache names, keep old and new application code compatible during the transition, and use immediate activation only when the app has a safe handoff plan. Cached data can be evicted or cleared. Treat the cache as a performance and offline aid, not the only copy of user-created data. Remove obsolete caches by prefix and keep the app functional when a cache entry is missing. ## Connectivity and live data `navigator.onLine` and the `online` and `offline` events are connectivity hints. Use actual request success and failure as the source of truth because these hints do not prove that the app's service or a particular request is reachable. Check HTTP status as well as network exceptions: ```javascript async function fetchData(url) { try { const response = await fetch(url); if (!response.ok) { throw new Error(`Request failed with ${response.status}`); } return await response.json(); } catch (error) { showOfflineMessage('This content is not available right now.'); return null; } } ``` Show an offline or stale state only when it helps the user understand what can be done. Show a connection indicator only where it helps explain available behavior. ## Test storage and offline behavior Before publishing, verify: - first installation while online; - a controlled reload after the worker takes control; - an offline cold start with the cached shell; - failure when a required precache URL is missing; - a waiting-worker update and the next activation; - cleanup of only the app's obsolete cache versions; - missing or evicted Web Storage and Cache Storage data; - `QuotaExceededError` and malformed stored values; - failed network requests and non-successful HTTP responses; - recovery when connectivity returns; and - that clearing or removing the app does not leave correctness dependent on retained local data. ### Overview ## Overview UI Toolkit (the toolkit) is a React component library for Meta Ray-Ban Display glasses. It provides finished components, semantic tokens, container materials, motion, and focus-navigation infrastructure. Applications supply content and callbacks while the toolkit keeps shape, material, focus, and typography consistent. Start with [what the toolkit provides](/docs/develop/webapps/design/getting-started/what-is-the-toolkit/), then follow [Installation](/docs/develop/webapps/design/getting-started/installation/) to install the packages from npm and render a first screen. If you use Muse Code, Codex, or Claude Code, install the optional [coding-agent skills](/docs/develop/webapps/design/getting-started/coding-agent-skills/) to give the agent toolkit-specific component, theming, navigation, and validation guidance. Use the **UI Toolkit for Meta Ray-Ban Display glasses** section in the sidebar to navigate the getting-started guides, foundations, icons, and component documentation. For the latest published documentation, visit the [Wearables Developer Center](https://wearables.developer.meta.com/). ## Design for Meta Ray-Ban Display Glasses Meta Ray-Ban Display draws content as light over the wearer's view and uses directional controls instead of a touch surface. Three characteristics shape its interfaces: - **Focus is the cursor.** Directional input moves focus among interactive elements. Use toolkit components so focus behavior remains coordinated. - **Light is additive.** Every displayed element adds brightness over the wearer's view. Use the toolkit's semantic colors and materials to preserve legibility as the surroundings change. - **Space is negotiated, not fixed.** Components size themselves from content, and layouts respond to the available viewport. Avoid assumptions based on a fixed device size. Read [What is the toolkit?](/docs/develop/webapps/design/getting-started/what-is-the-toolkit/) for the runtime model, library organization, and application responsibilities. ## Explore the documentation - **Get started:** Install the npm packages, render the application inside [`App`](/docs/develop/webapps/design/components/foundation/app/), learn [semantic tokens](/docs/develop/webapps/design/getting-started/semantic-tokens/), and [theme the application](/docs/develop/webapps/design/getting-started/theming/). - **Build complete screens:** Follow the guides for [application structure](/docs/develop/webapps/design/guides/application-structure/), [screen layout](/docs/develop/webapps/design/guides/screen-layout/), [navigation and focus](/docs/develop/webapps/design/guides/navigation-and-focus/), and [application patterns](/docs/develop/webapps/design/guides/application-patterns/). - **Choose components:** Use the [component selection guide](/docs/develop/webapps/design/components/overview/), then browse the [Meta Ray-Ban Display glasses components](/docs/develop/webapps/design/components/mrbd/overview/) and [Foundation](/docs/develop/webapps/design/foundations/overview/). - **Use icons:** Install and browse the optional icon collection, or learn how toolkit icon slots handle custom assets, in [Icons](/docs/develop/webapps/design/foundations/utilities/icons/). - **Customize and validate:** See [theming and customization](/docs/develop/webapps/design/guides/theming-and-customization/), [composition](/docs/develop/webapps/design/guides/customization-and-composition/), and [accessibility, performance, and validation](/docs/develop/webapps/design/guides/accessibility-performance-and-validation/). [Installation](/docs/develop/webapps/design/getting-started/installation/) documents package entry points and compatibility boundaries. The package's exported TypeScript declarations are the authoritative prop-level reference. The toolkit source is available in the [GitHub repository](https://github.com/facebook/meta-ray-ban-display-ui-toolkit-web). ### What is the toolkit? Use UI Toolkit for Meta Ray-Ban Display glasses (the toolkit) to build React interfaces for the device. The `@wearables-ui-toolkit/mrbd` package provides the shared visual language, interaction behavior, directional navigation, materials, motion, accessibility semantics, and layout infrastructure. Applications can then focus on their content and product behavior. The toolkit targets React 19 and builds as ECMAScript modules with TypeScript declarations. `App` loads the packaged component styles and design tokens. ## The target runtime Meta Ray-Ban Display glasses use an **additive-light display**, which draws pixels as light over the wearer's view of the world. An unlit pixel is absent rather than an opaque background. The toolkit's semantic tokens and material system provide the contrast that content needs without relying on a custom-painted page surface. Input is **focus-based and directional**. The runtime has no pointer or touch surface. One visible focus moves between interactive elements in four directions, and the user confirms the focused element. Every screen must remain reachable and legible through directional navigation. Toolkit components own their focused, pressed, and disabled appearances so interaction states remain consistent and accessible. Layout responds to **available space**. Prefer dimensions derived from the viewport, content, intrinsic component sizes, and semantic tokens. Avoid hardcoded device resolutions and viewport-specific branches so the same layout can adapt safely. The same toolkit components run on Meta Ray-Ban Display glasses and in desktop Chrome during local development. ## How the library is organized The toolkit is organized into a main package for Meta Ray-Ban Display glasses, a lower-level foundation package, and an optional icons package. The main package re-exports the foundation interfaces from its root entry point for convenient application imports. | Layer | What it contains | Documentation | | --- | --- | --- | | **Foundation** | [`@wearables-ui-toolkit/foundation`](https://www.npmjs.com/package/@wearables-ui-toolkit/foundation) provides lower-level primitives and infrastructure: the base `App`, theme tokens and text appearances, material and shape interfaces, motion constants and hooks, focus navigation, page transitions and route preloading, portals, scrolling and paging containers, `TextView`, `Container`, and `Surface`. | [Foundation](/docs/develop/webapps/design/foundations/overview/) | | **Meta Ray-Ban Display glasses components** | [`@wearables-ui-toolkit/mrbd`](https://www.npmjs.com/package/@wearables-ui-toolkit/mrbd) provides the app shell and complete component catalog built on Foundation: `Page`, `Button`, `ListItem`, `Modal`, `Toast`, `ContextMenu`, `SubNavigation`, indicators, controls, and the rest. | [Meta Ray-Ban Display glasses](/docs/develop/webapps/design/components/mrbd/overview/) | | **Icons** | The optional [`@wearables-ui-toolkit/icons`](https://www.npmjs.com/package/@wearables-ui-toolkit/icons) package provides the public Scalable Vector Graphics icon collection accepted by toolkit icon slots. | [Icons](/docs/develop/webapps/design/foundations/utilities/icons/) | Prefer a Meta Ray-Ban Display glasses component when one matches the intended interaction and information hierarchy. Use foundation interfaces for custom content when the component catalog does not express the need. Every toolkit surface must render inside `App`. `App` installs the fonts, focus navigation context, floating portal root for overlays, and toast host required by the component tree: ```tsx import { App, Button, Page } from '@wearables-ui-toolkit/mrbd'; export function Root() { return (