diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index 1dca78b..d78de3e 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -21,6 +21,7 @@ "./camera/camerax", "./cars/android-for-cars-parked-apps", "./device-ai/appfunctions", + "./device-ai/litert/litert-lm", "./device-ai/litert/litert-runtime", "./device-ai/ml-kit-genai-prompt-api", "./devtools/android-cli", @@ -33,6 +34,7 @@ "./media/media3-cast-integration", "./navigation/navigation-3", "./navigation/navigation-event", + "./performance/anr-audit-and-fix", "./performance/r8-analyzer", "./play/engage-sdk-integration", "./play/play-billing-library-version-upgrade", @@ -46,6 +48,7 @@ "./tv/leanback-to-compose-tv-migration", "./wear/wear-compose-m3", "./wear/wear-widgets", - "./xr/display-glasses-with-jetpack-compose-glimmer" + "./xr/display-glasses-with-jetpack-compose-glimmer", + "./xr/projected-hardware" ] } diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.json index 4d3656b..c399b0b 100644 --- a/.codex-plugin/plugin.json +++ b/.codex-plugin/plugin.json @@ -19,6 +19,7 @@ "./camera/camerax", "./cars/android-for-cars-parked-apps", "./device-ai/appfunctions", + "./device-ai/litert/litert-lm", "./device-ai/litert/litert-runtime", "./device-ai/ml-kit-genai-prompt-api", "./devtools/android-cli", @@ -31,6 +32,7 @@ "./media/media3-cast-integration", "./navigation/navigation-3", "./navigation/navigation-event", + "./performance/anr-audit-and-fix", "./performance/r8-analyzer", "./play/engage-sdk-integration", "./play/play-billing-library-version-upgrade", @@ -44,7 +46,8 @@ "./tv/leanback-to-compose-tv-migration", "./wear/wear-compose-m3", "./wear/wear-widgets", - "./xr/display-glasses-with-jetpack-compose-glimmer" + "./xr/display-glasses-with-jetpack-compose-glimmer", + "./xr/projected-hardware" ], "interface": { "displayName": "Android Skills", diff --git a/device-ai/litert/litert-lm/SKILL.md b/device-ai/litert/litert-lm/SKILL.md new file mode 100644 index 0000000..472fc25 --- /dev/null +++ b/device-ai/litert/litert-lm/SKILL.md @@ -0,0 +1,372 @@ +--- +name: litert-lm +description: Builds an Android app that runs an open text LLM on the device, on the + CPU or the GPU, with the LiteRT-LM Kotlin API (com.google.ai.edge.litertlm:litertlm-android). + Use this skill when creating a chat, summarization, or extraction feature with a + .litertlm model from Hugging Face litert-community (Gemma, Qwen, Llama, Phi), including + the Gradle dependency, the GPU manifest entries, getting the model file onto the + device, Engine initialization off the main thread, a streamed multi-turn Conversation + with sendMessageAsync, and the AndroidViewModel and Compose screen. Don't use for + running .tflite vision, audio, or embedding models with the CompiledModel API, for + image or audio input, tool calling, or NPU backends, or for converting models. +license: Complete terms in LICENSE.txt +metadata: + author: Google LLC + last-updated: '2026-10-20' + keywords: + - LiteRT-LM + - litertlm + - Gemma + - on-device LLM + - Android app + - GPU +--- + +This skill provides step-by-step guidance for building an Android app that runs +an open text LLM through the LiteRT-LM Kotlin API +(`com.google.ai.edge.litertlm`) on the CPU or the GPU. The model is one +`.litertlm` file. Images, audio, tool calling, and NPU backends aren't covered. +For vision and audio models with `.tflite` files, suggest the `litert-runtime` +skill. + +When a LiteRT-LM class or method that this skill uses doesn't compile, you must +read the [LiteRT-LM Android guide](https://ai.google.dev/edge/litert-lm/android) and the [LiteRT-LM README](https://github.com/google-ai-edge/LiteRT-LM/blob/main/README.md) before +changing the code. + +## Prerequisites + +- A Kotlin Android project (the Android Studio Empty Activity template is enough). The `litertlm-android` 0.18.0 AAR declares `minSdk` 24. +- The dependency in the app-level `build.gradle.kts` file from Google Maven: `implementation("com.google.ai.edge.litertlm:litertlm-android:0.18.0")`. Google Maven lists the later releases. +- For the GPU backend, two `uses-native-library` entries inside `` in `AndroidManifest.xml`. Add the `INTERNET` permission if the app downloads the model. +- The model file lives on the device's file system (hundreds of MB to a few GB), never in `assets/` or in the APK. + +## Critical mandates + +- **Dependency first** : Before writing code that imports `com.google.ai.edge.litertlm`, add the `litertlm-android` dependency from Step 1 if the project lacks it. Verify with `./gradlew assembleDebug`. +- **Background initialization** : `Engine(config).initialize()` loads the weights and blocks for seconds. Call it on a background thread, never on the main thread. +- **Single owner** : One single-thread executor owns the `Engine` and the `Conversation`. Creation, streaming, cancellation, and closing run on it. +- **Close once** : `Engine` and `Conversation` are `AutoCloseable`. A second `close` on either throws `IllegalStateException`. So does `close` on an engine whose `initialize` threw. Store the engine only after `initialize` returns. +- **Model outside the APK** : Pass an absolute path to `EngineConfig`. Never bundle a `.litertlm` file in `assets/`. + +## Detailed steps + +### Step 1: Configure Gradle and the manifest + +Search the project for `libs.versions.toml`, `app/build.gradle.kts`, and +`AndroidManifest.xml` before editing build files. Configure `minSdk` and add the +LiteRT-LM dependency in `app/build.gradle.kts`: + +
+ + android { + defaultConfig { + minSdk = 24 + } + } + + dependencies { + implementation("com.google.ai.edge.litertlm:litertlm-android:0.18.0") + } + +If the project uses Hilt, check the Dagger version first. `litertlm-android` +0.18.0 carries Kotlin 2.4 metadata, which Hilt 2.59.2 can't read. The build +then fails in `hiltJavaCompileDebug` with `Provided Metadata instance has +version 2.4.0, while maximum supported version is 2.3.0`. Pick one fix: + +- **Path A** : Upgrade the Hilt version (`com.google.dagger`) to 2.60.1 or later. Ask the user before editing `gradle/libs.versions.toml`. +- **Path B** : Keep the Hilt version and add a later version of the metadata reader to the annotation processor path in `app/build.gradle.kts`: + +
+ + dependencies { + ksp("org.jetbrains.kotlin:kotlin-metadata-jvm:2.4.21") + } + +For the GPU backend, add both lines inside `` in +`AndroidManifest.xml`. Add the `INTERNET` permission outside `` +when the app downloads the model: + +
+ + + + + + + + +### Step 2: Pick a model from litert-community + +Choose a `.litertlm` file from [litert-community](https://huggingface.co/litert-community) on Hugging Face. Start with +`litert-community/gemma-4-E2B-it-litert-lm` (the file `gemma-4-E2B-it.litertlm`, +2.6 GB). For a 0.6 GB model, start with `litert-community/Qwen3-0.6B` (the file +`Qwen3-0.6B.litertlm`). Both files run on the CPU and the GPU. A SoC suffix such +as `_Google_Tensor_G5` or `.mediatek.mt6993` marks an NPU build, which this +skill doesn't cover. The model card names the tested backends and the size. + +You must read [get a model](references/get-a-model.md) before you choose a model file other than these +two, before you write the in-app download code, and when no `.litertlm` file +exists for the model. + +### Step 3: Put the model file on the device + +Pick one path: + +- **Path A (local testing with `adb`)** : Run the following commands only when + all three are true: `adb devices` lists a device, the debug build of the app + is installed on it, and the user supplied the local `.litertlm` path. If any + of them is false, don't run the commands. Report them to the user as a + post-implementation step instead. Replace `` with the user's local path + and `` with the app's application ID: + + 1. `adb push /data/local/tmp/model.litertlm` + 2. `adb shell run-as cp /data/local/tmp/model.litertlm files/` +- **Path B (in-app download)** : Write the download code in the app. Use the + app's HTTP client to write into `filesDir`, or `DownloadManager` to write + into `getExternalFilesDir(null)`. `DownloadManager` can't write to internal + storage. Show the download progress and check the file size against the model + card before loading. Never download a model file in the agent's own session. + +- **Path C ([Play for On-Device AI](references/android/google/play/on-device-ai.md))** : Write the code that resolves the + path of a fast-follow or on-demand AI pack with the AI Delivery library + (`com.google.android.play:ai-delivery`): + `AiPackManagerFactory.getInstance(context).getPackLocation(name)?.assetsPath()`. + The call returns `null` until the pack has finished downloading, so show a + download state in that case. Creating the AI pack module and uploading the app + to Play Console are post-implementation steps for the user. + +On every path, pass the absolute path of the file to the engine. + +### Step 4: Initialize the engine off the main thread + +Create the engine with `EngineConfig(modelPath = ..., backend = ..., +cacheDir = ...)` and call `initialize` on a background thread: + +- `Backend.CPU()` is the default of `EngineConfig` and needs nothing from the device. +- `Backend.GPU()` needs the two manifest lines from Step 1. Without them, the engine initializes, and the first reply ends in the error `Can not find + OpenCL library on this device`. +- `cacheDir` speeds up the second load. + +### Step 5: Implement the conversation ViewModel + +Drive the engine and the conversation from a single-thread executor inside an +`AndroidViewModel`, stream the reply, and expose immutable state to the UI: + +
+ + import android.app.Application + import androidx.lifecycle.AndroidViewModel + import com.google.ai.edge.litertlm.Backend + import com.google.ai.edge.litertlm.Contents + import com.google.ai.edge.litertlm.Conversation + import com.google.ai.edge.litertlm.ConversationConfig + import com.google.ai.edge.litertlm.Engine + import com.google.ai.edge.litertlm.EngineConfig + import java.util.concurrent.Executors + import kotlin.coroutines.cancellation.CancellationException + import kotlinx.coroutines.CoroutineScope + import kotlinx.coroutines.SupervisorJob + import kotlinx.coroutines.asCoroutineDispatcher + import kotlinx.coroutines.cancel + import kotlinx.coroutines.delay + import kotlinx.coroutines.flow.MutableStateFlow + import kotlinx.coroutines.flow.StateFlow + import kotlinx.coroutines.flow.asStateFlow + import kotlinx.coroutines.flow.update + import kotlinx.coroutines.launch + import kotlinx.coroutines.sync.Mutex + import kotlinx.coroutines.sync.withLock + + data class ChatState( + val ready: Boolean = false, + val busy: Boolean = false, + val reply: String = "", + val error: String? = null + ) + + class ChatViewModel(app: Application) : AndroidViewModel(app) { + private val executor = Executors.newSingleThreadExecutor() + private val scope = CoroutineScope( + SupervisorJob() + executor.asCoroutineDispatcher() + ) + private val mutex = Mutex() + private var engine: Engine? = null + private var conversation: Conversation? = null + private val _state = MutableStateFlow(ChatState()) + val state: StateFlow = _state.asStateFlow() + + fun load(modelPath: String, backend: Backend = Backend.CPU()) { + scope.launch { + mutex.withLock { + if (engine?.engineConfig?.modelPath == modelPath) return@launch + release() + try { + val config = EngineConfig( + modelPath = modelPath, + backend = backend, + cacheDir = getApplication().cacheDir.path + ) + engine = Engine(config).also { it.initialize() } + conversation = checkNotNull(engine).createConversation( + ConversationConfig( + systemInstruction = + Contents.of("You are a helpful assistant.") + ) + ) + _state.value = ChatState(ready = true) + } catch (e: Exception) { + release() + _state.value = ChatState(error = e.message) + } + } + } + } + + fun send(text: String) { + scope.launch { + if (!_state.value.ready || _state.value.busy) return@launch + _state.update { it.copy(busy = true, reply = "", error = null) } + try { + val replies = checkNotNull(conversation).sendMessageAsync(text) + replies.collect { message -> + _state.update { it.copy(reply = it.reply + message) } + } + } catch (e: CancellationException) { + throw e + } catch (e: Exception) { + _state.update { it.copy(error = e.message) } + } finally { + _state.update { it.copy(busy = false) } + } + } + } + + private suspend fun release() { + _state.update { it.copy(ready = false) } + while (_state.value.busy) { + conversation?.cancelProcess() + delay(100) + } + conversation?.close() + conversation = null + engine?.close() + engine = null + _state.value = ChatState() + } + + override fun onCleared() { + scope.launch { + mutex.withLock { release() } + executor.shutdown() + }.invokeOnCompletion { scope.cancel() } + } + } + +`sendMessageAsync(text)` returns a `Flow` of chunks, and `toString` +gives a chunk's text. `sendMessage(text)` blocks and returns the whole reply. +The conversation keeps its history, so the next `send` continues the chat. +`busy` keeps the button disabled until the flow completes. + +To set the sampling, pass `SamplerConfig(topK = 10, topP = 0.95, +temperature = 0.8)` in `ConversationConfig`. All three parameters are required, +and `topP` and `temperature` are `Double`. + +`load` with the model path that is already loaded does nothing, whatever +backend it asks for. A second call after a rotation keeps the loaded model, +also after the app has fallen back to another backend. `send` runs only when +the state is `ready` and not `busy`, the same test as the button. A tap that +lands while `release` stops a reply does nothing. A tap queued behind a +`load` reaches the model that `load` loaded when no reply was streaming. + +### Step 6: Wire the Compose screen + +`viewModel` and `collectAsStateWithLifecycle` come from +`androidx.lifecycle:lifecycle-viewmodel-compose` and +`androidx.lifecycle:lifecycle-runtime-compose` (2.10.0): + +
+ + import androidx.compose.foundation.layout.Column + import androidx.compose.foundation.rememberScrollState + import androidx.compose.foundation.verticalScroll + import androidx.compose.material3.Button + import androidx.compose.material3.Text + import androidx.compose.material3.TextField + import androidx.compose.runtime.Composable + import androidx.compose.runtime.LaunchedEffect + import androidx.compose.runtime.getValue + import androidx.compose.runtime.mutableStateOf + import androidx.compose.runtime.remember + import androidx.compose.runtime.setValue + import androidx.compose.ui.Modifier + import androidx.lifecycle.compose.collectAsStateWithLifecycle + import androidx.lifecycle.viewmodel.compose.viewModel + import com.google.ai.edge.litertlm.Backend + + @Composable + fun ChatScreen(modelPath: String, viewModel: ChatViewModel = viewModel()) { + val state by viewModel.state.collectAsStateWithLifecycle() + var input by remember { mutableStateOf("") } + LaunchedEffect(modelPath) { viewModel.load(modelPath, Backend.GPU()) } + Column { + Text( + state.reply, + Modifier.weight(1f).verticalScroll(rememberScrollState()) + ) + state.error?.let { Text(it) } + TextField(value = input, onValueChange = { input = it }) + Button( + onClick = { viewModel.send(input); input = "" }, + enabled = state.ready && !state.busy + ) { + Text("Send") + } + } + } + +### Step 7: Manage the engine lifecycle + +- Keep the engine in the `ViewModel` for as long as its screen lives. When several screens share the engine, keep it in an application-scoped holder. +- The `ViewModel` keeps its own thread and scope. `viewModelScope` is cancelled before `onCleared` runs, so a close launched there wouldn't run. `onCleared` cancels `scope` once the close is done. +- `release` turns `ready` off and calls `cancelProcess` until the streaming reply's flow has ended. The flow ends at the latest when the reply ends on its own. `release` then closes the conversation and the engine, each once. +- `onCleared` and a `load` that replaces the model both call `release` under the `Mutex`, one at a time, because `release` suspends while a reply stops. +- To start a fresh chat, close the conversation and create a new one with `createConversation`. Keep the engine loaded. + +## Post-implementation steps for the user + +Report the steps that apply to the user after the code compiles: + +- When Path A in Step 3 didn't run, copy the model to the device with the two `adb` commands from Path A. +- When the app uses Path C in Step 3, create the AI pack module that holds the `.litertlm` file and upload the app to Play Console. +- When no `.litertlm` file exists for the model, convert the model on a workstation. This skill doesn't convert models. +- When the user asks for a complete reference app, point them to [Google AI Edge Gallery](https://github.com/google-ai-edge/gallery/blob/main/README.md). + +## Troubleshooting + +| Symptom | Cause | Fix | +|---|---|---| +| First reply is slow. | `initialize` ran on first use. | Load at app start and set `cacheDir`. | +| First GPU reply ends in `Can not find OpenCL library on this device`. | The manifest lacks the `uses-native-library` entries. | Add both entries from Step 1 inside ``. | +| `close` throws `IllegalStateException`. | The engine or conversation was closed twice, or `initialize` threw. | Store the engine only after `initialize` returns, and set each field to `null` after `close`. | +| Build fails with `[Hilt] Provided Metadata instance has version 2.4.0`. | Hilt 2.59.2 can't read the Kotlin 2.4 metadata in `litertlm-android` 0.18.0. | Follow Path A or Path B in Step 1. | + +## Checklist + +- \[ \] Is `minSdk` set to `24` or higher and is `com.google.ai.edge.litertlm:litertlm-android:0.18.0` added in `app/build.gradle.kts`? +- \[ \] For the GPU backend, does `AndroidManifest.xml` declare `libvndksupport.so` and `libOpenCL.so` with `android:required="false"` inside ``? +- \[ \] Is the `.litertlm` file loaded from an absolute path on the device's file system, never from `assets/`? +- \[ \] Do `initialize`, `sendMessageAsync`, `cancelProcess`, and `close` all run on the same single-thread background executor? +- \[ \] Is the conversation closed before the engine, each exactly once? +- \[ \] Does the project compile with `./gradlew assembleDebug`? + +## Antipatterns + +- **Never** call `Engine.initialize` on the Android main thread. +- **Never** place a `.litertlm` file in `assets/` or ship it inside the APK. +- **Never** call `close` on an engine whose `initialize` threw. +- **Never** close the engine while a reply is streaming; call `cancelProcess` until the flow ends first. +- **Never** close `Engine` or `Conversation` from `viewModelScope` in `onCleared`; that scope is already cancelled. diff --git a/device-ai/litert/litert-lm/references/android/google/play/on-device-ai.md b/device-ai/litert/litert-lm/references/android/google/play/on-device-ai.md new file mode 100644 index 0000000..f781d3b --- /dev/null +++ b/device-ai/litert/litert-lm/references/android/google/play/on-device-ai.md @@ -0,0 +1,818 @@ +## Introduction + +Play for On-device AI brings the benefits of +[Android App Bundles](https://developer.android.com/guide/app-bundle) and Google Play delivery to custom ML +model distribution so you can improve model performance with less device +ecosystem complexity at no additional cost. It lets you publish a single +artifact to Play containing your code, assets, and ML models and to choose +from a number of delivery modes and targeting options. + +### Benefits + +- Upload a single publishing artifact to Google Play and delegate hosting, delivery, updates, and targeting to Play at no additional cost. +- Deliver your ML models at install-time, fast-follow, or on-demand. + - Install-time delivery can guarantee that a very large model is present when your app is opened. Your model will be installed as an APK. + - Fast-follow delivery occurs automatically in the background after your app has been installed. Users may open your app before your model has been fully downloaded. Your model will be downloaded to your app's internal storage space. + - On-demand delivery lets you request the model at runtime, which is useful if the model is only required for certain user-flows. Your model will be downloaded to your app's internal storage space. +- Deliver variants of your ML models that are targeted to specific devices based on device model, system properties, or RAM. +- Keep app updates small and optimized with Play's automatic patching, which means only the differences in files need to be downloaded. + +### Considerations + +- By using Play for On-device AI you agree to the terms in the [Google Play Developer Distribution Agreement](https://play.google.com/about/developer-distribution-agreement.html) and the [Play Core Software Development Kit Terms of Service](https://developer.android.com/guide/playcore#license). +- Models downloaded by Play for On-device AI should only be used by your apps. Models shouldn't be offered to other apps. +- Individual AI packs can be up to 1.5GB, based on their compressed download sizes. The maximum cumulative app size of any version of your app generated from your app bundle is 4GB. +- Apps over 1GB in size must set min SDK Level to 21 or higher. + +### How to use Play for On-device AI + +Play for On-device AI uses AI packs. You package custom models that are ready +for distribution in AI packs in your app bundle. You can choose whether the AI +pack should be delivered at install-time, fast-follow, or on-demand. + +By packaging AI packs with your app bundle, you can use all of Play's existing +testing and release tools, such as test tracks and staged rollouts to manage +your app's distribution with your custom models. + +AI packs are updated together with the app binary. If your new app release +doesn't make changes to an AI pack, then Play's automatic patching process will +ensure the user doesn't have to re-download it. Play will just download what's +changed when it updates the app. + +AI packs only contain models. Java/Kotlin and native libraries are not allowed. +If you need to ship libraries or code to run your ML model, move it into the +base module or a [feature module](https://developer.android.com/guide/playcore/feature-delivery). You can +configure your feature module so that it has the same download and targeting +settings as the AI pack. + +> [!NOTE] +> **Note:** AI packs don't contain Java/Kotlin or native libraries. You can ship libraries or code to run your ML models in [feature modules](https://developer.android.com/guide/playcore/feature-delivery). + +### Use LiteRT and MediaPipe with AI packs + +You can use LiteRT and MediaPipe with AI packs. Package your model in an AI pack +and then access it using the instructions for +[install-time packs](https://developer.android.com/google/play/on-device-ai#install-time-delivery) or for +[fast-follow and on-demand packs](https://developer.android.com/google/play/on-device-ai#ff-od-delivery). + +Further reading: + +- [Getting started with LiteRT](https://ai.google.dev/edge/litert/android) + - The [sample app](https://developer.android.com/google/play/on-device-ai#example-app) shows how you can package a LiteRT model in an AI pack and load it at runtime. + - There are many [pretrained LiteRT models](https://ai.google.dev/edge/litert/models/trained) that you can use in AI packs to get started. +- [Getting started with MediaPipe](https://ai.google.dev/edge/mediapipe/framework/getting_started/android) + - For fast-follow and on-demand packs, you can use [AssetCache.java](https://github.com/google-ai-edge/mediapipe/blob/master/mediapipe/java/com/google/mediapipe/framework/AssetCache.java) to load your assets (eg. `.binarypb` files) by their file paths. + - For install-time packs, you can use [AndroidAssetUtil.java](https://github.com/google-ai-edge/mediapipe/blob/master/mediapipe/java/com/google/mediapipe/framework/AndroidAssetUtil.java#L56). + +## Get started with AI packs + +At a high level, here's how you can start using Play for On-device AI: + +1. Package your models into AI packs into your Android App Bundle and specify how the AI packs should be delivered. +2. \[Optional\] If you want to deliver different models to different devices, you can [configure device targeting](https://developer.android.com/google/play/on-device-ai#device-targeting-configuration) for your AI packs. For example, you could deliver AI pack A to a specific device model, AI pack B to devices with at least 6GB of RAM, and all other devices could receive no model. +3. \[Optional\] If you're using on-demand or fast-follow delivery, integrate the Play AI Delivery Library into your app to download your AI packs as needed. +4. Test and release your app bundle to Google Play. + +### Check Android Gradle Plugin version + +To use AI packs, ensure that your Android Gradle Plugin (AGP) version is at +least 8.8. This version is packaged with Android Studio Ladybug 2. + +### Extract your model into an AI pack + +Android Studio is not required for the following steps. + +1. In the top-level directory of your project, create a directory for the AI pack. This directory name is used as the AI pack name. AI pack names must start with a letter and can only contain letters, numbers, and underscores. +2. In the AI pack directory, create a `build.gradle` file and add the following + code. Make sure to specify the name of the AI pack and only one delivery + type: + + // In the AI pack's build.gradle file: + plugins { + id 'com.android.ai-pack' + } + + aiPack { + packName = "ai-pack-name" // Directory name for the AI pack + dynamicDelivery { + deliveryType = "[ install-time | fast-follow | on-demand ]" + } + } + +3. In the project's app `build.gradle` file, add the name of every AI pack in + your project as shown below: + + // In the app build.gradle file: + android { + ... + assetPacks = [":ai-pack-name", ":ai-pack2-name"] + } + +4. In the project's `settings.gradle` file, include all AI packs in your + project as shown below: + + // In the settings.gradle file: + include ':app' + include ':ai-pack-name' + include ':ai-pack2-name' + +5. Inside your AI pack, create a `src/main/assets/` directory. + +6. Place your models in the `src/main/assets` directory. You can create + subdirectories in here as well. The directory structure for your app should + now look like the following: + + - `build.gradle` + - `settings.gradle` + - `app/` + - `ai-pack-name/build.gradle` + - `ai-pack-name/src/main/assets/your-model-directories` +7. Add code to load and run your models. How you do this will depend on the + delivery mode of your AI packs. See instructions for + [install-time](https://developer.android.com/google/play/on-device-ai#install-time-delivery) and + [fast-follow/on-demand](https://developer.android.com/google/play/on-device-ai#ff-od-delivery) below. + +8. \[Optional\] [Configure device targeting](https://developer.android.com/google/play/on-device-ai#device-targeting-configuration) + to deliver different models to different devices. + +9. [Build the Android App Bundle with Gradle](https://developer.android.com/studio/build/building-cmdline#build_bundle). + In the generated app bundle, the root-level directory now includes the + following: + + - `ai-pack-name/manifest/AndroidManifest.xml`: Configures the AI pack's identifier and delivery mode + - `ai-pack-name/assets/your-model-directories`: Directory that contains all assets delivered as part of the AI pack + + Gradle generates the manifest for each AI pack and outputs the `assets/` + directory for you. + +### Configure install-time delivery + +AI packs configured as install-time are immediately available at app launch. Use +the Java AssetManager API to access AI packs served in this mode: + +```java +import android.content.res.AssetManager; +... +Context context = createPackageContext("com.example.app", 0); +AssetManager assetManager = context.getAssets(); +InputStream is = assetManager.open("model-name"); +``` + +### Configure fast-follow and on-demand delivery + +To download AI packs with fast-follow or on-demand delivery, use the Play AI +Delivery Library. + +#### Declare dependency on the Play AI Delivery Library + +In your app's `build.gradle` file, declare a dependency on the Play AI Delivery +Library: + + dependencies { + ... + implementation "com.google.android.play:ai-delivery:0.1.1-alpha01" + } + +#### Check status + +Each AI pack is stored in a separate folder in the app's internal storage. Use +the +[`getPackLocation()`](https://developer.android.com/reference/com/google/android/play/core/aipacks/AiPackManager#getPackLocation(java.lang.String)) +method to determine the root folder of an AI pack. This method returns the +following values: + +| Return value | Status | +|---|---| +| A valid [`AiPackLocation`](https://developer.android.com/reference/com/google/android/play/core/aipacks/AiPackLocation) object | AI pack root folder is ready for immediate access at [`assetsPath()`](https://developer.android.com/reference/com/google/android/play/core/aipacks/AiPackLocation#assetsPath) | +| `null` | Unknown AI pack or AI packs are not available | + +> [!NOTE] +> **Note:** Don't rely on cached AI pack locations between app launches. The app should always check for the existence of AI packs at every launch. AI packs may become invalid due to app updates or if the user clears the app data. + +#### Get download information about AI packs + +Use the + +[`getPackStates()`](https://developer.android.com/reference/com/google/android/play/core/aipacks/AiPackManager#getPackStates(java.util.List%3Cjava.lang.String%3E)) +method to determine the size of the download and whether the pack is already +downloading. + +```java +Task getPackStates(List packNames) +``` + +`getPackStates()` is an asynchronous method that returns a `Task`. +The +[`packStates()`](https://developer.android.com/reference/com/google/android/play/core/aipacks/AiPackStates#packStates()) +method of an `AiPackStates` object returns a `Map`. This +map contains the state of each requested AI pack, keyed by its name: + +```java +Map AiPackStates#packStates() +``` + +The final request is shown by the following: + +```java +final String aiPackName = "myAiPackName"; + +aiPackManager + .getPackStates(Collections.singletonList(aiPackName)) + .addOnCompleteListener(new OnCompleteListener() { + @Override + public void onComplete(Task task) { + AiPackStates aiPackStates; + try { + aiPackStates = task.getResult(); + AiPackState aiPackState = + aiPackStates.packStates().get(aiPackName); + } catch (RuntimeExecutionException e) { + Log.d("MainActivity", e.getMessage()); + return; + }); +``` + +The following +[`AiPackState`](https://developer.android.com/reference/com/google/android/play/core/aipacks/AiPackState) +methods provide the size of the AI pack, the downloaded amount so far (if +requested), and the amount already transferred to the app: + +- [`totalBytesToDownload()`](https://developer.android.com/reference/com/google/android/play/core/aipacks/AiPackState#totalBytesToDownload()) +- [`bytesDownloaded()`](https://developer.android.com/reference/com/google/android/play/core/aipacks/AiPackState#bytesDownloaded()) +- [`transferProgressPercentage()`](https://developer.android.com/reference/com/google/android/play/core/aipacks/AiPackState#transferProgressPercentage()) + +To get the status of an AI pack, use the +[`status()`](https://developer.android.com/reference/com/google/android/play/core/aipacks/AiPackState#status()) +method, which returns the status as an integer that corresponds to a constant +field in the +[`AiPackStatus`](https://developer.android.com/reference/com/google/android/play/core/aipacks/model/AiPackStatus) +class. An AI pack that's not installed yet has the status +`AiPackStatus.NOT_INSTALLED`. + +If a request fails, use the +[`errorCode()`](https://developer.android.com/reference/com/google/android/play/core/aipacks/AiPackState#errorCode()) +method, whose return value corresponds to a constant field in the +[`AiPackErrorCode`](https://developer.android.com/reference/com/google/android/play/core/aipacks/model/AiPackErrorCode) +class. + +#### Install + +Use the +[`fetch()`](https://developer.android.com/reference/com/google/android/play/core/aipacks/AiPackManager#fetch(java.util.List%3Cjava.lang.String%3E)) +method to download an AI pack for the first time or call for the update of an AI +pack to complete: + +```java +Task fetch(List packNames) +``` + +This method returns an +[`AiPackStates`](https://developer.android.com/reference/com/google/android/play/core/aipacks/AiPackStates) +object containing a list of packs and their initial download states and sizes. +If an AI pack requested via `fetch()` is already downloading, the download +status is returned and no additional download is started. + +> [!NOTE] +> **Note:** In most cases, you implement a `listener` to track the download and installation process as covered in the next section. + +#### Monitor download states + +You should implement an +[`AiPackStateUpdateListener`](https://developer.android.com/reference/com/google/android/play/core/aipacks/AiPackStateUpdateListener) +to track the installation progress of AI packs. The status updates are broken +down per pack to support tracking the status of individual AI packs. You can +start using available AI packs before all other downloads for your request have +completed. + +```java +void registerListener(AiPackStateUpdateListener listener) +void unregisterListener(AiPackStateUpdateListener listener) +``` + +> [!NOTE] +> **Note:** The Play Store automatically triggers the download of any `fast-follow` packs after the user installs or updates the app. However, these packs may not be ready to use immediately. You must check the status of the `fast-follow` packs at every app launch. If the download is in progress, monitor it with a listener. If the download is cancelled or paused, you can resume it by using the `fetch()` method, as covered in the [Install](https://developer.android.com/google/play/on-device-ai#install) section. + +##### Large downloads + +If the download is larger than 200 MB and the user is not on Wi-Fi, the download +does not start until the user explicitly gives their consent to proceed with the +download using a mobile data connection. Similarly, if the download is large and +the user loses Wi-Fi, the download is paused and explicit consent is required to +proceed using a mobile data connection. A paused pack has state +`WAITING_FOR_WIFI`. To trigger the UI flow to prompt the user for consent, use +the +[`showConfirmationDialog()`](https://developer.android.com/reference/com/google/android/play/core/aipacks/AiPackManager#showConfirmationDialog(androidx.activity.result.ActivityResultLauncher%3Candroidx.activity.result.IntentSenderRequest%3E)) +method. + +Note that if the app does not call this method, the download is paused and will +resume automatically only when the user is back on a Wi-Fi connection. + +##### Required user confirmation + +If a pack has the `REQUIRES_USER_CONFIRMATION` status, the download won't +proceed until the user accepts the dialog that is shown with +[`showConfirmationDialog()`](https://developer.android.com/reference/com/google/android/play/core/aipacks/AiPackManager#showConfirmationDialog(androidx.activity.result.ActivityResultLauncher%3Candroidx.activity.result.IntentSenderRequest%3E)). +This status can occur when the app is not recognized by Play---for example, if the +app was sideloaded. Note that calling +[`showConfirmationDialog()`](https://developer.android.com/reference/com/google/android/play/core/aipacks/AiPackManager#showConfirmationDialog(androidx.activity.result.ActivityResultLauncher%3Candroidx.activity.result.IntentSenderRequest%3E)) +in this case will cause the app to be updated. After the update, you will need +to request the AI packs again. + +The following is an example implementation of a listener: + +```java +AiPackStateUpdateListener aiPackStateUpdateListener = new AiPackStateUpdateListener() { + private final ActivityResultLauncher activityResultLauncher = + registerForActivityResult( + new ActivityResultContracts.StartIntentSenderForResult(), + new ActivityResultCallback() { + @Override + public void onActivityResult(ActivityResult result) { + if (result.getResultCode() == RESULT_OK) { + Log.d(TAG, "Confirmation dialog has been accepted."); + } else if (result.getResultCode() == RESULT_CANCELED) { + Log.d(TAG, "Confirmation dialog has been denied by the user."); + } + } + }); + + @Override + public void onStateUpdate(AiPackState aiPackState) { + switch (aiPackState.status()) { + case AiPackStatus.PENDING: + Log.i(TAG, "Pending"); + break; + + case AiPackStatus.DOWNLOADING: + long downloaded = aiPackState.bytesDownloaded(); + long totalSize = aiPackState.totalBytesToDownload(); + double percent = 100.0 * downloaded / totalSize; + + Log.i(TAG, "PercentDone=" + String.format("%.2f", percent)); + break; + + case AiPackStatus.TRANSFERRING: + // 100% downloaded and assets are being transferred. + // Notify user to wait until transfer is complete. + break; + + case AiPackStatus.COMPLETED: + // AI pack is ready to use. Run the model. + break; + + case AiPackStatus.FAILED: + // Request failed. Notify user. + Log.e(TAG, aiPackState.errorCode()); + break; + + case AiPackStatus.CANCELED: + // Request canceled. Notify user. + break; + + case AiPackStatus.WAITING_FOR_WIFI: + case AiPackStatus.REQUIRES_USER_CONFIRMATION: + if (!confirmationDialogShown) { + aiPackManager.showConfirmationDialog(activityResultLauncher); + confirmationDialogShown = true; + } + break; + + case AiPackStatus.NOT_INSTALLED: + // AI pack is not downloaded yet. + break; + case AiPackStatus.UNKNOWN: + Log.wtf(TAG, "AI pack status unknown") + break; + } + } +} +``` + +Alternatively, you can use the +[`getPackStates()`](https://developer.android.com/reference/com/google/android/play/core/aipacks/AiPackManager#getPackStates()) +method to get the status of current downloads. +[`AiPackStates`](https://developer.android.com/reference/com/google/android/play/core/aipacks/AiPackStates) +contains the download progress, download status, and any failure error codes. + +#### Access AI packs + +You can access an AI pack using file system calls after the download request +reaches the +[`COMPLETED`](https://developer.android.com/reference/com/google/android/play/core/aipacks/model/AiPackStatus#completed()) +state. Use the +[`getPackLocation()`](https://developer.android.com/reference/com/google/android/play/core/aipacks/AiPackManager#getPackLocation(java.lang.String)) +method to get the root folder of the AI pack. + +AI packs are stored in the `assets` directory within the AI pack root directory. +You can get the path to the `assets` directory by using the convenience method +[`assetsPath()`](https://developer.android.com/reference/com/google/android/play/core/aipacks/AiPackLocation#assetsPath()). +Use the following method to get the path to a specific asset: + +```java +private String getAbsoluteAiAssetPath(String aiPack, String relativeAiAssetPath) { + AiPackLocation aiPackPath = aiPackManager.getPackLocation(aiPack); + + if (aiPackPath == null) { + // AI pack is not ready + return null; + } + + String aiAssetsFolderPath = aiPackPath.assetsPath(); + // equivalent to: FilenameUtils.concat(aiPackPath.path(), "assets"); + String aiAssetPath = FilenameUtils.concat(aiAssetsFolderPath, relativeAiAssetPath); + return aiAssetPath; +} +``` + +### Configure device targeting + +You can follow the +[device targeting instructions](https://developer.android.com/google/play/on-device-ai#device-targeting-configuration) to specify +devices or groups of devices that should receive your AI packs. + +### Other Play AI Delivery API methods + +The following are some additional API methods you may want to use in your app. + +#### Cancel request + +Use +[`cancel()`](https://developer.android.com/reference/com/google/android/play/core/aipacks/AiPackManager#cancel(java.util.List%3Cjava.lang.String%3E)) +to cancel an active AI pack request. Note that this request is a best-effort +operation. + +#### Remove an AI pack + +Use +[`removePack()`](https://developer.android.com/reference/com/google/android/play/core/aipacks/AiPackManager#removePack(java.lang.String)) +to schedule the removal of an AI pack. + +#### Get locations of multiple AI packs + +Use +[`getPackLocations()`](https://developer.android.com/reference/com/google/android/play/core/aipacks/AiPackManager#getPackLocations()) +to query the status of multiple AI packs in bulk, which returns a map of AI +packs and their locations. The map returned by `getPackLocations()` contains an +entry for each pack that is currently downloaded and up-to-date. + +## Device targeting + +Device targeting gives you finer control over which parts of your app bundle are +delivered to specific devices. For example, you could ensure that a large model +is only delivered to devices with high RAM or you could deliver different +versions of a model to different devices. + +You can target device properties such as: + +- System on chip +- Device model +- Device RAM +- [System features](https://developer.android.com/guide/topics/manifest/uses-feature-element#features-reference) + +### Overview of required steps + +The following steps are required to enable device targeting: + +1. Define your device groups in an XML file. +2. Specify which parts of your bundle should go to which device groups. +3. \[Optional\] Test your configuration locally. +4. Upload your bundle (containing the XML file) to Google Play. + +### Check Android Gradle Plugin version + +To use device targeting, ensure that your Android Gradle Plugin (AGP) version is +at least 8.10.0. This is packaged with Android Studio (Meerkat 2 and later). +Download the latest [stable version of Android Studio](https://developer.android.com/studio). + +### Enable this feature in Android Gradle Plugin + +Device targeting must be enabled explicitly in your `gradle.properties` file: + + android.experimental.enableDeviceTargetingConfigApi=true + +### Create a device targeting configuration XML file + +The device targeting configuration file is an XML file in which you define your +custom device groups. For example, you could define a device group called +`qti_v79` that contains all devices with the Qualcomm SM8750 system on chip: + + + + + + + + + + + +A **device group** is composed of up to 5 **device selectors**. A device is +included in a device group if it satisfies any of its device selectors. + +A **device selector** can have one or more **device properties**. A device is +selected if it matches all of the selector's device properties. + +If a device matches multiple groups, it will be served the content for the group +that is defined first in the XML file. The order you define groups in the XML +file is your priority order. + +If a device matches no groups, it will receive the default "other" group. This +group is automatically generated and shouldn't be defined explicitly. + +#### Available device properties + +- **device_ram** : Device RAM requirements + - **min_bytes** (*inclusive)*: Minimum required RAM (in bytes) + - **max_bytes** (*exclusive)*: Maximum required RAM (in bytes) +- **included_device_ids** : Device models to be included in this selector **(max of 10000 device_ids per group)** . This property is satisfied if the device matches any device_id in the list. + - **build_brand**: Device manufacturer + - **build_device**: Device model code +- **excluded_device_ids** : Device models to be excluded in this selector **(max of 10000 device_ids per group)** . This property is satisfied if the device matches no device_id in the list. + - **build_brand**: Device manufacturer + - **build_device**: Device model code +- **required_system_features** : Features that a device needs to have to be + included by this selector **(max of 100 features per group)**. A device + needs to have all system features in this list to satisfy this property. + + System + [feature reference](https://developer.android.com/guide/topics/manifest/uses-feature-element#features-reference) + - **name**: A system feature +- **forbidden_system_features** : Features that a device mustn't have to be + included by this selector **(max of 100 features per group)**. If a device + has any of the system features in this list it doesn't satisfy this + property. + + System + [feature reference](https://developer.android.com/guide/topics/manifest/uses-feature-element#features-reference) + - **name**: A system feature +- **system-on-chip**: System on chips to be included in this selector. A + device needs to have any chip in this list to satisfy this property. + + - **manufacturer** : [System on chip manufacturer](https://developer.android.com/reference/android/os/Build#SOC_MANUFACTURER) + - **model** : [System on chip model](https://developer.android.com/reference/android/os/Build#SOC_MODEL) + +> [!TIP] +> **Tip:** Including multiple properties in a single +> selector creates a logical AND, for example: +> +> ```xml +> +> +> +> +> ``` +> +> will create the condition for all devices with \> 7GB of RAM AND it +> is a Pixel 4 - also written as follows: +> +> ![](https://developer.android.com/static/images/google/play/intersect.png) +> +> If you want an OR condition, create separate +> selectors in a single device group, for example: +> +> ```xml +> +> +> +> +> +> ``` +> +> will create the condition for all devices with \> 7GB of RAM OR +> it is a Pixel 4 - also written as follows: +> +> ![](https://developer.android.com/static/images/google/play/union.png) + +Here is an example showing all possible device properties: + + + + + + + + + + + + + + + + + + + + + + + + + + +#### Official device manufacturer and device model codes + +You can find the correct formatting for the device manufacturer and model code +by using the Device Catalog on the Google Play Console, by either: + +- Inspecting individual devices using the Device Catalog, and finding the + manufacturer and model code in the locations as shown in the example below + (For a Google Pixel 4a, the manufacturer is "Google" and the model code is + "sunfish")' + + ![pixel 4a page in the device catalog](https://developer.android.com/static/images/google/play/pixel-4.png) + + ![pixel 4a page in the device catalog](https://developer.android.com/static/images/google/play/pixel-4-2.png) +- Downloading a CSV of supported devices, and using the *Manufacturer* and + *Model Code* for the *build_brand* and *build_device* fields, + respectively. + +#### Include your device targeting configuration file in your app bundle + +Add the following to your main module's `build.gradle` file: + + android { + ... + bundle { + deviceTargetingConfig = file('device_targeting_config.xml') + deviceGroup { + enableSplit = true // split bundle by #group + defaultGroup = "other" // group used for standalone APKs + } + } + ... + } + +`device_targeting_config.xml` is the path of your configuration file relative to +the main module. This ensures that your configuration file is packaged with your +app bundle. + +The `deviceGroup` clause ensures that the APKs generated from your bundle are +split by device groups. + +### Use device targeting for your AI packs + +You can keep size optimized on devices by only delivering your large models to +devices that can run them. + +Subdivide your AI packs by device groups by taking the **existing** AI pack +directories created in the last step, and post-fixing the appropriate folders +(as described below) with #group_myCustomGroup1, #group_myCustomGroup2, etc. +When using the AI packs in your app, you won't need to address folders by +postfix (in other words, the postfix is automatically stripped during the build +process). + +After the previous step, this might look like: + + ... + .../ai-pack-name/src/main/assets/image-classifier#group_myCustomGroup1/ + .../ai-pack-name/src/main/assets/image-classifier#group_myCustomGroup2/ + ... + +In this example, you would reference `ai-pack-name/assets/image-classifier/` +without any postfixes. + +Devices in `myCustomGroup1` will receive all the assets under +`image-classifier#group_myCustomGroup1/`, while devices in `myCustomGroup2` will +receive all the assets under `image-classifier#group_myCustomGroup2/`. + +Devices that don't belong to either `myCustomGroup1` or `myCustomGroup2` will +receive an empty `ai-pack-name` pack. + +This is because devices that don't match any device group will receive the +default variant of your AI pack. This includes anything that is not inside a +directory with a `#group_suffix`. + +Once you have downloaded the AI pack, you can check whether your model is +present by using the [AssetManager](https://developer.android.com/google/play/on-device-ai#install-time-delivery) for install-time +packs or the [AiPackManager](https://developer.android.com/google/play/on-device-ai#access-AI-packs) for fast-follow and on-demand +packs. Examples for doing this are shown for all delivery modes in the +[sample app](https://developer.android.com/google/play/on-device-ai#example-app). + +> [!IMPORTANT] +> **Important:** It's not possible to prevent **any** variant of your AI pack being delivered to certain devices. Non-targeted devices will always receive the default variant. + +### Use device targeting for your feature modules + +You can also use device targeting for feature modules. Instead of subdividing +feature modules by device group, you specify whether the entire module should be +delivered based on device group membership. + +To deliver a feature module to devices that belong to either `myCustomGroup1` or +`myCustomGroup2`, modify its `AndroidManifest.xml`: + + + ... + + + + + + + + + ... + + + + + ... + + +> [!NOTE] +> **Note:** Devices that are not targeted won't receive the feature module at all. This is different from [targeting for AI packs](https://developer.android.com/google/play/on-device-ai#targeting-for-ai-packs), where devices that are not targeted receive a default, empty variant of the AI pack. + +## Test locally + +Before creating a release for your new bundle, you can test locally with either +Internal App Sharing or Bundletool. + +### Internal App Sharing + +Internal App Sharing lets you use an app bundle to quickly generate a URL that +you can tap on a local device to install exactly what Google Play would install +for that device if that version of the app was live in a test or prod track. + +Take a look at the +[internal app sharing instructions](https://support.google.com/googleplay/android-developer/answer/9844679). + +### Bundletool + +Alternatively, you can generate APKs using +[`bundletool`](https://developer.android.com/studio/command-line/bundletool) (1.18.0 or above) and sideload +them onto your device. Follow these steps to test your app locally using +bundletool: + +1. Build your app bundle with Android Studio or bundletool. + +2. Generate APKs with the `--local-testing` flag: + + ``` + java -jar bundletool-all.jar build-apks --bundle=path/to/your/bundle.aab \ + --output=output.apks --local-testing + ``` +3. Connect a device and run `bundletool` to sideload the APKs: + + ``` + # Example without Device Targeting Configuration + java -jar bundletool.jar install-apks --apks=output.apks + ``` + + ``` + # Example with Device Targeting Configuration (you must specify which groups the connected device belongs to) + java -jar bundletool.jar install-apks --apks=output.apks --device-groups=myCustomGroup1,myCustomGroup2 + ``` + +#### Limitations of local testing with bundletool + +The following are limitations of local testing with bundletool: + +- `fast-follow` packs behave as `on-demand` packs. That is, they won't be automatically fetched when the app is sideloaded. Developers need to request them manually when the app starts; this does not require any code changes in your app. +- Packs fetch from external storage instead of Play, so you cannot test how your code behaves in the case of network errors. +- Local testing does not cover the wait-for-Wi-Fi scenario. +- Updates are not supported. Before installing a new version of your build, manually uninstall the previous version. + +### Verify that the correct APKs are being installed + +Use the following method to ensure only the correct APKs are installed on the +device + + adb shell pm path {packageName} + +You should see something like: + + package:{...}/base.apk + package:{...}/split_config.en.apk + package:{...}/split_config.xxhdpi.apk + package:{...}/split_main_ai-pack-name.apk + package:{...}/split_main_ai-pack-name.config.group_myCustomGroup1.apk + +Note that you will only see APKs in this list, which are made from feature +modules and install-time AI packs. On-demand and fast-follow AI packs are not +installed as APKs. + +## Test and release on Google Play + +We recommend that you test your app end to end on Google Play with an +[internal test track](https://support.google.com/googleplay/android-developer/answer/9845334). + +Once you've done this, you can incrementally release your app update to +production with +[staged roll-outs](https://support.google.com/googleplay/android-developer/answer/6346149). + +## Sample app using Play for On-device AI + +Download the +[sample app](https://drive.google.com/drive/folders/1zb2O2mDvEh4vOmv1QbjeLU-O25KHrfKJ). + +It demonstrates how to use each of the delivery modes as well as the device +targeting configuration. See the [local testing](https://developer.android.com/google/play/on-device-ai#local-testing) section to get +started. + +## Related content + +Learn more about [Android App Bundles](https://developer.android.com/guide/app-bundle) and read the +references for the +[AI Delivery SDK](https://developer.android.com/reference/com/google/android/play/core/packages-ai_delivery). \ No newline at end of file diff --git a/device-ai/litert/litert-lm/references/get-a-model.md b/device-ai/litert/litert-lm/references/get-a-model.md new file mode 100644 index 0000000..a84ca8b --- /dev/null +++ b/device-ai/litert/litert-lm/references/get-a-model.md @@ -0,0 +1,36 @@ +This file splits model work between the agent and the user. The agent writes +app code only. The user picks, downloads, and converts model files. + +## Choose a file from litert-community + +Hugging Face `litert-community` holds one repository per model, such as +`litert-community/gemma-4-E2B-it-litert-lm` and `litert-community/Qwen3-0.6B`. +Use `gemma-4-E2B-it.litertlm` from the first or `Qwen3-0.6B.litertlm` from the +second; both run on the CPU and the GPU. + +The other files in a repository are other builds of the same model: + +- Other quantizations and context lengths. The Qwen3 card lists each file with its quantization, its context length in tokens, and its size. +- Web builds, named with the `-web` suffix. +- Builds for one chip, named after it, such as `_Google_Tensor_G5` or `.mediatek.mt6993`. These are NPU builds, which this skill doesn't cover. + +When the user hasn't named a file, use one of the two files in this section. +Don't pick a web build or an NPU build. + +## Write the in-app download code + +The download URL is `https://huggingface.co//resolve/main/`. When +writing the download code in the app, follow these rules: + +- Write the file into `filesDir`, or into `getExternalFilesDir(null)` when the app uses `DownloadManager`. +- Check the downloaded file size against the size on the model card before passing the path to the engine. +- Never place the file in `assets/` or in the APK. +- Never download the model file in the agent's own session. Files are hundreds of MB to a few GB. + +## When no .litertlm file exists + +This skill doesn't convert models. Tell the user that conversion is a separate +step on a workstation, never in the app. The user exports the model with +`litert-torch`, quantizes it with `ai-edge-quantizer`, and bundles the tokenizer +and metadata into a `.litertlm` file with `litert_lm_builder` from the +`litert-lm` package. \ No newline at end of file diff --git a/device-ai/litert/litert-runtime/SKILL.md b/device-ai/litert/litert-runtime/SKILL.md index 3148fc1..3f48421 100644 --- a/device-ai/litert/litert-runtime/SKILL.md +++ b/device-ai/litert/litert-runtime/SKILL.md @@ -10,7 +10,7 @@ description: Configures and implements LiteRT (.tflite) on-device inference on A license: Complete terms in LICENSE.txt metadata: author: Google LLC - last-updated: '2026-10-13' + last-updated: '2026-10-20' keywords: - LiteRT - CompiledModel @@ -40,6 +40,7 @@ skill. - **Buffer lifecycle** : Allocate `TensorBuffer` instances once using `createInputBuffers` and `createOutputBuffers`, reuse them across inferences, and close all input and output buffers before closing the `CompiledModel`. - **Synchronous timing** : `model.run` enqueues work on the GPU and returns before GPU execution completes. Always measure `model.run` and output buffer readback (`outputs[0].readFloat`) together. - **Warm-up and exception safety** : Execute one warm-up inference inside the helper constructor's `init` block so first-run GPU shader compilation doesn't block the first user interaction, and call `close` in `catch` before rethrowing so neither `CompiledModel` nor `TensorBuffer` handles leak if buffer allocation or warm-up throws. +- **Migration** : When the app already uses `org.tensorflow.lite` code, you must read the "Migrating from legacy TFLite Interpreter" section in full before you edit any file. Migrated code must not import any `org.tensorflow.lite` class (including `DataType`), `GpuDelegate` becomes `CompiledModel.Options(Accelerator.GPU, Accelerator.CPU)`, and an NNAPI choice maps to that same GPU option. ## Detailed steps @@ -74,12 +75,12 @@ Step 2. Obtain the pack's directory from the AI Delivery library `AiPackManagerFactory.getInstance(context).getPackLocation(name)?.assetsPath()`, which returns `null` until the pack has finished downloading. -With 2.3.0 nothing is added to `gradle.properties`: `litert` 2.3.0 declares the -namespace `com.google.ai.edge.litert.impl` and its dependency `litert-api` 2.3.0 -declares `com.google.ai.edge.litert`, so AGP 9.x builds the project. With 2.2.0 -or 2.1.6 both libraries declare `com.google.ai.edge.litert`, and AGP 9.x stops -at `processDebugMainManifest` (AGP 8.x reports it as a warning and builds); for -those two versions add `android.uniquePackageNames=false` to +With 2.3.0, nothing is added to `gradle.properties`: `litert` 2.3.0 declares +the namespace `com.google.ai.edge.litert.impl` and its dependency `litert-api` +2.3.0 declares `com.google.ai.edge.litert`, so AGP 9.x builds the project. With +2.2.0 or 2.1.6, both libraries declare `com.google.ai.edge.litert`, and AGP +9.x stops at `processDebugMainManifest` (AGP 8.x reports it as a warning and +builds); for those two versions, add `android.uniquePackageNames=false` to `gradle.properties`, and remove it when moving to 2.3.0 because it also hides the same clash between any other two libraries. Excluding `litert-api` instead doesn't work: `CompiledModel` and `Accelerator` are in `litert-api`, and the @@ -236,7 +237,7 @@ state to the Compose UI: One single-thread executor owns the model: creation, inference, and closing execute only on it, never on the main thread. The `ViewModel` keeps its own -scope because `viewModelScope` is cancelled before `onCleared` runs, so a close +scope because `viewModelScope` is canceled before `onCleared` runs, so a close launched there wouldn't run; `onCleared` cancels `scope` once `close` completes. `load` is a no-op when a model is already loaded across configuration changes; a failed `load` leaves `classifier` `null`. `classify` reports any @@ -331,14 +332,24 @@ outputs on a physical device, you must follow [device verification](references/v ## Migrating from legacy TFLite Interpreter -When upgrading an existing project from `org.tensorflow.lite.Interpreter`, -replace legacy constructs with their `CompiledModel` equivalents: +An existing app on `org.tensorflow.lite.Interpreter` (from +`org.tensorflow:tensorflow-lite*` or from `com.google.ai.edge.litert:litert` +1.x) moves to the `CompiledModel` code of Steps 1 to 5 in seven steps, in this +order. Before you change any file of an +existing app, you must read [migration](references/migration.md), which gives the commands and code +for each step: -- **Gradle dependencies** : Remove all `org.tensorflow:tensorflow-lite*` artifacts (`org.tensorflow:tensorflow-lite`, `org.tensorflow:tensorflow-lite-gpu`, `org.tensorflow:tensorflow-lite-support`, and `org.tensorflow:tensorflow-lite-metadata`). Replace them with `com.google.ai.edge.litert:litert:2.3.0` and `com.google.ai.edge.litert:litert-gpu:2.3.0`. If you still need Support or Metadata utilities (such as `ImageProcessor`, `TensorImage`, or `MetadataExtractor`), use `com.google.ai.edge.litert:litert-support:2.3.0` (or `litert-support-api:2.3.0` alongside `litert:2.3.0`) and `com.google.ai.edge.litert:litert-metadata:2.3.0`, and rename package imports from `org.tensorflow.lite.support.*` to `com.google.ai.edge.litert.support.*` per the [LiteRT Support migration guide](https://github.com/google-ai-edge/LiteRT/blob/main/litert/support_lib/MIGRATION_GUIDE.md). -- **Model loading** : Replace `FileUtil.loadMappedFile` and `Interpreter` with `CompiledModel.create(context.assets, name, options, environment)`. -- **GPU delegation** : Replace `GpuDelegate` on `Interpreter.Options` with `CompiledModel.Options(Accelerator.GPU)` and the `com.google.ai.edge.litert:litert-gpu:2.3.0` dependency. -- **Execution** : Replace `interpreter.run` on `ByteBuffer` inputs with `inputs[0].writeFloat(input)`, `model.run(inputs, outputs)`, and `outputs[0].readFloat`. -- **Label loading** : Replace `FileUtil.loadLabels` with `bufferedReader` and `readLines` on `context.assets.open(name)` (or `com.google.ai.edge.litert.support.common.FileUtil.loadLabels` if using `litert-support:2.3.0`). +1. **Find the runtime**: Read the build files and the resolved dependencies, and note the delegate, the thread count, and the Support library classes that the app uses. +2. **Capture a golden output first**: While the legacy runtime is still in the project, record the app's output for one fixed input on the CPU. This step needs a device or an emulator. +3. **Swap the dependencies** : Remove every legacy artifact and add the dependencies of Step 1. An app on the Support library keeps it as `com.google.ai.edge.litert:litert-support:2.3.0`, with the package `org.tensorflow.lite.support` renamed to `com.google.ai.edge.litert.support` ([LiteRT Support migration guide](https://github.com/google-ai-edge/LiteRT/blob/main/litert/support_lib/MIGRATION_GUIDE.md)). +4. **Replace the delegates** : `GpuDelegate` becomes `CompiledModel.Options(Accelerator.GPU, Accelerator.CPU)`, so the CPU runs the operators that the GPU doesn't support. An NNAPI choice in the app maps to the same GPU option, not to `Accelerator.CPU` alone or to an NPU accelerator. `setNumThreads(n)` becomes new CPU options with `cpuOptions = CompiledModel.CpuOptions(numThreads = n)`. +5. **Rewrite the inference code** : Replace `Interpreter` and `FileUtil.loadMappedFile` with the `Classifier` helper of Step 2, created and run only on the single-thread executor of Step 3, never in a constructor that the main thread calls. +6. **Verify in two stages**: The build passes and the searches for legacy references print nothing; then the migrated output matches the golden output on a device. +7. **Report what changed and what didn't migrate** : When the app used NNAPI, tell the user that it now runs on the GPU with the CPU as a second accelerator, which changes the speed and the numerical results of a quantized model that ran on a DSP or an NPU before; point them to [NPU acceleration with LiteRT](https://ai.google.dev/edge/litert/next/npu) for the NPU. Tell the user about LiteRT in Google Play services, the TensorFlow Lite Task Library, C++ modules, and Select TensorFlow operators, which this skill doesn't migrate. + +Migration mandates that apply to every migrated app: + +- **No `org.tensorflow.lite` imports** : Migrated code must not import any `org.tensorflow.lite` class, including `org.tensorflow.lite.DataType`; the LiteRT artifacts keep these classes only for backward compatibility. Replace Support calls that need `DataType` (`TensorImage(DataType.FLOAT32)`, `TensorBuffer.createFixedSize`, and `TensorLabel`) with `preprocess` of Step 4, the `CompiledModel` buffers, and labels paired with the output scores. Remove every existing `org.tensorflow.lite` import line, not only the ones that you add. ## Antipatterns diff --git a/device-ai/litert/litert-runtime/references/migration.md b/device-ai/litert/litert-runtime/references/migration.md new file mode 100644 index 0000000..41f2296 --- /dev/null +++ b/device-ai/litert/litert-runtime/references/migration.md @@ -0,0 +1,222 @@ +This file gives the commands and code for each step of the migration summary in +the skill. The migrated app ends with the code of Steps 1 to 5 of the skill: the +`Classifier` helper of Step 2, the executor and CPU fallback of Step 3, and +`preprocess` of Step 4. + +The code in this file uses these imports: + + import android.content.Context + import com.google.ai.edge.litert.Accelerator + import com.google.ai.edge.litert.CompiledModel + import com.google.ai.edge.litert.TensorBuffer + import com.google.ai.edge.litert.support.image.TensorImage + import com.google.ai.edge.litert.support.tensorbuffer.TensorBufferFloat + import java.io.File + import kotlin.math.abs + +## 1. Find the runtime + +Read `app/build.gradle.kts` (or `app/build.gradle`), +`gradle/libs.versions.toml`, and the output of `./gradlew :app:dependencies`, +which also shows what other libraries pull in. A legacy app gets the Interpreter +runtime from one of these: + +- **TensorFlow Lite** : `org.tensorflow:tensorflow-lite` and the other `org.tensorflow:tensorflow-lite*` artifacts. +- **LiteRT 1.x** : `com.google.ai.edge.litert:litert`, `litert-gpu`, `litert-support`, or `litert-metadata` at a 1.x version. + +When the app depends on LiteRT in Google Play services +(`com.google.android.gms:play-services-tflite-java`, `-gpu`, or `-support`, with +`TfLite.initialize` in the code) or on the TensorFlow Lite Task Library +(`org.tensorflow:tensorflow-lite-task-vision`, `-text`, or `-audio`), stop and +tell the user. This skill doesn't migrate either of them. + +Then find the code that the later steps change: + + grep -rnE 'org\.tensorflow\.lite\.|Interpreter|GpuDelegate|CompatibilityList|NnApiDelegate|NNAPI|[nN]umThreads|FlexDelegate|TfLite' --include='*.kt' --include='*.java' app/src + +Note which accelerator each code path uses (the GPU delegate, NNAPI, or the CPU +with a thread count), and whether the app uses the TensorFlow Lite Support +Library (`ImageProcessor`, `TensorImage`, `TensorLabel`, `MetadataExtractor`). + +## 2. Capture a golden output first + +The golden output is the legacy app's output for one fixed input, recorded +before any dependency changes. Step 6 compares the migrated app with it. Pick +one path: + +- **Path A (device or emulator connected)** : Write an instrumented test with the project's existing test setup. The test feeds one deterministic input through the app's own inference class and preprocessing on the CPU, without the GPU or NNAPI delegate. Use a bitmap of another size than the model input, never a camera frame. Write the output with `writeGoldenOutput`, then copy the file to `app/src/androidTest/assets/golden/output.txt`. +- **Path B (no device)**: Don't capture. Report the capture and the device comparison of Step 6 to the user as post-implementation steps. + + // Before the dependency swap: record the legacy app's output for one fixed + // input, one value per line. + fun writeGoldenOutput(file: File, output: FloatArray) { + file.writeText(output.joinToString("\n")) + } + + // After the migration: compare the migrated output with the recorded one. + // A difference smaller than 1e-3 passes for outputs of order 1. + fun maxAbsDifference(golden: FloatArray, migrated: FloatArray): Float { + check(golden.size == migrated.size) { "Size ${migrated.size} != ${golden.size}" } + return golden.indices.maxOf { abs(golden[it] - migrated[it]) } + } + + fun readGoldenOutput(context: Context, path: String = "golden/output.txt"): FloatArray = + context.assets.open(path).bufferedReader().use { reader -> + reader.readLines().filter { it.isNotBlank() }.map { it.toFloat() }.toFloatArray() + } + +## 3. Swap the dependencies + +Remove every legacy artifact, also when another library pulls it in. The LiteRT +artifacts carry the `org.tensorflow.lite` runtime classes under their original +names, so a build that mixes old and new artifacts fails with duplicate classes. +Add the dependencies of Step 1 of the skill (`litert:2.3.0` and +`litert-gpu:2.3.0`) with `minSdk` 24 or higher. + +### Support library + +An app on the TensorFlow Lite Support Library keeps it. The library ships as +LiteRT Support, with the same classes and behavior. When the app uses +`ImageProcessor`, `TensorImage`, `TensorLabel`, or `MetadataExtractor`, you must +read the [LiteRT Support migration guide](https://github.com/google-ai-edge/LiteRT/blob/main/litert/support_lib/MIGRATION_GUIDE.md) before swapping the coordinates: + +- `org.tensorflow:tensorflow-lite-support` or `litert-support` 1.x becomes `com.google.ai.edge.litert:litert-support:2.3.0`. +- `org.tensorflow:tensorflow-lite-support-api` or `litert-support-api` 1.x becomes `litert-support-api:2.3.0`, next to `litert:2.3.0`. +- `tensorflow-lite-metadata` or `litert-metadata` 1.x becomes `litert-metadata:2.3.0`. It depends on `com.google.flatbuffers:flatbuffers-java:25.2.10`; when the app pins an older `flatbuffers-java`, upgrade it. + +Keep `litert:2.3.0` in the build file. Then rename the package in the sources, +the R8 and ProGuard rules, and any reflection strings: + + grep -rl 'org\.tensorflow\.lite\.support' --include='*.java' --include='*.kt' --include='*.pro' app/ \ + | xargs sed -i 's/org\.tensorflow\.lite\.support/com.google.ai.edge.litert.support/g' + +An app that never used the Support library doesn't add it. Without the Support +library, load labels from assets: + + // Replaces FileUtil.loadLabels in an app that drops the Support library. + fun loadLabels(context: Context, name: String = "labels.txt"): List = + context.assets.open(name).bufferedReader().use { it.readLines() } + +### Toolchain + +The 2.3.0 artifacts need JDK 17 or later for the Gradle build, `compileSdk` +35, AGP 8.6.0 or later, and the Kotlin plugin 2.3 or later. Their classes carry +Kotlin 2.4 metadata. Ask the user before editing `gradle/libs.versions.toml` or +the root `build.gradle.kts`: + +- Kotlin 2.2 stops with `The binary version of its metadata is 2.4.0`. A project that stays on Kotlin 2.2 adds `freeCompilerArgs += "-Xskip-metadata-version-check"`. +- With Kotlin 2.x, a Compose app applies the `org.jetbrains.kotlin.plugin.compose` plugin in place of `composeOptions`. + +## 4. Replace the delegates + +- **GPU delegate** (`GpuDelegate`, `GpuDelegateFactory`): Create the GPU option of the Step 2 helper with the CPU as a second accelerator. `CompatibilityList` has no counterpart; the CPU fallback of Step 3 covers a device whose GPU can't run the model. +- **NNAPI** (`NnApiDelegate`, `setUseNNAPI(true)`): NNAPI is deprecated from Android 15, and the `CompiledModel` API has no NNAPI accelerator. Remove the delegate, and map an NNAPI choice in the app to the GPU option. You must tell the user about this change in the final report, because the speed and the numerical results of a quantized model that ran on a DSP or an NPU through NNAPI change on the GPU. NPU acceleration is a separate topic that this skill doesn't cover ([NPU acceleration with LiteRT](https://ai.google.dev/edge/litert/next/npu)). +- **Select TensorFlow operators** (`FlexDelegate`): No LiteRT 2.x artifact contains the Flex delegate. This skill doesn't cover it. Tell the user that the model needs to be exported again without Select TensorFlow operators. + + // Replaces GpuDelegate: the CPU runs the operators that the GPU doesn't + // support, as it did with the delegate. Keep the CPU fallback of Step 3 for a + // model that the GPU can't compile at all. + val gpuWithCpuFallback = CompiledModel.Options(Accelerator.GPU, Accelerator.CPU) + +## 5. Rewrite the inference code + +Never search and replace `Interpreter`. Move the model into the `Classifier` +helper of Step 2 of the skill, owned by the single-thread executor of Step 3. +Keep the app's own classes and their public functions as the callers. + +- **Model loading** : `Interpreter` from `FileUtil.loadMappedFile`, and the Support `Model.createModel`, become `CompiledModel.create` in the helper. No overload takes a `ByteBuffer`. +- **Inference** : `run` on `ByteBuffer` objects becomes `writeFloat`, `run`, and `readFloat` on buffers that the helper creates once. Use `writeInt8` and `readInt8` for int8 and uint8 tensors. +- **Thread count** : `setNumThreads` becomes `CpuOptions(numThreads = n)` on new CPU options. +- **Tensor shapes** : `getInputTensor` and its `shape` become a lookup by name. `resizeInput` has no counterpart in 2.3.0. +- **Signatures** : `runSignature` becomes `run` with maps from tensor names to buffers. +- **Closing** : Closing the interpreter becomes the helper's `close`. + + // Replaces Interpreter.Options.setNumThreads. Create new options; never change + // the shared CompiledModel.Options.CPU. + fun cpuOptionsWithThreads(numThreads: Int): CompiledModel.Options = + CompiledModel.Options(Accelerator.CPU).apply { + cpuOptions = CompiledModel.CpuOptions(numThreads = numThreads) + } + + // Replaces interpreter.getInputTensor(0).shape(). Look the tensor up by name. + fun inputShape(model: CompiledModel, inputName: String): List? = + model.getInputTensorType(inputName).layout?.dimensions + + // Replaces interpreter.runSignature. Create the buffers once with + // model.createInputBuffers(signature) and model.createOutputBuffers(signature), + // and reuse them for every call. + fun runSignature( + model: CompiledModel, + signature: String, + inputs: Map, + outputs: Map + ) { + model.run(inputs, outputs, signature) + } + +### With the Support library + +Migrated code must not import any `org.tensorflow.lite` class, including +`org.tensorflow.lite.DataType`. The LiteRT artifacts keep these classes only for +backward compatibility. Apply these rules to the Support code: + +- `ImageProcessor`, `TensorImage`, and `FileUtil` stay under `com.google.ai.edge.litert.support` when the calls don't need `DataType`. +- The Support `Model` class runs on the Interpreter API. Replace it with the `Classifier` helper of Step 2 of the skill. +- Replace a Support call that needs `DataType`, such as `TensorImage(DataType.FLOAT32)`, `TensorBuffer.createFixedSize`, or `TensorLabel`, with `preprocess` of Step 4 or with the `CompiledModel` buffers. + +The following code writes a processed image into the helper's buffers and pairs +the output scores with the labels, without `DataType`: + + // Replaces the Support library's Model class. ImageProcessor and TensorImage + // stay; the processed image goes into the CompiledModel buffers directly, and + // the scores pair with the labels without TensorLabel. + fun runWithSupportImage( + model: CompiledModel, + inputs: List, + outputs: List, + image: TensorImage, + inputElementCount: Int, + labels: List + ): Map { + // A uint8 image (one that skipped NormalizeOp) in a float input yields NaN. + val pixels = (image.tensorBuffer as? TensorBufferFloat)?.floatArray + ?: error("Normalize the image to float first") + // writeFloat accepts a shorter array without an error. + check(pixels.size == inputElementCount) { "Input size ${pixels.size}" } + inputs[0].writeFloat(pixels) + model.run(inputs, outputs) + return labels.zip(outputs[0].readFloat().toList()).toMap() + } + +### Without the Support library + +`preprocess` of Step 4 of the skill replaces the code that filled the input +`ByteBuffer`. Give it the model's input size and the legacy normalization: the +`mean` and `std` of a legacy `NormalizeOp(mean, std)`, or the app's own formula, +in the same channel order and layout. + +## 6. Verify in two stages + +**Stage 1, on the workstation** : `./gradlew assembleDebug` passes, and neither +of the following commands prints anything. The build alone doesn't reveal +leftover Interpreter code, because the `litert` 2.3.0 artifact still contains +`org.tensorflow.lite.Interpreter`: + + grep -rnE 'org\.tensorflow\.lite\.|GpuDelegate|NnApiDelegate|FlexDelegate|support\.model\.Model|org\.tensorflow:tensorflow-lite' --include='*.kt' --include='*.java' --include='*.kts' --include='*.gradle' --include='*.toml' --include='*.pro' --exclude-dir=build --exclude-dir=.gradle . + ./gradlew :app:dependencies --configuration releaseRuntimeClasspath | grep -E 'org\.tensorflow:tensorflow-lite|com\.google\.ai\.edge\.litert:litert[a-z-]*:1\.' + +**Stage 2, on a device**: Pick one path: + +- **Path A (golden output captured in Step 2)** : Extend the instrumented test of Step 2. Run the same input through the migrated preprocessing and the helper on `Accelerator.CPU`, and read the golden file with `readGoldenOutput`. The test must fail when `maxAbsDifference` is 1e-3 or more. When the golden file holds only one value, the top-1 index must match; a wrong normalization often still passes this check. Then compare the CPU and the GPU as in Step 5 of the skill. +- **Path B (no device)**: Run Stage 1 only, and report that Stage 2 didn't run. + +## Out of scope + +This skill doesn't migrate LiteRT in Google Play services or the +TensorFlow Lite Task Library (`tensorflow-lite-task-vision`, `-text`, or +`-audio`), which has no LiteRT release. + +This skill doesn't migrate C++ code. When the app has `CMakeLists.txt`, +`Android.mk`, or `.cpp` files that link `tensorflowlite_jni`, tell the user that +the C++ module moves to the LiteRT C++ SDK, and point them to +[Use prebuilt C++ library from LiteRT Maven](https://ai.google.dev/edge/litert/next/android_cpp_sdk). \ No newline at end of file diff --git a/devtools/android-cli/SKILL.md b/devtools/android-cli/SKILL.md index c9ebe6d..e0a5f9f 100644 --- a/devtools/android-cli/SKILL.md +++ b/devtools/android-cli/SKILL.md @@ -2,14 +2,14 @@ name: android-cli description: Provides instructions for installing and using the `android` CLI. The `android` command-line tool is a critical tool for Android development and helps - you create new Android projects, run Android apps on devices, manage and interact - with Android virtual devices (including screenshots and UI inspection), manage Android - SDK components, look up official Android documentation, and discover and install - official Android skills. + you create new Android projects, install APKs onto devices and emulators and launch + and run apps on them, manage and interact with Android virtual devices (including + screenshots and UI inspection), manage Android SDK components, look up official + Android documentation, and discover and install official Android skills. license: Complete terms in LICENSE.txt metadata: author: Google LLC - last-updated: '2026-09-12' + last-updated: '2026-10-16' keywords: - android-cli - sdk @@ -30,7 +30,8 @@ interacting with devices, and managing the CLI environment. If the android tool is not in the path, install it. To install run the following command: -- Linux: curl -fsSL https://dl.google.com/android/cli/latest/linux_x86_64/install.sh \| bash +- Linux x86-64: curl -fsSL https://dl.google.com/android/cli/latest/linux_x86_64/install.sh \| bash +- Linux ARM64: curl -fsSL https://dl.google.com/android/cli/latest/linux_arm64/install.sh \| bash - Mac Arm: curl -fsSL https://dl.google.com/android/cli/latest/darwin_arm64/install.sh \| bash - Mac Intel: curl -fsSL https://dl.google.com/android/cli/latest/darwin_x86_64/install.sh \| bash - Windows: curl -fsSL https://dl.google.com/android/cli/latest/windows_x86_64/install.cmd -o "%TEMP%\\i.cmd" \&\& "%TEMP%\\i.cmd" @@ -57,11 +58,103 @@ For example: `android create empty-activity --name="My App" --output=./my-app` Use the `android layout` command to inspect the UI layout of an Android application in JSON format. Use the `android screen` command to visually inspect the UI and obtain bounding coordinates for visual regions. +`android layout` dumps the UI as JSON. `android screen capture -o ` saves a PNG. +Both auto-select the device when only one is connected. -**IMPORTANT:** Before using `android layout` or `android screen`, you MUST read -this reference: [interact.md](references/interact.md) The reference file -[interact.md](references/interact.md) contains instructions for properly -interacting with Android devices. Follow its instructions exactly. +### UI Dump + +`android layout` returns a JSON tree of the UI elements on screen. + +Each JSON object represents a UI element in the Android app. The following properties may be present: +- `text` - any literal text the element contains +- `resource-id` - the Android resource id used to refer to the element +- `content-desc` - a description of a UI element for use by accessibility tools +- `interactions` - the set of user interactions the element supports. May contain one or more of: `CHECKABLE`, `CLICKABLE`, `FOCUSABLE`, `SCROLLABLE`, `LONG-CLICKABLE`, `PASSWORD` +- `state` - the set of states the element is in. May contain one or more of `CHECKED`, `FOCUSED`, `SELECTED` +- `bounds` - the screen coordinates of the bounding rectangle of the element, in the format `[min X,min Y][max X, max Y]` +- `center` - the screen coordinates of the center of the element, in the format `[x,y]` +- `off-screen` - if true, the element is in the UI hierarchy but not within the bounds of the screen; it may require scrolling to view. +- `hidden` - if true, the element is in the UI hierarchy but not visible or interactive on screen. + +Use `layout` as a primary means of examining an Android app. + +By default, `layout` only returns elements that have content or are interactive: +- elements with non-empty `text` or `contentDesc` properties +- elements with non-empty `interactions` or `state` +To return all elements, use the `--full` flag + +If `layout` continues to fail, use `android screen capture --annotate` to inspect the app. + +### Screenshot + +`android screen capture -o ` saves a PNG of the current device screen to `` + +Use `screen capture` as a secondary means of examining an Android app +Examples: +- Understanding the content of an on-screen image +- Looking at a `WebView` (web content does not always appear in the ui dump) +- Trying to find a UI element by its visual appearance + +**IMPORTANT** : Always *VISUALLY* examine the PNG image returned from `android screen` BEFORE doing anything else. + +### Annotated Screenshot + +`android screen capture --annotate -o ` +`android screen resolve --screenshot --string ` + +The `--annotate` command adds numerical labels and bounding boxes around UI elements. Use this command to locate UI elements that cannot +be located in the `layout` output. + +**IMPORTANT** : When using `android screen capture --annotate`, always *VISUALLY* examine the resulting PNG file. + +To refer to these labels in input commands, use `screen resolve` to convert labels into coordinates: + +`android screen resolve --screenshot --string "#3"` returns ` ` + +To save turns, you can combine shell commands: + +`adb shell input $(android screen resolve --screenshot screen.png --string "tap #34")` + +This command taps on region #34 from `screen.png` + +### Input + +Use `adb shell input` for interacting with Android devices. +Refer to the `"interactions"` property of an element for what interactions can be performed on a particular element. + +Interact with UI elements with their `center` coordinate or their `bounds` coordinates: +`json +{ +"key": -248568265, +"class": "android.widget.Button", +"bounds": "[138,9][167,38]", +"center": "[152,23]" +}` +To tap on this button, you would execute `adb shell input tap 152 23`. This taps the center. + + { + "key": 12487234, + "class": "com.example.ui.ScrollableList", + "bounds": "[100,200][400,600]", + "center": "[250,400]" + } + +To scroll down on this list, you would execute `adb shell input swipe 250 400 250 200 500`. This swipes from the center to the top over 500ms. + +#### Text Input + +To enter text, ensure the field is focused and execute `adb shell input text ""`. + +- **Spaces (`%s`):** Replace spaces with `%s` (Android ignores text after raw spaces): `adb shell input text "Hello%sworld"` (enters `"Hello world"`). +- **Special characters:** Escape shell metacharacters (`&`, `$`, `(`, `)`, `!`): `adb shell input text "AT\&T"` or `adb shell input text 'Price:\$10'`. +- **Submit / Enter:** `adb shell input keyevent 66` + +### Android Interaction Rules + +1. Always ensure text input fields have `"FOCUSED"` in their `"state"` list before entering text. +2. If an element has `"SCROLLABLE"` in its `"interactions"` list, try scrolling it when looking for missing UI elements. +3. Always scroll slowly when executing scroll inputs. The 5th argument to `adb shell input swipe` controls scroll duration. +4. Content may take time to load; if a `layout` is missing information after you take an action, wait a few seconds, then perform `layout` again to see if anything changes. ### Running journey tests @@ -69,10 +162,107 @@ Journey tests involve device interaction; use the `android layout` and `android screen` commands to evaluate Journeys. The instructions for device interaction apply here as well. -**IMPORTANT:** Before evaluating a journey, you MUST read this reference: -[journeys.md](references/journeys.md) The reference file -[journeys.md](references/journeys.md) contains instructions for correctly -evaluating an Android journey. Follow its instructions exactly. +A journey is an XML-specified test of an Android app's behavior. It consists of a list of `` elements. For example: +`xml + + +A sample journey to illustrate the format + + + +Tap the "Home" icon + + +Verify that the app is on its Home screen + + +` + +Evaluate a journey by proceeding through the `` list in sequential order. Evaluate each `` block individually. +A journey succeeds if all elements in the `` list succeed. + +A journey is a test case for an app. The journey XML is the source of truth; if the app disagrees with the journey, the app has failed. +Additionally, if the app exits, crashes, or freezes, journey evaluation stops and the journey fails. + +**IMPORTANT** - Execute each step EXACTLY as written, and independently of other steps! If an action says to `"tap the first search result"`, +you MUST find the search results and tap the first one. Do this even if you believe you know the intent behind the action. + +#### Taking Actions + +Some `` elements specify UI interactions to perform on the running Android app. Perform the interaction and verify that the app does +not crash or behave in an unexpected manner. This is the *only* verification you should perform for an ``. + +If the interaction cannot be performed as specified, the journey fails. +Example: +`Click the red button` +If you determine a red button is not present in the UI, the journey fails. + +If the text of an `` specifies a list of actions, break it into sub-actions and evaluate them individually: +Example: +`Search for soda and add the first result to the cart` +This should be evaluated as: +`Search for soda +Add the first result to the cart` + +If an `` contains something that is not a specification for a UI interaction, alert the user that the journey is malformed and exit +early, specifying the error in question. + +#### Verifying Expectations + +`` elements that begin with "check" or "verify" specify expectations for the current state of the Android app. Determine the current +state of the app and check if the expectations are met. + +Determine the current state of the app by inspecting the current screen of the device without interacting with it. +Example: +`Check if "Switch 2" is visible on the screen` +This requires only inspecting the current screen, not scrolling or interacting. If "Switch 2" is not currently visible, the action fails. + +If the expectations are not met, mark the `` as a failure and the journey evaluation ends. A single `` may contain +multiple expectations. +Example: +`Verify that the app is on the Home screen, the Home icon is blue, and the temperature is displayed` +This `` fails if ANY of the following are false: +- The app is on the Home screen +- There is a Home icon, and it is blue +- A temperature is displayed + +#### Handling failure + +When running a journey, evaluate it as a test. Failure is acceptable, and often expected. Proper reporting of failures is the priority. + +Keep debugging and troubleshooting to a minimum; assume that tools are showing you the correct output every time. The goal is to determine +if the *current* Android app can correctly handle the *current* steps outlined in the journey. Suggestions for bug fixes, clarification, or +other improvements should be kept to journey evaluation summary at the end. + +#### Summarizing + +After evaluating the whole journey, output a markdown (.md) file describing the results. + +The file should have the following format: + + # Journey: + + ## Results + + + ### Action: Click the blue button ✅ + - **Commands**: + + - `adb shell input swipe 490 200 500 500 500` + - `adb shell input tap 45 920` + + + - **Comment**: The journey step doesn't specify that the button requires scrolling to see + + ### Action: Check if "Switch 2" is visible on the screen ✅ + - **Screenshot**: [switch_2_screenshot.png](path/to/screenshot) + - **Comment**: Confirmed both in layout and visually + + ### Action: The home screen is shown ❌ + - **Comment**: The settings page was shown + + + ### Action: Swipe down to view active notifications ## Doc searching @@ -100,8 +290,15 @@ Manage Android Virtual Devices (AVDs) using the `android emulator` command ## Capturing screenshots -Capture an image of the current screen of a connected Android device and output -it to a file using the `android screen` command. +`android screen capture -o ` (or `--output=`) + +Saves a PNG of the current device screen. When exactly one device or emulator is +connected it is selected automatically. With more than one, pass +`--device=`; `adb devices` lists the serials (`android info` does not +report connected devices). + +**IMPORTANT:** Always *VISUALLY* examine the resulting PNG before doing anything +else. ## Managing skills @@ -114,53 +311,79 @@ Update the Android CLI using the `android update` command. ## `android help` output Usage: android [-Vhv] [--sdk=PARAM] [COMMAND] - -h, --help Show the help message for this command + -h, --help Shows the help message for the specified command --sdk=PARAM Path to the Android SDK -v, --verbose Enable verbose output for troubleshooting -V, --version Print version information and exit Commands: - completion Installs shell autocomplete configuration for the Android CLI in - the current user profile - create Create a new Android project from available templates. Allows - specifying project name, output directory, minSdk, and dry-run - execution + auth Authentication commands. Log in to and out of Google services for + Android CLI + completion Installs shell autocomplete configuration for Android CLI in the + current user profile + create Creates a new Android project from available templates. You can + specify the project name, output directory, `minSdk` value, and + dry-run execution describe Analyzes an Android project to generate descriptive metadata. This command identifies and outputs the paths to JSON files that detail the project's structure, including build targets and their - corresponding output artifact locations (e.g., APKs). This + corresponding output artifact locations (such as APKs). This information enables other tools and commands to locate build artifacts efficiently - docs Android documentation commands for searching and fetching - developer documentation from the official knowledge base - emulator Manage Android Virtual Devices (AVDs). Includes commands to start, - stop, list, and view details about emulators - help Shows the help of all commands - info Print environment information including SDK location, connected - devices, and configuration variables. Use specific fields to - narrow the output - init Initializes the environment for the Android CLI. Sets up required + device Manage physical Android devices. Create remote physical device + reservations + docs Searches and fetches developer documentation from the official + Android Knowledge Base + emulator Manages Android Virtual Devices (AVDs). Includes commands to + start, stop, list, and view details about emulators + help Shows the help information for a specified command + info Prints environment information including SDK location, connected + devices, and configuration variables + init Initializes the environment for Android CLI. Sets up required configurations, directories, and default skills - install Install an Android Application (one or more APKs) to a connected - device or emulator without activating any components using - incremental optimizations for faster deployment than adb - layout Returns the layout tree of an application - run Build, deploy, and launch an Android application on a connected - device or emulator - screen Commands to view the device - sdk Manage the Android SDK installation. Includes commands to install, - update, remove, and list available and installed SDK packages - skills Manage Android CLI skills. Includes commands to install, remove, + input Interact with Android apps by performing gestures and key events + on UI elements + install Installs an Android app (one or more APKs) on a connected device + or emulator without activating any components, using incremental + optimizations for faster deployment than `adb` + layout Returns the layout tree of an app + run Deploys and launches an Android app on a connected device or + emulator + screen Captures and inspects the screen of a connected Android device or + emulator + sdk Manages the Android SDK installation. Includes commands to + install, update, remove, and list available and installed SDK + packages + skills Manages Android CLI skills. Includes commands to install, remove, list, and search for skills by keyword - studio Android Studio commands - update Update the Android CLI to the latest version + studio Connects Android CLI to a running Android Studio instance to + analyze files, find declarations and usages, render Compose + previews, and look up library versions + update Updates Android CLI to the latest version + + auth + Usage: android auth [-h] [COMMAND] + Authentication commands. Log in to and out of Google services for + Android CLI + + Options: + -h, --help Shows the help message for the specified command + + android options: + --sdk=PARAM Path to the Android SDK + -v, --verbose Enable verbose output for troubleshooting + -V, --version Print version information and exit + Commands: + login Log in to Google services for Android CLI + logout Log out from Google services for Android CLI + status Prints the currently signed-in account completion Usage: android completion [-h] [] - Installs shell autocomplete configuration for the Android CLI in the + Installs shell autocomplete configuration for Android CLI in the current user profile Options: - -h, --help Show the help message for this command + -h, --help Shows the help message for the specified command android options: --sdk=PARAM Path to the Android SDK @@ -168,35 +391,42 @@ Update the Android CLI using the `android update` command. -V, --version Print version information and exit Positional Parameters: - If provided, print the given shell completion without - installing. Supported: bash, zsh + If provided, prints the completion configuration script for + the specified shell without installing it. Supports Bash + and Zsh create - Usage: android create [-h] [--applicationId=PARAM] [--list] [--minSdk=PARAM] - [--name=PARAM] [--namespace=PARAM] [--output=PARAM] - [] - Create a new Android project from available templates. Allows - specifying project name, output directory, minSdk, and dry-run - execution + Usage: android create [-h] [--application-id=PARAM] [--force] [--list] + [--min-sdk=PARAM] [--name=PARAM] [--namespace=PARAM] + [--output=PARAM] [] + Creates a new Android project from available templates. You can + specify the project name, output directory, `minSdk` value, and + dry-run execution Options: - --applicationId=PARAM The application ID for the application - (e.g. 'com.example.myapp') - -h, --help Show the help message for this command - --list List all available templates - --minSdk=PARAM The 'minSdk' supported by the application - (default is defined in the template) - --name=PARAM The name of the application (e.g. 'My - Application') - --namespace=PARAM The namespace for resources and package - name for Kotlin source files - -o, --output=PARAM The destination project directory path - (default is '.') + --application-id=PARAM The app ID for the app, for example + `com.example.myapp` + --force Allow creating the project in a + non-empty directory, overwriting + existing files + -h, --help Shows the help message for the specified + command + --list List all available templates + --min-sdk=PARAM The `minSdk` value supported by the app + (the default value is defined in the + template) + --name=PARAM The name of the app, for example `My + Application` + --namespace=PARAM The namespace for resources and package + name for Kotlin source files + -o, --output=PARAM The destination project directory path + (the default value is `.`) android options: - --sdk=PARAM Path to the Android SDK - -v, --verbose Enable verbose output for troubleshooting - -V, --version Print version information and exit + --sdk=PARAM Path to the Android SDK + -v, --verbose Enable verbose output for + troubleshooting + -V, --version Print version information and exit Positional Parameters: The template name @@ -206,11 +436,12 @@ Update the Android CLI using the `android update` command. Analyzes an Android project to generate descriptive metadata. This command identifies and outputs the paths to JSON files that detail the project's structure, including build targets and their corresponding - output artifact locations (e.g., APKs). This information enables other - tools and commands to locate build artifacts efficiently + output artifact locations (such as APKs). This information enables + other tools and commands to locate build artifacts efficiently Options: - -h, --help Show the help message for this command + -h, --help Shows the help message for the specified + command --project_dir=PARAM The project directory to describe android options: @@ -218,29 +449,46 @@ Update the Android CLI using the `android update` command. -v, --verbose Enable verbose output for troubleshooting -V, --version Print version information and exit + device + Usage: android device [-h] [COMMAND] + Manage physical Android devices. Create remote physical device + reservations + + Options: + -h, --help Shows the help message for the specified command + + android options: + --sdk=PARAM Path to the Android SDK + -v, --verbose Enable verbose output for troubleshooting + -V, --version Print version information and exit + Commands: + remote Manage remote physical devices. Includes commands to list, + reserve, connect to, and manage remote device reservations + docs Usage: android docs [-h] [COMMAND] - Android documentation commands for searching and fetching developer - documentation from the official knowledge base + Searches and fetches developer documentation from the official Android + Knowledge Base Options: - -h, --help Show the help message for this command + -h, --help Shows the help message for the specified command android options: --sdk=PARAM Path to the Android SDK -v, --verbose Enable verbose output for troubleshooting -V, --version Print version information and exit Commands: - search Search Android documentation. Enclose keywords in quotes - fetch Fetch an Android documentation article from a URL (kb://...) + fetch Fetches an Android documentation article from a URL starting + with `kb://` + search Searches Android documentation. Enclose keywords in quotes emulator Usage: android emulator [-h] [COMMAND] - Manage Android Virtual Devices (AVDs). Includes commands to start, + Manages Android Virtual Devices (AVDs). Includes commands to start, stop, list, and view details about emulators Options: - -h, --help Show the help message for this command + -h, --help Shows the help message for the specified command android options: --sdk=PARAM Path to the Android SDK @@ -248,18 +496,18 @@ Update the Android CLI using the `android update` command. -V, --version Print version information and exit Commands: create Creates a virtual device - start Launches the specified virtual device. This command will - return when the emulator is fully started and ready to use - stop Stops the specified virtual device list Lists available virtual devices - remove Delete a virtual device + remove Deletes a virtual device + start Launches the specified virtual device. This command returns + when the emulator is fully started and ready to use + stop Stops the specified virtual device help Usage: android help [-h] [COMMAND] - Shows the help of all commands + Shows the help information for a specified command Options: - -h, --help Show the help message for this command + -h, --help Shows the help message for the specified command android options: --sdk=PARAM Path to the Android SDK @@ -271,12 +519,11 @@ Update the Android CLI using the `android update` command. info Usage: android info [-h] [] - Print environment information including SDK location, connected - devices, and configuration variables. Use specific fields to narrow - the output + Prints environment information including SDK location, connected + devices, and configuration variables Options: - -h, --help Show the help message for this command + -h, --help Shows the help message for the specified command android options: --sdk=PARAM Path to the Android SDK @@ -284,39 +531,64 @@ Update the Android CLI using the `android update` command. -V, --version Print version information and exit Positional Parameters: - The specific field to print the value of. If omitted print - all + The specific field to print the value of. If omitted, + prints all fields init Usage: android init [-h] - Initializes the environment for the Android CLI. Sets up required + Initializes the environment for Android CLI. Sets up required configurations, directories, and default skills Options: - -h, --help Show the help message for this command + -h, --help Shows the help message for the specified command android options: --sdk=PARAM Path to the Android SDK -v, --verbose Enable verbose output for troubleshooting -V, --version Print version information and exit + input + Usage: android input [-h] [--device=PARAM] [COMMAND] + Interact with Android apps by performing gestures and key events on UI + elements + + Options: + --device=PARAM Device serial number or emulator ID (as reported + by 'adb devices') + -h, --help Shows the help message for the specified command + + android options: + --sdk=PARAM Path to the Android SDK + -v, --verbose Enable verbose output for troubleshooting + -V, --version Print version information and exit + Commands: + key Press a hardware button, navigation key, or Android keycode + scroll Scroll a container element, optionally until a target + element appears + swipe Swipe from a UI element matching a selector in a given + direction + tap Tap on a UI element matching a selector + text Type text into a UI element matching a selector + install Usage: android install [-h] [--apks=PARAM] [--device=PARAM] [--install-options=PARAM] [--use-delta-install] - Install an Android Application (one or more APKs) to a connected - device or emulator without activating any components using incremental - optimizations for faster deployment than adb + Installs an Android app (one or more APKs) on a connected device or + emulator without activating any components, using incremental + optimizations for faster deployment than `adb` Options: --apks=PARAM The paths to the APKs, comma separated --device=PARAM The device serial number - -h, --help Show the help message for this command - --install-options=PARAM Additional options/flags to pass to - package manager install (e.g. -g,-d) + -h, --help Shows the help message for the + specified command + --install-options=PARAM Additional options or flags to pass to + package manager install (for example, + `-g`, `-d`) --use-delta-install Use fast delta install (speeds up incremental updates by transferring only modified code and resources; - default: true) + default: `true`) android options: --sdk=PARAM Path to the Android SDK @@ -327,16 +599,16 @@ Update the Android CLI using the `android update` command. layout Usage: android layout [-dhp] [--device=PARAM] [--flat] [--full] [--no-idle] [--output=PARAM] - Returns the layout tree of an application + Returns the layout tree of an app Options: --device=PARAM The device serial number -d, --diff Deprecated; no-op flag. Will be removed in a future release --flat Returns a flat list instead of a tree - --full Return the full tree, including non-interactive + --full Returns the full tree, including non-interactive and hidden elements - -h, --help Show the help message for this command + -h, --help Shows the help message for the specified command --no-idle Don't wait for layout idle state when fetching layout -o, --output=PARAM Writes the layout to the specified file or @@ -351,24 +623,26 @@ Update the Android CLI using the `android update` command. run Usage: android run [-h] [--activity=PARAM] [--apks=PARAM] [--debug] [--device=PARAM] [--install-options=PARAM] [--type=PARAM] [--use-delta-install] - Build, deploy, and launch an Android application on a connected device - or emulator + Deploys and launches an Android app on a connected device or emulator Options: --activity=PARAM The activity name --apks=PARAM The paths to the APKs, comma separated --debug Run in debug mode --device=PARAM The device serial number - -h, --help Show the help message for this command - --install-options=PARAM Additional options/flags to pass to - package manager install (e.g. -g,-d) - --type=PARAM The component type (ACTIVITY, - WATCH_FACE, TILE, COMPLICATION, - DECLARATIVE_WATCH_FACE, WEAR_WIDGET) + -h, --help Shows the help message for the + specified command + --install-options=PARAM Additional options or flags to pass to + package manager install (for example, + `-g`, `-d`) + --type=PARAM The component type (`ACTIVITY`, + `WATCH_FACE`, `TILE`, `COMPLICATION`, + `DECLARATIVE_WATCH_FACE`, + `WEAR_WIDGET`) --use-delta-install Use fast delta install (speeds up incremental updates by transferring only modified code and resources; - default: true) + default: `true`) android options: --sdk=PARAM Path to the Android SDK @@ -378,10 +652,11 @@ Update the Android CLI using the `android update` command. screen Usage: android screen [-h] [COMMAND] - Commands to view the device + Captures and inspects the screen of a connected Android device or + emulator Options: - -h, --help Show the help message for this command + -h, --help Shows the help message for the specified command android options: --sdk=PARAM Path to the Android SDK @@ -389,80 +664,89 @@ Update the Android CLI using the `android update` command. -V, --version Print version information and exit Commands: capture Outputs the device screen to a PNG - resolve Target UI elements visually. Substitutes bounding box - coordinates from a annotated screenshot into a string. - Replaces all instances of '#N' with the center coordinates - of the bounding box labeled 'N' + resolve Targets UI elements visually. Substitutes bounding box + coordinates from an annotated screenshot into a string, + replacing all instances of `#N` with the center coordinates + of the bounding box labeled `N` sdk - Usage: android sdk [-h] [--platform=PARAM] [COMMAND] - Manage the Android SDK installation. Includes commands to install, + Usage: android sdk [-h] [--ignore-outdated-xmls] [--platform=PARAM] + [COMMAND] + Manages the Android SDK installation. Includes commands to install, update, remove, and list available and installed SDK packages Options: - -h, --help Show the help message for this command - --platform=PARAM Target platform _ (e.g. - linux_x86_64, mac_arm64, windows_x86), - defaults to current host + -h, --help Shows the help message for the specified + command + --ignore-outdated-xmls When installing or updating packages, do + not update other packages' XML files + --platform=PARAM Target platform `_` (for + example, `linux_x86_64`, `mac_arm64`, or + `windows_x86`); defaults to the current + host android options: - --sdk=PARAM Path to the Android SDK - -v, --verbose Enable verbose output for troubleshooting - -V, --version Print version information and exit + --sdk=PARAM Path to the Android SDK + -v, --verbose Enable verbose output for + troubleshooting + -V, --version Print version information and exit Commands: - install Install SDK packages - update Update one or all packages to the latest version - remove Remove packages from the SDK - list List installed and available SDK packages + install Installs SDK packages + list Lists installed and available SDK packages + remove Removes packages from the SDK + update Updates one or all packages to the latest version skills Usage: android skills [-h] [COMMAND] - Manage Android CLI skills. Includes commands to install, remove, list, - and search for skills by keyword + Manages Android CLI skills. Includes commands to install, remove, + list, and search for skills by keyword Options: - -h, --help Show the help message for this command + -h, --help Shows the help message for the specified command android options: --sdk=PARAM Path to the Android SDK -v, --verbose Enable verbose output for troubleshooting -V, --version Print version information and exit Commands: - add Install a specific skill by its ID to your environment - remove Remove an installed skill by its ID - list List installed and available skills - find Search for available skills in the repository matching a + add Installs a specific skill by its ID to your environment + find Searches for available skills in the repository matching a keyword - update Update installed skills + list Lists installed and available skills + remove Removes an installed skill by its ID + update Updates installed skills studio Usage: android studio [-h] [COMMAND] - Android Studio commands + Connects Android CLI to a running Android Studio instance to analyze + files, find declarations and usages, render Compose previews, and look + up library versions Options: - -h, --help Show the help message for this command + -h, --help Shows the help message for the specified command android options: --sdk=PARAM Path to the Android SDK -v, --verbose Enable verbose output for troubleshooting -V, --version Print version information and exit Commands: - find-declaration Find declaration of a symbol - find-usages Find usages of a symbol - open-file Open a file in Android Studio - check Check the status of running Studio instances - analyze-file Analyze a file in Android Studio - render-compose-preview Render a Compose preview in Android Studio - version-lookup Looks up the latest available versions on - the internet of maven artifacts, Android - versions, and more + analyze-file Analyzes a file in Android Studio + check Checks the status of running Android Studio + instances + find-declaration Finds the declaration of a symbol + find-usages Finds usages of a symbol + open-file Opens a file in Android Studio + render-compose-preview Renders a Compose preview in Android Studio + version-lookup Looks up the latest available versions of + Maven artifacts, Android versions, and SDK + tools update Usage: android update [-h] [--url=PARAM] - Update the Android CLI to the latest version + Updates Android CLI to the latest version Options: - -h, --help Show the help message for this command + -h, --help Shows the help message for the specified command --url=PARAM The URL to download the update from android options: diff --git a/devtools/android-cli/references/interact.md b/devtools/android-cli/references/interact.md deleted file mode 100644 index 2f2a4c4..0000000 --- a/devtools/android-cli/references/interact.md +++ /dev/null @@ -1,113 +0,0 @@ -Run `android layout --help` and `android screen --help`. - -## UI Dump - -`android layout` returns a flat JSON list of the UI elements on screen. `android -layout --diff` returns a flat JSON list of the UI elements that have changed -since the last call to `layout` or `layout --diff` - -Each JSON object represents a UI element in the Android app. The following -properties may be present: - -- `text` - any literal text the element contains -- `resourceId` - the Android resource id used to refer to the element -- `contentDesc` - a description of a UI element for use by accessibility tools -- `interactions` - the set of user interactions the element supports. May contain one or more of: `checkable`, `clickable`, `focusable`, `scrollable`, `long-clickable`, `password` -- `state` - the set of states the element is in. May contain one or more of `checked`, `focused`, `selected` -- `bounds` - the screen coordinates of the bounding rectangle of the element, in the format `[min X,min Y][max X, max Y]` -- `center` - the screen coordinates of the center of the element, in the format `[x,y]` -- `off-screen` - if true, the element is in the UI hierarchy but not visible; it may require scrolling to view. - -Use `layout` as a primary means of examining an Android app. Use `layout --diff` -to focus on changes and to keep your context small. Example: When entering -digits into a calculator, use `layout --diff` to output only the digit readout -element. - -`layout` may fail due to the app displaying a WebView or animation; in these -cases, use `android screen capture --annotate` to inspect the app. This failure -will likely resolve after navigating away from the current screen. - -## Screenshot - -`android screen capture -o ` saves a PNG of the current device screen -to `` - -Use `screen capture` as a secondary means of examining an Android app -Examples: - -- Understanding the content of an on-screen image -- Looking at a `WebView` (web content does not always appear in the ui dump) -- Trying to find a UI element by its visual appearance - -**IMPORTANT** : Always *VISUALLY* examine the PNG image returned from `android -screen` BEFORE doing anything else. - -## Annotated Screenshot - -`android screen capture --annotate -o ` -`android screen resolve --screen --string ` - -The `--annotate` command adds numerical labels and bounding boxes around UI -elements. Use this command to locate UI elements that cannot be located in the -`layout` output. - -**IMPORTANT** : When using `android screen --annotate`, always *VISUALLY* examine -the resulting PNG file. - -To refer to these labels in input commands, use `screen resolve` to convert -labels into coordinates: - -`android screen resolve --screen --string "#3"` returns ` ` - -To save turns, you can combine shell commands: - -`adb shell input $(android screen resolve --screen screen.png --string "tap #34")` - -This command taps on region #34 from `screen.png` - -## Input - -Use `adb shell input` for interacting with Android devices. Refer to the -`"interactions"` property of an element for what interactions can be performed -on a particular element. - -Interact with UI elements with their `center` coordinate or their `bounds` -coordinates: -`json -{ -"key": -248568265, -"class": "android.widget.Button", -"bounds": "[138,9][167,38]", -"center": "[152,23]" -}` - -To tap on this button, you would execute `adb shell input tap 152 23`. This taps -the center. - - { - "key": 12487234, - "class": "com.example.ui.ScrollableList", - "bounds": "[100,200][400,600]", - "center": "[250,400]" - } - -To scroll down on this list, you would execute `adb shell input swipe 250 400 -250 200 500`. This swipes from the center to the top over 500ms. - -### Text Input - -To enter text, ensure the field is focused and execute `adb shell input text -""`. - -- **Spaces (`%s`):** Replace spaces with `%s` (Android ignores text after raw spaces): `adb shell input text "Hello%sworld"` (enters `"Hello world"`). -- **Special characters:** Escape shell metacharacters (`&`, `$`, `(`, `)`, `!`): `adb shell input text "AT\&T"` or `adb shell input text 'Price:\$10'`. -- **Submit / Enter:** `adb shell input keyevent 66` - -## Android Interaction Rules - -1. Always ensure text input fields have `"focused"` in their `"state"` list before entering text. -2. If an element has `"scrollable"` in its `"interactions"` list, try scrolling it when looking for missing UI elements. -3. Always scroll slowly when executing scroll inputs. The 5th argument to `adb - shell input swipe` controls scroll duration. -4. Content may take time to load; if a `layout` is missing information after you take an action, wait a few seconds, then perform `layout --diff` to see if anything changes. \ No newline at end of file diff --git a/devtools/android-cli/references/journeys.md b/devtools/android-cli/references/journeys.md deleted file mode 100644 index f3cc163..0000000 --- a/devtools/android-cli/references/journeys.md +++ /dev/null @@ -1,120 +0,0 @@ -A journey is an XML-specified test of an Android app's behavior. It consists of -a list of `` elements. For example: -`xml - - -A sample journey to illustrate the format - - - -Tap the "Home" icon - - -Verify that the app is on its Home screen - - -` - -Evaluate a journey by proceeding through the `` list in sequential -order. Evaluate each `` block individually. -A journey succeeds if all elements in the `` list succeed. - -A journey is a test case for an app. The journey XML is the source of truth; if -the app disagrees with the journey, the app has failed. -Additionally, if the app exits, crashes, or freezes, journey evaluation stops -and the journey fails. - -**IMPORTANT** - Execute each step EXACTLY as written, and independently of other -steps! If an action says to `"tap the first search result"`, -you MUST find the search results and tap the first one. Do this even if you -believe you know the intent behind the action. - -## Taking Actions - -Some `` elements specify UI interactions to perform on the running -Android app. Perform the interaction and verify that the app does -not crash or behave in an unexpected manner. This is the *only* verification -you should perform for an ``. - -If the interaction cannot be performed as specified, the journey fails. -Example: -`Click the red button` -If you determine a red button is not present in the UI, the journey fails. - -If the text of an `` specifies a list of actions, break it into -sub-actions and evaluate them individually: -Example: -`Search for soda and add the first result to the cart` -This should be evaluated as: -`Search for soda -Add the first result to the cart` - -If an `` contains something that is not a specification for a UI -interaction, alert the user that the journey is malformed and exit -early, specifying the error in question. - -## Verifying Expectations - -`` elements that begin with "check" or "verify" specify expectations -for the current state of the Android app. Determine the current -state of the app and check if the expectations are met. - -Determine the current state of the app by inspecting the current screen of the -device without interacting with it. -Example: -`Check if "Switch 2" is visible on the screen` -This requires only inspecting the current screen, not scrolling or interacting. -If "Switch 2" is not currently visible, the action fails. - -If the expectations are not met, mark the `` as a failure and the -journey evaluation ends. A single `` may contain -multiple expectations. -Example: -`Verify that the app is on the Home screen, the Home icon is blue, and the temperature is displayed` -This `` fails if ANY of the following are false: - -- The app is on the Home screen -- There is a Home icon, and it is blue -- A temperature is displayed - -## Handling failure - -When running a journey, evaluate it as a test. Failure is acceptable, and often -expected. Proper reporting of failures is the priority. - -Keep debugging and troubleshooting to a minimum; assume that tools are showing -you the correct output every time. The goal is to determine -if the *current* Android app can correctly handle the *current* steps outlined -in the journey. Suggestions for bug fixes, clarification, or -other improvements should be kept to journey evaluation summary at the end. - -## Summarizing - -After evaluating the whole journey, output a markdown (.md) file describing the -results. - -The file should have the following format: - - # Journey: - - ## Results - - - ### Action: Click the blue button ✅ - - **Commands**: - - - `adb shell input swipe 490 200 500 500 500` - - `adb shell input tap 45 920` - - - - **Comment**: The journey step doesn't specify that the button requires scrolling to see - - ### Action: Check if "Switch 2" is visible on the screen ✅ - - **Screenshot**: [switch_2_screenshot.png](path/to/screenshot) - - **Comment**: Confirmed both in layout and visually - - ### Action: The home screen is shown ❌ - - **Comment**: The settings page was shown - - - ### Action: Swipe down to view active notifications \ No newline at end of file diff --git a/jetpack-compose/adaptive/SKILL.md b/jetpack-compose/adaptive/SKILL.md index 64d8059..d878f36 100644 --- a/jetpack-compose/adaptive/SKILL.md +++ b/jetpack-compose/adaptive/SKILL.md @@ -1,24 +1,30 @@ --- name: adaptive description: Instructions to make or update an app's UI so that it adapts to different - Android devices including phones, tablets, foldables, laptops, desktop, TV, Auto + Android devices including phones, tablets, foldables, laptops, desktop, TV, Auto, and XR. It includes how to handle different window sizes, pointing devices (such as mouse) and text entry devices (such as keyboard) using the Compose MediaQuery - API. It also covers multi-pane layouts using Navigation3 Scenes, adaptive UI components - (such as buttons) with varying target sizes, and adaptive layouts (including navigation - areas - nav rails and nav bars) using the Compose Grid and FlexBox APIs. + API, and how to preserve UI, navigation, scroll, and media playback state across + configuration changes (such as folding/unfolding a foldable, rotating a device, + or resizing a window). It also covers multi-pane layouts using Navigation3 Scenes, + adaptive UI components (such as buttons) with varying target sizes, and adaptive + layouts (including navigation areas - nav rails and nav bars) using the Compose + Grid and FlexBox APIs. license: Complete terms in LICENSE.txt metadata: author: Google LLC - last-updated: '2026-10-08' + last-updated: '2026-10-16' keywords: - - android - - ui - - adaptive + - Jetpack Compose + - Adaptive UI + - Navigation + - Multi-pane + - Layouts + - Responsive Design - Grid - FlexBox - MediaQuery - - navigation + - Android Development --- ## Prerequisites @@ -38,6 +44,7 @@ task. - Step 3: Add multi-pane layouts - Step 4: Make vertical lists adaptive by changing the number of columns - Step 5: Hide app bars when scrolling +- Step 6: Preserve UI state across configuration changes ## Step 1. Verify current UI @@ -265,6 +272,148 @@ app bar state independently. There are two main scroll behaviors: - `exitUntilCollapsedScrollBehavior`: Hides on scroll down, stays hidden while you scroll up until you reach the very top (0 offset). - `enterAlwaysScrollBehavior`: Hides on scroll down, shows immediately on scroll up. +## Step 6: Preserve UI state across configuration changes + +IMPORTANT: **Never** add `android:configChanges` or `android:screenOrientation` +to `AndroidManifest.xml` to suppress Activity recreation or lock orientation. +Opting out of recreation hides state-loss bugs instead of fixing them, breaks +adaptive resource reloading, and prevents an app's process from restarting. + +When fixing state loss across configuration changes, systematically audit +**every** usage of `remember` across all screens, components, and state holders +(in other words, do not only search for `remember { mutableStateOf(...) }`): + +### Step 6.1. Preserve primitive and custom UI state in Composables + +- Replace `remember { mutableStateOf(...) }` with `rememberSaveable { mutableStateOf(...) }` for user input and interactive UI state, such as `TextField` draft text, search queries, scroll position, and toggle states. +- Key state on relevant identifiers when appropriate (e.g., `rememberSaveable(conversationId) { mutableStateOf("") }`). +- For custom data classes or collections (for example, a `Set`) that are not natively `Bundle`-saveable: + - When delegating a `MutableState` via `var state by rememberSaveable(...) { mutableStateOf(...) }`, pass your custom `Saver` using **`stateSaver`** (do **not** pass `saver = ...` when the block returns `MutableState`): + + +```kotlin +// this Saver implementation converts CustomState object which we don't know how to save +// to List which we can save +private val CustomStateSaver = Saver>( + save = { customState -> customState.values.map { it.value } }, + restore = { values -> CustomState(values.map { CustomStateValue.valueOf(it) }.toSet()) } +) + +@Composable +fun MyComposable() { + var customState by rememberSaveable(stateSaver = CustomStateSaver) { + mutableStateOf(CustomState()) + } +} +``` + +
+ +- Alternatively, annotate the class with `@Serializable` and use `rememberSerializable`. + +### Step 6.2. Preserve scroll state in lazy and scrollable layouts + +Plain `remember { LazyListState() }`, `remember { LazyGridState() }`, and +`remember { ScrollState(...) }` do **not** save scroll position across +configuration changes because they bypass the state class's `Saver`. + +Inspect all scrollable containers (`LazyColumn`, `LazyRow`, `LazyVerticalGrid`, +`LazyHorizontalGrid`, `LazyVerticalStaggeredGrid`, `Modifier.verticalScroll`, +and `Modifier.horizontalScroll`) across all screens and reusable components, and +replace plain `remember` instantiations with the built-in saveable factory +functions: + +- `remember { LazyListState() }` -\> `rememberLazyListState()` +- `remember { LazyGridState() }` -\> `rememberLazyGridState()` +- `remember { LazyStaggeredGridState() }` -\> `rememberLazyStaggeredGridState()` +- `remember { ScrollState(initial = 0) }` -\> `rememberScrollState()` + +### Step 6.3. Preserve Navigation 3 state and multiple back stacks + +Check out the Navigation 3 guide to [Save and manage navigation state](references/android/guide/navigation/navigation-3/save-state.md) +for details. + +### Step 6.4. Preserve media playback state (`ExoPlayer` / Media3) + +When an MediaPlayer (such as `ExoPlayer`) is created inside a composable with `remember`, it is +released on disposal and recreated from scratch on configuration change, +resetting playback to `0:00` and auto-playing even if the user had paused the +video. + +To preserve `ExoPlayer` playback state (`currentPosition` and `playWhenReady`): + +1. Store `savedPositionMs` (`mutableLongStateOf(0L)`) and `savedPlayWhenReady` (`mutableStateOf(true)`) using `rememberSaveable(videoResId)`. +2. **Keep saved state updated during playback --- do not only save in `onDispose`** : update `savedPlayWhenReady` and `savedPositionMs` in a `Player.Listener` (`onPlayWhenReadyChanged`, `onPositionDiscontinuity`) **and** periodically inside a `LaunchedEffect(player)` loop while active. +3. **Initialize the player with the restored state** : Set `playWhenReady = savedPlayWhenReady` and call `seekTo(savedPositionMs)` before `prepare()`. +4. **Guard `LifecycleStartEffect` so it does not auto-play a paused video or overwrite `savedPlayWhenReady`** : + - In `LifecycleStartEffect(player)`, only call `player.play()` when `savedPlayWhenReady` is `true` (otherwise a paused video will start playing again after restoration). + - In `onStopOrDispose`, capture `val wasPlaying = player.playWhenReady` and `savedPositionMs = player.currentPosition` before calling `player.pause()`, then restore `savedPlayWhenReady = wasPlaying` so that `player.pause()` on stop/dispose does not overwrite `savedPlayWhenReady` to `false`: + + +```kotlin +@UnstableApi +@Composable +private fun rememberExoPlayer(@RawRes videoResId: Int, fromUri: Unit.(Uri) -> MediaItem): ExoPlayer { + val context = LocalContext.current + var savedPositionMs by rememberSaveable(videoResId) { mutableLongStateOf(0L) } + var savedPlayWhenReady by rememberSaveable(videoResId) { mutableStateOf(true) } + + val player = remember(videoResId) { + ExoPlayer.Builder(context.applicationContext).build().apply { + setMediaItem(MediaItem.fromUri(RawResourceDataSource.buildRawResourceUri(videoResId))) + repeatMode = Player.REPEAT_MODE_ONE + playWhenReady = savedPlayWhenReady + seekTo(savedPositionMs) + prepare() + } + } + + DisposableEffect(player) { + val listener = object : Player.Listener { + override fun onPlayWhenReadyChanged(playWhenReady: Boolean, reason: Int) { + savedPlayWhenReady = playWhenReady + savedPositionMs = player.currentPosition + } + override fun onPositionDiscontinuity( + oldPosition: Player.PositionInfo, + newPosition: Player.PositionInfo, + reason: Int + ) { + savedPositionMs = newPosition.positionMs + } + } + player.addListener(listener) + onDispose { + player.removeListener(listener) + player.release() + } + } + + LaunchedEffect(player) { + while (isActive) { + savedPositionMs = player.currentPosition + delay(200) + } + } + + LifecycleStartEffect(player) { + if (savedPlayWhenReady) { + player.play() + } + onStopOrDispose { + val wasPlaying = player.playWhenReady + savedPositionMs = player.currentPosition + player.pause() + savedPlayWhenReady = wasPlaying + } + } + + return player +} +``` + +
+ ## Final step: Build and test Build the app and run the local tests. If the project has screenshot tests, run @@ -273,7 +422,7 @@ they have viewed the screenshot diffs. ## Additional documentation for experimental adaptive APIs -The following APIs are available from Compose 1.11.0-beta01. +The following APIs are available from Compose 1.11.0. ### FlexBox diff --git a/jetpack-compose/adaptive/references/android/guide/navigation/navigation-3/save-state.md b/jetpack-compose/adaptive/references/android/guide/navigation/navigation-3/save-state.md new file mode 100644 index 0000000..30cc06f --- /dev/null +++ b/jetpack-compose/adaptive/references/android/guide/navigation/navigation-3/save-state.md @@ -0,0 +1,128 @@ +The following sections describe strategies for saving your back stack and +storing state associated with entries on your back stack. + +## Save your back stack + +Ensuring your app's navigation state persists across various lifecycle events, +including configuration changes and process death, is crucial for a good user +experience. In Navigation 3, you own your back stack, so there aren't strict +guidelines on how you should create or save it. However, Navigation 3 does offer +a convenience method that provides you with a saveable back stack: +[`rememberNavBackStack`](https://developer.android.com/reference/kotlin/androidx/navigation3/runtime/rememberNavBackStack.composable). + +### Use `rememberNavBackStack` + +The `rememberNavBackStack` composable function is designed to create a back +stack that persists across configuration changes and process death. + +For `rememberNavBackStack` to function correctly, each key in your back stack +must adhere to specific requirements: + +- **Implement `NavKey` interface** : Every key in the back stack must implement the [`NavKey`](https://developer.android.com/reference/kotlin/androidx/navigation3/runtime/NavKey) interface. This acts as a marker interface that signals to the library that the key can be saved. +- **Have the `@Serializable` annotation** : In addition to implementing `NavKey`, your key classes and objects must be marked with the [`@Serializable`](https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-core/kotlinx.serialization/-serializable/) annotation. + +The following snippet shows a correct implementation of `rememberNavBackStack`: + + +```kotlin +@Serializable +data object Home : NavKey + +@Composable +fun NavBackStack() { + val backStack = rememberNavBackStack(Home) +} +``` + +
+ +#### Remember a back stack with subtypes of `NavKey` + +The `rememberNavBackStack` composable function returns a `NavBackStack`. +If your app defines its own subtype of `NavKey` from which all of its keys +inherit, you can preserve that typing by implementing a custom remember +function, as follows: + + +```kotlin +@Serializable +sealed interface MyAppNavKey : NavKey + +@Serializable +data object ScreenA: MyAppNavKey + +@Serializable +data class ScreenB(val id: String): MyAppNavKey + +@Composable +fun rememberMyAppNavBackStack(vararg elements: MyAppNavKey): NavBackStack { + return rememberSerializable(serializer = serializer()) { + NavBackStack(*elements) + } +} + +@Composable +fun MyApp() { + // defaultNavBackStack is NavBackStack + val defaultNavBackStack = rememberNavBackStack(ScreenA) + // myAppNavBackStack is NavBackStack + val myAppNavBackStack = rememberMyAppNavBackStack(ScreenA) +} +``` + +
+ +For more examples---including how to handle open polymorphism---see +[`NavBackStackSamples`](https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:navigation3/navigation3-runtime/samples/src/main/kotlin/androidx/navigation3/runtime/samples/NavBackStackSamples.kt). + +### Alternative: Storing in a `ViewModel` + +Another approach to managing your back stack is to store it in a `ViewModel`. +For persistence through process death when using a `ViewModel` or any other +custom storage, you need to: + +- **Ensure your keys are serializable** : Just like with `rememberNavBackStack`, your navigation keys must be serializable. +- **Handle serialization and deserialization manually** : You're responsible for manually saving the serialized representation of each key to, and deserializing it from, persistent storage (e.g., `SharedPreferences`, a database, or a file) when your app is going into the background or being restored. + +## Scoping `ViewModel` instances to `NavEntry` objects + +`ViewModels` are used to retain UI-related state across configuration changes, +such as screen rotations. By default, `ViewModels` are scoped to the nearest +`ViewModelStoreOwner`, which is typically your `Activity` or `Fragment`. + +However, you might want to scope a `ViewModel` to a specific `NavEntry` (i.e., a +specific screen or destination) on the back stack, rather than the entire +`Activity`. This ensures that the `ViewModel`'s state is retained only while +that particular `NavEntry` is part of the back stack, and is cleared when the +`NavEntry` is popped. + +The `androidx.lifecycle:lifecycle-viewmodel-navigation3` add-on library provides +a [`NavEntryDecorator`](https://developer.android.com/guide/navigation/navigation-3/naventrydecorators) that facilitates this. This decorator provides a +`ViewModelStoreOwner` for each `NavEntry`. When you create a `ViewModel` inside a +`NavEntry`'s content (e.g., using `viewModel()` in Compose), it is automatically +scoped to that specific `NavEntry`'s key on the back stack. This means the +`ViewModel` is created when the `NavEntry` is added to the back stack, and +cleared when it's removed. + +To use [`NavEntryDecorator`](https://developer.android.com/guide/navigation/navigation-3/naventrydecorators#apply-decorators) for scoping `ViewModel`s to `NavEntry`s, follow +these steps: + +1. Add the `androidx.lifecycle:lifecycle-viewmodel-navigation3` dependency to your `app/build.gradle.kts` file. +2. Add the default [`rememberSaveableStateHolderNavEntryDecorator()`](https://developer.android.com/reference/kotlin/androidx/navigation3/runtime/rememberSaveableStateHolderNavEntryDecorator.composable) to the list of `entryDecorators` when constructing a `NavDisplay`. +3. Add [`rememberViewModelStoreNavEntryDecorator()`](https://developer.android.com/reference/kotlin/androidx/lifecycle/viewmodel/navigation3/rememberViewModelStoreNavEntryDecorator.composable) to the list of `entryDecorators`. + + +```kotlin +NavDisplay( + entryDecorators = listOf( + // Add the default decorators for managing scenes and saving state + rememberSaveableStateHolderNavEntryDecorator(), + // Then add the view model store decorator + rememberViewModelStoreNavEntryDecorator() + ), + backStack = backStack, + entryProvider = entryProvider { }, +) +``` + +
\ No newline at end of file diff --git a/jetpack-compose/jetpack-compose-performance/SKILL.md b/jetpack-compose/jetpack-compose-performance/SKILL.md index 3a07a5b..d8ec13b 100644 --- a/jetpack-compose/jetpack-compose-performance/SKILL.md +++ b/jetpack-compose/jetpack-compose-performance/SKILL.md @@ -6,7 +6,7 @@ description: Diagnose and optimize Android Jetpack Compose UI code. Use this ski license: Complete terms in LICENSE.txt metadata: author: Google LLC - last-updated: '2026-10-13' + last-updated: '2026-10-14' keywords: - Jetpack Compose - performance diff --git a/jetpack-compose/jetpack-compose-performance/references/state-and-stability.md b/jetpack-compose/jetpack-compose-performance/references/state-and-stability.md index bc63fdb..a05bbaf 100644 --- a/jetpack-compose/jetpack-compose-performance/references/state-and-stability.md +++ b/jetpack-compose/jetpack-compose-performance/references/state-and-stability.md @@ -121,7 +121,9 @@ unoptimized recomposition loop. Text("Count: $count") // 1. State read in Composition Button(onClick = {}) { // 2. State write in Composition AFTER read (Backwards write!) - countability.kt + count++ + } + } ```
@@ -300,9 +302,9 @@ top of the block. ```kotlin val label = when (transition.currentState) { State.Idle -> "Idle" - State>.Running - "Running: ${transition.currentState}" - > State.Finished - "Finished: ${tre}" - }StateAndStability.kt + State.Running -> "Running: ${transition.currentState}" + State.Finished -> "Finished: ${transition.currentState}" + } ```
@@ -314,9 +316,9 @@ top of the block. val currentState = transition.currentState val label = when (currentState) { State.Idle -> "Idle" - State>.Running - "Running: $currentState" - > State.Finished - "Fte" - }StateAndStability.kt + State.Running -> "Running: $currentState" + State.Finished -> "Finished: $currentState" + } ```
@@ -358,7 +360,9 @@ In Compose 1.13+, **`computedStateOf`** is introduced alongside if (showButton) { FloatingActionButton(onClick = { /* ... */ }) { Icon(Icons.Default.ArrowUpward, "Scroll to Top") - ability.kt + } + } + } ```
@@ -645,7 +649,8 @@ variable or parameter referenced inside the `remember` block as a key. val formattedDate = remember { SimpleDateFormat("yyyy-MM-dd", locale).format(Date(timestamp)) } - Text(text = formatability.kt + Text(text = formattedDate) + } ```
@@ -659,7 +664,8 @@ variable or parameter referenced inside the `remember` block as a key. val formattedDate = remember(timestamp, locale) { SimpleDateFormat("yyyy-MM-dd", locale).format(Date(timestamp)) } - Text(text = formatability.kt + Text(text = formattedDate) + } ```
\ No newline at end of file diff --git a/jetpack-compose/theming/styles/SKILL.md b/jetpack-compose/theming/styles/SKILL.md index 2af7e49..af6b808 100644 --- a/jetpack-compose/theming/styles/SKILL.md +++ b/jetpack-compose/theming/styles/SKILL.md @@ -8,7 +8,7 @@ description: Use this skill to integrate the Jetpack Compose Styles API into an license: Complete terms in LICENSE.txt metadata: author: Google LLC - last-updated: '2026-09-08' + last-updated: '2026-10-20' keywords: - Jetpack Compose - Styles diff --git a/jetpack-compose/theming/styles/references/android/develop/ui/compose/designsystems/custom.md b/jetpack-compose/theming/styles/references/android/develop/ui/compose/designsystems/custom.md index 59dbb25..6f8a5c0 100644 --- a/jetpack-compose/theming/styles/references/android/develop/ui/compose/designsystems/custom.md +++ b/jetpack-compose/theming/styles/references/android/develop/ui/compose/designsystems/custom.md @@ -6,7 +6,7 @@ the same manner. There are several approaches you might take: - [Extend `MaterialTheme`](https://developer.android.com/develop/ui/compose/designsystems/custom#extending-material) with additional theming values. -- [Replace one or more Material systems](https://developer.android.com/develop/ui/compose/designsystems/custom#replacing-systems) --- `Colors`, `Typography`, or `Shapes` --- with custom implementations while keeping the others. +- [Replace one or more Material systems](https://developer.android.com/develop/ui/compose/designsystems/custom#replacing-systems) --- `ColorScheme`, `Typography`, or `Shapes` --- with custom implementations while keeping the others. - [Implement a fully custom design system](https://developer.android.com/develop/ui/compose/designsystems/custom#implementing-fully-custom) to replace `MaterialTheme`. You may also want to continue using Material components with a custom design @@ -156,8 +156,8 @@ fun ExtendedApp() { ## Replace Material subsystems Instead of extending Material Theming, you may want to replace one or more -systems --- `Colors`, `Typography`, or `Shapes` --- with a custom implementation, -while maintaining the others. +systems --- `ColorScheme`, `Typography`, or `Shapes` --- with a custom +implementation, while maintaining the others. Suppose you want to replace the type and shape systems while keeping the color system: @@ -286,9 +286,9 @@ fun ReplacementApp() { You may want to replace Material Theming with a fully custom design system. Consider that `MaterialTheme` provides the following systems: -- `Colors`, `Typography`, and `Shapes`: Material Theming systems +- `ColorScheme`, `Typography`, and `Shapes`: Material Theming systems - `TextSelectionColors`: Colors used for text selection by `Text` and `TextField` -- `Ripple` and `RippleTheme`: Material implementation of `Indication` +- `ripple`, provided as `LocalIndication`: Material implementation of `Indication` If you want to continue using Material components, you must replace some of these systems in your custom themes or handle the systems in your @@ -443,7 +443,7 @@ val ButtonShape = RoundedCornerShape(percent = 50)
> [!NOTE] -> **Note:** `Button` uses `rememberRipple()` internally to provide a `Ripple` `Indication`. It's a good idea to check the source code when implementing other custom components that wrap existing components. +> **Note:** `Button` uses `ripple` internally, through `Surface`, as its `Indication`. To change the ripple color or alpha, provide `LocalRippleConfiguration`. Starting with `material3` 1.5.0-beta01, you can also provide `LocalRippleThemeConfiguration` to configure the ripple theme for all Material components, such as using an inset focus ring. All of these are in `androidx.compose.material3`. For more information, see [Migrate to `Indication` and `Ripple` APIs](https://developer.android.com/develop/ui/compose/touch-input/user-interactions/migrate-indication-ripple). It's a good idea to check the source code when implementing other custom components that wrap existing components. If you've introduced new class types --- such as `List` to represent gradients --- then it may be better to implement components from scratch instead diff --git a/media/media3-cast-integration/SKILL.md b/media/media3-cast-integration/SKILL.md index d4ce98a..02a3049 100644 --- a/media/media3-cast-integration/SKILL.md +++ b/media/media3-cast-integration/SKILL.md @@ -8,7 +8,7 @@ description: Implements Google Cast support in Android apps using Jetpack Media3 license: Complete terms in LICENSE.txt metadata: author: Google LLC - last-updated: '2026-09-16' + last-updated: '2026-10-14' keywords: - Android - Media3 diff --git a/media/media3-cast-integration/references/android/media/media3/cast/create-castplayer.md b/media/media3-cast-integration/references/android/media/media3/cast/create-castplayer.md index 0099bc5..10ac8f5 100644 --- a/media/media3-cast-integration/references/android/media/media3/cast/create-castplayer.md +++ b/media/media3-cast-integration/references/android/media/media3/cast/create-castplayer.md @@ -236,7 +236,7 @@ override fun onCreateOptionsMenu(menu: Menu): Boolean { Futures.addCallback( menuItemFuture, object : FutureCallback { - override fun onSuccess(menuItem: MenuItem?) { + override fun onSuccess(menuItem: MenuItem) { // Do something with the menu item. } diff --git a/navigation/navigation-3/SKILL.md b/navigation/navigation-3/SKILL.md index a7c77e7..7fc53e6 100644 --- a/navigation/navigation-3/SKILL.md +++ b/navigation/navigation-3/SKILL.md @@ -8,7 +8,7 @@ description: Learn how to install and migrate to Jetpack Navigation 3, and how t license: Complete terms in LICENSE.txt metadata: author: Google LLC - last-updated: '2026-10-13' + last-updated: '2026-10-17' keywords: - recipe - Android diff --git a/navigation/navigation-3/references/android/guide/navigation/navigation-3/recipes/deeplinks-custommatcher.md b/navigation/navigation-3/references/android/guide/navigation/navigation-3/recipes/deeplinks-custommatcher.md index fcfea49..f30ba27 100644 --- a/navigation/navigation-3/references/android/guide/navigation/navigation-3/recipes/deeplinks-custommatcher.md +++ b/navigation/navigation-3/references/android/guide/navigation/navigation-3/recipes/deeplinks-custommatcher.md @@ -44,7 +44,6 @@ import androidx.navigation3.runtime.NavKey import androidx.navigation3.runtime.deeplink.DeepLinkMatcher import androidx.navigation3.runtime.deeplink.DeepLinkRequest import androidx.navigation3.runtime.deeplink.RequestExtrasKey -import androidx.navigation3.runtime.deeplink.get import com.example.nav3recipes.common.deeplink.EntryScreen import com.example.nav3recipes.ui.setEdgeToEdgeConfig import kotlinx.serialization.KSerializer diff --git a/navigation/navigation-3/references/android/guide/navigation/navigation-3/recipes/modular-hilt.md b/navigation/navigation-3/references/android/guide/navigation/navigation-3/recipes/modular-hilt.md index fcc94a2..5dd8403 100644 --- a/navigation/navigation-3/references/android/guide/navigation/navigation-3/recipes/modular-hilt.md +++ b/navigation/navigation-3/references/android/guide/navigation/navigation-3/recipes/modular-hilt.md @@ -6,16 +6,18 @@ This recipe demonstrates how to structure a multi-module application using Navig The application is divided into several modules: -- **`app` module** : This is the main application module. It initializes a common `Navigator` and injects a set of `EntryProviderInstaller`s from the feature modules. It then uses these installers to build the final `entryProvider` for the `NavDisplay`. +- **`app` module** : This is the main application module. It provides `HiltDeepLinkActivity` (an entry point demonstrating regular and deep link intent launches), `HiltModularActivity` (which hosts the `NavDisplay` and delegates deep link intents to `Navigator.handleDeepLink(intent)`), the start destination binding, and builds the final `entryProvider`. - **`common` module**: This module contains the core navigation logic, including: - - A `Navigator` class that manages the back stack. + - A `Navigator` class that manages the back stack, resolves incoming deep link intents, and builds synthetic back stacks via `handleDeepLink(intent)`. + - A `startDestination: NavKey` used to initialize the back stack. - An `EntryProviderInstaller` type, which is a function that feature modules use to contribute their navigation entries to the application's `entryProvider`. + - `BackStackMatcher<*, NavKey>`, which feature modules use to contribute their deep link matchers with synthetic back stacks. - **Feature modules (e.g., `conversation`, `profile`)**: Each feature is split into two sub-modules: - - **`api` module**: Defines the public API for the feature, including its navigation routes. This allows other modules to navigate to this feature without needing to know about its implementation details. - - **`impl` module** : Provides the implementation of the feature, including its composables and an `EntryProviderInstaller` that maps the feature's routes to its composables. This installer is then provided to the `app` module using Dagger/Hilt. + - **`api` module**: Defines the public API for the feature, including its navigation routes and deep link URI patterns. This allows other modules to navigate to this feature without needing to know about its implementation details. + - **`impl` module** : Provides the implementation of the feature, including its composables, an `EntryProviderInstaller` that maps the feature's routes to its composables, and a `BackStackMatcher<*, NavKey>` configured with `.withBackStack` to establish the appropriate hierarchical back stack (e.g., `[ConversationList, ConversationDetail(id)]`). Both are provided using Dagger/Hilt multibindings (`@IntoSet`). This modular approach allows for a clean separation of concerns, making the codebase more scalable and maintainable. Each feature is responsible for its own navigation logic, and the `app` module only combines these pieces together. [![](https://developer.android.com/static/images/picto-icons/code.svg) Explore View the full recipe on GitHub.](https://github.com/android/nav3-recipes/tree/main/app/src/main/java/com/example/nav3recipes/modular/hilt) @@ -34,14 +36,23 @@ import androidx.compose.runtime.Composable import androidx.compose.ui.Alignment import androidx.compose.ui.Modifier import androidx.compose.ui.unit.dp +import androidx.navigation3.runtime.NavKey +import androidx.navigation3.runtime.deeplink.BackStackMatcher +import androidx.navigation3.runtime.deeplink.DeepLinkUri +import androidx.navigation3.runtime.deeplink.UriDeepLinkMatcher +import androidx.navigation3.runtime.deeplink.withBackStack import dagger.Module import dagger.Provides import dagger.hilt.InstallIn import dagger.hilt.android.components.ActivityRetainedComponent import dagger.multibindings.IntoSet +import kotlinx.serialization.Serializable // API -object Profile +@Serializable +object Profile : NavKey + +const val PROFILE_URI = "https://www.nav3recipes.com/profile" // IMPLEMENTATION @Module @@ -55,6 +66,16 @@ object ProfileModule { ProfileScreen() } } + + @IntoSet + @Provides + fun provideDeepLinkMatcher(): BackStackMatcher<*, NavKey> = + UriDeepLinkMatcher( + uriPattern = DeepLinkUri(PROFILE_URI), + serializer = Profile.serializer(), + ).withBackStack { matchResult -> + listOf(ConversationList, matchResult.key) + } } @Composable @@ -104,6 +125,7 @@ class HiltModularActivity : ComponentActivity() { override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) setEdgeToEdgeConfig() + navigator.handleDeepLink(intent) setContent { Scaffold { paddingValues -> NavDisplay( @@ -144,20 +166,30 @@ import androidx.compose.ui.Modifier import androidx.compose.ui.graphics.Color import androidx.compose.ui.unit.dp import androidx.lifecycle.compose.dropUnlessResumed +import androidx.navigation3.runtime.NavKey +import androidx.navigation3.runtime.deeplink.BackStackMatcher +import androidx.navigation3.runtime.deeplink.DeepLinkUri +import androidx.navigation3.runtime.deeplink.UriDeepLinkMatcher +import androidx.navigation3.runtime.deeplink.withBackStack import com.example.nav3recipes.ui.theme.colors import dagger.Module import dagger.Provides import dagger.hilt.InstallIn import dagger.hilt.android.components.ActivityRetainedComponent import dagger.multibindings.IntoSet +import kotlinx.serialization.Serializable // API -object ConversationList -data class ConversationDetail(val id: Int) { +object ConversationList : NavKey + +@Serializable +data class ConversationDetail(val id: Int) : NavKey { val color: Color get() = colors[id % colors.size] } +const val CONVERSATION_URI = "https://www.nav3recipes.com/conversation" + // IMPL @Module @InstallIn(ActivityRetainedComponent::class) @@ -175,9 +207,21 @@ object ConversationModule { ) } entry { key -> - ConversationDetailScreen(key) { navigator.goTo(Profile) } + ConversationDetailScreen(key) { + navigator.goTo(Profile) + } } } + + @IntoSet + @Provides + fun provideDeepLinkMatcher(): BackStackMatcher<*, NavKey> = + UriDeepLinkMatcher( + uriPattern = DeepLinkUri("$CONVERSATION_URI/{id}"), + serializer = ConversationDetail.serializer(), + ).withBackStack { matchResult -> + listOf(ConversationList, matchResult.key) + } } @Composable @@ -241,23 +285,43 @@ private fun ConversationDetailScreen( ``` package com.example.nav3recipes.modular.hilt +import android.content.Intent import androidx.compose.runtime.mutableStateListOf import androidx.compose.runtime.snapshots.SnapshotStateList import androidx.navigation3.runtime.EntryProviderScope +import androidx.navigation3.runtime.NavKey +import androidx.navigation3.runtime.deeplink.BackStackMatcher +import androidx.navigation3.runtime.deeplink.DeepLinkRequest +import androidx.navigation3.runtime.deeplink.invoke import dagger.hilt.android.scopes.ActivityRetainedScoped +import javax.inject.Inject - -typealias EntryProviderInstaller = EntryProviderScope.() -> Unit +typealias EntryProviderInstaller = EntryProviderScope.() -> Unit @ActivityRetainedScoped -class Navigator(startDestination: Any) { - val backStack : SnapshotStateList = mutableStateListOf(startDestination) +class Navigator @Inject constructor( + startDestination: NavKey, + private val deepLinkMatchers: Set<@JvmSuppressWildcards BackStackMatcher<*, NavKey>>, +) { + val backStack: SnapshotStateList = mutableStateListOf(startDestination) - fun goTo(destination: Any){ + fun goTo(destination: NavKey) { backStack.add(destination) } - fun goBack(){ + fun handleDeepLink(intent: Intent) { + if (intent.data == null && intent.extras == null) return + + val request = DeepLinkRequest(intent) + val matchResult = deepLinkMatchers.firstNotNullOfOrNull { matcher -> + matcher.match(request) + } ?: return + + backStack.clear() + backStack.addAll(matchResult.backStack) + } + + fun goBack() { backStack.removeLastOrNull() } } @@ -266,18 +330,17 @@ class Navigator(startDestination: Any) { ``` package com.example.nav3recipes.modular.hilt +import androidx.navigation3.runtime.NavKey import dagger.Module import dagger.Provides import dagger.hilt.InstallIn import dagger.hilt.android.components.ActivityRetainedComponent -import dagger.hilt.android.scopes.ActivityRetainedScoped @Module @InstallIn(ActivityRetainedComponent::class) object AppModule { @Provides - @ActivityRetainedScoped - fun provideNavigator() : Navigator = Navigator(startDestination = ConversationList) + fun provideStartDestination(): NavKey = ConversationList } ``` \ No newline at end of file diff --git a/navigation/navigation-3/references/android/guide/navigation/navigation-3/recipes/modular-koin.md b/navigation/navigation-3/references/android/guide/navigation/navigation-3/recipes/modular-koin.md index d1e7ad6..32bbd05 100644 --- a/navigation/navigation-3/references/android/guide/navigation/navigation-3/recipes/modular-koin.md +++ b/navigation/navigation-3/references/android/guide/navigation/navigation-3/recipes/modular-koin.md @@ -6,14 +6,17 @@ This recipe demonstrates how to structure a multi-module application using Navig The application is divided into several Android modules: -- **`app` module** : This is the main application module. It `includes()` the feature modules and initializes a common `Navigator`. +- **`app` module** : This is the main application module. It provides `KoinDeepLinkActivity` (an entry point demonstrating regular and deep link intent launches), `KoinModularActivity` (which hosts the `NavDisplay` and delegates deep link intents to `Navigator.handleDeepLink(intent)`), and initializes a common `Navigator` (resolving all registered `BackStackMatcher<*, NavKey>`s via `getAll()`). -- **`common` module** : This module contains the core navigation logic used by both the application module and the feature modules. Namely, it defines a `Navigator` class that manages the back stack. +- **`common` module**: This module contains the core navigation logic used by both the application module and the feature modules, including: + - A `Navigator` class that manages the back stack, resolves incoming deep link intents, and builds synthetic back stacks via `handleDeepLink(intent)`. + - A `startDestination: NavKey` used to initialize the back stack. + - `BackStackMatcher<*, NavKey>`, which feature modules use to contribute their deep link matchers with synthetic back stacks. - **Feature modules (e.g., `conversation`, `profile`)**: Each feature is split into two sub-modules: - - **`api` module**: Defines the public API for the feature, including its navigation routes. This allows other modules to navigate to this feature without needing to know about its implementation details. - - **`impl` module** : Provides the implementation of the feature, including its composables and Koin `Module`. The Koin module uses the [`navigation`](https://insert-koin.io/docs/reference/koin-compose/navigation3/#declaring-navigation-entries) DSL to define the entry provider installers for the feature module. + - **`api` module**: Defines the public API for the feature, including its navigation routes and deep link URI patterns. This allows other modules to navigate to this feature without needing to know about its implementation details. + - **`impl` module** : Provides the implementation of the feature, including its composables and Koin `Module`. The Koin module uses the [`navigation`](https://insert-koin.io/docs/reference/koin-compose/navigation3/#declaring-navigation-entries) DSL to define the entry provider installers and `single>(named<...>())` configured with `.withBackStack` to establish the appropriate hierarchical back stack (e.g., `[ConversationList, ConversationDetail(id)]`). This modular approach allows for a clean separation of concerns, making the codebase more scalable and maintainable. Each feature is responsible for its own navigation logic, and the `app` module only combines these pieces together. [![](https://developer.android.com/static/images/picto-icons/code.svg) Explore View the full recipe on GitHub.](https://github.com/android/nav3-recipes/tree/main/app/src/main/java/com/example/nav3recipes/modular/koin) @@ -32,13 +35,23 @@ import androidx.compose.runtime.Composable import androidx.compose.ui.Alignment import androidx.compose.ui.Modifier import androidx.compose.ui.unit.dp +import androidx.navigation3.runtime.NavKey +import androidx.navigation3.runtime.deeplink.BackStackMatcher +import androidx.navigation3.runtime.deeplink.DeepLinkUri +import androidx.navigation3.runtime.deeplink.UriDeepLinkMatcher +import androidx.navigation3.runtime.deeplink.withBackStack +import kotlinx.serialization.Serializable import org.koin.androidx.scope.dsl.activityRetainedScope import org.koin.core.annotation.KoinExperimentalAPI +import org.koin.core.qualifier.named import org.koin.dsl.module import org.koin.dsl.navigation3.navigation // API -object Profile +@Serializable +object Profile : NavKey + +const val PROFILE_URI = "https://www.nav3recipes.com/profile" // IMPL @OptIn(KoinExperimentalAPI::class) @@ -46,6 +59,15 @@ val profileModule = module { activityRetainedScope { navigation { ProfileScreen() } } + + single>(named()) { + UriDeepLinkMatcher( + uriPattern = DeepLinkUri(PROFILE_URI), + serializer = Profile.serializer(), + ).withBackStack { matchResult -> + listOf(ConversationList, matchResult.key) + } + } } @Composable @@ -92,19 +114,30 @@ import androidx.compose.ui.Modifier import androidx.compose.ui.graphics.Color import androidx.compose.ui.unit.dp import androidx.lifecycle.compose.dropUnlessResumed +import androidx.navigation3.runtime.NavKey +import androidx.navigation3.runtime.deeplink.BackStackMatcher +import androidx.navigation3.runtime.deeplink.DeepLinkUri +import androidx.navigation3.runtime.deeplink.UriDeepLinkMatcher +import androidx.navigation3.runtime.deeplink.withBackStack import com.example.nav3recipes.ui.theme.colors +import kotlinx.serialization.Serializable import org.koin.androidx.scope.dsl.activityRetainedScope import org.koin.core.annotation.KoinExperimentalAPI +import org.koin.core.qualifier.named import org.koin.dsl.module import org.koin.dsl.navigation3.navigation // API -object ConversationList -data class ConversationDetail(val id: Int) { +object ConversationList : NavKey + +@Serializable +data class ConversationDetail(val id: Int) : NavKey { val color: Color get() = colors[id % colors.size] } +const val CONVERSATION_URI = "https://www.nav3recipes.com/conversation" + // IMPL @OptIn(KoinExperimentalAPI::class) val conversationModule = module { @@ -123,6 +156,15 @@ val conversationModule = module { } } } + + single>(named()) { + UriDeepLinkMatcher( + uriPattern = DeepLinkUri("$CONVERSATION_URI/{id}"), + serializer = ConversationDetail.serializer(), + ).withBackStack { matchResult -> + listOf(ConversationList, matchResult.key) + } + } } @Composable @@ -186,17 +228,37 @@ private fun ConversationDetailScreen( ``` package com.example.nav3recipes.modular.koin +import android.content.Intent import androidx.compose.runtime.mutableStateListOf import androidx.compose.runtime.snapshots.SnapshotStateList +import androidx.navigation3.runtime.NavKey +import androidx.navigation3.runtime.deeplink.BackStackMatcher +import androidx.navigation3.runtime.deeplink.DeepLinkRequest +import androidx.navigation3.runtime.deeplink.invoke + +class Navigator( + startDestination: NavKey, + private val deepLinkMatchers: List>, +) { + val backStack: SnapshotStateList = mutableStateListOf(startDestination) -class Navigator(startDestination: Any) { - val backStack : SnapshotStateList = mutableStateListOf(startDestination) - - fun goTo(destination: Any){ + fun goTo(destination: NavKey) { backStack.add(destination) } - fun goBack(){ + fun handleDeepLink(intent: Intent) { + if (intent.data == null && intent.extras == null) return + + val request = DeepLinkRequest(intent) + val matchResult = deepLinkMatchers.firstNotNullOfOrNull { matcher -> + matcher.match(request) + } ?: return + + backStack.clear() + backStack.addAll(matchResult.backStack) + } + + fun goBack() { backStack.removeLastOrNull() } } @@ -209,11 +271,14 @@ import org.koin.androidx.scope.dsl.activityRetainedScope import org.koin.dsl.module val appModule = module { - includes(profileModule,conversationModule) + includes(profileModule, conversationModule) activityRetainedScope { scoped { - Navigator(startDestination = ConversationList) + Navigator( + startDestination = ConversationList, + deepLinkMatchers = getAll(), + ) } } } @@ -271,6 +336,7 @@ class KoinModularActivity : ComponentActivity(), AndroidScopeComponent, KoinComp super.onCreate(savedInstanceState) setEdgeToEdgeConfig() + navigator.handleDeepLink(intent) setContent { Scaffold { paddingValues -> NavDisplay( diff --git a/performance/anr-audit-and-fix/SKILL.md b/performance/anr-audit-and-fix/SKILL.md new file mode 100644 index 0000000..926459e --- /dev/null +++ b/performance/anr-audit-and-fix/SKILL.md @@ -0,0 +1,35 @@ +--- +name: anr-audit-and-fix +description: Audits Android codebases for ANR risks and diagnoses ANR stack dumps. + Use this skill when scanning Android code for ANR risks or triaging ANR stack dumps. + Don't use for non-ANR crashes, dropped frames, general CPU profiling, or Perfetto + system traces. +license: Complete terms in LICENSE.txt +metadata: + author: Google LLC + last-updated: '2026-10-16' + keywords: + - Android + - ANR + - Application Not Responding + - responsiveness + - main thread + - ANR stack dump + - Sync Barrier Leak +--- + +Select the execution path matching the user's request: + +## 1. Path A: Static codebase ANR risk audit + +When auditing source files or scanning for thread-blocking operations: + +- Follow [references/risk_analysis_guide.md](references/risk_analysis_guide.md) step by step: list candidates, **prove the executing thread** (trace callers when the thread depends on them), **prove the call blocks**, then attribute and grade. Report only candidates that pass these checks; list the dismissed ones separately with a one-line reason. +- Evaluate off-main-thread mutations of attached Views not merely as crash risks (`CalledFromWrongThreadException`), but as **Sync Barrier Leak ANR** risks with their own proof (off-main execution plus an attached View). + +## 2. Path B: Trace-driven diagnosis and remediation + +When analyzing ANR traces or stack dumps: + +- Diagnose systematically using [references/trace_analysis_guide.md](references/trace_analysis_guide.md) and format the report with [references/anr_trace_report_template.md](references/anr_trace_report_template.md). +- Base findings strictly on log evidence. Propose application fixes **only** for `APP_DEFECT` or `APP_RESOURCE_OVERLOAD` (never for `EXTERNAL_BOTTLENECK`). diff --git a/performance/anr-audit-and-fix/references/anr_trace_report_template.md b/performance/anr-audit-and-fix/references/anr_trace_report_template.md new file mode 100644 index 0000000..8cce7e4 --- /dev/null +++ b/performance/anr-audit-and-fix/references/anr_trace_report_template.md @@ -0,0 +1,33 @@ + > **Overview**: **``**<>/span> + (``) -- `<1-line conclusion and exact trace evidence to inspect>` + + ### 1. Trace validity and context + - **Process**: `` (PID: ``, Foreground: ``) + - **ANR Trigger**: `` + - **Unresponsive Thread**: ` (ContentProvider) | (BroadcastReceiver)>` + - **User-Perceived ANR**: **``** (`ination`) + - **Capture Delay**: `` (`ypassed`) + + ### 2. System health assessment + - **Metrics**: `` + - **Verdict**: **``** + - **Contributing Factor**: `) | APP ()>` + + ### 3. Root cause analysis + - **Unresponsive Thread State**: `` + - **Blocking Chain / Mechanism**: `` + - **Secondary / Compounding Threads**: `` + - **Diagnosis**: **``** + + ### 4. Remediation + 1. **Code Fix**: `` + 2. **Prevention**: `` \ No newline at end of file diff --git a/performance/anr-audit-and-fix/references/risk_analysis_guide.md b/performance/anr-audit-and-fix/references/risk_analysis_guide.md new file mode 100644 index 0000000..ab6180c --- /dev/null +++ b/performance/anr-audit-and-fix/references/risk_analysis_guide.md @@ -0,0 +1,58 @@ +Goal: decide whether code **in the file under review** stalls the main thread +(or a Binder thread that a caller waits on synchronously) long enough to cause +an ANR, and point to the exact lines. Work through the steps in order. A +candidate becomes a finding only if it passes Steps 2, 3, and 5. The one +exception is Step 4 (View threading): its proof is off-main execution on an +attached View, followed by Step 5. + +## Step 1: List candidates in the file + +- **Blocking primitives and locks** : `runBlocking`, blocking futures, latches, monitors, `Thread.sleep`, RxJava `blocking*` operators, or locks contended with background work. +- **Synchronous I/O or heavy compute** : network, disk, file, or database reads and writes (`SharedPreferences.Editor.commit()`, `ContentResolver.query`, non-suspend DAO calls), bitmap decoding, JSON or XML payload parsing, `MediaPlayer.prepare()`, or unbounded loops. +- **Synchronous IPC or slow component lifecycle** : cross-process Binder or `ContentProvider` calls, or heavy work inside `Application`, `Activity`, `Service`, `BroadcastReceiver`, or `ContentProvider` lifecycle callbacks. +- **Off-main mutation of attached Views** (Step 4). + +## Step 2: Prove the executing thread (mandatory) + +Establish which thread actually executes each candidate rather than assuming: + +- **Inspect the enclosing scope** : check coroutine dispatchers (`lifecycleScope` and `viewModelScope` default to Main), reactive schedulers, executors or handlers, and Android component callbacks (`ContentProvider` queries run on a Binder thread cross-process, or the caller's thread in-process). +- **Trace callers when the thread depends on the caller**: locate callers in the project (1 to 2 hops) to verify the dispatching thread, and state the verified call chain. +- **Listeners and callbacks**: verify which thread or executor dispatches the callback before assuming it runs on a worker thread. +- **Dismiss unproven candidates**: if you can't show that the candidate runs on the main thread or a synchronously awaited Binder thread (or off the main thread for Step 4 candidates), list it under dismissed as "thread not established". + +## Step 3: Prove the call blocks + +- **Seeded in-memory reactive reads return immediately** : `blockingFirst()` or `getValue()` on a seeded `BehaviorRelay` or `BehaviorSubject`, `StateFlow.value`, or `first()` on a `StateFlow` / populated `SharedFlow` (`replay >= 1`) return in-memory state immediately. Check how the source is initialized before flagging. +- **Suspension isn't blocking** : suspend functions, `delay`, and coroutine `Mutex`/`Semaphore` suspend the coroutine without blocking the underlying thread. +- **`runBlocking` blocks only its calling thread** : `runBlocking` called on a background thread (even with `Dispatchers.Main` inside) doesn't block the main thread unless the main thread is waiting on that worker. +- **Short bounded waits aren't findings**: a single sleep or wait of roughly a frame to a few frames (tens of milliseconds to about a hundred), not in a loop and not on a hot path, causes minor frame drops, not an ANR. Don't report it as a finding; mention it under dismissed candidates at most. +- **Latent hazard** : a blocking primitive on the main thread whose inner work completes in memory without I/O or suspension is a `LOW` finding. Explain why it doesn't block today and what change would make it block. + +## Step 4: View threading violations (sync barrier leak) + +Treat off-main-thread UI access on an **attached** View as a **Sync Barrier +Leak ANR** risk, not merely a `CalledFromWrongThreadException` crash risk. + +- **Mechanism** : `ViewRootImpl.requestLayout()` calls `checkThread()`, but invalidation paths (`invalidate()` -\> `onDescendantInvalidated()` -\> `scheduleTraversals()`) don't, allowing a background thread to enter `scheduleTraversals()` concurrently with the main thread. Both call `mHandler.getLooper().getQueue().postSyncBarrier()` and overwrite the unsynchronized `mTraversalBarrier` token; `doTraversal()` removes only one token, leaking an orphaned synchronization barrier in `MessageQueue` (freezing synchronous main-thread messages between VSYNC frames or throwing `IllegalStateException` at `MessageQueue.removeSyncBarrier`). Android 17 fixed this platform race; Android 16 and older remain vulnerable, so apps shipping to those versions must still remove the violation. Creating or configuring a *detached* View off-main doesn't trigger this race. +- **What to check** : View or adapter mutations inside background coroutine scopes (`Dispatchers.Default`/`IO`), background Rx schedulers, executors or threads, or asynchronous callbacks invoked on worker threads (such as network, image-loading, media, sensor, or Binder listeners). +- **Detection** : Android 17's `View#registerCalledFromWrongThreadListener` (requires `compileSdk 37`; guard calls with `Build.VERSION.SDK_INT >= 37`), `adb shell am compat enable ENFORCE_THREAD_CHECKS_ON_VIEW_ROOT_IMPL_APIS + ` (debuggable builds), or the Robolectric unit-test flag `-Drobolectric.enforceViewMethodsCalledOnMainThread=true`. +- **Remediation** : fix the threading context at the **root scope** : + - *Coroutines* : start UI flows on `Dispatchers.Main` (`lifecycleScope` or `viewModelScope`) and wrap only slow I/O or compute work in `withContext(Dispatchers.IO)` or `Dispatchers.Default`. + - *Threads, executors, and callbacks* : keep in-memory UI reads and mutations on the main thread before submitting background work, or configure the upstream stream, callback executor, or target API to deliver results on the main thread (`AndroidSchedulers.mainThread()`, main-thread UI targets, or main executor). + +## Step 5: Attribute and grade + +- **Attribute to the file that owns the defect**: report a finding only if the blocking statement or off-main View mutation originates in the file under review (or in a concrete dependency that unconditionally blocks when called on the main thread from this file). If this file invokes an abstract callback or listener interface and an external caller or host class chooses to block inside its callback implementation, attribute the defect to that external file and dismiss it here as "owned by caller or host file". +- **Severity** : + - `HIGH`: main-thread I/O or long waits on primary paths (startup, lifecycle, input), or off-main mutation of attached Views. + - `MEDIUM`: main-thread blocking on secondary paths, or bounded work stalling for hundreds of milliseconds. + - `LOW`: latent hazards that complete in memory today, or Binder-thread blocking reached only from external processes. + +## Output + +1. One-line verdict first: `ANR risk: yes ()` or `ANR risk: + none found in this file`. +2. For each finding: line range, executing thread and verified call chain, why it blocks (or, for Step 4, why it leaks the sync barrier), severity, and suggested fix. +3. **Dismissed candidates**: one line each with the reason (for example, "thread not established", "seeded relay, returns immediately", "runs on IO through caller X", "owned by caller or host file"). \ No newline at end of file diff --git a/performance/anr-audit-and-fix/references/trace_analysis_guide.md b/performance/anr-audit-and-fix/references/trace_analysis_guide.md new file mode 100644 index 0000000..8248a0d --- /dev/null +++ b/performance/anr-audit-and-fix/references/trace_analysis_guide.md @@ -0,0 +1,77 @@ +Diagnose ANR traces using the following rules and format the output with +[references/anr_trace_report_template.md](https://developer.android.com/agents/skills/performance/anr-audit-and-fix/references/anr_trace_report_template). + +*** ** * ** *** + +## 1. Pre-flight context checks + +- **Process match** : confirm that the dumped process (`--- pid N at ...` / `Cmd line:` header) matches the ANR `Subject`. `Input dispatching timed out` subjects often carry no package or PID; record `unknown` rather than aborting. +- **Capture delay (`Immediate` vs. `Delayed Snapshot`)** : + - `Immediate`: dump captured at the ANR timestamp (the unresponsive thread's stack reflects the freeze). + - `Delayed Snapshot (unresponsive thread stack bypassed)`: significant lag between the ANR and dump timestamps, OR `Debug Store` records a slow main-thread `LooperMsg` that finished right before the dump while the unresponsive thread is now idle in `nativePollOnce`. **Don't dismiss as + bug-free or misclassify a completed slow transaction as a Sync Barrier + Leak.** + - `Debug Store` exists only on Android 15+ (absence on older devices is not a signal). Its `LooperMsg` entry logs slow (over 2 seconds) `ActivityThread$H` messages (such as Activity lifecycle transactions or `BIND_APPLICATION`), whereas receiver and service start or bind callbacks use separate `Debug + Store` tags and generic `Handler.post` runnables are not logged. A completed `LooperMsg` therefore directly identifies a slow main-thread callback: report `Indeterminate (Delayed Snapshot)` and attribute it using Section 2 (`APP_DEFECT` unless external starvation holds). +- **User-perceived ANR** : + - **`Yes` (`Foreground UI freeze / Foreground termination`)** : any `Input + dispatching timed out` ANR, OR any component, service, job, or startup ANR when **`Foreground: Yes`** (killing a foreground process terminates the user's visible app). + - **`No` (`Silent background termination`)** : background component or watchdog ANRs when **`Foreground: No`**. + +*** ** * ** *** + +## 2. Attribution verdict + +Always commit to a single primary **`Verdict`** (never output `"Shared"` or +`"Mixed"`). Record any secondary environmental or app contention in +**`Contributing Factor`** (`None | EXTERNAL () | APP +()`). + +The verdict's job is to tell the developer whether there is something in the +app to fix. Evaluate the verdicts **in the following order** and stop at the +first one whose evidence is present; record the lower-ranked explanation, if +any, as a one-line `Contributing Factor`. Precedence never promotes a verdict +without real app-side evidence. + +1. **`APP_DEFECT`** (application code, lock, lifecycle contract, or threading bug). Applies when the unresponsive thread (`tid=1`, the `binder:` thread for ContentProvider ANRs, or the custom-`Handler` or `goAsync()` worker thread for broadcast receivers) has a concrete app-side blocking dependency or contract violation, regardless of background system load: + - Direct I/O or heavy computation on the unresponsive thread, a synchronous wait on in-process or cross-process work (such as `Future.get()`, a latch, synchronous SDK initialization, an avoidable synchronous Binder call, a lock, monitor, or futex whose holder is **actively executing** app work on the CPU, or a deadlock where the holder waits on a resource the unresponsive thread owns), or a View threading violation that leaked a sync barrier (Section 3). This is `APP_DEFECT` even when the waited-on work is slow because of app-driven CPU, GC, or memory pressure; the primary fix is removing the blocking dependency and the overload goes in `Contributing Factor: APP ()`. *(Distinguish `Blocked` + on `<0x...>` from normal `Native` idle polling in + `MessageQueue.nativePollOnce`.)* + - **Guard** : a lock or GC wait whose target thread (the lock holder or GC daemon, when the app's own heap is not exhausted) is itself starved by external load (waiting on the run queue or in kernel I/O `state=D`) is not app-side evidence **only if the device is actually saturated** ; fall through to the following verdicts. If total CPU has headroom and the holder still can't get scheduled because the app runs it at a much lower priority than the waiting unresponsive thread, that is a priority inversion the app created: stay with `APP_DEFECT`. +2. **`APP_RESOURCE_OVERLOAD`** (system resource contention driven by the target app). Applies when device CPU, memory, or I/O is saturated (or thrashing on major page faults) AND the target app is a dominant resource driver (for example, a top CPU consumer alongside heavy major page faults, GC thrashing, or excessive thread creation in the hundreds). If the target app has a low share of total CPU while `kswapd0` or other processes dominate---even if cold-start code paging produces several hundred major faults or standard framework/SDK thread pools---fall through to `EXTERNAL_BOTTLENECK`. Other processes being loaded when the target app is a dominant driver does not downgrade this verdict; note them in `Contributing Factor: EXTERNAL ()`. +3. **`EXTERNAL_BOTTLENECK`** (`External Starvation`; target app is a passive victim). Applies only when neither verdict 1 nor verdict 2 holds: other processes or the kernel saturate CPU, memory, or I/O while the target app consumes negligible or low CPU and has no blocking app-side dependency (including cold-start class, library, or provider initialization stalled in disk wait or starved under system I/O and memory pressure, or app threads suspended in kernel cgroup freezer channels). + - **Strict no-fix rule** : when Verdict is `EXTERNAL_BOTTLENECK`, **never + propose app code fixes or defensive workarounds** . Fill both Section 4 fields with `N/A -- No application code change required (External system + bottleneck)`. + +*** ** * ** *** + +## 3. Root cause and special forensic patterns + +- **`Sync Barrier Leak` vs. `Indeterminate` vs. `Async Lifecycle Timeout`** (when `tid=1` is in `nativePollOnce`, `dispatchVsync`, or rendering VSYNC frames): + - *Exclusions first*: + - All threads suspended in kernel cgroup freezer channels -\> **`EXTERNAL_BOTTLENECK`** (`External Starvation`). + - `Debug Store` shows a completed slow `LooperMsg` covering the ANR window, or dump capture was significantly delayed -\> **`Indeterminate (Delayed + Snapshot)`** (with the Verdict from Section 2 evidence, or `APP_DEFECT` when no external or resource-overload evidence is present). + - Uncompleted async component contract (`BroadcastReceiver.goAsync()` without `PendingResult.finish()`, `JobService` timeout, or `SHORT_SERVICE` exceeding its deadline without `Service.stopSelf()`) -\> **`Async Lifecycle Timeout`** . Only claim this when the trace shows it (for example, `GoAsync` or job-stop records in `Debug Store`, or a `SHORT_SERVICE` timeout named in the `Subject:` line); **never infer an + async contract leak from an empty `Debug Store`**. + - *Mechanism* : `ViewRootImpl.requestLayout()` calls `checkThread()`, but invalidation paths (`invalidate()` -\> `onDescendantInvalidated()` -\> `scheduleTraversals()`) don't, allowing a background thread to enter `scheduleTraversals()` concurrently with the main thread. Both call `mHandler.getLooper().getQueue().postSyncBarrier()` and overwrite the unsynchronized `mTraversalBarrier` token; `doTraversal()` removes only the latest token, leaking an orphaned sync barrier (`Message.target == null`) in `MessageQueue`. Synchronous messages starve behind the barrier while asynchronous VSYNC frames continue to run. Android 17 fixed this platform race; Android 16 and older remain vulnerable, so check the trace's Android version when weighing this pattern. + - *ANR subject signatures* : background component timeouts (`Broadcast of + Intent`, `executing service`) and `Input dispatching timed out (Application + does not have a focused window)`, because the window-focus request is also a synchronous main-thread message that starves behind the leaked barrier. A no-focused-window subject alone is not evidence of a sync barrier (for example, if companion code or window state explicitly sets `FLAG_NOT_FOCUSABLE`); apply the criteria that follow. + - **`View Threading Violation (Sync Barrier Leak [Confirmed])`** : background thread caught mutating UI views/`ViewRootImpl` in the dump, OR `tid=1` is actively rendering a VSYNC frame (`doFrame -> doTraversal`) while a synchronous message (`SchRcv` in `Debug Store`) starves alongside a spinning background worker. + - **`View Threading Violation (Sync Barrier Leak [probable])`** : when the offending background thread has already exited, **don't** diagnose by elimination alone. Require **corroborating VSYNC rendering evidence** (such as `tid=1` caught in `dispatchVsync`, or elevated context switches and CPU ticks on `main` and `RenderThread` showing asynchronous rendering continued while the synchronous message starved), immediate capture, `tid=1` in `nativePollOnce` (idle or dispatching VSYNC), an empty or absent `Debug + Store`, and no lock, IPC, or `state=D` I/O wait on `tid=1`. + - *Remediation and detection* : confine all View mutations to the main thread at the **root scope** (launching on `Dispatchers.Main` or `lifecycleScope` and wrapping only slow I/O in `withContext(Dispatchers.IO)`). When the offending call site isn't on the stack, audit background callbacks and pinpoint it using Android 17's compatibility check (`adb shell am compat + enable ENFORCE_THREAD_CHECKS_ON_VIEW_ROOT_IMPL_APIS `, debuggable builds only), Android 17's `View#registerCalledFromWrongThreadListener` (requires `compileSdk 37`; guard calls with `Build.VERSION.SDK_INT >= 37`) for production telemetry, the Robolectric unit-test flag `-Drobolectric.enforceViewMethodsCalledOnMainThread=true`, or crash logs for `IllegalStateException` at `MessageQueue.removeSyncBarrier`. +- **Blocked or missing unresponsive thread** : + - If the unresponsive thread is `Blocked` on a lock, identify the holder thread and diagnose what it is doing (app work, GC, I/O, waiting on a lock the unresponsive thread owns, or a lower-priority thread being starved) rather than the waiting thread's own frames; the remediation follows from the holder's workload. + - If `tid=1` is absent from `DALVIK THREADS`, locate the main thread among the `(not attached)` entries in `DALVIK THREADS` (`sysTid == PID`) before concluding anything; its frames (such as VM teardown waiting on other threads) reveal the actual stall. +- **Stackless traces (stack-dump timeout or missing thread stacks)** : + - Distinguish using kernel `Waiting Channels` (`sysTid= ` lines) and `/proc` stats: + - **External starvation (`EXTERNAL_BOTTLENECK`)** : severe system I/O or memory pressure with the main thread (`sysTid == PID`) in kernel disk or file system wait and negligible app CPU. + - **Internal futex contention (`APP_DEFECT` / `Lock Contention`)** : the main thread and a convoy of app worker threads parked in `futex_*` wait channels while the app actively consumes CPU. + - **App-driven overload (`APP_RESOURCE_OVERLOAD`)**: the app dominates CPU or major page faults without an internal futex convoy. + - **ART `Signal Catcher` vs. custom `SIGQUIT` interception** : ART's built-in `Signal Catcher` thread normally waits in `sigtimedwait` / `do_sigtimedwait`. **Never** flag it alone; only flag custom `SIGQUIT` interception when an additional app thread also appears in a signal-wait channel. +- **Secondary or compounding threads (`Secondary / Compounding Threads`)** : + - Scan worker threads and report concurrent aggravators (or `None`): kernel disk sync (`state=D` in `fsync`/`fdatasync`), third-party JNI or C/C++ hooks in ART/GC/signal paths, custom signal-wait threads, unbounded thread pools, or thread priority abuse. \ No newline at end of file diff --git a/play/engage-sdk-integration/SKILL.md b/play/engage-sdk-integration/SKILL.md index 70dd857..24e957a 100644 --- a/play/engage-sdk-integration/SKILL.md +++ b/play/engage-sdk-integration/SKILL.md @@ -6,7 +6,7 @@ description: Helps developers integrate, debug, and resolve Play Engage SDK impl license: Complete terms in LICENSE.txt metadata: author: Google LLC - last-updated: '2026-10-12' + last-updated: '2026-10-17' keywords: - android - engage diff --git a/play/engage-sdk-integration/references/android/guide/playcore/engage/food.md b/play/engage-sdk-integration/references/android/guide/playcore/engage/food.md index 6c86ff9..0f6644a 100644 --- a/play/engage-sdk-integration/references/android/guide/playcore/engage/food.md +++ b/play/engage-sdk-integration/references/android/guide/playcore/engage/food.md @@ -36,6 +36,9 @@ Use the Engage SDK [Android skill](https://developer.android.com/tools/agents/an ## Integration detail +This section details the steps, data models, and terminology required to +integrate with the Engage SDK for food content. + ### Terminology This integration includes the following five cluster types: **Recommendation** , @@ -43,8 +46,8 @@ This integration includes the following five cluster types: **Recommendation** , - **Recommendation** clusters show personalized food-related suggestions from an individual developer partner. These recommendations can be personalized to the - user or generalized (for example, new on sale). Use them to surface recipes, - stores, dishes, groceries, and so on as you see fit. + user or generalized (for example, new on sale). Use them to feature items such + as recipes, stores, dishes, or groceries. - A Recommendation cluster can be made of `ProductEntity`, `StoreEntity`, or `RecipeEntity` listings, but not a mix of different entity types. @@ -122,7 +125,8 @@ the following entities for the Food category: 5. `FoodShoppingList` 6. `FoodReorderCluster` -The charts below outline available attributes and requirements for each type. +The following charts outline available attributes and requirements for each +type. #### `ProductEntity` @@ -264,7 +268,7 @@ to publish. #### Image specifications -Required specifications for image assets are listed below: +The following table lists the required specifications for image assets: | Aspect ratio | Minimum pixels | Recommended pixels | |---|---|---| @@ -402,24 +406,17 @@ required for `ServiceAvailabilityRequest` from the following table. ###### Conditional Service Availability Feature Some integrated apps request a special configuration that enables and disables -the Engage service intermittently in order to reduce their serving cost. This -intermittent content ingestion strategy, although possible, negatively affects -the user and the product -- stale content will not be presented and some surfaces -will not be served at all. +the Engage service intermittently to reduce serving costs. However, this +intermittent content ingestion strategy negatively affects the user experience: +stale content won't be presented, and some surfaces won't be served. Starting with v1.6.0, the Engage SDK allows checking availability for specific -cluster types. This provides more flexibility so that if the intermittent -content strategy was adopted by a given application, some cluster types can -follow that intermittent strategy while other cluster types are always enabled -(i.e. continuation clusters). - -If the Engage service should not be 'continuously' enabled on all supported -devices for whatever reason, and is configured for intermittent ingestion for -any set of devices, all continuation cluster publications (e.g. Food Shopping -Cart, Food Shopping List, and Reorder) will be still enabled by default -configuration, and the rest of the cluster types will be enabled and disabled -intermittently. If intermittent ingestion applies to you but this default -configuration is not suitable for your needs, please contact +cluster types. If your app uses intermittent ingestion: + +- **Continuation clusters** (such as Food Shopping Cart, Food Shopping List, and Reorder) remain enabled by default across all supported devices. +- **Other cluster types** (such as Recommendations) are enabled and disabled intermittently. + +If this default configuration isn't suitable for your needs, contact engage-developers@google.com. ##### For SDK versions prior to v1.6.0 (Deprecated) @@ -594,7 +591,7 @@ This API is used to publish a list of `FoodShoppingList` objects. client.publishFoodShoppingLists( PublishFoodShoppingListsRequest.Builder() .addFoodShoppingList( - FoodShoppingListEntity.Builder() + FoodShoppingList.Builder() ... .build()) .build()) @@ -604,7 +601,7 @@ This API is used to publish a list of `FoodShoppingList` objects. client.publishFoodShoppingLists( new PublishFoodShoppingListsRequest.Builder() .addFoodShoppingList( - new FoodShoppingListEntity.Builder() + new FoodShoppingList.Builder() ... .build()) .build()); @@ -785,7 +782,7 @@ with the status code **NOT_PUBLISHED_REQUIRES_SIGN_IN** .setStatusCode(AppEngagePublishStatusCode.NOT_PUBLISHED_REQUIRES_SIGN_IN) .build()); -#### `deleteRecommendationClusters` +#### `deleteRecommendationsClusters` This API is used to delete the content of Recommendation Clusters. @@ -794,11 +791,11 @@ This API is used to delete the content of Recommendation Clusters. ### Kotlin - client.deleteRecommendationClusters() + client.deleteRecommendationsClusters() ### Java - client.deleteRecommendationClusters(); + client.deleteRecommendationsClusters(); When the service receives the request, it removes the existing data from the Recommendation Clusters. In case of an error, the entire request is rejected diff --git a/play/engage-sdk-integration/references/android/guide/playcore/engage/healthandfitness.md b/play/engage-sdk-integration/references/android/guide/playcore/engage/healthandfitness.md index 2b0bf02..1b181d0 100644 --- a/play/engage-sdk-integration/references/android/guide/playcore/engage/healthandfitness.md +++ b/play/engage-sdk-integration/references/android/guide/playcore/engage/healthandfitness.md @@ -35,6 +35,9 @@ Use the Engage SDK [Android skill](https://developer.android.com/tools/agents/an ## Integration detail +This section details the steps, data models, and terminology required to +integrate with the Engage SDK for health and fitness content. + ### Terminology This integration includes the following three cluster types: **Recommendation** , @@ -43,7 +46,7 @@ This integration includes the following three cluster types: **Recommendation** - **Recommendation** clusters show personalized health and fitness suggestions from an individual developer partner. These recommendations can be personalized to the user or generalized (for example, trending fitness \& - health). Use these to surface articles or people related to health and + health). Use these to feature articles or people related to health and fitness. - A Recommendation cluster can be made of `ArticleEntity`, `PersonEntity`, or `EventEntity` but not a mix of different entity types. @@ -140,7 +143,8 @@ the following entities for the Health \& Fitness category: 4. `EventEntity` 5. `EventReservationEntity` -The charts below outline available attributes and requirements for each type. +The following charts outline available attributes and requirements for each +type. #### `GenericFeaturedEntity` @@ -188,7 +192,7 @@ The charts below outline available attributes and requirements for each type. | Popularity - Count | Optional | Indicate the number of followers or popularity value, for example - "3.7 M". **Note:** If both Count and Count Value are provided, Count will be used | String **Recommended text size: max 20 chars for count + label combined** | | Popularity - Count Value | Optional | The number of followers or popularity value. **Note:** Provide Count Value if your app doesn't want to handle logic on how a large number should be optimized for different display sizes. If both Count and Count Value are provided, Count will be used. | Long | | Popularity - Label | Optional | Indicate what the popularity label is. For example - "Likes". | String **Recommended text size: Max 20 chars for count + label combined** | -| Popularity - Visual | Optional | Indicate what the interaction is for. For example - Image showing Likes icon, Emojis. Can provide more than 1 image, though not all may not be shown on all form factors. **Note:** Must be Square 1:1 image | See [Image Specifications](https://developer.android.com/guide/playcore/engage/healthandfitness#image-specs) for guidance. | +| Popularity - Visual | Optional | Indicate what the interaction is for. For example - Image showing Likes icon, emoji. Can provide more than 1 image, though not all may not be shown on all form factors. **Note:** Must be Square 1:1 image | See [Image Specifications](https://developer.android.com/guide/playcore/engage/healthandfitness#image-specs) for guidance. | | Rating - Max value | Required | The maximum value of the rating scale. **Must be provided if current value of rating is also provided.** | Number \>= 0.0 | | Rating - Current value | Required | The current value of the rating scale. **Must be provided if maximum value of rating is also provided.** | Number \>= 0.0 | | Rating - Count | Optional | The count of the ratings for the entity. **Note:** Provide this field if your app controls how the count is displayed to the users. Use a concise string. For example, if the count is 1,000,000, consider using an abbreviation like 1M so that the count isn't truncated on smaller display sizes. | String | @@ -448,24 +452,17 @@ required for `ServiceAvailabilityRequest` from the following table. ###### Conditional Service Availability Feature Some integrated apps request a special configuration that enables and disables -the Engage service intermittently in order to reduce their serving cost. This -intermittent content ingestion strategy, although possible, negatively affects -the user and the product -- stale content will not be presented and some surfaces -will not be served at all. +the Engage service intermittently to reduce serving costs. However, this +intermittent content ingestion strategy negatively affects the user experience: +stale content won't be presented, and some surfaces won't be served. Starting with v1.6.0, the Engage SDK allows checking availability for specific -cluster types. This provides more flexibility so that if the intermittent -content strategy was adopted by a given application, some cluster types can -follow that intermittent strategy while other cluster types are always enabled -(i.e. continuation clusters). - -If the Engage service should not be 'continuously' enabled on all supported -devices for whatever reason, and is configured for intermittent ingestion for -any set of devices, all continuation cluster publications (e.g. Continue -Reading and Event Reservations) will be still enabled by default -configuration, and the rest of the cluster types will be enabled and disabled -intermittently. If intermittent ingestion applies to you but this default -configuration is not suitable for your needs, please contact +cluster types. If your app uses intermittent ingestion: + +- **Continuation clusters** (such as Continue Reading and Event Reservations) remain enabled by default across all supported devices. +- **Other cluster types** (such as Recommendations) are enabled and disabled intermittently. + +If this default configuration isn't suitable for your needs, contact engage-developers@google.com. ##### For SDK versions prior to v1.6.0 (Deprecated) @@ -570,8 +567,8 @@ This API is used to publish a list of `FeaturedCluster` objects. ### Java client.publishFeaturedCluster( - new PublishFeaturedClustersRequest.Builder() - .addFeaturedCluster( + new PublishFeaturedClusterRequest.Builder() + .setFeaturedCluster( new FeaturedCluster.Builder() .addEntity(entity1) .addEntity(entity2) @@ -755,7 +752,7 @@ to publish the Sign In Card then we recommend calling the .setStatusCode(AppEngagePublishStatusCode.NOT_PUBLISHED_REQUIRES_SIGN_IN) .build()); -#### `deleteRecommendationClusters` +#### `deleteRecommendationsClusters` This API is used to delete the content of Recommendation Clusters. @@ -764,11 +761,11 @@ This API is used to delete the content of Recommendation Clusters. ### Kotlin - client.deleteRecommendationClusters() + client.deleteRecommendationsClusters() ### Java - client.deleteRecommendationClusters(); + client.deleteRecommendationsClusters(); When the service receives the request, it removes the existing data from the Recommendation Clusters. In case of an error, the entire request is rejected and diff --git a/play/engage-sdk-integration/references/android/guide/playcore/engage/listen.md b/play/engage-sdk-integration/references/android/guide/playcore/engage/listen.md index 4a1240f..d76db7f 100644 --- a/play/engage-sdk-integration/references/android/guide/playcore/engage/listen.md +++ b/play/engage-sdk-integration/references/android/guide/playcore/engage/listen.md @@ -36,6 +36,9 @@ Use the Engage SDK [Android skill](https://developer.android.com/tools/agents/an ## Integration detail +This section details the steps, data models, and terminology required to +integrate with the Engage SDK for audio content. + ### Terminology This integration includes the following three cluster types: **Recommendation** , @@ -109,7 +112,8 @@ the following entities for the Listen category: 8. `LiveRadioStationEntity` 9. `AudiobookEntity` -The charts below outline available attributes and requirements for each type. +The following charts outline available attributes and requirements for each +type. #### `MusicAlbumEntity` @@ -299,7 +303,7 @@ of *Becoming* by Michelle Obama). #### Image specifications -Required specifications for image assets are listed below: +The following table lists the required specifications for image assets: | Aspect ratio | Requirement | Minimum pixels | Recommended pixels | |---|---|---|---| @@ -329,13 +333,12 @@ PNG, JPG, static GIF, WebP .setImageHeightInPixel(960) .setImageWidthInPixel(408) .build()) - .setPlayBackUri("https://play.google/album/play") - .setInfoPageUri("https://play.google/album/info") + .setPlayBackUri(Uri.parse("https://play.google/album/play")) + .setInfoPageUri(Uri.parse("https://play.google/album/info")) .setDescription("A description of this album.") .addArtist("Artist") .addGenre("Genre") .addMusicLabel("Label") - .addContentRating("Rating") .setSongsCount(960) .setReleaseDateEpochMillis(1633032895L) .setDurationMillis(1633L) @@ -469,24 +472,18 @@ required for `ServiceAvailabilityRequest` from the following table. ###### Conditional Service Availability Feature Some integrated apps request a special configuration that enables and disables -the Engage service intermittently in order to reduce their serving cost. This -intermittent content ingestion strategy, although possible, negatively affects -the user and the product -- stale content will not be presented and some surfaces -will not be served at all. +the Engage service intermittently to reduce serving costs. However, this +intermittent content ingestion strategy negatively affects the user experience: +stale content won't be presented, and some surfaces won't be served. Starting with v1.6.0, the Engage SDK allows checking availability for specific -cluster types. This provides more flexibility so that if the intermittent -content strategy was adopted by a given application, some cluster types can -follow that intermittent strategy while other cluster types are always enabled -(i.e. continuation clusters). - -If the Engage service should not be 'continuously' enabled on all supported -devices for whatever reason, and is configured for intermittent ingestion for -any set of devices, all continuation cluster publications (e.g. Continue -Listening) will be still enabled by default configuration, and the rest of -the cluster types will be enabled and disabled intermittently. If -intermittent ingestion applies to you but this default configuration is not -suitable for your needs, please contact engage-developers@google.com. +cluster types. If your app uses intermittent ingestion: + +- **Continuation clusters** (such as Continue Listening) remain enabled by default across all supported devices. +- **Other cluster types** (such as Recommendations) are enabled and disabled intermittently. + +If this default configuration isn't suitable for your needs, contact +engage-developers@google.com. ##### For SDK versions prior to v1.6.0 (Deprecated) @@ -772,7 +769,7 @@ with the status code **NOT_PUBLISHED_REQUIRES_SIGN_IN** .setStatusCode(AppEngagePublishStatusCode.NOT_PUBLISHED_REQUIRES_SIGN_IN) .build()); -#### `deleteRecommendationClusters` +#### `deleteRecommendationsClusters` This API is used to delete the content of Recommendation Clusters. @@ -781,11 +778,11 @@ This API is used to delete the content of Recommendation Clusters. ### Kotlin - client.deleteRecommendationClusters() + client.deleteRecommendationsClusters() ### Java - client.deleteRecommendationClusters(); + client.deleteRecommendationsClusters(); When the service receives the request, it removes the existing data from the Recommendation Clusters. In case of an error, the entire request is rejected diff --git a/play/engage-sdk-integration/references/android/guide/playcore/engage/otherverticals.md b/play/engage-sdk-integration/references/android/guide/playcore/engage/otherverticals.md index 6dec631..728e69c 100644 --- a/play/engage-sdk-integration/references/android/guide/playcore/engage/otherverticals.md +++ b/play/engage-sdk-integration/references/android/guide/playcore/engage/otherverticals.md @@ -26,6 +26,9 @@ Use the Engage SDK [Android skill](https://developer.android.com/tools/agents/an ## Integration detail +This section details the steps, data models, and terminology required to +integrate with the Engage SDK for other content categories. + ### Terminology This integration includes the following three cluster types: **Recommendation** , @@ -177,7 +180,8 @@ the following entities for the others category: 11. `TransportationReservationEntity` 12. `VehicleRentalReservationEntity` -The charts below outline available attributes and requirements for each type. +The following charts outline available attributes and requirements for each +type. #### `GenericFeaturedEntity` @@ -330,7 +334,7 @@ want to publish, such as a restaurant or a grocery store. | Popularity - Count | Optional | Indicate the number of followers or popularity value, for example - "3.7 M". **Note:** If both Count and Count Value are provided, Count will be used | String **Recommended text size: max 20 chars for count + label combined** | | Popularity - Count Value | Optional | The number of followers or popularity value. **Note:** Provide Count Value if your app doesn't want to handle logic on how a large number should be optimized for different display sizes. If both Count and Count Value are provided, Count will be used. | Long | | Popularity - Label | Optional | Indicate what the popularity label is. For example - "Likes". | String **Recommended text size: Max 20 chars for count + label combined** | -| Popularity - Visual | Optional | Indicate what the interaction is for. For example - Image showing Likes icon, Emojis. Can provide more than 1 image, though not all may not be shown on all form factors. **Note:** Must be Square 1:1 image | See [Image Specifications](https://developer.android.com/guide/playcore/engage/otherverticals#image-specs) for guidance. | +| Popularity - Visual | Optional | Indicate what the interaction is for. For example - Image showing Likes icon, emoji. Can provide more than 1 image, though not all may not be shown on all form factors. **Note:** Must be Square 1:1 image | See [Image Specifications](https://developer.android.com/guide/playcore/engage/otherverticals#image-specs) for guidance. | | Rating - Max value | Required | The maximum value of the rating scale. **Must be provided if current value of rating is also provided.** | Number \>= 0.0 | | Rating - Current value | Required | The current value of the rating scale. **Must be provided if maximum value of rating is also provided.** | Number \>= 0.0 | | Rating - Count | Optional | The count of the ratings for the entity. **Note:** Provide this field if your app wants to control how this is displayed to the users. Provide the concise string that can be displayed to the user. For example, if the count is 1,000,000, consider using abbreviations like 1M, so that it won't be truncated on smaller display sizes. | String | @@ -685,24 +689,17 @@ required for `ServiceAvailabilityRequest` from the following table. ###### Conditional Service Availability Feature Some integrated apps request a special configuration that enables and disables -the Engage service intermittently in order to reduce their serving cost. This -intermittent content ingestion strategy, although possible, negatively affects -the user and the product -- stale content will not be presented and some surfaces -will not be served at all. +the Engage service intermittently to reduce serving costs. However, this +intermittent content ingestion strategy negatively affects the user experience: +stale content won't be presented, and some surfaces won't be served. Starting with v1.6.0, the Engage SDK allows checking availability for specific -cluster types. This provides more flexibility so that if the intermittent -content strategy was adopted by a given application, some cluster types can -follow that intermittent strategy while other cluster types are always enabled -(i.e. continuation clusters). - -If the Engage service should not be 'continuously' enabled on all supported -devices for whatever reason, and is configured for intermittent ingestion for -any set of devices, all continuation cluster publications (e.g. Continue -Reading and Reservations) will be still enabled by default configuration, -and the rest of the cluster types will be enabled and disabled -intermittently. If intermittent ingestion applies to you but this default -configuration is not suitable for your needs, please contact +cluster types. If your app uses intermittent ingestion: + +- **Continuation clusters** (such as Continue Reading and Reservations) remain enabled by default across all supported devices. +- **Other cluster types** (such as Recommendations) are enabled and disabled intermittently. + +If this default configuration isn't suitable for your needs, contact engage-developers@google.com. ##### For SDK versions prior to v1.6.0 (Deprecated) @@ -807,8 +804,8 @@ This API is used to publish a list of `FeaturedCluster` objects. ### Java client.publishFeaturedCluster( - new PublishFeaturedClustersRequest.Builder() - .addFeaturedCluster( + new PublishFeaturedClusterRequest.Builder() + .setFeaturedCluster( new FeaturedCluster.Builder() .addEntity(entity1) .addEntity(entity2) @@ -992,7 +989,7 @@ to publish the Sign In Card then we recommend calling the .setStatusCode(AppEngagePublishStatusCode.NOT_PUBLISHED_REQUIRES_SIGN_IN) .build()); -#### `deleteRecommendationClusters` +#### `deleteRecommendationsClusters` This API is used to delete the content of Recommendation Clusters. @@ -1001,11 +998,11 @@ This API is used to delete the content of Recommendation Clusters. ### Kotlin - client.deleteRecommendationClusters() + client.deleteRecommendationsClusters() ### Java - client.deleteRecommendationClusters(); + client.deleteRecommendationsClusters(); When the service receives the request, it removes the existing data from the Recommendation Clusters. In case of an error, the entire request is rejected and diff --git a/play/engage-sdk-integration/references/android/guide/playcore/engage/read.md b/play/engage-sdk-integration/references/android/guide/playcore/engage/read.md index 9c34f6f..ac4a984 100644 --- a/play/engage-sdk-integration/references/android/guide/playcore/engage/read.md +++ b/play/engage-sdk-integration/references/android/guide/playcore/engage/read.md @@ -59,6 +59,9 @@ Use the Engage SDK [Android skill](https://developer.android.com/tools/agents/an ## Integration detail +This section details the steps, data models, and terminology required to +integrate with the Engage SDK for reading content. + ### Terminology This integration includes the following three cluster types: **Recommendation** , @@ -125,7 +128,8 @@ the following entities for the Read category: 2. `AudiobookEntity` 3. `BookSeriesEntity` -The charts below outline available attributes and requirements for each type. +The following charts outline available attributes and requirements for each +type. #### `EbookEntity` @@ -201,7 +205,7 @@ Potter* book series, which has 7 books). #### Image specifications -Required specifications for image assets are listed below: +The following table lists the required specifications for image assets: | Aspect ratio | Supported cluster(s) | Minimum pixels | Recommended pixels | |---|---|---|---| @@ -350,24 +354,17 @@ required for `ServiceAvailabilityRequest` from the following table. ###### Conditional Service Availability Feature Some integrated apps request a special configuration that enables and disables -the Engage service intermittently in order to reduce their serving cost. This -intermittent content ingestion strategy, although possible, negatively affects -the user and the product -- stale content will not be presented and some surfaces -will not be served at all. +the Engage service intermittently to reduce serving costs. However, this +intermittent content ingestion strategy negatively affects the user experience: +stale content won't be presented, and some surfaces won't be served. Starting with v1.6.0, the Engage SDK allows checking availability for specific -cluster types. This provides more flexibility so that if the intermittent -content strategy was adopted by a given application, some cluster types can -follow that intermittent strategy while other cluster types are always enabled -(i.e. continuation clusters). - -If the Engage service should not be 'continuously' enabled on all supported -devices for whatever reason, and is configured for intermittent ingestion for -any set of devices, all continuation cluster publications (e.g. Continue -Reading) will be still enabled by default -configuration, and the rest of the cluster types will be enabled and disabled -intermittently. If intermittent ingestion applies to you but this default -configuration is not suitable for your needs, please contact +cluster types. If your app uses intermittent ingestion: + +- **Continuation clusters** (such as Continue Reading) remain enabled by default across all supported devices. +- **Other cluster types** (such as Recommendations) are enabled and disabled intermittently. + +If this default configuration isn't suitable for your needs, contact engage-developers@google.com. ##### For SDK versions prior to v1.6.0 (Deprecated) @@ -654,7 +651,7 @@ with the status code **NOT_PUBLISHED_REQUIRES_SIGN_IN** .setStatusCode(AppEngagePublishStatusCode.NOT_PUBLISHED_REQUIRES_SIGN_IN) .build()); -#### `deleteRecommendationClusters` +#### `deleteRecommendationsClusters` This API is used to delete the content of Recommendation Clusters. @@ -663,11 +660,11 @@ This API is used to delete the content of Recommendation Clusters. ### Kotlin - client.deleteRecommendationClusters() + client.deleteRecommendationsClusters() ### Java - client.deleteRecommendationClusters(); + client.deleteRecommendationsClusters(); When the service receives the request, it removes the existing data from the Recommendation Clusters. In case of an error, the entire request is rejected diff --git a/play/engage-sdk-integration/references/android/guide/playcore/engage/shopping.md b/play/engage-sdk-integration/references/android/guide/playcore/engage/shopping.md index 9dc53d3..1a41060 100644 --- a/play/engage-sdk-integration/references/android/guide/playcore/engage/shopping.md +++ b/play/engage-sdk-integration/references/android/guide/playcore/engage/shopping.md @@ -35,6 +35,9 @@ Use the Engage SDK [Android skill](https://developer.android.com/tools/agents/an ## Integration detail +This section details the steps, data models, and terminology required to +integrate with the Engage SDK for shopping content. + ### Terminology This integration includes the following five cluster types: **Recommendation** , @@ -43,8 +46,8 @@ This integration includes the following five cluster types: **Recommendation** , - **Recommendation** clusters show personalized shopping suggestions from an individual developer partner. These recommendations can be personalized to the - user or generalized (for example, trending items). Use these to surface - products, events, sales, promos, subscriptions as you see fit. + user or generalized (for example, trending items). Use these to feature + products, events, sales, promos, and subscriptions. Your recommendations take the following structure: - **Recommendation Cluster:** A UI view that contains a group of @@ -146,7 +149,8 @@ following entities are supported for the Shopping category: 4. `Reorder` 5. `ShoppingOrderTracking` -The charts below outline available attributes and requirements for each type. +The following charts outline available attributes and requirements for each +type. #### `ShoppingEntity` @@ -231,7 +235,7 @@ or event that developer partners want to publish. #### Image specifications -Required specifications for image assets are listed below: +The following table lists the required specifications for image assets: | Aspect ratio | Minimum pixels | Recommended pixels | |---|---|---| @@ -276,7 +280,7 @@ Following APIs are exposed to publish clusters in the client: - `deleteFeaturedCluster` - `deleteShoppingCartCluster` - `deleteShoppingListCluster` -- `deleteShoppingReorderCluster` +- `deleteReorderCluster` - `deleteShoppingOrderTrackingCluster` - `deleteUserManagementCluster` - `deleteClusters` @@ -372,25 +376,18 @@ required for `ServiceAvailabilityRequest` from the following table. ###### Conditional Service Availability Feature Some integrated apps request a special configuration that enables and disables -the Engage service intermittently in order to reduce their serving cost. This -intermittent content ingestion strategy, although possible, negatively affects -the user and the product -- stale content will not be presented and some surfaces -will not be served at all. +the Engage service intermittently to reduce serving costs. However, this +intermittent content ingestion strategy negatively affects the user experience: +stale content won't be presented, and some surfaces won't be served. Starting with v1.6.0, the Engage SDK allows checking availability for specific -cluster types. This provides more flexibility so that if the intermittent -content strategy was adopted by a given application, some cluster types can -follow that intermittent strategy while other cluster types are always enabled -(i.e. continuation clusters). - -If the Engage service should not be 'continuously' enabled on all supported -devices for whatever reason, and is configured for intermittent ingestion for -any set of devices, all continuation cluster publications (e.g. Shopping -Cart, Shopping List, Reorder, and Shopping Order Tracking) will be still -enabled by default configuration, and the rest of the cluster types will be -enabled and disabled intermittently. If intermittent ingestion applies to -you but this default configuration is not suitable for your needs, please -contact engage-developers@google.com. +cluster types. If your app uses intermittent ingestion: + +- **Continuation clusters** (such as Shopping Cart, Shopping List, Reorder, and Shopping Order Tracking) remain enabled by default across all supported devices. +- **Other cluster types** (such as Recommendations) are enabled and disabled intermittently. + +If this default configuration isn't suitable for your needs, contact +engage-developers@google.com. ##### For SDK versions prior to v1.6.0 (Deprecated) @@ -576,7 +573,7 @@ This API is used to publish a list of `ShoppingList` objects. client.publishShoppingLists( new PublishShoppingListsRequest.Builder() .addShoppingList( - new ShoppingListEntity.Builder() + new ShoppingList.Builder() ... .build()) .build()); @@ -793,7 +790,7 @@ with the status code **NOT_PUBLISHED_REQUIRES_SIGN_IN** .setStatusCode(AppEngagePublishStatusCode.NOT_PUBLISHED_REQUIRES_SIGN_IN) .build()); -#### `deleteRecommendationClusters` +#### `deleteRecommendationsClusters` This API is used to delete the content of Recommendation Clusters. @@ -802,11 +799,11 @@ This API is used to delete the content of Recommendation Clusters. ### Kotlin - client.deleteRecommendationClusters() + client.deleteRecommendationsClusters() ### Java - client.deleteRecommendationClusters(); + client.deleteRecommendationsClusters(); When the service receives the request, it removes the existing data from the Recommendation Clusters. In case of an error, the entire request is rejected @@ -881,7 +878,7 @@ and the existing state is maintained. > [!NOTE] > **Note:** This api is available from version 1.1.0 onwards. -#### `deleteShoppingReorderCluster` +#### `deleteReorderCluster` This API is used to delete the content of Shopping Reorder Cluster. @@ -890,11 +887,11 @@ This API is used to delete the content of Shopping Reorder Cluster. ### Kotlin - client.deleteShoppingReorderCluster() + client.deleteReorderCluster() ### Java - client.deleteShoppingReorderCluster(); + client.deleteReorderCluster(); When the service receives the request, it removes the existing data from the Shopping Reorder Cluster. In case of an error, the entire request is rejected diff --git a/play/engage-sdk-integration/references/android/guide/playcore/engage/social.md b/play/engage-sdk-integration/references/android/guide/playcore/engage/social.md index 778b040..f2dd9db 100644 --- a/play/engage-sdk-integration/references/android/guide/playcore/engage/social.md +++ b/play/engage-sdk-integration/references/android/guide/playcore/engage/social.md @@ -115,7 +115,8 @@ supports the following entities for the Social category: 1. `PortraitMediaEntity` 2. `SocialPostEntity` -The charts below outline available attributes and requirements for each type. +The following charts outline available attributes and requirements for each +type. #### `PortraitMediaEntity` @@ -302,14 +303,13 @@ required for `ServiceAvailabilityRequest` from the following table. ###### Conditional Service Availability Feature Some integrated apps request a special configuration that enables and disables -the Engage service intermittently in order to reduce their serving cost. This -intermittent content ingestion strategy, although possible, negatively affects -the user and the product -- stale content will not be presented and some surfaces -will not be served at all. +the Engage service intermittently to reduce serving costs. However, this +intermittent content ingestion strategy negatively affects the user experience: +stale content won't be presented, and some surfaces won't be served. Starting with v1.6.0, the Engage SDK allows checking availability for specific -cluster types. If you are interested in opting into this feature for any cluster type, -please contact engage-developers@google.com. +cluster types. If you are interested in opting into this feature for any cluster +type, contact engage-developers@google.com. ##### For SDK versions prior to v1.6.0 (Deprecated) @@ -533,7 +533,7 @@ with the status code **NOT_PUBLISHED_REQUIRES_SIGN_IN** .setStatusCode(AppEngagePublishStatusCode.NOT_PUBLISHED_REQUIRES_SIGN_IN) .build()); -#### `deleteRecommendationClusters` +#### `deleteRecommendationsClusters` This API is used to delete the content of Recommendation Clusters. @@ -542,11 +542,11 @@ This API is used to delete the content of Recommendation Clusters. ### Kotlin - client.deleteRecommendationClusters() + client.deleteRecommendationsClusters() ### Java - client.deleteRecommendationClusters(); + client.deleteRecommendationsClusters(); When the service receives the request, it removes the existing data from the Recommendation Clusters. In case of an error, the entire request is rejected diff --git a/play/engage-sdk-integration/references/android/guide/playcore/engage/travel.md b/play/engage-sdk-integration/references/android/guide/playcore/engage/travel.md index 84baa26..9d18117 100644 --- a/play/engage-sdk-integration/references/android/guide/playcore/engage/travel.md +++ b/play/engage-sdk-integration/references/android/guide/playcore/engage/travel.md @@ -35,6 +35,9 @@ Use the Engage SDK [Android skill](https://developer.android.com/tools/agents/an ## Integration detail +This section details the steps, data models, and terminology required to +integrate with the Engage SDK for travel and events content. + ### Terminology This integration includes the following cluster types: **Recommendation** , @@ -43,7 +46,7 @@ This integration includes the following cluster types: **Recommendation** , - **Recommendation** clusters show personalized travel \& event suggestions from an individual developer partner. These recommendations can be personalized to the user or generalized (for example, trending items). Use - these to surface articles, events, lodging, or places of interest + these to feature articles, events, lodging, or places of interest recommendations. - A Recommendation cluster can be made of `ArticleEntity`, `EventEntity`, `LodgingEntity`, `PointOfInterestEntity`, or `StoreEntity` listings, but not a mix of different entity types. @@ -189,7 +192,8 @@ the following entities for the Travel \& Events category: 10. `TransportationReservationEntity` 11. `VehicleRentalReservationEntity` -The charts below outline available attributes and requirements for each type. +The following charts outline available attributes and requirements for each +type. #### `GenericFeaturedEntity` @@ -718,24 +722,18 @@ required for `ServiceAvailabilityRequest` from the following table. ###### Conditional Service Availability Feature Some integrated apps request a special configuration that enables and disables -the Engage service intermittently in order to reduce their serving cost. This -intermittent content ingestion strategy, although possible, negatively affects -the user and the product -- stale content will not be presented and some surfaces -will not be served at all. +the Engage service intermittently to reduce serving costs. However, this +intermittent content ingestion strategy negatively affects the user experience: +stale content won't be presented, and some surfaces won't be served. Starting with v1.6.0, the Engage SDK allows checking availability for specific -cluster types. This provides more flexibility so that if the intermittent -content strategy was adopted by a given application, some cluster types can -follow that intermittent strategy while other cluster types are always enabled -(i.e. continuation clusters). - -If the Engage service should not be 'continuously' enabled on all supported -devices for whatever reason, and is configured for intermittent ingestion for -any set of devices, all continuation cluster publications (e.g. Reservation -and Continue Search) will be still enabled by default configuration, and the -rest of the cluster types will be enabled and disabled intermittently. If -intermittent ingestion applies to you but this default configuration is not -suitable for your needs, please contact engage-developers@google.com. +cluster types. If your app uses intermittent ingestion: + +- **Continuation clusters** (such as Reservation and Continue Search) remain enabled by default across all supported devices. +- **Other cluster types** (such as Recommendations) are enabled and disabled intermittently. + +If this default configuration isn't suitable for your needs, contact +engage-developers@google.com. ##### For SDK versions prior to v1.6.0 (Deprecated) @@ -839,8 +837,8 @@ This API is used to publish a list of `FeaturedCluster` objects. ### Java client.publishFeaturedCluster( - new PublishFeaturedClustersRequest.Builder() - .addFeaturedCluster( + new PublishFeaturedClusterRequest.Builder() + .setFeaturedCluster( new FeaturedCluster.Builder() .addEntity(entity1) .addEntity(entity2) @@ -1068,7 +1066,7 @@ to publish the Sign In Card then we recommend calling the .setStatusCode(AppEngagePublishStatusCode.NOT_PUBLISHED_REQUIRES_SIGN_IN) .build()); -#### `deleteRecommendationClusters` +#### `deleteRecommendationsClusters` This API is used to delete the content of Recommendation Clusters. @@ -1077,11 +1075,11 @@ This API is used to delete the content of Recommendation Clusters. ### Kotlin - client.deleteRecommendationClusters() + client.deleteRecommendationsClusters() ### Java - client.deleteRecommendationClusters(); + client.deleteRecommendationsClusters(); When the service receives the request, it removes the existing data from the Recommendation Clusters. In case of an error, the entire request is rejected and diff --git a/play/engage-sdk-integration/references/android/guide/playcore/engage/tv/entitlements.md b/play/engage-sdk-integration/references/android/guide/playcore/engage/tv/entitlements.md index a8f8c30..e2bf530 100644 --- a/play/engage-sdk-integration/references/android/guide/playcore/engage/tv/entitlements.md +++ b/play/engage-sdk-integration/references/android/guide/playcore/engage/tv/entitlements.md @@ -72,7 +72,7 @@ Example for the sample media provider feed. The following example creates a `SubscriptionEntity` for a user: val subscription = SubscriptionEntity.Builder() - setSubscriptionType( + .setSubscriptionType( SubscriptionType.SUBSCRIPTION_TYPE_ACTIVE ) .setProviderPackageName("com.google.android.example") @@ -179,7 +179,7 @@ In addition, if the user has another subscription to a subsidiary service, add another subscription and set the linked service package name accordingly. // Subscription for linked service package - val linkedSubscription = Subscription.Builder() + val linkedSubscription = SubscriptionEntity.Builder() .setSubscriptionType( SubscriptionType.SUBSCRIPTION_TYPE_ACTIVE ) @@ -203,7 +203,7 @@ another subscription and set the linked service package name accordingly. ) .build() ) - .build() + .build() Optionally, add entitlements to a linked service subscription too. @@ -222,7 +222,7 @@ described in the [Getting Started guide](https://developer.android.com/guide/pla .setAccountProfile(accountProfile) .setSubscription(subscription) .build() - ) + ) Use `setSubscription()` to verify that user should have only one subscription to the service. @@ -260,11 +260,11 @@ maintained. // to remove subscription and other Engage data from the current // google TV device. client.deleteClusters( - new DeleteClustersRequest.Builder() + DeleteClustersRequest.Builder() .setAccountProfile(accountProfile) - .setReason(DeleteReason.DELETE_REASON_USER_LOG_OUT) - .build() - ) + .setDeleteReason(DeleteReason.DELETE_REASON_USER_LOG_OUT) + .build() + ) The following code snippet demonstrates removal of user subscription when user revokes the consent: @@ -273,9 +273,9 @@ maintained. // to remove subscription and other Engage data from all google // TV devices. client.deleteClusters( - new DeleteClustersRequest.Builder() + DeleteClustersRequest.Builder() .setAccountProfile(accountProfile) - .setReason(DeleteReason.DELETE_REASON_LOSS_OF_CONSENT) + .setDeleteReason(DeleteReason.DELETE_REASON_LOSS_OF_CONSENT) .build() ) @@ -285,10 +285,10 @@ maintained. // If the user delete a specific profile, you must make the following call // to remove subscription data and other Engage data. client.deleteClusters( - new DeleteClustersRequest.Builder() - .setAccountProfile(accountProfile) - .setReason(DeleteReason.DELETE_REASON_ACCOUNT_PROFILE_DELETION) - .build() + DeleteClustersRequest.Builder() + .setAccountProfile(accountProfile) + .setDeleteReason(DeleteReason.DELETE_REASON_ACCOUNT_PROFILE_DELETION) + .build() ) ### Testing diff --git a/play/engage-sdk-integration/references/android/guide/playcore/engage/tv/recommendations.md b/play/engage-sdk-integration/references/android/guide/playcore/engage/tv/recommendations.md index a2fd19b..762a947 100644 --- a/play/engage-sdk-integration/references/android/guide/playcore/engage/tv/recommendations.md +++ b/play/engage-sdk-integration/references/android/guide/playcore/engage/tv/recommendations.md @@ -111,9 +111,9 @@ The following code removes recommendations on logout. // the recommendations data persists on the current Google TV device until 60 // days later. client.deleteClusters( - new DeleteClustersRequest.Builder() - .setAccountProfile(AccountProfile()) - .setReason(DeleteReason.DELETE_REASON_USER_LOG_OUT) + DeleteClustersRequest.Builder() + .setAccountProfile(accountProfile) + .setDeleteReason(DeleteReason.DELETE_REASON_USER_LOG_OUT) .build() ) @@ -121,9 +121,9 @@ The following code removes recommendations on logout. // the following call to remove recommendations data from all current Google TV // devices. Otherwise, the recommendations data persists until 60 days later. client.deleteClusters( - new DeleteClustersRequest.Builder() - .setAccountProfile(AccountProfile()) - .setReason(DeleteReason.DELETE_REASON_LOSS_OF_CONSENT) + DeleteClustersRequest.Builder() + .setAccountProfile(accountProfile) + .setDeleteReason(DeleteReason.DELETE_REASON_LOSS_OF_CONSENT) .build() ) @@ -190,7 +190,7 @@ repeat it for every platform. Arrays.asList(playbackUriTv, playbackUriAndroid, playbackUriIos) // Provide appropriate rating for the system. - val contentRating = new RatingSystem + val contentRating = RatingSystem .Builder() .setAgencyName("MPAA") .setRating("PG-13") @@ -205,14 +205,14 @@ correct display of the "Recommendations" entity, especially within Google's [Entertainment Space](https://support.google.com/entertainmentspace/answer/10346911). Images with a height less than 200 pixels may not be shown. - Image image1 = new Image.Builder() - .setImageUri(Uri.parse("http://www.example.com/entity_image1.png");) + val image1 = Image.Builder() + .setImageUri(Uri.parse("http://www.example.com/entity_image1.png")) .setImageHeightInPixel(300) .setImageWidthInPixel(169) .build() - Image image2 = new Image.Builder() - .setImageUri(Uri.parse("http://www.example.com/entity_image2.png");) + val image2 = Image.Builder() + .setImageUri(Uri.parse("http://www.example.com/entity_image2.png")) .setImageHeightInPixel(640) .setImageWidthInPixel(360) .build() @@ -244,9 +244,9 @@ the user. //Allows us to construct reason: "Because you watched Star Wars" val watchedSimilarTitles = RecommendationReasonWatchedSimilarTitles + .Builder() .addSimilarWatchedTitleName("Movie or TV Show Title") .addSimilarWatchedTitleName("Movie or TV Show Title") - .Builder() .build() //Allows us to construct reason: "Recommended for you by ChannelName" @@ -262,9 +262,9 @@ the user. .Builder() .build() - val userLikedOnPartner = RecommendationReasonUserLikedOnPartner + val userLikedTitle = RecommendationReasonUserLikedTitle .Builder() - .setTitleName("Movie or TV Show Title") + .setLikedTitleName("Movie or TV Show Title") .build() val generic = RecommendationReasonGeneric.Builder().build() @@ -279,14 +279,16 @@ availability windows. val window1 = DisplayTimeWindow .Builder() - .setStartTimeStampMillis(now()+ 1.days.toMillis()) - .setEndTimeStampMillis(now()+ 30.days.toMillis()) + .setStartTimestampMillis(now() + 1.days.toMillis()) + .setEndTimestampMillis(now() + 30.days.toMillis()) + .build() val window2 = DisplayTimeWindow .Builder() - .setEndTimeStampMillis(now()+ 30.days.toMillis()) + .setEndTimestampMillis(now() + 30.days.toMillis()) + .build() - val availabilityTimeWindows: List = listof(window1,window2) + val availabilityTimeWindows: List = listOf(window1, window2) ### DataFeedElementId @@ -306,10 +308,10 @@ Here's an example of creating a `MovieEntity` with all the required fields: val movieEntity = MovieEntity.Builder() .setName("Movie name") .setDescription("A sentence describing movie.") - .addPlatformSpecificPlaybackUri(platformSpecificPlaybackUris) + .addPlatformSpecificPlaybackUris(platformSpecificPlaybackUris) .addPosterImages(images) // Suppose the duration is 2 hours, it is 72000000 in milliseconds - .setDurationMills(72000000) + .setDurationMillis(72000000) .setCallToActionText("Watch Now") .addTag("Action") .build() @@ -318,9 +320,9 @@ You can provide additional data such as genres, content ratings, release date, recommendation reason and availability time windows, which may be used by Google TV for enhanced displays or filtering purposes. - val genres = Arrays.asList("Action", "Science fiction"); - val rating1 = RatingSystem.Builder().setAgencyName("MPAA").setRating("pg-13").build(); - val contentRatings = Arrays.asList(rating1); + val genres = Arrays.asList("Action", "Science fiction") + val rating1 = RatingSystem.Builder().setAgencyName("MPAA").setRating("pg-13").build() + val contentRatings = Arrays.asList(rating1) //Suppose release date is 11-02-2025 val releaseDate = 1739233800000L val movieEntity = MovieEntity.Builder() @@ -339,27 +341,27 @@ Here's an example of creating a `TvShowEntity` with all the required fields: val tvShowEntity = TvShowEntity.Builder() .setName("Show title") .setDescription("A sentence describing TV Show.") - .addPlatformSpecificPlaybackUri(platformSpecificPlaybackUris) + .addPlatformSpecificPlaybackUris(platformSpecificPlaybackUris) .addPosterImages(images) .setCallToActionText("Watch Now") .addTag("Drama") - .build(); + .build() Optionally provide additional data such as genres, content ratings, recommendation reason, offer price, season count or availability time window, which may be used by Google TV for enhanced displays or filtering purposes. - val genres = Arrays.asList("Action", "Science fiction"); + val genres = Arrays.asList("Action", "Science fiction") val rating1 = RatingSystem.Builder() .setAgencyName("MPAA") .setRating("pg-13") - .build(); + .build() val price = Price.Builder() .setCurrentPrice("$14.99") .setStrikethroughPrice("$16.99") - .build(); - val contentRatings = Arrays.asList(rating1); - val seasonCount = 5; + .build() + val contentRatings = Arrays.asList(rating1) + val seasonCount = 5 val tvShowEntity = TvShowEntity.Builder() ... .addGenres(genres) @@ -403,8 +405,8 @@ This represents a live TV channel. Here's an example of creating a .setEntityId("https://www.example.com/channel/12345") .setDescription("A sentence describing this live TV channel.") // channel playback uri must contain at least PlatformType.TYPE_ANDROID_TV - .addPlatformSpecificPlaybackUri(channelPlaybackUris) - .addLogoImage(logoImage) + .addPlatformSpecificPlaybackUris(channelPlaybackUris) + .setLogoImage(logoImage) .setCallToActionText("Watch Now") .addTag("News") .build() @@ -435,7 +437,7 @@ with all the required fields: .setChannelName("Channel Name") .setChannelId("https://www.example.com/channel/12345") // channel playback uri must contain at least PlatformType.TYPE_ANDROID_TV - .addPlatformSpecificPlaybackUri(channelPlaybackUris) + .addPlatformSpecificPlaybackUris(channelPlaybackUris) .setChannelLogoImage(channelLogoImage) // Then set the program or card specific information. .setName("Program Name") @@ -443,8 +445,10 @@ with all the required fields: .setDescription("Program Description") .addAvailabilityTimeWindow( DisplayTimeWindow.Builder() - .setStartTimestampMillis(1756713600000L)// 2025-09-01T07:30:00+0000 - .setEndTimestampMillis(1756715400000L))// 2025-09-01T08:00:00+0000 + .setStartTimestampMillis(1756713600000L) // 2025-09-01T07:30:00+0000 + .setEndTimestampMillis(1756715400000L) // 2025-09-01T08:00:00+0000 + .build() + ) .addPosterImage(programImage) .setCallToActionText("Watch Now") .addTag("Sports") diff --git a/play/engage-sdk-integration/references/android/guide/playcore/engage/watch.md b/play/engage-sdk-integration/references/android/guide/playcore/engage/watch.md index 21c0938..e90b6b0 100644 --- a/play/engage-sdk-integration/references/android/guide/playcore/engage/watch.md +++ b/play/engage-sdk-integration/references/android/guide/playcore/engage/watch.md @@ -43,6 +43,9 @@ Use the Engage SDK [Android skill](https://developer.android.com/tools/agents/an ## Integration detail +This section details the steps, data models, and terminology required to +integrate with the Engage SDK for video content. + ### Terminology This integration includes the following three cluster types: **Recommendation** , @@ -112,6 +115,9 @@ cluster types: ### Step 0: Migration from existing Media Home SDK integration +This section outlines how to migrate your existing Media Home SDK data models +and API calls to the Engage SDK. + #### Map data models from existing integration If you are migrating from an existing Media Home integration, the following @@ -128,7 +134,7 @@ table outlines how to map data models in existing SDKs to the new Engage SDK: | `com.google.android.mediahome.video.WatchNextProgram` | Divided into attributes in separate classes: `EventVideoEntity`, `LiveStreamingVideoEntity`, `MovieEntity`, `TvEpisodeEntity`, `TvSeasonEntity`, `TvShowEntity`, `VideoClipEntity` | | `com.google.android.mediahome.video.WatchNextProgram.Builder` | Divided into attributes in separate classes: `EventVideoEntity`, `LiveStreamingVideoEntity`, `MovieEntity`, `TvEpisodeEntity`, `TvSeasonEntity`, `TvShowEntity`, `VideoClipEntity` | -#### Publishing clusters in Media Home SDK vs Engage SDK +#### Publishing clusters in Media Home SDK compared to Engage SDK With Media Home SDK, clusters and entities were published through separate APIs: @@ -199,7 +205,7 @@ The following chart outlines attributes and requirements for each type. | Content ratings | Optional | Free text, follow the industry standard. ([Example](https://www.spectrum.net/support/tv/tv-and-movie-ratings-descriptions)) | | Call to action text | Optional | Free text to be displayed as a call to action. | | Tags | Optional | List of tags associated with the entity. | -| Watch next type | Conditionally required | Must be provided when the item is in the Continuation cluster and must be one of the following four types: CONTINUE: The user has already watched more than 1 minute of this content. NEW: The user has watched all available episodes from some episodic content, but a new episode has become available and there is exactly one unwatched episode. This works for TV shows, recorded soccer matches in a series, and so on. NEXT: The user has watched one or more complete episodes from some episodic content, but there remains either more than one episode remaining or exactly one episode remaining where the last episode is not "NEW" and was released before the user started watching the episodic content. WATCHLIST: The user has explicitly elected to add a movie, event, or series to a watchlist to manually curate what they want to watch next. | +| Watch next type | Conditionally required | Must be provided when the item is in the Continuation cluster and must be one of the following four types: CONTINUE: The user has already watched more than 1 minute of this content. NEW: The user has watched all available episodes from some episodic content, but a new episode has become available and there is exactly one unwatched episode. This works for episodic content such as TV shows or recorded soccer matches in a series. NEXT: The user has watched one or more complete episodes from some episodic content, but there remains either more than one episode remaining or exactly one episode remaining where the last episode is not "NEW" and was released before the user started watching the episodic content. WATCHLIST: The user has explicitly elected to add a movie, event, or series to a watchlist to manually curate what they want to watch next. | | Last engagement time | Conditionally required | Must be provided when the item is in the Continuation cluster. In epoch milliseconds. | | Last playback position time | Conditionally required | Must be provided when the item is in the Continuation cluster and WatchNextType is CONTINUE. In epoch milliseconds. | @@ -220,7 +226,7 @@ The following chart outlines attributes and requirements for each type. | Content ratings | Optional | Free text, follow the industry standard. ([Example](https://www.spectrum.net/support/tv/tv-and-movie-ratings-descriptions)) | | Call to action text | Optional | Free text to be displayed as a call to action. | | Tags | Optional | List of tags associated with the entity. | -| Watch next type | Conditionally required | Must be provided when the item is in the Continuation cluster and must be one of the following four types: CONTINUE: The user has already watched more than 1 minute of this content. NEW: The user has watched all available episodes from some episodic content, but a new episode has become available and there is exactly one unwatched episode. This works for TV shows, recorded soccer matches in a series, and so on. NEXT: The user has watched one or more complete episodes from some episodic content, but there remains either more than one episode remaining or exactly one episode remaining where the last episode is not "NEW" and was released before the user started watching the episodic content. WATCHLIST: The user has explicitly elected to add a movie, event, or series to a watchlist to manually curate what they want to watch next. | +| Watch next type | Conditionally required | Must be provided when the item is in the Continuation cluster and must be one of the following four types: CONTINUE: The user has already watched more than 1 minute of this content. NEW: The user has watched all available episodes from some episodic content, but a new episode has become available and there is exactly one unwatched episode. This works for episodic content such as TV shows or recorded soccer matches in a series. NEXT: The user has watched one or more complete episodes from some episodic content, but there remains either more than one episode remaining or exactly one episode remaining where the last episode is not "NEW" and was released before the user started watching the episodic content. WATCHLIST: The user has explicitly elected to add a movie, event, or series to a watchlist to manually curate what they want to watch next. | | Last engagement time | Conditionally required | Must be provided when the item is in the Continuation cluster. In epoch milliseconds. | | Last playback position time | Conditionally required | Must be provided when the item is in the Continuation cluster and WatchNextType is CONTINUE. In epoch milliseconds. | @@ -242,7 +248,7 @@ The following chart outlines attributes and requirements for each type. | Content ratings | Optional | Free text, follow the industry standard. ([Example](https://www.spectrum.net/support/tv/tv-and-movie-ratings-descriptions)) | | Call to action text | Optional | Free text to be displayed as a call to action. | | Tags | Optional | List of tags associated with the entity. | -| Watch next type | Conditionally required | Must be provided when the item is in the Continuation cluster and must be one of the following four types: CONTINUE: The user has already watched more than 1 minute of this content. NEW: The user has watched all available episodes from some episodic content, but a new episode has become available and there is exactly one unwatched episode. This works for TV shows, recorded soccer matches in a series, and so on. NEXT: The user has watched one or more complete episodes from some episodic content, but there remains either more than one episode remaining or exactly one episode remaining where the last episode is not "NEW" and was released before the user started watching the episodic content. WATCHLIST: The user has explicitly elected to add a movie, event, or series to a watchlist to manually curate what they want to watch next. | +| Watch next type | Conditionally required | Must be provided when the item is in the Continuation cluster and must be one of the following four types: CONTINUE: The user has already watched more than 1 minute of this content. NEW: The user has watched all available episodes from some episodic content, but a new episode has become available and there is exactly one unwatched episode. This works for episodic content such as TV shows or recorded soccer matches in a series. NEXT: The user has watched one or more complete episodes from some episodic content, but there remains either more than one episode remaining or exactly one episode remaining where the last episode is not "NEW" and was released before the user started watching the episodic content. WATCHLIST: The user has explicitly elected to add a movie, event, or series to a watchlist to manually curate what they want to watch next. | | Last engagement time | Conditionally required | Must be provided when the item is in the Continuation cluster. In epoch milliseconds. | | Last playback position time | Conditionally required | Must be provided when the item is in the Continuation cluster and WatchNextType is CONTINUE. In epoch milliseconds. | @@ -263,7 +269,7 @@ The following chart outlines attributes and requirements for each type. | Content ratings | Optional | Free text, follow the industry standard. ([Example](https://www.spectrum.net/support/tv/tv-and-movie-ratings-descriptions)) | | Call to action text | Optional | Free text to be displayed as a call to action. | | Tags | Optional | List of tags associated with the entity. | -| Watch next type | Conditionally required | Must be provided when the item is in the Continuation cluster and must be one of the following four types: CONTINUE: The user has already watched more than 1 minute of this content. NEW: The user has watched all available episodes from some episodic content, but a new episode has become available and there is exactly one unwatched episode. This works for TV shows, recorded soccer matches in a series, and so on. NEXT: The user has watched one or more complete episodes from some episodic content, but there remains either more than one episode remaining or exactly one episode remaining where the last episode is not "NEW" and was released before the user started watching the episodic content. WATCHLIST: The user has explicitly elected to add a movie, event, or series to a watchlist to manually curate what they want to watch next. | +| Watch next type | Conditionally required | Must be provided when the item is in the Continuation cluster and must be one of the following four types: CONTINUE: The user has already watched more than 1 minute of this content. NEW: The user has watched all available episodes from some episodic content, but a new episode has become available and there is exactly one unwatched episode. This works for episodic content such as TV shows or recorded soccer matches in a series. NEXT: The user has watched one or more complete episodes from some episodic content, but there remains either more than one episode remaining or exactly one episode remaining where the last episode is not "NEW" and was released before the user started watching the episodic content. WATCHLIST: The user has explicitly elected to add a movie, event, or series to a watchlist to manually curate what they want to watch next. | | Last engagement time | Conditionally required | Must be provided when the item is in the Continuation cluster. In epoch milliseconds. | | Last playback position time | Conditionally required | Must be provided when the item is in the Continuation cluster and WatchNextType is CONTINUE. In epoch milliseconds. | @@ -280,7 +286,7 @@ The following chart outlines attributes and requirements for each type. | View count | Optional | Free text, must be localized. | | Call to action text | Optional | Free text to be displayed as a call to action. | | Tags | Optional | List of tags associated with the entity. | -| Watch next type | Conditionally required | Must be provided when the item is in the Continuation cluster and must be one of the following four types: CONTINUE: The user has already watched more than 1 minute of this content. NEW: The user has watched all available episodes from some episodic content, but a new episode has become available and there is exactly one unwatched episode. This works for TV shows, recorded soccer matches in a series, and so on. NEXT: The user has watched one or more complete episodes from some episodic content, but there remains either more than one episode remaining or exactly one episode remaining where the last episode is not "NEW" and was released before the user started watching the episodic content. WATCHLIST: The user has explicitly elected to add a movie, event, or series to a watchlist to manually curate what they want to watch next. | +| Watch next type | Conditionally required | Must be provided when the item is in the Continuation cluster and must be one of the following four types: CONTINUE: The user has already watched more than 1 minute of this content. NEW: The user has watched all available episodes from some episodic content, but a new episode has become available and there is exactly one unwatched episode. This works for episodic content such as TV shows or recorded soccer matches in a series. NEXT: The user has watched one or more complete episodes from some episodic content, but there remains either more than one episode remaining or exactly one episode remaining where the last episode is not "NEW" and was released before the user started watching the episodic content. WATCHLIST: The user has explicitly elected to add a movie, event, or series to a watchlist to manually curate what they want to watch next. | | Last engagement time | Conditionally required | Must be provided when the item is in the Continuation cluster. In epoch milliseconds. | | Last playback position time | Conditionally required | Must be provided when the item is in the Continuation cluster and WatchNextType is CONTINUE. In epoch milliseconds. | @@ -301,7 +307,7 @@ such as TikTok or YouTube. | View count | Optional | Free text, must be localized. | | Call to action text | Optional | Free text to be displayed as a call to action. | | Tags | Optional | List of tags associated with the entity. | -| Watch next type | Conditionally required | Must be provided when the item is in the Continuation cluster and must be one of the following four types: CONTINUE: The user has already watched more than 1 minute of this content. NEW: The user has watched all available episodes from some episodic content, but a new episode has become available and there is exactly one unwatched episode. This works for TV shows, recorded soccer matches in a series, and so on. NEXT: The user has watched one or more complete episodes from some episodic content, but there remains either more than one episode remaining or exactly one episode remaining where the last episode is not "NEW" and was released before the user started watching the episodic content. WATCHLIST: The user has explicitly elected to add a movie, event, or series to a watchlist to manually curate what they want to watch next. | +| Watch next type | Conditionally required | Must be provided when the item is in the Continuation cluster and must be one of the following four types: CONTINUE: The user has already watched more than 1 minute of this content. NEW: The user has watched all available episodes from some episodic content, but a new episode has become available and there is exactly one unwatched episode. This works for episodic content such as TV shows or recorded soccer matches in a series. NEXT: The user has watched one or more complete episodes from some episodic content, but there remains either more than one episode remaining or exactly one episode remaining where the last episode is not "NEW" and was released before the user started watching the episodic content. WATCHLIST: The user has explicitly elected to add a movie, event, or series to a watchlist to manually curate what they want to watch next. | | Last engagement time | Conditionally required | Must be provided when the item is in the Continuation cluster. In epoch milliseconds. | | Last playback position time | Conditionally required | Must be provided when the item is in the Continuation cluster and WatchNextType is CONTINUE. In epoch milliseconds. | @@ -475,24 +481,18 @@ required for `ServiceAvailabilityRequest` from the following table. ###### Conditional Service Availability Feature Some integrated apps request a special configuration that enables and disables -the Engage service intermittently in order to reduce their serving cost. This -intermittent content ingestion strategy, although possible, negatively affects -the user and the product -- stale content will not be presented and some surfaces -will not be served at all. +the Engage service intermittently to reduce serving costs. However, this +intermittent content ingestion strategy negatively affects the user experience: +stale content won't be presented, and some surfaces won't be served. Starting with v1.6.0, the Engage SDK allows checking availability for specific -cluster types. This provides more flexibility so that if the intermittent -content strategy was adopted by a given application, some cluster types can -follow that intermittent strategy while other cluster types are always enabled -(i.e. continuation clusters). - -If the Engage service should not be 'continuously' enabled on all supported -devices for whatever reason, and is configured for intermittent ingestion for -any set of devices, all continuation cluster publications (e.g. Continue -Watching) will be still enabled by default configuration, and the rest of the -cluster types will be enabled and disabled intermittently. If intermittent -ingestion applies to you but this default configuration is not suitable for -your needs, please contact engage-developers@google.com. +cluster types. If your app uses intermittent ingestion: + +- **Continuation clusters** (such as Continue Watching) remain enabled by default across all supported devices. +- **Other cluster types** (such as Recommendations) are enabled and disabled intermittently. + +If this default configuration isn't suitable for your needs, contact +engage-developers@google.com. ##### For SDK versions prior to v1.6.0 (Deprecated) @@ -596,8 +596,8 @@ This API is used to publish a list of `FeaturedCluster` objects. ### Java client.publishFeaturedCluster( - new PublishFeaturedClustersRequest.Builder() - .addFeaturedCluster( + new PublishFeaturedClusterRequest.Builder() + .setFeaturedCluster( new FeaturedCluster.Builder() .addEntity(entity1) .addEntity(entity2) @@ -782,7 +782,7 @@ with the status code **NOT_PUBLISHED_REQUIRES_SIGN_IN** .setStatusCode(AppEngagePublishStatusCode.NOT_PUBLISHED_REQUIRES_SIGN_IN) .build()); -#### `deleteRecommendationClusters` +#### `deleteRecommendationsClusters` This API is used to delete the content of Recommendation Clusters. @@ -791,11 +791,11 @@ This API is used to delete the content of Recommendation Clusters. ### Kotlin - client.deleteRecommendationClusters() + client.deleteRecommendationsClusters() ### Java - client.deleteRecommendationClusters(); + client.deleteRecommendationsClusters(); When the service receives the request, it removes the existing data from the Recommendation Clusters. In case of an error, the entire request is rejected diff --git a/profilers/android-profiler/SKILL.md b/profilers/android-profiler/SKILL.md index e5791fa..6cab558 100644 --- a/profilers/android-profiler/SKILL.md +++ b/profilers/android-profiler/SKILL.md @@ -13,7 +13,7 @@ description: 'Records and analyzes Android performance data and diagnoses perfor license: Complete terms in LICENSE.txt metadata: author: Google LLC - last-updated: '2026-10-14' + last-updated: '2026-10-17' keywords: - android performance - debugging diff --git a/system/glance-canonical-layouts/SKILL.md b/system/glance-canonical-layouts/SKILL.md index d084271..955c2e7 100644 --- a/system/glance-canonical-layouts/SKILL.md +++ b/system/glance-canonical-layouts/SKILL.md @@ -7,7 +7,7 @@ description: Use this skill to create home screen widgets using Jetpack Glance C license: Complete terms in LICENSE.txt metadata: author: Google LLC - last-updated: '2026-10-14' + last-updated: '2026-10-13' keywords: - android - glance diff --git a/wear/wear-compose-m3/SKILL.md b/wear/wear-compose-m3/SKILL.md index cee3ee5..1ae6563 100644 --- a/wear/wear-compose-m3/SKILL.md +++ b/wear/wear-compose-m3/SKILL.md @@ -8,7 +8,7 @@ description: Guidance for Wear OS Compose Material3 (Wear Compose Material 3 / W license: Complete terms in LICENSE.txt metadata: author: Google LLC - last-updated: '2026-10-13' + last-updated: '2026-10-14' keywords: - Wear OS - Compose diff --git a/wear/wear-compose-m3/references/material3/TransformationSpecSample.kt.md b/wear/wear-compose-m3/references/material3/TransformationSpecSample.kt.md index 2572a29..84704cb 100644 --- a/wear/wear-compose-m3/references/material3/TransformationSpecSample.kt.md +++ b/wear/wear-compose-m3/references/material3/TransformationSpecSample.kt.md @@ -65,7 +65,7 @@ fun CustomTransformationSpecSample() { contentPadding = PaddingValues(20.dp), modifier = Modifier.background(Color.Black), ) { - items(co>unt = 100) { index - + items(count = 100) { index -> Button( onClick = {}, modifier = @@ -101,7 +101,7 @@ fun CustomCompositingStrategyTransformationSpecSample() { contentPadding = PaddingValues(20.dp), modifier = Modifier.background(Color.Black), ) { - > items(count = 100) { index - + items(count = 100) { index -> Button( onClick = {}, modifier = @@ -163,8 +163,8 @@ fun ResponsiveTransformationSpecButtonSample() { TransformingLazyColumn( contentPadding = PaddingValues(20.dp), modifier = Modifier.background(Color.Black), -> ) { - items(count = 100) { index - + ) { + items(count = 100) { index -> Button( onClick = {}, modifier = diff --git a/wear/wear-widgets/SKILL.md b/wear/wear-widgets/SKILL.md index d0d9ee1..164dafb 100644 --- a/wear/wear-widgets/SKILL.md +++ b/wear/wear-widgets/SKILL.md @@ -12,7 +12,7 @@ description: Create, migrate, update, and verify Wear OS widgets using Glance We license: Complete terms in LICENSE.txt metadata: author: Google LLC - last-updated: '2026-10-10' + last-updated: '2026-10-16' keywords: - Wear OS - Widgets @@ -32,8 +32,6 @@ They serialize declarative UI and state documents to the watch host process. - Otherwise, fetch the latest version from Google Maven metadata ([Glance Wear metadata](https://dl.google.com/dl/android/maven2/androidx/glance/wear/wear/maven-metadata.xml), [Compose Remote metadata](https://dl.google.com/dl/android/maven2/androidx/compose/remote/remote-creation-compose/maven-metadata.xml), [Wear Compose Remote metadata](https://dl.google.com/dl/android/maven2/androidx/wear/compose/remote/remote-material3/maven-metadata.xml)). 2. **SDK and Compose compiler requirements** : Ask permission before modifying repo-wide Gradle files unless the user requested widget creation, tile migration, or dependency setup. Set `compileSdk = 37` or higher and enable `buildFeatures { compose = true }`. Keep the project's existing Android and Kotlin Gradle plugin declarations intact. -
- dependencies { implementation("androidx.glance.wear:wear:{GLANCE_WEAR_VERSION}") implementation("androidx.glance.wear:wear-core:{GLANCE_WEAR_VERSION}") @@ -61,20 +59,19 @@ Register the `GlanceWearWidgetService` subclass in entries. When configuring the service, provider XML, or legacy `TileService` migration, you MUST follow [Widget service and manifest reference](references/widget-service-and-manifest.md). -
- - - +```xml + + - - + - + +``` ### Step 2: Implement GlanceWearWidget and GlanceWearWidgetService @@ -82,29 +79,30 @@ Extend `GlanceWearWidget` and override `provideWidgetData` returning `WearWidgetDocument`. Annotate the `GlanceWearWidgetService` subclass with `@AssociateWithGlanceWearWidget` for static provider resolution: -
- - // WRONG: Mobile GlanceAppWidget API (don't use on Wear OS) - class BrokenWidget : GlanceAppWidget() { - override suspend fun provideGlance(context: Context, id: GlanceId) {} - } - - // CORRECT: Wear OS GlanceWearWidget + GlanceWearWidgetService - @AssociateWithGlanceWearWidget(MyWidget::class) - class MyWidgetService : GlanceWearWidgetService() { - override val widget: GlanceWearWidget = MyWidget() - } - - class MyWidget : GlanceWearWidget() { - override suspend fun provideWidgetData( - context: Context, - params: WearWidgetParams, - ): WearWidgetData = WearWidgetDocument( +```kotlin +// WRONG: Mobile GlanceAppWidget API (don't use on Wear OS) +class BrokenWidget : GlanceAppWidget() { + override suspend fun provideGlance(context: Context, id: GlanceId) {} +} + +// CORRECT: Wear OS GlanceWearWidget + GlanceWearWidgetService +@AssociateWithGlanceWearWidget(MyWidget::class) +class MyWidgetService : GlanceWearWidgetService() { + override val widget: GlanceWearWidget = MyWidget() +} + +class MyWidget : GlanceWearWidget() { + override suspend fun provideWidgetData( + context: Context, + params: WearWidgetParams, + ): WearWidgetData = + WearWidgetDocument( background = WearWidgetBrush.color(Color(0xFF1C1B1F).rc) ) { MyWidgetContent(params = params) } - } +} +``` ### Step 3: Build Remote Compose layouts, state, and actions diff --git a/wear/wear-widgets/references/previews-and-testing.md b/wear/wear-widgets/references/previews-and-testing.md index f1a6168..ecb660d 100644 --- a/wear/wear-widgets/references/previews-and-testing.md +++ b/wear/wear-widgets/references/previews-and-testing.md @@ -8,52 +8,52 @@ Annotate preview functions with `@Preview` and pass one of the predefined - **`RoundAllWidgetPreviewParams`**: Pill-shaped containers. - **`RectangularAllWidgetPreviewParams`** : Uncropped rectangular containers with display-safe padding at `320 dpi` for generating widget picker preview assets. -
- - import androidx.compose.runtime.Composable - import androidx.compose.ui.tooling.preview.Preview - import androidx.compose.ui.tooling.preview.PreviewParameter - import androidx.glance.wear.core.WearWidgetParams - import androidx.glance.wear.tooling.preview.RectangularAllWidgetPreviewParams - import androidx.glance.wear.tooling.preview.RoundAllWidgetPreviewParams - import androidx.glance.wear.tooling.preview.SquircleAllWidgetPreviewParams - import androidx.glance.wear.tooling.preview.WearWidgetPreview - - @Preview( - name = "Squircle", - device = "spec:width=1000dp,height=1000dp,dpi=320", - ) - @Composable - fun MyWidgetSquirclePreview( - @PreviewParameter(SquircleAllWidgetPreviewParams::class) - params: WearWidgetParams, - ) { - WearWidgetPreview(MyWidget(), params) - } - - @Preview( - name = "Round", - device = "spec:width=1000dp,height=1000dp,dpi=320", - ) - @Composable - fun MyWidgetRoundPreview( - @PreviewParameter(RoundAllWidgetPreviewParams::class) - params: WearWidgetParams, - ) { - WearWidgetPreview(MyWidget(), params) - } - - @Preview( - name = "Widget Preview Asset", - device = "spec:width=1000dp,height=1000dp,dpi=320", - ) - @Composable - fun MyWidgetCatalogPreview( - @PreviewParameter(RectangularAllWidgetPreviewParams::class) - params: WearWidgetParams, - ) { - WearWidgetPreview(MyWidget(), params) - } +```kotlin +import androidx.compose.runtime.Composable +import androidx.compose.ui.tooling.preview.Preview +import androidx.compose.ui.tooling.preview.PreviewParameter +import androidx.glance.wear.core.WearWidgetParams +import androidx.glance.wear.tooling.preview.RectangularAllWidgetPreviewParams +import androidx.glance.wear.tooling.preview.RoundAllWidgetPreviewParams +import androidx.glance.wear.tooling.preview.SquircleAllWidgetPreviewParams +import androidx.glance.wear.tooling.preview.WearWidgetPreview + +@Preview( + name = "Squircle", + device = "spec:width=1000dp,height=1000dp,dpi=320", +) +@Composable +fun MyWidgetSquirclePreview( + @PreviewParameter(SquircleAllWidgetPreviewParams::class) + params: WearWidgetParams, +) { + WearWidgetPreview(MyWidget(), params) +} + +@Preview( + name = "Round", + device = "spec:width=1000dp,height=1000dp,dpi=320", +) +@Composable +fun MyWidgetRoundPreview( + @PreviewParameter(RoundAllWidgetPreviewParams::class) + params: WearWidgetParams, +) { + WearWidgetPreview(MyWidget(), params) +} + +@Preview( + name = "Widget Preview Asset", + device = "spec:width=1000dp,height=1000dp,dpi=320", +) +@Composable +fun MyWidgetCatalogPreview( + @PreviewParameter(RectangularAllWidgetPreviewParams::class) + params: WearWidgetParams, +) { + WearWidgetPreview(MyWidget(), params) +} +``` ## 2. Widget picker preview PNGs (`res/drawable-nodpi/`) @@ -82,50 +82,49 @@ When unit-testing a widget with JUnit 4 and Robolectric, call `WearWidgetDocument` composable lambda and verifies that the Remote Compose `rcDocument` payload serializes without errors. Because `WearWidgetParams` and `captureRawContent` carry `@RestrictTo(LIBRARY_GROUP)` in alpha releases, add -`@file:SuppressLint("RestrictedApi")` to the test file: - -
- - @file:SuppressLint("RestrictedApi") - - import android.annotation.SuppressLint - import android.content.Context - import androidx.glance.wear.core.ContainerInfo - import androidx.glance.wear.core.WearWidgetParams - import androidx.glance.wear.core.WidgetInstanceId - import androidx.test.core.app.ApplicationProvider - import kotlinx.coroutines.runBlocking - import org.junit.Assert.assertNotNull - import org.junit.Assert.assertTrue - import org.junit.Test - import org.junit.runner.RunWith - import org.robolectric.RobolectricTestRunner - import org.robolectric.annotation.Config - - @RunWith(RobolectricTestRunner::class) - @Config(sdk = [34]) - class MyWidgetTest { - @Test - fun provideWidgetData_serializesRemoteComposeDocument() = runBlocking { - val context = ApplicationProvider.getApplicationContext() - val widget = MyWidget() - for (containerType in listOf( - ContainerInfo.CONTAINER_TYPE_SMALL, - ContainerInfo.CONTAINER_TYPE_LARGE, - )) { - val params = WearWidgetParams( - instanceId = WidgetInstanceId(namespace = "test", id = 1), - containerType = containerType, - widthDp = 220f, - heightDp = 120f, - horizontalPaddingDp = 8f, - verticalPaddingDp = 8f, - cornerRadiusDp = 16f, - ) - val data = widget.provideWidgetData(context, params) - val rawContent = data.captureRawContent(context, params) - assertNotNull(rawContent.rcDocument) - assertTrue(rawContent.rcDocument.isNotEmpty()) - } +`@SuppressLint("RestrictedApi")` to the test class: + +```kotlin +import android.annotation.SuppressLint +import android.content.Context +import androidx.glance.wear.core.ContainerInfo +import androidx.glance.wear.core.WearWidgetParams +import androidx.glance.wear.core.WidgetInstanceId +import androidx.test.core.app.ApplicationProvider +import kotlinx.coroutines.runBlocking +import org.junit.Assert.assertNotNull +import org.junit.Assert.assertTrue +import org.junit.Test +import org.junit.runner.RunWith +import org.robolectric.RobolectricTestRunner +import org.robolectric.annotation.Config + +@SuppressLint("RestrictedApi") +@RunWith(RobolectricTestRunner::class) +@Config(sdk = [34]) +class MyWidgetTest { + @Test + fun provideWidgetData_serializesRemoteComposeDocument() = runBlocking { + val context = ApplicationProvider.getApplicationContext() + val widget = MyWidget() + for (containerType in listOf( + ContainerInfo.CONTAINER_TYPE_SMALL, + ContainerInfo.CONTAINER_TYPE_LARGE, + )) { + val params = WearWidgetParams( + instanceId = WidgetInstanceId(namespace = "test", id = 1), + containerType = containerType, + widthDp = 220f, + heightDp = 120f, + horizontalPaddingDp = 8f, + verticalPaddingDp = 8f, + cornerRadiusDp = 16f, + ) + val data = widget.provideWidgetData(context, params) + val rawContent = data.captureRawContent(context, params) + assertNotNull(rawContent.rcDocument) + assertTrue(rawContent.rcDocument.isNotEmpty()) } - } \ No newline at end of file + } +} +``` \ No newline at end of file diff --git a/wear/wear-widgets/references/remote-compose-layouts.md b/wear/wear-widgets/references/remote-compose-layouts.md index 553eb81..9e39509 100644 --- a/wear/wear-widgets/references/remote-compose-layouts.md +++ b/wear/wear-widgets/references/remote-compose-layouts.md @@ -9,48 +9,48 @@ Inspect `params.containerType` in `provideWidgetData` to distinguish `ContainerInfo.CONTAINER_TYPE_LARGE` (`2x1` cards). Adapt layout density, padding, visible rows, and typography for each container size: -
- - import androidx.compose.remote.creation.compose.layout.RemoteAlignment - import androidx.compose.remote.creation.compose.layout.RemoteArrangement - import androidx.compose.remote.creation.compose.layout.RemoteColumn - import androidx.compose.remote.creation.compose.layout.RemoteComposable - import androidx.compose.remote.creation.compose.modifier.RemoteModifier - import androidx.compose.remote.creation.compose.modifier.fillMaxSize - import androidx.compose.remote.creation.compose.modifier.padding - import androidx.compose.remote.creation.compose.state.rc - import androidx.compose.remote.creation.compose.state.rdp - import androidx.compose.remote.creation.compose.state.rs - import androidx.compose.runtime.Composable - import androidx.compose.ui.graphics.Color - import androidx.wear.compose.remote.material3.RemoteText - - @RemoteComposable - @Composable - fun AdaptiveCardContent( - isLarge: Boolean, - title: String, - subtitle: String, +```kotlin +import androidx.compose.remote.creation.compose.layout.RemoteAlignment +import androidx.compose.remote.creation.compose.layout.RemoteArrangement +import androidx.compose.remote.creation.compose.layout.RemoteColumn +import androidx.compose.remote.creation.compose.layout.RemoteComposable +import androidx.compose.remote.creation.compose.modifier.RemoteModifier +import androidx.compose.remote.creation.compose.modifier.fillMaxSize +import androidx.compose.remote.creation.compose.modifier.padding +import androidx.compose.remote.creation.compose.state.rc +import androidx.compose.remote.creation.compose.state.rdp +import androidx.compose.remote.creation.compose.state.rs +import androidx.compose.runtime.Composable +import androidx.compose.ui.graphics.Color +import androidx.wear.compose.remote.material3.RemoteText + +@RemoteComposable +@Composable +fun AdaptiveCardContent( + isLarge: Boolean, + title: String, + subtitle: String, +) { + RemoteColumn( + modifier = RemoteModifier.fillMaxSize() + .padding(if (isLarge) 12.rdp else 6.rdp), + verticalArrangement = RemoteArrangement.Center, + horizontalAlignment = RemoteAlignment.CenterHorizontally, ) { - RemoteColumn( - modifier = RemoteModifier.fillMaxSize() - .padding(if (isLarge) 12.rdp else 6.rdp), - verticalArrangement = RemoteArrangement.Center, - horizontalAlignment = RemoteAlignment.CenterHorizontally, - ) { + RemoteText( + text = title.rs, + color = Color(0xFFE6E1E5).rc, + ) + if (isLarge) { RemoteText( - text = title.rs, - color = Color(0xFFE6E1E5).rc, + text = subtitle.rs, + color = Color(0xFFD0BCFF).rc, + modifier = RemoteModifier.padding(top = 4.rdp), ) - if (isLarge) { - RemoteText( - text = subtitle.rs, - color = Color(0xFFD0BCFF).rc, - modifier = RemoteModifier.padding(top = 4.rdp), - ) - } } } +} +``` ## 2. Unmasked full-bleed background images @@ -60,49 +60,49 @@ When a widget displays background artwork or photography: 2. Convert the `Bitmap` using `RemoteImageBitmap` (`bitmap.asImageBitmap`). Render it as the first child of a root `RemoteBox` using `RemoteImage` with `RemoteModifier.fillMaxSize`. 3. **NEVER** apply `CircleShape`, `RoundedCornerShape`, or manual `cornerRadius` clipping to full-bleed backgrounds. The Wear OS host automatically masks the widget surface to match the watch's round or squircle border. -
- - import android.graphics.Bitmap - import androidx.compose.remote.creation.compose.layout.RemoteAlignment - import androidx.compose.remote.creation.compose.layout.RemoteArrangement - import androidx.compose.remote.creation.compose.layout.RemoteBox - import androidx.compose.remote.creation.compose.layout.RemoteColumn - import androidx.compose.remote.creation.compose.layout.RemoteComposable - import androidx.compose.remote.creation.compose.layout.RemoteImage - import androidx.compose.remote.creation.compose.modifier.RemoteModifier - import androidx.compose.remote.creation.compose.modifier.fillMaxSize - import androidx.compose.remote.creation.compose.modifier.padding - import androidx.compose.remote.creation.compose.state.RemoteImageBitmap - import androidx.compose.remote.creation.compose.state.rdp - import androidx.compose.runtime.Composable - import androidx.compose.ui.graphics.asImageBitmap - - @RemoteComposable - @Composable - fun FullBleedBackgroundLayout( - backgroundBitmap: Bitmap?, - isLarge: Boolean, - content: @RemoteComposable @Composable () -> Unit, - ) { - RemoteBox(modifier = RemoteModifier.fillMaxSize()) { - if (backgroundBitmap != null) { - RemoteImage( - remoteBitmap = - RemoteImageBitmap(backgroundBitmap.asImageBitmap()), - contentDescription = null, - modifier = RemoteModifier.fillMaxSize(), - ) - } - RemoteColumn( - modifier = RemoteModifier.fillMaxSize() - .padding(if (isLarge) 12.rdp else 6.rdp), - verticalArrangement = RemoteArrangement.Center, - horizontalAlignment = RemoteAlignment.CenterHorizontally, - ) { - content() - } +```kotlin +import android.graphics.Bitmap +import androidx.compose.remote.creation.compose.layout.RemoteAlignment +import androidx.compose.remote.creation.compose.layout.RemoteArrangement +import androidx.compose.remote.creation.compose.layout.RemoteBox +import androidx.compose.remote.creation.compose.layout.RemoteColumn +import androidx.compose.remote.creation.compose.layout.RemoteComposable +import androidx.compose.remote.creation.compose.layout.RemoteImage +import androidx.compose.remote.creation.compose.modifier.RemoteModifier +import androidx.compose.remote.creation.compose.modifier.fillMaxSize +import androidx.compose.remote.creation.compose.modifier.padding +import androidx.compose.remote.creation.compose.state.RemoteImageBitmap +import androidx.compose.remote.creation.compose.state.rdp +import androidx.compose.runtime.Composable +import androidx.compose.ui.graphics.asImageBitmap + +@RemoteComposable +@Composable +fun FullBleedBackgroundLayout( + backgroundBitmap: Bitmap?, + isLarge: Boolean, + content: @RemoteComposable @Composable () -> Unit, +) { + RemoteBox(modifier = RemoteModifier.fillMaxSize()) { + if (backgroundBitmap != null) { + RemoteImage( + remoteBitmap = + RemoteImageBitmap(backgroundBitmap.asImageBitmap()), + contentDescription = null, + modifier = RemoteModifier.fillMaxSize(), + ) + } + RemoteColumn( + modifier = RemoteModifier.fillMaxSize() + .padding(if (isLarge) 12.rdp else 6.rdp), + verticalArrangement = RemoteArrangement.Center, + horizontalAlignment = RemoteAlignment.CenterHorizontally, + ) { + content() } } +} +``` ## 3. Responsive multi-item grids and rows @@ -112,55 +112,80 @@ Display fewer items in `SMALL` containers and the full grid in `LARGE` containers. Wrap each item in a `.clickable(onItemClick)` container so both `RemoteImage` and fallback `RemoteText` branches respond to clicks: -
- - @RemoteComposable - @Composable - fun GridWithBottomActionLayout( - items: List>, - isLarge: Boolean, - onItemClick: androidx.compose.remote.creation.compose.action.Action, - onMoreClick: androidx.compose.remote.creation.compose.action.Action, +```kotlin +import android.graphics.Bitmap +import androidx.compose.remote.creation.compose.action.Action +import androidx.compose.remote.creation.compose.layout.RemoteAlignment +import androidx.compose.remote.creation.compose.layout.RemoteArrangement +import androidx.compose.remote.creation.compose.layout.RemoteBox +import androidx.compose.remote.creation.compose.layout.RemoteColumn +import androidx.compose.remote.creation.compose.layout.RemoteComposable +import androidx.compose.remote.creation.compose.layout.RemoteImage +import androidx.compose.remote.creation.compose.layout.RemoteRow +import androidx.compose.remote.creation.compose.modifier.RemoteModifier +import androidx.compose.remote.creation.compose.modifier.background +import androidx.compose.remote.creation.compose.modifier.clickable +import androidx.compose.remote.creation.compose.modifier.fillMaxSize +import androidx.compose.remote.creation.compose.modifier.padding +import androidx.compose.remote.creation.compose.modifier.size +import androidx.compose.remote.creation.compose.state.RemoteImageBitmap +import androidx.compose.remote.creation.compose.state.rc +import androidx.compose.remote.creation.compose.state.rdp +import androidx.compose.remote.creation.compose.state.rs +import androidx.compose.runtime.Composable +import androidx.compose.ui.graphics.Color +import androidx.compose.ui.graphics.asImageBitmap +import androidx.wear.compose.remote.material3.RemoteButton +import androidx.wear.compose.remote.material3.RemoteText + +@RemoteComposable +@Composable +fun GridWithBottomActionLayout( + items: List>, + isLarge: Boolean, + onItemClick: Action, + onMoreClick: Action, +) { + val visibleItems = if (isLarge) items.take(6) else items.take(3) + val rows = visibleItems.chunked(3) + + RemoteColumn( + modifier = RemoteModifier.fillMaxSize().padding(8.rdp), + verticalArrangement = RemoteArrangement.Center, + horizontalAlignment = RemoteAlignment.CenterHorizontally, ) { - val visibleItems = if (isLarge) items.take(6) else items.take(3) - val rows = visibleItems.chunked(3) - - RemoteColumn( - modifier = RemoteModifier.fillMaxSize().padding(8.rdp), - verticalArrangement = RemoteArrangement.Center, - horizontalAlignment = RemoteAlignment.CenterHorizontally, - ) { - for (rowItems in rows) { - RemoteRow( - horizontalArrangement = RemoteArrangement.spacedBy(8.rdp), - verticalAlignment = RemoteAlignment.CenterVertically, - ) { - for ((label, avatar) in rowItems) { - RemoteBox( - modifier = RemoteModifier.size(36.rdp) - .background(Color(0xFF4F378B).rc) - .clickable(onItemClick), - contentAlignment = RemoteAlignment.Center, - ) { - if (avatar != null) { - RemoteImage( - remoteBitmap = - RemoteImageBitmap(avatar.asImageBitmap()), - contentDescription = label.rs, - modifier = RemoteModifier.fillMaxSize(), - ) - } else { - RemoteText(text = label.rs, color = Color.White.rc) - } + for (rowItems in rows) { + RemoteRow( + horizontalArrangement = RemoteArrangement.spacedBy(8.rdp), + verticalAlignment = RemoteAlignment.CenterVertically, + ) { + for ((label, avatar) in rowItems) { + RemoteBox( + modifier = RemoteModifier.size(36.rdp) + .background(Color(0xFF4F378B).rc) + .clickable(onItemClick), + contentAlignment = RemoteAlignment.Center, + ) { + if (avatar != null) { + RemoteImage( + remoteBitmap = + RemoteImageBitmap(avatar.asImageBitmap()), + contentDescription = label.rs, + modifier = RemoteModifier.fillMaxSize(), + ) + } else { + RemoteText(text = label.rs, color = Color.White.rc) } } } } - RemoteButton( - onClick = onMoreClick, - modifier = RemoteModifier.padding(top = 6.rdp), - ) { - RemoteText(text = "More".rs) - } } - } \ No newline at end of file + RemoteButton( + onClick = onMoreClick, + modifier = RemoteModifier.padding(top = 6.rdp), + ) { + RemoteText(text = "More".rs) + } + } +} +``` \ No newline at end of file diff --git a/wear/wear-widgets/references/state-and-actions.md b/wear/wear-widgets/references/state-and-actions.md index daae02b..c4730ff 100644 --- a/wear/wear-widgets/references/state-and-actions.md +++ b/wear/wear-widgets/references/state-and-actions.md @@ -10,45 +10,45 @@ Mutate it directly inside the host player using - Call `remoteInt.toRemoteString` as a **member function** on `RemoteInt` (for example, `"Label: ".rs + remoteInt.toRemoteString`). **NEVER** import `toRemoteString` as a top-level extension. -
- - import androidx.compose.remote.creation.compose.action.valueChange - import androidx.compose.remote.creation.compose.layout.RemoteAlignment - import androidx.compose.remote.creation.compose.layout.RemoteArrangement - import androidx.compose.remote.creation.compose.layout.RemoteColumn - import androidx.compose.remote.creation.compose.layout.RemoteComposable - import androidx.compose.remote.creation.compose.layout.RemoteRow - import androidx.compose.remote.creation.compose.modifier.RemoteModifier - import androidx.compose.remote.creation.compose.modifier.fillMaxSize - import androidx.compose.remote.creation.compose.state.rdp - import androidx.compose.remote.creation.compose.state.rememberMutableRemoteInt - import androidx.compose.remote.creation.compose.state.rs - import androidx.compose.runtime.Composable - import androidx.wear.compose.remote.material3.RemoteButton - import androidx.wear.compose.remote.material3.RemoteText - - @RemoteComposable - @Composable - fun StatefulAdjusterSection() { - val value = rememberMutableRemoteInt(0) - - RemoteColumn( - modifier = RemoteModifier.fillMaxSize(), - verticalArrangement = RemoteArrangement.Center, - horizontalAlignment = RemoteAlignment.CenterHorizontally, - ) { - // Call value.toRemoteString() as a member method (no extension import) - RemoteText(text = "Value: ".rs + value.toRemoteString()) - RemoteRow(horizontalArrangement = RemoteArrangement.spacedBy(8.rdp)) { - RemoteButton(onClick = valueChange(value, value - 1)) { - RemoteText("-".rs) - } - RemoteButton(onClick = valueChange(value, value + 1)) { - RemoteText("+".rs) - } +```kotlin +import androidx.compose.remote.creation.compose.action.valueChange +import androidx.compose.remote.creation.compose.layout.RemoteAlignment +import androidx.compose.remote.creation.compose.layout.RemoteArrangement +import androidx.compose.remote.creation.compose.layout.RemoteColumn +import androidx.compose.remote.creation.compose.layout.RemoteComposable +import androidx.compose.remote.creation.compose.layout.RemoteRow +import androidx.compose.remote.creation.compose.modifier.RemoteModifier +import androidx.compose.remote.creation.compose.modifier.fillMaxSize +import androidx.compose.remote.creation.compose.state.rdp +import androidx.compose.remote.creation.compose.state.rememberMutableRemoteInt +import androidx.compose.remote.creation.compose.state.rs +import androidx.compose.runtime.Composable +import androidx.wear.compose.remote.material3.RemoteButton +import androidx.wear.compose.remote.material3.RemoteText + +@RemoteComposable +@Composable +fun StatefulAdjusterSection() { + val value = rememberMutableRemoteInt(0) + + RemoteColumn( + modifier = RemoteModifier.fillMaxSize(), + verticalArrangement = RemoteArrangement.Center, + horizontalAlignment = RemoteAlignment.CenterHorizontally, + ) { + // Call value.toRemoteString() as a member method (no extension import) + RemoteText(text = "Value: ".rs + value.toRemoteString()) + RemoteRow(horizontalArrangement = RemoteArrangement.spacedBy(8.rdp)) { + RemoteButton(onClick = valueChange(value, value - 1)) { + RemoteText("-".rs) + } + RemoteButton(onClick = valueChange(value, value + 1)) { + RemoteText("+".rs) } } } +} +``` ## 2. Activity launches and background refresh callbacks (`pendingIntentAction`) @@ -59,79 +59,79 @@ dispatch a background `BroadcastReceiver` refresh when tapped: - Attach the returned `Action` either to `RemoteButton(onClick = action)` or to any container using `RemoteModifier.clickable(action)`. - In a `BroadcastReceiver`, call `goAsync` and invoke `widget.triggerUpdateAll(context)` to refresh all active widget instances. Register the `` in `AndroidManifest.xml` when using a manifest broadcast. -
- - import android.app.PendingIntent - import android.content.BroadcastReceiver - import android.content.Context - import android.content.Intent - import androidx.compose.remote.creation.compose.action.pendingIntentAction - import androidx.compose.remote.creation.compose.layout.RemoteColumn - import androidx.compose.remote.creation.compose.layout.RemoteComposable - import androidx.compose.remote.creation.compose.modifier.RemoteModifier - import androidx.compose.remote.creation.compose.modifier.clickable - import androidx.compose.remote.creation.compose.modifier.fillMaxSize - import androidx.compose.remote.creation.compose.state.rs - import androidx.compose.runtime.Composable - import androidx.wear.compose.remote.material3.RemoteButton - import androidx.wear.compose.remote.material3.RemoteText - import kotlinx.coroutines.CoroutineScope - import kotlinx.coroutines.Dispatchers - import kotlinx.coroutines.launch - - @RemoteComposable - @Composable - fun InteractiveHeaderAndRefreshSection( - activityClass: Class<*>, - receiverClass: Class, - ) { - val launchActivityAction = pendingIntentAction { ctx -> - val intent = Intent(ctx, activityClass).apply { - flags = - Intent.FLAG_ACTIVITY_NEW_TASK or Intent.FLAG_ACTIVITY_CLEAR_TOP - } - PendingIntent.getActivity( - ctx, - 100, - intent, - PendingIntent.FLAG_UPDATE_CURRENT or PendingIntent.FLAG_IMMUTABLE, - ) +```kotlin +import android.app.PendingIntent +import android.content.BroadcastReceiver +import android.content.Context +import android.content.Intent +import androidx.compose.remote.creation.compose.action.pendingIntentAction +import androidx.compose.remote.creation.compose.layout.RemoteColumn +import androidx.compose.remote.creation.compose.layout.RemoteComposable +import androidx.compose.remote.creation.compose.modifier.RemoteModifier +import androidx.compose.remote.creation.compose.modifier.clickable +import androidx.compose.remote.creation.compose.modifier.fillMaxSize +import androidx.compose.remote.creation.compose.state.rs +import androidx.compose.runtime.Composable +import androidx.wear.compose.remote.material3.RemoteButton +import androidx.wear.compose.remote.material3.RemoteText +import kotlinx.coroutines.CoroutineScope +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.launch + +@RemoteComposable +@Composable +fun InteractiveHeaderAndRefreshSection( + activityClass: Class<*>, + receiverClass: Class, +) { + val launchActivityAction = pendingIntentAction { ctx -> + val intent = Intent(ctx, activityClass).apply { + flags = + Intent.FLAG_ACTIVITY_NEW_TASK or Intent.FLAG_ACTIVITY_CLEAR_TOP } + PendingIntent.getActivity( + ctx, + 100, + intent, + PendingIntent.FLAG_UPDATE_CURRENT or PendingIntent.FLAG_IMMUTABLE, + ) + } - val backgroundRefreshAction = pendingIntentAction { ctx -> - val intent = Intent(ctx, receiverClass).apply { - action = ACTION_REFRESH_WIDGET - } - PendingIntent.getBroadcast( - ctx, - 101, - intent, - PendingIntent.FLAG_UPDATE_CURRENT or PendingIntent.FLAG_IMMUTABLE, - ) + val backgroundRefreshAction = pendingIntentAction { ctx -> + val intent = Intent(ctx, receiverClass).apply { + action = ACTION_REFRESH_WIDGET } + PendingIntent.getBroadcast( + ctx, + 101, + intent, + PendingIntent.FLAG_UPDATE_CURRENT or PendingIntent.FLAG_IMMUTABLE, + ) + } - RemoteColumn(modifier = RemoteModifier.fillMaxSize()) { - RemoteText( - text = "Open App".rs, - modifier = RemoteModifier.clickable(launchActivityAction), - ) - RemoteButton(onClick = backgroundRefreshAction) { - RemoteText(text = "Refresh".rs) - } + RemoteColumn(modifier = RemoteModifier.fillMaxSize()) { + RemoteText( + text = "Open App".rs, + modifier = RemoteModifier.clickable(launchActivityAction), + ) + RemoteButton(onClick = backgroundRefreshAction) { + RemoteText(text = "Refresh".rs) } } - - const val ACTION_REFRESH_WIDGET = "com.example.wear.ACTION_REFRESH_WIDGET" - - class WidgetUpdateReceiver : BroadcastReceiver() { - override fun onReceive(context: Context, intent: Intent) { - val pendingResult = goAsync() - CoroutineScope(Dispatchers.Default).launch { - try { - MyWidget().triggerUpdateAll(context) - } finally { - pendingResult.finish() - } +} + +const val ACTION_REFRESH_WIDGET = "com.example.wear.ACTION_REFRESH_WIDGET" + +class WidgetUpdateReceiver : BroadcastReceiver() { + override fun onReceive(context: Context, intent: Intent) { + val pendingResult = goAsync() + CoroutineScope(Dispatchers.Default).launch { + try { + MyWidget().triggerUpdateAll(context) + } finally { + pendingResult.finish() } } - } \ No newline at end of file + } +} +``` \ No newline at end of file diff --git a/wear/wear-widgets/references/widget-service-and-manifest.md b/wear/wear-widgets/references/widget-service-and-manifest.md index e4a6a71..0c2c7bb 100644 --- a/wear/wear-widgets/references/widget-service-and-manifest.md +++ b/wear/wear-widgets/references/widget-service-and-manifest.md @@ -5,39 +5,40 @@ Every Wear OS widget requires two classes: 1. A `GlanceWearWidget` subclass overriding `provideWidgetData(context, params)` and returning a `WearWidgetDocument`. 2. A `GlanceWearWidgetService` subclass annotated with `@AssociateWithGlanceWearWidget` so build tools, linters, and `GlanceWearWidgetManager` resolve the widget provider statically without reflective instantiation. -
- - import android.content.Context - import androidx.compose.remote.creation.compose.state.rc - import androidx.compose.ui.graphics.Color - import androidx.glance.wear.AssociateWithGlanceWearWidget - import androidx.glance.wear.GlanceWearWidget - import androidx.glance.wear.GlanceWearWidgetService - import androidx.glance.wear.WearWidgetBrush - import androidx.glance.wear.WearWidgetData - import androidx.glance.wear.WearWidgetDocument - import androidx.glance.wear.color - import androidx.glance.wear.core.ContainerInfo - import androidx.glance.wear.core.WearWidgetParams - - @AssociateWithGlanceWearWidget(MyWidget::class) - class MyWidgetService : GlanceWearWidgetService() { - override val widget: GlanceWearWidget = MyWidget() - } - - class MyWidget : GlanceWearWidget() { - override suspend fun provideWidgetData( - context: Context, - params: WearWidgetParams, - ): WearWidgetData { - val isLarge = params.containerType == ContainerInfo.CONTAINER_TYPE_LARGE - return WearWidgetDocument( - background = WearWidgetBrush.color(Color(0xFF1C1B1F).rc) - ) { - MyWidgetContent(isLarge = isLarge) - } +```kotlin +import android.content.Context +import androidx.compose.remote.creation.compose.state.rc +import androidx.compose.ui.graphics.Color +import androidx.glance.wear.AssociateWithGlanceWearWidget +import androidx.glance.wear.GlanceWearWidget +import androidx.glance.wear.GlanceWearWidgetService +import androidx.glance.wear.WearWidgetBrush +import androidx.glance.wear.WearWidgetData +import androidx.glance.wear.WearWidgetDocument +import androidx.glance.wear.color +import androidx.glance.wear.core.ContainerInfo +import androidx.glance.wear.core.WearWidgetParams + + +@AssociateWithGlanceWearWidget(MyWidget::class) +class MyWidgetService : GlanceWearWidgetService() { + override val widget: GlanceWearWidget = MyWidget() +} + +class MyWidget : GlanceWearWidget() { + override suspend fun provideWidgetData( + context: Context, + params: WearWidgetParams, + ): WearWidgetData { + val isLarge = params.containerType == ContainerInfo.CONTAINER_TYPE_LARGE + return WearWidgetDocument( + background = WearWidgetBrush.color(Color(0xFF1C1B1F).rc) + ) { + MyWidgetContent(isLarge = isLarge) } } +} +``` ## 2. AndroidManifest.xml registration @@ -48,20 +49,19 @@ Register the `GlanceWearWidgetService` subclass in `androidx.glance.wear.widget.provider` `` pointing to the `@xml/` provider configuration file: -
- - - +```xml + + - - + - + +``` ## 3. Provider XML (`res/xml/_info.xml`) @@ -69,21 +69,20 @@ Create the `` XML resource in `app/src/main/res/xml/` declaring `SMALL` and `LARGE` `` elements with `@drawable/` picker preview images: -
- - - - - - +```xml + + + + +``` ## 4. Migrating a legacy ProtoLayout TileService diff --git a/xr/display-glasses-with-jetpack-compose-glimmer/SKILL.md b/xr/display-glasses-with-jetpack-compose-glimmer/SKILL.md index 2be3179..dfebcbd 100644 --- a/xr/display-glasses-with-jetpack-compose-glimmer/SKILL.md +++ b/xr/display-glasses-with-jetpack-compose-glimmer/SKILL.md @@ -1,15 +1,16 @@ --- name: display-glasses-with-jetpack-compose-glimmer description: Provides guidelines for developing projected Android XR apps for display - glasses using the Jetpack Compose Glimmer UI toolkit. This skill covers foundational - Glimmer design principles, workflows for implementing Jetpack Compose Glimmer, and - interaction models for the glasses form factor. Use this skill to build an Android - XR Augmented Experience app with Jetpack Compose Glimmer that adheres to the Glimmer - design system for optimized glasses styling. + glasses using the Jetpack Compose Glimmer UI toolkit. Covers Glimmer design principles, + workflows, and interaction models for the glasses form factor. Use this skill to + build an Android XR Augmented Experience app with Jetpack Compose Glimmer. Don't + use for audio-only AI glasses without a display, non-glasses Android XR headsets + (spatial panels or 3D volumes with Jetpack Compose for XR), or phone and tablet + Material 3 UIs. license: Complete terms in LICENSE.txt metadata: author: Google LLC - last-updated: '2026-10-11' + last-updated: '2026-10-17' keywords: - Jetpack Compose Glimmer - display glasses diff --git a/xr/display-glasses-with-jetpack-compose-glimmer/references/android/develop/xr/jetpack-xr-sdk/access-hardware-projected-context.md b/xr/display-glasses-with-jetpack-compose-glimmer/references/android/develop/xr/jetpack-xr-sdk/access-hardware-projected-context.md index 946617c..4fcd87d 100644 --- a/xr/display-glasses-with-jetpack-compose-glimmer/references/android/develop/xr/jetpack-xr-sdk/access-hardware-projected-context.md +++ b/xr/display-glasses-with-jetpack-compose-glimmer/references/android/develop/xr/jetpack-xr-sdk/access-hardware-projected-context.md @@ -77,106 +77,18 @@ context. ### Re-initialize on reconnect -When the glasses reconnect, your app can obtain another projected -context instance using [`createProjectedDeviceContext`](https://developer.android.com/reference/kotlin/androidx/xr/projected/ProjectedContext#createProjectedDeviceContext(android.content.Context)), and then -re-initialize any system services or resources using the new projected context. +When the glasses reconnect, your app can obtain another projected context +instance using [`createProjectedDeviceContext`](https://developer.android.com/reference/kotlin/androidx/xr/projected/ProjectedContext#createProjectedDeviceContext(android.content.Context)), and then re-initialize any +system services or resources using the new projected context. ## Record audio with the glasses' microphone -You can record audio from the glasses using two distinct methods: - -- Use a [projected context](https://developer.android.com/develop/xr/jetpack-xr-sdk/glasses/support-different-types#projected-context). -- Use Bluetooth Hands-Free Profile (HFP). - -### Choose a recording methods - -The method you choose depends on whether you need high-fidelity, XR-specific -audio processing, or standard Bluetooth audio input. - -| Recording method | Microphone access | Common use case | -|---|---|---| -| Projected Context | Multiple microphones | Recording using a projected context lets your app access multiple microphones from the glasses and its specialized hardware features, such as: - XR-specific spatialization. - Advanced denoising. - Voice separation that isolates the wearer's voice. - Maintain recording access in multidevice environments even when the glasses are not the active Bluetooth device. | -| Bluetooth HFP | Single microphone | Relies on the Bluetooth Hands-Free Profile (HFP) for immediate, out-of-the-box compatibility. In this mode, the glasses connect to the phone using standard Headset and Advanced Audio Distribution Profile (A2DP) [profiles](https://developer.android.com/develop/connectivity/bluetooth/profiles), functioning like a typical Bluetooth peripheral. If your app is already designed for standard Bluetooth recording, you can use this method to record audio from the glasses without integrating any XR-specific capabilities. | - -### Record audio using a projected context - -To record audio using a projected context, first request the required runtime -permissions, and then record the audio using the [`AudioRecord`](https://developer.android.com/reference/kotlin/android/media/AudioRecord) API, as -described in the following sections. - -#### Request runtime permissions - -To access multiple microphones on the glasses, you must request audio -permissions specifically for the projected device. The standard, phone-scoped -`RECORD_AUDIO` permission that a user has granted for your app on their mobile -device is insufficient. - -Follow these steps to request the permissions: - -1. [Declare the `RECORD_AUDIO` permission](https://developer.android.com/develop/xr/jetpack-xr-sdk/request-hardware-permissions#declare-permissions) in your app's manifest file. -2. Request the projected-device-scoped permissions in one of the following - ways, depending on where your code is executing: - - - **Code executing from a projected activity** : Use the [`ActivityResultLauncher`](https://developer.android.com/reference/kotlin/androidx/activity/result/ActivityResultLauncher) with the [`ProjectedPermissionsResultContract`](https://developer.android.com/reference/kotlin/androidx/xr/projected/permissions/ProjectedPermissionsResultContract#ProjectedPermissionsResultContract()). For more information on using this method, see the [register the permissions launcher](https://developer.android.com/develop/xr/jetpack-xr-sdk/request-hardware-permissions#register) section and subsequent sections in the guide for requesting hardware permissions. - - **Code executing from a host phone activity** : Use [`Activity#requestPermissions(permissions, requestCode, deviceId)`](https://developer.android.com/reference/kotlin/android/app/Activity#requestpermissions_1) and provide the device ID obtained from your `projectedDeviceContext`, as described in the [understand the permission request user flow](https://developer.android.com/develop/xr/jetpack-xr-sdk/request-hardware-permissions#permissions-user-flow) section of the guide for requesting hardware permissions. - -#### Initialize AudioRecord with a projected context - -To ensure that audio is recorded from the glasses rather than the host phone, -you must associate the `AudioRecord` object with the projected device context. - -The following code uses the [`AudioRecord.Builder`](https://developer.android.com/reference/kotlin/android/media/AudioRecord.Builder) and passes the -`projectedDeviceContext` to the [`setContext`](https://developer.android.com/reference/kotlin/android/media/AudioRecord.Builder#setcontext) method: - - -```kotlin -// Initialize AudioRecord with projected device context -val audioRecord = AudioRecord.Builder() - .setAudioSource(MediaRecorder.AudioSource.CAMCORDER) - .setAudioFormat(audioFormat) - .setBufferSizeInBytes(bufferSize) - // pass in the projected device context - .setContext(projectedDeviceContext) - .build() - -audioRecord.startRecording() -``` - -
- -##### Key points about the code - -- You can set the audio source to [`CAMCORDER`](https://developer.android.com/reference/kotlin/android/media/MediaRecorder.AudioSource#camcorder), - [`VOICE_RECOGNITION`](https://developer.android.com/reference/kotlin/android/media/MediaRecorder.AudioSource#voice_recognition), [`VOICE_COMMUNICATION`](https://developer.android.com/reference/kotlin/android/media/MediaRecorder.AudioSource#voice_communication), or - [`UNPROCESSED`](https://developer.android.com/reference/kotlin/android/media/MediaRecorder.AudioSource#unprocessed) to tailor the audio processing to your specific use - case. - - For example, use `VOICE_COMMUNICATION` if your use case needs automatic - noise reduction or to isolate the wearer's voice. `VOICE_RECOGNITION` is - processed with acoustic echo cancellation (AEC), which can be useful when - there's concurrent audio playback while recording. And if you need raw, - unaltered audio, select `UNPROCESSED` or `CAMCORDER`. -- To ensure compatibility with the glasses, the `audioFormat` object must - define a sample rate of 16kHz and a channel configuration of either mono or - stereo (using [`CHANNEL_IN_MONO`](https://developer.android.com/reference/kotlin/android/media/AudioFormat#channel_in_mono) or [`CHANNEL_IN_STEREO`](https://developer.android.com/reference/kotlin/android/media/AudioFormat#channel_in_stereo)). - -- Use [`AudioRecord.getMinBufferSize()`](https://developer.android.com/reference/kotlin/android/media/AudioRecord#getminbuffersize) to determine the minimum buffer - size to create the `AudioRecord` object. However, to prevent audio drops - from the glasses, you should read from this buffer in short, frequent chunks - (ideally 20ms slices) rather than waiting for the entire buffer to fill. - -> [!WARNING] -> **Preview:** Support for using the [`MediaRecorder`](https://developer.android.com/reference/kotlin/android/media/MediaRecorder) API with a projected context will be available in an upcoming Android Beta release. - -##### Clean up after use - -When your app no longer needs the microphone, or when the activity is stopped, -call [`stop`](https://developer.android.com/reference/kotlin/android/media/AudioRecord#stop) and [`release`](https://developer.android.com/reference/kotlin/android/media/AudioRecord#release) on the `AudioRecord` object. - -##### Check for runtime permissions before recording - -Before calling [`startRecording`](https://developer.android.com/reference/kotlin/android/media/AudioRecord#startrecording), [verify that the user has granted the -microphone permission for the glasses](https://developer.android.com/develop/xr/jetpack-xr-sdk/request-hardware-permissions#check-function) using the projected context. +You can record audio from the glasses using Bluetooth Hands-Free Profile (HFP). +In this mode, the glasses connect to the phone using standard Headset and +Advanced Audio Distribution Profile (A2DP) profiles, functioning like a typical +Bluetooth peripheral. If your app is already designed for standard Bluetooth +recording, you can use this method to record audio from the glasses without any +additional development. ### Record audio using Bluetooth HFP @@ -330,9 +242,9 @@ private fun startCameraOnGlasses(activity: ComponentActivity) { - Binds the `ImageCapture` use case to the activity's lifecycle. This automatically manages the opening and closing of the camera based on the activity's state (for example, stopping the camera when the activity is paused). - **Configuring for single-stream hardware:** The glasses camera pipeline is constrained to a single active stream at a time. When `ImageCapture` is bound alone, CameraX attaches an internal repeating stream (`MeteringRepeating`) for focus and metering by default, which causes the single-stream pipeline to stall. To bind `ImageCapture` by itself without a preview stream, disable the forced repeating stream by setting [`setRepeatingStreamForced(false)`](https://developer.android.com/reference/kotlin/androidx/camera/core/CameraXConfig.Builder#setRepeatingStreamForced(kotlin.Boolean)) on [`CameraXConfig.Builder`](https://developer.android.com/reference/kotlin/androidx/camera/core/CameraXConfig.Builder) (for example, by implementing [`CameraXConfig.Provider`](https://developer.android.com/reference/kotlin/androidx/camera/core/CameraXConfig.Provider) in your `Application` class or configuring `ProcessCameraProvider` before initialization). -After the glasses' camera is set up, you can capture an image with the -CameraX's `ImageCapture` class. Refer to the CameraX's documentation to learn -about using [`takePicture`](https://developer.android.com/media/camera/camerax/take-photo#take_a_picture) to [capture an image](https://developer.android.com/media/camera/camerax/take-photo#take_a_picture). +After the glasses' camera is set up, you can capture an image with the CameraX's +`ImageCapture` class. Refer to the CameraX's documentation to learn about using +[`takePicture`](https://developer.android.com/media/camera/camerax/take-photo#take_a_picture) to [capture an image](https://developer.android.com/media/camera/camerax/take-photo#take_a_picture). ### Capture a video with the glasses' camera @@ -380,9 +292,9 @@ private fun getPhoneContext(activity: ComponentActivity): Context? {
When accessing hardware or resources that are specific to the host device -(phone) in a hybrid app (an app containing both mobile and glasses -experiences), you must explicitly select the correct context to make sure your -app can access the correct hardware: +(phone) in a hybrid app (an app containing both mobile and glasses experiences), +you must explicitly select the correct context to make sure your app can access +the correct hardware: - Use the [`Activity`](https://developer.android.com/reference/kotlin/android/app/Activity) context from the phone `Activity` or the [`ProjectedContext.createHostDeviceContext`](https://developer.android.com/reference/kotlin/androidx/xr/projected/ProjectedContext#createHostDeviceContext(android.content.Context)) to get the phone's context. - Don't use [`getApplicationContext`](https://developer.android.com/reference/kotlin/android/content/Context#getapplicationcontext) because the application context can incorrectly return the glasses' context if a projected activity was the most-recently-launched component. \ No newline at end of file diff --git a/xr/display-glasses-with-jetpack-compose-glimmer/references/card-source.md b/xr/display-glasses-with-jetpack-compose-glimmer/references/card-source.md index 2b11e4a..8ed12a4 100644 --- a/xr/display-glasses-with-jetpack-compose-glimmer/references/card-source.md +++ b/xr/display-glasses-with-jetpack-compose-glimmer/references/card-source.md @@ -780,7 +780,7 @@ private fun CardImpl( } Row( modifier = Modifier.fillMaxWidth().padding(innerPadding), - verticalAlignment = CenterVertically, + verticalAlignment = Alignment.CenterVertically, ) { if (leadingIcon != null) { Box( @@ -861,8 +861,9 @@ private fun SmallImageCardImpl( ) { if (isLeading) { Box( - modifier = Modifier.weight(ImageWidthFraction).clip(ImageShape), - contentAlignment = Alignment.Center, + modifier = + Modifier.align(Alignment.Top).weight(ImageWidthFraction).clip(ImageShape), + contentAlignment = Alignment.TopStart, ) { image() } @@ -899,8 +900,9 @@ private fun SmallImageCardImpl( if (!isLeading) { Box( - modifier = Modifier.weight(ImageWidthFraction).clip(ImageShape), - contentAlignment = Alignment.Center, + modifier = + Modifier.align(Alignment.Top).weight(ImageWidthFraction).clip(ImageShape), + contentAlignment = Alignment.TopStart, ) { image() } diff --git a/xr/display-glasses-with-jetpack-compose-glimmer/references/projectedcontext-source.md b/xr/display-glasses-with-jetpack-compose-glimmer/references/projectedcontext-source.md index bb03e93..29be855 100644 --- a/xr/display-glasses-with-jetpack-compose-glimmer/references/projectedcontext-source.md +++ b/xr/display-glasses-with-jetpack-compose-glimmer/references/projectedcontext-source.md @@ -72,7 +72,11 @@ public object ProjectedContext { val deviceId = getProjectedDeviceId(context) ?: throw IllegalStateException("Projected device not found.") - return context.createDeviceContext(deviceId) + return try { + context.createDeviceContext(deviceId) + } catch (e: IllegalArgumentException) { + throw IllegalStateException("Projected device not found.", e) + } } /** diff --git a/xr/display-glasses-with-jetpack-compose-glimmer/references/surface-source.md b/xr/display-glasses-with-jetpack-compose-glimmer/references/surface-source.md index 3f5b368..f54498f 100644 --- a/xr/display-glasses-with-jetpack-compose-glimmer/references/surface-source.md +++ b/xr/display-glasses-with-jetpack-compose-glimmer/references/surface-source.md @@ -41,7 +41,9 @@ import androidx.compose.runtime.Immutable import androidx.compose.runtime.LaunchedEffect import androidx.compose.runtime.derivedStateOf import androidx.compose.runtime.getValue +import androidx.compose.runtime.mutableStateOf import androidx.compose.runtime.remember +import androidx.compose.runtime.setValue import androidx.compose.ui.Modifier import androidx.compose.ui.draw.clip import androidx.compose.ui.geometry.Size @@ -60,7 +62,6 @@ import androidx.compose.ui.graphics.layer.GraphicsLayer import androidx.compose.ui.graphics.layer.drawLayer import androidx.compose.ui.graphics.lerp import androidx.compose.ui.layout.layout -import androidx.compose.ui.node.DelegatingNode import androidx.compose.ui.node.DrawModifierNode import androidx.compose.ui.node.ModifierNodeElement import androidx.compose.ui.node.invalidateDraw @@ -294,7 +295,7 @@ private class SurfaceNode( private var focusedContentColor: Color, private var shape: Shape, private var interactionSource: InteractionSource?, -) : DrawModifierNode, DelegatingNode() { +) : Modifier.Node(), DrawModifierNode, ContentColorProviderNode { override val shouldAutoInvalidate = false @@ -303,10 +304,10 @@ private class SurfaceNode( if (field != value) { field = value if (value) { - contentColorNode.update(focusedContentColor) + currentContentColor = focusedContentColor startFocusAnimation() } else { - contentColorNode.update(contentColor) + currentContentColor = contentColor stopFocusAnimation() } // No need to invalidate the border cache - we build it ahead of time to account for @@ -316,7 +317,8 @@ private class SurfaceNode( } } - private val contentColorNode = delegate(ContentColorProviderNode(contentColor)) + override var currentContentColor: Color by mutableStateOf(contentColor) + private set // Cache borders and highlight for unfocused and focused states. This means we // can avoid recreating these for a given surface, if the border and shape never @@ -397,11 +399,11 @@ private class SurfaceNode( } if (this.contentColor != contentColor) { this.contentColor = contentColor - if (!isFocused) contentColorNode.update(contentColor) + if (!isFocused) currentContentColor = contentColor } if (this.focusedContentColor != focusedContentColor) { this.focusedContentColor = focusedContentColor - if (isFocused) contentColorNode.update(focusedContentColor) + if (isFocused) currentContentColor = focusedContentColor } if (this.shape != shape) { this.shape = shape diff --git a/xr/projected-hardware/SKILL.md b/xr/projected-hardware/SKILL.md new file mode 100644 index 0000000..12ff1be --- /dev/null +++ b/xr/projected-hardware/SKILL.md @@ -0,0 +1,43 @@ +--- +name: projected-hardware +description: Provides instructions and workflows for accessing and managing hardware + on Android XR projected devices, such as audio glasses and display glasses. Use + this skill when building experiences for intelligent eyewear in an Android app and + you need to access device hardware like the camera and microphone, handle glasses-specific + runtime permissions, or manage connection lifecycles using projected context. +license: Complete terms in LICENSE.txt +metadata: + author: Google LLC + last-updated: '2026-10-17' + keywords: + - Android XR + - intelligent eyewear + - audio glasses + - display glasses + - glasses hardware + - ProjectedContext + - projected context + - runtime permissions + - CameraX + - audio recording +--- + +## Prerequisites + +- You must use a `compileSdk` of 36 or higher. + +## Connection and lifecycle management + +Intelligent eyewear hardware is only accessible while the glasses are connected +to the host phone. Monitor connection state and manage resource lifecycles using +`ProjectedContext`. You must read +[Understand projected activities and projected contexts](references/android/develop/xr/jetpack-xr-sdk/glasses/support-different-types.md) to understand +how to get a projected context and [how to manage its lifecycle](references/android/develop/xr/jetpack-xr-sdk/glasses/check-availability.md). + +## Request glasses hardware permissions + +- **Hardware permissions and opt-in** : You must read [Request hardware permissions](references/android/develop/xr/jetpack-xr-sdk/request-hardware-permissions.md) to understand how to declare camera and microphone runtime permissions in the Android Manifest file, request runtime permissions in different scenarios, and handle permission results. + +## Glasses hardware access and capabilities + +- **Camera capture and CameraX** : You must read [Capture an image with the glasses' camera](references/android/develop/xr/jetpack-xr-sdk/access-hardware-projected-context.md) to understand the implementation patterns for capturing a photo with the glasses' camera using a `ProjectedContext`. diff --git a/xr/projected-hardware/references/android/develop/xr/jetpack-xr-sdk/access-hardware-projected-context.md b/xr/projected-hardware/references/android/develop/xr/jetpack-xr-sdk/access-hardware-projected-context.md new file mode 100644 index 0000000..4fcd87d --- /dev/null +++ b/xr/projected-hardware/references/android/develop/xr/jetpack-xr-sdk/access-hardware-projected-context.md @@ -0,0 +1,300 @@ +Applicable XR devices This guidance helps you build experiences for these types of XR devices. [Learn about XR device types →](https://developer.android.com/develop/xr/devices) ![](https://developer.android.com/static/images/develop/xr/ai-glasses-icon.svg) Audio \& +Display Glasses [](https://developer.android.com/develop/xr/devices#audio-display) [Learn about XR device types →](https://developer.android.com/develop/xr/devices) + +
+ +After you've [requested and been granted the necessary permissions](https://developer.android.com/develop/xr/jetpack-xr-sdk/request-hardware-permissions), your app +can access the hardware on the audio glasses or display glasses. The key to +accessing the glasses' hardware (instead of the phone's hardware), is to use a +[projected context](https://developer.android.com/develop/xr/jetpack-xr-sdk/glasses/support-different-types#projected-context). + +There are two primary ways to get a projected context, depending on where your +code is executing: + +## Get a projected context if your code is running in an projected activity + +If your app's code is running from within your [projected activity](https://developer.android.com/develop/xr/jetpack-xr-sdk/glasses/support-different-types#activity-lifecycle), its own +activity context is already a [projected context](https://developer.android.com/develop/xr/jetpack-xr-sdk/glasses/support-different-types#projected-context). In this scenario, calls +made within that activity can already access the glasses' hardware. + +## Get a projected context for code running in a phone app component + +If a part of your app outside of your projected activity (such as a phone +activity or a service) needs to access the glasses' hardware, it must explicitly +obtain a projected context. To do this, use the +[`createProjectedDeviceContext`](https://developer.android.com/reference/kotlin/androidx/xr/projected/ProjectedContext#createProjectedDeviceContext(android.content.Context)) method: + + +```kotlin +@RequiresApi(Build.VERSION_CODES.BAKLAVA) +@OptIn(ExperimentalProjectedApi::class, ExperimentalCoroutinesApi::class) +private fun monitorProjectedConnectivity(activity: ComponentActivity) { + activity.lifecycleScope.launch { + // Before creating a projected context, check to see if the projected device is connected. + // While this method returns true, the projected context remains valid. + ProjectedContext.isProjectedDeviceConnected(activity, coroutineContext) + .collectLatest { isConnected -> + if (isConnected) { + // From a phone Activity or Service, get a context for the audio and display glasses. + // Re-initialize on reconnect: Obtain another context instance. + val projectedContext = try { + ProjectedContext.createProjectedDeviceContext(activity) + } catch (e: IllegalStateException) { + Log.e(TAG, "Failed to create projected context", e) + return@collectLatest + } + + // Use the projectedContext to initialize system services (e.g., CameraManager). + Log.i(TAG, "Projected device connected. Initializing hardware...") + } else { + // The projected context is destroyed when the device disconnects. + // Clean up on disconnect: Listen for 'false' and release resources. + Log.i(TAG, "Projected device disconnected. Cleaning up hardware resources...") + } + } + } +} +``` + +
+ +### Check for validity + +Wrap the [`createProjectedDeviceContext`](https://developer.android.com/reference/kotlin/androidx/xr/projected/ProjectedContext#createProjectedDeviceContext(android.content.Context)) call within the +[`ProjectedContext.isProjectedDeviceConnected`](https://developer.android.com/reference/kotlin/androidx/xr/projected/ProjectedContext#isProjectedDeviceConnected(android.content.Context,kotlin.coroutines.CoroutineContext)). While this method returns +`true`, the projected context remains valid to the connected device, and your +phone app activity or service (such as a `CameraManager`) can access the AI +glasses hardware. + +### Clean up on disconnect + +The projected context is tied to the lifecycle of the connected device, so it is +destroyed when the device disconnects. When the device disconnects, +[`ProjectedContext.isProjectedDeviceConnected`](https://developer.android.com/reference/kotlin/androidx/xr/projected/ProjectedContext#isProjectedDeviceConnected(android.content.Context,kotlin.coroutines.CoroutineContext)) returns `false`. Your app +should listen for this change and clean up any system services (such as a +`CameraManager`) or resources that your app created using that projected +context. + +### Re-initialize on reconnect + +When the glasses reconnect, your app can obtain another projected context +instance using [`createProjectedDeviceContext`](https://developer.android.com/reference/kotlin/androidx/xr/projected/ProjectedContext#createProjectedDeviceContext(android.content.Context)), and then re-initialize any +system services or resources using the new projected context. + +## Record audio with the glasses' microphone + +You can record audio from the glasses using Bluetooth Hands-Free Profile (HFP). +In this mode, the glasses connect to the phone using standard Headset and +Advanced Audio Distribution Profile (A2DP) profiles, functioning like a typical +Bluetooth peripheral. If your app is already designed for standard Bluetooth +recording, you can use this method to record audio from the glasses without any +additional development. + +### Record audio using Bluetooth HFP + +To record audio using Bluetooth HFP, first request the required runtime +permissions, and then record the audio using the [`AudioManager`](https://developer.android.com/reference/kotlin/android/media/AudioManager) API, as +described in the following sections. + +#### Request permissions + +As with any standard Bluetooth audio device, the [`RECORD_AUDIO`](https://developer.android.com/reference/kotlin/android/Manifest.permission#record_audio), +[`BLUETOOTH_CONNECT`](https://developer.android.com/reference/kotlin/android/Manifest.permission#bluetooth_connect), and other related permissions are controlled by the +phone and not the connected device (such as audio glasses or display glasses). + +Follow these steps to request the permissions: + +1. [Declare following permissions](https://developer.android.com/develop/xr/jetpack-xr-sdk/request-hardware-permissions#declare-permissions) in your app's manifest file: + + - [`RECORD_AUDIO`](https://developer.android.com/reference/kotlin/android/Manifest.permission#record_audio) + - [`BLUETOOTH_CONNECT`](https://developer.android.com/reference/kotlin/android/Manifest.permission#bluetooth_connect) + - [`MODIFY_AUDIO_SETTINGS`](https://developer.android.com/reference/kotlin/android/Manifest.permission#modify_audio_settings) +2. Request both the `RECORD_AUDIO` and `BLUETOOTH_CONNECT` permissions at + runtime using the [standard Android permission flow](https://developer.android.com/training/permissions/requesting). + +#### Use AudioManager to route audio + +After the user has granted your app the necessary runtime permissions, use the +[`AudioManager`](https://developer.android.com/reference/kotlin/android/media/AudioManager) API to set the communication device to +[`TYPE_BLUETOOTH_SCO`](https://developer.android.com/reference/kotlin/android/media/AudioDeviceInfo#type_bluetooth_sco) to route the audio through Bluetooth HFP. This +directs the system to retrieve audio from the Bluetooth peripheral. + + +```kotlin +val audioManager = context.getSystemService(AudioManager::class.java) ?: return +val devices = audioManager.getDevices(AudioManager.GET_DEVICES_INPUTS) +val hfpDevice = devices.find { it.type == AudioDeviceInfo.TYPE_BLUETOOTH_SCO } + +hfpDevice?.let { device -> + val audioRecord = AudioRecord.Builder() + .setAudioSource(MediaRecorder.AudioSource.VOICE_COMMUNICATION) + .setAudioFormat(audioFormat) + .setBufferSizeInBytes(bufferSize) + .build() + + // Route recording to the Bluetooth device + audioRecord.setPreferredDevice(device) + audioManager.setCommunicationDevice(device) + + audioRecord.startRecording() +``` + +
+ +> [!NOTE] +> **Note:** To learn more about Bluetooth connectivity, refer to our [documentation](https://developer.android.com/develop/connectivity/bluetooth). + +## Capture an image with the glasses' camera + +To capture an image with the glasses' camera, set up and bind the CameraX's +[`ImageCapture`](https://developer.android.com/reference/kotlin/androidx/camera/core/ImageCapture) [use case](https://developer.android.com/media/camera/camerax#ease-of-use) to the glasses' camera using the correct +context for your app: + + +```kotlin +private fun startCameraOnGlasses(activity: ComponentActivity) { + activity.lifecycleScope.launch { + // Before creating a projected context, check to see if the projected device is connected. + ProjectedContext.isProjectedDeviceConnected(activity, coroutineContext) + .collectLatest { isConnected -> + if (isConnected) { + // 1. Get the CameraProvider using the projected context. + // When using the projected context, DEFAULT_BACK_CAMERA maps to the audio and display glasses' camera. + val projectedContext = try { + ProjectedContext.createProjectedDeviceContext(activity) + } catch (e: IllegalStateException) { + Log.e(TAG, "Projected context could not be created", e) + return@collectLatest + } + + val cameraProviderFuture = ProcessCameraProvider.getInstance(projectedContext) + + cameraProviderFuture.addListener({ + val cameraProvider: ProcessCameraProvider = cameraProviderFuture.get() + val cameraSelector = CameraSelector.DEFAULT_BACK_CAMERA + + // 2. Check for the presence of a camera. + if (!cameraProvider.hasCamera(cameraSelector)) { + Log.w(TAG, "The selected camera is not available.") + return@addListener + } + + // 3. Query supported streaming resolutions using Camera2 Interop. + val cameraInfo = cameraProvider.getCameraInfo(cameraSelector) + val camera2CameraInfo = Camera2CameraInfo.from(cameraInfo) + val cameraCharacteristics = camera2CameraInfo.getCameraCharacteristic( + CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP + ) + + // 4. Define the resolution strategy. + val targetResolution = Size(1920, 1080) + val resolutionStrategy = ResolutionStrategy( + targetResolution, + ResolutionStrategy.FALLBACK_RULE_CLOSEST_LOWER + ) + val resolutionSelector = ResolutionSelector.Builder() + .setResolutionStrategy(resolutionStrategy) + .build() + + // 5. If you have other continuous use cases bound, such as Preview or ImageAnalysis, + // you can use Camera2 Interop's CaptureRequestOptions to set the FPS + val fpsRange = Range(30, 60) + val captureRequestOptions = CaptureRequestOptions.Builder() + .setCaptureRequestOption(CaptureRequest.CONTROL_AE_TARGET_FPS_RANGE, fpsRange) + .build() + + // 6. Initialize the ImageCapture use case with options. + val imageCapture = ImageCapture.Builder() + // Optional: Configure resolution, format, etc. + .setResolutionSelector(resolutionSelector) + .build() + + try { + // Unbind use cases before rebinding. + cameraProvider.unbindAll() + + // Bind use cases to camera using the Activity as the LifecycleOwner. + cameraProvider.bindToLifecycle( + activity, + cameraSelector, + imageCapture + ) + } catch (exc: Exception) { + Log.e(TAG, "Use case binding failed", exc) + } + }, ContextCompat.getMainExecutor(activity)) + } + } + } +} +``` + +
+ +### Key points about the code + +- Obtains an instance of the [`ProcessCameraProvider`](https://developer.android.com/reference/kotlin/androidx/camera/lifecycle/ProcessCameraProvider) using the [**projected device context**](https://developer.android.com/develop/xr/jetpack-xr-sdk/access-hardware-projected-context#phone-activity-service). +- Within the projected context's scope, the glasses' primary, outward-pointing camera maps to the `DEFAULT_BACK_CAMERA` when selecting a camera. +- A pre-binding check uses [`cameraProvider.hasCamera(cameraSelector)`](https://developer.android.com/reference/kotlin/androidx/camera/lifecycle/ProcessCameraProvider#hasCamera(androidx.camera.core.CameraSelector)) to verify that the selected camera is available on the device before proceeding. +- Uses **Camera2 Interop** with [`Camera2CameraInfo`](https://developer.android.com/reference/kotlin/androidx/camera/camera2/interop/Camera2CameraInfo) to read the underlying [`CameraCharacteristics#SCALER_STREAM_CONFIGURATION_MAP`](https://developer.android.com/reference/kotlin/android/hardware/camera2/CameraCharacteristics#scaler_stream_configuration_map), which can be useful for advanced checks on supported resolutions. +- A custom [`ResolutionSelector`](https://developer.android.com/reference/kotlin/androidx/camera/core/resolutionselector/ResolutionSelector) is built to precisely control the output image resolution for [`ImageCapture`](https://developer.android.com/reference/kotlin/androidx/camera/core/ImageCapture). +- Creates an `ImageCapture` use case that is configured with a custom `ResolutionSelector`. +- Binds the `ImageCapture` use case to the activity's lifecycle. This automatically manages the opening and closing of the camera based on the activity's state (for example, stopping the camera when the activity is paused). +- **Configuring for single-stream hardware:** The glasses camera pipeline is constrained to a single active stream at a time. When `ImageCapture` is bound alone, CameraX attaches an internal repeating stream (`MeteringRepeating`) for focus and metering by default, which causes the single-stream pipeline to stall. To bind `ImageCapture` by itself without a preview stream, disable the forced repeating stream by setting [`setRepeatingStreamForced(false)`](https://developer.android.com/reference/kotlin/androidx/camera/core/CameraXConfig.Builder#setRepeatingStreamForced(kotlin.Boolean)) on [`CameraXConfig.Builder`](https://developer.android.com/reference/kotlin/androidx/camera/core/CameraXConfig.Builder) (for example, by implementing [`CameraXConfig.Provider`](https://developer.android.com/reference/kotlin/androidx/camera/core/CameraXConfig.Provider) in your `Application` class or configuring `ProcessCameraProvider` before initialization). + +After the glasses' camera is set up, you can capture an image with the CameraX's +`ImageCapture` class. Refer to the CameraX's documentation to learn about using +[`takePicture`](https://developer.android.com/media/camera/camerax/take-photo#take_a_picture) to [capture an image](https://developer.android.com/media/camera/camerax/take-photo#take_a_picture). + +### Capture a video with the glasses' camera + +To capture a video instead of an image with the glasses' camera, replace the +`ImageCapture` components with the corresponding [`VideoCapture`](https://developer.android.com/reference/kotlin/androidx/camera/video/VideoCapture) components +and modify the capture execution logic. + +The main changes involve using a different use case, creating a different output +file, and initiating the capture using the appropriate video recording method. +For more information about the `VideoCapture` API and how to use it, see the +[CameraX's video capture documentation](https://developer.android.com/media/camera/camerax/video-capture#using-videocapture-api). + +> [!IMPORTANT] +> **Important:** Battery power and thermal dissipation on glasses devices is often limited. When developing for glasses, pay close attention to the requested **output format** and **resolution**, as these can severely impact system power and temperature. + +The following table shows the recommended resolution and frame rate depending on +your app's use case: + +| Use case | Resolution | Frame rate | +|---|---|---| +| Video Communication | 1280 x 720 | 15 FPS | +| Computer Vision | 640 x 480 | 10 FPS | +| AI Video Streaming | 640 x 480 | 1 FPS | + +## Access a phone's hardware from a projected activity + +An [projected activity](https://developer.android.com/develop/xr/jetpack-xr-sdk/glasses/support-different-types#activity-lifecycle) can also access the phone's hardware (such as the +camera or microphone) by using [`createHostDeviceContext(context)`](https://developer.android.com/reference/kotlin/androidx/xr/projected/ProjectedContext#createHostDeviceContext(android.content.Context)) to get +the host device's (phone) context: + + +```kotlin +@OptIn(ExperimentalProjectedApi::class) +private fun getPhoneContext(activity: ComponentActivity): Context? { + return try { + // From a projected Activity, get a context for the phone. + ProjectedContext.createHostDeviceContext(activity) + } catch (e: IllegalStateException) { + Log.e(TAG, "Failed to create host device context", e) + null + } +} +``` + +
+ +When accessing hardware or resources that are specific to the host device +(phone) in a hybrid app (an app containing both mobile and glasses experiences), +you must explicitly select the correct context to make sure your app can access +the correct hardware: + +- Use the [`Activity`](https://developer.android.com/reference/kotlin/android/app/Activity) context from the phone `Activity` or the [`ProjectedContext.createHostDeviceContext`](https://developer.android.com/reference/kotlin/androidx/xr/projected/ProjectedContext#createHostDeviceContext(android.content.Context)) to get the phone's context. +- Don't use [`getApplicationContext`](https://developer.android.com/reference/kotlin/android/content/Context#getapplicationcontext) because the application context can incorrectly return the glasses' context if a projected activity was the most-recently-launched component. \ No newline at end of file diff --git a/xr/projected-hardware/references/android/develop/xr/jetpack-xr-sdk/glasses/check-availability.md b/xr/projected-hardware/references/android/develop/xr/jetpack-xr-sdk/glasses/check-availability.md new file mode 100644 index 0000000..2dc4b8f --- /dev/null +++ b/xr/projected-hardware/references/android/develop/xr/jetpack-xr-sdk/glasses/check-availability.md @@ -0,0 +1,76 @@ +
+ + +Applicable XR devices This guidance helps you build experiences for these types of XR devices. [Learn about XR device types →](https://developer.android.com/develop/xr/devices) ![](https://developer.android.com/static/images/develop/xr/ai-glasses-icon.svg) Audio \& +Display Glasses [](https://developer.android.com/develop/xr/devices#audio-display) [Learn about XR device types →](https://developer.android.com/develop/xr/devices) + +
+ +As a user goes through their day, their audio glasses or display glasses might +lose their connection to the host device (such as the user's phone) or their +glasses might be temporarily unavailable if they take their glasses off. To +account for these kinds of changes in device availability, your app can use the +XR Device Availability API, which consolidates device availability signals into +the standard Android [`Lifecycle.State`](https://developer.android.com/reference/kotlin/androidx/lifecycle/Lifecycle.State) values. Use this API to help manage +audio routing, hotword activation, and to know when to expect user input based +on when the glasses are available. + +## Understand lifecycle states + +The following table lists how device availability signals map to the +`Lifecycle.State` values. + +| Lifecycle state | Device status | Description | +|---|---|---| +| `INITIALIZED` | Created | The lifecycle object is created but not yet observed. | +| `CREATED` | Inactive | The service is connected, but the user is not wearing the device. | +| `STARTED` | Active | The user is wearing the device. | +| `DESTROYED` | Disconnected | The device is disconnected or the service connection is lost. | +| `RESUMED` | N/A | The device lifecycle doesn't currently use or emit this state. | + +## Check and monitor device availability + +To check and monitor a device's availability, you'll use a projected context +together with the lifecycle state to determine how your app should handle each +case: + + +```kotlin + // In your phone activity or service, check for projected device connection state before + // attempting to create a projected device context and get the device lifecycle. + ProjectedContext.isProjectedDeviceConnected(context, currentCoroutineContext()) + .flatMapLatest { isConnected -> + if (isConnected) { + try { + // Create the projected device context on connection + val projectedContext = ProjectedContext.createProjectedDeviceContext(context) + val xrDevice = XrDevice.getCurrentDevice(projectedContext) + + // Get the device lifecycle + xrDevice.getLifecycle().currentStateFlow + } catch (e: IllegalStateException) { + flowOf(Lifecycle.State.DESTROYED) + } + } else { + flowOf(Lifecycle.State.DESTROYED) + } + } + .collect { state -> + when (state) { + Lifecycle.State.STARTED -> { /* Device is available (worn) */ } + Lifecycle.State.CREATED -> { /* Device is unavailable (not worn) */ } + Lifecycle.State.DESTROYED -> { /* Device is disconnected from host phone */ } + else -> { /* Handle other states */ } + } + } +} +``` + +
+ +### Key points about the code + +- **Check for a connection** : Before accessing the device lifecycle, call [`ProjectedContext.isProjectedDeviceConnected`](https://developer.android.com/reference/kotlin/androidx/xr/projected/ProjectedContext#isProjectedDeviceConnected(android.content.Context,kotlin.coroutines.CoroutineContext)) to verify that the projected device is connected to the host device. +- **Obtain a `ProjectedContext`** : Only call [`ProjectedContext.createProjectedDeviceContext`](https://developer.android.com/reference/kotlin/androidx/xr/projected/ProjectedContext#createProjectedDeviceContext(android.content.Context)) after verifying the connection, and make sure you pass this context into your [`XrDevice`](https://developer.android.com/reference/kotlin/androidx/xr/runtime/XrDevice) instance. +- **Handle context invalidation** : A new `deviceId` is generated every time a projected device connects. Once the state reaches `DESTROYED`, the current `ProjectedContext` is invalid. Stop using it immediately, and wait for a new connection. +- **Optimize battery and resources** : Gracefully handle app functionality based on the lifecycle state to preserve system resources and reduce battery consumption. For example, you should release glasses-specific resources, such as a camera data stream, when the state transitions from `STARTED` back to `CREATED`. The `CREATED` state indicates the device is no longer being worn, so stopping these processes is essential to prevent unnecessary battery drain and to promote user privacy. \ No newline at end of file diff --git a/xr/projected-hardware/references/android/develop/xr/jetpack-xr-sdk/glasses/support-different-types.md b/xr/projected-hardware/references/android/develop/xr/jetpack-xr-sdk/glasses/support-different-types.md new file mode 100644 index 0000000..b3fe39d --- /dev/null +++ b/xr/projected-hardware/references/android/develop/xr/jetpack-xr-sdk/glasses/support-different-types.md @@ -0,0 +1,116 @@ +
+ + +Applicable XR devices This guidance helps you build experiences for these types of XR devices. [Learn about XR device types →](https://developer.android.com/develop/xr/devices) ![](https://developer.android.com/static/images/develop/xr/ai-glasses-icon.svg) Audio \& +Display Glasses [](https://developer.android.com/develop/xr/devices#audio-display) [Learn about XR device types →](https://developer.android.com/develop/xr/devices) + +
+ +Different types of audio glasses and display glasses have different +capabilities. For example, while all audio glasses and display glasses feature +audio experiences through voice, only display glasses have a display where your +app can show [UIs built with Jetpack Compose Glimmer](https://developer.android.com/develop/xr/jetpack-xr-sdk/jetpack-compose-glimmer). + +To create a seamless user experience across a range of devices, plan your +[`Activity`](https://developer.android.com/reference/kotlin/android/app/Activity) for glasses to check for different device capabilities. This +approach simplifies development by letting you build one activity that adapts +its behavior, rather than multiple activities that target specific devices. + +## Understand the lifecycle of projected activities + +The activity that you build for audio glasses and display glasses doesn't run +directly on the device, but is instead projected to the device from a host +device (such as user's phone). Dedicated activities that you build for this +purpose are *projected activities* . The lifecycle of projected activities is +built off the [standard activity lifecycle](https://developer.android.com/guide/components/activities/activity-lifecycle), but it also includes several key +differences that support the capabilities of different types of audio glasses +and display glasses. +![](https://developer.android.com/static/images/develop/xr/jetpack-xr-sdk/activity-lifecycle.png) **Figure 1**. The key events in the lifecycle of projected activities. + +Here is a breakdown of the key events, with callouts for interactions that are +specific to activities that are projected to audio glasses and display glasses: + +- [`onCreate()`](https://developer.android.com/reference/kotlin/android/app/Activity#onCreate(android.os.Bundle)) + - Called when the projected activity is created. + - Initialize your app's Jetpack Compose Glimmer UI and other components here. +- [`onStart()`](https://developer.android.com/reference/kotlin/android/app/Activity#onStart()) + - Called when the projected activity is starting and the user is [aware](https://developer.android.com/develop/xr/jetpack-xr-sdk/glasses/support-different-types#awareness) of the app. +- [`onResume()`](https://developer.android.com/reference/kotlin/android/app/Activity#onResume()) + - Called when the projected activity regains focus. While the activity is in focus, it is interactable and can consume touchpad or button input. + - Called when the glasses are put back on (donned), after they were previously taken off the head (doffed). +- [`onPause()`](https://developer.android.com/reference/kotlin/android/app/Activity#onPause()) + - Called when the projected activity loses focus, but the user is still [aware](https://developer.android.com/develop/xr/jetpack-xr-sdk/glasses/support-different-types#awareness) of your app. While the activity is out of focus, it isn't interactable stops consuming input. +- [`onStop()`](https://developer.android.com/reference/kotlin/android/app/Activity#onStop()) + - Called when the system believes the user is no longer [aware](https://developer.android.com/develop/xr/jetpack-xr-sdk/glasses/support-different-types#awareness) of your app. + - Called when the glasses are removed from the head (doffed). +- [`onDestroy()`](https://developer.android.com/reference/kotlin/android/app/Activity#onDestroy()) + - Called when the projected activity is about to be destroyed. When this is called, the system releases all resources that are tied to the activity. + +### Understand how display state affects the projected activity lifecycle + +In a standard [`Activity`](https://developer.android.com/reference/kotlin/android/app/Activity), the lifecycle state changes when the device's +screen turns off, typically moving to [`onPause()`](https://developer.android.com/reference/kotlin/android/app/Activity#onPause()) or [`onStop()`](https://developer.android.com/reference/kotlin/android/app/Activity#onStop()). In +contrast, the projected activity lifecycle doesn't change when the glasses' +display turns on or off. This behavior means your projected activity continues +to run in the Started or Resumed state even when the display is off, which lets +your app's audio experiences continue without interruption. + +Other [activity state changes](https://developer.android.com/guide/components/activities/state-changes) triggered by different system and user events +behave the usual way. + +### Understand how user awareness affects your projected activity + +A user can be aware of your projected activity even if it's not visible. +*Awareness* refers to all the ways a user can sense and interact with your app's +experiences, including the following: + +- Listening to audio, audible feedback, or other sound cues. +- App actions that trigger a user-facing LED, such as a privacy indicator if your app accesses the camera or microphone. + +In these situations, the user is aware that the audio glasses and display +glasses are active and responding, even if they aren't looking at a display: + +- The [`onStart()`](https://developer.android.com/reference/kotlin/android/app/Activity#onStart()) state for projected activities means the activity is active. +- The [`onResume()`](https://developer.android.com/reference/kotlin/android/app/Activity#onResume()) state means the activity is interactable and can receive touchpad input, or is receiving primary input dispatch. + +So long as the user is aware of your app, your activity remains active and in +the foreground. If the system doesn't detect any signals of awareness for a +short period of time, the system removes the activity from the foreground and +eventually triggers [`onStop()`](https://developer.android.com/reference/kotlin/android/app/Activity#onStop()). + +## Understand projected activities and projected contexts + +Audio glasses and display glasses are treated as a connected device that extends +the capabilities of a user's phone. A *projected context* is a device-aware +[`Context`](https://developer.android.com/reference/kotlin/android/content/Context) that lets apps interact with the hardware on a connected glasses +device---such as its sensors, camera, or microphone---rather than the phone's +hardware. As you develop experiences for audio glasses and display glasses, your +app must use a projected context to access the glasses' hardware. + +A projected context can be automatically granted to your app depending on the +calling activity's context: + +- **For projected activities**: If your app's code is running from within your + projected activity, its own activity context is already a projected context. + In this scenario, calls made within that activity can already access the + glasses' hardware. + +- **For phone apps or services** : If a part of your app outside of your + projected activity (such as a phone activity or a service) needs to access + the glasses' hardware, it must explicitly obtain a projected context. To do + this, use the [`createProjectedDeviceContext()`](https://developer.android.com/develop/xr/jetpack-xr-sdk/access-hardware-projected-context#phone-activity-service) method. + +For more information, see [Use a projected context to access hardware on audio +glasses and display glasses](https://developer.android.com/develop/xr/jetpack-xr-sdk/access-hardware-projected-context). + +### Understand device-aware APIs + +Some standard Android APIs change which device's hardware they access depending +on the calling activity's `Context`. When these APIs receive a projected +context, they access the glasses' hardware instead of the hardware on the host +phone device: + +- [`CameraManager`](https://developer.android.com/reference/kotlin/android/hardware/camera2/CameraManager): Accesses the camera on the glasses. +- [`SensorManager`](https://developer.android.com/reference/kotlin/android/hardware/SensorManager): Retrieves sensor data (for example, gyroscope or accelerometer data) from the glasses. +- [`AudioManager`](https://developer.android.com/reference/kotlin/android/media/AudioManager): Manages audio streams, volume, and routing on the glasses. +- [`AudioRecord`](https://developer.android.com/reference/kotlin/android/media/AudioRecord): Captures audio using the glasses' microphone. \ No newline at end of file diff --git a/xr/projected-hardware/references/android/develop/xr/jetpack-xr-sdk/request-hardware-permissions.md b/xr/projected-hardware/references/android/develop/xr/jetpack-xr-sdk/request-hardware-permissions.md new file mode 100644 index 0000000..5f25cb4 --- /dev/null +++ b/xr/projected-hardware/references/android/develop/xr/jetpack-xr-sdk/request-hardware-permissions.md @@ -0,0 +1,203 @@ +
+ + +Applicable XR devices This guidance helps you build experiences for these types of XR devices. [Learn about XR device types →](https://developer.android.com/develop/xr/devices) ![](https://developer.android.com/static/images/develop/xr/ai-glasses-icon.svg) Audio \& +Display Glasses [](https://developer.android.com/develop/xr/devices#audio-display) [Learn about XR device types →](https://developer.android.com/develop/xr/devices) + +
+ +Just like on a phone, accessing sensitive hardware like the camera and +microphone on audio glasses and display glasses requires explicit user consent. +These are considered +**glasses-specific permissions**, and your app must request them at runtime, +even if it already has the corresponding permissions on the phone. + +## Declare the permissions in your app's manifest + +Before requesting permissions, you must [declare them in your app's manifest](https://developer.android.com/training/permissions/declaring) +file using the [``](https://developer.android.com/guide/topics/manifest/uses-permission-element) element. This declaration remains the +same whether the permission is for a phone or a glasses-specific feature, but +you must still explicitly request it for glasses-specific hardware or +functionality. + + + + + + ... + + + +## Before requesting glasses permissions + +Review the [permission principles and workflows](https://developer.android.com/training/permissions/requesting#principles) to make sure you provide the +best experience for your users, like [checking whether the user has already +granted the runtime permissions](https://developer.android.com/training/permissions/requesting#already-granted) your app requires and [whether your app +should show a rationale](https://developer.android.com/training/permissions/requesting#explain) to the user before requesting glasses-specific +permissions. + +## Permission scenarios + +There are different scenarios you might encounter when requesting [dangerous +runtime permissions](https://developer.android.com/guide/topics/permissions/requesting#normal-dangerous) on audio and display glasses: + +- Request runtime permissions from a projected activity +- Request runtime permissions from a phone activity + +See the following sections in this guide for details about each scenario. + +### Request runtime permissions from a projected activity + +To request runtime permissions for audio or display glasses from a projected +activity, use the [`ProjectedActivityCompat.requestPermissions`](https://developer.android.com/reference/kotlin/androidx/xr/projected/ProjectedActivityCompat#requestPermissions(android.app.Activity,kotlin.Array,kotlin.Int)) method. + +Calling this method triggers a coordinated user flow across both the glasses and +the host phone. + +1. **On the glasses**: The system opens a projected companion activity on the glasses, instructing the user to check their phone to respond to the permission request. +2. **On the phone**: An Android system permission dialog appears on the phone screen. The dialog explicitly indicates that the user is granting the permission for the glasses rather than the phone itself. The user can then grant or deny the request. + +The following example shows how to request glasses permissions from a projected +activity: + + +```kotlin +@RequiresApi(Build.VERSION_CODES.VANILLA_ICE_CREAM) +private fun requestCameraPermission() { + lifecycleScope.launch { + try { + ProjectedActivityCompat.requestPermissions( + this@PermissionsFromProjectedActivity, + arrayOf(Manifest.permission.CAMERA), + // CAMERA_PERMISSION_REQUEST_CODE is a developer defined constant + CAMERA_PERMISSION_REQUEST_CODE, + ) + } catch (e: IllegalStateException) { + // Thrown when the projected system service can't be bound. + Log.e(TAG, "Camera permission request failed: projected service unavailable.", e) + } + } +} +``` + +
+ +#### Key points about the code + +- Use `ProjectedActivityCompat.requestPermissions` instead of the standard [`ActivityCompat`](https://developer.android.com/reference/kotlin/androidx/core/app/ActivityCompat) method. This method explicitly targets the runtime permissions for audio and display glasses rather than the host phone. + +#### Handle the permission results + +Once the user responds to the permission dialog on the phone, their decision is +delivered to your app by invoking the [`onRequestPermissionsResult`](https://developer.android.com/reference/kotlin/androidx/core/app/ActivityCompat.OnRequestPermissionsResultCallback#onRequestPermissionsResult(int,java.lang.String%5B%5D,int%5B%5D)) +callback. + +The system closes the projected companion activity automatically. On audio +glasses, this occurs after the audio alert finishes playing. On display glasses, +it happens after the user responds to all the permission dialogs. + +To handle the response, override `onRequestPermissionsResult()` within the +`Activity` instance that initiated the permission request. + +The following code snippet shows how to handle the callback and verify whether +the user granted the requested permissions: + + +```kotlin +@RequiresApi(Build.VERSION_CODES.VANILLA_ICE_CREAM) +override fun onRequestPermissionsResult( + requestCode: Int, + permissions: Array, + grantResults: IntArray, + deviceId: Int, +) { + super.onRequestPermissionsResult(requestCode, permissions, grantResults, deviceId) + if (requestCode != CAMERA_PERMISSION_REQUEST_CODE) return + + val isCameraGranted = + grantResults.getOrNull(permissions.indexOf(Manifest.permission.CAMERA)) == + PackageManager.PERMISSION_GRANTED + + if (isCameraGranted) { + initializeCameraFeatures() + } else { + onCameraPermissionDenied() + } +} +``` + +
+ +##### Key points about the code + +- Use the `deviceId` overload with the [`onRequestPermissionsResult`](https://developer.android.com/reference/kotlin/androidx/core/app/ActivityCompat.OnRequestPermissionsResultCallback#onRequestPermissionsResult(int,java.lang.String%5B%5D,int%5B%5D)) callback to ensure the permission status is correctly mapped to the specific context, such as the audio or display glasses rather than the host phone. +- Your app should handle both granted and denied results gracefully. + +### Request runtime permissions in a phone activity + +If your app is running in a phone activity but requires permissions for audio or +display glasses---for example, to let a user switch a video stream to the glasses' +camera for a first-person point of view---request the permissions using the +device-specific [`Activity#requestPermissions(permissions, requestCode, +deviceId)`](https://developer.android.com/reference/kotlin/android/app/Activity#requestpermissions_1) method: + + +```kotlin +// Request the projected permission from phone activity +requestPermissions( + arrayOf(Manifest.permission.CAMERA), + // REQUEST_CODE_GLASSES_CAMERA is a developer-defined constant + REQUEST_CODE_GLASSES_CAMERA, + projectedDeviceId +) +``` + +
+ +#### Key points about the code + +- To target audio and display glasses, obtain the appropriate device ID from a projected context that you've created using [`createProjectedDeviceContext`](https://developer.android.com/reference/kotlin/androidx/xr/projected/ProjectedContext#createProjectedDeviceContext(android.content.Context)). Pass this ID with your call to the `requestPermissions` method. + +#### Handle the permission results + +Once the user responds to the permission dialog on the phone, their decision is +delivered to your app by invoking the [`onRequestPermissionsResult`](https://developer.android.com/reference/kotlin/androidx/core/app/ActivityCompat.OnRequestPermissionsResultCallback#onRequestPermissionsResult(int,java.lang.String%5B%5D,int%5B%5D)) +callback. + +To handle the response, override `onRequestPermissionsResult` within the +`Activity` instance that initiated the permission request. + +The following code snippet shows how to handle the callback and verify whether +the user granted the requested permissions: + + +```kotlin +@RequiresApi(Build.VERSION_CODES.VANILLA_ICE_CREAM) +override fun onRequestPermissionsResult( + requestCode: Int, + permissions: Array, + grantResults: IntArray, + deviceId: Int, +) { + super.onRequestPermissionsResult(requestCode, permissions, grantResults, deviceId) + if (requestCode != CAMERA_PERMISSION_REQUEST_CODE) return + + val isCameraGranted = + grantResults.getOrNull(permissions.indexOf(Manifest.permission.CAMERA)) == + PackageManager.PERMISSION_GRANTED + + if (isCameraGranted) { + initializeCameraFeatures() + } else { + onCameraPermissionDenied() + } +} +``` + +
+ +##### Key points about the code + +- Use the `deviceId` overload with the `onRequestPermissionsResult` callback to ensure the permission status is correctly mapped to the specific context, such as the audio or display glasses rather than the host phone. +- Your app should handle both granted and denied results gracefully. \ No newline at end of file