diff --git a/README.md b/README.md index ce0dc79..38381f3 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,7 @@ The library is available on [Maven Central](https://central.sonatype.com/artifac ## License -OpenMapView is MIT licensed, so you can use it in commercial apps. Its five runtime dependencies (androidx core-ktx, lifecycle-runtime-ktx, Compose ui-graphics, ktor-client-android and DiskLruCache) are all Apache 2.0. See the [Dependency License Report](https://afarber.github.io/OpenMapView/licenses/) for details. +OpenMapView is MIT licensed, so you can use it in commercial apps. Its four runtime dependencies (androidx core-ktx, lifecycle-runtime-ktx, Compose ui-graphics and ktor-client-android) are all Apache 2.0. See the [Dependency License Report](https://afarber.github.io/OpenMapView/licenses/) for details. ## Features diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md index c65cbd9..84a2ad6 100644 --- a/docs/CONTRIBUTING.md +++ b/docs/CONTRIBUTING.md @@ -104,7 +104,6 @@ Before adding new dependencies to OpenMapView, verify the following: - androidx.lifecycle:lifecycle-runtime-ktx - androidx.compose.ui:ui-graphics - io.ktor:ktor-client-android -- com.jakewharton:disklrucache ## Automated Git Hooks (Recommended) diff --git a/docs/PERFORMANCE.md b/docs/PERFORMANCE.md index 644d453..561aea8 100644 --- a/docs/PERFORMANCE.md +++ b/docs/PERFORMANCE.md @@ -69,7 +69,7 @@ OpenMapView implements a two-tier cache hierarchy: │ Level 2: Disk Cache (LRU) │ │ - Persistent across app restarts │ │ - 50MB maximum size │ -│ - PNG compressed tiles │ +│ - Tiles as sent by the server │ └─────────────────┬───────────────────────┘ │ Cache miss v @@ -92,22 +92,25 @@ Implemented using Android's `LruCache`: New tiles are written to both levels as soon as they are downloaded, so they survive the process even if they are never evicted from memory: ```kotlin -fun put(tile: TileCoordinate, bitmap: Bitmap) { +fun put(tile: TileCoordinate, bitmap: Bitmap, bytes: ByteArray) { memoryCache.put(tile, bitmap) - diskCache?.put(tile, bitmap) // called from the IO download coroutine + diskCache?.put(tile, bytes) // the server's bytes, called from the IO download coroutine } ``` ### Disk Cache (Level 2) -Implemented using Jake Wharton's DiskLruCache library: +`DiskTileCache` keeps one file per tile, without any extra library: - **Access time:** Milliseconds (disk I/O) - **Size:** 50MB maximum -- **Format:** PNG compressed images -- **Location:** App cache directory (`context.cacheDir/tiles`) +- **Format:** The PNG or JPEG bytes as sent by the tile server, so tiles are never re-encoded. They are decoded with RGB_565, the same as fresh downloads +- **Location:** App cache directory (`context.cacheDir/tiles2//__.tile`) - **Persistence:** Survives app restarts -- **Eviction:** LRU with size limit +- **Eviction:** When the 50MB are exceeded, the least recently used tiles are deleted until 45MB are left +- **Expiry:** Tiles older than 7 days (the minimum of the OSM tile usage policy) are downloaded again. Reading a tile does not make it younger, the download time is stored in the file +- **Offline fallback:** If downloading an expired tile fails, the old tile is shown and retried with the usual backoff (see [Retry Backoff](#retry-backoff)) +- **Writes:** Each tile is written to a temporary file and renamed, so an interrupted write never leaves a broken tile **Benefits:** - Reduces network requests on app restart @@ -119,7 +122,7 @@ Implemented using Jake Wharton's DiskLruCache library: When a tile is found in disk cache, it's promoted back to memory cache: ```kotlin -val diskBitmap = diskCache?.get(tile) +val diskBitmap = diskCache?.get(tile) // null if missing or expired if (diskBitmap != null) { // Promote to memory for faster subsequent access memoryCache.put(tile, diskBitmap) @@ -274,10 +277,13 @@ Tile downloads use Kotlin coroutines for efficient async I/O: ```kotlin scope.launch(Dispatchers.IO) { - tileDownloadSemaphore.withPermit { - val bitmap = tileDownloader.downloadTile(url) - // ... - } + // Disk hits don't wait for a download permit + val bitmap = cache.get(tile) ?: tileDownloadSemaphore.withPermit { + tileDownloader.downloadTileBytes(url)?.let { bytes -> + decodeTile(bytes)?.also { cache.put(tile, it, bytes) } + } + } ?: cache.getStale(tile) // expired tile as offline fallback + // ... } ``` @@ -347,5 +353,4 @@ For optimal performance: - [Android Bitmap Best Practices](https://developer.android.com/topic/performance/graphics) - [OSM Tile Usage Policy](https://operations.osmfoundation.org/policies/tiles/) -- [DiskLruCache](https://github.com/JakeWharton/DiskLruCache) - [Kotlin Coroutines](https://kotlinlang.org/docs/coroutines-guide.html) diff --git a/docs/TESTING_UNIT.md b/docs/TESTING_UNIT.md index f35c01b..3603b5d 100644 --- a/docs/TESTING_UNIT.md +++ b/docs/TESTING_UNIT.md @@ -52,7 +52,7 @@ The `--continue` flag ensures all tests run even if some fail, useful for gettin ## Test Structure -### Current Test Coverage (461 tests across 35 test classes) +### Current Test Coverage (482 tests across 36 test classes) | Test Class | Tests | Description | |------------|-------|-------------| @@ -61,7 +61,7 @@ The `--continue` flag ensures all tests run even if some fail, useful for gettin | **CameraPositionTest** | 7 | Camera position validation and equality | | **CameraUpdateFactoryTest** | 11 | Camera update creation (zoom, pan, position) | | **CircleTest** | 4 | Circle defaults, radius validation, identity equality | -| **DiskTileCacheTest** | 8 | Persistent disk cache with DiskLruCache, per-source namespaces | +| **DiskTileCacheTest** | 18 | Persistent disk cache: raw tile bytes, expiry, LRU trimming, atomic writes, per-source namespaces | | **GeoJsonParserTest** | 14 | GeoJSON parsing (Point, LineString, Polygon, Multi*, Features) | | **GeodesicUtilsTest** | 17 | Great-circle interpolation for geodesic shapes | | **GroundOverlayTest** | 10 | Ground overlay position/bounds validation and equality | @@ -70,6 +70,7 @@ The `--continue` flag ensures all tests run even if some fail, useful for gettin | **LatLngTest** | 11 | Latitude clamping and longitude wrapping | | **MapControllerOverlaysTest** | 13 | Circles, ground and tile overlays, their touch detection, info windows, drawing all shape types | | **MapControllerTest** | 103 | Zoom, pan, marker/polyline/polygon management, touch detection, padding, camera bounds, edge effects | +| **MapControllerTileLoadingTest** | 7 | Base tile loading order (disk, network, stale fallback) and retry backoff | | **MapFeaturesTest** | 5 | Feature query methods (traffic, indoor, buildings, myLocation) | | **MapProjectionTest** | 8 | Projection screen/LatLng conversion, pan offset, padding, visible region | | **MapTypeTest** | 18 | Map type switching and validation | @@ -81,7 +82,7 @@ The `--continue` flag ensures all tests run even if some fail, useful for gettin | **PredefinedTileProvidersTest** | 2 | URL templates of the predefined overlay tile providers, Stadia key lookup | | **ProjectionTest** | 12 | Web Mercator projection calculations | | **SnapshotTest** | 8 | Map snapshot API and callbacks | -| **TileCacheTest** | 9 | Memory LRU cache behavior, clearing, namespaces, disk writes | +| **TileCacheTest** | 13 | Memory LRU cache behavior, clearing, namespaces, disk writes, stale tiles | | **TileDownloaderTest** | 5 | HTTP tile downloading with mocked Ktor client, HTTP errors, cancellation | | **TileOverlayTest** | 5 | Tile overlay defaults, transparency validation, copy helpers | | **TileRetryTrackerTest** | 4 | Exponential backoff for failed tiles | diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.toml index 39d1dc5..3dc2dde 100644 --- a/gradle/libs.versions.toml +++ b/gradle/libs.versions.toml @@ -25,7 +25,6 @@ compose-material-icons = "1.7.8" ktor = "3.6.0" # Utilities -disklrucache = "2.0.2" # Testing junit = "4.13.2" @@ -56,7 +55,6 @@ ktor-client-android = { group = "io.ktor", name = "ktor-client-android", version ktor-client-mock = { group = "io.ktor", name = "ktor-client-mock", version.ref = "ktor" } # Utilities -disklrucache = { group = "com.jakewharton", name = "disklrucache", version.ref = "disklrucache" } # Testing - Unit junit = { group = "junit", name = "junit", version.ref = "junit" } diff --git a/openmapview/build.gradle.kts b/openmapview/build.gradle.kts index b30d3e1..2369a85 100644 --- a/openmapview/build.gradle.kts +++ b/openmapview/build.gradle.kts @@ -109,7 +109,6 @@ dependencies { implementation(libs.androidx.lifecycle.runtime.ktx) implementation(libs.androidx.compose.ui.graphics) implementation(libs.ktor.client.android) - implementation(libs.disklrucache) // Unit testing testImplementation(libs.junit) diff --git a/openmapview/src/main/kotlin/de/afarber/openmapview/DiskTileCache.kt b/openmapview/src/main/kotlin/de/afarber/openmapview/DiskTileCache.kt index 025c574..993cb71 100644 --- a/openmapview/src/main/kotlin/de/afarber/openmapview/DiskTileCache.kt +++ b/openmapview/src/main/kotlin/de/afarber/openmapview/DiskTileCache.kt @@ -9,159 +9,265 @@ package de.afarber.openmapview import android.content.Context import android.graphics.Bitmap -import android.graphics.BitmapFactory import android.util.Log -import com.jakewharton.disklrucache.DiskLruCache +import java.io.ByteArrayOutputStream import java.io.File import java.io.IOException +import java.nio.ByteBuffer /** - * Persistent disk cache for map tiles using DiskLruCache. + * Persistent disk cache for map tiles. * - * Stores up to 50MB of tile images in PNG format in the app's cache directory. - * Tiles persist across app restarts, reducing network usage on subsequent launches. - * Uses LRU eviction when cache size limit is reached. + * Stores up to 50MB of tiles in the app's cache directory, one file per tile, with the + * bytes exactly as the tile server sent them. Tiles persist across app restarts, reducing + * network usage on subsequent launches. When the size limit is reached, the least recently + * used tiles are deleted. + * + * Tiles older than 7 days count as expired: [get] ignores them, so they are downloaded + * again, but they stay on disk as a fallback for when the download fails (e.g. offline), + * see the `allowStale` parameter of [get]. * * Tiles are stored under a [namespace] (e.g. the tile source), so tiles of different - * sources never replace each other. All instances in a process share one DiskLruCache - * per directory, because DiskLruCache does not support several instances on the same - * directory. + * sources never replace each other. All instances in a process share the size accounting + * per directory. * - * The cache stays usable after [clear], and a closed or failing cache only results - * in cache misses. + * The cache stays usable after [clear], and a failing cache only results in cache misses. * * @param context Context used to locate the app's cache directory * @param namespace Prefix for the cache keys, e.g. the name of the tile source */ -class DiskTileCache( - context: Context, - namespace: String = DEFAULT_NAMESPACE, +class DiskTileCache internal constructor( + directory: File, + namespace: String, + maxSize: Long, + private val maxAgeMs: Long, + private val clock: () -> Long, + legacyDirectory: File? = null, ) { - private val store = storeFor(File(context.cacheDir, "tiles")) - private val keyPrefix = namespace.lowercase().replace(INVALID_KEY_CHARS, "_").take(MAX_NAMESPACE_LENGTH) + constructor( + context: Context, + namespace: String = DEFAULT_NAMESPACE, + ) : this( + directory = File(context.cacheDir, DIRECTORY_NAME), + namespace = namespace, + maxSize = MAX_SIZE, + maxAgeMs = MAX_AGE_MS, + clock = System::currentTimeMillis, + legacyDirectory = File(context.cacheDir, LEGACY_DIRECTORY_NAME), + ) + + private val store = storeFor(directory, maxSize, legacyDirectory) + private val namespaceDirectory = + File(directory, namespace.lowercase().replace(INVALID_NAME_CHARS, "_").take(MAX_NAMESPACE_LENGTH)) @Volatile private var closed = false companion object { private val TAG = DiskTileCache::class.java.simpleName - private const val DISK_CACHE_VERSION = 2 - private const val VALUE_COUNT = 1 - private const val MAX_SIZE = 50L * 1024 * 1024 - private val INVALID_KEY_CHARS = Regex("[^a-z0-9_-]") + private const val DIRECTORY_NAME = "tiles2" - // DiskLruCache keys are limited to 64 characters, leave room for the tile coordinates + // Used by DiskLruCache in versions up to 0.15.0, deleted on first use + private const val LEGACY_DIRECTORY_NAME = "tiles" + private const val MAX_SIZE = 50L * 1024 * 1024 + private const val MAX_AGE_MS = 7L * 24 * 60 * 60 * 1000 + private const val FILE_SUFFIX = ".tile" + private const val TEMP_SUFFIX = ".tmp" + private val INVALID_NAME_CHARS = Regex("[^a-z0-9_-]") private const val MAX_NAMESPACE_LENGTH = 32 + // File header: format version (1 byte) and download time in epoch milliseconds (8 bytes) + private const val FORMAT_VERSION: Byte = 1 + private const val HEADER_SIZE = 9 + /** Namespace used when none is given. */ const val DEFAULT_NAMESPACE = "default" private val stores = HashMap() - private fun storeFor(directory: File): Store = + private fun storeFor( + directory: File, + maxSize: Long, + legacyDirectory: File?, + ): Store = synchronized(stores) { - stores.getOrPut(directory.absolutePath) { Store(directory) } + stores.getOrPut(directory.absolutePath) { Store(directory, maxSize, legacyDirectory) } } } /** - * One DiskLruCache per directory, shared by all DiskTileCache instances. - * Callers synchronize on the Store before using [cache]. + * Size accounting of one cache directory, shared by all DiskTileCache instances. + * Callers synchronize on the Store before reading or writing files. */ private class Store( - private val directory: File, + val directory: File, + val maxSize: Long, + private var legacyDirectory: File?, ) { - private var diskCache: DiskLruCache? = null - - // Opens the cache on first use, and again after it was deleted or - // the directory was removed (e.g. when the user clears the app cache) - val cache: DiskLruCache? - get() { - val current = diskCache - if (current != null && !current.isClosed && directory.exists()) { - return current - } - diskCache = - try { - DiskLruCache.open(directory, DISK_CACHE_VERSION, VALUE_COUNT, MAX_SIZE) - } catch (e: IOException) { - Log.e(TAG, "Failed to initialize disk cache at $directory", e) - null + // Total size of all tile files, -1 until the directory has been scanned + private var totalSize = -1L + + /** Scans the directory on first use, and again after it was removed (e.g. app cache cleared). */ + fun size(): Long { + legacyDirectory?.let { + it.deleteRecursively() + legacyDirectory = null + } + if (totalSize < 0 || !directory.exists()) { + totalSize = 0 + directory.walkTopDown().filter { it.isFile }.forEach { file -> + if (file.name.endsWith(FILE_SUFFIX)) { + totalSize += file.length() + } else { + // Left over from a write that was interrupted + file.delete() } - return diskCache + } + } + return totalSize + } + + fun add(delta: Long) { + totalSize = size() + delta + } + + fun reset() { + totalSize = 0 + } + + /** Deletes the least recently used tiles until the cache uses at most 90% of [maxSize]. */ + fun trim() { + if (size() <= maxSize) return + val target = maxSize / 10 * 9 + val files = + directory + .walkTopDown() + .filter { it.isFile && it.name.endsWith(FILE_SUFFIX) } + .map { it to it.lastModified() } + .sortedBy { it.second } + for ((file, _) in files) { + if (totalSize <= target) break + val length = file.length() + if (file.delete()) totalSize -= length } + } + } + + /** + * Returns the cached tile, decoded with RGB_565, or null if it is missing. + * + * @param allowStale Also return a tile older than the maximum age. Use it as a fallback + * when downloading a fresh tile failed. + */ + fun get( + tile: TileCoordinate, + allowStale: Boolean = false, + ): Bitmap? { + val bytes = getBytes(tile, allowStale) ?: return null + return decodeTile(bytes, HEADER_SIZE) } - fun get(tile: TileCoordinate): Bitmap? { + /** Returns the stored file contents (header and tile bytes), or null if missing or expired. */ + internal fun getBytes( + tile: TileCoordinate, + allowStale: Boolean = false, + ): ByteArray? { if (closed) return null - val key = getTileKey(tile) + val file = fileFor(tile) return synchronized(store) { try { - store.cache?.get(key)?.use { snapshot -> - val inputStream = snapshot.getInputStream(0) - BitmapFactory.decodeStream(inputStream) + if (!file.exists()) return null + val bytes = file.readBytes() + if (bytes.size <= HEADER_SIZE || bytes[0] != FORMAT_VERSION) { + store.add(-file.length()) + file.delete() + return null } + val downloadedAt = ByteBuffer.wrap(bytes, 1, 8).long + if (!allowStale && clock() - downloadedAt > maxAgeMs) return null + // Remember the access for LRU trimming, the download time stays in the header + file.setLastModified(clock()) + bytes } catch (e: IOException) { - Log.w(TAG, "Failed to read tile $key from disk cache", e) - null - } catch (e: IllegalStateException) { - Log.w(TAG, "Failed to read tile $key from disk cache", e) + Log.w(TAG, "Failed to read tile $tile from disk cache", e) null } } } + /** + * Stores the tile bytes as sent by the tile server. + */ fun put( tile: TileCoordinate, - bitmap: Bitmap, + bytes: ByteArray, ) { if (closed) return - val key = getTileKey(tile) + val file = fileFor(tile) synchronized(store) { + val temp = File(file.parentFile, file.name + TEMP_SUFFIX) try { - store.cache?.edit(key)?.let { editor -> - try { - editor.newOutputStream(0).use { outputStream -> - bitmap.compress(Bitmap.CompressFormat.PNG, 100, outputStream) - } - editor.commit() - } catch (e: IOException) { - Log.w(TAG, "Failed to write tile $key to disk cache", e) - editor.abort() - } + val header = + ByteBuffer + .allocate(HEADER_SIZE) + .put(FORMAT_VERSION) + .putLong(clock()) + .array() + // Scan before writing: the scan deletes temp files and must not count the new file + store.size() + file.parentFile?.mkdirs() + temp.outputStream().use { output -> + output.write(header) + output.write(bytes) } + val oldLength = if (file.exists()) file.length() else 0L + if (!temp.renameTo(file)) throw IOException("Failed to rename $temp") + file.setLastModified(clock()) + store.add(file.length() - oldLength) + store.trim() } catch (e: IOException) { - Log.w(TAG, "Failed to edit tile $key in disk cache", e) - } catch (e: IllegalStateException) { - Log.w(TAG, "Failed to edit tile $key in disk cache", e) + Log.w(TAG, "Failed to write tile $tile to disk cache", e) + temp.delete() } } } /** - * Removes all cached tiles of all namespaces. DiskLruCache.delete() also closes - * the cache, so it is reopened on the next access. + * Stores a decoded tile as PNG. Prefer the [put] overload with the original bytes, + * which avoids re-encoding. + */ + fun put( + tile: TileCoordinate, + bitmap: Bitmap, + ) { + if (closed) return + val output = ByteArrayOutputStream() + if (bitmap.compress(Bitmap.CompressFormat.PNG, 100, output)) { + put(tile, output.toByteArray()) + } + } + + /** + * Removes all cached tiles of all namespaces. */ fun clear() { if (closed) return synchronized(store) { - try { - store.cache?.delete() - } catch (e: IOException) { - Log.w(TAG, "Failed to clear disk cache", e) + if (!store.directory.deleteRecursively()) { + Log.w(TAG, "Failed to clear disk cache") } + store.reset() } } /** * Stops using the cache. Later calls to [get] and [put] are treated as cache misses. - * The shared DiskLruCache stays open for other instances; it flushes its journal - * after every write, so nothing is lost by not closing it. + * Other instances on the same directory keep working. */ fun close() { closed = true } - // Unique key per namespace and tile coordinate (e.g. "standard_10_512_384" for zoom 10, x=512, y=384) - private fun getTileKey(tile: TileCoordinate): String = "${keyPrefix}_${tile.zoom}_${tile.x}_${tile.y}" + // One file per namespace and tile coordinate, e.g. "standard/10_512_384.tile" + private fun fileFor(tile: TileCoordinate): File = File(namespaceDirectory, "${tile.zoom}_${tile.x}_${tile.y}$FILE_SUFFIX") } diff --git a/openmapview/src/main/kotlin/de/afarber/openmapview/MapController.kt b/openmapview/src/main/kotlin/de/afarber/openmapview/MapController.kt index 9dfda21..b4642d1 100644 --- a/openmapview/src/main/kotlin/de/afarber/openmapview/MapController.kt +++ b/openmapview/src/main/kotlin/de/afarber/openmapview/MapController.kt @@ -1265,11 +1265,11 @@ class MapController( } else { canvas.drawRect(screenX, screenY, screenX + TILE_SIZE, screenY + TILE_SIZE, tilePlaceholderPaint) canvas.drawRect(screenX, screenY, screenX + TILE_SIZE, screenY + TILE_SIZE, tileBorderPaint) + } - if (!downloadingTiles.contains(tile) && tileRetries.canRequest(tile, SystemClock.elapsedRealtime())) { - downloadingTiles.add(tile) - downloadTile(tile) - } + if (needsDownload(tile, SystemClock.elapsedRealtime())) { + downloadingTiles.add(tile) + downloadTile(tile) } } @@ -1346,17 +1346,25 @@ class MapController( // Download tiles that are adjacent but not yet visible val now = SystemClock.elapsedRealtime() for (tile in prefetchTiles) { - if (!visibleTiles.contains(tile) && - !downloadingTiles.contains(tile) && - tileCache.getMemory(tile) == null && - tileRetries.canRequest(tile, now) - ) { + if (!visibleTiles.contains(tile) && needsDownload(tile, now)) { downloadingTiles.add(tile) downloadTile(tile, lowPriority = true) } } } + /** + * Returns true if a base tile should be downloaded: it is missing from memory, or it is + * an expired tile shown as offline fallback, and its retry delay has passed. + */ + internal fun needsDownload( + tile: TileCoordinate, + now: Long, + ): Boolean = + (tileCache.getMemory(tile) == null || tileCache.isStale(tile)) && + !downloadingTiles.contains(tile) && + tileRetries.canRequest(tile, now) + /** * Converts a distance in meters to screen pixels at a given latitude and zoom level. * Uses the Mercator projection formula. @@ -1842,27 +1850,48 @@ class MapController( scope.launch(Dispatchers.IO) { var bitmap: Bitmap? = null var skipped = false + var stale = false try { - bitmap = - tileDownloadSemaphore.withPermit { - if (cache !== tileCache) { - skipped = true - return@withPermit null + // The source changed while this request was queued + if (cache !== tileCache) { + skipped = true + return@launch + } + // Disk hits don't need a download permit, a stale tile in memory is downloaded again + if (!cache.isStale(tile)) { + bitmap = cache.get(tile) + } + if (bitmap == null) { + bitmap = + tileDownloadSemaphore.withPermit { + if (cache !== tileCache) { + skipped = true + return@withPermit null + } + tileDownloader.downloadTileBytes(source.getTileUrl(tile))?.let { bytes -> + decodeTile(bytes)?.also { cache.put(tile, it, bytes) } + } } - // A disk hit is promoted to memory by get(), only new downloads need put() - cache.get(tile) - ?: tileDownloader.downloadTile(source.getTileUrl(tile))?.also { cache.put(tile, it) } - } + } + // Offline or server error: show an expired tile from disk, if there is one + if (bitmap == null && !skipped) { + bitmap = cache.getStale(tile) + stale = bitmap != null && cache.isStale(tile) + } } finally { launch(Dispatchers.Main) { downloading.remove(tile) if (skipped || cache !== tileCache) return@launch - if (bitmap != null) { + if (bitmap != null && !stale) { retries.onSuccess(tile) if (!lowPriority) onTileLoadedCallback?.invoke() } else { + // A stale fallback counts as a failure, so the tile is retried with back-off val retryDelay = retries.onFailure(tile, SystemClock.elapsedRealtime()) - if (!lowPriority) scheduleRedraw(retryDelay) + if (!lowPriority) { + if (stale) onTileLoadedCallback?.invoke() + scheduleRedraw(retryDelay) + } } } } diff --git a/openmapview/src/main/kotlin/de/afarber/openmapview/TileCache.kt b/openmapview/src/main/kotlin/de/afarber/openmapview/TileCache.kt index 08cbecc..0ce30f1 100644 --- a/openmapview/src/main/kotlin/de/afarber/openmapview/TileCache.kt +++ b/openmapview/src/main/kotlin/de/afarber/openmapview/TileCache.kt @@ -11,6 +11,8 @@ import android.content.Context import android.graphics.Bitmap import android.util.Log import android.util.LruCache +import java.util.Collections +import java.util.concurrent.ConcurrentHashMap /** * Two-level cache for map tiles: in-memory LRU cache backed by persistent disk cache. @@ -19,24 +21,30 @@ import android.util.LruCache * [put] also writes the tile to the disk cache, so call it from a background thread. * Disk cache stores up to 50MB of tiles to reduce network requests across app restarts. * + * A tile older than the disk cache's maximum age is only returned by [getStale], as a + * fallback when a fresh download failed. It is then marked as stale (see [isStale]) until + * a fresh tile is put. + * * @param context Context for the disk cache, or null for a memory-only cache * @param namespace Disk cache namespace, e.g. the tile source, so tiles of different * sources are stored separately */ -class TileCache( - context: Context? = null, - namespace: String = DiskTileCache.DEFAULT_NAMESPACE, +class TileCache internal constructor( + private var diskCache: DiskTileCache?, + // Use 1/8 of available heap memory for tile cache + cacheSize: Int = (Runtime.getRuntime().maxMemory() / 1024 / 8).toInt(), ) { + constructor( + context: Context? = null, + namespace: String = DiskTileCache.DEFAULT_NAMESPACE, + ) : this(context?.let { DiskTileCache(it, namespace) }) + companion object { private val TAG = TileCache::class.java.simpleName } - // Use 1/8 of available heap memory for tile cache - private val maxMemory = (Runtime.getRuntime().maxMemory() / 1024).toInt() - private val cacheSize = maxMemory / 8 - - private var diskCache: DiskTileCache? = - context?.let { DiskTileCache(it, namespace) } + // Tiles in the memory cache that came from an expired disk entry + private val staleTiles: MutableSet = Collections.newSetFromMap(ConcurrentHashMap()) private val memoryCache = object : LruCache(cacheSize) { @@ -44,8 +52,21 @@ class TileCache( key: TileCoordinate, bitmap: Bitmap, ): Int = bitmap.byteCount / 1024 + + // A tile that leaves the memory cache is no longer shown, so forget its stale mark + override fun entryRemoved( + evicted: Boolean, + key: TileCoordinate, + oldValue: Bitmap, + newValue: Bitmap?, + ) { + staleTiles.remove(key) + } } + /** + * Returns the tile from memory (which may be stale, see [isStale]), or a fresh tile from disk. + */ fun get(tile: TileCoordinate): Bitmap? { // Check memory cache first for fastest access val memoryBitmap = getMemory(tile) @@ -66,15 +87,45 @@ class TileCache( fun getMemory(tile: TileCoordinate): Bitmap? = memoryCache.get(tile) + /** + * Returns true if the tile in the memory cache is an expired one from [getStale], + * which should be replaced by a fresh download. + */ + fun isStale(tile: TileCoordinate): Boolean = tile in staleTiles + + /** + * Returns the tile even if its disk entry has expired, for use when downloading a fresh + * tile failed (e.g. offline). An expired tile is promoted to memory and marked as stale. + */ + fun getStale(tile: TileCoordinate): Bitmap? { + get(tile)?.let { return it } + val diskBitmap = diskCache?.get(tile, allowStale = true) ?: return null + memoryCache.put(tile, diskBitmap) + staleTiles.add(tile) + return diskBitmap + } + + /** + * Puts the tile into the memory cache and the disk cache. + * + * @param bytes The tile as sent by the server, stored on disk as is. If null, the + * bitmap is encoded as PNG for the disk cache. + */ fun put( tile: TileCoordinate, bitmap: Bitmap, + bytes: ByteArray? = null, ) { memoryCache.put(tile, bitmap) + staleTiles.remove(tile) // Write to disk right away, so the tile survives the process even if it is never // evicted from memory try { - diskCache?.put(tile, bitmap) + if (bytes != null) { + diskCache?.put(tile, bytes) + } else { + diskCache?.put(tile, bitmap) + } } catch (e: Exception) { Log.w(TAG, "Failed to write tile to disk cache", e) } @@ -82,6 +133,7 @@ class TileCache( fun clear() { memoryCache.evictAll() + staleTiles.clear() try { diskCache?.clear() } catch (e: Exception) { diff --git a/openmapview/src/main/kotlin/de/afarber/openmapview/TileDownloader.kt b/openmapview/src/main/kotlin/de/afarber/openmapview/TileDownloader.kt index 25b7cd2..d776e26 100644 --- a/openmapview/src/main/kotlin/de/afarber/openmapview/TileDownloader.kt +++ b/openmapview/src/main/kotlin/de/afarber/openmapview/TileDownloader.kt @@ -31,7 +31,13 @@ class TileDownloader { } } - suspend fun downloadTile(url: String): Bitmap? = + suspend fun downloadTile(url: String): Bitmap? = downloadTileBytes(url)?.let { decodeTile(it) } + + /** + * Downloads a tile and returns its bytes as sent by the server, or null on failure. + * Used to store tiles on disk without decoding and re-encoding them. + */ + suspend fun downloadTileBytes(url: String): ByteArray? = try { val response = client.get(url) { @@ -39,16 +45,10 @@ class TileDownloader { } if (!response.status.isSuccess()) { Log.w(TAG, "Failed to download tile from $url: HTTP ${response.status.value}") - return null + null + } else { + response.readRawBytes() } - val bytes = response.readRawBytes() - // Decode with RGB_565 to reduce memory usage (2 bytes per pixel vs 4 bytes for ARGB_8888) - val options = - BitmapFactory.Options().apply { - inPreferredConfig = Bitmap.Config.RGB_565 - inScaled = false - } - BitmapFactory.decodeByteArray(bytes, 0, bytes.size, options) } catch (e: CancellationException) { throw e } catch (e: Exception) { @@ -60,3 +60,20 @@ class TileDownloader { client.close() } } + +/** + * Decodes tile image bytes with RGB_565 to reduce memory usage + * (2 bytes per pixel vs 4 bytes for ARGB_8888). + */ +internal fun decodeTile( + bytes: ByteArray, + offset: Int = 0, + length: Int = bytes.size - offset, +): Bitmap? { + val options = + BitmapFactory.Options().apply { + inPreferredConfig = Bitmap.Config.RGB_565 + inScaled = false + } + return BitmapFactory.decodeByteArray(bytes, offset, length, options) +} diff --git a/openmapview/src/test/kotlin/de/afarber/openmapview/DiskTileCacheTest.kt b/openmapview/src/test/kotlin/de/afarber/openmapview/DiskTileCacheTest.kt index 52fffd6..d0b2161 100644 --- a/openmapview/src/test/kotlin/de/afarber/openmapview/DiskTileCacheTest.kt +++ b/openmapview/src/test/kotlin/de/afarber/openmapview/DiskTileCacheTest.kt @@ -9,19 +9,32 @@ package de.afarber.openmapview import android.graphics.Bitmap import org.junit.After +import org.junit.Assert.assertArrayEquals import org.junit.Assert.assertEquals +import org.junit.Assert.assertFalse import org.junit.Assert.assertNotNull import org.junit.Assert.assertNull +import org.junit.Assert.assertTrue import org.junit.Before +import org.junit.Rule import org.junit.Test +import org.junit.rules.TemporaryFolder import org.junit.runner.RunWith import org.robolectric.RobolectricTestRunner import org.robolectric.RuntimeEnvironment +import java.io.ByteArrayOutputStream +import java.io.File @RunWith(RobolectricTestRunner::class) class DiskTileCacheTest { + @get:Rule + val folder = TemporaryFolder() + private lateinit var diskCache: DiskTileCache + // Controlled clock for expiry and LRU tests, in steps far above file time granularity + private var now = 1_000_000_000_000L + @Before fun setUp() { diskCache = DiskTileCache(RuntimeEnvironment.getApplication()) @@ -140,6 +153,177 @@ class DiskTileCacheTest { assertNotNull(diskCache.get(tile)) } + @Test + fun testStoresServerBytesUnchanged() { + val cache = testCache() + val tile = TileCoordinate(1, 2, 3) + val bytes = createPngBytes() + + cache.put(tile, bytes) + val stored = cache.getBytes(tile)!! + + assertArrayEquals(bytes, stored.copyOfRange(stored.size - bytes.size, stored.size)) + } + + @Test + fun testDiskHitIsDecodedAsRgb565() { + val cache = testCache() + val tile = TileCoordinate(1, 2, 3) + + cache.put(tile, createPngBytes()) + + assertEquals(Bitmap.Config.RGB_565, cache.get(tile)?.config) + } + + @Test + fun testExpiredTileIsOnlyReturnedAsStale() { + val cache = testCache(maxAgeMs = 60_000) + val tile = TileCoordinate(1, 2, 3) + cache.put(tile, createPngBytes()) + + now += 60_000 + assertNotNull("Not expired yet", cache.get(tile)) + + now += 1 + assertNull(cache.get(tile)) + assertNotNull(cache.get(tile, allowStale = true)) + } + + @Test + fun testReadingDoesNotResetTheAge() { + val cache = testCache(maxAgeMs = 60_000) + val tile = TileCoordinate(1, 2, 3) + cache.put(tile, createPngBytes()) + + now += 50_000 + assertNotNull(cache.get(tile)) + now += 50_000 + + assertNull(cache.get(tile)) + } + + @Test + fun testTrimDeletesLeastRecentlyUsedTiles() { + val bytes = ByteArray(1_000) { it.toByte() } + // Room for three tiles, a fourth one triggers trimming to 90% + val cache = testCache(maxSize = 3_500) + val (a, b, c, d) = (1..4).map { TileCoordinate(it, 0, 5) } + + for (tile in listOf(a, b, c)) { + now += 10_000 + cache.put(tile, bytes) + } + now += 10_000 + assertNotNull("Reading a makes b the least recently used", cache.getBytes(a)) + now += 10_000 + cache.put(d, bytes) + + assertNull(cache.getBytes(b)) + assertNotNull(cache.getBytes(a)) + assertNotNull(cache.getBytes(c)) + assertNotNull(cache.getBytes(d)) + } + + @Test + fun testSizeIsCountedFromExistingFiles() { + val bytes = ByteArray(1_000) + val directory = folder.newFolder("shared") + // Fill the directory with tiles written by an earlier process + val first = testCache(directory = directory, maxSize = 3_500) + for (x in 1..3) { + now += 10_000 + first.put(TileCoordinate(x, 0, 5), bytes) + } + forgetStore(directory) + + now += 10_000 + val second = testCache(directory = directory, maxSize = 3_500) + second.put(TileCoordinate(4, 0, 5), bytes) + + assertNull("The oldest tile was trimmed", second.getBytes(TileCoordinate(1, 0, 5))) + assertNotNull(second.getBytes(TileCoordinate(4, 0, 5))) + } + + @Test + fun testNoTempFilesAreLeft() { + val directory = folder.newFolder("tiles") + val leftover = File(directory, "default/1_2_3.tile.tmp") + leftover.parentFile!!.mkdirs() + leftover.writeBytes(ByteArray(10)) + val cache = testCache(directory = directory) + + cache.put(TileCoordinate(1, 2, 3), createPngBytes()) + cache.put(TileCoordinate(4, 5, 6), createPngBytes()) + + val files = + directory + .walkTopDown() + .filter { it.isFile } + .map { it.name } + .toList() + assertEquals(listOf("3_1_2.tile", "6_4_5.tile").sorted(), files.sorted()) + } + + @Test + fun testCorruptFileIsAMiss() { + val directory = folder.newFolder("tiles") + val cache = testCache(directory = directory) + val tile = TileCoordinate(1, 2, 3) + cache.put(tile, createPngBytes()) + val file = File(directory, "default/3_1_2.tile") + file.writeBytes(byteArrayOf(42)) + + assertNull(cache.get(tile)) + assertFalse("Corrupt file is deleted", file.exists()) + } + + @Test + fun testLegacyDirectoryIsDeleted() { + val legacy = folder.newFolder("legacy") + File(legacy, "journal").writeText("libcore.io.DiskLruCache") + val cache = testCache(legacyDirectory = legacy) + + cache.put(TileCoordinate(1, 2, 3), createPngBytes()) + + assertFalse(legacy.exists()) + } + + @Test + fun testPutAfterDirectoryWasRemoved() { + val directory = folder.newFolder("tiles") + val cache = testCache(directory = directory) + val tile = TileCoordinate(1, 2, 3) + cache.put(tile, createPngBytes()) + + // E.g. the user cleared the app cache in the system settings + assertTrue(directory.deleteRecursively()) + cache.put(tile, createPngBytes()) + + assertNotNull(cache.get(tile)) + } + + private fun testCache( + directory: File = folder.newFolder(), + maxSize: Long = 1024 * 1024, + maxAgeMs: Long = 24 * 60 * 60 * 1000, + legacyDirectory: File? = null, + ) = DiskTileCache(directory, DiskTileCache.DEFAULT_NAMESPACE, maxSize, maxAgeMs, { now }, legacyDirectory) + + // Simulates a new process, which has to scan the directory again + @Suppress("UNCHECKED_CAST") + private fun forgetStore(directory: File) { + val field = DiskTileCache::class.java.getDeclaredField("stores") + field.isAccessible = true + val stores = field.get(null) as HashMap + synchronized(stores) { stores.remove(directory.absolutePath) } + } + + private fun createPngBytes(): ByteArray { + val output = ByteArrayOutputStream() + createMockBitmap(256, 256).compress(Bitmap.CompressFormat.PNG, 100, output) + return output.toByteArray() + } + private fun createMockBitmap( width: Int, height: Int, diff --git a/openmapview/src/test/kotlin/de/afarber/openmapview/MapControllerTileLoadingTest.kt b/openmapview/src/test/kotlin/de/afarber/openmapview/MapControllerTileLoadingTest.kt new file mode 100644 index 0000000..f1e65be --- /dev/null +++ b/openmapview/src/test/kotlin/de/afarber/openmapview/MapControllerTileLoadingTest.kt @@ -0,0 +1,215 @@ +/* + * Copyright (c) 2025 Alexander Farber + * SPDX-License-Identifier: MIT + * + * This file is part of the OpenMapView project (https://github.com/afarber/OpenMapView) + */ + +package de.afarber.openmapview + +import android.graphics.Bitmap +import android.os.Looper +import android.os.SystemClock +import io.ktor.client.HttpClient +import io.ktor.client.engine.mock.MockEngine +import io.ktor.client.engine.mock.respond +import io.ktor.http.HttpStatusCode +import io.ktor.utils.io.ByteReadChannel +import org.junit.After +import org.junit.Assert.assertEquals +import org.junit.Assert.assertFalse +import org.junit.Assert.assertNotNull +import org.junit.Assert.assertNull +import org.junit.Assert.assertTrue +import org.junit.Before +import org.junit.Rule +import org.junit.Test +import org.junit.rules.TemporaryFolder +import org.junit.runner.RunWith +import org.robolectric.RobolectricTestRunner +import org.robolectric.RuntimeEnvironment +import org.robolectric.Shadows.shadowOf +import java.io.ByteArrayOutputStream +import java.time.Duration + +/** + * Drives MapController's base tile download through a mocked HTTP client and a disk + * cache with a controlled clock, to check the order disk, network, stale fallback and + * the retry back-off. + */ +@RunWith(RobolectricTestRunner::class) +class MapControllerTileLoadingTest { + @get:Rule + val folder = TemporaryFolder() + + private lateinit var controller: MapController + private lateinit var disk: DiskTileCache + private lateinit var cache: TileCache + private val tile = TileCoordinate(1, 2, 3) + private var now = 1_000_000_000_000L + private var serverUp = true + private var requests = 0 + + @Before + fun setUp() { + controller = MapController(RuntimeEnvironment.getApplication()) + disk = DiskTileCache(folder.newFolder(), DiskTileCache.DEFAULT_NAMESPACE, 1024 * 1024, MAX_AGE_MS, { now }) + cache = TileCache(disk) + setField(controller, "tileCache", cache) + + val engine = + MockEngine { + requests++ + if (serverUp) { + respond(ByteReadChannel(createPngBytes()), HttpStatusCode.OK) + } else { + respond(ByteReadChannel(ByteArray(0)), HttpStatusCode.ServiceUnavailable) + } + } + setField(getField(controller, "tileDownloader"), "client", HttpClient(engine)) + } + + @After + fun tearDown() { + controller.onDestroy() + } + + @Test + fun testFreshDiskTileIsNotDownloaded() { + disk.put(tile, createPngBytes()) + + download() + + assertEquals(0, requests) + assertNotNull(cache.getMemory(tile)) + assertTrue(retries().canRequest(tile, SystemClock.elapsedRealtime())) + } + + @Test + fun testMissingTileIsDownloadedAndStored() { + download() + + assertEquals(1, requests) + assertNotNull(cache.getMemory(tile)) + assertNotNull("Stored on disk", TileCache(disk).get(tile)) + } + + @Test + fun testFailedDownloadAfterDiskMissBacksOff() { + serverUp = false + + download() + + assertNull(cache.getMemory(tile)) + assertFalse("Retried only after a delay", retries().canRequest(tile, SystemClock.elapsedRealtime())) + } + + @Test + fun testExpiredTileIsDownloadedAgain() { + disk.put(tile, createPngBytes()) + now += MAX_AGE_MS + 1 + + download() + + assertEquals(1, requests) + assertFalse(cache.isStale(tile)) + assertNotNull("Fresh again", TileCache(disk).get(tile)) + } + + @Test + fun testStaleTileIsShownWhenDownloadFailsAndRetriedWithBackOff() { + disk.put(tile, createPngBytes()) + now += MAX_AGE_MS + 1 + serverUp = false + + download() + + assertNotNull("Stale tile shown", cache.getMemory(tile)) + assertTrue(cache.isStale(tile)) + assertFalse("Retried only after a delay", retries().canRequest(tile, SystemClock.elapsedRealtime())) + + // Once the network is back, the next attempt replaces the stale tile + serverUp = true + download() + + assertEquals(2, requests) + assertFalse(cache.isStale(tile)) + assertTrue(retries().canRequest(tile, SystemClock.elapsedRealtime())) + } + + @Test + fun testStaleTileNeedsDownloadAfterBackOff() { + disk.put(tile, createPngBytes()) + now += MAX_AGE_MS + 1 + serverUp = false + download() + + assertFalse("Still waiting for the retry delay", controller.needsDownload(tile, SystemClock.elapsedRealtime())) + + // Prefetch and visible tiles share needsDownload(), so both retry the stale tile + shadowOf(Looper.getMainLooper()).idleFor(Duration.ofMillis(TileRetryTracker.INITIAL_DELAY_MS)) + assertTrue(controller.needsDownload(tile, SystemClock.elapsedRealtime())) + } + + @Test + fun testFreshTileInMemoryNeedsNoDownload() { + assertTrue(controller.needsDownload(tile, SystemClock.elapsedRealtime())) + + download() + + assertFalse(controller.needsDownload(tile, SystemClock.elapsedRealtime())) + } + + // Calls the private downloadTile() and waits until its result reached the main thread + @Suppress("UNCHECKED_CAST") + private fun download() { + val downloading = getField(controller, "downloadingTiles") as MutableSet + downloading.add(tile) + val method = + MapController::class.java.getDeclaredMethod( + "downloadTile", + TileCoordinate::class.java, + Boolean::class.javaPrimitiveType, + ) + method.isAccessible = true + method.invoke(controller, tile, false) + + val deadline = System.currentTimeMillis() + 5_000 + while (downloading.contains(tile)) { + check(System.currentTimeMillis() < deadline) { "Download did not finish" } + shadowOf(Looper.getMainLooper()).idle() + Thread.sleep(10) + } + } + + private fun retries(): TileRetryTracker = getField(controller, "tileRetries") as TileRetryTracker + + private fun getField( + target: Any, + name: String, + ): Any { + val field = target.javaClass.getDeclaredField(name) + field.isAccessible = true + return field.get(target)!! + } + + private fun setField( + target: Any, + name: String, + value: Any, + ) { + val field = target.javaClass.getDeclaredField(name) + field.isAccessible = true + field.set(target, value) + } + + private fun createPngBytes(): ByteArray { + val output = ByteArrayOutputStream() + Bitmap.createBitmap(256, 256, Bitmap.Config.ARGB_8888).compress(Bitmap.CompressFormat.PNG, 100, output) + return output.toByteArray() + } + + private companion object { + const val MAX_AGE_MS = 60_000L + } +} diff --git a/openmapview/src/test/kotlin/de/afarber/openmapview/TileCacheTest.kt b/openmapview/src/test/kotlin/de/afarber/openmapview/TileCacheTest.kt index 711dd44..513ef5a 100644 --- a/openmapview/src/test/kotlin/de/afarber/openmapview/TileCacheTest.kt +++ b/openmapview/src/test/kotlin/de/afarber/openmapview/TileCacheTest.kt @@ -10,16 +10,24 @@ package de.afarber.openmapview import android.graphics.Bitmap import org.junit.After import org.junit.Assert.assertEquals +import org.junit.Assert.assertFalse import org.junit.Assert.assertNotNull import org.junit.Assert.assertNull +import org.junit.Assert.assertTrue import org.junit.Before +import org.junit.Rule import org.junit.Test +import org.junit.rules.TemporaryFolder import org.junit.runner.RunWith import org.robolectric.RobolectricTestRunner import org.robolectric.RuntimeEnvironment +import java.io.ByteArrayOutputStream @RunWith(RobolectricTestRunner::class) class TileCacheTest { + @get:Rule + val folder = TemporaryFolder() + private lateinit var cache: TileCache @Before @@ -157,6 +165,77 @@ class TileCacheTest { assertEquals(bitmap15, cache.get(tile15)) } + @Test + fun testExpiredTileIsOnlyAvailableAsStale() { + var now = 1_000_000_000_000L + val disk = DiskTileCache(folder.newFolder(), DiskTileCache.DEFAULT_NAMESPACE, 1024 * 1024, 60_000, { now }) + val tile = TileCoordinate(1, 2, 3) + TileCache(disk).put(tile, createMockBitmap(256, 256), createPngBytes()) + now += 60_001 + + // A new cache has an empty memory cache, so the tile must come from disk + val stale = TileCache(disk) + assertNull(stale.get(tile)) + assertFalse(stale.isStale(tile)) + + assertNotNull(stale.getStale(tile)) + assertTrue(stale.isStale(tile)) + assertNotNull("Stale tile is in memory now", stale.getMemory(tile)) + + stale.put(tile, createMockBitmap(256, 256), createPngBytes()) + assertFalse("A fresh tile replaces the stale one", stale.isStale(tile)) + assertNotNull(TileCache(disk).get(tile)) + } + + @Test + fun testGetStaleReturnsFreshTile() { + val tile = TileCoordinate(1, 2, 3) + cache.put(tile, createMockBitmap(256, 256), createPngBytes()) + + assertNotNull(cache.getStale(tile)) + assertFalse(cache.isStale(tile)) + } + + @Test + fun testClearForgetsStaleTiles() { + var now = 1_000_000_000_000L + val disk = DiskTileCache(folder.newFolder(), DiskTileCache.DEFAULT_NAMESPACE, 1024 * 1024, 60_000, { now }) + val tile = TileCoordinate(1, 2, 3) + val stale = TileCache(disk) + disk.put(tile, createPngBytes()) + now += 60_001 + stale.getStale(tile) + + stale.clear() + + assertFalse(stale.isStale(tile)) + assertNull(stale.getStale(tile)) + } + + @Test + fun testEvictedTileLosesStaleMark() { + var now = 1_000_000_000_000L + val disk = DiskTileCache(folder.newFolder(), DiskTileCache.DEFAULT_NAMESPACE, 1024 * 1024, 60_000, { now }) + val a = TileCoordinate(1, 2, 3) + disk.put(a, createPngBytes()) + now += 60_001 + // Room for one 256x256 RGB_565 tile (128 KB) + val small = TileCache(disk, cacheSize = 200) + small.getStale(a) + assertTrue(small.isStale(a)) + + small.put(TileCoordinate(4, 5, 6), Bitmap.createBitmap(256, 256, Bitmap.Config.RGB_565)) + + assertNull("Evicted from memory", small.getMemory(a)) + assertFalse(small.isStale(a)) + } + + private fun createPngBytes(): ByteArray { + val output = ByteArrayOutputStream() + createMockBitmap(256, 256).compress(Bitmap.CompressFormat.PNG, 100, output) + return output.toByteArray() + } + private fun createMockBitmap( width: Int, height: Int,