fix: conformance on Ox, Zimbra, SOGo, CCS and Bedework

docs/design/API_DESIGN.md

@@ -1218,7 +1218,11 @@ from both is dropped) are enough for the generic layer to report the losses
 itself, so a backend whose mapping is that simple writes no mapper. A
 `LOSSY` entry without `values` is allowed, for a backend with mapping rules
 of its own; the conformance suite can then check only that each value is
-either stored or refused before any I/O.
+either stored or refused before any I/O. A property kept only beside
+another is `LOSSY` with `requires`, the properties it needs: Zimbra keeps a
+task's `DTSTART` only where it has a `DUE`, so `task.start` there is
+`{"level": "lossy", "requires": ["DUE"]}`, and the generic layer reports a
+`DTSTART` without a `DUE` as lost.
 
 `supports()` counts `UNKNOWN` as not supported: it is the view for a caller
 deciding whether to rely on a feature, and nothing promises an unprobed one
@@ -1236,6 +1240,7 @@ initial set:
 | `collection.set-name` | `set_name()` |
 | `collection.mixed` | one collection holds several component kinds; `UNSUPPORTED` where each holds one (CCS, a task tracker), and `create_collection()` then makes one for the first kind asked, in sorted order, of those the backend holds |
 | `search.server-side`, `search.time-range`, `search.text` | how much the server filters; never affects results ([§4 Search](#4-search)) |
+| `listing.complete` | a listing (`search()`, `iter_search()`, the listing shorthands, `changes()`) returns every stored object, however many; `UNSUPPORTED` where the server cuts a long one at a limit of its own: the listing then raises `ListingLimitError` where the server says so with a 507 (Stalwart, at its `WebDav.maxResults`, 2000 by default), and is short where it does not |
 | `changes` | `changes()`: `FULL` with a native token, `EMULATED` by listing |
 | `changes.known` | `changes(known=...)` fetches only what changed: `FULL` where etags can be listed without content (CalDAV, vdir), `EMULATED` where everything is read and compared (JMAP, single `.ics` file, feed, Gitea) |
 | `write.wire` | `add(wire=...)`, `save(wire=...)`: given bytes are sent as they are; `FULL` on vdir, and on CalDAV through caldav's `save(raw_data=...)` (caldav 3.6, [caldav #763](https://github.com/python-caldav/caldav/issues/763)); `UNSUPPORTED` elsewhere |
@@ -1244,6 +1249,7 @@ initial set:
 | `identity.client-native-id` | `add(native_id=...)`: the caller may name the stored object (CalDAV href, vdir file name) |
 | `properties.passthrough` | unknown properties and components survive a round trip |
 | `recurrence`, `recurrence.edit-this-and-future` | stores a recurrence set: `RRULE`, `RDATE`, `EXDATE` and the overrides (`RECURRENCE-ID`); `UNSUPPORTED` drops all four, overrides included; [§3.7 Recurrence](#37-recurrence) |
+| `recurrence.rdate` | an event's `RDATE` survives where `recurrence` stores one; `UNSUPPORTED` drops it, and the overrides of the occurrences only it gives, the `RRULE` kept (Zimbra) |
 | `recurrence.edit-occurrence` | `save(occurrence)` merges it into the stored object ([§3.7](#37-recurrence)); `UNSUPPORTED` where the merge is not available yet (Q3) |
 | `task.complete-recurring` | `complete()` on a recurring task or one occurrence of it ([§3.7](#37-recurrence)) |
 | `task.uncomplete` | `uncomplete()` makes a completed task pending again; `UNSUPPORTED` where the server keeps it completed (GMX) |
@@ -1252,8 +1258,9 @@ initial set:
 | `task.start`, `task.due`, `task.completed`, `task.planned`, `task.estimate` | |
 | `task.relations.parent`, `task.relations.depends-on` | |
 | `categories`, `attendees` | for every component; on a task, attendees are its assignees |
-| `alarms` | `VALARM` components survive; `UNSUPPORTED` drops them |
+| `alarms` | `VALARM` components survive; `UNSUPPORTED` drops them; `LOSSY` sends them as they are, to come back rewritten, which `verify` lets pass (Ox makes a task's alarm trigger absolute and replaces its description) |
 | `timezones` | a time with a `TZID` keeps it (with its `VTIMEZONE`), a floating time stays floating; `LOSSY` stores both in UTC, a floating time read as local time ([§4](#4-search)) |
+| `timezones.floating` | a floating time stays floating, whatever `timezones` says of a time with a `TZID`; `LOSSY` stores it in UTC, read as local time (Zimbra and SOGo keep a `TZID` but not a floating time) |
 | `dates` | a `DATE` value stays a date; `LOSSY` stores midnight UTC |
 | `time-log` | the time log of a task or event ("Time tracking" in [§3.4](#34-task)); `LOSSY` with `details["lost"]`, or `details["total-only"]` |
 
@@ -1326,6 +1333,8 @@ Three cases, matching the roadmap's "raise, degrade, or emulate":
       property: str                     # e.g. "PRIORITY", "STATUS", "RELATED-TO"
       sent: object                      # the value in the item
       stored: object | None             # what the backend will store instead; None if dropped
+      note: str | None = None           # how the server changes it, where the backend cannot say
+                                        # what it stores (Ox rewrites a task's alarm): stored is sent
 
   class LossPolicy(Protocol):
       def __call__(self, losses: Sequence[Loss], item: Item) -> None: ...
@@ -1359,8 +1368,10 @@ property that was sent, after normalisation, except those the mapper had
 already reported as not storable (a lossy write, above), and skips what
 servers legitimately change: `DTSTAMP`, `SEQUENCE`, `LAST-MODIFIED`, property
 and parameter order, a parameter spelt out at its RFC 5545 default
-(`PARTSTAT=NEEDS-ACTION`, `RSVP=FALSE`, ...), and the identity a backend
-assigns (`uid`, `native_id`, `etag`;
+(`PARTSTAT=NEEDS-ACTION`, `RSVP=FALSE`, ...), a parameter the server
+adds (SOGo's `RSVP=TRUE`, Ox's `EMAIL=`: that loses nothing), an
+attendee it adds (Ox adds the organizer, as a scheduling server does),
+and the identity a backend assigns (`uid`, `native_id`, `etag`;
 [§3.2 Identity](#32-identity)). A difference raises `VerificationError`
 listing what the server dropped or changed. It is not a `LossyWriteError`:
 that one means nothing was sent, while a `VerificationError` means the
@@ -1389,7 +1400,8 @@ CalendaringError
 ├── RateLimitError                # carries .retry_after: float | None
 ├── VerificationError             # verify=True: written, but stored differently; carries .differences
 └── BackendError                  # anything else from the backend or transport
-    └── TransportError            # network, TLS, timeout
+    ├── TransportError            # network, TLS, timeout
+    └── ListingLimitError         # the server cut a listing at a limit of its own; nothing of it is returned
 
 CalendaringWarning
 └── LossyWriteWarning
@@ -1434,6 +1446,13 @@ CalendaringWarning
   it; a caller that can live with the gap takes `.results`.  The listing
   primitives return a `BrokenObject` in place of a `StoredObject` rather
   than raising, so the backend decides per object.
+- **A cut listing is not a partial result.** `ListingLimitError` is a
+  `BackendError`, not a `PartialResultError`: the server answered a 507 for
+  the collection, and caldav keeps none of what came before it, so there
+  is no `.results` to hand on and no `.broken` to name.  Where the
+  backend declares `listing.complete` unsupported, a caller expecting
+  more objects than the server's limit narrows the search itself, e.g. by
+  time range.
 - Fan-out ([§2.1 Workspace](#21-workspace)) does not raise; `MultiResult.raise_for_errors()` raises an `ExceptionGroup` of these.  `Workspace.search()` keeps a `PartialResultError`'s `.results` among the items, and the error among the errors.
 
 caldav's errors map one to one where they overlap:
@@ -1442,8 +1461,9 @@ caldav's errors map one to one where they overlap:
 `DAVError` carrying the `DAV:valid-sync-token` precondition, any 412, and a
 500 from a server flagged as above → `SyncTokenError`;
 `AuthorizationError` → `AuthenticationError` or `AuthorizationError` by
-status code, `RateLimitError` → `RateLimitError`, other `DAVError` →
-`BackendError`.
+status code, `RateLimitError` → `RateLimitError`, a 507 in a listing (inside
+the multistatus, or for the whole `REPORT` or `PROPFIND`) → `ListingLimitError`,
+other `DAVError` → `BackendError`.
 
 ---
 

news/+added-parameters.bugfix.rst

@@ -0,0 +1 @@
+``verify=True`` and the conformance suite no longer count a parameter the server adds as a difference where it says nothing about the value: SOGo adding ``RSVP=TRUE`` to an ATTENDEE, or Ox adding ``EMAIL=``, loses nothing, and neither does Ox adding the organizer as an attendee.  A parameter the server drops or changes still is one, and so is one it adds with a meaning of its own, such as a TZID or ``PARTSTAT=DECLINED``.

news/+completed-requires-status.bugfix.rst

@@ -0,0 +1 @@
+On a server that drops a task's completion time where the task has no status (Ox), ``task.completed`` is declared ``lossy``, and such a write is reported as losing the completion time instead of losing it silently.

news/+dropped-start.bugfix.rst

@@ -0,0 +1 @@
+``verify=True`` reports a start, end or due time the server dropped as a ``VerificationError``, instead of failing with ``KeyError``.

news/+dropped-vtimezone.bugfix.rst

@@ -0,0 +1 @@
+A CalDAV read puts back a VTIMEZONE the server dropped (CCS), from zoneinfo; ``wire`` keeps the bytes as received.

news/+listing-limit.feature.rst

@@ -0,0 +1 @@
+New capability ``listing.complete`` and error ``ListingLimitError``: a server that cuts a long listing at a limit of its own (Stalwart, at its ``WebDav.maxResults``, 2000 objects by default) declares ``listing.complete`` unsupported, and a listing that hits the limit raises ``ListingLimitError`` instead of a ``BackendError`` where the server answers it with a 507.  None of the listing is returned.

news/+loss-note.feature.rst

@@ -0,0 +1 @@
+A Loss has a note saying how the server changes a value where the backend cannot tell what it will store; the warn policy shows it.

news/+occurrence-restamped.bugfix.rst

@@ -0,0 +1 @@
+Deleting an occurrence of a recurring event no longer fails with ``ConflictError`` on servers that re-stamp every occurrence when one is deleted (Ox).

news/+recurrence-rdate.feature.rst

@@ -0,0 +1 @@
+New capability ``recurrence.rdate``: whether an ``RDATE`` survives where an ``RRULE`` does.  The CalDAV backend declares it unsupported on Zimbra, which drops every ``RDATE``, so the loss policy hears of it instead of the date silently disappearing.

news/+task-alarm-rewritten.bugfix.rst

@@ -0,0 +1 @@
+On a server that rewrites a task's alarm (Ox makes its trigger absolute and replaces its description), ``alarms`` is declared ``lossy`` on a task list, and such a write is reported as changing the alarm instead of failing verification.

news/+task-start-requires-due.bugfix.rst

@@ -0,0 +1 @@
+On a server that drops a task's start time where the task has no due time (Zimbra), ``task.start`` is declared ``lossy``, and a task with a start but no due is reported as losing its start instead of losing it silently.

news/+timezones-floating.feature.rst

@@ -0,0 +1 @@
+New capability ``timezones.floating``: whether a floating time stays floating.  The CalDAV backend declares it ``lossy`` where the server stores a floating time in UTC but keeps a ``TZID`` (Zimbra, SOGo), so such a time is written in UTC, read as local time, instead of being reported as changed.

src/calendaring/__init__.py

@@ -38,6 +38,7 @@
     ConfigurationError,
     ConflictError,
     InvalidDataError,
+    ListingLimitError,
     LossyWriteError,
     LossyWriteWarning,
     NotFoundError,
@@ -89,6 +90,7 @@
     "Event",
     "Feature",
     "InvalidDataError",
+    "ListingLimitError",
     "Item",
     "Journal",
     "Loss",

src/calendaring/_async/caldav.py

@@ -86,6 +86,8 @@ def _stored(self, obj: Any) -> StoredObject:
         # raw_data is what the server sent; data is what caldav made of it, which is what gets parsed.
         data = obj.data
         parsed = _parse(data.encode() if isinstance(data, str) else bytes(data))
+        # CCS drops a VTIMEZONE, which RFC 5545 section 3.2.19 requires for each TZID; zoneinfo has it.
+        parsed.add_missing_timezones()
         return StoredObject(_name_of(obj.url), obj.etag, parsed, obj.raw_data, obj)
 
     def _listed(self, obj: Any) -> StoredObject | BrokenObject:

src/calendaring/_async/collection.py

@@ -25,7 +25,7 @@
 import recurring_ical_events
 from icalendar_searcher import Searcher
 
-from .._mapping import Change, differences, plan_calendar_losses, plan_losses
+from .._mapping import NOT_VERIFIED, Change, differences, plan_calendar_losses, plan_losses, undeclared
 from ..capabilities import BACKEND_FEATURES, COMPONENT_FEATURE, Capabilities, Feature, Support
 from ..errors import (
     AlreadyExistsError,
@@ -93,6 +93,14 @@ def _occurrence_now(calendar: icalendar.Calendar, item: Item) -> icalendar.Compo
     return None
 
 
+def _unstamped(component: icalendar.Component) -> icalendar.Component:
+    """``component`` without what a server rewrites in every component of an object it stores (Ox)."""
+    out = copy.deepcopy(component)
+    for name in NOT_VERIFIED:
+        out.pop(name, None)
+    return out
+
+
 def _occurs_at(item: Item) -> datetime:
     """Where an occurrence sorts: its ``RECURRENCE-ID`` as an aware instant (floating counts as UTC)."""
     rid = item.component.get("RECURRENCE-ID")
@@ -431,7 +439,7 @@ def _prepare(
     async def _check(self, sent: icalendar.Calendar, result: AsyncItem) -> None:
         assert result.native_id is not None
         stored = await self._fetch(result.native_id)
-        found = differences(sent, stored.calendar)
+        found = undeclared(differences(sent, stored.calendar), self.capabilities)
         if found:
             raise VerificationError(found)
 
@@ -819,7 +827,8 @@ async def _delete_occurrence(self, item: Item, *, overwrite: bool) -> None:
         Siblings share the stored object's etag, so a stale etag is not a
         conflict by itself: only a change to this occurrence on the server is.
         It is found by comparing the stored occurrence with the item as the
-        caller holds it, so once the etag moved, a local edit before
+        caller holds it, leaving out what verify never compares (DTSTAMP, SEQUENCE,
+        LAST-MODIFIED, UID), so once the etag moved, a local edit before
         ``delete()`` also counts as one; with the etag unchanged nothing
         changed on the server, and a local edit is deleted with the rest.
         An occurrence already gone is ``NotFoundError``.
@@ -830,7 +839,7 @@ async def _delete_occurrence(self, item: Item, *, overwrite: bool) -> None:
             now = _occurrence_now(stored.calendar, item)
             if now is None:
                 raise NotFoundError(f"the occurrence {item.component.decoded('RECURRENCE-ID')} is gone")
-            if not overwrite and now != item.component:
+            if not overwrite and _unstamped(now) != _unstamped(item.component):
                 raise ConflictError(
                     f"the occurrence {item.component.decoded('RECURRENCE-ID')} changed since it was read"
                 )

src/calendaring/_async/conformance.py

@@ -47,6 +47,7 @@ async def open_backend(self) -> AsyncBackend:
     BackendError,
     CalendaringError,
     ConflictError,
+    ListingLimitError,
     LossyWriteError,
     LossyWriteWarning,
     NotFoundError,
@@ -773,8 +774,8 @@ async def test_property_round_trips_as_declared(self, collection: AsyncCollectio
             assert support is Support.LOSSY, f"{feature} is declared {support.value} but the write is lossy"
             return
         read = await collection.reload(stored)
-        if support is not Support.LOSSY:
-            assert property_lines(read.component, name) == property_lines(item.component, name)
+        if support is not Support.LOSSY:  # stored as verify sees it: a parameter the server adds loses nothing
+            assert [d for d in differences(item.icalendar, read.icalendar) if d.property == name] == []
 
     @pytest.mark.parametrize("feature", list(VALUE_SAMPLES))
     async def test_values_map_as_declared(self, collection: AsyncCollection, feature: Feature) -> None:
@@ -921,6 +922,14 @@ def fields(items: Sequence[AsyncItem]) -> list[tuple[Any, ...]]:
     async def test_listings_are_not_cut_at_a_page(self, collection: AsyncCollection) -> None:
         kind = next(k for k in KINDS if holds(collection, k))
         seeded = await self.seed(collection, [minimal(kind, f"paged {n}") for n in range(self.page_size_hint)])
+        try:
+            await self._listings_are_complete(collection, seeded)
+        except ListingLimitError:
+            # Declared: the server cuts a long listing, and says so (Stalwart, at its WebDav.maxResults)
+            if level(collection, Feature.LISTING_COMPLETE) is not Support.UNSUPPORTED:
+                raise
+
+    async def _listings_are_complete(self, collection: AsyncCollection, seeded: list[AsyncItem]) -> None:
         mine = {nid(i) for i in seeded}
         assert {nid(i) for i in stored_by(await collection.search(), seeded)} == mine
         assert {nid(i) async for i in collection.iter_search() if i.native_id in mine} == mine