Run the live cloud 51Did test in CI - #80
Merged
Merged
Conversation
…flags The live test created a 51Did through the json endpoint without sending id.usage. The service takes a caller who sends none as not having asked for a 51Did at all, so it creates none and the fodid element is absent, which would have failed the assertion in create() had the job ever got that far. It did not, because the workflow could not build the package. The request now sends id.usage=non-marketing, which is what every other port's live test sends. Two tests are added on top. The first asks for each id.usage the service offers and asserts the usage this package answers with, the terms address that usage must produce, that the usage was recorded as stated by the caller, and that the type is probabilistic. Read through the accessors rather than by masking, because the usage values are cumulative, being 001, 011 and 111, so a caller masking the byte for the non-marketing bit reads every marketing identifier as non-marketing. The second covers the half a caller cannot state for itself. A consent management platform sends an IAB TCF consent string and no usage of its own, the service decodes the purposes, decides the usage, and records in the identifier that it decided rather than was told. Two strings are sent, one granting all twelve purposes and one granting the Appendix 1 standard set, and each identifier must read as the matching usage with the consent bit set. No id.usage goes with them, because a stated usage wins over a consent string and sending one would prove the opposite. The address and the consent strings are written out here rather than taken from the package or shared with the service's own tests, because a test that asked either what it expects would agree with it whatever it said. Both write NOTHING PROVEN to stderr where they read no marketing identifier, so a key with no marketing entitlement cannot produce a pass that checked nothing. Verified: tox runs the file with 4 skipped where it ran 2, no resource key being set, which is the state off CI. The assertions need the live service.
The approved run failed before any test ran, with error: package directory 'src/fiftyone_pipeline_did/_owid' does not exist setup.py declares fiftyone_pipeline_did._owid as a package and its contents are copied out of the owid-python submodule rather than committed, so checking the submodule out is not enough on its own. The repository has ci/copy-owid-source.ps1 for exactly this and the workflow did not call it. The step runs from the repository root rather than the package directory the other steps default to, because that is where the script lives. Verified: the same sequence run locally, being submodule checkout then ci/copy-owid-source.ps1, produces the _owid directory and tox then runs the live test file, and the workflow file parses as YAML.
…w branch The workflow and the tests it runs were on separate branches, so a run of this pull request exercised the old assertions and would have proved nothing new. This brings them together so one approval proves both.
A runner prints the test counts and not the captured output, so a key that returned no marketing identifier would report a pass identical to one that proved the byte. The two summary reports are now skips, which show in the visible counts. The per-usage notes inside the loops stay on stderr, because they explain a skip rather than being one. Verified: the module compiles.
The approved run built the package and then reported 4 skipped on all three organisation resource keys, and the job passed. tox runs the tests in a clean environment and strips anything it is not told to keep, so _51DEGREES_RESOURCE_KEY never reached the tests and the class level skipUnless gate closed on every run. passenv now names the resource key, the licence key, their lower case forms and FOD_CLOUD_API_URL, which are the variables the live tests read. Verified: with the variable set to a junk value the four tests now run and fail against the cloud, where before they were skipped. Failing on a bad key is the proof that the gate opened.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Only
pipeline-dotnethas a workflow that runs its live cloud 51Did test. For this package, merged and green therefore means the unit tests passed, rather than that the package has read an identifier the production cloud wrote. This adds the missing workflow so that all six say the same thing.References 51Degrees/cloud#363.
What this adds
One file,
.github/workflows/cloud-51did.yml. Nothing else changes.It mirrors
pipeline-dotnet'scloud-51did.ymlso the six behave the same way:workflow_dispatch, pluspull_requestonmainwith path filters that include the workflow file itself, so a change to it is proven by its own pull request.51Degrees/common-ciforsteps/set-resource-keys.ps1, the central script that exports the_51DEGREES_RESOURCE_KEY*secrets. A resource key added at the organisation level is picked up with no change here.tox -- tests/test_did_client_live.pyonce for every key, saying which key each run used, and fails if any run fails.The test it runs already exists
fiftyone_pipeline_did/tests/test_did_client_live.pyreads_51DEGREES_RESOURCE_KEYand skips itself when that is unset. Nothing in the test changes.How it was verified
The file parses as YAML and the test it names exists at the path given. It has not been run, because it cannot be until it is on a branch in this repository. The first run of this pull request is the proof, and it runs with secrets available because the branch is here rather than on a fork.
What to watch on that first run
A run with no keys available exits zero, and the test skips rather than fails when no key is set. Both are right for a fork's pull request and both mean a green run can prove nothing on its own.
pipeline-dotnetfound exactly this: a resource key with no marketing usage reported a pass having read no terms at all. It now counts the identifiers whose terms were actually read and reports inconclusive at zero. This workflow matches the existing one rather than diverging, so that improvement belongs in the tests rather than here.