diff --git a/.claude/skills/pattern-author/SKILL.md b/.claude/skills/pattern-author/SKILL.md new file mode 100644 index 00000000..281eef3b --- /dev/null +++ b/.claude/skills/pattern-author/SKILL.md @@ -0,0 +1,243 @@ +--- +name: pattern-author +description: > + Author and modify Validated Patterns — GitOps-based deployment configurations + for OpenShift. Use when creating new Patterns, adding applications/subscriptions/namespaces + to existing Patterns, configuring secrets, setting up hub/spoke clusters, or working with + clustergroup values files. +when_to_use: > + When the user asks to create a new Validated Pattern, add a helm chart or application to a + Pattern, configure secrets or vault, set up spoke clusters, modify clustergroup values, + or work with values-global.yaml, values-*.yaml, or values-secret.yaml.template files. + Also when the user mentions "Validated Patterns", "clustergroup", "pattern init", or + "patternizer". +allowed-tools: Read Bash(pattern *) Bash(helm *) Bash(find *) Bash(ls *) +--- + +# Validated Patterns Author + +You are helping author Validated Patterns — GitOps-based deployment configurations for OpenShift built on the [clustergroup helm chart](https://github.com/validatedpatterns/clustergroup-chart/). A Pattern is a Git repository containing values files that define what namespaces, operators, and applications to deploy on one or more OpenShift clusters via ArgoCD. + +For complete framework documentation, read [reference.md](reference.md) in this skill directory. Read it before your first Pattern authoring task in a session, or when you need details on a specific framework feature. + +## Authoring Workflow + +When creating a new Pattern: + +1. **Initialize** — Determine the Pattern name and whether secrets are needed. Run `pattern init` or `pattern init --with-secrets` in the Pattern directory. +2. **Identify requirements** — What operators, applications, and custom helm charts does this Pattern need? +3. **Define namespaces** — Add all required namespaces to the clustergroup values file (`values-.yaml`). Include OperatorGroup configuration for operator namespaces. +4. **Define subscriptions** — Add operator subscriptions with at minimum the operator `name`. Set `namespace`, `channel`, and `source` as needed. +5. **Define applications** — Wire in helm charts as applications: + - Local charts: set `path` to the chart location in the repository + - VP-published charts: set `chart` and `chartVersion` + - External Git charts: set `repoURL`, `path`, and `chartVersion` (Git ref) +6. **Configure secrets** (if applicable) — Define secrets in `values-secret.yaml.template` and create corresponding ExternalSecret CRDs in chart templates. +7. **Set up hub/spoke** (if multi-cluster) — Add ACM subscription and `managedClusterGroups` to the hub. Create spoke values files. +8. **Add imperative jobs** (if needed) — Configure Ansible playbooks in the imperative framework for tasks that don't fit the declarative model. + +When modifying an existing Pattern, read the current `values-global.yaml` and clustergroup values files first to understand the existing structure before making changes. + +## Rules + +These rules must always be followed: + +- **Map form for namespaces** — Always define namespaces as a map, never a list. Maps merge across values files; lists override entirely. +- **No secrets in Git** — Never put real secrets or credentials in the Pattern repository. Secrets belong in `~/values-secret-.yaml` on the user's machine. +- **`singleArgoCD: true`** — Always set this for new Patterns. +- **`multiSourceConfig.enabled: true`** — Always set this for new Patterns. +- **Vault only on hub** — The Vault application and namespace belong only on the hub/main cluster. Spoke clusters need ESO only (no Vault). The VP `openshift-external-secrets` chart auto-configures spokes to use the hub's Vault. +- **Chart values stubs** — A chart's `values.yaml` must include default stubs for any `.Values.global.*` or `.Values.clusterGroup.*` values referenced in its templates, so `helm template` works standalone during development. +- **ESO backtick escaping** — In ExternalSecret templates, escape ESO template expressions with backticks to prevent Helm from interpreting them: + + ```text + "{{ `{{ .field_name }}` }}" + ``` + +- **Idempotent imperative jobs** — All imperative jobs run on a schedule (every 10 minutes by default) and must be idempotent. +- **Re-run `pattern init`** — After adding new local helm charts, re-run `pattern init` to wire them into the clustergroup values file. It is idempotent. + +## Common Tasks + +### Adding an Operator + +Add three things to the clustergroup values file: a namespace, a subscription, and (if the operator needs its own chart for configuration) an application. + +```yaml +clusterGroup: + namespaces: + my-operator: + operatorGroup: true + targetNamespaces: [] + + subscriptions: + my-operator: + name: my-operator + namespace: my-operator + channel: stable + + applications: + my-operator-config: + name: my-operator-config + namespace: my-operator + path: charts/my-operator-config +``` + +Not every operator needs a local chart. If the operator requires no additional configuration beyond installation, the namespace and subscription are sufficient. + +### Adding a Local Helm Chart + +Place the chart anywhere in the repository (convention: `charts/`). Run `pattern init` to auto-discover it, or manually add it to the clustergroup values: + +```yaml +clusterGroup: + namespaces: + my-app: + + applications: + my-app: + name: my-app + namespace: my-app + path: charts/my-app +``` + +### Adding a VP-Published Chart + +```yaml +clusterGroup: + applications: + vault: + name: vault + namespace: vault + chart: hashicorp-vault + chartVersion: 0.1.* +``` + +### Adding a Chart from an External Git Repository + +```yaml +clusterGroup: + applications: + external-app: + name: external-app + namespace: external-app + repoURL: https://github.com/org/repo.git + chartVersion: main + path: charts/the-chart +``` + +### Adding a Secret + +1. Define the secret in `values-secret.yaml.template`: + + ```yaml + version: "2.0" + + secrets: + - name: my-secret + vaultPrefixes: + - global + fields: + - name: api-key + onMissingValue: prompt + - name: password + onMissingValue: generate + vaultPolicy: validatedPatternDefaultPolicy + ``` + +2. Add `secretStore` defaults to the chart's `values.yaml`: + + ```yaml + secretStore: + name: vault-backend + kind: ClusterSecretStore + + mysecret: + key: secret/data/global/my-secret + refreshInterval: 2m0s + ``` + +3. Create an ExternalSecret template in the chart: + + ```yaml + apiVersion: external-secrets.io/v1 + kind: ExternalSecret + metadata: + name: my-secret + spec: + refreshInterval: {{ .Values.mysecret.refreshInterval }} + secretStoreRef: + name: {{ .Values.secretStore.name }} + kind: {{ .Values.secretStore.kind }} + target: + name: my-secret + template: + type: Opaque + data: + api-key: "{{ `{{ .api_key }}` }}" + password: "{{ `{{ .password }}` }}" + data: + - secretKey: api_key + remoteRef: + key: {{ .Values.mysecret.key }} + property: api-key + - secretKey: password + remoteRef: + key: {{ .Values.mysecret.key }} + property: password + ``` + +The Vault path is `secret/data//`. The `secretKey` values become the template variables in `target.template.data`. + +### Adding a Spoke Cluster + +1. Add ACM and `managedClusterGroups` to the hub clustergroup values: + + ```yaml + clusterGroup: + name: hub + + namespaces: + open-cluster-management: + + subscriptions: + acm: + name: advanced-cluster-management + namespace: open-cluster-management + channel: release-2.16 + + applications: + acm: + name: acm + namespace: open-cluster-management + chart: acm + chartVersion: 0.2.* + + managedClusterGroups: + region-one: + name: group-one + acmlabels: + - name: clusterGroup + value: group-one + ``` + +2. Create `values-group-one.yaml` with the spoke's namespaces, subscriptions, and applications. + +3. If using secrets on the spoke, include ESO components (without Vault) in the spoke values. + +### Adding Conditional Overrides + +Define a custom global variable and use `sharedValueFiles`: + +```yaml +# values-global.yaml +global: + device: gpu + +# values-.yaml +clusterGroup: + sharedValueFiles: + - /overrides/values-{{ $.Values.global.device }}.yaml +``` + +Create the override file (e.g., `/overrides/values-gpu.yaml`) with the conditional namespaces, subscriptions, and applications. diff --git a/.claude/skills/pattern-author/reference.md b/.claude/skills/pattern-author/reference.md new file mode 100644 index 00000000..1a24ddaf --- /dev/null +++ b/.claude/skills/pattern-author/reference.md @@ -0,0 +1,951 @@ +# What are Validated Patterns + +[Validated Patterns](https://validatedpatterns.io/) are an advanced form of reference architectures that offer a streamlined approach to deploying complex business solutions. + +Validated Patterns are GitOps-based. A Validated Pattern is a Git repository containing the values files for the [clustergroup helm chart](https://github.com/validatedpatterns/clustergroup-chart/). + +Validated Patterns are installed via the [Validated Patterns Operator](https://github.com/validatedpatterns/patterns-operator) (available as a community operator in OpenShift's operator catalog). The operator creates and manages a subscription for OpenShift GitOps (ArgoCD) and creates the app-of-apps Application in ArgoCD — the clustergroup chart with values taken from the Pattern repository (as a multi-source ArgoCD Application). + +## Creating a New Pattern + +1. Create an empty directory (or create a new repository on GitHub/GitLab and clone it). The directory name becomes the Pattern name. +2. Run `podman run --pull=newer -v "$PWD:$PWD:z" -w "$PWD" quay.io/validatedpatterns/patternizer init` to create the new Pattern. Add `--with-secrets` to include the necessary components for the Validated Patterns secret framework: + `podman run --pull=newer -v "$PWD:$PWD:z" -w "$PWD" quay.io/validatedpatterns/patternizer init --with-secrets` + +[The patternizer source is available on GitHub.](https://github.com/validatedpatterns/patternizer) + +You can shorten the patternizer command by adding a shell function to your shell's startup: + +```bash +pattern() { + podman run --pull=newer \ + -v "$PWD:$PWD:z" \ + -w "$PWD" \ + quay.io/validatedpatterns/patternizer "$@" +} +``` + +Then run `pattern init` or `pattern init --with-secrets`. + +`pattern init` is idempotent. You can add secrets to a Pattern initialized without them by running `pattern init --with-secrets` later. You can also re-run `pattern init` as you add helm charts to the repository and it will wire them into the clustergroup values file. + +## Pattern File Structure + +Given a fresh Pattern directory with a user-defined helm chart: + +```bash +mkdir -p fresh-pattern/charts +cd fresh-pattern/charts +helm create user-defined-chart +cd .. +pattern init --with-secrets +``` + +The resulting structure: + +```text +fresh-pattern +├── ansible.cfg +├── charts +│ └── user-defined-chart +├── Makefile +├── Makefile-common +├── pattern.sh +├── values-global.yaml +├── values-prod.yaml +└── values-secret.yaml.template +``` + +### File Descriptions + +`ansible.cfg` contains defaults for the Ansible playbooks invoked via the Makefile (e.g., `make install`). These playbooks are defined in the [rhvp.cluster_utils collection](https://github.com/validatedpatterns/rhvp.cluster_utils). + +`charts/` is the recommended location for local helm charts. `pattern init` discovers charts anywhere in the repository, but `charts/` is the convention. + +`Makefile` includes `Makefile-common` and provides a place for Pattern-specific make targets or overrides. Most Patterns never need to override the common targets, but it's useful for custom tests, linting, or convenience functions. + +`Makefile-common` provides the core make targets for the Patterns framework. Documented more fully on the [VP blog](https://validatedpatterns.io/blog/2025-08-29-new-common-makefile-structure/). The primary targets are `make install`, `make uninstall`, and `make load-secrets`. These targets are run via the `./pattern.sh` wrapper script, which executes them inside the [VP Utility Container](https://github.com/validatedpatterns/utility-container). The utility container includes make, oc, helm, aws, and other CLIs, so the only local dependency is podman. A user can run `./pattern.sh make install` to install a Pattern (assuming they are logged into an OpenShift cluster). + +`pattern.sh` is the wrapper script that runs make targets inside the utility container. + +`values-secret.yaml.template` is a template showing how the Pattern's secrets should be formatted for the [secrets framework](#the-secrets-framework). This file should never contain real secrets. Copy it to your home directory (`cp values-secret.yaml.template ~/values-secret-fresh-pattern.yaml`) so secrets stay on your local machine, not in your Git repository. + +### values-global.yaml + +```yaml +global: + pattern: fresh-pattern + singleArgoCD: true + secretLoader: + disabled: false +main: + clusterGroupName: prod + multiSourceConfig: + enabled: true + clusterGroupChartVersion: 0.9.* +``` + +#### Field Reference + +`global.pattern` — The Pattern name, taken from the directory name by patternizer. Used as a label on ArgoCD Applications and as part of the ArgoCD namespace name. + +`global.singleArgoCD` — When `true` (the patternizer default), each cluster uses a single ArgoCD instance for both the app-of-apps and all child applications. When `false` (legacy behavior), each cluster runs two ArgoCD instances: one for the clustergroup chart (app-of-apps) and a separate one for the applications it defines. A hub/spoke Pattern with `singleArgoCD: false` would have two ArgoCD instances on the hub and two on the spoke. With `singleArgoCD: true`, there is one ArgoCD instance on the hub and one on the spoke. New Patterns should always use `true`. + +`global.secretLoader.disabled` — When `true`, skip loading secrets. Useful for Patterns that don't use the secrets framework. + +`main.clusterGroupName` — The name of the primary clustergroup. Determines which `values-.yaml` file defines the main cluster. If you change this value, the next `pattern init` run creates the corresponding values file (you would need to manually delete the old unused one). + +`main.multiSourceConfig.enabled` — Enable ArgoCD multi-source applications. Should always be `true` for modern Patterns. + +`main.multiSourceConfig.clusterGroupChartVersion` — SemVer constraint for the clustergroup chart version from the VP chart repository. + +There are also some global options that can be set in `values-global.yaml` but are not included by default: + +`global.options.syncPolicy` — Controls the ArgoCD sync policy for all applications. `"Automatic"` (the default) enables auto-sync with retry. `"Manual"` disables auto-sync. Per-application `syncPolicy` overrides this global default. + +`global.options.installPlanApproval` — Default `installPlanApproval` for all subscriptions. `"Automatic"` (the default) allows operators to auto-upgrade when new updates are available in their channel. Set to `"Manual"` to prevent auto-upgrades. Per-subscription values override this. + +`global.options.useCSV` — When `True` (the default), subscriptions include a `startingCSV` field if their `.csv` value is set. + +### values-prod.yaml + +`values-prod.yaml` is the values file that defines the main clustergroup of the Pattern. In hub/spoke Patterns (see [Spoke clusters](#spoke-clusters)) this would be the hub cluster. In single cluster Patterns this defines the solitary cluster. If there is one file to look at which defines the Pattern, it is this one. Its contents are explored in [The clustergroup values](#the-clustergroup-values) section. + +### The Pattern CR + +When you run `./pattern.sh make install` to deploy the Pattern, the values in `values-global.yaml` are passed to the [pattern-install chart](https://github.com/validatedpatterns/pattern-install-chart) by the [rhvp.cluster_utils.install](https://github.com/validatedpatterns/rhvp.cluster_utils/blob/main/playbooks/install.yml) playbook. This chart creates a subscription for the Validated Patterns Operator, a configmap with default operator configuration, and the Pattern CR: + +```yaml +apiVersion: gitops.hybrid-cloud-patterns.io/v1alpha1 +kind: Pattern +metadata: + name: fresh-pattern + namespace: openshift-operators +spec: + clusterGroupName: prod + gitSpec: + targetRepo: https://github.com/validatedpatterns/fresh-pattern.git + targetRevision: main + multiSourceConfig: + enabled: true + clusterGroupChartVersion: 0.9.* +``` + +Your branch (`main` in this example) must have an upstream remote set and be pushed to that remote. + +## Values File Hierarchy + +When the Patterns Operator creates the app-of-apps representing the Pattern, it renders the [clustergroup chart](https://github.com/validatedpatterns/clustergroup-chart/) and automatically includes certain values files (if they exist) to customize resources for a given clustergroup, OCP version, and platform. + +The following values files are automatically included for each application, in this order: + +1. `values-global.yaml` — included for all clustergroups +2. `values-.yaml` — values specific to the clustergroup +3. `values-.yaml` — values specific to the platform (e.g., `values-AWS.yaml`) +4. `values--.yaml` — platform and version specific (e.g., `values-AWS-4.21.yaml`) +5. `values--.yaml` — platform and clustergroup specific (e.g., `values-AWS-hub.yaml`) +6. `values--.yaml` — version and clustergroup specific (e.g., `values-4.21-hub.yaml`) +7. `values-.yaml` — values for a specifically named cluster (e.g., `values-test-cluster.yaml`) +8. Files from `global.extraValueFiles` — additional global value files +9. Per-clustergroup `sharedValueFiles` — from `clusterGroup.sharedValueFiles` +10. Per-application `extraValueFiles` — from the application's `extraValueFiles` field + +ArgoCD is configured with `ignoreMissingValueFiles: true`, so it silently skips any of these files that do not exist. Only create the files you actually need. + +In multi-source mode (the default), values files from the Pattern repository are prefixed with `$patternref/` to tell ArgoCD which source they come from. This is handled automatically by the clustergroup chart. + +Some of these values files are cross-clustergroup. For example, `values-AWS.yaml` would apply to both hub and spoke clustergroups running on AWS. + +### Shared Value Files + +The clustergroup provides a `sharedValueFiles` field for including additional overrides for all applications in that clustergroup: + +```yaml +clusterGroup: + sharedValueFiles: + - '/overrides/values-{{ $.Values.global.clusterPlatform }}.yaml' + - '/overrides/values-{{ $.Values.global.clusterPlatform }}-{{ $.Values.global.clusterVersion }}.yaml' + - '/overrides/values-{{ $.Values.global.clusterPlatform }}-{{ $.Values.clusterGroup.name }}.yaml' + - '/overrides/values-{{ $.Values.global.clusterVersion }}-{{ $.Values.clusterGroup.name }}.yaml' + - '/overrides/values-{{ $.Values.global.localClusterName }}.yaml' +``` + +These paths support Helm template interpolation. The Validated Patterns operator handles resolving the global variables. As with the auto-included files, any that do not exist are silently skipped. + +By convention, extra value files are placed in an `overrides/` directory in the Pattern repository, but any location works. + +### Custom Globals for Conditional Overrides + +You can define your own global variables in `values-global.yaml` and use them in `sharedValueFiles` to enable conditional configuration. Consider this example: + +```yaml +# values-global.yaml +global: + pattern: ai-quickstart-rag + device: cpu # one of 'cpu' (no GPU) or 'gpu' (NVIDIA GPU) +main: + clusterGroupName: prod + multiSourceConfig: + enabled: true + clusterGroupChartVersion: 0.9.* +``` + +This enables conditional value file inclusion: + +```yaml +clusterGroup: + sharedValueFiles: + - /overrides/values-{{ $.Values.global.device }}.yaml + - /overrides/values-{{ $.Values.global.device }}-{{ $.Values.global.clusterPlatform }}.yaml +``` + +```yaml +# /overrides/values-gpu.yaml +global: + models: + llama-3-2-3b-instruct: + enabled: true + +llm-service: + device: gpu + +clusterGroup: + namespaces: + openshift-nfd: + nvidia-gpu-operator: + + subscriptions: + nfd: + name: nfd + namespace: openshift-nfd + nvidia: + name: gpu-operator-certified + namespace: nvidia-gpu-operator + source: certified-operators + + applications: + nfd: + name: nfd + namespace: openshift-nfd + path: charts/nfd + nvidia-config: + name: nvidia-config + namespace: nvidia-gpu-operator + path: charts/nvidia-config +``` + +```yaml +# /overrides/values-gpu-AWS.yaml +clusterGroup: + imperative: + jobs: + - name: deploy-nvidia-gpu + playbook: rhvp.cluster_utils.create_machineset + verbosity: -vvv + extravars: + - max_machineset_count=1 + - machineset_replicas=1 + - ensure_two_machine_minimum=false + - machineset_name=nvidia-gpu + - machineset_labels= + - machineset_instance_type=g6.2xlarge + - 'machineset_taints=[{"effect":"NoSchedule","key":"nvidia.com/gpu","value":"true"}]' + - 'machineset_node_labels={"node-role.kubernetes.io/nvidia-gpu":""}' + - machineset_api_version=machine.openshift.io/v1beta1 + clusterRoleYaml: + - apiGroups: + - "*" + resources: + - machinesets + - persistentvolumeclaims + - datavolumes + - dataimportcrons + - datasources + verbs: + - "*" + - apiGroups: + - "*" + resources: + - "*" + verbs: + - get + - list + - watch +``` + +This example leverages the `global.device` value to install GPU-specific subscriptions and create an additional machineset when deployed to AWS with a GPU. + +## The Clustergroup Values + +A Pattern IS a clustergroup. The Git repository containing a Validated Pattern contains the values files for the [clustergroup helm chart](https://github.com/validatedpatterns/clustergroup-chart/). The clustergroup values file (`values-.yaml`) is the central definition of what a Pattern deploys — which namespaces to create, which operators to install, and which applications (helm charts) to run. + +Here is the `values-prod.yaml` generated by `pattern init --with-secrets` for our example Pattern. It defines a single clustergroup named `prod` with the namespaces, operator subscriptions, and applications needed for the secrets framework alongside the user's own chart: + +```yaml +clusterGroup: + name: prod + namespaces: + fresh-pattern: + vault: + external-secrets-operator: + operatorGroup: true + targetNamespaces: [] + external-secrets: + subscriptions: + eso: + name: openshift-external-secrets-operator + namespace: external-secrets-operator + channel: stable-v1 + applications: + openshift-external-secrets: + name: openshift-external-secrets + namespace: external-secrets + chart: openshift-external-secrets + chartVersion: 0.0.* + user-defined-chart: + name: user-defined-chart + namespace: fresh-pattern + path: charts/user-defined-chart + vault: + name: vault + namespace: vault + chart: hashicorp-vault + chartVersion: 0.1.* +``` + +These three sections — `namespaces`, `subscriptions`, and `applications` — are the core building blocks of every clustergroup. Each is detailed below. + +### Namespaces + +The namespace section accepts simple strings or more complex mappings: + +```yaml +clusterGroup: + namespaces: + fresh-pattern: + external-secrets-operator: + operatorGroup: true + targetNamespaces: [] +``` + +Setting `operatorGroup: true` creates an OperatorGroup with the same name as the namespace using the specified `targetNamespaces`. + +Labels and annotations can also be set: + +```yaml +clusterGroup: + namespaces: + rag-llm: + labels: + opendatahub.io/dashboard: "true" + modelmesh-enabled: "false" +``` + +Namespaces can also be defined as a list: + +```yaml +clusterGroup: + namespaces: + - fresh-pattern + - vault + - external-secrets-operator: + operatorGroup: true + targetNamespaces: [] + - external-secrets +``` + +The map form is recommended over the list form. When merging values files, lists are overridden entirely whereas maps are merged. + +### Subscriptions + +Subscriptions define operators that should be installed on the cluster. + +The only required field is the name of the subscription: + +```yaml +clusterGroup: + subscriptions: + servicemesh-console: + name: kiali-ossm +``` + +The clustergroup chart provides defaults for the other fields, making this equivalent to: + +```yaml +clusterGroup: + subscriptions: + servicemesh-console: + name: kiali-ossm + namespace: openshift-operators + source: redhat-operators + sourceNamespace: openshift-marketplace + channel: stable + installPlanApproval: Automatic +``` + +You can also specify `.csv` for the `startingCSV` of the subscription if needed. + +The `name` must be the operator's name in the `source` catalog. The default source is `redhat-operators`, but some operators are in `certified-operators`, `community-operators`, or `redhat-marketplace`. + +`namespace` is where the operator is installed. For operators like ESO, this is an OperatorGroup namespace. + +`source` and `sourceNamespace` only need updating in disconnected or custom install scenarios where the standard catalog sources are unavailable. + +`channel` is operator-specific. Some operators publish on multiple channels (e.g., `fast`, `stable-3.x`). + +Set `installPlanApproval` to `Manual` to prevent the operator from upgrading automatically when new updates are available in its channel. The default is `Automatic`. + +### Applications + +#### Local Helm Charts + +If `path` is provided (and `repoURL` and `chartVersion` are not), the helm chart is sourced from the Pattern repository: + +```yaml +clusterGroup: + applications: + user-defined-chart: + name: user-defined-chart + namespace: fresh-pattern + path: charts/user-defined-chart +``` + +#### VP-Published Helm Charts + +If neither `repoURL` nor `path` are provided, charts are sourced from the [Validated Patterns chart repository](https://charts.validatedpatterns.io/). Specify a version with `chartVersion`: + +```yaml +clusterGroup: + applications: + vault: + name: vault + namespace: vault + chart: hashicorp-vault + chartVersion: 0.1.* +``` + +#### Helm Charts from External Git Repositories + +Helm charts from external Git repositories can be referenced with `repoURL` (the publicly reachable Git URL), `path` (the path within the repository), and `chartVersion` (the Git revision to use): + +```yaml +clusterGroup: + applications: + maas-quickstart: + name: maas-quickstart + repoURL: https://github.com/dminnear-rh/maas-code-assistant.git + chartVersion: main + path: charts/maas-code-assistant +``` + +#### Additional Application Fields + +Several additional fields are available for any application type: + +`extraValueFiles` — Paths (relative to the Pattern repository root) to additional values files passed to the helm chart: + +```yaml +extraValueFiles: + - /overrides/maas-quickstart.yaml +``` + +`overrides` — Direct helm value overrides: + +```yaml +overrides: + - name: grafana.namespace + value: grafana +``` + +`ignoreDifferences` — Instructs ArgoCD to ignore certain differences when computing the sync diff: + +```yaml +ignoreDifferences: + - kind: Secret + name: grafana-proxy + namespace: grafana + jsonPointers: + - /data/session_secret +``` + +A full example combining these: + +```yaml +clusterGroup: + applications: + maas-quickstart: + name: maas-quickstart + repoURL: https://github.com/dminnear-rh/maas-code-assistant.git + chartVersion: main + path: charts/maas-code-assistant + extraValueFiles: + - /overrides/maas-quickstart.yaml + overrides: + - name: grafana.namespace + value: grafana + ignoreDifferences: + - kind: Secret + name: grafana-proxy + namespace: grafana + jsonPointers: + - /data/session_secret +``` + +## Using Helm Charts in VP + +This section covers what the VP framework provides to your charts, not helm chart authoring in general. + +### Where Charts Live + +The default location is `charts/` in the Pattern repository. `patternizer init` auto-discovers charts anywhere in the repository, so you can organize them however you like. + +Charts can also live in separate Git repositories and be referenced via `repoURL` in the application definition. VP-published charts are available from [charts.validatedpatterns.io](https://charts.validatedpatterns.io/) (referenced with `chart:` and `chartVersion:`). + +### How Values Flow into Charts + +Every ArgoCD Application created by the clustergroup chart receives the full merged tree of values files described in [Values File Hierarchy](#values-file-hierarchy). This means every chart sees: + +- `.Values.global.*` — global Pattern values +- `.Values.clusterGroup.*` — clustergroup configuration +- Chart-specific values from the chart's own `values.yaml` + +In addition to the values files, the clustergroup chart injects helm parameters for cluster-specific values that are known at deploy time: + +- `global.repoURL` — Pattern repository URL +- `global.targetRevision` — Git branch/commit/ref +- `global.namespace` — the ArgoCD app namespace +- `global.pattern` — Pattern name +- `global.clusterDomain` — cluster FQDN +- `global.localClusterName` — local cluster identifier +- `global.clusterVersion` — OpenShift version +- `global.clusterPlatform` — platform type (e.g., AWS, Azure, GCP) +- `global.hubClusterDomain` — hub cluster FQDN +- `global.localClusterDomain` — local cluster FQDN + +These are available in templates as `.Values.global.`. + +A chart's `values.yaml` should include default stubs for any `global.*` or `clusterGroup.*` values referenced in its templates. These defaults enable standalone `helm template` to work during development and are overridden at deploy time by the merged values tree. + +### Example: config-demo Chart + +The [config-demo chart](https://github.com/validatedpatterns/multicloud-gitops/tree/main/charts/all/config-demo) from multicloud-gitops is a minimal working example. Its `values.yaml`: + +```yaml +secretStore: + name: vault-backend + kind: ClusterSecretStore + +configdemosecret: + key: secret/data/global/config-demo + refreshInterval: 2m0s + +global: + hubClusterDomain: hub.example.com + localClusterDomain: region-one.example.com + +clusterGroup: + isHubCluster: true + +image: + repository: registry.access.redhat.com/ubi10/httpd-24 + tag: "10.0-1755779646" + pullPolicy: IfNotPresent +``` + +The `global` and `clusterGroup` stubs provide defaults for development. The `secretStore` and `configdemosecret` sections are chart-specific values used by the ExternalSecret template (see [Consuming Secrets in Charts](#consuming-secrets-in-charts)). + +The deployment template uses chart-specific values: + +```yaml +containers: +- name: apache + image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" + imagePullPolicy: {{ .Values.image.pullPolicy }} +``` + +The ConfigMap template uses global values injected by the framework, demonstrating how charts can reference cluster-specific information without hardcoding it: + +```yaml +data: + "index.html": |- +

+ Hub Cluster domain is '{{ .Values.global.hubClusterDomain }}'
+ Pod is running on Local Cluster Domain '{{ .Values.global.localClusterDomain }}'
+

+``` + +## The Secrets Framework + +Secrets in the VP framework are stored in Vault and consumed in charts via External Secrets Operator (ESO). The workflow is: define secrets in `values-secret.yaml.template`, load them into Vault with `./pattern.sh make install` or `./pattern.sh make load-secrets`, and consume them in charts using ExternalSecret CRDs. + +### Defining Secrets + +The `values-secret.yaml.template` file defines the secrets a Pattern needs. The install/load-secrets command looks for secrets in `~/values-secret-.yaml` and falls back to the template in the Pattern repository. This encourages users to copy the template to their home directory and keep secrets out of the Git repository. + +Example from [multicloud-gitops](https://github.com/validatedpatterns/multicloud-gitops/blob/main/values-secret.yaml.template): + +```yaml +version: "2.0" + +secrets: + - name: config-demo + vaultPrefixes: + - global + fields: + - name: secret + onMissingValue: generate + vaultPolicy: validatedPatternDefaultPolicy +``` + +#### Secret Field Reference + +Each secret entry supports these fields: + +- `name` — secret name (becomes the Vault path segment) +- `vaultPrefixes` — list of Vault path prefixes controlling which clustergroups can read the secret +- `fields[].name` — field name within the secret +- `fields[].value` — literal value +- `fields[].path` — path to a file containing the value +- `fields[].ini_file`, `ini_section`, `ini_key` — read a value from an INI file +- `fields[].onMissingValue` — set to `generate` to auto-generate the value +- `fields[].vaultPolicy` — policy name for generation (either `validatedPatternDefaultPolicy` or a custom policy) + +Example with various field types: + +```yaml +secrets: + # AWS credentials from INI file + - name: aws + fields: + - name: aws_access_key_id + ini_file: ~/.aws/credentials + ini_section: default + ini_key: aws_access_key_id + - name: aws_secret_access_key + ini_file: ~/.aws/credentials + ini_key: aws_secret_access_key + + # SSH keys from files + - name: publickey + fields: + - name: content + path: ~/.ssh/id_rsa.pub + - name: privatekey + fields: + - name: content + path: ~/.ssh/id_rsa + + # OpenShift pull secret from file + - name: openshiftPullSecret + fields: + - name: content + path: ~/.pullsecret.json +``` + +### Vault Prefixes and Access Control + +The `vaultPrefixes` field controls which clustergroups can access a secret. The Vault path convention is `secret/data//`. + +- `global` — readable by all clustergroups (hub and spoke) +- A clustergroup name (e.g., `hub`) — readable only by that clustergroup +- Multiple prefixes write the secret to multiple paths, making it accessible from multiple clustergroups + +### Secret Generation and Policies + +Secrets can be auto-generated if no value is provided by setting `onMissingValue: generate`. The generation is controlled by a vault policy: + +```yaml +version: "2.0" + +vaultPolicies: + basicPolicy: | + length=16 + rule "charset" { charset = "abcdefghijklmnopqrstuvwxyz" min-chars = 1 } + rule "charset" { charset = "ABCDEFGHIJKLMNOPQRSTUVWXYZ" min-chars = 1 } + rule "charset" { charset = "0123456789" min-chars = 1 } + +secrets: + - name: pgvector + fields: + - name: user + value: postgres + - name: password + onMissingValue: generate + vaultPolicy: basicPolicy + - name: dbname + value: rag_blueprint + - name: host + value: pgvector + - name: port + value: "5432" +``` + +### Consuming Secrets in Charts + +Secrets stored in Vault are consumed in charts using ESO ExternalSecret CRDs. The VP `openshift-external-secrets` chart sets up a `ClusterSecretStore` named `vault-backend` that points to the Vault instance. + +A chart that needs secrets should include `secretStore` defaults in its `values.yaml`: + +```yaml +secretStore: + name: vault-backend + kind: ClusterSecretStore + +configdemosecret: + key: secret/data/global/config-demo + refreshInterval: 2m0s +``` + +The ExternalSecret template maps secrets from Vault into Kubernetes Secrets: + +```yaml +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: config-demo-secret + namespace: config-demo +spec: + refreshInterval: {{ .Values.configdemosecret.refreshInterval }} + secretStoreRef: + name: {{ .Values.secretStore.name }} + kind: {{ .Values.secretStore.kind }} + target: + name: config-demo-secret + template: + type: Opaque + data: + secret: "{{ `{{ .configdemo_secret }}` }}" + data: + - secretKey: configdemo_secret + remoteRef: + key: {{ .Values.configdemosecret.key }} + property: secret +``` + +#### How the Mapping Works + +Given a secret defined in `values-secret.yaml.template` as: + +```yaml +secrets: + - name: config-demo + vaultPrefixes: + - global + fields: + - name: secret +``` + +The Vault path is: `secret/data/global/config-demo` (constructed as `secret/data//`). + +In the ExternalSecret: + +- `remoteRef.key` is set to this Vault path (`secret/data/global/config-demo`) +- `remoteRef.property` is the field name (`secret`) +- `data[].secretKey` (`configdemo_secret`) is the local key used to reference the fetched value in the `target.template` + +#### Backtick Escaping for ESO Templates + +The `target.template.data` section uses ESO's own template syntax (`{{ .fieldname }}`) to map fetched values into the Kubernetes Secret. Since the ExternalSecret is itself rendered by Helm, the ESO template expressions must be escaped to prevent Helm from interpreting them. The pattern is: + +```text +"{{ `{{ .configdemo_secret }}` }}" +``` + +The backticks create a Go raw string literal that Helm passes through unchanged. ESO then processes `{{ .configdemo_secret }}` at runtime. + +### Hub vs. Spoke Secret Infrastructure + +The Vault/ESO components should only be defined on the hub/main cluster: + +```yaml +# Hub cluster needs both Vault and ESO +clusterGroup: + namespaces: + vault: + external-secrets-operator: + operatorGroup: true + targetNamespaces: [] + external-secrets: + subscriptions: + eso: + name: openshift-external-secrets-operator + namespace: external-secrets-operator + channel: stable-v1 + applications: + openshift-external-secrets: + name: openshift-external-secrets + namespace: external-secrets + chart: openshift-external-secrets + chartVersion: 0.0.* + vault: + name: vault + namespace: vault + chart: hashicorp-vault + chartVersion: 0.1.* +``` + +Spoke clusters only need ESO — no Vault: + +```yaml +# Spoke cluster needs ESO only +clusterGroup: + namespaces: + external-secrets-operator: + operatorGroup: true + targetNamespaces: [] + external-secrets: + subscriptions: + eso: + name: openshift-external-secrets-operator + namespace: external-secrets-operator + channel: stable-v1 + applications: + openshift-external-secrets: + name: openshift-external-secrets + namespace: external-secrets + chart: openshift-external-secrets + chartVersion: 0.0.* +``` + +The VP `openshift-external-secrets` chart automatically configures spoke clusters to use the Vault instance on the hub cluster as the external secret store. + +For more information, see [Secrets management in the Validated Patterns framework](https://validatedpatterns.io/learn/secrets-management-in-the-validated-patterns-framework/). + +## Spoke Clusters + +Hub/spoke cluster support uses ACM (Advanced Cluster Management). Include the ACM components in your hub clustergroup: + +```yaml +clusterGroup: + name: hub + + namespaces: + open-cluster-management: + + subscriptions: + acm: + name: advanced-cluster-management + namespace: open-cluster-management + channel: release-2.16 + + applications: + acm: + name: acm + namespace: open-cluster-management + chart: acm + chartVersion: 0.2.* + + managedClusterGroups: + exampleRegion: + name: group-one + acmlabels: + - name: clusterGroup + value: group-one +``` + +This installs ACM on the hub cluster via the Validated Patterns ACM chart. The `managedClusterGroups` defines the mapping the framework uses to determine which clusters imported into ACM correspond to which clustergroup values files. + +To create a spoke clustergroup, create `values-group-one.yaml` with its own namespaces, subscriptions, and applications. For secrets, include the ESO components (without Vault) as described in [Hub vs. Spoke Secret Infrastructure](#hub-vs-spoke-secret-infrastructure). + +```yaml +clusterGroup: + name: group-one + + namespaces: + config-demo: + hello-world: + external-secrets-operator: + operatorGroup: true + targetNamespaces: [] + external-secrets: + + subscriptions: + eso: + name: openshift-external-secrets-operator + namespace: external-secrets-operator + channel: stable-v1 + + applications: + openshift-external-secrets: + name: openshift-external-secrets + namespace: external-secrets + chart: openshift-external-secrets + chartVersion: 0.0.* + config-demo: + name: config-demo + namespace: config-demo + path: charts/all/config-demo + hello-world: + name: hello-world + namespace: hello-world + path: charts/all/hello-world +``` + +Any cluster imported into ACM with the label `clusterGroup: group-one` will have the clustergroup chart applied using `values-global.yaml` and `values-group-one.yaml` (plus all the other automatically included platform/version values files). + +In short, spoke and hub clusters are both just clustergroups. The difference is that the hub cluster includes ACM and the Vault components. While the Patterns Operator installs the Pattern via the clustergroup chart in ArgoCD on the hub cluster, the ACM chart pushes policies to spoke clusters for installing OpenShift GitOps (ArgoCD) and the clustergroup chart for that spoke cluster. + +## The Imperative Framework + +Sometimes tasks don't fit neatly into a declarative framework. The imperative framework runs Ansible playbooks on a schedule against the cluster. + +```yaml +clusterGroup: + imperative: + jobs: + - name: trilio-enable-cr + playbook: ansible/playbooks/imperative-enable-cr.yaml + timeout: 900 + + - name: trilio-cr-backup + playbook: ansible/playbooks/imperative-cr-backup.yaml + timeout: 1200 + + - name: trilio-backup + playbook: ansible/playbooks/imperative-backup.yaml + timeout: 1200 + + - name: trilio-restore-standard + playbook: ansible/playbooks/imperative-restore-standard.yaml + timeout: 1800 + + - name: trilio-e2e-status + playbook: ansible/playbooks/imperative-e2e-status.yaml + timeout: 120 +``` + +Imperative jobs typically reference playbooks stored in the `ansible/` directory of the Pattern repository, or playbooks from the `rhvp.cluster_utils` Ansible collection. Jobs are defined as a list since they run in order — if one fails, the imperative job fails and remaining jobs are aborted. Jobs must be idempotent since they run on a schedule (every 10 minutes by default). + +The full set of imperative framework defaults: + +```yaml +imperative: + jobs: [] + image: quay.io/validatedpatterns/imperative-container:v1 + ansibleDevMode: + enabled: false + requirementsFile: "requirements.yml" + requirementsContent: "" + ansibleCfgFile: "ansible.cfg" + ansibleCfgContent: "" + namespace: "imperative" + valuesConfigMap: "helm-values-configmap" + cronJobName: "imperative-cronjob" + jobName: "imperative-job" + imagePullPolicy: Always + activeDeadlineSeconds: 3600 + schedule: "*/10 * * * *" + insecureUnsealVaultInsideClusterSchedule: "*/5 * * * *" + verbosity: "" + extraPlaybookArgs: [] + serviceAccountCreate: true + serviceAccountName: imperative-sa + clusterRoleName: imperative-cluster-role + clusterRoleYaml: "" + roleName: imperative-role + roleYaml: "" + adminServiceAccountCreate: true + adminServiceAccountName: imperative-admin-sa + adminClusterRoleName: imperative-admin-cluster-role + vaultNamespace: "vault" +``` diff --git a/.cursor/skills/pattern-author/SKILL.md b/.cursor/skills/pattern-author/SKILL.md new file mode 100644 index 00000000..281eef3b --- /dev/null +++ b/.cursor/skills/pattern-author/SKILL.md @@ -0,0 +1,243 @@ +--- +name: pattern-author +description: > + Author and modify Validated Patterns — GitOps-based deployment configurations + for OpenShift. Use when creating new Patterns, adding applications/subscriptions/namespaces + to existing Patterns, configuring secrets, setting up hub/spoke clusters, or working with + clustergroup values files. +when_to_use: > + When the user asks to create a new Validated Pattern, add a helm chart or application to a + Pattern, configure secrets or vault, set up spoke clusters, modify clustergroup values, + or work with values-global.yaml, values-*.yaml, or values-secret.yaml.template files. + Also when the user mentions "Validated Patterns", "clustergroup", "pattern init", or + "patternizer". +allowed-tools: Read Bash(pattern *) Bash(helm *) Bash(find *) Bash(ls *) +--- + +# Validated Patterns Author + +You are helping author Validated Patterns — GitOps-based deployment configurations for OpenShift built on the [clustergroup helm chart](https://github.com/validatedpatterns/clustergroup-chart/). A Pattern is a Git repository containing values files that define what namespaces, operators, and applications to deploy on one or more OpenShift clusters via ArgoCD. + +For complete framework documentation, read [reference.md](reference.md) in this skill directory. Read it before your first Pattern authoring task in a session, or when you need details on a specific framework feature. + +## Authoring Workflow + +When creating a new Pattern: + +1. **Initialize** — Determine the Pattern name and whether secrets are needed. Run `pattern init` or `pattern init --with-secrets` in the Pattern directory. +2. **Identify requirements** — What operators, applications, and custom helm charts does this Pattern need? +3. **Define namespaces** — Add all required namespaces to the clustergroup values file (`values-.yaml`). Include OperatorGroup configuration for operator namespaces. +4. **Define subscriptions** — Add operator subscriptions with at minimum the operator `name`. Set `namespace`, `channel`, and `source` as needed. +5. **Define applications** — Wire in helm charts as applications: + - Local charts: set `path` to the chart location in the repository + - VP-published charts: set `chart` and `chartVersion` + - External Git charts: set `repoURL`, `path`, and `chartVersion` (Git ref) +6. **Configure secrets** (if applicable) — Define secrets in `values-secret.yaml.template` and create corresponding ExternalSecret CRDs in chart templates. +7. **Set up hub/spoke** (if multi-cluster) — Add ACM subscription and `managedClusterGroups` to the hub. Create spoke values files. +8. **Add imperative jobs** (if needed) — Configure Ansible playbooks in the imperative framework for tasks that don't fit the declarative model. + +When modifying an existing Pattern, read the current `values-global.yaml` and clustergroup values files first to understand the existing structure before making changes. + +## Rules + +These rules must always be followed: + +- **Map form for namespaces** — Always define namespaces as a map, never a list. Maps merge across values files; lists override entirely. +- **No secrets in Git** — Never put real secrets or credentials in the Pattern repository. Secrets belong in `~/values-secret-.yaml` on the user's machine. +- **`singleArgoCD: true`** — Always set this for new Patterns. +- **`multiSourceConfig.enabled: true`** — Always set this for new Patterns. +- **Vault only on hub** — The Vault application and namespace belong only on the hub/main cluster. Spoke clusters need ESO only (no Vault). The VP `openshift-external-secrets` chart auto-configures spokes to use the hub's Vault. +- **Chart values stubs** — A chart's `values.yaml` must include default stubs for any `.Values.global.*` or `.Values.clusterGroup.*` values referenced in its templates, so `helm template` works standalone during development. +- **ESO backtick escaping** — In ExternalSecret templates, escape ESO template expressions with backticks to prevent Helm from interpreting them: + + ```text + "{{ `{{ .field_name }}` }}" + ``` + +- **Idempotent imperative jobs** — All imperative jobs run on a schedule (every 10 minutes by default) and must be idempotent. +- **Re-run `pattern init`** — After adding new local helm charts, re-run `pattern init` to wire them into the clustergroup values file. It is idempotent. + +## Common Tasks + +### Adding an Operator + +Add three things to the clustergroup values file: a namespace, a subscription, and (if the operator needs its own chart for configuration) an application. + +```yaml +clusterGroup: + namespaces: + my-operator: + operatorGroup: true + targetNamespaces: [] + + subscriptions: + my-operator: + name: my-operator + namespace: my-operator + channel: stable + + applications: + my-operator-config: + name: my-operator-config + namespace: my-operator + path: charts/my-operator-config +``` + +Not every operator needs a local chart. If the operator requires no additional configuration beyond installation, the namespace and subscription are sufficient. + +### Adding a Local Helm Chart + +Place the chart anywhere in the repository (convention: `charts/`). Run `pattern init` to auto-discover it, or manually add it to the clustergroup values: + +```yaml +clusterGroup: + namespaces: + my-app: + + applications: + my-app: + name: my-app + namespace: my-app + path: charts/my-app +``` + +### Adding a VP-Published Chart + +```yaml +clusterGroup: + applications: + vault: + name: vault + namespace: vault + chart: hashicorp-vault + chartVersion: 0.1.* +``` + +### Adding a Chart from an External Git Repository + +```yaml +clusterGroup: + applications: + external-app: + name: external-app + namespace: external-app + repoURL: https://github.com/org/repo.git + chartVersion: main + path: charts/the-chart +``` + +### Adding a Secret + +1. Define the secret in `values-secret.yaml.template`: + + ```yaml + version: "2.0" + + secrets: + - name: my-secret + vaultPrefixes: + - global + fields: + - name: api-key + onMissingValue: prompt + - name: password + onMissingValue: generate + vaultPolicy: validatedPatternDefaultPolicy + ``` + +2. Add `secretStore` defaults to the chart's `values.yaml`: + + ```yaml + secretStore: + name: vault-backend + kind: ClusterSecretStore + + mysecret: + key: secret/data/global/my-secret + refreshInterval: 2m0s + ``` + +3. Create an ExternalSecret template in the chart: + + ```yaml + apiVersion: external-secrets.io/v1 + kind: ExternalSecret + metadata: + name: my-secret + spec: + refreshInterval: {{ .Values.mysecret.refreshInterval }} + secretStoreRef: + name: {{ .Values.secretStore.name }} + kind: {{ .Values.secretStore.kind }} + target: + name: my-secret + template: + type: Opaque + data: + api-key: "{{ `{{ .api_key }}` }}" + password: "{{ `{{ .password }}` }}" + data: + - secretKey: api_key + remoteRef: + key: {{ .Values.mysecret.key }} + property: api-key + - secretKey: password + remoteRef: + key: {{ .Values.mysecret.key }} + property: password + ``` + +The Vault path is `secret/data//`. The `secretKey` values become the template variables in `target.template.data`. + +### Adding a Spoke Cluster + +1. Add ACM and `managedClusterGroups` to the hub clustergroup values: + + ```yaml + clusterGroup: + name: hub + + namespaces: + open-cluster-management: + + subscriptions: + acm: + name: advanced-cluster-management + namespace: open-cluster-management + channel: release-2.16 + + applications: + acm: + name: acm + namespace: open-cluster-management + chart: acm + chartVersion: 0.2.* + + managedClusterGroups: + region-one: + name: group-one + acmlabels: + - name: clusterGroup + value: group-one + ``` + +2. Create `values-group-one.yaml` with the spoke's namespaces, subscriptions, and applications. + +3. If using secrets on the spoke, include ESO components (without Vault) in the spoke values. + +### Adding Conditional Overrides + +Define a custom global variable and use `sharedValueFiles`: + +```yaml +# values-global.yaml +global: + device: gpu + +# values-.yaml +clusterGroup: + sharedValueFiles: + - /overrides/values-{{ $.Values.global.device }}.yaml +``` + +Create the override file (e.g., `/overrides/values-gpu.yaml`) with the conditional namespaces, subscriptions, and applications. diff --git a/.cursor/skills/pattern-author/reference.md b/.cursor/skills/pattern-author/reference.md new file mode 100644 index 00000000..1a24ddaf --- /dev/null +++ b/.cursor/skills/pattern-author/reference.md @@ -0,0 +1,951 @@ +# What are Validated Patterns + +[Validated Patterns](https://validatedpatterns.io/) are an advanced form of reference architectures that offer a streamlined approach to deploying complex business solutions. + +Validated Patterns are GitOps-based. A Validated Pattern is a Git repository containing the values files for the [clustergroup helm chart](https://github.com/validatedpatterns/clustergroup-chart/). + +Validated Patterns are installed via the [Validated Patterns Operator](https://github.com/validatedpatterns/patterns-operator) (available as a community operator in OpenShift's operator catalog). The operator creates and manages a subscription for OpenShift GitOps (ArgoCD) and creates the app-of-apps Application in ArgoCD — the clustergroup chart with values taken from the Pattern repository (as a multi-source ArgoCD Application). + +## Creating a New Pattern + +1. Create an empty directory (or create a new repository on GitHub/GitLab and clone it). The directory name becomes the Pattern name. +2. Run `podman run --pull=newer -v "$PWD:$PWD:z" -w "$PWD" quay.io/validatedpatterns/patternizer init` to create the new Pattern. Add `--with-secrets` to include the necessary components for the Validated Patterns secret framework: + `podman run --pull=newer -v "$PWD:$PWD:z" -w "$PWD" quay.io/validatedpatterns/patternizer init --with-secrets` + +[The patternizer source is available on GitHub.](https://github.com/validatedpatterns/patternizer) + +You can shorten the patternizer command by adding a shell function to your shell's startup: + +```bash +pattern() { + podman run --pull=newer \ + -v "$PWD:$PWD:z" \ + -w "$PWD" \ + quay.io/validatedpatterns/patternizer "$@" +} +``` + +Then run `pattern init` or `pattern init --with-secrets`. + +`pattern init` is idempotent. You can add secrets to a Pattern initialized without them by running `pattern init --with-secrets` later. You can also re-run `pattern init` as you add helm charts to the repository and it will wire them into the clustergroup values file. + +## Pattern File Structure + +Given a fresh Pattern directory with a user-defined helm chart: + +```bash +mkdir -p fresh-pattern/charts +cd fresh-pattern/charts +helm create user-defined-chart +cd .. +pattern init --with-secrets +``` + +The resulting structure: + +```text +fresh-pattern +├── ansible.cfg +├── charts +│ └── user-defined-chart +├── Makefile +├── Makefile-common +├── pattern.sh +├── values-global.yaml +├── values-prod.yaml +└── values-secret.yaml.template +``` + +### File Descriptions + +`ansible.cfg` contains defaults for the Ansible playbooks invoked via the Makefile (e.g., `make install`). These playbooks are defined in the [rhvp.cluster_utils collection](https://github.com/validatedpatterns/rhvp.cluster_utils). + +`charts/` is the recommended location for local helm charts. `pattern init` discovers charts anywhere in the repository, but `charts/` is the convention. + +`Makefile` includes `Makefile-common` and provides a place for Pattern-specific make targets or overrides. Most Patterns never need to override the common targets, but it's useful for custom tests, linting, or convenience functions. + +`Makefile-common` provides the core make targets for the Patterns framework. Documented more fully on the [VP blog](https://validatedpatterns.io/blog/2025-08-29-new-common-makefile-structure/). The primary targets are `make install`, `make uninstall`, and `make load-secrets`. These targets are run via the `./pattern.sh` wrapper script, which executes them inside the [VP Utility Container](https://github.com/validatedpatterns/utility-container). The utility container includes make, oc, helm, aws, and other CLIs, so the only local dependency is podman. A user can run `./pattern.sh make install` to install a Pattern (assuming they are logged into an OpenShift cluster). + +`pattern.sh` is the wrapper script that runs make targets inside the utility container. + +`values-secret.yaml.template` is a template showing how the Pattern's secrets should be formatted for the [secrets framework](#the-secrets-framework). This file should never contain real secrets. Copy it to your home directory (`cp values-secret.yaml.template ~/values-secret-fresh-pattern.yaml`) so secrets stay on your local machine, not in your Git repository. + +### values-global.yaml + +```yaml +global: + pattern: fresh-pattern + singleArgoCD: true + secretLoader: + disabled: false +main: + clusterGroupName: prod + multiSourceConfig: + enabled: true + clusterGroupChartVersion: 0.9.* +``` + +#### Field Reference + +`global.pattern` — The Pattern name, taken from the directory name by patternizer. Used as a label on ArgoCD Applications and as part of the ArgoCD namespace name. + +`global.singleArgoCD` — When `true` (the patternizer default), each cluster uses a single ArgoCD instance for both the app-of-apps and all child applications. When `false` (legacy behavior), each cluster runs two ArgoCD instances: one for the clustergroup chart (app-of-apps) and a separate one for the applications it defines. A hub/spoke Pattern with `singleArgoCD: false` would have two ArgoCD instances on the hub and two on the spoke. With `singleArgoCD: true`, there is one ArgoCD instance on the hub and one on the spoke. New Patterns should always use `true`. + +`global.secretLoader.disabled` — When `true`, skip loading secrets. Useful for Patterns that don't use the secrets framework. + +`main.clusterGroupName` — The name of the primary clustergroup. Determines which `values-.yaml` file defines the main cluster. If you change this value, the next `pattern init` run creates the corresponding values file (you would need to manually delete the old unused one). + +`main.multiSourceConfig.enabled` — Enable ArgoCD multi-source applications. Should always be `true` for modern Patterns. + +`main.multiSourceConfig.clusterGroupChartVersion` — SemVer constraint for the clustergroup chart version from the VP chart repository. + +There are also some global options that can be set in `values-global.yaml` but are not included by default: + +`global.options.syncPolicy` — Controls the ArgoCD sync policy for all applications. `"Automatic"` (the default) enables auto-sync with retry. `"Manual"` disables auto-sync. Per-application `syncPolicy` overrides this global default. + +`global.options.installPlanApproval` — Default `installPlanApproval` for all subscriptions. `"Automatic"` (the default) allows operators to auto-upgrade when new updates are available in their channel. Set to `"Manual"` to prevent auto-upgrades. Per-subscription values override this. + +`global.options.useCSV` — When `True` (the default), subscriptions include a `startingCSV` field if their `.csv` value is set. + +### values-prod.yaml + +`values-prod.yaml` is the values file that defines the main clustergroup of the Pattern. In hub/spoke Patterns (see [Spoke clusters](#spoke-clusters)) this would be the hub cluster. In single cluster Patterns this defines the solitary cluster. If there is one file to look at which defines the Pattern, it is this one. Its contents are explored in [The clustergroup values](#the-clustergroup-values) section. + +### The Pattern CR + +When you run `./pattern.sh make install` to deploy the Pattern, the values in `values-global.yaml` are passed to the [pattern-install chart](https://github.com/validatedpatterns/pattern-install-chart) by the [rhvp.cluster_utils.install](https://github.com/validatedpatterns/rhvp.cluster_utils/blob/main/playbooks/install.yml) playbook. This chart creates a subscription for the Validated Patterns Operator, a configmap with default operator configuration, and the Pattern CR: + +```yaml +apiVersion: gitops.hybrid-cloud-patterns.io/v1alpha1 +kind: Pattern +metadata: + name: fresh-pattern + namespace: openshift-operators +spec: + clusterGroupName: prod + gitSpec: + targetRepo: https://github.com/validatedpatterns/fresh-pattern.git + targetRevision: main + multiSourceConfig: + enabled: true + clusterGroupChartVersion: 0.9.* +``` + +Your branch (`main` in this example) must have an upstream remote set and be pushed to that remote. + +## Values File Hierarchy + +When the Patterns Operator creates the app-of-apps representing the Pattern, it renders the [clustergroup chart](https://github.com/validatedpatterns/clustergroup-chart/) and automatically includes certain values files (if they exist) to customize resources for a given clustergroup, OCP version, and platform. + +The following values files are automatically included for each application, in this order: + +1. `values-global.yaml` — included for all clustergroups +2. `values-.yaml` — values specific to the clustergroup +3. `values-.yaml` — values specific to the platform (e.g., `values-AWS.yaml`) +4. `values--.yaml` — platform and version specific (e.g., `values-AWS-4.21.yaml`) +5. `values--.yaml` — platform and clustergroup specific (e.g., `values-AWS-hub.yaml`) +6. `values--.yaml` — version and clustergroup specific (e.g., `values-4.21-hub.yaml`) +7. `values-.yaml` — values for a specifically named cluster (e.g., `values-test-cluster.yaml`) +8. Files from `global.extraValueFiles` — additional global value files +9. Per-clustergroup `sharedValueFiles` — from `clusterGroup.sharedValueFiles` +10. Per-application `extraValueFiles` — from the application's `extraValueFiles` field + +ArgoCD is configured with `ignoreMissingValueFiles: true`, so it silently skips any of these files that do not exist. Only create the files you actually need. + +In multi-source mode (the default), values files from the Pattern repository are prefixed with `$patternref/` to tell ArgoCD which source they come from. This is handled automatically by the clustergroup chart. + +Some of these values files are cross-clustergroup. For example, `values-AWS.yaml` would apply to both hub and spoke clustergroups running on AWS. + +### Shared Value Files + +The clustergroup provides a `sharedValueFiles` field for including additional overrides for all applications in that clustergroup: + +```yaml +clusterGroup: + sharedValueFiles: + - '/overrides/values-{{ $.Values.global.clusterPlatform }}.yaml' + - '/overrides/values-{{ $.Values.global.clusterPlatform }}-{{ $.Values.global.clusterVersion }}.yaml' + - '/overrides/values-{{ $.Values.global.clusterPlatform }}-{{ $.Values.clusterGroup.name }}.yaml' + - '/overrides/values-{{ $.Values.global.clusterVersion }}-{{ $.Values.clusterGroup.name }}.yaml' + - '/overrides/values-{{ $.Values.global.localClusterName }}.yaml' +``` + +These paths support Helm template interpolation. The Validated Patterns operator handles resolving the global variables. As with the auto-included files, any that do not exist are silently skipped. + +By convention, extra value files are placed in an `overrides/` directory in the Pattern repository, but any location works. + +### Custom Globals for Conditional Overrides + +You can define your own global variables in `values-global.yaml` and use them in `sharedValueFiles` to enable conditional configuration. Consider this example: + +```yaml +# values-global.yaml +global: + pattern: ai-quickstart-rag + device: cpu # one of 'cpu' (no GPU) or 'gpu' (NVIDIA GPU) +main: + clusterGroupName: prod + multiSourceConfig: + enabled: true + clusterGroupChartVersion: 0.9.* +``` + +This enables conditional value file inclusion: + +```yaml +clusterGroup: + sharedValueFiles: + - /overrides/values-{{ $.Values.global.device }}.yaml + - /overrides/values-{{ $.Values.global.device }}-{{ $.Values.global.clusterPlatform }}.yaml +``` + +```yaml +# /overrides/values-gpu.yaml +global: + models: + llama-3-2-3b-instruct: + enabled: true + +llm-service: + device: gpu + +clusterGroup: + namespaces: + openshift-nfd: + nvidia-gpu-operator: + + subscriptions: + nfd: + name: nfd + namespace: openshift-nfd + nvidia: + name: gpu-operator-certified + namespace: nvidia-gpu-operator + source: certified-operators + + applications: + nfd: + name: nfd + namespace: openshift-nfd + path: charts/nfd + nvidia-config: + name: nvidia-config + namespace: nvidia-gpu-operator + path: charts/nvidia-config +``` + +```yaml +# /overrides/values-gpu-AWS.yaml +clusterGroup: + imperative: + jobs: + - name: deploy-nvidia-gpu + playbook: rhvp.cluster_utils.create_machineset + verbosity: -vvv + extravars: + - max_machineset_count=1 + - machineset_replicas=1 + - ensure_two_machine_minimum=false + - machineset_name=nvidia-gpu + - machineset_labels= + - machineset_instance_type=g6.2xlarge + - 'machineset_taints=[{"effect":"NoSchedule","key":"nvidia.com/gpu","value":"true"}]' + - 'machineset_node_labels={"node-role.kubernetes.io/nvidia-gpu":""}' + - machineset_api_version=machine.openshift.io/v1beta1 + clusterRoleYaml: + - apiGroups: + - "*" + resources: + - machinesets + - persistentvolumeclaims + - datavolumes + - dataimportcrons + - datasources + verbs: + - "*" + - apiGroups: + - "*" + resources: + - "*" + verbs: + - get + - list + - watch +``` + +This example leverages the `global.device` value to install GPU-specific subscriptions and create an additional machineset when deployed to AWS with a GPU. + +## The Clustergroup Values + +A Pattern IS a clustergroup. The Git repository containing a Validated Pattern contains the values files for the [clustergroup helm chart](https://github.com/validatedpatterns/clustergroup-chart/). The clustergroup values file (`values-.yaml`) is the central definition of what a Pattern deploys — which namespaces to create, which operators to install, and which applications (helm charts) to run. + +Here is the `values-prod.yaml` generated by `pattern init --with-secrets` for our example Pattern. It defines a single clustergroup named `prod` with the namespaces, operator subscriptions, and applications needed for the secrets framework alongside the user's own chart: + +```yaml +clusterGroup: + name: prod + namespaces: + fresh-pattern: + vault: + external-secrets-operator: + operatorGroup: true + targetNamespaces: [] + external-secrets: + subscriptions: + eso: + name: openshift-external-secrets-operator + namespace: external-secrets-operator + channel: stable-v1 + applications: + openshift-external-secrets: + name: openshift-external-secrets + namespace: external-secrets + chart: openshift-external-secrets + chartVersion: 0.0.* + user-defined-chart: + name: user-defined-chart + namespace: fresh-pattern + path: charts/user-defined-chart + vault: + name: vault + namespace: vault + chart: hashicorp-vault + chartVersion: 0.1.* +``` + +These three sections — `namespaces`, `subscriptions`, and `applications` — are the core building blocks of every clustergroup. Each is detailed below. + +### Namespaces + +The namespace section accepts simple strings or more complex mappings: + +```yaml +clusterGroup: + namespaces: + fresh-pattern: + external-secrets-operator: + operatorGroup: true + targetNamespaces: [] +``` + +Setting `operatorGroup: true` creates an OperatorGroup with the same name as the namespace using the specified `targetNamespaces`. + +Labels and annotations can also be set: + +```yaml +clusterGroup: + namespaces: + rag-llm: + labels: + opendatahub.io/dashboard: "true" + modelmesh-enabled: "false" +``` + +Namespaces can also be defined as a list: + +```yaml +clusterGroup: + namespaces: + - fresh-pattern + - vault + - external-secrets-operator: + operatorGroup: true + targetNamespaces: [] + - external-secrets +``` + +The map form is recommended over the list form. When merging values files, lists are overridden entirely whereas maps are merged. + +### Subscriptions + +Subscriptions define operators that should be installed on the cluster. + +The only required field is the name of the subscription: + +```yaml +clusterGroup: + subscriptions: + servicemesh-console: + name: kiali-ossm +``` + +The clustergroup chart provides defaults for the other fields, making this equivalent to: + +```yaml +clusterGroup: + subscriptions: + servicemesh-console: + name: kiali-ossm + namespace: openshift-operators + source: redhat-operators + sourceNamespace: openshift-marketplace + channel: stable + installPlanApproval: Automatic +``` + +You can also specify `.csv` for the `startingCSV` of the subscription if needed. + +The `name` must be the operator's name in the `source` catalog. The default source is `redhat-operators`, but some operators are in `certified-operators`, `community-operators`, or `redhat-marketplace`. + +`namespace` is where the operator is installed. For operators like ESO, this is an OperatorGroup namespace. + +`source` and `sourceNamespace` only need updating in disconnected or custom install scenarios where the standard catalog sources are unavailable. + +`channel` is operator-specific. Some operators publish on multiple channels (e.g., `fast`, `stable-3.x`). + +Set `installPlanApproval` to `Manual` to prevent the operator from upgrading automatically when new updates are available in its channel. The default is `Automatic`. + +### Applications + +#### Local Helm Charts + +If `path` is provided (and `repoURL` and `chartVersion` are not), the helm chart is sourced from the Pattern repository: + +```yaml +clusterGroup: + applications: + user-defined-chart: + name: user-defined-chart + namespace: fresh-pattern + path: charts/user-defined-chart +``` + +#### VP-Published Helm Charts + +If neither `repoURL` nor `path` are provided, charts are sourced from the [Validated Patterns chart repository](https://charts.validatedpatterns.io/). Specify a version with `chartVersion`: + +```yaml +clusterGroup: + applications: + vault: + name: vault + namespace: vault + chart: hashicorp-vault + chartVersion: 0.1.* +``` + +#### Helm Charts from External Git Repositories + +Helm charts from external Git repositories can be referenced with `repoURL` (the publicly reachable Git URL), `path` (the path within the repository), and `chartVersion` (the Git revision to use): + +```yaml +clusterGroup: + applications: + maas-quickstart: + name: maas-quickstart + repoURL: https://github.com/dminnear-rh/maas-code-assistant.git + chartVersion: main + path: charts/maas-code-assistant +``` + +#### Additional Application Fields + +Several additional fields are available for any application type: + +`extraValueFiles` — Paths (relative to the Pattern repository root) to additional values files passed to the helm chart: + +```yaml +extraValueFiles: + - /overrides/maas-quickstart.yaml +``` + +`overrides` — Direct helm value overrides: + +```yaml +overrides: + - name: grafana.namespace + value: grafana +``` + +`ignoreDifferences` — Instructs ArgoCD to ignore certain differences when computing the sync diff: + +```yaml +ignoreDifferences: + - kind: Secret + name: grafana-proxy + namespace: grafana + jsonPointers: + - /data/session_secret +``` + +A full example combining these: + +```yaml +clusterGroup: + applications: + maas-quickstart: + name: maas-quickstart + repoURL: https://github.com/dminnear-rh/maas-code-assistant.git + chartVersion: main + path: charts/maas-code-assistant + extraValueFiles: + - /overrides/maas-quickstart.yaml + overrides: + - name: grafana.namespace + value: grafana + ignoreDifferences: + - kind: Secret + name: grafana-proxy + namespace: grafana + jsonPointers: + - /data/session_secret +``` + +## Using Helm Charts in VP + +This section covers what the VP framework provides to your charts, not helm chart authoring in general. + +### Where Charts Live + +The default location is `charts/` in the Pattern repository. `patternizer init` auto-discovers charts anywhere in the repository, so you can organize them however you like. + +Charts can also live in separate Git repositories and be referenced via `repoURL` in the application definition. VP-published charts are available from [charts.validatedpatterns.io](https://charts.validatedpatterns.io/) (referenced with `chart:` and `chartVersion:`). + +### How Values Flow into Charts + +Every ArgoCD Application created by the clustergroup chart receives the full merged tree of values files described in [Values File Hierarchy](#values-file-hierarchy). This means every chart sees: + +- `.Values.global.*` — global Pattern values +- `.Values.clusterGroup.*` — clustergroup configuration +- Chart-specific values from the chart's own `values.yaml` + +In addition to the values files, the clustergroup chart injects helm parameters for cluster-specific values that are known at deploy time: + +- `global.repoURL` — Pattern repository URL +- `global.targetRevision` — Git branch/commit/ref +- `global.namespace` — the ArgoCD app namespace +- `global.pattern` — Pattern name +- `global.clusterDomain` — cluster FQDN +- `global.localClusterName` — local cluster identifier +- `global.clusterVersion` — OpenShift version +- `global.clusterPlatform` — platform type (e.g., AWS, Azure, GCP) +- `global.hubClusterDomain` — hub cluster FQDN +- `global.localClusterDomain` — local cluster FQDN + +These are available in templates as `.Values.global.`. + +A chart's `values.yaml` should include default stubs for any `global.*` or `clusterGroup.*` values referenced in its templates. These defaults enable standalone `helm template` to work during development and are overridden at deploy time by the merged values tree. + +### Example: config-demo Chart + +The [config-demo chart](https://github.com/validatedpatterns/multicloud-gitops/tree/main/charts/all/config-demo) from multicloud-gitops is a minimal working example. Its `values.yaml`: + +```yaml +secretStore: + name: vault-backend + kind: ClusterSecretStore + +configdemosecret: + key: secret/data/global/config-demo + refreshInterval: 2m0s + +global: + hubClusterDomain: hub.example.com + localClusterDomain: region-one.example.com + +clusterGroup: + isHubCluster: true + +image: + repository: registry.access.redhat.com/ubi10/httpd-24 + tag: "10.0-1755779646" + pullPolicy: IfNotPresent +``` + +The `global` and `clusterGroup` stubs provide defaults for development. The `secretStore` and `configdemosecret` sections are chart-specific values used by the ExternalSecret template (see [Consuming Secrets in Charts](#consuming-secrets-in-charts)). + +The deployment template uses chart-specific values: + +```yaml +containers: +- name: apache + image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" + imagePullPolicy: {{ .Values.image.pullPolicy }} +``` + +The ConfigMap template uses global values injected by the framework, demonstrating how charts can reference cluster-specific information without hardcoding it: + +```yaml +data: + "index.html": |- +

+ Hub Cluster domain is '{{ .Values.global.hubClusterDomain }}'
+ Pod is running on Local Cluster Domain '{{ .Values.global.localClusterDomain }}'
+

+``` + +## The Secrets Framework + +Secrets in the VP framework are stored in Vault and consumed in charts via External Secrets Operator (ESO). The workflow is: define secrets in `values-secret.yaml.template`, load them into Vault with `./pattern.sh make install` or `./pattern.sh make load-secrets`, and consume them in charts using ExternalSecret CRDs. + +### Defining Secrets + +The `values-secret.yaml.template` file defines the secrets a Pattern needs. The install/load-secrets command looks for secrets in `~/values-secret-.yaml` and falls back to the template in the Pattern repository. This encourages users to copy the template to their home directory and keep secrets out of the Git repository. + +Example from [multicloud-gitops](https://github.com/validatedpatterns/multicloud-gitops/blob/main/values-secret.yaml.template): + +```yaml +version: "2.0" + +secrets: + - name: config-demo + vaultPrefixes: + - global + fields: + - name: secret + onMissingValue: generate + vaultPolicy: validatedPatternDefaultPolicy +``` + +#### Secret Field Reference + +Each secret entry supports these fields: + +- `name` — secret name (becomes the Vault path segment) +- `vaultPrefixes` — list of Vault path prefixes controlling which clustergroups can read the secret +- `fields[].name` — field name within the secret +- `fields[].value` — literal value +- `fields[].path` — path to a file containing the value +- `fields[].ini_file`, `ini_section`, `ini_key` — read a value from an INI file +- `fields[].onMissingValue` — set to `generate` to auto-generate the value +- `fields[].vaultPolicy` — policy name for generation (either `validatedPatternDefaultPolicy` or a custom policy) + +Example with various field types: + +```yaml +secrets: + # AWS credentials from INI file + - name: aws + fields: + - name: aws_access_key_id + ini_file: ~/.aws/credentials + ini_section: default + ini_key: aws_access_key_id + - name: aws_secret_access_key + ini_file: ~/.aws/credentials + ini_key: aws_secret_access_key + + # SSH keys from files + - name: publickey + fields: + - name: content + path: ~/.ssh/id_rsa.pub + - name: privatekey + fields: + - name: content + path: ~/.ssh/id_rsa + + # OpenShift pull secret from file + - name: openshiftPullSecret + fields: + - name: content + path: ~/.pullsecret.json +``` + +### Vault Prefixes and Access Control + +The `vaultPrefixes` field controls which clustergroups can access a secret. The Vault path convention is `secret/data//`. + +- `global` — readable by all clustergroups (hub and spoke) +- A clustergroup name (e.g., `hub`) — readable only by that clustergroup +- Multiple prefixes write the secret to multiple paths, making it accessible from multiple clustergroups + +### Secret Generation and Policies + +Secrets can be auto-generated if no value is provided by setting `onMissingValue: generate`. The generation is controlled by a vault policy: + +```yaml +version: "2.0" + +vaultPolicies: + basicPolicy: | + length=16 + rule "charset" { charset = "abcdefghijklmnopqrstuvwxyz" min-chars = 1 } + rule "charset" { charset = "ABCDEFGHIJKLMNOPQRSTUVWXYZ" min-chars = 1 } + rule "charset" { charset = "0123456789" min-chars = 1 } + +secrets: + - name: pgvector + fields: + - name: user + value: postgres + - name: password + onMissingValue: generate + vaultPolicy: basicPolicy + - name: dbname + value: rag_blueprint + - name: host + value: pgvector + - name: port + value: "5432" +``` + +### Consuming Secrets in Charts + +Secrets stored in Vault are consumed in charts using ESO ExternalSecret CRDs. The VP `openshift-external-secrets` chart sets up a `ClusterSecretStore` named `vault-backend` that points to the Vault instance. + +A chart that needs secrets should include `secretStore` defaults in its `values.yaml`: + +```yaml +secretStore: + name: vault-backend + kind: ClusterSecretStore + +configdemosecret: + key: secret/data/global/config-demo + refreshInterval: 2m0s +``` + +The ExternalSecret template maps secrets from Vault into Kubernetes Secrets: + +```yaml +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: config-demo-secret + namespace: config-demo +spec: + refreshInterval: {{ .Values.configdemosecret.refreshInterval }} + secretStoreRef: + name: {{ .Values.secretStore.name }} + kind: {{ .Values.secretStore.kind }} + target: + name: config-demo-secret + template: + type: Opaque + data: + secret: "{{ `{{ .configdemo_secret }}` }}" + data: + - secretKey: configdemo_secret + remoteRef: + key: {{ .Values.configdemosecret.key }} + property: secret +``` + +#### How the Mapping Works + +Given a secret defined in `values-secret.yaml.template` as: + +```yaml +secrets: + - name: config-demo + vaultPrefixes: + - global + fields: + - name: secret +``` + +The Vault path is: `secret/data/global/config-demo` (constructed as `secret/data//`). + +In the ExternalSecret: + +- `remoteRef.key` is set to this Vault path (`secret/data/global/config-demo`) +- `remoteRef.property` is the field name (`secret`) +- `data[].secretKey` (`configdemo_secret`) is the local key used to reference the fetched value in the `target.template` + +#### Backtick Escaping for ESO Templates + +The `target.template.data` section uses ESO's own template syntax (`{{ .fieldname }}`) to map fetched values into the Kubernetes Secret. Since the ExternalSecret is itself rendered by Helm, the ESO template expressions must be escaped to prevent Helm from interpreting them. The pattern is: + +```text +"{{ `{{ .configdemo_secret }}` }}" +``` + +The backticks create a Go raw string literal that Helm passes through unchanged. ESO then processes `{{ .configdemo_secret }}` at runtime. + +### Hub vs. Spoke Secret Infrastructure + +The Vault/ESO components should only be defined on the hub/main cluster: + +```yaml +# Hub cluster needs both Vault and ESO +clusterGroup: + namespaces: + vault: + external-secrets-operator: + operatorGroup: true + targetNamespaces: [] + external-secrets: + subscriptions: + eso: + name: openshift-external-secrets-operator + namespace: external-secrets-operator + channel: stable-v1 + applications: + openshift-external-secrets: + name: openshift-external-secrets + namespace: external-secrets + chart: openshift-external-secrets + chartVersion: 0.0.* + vault: + name: vault + namespace: vault + chart: hashicorp-vault + chartVersion: 0.1.* +``` + +Spoke clusters only need ESO — no Vault: + +```yaml +# Spoke cluster needs ESO only +clusterGroup: + namespaces: + external-secrets-operator: + operatorGroup: true + targetNamespaces: [] + external-secrets: + subscriptions: + eso: + name: openshift-external-secrets-operator + namespace: external-secrets-operator + channel: stable-v1 + applications: + openshift-external-secrets: + name: openshift-external-secrets + namespace: external-secrets + chart: openshift-external-secrets + chartVersion: 0.0.* +``` + +The VP `openshift-external-secrets` chart automatically configures spoke clusters to use the Vault instance on the hub cluster as the external secret store. + +For more information, see [Secrets management in the Validated Patterns framework](https://validatedpatterns.io/learn/secrets-management-in-the-validated-patterns-framework/). + +## Spoke Clusters + +Hub/spoke cluster support uses ACM (Advanced Cluster Management). Include the ACM components in your hub clustergroup: + +```yaml +clusterGroup: + name: hub + + namespaces: + open-cluster-management: + + subscriptions: + acm: + name: advanced-cluster-management + namespace: open-cluster-management + channel: release-2.16 + + applications: + acm: + name: acm + namespace: open-cluster-management + chart: acm + chartVersion: 0.2.* + + managedClusterGroups: + exampleRegion: + name: group-one + acmlabels: + - name: clusterGroup + value: group-one +``` + +This installs ACM on the hub cluster via the Validated Patterns ACM chart. The `managedClusterGroups` defines the mapping the framework uses to determine which clusters imported into ACM correspond to which clustergroup values files. + +To create a spoke clustergroup, create `values-group-one.yaml` with its own namespaces, subscriptions, and applications. For secrets, include the ESO components (without Vault) as described in [Hub vs. Spoke Secret Infrastructure](#hub-vs-spoke-secret-infrastructure). + +```yaml +clusterGroup: + name: group-one + + namespaces: + config-demo: + hello-world: + external-secrets-operator: + operatorGroup: true + targetNamespaces: [] + external-secrets: + + subscriptions: + eso: + name: openshift-external-secrets-operator + namespace: external-secrets-operator + channel: stable-v1 + + applications: + openshift-external-secrets: + name: openshift-external-secrets + namespace: external-secrets + chart: openshift-external-secrets + chartVersion: 0.0.* + config-demo: + name: config-demo + namespace: config-demo + path: charts/all/config-demo + hello-world: + name: hello-world + namespace: hello-world + path: charts/all/hello-world +``` + +Any cluster imported into ACM with the label `clusterGroup: group-one` will have the clustergroup chart applied using `values-global.yaml` and `values-group-one.yaml` (plus all the other automatically included platform/version values files). + +In short, spoke and hub clusters are both just clustergroups. The difference is that the hub cluster includes ACM and the Vault components. While the Patterns Operator installs the Pattern via the clustergroup chart in ArgoCD on the hub cluster, the ACM chart pushes policies to spoke clusters for installing OpenShift GitOps (ArgoCD) and the clustergroup chart for that spoke cluster. + +## The Imperative Framework + +Sometimes tasks don't fit neatly into a declarative framework. The imperative framework runs Ansible playbooks on a schedule against the cluster. + +```yaml +clusterGroup: + imperative: + jobs: + - name: trilio-enable-cr + playbook: ansible/playbooks/imperative-enable-cr.yaml + timeout: 900 + + - name: trilio-cr-backup + playbook: ansible/playbooks/imperative-cr-backup.yaml + timeout: 1200 + + - name: trilio-backup + playbook: ansible/playbooks/imperative-backup.yaml + timeout: 1200 + + - name: trilio-restore-standard + playbook: ansible/playbooks/imperative-restore-standard.yaml + timeout: 1800 + + - name: trilio-e2e-status + playbook: ansible/playbooks/imperative-e2e-status.yaml + timeout: 120 +``` + +Imperative jobs typically reference playbooks stored in the `ansible/` directory of the Pattern repository, or playbooks from the `rhvp.cluster_utils` Ansible collection. Jobs are defined as a list since they run in order — if one fails, the imperative job fails and remaining jobs are aborted. Jobs must be idempotent since they run on a schedule (every 10 minutes by default). + +The full set of imperative framework defaults: + +```yaml +imperative: + jobs: [] + image: quay.io/validatedpatterns/imperative-container:v1 + ansibleDevMode: + enabled: false + requirementsFile: "requirements.yml" + requirementsContent: "" + ansibleCfgFile: "ansible.cfg" + ansibleCfgContent: "" + namespace: "imperative" + valuesConfigMap: "helm-values-configmap" + cronJobName: "imperative-cronjob" + jobName: "imperative-job" + imagePullPolicy: Always + activeDeadlineSeconds: 3600 + schedule: "*/10 * * * *" + insecureUnsealVaultInsideClusterSchedule: "*/5 * * * *" + verbosity: "" + extraPlaybookArgs: [] + serviceAccountCreate: true + serviceAccountName: imperative-sa + clusterRoleName: imperative-cluster-role + clusterRoleYaml: "" + roleName: imperative-role + roleYaml: "" + adminServiceAccountCreate: true + adminServiceAccountName: imperative-admin-sa + adminClusterRoleName: imperative-admin-cluster-role + vaultNamespace: "vault" +``` diff --git a/.gitignore b/.gitignore index 9724d136..e729fc16 100644 --- a/.gitignore +++ b/.gitignore @@ -8,3 +8,6 @@ vault.init super-linter.log common/pattern-vault.init common/scripts/pattern-util.sh +tests/ci/.results +tests/ci/__pycache__ +tests/ci/.pytest_cache diff --git a/Makefile-common b/Makefile-common index 537ca883..cf9b7bfe 100644 --- a/Makefile-common +++ b/Makefile-common @@ -1,6 +1,6 @@ MAKEFLAGS += --no-print-directory -ANSIBLE_STDOUT_CALLBACK ?= null # null silences all ansible output. Override this with default, minimal, oneline, etc. when debugging. -ANSIBLE_RUN := ANSIBLE_STDOUT_CALLBACK=$(ANSIBLE_STDOUT_CALLBACK) ansible-playbook $(EXTRA_PLAYBOOK_OPTS) +ANSIBLE_STDOUT_CALLBACK ?= rhvp.cluster_utils.readable +ANSIBLE_RUN ?= ANSIBLE_STDOUT_CALLBACK=$(ANSIBLE_STDOUT_CALLBACK) ansible-playbook $(EXTRA_PLAYBOOK_OPTS) DOCS_URL := https://validatedpatterns.io/blog/2025-08-29-new-common-makefile-structure/ .PHONY: help @@ -20,9 +20,9 @@ operator-deploy operator-upgrade: ## Installs/updates the pattern on a cluster ( .PHONY: install install: pattern-install ## Installs the pattern onto a cluster (Loads secrets as well if configured) -.PHONY: uninstall ## Prints a notice that patterns cannot currently be uninstalled -uninstall: - @echo "Uninstall is not possible at the moment so this target is empty. We are working to implement it as well as we can." +.PHONY: uninstall +uninstall: ## (EXPERIMENTAL) See https://validatedpatterns.io/blog/2026-02-16-pattern-uninstall/. + @$(ANSIBLE_RUN) rhvp.cluster_utils.uninstall .PHONY: pattern-install pattern-install: @@ -32,6 +32,11 @@ pattern-install: load-secrets: ## Loads secrets onto the cluster (unless explicitly disabled in values-global.yaml) @$(ANSIBLE_RUN) rhvp.cluster_utils.load_secrets +##@ Debug Tasks +.PHONY: display-secrets-info +display-secrets-info: ## Display your secret material on terminal or show secret loading error. Use with caution! + @$(ANSIBLE_RUN) rhvp.cluster_utils.display_secrets_info + ##@ Validation Tasks .PHONY: validate-prereq validate-prereq: ## verify pre-requisites @@ -52,3 +57,8 @@ validate-schema: ## validates values files against schema in common/clustergroup .PHONY: argo-healthcheck argo-healthcheck: ## Checks if all argo applications are synced @$(ANSIBLE_RUN) rhvp.cluster_utils.argo_healthcheck + +##@ Testing (CI) Tasks +.PHONY: run-ci-tests +run-ci-tests: ## To run ci-tests, set any needed env vars + @$(ANSIBLE_RUN) rhvp.cluster_utils.run_ci_tests diff --git a/ansible.cfg b/ansible.cfg index 516f8b84..528a8cb6 100644 --- a/ansible.cfg +++ b/ansible.cfg @@ -1,6 +1,15 @@ [defaults] localhost_warning=False retry_files_enabled=False -library=~/.ansible/plugins/modules:./ansible/plugins/modules:./common/ansible/plugins/modules:/usr/share/ansible/plugins/modules -roles_path=~/.ansible/roles:./ansible/roles:./common/ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles -filter_plugins=~/.ansible/plugins/filter:./ansible/plugins/filter:./common/ansible/plugins/filter:/usr/share/ansible/plugins/filter +# Retry files disabled to avoid cluttering CI/CD environments +interpreter_python=auto_silent +timeout=30 +library=~/.ansible/plugins/modules:./ansible/plugins/modules:/usr/share/ansible/plugins/modules +roles_path=~/.ansible/roles:./ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles +filter_plugins=~/.ansible/plugins/filter:./ansible/plugins/filter:/usr/share/ansible/plugins/filter +# use the collections from the util. container, +# change below if you want to test local collections +collections_path=/usr/share/ansible/collections + +[inventory] +inventory_unparsed_warning=False diff --git a/ansible/playbooks/create-gpu-machineset-azure.yaml b/ansible/playbooks/create-gpu-machineset-azure.yaml index 5a96896d..318ff9c2 100644 --- a/ansible/playbooks/create-gpu-machineset-azure.yaml +++ b/ansible/playbooks/create-gpu-machineset-azure.yaml @@ -57,7 +57,7 @@ src: templates/gpu-machineset-azure.j2 dest: /tmp/gpu-machineset-azure.yaml vars: - ms_name: "nvidia-worker-{{ azure_location | replace(' ', '') }}{{ gpu_zone }}" + ms_name: "nvidia-gpu-worker-{{ azure_location | replace(' ', '') }}{{ gpu_zone }}" - name: Apply the GPU MachineSet kubernetes.core.k8s: diff --git a/ansible/playbooks/create-gpu-machineset.yaml b/ansible/playbooks/create-gpu-machineset.yaml index 62c8d437..8fb2ea7d 100644 --- a/ansible/playbooks/create-gpu-machineset.yaml +++ b/ansible/playbooks/create-gpu-machineset.yaml @@ -39,19 +39,6 @@ ansible.builtin.set_fact: cloudRegion: "{{ infraInfo.resources[0].status.platformStatus[cloudProvider].region }}" - - name: "[create-gpu-machine-set] Search for MachineSets" - kubernetes.core.k8s_info: - api: machine.openshift.io/v1beta1 - kind: MachineSet - namespace: openshift-machine-api - register: machineset - - - name: "[create-gpu-machine-set] Get the availability zone from the first worker MachineSets" - ansible.builtin.set_fact: - cloudAvailabilityZone: "{{ item.spec.template.spec.providerSpec.value.placement.availabilityZone }}" - with_items: - - "{{ machineset.resources[0] }}" - - name: "[create-gpu-machine-set] Search for worker MachineSets" kubernetes.core.k8s_info: api: machine.openshift.io/v1beta1 @@ -69,6 +56,7 @@ securityGroups: "{{ machines.resources[0].spec.providerSpec.value.securityGroups }}" subnets: "{{ machines.resources[0].spec.providerSpec.value.subnet }}" tags: "{{ machines.resources[0].spec.providerSpec.value.tags }}" + cloudAvailabilityZone: "{{ machines.resources[0].spec.providerSpec.value.placement.availabilityZone }}" - name: "[create-gpu-machine-set] Generate machineset" ansible.builtin.template: diff --git a/charts/all/rag-llm/charts/azure-sql/templates/external-secret.yaml b/charts/all/rag-llm/charts/azure-sql/templates/external-secret.yaml index da4d6d56..5c1b0b42 100644 --- a/charts/all/rag-llm/charts/azure-sql/templates/external-secret.yaml +++ b/charts/all/rag-llm/charts/azure-sql/templates/external-secret.yaml @@ -1,10 +1,10 @@ {{- if eq .Values.global.db.type "AZURESQL" }} -apiVersion: "external-secrets.io/v1beta1" +apiVersion: "external-secrets.io/v1" kind: ExternalSecret metadata: name: azuresql-external-secret spec: - refreshInterval: 15s + refreshInterval: 2m0s secretStoreRef: name: {{ .Values.secretStore.name }} kind: {{ .Values.secretStore.kind }} diff --git a/charts/all/rag-llm/charts/mssql/templates/external-secret.yaml b/charts/all/rag-llm/charts/mssql/templates/external-secret.yaml index d67f4394..4fc12072 100644 --- a/charts/all/rag-llm/charts/mssql/templates/external-secret.yaml +++ b/charts/all/rag-llm/charts/mssql/templates/external-secret.yaml @@ -1,10 +1,10 @@ {{- if eq .Values.global.db.type "MSSQL" }} -apiVersion: "external-secrets.io/v1beta1" +apiVersion: "external-secrets.io/v1" kind: ExternalSecret metadata: name: mssql-external-secret spec: - refreshInterval: 15s + refreshInterval: 2m0s secretStoreRef: name: {{ .Values.secretStore.name }} kind: {{ .Values.secretStore.kind }} diff --git a/charts/all/rag-llm/charts/pgvector/templates/secret.yaml b/charts/all/rag-llm/charts/pgvector/templates/secret.yaml index dbfeb3a7..d1300cc4 100644 --- a/charts/all/rag-llm/charts/pgvector/templates/secret.yaml +++ b/charts/all/rag-llm/charts/pgvector/templates/secret.yaml @@ -1,10 +1,10 @@ {{- if eq .Values.global.db.type "PGVECTOR" }} -apiVersion: "external-secrets.io/v1beta1" +apiVersion: "external-secrets.io/v1" kind: ExternalSecret metadata: name: pgvector-external-secret spec: - refreshInterval: 15s + refreshInterval: 2m0s secretStoreRef: name: {{ .Values.secretStore.name }} kind: {{ .Values.secretStore.kind }} diff --git a/charts/all/rag-llm/templates/edb-pull-secret.yaml b/charts/all/rag-llm/templates/edb-pull-secret.yaml index aa96b574..77238a08 100644 --- a/charts/all/rag-llm/templates/edb-pull-secret.yaml +++ b/charts/all/rag-llm/templates/edb-pull-secret.yaml @@ -1,11 +1,11 @@ {{- if eq .Values.global.db.type "EDB" }} -apiVersion: external-secrets.io/v1beta1 +apiVersion: external-secrets.io/v1 kind: ExternalSecret metadata: name: edb-operator-pull-secret namespace: openshift-operators spec: - refreshInterval: 15s + refreshInterval: 2m0s secretStoreRef: name: {{ .Values.secretStore.name }} kind: {{ .Values.secretStore.kind }} diff --git a/charts/all/rag-llm/templates/external-secret.yaml b/charts/all/rag-llm/templates/external-secret.yaml index 92432720..11ba8b79 100644 --- a/charts/all/rag-llm/templates/external-secret.yaml +++ b/charts/all/rag-llm/templates/external-secret.yaml @@ -1,13 +1,13 @@ --- -apiVersion: "external-secrets.io/v1beta1" +apiVersion: "external-secrets.io/v1" kind: ExternalSecret metadata: name: huggingface-secret spec: - refreshInterval: 15s + refreshInterval: 2m0s secretStoreRef: name: {{ .Values.secretStore.name }} kind: {{ .Values.secretStore.kind }} dataFrom: - extract: - key: {{ .Values.hfmodel.key }} \ No newline at end of file + key: {{ .Values.hfmodel.key }} diff --git a/charts/all/rhods/templates/dsc.yaml b/charts/all/rhods/templates/dsc.yaml index c7ea8ade..e032ce8a 100644 --- a/charts/all/rhods/templates/dsc.yaml +++ b/charts/all/rhods/templates/dsc.yaml @@ -10,8 +10,6 @@ spec: managementState: Managed workbenches: managementState: Managed - datasciencepipelines: - managementState: Managed kueue: managementState: Removed ray: diff --git a/overrides/ci.yaml b/overrides/ci.yaml deleted file mode 100644 index c447c4ad..00000000 --- a/overrides/ci.yaml +++ /dev/null @@ -1,13 +0,0 @@ -global: - model: - vllm: ibm-granite/granite-4.0-h-350m - -vllmServingRuntime: - args: - - /cache/models - - command: [] - - image: - repository: vllm/vllm-openai - tag: v0.21.0 diff --git a/pattern-metadata.yaml b/pattern-metadata.yaml index c315406a..22669575 100644 --- a/pattern-metadata.yaml +++ b/pattern-metadata.yaml @@ -12,7 +12,7 @@ docs_url: https://validatedpatterns.io/patterns/rag-llm-gitops/ ci_url: https://validatedpatterns.io/ci/?pattern=ragllm # can be sandbox, tested or maintained tier: tested -owners: day0hero, sauagarwa +owners: dminnear-rh requirements: hub: # Main cluster compute: @@ -20,11 +20,17 @@ requirements: aws: replicas: 3 type: m5.2xlarge + azure: + replicas: 3 + type: Standard_D8s_v4 controlPlane: platform: aws: - replicas: 1 - type: m5.2xlarge + replicas: 3 + type: m5.xlarge + azure: + replicas: 3 + type: Standard_D4s_v4 # Loosely defined extra features like hypershift support, non-openshift # kubernetes support, spoke support diff --git a/pattern.sh b/pattern.sh index d6daa15e..7d8937c5 100755 --- a/pattern.sh +++ b/pattern.sh @@ -1,19 +1,73 @@ #!/bin/bash +set -euo pipefail function is_available { - command -v $1 >/dev/null 2>&1 || { echo >&2 "$1 is required but it's not installed. Aborting."; exit 1; } + command -v "$1" >/dev/null 2>&1 || { echo >&2 "$1 is required but it's not installed. Aborting."; exit 1; } } function version { - echo "$@" | awk -F. '{ printf("%d%03d%03d%03d\n", $1,$2,$3,$4); }' + echo "$1" | awk -F. '{ printf("%d%03d%03d%03d\n", $1,$2,$3,$4); }' } -if [ -z "$PATTERN_UTILITY_CONTAINER" ]; then +# We need this check mostly for CI testing, we do not want to run container in container +function is_container() { + [ -n "${KUBERNETES_SERVICE_HOST:-}" ] && return 0 + [ -f /.dockerenv ] && return 0 + [ -f /run/.containerenv ] && return 0 + return 1 +} + +function verify_image() { + local image="$1" + + case "${image}" in + quay.io/validatedpatterns/*|quay.io/hybridcloudpatterns/*) + ;; + *) + echo "Skipping image verification for third-party registry" + return 0 + ;; + esac + + if ! command -v cosign >/dev/null 2>&1; then + echo "WARNING: cosign is not installed, cannot verify image signature" + echo "Install cosign to enable image verification: https://docs.sigstore.dev/cosign/system_config/installation/" + return 0 + fi + + echo "Verifying image signature for ${image}..." + local output rc + local oidc_issuer="${VP_COSIGN_OIDC_ISSUER:-https://token.actions.githubusercontent.com}" + local cert_identity="${VP_COSIGN_CERT_IDENTITY:-https://github.com/validatedpatterns/utility-container/.*}" + output=$(cosign verify \ + --certificate-oidc-issuer "${oidc_issuer}" \ + --certificate-identity-regexp "${cert_identity}" \ + "${image}" 2>&1) && rc=$? || rc=$? + + if [ "${rc}" -eq 0 ]; then + echo "Image signature verified successfully" + elif [ "${rc}" -ge 10 ] && [ "${rc}" -le 13 ]; then + echo "ERROR: Image signature verification failed for ${image} (exit code ${rc})" + echo "${output}" + echo "Set VP_VERIFY_IMAGE=false to skip this check" + exit 1 + else + echo "WARNING: Could not verify image signature for ${image} (likely a network issue)" + echo "Set VP_VERIFY_IMAGE=false to skip this check" + fi +} + +if is_container; then + echo "Already running in a container" + exec "$@" +fi + +if [ -z "${PATTERN_UTILITY_CONTAINER:-}" ]; then PATTERN_UTILITY_CONTAINER="quay.io/validatedpatterns/utility-container" fi # If PATTERN_DISCONNECTED_HOME is set it will be used to populate both PATTERN_UTILITY_CONTAINER # and PATTERN_INSTALL_CHART automatically -if [ -n "${PATTERN_DISCONNECTED_HOME}" ]; then +if [ -n "${PATTERN_DISCONNECTED_HOME:-}" ]; then PATTERN_UTILITY_CONTAINER="${PATTERN_DISCONNECTED_HOME}/utility-container" PATTERN_INSTALL_CHART="oci://${PATTERN_DISCONNECTED_HOME}/pattern-install" echo "PATTERN_DISCONNECTED_HOME is set to ${PATTERN_DISCONNECTED_HOME}" @@ -23,10 +77,10 @@ if [ -n "${PATTERN_DISCONNECTED_HOME}" ]; then fi readonly commands=(podman) -for cmd in ${commands[@]}; do is_available "$cmd"; done +for cmd in "${commands[@]}"; do is_available "$cmd"; done UNSUPPORTED_PODMAN_VERSIONS="1.6 1.5" -PODMAN_VERSION_STR=$(podman --version) +PODMAN_VERSION_STR=$(podman --version) || { echo "Failed to get podman version"; exit 1; } for i in ${UNSUPPORTED_PODMAN_VERSIONS}; do # We add a space if echo "${PODMAN_VERSION_STR}" | grep -q -E "\b${i}"; then @@ -41,19 +95,20 @@ done PODMAN_VERSION=$(echo "${PODMAN_VERSION_STR}" | awk '{ print $NF }') # podman < 4.3.0 do not support keep-id:uid=... -if [ $(version "${PODMAN_VERSION}") -lt $(version "4.3.0") ]; then - PODMAN_ARGS="-v ${HOME}:/root" +PODMAN_ARGS=() +if [ "$(version "${PODMAN_VERSION}")" -lt "$(version "4.3.0")" ]; then + PODMAN_ARGS=(-v "${HOME}:/root") else # We do not rely on bash's $UID and $GID because on MacOSX $GID is not set MYNAME=$(id -n -u) MYUID=$(id -u) MYGID=$(id -g) - PODMAN_ARGS="--passwd-entry ${MYNAME}:x:${MYUID}:${MYGID}::/pattern-home:/bin/bash --user ${MYUID}:${MYGID} --userns keep-id:uid=${MYUID},gid=${MYGID}" - + PODMAN_ARGS=(--passwd-entry "${MYNAME}:x:${MYUID}:${MYGID}::/pattern-home:/bin/bash" --user "${MYUID}:${MYGID}" --userns "keep-id:uid=${MYUID},gid=${MYGID}") fi -if [ -n "$KUBECONFIG" ]; then - if [[ ! "${KUBECONFIG}" =~ ^$HOME* ]]; then +if [ -n "${KUBECONFIG:-}" ]; then + # Check if KUBECONFIG path starts with HOME directory + if [[ ! "${KUBECONFIG}" =~ ^"${HOME}" ]]; then echo "${KUBECONFIG} is pointing outside of the HOME folder, this will make it unavailable from the container." echo "Please move it somewhere inside your $HOME folder, as that is what gets bind-mounted inside the container" exit 1 @@ -62,20 +117,30 @@ fi # Detect if we use podman machine. If we do not then we bind mount local host ssl folders # if we are using podman machine then we do not bind mount anything (for now!) -REMOTE_PODMAN=$(podman system connection list | tail -n +2 | wc -l) -if [ $REMOTE_PODMAN -eq 0 ]; then # If we are not using podman machine we check the hosts folders +REMOTE_PODMAN=$(podman system connection list | tail -n +2 | wc -l) || REMOTE_PODMAN=0 +PKI_HOST_MOUNT_ARGS=() +if [ "${REMOTE_PODMAN}" -eq 0 ]; then # If we are not using podman machine we check the hosts folders # We check /etc/pki/tls because on ubuntu /etc/pki/fwupd sometimes # exists but not /etc/pki/tls and we do not want to bind mount in such a case # as it would find no certificates at all. if [ -d /etc/pki/tls ]; then - PKI_HOST_MOUNT_ARGS="-v /etc/pki:/etc/pki:ro" + PKI_HOST_MOUNT_ARGS=(-v /etc/pki:/etc/pki:ro) elif [ -d /etc/ssl ]; then - PKI_HOST_MOUNT_ARGS="-v /etc/ssl:/etc/ssl:ro" + PKI_HOST_MOUNT_ARGS=(-v /etc/ssl:/etc/ssl:ro) else - PKI_HOST_MOUNT_ARGS="-v /usr/share/ca-certificates:/usr/share/ca-certificates:ro" + PKI_HOST_MOUNT_ARGS=(-v /usr/share/ca-certificates:/usr/share/ca-certificates:ro) fi -else - PKI_HOST_MOUNT_ARGS="" +fi + +# Parse EXTRA_ARGS into an array if set +EXTRA_ARGS_ARRAY=() +if [ -n "${EXTRA_ARGS:-}" ]; then + # shellcheck disable=SC2206 + EXTRA_ARGS_ARRAY=(${EXTRA_ARGS}) +fi + +if [ "${VP_VERIFY_IMAGE:-true}" != "false" ]; then + verify_image "$PATTERN_UTILITY_CONTAINER" fi # Copy Kubeconfig from current environment. The utilities will pick up ~/.kube/config if set so it's not mandatory @@ -101,17 +166,19 @@ podman run -it --rm --pull=newer \ -e PATTERN_NAME \ -e TARGET_BRANCH \ -e TARGET_CLUSTERGROUP \ + -e TARGET_VARIANT \ -e TARGET_ORIGIN \ -e TOKEN_NAMESPACE \ -e TOKEN_SECRET \ -e UUID_FILE \ -e VALUES_SECRET \ - ${PKI_HOST_MOUNT_ARGS} \ + -e 'VP_*' \ + ${PKI_HOST_MOUNT_ARGS[@]+"${PKI_HOST_MOUNT_ARGS[@]}"} \ -v "$(pwd -P)":"$(pwd -P)" \ -v "${HOME}":"${HOME}" \ -v "${HOME}":/pattern-home \ - ${PODMAN_ARGS} \ - ${EXTRA_ARGS} \ + "${PODMAN_ARGS[@]}" \ + ${EXTRA_ARGS_ARRAY[@]+"${EXTRA_ARGS_ARRAY[@]}"} \ -w "$(pwd -P)" \ "$PATTERN_UTILITY_CONTAINER" \ - $@ + "$@" diff --git a/tests/all-config-demo.expected.diff b/tests/all-config-demo.expected.diff deleted file mode 100644 index 28bdbbfc..00000000 --- a/tests/all-config-demo.expected.diff +++ /dev/null @@ -1,13 +0,0 @@ ---- tests/all-config-demo-naked.expected.yaml -+++ tests/all-config-demo-normal.expected.yaml -@@ -23,8 +23,8 @@ - - -

-- Hub Cluster domain is 'hub.example.com'
-- Pod is running on Local Cluster Domain 'region-one.example.com'
-+ Hub Cluster domain is 'apps.hub.example.com'
-+ Pod is running on Local Cluster Domain 'apps.region.example.com'
-

-

- The secret is secret diff --git a/tests/ci/README.md b/tests/ci/README.md new file mode 100644 index 00000000..c62c7fc0 --- /dev/null +++ b/tests/ci/README.md @@ -0,0 +1,42 @@ +# CI Tests + +The requirements.txt file is just a placeholder to show the installed packages in the utility container: +[utility-container requirements.txt](https://github.com/validatedpatterns/utility-container/blob/main/requirements.txt) + +## The ci will run pytest based on file pattern + +pytest -lv test\_\.py --junit-xml .results/test\_\.xml + +## To run upstream tests locally + +Set the env variables pointing to the clusters needed to run the tests on:\ +`VP_HUBCONFIG` (`VP_SPOKECONFIG` if applicable) pointing to the kubconfig of hub (and spoke) clusters\ +(all VP\_\* env var will be available inside the container)\ +`TARGET_CLUSTERGROUP` if its different from the default (values-global.yaml main.clusterGroupName)\ +Test logs and junit-xml will be saved in ci/.results/\ +Run from root repository: + +```bash +./pattern.sh make run-ci-tests +``` + +## Writing additional tests + +The openshift_dyn_client fixture will return a DynamicClient which can be used inside test functions.\ +It requires only an env variable param to use it as a kubeconfig. + +```python +@pytest.mark.parametrize( + "openshift_dyn_client", + ["VP_HUBCONFIG"], + indirect=True, +) +``` + +If your tests requires additional python packages, you might want to either run them fully locally in a venv or similar.\ +Or if you want to use the pattern framework (make/Ansible wrappers) you need to use your own util container + +```bash +PATTERN_UTILITY_CONTAINER="your_utility container" ./pattern.sh make run-ci-tests + +``` diff --git a/tests/ci/__init__.py b/tests/ci/__init__.py new file mode 100644 index 00000000..3dc1f76b --- /dev/null +++ b/tests/ci/__init__.py @@ -0,0 +1 @@ +__version__ = "0.1.0" diff --git a/tests/ci/conftest.py b/tests/ci/conftest.py new file mode 100644 index 00000000..12cc3c06 --- /dev/null +++ b/tests/ci/conftest.py @@ -0,0 +1 @@ +from validatedpatterns_tests.interop.conftest_openshift import * # noqa: F401,F403 diff --git a/tests/ci/pytest.ini b/tests/ci/pytest.ini new file mode 100644 index 00000000..6408ef04 --- /dev/null +++ b/tests/ci/pytest.ini @@ -0,0 +1,6 @@ +[pytest] +# Cluster routes are fronted by self-signed certs, so we make requests with +# verify=False (mirroring playwright's ignore_https_errors). Silence the +# resulting InsecureRequestWarning noise from urllib3. +filterwarnings = + ignore::urllib3.exceptions.InsecureRequestWarning diff --git a/tests/ci/requirements.txt b/tests/ci/requirements.txt new file mode 100644 index 00000000..47480078 --- /dev/null +++ b/tests/ci/requirements.txt @@ -0,0 +1,17 @@ +ansible-core==2.18.* +ansible-runner +awxkit +kubernetes +openshift +boto3>=1.21 +botocore>=1.24 +awscli>=1.22 +azure-cli>=2.34 +gcloud +humanize +pytz +pytest +pytest-playwright +requests +junitparser +vp-qe-test-common @ git+https://github.com/validatedpatterns/vp-qe-test-common.git@v1 diff --git a/tests/ci/test_ci.py b/tests/ci/test_ci.py new file mode 100644 index 00000000..186f254e --- /dev/null +++ b/tests/ci/test_ci.py @@ -0,0 +1,330 @@ +import os + +import pytest +import requests +from ocp_resources.machine_set import MachineSet +from ocp_resources.node import Node +from ocp_resources.pod import Pod +from ocp_resources.resource import Resource +from ocp_resources.route import Route +from playwright.sync_api import expect, sync_playwright +from validatedpatterns_tests.interop import components, subscription + + +def _route_url(openshift_dyn_client, namespace, name): + routes = [ + route + for route in Route.get( + dyn_client=openshift_dyn_client, namespace=namespace, name=name + ) + ] + + assert ( + len(routes) == 1 + ), f"Expected to find the route '{name}' in the namespace '{namespace}'" + + spec = routes[0].instance.spec + scheme = "https" if getattr(spec, "tls", None) else "http" + return f"{scheme}://{spec.host}" + + +@pytest.mark.parametrize( + "openshift_dyn_client", + ["VP_HUBCONFIG"], + indirect=True, +) +def test_subscription_status(openshift_dyn_client): + expected_subs = { + "openshift-gitops-operator": ["openshift-gitops-operator"], + "prometheus": ["llm-monitoring"], + "grafana-operator": ["llm-monitoring"], + "nfd": ["openshift-nfd"], + "gpu-operator-certified": ["nvidia-gpu-operator"], + } + + subscription.assert_subscription_status(openshift_dyn_client, expected_subs) + + +@pytest.mark.parametrize( + "openshift_dyn_client", + ["VP_HUBCONFIG"], + indirect=True, +) +def test_site_reachable(openshift_dyn_client): + components.assert_site_reachable(openshift_dyn_client) + + +@pytest.mark.parametrize( + "openshift_dyn_client", + ["VP_HUBCONFIG"], + indirect=True, +) +def test_pod_count(openshift_dyn_client): + projects = {"rag-llm": 4} + + errors = [] + for key in projects.keys(): + pods = [pod for pod in Pod.get(dyn_client=openshift_dyn_client, namespace=key)] + expected_count = projects[key] + actual_count = len(pods) + + if actual_count < expected_count: + errors.append( + f"Expected the namespace '{key}' to contain at least {expected_count} pods but it actually contains {actual_count} pods" + ) + + assert not errors, "\n".join(errors) + + +@pytest.mark.parametrize( + "openshift_dyn_client", + ["VP_HUBCONFIG"], + indirect=True, +) +def test_llm_ui_route(openshift_dyn_client): + # _route_url asserts that exactly one 'llm-ui' route exists in 'rag-llm'. + url = _route_url(openshift_dyn_client, "rag-llm", "llm-ui") + + response = requests.get(url, verify=False, timeout=60) + assert ( + response.status_code == 200 + ), f"Expected a 200 from the llm-ui route '{url}' but got {response.status_code}" + + +@pytest.mark.parametrize( + "openshift_dyn_client", + ["VP_HUBCONFIG"], + indirect=True, +) +def test_vllm_inference_service_route(openshift_dyn_client): + base_url = _route_url(openshift_dyn_client, "rag-llm", "vllm-inference-service") + + models_url = f"{base_url}/v1/models" + models_response = requests.get(models_url, verify=False, timeout=60) + assert ( + models_response.status_code == 200 + ), f"Expected a 200 from '{models_url}' but got {models_response.status_code}" + + models = models_response.json().get("data", []) + assert models, f"Expected vLLM to be serving at least one model at '{base_url}'" + model = models[0]["id"] + + completions_url = f"{base_url}/v1/chat/completions" + payload = { + "model": model, + "messages": [{"role": "user", "content": "Reply with a short greeting."}], + "max_tokens": 20, + } + response = requests.post(completions_url, json=payload, verify=False, timeout=300) + assert ( + response.status_code == 200 + ), f"Expected a 200 from '{completions_url}' but got {response.status_code}: {response.text}" + + content = response.json()["choices"][0]["message"]["content"] + assert ( + content + ), f"Expected a non-empty completion from the LLM at '{completions_url}'" + + +@pytest.mark.parametrize( + "openshift_dyn_client", + ["VP_HUBCONFIG"], + indirect=True, +) +def test_nodefeaturediscovery(openshift_dyn_client): + name = "nfd-instance" + namespace = "openshift-nfd" + + class NodeFeatureDiscovery(Resource): + api_group = "nfd.openshift.io" + api_version = "v1" + kind = "NodeFeatureDiscovery" + + nfds = [ + nfd + for nfd in NodeFeatureDiscovery.get( + dyn_client=openshift_dyn_client, namespace=namespace, name=name + ) + ] + + assert ( + len(nfds) == 1 + ), f"Expected to find the NodeFeatureDiscovery '{name}' in the namespace '{namespace}'" + + +@pytest.mark.parametrize( + "openshift_dyn_client", + ["VP_HUBCONFIG"], + indirect=True, +) +def test_gpu_clusterpolicy(openshift_dyn_client): + name = "rag-llm-gpu-cluster-policy" + expected_tolerations = [ + {"effect": "NoSchedule", "key": "odh-notebook", "value": "true"} + ] + + class ClusterPolicy(Resource): + api_group = "nvidia.com" + api_version = "v1" + kind = "ClusterPolicy" + + policies = [ + policy + for policy in ClusterPolicy.get(dyn_client=openshift_dyn_client, name=name) + ] + + assert len(policies) == 1, f"Expected to find the ClusterPolicy '{name}'" + + actual_tolerations = [ + item.to_dict() for item in policies[0].instance.spec.daemonsets.tolerations + ] + + assert ( + actual_tolerations == expected_tolerations + ), f"Expected the ClusterPolicy '{name}' to contain the tolerations '{expected_tolerations}' but it actually contains '{actual_tolerations}'" + + +@pytest.mark.parametrize( + "openshift_dyn_client", + ["VP_HUBCONFIG"], + indirect=True, +) +def test_gpu_machineset(openshift_dyn_client): + namespace = "openshift-machine-api" + machinesets = [ + ms + for ms in MachineSet.get(dyn_client=openshift_dyn_client, namespace=namespace) + ] + + gpu_machinesets = [ms for ms in machinesets if "-gpu-" in ms.instance.metadata.name] + assert ( + len(gpu_machinesets) == 1 + ), f"Expected to find a GPU MachineSet in the namespace '{namespace}'" + + gpu_machineset = gpu_machinesets[0] + gpu_machineset_name = gpu_machineset.instance.metadata.name + + actual_taints = [ + item.to_dict() for item in gpu_machineset.instance.spec.template.spec.taints + ] + expected_taints = [{"effect": "NoSchedule", "key": "odh-notebook", "value": "true"}] + assert ( + actual_taints == expected_taints + ), f"Expected GPU MachineSet '{gpu_machineset_name}' to contain the taints '{expected_taints}' but it actually contains '{actual_taints}'" + + actual_labels = [ + item for item in gpu_machineset.instance.spec.template.spec.metadata.labels + ] + expected_labels = [("node-role.kubernetes.io/odh-notebook", "")] + assert ( + actual_labels == expected_labels + ), f"Expected GPU MachineSet '{gpu_machineset_name}' to contain the labels '{expected_labels}' but it actually contains '{actual_labels}'" + + +@pytest.mark.parametrize( + "openshift_dyn_client", + ["VP_HUBCONFIG"], + indirect=True, +) +def test_gpu_node_role_labels_pods(openshift_dyn_client): + def is_gpu_node(node: Node) -> bool: + node_labels = [label for label in node.instance.metadata.labels] + odh_label = ("node-role.kubernetes.io/odh-notebook", "") + worker_label = ("node-role.kubernetes.io/worker", "") + + return odh_label in node_labels and worker_label in node_labels + + gpu_nodes = [ + node for node in Node.get(dyn_client=openshift_dyn_client) if is_gpu_node(node) + ] + + num_gpu_nodes = len(gpu_nodes) + assert ( + num_gpu_nodes == 1 + ), f"Expected to find 1 GPU Node but actually found '{num_gpu_nodes}'" + + namespace = "nvidia-gpu-operator" + gpu_node_name = gpu_nodes[0].instance.metadata.name + nvidia_pods = [ + pod + for pod in Pod.get(dyn_client=openshift_dyn_client, namespace=namespace) + if pod.instance.spec.nodeName == gpu_node_name + and "nvidia" in pod.instance.metadata.name + ] + + actual_count = len(nvidia_pods) + expected_count = 8 + assert ( + actual_count == expected_count + ), f"Expected to find {expected_count} Nvidia pods on Node '{gpu_node_name}' but actually found {actual_count}" + + +@pytest.mark.parametrize( + "openshift_dyn_client", + ["VP_HUBCONFIG"], + indirect=True, +) +def test_ragllm_ui(openshift_dyn_client): + rag_ui_url = _route_url(openshift_dyn_client, "rag-llm", "llm-ui") + grafana_url = _route_url( + openshift_dyn_client, "llm-monitoring", "ai-llm-grafana-route" + ) + + results_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), ".results") + os.makedirs(results_dir, exist_ok=True) + + with sync_playwright() as p: + browser = p.chromium.launch(headless=True) + context = browser.new_context(ignore_https_errors=True) + context.set_default_timeout(120_000) + page = context.new_page() + try: + # Generate a proposal in the RAG-LLM demo UI + page.goto(rag_ui_url) + page.get_by_role("textbox", name="Customer Enter the customer").fill( + "validated-patterns-qe" + ) + page.get_by_role("textbox", name="Product Enter the Red Hat").fill( + "RedHat OpenShift AI" + ) + page.get_by_role("button", name="Generate").click() + + # Wait for generation to complete, then submit a rating + rating = page.get_by_role("radio", name="3") + expect(rating).to_be_visible(timeout=180_000) + rating.check() + + # Add a provider on the Configuration tab + page.get_by_role("tab", name="Configuration").click() + page.get_by_role("button", name="Add Provider").click() + page.get_by_role("listbox", name="Providers").click() + page.get_by_role("option", name="OpenAI").click() + page.get_by_role("textbox", name="Model Enter the model name").fill( + "gpt-4o-mini" + ) + page.get_by_role("textbox", name="URL Enter the URL").fill( + "https://api.openai.com/v1/chat/completions" + ) + page.get_by_test_id("password").fill("12121212") + page.get_by_role("button", name="Add", exact=True).click() + + toast_close = page.get_by_test_id("toast-close") + expect(toast_close).to_be_visible() + toast_close.click() + page.screenshot(path=os.path.join(results_dir, "ragllm-add-provider.png")) + + # Check the Grafana LLM ratings dashboard + gpage = context.new_page() + gpage.goto(grafana_url) + gpage.get_by_role("link", name="Dashboards").click() + gpage.get_by_role("link", name="llm-monitoring").click() + + feedback = gpage.get_by_role("link", name="MODEL FEEDBACK/RATING") + expect(feedback).to_be_visible() + feedback.click() + gpage.screenshot( + path=os.path.join(results_dir, "ragllm-grafana-dashboard.png") + ) + finally: + context.close() + browser.close() diff --git a/tests/common-acm.expected.diff b/tests/common-acm.expected.diff deleted file mode 100644 index 2c8924b9..00000000 --- a/tests/common-acm.expected.diff +++ /dev/null @@ -1,151 +0,0 @@ ---- tests/common-acm-naked.expected.yaml -+++ tests/common-acm-normal.expected.yaml -@@ -1,7 +1,4 @@ - --- --# Source: acm/templates/policies/application-policies.yaml --# TODO: Also create a GitOpsCluster.apps.open-cluster-management.io ----- - # Source: acm/templates/multiclusterhub.yaml - apiVersion: operator.open-cluster-management.io/v1 - kind: MultiClusterHub -@@ -12,6 +9,22 @@ - argocd.argoproj.io/sync-wave: "-1" - spec: {} - --- -+# Source: acm/templates/policies/application-policies.yaml -+apiVersion: policy.open-cluster-management.io/v1 -+kind: PlacementBinding -+metadata: -+ name: group-one-placement-binding -+ annotations: -+ argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true -+placementRef: -+ name: group-one-placement -+ kind: PlacementRule -+ apiGroup: apps.open-cluster-management.io -+subjects: -+ - name: group-one-clustergroup-policy -+ kind: Policy -+ apiGroup: policy.open-cluster-management.io -+--- - # Source: acm/templates/policies/ocp-gitops-policy.yaml - apiVersion: policy.open-cluster-management.io/v1 - kind: PlacementBinding -@@ -28,6 +41,19 @@ - kind: Policy - apiGroup: policy.open-cluster-management.io - --- -+# Source: acm/templates/policies/application-policies.yaml -+apiVersion: apps.open-cluster-management.io/v1 -+kind: PlacementRule -+metadata: -+ name: group-one-placement -+spec: -+ clusterConditions: -+ - status: 'True' -+ type: ManagedClusterConditionAvailable -+ clusterSelector: -+ matchLabels: -+ clusterGroup: group-one -+--- - # Source: acm/templates/policies/ocp-gitops-policy.yaml - apiVersion: apps.open-cluster-management.io/v1 - kind: PlacementRule -@@ -44,6 +70,97 @@ - values: - - OpenShift - --- -+# Source: acm/templates/policies/application-policies.yaml -+# TODO: Also create a GitOpsCluster.apps.open-cluster-management.io -+apiVersion: policy.open-cluster-management.io/v1 -+kind: Policy -+metadata: -+ name: group-one-clustergroup-policy -+ annotations: -+ argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true -+ argocd.argoproj.io/compare-options: IgnoreExtraneous -+spec: -+ remediationAction: enforce -+ disabled: false -+ policy-templates: -+ - objectDefinition: -+ apiVersion: policy.open-cluster-management.io/v1 -+ kind: ConfigurationPolicy -+ metadata: -+ name: group-one-clustergroup-config -+ spec: -+ remediationAction: enforce -+ severity: medium -+ namespaceSelector: -+ include: -+ - default -+ object-templates: -+ - complianceType: mustonlyhave -+ objectDefinition: -+ apiVersion: argoproj.io/v1alpha1 -+ kind: Application -+ metadata: -+ name: mypattern-group-one -+ namespace: openshift-gitops -+ finalizers: -+ - resources-finalizer.argocd.argoproj.io/foreground -+ spec: -+ project: default -+ source: -+ repoURL: https://github.com/pattern-clone/mypattern -+ targetRevision: main -+ path: common/clustergroup -+ helm: -+ ignoreMissingValueFiles: true -+ valueFiles: -+ - "/values-global.yaml" -+ - "/values-group-one.yaml" -+ - '/values-{{ (lookup "config.openshift.io/v1" "Infrastructure" "" "cluster").spec.platformSpec.type }}-group-one.yaml' -+ # We cannot use $.Values.global.clusterVersion because that gets resolved to the -+ # hub's cluster version, whereas we want to include the spoke cluster version -+ - '/values-{{ printf "%d.%d" ((semver (lookup "operator.openshift.io/v1" "OpenShiftControllerManager" "" "cluster").status.version).Major) ((semver (lookup "operator.openshift.io/v1" "OpenShiftControllerManager" "" "cluster").status.version).Minor) }}-group-one.yaml' -+ parameters: -+ - name: global.repoURL -+ value: $ARGOCD_APP_SOURCE_REPO_URL -+ - name: global.targetRevision -+ value: $ARGOCD_APP_SOURCE_TARGET_REVISION -+ - name: global.namespace -+ value: $ARGOCD_APP_NAMESPACE -+ - name: global.pattern -+ value: mypattern -+ - name: global.hubClusterDomain -+ value: apps.hub.example.com -+ - name: global.localClusterDomain -+ value: '{{ (lookup "config.openshift.io/v1" "Ingress" "" "cluster").spec.domain }}' -+ # Requires ACM 2.6 or higher -+ - name: global.clusterDomain -+ value: '{{ (lookup "config.openshift.io/v1" "Ingress" "" "cluster").spec.domain | replace "apps." "" }}' -+ # Requires ACM 2.6 or higher (I could not come up with something less terrible to get maj.min) -+ - name: global.clusterVersion -+ value: '{{ printf "%d.%d" ((semver (lookup "operator.openshift.io/v1" "OpenShiftControllerManager" "" "cluster").status.version).Major) ((semver (lookup "operator.openshift.io/v1" "OpenShiftControllerManager" "" "cluster").status.version).Minor) }}' -+ - name: global.clusterPlatform -+ value: -+ - name: clusterGroup.name -+ value: group-one -+ - name: clusterGroup.isHubCluster -+ value: "false" -+ destination: -+ server: https://kubernetes.default.svc -+ namespace: mypattern-group-one -+ syncPolicy: -+ automated: -+ prune: false -+ selfHeal: true -+ ignoreDifferences: -+ - group: apps -+ kind: Deployment -+ jsonPointers: -+ - /spec/replicas -+ - group: route.openshift.io -+ kind: Route -+ jsonPointers: -+ - /status -+--- - # Source: acm/templates/policies/ocp-gitops-policy.yaml - apiVersion: policy.open-cluster-management.io/v1 - kind: Policy diff --git a/tests/common-clustergroup.expected.diff b/tests/common-clustergroup.expected.diff deleted file mode 100644 index dee42fc0..00000000 --- a/tests/common-clustergroup.expected.diff +++ /dev/null @@ -1,764 +0,0 @@ ---- tests/common-clustergroup-naked.expected.yaml -+++ tests/common-clustergroup-normal.expected.yaml -@@ -1,17 +1,204 @@ - --- -+# Source: pattern-clustergroup/templates/core/namespaces.yaml -+apiVersion: v1 -+kind: Namespace -+metadata: -+ labels: -+ argocd.argoproj.io/managed-by: mypattern-hub -+ name: open-cluster-management -+spec: -+--- -+# Source: pattern-clustergroup/templates/core/namespaces.yaml -+apiVersion: v1 -+kind: Namespace -+metadata: -+ labels: -+ argocd.argoproj.io/managed-by: mypattern-hub -+ name: vault -+spec: -+--- -+# Source: pattern-clustergroup/templates/core/namespaces.yaml -+apiVersion: v1 -+kind: Namespace -+metadata: -+ labels: -+ argocd.argoproj.io/managed-by: mypattern-hub -+ name: golang-external-secrets -+spec: -+--- -+# Source: pattern-clustergroup/templates/core/namespaces.yaml -+apiVersion: v1 -+kind: Namespace -+metadata: -+ labels: -+ argocd.argoproj.io/managed-by: mypattern-hub -+ name: config-demo -+spec: -+--- -+# Source: pattern-clustergroup/templates/imperative/namespace.yaml -+apiVersion: v1 -+kind: Namespace -+metadata: -+ labels: -+ name: imperative -+ argocd.argoproj.io/managed-by: mypattern-hub -+ name: imperative -+--- - # Source: pattern-clustergroup/templates/plumbing/gitops-namespace.yaml - apiVersion: v1 - kind: Namespace - metadata: - labels: -- name: common-example -+ name: mypattern-hub - # The name here needs to be consistent with - # - acm/templates/policies/application-policies.yaml - # - clustergroup/templates/applications.yaml - # - any references to secrets and route URLs in documentation -- name: common-example -+ name: mypattern-hub - spec: {} - --- -+# Source: pattern-clustergroup/templates/imperative/serviceaccount.yaml -+apiVersion: v1 -+kind: ServiceAccount -+metadata: -+ name: imperative-sa -+ namespace: imperative -+--- -+# Source: pattern-clustergroup/templates/imperative/configmap.yaml -+apiVersion: v1 -+kind: ConfigMap -+metadata: -+ name: helm-values-configmap-hub -+ namespace: imperative -+data: -+ values.yaml: | -+ clusterGroup: -+ applications: -+ acm: -+ ignoreDifferences: -+ - group: internal.open-cluster-management.io -+ jsonPointers: -+ - /spec/loggingCA -+ kind: ManagedClusterInfo -+ name: acm -+ namespace: open-cluster-management -+ path: common/acm -+ project: hub -+ config-demo: -+ name: config-demo -+ namespace: config-demo -+ path: charts/all/config-demo -+ project: config-demo -+ golang-external-secrets: -+ name: golang-external-secrets -+ namespace: golang-external-secrets -+ path: common/golang-external-secrets -+ project: hub -+ vault: -+ name: vault -+ namespace: vault -+ path: common/hashicorp-vault -+ project: hub -+ imperative: -+ activeDeadlineSeconds: 3600 -+ clusterRoleName: imperative-cluster-role -+ clusterRoleYaml: "" -+ cronJobName: imperative-cronjob -+ image: registry.redhat.io/ansible-automation-platform-22/ee-supported-rhel8:latest -+ imagePullPolicy: Always -+ insecureUnsealVaultInsideClusterSchedule: '*/5 * * * *' -+ jobName: imperative-job -+ jobs: -+ - name: test -+ playbook: ansible/test.yml -+ timeout: 234 -+ namespace: imperative -+ roleName: imperative-role -+ roleYaml: "" -+ schedule: '*/10 * * * *' -+ serviceAccountCreate: true -+ serviceAccountName: imperative-sa -+ valuesConfigMap: helm-values-configmap -+ verbosity: "" -+ insecureUnsealVaultInsideCluster: true -+ isHubCluster: true -+ managedClusterGroups: -+ exampleRegion: -+ helmOverrides: -+ - name: clusterGroup.isHubCluster -+ value: false -+ labels: -+ - name: clusterGroup -+ value: group-one -+ name: group-one -+ name: hub -+ namespaces: -+ - open-cluster-management -+ - vault -+ - golang-external-secrets -+ - config-demo -+ projects: -+ - hub -+ - config-demo -+ subscriptions: -+ acm: -+ channel: release-2.6 -+ name: advanced-cluster-management -+ namespace: open-cluster-management -+ targetCluster: in-cluster -+ enabled: all -+ global: -+ clusterDomain: region.example.com -+ hubClusterDomain: apps.hub.example.com -+ localClusterDomain: apps.region.example.com -+ namespace: pattern-namespace -+ options: -+ installPlanApproval: Automatic -+ syncPolicy: Automatic -+ useCSV: false -+ pattern: mypattern -+ repoURL: https://github.com/pattern-clone/mypattern -+ targetRevision: main -+ main: -+ clusterGroupName: hub -+ git: -+ repoURL: https://github.com/pattern-clone/mypattern -+ revision: main -+ secretStore: -+ kind: ClusterSecretStore -+ name: vault-backend -+ secretsBase: -+ key: secret/data/hub -+--- -+# Source: pattern-clustergroup/templates/imperative/clusterrole.yaml -+apiVersion: rbac.authorization.k8s.io/v1 -+kind: ClusterRole -+metadata: -+ name: imperative-cluster-role -+rules: -+ - apiGroups: -+ - '*' -+ resources: -+ - '*' -+ verbs: -+ - get -+ - list -+ - watch -+--- -+# Source: pattern-clustergroup/templates/imperative/rbac.yaml -+apiVersion: rbac.authorization.k8s.io/v1 -+kind: ClusterRoleBinding -+metadata: -+ name: imperative-cluster-admin-rolebinding -+roleRef: -+ apiGroup: rbac.authorization.k8s.io -+ kind: ClusterRole -+ name: imperative-cluster-role -+subjects: -+ - kind: ServiceAccount -+ name: imperative-sa -+ namespace: imperative -+--- - # Source: pattern-clustergroup/templates/plumbing/argocd-super-role.yaml - # WARNING: ONLY USE THIS FOR MANAGING CLUSTERS NOT FOR REGULAR USERS - apiVersion: rbac.authorization.k8s.io/v1 -@@ -36,7 +223,7 @@ - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRoleBinding - metadata: -- name: common-example-cluster-admin-rolebinding -+ name: mypattern-hub-cluster-admin-rolebinding - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole -@@ -44,17 +231,436 @@ - subjects: - - kind: ServiceAccount - # This is the {ArgoCD.name}-argocd-application-controller -- name: example-gitops-argocd-application-controller -- namespace: common-example -+ name: hub-gitops-argocd-application-controller -+ namespace: mypattern-hub - # NOTE: THIS MUST BE FIXED FOR MULTITENANT SETUP - - kind: ServiceAccount - # This is the {ArgoCD.name}-argocd-server -- name: example-gitops-argocd-server -- namespace: common-example -+ name: hub-gitops-argocd-server -+ namespace: mypattern-hub - # NOTE: This is needed starting with gitops-1.5.0 (see issue common#76) - - kind: ServiceAccount -- name: example-gitops-argocd-dex-server -- namespace: common-example -+ name: hub-gitops-argocd-dex-server -+ namespace: mypattern-hub -+--- -+# Source: pattern-clustergroup/templates/imperative/role.yaml -+apiVersion: rbac.authorization.k8s.io/v1 -+kind: Role -+metadata: -+ name: imperative-role -+ namespace: imperative -+rules: -+ - apiGroups: -+ - '*' -+ resources: -+ - '*' -+ verbs: -+ - '*' -+--- -+# Source: pattern-clustergroup/templates/imperative/rbac.yaml -+apiVersion: rbac.authorization.k8s.io/v1 -+kind: RoleBinding -+metadata: -+ name: imperative-admin-rolebinding -+ namespace: imperative -+roleRef: -+ apiGroup: rbac.authorization.k8s.io -+ kind: Role -+ name: imperative-role -+subjects: -+ - kind: ServiceAccount -+ name: imperative-sa -+ namespace: imperative -+--- -+# Source: pattern-clustergroup/templates/imperative/job.yaml -+apiVersion: batch/v1 -+kind: CronJob -+metadata: -+ name: imperative-cronjob -+ namespace: imperative -+spec: -+ schedule: "*/10 * * * *" -+ # if previous Job is still running, skip execution of a new Job -+ concurrencyPolicy: Forbid -+ jobTemplate: -+ spec: -+ activeDeadlineSeconds: 3600 -+ template: -+ metadata: -+ name: imperative-job -+ spec: -+ serviceAccountName: imperative-sa -+ initContainers: -+ # git init happens in /git/repo so that we can set the folder to 0770 permissions -+ # reason for that is ansible refuses to create temporary folders in there -+ - name: git-init -+ image: registry.redhat.io/ansible-automation-platform-22/ee-supported-rhel8:latest -+ imagePullPolicy: Always -+ env: -+ - name: HOME -+ value: /git/home -+ command: -+ - 'sh' -+ - '-c' -+ - "mkdir /git/{repo,home};git clone --single-branch --branch main --depth 1 -- https://github.com/pattern-clone/mypattern /git/repo;chmod 0770 /git/{repo,home}" -+ volumeMounts: -+ - name: git -+ mountPath: "/git" -+ - name: test -+ image: registry.redhat.io/ansible-automation-platform-22/ee-supported-rhel8:latest -+ imagePullPolicy: Always -+ env: -+ - name: HOME -+ value: /git/home -+ workingDir: /git/repo -+ # We have a default timeout of 600s for each playbook. Can be overridden -+ # on a per-job basis -+ command: -+ - timeout -+ - "234" -+ - ansible-playbook -+ - -e -+ - "@/values/values.yaml" -+ - ansible/test.yml -+ volumeMounts: -+ - name: git -+ mountPath: "/git" -+ - name: values-volume -+ mountPath: /values/values.yaml -+ subPath: values.yaml -+ containers: -+ - name: "done" -+ image: registry.redhat.io/ansible-automation-platform-22/ee-supported-rhel8:latest -+ imagePullPolicy: Always -+ command: -+ - 'sh' -+ - '-c' -+ - 'echo' -+ - 'done' -+ - '\n' -+ volumes: -+ - name: git -+ emptyDir: {} -+ - name: values-volume -+ configMap: -+ name: helm-values-configmap-hub -+ restartPolicy: Never -+--- -+# Source: pattern-clustergroup/templates/imperative/unsealjob.yaml -+apiVersion: batch/v1 -+kind: CronJob -+metadata: -+ name: unsealvault-cronjob -+ namespace: imperative -+spec: -+ schedule: "*/5 * * * *" -+ # if previous Job is still running, skip execution of a new Job -+ concurrencyPolicy: Forbid -+ jobTemplate: -+ spec: -+ activeDeadlineSeconds: 3600 -+ template: -+ metadata: -+ name: unsealvault-job -+ spec: -+ serviceAccountName: imperative-sa -+ initContainers: -+ # git init happens in /git/repo so that we can set the folder to 0770 permissions -+ # reason for that is ansible refuses to create temporary folders in there -+ - name: git-init -+ image: registry.redhat.io/ansible-automation-platform-22/ee-supported-rhel8:latest -+ imagePullPolicy: Always -+ env: -+ - name: HOME -+ value: /git/home -+ command: -+ - 'sh' -+ - '-c' -+ - "mkdir /git/{repo,home};git clone --single-branch --branch main --depth 1 -- https://github.com/pattern-clone/mypattern /git/repo;chmod 0770 /git/{repo,home}" -+ volumeMounts: -+ - name: git -+ mountPath: "/git" -+ - name: unseal-playbook -+ image: registry.redhat.io/ansible-automation-platform-22/ee-supported-rhel8:latest -+ imagePullPolicy: Always -+ env: -+ - name: HOME -+ value: /git/home -+ workingDir: /git/repo -+ # We have a default timeout of 600s for each playbook. Can be overridden -+ # on a per-job basis -+ command: -+ - timeout -+ - "600" -+ - ansible-playbook -+ - -e -+ - "@/values/values.yaml" -+ - -e -+ - '{"file_unseal": false}' -+ - -t -+ - 'vault_init,vault_unseal,vault_secrets_init' -+ - "common/ansible/playbooks/vault/vault.yaml" -+ volumeMounts: -+ - name: git -+ mountPath: "/git" -+ - name: values-volume -+ mountPath: /values/values.yaml -+ subPath: values.yaml -+ containers: -+ - name: "done" -+ image: registry.redhat.io/ansible-automation-platform-22/ee-supported-rhel8:latest -+ imagePullPolicy: Always -+ command: -+ - 'sh' -+ - '-c' -+ - 'echo' -+ - 'done' -+ - '\n' -+ volumes: -+ - name: git -+ emptyDir: {} -+ - name: values-volume -+ configMap: -+ name: helm-values-configmap-hub -+ restartPolicy: Never -+--- -+# Source: pattern-clustergroup/templates/core/subscriptions.yaml -+--- -+--- -+# Source: pattern-clustergroup/templates/plumbing/projects.yaml -+apiVersion: argoproj.io/v1alpha1 -+kind: AppProject -+metadata: -+ name: hub -+ namespace: mypattern-hub -+spec: -+ description: "Pattern hub" -+ destinations: -+ - namespace: '*' -+ server: '*' -+ clusterResourceWhitelist: -+ - group: '*' -+ kind: '*' -+ namespaceResourceWhitelist: -+ - group: '*' -+ kind: '*' -+ sourceRepos: -+ - '*' -+status: {} -+--- -+# Source: pattern-clustergroup/templates/plumbing/projects.yaml -+apiVersion: argoproj.io/v1alpha1 -+kind: AppProject -+metadata: -+ name: config-demo -+ namespace: mypattern-hub -+spec: -+ description: "Pattern config-demo" -+ destinations: -+ - namespace: '*' -+ server: '*' -+ clusterResourceWhitelist: -+ - group: '*' -+ kind: '*' -+ namespaceResourceWhitelist: -+ - group: '*' -+ kind: '*' -+ sourceRepos: -+ - '*' -+status: {} -+--- -+# Source: pattern-clustergroup/templates/plumbing/applications.yaml -+apiVersion: argoproj.io/v1alpha1 -+kind: Application -+metadata: -+ name: acm -+ namespace: mypattern-hub -+ finalizers: -+ - resources-finalizer.argocd.argoproj.io/foreground -+spec: -+ destination: -+ name: in-cluster -+ namespace: open-cluster-management -+ project: hub -+ source: -+ repoURL: https://github.com/pattern-clone/mypattern -+ targetRevision: main -+ path: common/acm -+ helm: -+ ignoreMissingValueFiles: true -+ valueFiles: -+ - "/values-global.yaml" -+ - "/values-hub.yaml" -+ # Watch the progress of https://issues.redhat.com/browse/GITOPS-891 and update accordingly -+ parameters: -+ - name: global.repoURL -+ value: $ARGOCD_APP_SOURCE_REPO_URL -+ - name: global.targetRevision -+ value: $ARGOCD_APP_SOURCE_TARGET_REVISION -+ - name: global.namespace -+ value: $ARGOCD_APP_NAMESPACE -+ - name: global.pattern -+ value: mypattern -+ - name: global.clusterDomain -+ value: region.example.com -+ - name: global.clusterVersion -+ value: "" -+ - name: global.clusterPlatform -+ value: "" -+ - name: global.hubClusterDomain -+ value: apps.hub.example.com -+ - name: global.localClusterDomain -+ value: apps.region.example.com -+ ignoreDifferences: [ -+ { -+ "group": "internal.open-cluster-management.io", -+ "jsonPointers": [ -+ "/spec/loggingCA" -+ ], -+ "kind": "ManagedClusterInfo" -+ } -+] -+ syncPolicy: -+ automated: {} -+ # selfHeal: true -+--- -+# Source: pattern-clustergroup/templates/plumbing/applications.yaml -+apiVersion: argoproj.io/v1alpha1 -+kind: Application -+metadata: -+ name: config-demo -+ namespace: mypattern-hub -+ finalizers: -+ - resources-finalizer.argocd.argoproj.io/foreground -+spec: -+ destination: -+ name: in-cluster -+ namespace: config-demo -+ project: config-demo -+ source: -+ repoURL: https://github.com/pattern-clone/mypattern -+ targetRevision: main -+ path: charts/all/config-demo -+ helm: -+ ignoreMissingValueFiles: true -+ valueFiles: -+ - "/values-global.yaml" -+ - "/values-hub.yaml" -+ # Watch the progress of https://issues.redhat.com/browse/GITOPS-891 and update accordingly -+ parameters: -+ - name: global.repoURL -+ value: $ARGOCD_APP_SOURCE_REPO_URL -+ - name: global.targetRevision -+ value: $ARGOCD_APP_SOURCE_TARGET_REVISION -+ - name: global.namespace -+ value: $ARGOCD_APP_NAMESPACE -+ - name: global.pattern -+ value: mypattern -+ - name: global.clusterDomain -+ value: region.example.com -+ - name: global.clusterVersion -+ value: "" -+ - name: global.clusterPlatform -+ value: "" -+ - name: global.hubClusterDomain -+ value: apps.hub.example.com -+ - name: global.localClusterDomain -+ value: apps.region.example.com -+ syncPolicy: -+ automated: {} -+ # selfHeal: true -+--- -+# Source: pattern-clustergroup/templates/plumbing/applications.yaml -+apiVersion: argoproj.io/v1alpha1 -+kind: Application -+metadata: -+ name: golang-external-secrets -+ namespace: mypattern-hub -+ finalizers: -+ - resources-finalizer.argocd.argoproj.io/foreground -+spec: -+ destination: -+ name: in-cluster -+ namespace: golang-external-secrets -+ project: hub -+ source: -+ repoURL: https://github.com/pattern-clone/mypattern -+ targetRevision: main -+ path: common/golang-external-secrets -+ helm: -+ ignoreMissingValueFiles: true -+ valueFiles: -+ - "/values-global.yaml" -+ - "/values-hub.yaml" -+ # Watch the progress of https://issues.redhat.com/browse/GITOPS-891 and update accordingly -+ parameters: -+ - name: global.repoURL -+ value: $ARGOCD_APP_SOURCE_REPO_URL -+ - name: global.targetRevision -+ value: $ARGOCD_APP_SOURCE_TARGET_REVISION -+ - name: global.namespace -+ value: $ARGOCD_APP_NAMESPACE -+ - name: global.pattern -+ value: mypattern -+ - name: global.clusterDomain -+ value: region.example.com -+ - name: global.clusterVersion -+ value: "" -+ - name: global.clusterPlatform -+ value: "" -+ - name: global.hubClusterDomain -+ value: apps.hub.example.com -+ - name: global.localClusterDomain -+ value: apps.region.example.com -+ syncPolicy: -+ automated: {} -+ # selfHeal: true -+--- -+# Source: pattern-clustergroup/templates/plumbing/applications.yaml -+apiVersion: argoproj.io/v1alpha1 -+kind: Application -+metadata: -+ name: vault -+ namespace: mypattern-hub -+ finalizers: -+ - resources-finalizer.argocd.argoproj.io/foreground -+spec: -+ destination: -+ name: in-cluster -+ namespace: vault -+ project: hub -+ source: -+ repoURL: https://github.com/pattern-clone/mypattern -+ targetRevision: main -+ path: common/hashicorp-vault -+ helm: -+ ignoreMissingValueFiles: true -+ valueFiles: -+ - "/values-global.yaml" -+ - "/values-hub.yaml" -+ # Watch the progress of https://issues.redhat.com/browse/GITOPS-891 and update accordingly -+ parameters: -+ - name: global.repoURL -+ value: $ARGOCD_APP_SOURCE_REPO_URL -+ - name: global.targetRevision -+ value: $ARGOCD_APP_SOURCE_TARGET_REVISION -+ - name: global.namespace -+ value: $ARGOCD_APP_NAMESPACE -+ - name: global.pattern -+ value: mypattern -+ - name: global.clusterDomain -+ value: region.example.com -+ - name: global.clusterVersion -+ value: "" -+ - name: global.clusterPlatform -+ value: "" -+ - name: global.hubClusterDomain -+ value: apps.hub.example.com -+ - name: global.localClusterDomain -+ value: apps.region.example.com -+ syncPolicy: -+ automated: {} -+ # selfHeal: true - --- - # Source: pattern-clustergroup/templates/plumbing/argocd.yaml - apiVersion: argoproj.io/v1alpha1 -@@ -64,8 +670,8 @@ - - argoproj.io/finalizer - # Changing the name affects the ClusterRoleBinding, the generated secret, - # route URL, and argocd.argoproj.io/managed-by annotations -- name: example-gitops -- namespace: common-example -+ name: hub-gitops -+ namespace: mypattern-hub - annotations: - argocd.argoproj.io/compare-options: IgnoreExtraneous - spec: -@@ -90,15 +696,15 @@ - command: ["/bin/bash", "-c"] - args: ["helm template . --name-template ${ARGOCD_APP_NAME:0:52} - -f $(git rev-parse --show-toplevel)/values-global.yaml -- -f $(git rev-parse --show-toplevel)/values-example.yaml -+ -f $(git rev-parse --show-toplevel)/values-hub.yaml - --set global.repoURL=$ARGOCD_APP_SOURCE_REPO_URL - --set global.targetRevision=$ARGOCD_APP_SOURCE_TARGET_REVISION - --set global.namespace=$ARGOCD_APP_NAMESPACE -- --set global.pattern=common -- --set global.clusterDomain= -- --set global.hubClusterDomain= -- --set global.localClusterDomain= -- --set clusterGroup.name=example -+ --set global.pattern=mypattern -+ --set global.clusterDomain=region.example.com -+ --set global.hubClusterDomain=apps.hub.example.com -+ --set global.localClusterDomain=apps.region.example.com -+ --set clusterGroup.name=hub - --post-renderer ./kustomize"] - applicationSet: - resources: -@@ -173,12 +779,65 @@ - apiVersion: console.openshift.io/v1 - kind: ConsoleLink - metadata: -- name: example-gitops-link -- namespace: common-example -+ name: hub-gitops-link -+ namespace: mypattern-hub - spec: - applicationMenu: - section: OpenShift GitOps - imageURL: data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAQwAAAEMCAYAAAAxjIiTAABtCklEQVR4nOy9B5gkx30f+qvqMHHj5RwA3OGAQwaIQ86JYBJFUgyiRJHm06Msy7QtPkkkre9ZFml9T5ItW6YtySZNijkiA0Q85EM6AAfgIu4Ol/Pepokd6v++qu7Zm9udmZ3QPTML9I/fcHE7O9011VW/+uc/R4QIESLUiYgwIkSIUDciwogQIULdiAgjQoQIdSMijAgRItSNiDAiRIhQNyLCiBAhQt2ICCNChAh1IyKMCBEi1I2IMCJEiFA3IsKIECFC3YgII0KECHUjIowIESLUjYgwIkSIUDciwogQIULdiAgjQoQIdSMijAgRItSNiDAiRIhQNyLCiBAhQt2ICCNChAh1IyKMCBEi1I2IMCJEiFA3IsKIECFC3YgII0KECHUjIowIESLUjYgwIkSIUDciwogQIULdiAgjQoQIdSMijAgRItSNiDAiRIhQNyLCiBAhQt2ICCNChAh1IyKMCBEi1I2IMCJEiFA39E4PIEK4uPduQnzVCDRiIOIQjMDAAJA6LggAo1M/S2AT/1cGOvU7kv8jBsbkdcn7tfw3995jROqCrutgDWZj6XmTLxZhJiJ6iu8y/HDDBswaOBu6yyH3rEtFMIfDYRx6UWeWUdQ1xnXOSbc1YRK0mO5S3AXFGbEYgBgHmRzQAGYAjHk8IWmBbDDmcIIlOCxBKALIOy4VdWIFMGZpGhwXwo05wnE0jbjG4QoHBo/B4QyCGI4sjuPz/UanpypCE4gIYwbiVy8dgx5jSHAd4Jp39MsnKQg3n9uHe986Eou5RpoIAwAGGKPZAJtHDHMBzGHALACDYOgjIA1CEkCcATFf6tT8taFNrBBP+nDlXbyf5BCYJAz5yjJgnAijjGEYwBBAxwCoFyMcJ2LDNuMjNljmxl0566U1aUlC4IqK5OUZNMHw/No0vs6iZdmtiJ7MDMJTb2dgFQVcYSNl6Bgby2lIxOIQop8YLdQJywWjlYyxFYywRJKEJAwAvQBS8AihXXYrt0QmAMYAnARwlED7wPg7JGi3YLSHEzukA2OOqxeEbglT0lA8DodiuOPcmBRw2jTcCPUgehpdigf3ONCzOXW0M9/kQKKgua4+QKDFYOIMRmwNY2wNAWcxYCGAPikpzADblA2gANAIAztAwE4CthBhK4F2c7BDI+gdXkCjwjYNtUiZYMi6PfjQhZGdvpOICKOL8K1rCCv+5zg0JsCtIrJunMMspHXwxZpgaxnDxWA4D4QzAMwH0FOvxEAT/zcJPhlVOsjLf0cVPktlRtAp12YNLy5BwCgDDoNhFwibiOg1AbxlAIfZsMiwOZwcMlEQWXzkgoWNXT1CIIgIo8NY/04WTtZWOjyLWRgb1vV4zJnHGFvNCJcBeB8DzgOwAFC2hmkJopwc5KbncvMyBo0zcM6gaVD/Xfr3xEv9redDUWThf04yA/meFPWTSO1uVxCEfBHBdcn/t/d7+SLh/V052TSgYbieOkMHQXgTjL8gBNsoSOw4kjlwfNnslS6Ts+YCKZ7EunMjI2o7EBFGh3DXGwWktDzcvAOXyNC4NodrdCEB14DhcgCrAWWkrKpeTGxE/zSXm13TGHSNwdA5TIPB1Dl0Xf6OeyShMfV3vJwQGtvI/s1PCRUlEpE/FXkowgAcR8BxBWybYDkCtnrRBNFMJrZpINWYIwC2AdgggGeInDdN2zhRSFpukhKw+lO4Y3FEHGEiIow24tEdeTDHUv/99F6NXbEwNw9g5zGwGwi4lgFrAPTXkiKITkkNmiZJgSMmX6b3U/5b88mBsSobkSprJ0Gg0v3IlzIkSSgCcQSKNqFouSjaApYticUnkSrq0SS4BJxkwGYQnmSMnmYCb26+cPbQeZtHldGHx5K48cyIPIJGRBhtwN07c0gWbMSdHPIsnnTJWa0x3CjAbmHA+QDmVSKJiRPYJwgpNUhSSMQ0xGOa+m/5u5I6MRFUFRYbBICJgDCftCRJeAQiUCy6yBddFCyPVMrVmRokIlWXwwBeg8CjxOkJAtut28U8j/cgbzn44MWDbft+73ZEhBESHt6TBc/YKtrxNV2wtTlawDitA9idDLgOwBIAZqXPlk5ZqVoogojrSMY1xM1TBMHKjI1dzA91ofy7SJVGqi1S+sgVXOSKLoqWUOqNmF76KALYA+AJIjwAwV65/aLBo49uHlVLXaTjuH15rC3f6d2KiDBCwBM7crDzOeRhGRqMFTqx2xjwQTBcDC9o6jSUJIkSSUgJIp3QkfBJQqoYvu3xPYPS93UFKZUll3eQlQRScOA4njEVtSWPYwBeIsHuFZweExb2mZrraskUbj473b4v8i5DRBgB4bHNNohyakZtx4mD03ncxYfA6AMAO9uPjzgNJa/kBEkkdaQTGkxDUzaIctH9vYwSKQifPLJ5F5m8g3zBVcbUaeweOYA2E9jdBHrAFWJr3IxbBEImlsRHz6wo5EWogogwAsBj2/JwrTG4jpEApws46BNgeD+g4iVO83KUpAlJCPEYR48kiaShJImSqvFekiQaRYkYlORhCUUc41lH2T7c2kZTm4BtINxPhF/mdXpzrk2WlUzipkjiqBsRYTSJB3cRYoVxCBAKtpvQiS5mjD5JDB9gwNLJRszSQjZ1jlRSQ2/KUHYJ/T2obgSFUgSsI0hJG2NZWxGIJBJRfXG7AHYR4W4CfkEkNsWMmEXE4FAP7jg/2hK1EM1OE3jknTzY6CgsGAYHzuMcnyGiDwFYWYkoOAdipoa+lI6e1ClpIiKJ4CDJQwjAsl2M5xyMZmwUVN4NVZM4JHHsIKJfMmI/Fba2VY/ZLtPjuOXc3raPf6YgIowG8MiOLLjtYtR0eCpLq8DokwB+C8BZfobnBCQZaBpDMqahP20gndKVhyOSJsLFhNThEjI5GyMZB9mCo/5dZbE7ALaA8EMi9suhkeHd8+bMI8OI4frVkX1jMiLCqBNPbilini2wV+TmgdNHAfwugIsmu0ZLRJGKaxjoMZBK6jA0T+iIeKK9YL6tI5t3MJKxleRRgzgKAF4Ese+Qyx/gsfyQafbjhlXJdg+7qxERRi3QX+DxLV/2KkflKeXq7o0M9EUAN/rp4qf+1CeKdEKfIApdqh2dG30EH566QsotOzxmTUcco0TsEcbwj8TwvK7reUPTcf3qVLuH3ZWICKMGntmcw2ExwvqFeY4g9gUw+gSAReV/o4iCA8mEjsEeQ3k8dC0iim6EJI6SxDE85kkcrlvVxrEHYD9yGL5jFrHb6EnSDWcn2j7mbkNEGBWwfnsWju2gAGvQcNlHGMMfEOHCcjsF+QswGdMw2Gsqr0dEFDMDijiUjcPByTFLeVYEVdwMtlJTQP+DhPaAHuNjOo/hvUwcEWFMwtPb8jhycjtPJRZeqHH+hwA+4letOg2mwRVR9KcN9d8RUcw8yMVvuwJjGRtDYzYKRbe8znE5jgP4KZH4h0R2zhZ7MEe3rHlvqigRYfh4ansejmPBtZx+wfFxEP2hKlZTNkdyMemcoS9tYFafqRLAWGTMnPGQz7BoCyVtjIxbsJyK9g1BDK9AiP/quuy+WMIcJ8Zx65qeTgy5Y4gIA8AT2zLoORbDyf7Rc4jwr3xX6YRUUTp1UnENs/pjKjpTiwya7yr4NZSVfWNotKjsG5XVFDpGjP0AwLdu75+1+6mxPK5f+97xpLynCWPDdgsZkYddKCY457cB+AqAdeXBV0RQ4VmDPQYG+0wVqRkRxbsXjEElt0lJY2jMUpmyFWBL7dUV9Demw59gSd2Sf3fnRVM013cd3rOEcf9OQj5zBGnNmAPBvshAXwKwuPR+SapIJ3TMGYipn+/d2XpvIl9wcWKkqELO3cpG0V1E+G+c0fc1XR9maQM3LXt356W8J0swP7k1i/s0oBfG+RD4zwz0tclkYWgMcwdjWDIvoVSQiCzee0gmNCyam8D82XFVl6SCZHkGY/iPBPZXdtE96++W3oXHt+c7MdS24T23DZ7cnsdQLq8nubgJwNcZcMXksO5kXMNcKVUkDVXJKmwVRHUM4gx+SyK4ROpEi9A9yOUdHBspqszYCpAqynqN2DfGdPZsWmPitjXvTvXkPUMYv9i4FX2xhXBdN80gPkOeveKM0vvkb9r+Hh1z+mOIGVpbbBUGZ0jpDDGNqS5gEg4R8i4h51eZaiem5rlMdTS+F3sLMVXnhDA0UlS2jSqRolsE6BuWW7wrFU/nIdK4ZW23t4hpDO+JR//jLW9gCT8PY7mTc7km/iXA/gDA7NL7ckuYOlNEMdBrqkzSdkCSRb/J1c9KkIQxZgdDGl6LgFK7gFL5f1Jp4Or3pWK901XsUXV9/ALD8KqO89JPvwp56ffvxsUl52gsY+HocFHVHq3Qr/oQIP6rzdg/9SXNkevO7OvQSMPBu/GZnoaHdo1jtZXGlvzRlZqmf40Bn/T7e0xAqiDzBj0VpF2Qm6vf1BDXqj8CuW/HLYGMU9FSXxXC7xvi/SSl4oiJl0cQCDh+pPQtSsThtTJg0Bib+O/S798NyBddHDtZwFhlFWUMDN9hTPtbztiBmBHDtavfHdGh746nVwWP7y7ixsdM/PryoQsY2P8L0J3yYJ/4Awb0pQxFFnGzPSpICTHOMBDTMJ0wU3QJw5ZbVcooSQ6SFBzVD0Qo+4dQ0gR1hQuY+VKJRyBS9eMqAE6SyUyVROR3smyB48NFlZci53/S9yiA6BfE6D/kkNuZzC3BHVdonRpuYJiJz6ouPLZtDBaBk128QiP2DQDXln9fqXbM6jOVGqLr7S9mk9I5+szpnVRyIZ4sCthljCHKCMIRXpEY0SXkUC9KjZcUcZQRyEySQJj/LIZGLUUczlRLtQvCr4m0P7/9wnWvPrzjddw+wyWNmfN0GsCj28cwUjjJepC+GcBfAqrloPquKhBLZ8oLMthnqgXaiY3WCGEMFV0labg+QdjilIrxbkFJbTG4JBGPQGYKeXh2DRtHTxZQsKfYNaQ++bQQ2p/tjw2/uNSZTXecP3Mres2MJ9IAntyWw2hhVDdIu4Nz/k0Aa8vfjxkc82fF0ZvubFesmMYwYE6vkuRdgcNZGwXXPdVe8F2OkpvZ4Fy9tBlCHtm8gyNDBVV3o4Ix9GUC/mxkvLh+4ax+cf0MTV7r/qfQAJ7cmkMxm9dIFx8Gk5IFW1N6T260ZExTZJFOdt7VJYlCEkZsGqPn0ZyN43mrrWPrJqg2DJI4NA7TJ49uBfONoYeHCip1vgJeg8CfuIX842Zvn5iJtUO7d/YbxFPbcsjncgZxfIQxSMnizNJ7pEK8NSyYlVAekW45pSVZSLVEr3J6jsrFlyueZr94L0NKGaZPHgZnE42kuwle5quLI0NFVYi4At4gwp8ULfuRVH9a3LJqZmW7dt+MN4GHNmdg5jLcNrTfAGP/yS/KOwEpUSycHW+bJ6QkUnM/A9KpYWvQGZDQGRI6h+Y/DkkQY7aDE3kHtmjMpfpeQEnqiGkeeXSjumI7QqknI+MVSWMTCXxlXIw+tii5lK5aM3OaRnffTDeIJ3YUMDw6qqdM/f0A/TWAVeXv96Z0LFC5AO2O3OQTVvS8S8jY4rT7u0SwXIGi6yoRSP697ovbRVeo92r01ogwQcwecZhdRhxecR7C0aEChsetSl64112Irww4vY8X0kQ3zhDvSffMcBN4/u1R7M/FWS/GbmVgfzPZwNmb1pUaUiVxKFDIvZ7UOZI6m6JilAdgiTKicMpUjfLxzeiH0iHoXUocjksqwOvkqDVlDRLwEhG+nEmmNgwIC7ec3f1Rod0zsw3ivjfzGGAnWEYkrgaxvwPo4vL3lWQxJ4FYyPUrmG+LSOm8pgHTEqS8HTnHOY0oIgQLSRxxnzi6wcbBfNKQksbJsamkAeAZIvZvDE3bWDQ03Hl2d9s0Zmx6+4p5Qxh3kxeB8JcAXVT6vXwgvUmphoRPFpIfegyuQrxrkUUJUqqIDJjhwhECWdtBxnaUJNfp2VZJjRrD3Flx9PdWbIx0FWP0F7ZwzlrT/uE1jM5TcIO4fwfBdEZRKNpnmlxKFqrpsReUBaAnoWPRnLhqTRjmYpEEIcnCrNPNl7UF9o0XahpAIwQLKWDENE299A67Y0s2jcMn8pUMoS4BPyMSfxoz4vs2bn8e/89Hb+/MQKfBjJMw4sUhFB1nvs7xNQC3lpNFKq55Bs4QyUKuu7QvVdRLFlKoGLWciCzaDDndBcdFxrLVT+rg/KsC0hrzggZTU7wiUj79DQ3831lFZ+Cy867szCDrwIwijPXbx2A51KMR/i0H+2R5IlnC5IosErHwyMLgDH2mpiSLOjQQhaJLOJKzMFys6F6L0Aa4RJ6aIkm7w25qU+dYMCum4oImrdM4Mfwe4+L/zhdyyce2jXVqiDUxYwjjV5sc2IWsyTn9Dge+ICcY/ikiH4Jk7mRcD40s4ppXuyKh1ZddqZLGCg72ZQoYKthtL4QTYSosITBuOcg7TsekDXlXKQHPnx1HMsYnu1t7wPBH3NV/czw7zp/a3X3l/mYEYTz9dg5HR10moL8f4F8BMFh6T9cZ5s2KoWeqmBcIVCFgXwWpVuhmMrKOwIGMhUO5IvIN1rKIEC4EEXK2q4yinZI2vDQF3+U/NQFxPoCvxrl5neMW2XO7u0vSmBGEcfL4OFb2jl0AsD8DsKz0e8a8Kll96XDa8ku1o9fkSgWphyscQTiet3FgvKhsFlS50nSELoDlCqWiFN3OkUYqqataLNrkFpsMqxljXyvm7NUjue6KAu16wli/PYdESltCjH3NT1OfwGCv14EsDHe77tsrUjqva9PnHIGDWQtHcxaKYmrptpkJVvZ690HZNiwbOdvpWKkAedjJQ2/SgST13usZ8BVOuVlP7Mh2ZGyV0NWE8cTWHEat8QQBvw/gzvKV25P0+oWEkb1o+rU2a5XPK0EVUCk42J/xpYqZsr0ky3IO4pp6Qb04qMS+RGDkggnHe5HwzkVV+YZ7f6/ppz7L+IysDiyfV95xlVHU7YChSS5feegN9FTynLCPw6XPZfPZ2DO7c20fWyV07RN+9BULNh/XOKdPgOHvAMyF/4ATpobFcxOqb0TQB0NMY+g1qhfmLYflqyAjRadSibbugqqTJ0VfpjY/s4vghSx4bhxabhQ8NwYtPw5eyIAV8kCxAOY4YK6jVjVxHWSYICMGiifhJnogUr3eK9kLN9kDMpMg3fDvQX4J8plj7ZVSZVLXVUJbOyHXjWULHDiWVy0aJ/HuXgH8YSqtP0DjBl1/YWfraHS+MEQVaEszEAfpAmL4tyWygO/LnjsY89LUA16LUqLorZFuXo6sI3AsZyFju+rf3UcWzDu+5E/hKnLQxk7AGDoI4/h+GEOHoY0PgWdHwYs5RSBMJcIJ+BWEQVK/91V8mnxdKY1IcjDjoEQabk8/nIG5cGYvhj13CZxZC+Gm+xXBqM8oAuluA7AjSBlDk6Qhprev/qaqWm9wZc+wHKEaQ5etp2Uc+OPMeHE7UrG32zaoKui+dQ5g/bY88vn8bM7dvwPYp0vjlPt47kAMcwbigUu/CUUW2rTxFaTqVDg4mreVwazrJlBJEhxwbejjJ2Ec24fYwR0wDu/ySCI/Dji22rxe53lWmt2pKoXa45I4PAI5/T0q+0meRCElGE1TJOL2DcKZtxTFxWfBXngm7DmLIeJpb2ySOLo4iE3OQkLXEde1tmpZ8lYnxywcPlGYrB5JXfcfXcG/lk6lR69bHY6Rv94xdhWefyGH8WTRcMn9EvfqcapsHDl9/WkDC+cklJQRJBK6VEOmJ4uSvUKqIU5XqSDeiS83olQtzMO7EH/nDcQObId+8ognQRB59gnWhBGTCMIh1N2OzVdHpAJEmg7R0wd7/jIUl5+D4srzYc9aBDITXS11yBmShJFQpNG+Jy2El6h2YnRKlbUhAP8uyXq+f+35sY5NWveseR8/y55A7w52LTj9r/LaFjGTY+m8JBIBqyL1ShZSXD2Wt3Gy6AVhdcfE+UTh2jCGDiG++3Ukdm6EeXQfWCHnbdgAjZHk+GpKo/OvvEakpA/RO4DisjUonH0ZikvXwO0Z9HsldCdxxDWOhKG3LWVe2TMcgf1Hc8jkJ9cGZa8R4fPxROr1G1bH2zKeSuPrGjy2Iw9nPDuHdPwPBvxmaXycM1Uxa6Bytl/TiGue63Q6srBcUu7Skhek8/CIgjk2jON7kNyyAfGdr8EYPgK4TqgeC6mekNMEaUxcQChpRySSsBedgfy565A/6xK4fXO897uwwlhM40i2mTTGczb2H82rhLWyu7pE+A4Y+xPu9A3fdkn7TZBdQxgP7RiFm3cNjdw/YEz1EZkwBw/2mipPJMgWhjEV6j09WRRdgUPZU8bNjkMShevCOLoHqc1PI7H9ZWhjJ70TmvP2PFIhVZRTBtGmoNy2Qnle7IXLkDv/WuTPfh/c3tkTKk03od2kIXFsuICjJ4uTyXmYiP3b/HD8n5ckkuKyde3dwl3jJel3NIwy6yKA/YsSWSgXaoxjdr8XbxHUEjK55zqdjiwKqsR/l5AF81x9+vARpN58CsnNz0EfPubHRkiJoo1dtTgD17ln12g2doExkByz68DYuxN9h/ciseVFZC+5GfmzLgbF07600R3E4UWEOm0jDXmHwR4TubyrXK1lGGCMvpTsL7ywb3B4W+gDqTCujmP9tjwK1ngfU5Wz2O+WxiVJYuGcOPp7glNFvIzT6etY5B2fLJypPSbaDs7BCzkktr+I9MZHYB7d420m3uG4Oylp2AFJAyWJI55E/uyLkHnf+2EtPMsLCusi+0Y7JQ15i0zOUfYMyzlNNZGz/vfksD/n6b7s7avbd+53hYRRyKlONXeAsQ+U17foTeuVagc0jVKFrGnJQkoWOQvZTpOF79Ewj7yD9MsPIrnjZWXMLEVldhzysRnwSaPFa5UkjmIByU3PwzywC9nLbkH2ghtVcFi32DaUK525SLbBeyJ5OJXQlUquVJNTkJviE0Lnj99h/4cHQx3EJHT88Hx6SxY5O7cSxL4Nhuvhk0Xc4Fg6PxlYfQv5RXtVbkjtr1wos1l0liw4mJ1HcusL6HnxfhgnDlSOlegCtGwIrQThAoaJwqoLkbn6Iygu8h1mXWLbSCiXq96Wx2FX9ZrgPpf4F1OJ2NHrV7cnArTjx5Rj2TojfAIMV5R+JwWAwT4T8QCL4aT8it61YLmEI1kL2U6TBecqCrPvqZ+i/7F/hnH8QFfnajCNgekBLyUpRTkOEltexsA9/xPJN59SXqGSLafTKDiual/ZDpg6x6y+WKUyg9drTHyk4Ii2TUpHV+AT28Zh5YsXg+EHYFA1UEtFfJfMS6q03yCQ8N2ntTQRW1X19lynHQXnMA+9jb5nfo747jc9/b1LNsl0IFt4UaEBgwkXIplG9n23YHzdByFSfV2hokj+Thm6qhkaNogIh44XMDRmTd60LzKw3zNjia03nJ0MfRwdW4m/fJtg5wopMPZZsFMBWpJFZ/WZgUVzGpypAji1yMIlLyiro2ThSw/xna9i8KH/jfjOTac8IDMESsoIIXuYuAaWzyL93P3of+R7KnpVSSAdhtSO8rYLuw01NThjSuo2p/bYuUiAPmHbblsKZ3RsNV6YewmuhnVg9FEAE0+/L22o1oZBnFMlI2etzFP50E8UnM7W3GRegljqracx+PC3YRx5p30xFUGCyX3Mwhm2JE7HRfL1Z9D/4P+CcWR3V5CpPGxyjpetHCa8EANNuVonTa/JgE8JUTz/8e2FUMeAThHGl39F2MrOTDOw3wawBKWMPZ1joNcILEArqU9f02LEcjCU72DNTcZUCnl60xPoW/8jaCMnuuL0bBoaAwurpL+fnh/fsQkDD34b5sEdXTFXjiBVhCdse6yc1f4ew3MEnH6vM0D4tGNZoeskHSGM//IbDIz0axlwB07lSqKvx1C1DoOY+LjfjawWMrbAsVwHE8lKZPHqI+h76ifQMqPd4S5tEUo1CXFCiXGY72xF/0PfQWz/tq6QNCxXqOLCYUIVEDa4crNOWiY6GH5DuNYlT20Lt3Bw22f6nh153P/a2ACH86mJojjkTcRAjxGII0Bn09stSvkhHSunJ8lCqiGbnkDvc78Cz2ffFWShILWSkIvQENdg7t+Jvoe/q4zE3SBpFF1XEUeYUE6BlFGpQv4yBvoE2SLUrLS2r9DdAtA0+yoGuqW8zkV/rxlIAyJ5wZTBagZneUbOTgZmMfXkk289g75nfgGezzR+SpbnW5RqYKB74hSYFn7MiJI09u1QhlDj2J6OSxpSrc23wZ5h6EzVs51UnpKD4YMFN3/pE9vDK+fX9hk+3y2mOfAJAPPgr++4qaEvHUwQTExjSExzug0XHWW76JhJkTHEd21E77O/AM+ONbXQmWmCz5oFfelS6CvPgL5yJfSly8BnzwYMo/PEwXzSCBnENJh7tqPviR9BHz3WcUnDEYR8yPYM8mvapqaUemCLieFjjm2HZstoa2j4kzuyyOez6xj4zaXfKemix0DMaL3Ohea3MaylimRtFyfyHaxpIUXpg9vR/+RPoI8cb3yBMwbePwBt/nzwZFKKa6e9rQkBkc3CPXoUYni4o3kYkjDIZaGTlzKEbnsVvck+jNzyWa+yVwcJU6olOndVAZ6woGtc7Zts3ik32MstcKdw7R8/sbXw4o1rgtdO2iphOHYuxcA+Wi5dxEyO3lQw0kVSr50nYvtFey3RKSMnhzZ6TAVlGcf2NUUW2ty50JcvB+/p8ats0ekvSSg9PTCWLYc2b15no0NZ+5JoJS8m3ngW6Y0Pe4WLO/i9yY8EDbNRkrxHOmkgMdWWsQKED7mOFUpcRtsI48mtGbgOPw9gt5buK59pX8rwbBctHgimxhRhVIO8/MmCg/FOhX3LjWzl0fvCPYi/82bjBk4i8P5+aAsXgU2ncsj3DB3aggXgAwOdTRHnIcVlTIZcTJaF9IaHkNjxUsfD6F0irwF0iPfQNaYcBZMyZzUwfMh17TMf2xG8x6RthGHbri6I7gSwHGWVkvvSrXtGVIiuzmrWt8jaQpXX69zWYUhueQ6pN5/192+DX9owlMQwLVmUQKT+Vp83H8yIdUxEZ6yNCXOMg4+PoufZu2Ec29txr5NUTSwnvHwTpqQMXdWMmfR4zwTo9iEKXr5ry4w+9vY4HNdezqAIY+JL9CR1xKYGoTSMOGeI11gcjiCcyFtKJemM3YLDPPy2yjplxVzjG0hKFz094KkGdXNJGqkUWE9Pw0MODMqB075ZJ8ZhHNyDnufvbc77FORYVPazG1qDJFIek4qHbhwMH+wtZhY+viVYj0lbZtN2MgycbgJjq0u/U60I00bLA5BrMWnwmntwpOh0Ll1dnnq5cfS89AD0k4ebs+IzBpZMNXdicg6eSnVURGdtjnKXnJrY8hKSW5/veHS9PKyKIWa1ysfak9K9HJPTeekiDXTFz3YEK2SEThgPbBaArc9mjEnpQrl7lMEmoQdS6yKh1TZ0FlypijgtlZ9sFXLhJnZsbH7XMAYuVZFmN73ZwmeDAG9zHQ9JsIUc0i895KkmHY7PKLoiNAOoF/SoVXIc9AvBPvhbZ+YDLZQR+kwuOsoBMi8EnWqkzBlT1bRa7YuqSelCZ1W3oZQETxaczjUcYhz60EGkX39cdRbr2Kbtgliudu9Zqb5rRw4i/epjYE4H597vZ1NwRGiPQX613pQxNcOb4eqi45zzzNbxwO4V+mPcO3DcAJzbTw/U4qr0WKu7WEoXtTJRc46rupR1BCpPxFZJZU25UMtBBGHbzRsuLavzgVzt8paUQxASbz2P2N7NHZcyLBFeGrxSwWJapfahiwDc4tp2YHpJqLP4g82vI8axhIGuLw8D70nqyljTyhqWZJqoUUGrJF3YndoojKsOZMmtG1rfrESgbAZoRhd2XRXI1WnCaGf3sLKbgo+NIv3a46rJdEdjM8iLzQjrMWgaU1LGpPPTAMNNlmDzz3j404HcJ1TCOG7tBQO/sryDmfxiPQEEasWnkS7GbbdzMRdgSgVJvfEktNETrZ9ujEGMj0NkGlz08nOZDEQmOJG0abDOkAaBIbbrTcR3vd7x2AxHCFgh2jKk1G6apxfYYcD5DtkX7bz8h4HcJ1TCWK1fkRYkbpzoM0JAMqap3JEwpQuXCMMFO/QkoKpQbtSdwS5Sx4F79AhIqhf1XJMxkGWrEHHU+5mw0YkhSNUwl0PyzadV39lOu1mLUuILaV2aBlfOhEmYxRi/4b59I4FEfoY2e49uHQe5fCUYu3yi5aGvjrRq7IxNJ11YAlmng2nrdhHJzc9CywwHt0CltDA6Cmf/flBxGiNeiSwOHoAYHekOsgBCKd9XD5SUsWerb8vosJThChUPFAbkV0sn9cnFghlj7FounIXffeNoy/cIjTC0jJAXv4wBKzARrdy6sVNOSlyr7hmRUsWIFX6KcVUwDuPEftU9PQyIoRNw9rwDMTICKCOa77IsvYRQJOG8sxvuieMdt12chk7t1ZKUsfUFMKvQ8TwTy3VVUd8wLi4l+Jg52T5IZ3JiF//ueXNbvkVo2arFJJKw6TqAJUq/S8YrfZnGYPLatS6ytuhsmwASSLz9CvSRAGwXVSDJgjIZsFQaLJ1Wqe4KtgWRyUJkM4Btd/w0nQxJ88Q64+ZV1ar2bIF5ZBeKS88FqHPtL20pZWik8p+ChPyOujyU4zqy+dO+Xy9n7Lr73hp+UG7NVu4RGmFoYEsEY5eW/q3yPRK6qtfZLGEwv/ReNb6Qkt6oL110LBt1/IRnu1DtAUJK1ZQqhzylpLoxNuoTg999zM9Y7TayUGA+aXSCMRgHGxtB/O1XvaZI6tl0RvoqSRmGxgNfp15+iYahMQZxSvXhYHRZgusLAOxp5fqhHIFP73bhOsVLSwV+4VcJSia0lp6RxpkqkFMNeUd0tnEyY6rGpHHiYHuMayVSoLJWhd1IFOXoZLa9KxDf/Qb0saGO2VNKsAXBDcFjoroGmpoqeTlpq61ybPuc9Ttaq44fyqq28rkYgzJ2eqHgfmCJqU/5Eg0hxr16ndUwZjmdSzDzjZ3xPW+CWZ2NLOxasM4SBjEO/fgh5cHqdJKJIAqt/qdSSxJTpNtBxvA+x7FaEntDIIx/A5ec+QAuLq97IfWqVrwjnHmxF9VguaTiLjoGvzhO7MCOzo0hQm2oHJOC11HO7WAfGh+WEKG4WOV+S8r9dvqhxRlhHSPqa+XagRPGq4f/QurXZ6heCSVDjMaQiGstkbrOWU1XqlRFrE7ljPgwj7wDbfR4x8XdrkaHp0Z56w7shD5+suPh4kJQaC7WhMlhGKfbC4nhbMspLH9px2tNXzfwGRs6Ns4IJKWL2eoXfguBVr0jMV7d2OkSMGa7Hc1Iheso+wWzrc7viq5Gh+eGMegjx2AcfafjaiP5HpOgKUORosGVLWMS5migC9YfvrDpawdOGLrWm2BgF5V7YOIxTRUtbRaSKGq5UouqiUwHXamMQcuPw5SLMEJ3Q6kleZiHdqv2lJ0mMEeIUArscMZUGMMkTkwQ2MXr+kdjTV83iMGVw4E7D4S1EzdgXjBJK2Sus9rqyLjlqkIlnYIypp085FUBj4yd3Q9XqP61vJjt+PNyicKplcE8R8Mku6H8x1qL89nNXjZQwrh/I8FxrDPBsLD0O01jSsJoBWaN2As54dmQi61OBzk04/h+8EKu4ydW16MLpoekWjJ0GFqmO8LmbSECD8iV1zMNPiUrnIDltmstfviVI01dt6HArce2ZCpHyHmNvDB0jFjfADsXQC/KWiC2ksrOfPtFNRQcrzhJJ9URuLYiDGV574KWfRGmA4M2PgJ9+AjsOUs7PRglHcuDr1bIQDPQ1WHNkS+e2rNM1aWh1UgmXnxs8yjK3xCq+76Om1dVL9JVN2E8usPGtrffxhmL585nhAu9gjjEAeaC0WGH2Ka+3uFxBpyr8vD9QUjpQmshBFbnbHIyzWmQ0kXH8kYUGHgxB334KDoU9Tyj0BVzxJiKlVE1VrsAwldL9IAPG8YYEqaGEXaaCznOGHsf2fYeF1hCRKafnzdsc/5W3iq+88z2vLhmdaLiNesijB/nx5DfUeRnLpp7E4A/BlPl9uKn5EuW0xmeB6cfM2A1lWWnxk3e0iIxeHUvpSSKnK+OdE7CgKpOrY0NoUMhYxGagSuUWgLhdIWeJAmDVEuR4CAFlpjJlR2jzLAqb/FJAn5TaQKnipTYOtGOHvC/LxbyP39wt5V//0pzyjXrIoxF6wWyi+zzwfCfAFxS4U8kHX0QDBcSoGrak6pbwWAaWtPHirIN8OqZqZZLSiXpLBh4bhQ8P9YV+nCEekHQR08oNzgZ8Y7LPVItkZKGFvAaMg2uVBPHpfLlOavCXRKMcCmAv3Qhxk4Qv7vS9eoyep48gwxAfAzAdA7cJQD61X+Rlz8iB9wspGRRyzuSd7xqzJ3cpiowLTMCrtKmOziQCA2BiIGPjyh1shuIXpJF0O5VqanrmmdDbABLBPDZ+Xa2t9KbdV0pZfMUA84pb0JUD+RAJbs1a2KQbFvLEJR3RWeDtXzw7AiY23mf/oxAt0yRVCULWd+12unBeAdPGO5VTWOqbF8jYMAqLjCr0nt1XcmFK/WKxkp8MU8c4i2ESes1ojsdv3R7p8GkGJkdVYVrIswsMKuo7E/dYnuSazpoxUjZMaZp9DUZUjlwBatorqiLMKiJCgbKHdrgQCejljZjuaSSdzr+qEmoRcc6b/uP0BAY4NhKyugWCEHlNSwCQzP7UK+iFoSWfcN9CaNZMNROZS+6osPuVB8kwKzgu2RHCB9SjZTPruOHjg9lxwg8gsszDbRaR7eE8AiDM8/Y0rT9AjW7sRfc4KPjmgETAlwlnDUAKnuJslf579/N6JodKsm+wWcXIsgPFQj6mprGlfEziEuHUqLPS2nnyuDS7Bg1Zb+ovLLkpBb9LL/Orj3mSRiuVf17kletTxKC/KkCZQU7VQR2UhMJlJr+cPKyrzUvC7vdDY27AqUpIubPI5vy3gQm5o7UMegtHao9Z0RgTufrYpTDDSEeQ+OexzLfUjVPD+HU9CQvLLVZg2cphqOaRuIKz4bRNZhM3eQRAzn+T9cnDSr7g1qXK3+/VJ5T88pQcsMvR/luJQ/yCUJ4BDFBEnU+bgI7nTw076dHuJOfE4GJDrXSrAJ5GMrDJMimT560H8z1QisCLAfYrNrk2S+q7wmbSFmUu2fPeCNREoQkCauMJFoF+Xwkr20DoggwHeC657cKq85w2yHJwWXeHIoWn2y5ZCJ8ElFSGoFp5BHJBA91zyqCX8iaQhCdDb01B0QJoRCGHJiuBtjcCJkvYVSD7YpQrMkNQ6kOmlqYciMLX6II1QZBHnG4NsAsjzS4OZOI4/TnSiWicFm48yZO3csjDqFOJdLMrgjcKkFKFyriM2DGkBK//Jqt2jFCkzCmtJ5vAGof1vi4JTpSqP50cA5WyCG+5RWwvYfg5tvfnXxC3bF90jA7XnWufghAuDx8opgMpS5K4uBgLgMfHlLNjcgwuyKWpmT4DKSvoQ91gGveAd5qA6XQJAytBUZjqK3O2D5hdKo6uIR+eC9SzzyI+JsvqQpOje9UqrxRmjjtJGm4cggOoMUaDrFrK1RHBIdD2F6Ryc6BgRxC4vknwLJ5ZK+6De7cRf4AO3schVEYWPO7Bba6b0IiDNZySb5qHhLhE0ZHwJiyqsfeeAHpJ++Ffnh//U2DSgux1GhI08B0Tf30WhySKhlHjuOddOUNieokESlpyI/zGKDFu88wSg7gFuQ4GxzYhEdp8maetPwn5gv1fXnGwLNjSD7/CIy9O5C94UMonnsZSNM7ShphxBcpr6PcWO40nqNpEDhhkL/hW+kCx2tI9yKskmbTgWtg2TGknnsIyeceUQtt2mI5pY2v66qtoTZ7LrR5C6HNmQ/ePwieSoOZMU86IaGaLIvsOMTwENzjR+AeOQT35HFQLgu4rq+rTUPEkncKHrPyRPeoKMLyxlV3h8Iy0mSmCZZMg/f0gvX2gyVT4GbcWyhCQBQLoMw4xPioelE+57WKlJ/nfJrG1d4EGQd2o/eu/4Pc8UPIXXkbRLLXr/nZfpSfK0GBK8Jo/TrBSxjkSRit5JDwGi5VdRC3RpJNDIhDGz2B9CM/Q2Ljs/4xXoMsfEKTpKAvPxPG6rUwVq6GNneBWuxM12ufgEQgxwblMnCPHoK9cxvsHW/B3rsLNDY6MaZakBuUBKAlPK9Kx0CeZ0dKFnXZKuTcyQOjtw/6omXQV5wJfclK6PMXgfcNgMUTgKZPGNSp9BnHARVycCXZHjkAe89OOHt2wj18wCNcTDNnXAPPjCH92F3QTh5H5taPw+2f3RG7hvCTMaoXdmgctaT2RhDKUuJ1HIQ1P19jO7kihPDZmoPRoJ08ip4Hf4T4Gy+eOrUqwV/s2tz5MC98H8yL1sFYvFydjg1BEqZhgvUNgvcNwli1FnTtrbD37ULx1RdgbXoZ4uQJf3zVJ1qpADlAS3aINMgjCkkY05KFlKB0HdrCpTDPvwTm2osVYfDe/pofU+tEcrecr0QSfGC2Iuf4uhsgRk8qkrXeeAXW5tcgho7Xfn7y966LxCtPK7vU+J2fgTtrftslDQrYtVqSVlo5xEsIzejZSuBJre/lBbY0fenGwDi0kRPoeeAHHlmgij3Bf8J8YBZil12F+JU3qcUeiAxYGkq6F+Y5F8FcfT7sq25C4bnHYW3cADE2XFPaUQbRTpBGiSwK0/2d8OZ56QrEL78WsUuuhDZ7futzx7kij5h8nXcJnIN7UXz5GRRefhbi+LHqtiHfUh9/80WVazL24d+BOzivrZIGTQTvBSdhKKm/W+MwJJO1Iv3U+qhSSZq/dP2QCy47hvSjP0f8zZerk4VcSJoOY835SN72YZirzlMnZWjQNBgrVkFfvBzW2ouRf+Qe2G9vmdh4lVDyoijSaFO8hopLmS4UWbhgPX2Ir7se8Wtvhb5wSTiWWk2HvvQMNWfmhZcjv/4hWK++ACoWKhOT/5xjWzeix4xh/IOfhds70D7SULEYwV5yRkgYTVcKn0bCCN1xrxorW0g+/QASG5+pboESAizdg8QN71cvqWO3C1JliV14OYylK5F79B4UnnnMM/ZVOZmleiJ80ggv5dDDtDYLf2HoZ6xG8v0fR2ztxYDeBl8w12CcsQb6ouUorjoXuUfuhnv4YJU58yWNTRsgUmmM3/EpkBlvm/ckhMoY3hJukTPCkTDk4Fr4vtPkC7XlmcXeeAHJDY+pFogVT27XBZ8zD6kPfRKxy68Da8eCrwA+OAepj/w2+Jz5yD/wc4iRk1VVFBX7UAzX5arC16cjC84Ru/gKJD/0SegL21/mn8UTSqLR5i9G9u4fwN6xxX9j0qT46kni5afgzFmA3Lrb/L8JdwFShfSkltGimaCEcM6aFpms1hcLPQRDnkIHdyO9/h7w3HhlshAC2sLF6PnM7yN+1c0dI4sSWCyO5A13Iv1bXwCfPbem6KxUhZASNKVWpOIsqt3edzEnbrgd6c/8fkfI4hQYjFXnoud3/xVil1zhr9cKi0tKm8UCUk/eD3P35kDtUrUQAl8E4qYN5du3ar+oKWGEye6q72YOyWcehH7kQOWT2nWhzVuA9Ce+APP8y8IbS6NgDLHLrkb6Y59Txteqln1qMB6iAVDRU30qv+mTxfV3IPWhT4P39AU/gCagzVuI1G99XhlbFSod7ZxDGz6O1FP3gY8Ptym4Jfh1Xo0TG0FI37w1D3JtwggX8bdeQvytV6raLPjAIFK/+Tswz7805JE0AcYRu+waJD/yabBUT9WjXpKFCLhujEqIq3pNzwYUv/ompcKpsXURtMG5SH/897xnWk0XYBzmzs1IvPKUz7bhRgKFoXZ3r0oyE8E9F2rixcfBivmphEGkRP/ErR9B7KJ1nRplXYivux6JG+8ENKPqylOBXUGVgiCfLKqpIoJgnncJknd+ovGYlDaBz5qrbEH6spWVVTo/LUAShn5kf9tUk25DSN+647mkTSH25osw9+2svBiIVIxF4rrbur5/KtMNJG7+oAqAqmpQEHUGVNUBYXsSRuU3XWiLliH14U9DG2i6aXhboC9ZgeSHP6UidCuSBtegHzuIxManPWP4DEOrmaoIizCoSiJm10IFaA0hsWmDvxAmSRdCQFu0FIlbPgwWT3ZqlA2Bp3uRvO0jyntSzQiqNnqrtgzyCgZVfOBSKosnkLz1g9CXndHijdqD2NpLEb/65uoSBEGprPqRfTNOylCPqEWtpCu/cS2yCUVzZIC5YxP0Q/umGrTkojdjylinL14e6G1HRkawb98+7NmzB8ePH4fjBHtqGSvPRvyam71AskqnC7VuyxCO3560EohUiHzs0qtbu0kFjI6Oqrl7J+i50zQlRRpnrK4iZXgG0PhbL/mG5S5LCa6Fri0C3EJs1XQfDTIhx7sgA8tnEdvyKphdnKpuCAH9jFWIXXplILdzXRevvvYaHnzwQbyycSOOHj2qftff349zzzkHt912G6675lqkewLQ9TlH/PLrYb36Ipw9b1cM8yzVHW0qArSWdKEMxLPU5gtKKpPz9PqmTXjggQfw8iuv4OixY3AdR83dOWvW4NZbbsH111+Pnp7WjKp8cI6K03D27/GiQSfbs4SL2LbXkb/0eriz5rXB1986vPil1scZCmEIOpVt18wQqUbmTUDtFU6BcWXEMva9XcEz4kkX8StuBO9tPYrz2LFj+Id//Ed857vfxf79+yHc0/WBJ9avxw9++CN88AN34it//MdYu3Zty/fUZs9DbN21cPa/U1HKKBUrboYwSjVMq8G84DIVWRkEhoaG1Nx9+zvfwb79+xVRlEPN3Y9+hDvvuEPN3QUXXNDS/WLnX4bCS8/AfmOjV7OkHGrNHFA1NFRy2oyAH27eYopKeDaMkEpiNVBPpk4QzD3boY2PTlVHhIC2ZDnMc6brQT09pNj89X//7/HNv/orJUpzzqEbxukvXcfo2Ci+/8Mf4kv/8l/i1Vdfbfm+ErHzLoM2f2FVW4ba9E0wu5JOKl1SqnG9fYhfdrXK42gVkiy+/ud/jr/85jexZ+9er0BThbkbGxvDD3/8YzV3r7zySkv3ZOleb/ymWeFNpqTR2M63VPe0MBBk1XAEKGGEQhiixeSZWp/lQQTET4ApF6r5zraqVnHz/Es9q3kLKBaL+G9///f43ve/D9u2oU0+scpvKXVkTcOzzz6rNsmhQ4daureENmc+zHMvqvp+1Y0/DapKF0LAOGtNIIZOx3Hw37/1LXznO9+BZVl1zd2GDRvwta9/XRFzKzDXXOAlxFUkWgZj305oYydDCeQKwzISRO5caBJGK3UJa31SYwEOmjPlHdGPHaygpwrw3j6Yq9e2LNK8/PLL+O4//7MiC16nZV3TdSVm/+SnPwW1+qQ1DcbZ54OlUpXVElV5trFLTjRlqnQx01RSGUukmh+zj5dfeQXf/d73YNU5d/JklnO3/qmn1Ny1AnlQyHmrciNooyf9mIzgt3eQV/QqQFIgtUJDocZWm8rW+mitBkcNQ4q2xw+qSkuVArW0+YtVQZdWIMXAu+65BwcOHKh5Ok4dGkOxUMBdd9+tjHutwli6EtqceZVFCWri9KnWd4XIK/oTgO1CShf33nuvslk0One2ZeFXd92Fw4cPNz8Arql8E5ZMTf2yfo6Jfmhv4EZPFoZKIhBIa47ACYOVJIwWDsVaH5WEUatnSUOQpHD8sGr7PzVTEdCXrlDxDK1geHhYSRjNxPpyTcP27duxa/fulsagrtU3AG3R8uriW4P9VE7v5Fb+BkFb4NUtbRWjo6N46eWXlXG40Q0k52737t3Ytm1bS2PQFy6FNji78vNzXXXgKO9awBs8aJnFFcHU2AjNhuG2JGFQVdKQUikPopWFH+qrDx2rnKilGyryr1WcOHFCuU5ZE0E+UgQfz2SUdNIyuOZ9n2r1MqoRQBVUDfiSUtviFV7tzRYxNj6Ow0eONDV3kmCyuVzLNiBVrHnewqpzow2fUAmLQRIGU1J0sJThlqT+rgzcIsBxRdO7WtSoecHlggwkws4rkqOyDyffy88bCeKUtG27paAiIYQy9gUBfe4CVYG74uSKBoQgqiEGapoq2BsEpGTR6tzJ+W8FzIypjNZq5fykOsvyuWDL6YWwMUsSRqujDMfoqfTP5hPRyW9IWwlywHoQRiZ5CasInh2vNADwZDqQ2Iu+vj4VSNSMS0t+xjRNzBpszUtTAu8fUIVyKzJ5g8F2Fb+OHxXLZ81paZwlxOPxlueuf6D1Z6jNnuu1QJ8MBvB8Vr2CROChAyS1p+p7qhGE5iVREkaTENPYMcyArNJSJWHFYgXaJa8dQDze8j1mz56NM888sykbhjwh58+bhxUrWleNJFgi5UVdVuOLejukqz+uXAxZzhlPB5O+3t/fj9VnndXU3MnNMXvWLJx5RuuuXXlwsIrxJEz1P/Gym1u+zan7Vasf2wLkfgwiZT60XBLbad5TIr+YW+OjkjBaHzjz+otUSjaT95Y6eACVtOQpecdttyGeSDTM8PLvr7nmGixfFlAOi2GCxWJV80oaQ+UPMDOuXkEglUrh1ltvRSqdVuTZ0OiIcOWVV3pk3SKUl6RKNzQmXM9oHiBUEe0Ar0f+fgzClxMaYThu835f8nWuajA4D6Qpi9/Su+I7rNTCMADcfvvtuPqqK6eEM9eC1N2XLFmCz37mM0gkWzcgQkU082AqmtcMlNECTf+//bbbcM1VV00Jo68Fx3WxcMEC/M5v/7Yi7FahGk9Vc+uSAFNt+1u+zQSCWdunIPeh7TRvUyxHaCX6pAgk9aZmv7pTI0Xe4EzZMVr//kFGjVbHokWL8NU/+ypWr14Npw4jnOu6Snf/8h/9Ea699trgBjKd3tGFOVTz58/Hn/3pn+Lss89WJDqdlCbnrjedxr/58pdx3XXXtW2cQSJwwhCehBEEQpMwXOEZPpvdj7UaFmk8IDsG95shV9gp5NiBBuRcf911+Ju//mtcdNFFE9Z/KWaXDLzyJRe7JJQF8+fj61/9Kn7/i19sKGBpOpC8n11FymmEO2s2jqkutTULqZb957/9W1x6ySVqzirNnePP3by5cxXB/MGXvqSMnoFANciuIuEwDtKMwKRR1T8kQL5gikSFkviDGGJoHXeEIBRtgWaTtD2vbOUMNsnApmQNu5XqLwQYBsgwp/IFY6BCHrCDK3zJGMMH7rwTK1euVBmXDz/yiMpYzeXz6tQ3DEMt9nXr1uHzn/scbrjhBpVQFSisQuV07YkxNnKxyuX2ySqqV5CQc3fH7bdj+bJl+D/f/S5+/fDD2Lt3L/KFgiILOXdz5szBussvV3N34403qt8FBbUWSs2wJ7+naSAzFph4xsGClTCYJ120EhdVjtAIQ0oHlt28ZdZVXdorq45yOuMab02ZkCqPYYJUvkOFhZ/NQOSzXgXuAHHOmjX4q29+A1/8whdUFOLBw4cgXIHBwUGsXrVKqS2t1nOoBpHLAPlsxYXfSE6f97eVS/JTMa/mLgysWbMG3/zGN/CFz38eW7dtU0FZruNgcNYsrDrrLDV3vb2tReZWghgfAVWyP0npxoiBAqzCJsmixZU9BXIfBhEWjjAJQ6Jou2qgzbRoU7EcRIhVmbyYzqFx1gJzeg9bqHL3UxvYUC7rNQUKoXeGYZhKJ5evdkKcPAFRqFDgGI2bcxivQLOKMIpwh08grE4tUuqSxCBf7YI7dNyTMKYEDBIokYRIVHZVNwOtxTajkyEP7KJ/cHdtX5ISSqJQs+O0a6jCMc6UHaPp5yRPB92A0z97qtKoFn4B7tHWU8u7Ce6RgypuoCJ4AwuqViii48A98i6aN8f21kG1Eoc9faB4ZSm1GQSWJ+XDMw0E14QmPMJQupPwrLNNzoFTI2FG5wzxVg2CmgZ3zgKQXiFc2nXh7Nvd1q7docKxYVepugVfYmhUwqgIqUoe2BO4HaNTcMdGqhMgg+rsTvFEII1E5LkVSBSzD89bSbBrnbwNIjTCYH4sRdFqnt1cKjVfroyE3rodw5mzUImVlU4IZ/87EKMnW7lD18AdOgb34L7qBs8GuVf9faVLMQb38H6Ik8ebG2iXwT18AK78LhXKN4LrsOcurHzgNIHADZ4ALHlou40f2tW+TV2Ewb1A4IZnREoHRat5w6cKOKlho0jqvLV4DBKqJqMzOHfqA+cM7rHDsPe1nlreDbDfeVvZMCoaPHkThMGrSBmMQQwPwd69o/nBdgvk+nt7MygzPtV+IdWReALuwuWBuVQ1HjBhqP3n2REbvapV5QN1EYbhcilf1nPUyl2Xm0gFIaDgD7gZqJBWUT2k1VBqSQtCEhFEMg17aYV8A8ZBuQzsLZs8g9cMhlQP7M2vKQ9GxcXdBGGoz1RMr/DsP/bWTTNeLRHjI7C3vVWl6JCAOzgb9rxFgfU1DCYL+xTkqApF0YxWPUIaq+jqqpMwjBwx8bi80DR/egxEvwQwqv7FgIItlC2jWd60aqRdS+kiaWitqSWaBuuMcz3X2OQbEWBtfhXOsRaqNnUBnIN7YW17s6qRQm38JiaxImF478Da/hacQ63V1Ow07Le3enasKgYbe9lqiHR/IIFqQdsvAC9DtdC4ScAG8JhOOFHpzboIg5mcOMQDAP4ZQIV8cAVJEt8WjH1LEYe/Bh1HeHaMFiI+p1NLWrIsE8FZuALOvMVTDZycwz16GNamF5u/fqdBAsWNzys1oWLxHAbwJn2gkjAqSiacK/VH3jeUrsJtgJTGCi8/4wVtVSjfKA+Y4llrg8nNUd6RoPKjPDA//kK+Jl1WHvoVyUB+BMADxPE9GGZFd1pd3/bqtUk8+MaRk4LjLwzB3yASt4FjjhoXKfvGEU3Dr/NW/m5dS0hK2w5AOcrlHswVBXqbDPmUXGEJQkyrPJkJjSOmcWQdtzlOEgS3bxDF1RfA2L9r6vuui8ILT8G88HLo8xc3c4eOQp6QVmnjVliQvNqmrwPKjmFULwYs7+tcdjX0Sipfl8Paugn21jcqx2kLAXvhMthLmitbUAl60PYLBuQtV3lJyiBH+6Cu4Veuy24HaCUYvKdPGAPYc4LhJ2u3D+1d+rHKfXDrVppm9Q0grsWHfvXU738bXPuiS/g0EX1KEH2aC/q/bv4vqe+lEulRjXgewGvlhtZ80VXiUbOwpnGvplpSS0idiMVzLobbP6uylHFoH4rPPz7jGvCSVUThmUfhHj9aVbpgZouNbYwqn5fzdvwICs8+BgowxL4dEJkxFJ5+xDN2VmidKaWK4jmXQPQOBOJ2Z2HYLwSQLziT+czSdbxwy9pZv9TA/zUj+jTz9vCnBOFzFud/Y2r63l3nV6+YVvcoL18Ww83npHHHVf8RmqaN6oZxWL4M3TjMNGP8l/8auHHNADbtzbnEaOOEvYN5npIKolHdcASpVzX0GLw1/U+eGAuWobjm4qnvqRrtAoXnnoC1/c3m79EBWG+8rLp3VYOULJpVR+q6BhEKLz0N661gGjK1BUQoblgPa8umygZiqcLOXYjCuZcG6h0JOv7CdgXyxSlkNuw6eP3B14R8aDndMI+qfayrvXwyGU+4N5/bhxtWVW8P0bAC9pFLaoczX3jmIBwnK1WSPQAGmF/tR4pHyXhzsq/rqyVmFbUkrnOlmoyJJtUSCc1A/uKrENu6EdrJE6efyIxDDJ9E7td3KbWEDwZTgi5MuEcPIvfI3d4pWSXAjbcoXSgw7zrCruC8Zxw0Nobcw3epRtZB1EgNG/aurcivf9BLPKxU14Nz5C+4Au7sBYEF9emB1Xc5hYLlVjqkd2q6/vbt5zYf8Bh44NYt5yblxj4EwqbSElJ2jLzTUiOVYg21RGMMabNFbwkJ2IvPQOGiq32ymByXwZVOm33wF6BCrpU7hQ4pUmfv+ymc3W9XJQtm+IQRAJhe41oah7NzG3IP/Ey5qbsZ7omjyN7zY2XorkgWwoW9aAUKF14VaDq7EbQ6QkA+707Os5K/ftkGDbVy7VAiPdOHZ2XB2IuS6NQvGJArui0V8bCnUUvShqZS3lvJLYGmI3fZ9bAXr/Dy68shn6wUV59/wjuBWqhmHSbIKiL/yN0ovvJc9T9igBYLtOMkeKya99G7SeHFp5F77F6vzkgXQpJs7v6f+obOCl+ECBSLI7/uJr9jezDShcaCVUfgR1hnC+5k+0WGAS+OpvtbegDhVNxKqeTxlwAcRZmLJ190myZmyRWFGobTmMYVabQEIZSomb32/aBUeqoF3M/GzD30K+SeerDrFj9ZBaWG5B9/wGsSXGWyJVmwgNNJlS0jVu1NBlgW8o/ei/zj93WdEVRKPrn7foLChidr/BGhcO5lSh0JEsGVm/TAmBfdmbem7LXdBGxKtpgkFwphvO9KBo3FdvpqiYIk5EzOaYmYpVpSLbdEzk2fqbXO1lKKWHMpcpdeNyFVnAbuRYDm7v0p8k880DXRjFJNyv36V8j/+i6/SE7lR6tiJ6pt7BYhCaOqaiLnLZ9D7v6fIy8ljS6ZNzE+guw9P0T+qYerFsmRqoizYAly197pZaYG5EqVS9VsJVK5EgjI5it5JelVXdMOfHhZa/cLLfmMZ90MGJ72I8cUcgXHi/pswVtSrCFlJHSOtK61xqG+6Jm77gMqNqPi4mBcGRNz9/4Y2Xt+BDE23ModW4YYOobML76nJB+5KWslmGmJUJqN+zcAeLxGXIcKt88id//PkL3rBx1P7HOPHEDmR/+E/PpfeypmRbIQEOk+ZG76qLJfVC3V1wR0zlXAVpCQeySbn+JOzRKxZ5b05ls2IoVWQIcl4gKU2wAmDgJYrqRSmxRpxJqstahi411CokpBb6kPSilj3HZb61QtVZO+2cjc+jFo4yPQD+yeagSTJ2ahoMRs9/gRpN7/MejLz2r+ns2ACPbOrcg98FNYmzd55FbNgMb9zRxqyaRTpOTkqjSXkfNWLCL/+P1q3pJ3fhzGilXhDmoyhIvi5teQe/DncN72e69WcaGSYSB39e0orn1f4EWSpXQRaKwWAwpFV6n+p9unaL8Ae/Gk1WzBzFMIrsLsJHzvH/4Sn/vDr+YEicsAqFbewu+50JPUm+4dKfy+JNVUD4Mz5Byh1JfWvCYE0T8Lzqx5MA7sBs+MTj2afZXFPbQf9va3QCRUlywWC6YtQC2I4RPIrX9AndTOnp1+FFaVb8y8TVzVxhAwmE/oVM0uXJq3wwe8eROu6izfjnlzjx1WRuHcPT9Wz00RbBWygKYhd9WtyF7/4UDrdsKXLhK6FngP1eFxC+M557S1T8B9RZg/vPWcvpYt9aERhsS/+PzXiw535oDhRjlHzDdeppM6jCZ1N/LrHsarxGSUDEiZFupwnLoZqfR3t38WzIPvgGfHKpMGY6DMGOwdb8HZu0v1NNH6BlTbwKAhxkZQfPlZpXcXNzzlp17X6KEiySLuk0X4HRVO3VavgzT8eXO2b4a9d5dywfL+WeHM2/AJFF5Yj+zdP0Rx4wYvR6Ra/xRfUstdfgOyt3zMq/sacE5MXNcCt184rsDx4aKS5MuWQ4aB/Xe3SK/95H//fy3fI1QBVTMNQaLwNIB9AJTcadlCGT8TZvNcVXAFkoIpaaISegxNhYtL1aTlPUKkwoBHDRO99/8A+qE9VQN6YNuw33oNzq7t0FechdglV8BYvRbanAVgRvNBD6pc4LHDsLa+AevVDXD27lRivdfKvsai4z5ZBBGg1QRKEo1bqNH7UqooTmnetkFfuRqxi7150+fMV93amoUkBffIARW1WZTzdmCP8taoPhXV5o2EKoiTX3cjMjf9JkSqL1C7BXzV2Qw49kISRC7vqujOSWfHVsbEC0YsHgjjhb6MHn0rkxSi8C0ifA4+efekdCydn1Qhsc0ibXD0GtUnfbjo4FC2GFxrEc5h7tmG9K9/CnPXVu931U51+SVJALqpRG19+Zkwzjgb+tIV0GbN9Xqc6vpUyUB+TriqQjVlM3BPHFFl9ZydW5Xk4p445hnnqonRZVBuznhwwVmtQFiAyNeRBa6+v1AkoeZthZy3NdCXLIc2a55qJM10Y+r3l5tcdeuxIbLjat6cvbth79qm1DUpXXgekGnmTc59IoXsVbcje90HfI9I8CUapSqSNII9q4kIh47lMTRul29qF6C/c3Xja3ee2x+IWyp0wti2+wT2jNPHAfZPAPrhx84vmZdAb9poWtLTGcNgrHoOiUuE/ZkixqwApIwSOIc2dBSpJ+5C4vUNYMVC7RO+RBykyl2rjvCsvx/awGzwvkHVtJjFExP5KlTMQ2TG4Y6c9Cp8jw57Xg9lwcf0C94HMzzJImwDZyOQqombr6GinPbH5fNmgKdS4H0D4INl8xaLn5o3KUlkxiBGhvx5G/HmreQmne409+/nzFmA7I2/gcKFV/pl94InC6ky95hG4Lkj+YKLvYdzqkJ42RI5CuCzMXH00RsuOjeYewVylRp4ekcBmczoMq5p3wdwDXw7xKxeEwvnJFqyEvcYXL2qQZLFgUyxZl3QhsE5WD6LxGvPIvnsQ9CPHT61maeDWph0Sh9mZVIK+f9XGmqp538jE+RHXFaPuuws5P4TBU/iaMh+WGnelJG3/H3/vxudN6lu6AaKq85Txk1rxdmnrhkC4rpUl4Nn8uPDRRwZKpz2OwIeZOCfddMDJ+88I5itHvoZdO2qOB5+deQgwfk1gMsBKCF5POeoiLREXGv62eRdQlyjqraMtKGh19SUehIY5IkWTyF3xa2wl56F5IZHEHtrI3hufPpFWmsht/I8mZ/PEWs9+zRMSBJTcSA6IIp1ShuYZt7Q5Nz5a86ZuxD5992A/KXXq3iLMKvEa4whFmDryxIcR2AsY08ueZJnhPvStjZ8dUBkgbD7kpSgxXWHET0MYD/852vLL5m1WyJyRxDyNfJTJI8Mxg1FKIGeF+TVDbQXr8TYh38Pmds/BqSCdbvVC7n55CbUUt1NFhPws1vleBV5hOqnqwGNoXDRFRj59B8he+0HfONmuC0lTK3FMgwVIK+WyTtTQsEJ2EagpygdjLGzhLYQxs3npJFnsc0EPDohfBMwlnVaqpMBX8qwpinhJ0kjFN1LCIhYAs6KVeADCegpUpshdHWA+XaKpE8UscCSJ9sG5geSTRBHk3VFG76vynkhaP0GCu+7BvayVd6NQ7BXlENJF3oI0oVLGBm3J3OdlN0eEpq265o1wbqo26bpfvj8noJLdO9EvU/m5eyP5+yWDmaXCDmnemVxiYGYrtysYZz/jAjEuYoI5GWbWG0CI0DyKKkdcUBPea9utVU0gpI3R81ZEuEQLj+dYOWzQUxXwVisTRXhJVkE3dVMXi5bcJArTE40o4Mc9P+z9yVQclTnud+ttbfZRyONdoSYRUIImc2WhYUWsEViHib4OAbs45N4g2MH85I8h9gJNjbmxA7BwHNYEoixXjACO/HDBssGIctgErMaSSCBJCQQQpq1Z6ant+qq+nPurapRa6ZnNEtVd8+ov3OaQV3dXcu997v//v9cVzXfs/yKZkd/4JXX0SzNeZ5Av2XAlfx+uWDA2bEmqkJRJj9LMpaNkDV6MBdXSRpDKrKWPWaK/ORAjkVdUd2oMjdoiS9uz9hvun8tNx6BhtnUhjWpH2qMzNyoybxXKeIpigHhANIcCxd/Tvkv/szIHqfG55k7pDGeHSd5WXXGrQhQJafurN+wbEe6ME+MaiYi9mtbrX71w23+31/RCOPP37cct790sLddrvopGC4GUMvcep+JlIn6am3SEgDngKRpQ5PkgjVbOao0GbWmgu60/ynpxyff8HT4vMmKPKKwh/37+MePN0VmE29fOFMw4pnZJ/6FfdyphHyClfLIgo0hP/PfUlTxCvxemOMZ8builgjUSlnCfjHslztsSf3PS7u1tK8ndFFUT32jHiPLYk/LoP8CsAlu2ns8kRP5JVORMgyLkLZsREf5Df5QG3QV6ZyNwclWGC8Ecnqb2OOJSGQjJ/IpyAcTAztOHj7W+xESISf6oME3scmmQYwFLl3EE4awYZwY+seelmD990+bKZDZVVQN+FNnzuKyejeBHvH6mzhBJyYGUlNzffLHkzTH7mGiyQyzwuqobthJn5tLGGqoFE6SCiYFR410JIzgBk1mTEgXfi9bLwx8cOSa6QHhkRfS/xa/siWYrajoJjNNVYmBPQngOe89i4D4gCH8yVO5Ta7LDZr2mOHgMU1Gg69eEwJJMkgtUipoBb7A5iqkrAbGF8wN0vLbjQq3o1nvgCGaLJ9o68R2yWK/ba3a6Ps5PRSdMC5uj6BKo6ME/HhIyoBT87M/mZvy+GVMGrOUHz9Xva6gWvNRGxNeEr2iX0wXEECqJog+KKiiwZb/v+/FXQxPYQfQZYMeshQ5/ollK30/r4eSOOWYotlkS79kwFDTDLKB3oEcjClmmDqqiT2masJZf1ZEFZZrf1L4ZNiqXtFIphFIC4FkJRCVhKsiYUUJJDbGdKWLYSX4bID9GjLboanB2mVKQhirz6iBoutdNrDZ6wrvVQuKD0xdyuBkMZgbvS0B3BaLUy4aDHe3kpiYgBURY5qAwZEIfU4xR8CqCP/FgWQOydSIAr8dJNPmTX/fEN+4LOL7efNRsrAfRVHIluwnAfzKczQSHI/JyECUiSNj2UgNbxUw/Bok5sMSJ6fRkRaafuGWpyiEZ1YNJupNV+RAIjpFNfCcjZ5+Y3gypc3A/r+VTj7zr1uCry1bMsJY36Lj0lWzemyZHgRwDF47AtN2RK4pBlgJ1SQ3etFgcnu2+lT/2fGSCMKoKCblDybGi7iE4eNwKZIUiFcEbipFPGE4rTpOPHSACJujsfrU57T6AM58IkoaWPzi8zaytvlbEH7qFPtwxa7BHBLJEUadCYMzcSJnwyyQ4ZbK2Rg0pn4OAS7iaq6IW+GL8ofEYIt6GgU63E0SMmOIBBD+Dc8pkDbRN1JdN8DYQ7KivLSuvcr38xZCSQnjvPfJqNGr0mBCytjjvW/ahJ7+rJA2pgouRfQbtvCccGGDk0jatEXKu19h4kLE1ULTP7HjlAANSRi+tTt0q2gFEaAFN0iLqyLGyLCDVxiZD8mKVrQmLyWf4etbY0jJyk4C+zdOpHAHIJm2hGriB7haEjcs9GT4y0afYYMxyVdd0xFxS5WrXcG4QQAxyVcjtSjoG4DdwkP/YA79yRFBWn0g3KdC27e+Lfhq6x5KThgc1aqSY8zeAhKNjwSEAXTAEE1Z/NgIuFbCVRP+IteIFPaxcjOpuqMTV1D2YJLkmxtcl4OzW3htD7v7jeF9dvgk3moy/FwOKUVVgstihm9srcLL/z54hGT8AMARDBlASZQey00xAnQ0SFzvVBVhrJoqbE0XzZwrRozyh4jM1aa+K6uSM3/8TizzYFmE7r6sCDcYdoa3iPCDJWc1dK1tLY7twkNZEAbHBZ9pgpKj7cREBKhIKeUPKZE2haQR1DIUxip16sYqUriEUQZVdwOqRTmT4NQvmVrqtyzIQg2MLLyYi77BEapIhoh+OCjj+WMHit/UumwIY0N7FMbsZNKycB/Afu+9z+d/T78hEm2CinJQJUkUZp3s4DMuIXIRVy4tYRBjhdPsK8gDifYOthqa9C/wzSUqJNPgyCKdtdAVHxHRyf/xNDE82KA3GOuXFj9/qWwIg2POW4uQaIjsJ6K7hipzuapJZzxbyErsG1RZEpLGpEjDbSPgFNEp0WJ1i8Jg1YcgzV7ge/MdX+D1fi2xrYcT+2STBT01Vg3wHiyb0MVVEWNEAONhEO5sjpnvrn+jLrDzj4WyIoxzLmSoZhoZdu4JG/SjfNUkmTaFPjelJssngS7LLmlM9JsEW9acDMgSgmwb0qJ2hC/7LOQ5iwIvajsh2DZYrBryuetBsdrAa2iOCU6sqoaJloaWhGThf4vD4egdMNA/UhVJA/QvhiJvz7II4bK+QK9hNJQVYXD80QIFkUjVIBjuAfC7/GPxgRz6T+zs5DsEaSjKxElD7FqlIwzGGGwzh8G+OOQzzkb4ii9AXniGK2mUWEWxLbCaeoQ/cg1wzkbkiJXukrgkpihuo6Lxf01ybV1aABmoHkQmasos5BXh+JUF9mCVGjLev6QmsGs4GcqOMDgubovByDUeIOB7AA5575s2oSOeFYVPAyUNt5XduNUTkYA2eTHXL9iWhe6Oo8jlTChLViDy8S9BXX6+E1BWClXJbX0oz1+KyJ9cB+2CDyOZSsPIpEuadsMlQVsef/EcjyyCSFf3wB9HNmejozfjVNI/8fAek+F7lx5qOLyutXgxF4VQloTBUVWVhUzSNiLikkYS3kM1bHT2BmvPwAnqyXjOQqLBr62GSpqvKoHQ+c4hpJLicUFuPg3hK66FvvpSMD1cXLuG6CimQF35QYe4lp0v3u7p7EAunS5pop6QBMdpoC4GWcC1W3TGs07i5YmH4jZwZ5ZZzz/TWrSAzlFRtoSxoSUEPRTO5oD7AfzHUK4Jc7qm8Yc71QS1k4FPkqiqjMvl6kQPllbCUGQZx94+iK7OjqH3pJpGhC79NMIf+xzkuacNNWEKDK5UITXORXjTp4RkIc87XRzK2TYOH3wLOSMjVKhSwYnKVU4qYHjekKDJgj+y7v4s+hIj3KQ5Am2WGB6OhKLmh9om79nxC2VLGBxr28KI6OFugN0G4Pn8Y/EBQxiHgha0NVkaH2kIwiiduEguYaR6OrFr56snXpqqQztnAyKfvAHa+y8Bi8ScRsV+EgcnIssCC0WgnXsRolfdAP3Cy8AixwOLEolB7N/zGmRh8CwNYTDk18IYu8hSVFMCN3DCjbfo6TMKDcdvJJvdqYYifZvOiAV+HeNBWRMGx8b2GCLM2knALfn2DC5cdMWzGEj43zZgOFRZQkxTx3alMSYkjFKaF0VncJjY8fQ2ZLIjg3qEivK/Po/In14P9az3i8UtVIepeCxs2zFqhmNQV64WpBS+4jrIC9tGJOO9/fYhHNyzG6EAmhFPBLY2dt6PJkmIqScZbx/gef86ejLIjWz5+RqzpVu+c/bLBza0BFsUZyIog9DEk4P0KCnZ5JMmSd9nwDe8niamRcJIpCgMsbAS6GL1dpx0zhQNkUaAsZIX0SEQ5sRCePm/n8Ou3btx3jnvG/EZpmhQ28+Hsqgd5oFdMHb+DuZbr4ESfYBl5jU+LtAAeXgXdVWFVDcLyukroK54P5TFywRxjIYdO3Yg3dOJUHstqIQRqU791cLjpIt4nODCvT3wX88YFjp6ssiMbBfaScB3SbJ/d8vrF2JNoFcyMUwLwljbGsWONwcNK515EIQFYPgSH1t+jD/sYz0ZzJ8VRkgPph2iB0+nlZiFjGnhxOZlDLYeKSlhcKmruSqMROfreHjLFqxaeRYUpfAQc1VBXbEaats5sI69jdyB3bAO7YXVdQQ02AfKpl3pw/2CxABFE1KJVNMAefYCyIvboJy2HHJDsxO0NgaOHTuGx372M6zRJLFzBxlPMxZI1PMMu56j48TvpaiLRLIijKFh8nmbxeBIj1+SiP1gkOUebQxVm2tK7BUZjmlBGBxrW2LY/maqbzCTuF0leSGAP3GnsbAsH+3JYN6sMDTVp8K+o4BPprBr00iZ1gkTn7yyb6VaDESoD2mYG9PxyKOP4sqPXY4PfOADY39J1SEvaBEvyqZg9/eC+rpgD/SCkgOAlXPuSQ9DilWDVTcIQyqrqhXSynjx2M9/jl1/eAVXrVkKmWHMequBgjnFc4gLUO41iKK9RfCEeBAekd6ssF0MI4scQA8Rwz21oVh6XWv5qCIepg1hcKxriWDnS5kjR+TkzYxRA3/Ls54lUqaQNJobw0JFCRLMjdWQJCZUFK9CubC+MwmMzJIY9cjdJZfNqsHW3+3F7XfcgdbWVtTXj690G9MjkJsiQNN8X69rz949+Od77wUzDSyujZY4jIy5xmlnfLi0E1FlXzKWxwNbkEVGGO2HHyLglxZwa5KqOj/RWp59bsre6Dkc++vSqIprrwHsGwB25h/rG8wJm4ZlU1GWK59sMU1FSHbEWFFYVpJLGljJd+/ljVXQdRWPP/EE7rvvPuRywRuGR0MikcDt3/8+Xv3Dq5hbHcG8WKh00gWcHjK2FnIkRUVGTPOnvMF4wCXA7r4sevsLePcI/wXg5lmmfnBBrPTxFqNh2hHGFUvqwOo1MiPaszbR3wPYl3+cM3dnTyaALu2FIQJ7NAUxVYHERd0iibWjgd91W30Mc6rCSKXS+Kc77sC/P/QQLKv4yWiZTAZ33nUXfvzww+LfS+tiaAxrJTR4kpAAZT0kxqsYxs2hMwuyMNDVZxQgTNppgb7WFDNeOlCfweql1UW5pslg2hEGxwdX6ogylWSb/ZKIbvaK7sBdMD2cNHozQvwrBphrXQ9HIq7xr3RbKL/l+VUhtDXExG7a3d2Nm775TWzZsqWopJHNZnH3PffgtttvF8QlyTJWNlUhpARrYxoT/MSSDD0cDaQVwKindUs0dMWzhebkARBuSqeSzwzaNbjqtKaiXddkMC0Jg2Pd0hD0cCRnZ9gjRPRtAEPhjd4AeepJsSCpOpgSXL/O8YDvZDW6igua64QrmC/Uw4cP46+++lWxgJNu2HiQ4CR1y3e+g299+9vo6+sTRtO6kIpVTTWlb/WkyGCinmdxBomPR09/Fh29BSOT3ybQ15MmfhGLVdtrz4gW5ZqmgmlLGBwXtUcQqQkZKsk/ssn+B9G92oVHGp1FtGmILu6lTnF37RgfmFuPhrAuJqwsy8Kt+Xc33YS/ufFG7Nu/P5hzE+Hll1/GX3zlevzjbbehf2BAnNsmwhn1MfEqbUEwEqntIpekCNdxfA4WlCze4xudRew/6mNR8yNnBd9TxA9Ma8LgWNce45JGyrJy94Lou4VIQ0TSjdGg2R+Qo46UQcUriwjLZ1VhZVP1kL7MF+7AwADuvvdeXH3NNfjX++8XJOIHOFHs378f//Dd7+KTV1+Nh7c8AiOXgyRJLoExfHB+PZoiWsniL5wLdWthKMF7IPh9dsczo0kWRwn4Zlqy/l8oFDHWtZWf+3Q0TCu36mi4sC2Ep/b0p8xs5m7R6JSx/wNAUDafn70DhohgntMQgqpMtGzK+MFkRRRmKTX4PdfqCi5e0oTt73SLycuEg8DZH1548UXs2bMHWx55BFd87HJsWL8BixYtgq6PfyFxkkgkEti7dy+2bt2Kx37xC+zevVsQhZxn+OWfa4zouGhhoyiaWyxj9KjXrWqgkwSZTQXMq5gVzzp1LUbe7zGb0a2KxTZXRWKZjWUYazEWZgRhQOSc1GD7nv5ENpv+vyCZwNhfA2hEXps5vnA4aehBBHd5u5eql00h3vULG9BaH8NrXQOiaK0HRVGQSqfx1LZteObZZ7F40SKcd955uOD889HW1obmOXNQXVODcCgEVVWFsTSbzSKVSqG3txfvHD6MXbt24fkXXsCrO3eio6NDfIYThTzMS8Sf+bnNtVjVVD28J2gJQI4EGKBhOifKSWacxMiRpzgC4FvMZD/SI+H02tbyt1kMx4whDI6d+/fjzCWnJ7NG7p8ZkGMMfwNgyOzcP5gT7N/cEEI4gDBykuWSp7h74Dvb4poILj19Nvb2JNx+X8fBpQ3+Mk0Tb+7bhzfeeENIHDU1NWior0ddXR2i0aiQOvhnOMEkBgbQG4+jLx5HMpWCbdtDv1MoBJ0vmIiq4LKlc1CjKyWXLsRDUDUwWfV97EXt2ZyNY70ZURWuwO8fBti3sqa1OVoVzaxtmX5kgZlGGNd/9Bzx9+m9ycHBdPZuldlpBnwNwFDoYiJlwrLSQtKIRfy8facaNXzsqDXFqxEqwBUtzXhs/zHs7U4UrHLNGHOkAlkWBMAliO7u7sJSEmPi8/zlEcVY4BLF+XPrsH5hY5kIXeSojLL/wXWprOnkhqRG1OLk2E+Em3Ky/JNwWDc2TlOywEwwehbC+rYoYmE9Y1nshwT6WwIOeMe8Eu5HOtNOc1s/J44kl7xMXz64NNVWH8XH2+ZBkU+uhnlEwKUFRVVHvhRFkMvJiAKuKlKtK7hq2XzMjuploI44IC3kezsIvgm925kWfwtgNxF9VYL5aLUeMi5ZVh51LSaLGUkYcEkjFI5ksoweItD/BvBK/vFszsZ73Wl0xTOi98OUZQK3ZydKnOI+HBJj+ETbXFwwt66oMSn8VBef1oRNS5pK7jUaAh8XPj6iFsbUrom5Bt3efkNsPunsiJKRNoDnQOwv+tOJn4XC0dxFLaWvmDVVzFjCgDCEhlEdiloZK/W4CXwFwA53IMXc8eooHu3OCAKZYu8zEVnppE77dQdTB9/ZF1SFcO2qxZhVpJ2eP9cldVF84ezFwltTatPFCRiqtjV58LmTs2zhMj3akylUX5aLGr8khq+8p9T/prG20S52S8OgMKMJg2N9SxThcLX1R8sHfkvAlwh4lAsY3nHPg3K4IzWaSDl+8Jmkh9xKU+WzSvhO+JHTmvCZFQtFwlyQV8ZVkSpdwZfPWYLz5tQUVaoZD4RKwqZmw0ilHZW2q88JyBpezwIMPwLh+tnp9AvNGmhje/nmhkwUM54wODa112HrgTmYk1V3G8T+CsDdAPrzP5NMW0IP7Sk8CcYJJiakUxPDr6ufOkiUnWP4/MqF+OgZcwSBBHF5nHxlJuHTKxYKNYiVsP1IQTB3fITxd2JXxtz7608YeLczhf6kWegnugH8k2GzrymKcqCnoRGbWspI3PQBM8pLMhY2tUZwV2cvVnSZ76Zz9A0myQdB7AYAi+EKB1y0PNqTFUbRWXU6QtoEXa8M7oSUyq5VIVdFZkd0/N3qFiQME08e7BT2Db+ms01Og+I/bZ+HG85dgpgql42h0wENFQJy9snx1zFlboUsvpn0DuSEe7jAc9sLYt8zrNzD1ZHq1EXt0ysga7w4JSQMD19uqse+dBNULdSvEt0DYl8B8Pshu4YrvscTORzuSIv6GnzOj39ReUa18nysfAGfXhvBty5sEwZJclUIP35XlRiuXjYfX1/dIlLYy4ssXIFCVHYPjXtAvY8Npky825EWqekF8pIsgG2XbPaltK5sDumh1BE75fvllwvKc2YHiM+dy7BxWS30cMTY06k+RoTPAXiIz4v8z6Uyjp56tCc9IYMoO0lF6lKD747t9THctm45rlm+QNSwnOziJvf3miI6/vKCpfjGmlY0R/XS5ouMBckljHGMJhNRm7ZIXjzcmUYiXdC+xdXafyGwaw+dZWyLauHcJWfW4urljUFcfVnglFFJhmNNSwxP7UkT2bTLytl/DWa/DtjXAmwB8rwoPX0G0hkLjbU6qqOKKMs36nogcuIwZM9tV576KyeIRdVh3LymVVTnuv/Vt7EvPujYIKSTqymcEPhLV2SsnlePL65ajEsWzxI9PMpOssiHdHK3N3PVq0G3+bcgikJSJsN+EO6CLW1WdSXevjeMC5eXPo8oaJyyhAHX7crx9K7+Y6k0u00OYScj/CVAawAMZSglMxYynWnUxlQ01GiiOvmoULlKUv6PlS/supCKL5y9EGvm12PLnvfw60OdONiXQtZ07C/568rjAYkBtSENZ86qwmVnNOOjp89Gc0wfIpGyBpf8+PiMcp1ef9PeAUNUbssVjs/JANgGYv8oE3tWCYXNde3lVdk7SJT/zC4CspEqhKRB49iRmscbmrreZJCuA8PVAGbB23VsEhMpmTEFadTGNFFs+MS550oYAWZD+glvga9orELrB1tw1fL5eP5oHC8d7cP+eBLxjIGMZYuQ8piqiHqcyxqrRDLZyqZqYUSFG3dR/iBHVdRGBk8JadIipwNZvyHUURSWD98j4AFidP/83tSh3rmNuKjl1CELlK3MXCL8+rWsKDWfysZjErFNNsP1DDg/X9ogd5eNhhQ01mqigZJQU+AY1aS+Y9A3fxus87AbUTg9wCeC7N6HYdkYNCwkc5Zo2sQJI6LIiKqyKMevuG0Cylr9GA6yQdX1MK66Eda8Fqdbm6t+pDKmIAonz4gKaSxZAM8wRncyQ3vSCOUy7y5pwBcjp97yqUgYebhkuY5HX9iJusjiwVd27f3Jme2n7SawzzLgkwCakeeP55MrbVjCrtFQrSGsK068luzUW5huU8kzYHIojKE2pAiVhQ0dJ7fpGSE3jXjiBLjlB7wojIxhCTdp/2BuKFqzAFm8Q8BmInqgoaHxrYF4ArVSCH98CpIFKoQxEh8/7yzx9xev9FAItKeXWV8Py+oOybKvBeFDAISD3RNje/tzGExZqIkpqKvWEVbKo4jOVEDuf4IJ7yoRyKmIRqqGrGGifyAr3OfZnD2aeToBYBsBP5AZns3AyBhZAxevmDlRm5PBKedWHS/+eFUD+vUQoqFIetPyHz5myezzRLiZgD35UT8ir8C0RQn5d46l0NFnIidNb8KYqbCYgu6ELcbpWG9WkAVGkoVFwB9AuJEB183pyT7FFDUzT5+LC08vn0zkUqEiYYyBy9ucVOQdb6QQSSQODyjybRroSTD250S4nAFz8z/PRdyujIUaU4VuuzPx1JRcyw6MgIytoGOAkNOs0YblEICfMOBBi+zXNcj24+vm4utllH1calQIYxxY2xrBjURY99qAyRheNixjnwTpcYD+DMAGALXw7BtMgiXrQgYhrwF6RY4rHcjtuWxBjAtJciGy6Aaw1WL0QzD2nCapab4wdM2okMUwVAhjnLiVMdzq/v+2N5OJeL/xRFi1f8+YfYkE9mkAqwFUC8LwXHfkuvzdNIaKtFFEUB5ZuLAU3Y3CHbLNxAn4jQ1sBqOnGyy1PxlTsWEa9AcpFSp73ySwoSWKpjodqqb0bNra+GML9FkQbgDYdgJL2uqwfAU+cS3nNZPsiOUKThLeKx+CMARzi5DuJ4jhy8xmX5zb9M5/arLW310lV8jiJKhIGJPEh5Y62Yj3HXkPp/WGjuy05QeWU+5XpKgflrOD1xGTzhnxJZc4OJlUJI4AYB+X6EbBAEnybxhhC9n2UwsPNnYebR3A0c5VuHRlZTDGg8pT8gEPPLUP85pnw4aEubduUuc1Nd0hkXntSb8ouTaOyihMDZ7qcRLpzdCqXzzSdsk1zzVf+cayBQYkTcG6ikQxIVSmqo/o/LOPgGy5RY7KWySNnT2uL7E80qiMxsQwTqIY+niWOqxk5jNq63lb6/72lqCvbkaiYsPwEUcHeyTI8kYi1jLuxU+j69wVjALvmU3QJkRgs0gNfczc/+r0Lt1dQlQIwye896kNmB2b3QyGy0GYeLklcnVwa0LFoE4t5BPFRJ+RQ+ASA7vEIuns+OcvC+QSZzoqhOET1m7eBkWS1zKw84dijSejYuQtiomI2zMa+V6myZCpZ2R2sJBJ7Mp4OlkJ25wE/icAAP//iFU60gIwwN4AAAAASUVORK5CYII= -- href: 'https://example-gitops-server-common-example.' -+ href: 'https://hub-gitops-server-mypattern-hub.apps.region.example.com' - location: ApplicationMenu -- text: 'Example ArgoCD' -+ text: 'Hub ArgoCD' -+--- -+# Source: pattern-clustergroup/templates/core/operatorgroup.yaml -+apiVersion: operators.coreos.com/v1 -+kind: OperatorGroup -+metadata: -+ name: open-cluster-management-operator-group -+ namespace: open-cluster-management -+spec: -+ targetNamespaces: -+ - open-cluster-management -+--- -+# Source: pattern-clustergroup/templates/core/operatorgroup.yaml -+apiVersion: operators.coreos.com/v1 -+kind: OperatorGroup -+metadata: -+ name: vault-operator-group -+ namespace: vault -+spec: -+ targetNamespaces: -+ - vault -+--- -+# Source: pattern-clustergroup/templates/core/operatorgroup.yaml -+apiVersion: operators.coreos.com/v1 -+kind: OperatorGroup -+metadata: -+ name: golang-external-secrets-operator-group -+ namespace: golang-external-secrets -+spec: -+ targetNamespaces: -+ - golang-external-secrets -+--- -+# Source: pattern-clustergroup/templates/core/operatorgroup.yaml -+apiVersion: operators.coreos.com/v1 -+kind: OperatorGroup -+metadata: -+ name: config-demo-operator-group -+ namespace: config-demo -+spec: -+ targetNamespaces: -+ - config-demo -+--- -+# Source: pattern-clustergroup/templates/core/subscriptions.yaml -+apiVersion: operators.coreos.com/v1alpha1 -+kind: Subscription -+metadata: -+ name: advanced-cluster-management -+ namespace: open-cluster-management -+spec: -+ name: advanced-cluster-management -+ source: redhat-operators -+ sourceNamespace: openshift-marketplace -+ channel: release-2.6 -+ installPlanApproval: Automatic diff --git a/tests/common-examples-blank.expected.diff b/tests/common-examples-blank.expected.diff deleted file mode 100644 index e69de29b..00000000 diff --git a/tests/common-examples-kustomize-renderer.expected.diff b/tests/common-examples-kustomize-renderer.expected.diff deleted file mode 100644 index dd709677..00000000 --- a/tests/common-examples-kustomize-renderer.expected.diff +++ /dev/null @@ -1,14 +0,0 @@ ---- tests/common-examples-kustomize-renderer-naked.expected.yaml -+++ tests/common-examples-kustomize-renderer-normal.expected.yaml -@@ -10,9 +10,9 @@ - GIT_EMAIL: SOMEWHERE@EXAMPLE.COM - GIT_DEV_REPO_URL: https:///PLAINTEXT/manuela-dev.git - GIT_DEV_REPO_REVISION: main -- GIT_OPS_REPO_TEST_URL: -+ GIT_OPS_REPO_TEST_URL: https://github.com/pattern-clone/mypattern - GIT_OPS_REPO_TEST_REVISION: -- GIT_OPS_REPO_PROD_URL: -+ GIT_OPS_REPO_PROD_URL: https://github.com/pattern-clone/mypattern - GIT_OPS_REPO_PROD_REVISION: - IOT_CONSUMER_IMAGE: iot-consumer - IOT_CONSUMER_YAML_PATH: images.(name==messaging).newTag diff --git a/tests/common-golang-external-secrets.expected.diff b/tests/common-golang-external-secrets.expected.diff deleted file mode 100644 index e80e716f..00000000 --- a/tests/common-golang-external-secrets.expected.diff +++ /dev/null @@ -1,11 +0,0 @@ ---- tests/common-golang-external-secrets-naked.expected.yaml -+++ tests/common-golang-external-secrets-normal.expected.yaml -@@ -6337,7 +6337,7 @@ - spec: - provider: - vault: -- server: https://vault-vault.hub.example.com -+ server: https://vault-vault.apps.hub.example.com - path: secret - # Version of KV backend - version: v2 diff --git a/tests/common-hashicorp-vault.expected.diff b/tests/common-hashicorp-vault.expected.diff deleted file mode 100644 index d9923297..00000000 --- a/tests/common-hashicorp-vault.expected.diff +++ /dev/null @@ -1,11 +0,0 @@ ---- tests/common-hashicorp-vault-naked.expected.yaml -+++ tests/common-hashicorp-vault-normal.expected.yaml -@@ -341,7 +341,7 @@ - applicationMenu: - section: HashiCorp Vault - imageURL: data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAf0AAAHhCAQAAADO0a/jAAAcYElEQVR42u2dB5hU5fWHzy5NmoBd7LH3WIIFFVS6goJRFBVjwYqKIhgbCFgQO2LBBoKKBQWFYIt/E2NHo4SABERAaui9LOzm/41DCLA7M7ffr7zv++Qxj48c7j3n++3Mztz7XZGh8h9EdMyhIodKGY1AdMoylXvFAFqB6JQD5Ff2lhKageiMJSrzG7mHdiA64z2yiZ1kFQ1BdMJVKu+b0Y2WIDphN9mCbWURTUG03kUq61txJW1BtN4rpRzVZBaNQbTaWSrnFdCe1iBabXupkEoymeYgWutklfEcNKc9iNbaXPLwHQ1CtNLvJC9NaRGilTaVAoykSYjWOVIKchxtQrTO48QDL9EoRKt8STxxiJTSLERrLFWZ9kh/2oVojf3FM3vKOhqGaIXrVJ590IeWIVphH/HFjrKSpiEa70qVZZ90pW2IxttVfFNbFtI4RKNdqHIcgCtoHaLRXiGBqCYzaR6isc6seGMOL5xL+xCN9VwJTLFMooGIRjpJ5TcEzWghopE2k5CMpYmIxjlWQnM6bUQ0ztMlAkbQSESjHCGR8DtaiWiUv5OIGEwzEY1xsETGQWzegWiIpSqvEfIYLUU0wsckUvZg8w5EA1ynshoxvWgrovb2ksjZQVbQWEStXaFyGgM301pErb1ZYqGWLKC5iNq6QGU0Ji6nvYjaernERlWZQYMRtXSGymeMnEOLEbX0HImVYplIkxG1c2K4jTm80IQ2I2pnE0mAr2k0olZ+LYlwKq1G1MpTJSHeotmI2viWJMYxtBtRG4+RBHmRhiNq4YuSKAfIBpqOmLobVBYT5lHajpi6j0ri7C5raTxiqq5VOUyBnrQeMVV7SipsJ8tpPmJqLlcZTIkutB8xNbtIatSU+QwAMRXnq/ylyKWMADEVL5VUqSLTGQJi4k5X2UuZdowBMXHbSeoUywQGgZioE+LfmMMLZzEKxEQ9SzThI4aBmJgfiTY0YhyIidlINGI4A0FMxOGiFUcxEsREPEo043mGghi7z4t27M/mHYgxu0HlTEMeZjSIsfqwaEl9WcNwEGNzjcqYpvRgPIix2UO0pZ4sY0CIsbhM5UtjbmREiLF4o2hNDZnHkBAjd57KluZcwpgQI/cS0Z4q8jODQozUn9PfmMMLZzMqxEg9W4ygSMYzLMTIHK8yZQitGRdiZLYWg/iAgSFG4gdiFCczMsRIPFkM4w2GhhjaN8Q4jpQyBocYyjKVIwN5jtEhhvI5MZJ9ZT3DQwzsepUhQ3mQ8SEG9kExll1lNQNEDORqlR+DuZMRIgbyTjGaurKEISL6donKjuFczxgRfXu9GE91mcsgEX05V+XGAjoySkRfdhQrqCxTGSaiZ6eqzFhCG8aJ6Nk2Yg1FMo6BInpynDkbc3ihFSNF9GQrsYz3GSpiQd8X62jIWBEL2lAs5DUGi5jX18RKjmDzDsQ8lqmMWMpAxouY04FiLb9h8w7EHJaofFhMP0aMWKH9xGp2kVUMGbGcq1Q2LOcOxoxYzjvEeurIYgaNuIWLVS4c4DpGjbiF14kTVJc5DBtxk3Ps2JjDCxcx7kAXfCyUf8rH8lf5lyzTuGbWNTJNvpAP5QeZKxuYXl4vEmeoLJMZuA+XyMtygdTbood7yNUyStZpVTPrFLlfGmxx4+k20kqelJlMskIn27MxhxdaM3KPrpA+eT4C2lNeDPCaGkfNrAvkKinOUbeK+sEym4mWs7U4RZF8ydA9+L7sVLCXh8qPqdfM+sFW7yIq+pynP1Pdwi/t2pjDCy0Ye0EflkqeermtjE61ZtZX1eu6Fy4P/SuFTbYQBxnD4PPa00cvi2VkajWzjvLx6tWGezg3Okac5ARGn8fhPt8I1pZ/plIz63TZzlfl25nwr54gjjKM4edwmtT03c39C7yRjqPmfz3Jd+X3mLJa/85yGG/8chjsEQz9E6+Z9a0AdQ+XUuev0zhMHOZpYl6BE3J+QZafnWVlojWzrpcDA1V+xfEpPy1Os7eUEPVyBn/U4kuJ1sz6ZsC6Jzo94xK19h2nL1EvZ/DdWs5NtGbW5gHrFstCh2fcV5xnp4JvKF3zxxDdrJNjC7Q4amadEfAXiQwvOzvjlR4uq3KA24j7VpfGhGFyYjWzPhKibldnZ3wbsc9eN7aIwEcUJpFPE6uZtVmIuhc6OuFFas3Dr1xL4Dfz1lC9fCOxmhlXyzYh6p7u6ISvJfL/parMIvKbvCFUL/8UQ82X8nxlGAY3H8Y2S6132EQHIr/Je0N9JLcu8pr5HpW6MFTddk7OtwNx35xKbN6xyRdC9LFjDDVFfshzRVqVEHVd/EVvssc7Jx3iDEK/0Y9DdPGjGGoWy5I8R7tfiMouPpDlDKJenu+I/a+uDXCbTZZjY6gpclzeow2zl+w452b7HTGviGbEfqNnBezg2zHUFOmT91iD33G+h4OTbUbMK2Y0sQ9xAU6DvPdBBr2op7jAVl1rZEcu6PHoaCIe7K2lO5bKob57V0XGR14zQ+GN0x8NVLeW/Nu5uR5HxHPzCsH/1RG+O9czhpqZHyhTPXw2sUeAync6N9NXiHc+DnF+A4dg3/628LR9tv9vlB/2dKx/9r2T/FHOPXG5VK1tyMtTxH7j79DHeu7Z4bI88pr5rhIo7+O+6u4kM5yb51NEuxC7s1XzRmer10YvHOHjwRZea2Zoq97Kez/aOz3X3UXGOjfLdWpdQ0HuJ/YbXSXnefhKdHnkNTOPSOnpe9/E16SGp2sPXLxf435i7YUdZAWx3+TQPPvr1JIH826jEaRmhuPlL4GOdZK0y1u3nvRVv3S4N8MVak2DJ24l8ptZIs/IyeWu/K4jV4Z4gGXFNUW9bp8p74Q62rFykWxf4a8lvWWpoxO8lUh7pbbMIfJbuVhel/vkBvV2/Q9ytwyP5NXzfzUvlK7ST8ZE9JpcKp/Lo2rBXyLnyDXSSwY5/ZTdOWo9g2euJuxoiVcTZz9U42nsaIUz1VoGX1zAskELvIAo+6WSTGLhoOFOYmOOILRi6aDhtiLGwRjL4kGDHUuEg9KE5YMG24QIB2cUCwgNdRTxDcORLKGNrpe/ySDpK13kFnlIXs6zS266NbP+LK/L43KbXC995Dl537lbdDMeQXzDMZTYy7vSXuqW68zucqV8o1XNjNOkewV3pm8jLdSPAJceqD6U6IblYE+bUNjrxwW2dWorE7SomXGedM77ZJl9ZIgjW7FsUOsWQjPA2dhvkBs99KeKDEy5Ztb/q/C2na1pkXdXf1scQGyjYE8nb/T8j4qI90+Ir/X43iiOmlmf9LxJ1wHWP2lpTaDdCqEC7nXyFd/fV0NdU6qZ1d9W3/vIQqtndy+RjYodfO5EY4M3+u7SkFRqZvxWqvus2zjQFiNmuJyNOaKku3Mf7vmnRoFdDuKomX0vcWCAyvY+b687cY2SWrLAqegHe1jDNYnXzPh8oLrbWbpnzwK1ViFSrnLqe/xgVJHpidbMWBL4I62eVs7uSqIaNdUc2ru9feAu9Uu0ZtBfI7LsZ+HkZrAxRxy0d+aS3bqBe9QowZpZbw4x0X/xQxu8UFzggZK2+LcQPaqc47uQOGpmDXPV2mOWTW68WqMQCy2ciP6gUD36R2I1s1YNUbezZZNrQUTj42sHoh/uSS0fJVYz4+JQdX9v1dy+Jp5xcoID0e8SqkNDE6uZcWKouidbNbcTiGe8vGN99G8J1Z/hMdR8LeexTglV9zSLpvYO0YybY62P/kOh+vNtDDU/yXmsK0LV7WDR1I4hmvEzxPLovxyqOwtiqPljnqOtGaJuV2tmNoRYJsEBlm/e8UOI3uwbQ82asjbP0TYIUXmwNXdZHkAsk+EJy1/3dwvcmd4x1GyT91h7Ba5bLPMtmdcTRDIpds/7OuTudeBFMi2Ga8sHFrhh1/Vva9aq9QiJ0cfq6H8TsCsXx1CzjizOe6xlcrTjb/f7EMck2V6WWR3+toF6siDymiL3FDzW9wLVPdSSDTqXedqRECLkFqujP0Gq+O7I0Bhq1peVHo721ADzG81VGBCMWtZ8SFSxA332o0cMNavK556Odbbs6rNyN0umNJ+NOdKgk+Wf81/roxedY6gp8oLnY/3K133qZ1izG38nYpgGVQvsH2P+t8VdPfWhkvSVsohrZrr7gq+j/avs6LFyB1ltyYSmh7pvEUJwnvUX9Q6RGgV/G/9L5DUzVT8PEITCn/VXtmo7zvOIYFoUB3xAlEnOkWtyfjy3rfTx9DGcn5qZr/PuCVQ1855ikOyV56qDDvKTVR/FsjFHijRz4CbezCMs+0mjLZ5uUyynSP9QD7IoXzNzyW4bGVjge/xCrpPX5eKtdqIvkt9JL5lo2VSaEb90+cSJ8GdcLv+Qj2SovClfy6KIaw5TffwxwmskS9Xr+2cyXAbLGPk+sqPVyU+IXto0dib6qJONiV76jGAhYsKOIHY6cDRLERP2aGKnB4NZjJigg4mcLuxv+eYdqNelVvsTOX14nCWJCfk4cdOJnWWN1cttvXwi3eUcaSj7ykHSSNpLH/m7hjWzTpX+cpE0kcNkb2kgbaSzjAx4kZB+rlFrDbSit7Wx/0U6SZ0Kz7m+9Ai4b0EcNbM/TgbmeBRXNTlLvrNgGr2Jmm5sF/IqND1dKV0K3BO3vTzi+ead+GpmfVv2y1u3SM5VP3RMnscitc5AO2628ALeIzydeZu8D8SMv2b2+r3unuruJH8zeCI3EzMdqSmzrAr+t1tdCZ+PQ2ReajUzlsiZnutWlWGGTmRWqKcOQIxcbtUde/V9nXtDWZdKzaxX+6pbTb40ciaXEzFdqSJTLQl+iRzv++w7pVAz6zO+6+6ifgyZNpOpAfY2hMSw5bHNAwKd/ZeJ18y4ULYNUPdS42bye+KlM8Uy3oLgr1KvikE4NeGaWYPtSVsp75P89HM8G3PoTlOnrxf7LNGaGRdL9YB1Oxo1k6ZES38+Nz76jQOf+y2J1sw4LHDdegbdefE5sTKBUwwP/tKtts3yw0EJ1sx6YYhJfWrMTE4hVmbwttHRHx3q3OcmVjNr/RB1exkykbeJlCn8NtBlqLr4TKhzH5tYzewNrGE+/DLjISplaj2BMQwyOPp3hzrzUYnVzDg3VN3WRsxjEHEyib1kvbHR7xzqzAfHUHNQnq+8wnCCEbdL70WczOJRY6N/Z6jzHhNDzRF5bv4NQ0sDpvEoUTKNXY19stsToc57Qgw1c1/Rty5UXf2v6Fvt+xnCoAF3Gxr94SHOuSjHbjjDQ3VyWp6j3T5E3Tss/9wFUqKeLDEy+rNDnHOjGGrWz3u0rSP/5UQfl6g1BEZyk6Gv+0cFPuMXY6jZKaavImtG+KCveLyJCJlKDfVqZ2L0ewQ831p5dtbpEbiL7+Q91pnql4xgtNP+3VcNImQulxkZ/XkBd4N5OIaaBxe80v6CgLP5SvMpXEZ8TMbUzTtuC3CuRxcI6W2BOvhWwWOdEuj+AN1f89mYw3jaGRn9JbKHz/OsLeMir+nlbv2MXX3XrS2TNZ9AO6JjOkUFI6Gn3/m6D76afBx5zcw1kQs8XvF2mq+6xfKu5t0fF/gTDNCIZoZ+zv+6VPIc/JGR1xSpKz94PtaFcqCPiTykfe+bERs7+MjQ8P/Z0wMf9vH1VJs/e3yIxEHyL1/HuthjXKrLq9r3/SMiYwvHGXs9/09ycoFfZy70feFSoZrZqkt9H+sGuUO2KVD5SCMevnUckbGH4QbfxDtGjs5xVi0CPx4zd02RM+T7wMc6Uzrl/GT8ABlmxD4Kw4mLTRxh0D5wFfmj+v34dNlr46vqTnKS9JOfI61ZS/aTs+W5CC6CWipvSEc5eONFsFVkN2kovSN6hm/8bvD4YDIwhhcM37Hvf8FaE0PNeB6BvVYWGbdf0gtExTZ+IyWWhB/js0StE7COR1jaWMBHiImN7CKrWNyYx6BPKALt6cHyxhjumATtqSuLWeCY86KkukTEXm5kiWMObyQeNlPdwOe6YxLOCfzYUDCEP7DME/te3yT/QDRsp5JMIeyxXs1nolN83NEIxtKW2Oe5hr8o1DX8ptqWWLhAUZ7HR7pgXHfumetYNuZwhZYOBz+u+/VNtiWRcIcPHQ1+XLv0mOyHxMElTnQy+HHtzWe2JxIHt3jTueDHtyOvyb5JFFzjMCl1LPpx7cNvsqVqHYBzPOdU8ON7+o7JPkcMXGQvpzbviOuZeyZbotYAOMlDDkU/riftmuxDRMBVdnbm2vXZIbpU39KerFTzB2e5y5Hoh9tgepqVPbmL5e8ydWSRE9F/IlSXvrSwI4vU7MFpbnAi+neE6tEICztyA0vfdbaR6Q5Ev3OoHg22rh/TCz4iDBygowPRvztUh0Zb14+OLHsQqSyTrY/+M6E6ZNtNzpPVzAEUZ1kf/dGh+jPXsm6cxZKHLEVGPPo5jEtDvM4dZN39i2zMAZtobv3rfuPAvbnFsk40Z7nD5rxvefQfD9yZz6zqw/ssddiS4y2PftDnydl2x/7xLHXYmtctD/+AQF2x60q+11nmUJ6DLd+8oyTAK55dd+2VqhkDVMCzlr/uz5H6vvrRUNZZdf7PssShYna3bKmX91vZwXM3DpF5Vp37OjVfgBz0s/5LvmlyhKdOtJHllp15P5Y35GZ7WWF9+FdKF6lWoAuPSJllZ71CnRVAHu5w4ibeX6RTjvvV60sPWcZty+Ae28q/Hdm5Z718It2lnTSUfeVAaSTtpY/83dJz/beaK0ABOjsSfZfszLKGwmwjMwmLVc5kYw7wxsXExSovZkmDNyrJJAJjjZN8PF0YnKc1kbHG1ixn8MNYQmOFY1nK4I8mxMYKm7CUwS/vERzjfY9lDP5pQHSMtwHLGIIwjPAY7TCWMATjYNlAgIx1AxtzQHCeIULG+gzLF4Kzp6wlREa6Vs0OIAQPECMjfYClC+HYwbrdalxwuY+NyABycDtRMs7bWbYQntqygDAZ5QI1M4AIuJY4GeW1LFmIhmoyg0AZ44wCm44C+OBCImWMF7JcIToqyURCZYQT2ZgDouVMYmWEZ7JUIWq+IVja+w3LFKLnNKKlvaexTCEO/kS4tPZPLFGIh2OJl9YeyxKFuHiVgGnrqyxPiI8D2bxD2405DmR5Qpw8Rcy09CmWJsTLHmzeoeXGHLuzNCFu7idq2nk/yxLiZ3tZRti0cpmaCUAC/JG4aeUfWZKQDLVkPoHTxvlqHgAJcQ2R08ZrWI6QHFVlOqHTwulqFgAJcgGx08ILWIqQLMUygeCl7gQ1B4CEaUX0UrcVyxDS4CvCl6pfsQQhHRoTv1RtzBKEtBhFAFNzFMsP0uNoKSOEqVimeg+QIi8Tw1R8maUH6XKArCeIibte9R0gZZ4kion7JMsO0mc3WUMYE3WN6jmABtxHHBP1PpYc6MF2spRAJuZS1W8ATehOJBOzO8sN9KGmzCOUiThP9RpAI64ilol4FUsN9KKq/EwwY/dnNuYA/TifaMbu+Swz0I9iGU84Y3U8G3OAnrQknrHakiUGuvIFAY3NL1heoC+nENHYPIXlBTrzLiGNxXdZWqA3v2Xzjlg25vgtSwt0ZwhRjdwhLCvQn/3YvCPyjTn2Y1mBCTxBXCP1CZYUmEF9WU1gI3O16ieAIdxDZCPzHpYTmEM9WUJoI3GJ6iWAQXQjtpHYjaUEZlFD5hLc0M5VfQQwjE5EN7SdWEZgHlVkKuEN5VTVQwADOY/4hvI8lhCYSZGMI8CBHaf6B2AozYlwYJuzfMBkPiPEgfyMpQNmcxIxDuRJLB0wnZEE2bcjWTZgPkeyeYfvjTmOZNmADbxEnH35EksG7GBfKSHQni1R/QKwhP5E2rP9WS5gD7vKKkLtyVWyC8sFbKI3sfZkb5YK2EVdWUywC7pY9QnAMroS7YJ2ZZmAfVSXOYQ7r3NUjwAs5ArindcrWCJgJ5VlCgHP6RTVHwBLOZeI5/RclgfYS5F8T8gr9Hs25gC7aUrMK7QpSwNs51OCXs5PWRZgPycS9XKeyLIAF3ibsG/h2ywJcIPDpZTAb7JU9QPAEQYR+U0OYjmAO+zD5h2bNubYh+UALvEYsf/Vx1gK4BY7y0qCr3qwM0sBXKMX0Vc9AHCOOrLI8eAvUj0AcJCbHI/+TSwBcJPqMsvh4M9iYw5wl8scjv5ljB/cpbJMdjT4k9mYA9zmHEejfw6jB7cpkr87GPzv2JgDoImD0W/C2AFEPnEs+J8wcoAMxzsW/eMZOUCWtxwK/luMG+C/HOrM5h2l6lwBYBMvOhL9Fxk1wObsLescCP46dZ4AsAWPOBD9RxgzwNbsJCssD/4KdY4AUI6elke/JyMGqIhtZaHFwV+ozg8AKqSLxdHvwngBcrGNzLQ0+DPVuQFATi61NPqXMlqAfFSSSRYGf5I6LwDIS1sLo9+WsQIUoki+tSz437IxB4AXmloW/aaMFMAbH1oU/A8ZJ4BXGlgU/QaME8A7b1oS/DcZJYAfDpYNFgR/gzoPAPDF8xZE/3nGCOCXPWWt4cFfq84BAHzzsOHRf5gRAgRhR1lucPCXq+MHgEDcZXD072J8AEGpLQsMDf4CdewAEJgbDI3+DYwOIAzV5BcDg/+LOm4ACMUlBkb/EsYGEJZK8qNhwZ/IxhwAUXC2YdE/m5EBRMM3BgX/G8YFEBWnGxT90xkXQHR8YEjwP2BUAFFyrCHRP5ZRAUTLGwYE/w3GBBA1B2m/eccGdYwAEDnPah79ZxkRQBzsofXmHWvV8QFALDyocfQfZDwAcbGDLNM0+MvUsQFAbNypafTvZDQAcVJL5msY/PnquAAgVq7XMPrXMxaAuKkq0zUL/nR1TAAQOxdrFv2LGQlAEhTLBI2CP0EdDwAkQhuNot+GcQAkx1eaBP8rRgGQJKdqEv1TGQVAsrynQfDfYwwASXOMlKUc/DJ1DACQOK+lHP3XGAFAGhwg61MM/nr19wNAKgxMMfoDaT9AWuwma1IK/hr1dwNAajyQUvQfoPUAabKdLE0h+EvV3wsAqXJ7CtG/nbYDpE1NmZdw8OepvxMAUue6hKN/HS0H0IGqMi3B4E9jYw4AXbgowehfRLsBdKFYxicU/PFszAGgE2cmFP0zaTWAXnyRQPC/oM0AutEogeg3os0A+jEm5uCPocUAOnJUrJt3lKn6AKAlr8YY/VdpL4Cu7B/b5h3rVW0A0JanY4r+07QWQGfqy+oYgr9a1QUArekbQ/T70lYA3aknSyIO/hJVEwC057aIo38bLQUwgRoyN8Lgz1X1AMAIrokw+tfQTgBTqCJTIwr+VFULAIyhQ0TR70ArAUyiWMZFEPxxbMwBYBpnRBD9M2gjgHl8FjL4n9FCABM5OWT0T6aFAGYyOkTwR9M+AFM5MvDmHWXqzwKAsbwSMPqv0DoAk9lXSgIEv0T9OQAwmicDRP9J2gZgOrvKKp/BX6X+DAAYz30+o38fLQOwgbqy2EfwF6v/HgCs4FYf0b+VdgHYQg2Z4zH4c9iYA8AmrvYY/atpFYBNVJGfPAT/JzbmALCN8z1E/3zaBGAbRfJDgeD/oP4bALCOlgWi35IWAdjJp3mC/yntAbCVhnmi35D2ANjLuzmC/y6tAbCZw6W0guCXqn8PAFYztILoD6UtALazT7nNO0rUvwMA6xmwVfQH0BIAF9hFVm4W/JWyMy0BcIN7N4v+vbQDwBXqyKKNwV+k/j8AOEO3jdHvRisAXKK6zFbBn63+CQBOcaWK/pW0AcA1Ksto9T8AcA525HGY/wcAxiEwaH/ifAAAAABJRU5ErkJggg== -- href: 'https://vault-vault.apps.foo.cluster.com' -+ href: 'https://vault-vault.apps.region.example.com' - location: ApplicationMenu - text: 'Vault' - --- diff --git a/tests/common-install.expected.diff b/tests/common-install.expected.diff deleted file mode 100644 index f002ff47..00000000 --- a/tests/common-install.expected.diff +++ /dev/null @@ -1,43 +0,0 @@ ---- tests/common-install-naked.expected.yaml -+++ tests/common-install-normal.expected.yaml -@@ -11,14 +11,14 @@ - apiVersion: argoproj.io/v1alpha1 - kind: Application - metadata: -- name: common-install-default -+ name: common-install-hub - namespace: openshift-gitops - finalizers: - - resources-finalizer.argocd.argoproj.io/foreground - spec: - destination: - name: in-cluster -- namespace: common-install-default -+ namespace: common-install-hub - project: default - source: - repoURL: https://github.com/pattern-clone/mypattern -@@ -28,7 +28,7 @@ - ignoreMissingValueFiles: true - valueFiles: - - "/values-global.yaml" -- - "/values-default.yaml" -+ - "/values-hub.yaml" - # Track the progress of https://github.com/argoproj/argo-cd/pull/6280 - parameters: - - name: global.repoURL -@@ -40,7 +40,7 @@ - - name: global.pattern - value: common-install - - name: global.hubClusterDomain -- value: -+ value: apps.hub.example.com - - name: global.clusterVersion - value: "" - syncPolicy: -@@ -63,4 +63,4 @@ - config: - env: - - name: ARGOCD_CLUSTER_CONFIG_NAMESPACES -- value: common-install-default,openshift-gitops -+ value: common-install-hub,openshift-gitops diff --git a/tests/common-operator-install.expected.diff b/tests/common-operator-install.expected.diff deleted file mode 100644 index 7cfc98df..00000000 --- a/tests/common-operator-install.expected.diff +++ /dev/null @@ -1,11 +0,0 @@ ---- tests/common-operator-install-naked.expected.yaml -+++ tests/common-operator-install-normal.expected.yaml -@@ -6,7 +6,7 @@ - name: common-operator-install - namespace: openshift-operators - spec: -- clusterGroupName: default -+ clusterGroupName: hub - gitSpec: - targetRepo: https://github.com/pattern-clone/mypattern - targetRevision: main diff --git a/tests/interop/__init__.py b/tests/interop/__init__.py deleted file mode 100644 index 890362ce..00000000 --- a/tests/interop/__init__.py +++ /dev/null @@ -1,2 +0,0 @@ -__version__ = "0.1.0" -__loggername__ = "css_logger" diff --git a/tests/interop/conftest.py b/tests/interop/conftest.py deleted file mode 100644 index fb301d57..00000000 --- a/tests/interop/conftest.py +++ /dev/null @@ -1,2 +0,0 @@ -from validatedpatterns_tests.interop.conftest_logger import * # noqa: F401, F403 -from validatedpatterns_tests.interop.conftest_openshift import * # noqa: F401, F403 diff --git a/tests/interop/test_subscription_status_hub.py b/tests/interop/test_subscription_status_hub.py deleted file mode 100644 index 22d4fc80..00000000 --- a/tests/interop/test_subscription_status_hub.py +++ /dev/null @@ -1,29 +0,0 @@ -import logging - -import pytest -from validatedpatterns_tests.interop import subscription - -from . import __loggername__ - -logger = logging.getLogger(__loggername__) - - -@pytest.mark.subscription_status_hub -def test_subscription_status_hub(openshift_dyn_client): - # These are the operator subscriptions and their associated namespaces - expected_subs = { - "openshift-gitops-operator": ["openshift-gitops-operator"], - "prometheus": ["llm-monitoring"], - "grafana-operator": ["llm-monitoring"], - "nfd": ["openshift-nfd"], - "gpu-operator-certified": ["nvidia-gpu-operator"], - } - - err_msg = subscription.subscription_status( - openshift_dyn_client, expected_subs, diff=False - ) - if err_msg: - logger.error(f"FAIL: {err_msg}") - assert False, err_msg - else: - logger.info("PASS: Subscription status check passed") diff --git a/tests/interop/test_validate_gpu_nodes.py b/tests/interop/test_validate_gpu_nodes.py deleted file mode 100644 index c3ae585c..00000000 --- a/tests/interop/test_validate_gpu_nodes.py +++ /dev/null @@ -1,166 +0,0 @@ -import logging -import os -import re - -import pytest -from ocp_resources.machine_set import MachineSet -from ocp_resources.node import Node -from ocp_resources.pod import Pod - -from . import __loggername__ - -logger = logging.getLogger(__loggername__) - -oc = os.environ["HOME"] + "/oc_client/oc" - - -@pytest.mark.validate_gpu_machineset -def test_validate_gpu_nodes(openshift_dyn_client): - """ - Check for the existence of the GPU machineset - """ - logger.info("Checking GPU machineset") - machinesets = MachineSet.get( - dyn_client=openshift_dyn_client, namespace="openshift-machine-api" - ) - - found = False - for machineset in machinesets: - logger.info(machineset.instance.metadata.name) - # "gpu" for AWS machineset - # "nvidia" for Azure machineset - if re.search("gpu", machineset.instance.metadata.name) or re.search( - "nvidia", machineset.instance.metadata.name - ): - gpu_machineset = machineset - found = True - break - - err_msg = "GPU machineset not found" - if found: - logger.info( - f"PASS: Found GPU machineset: {gpu_machineset.instance.metadata.name}" - ) - else: - logger.error(f"FAIL: {err_msg}") - assert False, err_msg - - """ - Check for the existence of the GPU machineset taint - """ - logger.info("Checking GPU machineset taint") - - err_msg = "No taints found for GPU machineset" - try: - logger.info(gpu_machineset.instance.spec.template.spec.taints) - except AttributeError: - logger.error(f"FAIL: {err_msg}") - assert False, err_msg - - if gpu_machineset.instance.spec.template.spec.taints == "None": - logger.error(f"FAIL: {err_msg}") - assert False, err_msg - - logger.info( - f"PASS: Found GPU machineset taint: {gpu_machineset.instance.spec.template.spec.taints}" - ) - - """ - Check for the existence of the GPU machineset label - """ - logger.info("Checking GPU machineset label") - - err_msg = "No label found for GPU machineset" - try: - logger.info(gpu_machineset.instance.spec.template.spec.metadata.labels) - except AttributeError: - logger.error(f"FAIL: {err_msg}") - assert False, err_msg - - labels = str(gpu_machineset.instance.spec.template.spec.metadata.labels) - if labels == "None": - logger.error(f"FAIL: {err_msg}") - assert False, err_msg - - logger.info(f"PASS: Found GPU machineset labels: {labels}") - - """ - Check for the existence of the GPU machineset instance type - """ - logger.info("Checking GPU machineset instance type") - - err_msg = "No instanceType found for GPU machineset" - - # for AWS - instance_type = ( - gpu_machineset.instance.spec.template.spec.providerSpec.value.instanceType - ) - if instance_type is None: - # for Azure - instance_type = ( - gpu_machineset.instance.spec.template.spec.providerSpec.value.vmSize - ) - - logger.info(instance_type) - - if instance_type is None: - logger.error(f"FAIL: {err_msg}") - assert False, err_msg - - logger.info(f"PASS: Found GPU machineset instance type: {instance_type}") - - -@pytest.mark.validate_gpu_node_role_labels_pods -def test_validate_gpu_node_role_labels_pods(openshift_dyn_client): - """ - Check for the expected node-role labels for GPU nodes - """ - logger.info("Checking GPU node-role labels") - - nodes = Node.get(dyn_client=openshift_dyn_client) - gpu_nodes = [] - expected_count = 1 - for node in nodes: - logger.info(node.instance.metadata.name) - labels = node.instance.metadata.labels - logger.info(labels) - label_str = str(labels) - - odh_label = "'node-role.kubernetes.io/odh-notebook': ''" - worker_label = "'node-role.kubernetes.io/worker': ''" - - if odh_label in label_str and worker_label in label_str: - gpu_nodes.append(node) - - if len(gpu_nodes) == int(expected_count): - logger.info("PASS: Found 'worker' and 'odh-notebook' GPU node-role labels") - else: - err_msg = "Could not find 'worker' and 'odh-notebook' GPU node-role label" - logger.error(f"FAIL: {err_msg}") - assert False, err_msg - - """ - Check for the expected number of pods deployed on GPU nodes - """ - logger.info("Checking pod count on GPU nodes") - - # We are assuming one GPU node - gpu_node = gpu_nodes[0].instance.metadata.name - nvidia_pods = [] - expected_count = 8 - project = "nvidia-gpu-operator" - pods = Pod.get(dyn_client=openshift_dyn_client, namespace=project) - - for pod in pods: - if "nvidia" in pod.instance.metadata.name: - logger.info(f"nvidia pod: {pod.instance.metadata.name}") - if gpu_node in pod.instance.spec.nodeName: - logger.info(f"nvidia pod node name: {pod.instance.spec.nodeName}") - nvidia_pods.append(pod.instance.metadata.name) - - if len(nvidia_pods) == int(expected_count): - logger.info("PASS: Found the expected nvidia pod count for GPU nodes") - else: - err_msg = "Did not find the expected nvidia pod count for GPU nodes" - logger.error(f"FAIL: {err_msg}") - assert False, err_msg diff --git a/tests/interop/test_validate_hub_site_components.py b/tests/interop/test_validate_hub_site_components.py deleted file mode 100644 index 3519b673..00000000 --- a/tests/interop/test_validate_hub_site_components.py +++ /dev/null @@ -1,166 +0,0 @@ -import logging -import os -import subprocess - -import pytest -from ocp_resources.pod import Pod -from ocp_resources.route import Route -from ocp_resources.storage_class import StorageClass -from openshift.dynamic.exceptions import NotFoundError -from validatedpatterns_tests.interop import application, components - -from . import __loggername__ - -logger = logging.getLogger(__loggername__) - -oc = os.environ["HOME"] + "/oc_client/oc" - - -@pytest.mark.test_validate_hub_site_components -def test_validate_hub_site_components(openshift_dyn_client): - logger.info("Checking Openshift version on hub site") - version_out = components.dump_openshift_version() - logger.info(f"Openshift version:\n{version_out}") - - logger.info("Dump PVC and storageclass info") - pvcs_out = components.dump_pvc() - logger.info(f"PVCs:\n{pvcs_out}") - - for sc in StorageClass.get(dyn_client=openshift_dyn_client): - logger.info(sc.instance) - - -@pytest.mark.validate_hub_site_reachable -def test_validate_hub_site_reachable(kube_config, openshift_dyn_client): - logger.info("Check if hub site API end point is reachable") - err_msg = components.validate_site_reachable(kube_config, openshift_dyn_client) - if err_msg: - logger.error(f"FAIL: {err_msg}") - assert False, err_msg - else: - logger.info("PASS: Hub site is reachable") - - -@pytest.mark.check_pod_status_hub -def test_check_pod_status(openshift_dyn_client): - logger.info("Checking pod status") - projects = ["nvidia-gpu-operator", "rag-llm"] - err_msg = components.check_pod_status(openshift_dyn_client, projects) - if err_msg: - logger.error(f"FAIL: {err_msg}") - assert False, err_msg - else: - logger.info("PASS: Pod status check succeeded.") - - -@pytest.mark.check_pod_count_hub -def test_check_pod_count_hub(openshift_dyn_client): - logger.info("Checking pod count") - projects = {"rag-llm": 4} - - failed = [] - for key in projects.keys(): - logger.info(f"Checking project: {key}") - pods = Pod.get(dyn_client=openshift_dyn_client, namespace=key) - - count = 0 - for pod in pods: - logger.info(pod.instance.metadata.name) - count += 1 - - logger.info(f"Found {count} pods") - if count < projects[key]: - failed.append(key) - - if len(failed) > 0: - err_msg = f"Failed to find the expected pod count for: {failed}" - logger.error(f"FAIL: {err_msg}") - assert False, err_msg - else: - logger.info("PASS: Found the expected pod count") - - -@pytest.mark.validate_argocd_reachable_hub_site -def test_validate_argocd_reachable_hub_site(openshift_dyn_client): - logger.info("Check if argocd route/url on hub site is reachable") - err_msg = components.validate_argocd_reachable(openshift_dyn_client) - if err_msg: - logger.error(f"FAIL: {err_msg}") - assert False, err_msg - else: - logger.info("PASS: Argocd is reachable") - - -@pytest.mark.validate_llm_ui_route -def test_validate_llm_ui_route(openshift_dyn_client): - namespace = "rag-llm" - logger.info("Check for the existence of the llm-ui route") - try: - for route in Route.get( - dyn_client=openshift_dyn_client, - namespace=namespace, - name="llm-ui", - ): - logger.info(route.instance.spec.host) - except NotFoundError: - err_msg = "llm-ui url/route is missing in rag-llm namespace" - logger.error(f"FAIL: {err_msg}") - assert False, err_msg - - logger.info("PASS: Found llm-ui route") - - -@pytest.mark.validate_nodefeaturediscovery -def test_validate_nodefeaturediscovery(): - namespace = "openshift-nfd" - name = "nfd-instance" - logger.info("Check for nodefeaturediscovery instance") - - cmd_out = subprocess.run( - [oc, "get", "NodeFeatureDiscovery", "-n", namespace, name, "--no-headers"], - capture_output=True, - ) - if cmd_out.stdout: - logger.info(cmd_out.stdout.decode("utf-8")) - logger.info("PASS: Found nodefeaturediscovery instance") - else: - assert False, cmd_out.stderr - - -@pytest.mark.validate_gpu_clusterpolicy -def test_validate_gpu_clusterpolicy(): - name = "rag-llm-gpu-cluster-policy" - tolerations = ( - '"tolerations":[{"effect":"NoSchedule","key":"odh-notebook","value":"true"}]' - ) - logger.info("Check for GPU clusterpolicy") - - cmd_out = subprocess.run( - [oc, "get", "ClusterPolicy", "-o", "yaml", name], capture_output=True - ) - if cmd_out.stdout: - logger.info(cmd_out.stdout.decode("utf-8")) - - if tolerations in cmd_out.stdout.decode("utf-8"): - logger.info("PASS: Found GPU clusterpolicy and tolerations") - else: - err_msg = "FAIL: Expected tolerations not found" - logger.error(f"FAIL: {err_msg}") - assert False, err_msg - else: - assert False, cmd_out.stderr - - -@pytest.mark.validate_argocd_applications_health_hub_site -def test_validate_argocd_applications_health_hub_site(openshift_dyn_client): - logger.info("Get all applications deployed by argocd on hub site") - projects = ["openshift-gitops", "rag-llm-gitops-hub"] - unhealthy_apps = application.get_argocd_application_status( - openshift_dyn_client, projects - ) - if unhealthy_apps: - err_msg = "Some or all applications deployed on hub site are unhealthy" - logger.error(f"FAIL: {err_msg}:\n{unhealthy_apps}") - assert False, err_msg - else: - logger.info("PASS: All applications deployed on hub site are healthy.") diff --git a/tests/interop/ui/playwright.config.ts b/tests/interop/ui/playwright.config.ts deleted file mode 100644 index 3e862795..00000000 --- a/tests/interop/ui/playwright.config.ts +++ /dev/null @@ -1,82 +0,0 @@ -import { defineConfig, devices } from '@playwright/test'; - -/** - * Read environment variables from file. - * https://github.com/motdotla/dotenv - */ -// import dotenv from 'dotenv'; -// import path from 'path'; -// dotenv.config({ path: path.resolve(__dirname, '.env') }); - -/** - * See https://playwright.dev/docs/test-configuration. - */ -export default defineConfig({ - testDir: './tests', - /* Run tests in files in parallel */ - fullyParallel: true, - /* Fail the build on CI if you accidentally left test.only in the source code. */ - forbidOnly: !!process.env.CI, - /* Retry on CI only */ - retries: process.env.CI ? 2 : 0, - /* Opt out of parallel tests on CI. */ - workers: process.env.CI ? 1 : undefined, - /* Reporter to use. See https://playwright.dev/docs/test-reporters */ - reporter: [ ['html', { open: 'never' }] ], - /* Shared settings for all the projects below. See https://playwright.dev/docs/api/class-testoptions. */ - use: { - /* Base URL to use in actions like `await page.goto('/')`. */ - // baseURL: 'http://127.0.0.1:3000', - - /* Collect trace when retrying the failed test. See https://playwright.dev/docs/trace-viewer */ - trace: 'on-first-retry', - ignoreHTTPSErrors: true, - }, - timeout: 600 * 1000, - globalTimeout: 600 * 1000, - - /* Configure projects for major browsers */ - projects: [ - { - name: 'chromium', - use: { ...devices['Desktop Chrome'] }, - }, - -// { -// name: 'firefox', -// use: { ...devices['Desktop Firefox'] }, -// }, -// -// { -// name: 'webkit', -// use: { ...devices['Desktop Safari'] }, -// }, - - /* Test against mobile viewports. */ - // { - // name: 'Mobile Chrome', - // use: { ...devices['Pixel 5'] }, - // }, - // { - // name: 'Mobile Safari', - // use: { ...devices['iPhone 12'] }, - // }, - - /* Test against branded browsers. */ - // { - // name: 'Microsoft Edge', - // use: { ...devices['Desktop Edge'], channel: 'msedge' }, - // }, - // { - // name: 'Google Chrome', - // use: { ...devices['Desktop Chrome'], channel: 'chrome' }, - // }, - ], - - /* Run your local dev server before starting the tests */ - // webServer: { - // command: 'npm run start', - // url: 'http://127.0.0.1:3000', - // reuseExistingServer: !process.env.CI, - // }, -}); diff --git a/tests/interop/ui/ragllm.spec.ts-snapshots/ragllm-add-provider-1-chromium-linux.png b/tests/interop/ui/ragllm.spec.ts-snapshots/ragllm-add-provider-1-chromium-linux.png deleted file mode 100644 index a19b79ff..00000000 Binary files a/tests/interop/ui/ragllm.spec.ts-snapshots/ragllm-add-provider-1-chromium-linux.png and /dev/null differ diff --git a/tests/interop/ui/ragllm.spec.ts-snapshots/ragllm-grafana-dashboard-1-chromium-linux.png b/tests/interop/ui/ragllm.spec.ts-snapshots/ragllm-grafana-dashboard-1-chromium-linux.png deleted file mode 100644 index a87ef552..00000000 Binary files a/tests/interop/ui/ragllm.spec.ts-snapshots/ragllm-grafana-dashboard-1-chromium-linux.png and /dev/null differ diff --git a/tests/interop/ui/ragllm.spec.ts.j2 b/tests/interop/ui/ragllm.spec.ts.j2 deleted file mode 100644 index c80b168f..00000000 --- a/tests/interop/ui/ragllm.spec.ts.j2 +++ /dev/null @@ -1,60 +0,0 @@ -import { test, expect } from '@playwright/test'; - -test.use({ - ignoreHTTPSErrors: true -}); - -test('ragllm: test proposal generation and grafana dashboard', async ({ page }) => { - test.setTimeout(240000); - // Log in to Openshift console - await page.goto("{{ hub_console.stdout }}"); - await page.getByLabel('Username *').fill('kubeadmin'); - await page.getByLabel('Password *').fill("{{ kubeadmin }}"); - await page.getByRole('button', { name: 'Log in' }).click(); - - // Launch RAG-LLM demo UI - await page.locator('[data-test-id="application-launcher"]').click(); - const page1Promise = page.waitForEvent('popup'); - await page.getByRole('menuitem', { name: 'Retrieval-Augmented' }).click(); - const page1 = await page1Promise; - - // Supply customer and product names - await page1.getByRole('textbox', { name: 'Customer Enter the customer' }).click(); - await page1.getByRole('textbox', { name: 'Customer Enter the customer' }).fill('validated-patterns-qe'); - await page1.getByRole('textbox', { name: 'Product Enter the Red Hat' }).click(); - await page1.getByRole('textbox', { name: 'Product Enter the Red Hat' }).fill('RedHat OpenShift AI'); - - // Generate proposal and add rating - await page1.getByRole('button', { name: 'Generate' }).click(); - await page1.waitForTimeout(60000); - await page1.getByRole('radio', { name: '3' }).check(); - await page1.waitForTimeout(60000); - - // Add a provider - await page1.getByRole('tab', { name: 'Configuration' }).click(); - await page1.getByRole('button', { name: 'Add Provider' }).click(); - await page1.getByRole('listbox', { name: 'Providers' }).click(); - await page1.getByRole('option', { name: 'OpenAI' }).click(); - await page1.getByRole('textbox', { name: 'Model Enter the model name' }).click(); - await page1.getByRole('textbox', { name: 'Model Enter the model name' }).fill('gtp-4o-mini'); - await page1.getByRole('textbox', { name: 'URL Enter the URL' }).click(); - await page1.getByRole('textbox', { name: 'URL Enter the URL' }).fill('https://api.openai.com/vi/chat/completions'); - await page1.getByTestId('password').click(); - await page1.getByTestId('password').fill('12121212'); - await page1.getByRole('button', { name: 'Add', exact: true }).click(); - await page1.getByTestId('toast-close').click(); - await page1.screenshot({ path: 'ragllm-add-provider-1-chromium-linux.png' }); - // await page1.screenshot({ path: 'add-provider.png', fullPage: true }); - - // Check grafana dashboard - await page.locator('[data-test-id="application-launcher"]').click(); - const page2Promise = page.waitForEvent('popup'); - await page.getByRole('menuitem', { name: 'Grafana UI for LLM Ratings' }).click(); - const page2 = await page2Promise; - await page2.getByTestId('data-testid navigation mega-menu').getByRole('link', { name: 'Dashboards' }).click(); - await page2.getByRole('link', { name: 'llm-monitoring' }).click(); - await page2.getByRole('link', { name: 'MODEL FEEDBACK/RATING' }).click(); - await page2.waitForTimeout(10000); - await page2.screenshot({ path: 'ragllm-grafana-dashboard-1-chromium-linux.png' }); - // await page2.screenshot({ path: 'grafana-dashboard.png', fullPage: true }); -}); diff --git a/values-global.yaml b/values-global.yaml index 3f10827b..b754b878 100644 --- a/values-global.yaml +++ b/values-global.yaml @@ -1,6 +1,7 @@ --- global: pattern: rag-llm-gitops + singleArgoCD: true options: useCSV: false syncPolicy: Automatic @@ -23,7 +24,7 @@ global: storageClass: gp3-csi main: - clusterGroupName: hub + variant: hub multiSourceConfig: enabled: true clusterGroupChartVersion: 0.9.* diff --git a/variants/ci/values-AWS.yaml b/variants/ci/values-AWS.yaml new file mode 100644 index 00000000..8206e7ca --- /dev/null +++ b/variants/ci/values-AWS.yaml @@ -0,0 +1,24 @@ +clusterGroup: + imperative: + jobs: + - name: create-gpu-machineset + playbook: ansible/playbooks/create-gpu-machineset.yaml + clusterRoleYaml: + - apiGroups: + - "*" + resources: + - machinesets + - persistentvolumeclaims + - datavolumes + - dataimportcrons + - datasources + verbs: + - "*" + - apiGroups: + - "*" + resources: + - "*" + verbs: + - get + - list + - watch diff --git a/variants/ci/values-Azure.yaml b/variants/ci/values-Azure.yaml new file mode 100644 index 00000000..0df42d6f --- /dev/null +++ b/variants/ci/values-Azure.yaml @@ -0,0 +1,29 @@ +global: + db: + type: MSSQL + storageClass: managed-csi + +clusterGroup: + imperative: + jobs: + - name: create-gpu-machineset + playbook: ansible/playbooks/create-gpu-machineset-azure.yaml + clusterRoleYaml: + - apiGroups: + - "*" + resources: + - machinesets + - persistentvolumeclaims + - datavolumes + - dataimportcrons + - datasources + verbs: + - "*" + - apiGroups: + - "*" + resources: + - "*" + verbs: + - get + - list + - watch diff --git a/variants/ci/values-ci.yaml b/variants/ci/values-ci.yaml new file mode 100644 index 00000000..7b8c12f8 --- /dev/null +++ b/variants/ci/values-ci.yaml @@ -0,0 +1,122 @@ +clusterGroup: + name: ci + + namespaces: + open-cluster-management: + vault: + external-secrets-operator: + operatorGroup: true + targetNamespaces: [] + external-secrets: + openshift-nfd: + nvidia-gpu-operator: + redhat-ods-operator: + operatorGroup: true + targetNamespaces: [] + rag-llm: + operatorGroup: true + targetNamespaces: + - rag-llm + labels: + opendatahub.io/dashboard: "true" + modelmesh-enabled: 'false' + openshift-serverless: + operatorGroup: true + targetNamespaces: [] + + subscriptions: + eso: + name: openshift-external-secrets-operator + namespace: external-secrets-operator + channel: stable-v1 + nfd: + name: nfd + namespace: openshift-nfd + nvidia: + name: gpu-operator-certified + namespace: nvidia-gpu-operator + source: certified-operators + edb: + name: cloud-native-postgresql + namespace: openshift-operators + source: certified-operators + elastic: + name: elasticsearch-eck-operator-certified + namespace: rag-llm + source: certified-operators + rhoai: + name: rhods-operator + namespace: redhat-ods-operator + + sharedValueFiles: + - '/variants/ci/values-{{ $.Values.global.clusterPlatform }}.yaml' + + applications: + vault: + name: vault + namespace: vault + chart: hashicorp-vault + chartVersion: 0.1.* + openshift-external-secrets: + name: openshift-external-secrets + namespace: external-secrets + chart: openshift-external-secrets + chartVersion: 0.0.* + vllm-inference-service: + name: vllm-inference-service + namespace: rag-llm + path: charts/all/vllm-inference-service + syncPolicy: + automated: + selfHeal: true + retry: + limit: 20 + rag-llm: + name: rag-llm + namespace: rag-llm + path: charts/all/rag-llm + llm-monitoring: + name: llm-monitoring + namespace: llm-monitoring + kustomize: true + path: charts/all/llm-monitoring/kustomize/overlays/dev + nfd-config: + name: nfd-config + namespace: openshift-cfd + path: charts/all/nfd-config + nvidia-config: + name: nvidia-config + namespace: nvidia-network-operator + path: charts/all/nvidia-gpu-config + llm-ui-config: + name: rag-llm-ui-config + namespace: rag-llm + path: charts/all/rag-llm-ui-config + llm-monitoring-config: + name: grafana-ui-config + namespace: llm-monitoring + path: charts/all/llm-monitoring-config + openshift-ai: + name: openshift-ai + namespace: redhat-ods-operator + path: charts/all/rhods + +global: + model: + vllm: ibm-granite/granite-4.0-h-350m + +vllmServingRuntime: + args: + - /cache/models + # Cap KV-cache reservation so there is free VRAM headroom for the Mamba + # Triton kernel autotuner, which allocates scratch outside vLLM's pool on + # the first request. Needed on 16 GB GPUs (e.g. Azure Tesla T4); the + # default 0.9 leaves too little free and OOMs. The model is ~0.65 GiB, so + # a small KV cache is plenty for CI. + - --gpu-memory-utilization=0.5 + + command: [] + + image: + repository: vllm/vllm-openai + tag: v0.21.0 diff --git a/values-hub.yaml b/variants/hub/values-hub.yaml similarity index 64% rename from values-hub.yaml rename to variants/hub/values-hub.yaml index c1bc6588..730dc87d 100644 --- a/values-hub.yaml +++ b/variants/hub/values-hub.yaml @@ -1,28 +1,34 @@ clusterGroup: name: hub - isHubCluster: true namespaces: - - open-cluster-management - - vault - - golang-external-secrets - - openshift-nfd - - nvidia-gpu-operator - - redhat-ods-operator: - operatorGroup: true - targetNamespaces: [] - - rag-llm: - operatorGroup: true - targetNamespaces: - - rag-llm - labels: - opendatahub.io/dashboard: "true" - modelmesh-enabled: 'false' - - openshift-serverless: - operatorGroup: true - targetNamespaces: [] + open-cluster-management: + vault: + external-secrets-operator: + operatorGroup: true + targetNamespaces: [] + external-secrets: + openshift-nfd: + nvidia-gpu-operator: + redhat-ods-operator: + operatorGroup: true + targetNamespaces: [] + rag-llm: + operatorGroup: true + targetNamespaces: + - rag-llm + labels: + opendatahub.io/dashboard: "true" + modelmesh-enabled: 'false' + openshift-serverless: + operatorGroup: true + targetNamespaces: [] subscriptions: + eso: + name: openshift-external-secrets-operator + namespace: external-secrets-operator + channel: stable-v1 nfd: name: nfd namespace: openshift-nfd @@ -42,13 +48,6 @@ clusterGroup: name: rhods-operator namespace: redhat-ods-operator - projects: - - hub - - rag-llm - - llm-monitoring - - gpu-config - - openshift-ai - sharedValueFiles: - '/overrides/values-{{ $.Values.global.clusterPlatform }}.yaml' @@ -56,19 +55,16 @@ clusterGroup: vault: name: vault namespace: vault - project: hub chart: hashicorp-vault chartVersion: 0.1.* - golang-external-secrets: - name: golang-external-secrets - namespace: golang-external-secrets - project: hub - chart: golang-external-secrets - chartVersion: 0.1.* + openshift-external-secrets: + name: openshift-external-secrets + namespace: external-secrets + chart: openshift-external-secrets + chartVersion: 0.0.* vllm-inference-service: name: vllm-inference-service namespace: rag-llm - project: hub path: charts/all/vllm-inference-service syncPolicy: automated: @@ -78,36 +74,29 @@ clusterGroup: rag-llm: name: rag-llm namespace: rag-llm - project: rag-llm path: charts/all/rag-llm llm-monitoring: name: llm-monitoring namespace: llm-monitoring - project: llm-monitoring kustomize: true path: charts/all/llm-monitoring/kustomize/overlays/dev nfd-config: name: nfd-config namespace: openshift-cfd - project: gpu-config path: charts/all/nfd-config nvidia-config: name: nvidia-config namespace: nvidia-network-operator - project: gpu-config path: charts/all/nvidia-gpu-config llm-ui-config: name: rag-llm-ui-config namespace: rag-llm - project: gpu-config path: charts/all/rag-llm-ui-config llm-monitoring-config: name: grafana-ui-config namespace: llm-monitoring - project: gpu-config path: charts/all/llm-monitoring-config openshift-ai: name: openshift-ai namespace: redhat-ods-operator - project: openshift-ai path: charts/all/rhods