Add internal scripts and info plugins for cross-cluster bundle integration

[dimoffon]

What this is

Bundles that integrate two clusters managed by the same ADCM — an umbrella cluster running
agents on the hosts of a product cluster — could only do it by driving ADCM's public REST API
with a service account, or by importing ADCM's internals from a bundle plugin. Everything such
a bundle needs is already implemented inside ADCM; this exposes it the supported way.

This revision is a redesign following @kuhella's review. The first attempt is the commit
underneath; what it became is described below.

Two internal scripts

host_manage

field	required	default	meaning
operation	yes	—	add_duplicates, remove_duplicates, add_to_groups
source	yes	—	non-empty list of selectors; the result is their union, deduplicated by host id
source[].type	yes	—	cluster, service, component, host, config_host_group, action_host_group
source[].cluster_name	no	current cluster	cluster-relative entries only — a variant that cannot take it does not declare it
target	no	current cluster	add_duplicates only
mapping_rules	no	empty	add_duplicates only; writes the task mapping delta, does not commit
groups	no	empty	membership only; the group must already exist

add_duplicates resolves source to a set of originals and gives every target cluster a
copy of each one it does not have yet. Identity is the original, not the fqdn — a duplicate can
be renamed — so a re-run creates nothing and says so. A host the target already owns in its own
right is skipped.

remove_duplicates resolves source, intersects it with this cluster's duplicates, then
unmaps, detaches and deletes them. A selector may name either side of the copy: the originals in
the cluster they belong to, or the duplicates themselves through a group that holds them — which
is what lets an attachment be undone after its source cluster is gone.

add_to_groups resolves source, intersects it with this cluster's duplicates, and adds the
difference to each group in groups.

host_group_manage

field	required	default	meaning
operation	yes	—	add (create-or-update) or remove
groups	yes	—	non-empty list, each addressed by (type, object, name)
groups[].type	yes	—	config_host_group or action_host_group
groups[].object	yes	—	the group's owner
groups[].description	no	""	declarative: applied on every run
groups[].hosts	no	—	three states: omitted leaves membership alone, a list replaces it exactly, [] empties it
groups[].parameters	no	empty	configuration groups only

remove deletes each named group if present; absent is success. For remove, hosts,
parameters and description are rejected at schema level.

How the review is answered

config_key is gone. A scripts_template is rendered with the submitted action
configuration in scope, so a bundle writes {{ task.config.cluster_name }}. The runtime error it
existed to raise becomes a schema rule: a mistyped key renders to an empty string rather than
failing, so every name is min_length=1 — without that a typo would quietly mean "the current
cluster".

Both read-only plugins are deleted. Reading another cluster's topology from a playbook was
the wrong shape twice over: ansible had to be told what ADCM already knows, and it learned it
after its inventory was frozen. source replaces it, resolved server-side against live topology.
Probing whether a configuration host group exists is likewise unnecessary now that add is
create-or-update.

The decisions happen before ansible starts, in use_cases/transition/.

Mapping is separated from its commit. mapping_rules writes the task's mapping delta and
stops; the inventory of every later job gains the <service>.<component>.add groups that delta
implies, so the agents job installs onto exactly the hosts just attached, and hc_apply commits
afterwards. A failed install leaves the cluster's mapping untouched, and a parallel action sees
nothing until installation completes.

One vocabulary. source, target, groups and object address objects the way
config_apply does, and mapping_rules reuses the hc_acl rule shape.

The three hard properties

Idempotence is not automatic — both group APIs raise when a host is added twice, and neither
creating a duplicate nor filling a group may rely on a constraint to reject a repeat. Every
operation computes the difference between wanted and current state and applies only that.

Atomicity had to be built. InternalExecutor swallows every exception, so the transaction the
celery runner wraps around a job commits whatever a failing script already wrote. Each script
opens its own atomic().

The query count no longer grows with the number of hosts. create_duplicates inserts the rows
in one statement, joins the cluster once, distributes provider concerns once and reads a
configuration specification once per prototype. What remains is re_apply_object_policy, which
walks the cluster's hosts for any host-parametrised policy and belongs to the cluster rather than
to the hosts being added — the new code adds no per-host query, pinned by a test asserting an
identical count for two hosts and for ten.

Two notes

The inventory used to take the task's mapping delta only for an action declaring hc_acl.
Declaring it is not free — it switches on the launch-time mapping path, and the caller must then
submit the cluster's complete mapping on every run — so the gate now also opens for a delta a
script wrote, which is the only other way one can exist.

core/legacy/bundle_alt/schema.py is not updated, and develop has since deleted it outright.

Tests

Parsing, including every rejection the shape implies; each operation and its idempotence; the
delta's two readers; three-state group membership; and the query count. Rebased onto develop;
the suite is green apart from three api_v2.tests.test_service failures that are present on
pristine develop and unrelated to this change. ruff, the licence checker and the import-linter
contracts are clean.

First consumer

The ADB Enterprise Services bundle's adb_clusters service: Register/Sync run
claim → host_group_manage add → host_manage add_duplicates → install agents → hc_apply → host_manage add_to_groups → finalize, Deregister runs uninstall → host_manage remove_duplicates → host_group_manage remove → finalize. The whole cycle is verified on a live stand.

[kuhella]

Thank you for your interest in our project, but unfortunately, we cannot incorporate your changes in their current form.

1. Current solution review

1. The config_key duplicates a mechanism that already exists, already exposes the submitted config as task|action.config in scripts_template|ansible runtime
2. *_apply is for a single operation, *_manage for several. Both scripts take anoperation discriminator, so both are misnamed. And *_manage can be extended in future with new operations
3. adcm_config_host_group_info , adcm_cluster_info implemented as action plugin, but according to ansible architecture it must be vars plugins or facts. Now I am not recommend to implement it as fact or var plugins, cause we already know about Ansible performance issue with static inventory and access to vars in runtime. For that purses we need ansible facts storage and caching facts mechanism.
4. The logic for interacting with ADCM and making decisions is not encapsulated before launch of Ansible playbook.
5. Usage problem: There is no separation between already registered ADB cluster, and new one (during action execution) by inventory host group, so agents every time will be installed on all registered ADB cluster hosts.

2. Proposed changes

Two internal scripts, without ansible info plugins.

Script	Operations
host_manage	add_duplicates, delete_duplicates
host_group_manage	add, remove — CHG and AHG in one discriminated list

2.1 host_manage params

field	required	default	logic
operation	yes	—	Enum [add_duplicates, remove_duplicates] type of operation .
source	yes	—	Non-empty list. Selects hosts. Entries are a discriminated union on type; the result is the union of all entries, deduplicated by host id.
source[].cluster_name	no	current cluster	Which cluster the entry is resolved against. Rejected for type: host and for the group types: a config host group only ever holds hosts of its owner's cluster, so a group source is always in the current cluster and a qualifier could only be redundant or contradictory.
source[].service_name	for type in service, component	—	Narrows the selection to hosts mapped to that service.
source[].component_name	for type: component	—	Narrows to hosts mapped to that component.
source[].host_name	for type: host	—	Selects exactly one host by fqdn.
source[].name	for group types	—	The group's name.
source[].object	for group types	—	The group's owner, as CoreObjectTargetDescription.
target	no	current cluster	Destination clusters for add_duplicates. Rejected for remove_duplicates and add_to_groups: both act only on the current cluster.
mapping_rules	no	empty	add_duplicates only. Writes the task mapping delta; does not commit mapping — hc_apply does. Empty means the duplicates arrive unmapped.
groups	no	empty	add_duplicates. Membership only: the group must already exist, and the script never creates, renames or reconfigures it.

add_duplicates — resolve source to a set of original hosts; for each target cluster, create a duplicate of every original that does not already have one there; write the mapping delta implied by mapping_rules; add the resulting hosts to groups. Hosts that already have a duplicate in the target are counted as existing, not re-created.

remove_duplicates — resolve source; intersect with the duplicates of the current cluster; unmap them, detach them from the cluster, delete the host records. Group memberships disappear by cascade; the groups themselves are left alone (removing them is host_group_manage's job, and must run after this, never before).

2.2 host_group_manage params

field	required	default	logic
operation	yes	—	add (create-or-update) or remove.
groups	yes	—	Non-empty list. Each entry is addressed by (type, object, name), which is unique by unique_together on both models.
groups[].name	yes	—	Group name; the address, not a mutable attribute. Renaming is out of scope.
groups[].type	yes	—	config_host_group or action_host_group. Discriminates the entry: parameters is admissible only for the former.
groups[].object	yes	—	object - owner of group
groups[].description	no	""	Declarative: set to the given value on every run, including updates.
groups[].hosts	no	—	Three distinct states. Omitted — membership untouched (this is how a group is created empty). A list — membership replaced to exactly that set. An empty list — membership emptied.
groups[].parameters	no	empty	Config groups only. Sets the object configuration key and value (semantic of config_apply)

For operation: remove, hosts, parameters and description are rejected at schema level — there is nothing for them to mean.

operation: add — for each entry, create the group if absent, apply description and parameters, and reconcile hosts per the three-state rule above.

operation: remove — delete each named group if present. Absent is success, not an error.

2.3 Examples

Attach hosts of several clusters to the current one:

- name: share_hosts
  script_type: internal
  script: host_manage
  params:
    operation: add_duplicates
    source:
      {% for cluster in task.config.source_cluster %}
      - type: cluster
        cluster_name: "{{ cluster }}"
      {% endfor %}

Attach the current cluster's hosts to another one:

    operation: add_duplicates
    source:
      - type: cluster
    target:
      - cluster_name: "{{ task.config.target_cluster }}"

Attach only the hosts of component of a foreign cluster, and map them here:

    operation: add_duplicates
    source:
      - type: component
        cluster_name: "{{ task.config.cluster_name }}"
        service_name: adb
        component_name: segment
    mapping_rules:
      - {service: adbm,  component: agent, action: add}
      - {service: adbcc, component: agent, action: add}
    groups:
      - {name: "{{ task.config.cluster_name }}", type: config_host_group }}"
      - {name: "{{ task.config.cluster_name }}", type: action_host_group }}"

Create the per-attachment groups, idempotently:

- name: prepare_groups
  script_type: internal
  script: host_group_manage
  params:
    operation: add
    groups:
      - name: "{{ task.config.cluster_name }}"
        type: config_host_group
        description: generated by system
        object: 
           type: component
           service_name: adbm
           component_name: agent
        parameters:
           - key: Cluster_configuration/replication_factor
              value: 2
      - name: "{{ task.config.cluster_name }}"
         type: action_host_group
         object: 
            type: component
            service_name: adbm
            component_name: agent

Release by group, or by source:

    operation: remove_duplicates
    source:
      - type: config_host_group
         name: "{{ task.config.cluster_name }}"
         object:
           type: component
           service_name: adb
           component_name: agent

    operation: remove_duplicates
    source:
      - type: cluster
         cluster_name: "{{ task.config.cluster_name }}"

Important

** Every operation must be safe to re-run with identical params, and a second run must change nothing and say so. This is not automatic: both group APIs raise when a host is added twice and the difference applied, never the whole wanted set.The same holds for duplicate creation, which must skip originals that already have a duplicate in the target rather than relying on a constraint to reject them.
** Atomicity: everything an operation does — host creation, mapping delta, group membership — either lands together or not at all.
** Concurrency support important as we have non-blocking actions;
** The DB query count must not depend on number of hosts**, for every operation.
** The context of an action is usually important: a bundle developer might define the same bundle scripts at the cluster, service, or component level, yet in this case, the target object is always the cluster, regardless of the invocation context.

2.4 Full sequence with deferred mapping

Requirement: the cluster's mapping must not change until the agents are installed, so a failed
install leaves nothing applied. mapping_rules therefore writes the task delta and hc_apply
commits it afterwards.

- name: share_hosts          # job 1 — creates duplicates, writes the mapping delta, add to host-group in cluster level
  script_type: internal
  script: host_manage
  params:
    operation: add_duplicates
    source: [{type: component, cluster_name: "{{ task.config.cluster_name }}",
              service_name: adb, component_name: segment}]
    mapping_rules:
      - service: adbm
        component: agent
        action: add
    groups:
      - name: "{{ task.config.cluster_name }}"
         type: config_host_group
      - name: "{{ task.config.cluster_name }}"
         type: action_host_group

- name: install_agents       # job 2 — hosts appear in the adbm.agent.add inventory group
  script_type: ansible
  script: install.yaml

- name: apply_mapping        # job 3 — commits the delta
  script_type: internal
  script: hc_apply
  params:
    rules:                   # must repeat the components: empty rules apply nothing, silently
      - service: adbm
         component: agent
         action: add

Important: if the host group can live on the cluster instead of on adbm.agent, imposes no mapping requirement at all (during group creation). As alternative you can add add_to_groups operation to host_manage, which is resolve source; intersect with the duplicates of the current cluster; add the difference between the wanted and the current membership to each group in groups.

Benefits

• Generating *.add, *.remove inventory groups, according to mapping_rules of host_manage, that group should be used to install agents (over duplicates from source cluster according to cluster topology) and committing mapping changes only after installation (other parallel action do not see changes until full agent installation);
• Clean ansible runtime, isolated from adcm business-logic, both read-only plugins become unnecessary:
  • adcm_config_host_group_info need for getting info about group exists —> host_group a question idempotent.
  • adcm_cluster_info -> sourceas selector for host inhost_manage`. It select host to create duplicate hosts
• Reusing existing approches:
  • config_key is redundant{{ task.config.X }} already does the substitution at render time.
  • Errors move from run time to scripts_template render time.
  • One vocabulary instead of three source/target/groups all address objects the way does; mapping_rules is used by hc_acl. A bundle author who knows ADCM already knows this syntax.
  • Naming follows the project rule *_manage for multi-operation scripts.
• The two jobs of group_owner are separated. The config host group keeps only its real purpose — deciding whose configuration the attached hosts override. Teardown gets its own address: by source cluster, or by group as an explicit alternative.
• Deletion is symmetric with creation. remove_duplicates takes the same source union as add_duplicates, so the same expression means the same thing in both directions.

[sonarqubecloud]

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.4% Duplication on New Code

See analysis details on SonarQube Cloud