Operations

Overview

This document provides explanation of the Touchpoints agile development process and the production deployment process.

---

Environments

Touchpoints is currently deployed in at least 3 environments.

1. Production - the live site
2. Staging - an integration environment for the development team and product owner to validate functionality before pushing to production
3. Demo - a lightweight copy of Production for Touchpoint's gov customers to test in. Demo and Production always share the same code but feature flags on Demo may be set to expose in-progress features.

These environments are managed via cloud.gov, and are currently configured to run on Cloud Foundry, but can run elsewhere with relatively little modification.

Each Developer is expected to be able to develop Touchpoints on their own cloud.gov instance as well.

Development Process

Software Product teams develop conventions related to the development process. The steps below represent the flow of code from a developer's machine through multiple gates, to production.

How git branches are used

Touchpoints follows a version of GitHub Flow.

Key points:

• There is a single long-lived branch named main.
• Developers work in feature branches, created off of main.
• main is to be treated as always deployable. Developers should not merge anything to main that they would be uncomfortable being deployed immediately to production.
• Notwithstanding the previous point, we are not doing continuous deployment. Code that is merged to main will generally be verified in staging before it is pushed to production.
• The Staging environment always reflects the HEAD of main.
• The Production environment always reflects the latest release tag applied to main.

How code flows from a developer's machine to production deployment

• LOCAL DEVELOPMENT
• Feature branches are submitted as Pull Requests and reviewed in GitHub.
• All commits are pushed here, to this Touchpoints code repository.
• Upon creation of PR, CircleCI runs automated tests.
• Upon creation of PR, CodeQL runs automated code scans.
• When a feature branch is code reviewed, it may be merged to main.
• Green (passing) main builds get deployed to Staging automatically via CircleCI.
• ON STAGING...
• The Product Owner reviews the changes on Staging.
• If fixes are required, the process starts over from LOCAL DEVELOPMENT.
• Once the Product Owner is happy, main is tagged with a release tag.
• RELEASE TO PRODUCTION...
• Pushing a release tag triggers a deployment to Production automatically via CircleCI.
• Pushing a release tag also triggers a deployment to Demo automatically via CircleCI.

Deployment

How to deploy Touchpoints

Deploying Touchpoints requires pushing a directory of code (the Touchpoints repo or CI build output) to an application instance in cloud.gov's Cloud Foundry using the cf push command.

This can be done 1) manually via command line or 2) in an automated way, using a continuous integration tool like CircleCI.

Using Cloud Foundry

cf push references a local manifest.yml that declares one or more applications. Thus, having a "properly configured" manifest.yml file is a prerequisite dependency; a requirement.

To deploy a specific instance, use cf push. For example: cf push touchpoints-staging.

Secrets

Touchpoints loads its runtime secrets from a Cloud Foundry user-provided service named touchpoints-secrets, one per environment. At boot, the application reads VCAP_SERVICES, finds this service, validates that every required secret is present, and copies the values into environment variables.

This replaces the previous approach of setting secrets individually with cf set-env.

Why a user-provided service

• All application secrets live in one place per environment.
• Secrets are bound to the app rather than injected ad hoc, so they are not lost on cf push.
• The application fails fast at startup if a required secret is missing, instead of failing later at runtime.
• The same mechanism works for the web app, the Sidekiq worker, and the migration task.

How it works

1. Each app's manifest binds the environment-specific secrets service under the binding name touchpoints-secrets.
2. On boot, config/vcap_services.rb runs early in config/application.rb, before the Rails environment configuration reads any required ENV values.
3. It locates the bound service named touchpoints-secrets in VCAP_SERVICES, reads its credentials, validates all required keys, and sets the corresponding environment variables.
4. If the service, its credentials, or any required value is missing, startup raises an error and the app does not come up.

Credential structure

The service credentials must be a JSON object with this nested shape:

{
  "turnstile": {
    "secret_key": "...",
    "site_key": "..."
  },
  "api": {
    "username": "...",
    "password": "..."
  },
  "login_gov": {
    "private_key": "..."
  },
  "ses": {
    "region": "...",
    "access_key_id": "...",
    "secret_access_key": "..."
  },
  "new_relic": {
    "key": "..."
  },
  "rails_active_record": {
    "primary_key": "...",
    "deterministic_key": "...",
    "key_derivation_salt": "..."
  },
  "email_sender": "...",
  "gtm_container_id": "..."
}

Required keys and the environment variables they populate

Credential path	Environment variable
turnstile.secret_key	TURNSTILE_SECRET_KEY
turnstile.site_key	TURNSTILE_SITE_KEY
api.username	API_HTTP_USERNAME
api.password	API_HTTP_PASSWORD
login_gov.private_key	LOGIN_GOV_PRIVATE_KEY
ses.region	AWS_SES_REGION
ses.access_key_id	AWS_SES_ACCESS_KEY_ID
ses.secret_access_key	AWS_SES_SECRET_ACCESS_KEY
email_sender	TOUCHPOINTS_EMAIL_SENDER
new_relic.key	NEW_RELIC_KEY
rails_active_record.primary_key	RAILS_ACTIVE_RECORD_PRIMARY_KEY
rails_active_record.deterministic_key	RAILS_ACTIVE_RECORD_DETERMINISTIC_KEY
rails_active_record.key_derivation_salt	RAILS_ACTIVE_RECORD_KEY_DERIVATION_SALT
gtm_container_id	TOUCHPOINTS_GTM_CONTAINER_ID

All of the keys above are required. If any is missing, the app logs which ones were absent and refuses to start.

S3 and Redis credentials are not part of this service. They are supplied by their own bound services and mapped separately.

Service names per environment

Each environment has its own service instance, bound under the common binding name touchpoints-secrets:

Environment	Service instance name
Production	touchpoints-prod-secrets
Staging	touchpoints-staging-secrets
Demo	touchpoints-demo-secrets

The binding name the application looks up is always touchpoints-secrets, regardless of the underlying instance name.

Creating the service

Create the user-provided service once per environment. Store the credentials JSON in a local file (do not commit it), then pass it with -p:

# secrets.json contains the credential structure shown above
cf create-user-provided-service touchpoints-prod-secrets -p ./secrets.json

Targeting:

cf target -o <org> -s <space>

Use the matching instance name for the space you are targeting (touchpoints-staging-secrets, touchpoints-demo-secrets, etc.).

Updating secrets

To rotate or change a value, update the whole credentials object:

cf update-user-provided-service touchpoints-prod-secrets -p ./secrets.json

cf update-user-provided-service replaces the entire credentials payload, so include all required keys in the file, not just the one you are changing.

After updating, restart the bound apps so they pick up the new values:

cf restart touchpoints
cf restart touchpoints-production-sidekiq-worker

The next deployment will also pick up the new values when the apps restage.

Binding the service to an app

The deployment manifests already bind the service. For reference, the manifest entry uses name (the instance) and binding_name (what the app looks up):

services:
  - name: touchpoints-prod-secrets
    binding_name: touchpoints-secrets

If you add a new app that boots Rails, it must bind the environment's secrets service under the binding name touchpoints-secrets, or it will fail to start.

Inspecting the current secrets

To view the bound credentials for an app (values are sensitive, so do this carefully):

cf env touchpoints

To view the service instance directly:

cf service touchpoints-prod-secrets

Local development

Local development and test do not use VCAP_SERVICES. When VCAP_SERVICES is absent, the loader is skipped and the app falls back to loading .env via Dotenv. Set the same variables in your local .env as needed.

Troubleshooting

• required startup secrets service is missing The app is not bound to touchpoints-secrets. Bind the environment's service and restage.

• required startup secrets credentials are missing The service exists but has an empty or malformed credentials object. Re-create or update it with a valid payload.

• required startup secrets are missing: <names> One or more required keys are absent from the credentials. The message lists the missing environment variable names. Add them and update the service.

• App boots locally but fails in Cloud Foundry Confirm the service is bound in the target space and that the credentials JSON matches the required structure exactly, including the nested objects.

Deployer accounts

Touchpoints uses Cloud.gov Service Accounts to deploy the application from the Continuous Integration server.

Staging and Demo environments

• cf create-service cloud-gov-service-account space-deployer touchpoints-staging-deployer - creates the Service Account service
• cf create-service-key touchpoints-staging-deployer staging-deploy-key - creates the Service Account's Service Key
• cf service-key touchpoints-staging-deployer staging-deploy-key - diplays the username & password of the Service Account's Service Key

The Staging Service Account's username and password are used on CI to deploy to Staging and Demo.

Production environment

• cf create-service cloud-gov-service-account space-deployer touchpoints-deployer
• cf create-service-key touchpoints-deployer staging-deploy-key
• cf service-key touchpoints-deployer staging-deploy-key

Configuring S3

Touchpoints uses s3 buckets, which are provided as Cloud.gov service.

Touchpoints stores Organization logo image files, and .pdfs of Service Maps for a given Touchpoint.

Here's how to configure cloud.gov S3 for Touchpoints:

1. Create a public S3 service in cloud.gov
2. Bind that s3 instance to your Touchpoints application application
3. Then look at the VCAP settings of that cloud.gov application instance, and set those configs in Touchpoints /config/initializers/carrierwave.rb

• aws_access_key_id
• aws_secret_access_key
• region
• host
• fog_directory (the name of the S3 bucket)

Maintenance Page

Touchpoints has a Maintenance Page for display when the app is under extended maintenance or otherwise experiencing downtime.

The maintenance page is a separate application; with 1 html file. 1 maintenance page is maintained for each Cloud.gov space. In cloud.gov, the web application simply updates a virtual route to point the maintenance page app. When complete, the virtual route is pointed back at the web application.

Because Touchpoints maintains 3 "Spaces" (dev, staging, prod), the 3 maintenance pages exist as:

• touchpoints-dev-maintenance-page
• touchpoints-staging-maintenance-page
• touchpoints-prod-maintenance-page

Disabling health checks for workers

Because Worker instances do not have publicly accessible HTTP endpoints, health checks should be disabled for the app. Otherwise, cloud.gov will kill the application instance when it fails a health check.

• Staging - cf set-health-check touchpoints-staging-sidekiq-worker none
• Demo - cf set-health-check touchpoints-demo-sidekiq-worker none
• Production - cf set-health-check touchpoints-production-sidekiq-worker none

---

See the README at https://github.com/gsa/touchpoints-maintenance-page for information about the specific commands to create and use the maintenance page.

Scaling

To scale Touchpoints instances on cloud.gov:

• To view how many instances are set to run - cf scale instance-name
  • example: cf scale touchpoints
• To set how many instances should run - cf scale instance-name -i number-of-instances
  • example: cf scale touchpoints-staging -i 2

References

• https://cloud.gov
• https://docs.cloudfoundry.org