Skip to content

Make lint enforcing and run the sample tests in CI - #1

Merged
HarshCasper merged 4 commits into
mainfrom
ci/enforce-lint-and-run-tests
Oct 12, 2026
Merged

HarshCasper merged 4 commits into
mainfrom
ci/enforce-lint-and-run-tests

Conversation

@HarshCasper

@HarshCasper HarshCasper commented Oct 12, 2026 •

Copy link
Copy Markdown
Member

Both lint checks ended in || true, so they could never fail. And no workflow ran any sample's tests.

Lint (lint.yml)

  • shellcheck runs with --severity=error, without || true.
  • The Terraform check is now terraform fmt -check -recursive -diff ., and the pinned version goes from 1.5.0 to 1.14.9.
  • Ran terraform fmt -recursive (14 files, formatting only).
  • Added shebangs to emr-serverless-python-dependencies/start_job.sh and stop-application.sh.

Tests (new test.yml)

  • A prepare-list job finds the samples to test.
  • script-samples matrix: the samples with scripts/test.sh (10, with glue-etl-jobs skipped) run lstk start, then scripts/deploy.sh, then scripts/test.sh.
  • make-samples matrix: the 8 samples with a test-ci Makefile target run make test-ci.

CI runner setup

  • The runner's AWS CLI is v2, so the workflow sets cli_binary_format raw-in-base64-out. Without it, raw JSON lambda invoke payloads failed with "Invalid base64".
  • Installs uv for the websocket client and Terraform for emr-serverless.

Sample fixes

  • apigw-websockets/javascript:
    • test.sh connected without the stage in the URL, exited before printing the failure and summary, and counted uv's install output as part of the result.
    • handler.js returned the raw event. It now returns {statusCode, body}, without which LocalStack answers "Internal server error".
  • lambda-debugging-sam-*: test-ci now builds the function zip before deploying it.
  • terraform-resources:
    • Builds lambda.zip with archive_file, which nothing created before.
    • Uses python3.12 instead of python3.7.
    • Uses real trust policies for the two IAM roles.
  • emr-serverless-python-dependencies:
    • Accepts Terraform >= 1.9.1.
    • Waits for LocalStack after docker compose up.
    • Stops polling the job after 15 minutes.
  • glue-etl-jobs/python:
    • deploy.sh creates the cluster with --enable-http-endpoint.
    • teardown.sh works without .env, so a failed deploy can be cleaned up.

glue-etl-jobs is skipped in CI for now. This is a LocalStack bug, not a sample bug:

  • Glue always tries SSL to RDS first.
  • An aborted TLS handshake stops LocalStack's Postgres proxy for that cluster, so Glue's non-SSL fallback gets "Connection refused".
  • Repro without Glue: send a Postgres SSLRequest, close the socket, and later queries to the cluster fail.

Result: all 18 matrix jobs pass, including the 8 make test-ci samples.

The other 26 samples have no test scripts yet; they'll come in follow-ups.

https://claude.ai/code/session_01DZfDmzUBPqNAu3yFKyfCNM

- Set cli_binary_format for AWS CLI v2 on the runner, so raw JSON
  payloads in lambda invoke are not parsed as base64
- Install uv for the apigw-websockets client and Terraform for
  emr-serverless-python-dependencies
- lambda-debugging-sam-*: build the function zip in test-ci before deploying
- terraform-resources: build lambda.zip with archive_file, use python3.12,
  and use real trust policies for the IAM roles
- glue-etl-jobs: disable SSL on the JDBC connection; the Glue job container
  fails the TLS handshake with the LocalStack Postgres proxy

Claude-Session: https://claude.ai/code/session_01DZfDmzUBPqNAu3yFKyfCNM
- emr-serverless-python-dependencies: allow Terraform >= 1.9.1, wait for
  LocalStack after docker compose up, and stop polling the job after 15 minutes
- Give make test-ci 30 minutes, since the first EMR Spark run is slow
- glue-etl-jobs: Glue always attempts SSL to RDS first, and an aborted TLS
  handshake stops the LocalStack Postgres proxy for that cluster, so the
  non-SSL fallback is refused. Skip it in the matrix until that is fixed.
  Revert the sslmode change, which did not help.
- glue-etl-jobs teardown: fall back to the fixed resource names when .env
  is missing, so a failed deploy can be cleaned up

Claude-Session: https://claude.ai/code/session_01DZfDmzUBPqNAu3yFKyfCNM
@HarshCasper
HarshCasper merged commit f0727f3 into main Oct 12, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant