[YAML] Add SplunkIO write SchemaTransform and WriteToSplunk

[schizophrenicmaniac]

Adds a SchemaTransform for SplunkIO writes and exposes it in Beam YAML as WriteToSplunk. It follows the same approach as the Datadog one (#38362). SplunkIO in Java only supports writes, so there is no ReadFromSplunk here.

Each input row becomes a SplunkEvent. event (string) is required. time (int64), host, source, sourcetype, index and fields are optional. fields can be a nested row or a JSON object string, and other columns are ignored. The config covers everything SplunkIO.Write supports (url, token, batch_count, parallelism, disable_certificate_validation, root_ca_certificate_path, enable_batch_logs, enable_gzip_http_compression) plus error_handling. The error output has the same shape as Datadog's: rows that can't be converted come back with failed_row and a 400, and HEC write failures come back with the HTTP status and the payload. Without error_handling, a failed write fails the pipeline.

While testing this I found that SplunkWriteError loses all of its values once it goes through a coder. It has @DefaultSchema(AutoValueSchema.class), but its accessors aren't get-prefixed, so the inferred schema has no fields and every error decodes as null/null/null. SplunkEventWriterTest doesn't catch this because containsInAnyOrder decodes both sides with the same coder. I replaced the annotation with a hand-written SplunkWriteErrorSchemaProvider, so SplunkWriteError is still encoded as a row, now with statusCode, statusMessage and payload fields. This changes the output coder of SplunkIO.Write, so I listed it under bugfixes in CHANGES.md.

Changes:

• SplunkWriteSchemaTransformProvider / SplunkWriteSchemaTransformConfiguration, registered in the IO expansion service
• WriteToSplunk in standard_io.yaml and the matching entry in standard_external_transforms.yaml
• SplunkWriteErrorSchemaProvider, used as the default schema of SplunkWriteError
• Java tests against a mock HEC server, and a YAML integration test with a fake HEC fixture (tests/splunk.yaml)

Fixes #40470

[codecov]

Codecov Report

❌ Patch coverage is 0% with 109 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.75%. Comparing base (76f6e6d) to head (ef96d29).
⚠️ Report is 181 commits behind head on master.

Files with missing lines	Patch %	Lines
...n/apache_beam/yaml/test_utils/splunk_test_utils.py	0.00%	103 Missing ⚠️
sdks/python/apache_beam/yaml/integration_tests.py	0.00%	6 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #40481      +/-   ##
============================================
+ Coverage     56.16%   58.75%   +2.58%     
- Complexity     2288    13977   +11689     
============================================
  Files          1124     2596    +1472     
  Lines        178336   272570   +94234     
  Branches       1489    11305    +9816     
============================================
+ Hits         100170   160150   +59980     
- Misses        75645   106395   +30750     
- Partials       2521     6025    +3504     

Flag	Coverage Δ
java	65.05% <ø> (-10.74%)	⬇️
python	79.79% <0.00%> (+0.23%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Assigning reviewers:

R: @jrmccluskey for label python.
R: @kennknowles for label java.
R: @derrickaw for label yaml.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[derrickaw]

waiting on author

[schizophrenicmaniac]

Merged master and resolved the CHANGES.md conflict, keeping both entries.

[derrickaw]

A few comments on yaml related code.

[derrickaw]

LGTM for yaml/python

@kennknowles, can you review the java part? Thanks

[kennknowles]

As you've probably noticed, the trouble with this is that it requires a special wrapper to be encoded as a row to be used by other languages. This is not such a terrible thing, but it would be better to do a hand-written schema coder so that SplunkWriteError is still encoded as a row, it just cannot use the @AutoValueSchema annotation to do it (though fixing the get-prefix limitation might be a one-shot prompt...)

This would look like making a new SchemaCoder(toFunction, fromFunction) where the to/from are just the same logic that you have in the SchemaTransform wrapper.

Does this work? Am I overlooking at issue?

---

Aside: if we were to switch to the new coder since the @AutoValueSchema annotation is not working as intended, we should replace it in the other file like so:

@DefaultCoder(SplunkWriteErrorCoder.class)
public class SplunkWriteError {
 ...
}

Then the coder will be automatic, here and everywhere.

But again, I think we can keep it encoded as a row so we don't have the breaking change and it works smoothly with the schema transform.

[schizophrenicmaniac]

Yes, that works. I dropped SplunkWriteErrorCoder and added a SplunkWriteErrorSchemaProvider with hand-written to/from row functions (fields statusCode, statusMessage, payload). SplunkWriteError now uses it via @DefaultSchema instead of AutoValueSchema, so the setCoder call in SplunkIO.Write is gone and the output is inferred as a SchemaCoder like before, just with actual fields now. The schema transform still builds its own error rows since it adds failed_row.

I didn't touch the get-prefix handling in AutoValueSchema. That would change the inferred schema, and so the coder, of any other AutoValue type that has non-get accessors, so it seemed better kept out of this PR.

[schizophrenicmaniac]

@kennknowles I switched SplunkWriteError over to a hand-written schema provider as you suggested, could you take another look when you get a chance?