Fix:stale parameter handling in the Pipeline Executor when retries are enabled and variables are inherited from parent scopes by beatum · Pull Request #9059 · apache/hop

beatum · 2026-10-21T07:35:43Z

Summary

This PR fixes stale parameter handling in the Pipeline Executor when retries are enabled and variables are inherited from parent scopes.

The previous retry flow could reuse stale inherited variable values or previous attempt parameter values, causing child pipeline executions to behave inconsistently across retries. This patch ensures each retry reflects the newest available context while preserving explicit mappings.

Testing

Added/updated PipelineExecutor unit tests covering variable refresh, retry value selection, and retry execution sequencing.

…n and improve parameter handling

…inMeta

…re parameter specification is initialized

…hance metadata discovery

…JoinMeta

- Skip only explicit JSONB operator tokens: `?|`, `?&`, `??`, `??|`, and `??&`. - Consume complete operator tokens so `col ?? ?` still counts the trailing `?` as a bind parameter. - Stop treating all `[...]` expressions as bracket-quoted identifiers. - Add dialect-aware bracket handling for SQL Server. - Preserve PostgreSQL parameter markers in `ARRAY[?]` and `ARRAY[?, ?]`. - Apply the same parsing behavior to the Database Join path. - Expand regression coverage for JSONB operators, `LIKE`, `NOT LIKE`, `BETWEEN`, `SELECT ? FROM dual`, and ARRAY expressions.

…et-quoted identifiers

…r parsing

…B operators and bracket-quoted identifiers

…parameter handling

… logic for pipeline execution

…for improved error handling

…ion attempts

…error handling

…nhance retry logic tests

…imeout

…us attempts

…ry behavior and timing

…ttemptPipeline

…s from external scopes

…PipelineExecutor

hansva · 2026-10-21T11:42:15Z

Can we have a ticket and a reproduction path for this please. I'm not sure the code change reflect the expectations you/we have. As far as I can see every retry read the same row, and that row doesn't change while the retries run.

…gic and improve retry handling

…PipelineExecutor

beatum · 2026-10-22T03:32:05Z

Thanks again for the detailed review and feedback. i have implemented all three fixes in PipelineExecutor and updated the tests to lock in the expected behavior.

Retry parameter handling

executeWithRetries() now always reuses the original attempt-0 parameter values on retries.
Fixed the buffered-row fallback to preserve column positions (including null values) and use row metadata conversion instead of Object.toString().

Variable refresh on retries

Retry attempts now refresh variables from the live parent variable scopes and JVM properties instead of relying on the pipeline's copied snapshot.
Transform variables are also refreshed so inherited child parameters and static values such as ${ACCESS_TOKEN} pick up updated values before each retry.

Test updates

Replaced the stub-based retry test with a real-path scenario that verifies both retry attempts receive the original attempt-0 parameter list, including correct null handling.
Added a test to verify row metadata conversion is used instead of raw toString().
Strengthened the variable refresh tests to cover updated parent-scope values and JVM property fallback behavior across retry attempts.

beatum · 2026-10-23T01:34:27Z

Thanks for the quick response @hansva .

Of course. I've prepared a small example for testing purposes and uploaded it to the following repository:

https://github.com/beatum/apache-hop-demo.git

When the variable is set to rabbit, the pipeline logs that the door opens.
When the variable is set to monkey, the pipeline follows the monkey branch and demonstrates the retry/recovery behavior.

2026/10/23 08:59:24 - Hop - Pipeline opened.
2026/10/23 08:59:24 - Hop - Launching pipeline [main]...
2026/10/23 08:59:24 - Hop - Started executing pipeline 'main'.
2026/10/23 08:59:24 - GUI - Getting revisions for path - original: 'main.hpl', normalized: 'main.hpl'
2026/10/23 08:59:24 - main - Executing this pipeline using the Local Pipeline Engine with run configuration 'local'
2026/10/23 08:59:24 - main - Execution started for pipeline [main]
2026/10/23 08:59:24 - SysInfo.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2026/10/23 08:59:24 - room - Executing this pipeline using the Local Pipeline Engine with run configuration 'local'
2026/10/23 08:59:24 - room - Execution started for pipeline [room]
2026/10/23 08:59:24 - SysInfo.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2026/10/23 08:59:24 - JavaScript.0 - ECMAScript language level set to ES6.
2026/10/23 08:59:24 - JavaScript.0 - Optimization level set to 9.
2026/10/23 08:59:25 - JavaScript.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2026/10/23 08:59:25 - Monkey.0 - 
2026/10/23 08:59:25 - Monkey.0 - ------------> Linenr 1------------------------------
2026/10/23 08:59:25 - Monkey.0 - Monkey 🐒🐒🐒 can't open the door without a key yet.
2026/10/23 08:59:25 - Monkey.0 - 🔒🔒🔒🔒🔒🔒🔒🔒🔒🔒🔒🔒🔒🔒🔒
2026/10/23 08:59:25 - Monkey.0 - 
2026/10/23 08:59:25 - Monkey.0 - animal = monkey
2026/10/23 08:59:25 - Monkey.0 - 
2026/10/23 08:59:25 - Monkey.0 - ====================
2026/10/23 08:59:25 - Door.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2026/10/23 08:59:25 - Monkey.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2026/10/23 08:59:25 - changeAnimal - Executing this pipeline using the Local Pipeline Engine with run configuration 'local'
2026/10/23 08:59:25 - changeAnimal - Execution started for pipeline [changeAnimal]
2026/10/23 08:59:25 - SysInfo.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2026/10/23 08:59:25 - JavaScript.0 - ECMAScript language level set to ES6.
2026/10/23 08:59:25 - JavaScript.0 - Optimization level set to 9.
2026/10/23 08:59:25 - Write to log.0 - 
2026/10/23 08:59:25 - Write to log.0 - ------------> Linenr 1------------------------------
2026/10/23 08:59:25 - Write to log.0 - In the Change Animal pipelie===============================
2026/10/23 08:59:25 - Write to log.0 - 
2026/10/23 08:59:25 - Write to log.0 - animal = rabbit
2026/10/23 08:59:25 - Write to log.0 - 
2026/10/23 08:59:25 - Write to log.0 - ====================
2026/10/23 08:59:25 - JavaScript.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2026/10/23 08:59:25 - Write to log.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2026/10/23 08:59:25 - changeAnimal - Pipeline duration : 0.039 seconds [  0.039" ]
2026/10/23 08:59:25 - changeAnimal - Execution finished on a local pipeline engine with run configuration 'local'
2026/10/23 08:59:25 - MonkeyRecovered.0 - 
2026/10/23 08:59:25 - MonkeyRecovered.0 - ------------> Linenr 1------------------------------
2026/10/23 08:59:25 - MonkeyRecovered.0 - Monkey recovered the key and the animal was reset to rabbit. The door can now be opened.
2026/10/23 08:59:25 - MonkeyRecovered.0 - 
2026/10/23 08:59:25 - MonkeyRecovered.0 - ExecutionTime = 50
2026/10/23 08:59:25 - MonkeyRecovered.0 - ExecutionResult = Y
2026/10/23 08:59:25 - MonkeyRecovered.0 - ExecutionNrErrors = 0
2026/10/23 08:59:25 - MonkeyRecovered.0 - ExecutionLinesRead = 1
2026/10/23 08:59:25 - MonkeyRecovered.0 - ExecutionLinesWritten = 1
2026/10/23 08:59:25 - MonkeyRecovered.0 - ExecutionLinesInput = 0
2026/10/23 08:59:25 - MonkeyRecovered.0 - ExecutionLinesOutput = 0
2026/10/23 08:59:25 - MonkeyRecovered.0 - ExecutionLinesRejected = 0
2026/10/23 08:59:25 - MonkeyRecovered.0 - ExecutionLinesUpdated = 0
2026/10/23 08:59:25 - MonkeyRecovered.0 - ExecutionLinesDeleted = 0
2026/10/23 08:59:25 - MonkeyRecovered.0 - ExecutionFilesRetrieved = 0
2026/10/23 08:59:25 - MonkeyRecovered.0 - ExecutionExitStatus = 0
2026/10/23 08:59:25 - MonkeyRecovered.0 - ExecutionLogText = 2026/10/23 08:59:25 - changeAnimal - Executing this pipeline using the Local Pipeline Engine with run configuration 'local'
2026/10/23 08:59:25 - MonkeyRecovered.0 - 2026/10/23 08:59:25 - changeAnimal - Execution started for pipeline [changeAnimal]
2026/10/23 08:59:25 - MonkeyRecovered.0 - 2026/10/23 08:59:25 - SysInfo.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2026/10/23 08:59:25 - MonkeyRecovered.0 - 2026/10/23 08:59:25 - JavaScript.0 - ECMAScript language level set to ES6.
2026/10/23 08:59:25 - MonkeyRecovered.0 - 2026/10/23 08:59:25 - JavaScript.0 - Optimization level set to 9.
2026/10/23 08:59:25 - MonkeyRecovered.0 - 2026/10/23 08:59:25 - Write to log.0 - 
2026/10/23 08:59:25 - MonkeyRecovered.0 - 2026/10/23 08:59:25 - Write to log.0 - ------------> Linenr 1------------------------------
2026/10/23 08:59:25 - MonkeyRecovered.0 - 2026/10/23 08:59:25 - Write to log.0 - In the Change Animal pipelie===============================
2026/10/23 08:59:25 - MonkeyRecovered.0 - 2026/10/23 08:59:25 - Write to log.0 - 
2026/10/23 08:59:25 - MonkeyRecovered.0 - 2026/10/23 08:59:25 - Write to log.0 - animal = rabbit
2026/10/23 08:59:25 - MonkeyRecovered.0 - 2026/10/23 08:59:25 - Write to log.0 - 
2026/10/23 08:59:25 - MonkeyRecovered.0 - 2026/10/23 08:59:25 - Write to log.0 - ====================
2026/10/23 08:59:25 - MonkeyRecovered.0 - 2026/10/23 08:59:25 - JavaScript.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2026/10/23 08:59:25 - MonkeyRecovered.0 - 2026/10/23 08:59:25 - Write to log.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2026/10/23 08:59:25 - MonkeyRecovered.0 - 2026/10/23 08:59:25 - changeAnimal - Pipeline duration : 0.039 seconds [  0.039" ]
2026/10/23 08:59:25 - MonkeyRecovered.0 - 2026/10/23 08:59:25 - changeAnimal - Execution finished on a local pipeline engine with run configuration 'local'
2026/10/23 08:59:25 - MonkeyRecovered.0 - 
2026/10/23 08:59:25 - MonkeyRecovered.0 - ExecutionLogChannelId = f4ce3c6f-a3ca-4ee1-9419-50473af9f315
2026/10/23 08:59:25 - MonkeyRecovered.0 - 
2026/10/23 08:59:25 - MonkeyRecovered.0 - ====================
2026/10/23 08:59:25 - changeAnimal.hpl.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2026/10/23 08:59:25 - MonkeyRecovered.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2026/10/23 08:59:25 - Abort.0 - ERROR: Row nr 1 causing abort "Abort and log as an error": [50], [Y], [0], [1], [1], [0], [0], [0], [0], [0], [0], [0], [2026/10/23 08:59:25 - changeAnimal - Executing this pipeline using the Local Pipeline Engine with run configuration 'local'
2026/10/23 08:59:25 - Abort.0 - 2026/10/23 08:59:25 - changeAnimal - Execution started for pipeline [changeAnimal]
2026/10/23 08:59:25 - Abort.0 - 2026/10/23 08:59:25 - SysInfo.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2026/10/23 08:59:25 - Abort.0 - 2026/10/23 08:59:25 - JavaScript.0 - ECMAScript language level set to ES6.
2026/10/23 08:59:25 - Abort.0 - 2026/10/23 08:59:25 - JavaScript.0 - Optimization level set to 9.
2026/10/23 08:59:25 - Abort.0 - 2026/10/23 08:59:25 - Write to log.0 - 
2026/10/23 08:59:25 - Abort.0 - 2026/10/23 08:59:25 - Write to log.0 - ------------> Linenr 1------------------------------
2026/10/23 08:59:25 - Abort.0 - 2026/10/23 08:59:25 - Write to log.0 - In the Change Animal pipelie===============================
2026/10/23 08:59:25 - Abort.0 - 2026/10/23 08:59:25 - Write to log.0 - 
2026/10/23 08:59:25 - Abort.0 - 2026/10/23 08:59:25 - Write to log.0 - animal = rabbit
2026/10/23 08:59:25 - Abort.0 - 2026/10/23 08:59:25 - Write to log.0 - 
2026/10/23 08:59:25 - Abort.0 - 2026/10/23 08:59:25 - Write to log.0 - ====================
2026/10/23 08:59:25 - Abort.0 - 2026/10/23 08:59:25 - JavaScript.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2026/10/23 08:59:25 - Abort.0 - 2026/10/23 08:59:25 - Write to log.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2026/10/23 08:59:25 - Abort.0 - 2026/10/23 08:59:25 - changeAnimal - Pipeline duration : 0.039 seconds [  0.039" ]
2026/10/23 08:59:25 - Abort.0 - 2026/10/23 08:59:25 - changeAnimal - Execution finished on a local pipeline engine with run configuration 'local'
2026/10/23 08:59:25 - Abort.0 - ], [f4ce3c6f-a3ca-4ee1-9419-50473af9f315]
2026/10/23 08:59:25 - Abort.0 - ERROR: Aborting after having seen 1 rows.
2026/10/23 08:59:25 - Abort.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=1)
2026/10/23 08:59:25 - room - Pipeline duration : 0.589 seconds [  0.589" ]
2026/10/23 08:59:25 - room - Execution finished on a local pipeline engine with run configuration 'local'
2026/10/23 08:59:25 - room - Pipeline detected one or more transforms with errors.
2026/10/23 08:59:25 - room - Pipeline is killing the other transforms!
2026/10/23 08:59:25 - room - Executing this pipeline using the Local Pipeline Engine with run configuration 'local'
2026/10/23 08:59:25 - room - Execution started for pipeline [room]
2026/10/23 08:59:25 - SysInfo.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2026/10/23 08:59:25 - JavaScript.0 - ECMAScript language level set to ES6.
2026/10/23 08:59:25 - JavaScript.0 - Optimization level set to 9.
2026/10/23 08:59:25 - Lucky.0 - 
2026/10/23 08:59:25 - Lucky.0 - ------------> Linenr 1------------------------------
2026/10/23 08:59:25 - Lucky.0 - The door is opened===================== 🐇🐇🐇🐇🐇🐇🐇🐇🐇🐇
2026/10/23 08:59:25 - Lucky.0 - Lucky 🎆🎆🎆🎆🎆🎆🎆🎆🎆
2026/10/23 08:59:25 - Lucky.0 - 
2026/10/23 08:59:25 - Lucky.0 - animal = rabbit
2026/10/23 08:59:25 - Lucky.0 - 
2026/10/23 08:59:25 - Lucky.0 - ====================
2026/10/23 08:59:25 - JavaScript.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2026/10/23 08:59:25 - Door.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2026/10/23 08:59:25 - Lucky.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2026/10/23 08:59:25 - room - Pipeline duration : 0.098 seconds [  0.098" ]
2026/10/23 08:59:25 - room - Execution finished on a local pipeline engine with run configuration 'local'
2026/10/23 08:59:25 - room.hpl.0 - Finished processing (I=0, O=0, R=1, W=0, U=0, E=0)
2026/10/23 08:59:25 - main - Pipeline duration : 0.849 seconds [  0.849" ]
2026/10/23 08:59:25 - Hop - Pipeline 'main' completed.
2026/10/23 08:59:25 - main - Execution finished on a local pipeline engine with run configuration 'local'

Feel free to try it out, and let me know if you need any additional information.

Have fun! 😄

happy.he and others added 30 commits September 11, 2026 10:07

Enhance DatabaseJoin to support named and positional SQL parameters

be7999e

Enhance DatabaseJoin to support runtime output metadata initializatio…

637110e

…n and improve parameter handling

Add tests for mixed named and positional SQL parameters in DatabaseJo…

c0464aa

…inMeta

Added a new core SQL parser: parseSqlParameterSpec(String) that returns

ba7499e

Refactor DatabaseJoin to simplify output field name handling and ensu…

594367b

…re parameter specification is initialized

Refactor DatabaseJoinMeta to streamline SQL parameter handling and en…

7d200c9

…hance metadata discovery

Add tests for SQL parameter parsing and metadata handling in Database…

e099c7a

…JoinMeta

Merge branch 'apache:main' into main

0a9322b

Enhance parseSqlParameterSpec() call in DatabaseJoin to support brack…

ec73098

…et-quoted identifiers

Refactor parseSqlParameterSpec() in DatabaseJoinMeta to support brack…

472cfc1

…et-quoted identifiers

Add tests for mixed named and positional placeholders in SQL paramete…

6a15572

…r parsing

Refactor SQL parameter tests to improve clarity and coverage for JSON…

39e61db

…B operators and bracket-quoted identifiers

Add resolved parameters panel to DatabaseJoinDialog for improved SQL …

b532564

…parameter handling

Merge branch 'apache:main' into main

3bd7685

Refactor PipelineExecutor to improve parameter handling and add retry…

857fe36

… logic for pipeline execution

Add retry attempts and delay configuration to PipelineExecutorDialog …

557085e

…for improved error handling

Add unit tests for retry logic in PipelineExecutor to validate execut…

48b5419

…ion attempts

Add retry configuration panel to PipelineExecutorDialog for enhanced …

76371b6

…error handling

Add support for retry options in PipelineExecutorMeta with XML handling

8253e09

Add timeout handling to executePipelineAttempt for retry logic

d60b38a

Add support for additional parameters in executePipelineAttempt and e…

a052d64

…nhance retry logic tests

Merge branch 'apache:main' into main

268cf79

Merge branch 'apache:main' into main

9172069

Add log message for interrupted retries and update tooltip for wait t…

951e1b3

…imeout

Refactor retry logic in executeWithRetries and add cleanup for previo…

9607644

…us attempts

Add tests for cleanup and retry logic in PipelineExecutor

4689003

Update documentation for pipeline execution parameters to clarify ret…

fc312a0

…ry behavior and timing

Refactor cleanup logic in executePipelineAttempt to use stopPreviousA…

7c69fc9

…ttemptPipeline

Add tests for cleanup behavior in executeWithRetries method

35823f9

beatum added 3 commits October 21, 2026 09:05

Merge branch 'apache:main' into main

57135db

Enhance retry logic in PipelineExecutor to refresh parameter variable…

d780d1d

…s from external scopes

Add tests for refreshing parameter variables from external scopes in …

58117a1

…PipelineExecutor

mattcasters requested changes Oct 21, 2026

View reviewed changes

beatum added 2 commits October 22, 2026 11:06

Refactor PipelineExecutor to streamline parameter variable refresh lo…

7af57c0

…gic and improve retry handling

Add tests for refreshing parameter variables from external scopes in …

47376c4

…PipelineExecutor

Navigation Menu

Repository navigation

Fix:stale parameter handling in the Pipeline Executor when retries are enabled and variables are inherited from parent scopes - #9059

Fix:stale parameter handling in the Pipeline Executor when retries are enabled and variables are inherited from parent scopes#9059
beatum wants to merge 35 commits into
apache:mainfrom
beatum:fix-cache-variable

beatum commented Oct 21, 2026

Uh oh!

hansva commented Oct 21, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

beatum commented Oct 22, 2026

Uh oh!

beatum commented Oct 23, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

beatum commented Oct 21, 2026

Summary

Testing

Uh oh!

hansva commented Oct 21, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

beatum commented Oct 22, 2026

Retry parameter handling

Variable refresh on retries

Test updates

Uh oh!

beatum commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

beatum commented Oct 23, 2026 •

edited

Loading