diff --git a/CHANGELOG.md b/CHANGELOG.md index 7b28f572..a5059eac 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,8 +3,9 @@ Latest ## Changes * [#190](https://github.com/cleverage/process-bundle/issues/190) Update quality stack: use Rector `withComposerBased()` sets (removed `SYMFONY_64` / `PHPUNIT_100` sets), declare used Symfony packages and PHPUnit range in composer.json, apply quality tools fixes +* [#145](https://github.com/cleverage/process-bundle/issues/145) Add missing documentations: reference pages for every Task & Transformer, ConditionTrait & GenericTransformer, complete guides and cookbooks. Harmonize and fix existing documentation. -v5.0 +* v5.0 ----- ## Changes diff --git a/docs/01-quick_start.md b/docs/01-quick_start.md index efcc83c7..4b7359b9 100644 --- a/docs/01-quick_start.md +++ b/docs/01-quick_start.md @@ -3,8 +3,9 @@ Quick start ## Base concepts -In most application, there's always a set of workflows defining how to manage your data. It can be imports/exports, -asynchronous treatments or periodically checking an API... With its life, it may grow, code may duplicate quite quickly. +In most applications, there's always a set of workflows defining how to manage your data. It can be imports/exports, +asynchronous treatments or periodically checking an API... Over time these workflows grow, and code may duplicate +quite quickly. This bundle aims to provide a framework to build efficient, quick to build, easy to change workflows. @@ -29,51 +30,73 @@ Open a command console, enter your project directory and install it using compos composer require cleverage/process-bundle ``` -Remember to add the following line to config/bundles.php (not required if Symfony Flex is used) +Remember to add the following line to `config/bundles.php` (not required if Symfony Flex is used): ```php CleverAge\ProcessBundle\CleverAgeProcessBundle::class => ['all' => true], ``` -Some tasks and transformers use the main Symfony serializer service. You might need to explicitly enable it, or dependency -resolution might fail -* https://symfony.com/doc/current/reference/configuration/framework.html#reference-serializer-enabled +Some tasks and transformers use the main Symfony serializer service, and the bundle checks at container compilation +that it is available: if it is not, the build fails with an explicit message. Make sure it is enabled +(see [`framework.serializer.enabled`](https://symfony.com/doc/current/reference/configuration/framework.html#reference-serializer-enabled)): + +```yaml +# config/packages/framework.yaml +framework: + serializer: + enabled: true +``` ## Global configuration You can use `./bin/console config:dump-reference clever_age_process` to have a summary of current configuration. -Aside from process and transformer configurations, there is the `default_error_strategy` setting that allow you to define -behavior if a task encounter an error. Up to v3.0, the default value was to `skip` iterations with errors. Starting from v3.1, -the configuration should be defined by the user. +The configuration has three root keys: +- `configurations`: your processes (see [process definition](reference/01-process_definition.md)) +- `generic_transformers`: reusable transformers built from configuration (see + [generic transformers definition](reference/03-generic_transformers_definition.md)) +- `default_error_strategy`: the behavior of a task that encounters an error when it does not define its own + `error_strategy`. Allowed values are `stop` (the default) and `skip`. -We recommend to use the `stop` configuration (see bellow), and then specify task by task which one can be `skipped`. +We recommend keeping the `stop` default, and then specify task by task which one can be skipped: -Recommended example : ```yaml +# config/packages/clever_age_process.yaml clever_age_process: default_error_strategy: stop ``` -When creating custom tasks and transformers, you can use Symfony automatic registration, but remember there is a few required configurations : +When creating custom tasks and transformers, you can use Symfony automatic registration, but remember there are a few +required configurations: + ```yaml +# config/services.yaml services: App\Transformer\: - resource: 'relative/path/to/Transformer/*' - autowire: true + resource: '../src/Transformer/*' + autowire: true autoconfigure: true - public: false + public: false tags: - - { name: cleverage.transformer } # Needed by the process registry to find transformers + - { name: cleverage.transformer } # Needed by the transformer registry to find transformers + - { name: monolog.logger, channel: cleverage_process_transformer } # Optional, see logging App\Task\: - resource: 'relative/path/to/Task/*' - autowire: true + resource: '../src/Task/*' + autowire: true autoconfigure: true - shared: false # Important to avoid shared data between task usage - public: true # Needed by the Process Manager to find tasks + shared: false # Important to avoid shared data between task usages + public: true # Needed by the Process Manager to fetch tasks from the container + tags: + - { name: monolog.logger, channel: cleverage_process_task } # Optional, see logging ``` +The `cleverage.transformer` tag is not added by autoconfiguration: you have to declare it yourself (or use an +`_instanceof` rule on `CleverAge\ProcessBundle\Transformer\TransformerInterface`, which only applies to services +defined in the same file). If your `services.yaml` also has the default `App\:` resource, declare these resources after +it, so that they override its definitions. The `monolog.logger` tags bind injected loggers to the bundle channels (see +[logging](03-custom_tasks.md#logging)). + ## Process definition Most of the work is done through the bundle configuration. @@ -88,7 +111,7 @@ clever_age_process: ``` Then you can add tasks in this array. They consist of a `service`, optionally configured by `options`, and eventually - chained with others through their `outputs`. Minimal syntax is: +chained with others through their `outputs`. Minimal syntax is: ```yaml : service: @@ -99,14 +122,21 @@ Then you can add tasks in this array. They consist of a `service`, optionally co outputs: [, , ] ``` -Below you can see a minimal working ETL example. It consist of 3 tasks: -- the first *extract* some data (the [constant output task](reference/tasks/constant_output_task.md) outputs... a constant value): it's an array with 3 +Below you can see a minimal working ETL example. It consists of 3 tasks: +- the first *extracts* some data (the [constant output task](reference/tasks/constant_output_task.md) outputs... a constant value): it's an array with 3 keys/values -- the second *transform* the given value (the [transformer task](reference/tasks/transformer_task.md) is one of the most important!): the output is then an +- the second *transforms* the given value (the [transformer task](reference/tasks/transformer_task.md) is one of the most important!): the output is then an array with 2 keys/values, created using the value from previous task - finally, the last will just display the result (it's a cheap *load*, using the [debug task](reference/tasks/debug_task.md), only for development purpose!) +The debug task only dumps its input if the Symfony VarDumper component is installed (otherwise it silently does +nothing): `composer require --dev symfony/var-dumper`. + +Put this configuration in any file loaded by Symfony, e.g. `config/packages/clever_age_process.yaml`, or one file per +process in a `config/packages/process/` folder imported with `imports: [{ resource: process/ }]` (see +[common setup](cookbooks/01-common_setup.md#configuration)). + ```yaml clever_age_process: configurations: @@ -143,22 +173,24 @@ clever_age_process: service: '@CleverAge\ProcessBundle\Task\Debug\DebugTask' ``` -There is more to know about process configuration. See [the full process configuration reference](). +There is more to know about process configuration. See [the full process configuration reference](reference/01-process_definition.md) +and [the task definition reference](reference/02-task_definition.md). ## Command line usage -Once your process are defined, you want to use them. Some console commands are provided for their manipulation: -- `cleverage:process:list`: gives you a list of all defined process -- `cleverage:process:help `: tries to show you what's inside `` using a nice charting -- `cleverage:process:execute [ ...]`: starts one by one ``, -``, ... , unrolling tasks one by one. Note that you can use verbosity options (`-v`, `-vv`, `-vvvv`) -to look in depth what's happening. +Once your processes are defined, you want to use them. Some console commands are provided for their manipulation: +- `cleverage:process:list [--all|-a]`: gives you a list of all defined public processes (`--all` also shows private + ones) +- `cleverage:process:help `: shows the description, the help and the tree of tasks of `` +- `cleverage:process:execute [ ...]`: executes one by one ``, + ``, ... Note that you can use verbosity options (`-v`, `-vv`, `-vvv`) to look in depth at what's + happening. Applied to previous example, it will show: ``` $ ./bin/console cleverage:process:list -There are 1 process configurations defined : +There are 1 process configurations defined (and 0 private) : - project_prefix.process_name with 3 tasks ``` @@ -178,7 +210,6 @@ Tasks tree: ``` $ ./bin/console cleverage:process:execute project_prefix.process_name Starting process 'project_prefix.process_name'... -DEBUG from project_prefix.process_name::load array:2 [ "id" => 123 "slug" => "123-Test1-Test2" @@ -186,14 +217,57 @@ array:2 [ Process 'project_prefix.process_name' executed successfully ``` +### Options of the execute command + +| Option | Shortcut | Description | +| ------ | :------: | ----------- | +| `--input=` | `-i` | Value given as input to the task defined by the process `entry_point` (ignored with a warning if the process has no entry point) | +| `--input-from-stdin` | | Read the input value from STDIN instead (e.g. `cat file.json \| bin/console cleverage:process:execute ...`) | +| `--context=:` | `-c` | Contextual value, can be repeated. The key must only contain word characters (`\w+`: letters, digits, `_`). The value is parsed as YAML, so `-c limit:10` gives an integer and `-c name:"'foo'"` forces a string (also needed for dates: `-c date:2024-01-01` gives an integer timestamp) | +| `--output=` | `-o` | Where to dump the value returned by the task defined by the process `end_point`: `-` (default) for STDOUT, or a file path | +| `--output-format=` | `-t` | Format of the dumped output: `dump` (Symfony VarDumper, STDOUT only, requires `-vv`) or `json-stream` (written to STDOUT with `-vv`, or into the `--output` file, only if the value is an array). Nothing is dumped when this option is omitted. Without `end_point`, the output is `null` | + +Example: + +```bash +./bin/console cleverage:process:execute project_prefix.import --input=/tmp/products.csv -c "delimiter:';'" -c limit:100 +``` + +### Contextual values + +Values passed with `--context` (or with the `$context` argument of `ProcessManager::execute()`, see +[executing a process from PHP](04-advanced_workflow.md#executing-a-process-from-php)) are available in every +task of the process: +- in the task options, a string `{{ key }}` is replaced by the value of the `key` context entry. If the whole option + value is a placeholder, the raw value is injected (it can be an array or an integer), otherwise it is replaced inside + the string. This is done before options are resolved, so it works for every task extending + `AbstractConfigurableTask` (through `ProcessState::getContextualizedOptions()`) +- in PHP, with `ProcessState::getContext()` or `ProcessState::getContextualizedOption($code, $default)` + +```yaml +read: + service: '@CleverAge\ProcessBundle\Task\File\Csv\CsvReaderTask' + options: + file_path: '%kernel.project_dir%/var/import/{{ file_name }}' +``` + +Note that placeholders are only resolved for keys existing in the context: if you run the process without the +`file_name` context value, the option will keep the literal `{{ file_name }}` string. + ## Automation Once everything is working fine, you may want to automate your processes. The standard way is using the Unix cron jobs: ``` # Every two hours, execute -0 */2 * * * ./bin/console cleverage:process:execute +0 */2 * * * /path/to/project/bin/console cleverage:process:execute --env=prod ``` -To check if everything went fine, logs are stored in database: -- `clever_process_history`: logs process started, with `process_code`, `start_date`, `end_date` and `statut` -- `clever_task_history`: logs custom tasks logs (see [logging]), with `task_code`, `message`, `logged_at` date, `level`, a `reference` and `context` +This bundle does not store any execution history in database. Process and task logs are sent to dedicated Monolog +channels (`cleverage_process`, `cleverage_process_task` and `cleverage_process_transformer`, see +[logging](03-custom_tasks.md#logging)), so you can route them to any handler. Each record is enriched with the +process code, a process execution id and the context (plus the task code and service for task and transformer logs). + +If you need a web interface to launch, schedule and follow executions, have a look at +[cleverage/ui-process-bundle](https://github.com/cleverage/ui-process-bundle): it listens to the process events +(see [events](04-advanced_workflow.md#events)) to persist every execution and its logs in database, and provides a +scheduler based on Symfony Scheduler. diff --git a/docs/02-task_types.md b/docs/02-task_types.md index 6639f027..53400464 100644 --- a/docs/02-task_types.md +++ b/docs/02-task_types.md @@ -3,98 +3,166 @@ Task types ## Task definition -Tasks are symfony services that implements `CleverAge\ProcessBundle\Model\TaskInterface`. +Tasks are Symfony services that implement `CleverAge\ProcessBundle\Model\TaskInterface`, which only defines one +method: `execute(ProcessState $state): void`. -Most of them takes an input to produce an output, but others might download a file, write a CSV, load database data... +Most of them take an input to produce an output, but others might download a file, write a CSV, load database data... Step by step, tasks are chained together according to the process workflow definition. -A task can be executed one or multiple times. +A task can be executed one or multiple times during a process: once for each input it receives from its parents. + +On top of `TaskInterface`, a task can implement several optional interfaces (all in the `CleverAge\ProcessBundle\Model` +namespace) that change how the process manager drives it. They are described below. + +A task is *resolved* when all its executions are over, as well as those of all its ancestors (see +[task resolution](04-advanced_workflow.md#task-resolution--blocking)). + +| Interface | Method | Called | +| --------- | ------ | ------ | +| `TaskInterface` | `execute(ProcessState $state): void` | For each input | +| `IterableTaskInterface` | `next(ProcessState $state): bool` | After each execution, the task is executed again while it returns `true` | +| `BlockingTaskInterface` | `proceed(ProcessState $state): void` | Once, when all parent tasks are resolved, only if the task received at least one input | +| `FlushableTaskInterface` | `flush(ProcessState $state): void` | When an ancestor (or the task itself) is resolved or finishes iterating: possibly several times | +| `InitializableTaskInterface` | `initialize(ProcessState $state): void` | Once, before the process starts | +| `FinalizableTaskInterface` | `finalize(ProcessState $state): void` | Once, at the end of the process | ## Iterable tasks -In most of process, when loading a file or querying database, you want to manipulate a collection of data. If you're -managing huge amount of data you might face memory issues. +In most processes, when loading a file or querying a database, you want to manipulate a collection of data. If you're +managing huge amounts of data you might face memory issues. Iterable tasks are a way to resolve this issue. They implement `CleverAge\ProcessBundle\Model\IterableTaskInterface`. -Every time it's needed they will dispatch a new chunk of data to the next task (cascading to other tasks), until all -data have been processed. +Each execution produces one output (one item) that is immediately sent to the next tasks (cascading to other tasks). +Then the `next` method is called: while it returns `true`, the task is executed again to produce the next item, until +all data have been processed. This way only one item at a time is in memory. -The main condition to use them is to have no interaction between each chunk of data. If it's not the case, you might -have to look for another way to organize your data, in order to find bigger chunks. +The main condition to use them is to have no interaction between each chunk of data. If it's not the case, you might +have to look for another way to organize your data, in order to find bigger chunks. -A few examples where iterable are useful: +A few examples where iterable tasks are useful: - You load a collection of database entities: for each of them you want to edit a field and save it back in database. -- In a model with 2 types A and B, where A contains a collection of B: you want to export every B entity that match a -condition on itself and on its parent. You can iterate on A entities, then if it match the first condition, iterate on B -entities, check the second condition and finally export them. +- In a model with 2 types A and B, where A contains a collection of B: you want to export every B entity that matches a +condition on itself and on its parent. You can iterate on A entities, then if it matches the first condition, iterate +on B entities, check the second condition and finally export them. + +Examples of iterable tasks: [CsvReaderTask](reference/tasks/csv_reader_task.md), +[InputIteratorTask](reference/tasks/input_iterator_task.md), +[ConstantIterableOutputTask](reference/tasks/constant_iterable_output_task.md). ## Blocking tasks -Once you produced an iterated flow of data, there can be some point where you need to get the whole result to do a +Once you produced an iterated flow of data, there can be some point where you need to get the whole result to do a onetime operation. -Blocking tasks aims to provide a way to block the flow, waiting for all preceding task to complete. They implement -`CleverAge\ProcessBundle\Model\BlockingTaskInterface`. +Blocking tasks provide a way to block the flow, waiting for all preceding tasks to complete. They implement +`CleverAge\ProcessBundle\Model\BlockingTaskInterface`: `execute` is called for each input but its output is never +passed to the next tasks. Once every parent task is resolved (all their iterations are over), `proceed` is called once +and its output is sent to the next tasks. -The main category of blocking task is aggregator tasks: they accumulate data until execution. Yet one huge caveat is -they can provoke memory issues (due from their very nature). A strong advice when using them is to control the uphill +`proceed` is only called if the task received at least one input: e.g. a [CsvWriterTask](reference/tasks/csv_writer_task.md) +that receives no line creates no file and outputs nothing, so its next tasks are never executed. + +The main category of blocking task is aggregator tasks: they accumulate data until execution. Yet one huge caveat is +they can provoke memory issues (due to their very nature). A strong advice when using them is to control the uphill amount of data (either with a hard limit or by storing the minimum amount of data). Some examples: -- After loading a collection of entity from database, you can iterate on them to extract and transform some values, +- After loading a collection of entities from database, you can iterate on them to extract and transform some values, before finally doing a onetime upload of the result as a JSON. -- Once you retrieved some collection of data, you want to check for a global condition such as "there is exactly `XX` -data that fulfill `YY` condition on one field". In this case, instead of storing the full data, you could only store the -field values. +- Once you retrieved some collection of data, you want to check for a global condition such as "there is exactly `XX` +data that fulfill `YY` condition on one field". In this case, instead of storing the full data, you could only store +the field values. + +Other blocking tasks might be accumulators: with each input they change some internal data (value, file, ...) without +storing a huge collection. Once there is no input, only the final data is outputted. +The simplest example is a counter, but it can also be a CSV writer: [CsvWriterTask](reference/tasks/csv_writer_task.md) +writes each line on `execute` and outputs the file path on `proceed`. + +Examples of blocking tasks: [AggregateIterableTask](reference/tasks/aggregate_iterable_task.md), +[RowAggregatorTask](reference/tasks/row_aggregator_task.md), +[CsvWriterTask](reference/tasks/csv_writer_task.md). + +Combining the blocking and iterable behaviors in the same task is not supported. -Other blocking tasks might be accumulators: with each input they change some internal data (value, file, ...) without -storing a huge collection. Once there is no input, only the final data is outputted. -The simplest example is a counter, but it can also be a CSV writer. +## Flushable tasks + +Flushable tasks sit between normal and blocking tasks: they keep an internal buffer and may output something on some +of their executions (and skip the others), but they need a last chance to output what remains in their buffer once the +flow of data is over. + +They implement `CleverAge\ProcessBundle\Model\FlushableTaskInterface`. Each time a task is resolved, and each time an +iterable task finishes its iterations, the process manager browses this task and its following tasks until it reaches a +blocking task, and calls `flush` on every flushable task it finds. The output set during `flush` is passed to the next +tasks as a normal output (call `ProcessState::setSkipped(true)` when there is nothing left to output). + +Since every resolved ancestor triggers this walk, `flush` may be called several times on the same task, including when +its buffer is already empty: implementations must be idempotent (output nothing, and skip, when there is nothing new to +flush). -For now, due to model limitations, a task cannot be blocking and iterable. +Examples: +- [SimpleBatchTask](reference/tasks/simple_batch_task.md) groups inputs by batches of `batch_count` elements: each + full batch is outputted during `execute`, and the last incomplete batch during `flush`. +- [CounterTask](reference/tasks/counter_task.md) outputs the count every `flush_every` items, and the current count on + `flush` (unless it is a multiple of `flush_every`): as `flush` may be called several times, the same count can be + outputted more than once. ## Initializable tasks -Some tasks may have mandatory initial actions. It may be opening a connexion to a remote server, testing if file -permissions are ok, ... But in most of those case you want to check those setup before actually starting the process. +Some tasks may have mandatory initial actions. It may be opening a connection to a remote server, testing if file +permissions are ok, ... But in most of those cases you want to check this setup before actually starting the process. -Initializable tasks, which implements `CleverAge\ProcessBundle\Model\InitializableTaskInterface`, can setup, check -and prepare anything needed for the main execution of the task. +Initializable tasks, which implement `CleverAge\ProcessBundle\Model\InitializableTaskInterface`, can set up, check +and prepare anything needed for the main execution of the task. All tasks of a process are initialized, in the order +they are configured, before any execution. -This is especially useful (for example) when the process starts with heavy tasks before actually uploading a file. If -the remote server cannot be reached the process can fail at the initialization step, and not at the end of the process. +This is especially useful (for example) when the process starts with heavy tasks before actually uploading a file, to +detect a problem early. Note however that an exception thrown by `initialize` does not abort the process: it is logged +(`critical` level, on the `cleverage_process_task` channel) and the task state is flagged as stopped, then the process +goes on. The failure only surfaces when the task is first reached: for [configurable tasks](#configurable-tasks-and-options), +the options are resolved again and the process fails at that point (upstream tasks may already have run); for other +tasks, the branch stops after this first execution. If the task is never reached, the process ends normally. ## Configurable tasks and options -Most tasks aims to have a generic behavior. This can provide reusablility but every times it needs a slightly different +Most tasks aim to have a generic behavior. This provides reusability, but each usage needs a slightly different behavior. Options are a way to configure a task. -Configurable tasks extends `CleverAge\ProcessBundle\Model\AbstractConfigurableTask`. As you might notice it's an -initializable task that rely on -[Symfony's OptionsResolver Component](https://symfony.com/doc/current/components/options_resolver.html). This allows to -check the definition of the options before actually executing the process. +Configurable tasks extend `CleverAge\ProcessBundle\Model\AbstractConfigurableTask`. It is an initializable task that +relies on [Symfony's OptionsResolver Component](https://symfony.com/doc/current/components/options_resolver.html): +- you define the options in the abstract `configureOptions(OptionsResolver $resolver)` method +- options are resolved (and validated) during `initialize`, so a misconfigured task is logged as critical before any + execution; the process itself only fails when the task is first executed (see + [initializable tasks](#initializable-tasks)) +- the options are read from the task `options` configuration, after replacing the `{{ key }}` placeholders with the + process context values (see [contextual values](01-quick_start.md#contextual-values)) +- in `execute`, use `$this->getOptions($state)` or `$this->getOption($state, 'code')` to read the resolved options -_TODO_ more about definition in XXX section +See [custom tasks](03-custom_tasks.md#options) for an implementation example. ## Transformers -Transformers are a special subset of this bundle. They're not tasks strictly speaking, but used by them. The main entry -point for Transformers is the `CleverAge\ProcessBundle\Task\TransformerTask`, whose only purpose is to take some input, -pass it to a transformer and transfer the output to next task. +Transformers are a special subset of this bundle. They're not tasks strictly speaking, but used by them. The main entry +point for Transformers is the [TransformerTask](reference/tasks/transformer_task.md), whose only purpose is to take +some input, pass it through a chain of transformers and transfer the output to the next task. -The idea is to allow a great flexibility (especially using the [MappingTransformer]), without using too much code. +The idea is to allow a great flexibility (especially using the [MappingTransformer](reference/transformers/mapping_transformer.md)), +without using too much code. -They implement `CleverAge\ProcessBundle\Transformer\TransformerInterface` or -`CleverAge\ProcessBundle\Transformer\ConfigurableTransformerInterface`. - -## Flushable tasks - -_TODO_ +They implement `CleverAge\ProcessBundle\Transformer\TransformerInterface` or +`CleverAge\ProcessBundle\Transformer\ConfigurableTransformerInterface`. See +[custom tasks](03-custom_tasks.md#transformers) to create your own, and +[generic transformers](reference/03-generic_transformers_definition.md) to build reusable ones from configuration only. ## Finalizable tasks -On the opposite, some tasks may require cleanup work at the very end of the process (e.g. cleanup a temporary folder). +On the opposite, some tasks may require cleanup work at the very end of the process (e.g. close a file, cleanup a +temporary folder, log some statistics). + +Finalizable tasks implement `CleverAge\ProcessBundle\Model\FinalizableTaskInterface`. Their `finalize` method is +called once all tasks are resolved, for every task of the process, in the order they are configured. -Finalizable tasks implements `CleverAge\ProcessBundle\Model\FinalizableTaskInterface` and can trigger any work at the -very end of a process. +Note that finalization only happens when the process reaches its end: if a task stops the process because of an +exception (error strategy `stop`), the process fails immediately and `finalize` is not called. A process stopped +without exception (e.g. by the [StopTask](reference/tasks/stop_task.md)) is still finalized. -_TODO_ check behavior during failed process +Examples of finalizable tasks: [CsvReaderTask](reference/tasks/csv_reader_task.md) and the other CSV tasks (close the +file), [StatCounterTask](reference/tasks/stat_counter_task.md) (logs the number of processed items). diff --git a/docs/03-custom_tasks.md b/docs/03-custom_tasks.md index 607ac3b0..11b72502 100644 --- a/docs/03-custom_tasks.md +++ b/docs/03-custom_tasks.md @@ -1,90 +1,287 @@ Custom tasks ============ -Once you've worked with provided tasks to build simple processes, you may face cases where you want to build a more -complex workflow. Common tasks would not be powerful enough (or would imply much unoptimized setup), so you'll have to +Once you've worked with provided tasks to build simple processes, you may face cases where you want to build a more +complex workflow. Common tasks would not be powerful enough (or would imply much unoptimized setup), so you'll have to create your own. ## Service declaration -As stated before, Tasks are simple services implementing `CleverAge\ProcessBundle\Model\TaskInterface`, so you can -design it like any other service, with only small specificities. +As stated before, tasks are simple services implementing `CleverAge\ProcessBundle\Model\TaskInterface`, so you can +design them like any other service, with only small specificities: +- the service must be **public**: the process manager fetches it from the container using the `service` reference of + the task configuration (with or without the leading `@`) +- the service should **not be shared** ([`shared: false`](https://symfony.com/doc/current/service_container/shared.html)) -Most of the time you don't want that a task declared in a process shares some data with another declaration, using the -same service, or even with another instance of your process. Since a task is a service, by default, symfony will only -create one instance (keeping values in every attribute of your class). You can force it to create one instance by -process by using the option [`shared: false`](https://symfony.com/doc/current/service_container/shared.html). +Most of the time you don't want a task declared in a process to share some data with another declaration using the +same service, or even with another instance of your process. Since a task is a service, by default, Symfony will only +create one instance (keeping values in every attribute of your class). With `shared: false`, a new instance is created +each time a task configuration is initialized. + +```yaml +# config/services.yaml +services: + App\Task\: + resource: '../src/Task/*' + autowire: true + autoconfigure: true + public: true + shared: false + tags: + - { name: monolog.logger, channel: cleverage_process_task } +``` + +If your `services.yaml` also has the default `App\:` resource (which includes `src/Task/`), declare `App\Task\:` after +it: the last definition of a service wins. ## Using the state object -With the `TaskInterface::execute` method comes a small container object : the State +With the `TaskInterface::execute` method comes a small container object: the state (`CleverAge\ProcessBundle\Model\ProcessState`). -It's the only way to interact with the rest of the process. Each time the task will need to process data, the `execute` -method will be called and the `$state` will contain a new input (`ProcessState::getInput`). Once the task is done, it -may pass a new output to the next task (`ProcessState::setOutput`). +It's the only way to interact with the rest of the process. Each time the task needs to process data, the `execute` +method is called and the `$state` contains a new input (`ProcessState::getInput`). Once the task is done, it +may pass a new output to the next tasks (`ProcessState::setOutput`). -The State also provide reporting tools: -* `ProcessState::log`: register a new log message (see[logging]) -* `ProcessState::getConsoleOutput`: direct link to Symfony's Console Output (deprecated, prefer log) +```php +namespace App\Task; -Sometimes, when you execute a task, you need to change how the process may continue. It will be detailed in depth in -the [next chapter about error management] but here are the main methods -* `ProcessState::setSkipped`: process won't continue to next step -* `ProcessState::setStopped`: process will fully stop -* `ProcessState::setErrorOutput`: allow to direct an output to an error branch from your workflow +use CleverAge\ProcessBundle\Model\ProcessState; +use CleverAge\ProcessBundle\Model\TaskInterface; + +class UppercaseNameTask implements TaskInterface +{ + public function execute(ProcessState $state): void + { + $item = $state->getInput(); + $item['name'] = mb_strtoupper($item['name']); + + $state->setOutput($item); + } +} +``` + +The state also gives access to the context of the execution: +* `ProcessState::getContext()`: the contextual values given to the process (see + [contextual values](01-quick_start.md#contextual-values)) +* `ProcessState::getContextualizedOptions()` / `ProcessState::getContextualizedOption($code, $default)`: the raw task + options, with context placeholders replaced +* `ProcessState::getProcessConfiguration()` / `ProcessState::getTaskConfiguration()`: the current process and task + configurations +* `ProcessState::getProcessHistory()`: the current execution (process code, start date, state, duration...) +* `ProcessState::getPreviousState()`: the state of the task that produced the current input + +Sometimes, when you execute a task, you need to change how the process continues. Here are the main methods (see +[error management](04-advanced_workflow.md#errors-and-skips) for more details): +* `ProcessState::setSkipped(true)`: the current output won't be passed to the next tasks (the process continues with + the next input) +* `ProcessState::stop(?\Throwable $e = null)`: with an exception, the process fails; without exception, only the current + flow stops (other root tasks, blocking tasks and finalization still run, see + [errors and skips](04-advanced_workflow.md#errors-and-skips)) +* `ProcessState::setException(\Throwable $e)`: flag the current execution as failed without throwing, the task + `error_strategy` is then applied (throwing an exception from `execute` has the same effect) +* `ProcessState::setErrorOutput($value)`: send a value to the error branch of your workflow (the tasks listed in + `error_outputs`) +* `ProcessState::addErrorContextValue($key, $value)` / `removeErrorContext($key)`: add information to the log record + written when an error occurs ## Options -To reuse more easily tasks, the best way is to use options. A basic option management implementation is already +To reuse tasks more easily, the best way is to use options. A basic option management implementation is already available in `CleverAge\ProcessBundle\Model\AbstractConfigurableTask`. Based on [Symfony's OptionsResolver Component](https://symfony.com/doc/current/components/options_resolver.html) this -abstract allows you to override its `configureOptions` method to add your requirements, default values and normalizer. -It's a very important step to allow manipulating your custom task. Even when you may have only one instance, and one +abstract class allows you to implement its `configureOptions` method to add your requirements, default values and +normalizers. Options are resolved (once) during the task initialization, and can be read with `getOptions($state)` or +`getOption($state, $code)`. If the resolution fails during initialization, the error is logged and the options are +resolved again (failing the process) when the task is first executed. + +```php +namespace App\Task; + +use CleverAge\ProcessBundle\Model\AbstractConfigurableTask; +use CleverAge\ProcessBundle\Model\ProcessState; +use Symfony\Component\OptionsResolver\OptionsResolver; + +class PrefixTask extends AbstractConfigurableTask +{ + public function execute(ProcessState $state): void + { + $state->setOutput($this->getOption($state, 'prefix').$state->getInput()); + } + + protected function configureOptions(OptionsResolver $resolver): void + { + $resolver->setRequired('prefix'); + $resolver->setAllowedTypes('prefix', 'string'); + } +} +``` + +```yaml +prefix: + service: '@App\Task\PrefixTask' + options: + prefix: 'SKU-' +``` + +It's a very important step to allow manipulating your custom task. Even when you may have only one instance, and one purpose, you'll find that having some options will help you debug a situation, or evolve your process. +Note that `AbstractConfigurableTask` implements `Symfony\Contracts\Service\ResetInterface`: resolved options are +cleared when the service is reset. If you override `initialize`, remember to call `parent::initialize($state)`. + ## Iterable and Blocking tasks implementations -Defining your tasks as Iterable or Blocking is as simple as implementing one of corresponding interface: -* `CleverAge\ProcessBundle\Model\IterableTaskInterface`: the `next` method should behave almost the same as PHP's native -[next](https://secure.php.net/manual/en/function.next.php) function for arrays (except it only returns a boolean) -* `CleverAge\ProcessBundle\Model\BlockingTaskInterface`: every `execute` method call should only accumulate data from -the input and once every previous task is _resolved_, the `proceed` method should provide an output (see [TODO] for -the exact definition of a resolved method) +Defining your tasks as Iterable or Blocking is as simple as implementing one of the corresponding interfaces (see +[task types](02-task_types.md) for the complete lifecycle): +* `CleverAge\ProcessBundle\Model\IterableTaskInterface`: the `next` method should behave almost the same as PHP's +native [next](https://www.php.net/manual/en/function.next.php) function for arrays, except it only returns a boolean: +`true` if there is another item to output (the task is then executed again), `false` when the iteration is over +* `CleverAge\ProcessBundle\Model\BlockingTaskInterface`: every `execute` method call should only accumulate data from +the input, and once every previous task is _resolved_ (all their executions and iterations are over), the `proceed` +method should provide an output + +```php +namespace App\Task; + +use CleverAge\ProcessBundle\Model\BlockingTaskInterface; +use CleverAge\ProcessBundle\Model\ProcessState; + +class SumTask implements BlockingTaskInterface +{ + protected int|float $sum = 0; + + public function execute(ProcessState $state): void + { + $this->sum += $state->getInput(); + } + + public function proceed(ProcessState $state): void + { + $state->setOutput($this->sum); + } +} +``` It's up to you to know when you should be using one of those, but basically: -* When you loop over a collection of independent elements, you should use an Iterable task. It may help you reduce the +* When you loop over a collection of independent elements, you should use an Iterable task. It may help you reduce the memory footprint. -* When you need to collect, upload, ... data as a whole, then you might need a Blocking task. Be sure to read [previous -chapter's notice] about performance. +* When you need to collect, upload, ... data as a whole, then you might need a Blocking task. Be sure to read the +[notice about blocking tasks](02-task_types.md#blocking-tasks) about memory usage. -Tasks cannot be both Iterable and Blocking. +Tasks should not be both Iterable and Blocking. If you need to buffer data and output it by chunks, look at +`CleverAge\ProcessBundle\Model\FlushableTaskInterface` (see [flushable tasks](02-task_types.md#flushable-tasks)). ## Transformers -Transformer are another kind of service. They implement `CleverAge\ProcessBundle\Transformer\TransformerInterface` or -`CleverAge\ProcessBundle\Transformer\ConfigurableTransformerInterface` and are declared with a `cleverage.transformer` -tag. +Transformers are another kind of service. They implement `CleverAge\ProcessBundle\Transformer\TransformerInterface` +(`transform(mixed $value, array $options = []): mixed` and `getCode(): string`) or +`CleverAge\ProcessBundle\Transformer\ConfigurableTransformerInterface` (which adds +`configureOptions(OptionsResolver $resolver): void`) and are declared with a `cleverage.transformer` tag. + +They're meant to be lightweight, composable, and stateless pieces of your process. Feel free to implement custom ones +as soon as provided ones don't fit your goal. + +```php +namespace App\Transformer; + +use CleverAge\ProcessBundle\Transformer\ConfigurableTransformerInterface; +use Symfony\Component\OptionsResolver\OptionsResolver; + +class VatTransformer implements ConfigurableTransformerInterface +{ + public function transform(mixed $value, array $options = []): mixed + { + return round($value * (1 + $options['rate']), 2); + } + + public function configureOptions(OptionsResolver $resolver): void + { + $resolver->setDefault('rate', 0.2); + $resolver->setAllowedTypes('rate', ['float', 'int']); + } + + public function getCode(): string + { + return 'app_vat'; + } +} +``` + +```yaml +# config/services.yaml +services: + App\Transformer\: + resource: '../src/Transformer/*' + autowire: true + tags: + - { name: cleverage.transformer } +``` -They're meant to be lightweight, composable, and stateless piece of your process. Be free to implement custom one as -soon as provided one doesn't fit your goal. +Once properly set up, they are registered in the `CleverAge\ProcessBundle\Registry\TransformerRegistry` (used by the +`TransformerTask` and every transformer using a sub-list of transformers), using the code from +`TransformerInterface::getCode`. Two transformers cannot share the same code. When creating a new transformer for your +project you should use an internal prefix in your codes to avoid conflicts with potential standard additions. -Once properly setup, they should be accessible from the `CleverAge\ProcessBundle\Registry\TransformerRegistry` (already -available in the `TransformerTask`), using the code from `TransformerInterface::getCode`. When creating a new transformer -for your project you should use an internal prefix in your codes to avoid conflict with potential standard additions. - -Just as tasks, options can be managed with -[Symfony's OptionsResolver Component](https://symfony.com/doc/current/components/options_resolver.html), so be sure to -implement a few of them. +```yaml +transform: + service: '@CleverAge\ProcessBundle\Task\TransformerTask' + options: + transformers: + app_vat: + rate: 0.055 +``` ## Logging -_TODO_ +The bundle declares three [Monolog channels](https://symfony.com/doc/current/logging/channels_handlers.html): + +| Channel | Used by | +| ------- | ------- | +| `cleverage_process` | the process manager: process start/end, tasks processing (debug level), unreachable tasks warnings, critical failures | +| `cleverage_process_task` | the process manager for task errors (with the `log_level` of the task configuration), and every task of the bundle | +| `cleverage_process_transformer` | every transformer of the bundle | + +Each channel has a Monolog processor that adds the following values in the `extra` of every record: `process_id`, +`process_code` and `process_context`, plus `task_code`, `task_service` (and the current `error` output or `exception` +if any) for the task and transformer channels. -A logging chanel has been defined for tasks +To benefit from those in your own tasks, inject a `Psr\Log\LoggerInterface` and bind it to the task channel with the +`monolog.logger` tag: ```yaml +services: + App\Task\: + resource: '../src/Task/*' + autowire: true + public: true + shared: false tags: - { name: monolog.logger, channel: cleverage_process_task } ``` + +```php +namespace App\Task; + +use CleverAge\ProcessBundle\Model\ProcessState; +use CleverAge\ProcessBundle\Model\TaskInterface; +use Psr\Log\LoggerInterface; + +class MyTask implements TaskInterface +{ + public function __construct( + protected LoggerInterface $logger, + ) { + } + + public function execute(ProcessState $state): void + { + $this->logger->info('Processing item', ['input' => $state->getInput()]); + // ... + } +} +``` + +Use `cleverage_process_transformer` for your transformers. See the [common setup cookbook](cookbooks/01-common_setup.md#logging) +for a handler configuration example. diff --git a/docs/04-advanced_workflow.md b/docs/04-advanced_workflow.md index f31ce2a3..b73ca71e 100644 --- a/docs/04-advanced_workflow.md +++ b/docs/04-advanced_workflow.md @@ -3,26 +3,276 @@ Advanced Workflow ## Process execution flow -_TODO_ -* task resolution & blocking -* wrapping execution in subprocess -* errors & skips -* orphan tasks +When a process is executed (with the `cleverage:process:execute` command or with +`CleverAge\ProcessBundle\Manager\ProcessManager::execute($processCode, $input, $context)`), the process manager: + +1. dispatches the `cleverage_process.start` event +2. checks the process: circular dependencies are forbidden, and the `entry_point` / `end_point` tasks must be part of + the *main branch*, the group of connected tasks that is actually executed (see [orphan tasks](#orphan-tasks)) +3. **initializes** every task, in the order they are configured: the service is fetched from the container and + `initialize` is called on [initializable tasks](02-task_types.md#initializable-tasks) (for configurable tasks, this is + where options are validated). An exception thrown by `initialize` is logged as critical and flags the task as + stopped, but does not abort the process: it only fails when this task is first executed +4. gives the process input to the `entry_point` task, if one is defined (otherwise the input is ignored and a warning is + logged) +5. **resolves** the tasks of the main branch (see below) +6. **finalizes** every task, in the order they are configured (see + [finalizable tasks](02-task_types.md#finalizable-tasks)) +7. returns the last output of the `end_point` task (or `null` if there is none) and dispatches the + `cleverage_process.end` event + +If an exception is thrown at any point, the `cleverage_process.fail` event is dispatched and the exception is rethrown. +Note that a task error handled by the `stop` strategy does not surface as the original exception: the process manager +throws a new `Symfony\Component\ErrorHandler\Error\FatalError` (an `\Error`, not an `\Exception`), whose message +contains the process code, the task code and the original message; the original exception is not attached as +`previous` (it is only available in the task error log record). + +### Executing a process from PHP + +The process manager service id is `cleverage_process.manager.process`. It is private and has no class alias, so it +cannot be autowired by type: inject it explicitly (or declare the alias yourself). + +```yaml +# config/services.yaml +services: + App\Service\ProductImporter: + arguments: + $processManager: '@cleverage_process.manager.process' + + # Or, to enable autowiring of the ProcessManager type everywhere: + CleverAge\ProcessBundle\Manager\ProcessManager: '@cleverage_process.manager.process' +``` + +```php +$result = $this->processManager->execute('app.import_file', '/tmp/products.csv', ['dry_run' => true]); +``` + +### Task resolution & blocking + +A task is *resolved* when it and all its ancestors are over. Tasks are resolved starting from the end of the +configuration, each task first resolving its parents: +- a **root task** (a task without parent, outside of an error branch) is executed; its output is immediately passed to + each of its `outputs`, which are executed in turn, and so on (depth-first). An + [iterable task](02-task_types.md#iterable-tasks) is executed again as long as its `next` method returns `true`, + each item going through the whole following chain before the next one is produced +- a [blocking task](02-task_types.md#blocking-tasks) receives all the inputs of its parents but does not pass anything + to its outputs; once all its parents are resolved, `proceed` is called and its output starts a new flow to its + outputs. If the blocking task never received any input, `proceed` is not called +- when a task is resolved (and when an iterable task finishes its iterations), the task itself and the + [flushable tasks](02-task_types.md#flushable-tasks) found in its descendants (until a blocking task) are flushed. + As each resolved ancestor triggers this, a flushable task can be flushed several times: `flush` must be idempotent + +The following process reads a CSV file line by line, transforms each line and writes it into another CSV file: the +reader is iterable, the transformer is executed once for each line, and the writer is blocking: it writes each line and +outputs the file path only once, at the end, to the `log` task. + +```yaml +clever_age_process: + configurations: + app.csv_copy: + tasks: + read: + service: '@CleverAge\ProcessBundle\Task\File\Csv\CsvReaderTask' + options: + file_path: '%kernel.project_dir%/var/data/input.csv' + outputs: [transform] + transform: + service: '@CleverAge\ProcessBundle\Task\TransformerTask' + options: + transformers: + mapping: + mapping: + id: { code: '[id]' } + name: { code: '[name]' } + outputs: [write] + write: + service: '@CleverAge\ProcessBundle\Task\File\Csv\CsvWriterTask' + options: + file_path: '%kernel.project_dir%/var/data/output.csv' + outputs: [log] + log: + service: '@CleverAge\ProcessBundle\Task\Debug\DebugTask' +``` + +### Orphan tasks + +Only one group of connected tasks, the *main branch*, is executed. It is the group containing the `entry_point` task, +or the `end_point` task if there is no entry point, or else the first configured task. Tasks that are not connected +to it (not referenced in any `outputs` or `error_outputs` of the main branch) are never executed: a warning +`Task '' is unreachable` is logged for each of them. They are still initialized and finalized, like every task of +the process (so a misconfigured orphan task still logs its initialization error). The `cleverage:process:help` command +only displays the main branch. + +### Errors and skips + +An error happens when a task throws an exception, or flags one with `ProcessState::setException()`. The error is logged +on the `cleverage_process_task` channel with the `log_level` of the task (`critical` by default), then the +`error_strategy` of the task is applied (it defaults to the global `default_error_strategy`, itself `stop` by +default): +- `skip`: the current output is dropped, and the process continues with the next input (e.g. the next line of a CSV + file) +- `stop`: the whole process stops and fails (a `FatalError` is thrown by the process manager, see + [process execution flow](#process-execution-flow)) + +Before applying the strategy, the task input is sent to the tasks listed in `error_outputs` (unless the task already +set a specific value with `ProcessState::setErrorOutput()`). Those tasks form an *error branch*: they are only executed +when an error output is set, and they are never considered as root tasks. A task can also send a value to its error +branch without failing, by calling `setErrorOutput()` (e.g. [FilterTask](reference/tasks/filter_task.md) sends +rejected items to its error outputs). With the `stop` strategy, the error branch is executed before the process +stops. + +```yaml +transform: + service: '@CleverAge\ProcessBundle\Task\TransformerTask' + error_strategy: skip # Continue with the next item on error + log_level: warning # Log errors as warning instead of critical + options: + transformers: + mapping: + mapping: + price: { code: '[price]', transformers: { cast: { type: float } } } + outputs: [write] + error_outputs: [write_errors] # Receives the input that failed +write_errors: + service: '@CleverAge\ProcessBundle\Task\File\JsonStream\JsonStreamWriterTask' + options: + file_path: '%kernel.project_dir%/var/data/errors_{date_time}.json-stream' +``` + +A task may also control the flow without any error: +- `ProcessState::setSkipped(true)`: the output of this execution is not sent to the next tasks (used by filters, by + iterable tasks on empty items, by batch tasks while filling a batch...) +- `ProcessState::setStopped(true)` / `ProcessState::stop()`: without exception, only the current flow stops: the stop + signal goes back up to the root task, which stops iterating. The rest of the resolution still happens: other root + tasks are executed, blocking tasks that already received inputs are proceeded, flushable tasks are flushed and every + task is finalized. It is not a failure for the process manager: the `cleverage_process.end` event is dispatched (not + `cleverage_process.fail`) and the command exits with code 0. The [StopTask](reference/tasks/stop_task.md) also flags + the process history as failed, but this does not change the event nor the exit code + +The legacy `errors` key is a deprecated alias of `error_outputs` (defining both throws an exception). + +### Wrapping execution in subprocesses + +A process can execute another process with the [ProcessExecutorTask](reference/tasks/process_executor_task.md): for +each input, the sub-process is executed in the same PHP process, with this input given to its `entry_point`, and the +last output of its `end_point` becomes the output of the task (`null` if the sub-process has no `end_point`). Context +values are not inherited: pass them explicitly with the `context` option (placeholders like `{{ key }}` can be used to +forward the parent context). + +```yaml +clever_age_process: + configurations: + app.import_all: + tasks: + list_files: + service: '@CleverAge\ProcessBundle\Task\File\FolderBrowserTask' + options: + folder_path: '%kernel.project_dir%/var/import' + name_pattern: '*.csv' + outputs: [import_file] + import_file: + service: '@CleverAge\ProcessBundle\Task\Process\ProcessExecutorTask' + options: + process: app.import_file + context: + dry_run: '{{ dry_run }}' # Always pass -c dry_run:, see below + + app.import_file: + public: false + entry_point: read + tasks: + read: + service: '@CleverAge\ProcessBundle\Task\File\Csv\InputCsvReaderTask' + outputs: [save] + save: + service: '@App\Task\SaveProductTask' +``` + +If the parent process is executed without a `dry_run` context value, the placeholder is not replaced and the +sub-process receives the literal string `'{{ dry_run }}'`, which is truthy: always provide the forwarded context +values when executing the parent process. + +This is a good way to split a big workflow into small, testable processes. Private processes (`public: false`) are +hidden from `cleverage:process:list` (unless `--all` is used) but can still be executed. ## Events -Events are being send around process execution (see `CleverAge\ProcessBundle\Event\ProcessEvent`) : -* `cleverage_process.start` : on process start -* `cleverage_process.end` : on successful process end -* `cleverage_process.fail` : on failed process end (with the associated error) +Events are dispatched around process execution (see `CleverAge\ProcessBundle\Event\ProcessEvent`, which gives access +to the process code, input, context and, depending on the event, output or error): + +| Event name | Constant | Dispatched | +| ---------- | -------- | ---------- | +| `cleverage_process.start` | `ProcessEvent::EVENT_PROCESS_STARTED` | on process start | +| `cleverage_process.end` | `ProcessEvent::EVENT_PROCESS_ENDED` | on successful process end (with the process output) | +| `cleverage_process.fail` | `ProcessEvent::EVENT_PROCESS_FAILED` | on failed process end (with the associated error) | + +Those events are also dispatched for sub-processes executed with the `ProcessExecutorTask`. -Another event is send when a process is executed with the CLI (see `CleverAge\ProcessBundle\Event\ConsoleProcessEvent`) : -* `cleverage_process.cli.init` : before executing any process, giving access to console Input/Output objects +Another event is dispatched once by the `cleverage:process:execute` command, before executing any process: +`CleverAge\ProcessBundle\Event\ConsoleProcessEvent`. It is dispatched without a specific name, so listeners must use +the class name as event name. It gives access to the console input/output objects, and to the process input and +context. -You can also use [EventDispatcherTask](reference/tasks/event_dispatcher_task.md) to manually trigger an event in the middle of a process. +```php +namespace App\EventListener; + +use CleverAge\ProcessBundle\Event\ProcessEvent; +use Psr\Log\LoggerInterface; +use Symfony\Component\EventDispatcher\Attribute\AsEventListener; + +#[AsEventListener(event: ProcessEvent::EVENT_PROCESS_FAILED)] +class ProcessFailureListener +{ + public function __construct(private LoggerInterface $logger) + { + } + + public function __invoke(ProcessEvent $event): void + { + $this->logger->alert("Process {$event->getProcessCode()} failed", [ + 'error' => $event->getProcessError()?->getMessage(), + ]); + } +} +``` + +You can also use the [EventDispatcherTask](reference/tasks/event_dispatcher_task.md) to trigger an event in the middle +of a process: it dispatches a `CleverAge\ProcessBundle\Event\EventDispatcherTaskEvent`, giving access to the current +`ProcessState`. Note that in the current implementation the event is dispatched under its class name +(`CleverAge\ProcessBundle\Event\EventDispatcherTaskEvent`): the `event_name` option is required but not passed to the +event dispatcher. ## Parallelization -_TODO_ -* ProcessLauncherTask -* pthread +> **Warning**: in the current version, the `ProcessLauncherTask` and the `CommandRunnerTask` fail when their options +> are resolved, because of known bugs: see the Notes of the [ProcessLauncherTask](reference/tasks/process_launcher_task.md#notes) +> and [CommandRunnerTask](reference/tasks/command_runner_task.md#notes) reference pages. + +PHP executes a process in a single thread. To use several CPU cores, the +[ProcessLauncherTask](reference/tasks/process_launcher_task.md) launches a process in a separate system process +(`bin/console cleverage:process:execute --input-from-stdin ...`, in the same environment) for each input it receives: +- at most `max_processes` sub-processes run at the same time; the task waits for a free slot before launching a new one +- the input is cast to string and sent to the sub-process through STDIN, `context` values are passed as + `--context=:` options (values are cast to string, so arrays cannot be forwarded this way) +- with `json_buffering` enabled, the sub-process runs with `--output-format=json-stream --output=`: the last output + of its `end_point` is written to a JSON stream file whose path is outputted by the task once the sub-process is over + (use a [JsonStreamReaderTask](reference/tasks/json_stream_reader_task.md) to read it back). The file is only written + if the sub-process has an `end_point` whose last output is an array; otherwise, as when `json_buffering` is disabled, + the task outputs nothing +- if a sub-process fails (non-zero exit code), all the running sub-processes are stopped and the task throws an + exception + +```yaml +parallel_import: + service: '@CleverAge\ProcessBundle\Task\Process\ProcessLauncherTask' + options: + process: app.import_file + max_processes: 4 +``` + +Since sub-processes do not share memory with the parent process, their input must be small and serializable as a +string (an identifier, a file path, a chunk produced by the [CsvSplitterTask](reference/tasks/csv_splitter_task.md)...). + +To execute any other system command (not a process), use the +[CommandRunnerTask](reference/tasks/command_runner_task.md): it runs the `commandline` option synchronously with the +Symfony Process component, sends the task input to its STDIN, and outputs what the command wrote on STDOUT. diff --git a/docs/05-good_practices.md b/docs/05-good_practices.md index 5a029c6f..eb456b82 100644 --- a/docs/05-good_practices.md +++ b/docs/05-good_practices.md @@ -1,8 +1,82 @@ -pure functions -- same in => same out -- no side effect +Good practices +============== -découper les process -- testing +## Keep transformers pure -do not rely on execution order ? +A transformer should behave like a pure function: +- the same input (and options) always gives the same output +- it has no side effect: no database write, no file, no API call, no state kept between two calls + +This makes transformers predictable, reusable in any process, and trivial to unit test. Anything with a side effect +(reading, writing, calling a remote service) belongs in a task. If a transformation is expensive and deterministic, +wrap it in the [CachedTransformer](reference/transformers/cached_transformer.md) instead of adding a cache in your own +transformer. + +Prefer composing existing transformers (`mapping`, `callback`, `cast`, `implode`, ...) and, when the same chain is used +in several places, declare it once as a [generic transformer](reference/03-generic_transformers_definition.md). + +## Declare tasks as non-shared services + +Task services must be `public: true` and should be `shared: false`: a task often keeps state in its properties (an +open file, a buffer, a counter...). A shared service would share this state between two tasks using the same service +in a process, or between two executions of a process in the same PHP process (sub-processes, workers, tests). See +[service declaration](03-custom_tasks.md#service-declaration). + +## Split big workflows into small processes + +A process with dozens of tasks is hard to read, to debug and to test. Split it into small processes that do one thing +(read a file, import one item, export a batch...) and chain them with the +[ProcessExecutorTask](reference/tasks/process_executor_task.md) (see +[subprocesses](04-advanced_workflow.md#wrapping-execution-in-subprocesses)): +- each sub-process can be executed and tested on its own, with an `entry_point` for its input and an `end_point` for + its output +- sub-processes that should not be launched directly can be marked `public: false` +- the same small process can be reused by several parent processes, or parallelized with the + [ProcessLauncherTask](reference/tasks/process_launcher_task.md) (currently broken by a known bug, see the + [Notes](reference/tasks/process_launcher_task.md#notes) of its reference page) + +Use `description` and `help` on processes and tasks: they are displayed by `cleverage:process:list` and +`cleverage:process:help`. + +## Name processes with a prefix + +Process codes are global to the application, and so are transformer codes. Prefix them with your project or domain +(e.g. `app.catalog.import_products`, `app_vat`) to avoid conflicts with processes and transformers provided by +bundles. Organizing configuration files like the codes (e.g. `config/packages/process/app/catalog/import_products.yaml`) +makes them easy to find. + +## Handle errors explicitly + +- Keep the global `default_error_strategy: stop` (the default), so any unexpected error stops the process instead of + silently dropping data. +- Set `error_strategy: skip` only on the tasks where an item can fail without compromising the others (e.g. the + transformation of one line of a file), and tune their `log_level`. +- Plug an error branch (`error_outputs`) on those tasks to keep track of rejected items (write them to a file, log + them...), rather than losing them. + +See [errors and skips](04-advanced_workflow.md#errors-and-skips). + +## Mind the memory + +- Stream data with [iterable tasks](02-task_types.md#iterable-tasks) (CSV/JSON stream/line readers, Doctrine + iterators...) instead of loading whole collections: only one item at a time goes through the process. +- Use [blocking tasks](02-task_types.md#blocking-tasks) (aggregators) only when you really need the whole data set, + and store only what you need in them. Prefer accumulators that write as they receive data (like the + [CsvWriterTask](reference/tasks/csv_writer_task.md)). +- When you need batches (e.g. for bulk database writes), use the + [SimpleBatchTask](reference/tasks/simple_batch_task.md) rather than an aggregator. +- Measure: see the [memory usage cookbook](cookbooks/memory_usage_graph.md), and always check memory in the `prod` + environment (the `dev` environment keeps logs and debug data in memory). + +## Do not rely on the execution order of branches + +When a task has several `outputs`, each output is fully processed one after the other, in the configured order, but +this is an implementation detail. If a task needs the result of another branch, make the dependency explicit in the +graph (e.g. with a blocking task or an [InputAggregatorTask](reference/tasks/input_aggregator_task.md)) rather than +relying on the order in which branches are executed. + +## Use the context for runtime parameters + +Values that change between executions (a file name, a date, a limit...) should not be hard-coded in the process +configuration: pass them as [contextual values](01-quick_start.md#contextual-values) (`-c key:value`) and use +`{{ key }}` placeholders in the task options. diff --git a/docs/06-testing.md b/docs/06-testing.md index 08f602b7..bf0fb5d0 100644 --- a/docs/06-testing.md +++ b/docs/06-testing.md @@ -1 +1,159 @@ -phpunit +Testing +======= + +Processes are made of small pieces (transformers, tasks, sub-processes), which makes them easy to test at each level. +The examples below use [PHPUnit](https://phpunit.de) and Symfony's +[testing tools](https://symfony.com/doc/current/testing.html). + +## Unit testing transformers + +A transformer is a plain PHP object: instantiate it and call `transform()` with an input and options. If it implements +`ConfigurableTransformerInterface`, resolve the options with its `configureOptions()` method first, to test defaults +and validation the same way the bundle does. + +```php +namespace App\Tests\Transformer; + +use App\Transformer\VatTransformer; +use PHPUnit\Framework\Attributes\CoversClass; +use PHPUnit\Framework\TestCase; +use Symfony\Component\OptionsResolver\OptionsResolver; + +#[CoversClass(VatTransformer::class)] +class VatTransformerTest extends TestCase +{ + public function testTransformWithDefaultRate(): void + { + $transformer = new VatTransformer(); + + $resolver = new OptionsResolver(); + $transformer->configureOptions($resolver); + $options = $resolver->resolve([]); + + self::assertSame(12.0, $transformer->transform(10, $options)); + } +} +``` + +Passing the options array directly to `transform()` also works, but skips the defaults and validation: most of the +bundle's own tests in `tests/Transformer/` do so, and test `configureOptions()` separately. + +## Unit testing tasks + +A task only interacts with the process through its `ProcessState`. To test it in isolation, build a state with a +minimal task and process configuration: + +```php +namespace App\Tests\Task; + +use App\Task\PrefixTask; +use CleverAge\ProcessBundle\Configuration\ProcessConfiguration; +use CleverAge\ProcessBundle\Configuration\TaskConfiguration; +use CleverAge\ProcessBundle\Context\ContextualOptionResolver; +use CleverAge\ProcessBundle\Model\ProcessHistory; +use CleverAge\ProcessBundle\Model\ProcessState; +use PHPUnit\Framework\Attributes\CoversClass; +use PHPUnit\Framework\TestCase; + +#[CoversClass(PrefixTask::class)] +class PrefixTaskTest extends TestCase +{ + public function testExecute(): void + { + $state = $this->createState(['prefix' => 'SKU-']); + $state->setInput('123'); + + $task = new PrefixTask(); + $task->initialize($state); // Resolves (and validates) options + $task->execute($state); + + self::assertSame('SKU-123', $state->getOutput()); + self::assertFalse($state->isSkipped()); + } + + private function createState(array $options, array $context = []): ProcessState + { + $taskConfiguration = new TaskConfiguration('prefix', '@'.PrefixTask::class, $options); + $processConfiguration = new ProcessConfiguration('test', ['prefix' => $taskConfiguration]); + + $state = new ProcessState($processConfiguration, new ProcessHistory($processConfiguration, $context)); + $state->setTaskConfiguration($taskConfiguration); + $state->setContext($context); + $state->setContextualOptionResolver(new ContextualOptionResolver()); + $state->reset(false); // Initializes the output, skipped and error flags, as the process manager does + + return $state; + } +} +``` + +For iterable tasks, call `execute()` then `next()` in a loop; for blocking tasks, call `execute()` for each input then +`proceed()`; for finalizable tasks, do not forget to call `finalize()`. + +## Functional testing of processes + +The best way to test a whole workflow is to execute the process, like the console command does, with the process +manager. Its service id is `cleverage_process.manager.process` (class `CleverAge\ProcessBundle\Manager\ProcessManager`); +it is a private service, available through the test container of a `KernelTestCase`: + +```php +public function execute(string $processCode, mixed $input = null, array $context = []): mixed +``` + +`$input` is given to the process `entry_point`, `$context` is the same as the `--context` option of the command, and +the returned value is the last output of the process `end_point` (`null` if there is none). If the process fails, the +exception is rethrown; but a task error handled by the `stop` error strategy is thrown as a new +`Symfony\Component\ErrorHandler\Error\FatalError` (an `\Error`), which only keeps the original message (the original +exception is not attached as `previous`). Test it with `expectException(FatalError::class)` and +`expectExceptionMessageMatches()` rather than with the original exception class. + +```yaml +# config/packages/test/clever_age_process.yaml +clever_age_process: + configurations: + test.app.vat: + entry_point: transform + end_point: transform + tasks: + transform: + service: '@CleverAge\ProcessBundle\Task\TransformerTask' + options: + transformers: + app_vat: + rate: '{{ rate }}' +``` + +```php +namespace App\Tests\Process; + +use CleverAge\ProcessBundle\Manager\ProcessManager; +use PHPUnit\Framework\Attributes\CoversNothing; +use Symfony\Bundle\FrameworkBundle\Test\KernelTestCase; + +#[CoversNothing] +class VatProcessTest extends KernelTestCase +{ + public function testProcess(): void + { + self::bootKernel(); + /** @var ProcessManager $processManager */ + $processManager = self::getContainer()->get('cleverage_process.manager.process'); + + $result = $processManager->execute('test.app.vat', 100, ['rate' => 0.055]); + + self::assertSame(105.5, $result); + } +} +``` + +A few tips: +- Put test-only processes (prefixed, e.g. `test.`) in `config/packages/test/` so they are only loaded in the `test` + environment, and use `entry_point`/`end_point` to feed and check the data. +- Use [contextual values](01-quick_start.md#contextual-values) for file paths, so each test can use its own fixture + files and temporary output folder. +- Iterable processes only return the last output of the end point: to check every item, end the process with an + aggregator (e.g. [AggregateIterableTask](reference/tasks/aggregate_iterable_task.md)) as `end_point`, or write the + result to a file and check its content. +- To test a sub-process on its own, execute it directly with the process manager: it is a regular process. +- You can also test the console command with Symfony's `CommandTester` on `cleverage:process:execute`, e.g. to check + the `--input` and `--context` parsing. diff --git a/docs/cookbooks/01-common_setup.md b/docs/cookbooks/01-common_setup.md index d34c6d77..f35ab40f 100644 --- a/docs/cookbooks/01-common_setup.md +++ b/docs/cookbooks/01-common_setup.md @@ -1,47 +1,146 @@ Common Setup ============ -Bundle & task declaration -------------------------- +Task & transformer declaration +------------------------------ -Optional step: create a bundle dedicated to the process bundle +Example of a generic declaration for all the tasks and transformers of your application: -Example of a generic declaration for all tasks: ```yaml +# config/services.yaml services: - \ProcessBundle\Task\: - resource: '../../../Task/*' + App\Task\: + resource: '../src/Task/*' autowire: true public: true shared: false tags: - { name: monolog.logger, channel: cleverage_process_task } + + App\Transformer\: + resource: '../src/Transformer/*' + autowire: true + public: false + tags: + - { name: cleverage.transformer } + - { name: monolog.logger, channel: cleverage_process_transformer } +``` + +If the file also contains the default `App\:` resource (which includes `src/Task/` and `src/Transformer/`), declare +these resources after it: the last definition of a service wins. + +Alternatively, you can use `_instanceof` rules to configure every class implementing +`CleverAge\ProcessBundle\Model\TaskInterface` or `CleverAge\ProcessBundle\Transformer\TransformerInterface`, wherever +it is located in your sources (`_instanceof` only applies to the services defined in the same file, typically by the +`App\:` resource of `config/services.yaml`): + +```yaml +services: + _instanceof: + CleverAge\ProcessBundle\Model\TaskInterface: + public: true + shared: false + tags: + - { name: monolog.logger, channel: cleverage_process_task } + CleverAge\ProcessBundle\Transformer\TransformerInterface: + tags: + - { name: cleverage.transformer } + - { name: monolog.logger, channel: cleverage_process_transformer } ``` Configuration ------------- -Process are mostly defined in a `app/config/process` subfolder. You should replicate process codes depending on folder -layout (`my.feature.process` should be in `my/feature/process.yml`). +Processes are usually defined in a `config/packages/process/` subfolder, imported from the bundle configuration file: -Private subprocess could be defined in the same file than their parents, only if parents are only wrappers. +```yaml +# config/packages/clever_age_process.yaml +imports: + - { resource: process/ } + +clever_age_process: + default_error_strategy: stop +``` + +You should replicate process codes in the folder layout (`my.feature.process` should be in +`config/packages/process/my/feature/process.yaml`). + +Private subprocesses could be defined in the same file as their parents, only if parents are only wrappers. Logging ------- -A simple default configuration, with rotating file, would be +A simple default configuration, with rotating files, would be: ```yaml +# config/packages/monolog.yaml monolog: handlers: - cdm_process: + process: type: rotating_file - path: '%kernel.logs_dir%/cdm_process-%kernel.environment%.log' + path: '%kernel.logs_dir%/process-%kernel.environment%.log' max_files: 10 channels: ['cleverage_process'] - cdm_tasks: + process_tasks: type: rotating_file - path: '%kernel.logs_dir%/cdm_tasks-%kernel.environment%.log' + path: '%kernel.logs_dir%/process_tasks-%kernel.environment%.log' max_files: 10 channels: ['cleverage_process_task', 'cleverage_process_transformer'] ``` + +Remember to exclude those channels from your main handlers if you don't want them to be logged twice (e.g. +`channels: ['!cleverage_process', '!cleverage_process_task', '!cleverage_process_transformer']`). + +Example: lightweight file import +-------------------------------- + +Here is a minimal CSV-to-CSV workflow built only with tasks shipped with the bundle: +[CsvReaderTask](../reference/tasks/csv_reader_task.md) (iterable, reads one line at a time), +[TransformerTask](../reference/tasks/transformer_task.md) and [CsvWriterTask](../reference/tasks/csv_writer_task.md) +(blocking, writes each line and outputs the file path at the end). + +```yaml +# config/packages/process/app/file_import.yaml +clever_age_process: + configurations: + app.file_import: + description: 'Prepare the products CSV file' + help: 'bin/console cleverage:process:execute app.file_import -c "file:''products.csv''"' + tasks: + read_csv: + service: '@CleverAge\ProcessBundle\Task\File\Csv\CsvReaderTask' + options: + file_path: '%kernel.project_dir%/var/data/{{ file }}' + delimiter: ';' + outputs: [transform] + + transform: + service: '@CleverAge\ProcessBundle\Task\TransformerTask' + error_strategy: skip + options: + transformers: + mapping: + mapping: + id: + code: '[id]' + slug: + code: + - '[name]' + - '[category]' + transformers: + implode: + separator: ' ' + slugify: + separator: '-' + outputs: [write_csv] + + write_csv: + service: '@CleverAge\ProcessBundle\Task\File\Csv\CsvWriterTask' + options: + file_path: '%kernel.project_dir%/var/output/products_prepared_{date_time}.csv' + headers: [id, slug] +``` + +The CSV reader uses the first line of the file as headers (when the `headers` option is not set) and outputs each line +as an associative array. Lines that cannot be transformed are skipped (and logged) thanks to `error_strategy: skip`, +while any other error stops the process. diff --git a/docs/cookbooks/etl_aggregate_reports.md b/docs/cookbooks/etl_aggregate_reports.md new file mode 100644 index 00000000..26941dc4 --- /dev/null +++ b/docs/cookbooks/etl_aggregate_reports.md @@ -0,0 +1,95 @@ +ETL report aggregation +====================== + +This example shows an ETL path that reads several JSON stream log files (one JSON object per line), aggregates the +calls per service, and writes a CSV summary that can feed a dashboard. + +Each line of the log files looks like: + +```json +{"service": "catalog", "duration": 120, "status": 200, "date": "2024-01-01T10:00:00+00:00"} +``` + +```yaml +clever_age_process: + configurations: + app.etl_report_aggregate: + description: 'Aggregate API logs per service' + tasks: + list_sources: + service: '@CleverAge\ProcessBundle\Task\File\FolderBrowserTask' + options: + folder_path: '%kernel.project_dir%/var/logs/api' + name_pattern: '*.json-stream' + outputs: [read_log] + + read_log: + service: '@CleverAge\ProcessBundle\Task\File\JsonStream\JsonStreamReaderTask' # Reads the file path given as input + outputs: [map_log] + + map_log: + service: '@CleverAge\ProcessBundle\Task\TransformerTask' + options: + transformers: + mapping: + mapping: + service: { code: '[service]' } + duration: { code: '[duration]' } + status: { code: '[status]' } + outputs: [group_by_service] + + group_by_service: + service: '@CleverAge\ProcessBundle\Task\RowAggregatorTask' + options: + aggregate_by: service + aggregate_columns: [duration, status] + aggregation_key: calls + outputs: [iterate_services] + + iterate_services: + service: '@CleverAge\ProcessBundle\Task\InputIteratorTask' + outputs: [compute_stats] + + compute_stats: + service: '@CleverAge\ProcessBundle\Task\TransformerTask' + options: + transformers: + mapping: + mapping: + service: + code: '[service]' + calls: + code: '[calls]' + transformers: + callback: + callback: count + total_duration: + code: '[calls]' + transformers: + callback#1: + callback: array_column + right_parameters: [duration] + callback#2: + callback: array_sum + outputs: [write_summary] + + write_summary: + service: '@CleverAge\ProcessBundle\Task\File\Csv\CsvWriterTask' + options: + file_path: '%kernel.project_dir%/var/exports/report_summary_{date}.csv' + headers: [service, calls, total_duration] +``` + +How it works: +- [FolderBrowserTask](../reference/tasks/folder_browser_task.md) iterates over the files of the folder matching + `name_pattern`, and outputs their path. +- [JsonStreamReaderTask](../reference/tasks/json_stream_reader_task.md) opens the file given as input and iterates over + its lines, each one decoded as an array. +- [RowAggregatorTask](../reference/tasks/row_aggregator_task.md) is blocking: it groups the rows by `service`, storing + the `duration` and `status` of each call under the `calls` key, and outputs the list of groups once all files have + been read. For big volumes, keep the aggregated columns to the strict minimum, since everything is kept in memory. +- [InputIteratorTask](../reference/tasks/input_iterator_task.md) iterates over the groups, and the + [TransformerTask](../reference/tasks/transformer_task.md) computes the number of calls and the total duration with + [callback transformers](../reference/transformers/callback_transformer.md) (the `#1`/`#2` suffixes allow using the + same transformer twice, see [TransformerTrait](../reference/traits/transformer_trait.md)). +- [CsvWriterTask](../reference/tasks/csv_writer_task.md) writes one line per service. diff --git a/docs/cookbooks/etl_file_sync.md b/docs/cookbooks/etl_file_sync.md new file mode 100644 index 00000000..30d36154 --- /dev/null +++ b/docs/cookbooks/etl_file_sync.md @@ -0,0 +1,94 @@ +File synchronization ETL +======================== + +This recipe describes a typical ETL flow: read a CSV file, reject invalid lines, normalize the data, remove duplicates, +then write the result to another CSV file while logging some statistics. + +```yaml +clever_age_process: + configurations: + app.etl_file_sync: + description: 'Normalize the catalog CSV file' + tasks: + read_source: + service: '@CleverAge\ProcessBundle\Task\File\Csv\CsvReaderTask' + options: + file_path: '%kernel.project_dir%/var/data/catalog.csv' + delimiter: ';' + outputs: [filter_valid] + + filter_valid: + service: '@CleverAge\ProcessBundle\Task\FilterTask' + options: + not_empty: + '[sku]': ~ + outputs: [normalize] + error_outputs: [log_rejected] # Lines without sku + + log_rejected: + service: '@CleverAge\ProcessBundle\Task\Reporting\LoggerTask' + options: + level: warning + message: 'Line without sku' + + normalize: + service: '@CleverAge\ProcessBundle\Task\TransformerTask' + error_strategy: skip # A line with an invalid date is logged and skipped + options: + transformers: + mapping: + mapping: + sku: + code: '[sku]' + price: + code: '[price]' + transformers: + cast: + type: float + updated_at: + code: '[updated_at]' + transformers: + date_parser: + format: 'd/m/Y H:i' + date_format: + format: 'Y-m-d' + outputs: [deduplicate, count_rows] + + count_rows: + service: '@CleverAge\ProcessBundle\Task\Reporting\StatCounterTask' + + deduplicate: + service: '@CleverAge\ProcessBundle\Task\GroupByAggregateIterableTask' + options: + group_by_accessors: ['[sku]'] # The last line of each sku is kept + outputs: [iterate] + + iterate: + service: '@CleverAge\ProcessBundle\Task\InputIteratorTask' + outputs: [write_target] + + write_target: + service: '@CleverAge\ProcessBundle\Task\File\Csv\CsvWriterTask' + options: + file_path: '%kernel.project_dir%/var/exports/catalog_normalized_{date}.csv' + headers: [sku, price, updated_at] +``` + +How it works: +- [CsvReaderTask](../reference/tasks/csv_reader_task.md) is iterable: each line goes through the following tasks + before the next one is read. +- [FilterTask](../reference/tasks/filter_task.md) skips lines with an empty `sku` and sends them to its error branch, + where the [LoggerTask](../reference/tasks/logger_task.md) logs them. +- The [TransformerTask](../reference/tasks/transformer_task.md) casts the price and reformats the date (see the + [mapping](../reference/transformers/mapping_transformer.md), + [date_parser](../reference/transformers/date_parser_transformer.md) and + [date_format](../reference/transformers/date_format_transformer.md) transformers). +- [StatCounterTask](../reference/tasks/stat_counter_task.md) counts the normalized lines and logs the total at the + end of the process. +- [GroupByAggregateIterableTask](../reference/tasks/group_by_aggregate_iterable_task.md) is blocking: it keeps the + last line of each `sku` and outputs them all at once when the file has been fully read. The + [InputIteratorTask](../reference/tasks/input_iterator_task.md) then iterates again over this array, one line at a + time, to the [CsvWriterTask](../reference/tasks/csv_writer_task.md). + +Note that deduplication requires keeping one line per `sku` in memory: for very big files, prefer deduplicating at the +destination (e.g. with a database unique key). diff --git a/docs/cookbooks/memory_usage_graph.md b/docs/cookbooks/memory_usage_graph.md index c72161f9..2976f127 100644 --- a/docs/cookbooks/memory_usage_graph.md +++ b/docs/cookbooks/memory_usage_graph.md @@ -4,7 +4,7 @@ Memory usage analysis This method is clearly not the most elegant one but it doesn't require any special tool apart from Gnuplot on your desktop environment. -Add this to your process: +Add these tasks to your process: ```yaml memory: service: '@CleverAge\ProcessBundle\Task\TransformerTask' @@ -13,7 +13,7 @@ memory: mapping: mapping: memory_usage: - set_null: true + constant: false # Argument of memory_get_usage(), the input is ignored transformers: callback: callback: memory_get_usage @@ -23,18 +23,65 @@ write_memory: service: '@CleverAge\ProcessBundle\Task\File\Csv\CsvWriterTask' options: file_path: '%kernel.project_dir%/memory.dat' - headers: - - memory_usage + headers: [memory_usage] + write_headers: false ``` -Then just output regularly to the ```memory``` and this will write the memory usage to the ```memory.dat``` at the root -of your project. +Then add `memory` to the `outputs` of the task you want to monitor (e.g. the one reading your data): each time this +task produces an output, the current memory usage is written to the `memory.dat` file at the root of your project. Then launch your process using the production environment (you can't rely on the development environment memory wise). -To graph the output of this process, use this Gnuplot command in your host environment: -(not in a container because Gnuplot uses the X server to output the window containing the graph) +To graph the output of this process, use this Gnuplot command in your host environment +(not in a container because Gnuplot uses the X server to output the window containing the graph): ```bash $ gnuplot -e 'while(1) {plot "memory.dat" using 0:1 with lines; pause 1; reread}' ``` + +Alternative: log memory per process phase +----------------------------------------- + +For a more granular analysis, measure the memory at several points of your process, with a constant `phase` column to +identify each of them. All measure tasks send their output to the same `write_memory` task: + +```yaml +memory_after_read: + service: '@CleverAge\ProcessBundle\Task\TransformerTask' + options: + transformers: + mapping: + mapping: + phase: + constant: after_read + memory_usage: + constant: false + transformers: + callback: + callback: memory_get_usage + outputs: [write_memory] + +memory_after_transform: + service: '@CleverAge\ProcessBundle\Task\TransformerTask' + options: + transformers: + mapping: + mapping: + phase: + constant: after_transform + memory_usage: + constant: false + transformers: + callback: + callback: memory_get_usage + outputs: [write_memory] + +write_memory: + service: '@CleverAge\ProcessBundle\Task\File\Csv\CsvWriterTask' + options: + file_path: '%kernel.project_dir%/memory_phase.csv' + headers: [phase, memory_usage] +``` + +Add `memory_after_read` to the outputs of your reader task, and `memory_after_transform` to the outputs of your +transformer task, then compare the values of each phase in `memory_phase.csv`. diff --git a/docs/cookbooks/performances_monitoring.md b/docs/cookbooks/performances_monitoring.md index 8a7e798d..b6fdc807 100644 --- a/docs/cookbooks/performances_monitoring.md +++ b/docs/cookbooks/performances_monitoring.md @@ -1,7 +1,8 @@ Performances Monitoring ======================= -For heavy work there is multiple solutions to improve speed (_TODO add link to multithreading cookbook_) and memory consumption. +For heavy work there are multiple solutions to improve speed (see [parallelization](../04-advanced_workflow.md#parallelization)) +and memory consumption (see [memory usage analysis](memory_usage_graph.md)). While developing custom tasks you might want to see how well your PHP code behaves, and one solution is to use [Blackfire](https://blackfire.io) to analyse call graphs with timings & memory analysis. @@ -46,4 +47,25 @@ CPU Time n/a Memory 5.35MB Network n/a n/a n/a SQL n/a n/a -``` +``` + +## Built-in timing information + +Before profiling, the process logs already give some timing information: +- on success, the process manager logs `Process succeed` (level `info`, channel `cleverage_process`) + with the total `duration` of the process, in seconds, in the record context +- at `debug` level, the same channel logs each task execution (`Processing task `, `Proceeding task ...`, + `Flushing task ...`): with a formatter displaying milliseconds, it shows where the time is spent. With the Monolog + console handler of the Symfony recipe, `-vvv` displays debug records in the console: + +```bash +$ ./bin/console cleverage:process:execute app.file_import -vvv +``` + +For a precise analysis of a heavy task (e.g. a CSV reader or writer, or a custom task doing database or API calls), +profile the process with Blackfire as shown above, preferably with a production-like data set and in the `prod` +environment: + +```bash +$ blackfire run php bin/console --env=prod cleverage:process:execute app.file_import +``` diff --git a/docs/index.md b/docs/index.md index 26d38e0b..5e96db36 100644 --- a/docs/index.md +++ b/docs/index.md @@ -4,21 +4,21 @@ - [Task types](02-task_types.md) - [Custom tasks and development](03-custom_tasks.md) - [Advanced workflow](04-advanced_workflow.md) +- [Good practices](05-good_practices.md) +- [Testing](06-testing.md) - Cookbooks - [Common Setup](cookbooks/01-common_setup.md) - - [Transformations] - - [Flow manipulation] - - [Dummy tasks] - - [Debugging] - - [Logging] - - [Subprocess] - - [File manipulation] - - [Direct call (in controller)] + - [File synchronization ETL](cookbooks/etl_file_sync.md) + - [ETL report aggregation](cookbooks/etl_aggregate_reports.md) - [Performances monitoring](cookbooks/performances_monitoring.md) - [Memory usage analysis](cookbooks/memory_usage_graph.md) - Reference - [Process definition](reference/01-process_definition.md) - [Task definition](reference/02-task_definition.md) + - [Generic transformers definition](reference/03-generic_transformers_definition.md) + - Traits + - [ConditionTrait](reference/traits/condition_trait.md) + - [TransformerTrait](reference/traits/transformer_trait.md) - Basic and debug - [ConstantOutputTask](reference/tasks/constant_output_task.md) - [ConstantIterableOutputTask](reference/tasks/constant_iterable_output_task.md) @@ -32,18 +32,18 @@ - Data manipulation and transformations - [DenormalizerTask](reference/tasks/denormalizer_task.md) - [NormalizerTask](reference/tasks/normalizer_task.md) - - [DeserializerTask] - - [SerializerTask] + - [DeserializerTask](reference/tasks/deserializer_task.md) + - [SerializerTask](reference/tasks/serializer_task.md) - [PropertyGetterTask](reference/tasks/property_getter_task.md) - [PropertySetterTask](reference/tasks/property_setter_task.md) - - [ObjectUpdaterTask] - - [SplitJoinLineTask] + - [ObjectUpdaterTask](reference/tasks/object_updater_task.md) + - [SplitJoinLineTask](reference/tasks/split_join_line_task.md) - [TransformerTask](reference/tasks/transformer_task.md) - - [ValidatorTask] + - [ValidatorTask](reference/tasks/validator_task.md) - File/CSV - [CsvReaderTask](reference/tasks/csv_reader_task.md) - [CsvWriterTask](reference/tasks/csv_writer_task.md) - - [CSVSplitterTask] + - [CsvSplitterTask](reference/tasks/csv_splitter_task.md) - [InputCsvReaderTask](reference/tasks/input_csv_reader_task.md) - File/JsonStream - [JsonStreamReaderTask](reference/tasks/json_stream_reader_task.md) @@ -52,14 +52,14 @@ - [XmlReaderTask](reference/tasks/xml_reader_task.md) - [XmlWriterTask](reference/tasks/xml_writer_task.md) - File/Yaml - - [YamlReaderTask] - - [YamlWriterTask] + - [YamlReaderTask](reference/tasks/yaml_reader_task.md) + - [YamlWriterTask](reference/tasks/yaml_writer_task.md) - File - - [FileMoverTask] + - [FileMoverTask](reference/tasks/file_mover_task.md) - [FileReaderTask](reference/tasks/file_reader_task.md) - - [FileRemoverTask] + - [FileRemoverTask](reference/tasks/file_remover_task.md) - [FileSplitterTask](reference/tasks/file_splitter_task.md) - - [FileWriterTask] + - [FileWriterTask](reference/tasks/file_writer_task.md) - [FolderBrowserTask](reference/tasks/folder_browser_task.md) - [InputFileReaderTask](reference/tasks/input_file_reader_task.md) - [InputFolderBrowserTask](reference/tasks/input_folder_browser_task.md) @@ -69,69 +69,69 @@ - [AggregateIterableTask](reference/tasks/aggregate_iterable_task.md) - [InputAggregatorTask](reference/tasks/input_aggregator_task.md) - [InputIteratorTask](reference/tasks/input_iterator_task.md) - - [ArrayMergeTask] - - [ColumnAggregatorTask] - - [RowAggregatorTask] - - [FilterTask] - - [GroupByAggregateIterableTask] - - [SimpleBatchTask] - - [IterableBatchTask] - - [SkipEmptyTask] - - [StopTask] + - [ArrayMergeTask](reference/tasks/array_merge_task.md) + - [ColumnAggregatorTask](reference/tasks/column_aggregator_task.md) + - [RowAggregatorTask](reference/tasks/row_aggregator_task.md) + - [FilterTask](reference/tasks/filter_task.md) + - [GroupByAggregateIterableTask](reference/tasks/group_by_aggregate_iterable_task.md) + - [SimpleBatchTask](reference/tasks/simple_batch_task.md) + - [IterableBatchTask](reference/tasks/iterable_batch_task.md) + - [SkipEmptyTask](reference/tasks/skip_empty_task.md) + - [StopTask](reference/tasks/stop_task.md) - Process - - [CommandRunnerTask] - - [ProcessExecutorTask] - - [ProcessLauncherTask] + - [CommandRunnerTask](reference/tasks/command_runner_task.md) + - [ProcessExecutorTask](reference/tasks/process_executor_task.md) + - [ProcessLauncherTask](reference/tasks/process_launcher_task.md) - Reporting - - [AdvancedStatCounterTask] + - [AdvancedStatCounterTask](reference/tasks/advanced_stat_counter_task.md) - [LoggerTask](reference/tasks/logger_task.md) - - [StatCounterTask] + - [StatCounterTask](reference/tasks/stat_counter_task.md) - Transformers - Basic and debug - - [CachedTransformer] - - [CallbackTransformer] - - [CastTransformer] - - [ConstantTransformer] - - [ConvertValueTransformer] - - [DebugTransformer] - - [DefaultTransformer] - - [GenericTransformer] - - [EvaluatorTransformer] - - [ExpressionLanguageMapTransformer] + - [CachedTransformer](reference/transformers/cached_transformer.md) + - [CallbackTransformer](reference/transformers/callback_transformer.md) + - [CastTransformer](reference/transformers/cast_transformer.md) + - [ConstantTransformer](reference/transformers/constant_transformer.md) + - [ConvertValueTransformer](reference/transformers/convert_value_transformer.md) + - [DebugTransformer](reference/transformers/debug_transformer.md) + - [DefaultTransformer](reference/transformers/default_transformer.md) + - [GenericTransformer](reference/transformers/generic_transformer.md) + - [EvaluatorTransformer](reference/transformers/evaluator_transformer.md) + - [ExpressionLanguageMapTransformer](reference/transformers/expression_language_map_transformer.md) - [MappingTransformer](reference/transformers/mapping_transformer.md) - [MultiReplaceTransformer](reference/transformers/multi_replace_transformer.md) - - [PregFilterTransformer] + - [PregFilterTransformer](reference/transformers/preg_filter_transformer.md) - [RulesTransformer](reference/transformers/rules_transformer.md) - - [TypeSetterTransformer] - - [UnsetTransformer] - - [WrapperTransformer] + - [TypeSetterTransformer](reference/transformers/type_setter_transformer.md) + - [UnsetTransformer](reference/transformers/unset_transformer.md) + - [WrapperTransformer](reference/transformers/wrapper_transformer.md) - Array - - [ArrayElementTransformer] + - [ArrayElementTransformer](reference/transformers/array_element_transformer.md) - [ArrayFilterTransformer](reference/transformers/array_filter_transformer.md) - - [ArrayFirstTransformer] - - [ArrayLastTransformer] + - [ArrayFirstTransformer](reference/transformers/array_first_transformer.md) + - [ArrayLastTransformer](reference/transformers/array_last_transformer.md) - [ArrayMapTransformer](reference/transformers/array_map_transformer.md) - - [ArrayUnsetTransformer] + - [ArrayUnsetTransformer](reference/transformers/array_unset_transformer.md) - Date - - [DateFormatTransformer](reference/transformers/date_format.md) - - [DateParserTransformer](reference/transformers/date_parser.md) + - [DateFormatTransformer](reference/transformers/date_format_transformer.md) + - [DateParserTransformer](reference/transformers/date_parser_transformer.md) - Object - - [InstantiateTransformer] - - [PropertyAccessorTransformer] - - [RecursivePropertySetterTransformer] + - [InstantiateTransformer](reference/transformers/instantiate_transformer.md) + - [PropertyAccessorTransformer](reference/transformers/property_accessor_transformer.md) + - [RecursivePropertySetterTransformer](reference/transformers/recursive_property_setter_transformer.md) - Serialization - - [DenormalizeTransformer] - - [NormalizeTransformer] + - [DenormalizeTransformer](reference/transformers/denormalize_transformer.md) + - [NormalizeTransformer](reference/transformers/normalize_transformer.md) - String - - [ExplodeTransformer] - - [HashTransformer] + - [ExplodeTransformer](reference/transformers/explode_transformer.md) + - [HashTransformer](reference/transformers/hash_transformer.md) - [ImplodeTransformer](reference/transformers/implode_transformer.md) - [PregMatchTransformer](reference/transformers/preg_match_transformer.md) - [SlugifyTransformer](reference/transformers/slugify_transformer.md) - [SprintfTransformer](reference/transformers/sprintf_transformer.md) - [TrimTransformer](reference/transformers/trim_transformer.md) - XML - - [XpathEvaluatorTransformer](reference/transformers/xpath_evaluator.md) + - [XpathEvaluatorTransformer](reference/transformers/xpath_evaluator_transformer.md) - Other bridges - [Archive](https://github.com/cleverage/archive-process-bundle) - [Cache](https://github.com/cleverage/cache-process-bundle) @@ -139,5 +139,4 @@ - [Flysystem](https://github.com/cleverage/flysystem-process-bundle) - [Rest](https://github.com/cleverage/rest-process-bundle) - [Soap](https://github.com/cleverage/soap-process-bundle) - - [Generic transformers definition](reference/03-generic_transformers_definition.md) - [UI](https://github.com/cleverage/ui-process-bundle) diff --git a/docs/reference/01-process_definition.md b/docs/reference/01-process_definition.md index adc6913b..3b8675ff 100644 --- a/docs/reference/01-process_definition.md +++ b/docs/reference/01-process_definition.md @@ -6,34 +6,56 @@ YAML Configuration ```yaml clever_age_process: + default_error_strategy: configurations: - : + : description: help: entry_point: end_point: public: - options: - tasks: - + options: + tasks: + : ``` +Global attributes +----------------- + +**default_error_strategy**: optional, either *stop* (default) or *skip*. Error strategy used by every task that does +not define its own `error_strategy` (see [task definition](02-task_definition.md)). + Process attributes ------------------ -**description**: optional string to describe a process. Displayed in process list and help. Should not be too long -(~ one line). +The process code (key of the `configurations` array) must be unique in the whole application: a process cannot be +defined twice, even in different files. + +**description**: optional string to describe a process. Displayed in process list and help. Should not be too long +(~ one line). Default is empty. + +**help**: optional string to describe in depth a process. Displayed in process help. Can be multiline. Default is +empty. + +**entry_point**: optional task code (default is none) that will receive the process input (`--input` option of the +execute command, or `$input` argument of `ProcessManager::execute()`). The referenced task cannot have ancestors, and +must be in the [main branch](#main-branch) of the process. -**help**: optional string to describe in depth a process. Displayed in process help. Can be multiline. +**end_point**: optional task code (default is none) whose last output will be returned as the process output. It must +be in the main branch of the process. -**entry_point**: optional task code (default is none) that will receive the process input. The referenced task cannot have -ancestors. +**public**: optional boolean (default is true) to mark a process as public or private. Private processes are filtered +from the process list (unless `--all` is used) but execution is still allowed. -**end_point**: optional task code (default is none) that will provide the process output +**options**: optional free array (default is empty), not used by this bundle itself. It is available to other bundles +through `ProcessConfiguration::getOptions()`, e.g. [cleverage/ui-process-bundle](https://github.com/cleverage/ui-process-bundle) +reads its `ui` key to configure the launch form. -**public**: optional boolean (default is true) to mark a process as public or private. Private process are filtered from -process list but execution is still allowed +**tasks**: list of task definitions contained in the process, indexed by task code. See +[task definition](02-task_definition.md). -**options**: deprecated variable node +### Main branch -**tasks**: list of task definitions contained in the process. See [task definition](02-task_definition.md) +Only the *main branch* of the process is executed: the group of connected tasks containing the `entry_point`, or else +the `end_point`, or else the first configured task. Other tasks are ignored with a warning (see +[orphan tasks](../04-advanced_workflow.md#orphan-tasks)). diff --git a/docs/reference/02-task_definition.md b/docs/reference/02-task_definition.md index f222848c..5a43c7a9 100644 --- a/docs/reference/02-task_definition.md +++ b/docs/reference/02-task_definition.md @@ -11,30 +11,35 @@ YAML Configuration help: options: outputs: - errors: + error_outputs: + errors: # Deprecated, use error_outputs error_strategy: - log_errors: # Deprecated log_level: ``` -Process attributes ------------------- +Task attributes +--------------- -**service**: reference service used for the task, must implement `CleverAge\ProcessBundle\Model\TaskInterface` +**service**: required, reference to the service used for the task, with or without a leading `@`. The service must be +public and implement `CleverAge\ProcessBundle\Model\TaskInterface`. -**description**: optional string to describe a task, displayed in process help (should not exceed one line) +**description**: optional string to describe a task, displayed in process help (should not exceed one line). -**help**: optional string to describe in depth a task, displayed in verbose process help (can be multiline) +**help**: optional string to describe in depth a task (can be multiline). -**options**: optional list of parameters to pass to a task +**options**: optional list of parameters to pass to the task. String values can contain `{{ key }}` placeholders that +are replaced by the process [contextual values](../01-quick_start.md#contextual-values). -**outputs**: optional list of following tasks, it can be a simple string +**outputs**: optional list of following tasks, receiving the output of this task. It can be a simple string. -**errors**: optional list of following tasks, in case of error, it can be a simple string +**error_outputs**: optional list of following tasks, receiving the error output of this task (by default its input when +an error occurs). It can be a simple string. See [errors and skips](../04-advanced_workflow.md#errors-and-skips). -**error_strategy**: either *skip* (default) or *stop*, defines if a task can be continued or not +**errors**: deprecated alias of `error_outputs`. Defining both on the same task throws an exception. -**log_errors**: DEPRECATED: use log_level instead. Optional boolean (defaults to true), to allow logging thrown errors +**error_strategy**: optional, either *skip* or *stop*. Defines if the process continues with the next input or stops +when the task fails. When not defined, the global `default_error_strategy` is used (*stop* by default). -**log_level**: rfc5424 severity (emergency, alert, critical, error, warning, notice, info, debug) for error logged when -an exception is thrown by a task. Default 'critical'. Case-independant. +**log_level**: optional [RFC 5424](https://datatracker.ietf.org/doc/html/rfc5424) severity (emergency, alert, critical, +error, warning, notice, info, debug) of the log record written on the `cleverage_process_task` channel when the task +fails. Default is *critical*. diff --git a/docs/reference/03-generic_transformers_definition.md b/docs/reference/03-generic_transformers_definition.md index 9615761c..1883a819 100644 --- a/docs/reference/03-generic_transformers_definition.md +++ b/docs/reference/03-generic_transformers_definition.md @@ -1,6 +1,10 @@ Generic transformers definition =============================== +Generic transformers are reusable transformers defined only with configuration: a named chain of existing +transformers, optionally parameterized by *contextual options*. Each one is registered in the transformer registry +under its code, and can then be used like any other transformer (in a `TransformerTask`, in a `mapping`, ...). + YAML Configuration ------------------ @@ -10,12 +14,13 @@ clever_age_process: : contextual_options: : - required: - default: - default_is_null: - transformers: + required: + default: + default_is_null: + transformers: ``` + Options ------- @@ -27,6 +32,45 @@ For each contextual option, you can define | `default` | `any` | | `null` | If not `null`, define the default value | | `default_is_null` | `bool` | | `false` | If you need `null` to be the default value, use this option | -The transformer options are the same than any other transformer using a sub-list of transformers (see [TransformerTrait](traits/transformer_trait.md)). +Note that an option with a default value is still required by default, which has no effect since the default is used: +set `required: false` for an option without default value that may be omitted. + +The `transformers` list uses the same syntax as any other transformer using a sub-list of transformers (see +[TransformerTrait](traits/transformer_trait.md)). You can use the syntax for contextual values (`{{ contextual_option_code }}`) to put placeholders that will be filled by -those contextual options. +those contextual options, with the values given when the generic transformer is used. If the whole value is a +placeholder, the raw option value is injected (it can be an array, an integer...). + +Example +------- + +```yaml +clever_age_process: + generic_transformers: + app_slug_prefix: + contextual_options: + prefix: ~ # Required option + separator: + default: '-' + transformers: + slugify: ~ + sprintf: + format: '{{ prefix }}{{ separator }}%s' + + configurations: + app.demo: + tasks: + transform: + service: '@CleverAge\ProcessBundle\Task\TransformerTask' + options: + transformers: + mapping: + mapping: + slug: + code: '[name]' + transformers: + app_slug_prefix: + prefix: product +``` + +As any transformer code, generic transformer codes must be unique: use a prefix to avoid conflicts. diff --git a/docs/reference/tasks/_template.md b/docs/reference/tasks/_template.md index 842c5f37..919390a2 100644 --- a/docs/reference/tasks/_template.md +++ b/docs/reference/tasks/_template.md @@ -1,43 +1,47 @@ TaskName ======== -_Describe main goal an use cases of the task_ +_Describe the main goal and use cases of the task._ Task reference -------------- -* **Service**: `ClassName` +* **Service**: `Fully\Qualified\ClassName` +* **Iterable task** _(only if it implements `IterableTaskInterface`)_ +* **Blocking task** _(only if it implements `BlockingTaskInterface`)_ +* **Flushable task** _(only if it implements `FlushableTaskInterface`)_ Accepted inputs --------------- -_Description of allowed types_ +_Description of allowed types, or "Input is ignored"._ Possible outputs ---------------- -_Description of possible types_ +_Description of possible types._ Options ------- -| Code | Type | Required | Default | Description | -|--------|--------|--------------------|--------------------------------|---------------| -| `code` | `type` | **X** _or nothing_ | `default value` _if available_ | _description_ | + +| Code | Type | Required | Default | Description | +|--------|--------|:--------:|-----------------|---------------| +| `code` | `type` | **X** | `default value` | _description_ | + +_If the task has no option, replace the table with "This task has no option."._ Examples -------- -_YAML samples and explanations_ - * Example 1 - details - - details - + ```yaml # Task configuration level code: - service: '@service_ref' - options: - a: 1 - b: 2 + service: '@Fully\Qualified\ClassName' + options: + a: 1 + b: 2 + outputs: [next_task] ``` diff --git a/docs/reference/tasks/advanced_stat_counter_task.md b/docs/reference/tasks/advanced_stat_counter_task.md new file mode 100644 index 00000000..6d47abff --- /dev/null +++ b/docs/reference/tasks/advanced_stat_counter_task.md @@ -0,0 +1,50 @@ +AdvancedStatCounterTask +======================= + +Logs performance statistics (`info` level) every N executions: time since the last log, processing rate, number of +processed items and total elapsed time. + +Task reference +-------------- + +* **Service**: `CleverAge\ProcessBundle\Task\Reporting\AdvancedStatCounterTask` + +Accepted inputs +--------------- + +Input is ignored, only the number of executions matters. + +Possible outputs +---------------- + +`null` when statistics are logged, otherwise the task is skipped (nothing is sent to the outputs). It is meant to be +used as the last task of a branch. + +The logged message has the following format: + +``` +Last iteration 00:00:12 ago - 1,50 items/s - 500 items processed in 00:05:33 +``` + +Options +------- + +| Code | Type | Required | Default | Description | +|--------------|-------|:--------:|---------|---------------------------------------------------------------------------------------| +| `num_items` | `int` | | `1` | Number of items represented by one execution (multiplier of the counter) | +| `skip_first` | `int` | | `0` | Number of first executions to ignore (the elapsed time still starts at the first one) | +| `show_every` | `int` | | `1` | Log the statistics every N executions (the first counted execution is never logged) | + +Examples +-------- + +* Log statistics every 100 batches of 50 items + +```yaml +# Task configuration level +stats: + service: '@CleverAge\ProcessBundle\Task\Reporting\AdvancedStatCounterTask' + options: + num_items: 50 + show_every: 100 +``` diff --git a/docs/reference/tasks/aggregate_iterable_task.md b/docs/reference/tasks/aggregate_iterable_task.md index 027ec782..246d6b0d 100644 --- a/docs/reference/tasks/aggregate_iterable_task.md +++ b/docs/reference/tasks/aggregate_iterable_task.md @@ -1,7 +1,8 @@ AggregateIterableTask -=============== +===================== -Aggregate every input given +Collects every received input in a list, and outputs the whole list once all previous tasks are resolved (typically +at the end of an upstream iteration). Task reference -------------- @@ -17,4 +18,25 @@ Accepted inputs Possible outputs ---------------- -`array`: list of received inputs +`array`: list of all received inputs, in reception order. The task is skipped if no input was received. + +Options +------- + +This task has no option. + +Examples +-------- + +* Aggregate iterated values: outputs `[1, 2, 3]` + +```yaml +# Task configuration level +data: + service: '@CleverAge\ProcessBundle\Task\ConstantIterableOutputTask' + options: + output: [1, 2, 3] + outputs: [aggregate] +aggregate: + service: '@CleverAge\ProcessBundle\Task\AggregateIterableTask' +``` diff --git a/docs/reference/tasks/array_merge_task.md b/docs/reference/tasks/array_merge_task.md new file mode 100644 index 00000000..6b85199c --- /dev/null +++ b/docs/reference/tasks/array_merge_task.md @@ -0,0 +1,48 @@ +ArrayMergeTask +============== + +Merges every input array into a single result using a configurable PHP merge function, and outputs the result once +all previous tasks are resolved. + +Task reference +-------------- + +* **Service**: `CleverAge\ProcessBundle\Task\ArrayMergeTask` +* **Blocking task** + +Accepted inputs +--------------- + +`array`: any other type throws an `\UnexpectedValueException` + +Possible outputs +---------------- + +`array`: result of `merge_function(previous_result, input)` applied to each input in turn (starting from `[]`) + +Options +------- + +| Code | Type | Required | Default | Description | +|------------------|----------|:--------:|---------------|-----------------------------------------------------------------------------------------------------------------------| +| `merge_function` | `string` | | `array_merge` | PHP function used to merge; one of `array_merge`, `array_merge_recursive`, `array_replace`, `array_replace_recursive` | + +Examples +-------- + +* Merge iterated arrays, later keys overriding earlier ones recursively + +```yaml +# Task configuration level +data: + service: '@CleverAge\ProcessBundle\Task\ConstantIterableOutputTask' + options: + output: + - { a: 1, b: { c: 2 } } + - { b: { d: 3 } } + outputs: [merge] +merge: + service: '@CleverAge\ProcessBundle\Task\ArrayMergeTask' + options: + merge_function: array_replace_recursive +``` diff --git a/docs/reference/tasks/column_aggregator_task.md b/docs/reference/tasks/column_aggregator_task.md new file mode 100644 index 00000000..47244242 --- /dev/null +++ b/docs/reference/tasks/column_aggregator_task.md @@ -0,0 +1,62 @@ +ColumnAggregatorTask +==================== + +For each configured column, collects the input rows that contain this column (and match the optional condition), and +outputs all the groups once all previous tasks are resolved. + +Task reference +-------------- + +* **Service**: `CleverAge\ProcessBundle\Task\ColumnAggregatorTask` +* **Blocking task** + +Accepted inputs +--------------- + +`array`: an associative array that should contain the configured `columns` (a column whose value is `null` is +considered missing) + +Possible outputs +---------------- + +`array`: associative array indexed by column **name**, each entry being: +* ``: the column name +* ``: list of the whole input rows that contain this column and matched the condition + +Columns that never matched are absent; if nothing matched, the output is an empty array. + +Options +------- + +| Code | Type | Required | Default | Description | +|-------------------|----------|:--------:|----------|-------------------------------------------------------------------------------------------------------------------------------------------------| +| `columns` | `array` | **X** | | List of column keys to aggregate on | +| `reference_key` | `string` | | `column` | Key holding the column name in each output group | +| `aggregation_key` | `string` | | `values` | Key holding the aggregated rows in each output group | +| `condition` | `array` | | `[]` | Conditions (`match`, `not_match`, `empty`, `not_empty`, `match_regexp`, `not_match_regexp`), see [ConditionTrait](../traits/condition_trait.md) | +| `ignore_missing` | `bool` | | `false` | If `true`, missing columns only log a warning instead of throwing an `\UnexpectedValueException` | + +The condition is checked, for each column, against an array with two keys: `input_column_value` (the value of the +column) and `input` (the whole row). Use property paths such as `[input_column_value]` or `[input][status]`. + +Examples +-------- + +* Aggregate rows whose `col1` equals `A` + +```yaml +# Task configuration level +iterate: + service: '@CleverAge\ProcessBundle\Task\InputIteratorTask' + outputs: [aggregate_a] +aggregate_a: + service: '@CleverAge\ProcessBundle\Task\ColumnAggregatorTask' + options: + columns: [col1] + condition: + match: + '[input_column_value]': A +``` + +With inputs `{col1: A, col2: 1}`, `{col1: B, col2: 2}` and `{col1: A, col2: 3}`, the output is +`{col1: {column: col1, values: [{col1: A, col2: 1}, {col1: A, col2: 3}]}}`. diff --git a/docs/reference/tasks/command_runner_task.md b/docs/reference/tasks/command_runner_task.md new file mode 100644 index 00000000..50a45399 --- /dev/null +++ b/docs/reference/tasks/command_runner_task.md @@ -0,0 +1,57 @@ +CommandRunnerTask +================= + +Runs a system command (with the Symfony Process component) for each input. The input is passed to the command as +stdin, and the command standard output becomes the task output. + +Task reference +-------------- + +* **Service**: `CleverAge\ProcessBundle\Task\Process\CommandRunnerTask` + +Accepted inputs +--------------- + +`string|int|float|bool|resource|\Traversable|null`: passed as stdin to the command (see `Process::setInput()`). + +Possible outputs +---------------- + +`string`: the standard output of the command. + +The command is run with `Process::mustRun()`: a non-zero exit code or a timeout throws an exception, handled +according to the task `error_strategy`. + +Options +------- + +| Code | Type | Required | Default | Description | +|---------------|--------------------|:--------:|---------------------------|--------------------------------------------------------------------------------| +| `commandline` | `string\|array` | **X** | | Command to run, as an array of arguments (recommended) or a string | +| `cwd` | `string\|null` | | Symfony project directory | Working directory of the command | +| `env` | `array\|null` | | `null` | Environment variables of the command (`null` inherits the current environment) | +| `timeout` | `int\|float\|null` | | `60` | Timeout in seconds (`null` disables it) | +| `options` | `mixed` | | `null` | Unused by the task itself, see Notes | + +Examples +-------- + +* Count the lines of the input + +```yaml +# Task configuration level +count_lines: + service: '@CleverAge\ProcessBundle\Task\Process\CommandRunnerTask' + options: + commandline: ['wc', '-l'] + timeout: 30 + outputs: [next_task] +``` + +Notes +----- + +The task calls `Process::setOptions()` with **all** its resolved options (`commandline`, `cwd`, `env`, `timeout` and +`options`). Recent versions of `symfony/process` only accept `blocking_pipes`, `create_process_group` and +`create_new_console` there and throw a `LogicException` for any other key, which makes the task fail before the +command is started. diff --git a/docs/reference/tasks/constant_iterable_output_task.md b/docs/reference/tasks/constant_iterable_output_task.md index 30b04829..cb4c9d78 100644 --- a/docs/reference/tasks/constant_iterable_output_task.md +++ b/docs/reference/tasks/constant_iterable_output_task.md @@ -1,12 +1,14 @@ ConstantIterableOutputTask ========================== -Same as ConstantOutputTask but only accepts an array of values and iterates over each element. +Same as [ConstantOutputTask](constant_output_task.md), but the `output` option must be an array: the task iterates +over it and outputs each value one by one, regardless of the input. Task reference -------------- * **Service**: `CleverAge\ProcessBundle\Task\ConstantIterableOutputTask` +* **Iterable task** Accepted inputs --------------- @@ -16,25 +18,28 @@ Input is ignored Possible outputs ---------------- -`any`: iterate on the `output` option +`any`: each value of the `output` array (keys are not transmitted). If the array is empty, the task is skipped. Options ------- | Code | Type | Required | Default | Description | |----------|---------|:--------:|---------|---------------------------------| -| `output` | `array` | **X** | | Array of values to iterate onto | +| `output` | `array` | **X** | | Array of values to iterate over | -Example -------- +Examples +-------- + +* Iterate over a static list: `123`, `Test1` and `Test2` are sent one after the other to `debug` ```yaml # Task configuration level -code: +entry: service: '@CleverAge\ProcessBundle\Task\ConstantIterableOutputTask' options: output: id: 123 firstname: Test1 lastname: Test2 + outputs: [debug] ``` diff --git a/docs/reference/tasks/constant_output_task.md b/docs/reference/tasks/constant_output_task.md index ed2f042f..165f49c0 100644 --- a/docs/reference/tasks/constant_output_task.md +++ b/docs/reference/tasks/constant_output_task.md @@ -1,7 +1,8 @@ ConstantOutputTask ================== -Simply outputs the same configured value all the time, ignores any input +Always outputs the same configured value, regardless of the input. Commonly used as an entry point to feed a +process with static data. Task reference -------------- @@ -16,25 +17,28 @@ Input is ignored Possible outputs ---------------- -`any`: directly output given `output` option +`any`: the value of the `output` option, as is Options ------- | Code | Type | Required | Default | Description | -|----------|-------|:---------|---------|-----------------| -| `output` | `any` | **X** | | Value to output | +|----------|-------|:--------:|---------|-----------------| +| `output` | `any` | **X** | | Value to output | -Example -------- +Examples +-------- + +* Output a static array, then dump it ```yaml # Task configuration level -code: +entry: service: '@CleverAge\ProcessBundle\Task\ConstantOutputTask' options: output: id: 123 firstname: Test1 lastname: Test2 + outputs: [debug] ``` diff --git a/docs/reference/tasks/counter_task.md b/docs/reference/tasks/counter_task.md index 455539e4..9cc9a5af 100644 --- a/docs/reference/tasks/counter_task.md +++ b/docs/reference/tasks/counter_task.md @@ -1,39 +1,57 @@ CounterTask -================== +=========== -Count the number of times the task is processed and continue every N iteration (skip the rest of the time) - -Flush at the end with the actual count. +Counts the number of times the task is executed and only outputs the current count every `flush_every` executions +(the task is skipped the rest of the time). When flushed (at the end of the upstream iteration), it outputs the +final count, unless this count is a multiple of `flush_every` (in which case it has already been sent). Task reference -------------- * **Service**: `CleverAge\ProcessBundle\Task\CounterTask` +* **Flushable task** Accepted inputs --------------- -`any`, must implement IterableTaskInterface +Input is ignored (only the number of executions matters) Possible outputs ---------------- -`int`: outputs the number of times the counter is called +`int`: the number of times the task has been executed so far Options ------- -| Code | Type | Required | Default | Description | -|---------------|-------|----------|----------|---------------------------------------------------| -| `flush_every` | `int` | **X** | | The period at which the task will produce outputs | +| Code | Type | Required | Default | Description | +|---------------|-------|:--------:|---------|-------------------------------------------------------| +| `flush_every` | `int` | **X** | | Output the count every N executions (N = this option) | -Example -------- +Examples +-------- + +* Count 9 iterated items, outputting `3`, `6` and `9`; with `flush_every: 4` it would output `4`, `8`, then `9` on + flush ```yaml # Task configuration level -code: +entry: + service: '@CleverAge\ProcessBundle\Task\ConstantIterableOutputTask' + options: + output: [1, 2, 3, 4, 5, 6, 7, 8, 9] + outputs: [counter] +counter: service: '@CleverAge\ProcessBundle\Task\CounterTask' options: - flush_every: 2 + flush_every: 3 + outputs: [debug] ``` + +Notes +----- + +* `flush()` can be called several times during a process (see + [Advanced workflow](../../04-advanced_workflow.md)), e.g. once at the end of the upstream iteration and once when + the counter itself is resolved. Each call outputs the current count again (unless it is a multiple of `flush_every`), + so the final count may be sent more than once to the next tasks. diff --git a/docs/reference/tasks/csv_reader_task.md b/docs/reference/tasks/csv_reader_task.md index 0e7b4c3a..14980a6c 100644 --- a/docs/reference/tasks/csv_reader_task.md +++ b/docs/reference/tasks/csv_reader_task.md @@ -1,7 +1,8 @@ CsvReaderTask ============= -Reads a CSV file and iterate on each line, returning an array of key -> values. Skips empty lines. +Reads a CSV file, whose path is set in the options, and iterates over its lines, outputting each line as an associative +array indexed by the CSV headers. Task reference -------------- @@ -17,32 +18,58 @@ Input is ignored Possible outputs ---------------- -`array`: foreach line, it will return a php array where key comes from headers and values are strings. -Underlying method is [fgetcsv](https://secure.php.net/manual/en/function.fgetcsv.php). +`array`: for each line, an associative array whose keys are the headers and values are strings. +Underlying method is [fgetcsv](https://www.php.net/manual/en/function.fgetcsv.php). + +When no line can be read (typically the trailing empty line at the end of the file), no output is produced and the +task is skipped for this iteration. Options ------- -| Code | Type | Required | Default | Description | -|-------------------|-------------------|:---------:|----------|--------------------------------------------------------------------------------------------------| -| `file_path` | `string` | **X** | | Path of the file to read from (relative to symfony root or absolute) | -| `delimiter` | `string` | | `;` | CSV delimiter | -| `enclosure` | `string` | | `"` | CSV enclosure character | -| `escape` | `string` | | `\\` | CSV escape character | -| `headers` | `array` or `null` | | `null` | Static list of CSV headers, without the option, it will be dynamically read from first input | -| `mode` | `string` | | `rb` | File open mode (see [fopen mode parameter](https://secure.php.net/manual/en/function.fopen.php)) | -| `log_empty_lines` | `bool` | | `false` | Log when the output is empty | - -Example -------- +| Code | Type | Required | Default | Description | +|-------------------|---------------|:--------:|---------|-----------------------------------------------------------------------------------------------------------------------------------| +| `file_path` | `string` | **X** | | Path of the file to read from (absolute or relative to the current working directory) | +| `delimiter` | `string` | | `;` | CSV delimiter | +| `enclosure` | `string` | | `"` | CSV enclosure character | +| `escape` | `string` | | `\` | CSV escape character | +| `headers` | `array\|null` | | `null` | Static list of CSV headers. If `null`, headers are read from the first line of the file; otherwise the first line is read as data | +| `mode` | `string` | | `rb` | File open mode (see [fopen mode parameter](https://www.php.net/manual/en/function.fopen.php)) | +| `log_empty_lines` | `bool` | | `false` | Log a warning when a line cannot be read (empty line) | + +Examples +-------- + +* Read a CSV file with a contextualized delimiter + - the delimiter must be passed on execution: `-c delimiter:";"` + +```yaml +# Task configuration level +entry: + service: '@CleverAge\ProcessBundle\Task\File\Csv\CsvReaderTask' + options: + file_path: '%kernel.project_dir%/var/data/sample.csv' + delimiter: '{{ delimiter }}' + outputs: [log_line] +``` + +* Read a CSV file without header line ```yaml # Task configuration level -code: +entry: service: '@CleverAge\ProcessBundle\Task\File\Csv\CsvReaderTask' options: - file_path: 'path/to/file.csv' - delimiter: '{{ delimiter }}' ## delimiter is contextualized you must add -c delimiter:";" on console execute + file_path: '%kernel.project_dir%/var/data/no_header.csv' + delimiter: ',' + headers: [sku, name, price] + outputs: [transform] ``` +Notes +----- +* Each line must contain exactly as many columns as there are headers, otherwise an `\UnexpectedValueException` is thrown. +* A UTF-8 BOM is removed from the first header when headers are read from the file. +* `csv_file` and `csv_line` are added to the error context of the process. +* See also [InputCsvReaderTask](input_csv_reader_task.md) to read a file path given as input. diff --git a/docs/reference/tasks/csv_splitter_task.md b/docs/reference/tasks/csv_splitter_task.md new file mode 100644 index 00000000..77634e7e --- /dev/null +++ b/docs/reference/tasks/csv_splitter_task.md @@ -0,0 +1,69 @@ +CsvSplitterTask +=============== + +Splits a large CSV file, whose path is given as input, into smaller temporary CSV files, keeping the headers in each +of them. Iterates over the chunks until the whole source file has been processed. + +Task reference +-------------- + +* **Service**: `CleverAge\ProcessBundle\Task\File\Csv\CsvSplitterTask` +* **Iterable task** + +Accepted inputs +--------------- + +`string`: path of the CSV file to split, prefixed by the `base_path` option if set +(same as [InputCsvReaderTask](input_csv_reader_task.md)). + +Possible outputs +---------------- + +`string`: path of a temporary CSV file (created in the system temporary directory) containing the headers and a chunk +of lines of the source file. + +Options +------- + +| Code | Type | Required | Default | Description | +|-------------------|---------------|:--------:|---------|------------------------------------------------------------------------------------------------------| +| `max_lines` | `int` | | `1000` | Maximum number of lines per produced file (see notes) | +| `base_path` | `string` | | `''` | Prepended (with a `/` separator) to the input path. If empty, the input path is used as is | +| `delimiter` | `string` | | `;` | CSV delimiter (used for both the source and the produced files) | +| `enclosure` | `string` | | `"` | CSV enclosure character | +| `escape` | `string` | | `\` | CSV escape character | +| `headers` | `array\|null` | | `null` | Static list of CSV headers. If `null`, headers are read from the first line of the source file | +| `mode` | `string` | | `rb` | Source file open mode (see [fopen mode parameter](https://www.php.net/manual/en/function.fopen.php)) | +| `log_empty_lines` | `bool` | | `false` | Inherited from [CsvReaderTask](csv_reader_task.md), not used by this task | + +Examples +-------- + +* Split a CSV file into chunks, then read each chunk + +```yaml +# Task configuration level +entry: + service: '@CleverAge\ProcessBundle\Task\ConstantOutputTask' + options: + output: '%kernel.project_dir%/var/data/large_file.csv' + outputs: [split] +split: + service: '@CleverAge\ProcessBundle\Task\File\Csv\CsvSplitterTask' + options: + delimiter: ';' + max_lines: 500 + outputs: [read_chunk] +read_chunk: + service: '@CleverAge\ProcessBundle\Task\File\Csv\InputCsvReaderTask' + options: + delimiter: ';' +``` + +Notes +----- + +* Lines are copied as is: they are not checked against the headers. +* Temporary files are not deleted by the task, use [FileRemoverTask](file_remover_task.md) if needed. +* The line counter of the produced file includes the header line and starts at 1, so each produced file actually + contains `max_lines - 2` data lines. diff --git a/docs/reference/tasks/csv_writer_task.md b/docs/reference/tasks/csv_writer_task.md index 0ff7e2ed..22d1d762 100644 --- a/docs/reference/tasks/csv_writer_task.md +++ b/docs/reference/tasks/csv_writer_task.md @@ -1,8 +1,8 @@ CsvWriterTask ============= -Write given array to a CSV file, will wait until the end of the previous iteration (this is a blocking task) and outputs -the file path. +Writes each received array as a line of a CSV file. As a blocking task, it waits until all inputs have been received +and then outputs the file path. Task reference -------------- @@ -13,49 +13,68 @@ Task reference Accepted inputs --------------- -`array`: foreach line, it will need a php array where key match the headers and values are convertible to string. -Underlying method is [fputcsv](https://secure.php.net/manual/en/function.fputcsv.php). +`array`: an associative array whose keys match the headers and values are convertible to string. Values that are +arrays are imploded using the `split_character` option. Underlying method is +[fputcsv](https://www.php.net/manual/en/function.fputcsv.php). Possible outputs ---------------- -`string`: absolute path of the produced file +`string`: path of the written file (with placeholders replaced), once all inputs have been processed. Options ------- -| Code | Type | Required | Default | Description | -|-------------------|-------------------|:--------:|---------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `file_path` | `string` | **X** | | Path of the file to write to (relative to symfony root or absolute).
It can also take placeholders (`{date}`, `{date_time}`, `{timestamp}` `{unique_token}`) to insert data into the filename | -| `delimiter` | `string` | | `;` | CSV delimiter | -| `enclosure` | `string` | | `"` | CSV enclosure character | -| `escape` | `string` | | `\\` | CSV escape character | -| `headers` | `array` or `null` | | `null` | Static list of CSV headers, without the option, it will be dynamically read from first line | -| `mode` | `string` | | `wb` | File open mode (see [fopen mode parameter](https://secure.php.net/manual/en/function.fopen.php)) | -| `split_character` | `string` | | `\|` | Used to implode array values | -| `write_headers` | `bool` | | `true` | Write the headers as a first line | - -Example ----------------- +| Code | Type | Required | Default | Description | +|-------------------|---------------|:--------:|---------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `file_path` | `string` | **X** | | Path of the file to write to (absolute or relative to the current working directory).
Placeholders `{date}`, `{date_time}`, `{timestamp}` and `{unique_token}` are replaced in the path | +| `delimiter` | `string` | | `;` | CSV delimiter | +| `enclosure` | `string` | | `"` | CSV enclosure character | +| `escape` | `string` | | `\` | CSV escape character | +| `headers` | `array\|null` | | `null` | Static list of CSV headers. If `null`, the keys of the first input are used | +| `mode` | `string` | | `wb` | File open mode (see [fopen mode parameter](https://www.php.net/manual/en/function.fopen.php)) | +| `split_character` | `string` | | `\|` | Used to implode array values | +| `write_headers` | `bool` | | `true` | Write the headers as first line, only if the file is empty (useful with an append `mode`) | + +Examples +-------- + +* Write a CSV file with a date in its name ```yaml # Task configuration level entry: service: '@CleverAge\ProcessBundle\Task\ConstantIterableOutputTask' - outputs: [ write ] options: output: - column1: value1-1 column2: value2-1 - column3: value3-1 - column1: value1-2 column2: value2-2 - column3: value3-2 - - column1: '' - column2: null - column3: value3-3 + outputs: [write] write: service: '@CleverAge\ProcessBundle\Task\File\Csv\CsvWriterTask' options: file_path: '%kernel.project_dir%/var/data/csv_writer_{date_time}.csv' ``` + +* Append lines to an existing file, with fixed headers + +```yaml +# Task configuration level +write: + service: '@CleverAge\ProcessBundle\Task\File\Csv\CsvWriterTask' + options: + file_path: '%kernel.project_dir%/var/data/export.csv' + mode: 'ab' + headers: [sku, name, price] +``` + +Notes +----- + +* `{date}` is replaced by `Ymd`, `{date_time}` by `Ymd_His`, `{timestamp}` by the Unix timestamp and `{unique_token}` + by a `uniqid()` value. +* The parent directory of the file is created if needed. +* Each input must contain every header key (extra keys are not allowed: the number of columns must match the number of + headers), otherwise an `\UnexpectedValueException` is thrown. Columns are written in the headers order. diff --git a/docs/reference/tasks/debug_task.md b/docs/reference/tasks/debug_task.md index cfaf44d7..9dfe0d24 100644 --- a/docs/reference/tasks/debug_task.md +++ b/docs/reference/tasks/debug_task.md @@ -1,9 +1,8 @@ DebugTask ========= -Dumps the input value to the console, obviously for debug purposes. -Only usable in dev environment (where the [VarDumper Component](https://symfony.com/doc/current/components/var_dumper.html) is enabled) - +Dumps the input using the [VarDumper Component](https://symfony.com/doc/current/components/var_dumper.html), then +passes it to the output. If VarDumper is not installed, nothing is dumped and the input is simply forwarded. Task reference -------------- @@ -18,23 +17,27 @@ Accepted inputs Possible outputs ---------------- -`any`: re-output given input +`any`: the input, unchanged Options ------- -None +This task has no option. -Example ----------------- +Examples +-------- + +* Dump a constant value ```yaml # Task configuration level -code: +entry: service: '@CleverAge\ProcessBundle\Task\ConstantOutputTask' options: output: id: 123 firstname: Test1 - lastname: Test2 + outputs: [debug] +debug: + service: '@CleverAge\ProcessBundle\Task\Debug\DebugTask' ``` diff --git a/docs/reference/tasks/denormalizer_task.md b/docs/reference/tasks/denormalizer_task.md index 9be4581b..07436b77 100644 --- a/docs/reference/tasks/denormalizer_task.md +++ b/docs/reference/tasks/denormalizer_task.md @@ -1,7 +1,8 @@ DenormalizerTask ================ -Denormalize data from the input and pass it to the output +Denormalizes the input (usually an array) into an object of the configured class, using the Symfony Serializer +`DenormalizerInterface`. Task reference -------------- @@ -11,19 +12,44 @@ Task reference Accepted inputs --------------- -`array` +`mixed`: any data supported by the configured denormalizers for the given `class` (typically an `array`). Possible outputs ---------------- -`object`, instance of `class`, as a product of the denormalization +`mixed`: result of `DenormalizerInterface::denormalize()`, usually an instance of `class` (or an array of instances when +`class` ends with `[]`). Options ------- -| Code | Type | Required | Default | Description | -| ---- | ---- | :------: | ------- | ----------- | -| `class` | `string` | **X** | | Destination class for denormalization | -| `format` | `string` | | `null` | Format for denormalization ("json", "xml", ... an empty string should also work) | -| `context` | `array` | | `[]` | Will be passed directly to the 4th parameter of the denormalize method | - +| Code | Type | Required | Default | Description | +|-----------|----------------|:--------:|---------|------------------------------------------------------------------------------| +| `class` | `string` | **X** | | Target type of the denormalization (FQCN, or `FQCN[]` for a list of objects) | +| `format` | `string\|null` | | `null` | Format passed to the denormalizer (`json`, `xml`, ...) | +| `context` | `array` | | `[]` | Denormalization context, passed as 4th argument of `denormalize()` | + +Examples +-------- + +* Denormalize an array into an entity + +```yaml +# Task configuration level +denormalize: + service: '@CleverAge\ProcessBundle\Task\Serialization\DenormalizerTask' + options: + class: App\Entity\Author + outputs: [save] +``` + +* Denormalize a decoded JSON list into an array of DTOs + +```yaml +# Task configuration level +dto: + service: '@CleverAge\ProcessBundle\Task\Serialization\DenormalizerTask' + options: + class: 'App\Dto\Commune[]' + outputs: [debug] +``` diff --git a/docs/reference/tasks/deserializer_task.md b/docs/reference/tasks/deserializer_task.md new file mode 100644 index 00000000..051d337d --- /dev/null +++ b/docs/reference/tasks/deserializer_task.md @@ -0,0 +1,43 @@ +DeserializerTask +================ + +Deserializes a string input into a PHP value (object, array...) using the Symfony Serializer `SerializerInterface`. + +Task reference +-------------- + +* **Service**: `CleverAge\ProcessBundle\Task\Serialization\DeserializerTask` + +Accepted inputs +--------------- + +`string`: the serialized data, in the configured `format`. + +Possible outputs +---------------- + +`mixed`: result of `SerializerInterface::deserialize()`, matching the configured `type`. + +Options +------- + +| Code | Type | Required | Default | Description | +|-----------|----------|:--------:|---------|--------------------------------------------------------------------| +| `type` | `string` | **X** | | Target type of the deserialization (FQCN, `FQCN[]`, ...) | +| `format` | `string` | **X** | | Format of the input data (`json`, `xml`, `csv`, ...) | +| `context` | `array` | | `[]` | Deserialization context, passed as 4th argument of `deserialize()` | + +Examples +-------- + +* Deserialize a JSON string into an entity + +```yaml +# Task configuration level +deserialize: + service: '@CleverAge\ProcessBundle\Task\Serialization\DeserializerTask' + options: + type: App\Entity\Author + format: json + outputs: [save] +``` diff --git a/docs/reference/tasks/die_task.md b/docs/reference/tasks/die_task.md index 741fdeed..f11024fd 100644 --- a/docs/reference/tasks/die_task.md +++ b/docs/reference/tasks/die_task.md @@ -1,8 +1,9 @@ DieTask -========= - -Stops the process brutally +======= +Stops the process brutally by calling PHP `exit`. The whole PHP script ends immediately: no following task, flush, +blocking task or process end handling is executed. Intended for debugging only; use [StopTask](stop_task.md) to stop a +process cleanly. Task reference -------------- @@ -12,23 +13,30 @@ Task reference Accepted inputs --------------- -`any` +Input is ignored Possible outputs ---------------- -None +None, the script is terminated Options ------- -None +This task has no option. -Example ----------------- +Examples +-------- + +* Terminate right after the first task ```yaml # Task configuration level -code: +entry: + service: '@CleverAge\ProcessBundle\Task\ConstantOutputTask' + options: + output: { id: 123 } + outputs: [die] +die: service: '@CleverAge\ProcessBundle\Task\Debug\DieTask' ``` diff --git a/docs/reference/tasks/dummy_task.md b/docs/reference/tasks/dummy_task.md index 323cafc5..e46ad30f 100644 --- a/docs/reference/tasks/dummy_task.md +++ b/docs/reference/tasks/dummy_task.md @@ -1,7 +1,8 @@ DummyTask ========= -Passes the input to the output, can be used as an entry point allow multiple tasks to be run at the entry point +Passes the input to the output without any change. Useful as an entry point to start several branches from the same +input, or as a placeholder / junction task. Task reference -------------- @@ -16,33 +17,29 @@ Accepted inputs Possible outputs ---------------- -`any`: re-output given input +`any`: the input, unchanged Options ------- -None +This task has no option. -Example -------- +Examples +-------- + +* Use as an entry point to run two branches ```yaml # Task configuration level -dummy: +entry: service: '@CleverAge\ProcessBundle\Task\DummyTask' outputs: [output1, output2] output1: service: '@CleverAge\ProcessBundle\Task\ConstantOutputTask' options: - output: - id: 123 - firstname: Test1 - lastname: Test2 + output: { id: 123 } output2: service: '@CleverAge\ProcessBundle\Task\ConstantOutputTask' options: - output: - id: 456 - firstname: Test3 - lastname: Test4 + output: { id: 456 } ``` diff --git a/docs/reference/tasks/error_forwarder_task.md b/docs/reference/tasks/error_forwarder_task.md index 592c9cb2..895e63ef 100644 --- a/docs/reference/tasks/error_forwarder_task.md +++ b/docs/reference/tasks/error_forwarder_task.md @@ -1,9 +1,8 @@ -CounterTask +ErrorForwarderTask ================== -This is a dummy task mostly intended for testing purpose. - -Forward any input to the error output. +Forwards any input to the error output and skips the normal output. Mostly intended for testing purposes (e.g. to +test error branches). Task reference -------------- @@ -18,18 +17,30 @@ Accepted inputs Possible outputs ---------------- -`any`: directly error_output given `output` option +None on the normal output (the task is always skipped). + +Error output: `any`, the input, unchanged Options ------- -None +This task has no option. -Example -------- +Examples +-------- + +* Send every item to an error branch ```yaml # Task configuration level -code: +entry: + service: '@CleverAge\ProcessBundle\Task\ConstantIterableOutputTask' + options: + output: [Error 1, Error 2, Error 3] + outputs: [error_forwarder] +error_forwarder: service: '@CleverAge\ProcessBundle\Task\Debug\ErrorForwarderTask' + error_outputs: [debug] +debug: + service: '@CleverAge\ProcessBundle\Task\Debug\DebugTask' ``` diff --git a/docs/reference/tasks/event_dispatcher_task.md b/docs/reference/tasks/event_dispatcher_task.md index 88c87e98..23ac3f98 100644 --- a/docs/reference/tasks/event_dispatcher_task.md +++ b/docs/reference/tasks/event_dispatcher_task.md @@ -1,7 +1,9 @@ EventDispatcherTask =================== -Call the Symfony event dispatcher +Dispatches a `CleverAge\ProcessBundle\Event\EventDispatcherTaskEvent` through the Symfony event dispatcher. The event +gives listeners access to the current `ProcessState` (`getState()`), so they can read the input and, when the task is +not passive, set the output themselves. Task reference -------------- @@ -16,24 +18,38 @@ Accepted inputs Possible outputs ---------------- -* `any` when `passive` option is set to true -* `null` in other cases +* `any`: the input, unchanged, when `passive` is `true` +* when `passive` is `false`: whatever a listener set with `$event->getState()->setOutput()`, `null` otherwise Options ------- -| Code | Type | Required | Default | Description | -|--------------|----------|:---------:|----------|----------------------| -| `event_name` | `string` | **X** | | | -| `passive` | `bool` | | `true` | Pass input to output | +| Code | Type | Required | Default | Description | +|--------------|----------|:--------:|---------|---------------------------------------------------------------| +| `event_name` | `string` | **X** | | Name of the event (see Notes: currently not used to dispatch) | +| `passive` | `bool` | | `true` | If `true`, the input is passed to the output before dispatch | -Example -------- +Examples +-------- + +* Dispatch an event for each item ```yaml # Task configuration level -code: +data: + service: '@CleverAge\ProcessBundle\Task\ConstantIterableOutputTask' + options: + output: [1, 2, 3] + outputs: [push_data_event] +push_data_event: service: '@CleverAge\ProcessBundle\Task\Event\EventDispatcherTask' options: - event_name: 'myapp.myevent' + event_name: myapp.data_queue ``` + +Notes +----- + +The event is dispatched without an explicit name (`$eventDispatcher->dispatch($event)`), so its name is the event +class name. Listeners must therefore subscribe to `CleverAge\ProcessBundle\Event\EventDispatcherTaskEvent`; the +`event_name` option is required and validated but not used for dispatching. diff --git a/docs/reference/tasks/file_mover_task.md b/docs/reference/tasks/file_mover_task.md new file mode 100644 index 00000000..000cbd73 --- /dev/null +++ b/docs/reference/tasks/file_mover_task.md @@ -0,0 +1,48 @@ +FileMoverTask +============= + +Moves (renames) the file passed as input to a destination path. Supports overwriting and auto-incrementing the file +name to avoid collisions. + +Task reference +-------------- + +* **Service**: `CleverAge\ProcessBundle\Task\File\FileMoverTask` + +Accepted inputs +--------------- + +`string`: path of the file to move. An `\UnexpectedValueException` is thrown if it does not exist. + +Possible outputs +---------------- + +`string`: the final destination path of the file. + +Options +------- + +| Code | Type | Required | Default | Description | +|-----------------|----------|:--------:|---------|--------------------------------------------------------------------------------------------------------------------------| +| `destination` | `string` | **X** | | Destination path. If it is an existing directory, the original file name is kept | +| `overwrite` | `bool` | | `false` | Allow overwriting an existing file at destination (otherwise an exception is thrown) | +| `autoincrement` | `bool` | | `false` | If the destination file exists, add or increment a numeric suffix before the extension (e.g. `file-1.csv`, `file-2.csv`) | + +Examples +-------- + +* Archive processed files + +```yaml +# Task configuration level +move_file: + service: '@CleverAge\ProcessBundle\Task\File\FileMoverTask' + options: + destination: '%kernel.project_dir%/var/data/archive/' + autoincrement: true +``` + +Notes +----- + +* Underlying method is Symfony `Filesystem::rename()`. diff --git a/docs/reference/tasks/file_reader_task.md b/docs/reference/tasks/file_reader_task.md index e0097a13..b1a05e8f 100644 --- a/docs/reference/tasks/file_reader_task.md +++ b/docs/reference/tasks/file_reader_task.md @@ -1,7 +1,7 @@ FileReaderTask -============= +============== -Reads a file and return raw content as a string +Reads the whole content of a file, whose path is set in the options, and outputs it as a string. Task reference -------------- @@ -22,19 +22,23 @@ Underlying method is [file_get_contents](https://www.php.net/manual/en/function. Options ------- -| Code | Type | Required | Default | Description | -|------------|----------|:---------:|----------|------------------------------------------| -| `filename` | `string` | **X** | | Path of the file to read from (absolute) | +| Code | Type | Required | Default | Description | +|------------|----------|:--------:|---------|------------------------------------------------------------------------------------------------------------------| +| `filename` | `string` | **X** | | Path of the file to read. An `\UnexpectedValueException` is thrown if the file does not exist or is not readable | -Example -------- +Examples +-------- ```yaml # Task configuration level -code: +entry: service: '@CleverAge\ProcessBundle\Task\File\FileReaderTask' options: - filename: 'path/to/file.txt' + filename: '%kernel.project_dir%/var/data/sample.txt' + outputs: [debug] ``` +Notes +----- +* See also [InputFileReaderTask](input_file_reader_task.md) to read a file path given as input. diff --git a/docs/reference/tasks/file_remover_task.md b/docs/reference/tasks/file_remover_task.md new file mode 100644 index 00000000..4ee709e1 --- /dev/null +++ b/docs/reference/tasks/file_remover_task.md @@ -0,0 +1,41 @@ +FileRemoverTask +=============== + +Deletes the file(s) or directory(ies) passed as input. Directories are removed recursively. + +Task reference +-------------- + +* **Service**: `CleverAge\ProcessBundle\Task\File\FileRemoverTask` + +Accepted inputs +--------------- + +`string|iterable`: path, or list of paths, of files or directories to remove. +Underlying method is Symfony `Filesystem::remove()`: paths that do not exist are ignored. + +Possible outputs +---------------- + +No output is set. + +Options +------- + +This task has no option. + +Examples +-------- + +* Remove each file once it has been processed + +```yaml +# Task configuration level +entry: + service: '@CleverAge\ProcessBundle\Task\File\FolderBrowserTask' + options: + folder_path: '%kernel.project_dir%/var/tmp' + outputs: [cleanup] +cleanup: + service: '@CleverAge\ProcessBundle\Task\File\FileRemoverTask' +``` diff --git a/docs/reference/tasks/file_splitter_task.md b/docs/reference/tasks/file_splitter_task.md index a652eb4b..eecd8b9f 100644 --- a/docs/reference/tasks/file_splitter_task.md +++ b/docs/reference/tasks/file_splitter_task.md @@ -1,7 +1,8 @@ FileSplitterTask -============= +================ -Split long file into smaller ones +Splits a long text file into smaller temporary files of at most `max_lines` lines. Iterates over the produced files +until the whole source file has been processed. Task reference -------------- @@ -12,23 +13,25 @@ Task reference Accepted inputs --------------- -`array`: inputs are merged with task defined options. +`array`: optional, merged over the task options (e.g. `{ file_path: ..., max_lines: ... }`), which allows to give the +file path as input. Any other input is ignored. Possible outputs ---------------- -`string`: absolute path of the produced file +`string`: path of a temporary file (created in the system temporary directory, with a `.tmp` extension) containing a +chunk of lines of the source file. Options ------- -| Code | Type | Required | Default | Description | -|-------------------------|-----------------|:--------:|----------|------------------------------------------| -| `file_path` | `string` | **X** | | Path of the file to read from (absolute) | -| `max_lines` | `int` | **X** | 1000 | Max number of line on a produced file | +| Code | Type | Required | Default | Description | +|-------------|----------|:--------:|---------|-------------------------------------------| +| `file_path` | `string` | **X** | | Path of the file to split | +| `max_lines` | `int` | | `1000` | Maximum number of lines per produced file | -Example -------- +Examples +-------- ```yaml # Task configuration level @@ -36,7 +39,15 @@ entry: service: '@CleverAge\ProcessBundle\Task\File\FileSplitterTask' options: file_path: '%kernel.project_dir%/var/data/json_stream_reader.json' - max_lines: 1 + max_lines: 100 + outputs: [read_chunk] +read_chunk: + service: '@CleverAge\ProcessBundle\Task\File\InputLineReaderTask' ``` +Notes +----- +* Values given as input are merged after option resolution, so they are not validated. +* Temporary files are not deleted by the task, use [FileRemoverTask](file_remover_task.md) if needed. +* For CSV files, prefer [CsvSplitterTask](csv_splitter_task.md) which keeps the headers in each produced file. diff --git a/docs/reference/tasks/file_writer_task.md b/docs/reference/tasks/file_writer_task.md new file mode 100644 index 00000000..d66cf076 --- /dev/null +++ b/docs/reference/tasks/file_writer_task.md @@ -0,0 +1,44 @@ +FileWriterTask +============== + +Writes the input content to a file, whose path is set in the options. The file is overwritten on each execution. + +Task reference +-------------- + +* **Service**: `CleverAge\ProcessBundle\Task\File\FileWriterTask` + +Accepted inputs +--------------- + +`string`: content to write to the file. +Underlying method is Symfony `Filesystem::dumpFile()` (parent directories are created if needed). + +Possible outputs +---------------- + +`string`: path of the written file (value of the `filename` option). + +Options +------- + +| Code | Type | Required | Default | Description | +|------------|----------|:--------:|---------|---------------------------| +| `filename` | `string` | **X** | | Path of the file to write | + +Examples +-------- + +```yaml +# Task configuration level +write_file: + service: '@CleverAge\ProcessBundle\Task\File\FileWriterTask' + options: + filename: '%kernel.project_dir%/var/data/result.txt' +``` + +Notes +----- + +* This task is not blocking: when receiving several inputs, each one overwrites the file. Aggregate the data first + (e.g. with [AggregateIterableTask](aggregate_iterable_task.md)) if needed. diff --git a/docs/reference/tasks/filter_task.md b/docs/reference/tasks/filter_task.md new file mode 100644 index 00000000..6a691b21 --- /dev/null +++ b/docs/reference/tasks/filter_task.md @@ -0,0 +1,55 @@ +FilterTask +========== + +Passes the input to the output only if it matches all the configured conditions. Otherwise, the input is sent to the +error output and the task is skipped. + +Task reference +-------------- + +* **Service**: `CleverAge\ProcessBundle\Task\FilterTask` + +Accepted inputs +--------------- + +`array` or `object`: values are read with the Symfony PropertyAccessor. A non-readable property is considered `null`. + +Possible outputs +---------------- + +`any`: the input, unchanged, when all conditions match. + +Error output: the input, unchanged, when a condition does not match. + +Options +------- + +Condition options are provided by [ConditionTrait](../traits/condition_trait.md), directly at the root of the task +options. Each option is a map of property path => value; all conditions must be satisfied. + +| Code | Type | Required | Default | Description | +|--------------------|---------|:--------:|---------|---------------------------------------------------------------------------| +| `match` | `array` | | `[]` | Property path => value: the property must be strictly equal (`===`) to it | +| `not_match` | `array` | | `[]` | Property path => value: the property must not be strictly equal to it | +| `empty` | `array` | | `[]` | Property path => (ignored): the property must be empty (PHP `empty()`) | +| `not_empty` | `array` | | `[]` | Property path => (ignored): the property must not be empty | +| `match_regexp` | `array` | | `[]` | Property path => regular expression: the property must match it | +| `not_match_regexp` | `array` | | `[]` | Property path => regular expression: the property must not match it | + +Examples +-------- + +* Keep only active items, send the others to an error branch + +```yaml +# Task configuration level +filter_active: + service: '@CleverAge\ProcessBundle\Task\FilterTask' + options: + match: + '[status]': active + not_empty: + '[sku]': ~ + outputs: [next_task] + error_outputs: [handle_inactive] +``` diff --git a/docs/reference/tasks/folder_browser_task.md b/docs/reference/tasks/folder_browser_task.md index 3dcc44ef..9d318139 100644 --- a/docs/reference/tasks/folder_browser_task.md +++ b/docs/reference/tasks/folder_browser_task.md @@ -1,7 +1,8 @@ FolderBrowserTask -============= +================= -Reads a folder and iterate on each file, returning absolute path as string. +Browses a folder, whose path is set in the options, recursively and iterates over each file found (sorted by name), +outputting its path. Task reference -------------- @@ -17,27 +18,39 @@ Input is ignored Possible outputs ---------------- -`string`: absolute path of the file. -Underlying method is [Symfony Finder component](https://symfony.com/doc/current/components/finder.html). +`string`: path of the file (`folder_path` followed by the path of the file relative to it). +Underlying component is [Symfony Finder](https://symfony.com/doc/current/components/finder.html). + +If no file is found, a message is logged with the `empty_log_level` level, the task is skipped and the folder path is +set as error output. Options ------- -| Code | Type | Required | Default | Description | -|-------------------|-----------------------------|:---------:|---------------------------|----------------------------------------------------------------------------------------| -| `folder_path` | `string` | **X** | | Path of the directory to read from | -| `name_pattern` | `null`, `string` or `array` | | null | Restrict files using a pattern (a regexp, a glob, or a string) or an array of patterns | -| `empty_log_level` | `string` | | Psr\Log\LogLevel::WARNING | From Psr\Log\LogLevel constants | +| Code | Type | Required | Default | Description | +|-------------------|-----------------------|:--------:|-----------|----------------------------------------------------------------------------------------------------------------| +| `folder_path` | `string` | **X** | | Path of the folder to browse. Must be an existing readable directory (checked at initialization) | +| `name_pattern` | `string\|array\|null` | | `null` | Restrict files by name using a pattern (glob, regexp or string) or an array of patterns (see `Finder::name()`) | +| `empty_log_level` | `string` | | `warning` | Log level used when no file is found, one of the `Psr\Log\LogLevel` constants | -Example -------- +Examples +-------- + +* Browse all CSV files of a folder ```yaml # Task configuration level -code: +entry: service: '@CleverAge\ProcessBundle\Task\File\FolderBrowserTask' options: folder_path: '%kernel.project_dir%/var/data' + name_pattern: '*.csv' + empty_log_level: info + outputs: [read] ``` +Notes +----- +* `current_file_path` is added to the error context of the process. +* See also [InputFolderBrowserTask](input_folder_browser_task.md) to browse a folder path given as input. diff --git a/docs/reference/tasks/group_by_aggregate_iterable_task.md b/docs/reference/tasks/group_by_aggregate_iterable_task.md new file mode 100644 index 00000000..a66701f2 --- /dev/null +++ b/docs/reference/tasks/group_by_aggregate_iterable_task.md @@ -0,0 +1,48 @@ +GroupByAggregateIterableTask +============================ + +Aggregates inputs in an associative array, indexed by a key built from configurable properties of each input, and +outputs it once all previous tasks are resolved. An input with the same key as a previous one replaces it, so this +task can be used to remove duplicates. + +Task reference +-------------- + +* **Service**: `CleverAge\ProcessBundle\Task\GroupByAggregateIterableTask` +* **Blocking task** + +Accepted inputs +--------------- + +`array` or `object`: values are read with the Symfony PropertyAccessor. If a property cannot be read, the exception +is handled according to the task `error_strategy`. + +Possible outputs +---------------- + +`array`: inputs indexed by the values of `group_by_accessors` joined with `-` (last input wins for a given key). The +task is skipped if no input was received. + +Options +------- + +| Code | Type | Required | Default | Description | +|----------------------|---------|:--------:|---------|-------------------------------------------------------| +| `group_by_accessors` | `array` | **X** | | List of property paths used to build the grouping key | + +Examples +-------- + +* Deduplicate items on `type` and `code` + +```yaml +# Task configuration level +deduplicate: + service: '@CleverAge\ProcessBundle\Task\GroupByAggregateIterableTask' + options: + group_by_accessors: + - '[type]' + - '[code]' +``` + +With inputs `{type: A, code: 1, v: x}` and `{type: A, code: 1, v: y}`, the output is `{A-1: {type: A, code: 1, v: y}}`. diff --git a/docs/reference/tasks/input_aggregator_task.md b/docs/reference/tasks/input_aggregator_task.md index 816e6ced..103529dd 100644 --- a/docs/reference/tasks/input_aggregator_task.md +++ b/docs/reference/tasks/input_aggregator_task.md @@ -1,11 +1,11 @@ InputAggregatorTask -=============== +=================== -Accumulate heterogeneous inputs. It skips output until input is received from every parent task. Then, the output is flush (except for `keep_inputs` indexes). +**Deprecated**: this class is marked `@deprecated` (too error-prone, should be refactored as a blocking task). -Warning : the `clean_input_on_override` option can be dangerous if set to `false`. Especially in loops (iterable process), there can be cases where the inputs are mixed between the iterations of an input... (ex: one of the parent task has skipped output due to an error). Even on `true`, some case have been determined to be problematic. - -The usage of this task is therefore **strongly discouraged**, unless you are using it in a non-iterable process. It may one day evolve in a Blocking Task. +Accumulates inputs coming from several parent tasks. The task is skipped until an input has been received from every +parent declared in `input_codes`; it then outputs an array of all received inputs, indexed by their destination key, +and clears its buffer (except for the keys listed in `keep_inputs`). Task reference -------------- @@ -15,19 +15,55 @@ Task reference Accepted inputs --------------- -`any` +`any`: the parent task (previous task) code must be declared in `input_codes`, otherwise an +`\UnexpectedValueException` is thrown. The task cannot be used without a previous task (e.g. as an entry point). Possible outputs ---------------- -`array`: list of index destination => values from previous tasks +`array`: destination key (from `input_codes`) => input received from the corresponding parent task Options ------- -| Code | Type | Required | Default | Description | -| ---- | ---- | :------: | ------- | ----------- | -| `input_codes` | `array` | **X** | | List of task code => index destination | -| `clean_input_on_override` | `bool` | | `true` | Empty the future output if there any override | -| `keep_inputs` | `array` or `null` | | `null` | List of index destination to keep on flush | +| Code | Type | Required | Default | Description | +|---------------------------|---------------|:--------:|---------|-------------------------------------------------------------------------------------------------------------------------------------------------| +| `input_codes` | `array` | **X** | | Map of parent task code => destination key in the output | +| `clean_input_on_override` | `bool` | | `true` | When an input is received again for an already filled key: if `true`, all buffered inputs are cleared first; if `false`, an exception is thrown | +| `keep_inputs` | `array\|null` | | `null` | List of destination keys that are kept in the buffer after an output | + +Examples +-------- + +* Wait for the results of two branches + +```yaml +# Task configuration level +entry: + service: '@CleverAge\ProcessBundle\Task\DummyTask' + outputs: [branch_a, branch_b] +branch_a: + service: '@CleverAge\ProcessBundle\Task\ConstantOutputTask' + options: + output: A + outputs: [aggregate] +branch_b: + service: '@CleverAge\ProcessBundle\Task\ConstantOutputTask' + options: + output: B + outputs: [aggregate] +aggregate: + service: '@CleverAge\ProcessBundle\Task\InputAggregatorTask' + options: + input_codes: + branch_a: a + branch_b: b +``` + +Notes +----- +In iterable processes, inputs of different iterations may get mixed (for instance when a parent task skips an item +because of an error), even with `clean_input_on_override: true`. Its usage is **strongly discouraged** outside of +non-iterable processes; prefer a blocking task such as [ArrayMergeTask](array_merge_task.md) or +[AggregateIterableTask](aggregate_iterable_task.md) when possible. diff --git a/docs/reference/tasks/input_csv_reader_task.md b/docs/reference/tasks/input_csv_reader_task.md index 59e67d2a..f03a8781 100644 --- a/docs/reference/tasks/input_csv_reader_task.md +++ b/docs/reference/tasks/input_csv_reader_task.md @@ -1,7 +1,8 @@ InputCsvReaderTask -============= +================== -Reads a CSV file and iterate on each line, returning an array of key -> values. Skips empty lines. +Reads a CSV file whose path is given as input and iterates over its lines, outputting each line as an associative array +indexed by the CSV headers. Same behaviour as [CsvReaderTask](csv_reader_task.md), except for the file path. Task reference -------------- @@ -12,35 +13,59 @@ Task reference Accepted inputs --------------- -`string`: file path +`string`: path of the file to read, prefixed by the `base_path` option if set. +When a different path is received, the previous file is dropped and the new one is opened. Possible outputs ---------------- -`array`: foreach line, it will return a php array where key comes from headers and values are strings. -Underlying method is [fgetcsv](https://secure.php.net/manual/en/function.fgetcsv.php). +`array`: for each line, an associative array whose keys are the headers and values are strings. +Underlying method is [fgetcsv](https://www.php.net/manual/en/function.fgetcsv.php). + +When no line can be read (typically the trailing empty line at the end of the file), no output is produced and the +task is skipped for this iteration. Options ------- -Same as [CsvReaderTask](reference/tasks/csv_reader_task.md) except following : +| Code | Type | Required | Default | Description | +|-------------------|---------------|:--------:|---------|-----------------------------------------------------------------------------------------------------------------------------------| +| `base_path` | `string` | | `''` | Prepended (with a `/` separator) to the input path. If empty, the input path is used as is | +| `delimiter` | `string` | | `;` | CSV delimiter | +| `enclosure` | `string` | | `"` | CSV enclosure character | +| `escape` | `string` | | `\` | CSV escape character | +| `headers` | `array\|null` | | `null` | Static list of CSV headers. If `null`, headers are read from the first line of the file; otherwise the first line is read as data | +| `mode` | `string` | | `rb` | File open mode (see [fopen mode parameter](https://www.php.net/manual/en/function.fopen.php)) | +| `log_empty_lines` | `bool` | | `false` | Log a warning when a line cannot be read (empty line) | -| Code | Type | Required | Default | Description | -|-------------|----------|:--------:|---------|----------------------------| -| `file_path` | | | | Removed, use input instead | -| `base_path` | `string` | | `` | | +The `file_path` option of [CsvReaderTask](csv_reader_task.md) is removed: the path comes from the input. -Example -------- +Examples +-------- + +* Read every CSV file of a folder + +```yaml +# Task configuration level +entry: + service: '@CleverAge\ProcessBundle\Task\File\FolderBrowserTask' + options: + folder_path: '%kernel.project_dir%/var/data' + name_pattern: '*.csv' + outputs: [read] +read: + service: '@CleverAge\ProcessBundle\Task\File\Csv\InputCsvReaderTask' + outputs: [dump] +``` + +* Read an uploaded file (process entry point), with a contextualized delimiter + - the delimiter must be passed on execution: `-c delimiter:";"` ```yaml -clever_age_process: - configurations: - process.name: - entry_point: entrypoint # for upload_and_run process entry_point is required - tasks: - entrypoint: - service: '@CleverAge\ProcessBundle\Task\File\Csv\InputCsvReaderTask' - options: - delimiter: '{{ delimiter }}' ## delimiter is contextualized you must add -c delimiter:";" on console execute +# Task configuration level +read: + service: '@CleverAge\ProcessBundle\Task\File\Csv\InputCsvReaderTask' + options: + delimiter: '{{ delimiter }}' + outputs: [dump] ``` diff --git a/docs/reference/tasks/input_file_reader_task.md b/docs/reference/tasks/input_file_reader_task.md index d960ba21..39e5e06b 100644 --- a/docs/reference/tasks/input_file_reader_task.md +++ b/docs/reference/tasks/input_file_reader_task.md @@ -1,7 +1,8 @@ InputFileReaderTask -============= +=================== -Reads a file and return raw content as a string +Reads the whole content of a file, whose path is given as input, and outputs it as a string. +Same behaviour as [FileReaderTask](file_reader_task.md), except for the file path. Task reference -------------- @@ -11,7 +12,8 @@ Task reference Accepted inputs --------------- -`string`: file path +`string`: path of the file to read. An `\UnexpectedValueException` is thrown if the file does not exist or is not +readable. Possible outputs ---------------- @@ -22,10 +24,12 @@ Underlying method is [file_get_contents](https://www.php.net/manual/en/function. Options ------- -None +This task has no option. -Example -------- +Examples +-------- + +* Read every file of a folder ```yaml # Task configuration level @@ -33,9 +37,8 @@ entry: service: '@CleverAge\ProcessBundle\Task\File\FolderBrowserTask' options: folder_path: '%kernel.project_dir%/var/data' - outputs: read + outputs: [read] read: service: '@CleverAge\ProcessBundle\Task\File\InputFileReaderTask' + outputs: [debug] ``` - - diff --git a/docs/reference/tasks/input_folder_browser_task.md b/docs/reference/tasks/input_folder_browser_task.md index e95e2401..b182f2ef 100644 --- a/docs/reference/tasks/input_folder_browser_task.md +++ b/docs/reference/tasks/input_folder_browser_task.md @@ -1,34 +1,46 @@ InputFolderBrowserTask -============= +====================== -Reads a folder and iterate on each file, returning absolute path as string. +Browses a folder, whose path is given as input, recursively and iterates over each file found (sorted by name), +outputting its path. Same behaviour as [FolderBrowserTask](folder_browser_task.md), except for the folder path. Task reference -------------- * **Service**: `CleverAge\ProcessBundle\Task\File\InputFolderBrowserTask` * **Iterable task** +* **Flushable task** Accepted inputs --------------- -`string`: folder path +`string`: path of the folder to browse, prefixed by the `base_folder_path` option. It must be an existing readable +directory. + +Receiving a different folder path before the task has been flushed throws a `\LogicException`. Possible outputs ---------------- -`string`: absolute path of the file. -Underlying method is [Symfony Finder component](https://symfony.com/doc/current/components/finder.html). +`string`: path of the file (folder path followed by the path of the file relative to it). +Underlying component is [Symfony Finder](https://symfony.com/doc/current/components/finder.html). + +If no file is found, a message is logged with the `empty_log_level` level, the task is skipped and the folder path is +set as error output. Options ------- -| Code | Type | Required | Default | Description | -|--------------------|----------|:---------:|---------|---------------------------------------| -| `base_folder_path` | `string` | | | Concatenated with input `folder_path` | +| Code | Type | Required | Default | Description | +|--------------------|-----------------------|:--------:|-----------|----------------------------------------------------------------------------------------------------------------| +| `base_folder_path` | `string` | | `''` | Prepended to the input path, without adding any separator | +| `name_pattern` | `string\|array\|null` | | `null` | Restrict files by name using a pattern (glob, regexp or string) or an array of patterns (see `Finder::name()`) | +| `empty_log_level` | `string` | | `warning` | Log level used when no file is found, one of the `Psr\Log\LogLevel` constants | -Example -------- +The `folder_path` option of [FolderBrowserTask](folder_browser_task.md) is removed: the path comes from the input. + +Examples +-------- ```yaml # Task configuration level @@ -36,12 +48,12 @@ entry: service: '@CleverAge\ProcessBundle\Task\ConstantOutputTask' options: output: '/var/data' - outputs: directory + outputs: [directory] directory: service: '@CleverAge\ProcessBundle\Task\File\InputFolderBrowserTask' options: base_folder_path: '%kernel.project_dir%' - outputs: read + outputs: [read] +read: + service: '@CleverAge\ProcessBundle\Task\File\InputFileReaderTask' ``` - - diff --git a/docs/reference/tasks/input_iterator_task.md b/docs/reference/tasks/input_iterator_task.md index 0dd6f4c0..6451f86d 100644 --- a/docs/reference/tasks/input_iterator_task.md +++ b/docs/reference/tasks/input_iterator_task.md @@ -1,7 +1,7 @@ InputIteratorTask -=============== +================= -Iterate on every value from given input +Iterates over the input and outputs each value one by one. Task reference -------------- @@ -12,9 +12,29 @@ Task reference Accepted inputs --------------- -`\Iterable` or `array`: an input to iterate onto +`array`, `\Iterator` or `\IteratorAggregate`: any other type throws an `\UnexpectedValueException` Possible outputs ---------------- -`any`: each value from the iterable +`any`: each value of the input (keys are not transmitted). If the input is empty, the task is skipped. + +Options +------- + +This task has no option. + +Examples +-------- + +* Aggregate a list, then iterate over it again + +```yaml +# Task configuration level +aggregate: + service: '@CleverAge\ProcessBundle\Task\AggregateIterableTask' + outputs: [iterate] +iterate: + service: '@CleverAge\ProcessBundle\Task\InputIteratorTask' + outputs: [next_task] +``` diff --git a/docs/reference/tasks/input_line_reader_task.md b/docs/reference/tasks/input_line_reader_task.md index da185453..97de9d46 100644 --- a/docs/reference/tasks/input_line_reader_task.md +++ b/docs/reference/tasks/input_line_reader_task.md @@ -1,7 +1,8 @@ InputLineReaderTask -============= +=================== -Reads a file and iterate on each line, returning content as string. Skips empty lines. +Reads a file, whose path is given as input, and iterates over its lines, outputting each line as a string. +Same behaviour as [LineReaderTask](line_reader_task.md), except for the file path. Task reference -------------- @@ -12,21 +13,25 @@ Task reference Accepted inputs --------------- -`string`: file path +`string`: path of the file to read. An `\UnexpectedValueException` is thrown if the file does not exist or is not +readable. When a different path is received, the previous file is dropped and the new one is opened. Possible outputs ---------------- -`string`: foreach line, it will return content as string. -Underlying method is [SplFileObject](https://www.php.net/manual/en/class.splfileobject.php). +`string`: for each line, its raw content. The line break is not removed. +Underlying class is [SplFileObject](https://www.php.net/manual/en/class.splfileobject.php), with the `READ_AHEAD` and +`SKIP_EMPTY` flags. Options ------- -None +This task has no option. -Example -------- +Examples +-------- + +* Read every line of every file of a folder ```yaml # Task configuration level @@ -34,9 +39,8 @@ entry: service: '@CleverAge\ProcessBundle\Task\File\FolderBrowserTask' options: folder_path: '%kernel.project_dir%/var/data' - outputs: read + outputs: [read] read: service: '@CleverAge\ProcessBundle\Task\File\InputLineReaderTask' + outputs: [log_line] ``` - - diff --git a/docs/reference/tasks/iterable_batch_task.md b/docs/reference/tasks/iterable_batch_task.md index 4f5b3170..1cd08b39 100644 --- a/docs/reference/tasks/iterable_batch_task.md +++ b/docs/reference/tasks/iterable_batch_task.md @@ -1,8 +1,10 @@ IterableBatchTask ================= -Accumulate inputs and periodically flush them using iterations. -It's mainly an example task since it's not useful as-is, but the processInput method may allow custom overrides. +Buffers inputs and, every `batch_count` inputs, iterates over the buffer to output its elements one by one. Remaining +elements are output (one by one as well) when the task is flushed, at the end of the upstream iteration. It is mainly +an example task: it is not really useful as is, but its `processInput()` method can be overridden to customize how +each input is transformed before being buffered. Task reference -------------- @@ -19,12 +21,28 @@ Accepted inputs Possible outputs ---------------- -`any`: same type as input +`any`: each buffered input (as returned by `processInput()`, the input unchanged by default) Options ------- -| Code | Type | Required | Default | Description | -| ---- | ---- | :------: | ------- | ----------- | -| `batch_count` | `integer` | | `10` | Accumulated batch size | - +| Code | Type | Required | Default | Description | +|---------------|-------|:--------:|---------|---------------------------------------------| +| `batch_count` | `int` | | `10` | Number of inputs to buffer before iterating | + +Examples +-------- + +* Buffer iterated values by 2 + +```yaml +# Task configuration level +iterator: + service: '@CleverAge\ProcessBundle\Task\InputIteratorTask' + outputs: [batch] +batch: + service: '@CleverAge\ProcessBundle\Task\IterableBatchTask' + options: + batch_count: 2 + outputs: [next_task] +``` diff --git a/docs/reference/tasks/json_stream_reader_task.md b/docs/reference/tasks/json_stream_reader_task.md index 77fc2329..76bec6ae 100644 --- a/docs/reference/tasks/json_stream_reader_task.md +++ b/docs/reference/tasks/json_stream_reader_task.md @@ -1,7 +1,8 @@ JsonStreamReaderTask -============= +==================== -Reads a json file and iterate on each line, returning decoded content as array. Skips empty lines. +Reads a JSON Lines file (one JSON document per line), whose path is given as input, and iterates over its lines, +outputting each decoded line. Task reference -------------- @@ -12,40 +13,47 @@ Task reference Accepted inputs --------------- -`string`: Path of the file to read from (absolute) +`string`: path of the file to read. Possible outputs ---------------- -`array`: foreach line, it will return content as array. -Underlying method are [SplFileObject::fgets](https://www.php.net/manual/fr/splfileobject.fgets.php) and [json_decode](https://www.php.net/manual/en/function.json-decode.php). +`array`: for each line, the decoded JSON as an associative array. +Underlying methods are [SplFileObject::fgets](https://www.php.net/manual/en/splfileobject.fgets.php) and +[json_decode](https://www.php.net/manual/en/function.json-decode.php). + +When a line is empty or decodes to `null`, no output is produced and the task is skipped for this iteration. Each +line must be a JSON object or array: a line decoding to a scalar (e.g. `42` or `"foo"`) raises a `\TypeError`. Options ------- -| Code | Type | Required | Default | Description | -|-------------------------|-----------------|:--------:|---------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `spl_file_object_flags` | `array`, `null` | | `\SplFileObject::DROP_NEW_LINE \SplFileObject::READ_AHEAD \SplFileObject::SKIP_EMPTY` | Flags to pass to `SplFileObject` constructor, can be empty.
See [PHP documentation](https://www.php.net/manual/en/splfileobject.construct.php) for more information on available flags. | -| `json_flags` | `array`, `null` | | `\JSON_THROW_ON_ERROR` | Flags to pass to `json_encode` function, can be empty.
See [PHP documentation](https://www.php.net/manual/en/function.json-encode.php) for more information on available flags. | - +| Code | Type | Required | Default | Description | +|-------------------------|---------------|:--------:|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `spl_file_object_flags` | `array\|null` | | `null` | List of `SplFileObject` flags, summed and passed to [SplFileObject::setFlags](https://www.php.net/manual/en/splfileobject.setflags.php).
`null` means `DROP_NEW_LINE`, `READ_AHEAD` and `SKIP_EMPTY`; an empty array means no flag | +| `json_flags` | `array\|null` | | `null` | List of JSON flags, summed and passed to [json_decode](https://www.php.net/manual/en/function.json-decode.php).
`null` means `JSON_THROW_ON_ERROR`; an empty array means no flag (invalid lines are then skipped instead of throwing an exception) | -Example -------- +Examples +-------- ```yaml # Task configuration level entry: - service: '@CleverAge\ProcessBundle\Task\ConstantIterableOutputTask' - outputs: read + service: '@CleverAge\ProcessBundle\Task\ConstantOutputTask' options: - output: - file_path: '%kernel.project_dir%/var/data/json_stream_reader.json' + output: '%kernel.project_dir%/var/data/json_stream_reader.json' + outputs: [read] read: service: '@CleverAge\ProcessBundle\Task\File\JsonStream\JsonStreamReaderTask' options: - spl_file_object_flags: [] json_flags: - - !php/const JSON_ERROR_NONE + - !php/const JSON_THROW_ON_ERROR + - !php/const JSON_BIGINT_AS_STRING + outputs: [dump] ``` +Notes +----- +* Files written by [JsonStreamWriterTask](json_stream_writer_task.md) can be read by this task, as long as no flag + producing multi-line JSON (like `JSON_PRETTY_PRINT`) was used. diff --git a/docs/reference/tasks/json_stream_writer_task.md b/docs/reference/tasks/json_stream_writer_task.md index 9b4a2302..96f79680 100644 --- a/docs/reference/tasks/json_stream_writer_task.md +++ b/docs/reference/tasks/json_stream_writer_task.md @@ -1,8 +1,8 @@ JsonStreamWriterTask -=============== +==================== -Write given array to a json file, will wait until the end of the previous iteration (this is a blocking task) and outputs -the file path. +Writes each received array as a JSON document on its own line (JSON Lines format), in a file whose path is set in the +options. As a blocking task, it waits until all inputs have been received and then outputs the file path. Task reference -------------- @@ -13,47 +13,52 @@ Task reference Accepted inputs --------------- -`array` +`array`: data to encode. An `\UnexpectedValueException` is thrown for any other type. +Underlying method is [json_encode](https://www.php.net/manual/en/function.json-encode.php). Possible outputs ---------------- -`string`: absolute path of the produced file +`string`: path of the written file (with placeholders replaced), once all inputs have been processed. Options ------- -| Code | Type | Required | Default | Description | -|-------------------------|-----------------|:--------:|---------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `file_path` | `string` | **X** | | Path of the file to write to (relative to symfony root or absolute).
It can also take placeholders (`{date}`, `{date_time}`, `{timestamp}` `{unique_token}`) to insert data into the filename | -| `spl_file_object_flags` | `array`, `null` | | `\SplFileObject::DROP_NEW_LINE \SplFileObject::READ_AHEAD \SplFileObject::SKIP_EMPTY` | Flags to pass to `SplFileObject` constructor, can be empty.
See [PHP documentation](https://www.php.net/manual/en/splfileobject.construct.php) for more information on available flags. | -| `json_flags` | `array`, `null` | | `\JSON_THROW_ON_ERROR` | Flags to pass to `json_encode` function, can be empty.
See [PHP documentation](https://www.php.net/manual/en/function.json-encode.php) for more information on available flags. | +| Code | Type | Required | Default | Description | +|-------------------------|---------------|:--------:|---------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `file_path` | `string` | **X** | | Path of the file to write to, opened in `wb` mode (overwritten).
Placeholders `{date}`, `{date_time}`, `{timestamp}` and `{unique_token}` are replaced in the path | +| `spl_file_object_flags` | `array\|null` | | `null` | List of `SplFileObject` flags, summed and passed to [SplFileObject::setFlags](https://www.php.net/manual/en/splfileobject.setflags.php).
`null` means `DROP_NEW_LINE`, `READ_AHEAD` and `SKIP_EMPTY`; an empty array means no flag | +| `json_flags` | `array\|null` | | `null` | List of JSON flags, summed and passed to [json_encode](https://www.php.net/manual/en/function.json-encode.php).
`null` means `JSON_THROW_ON_ERROR`; an empty array means no flag | -Example ----------------- +Examples +-------- ```yaml # Task configuration level entry: service: '@CleverAge\ProcessBundle\Task\ConstantIterableOutputTask' - outputs: [ write ] options: output: - column1: value1-1 column2: value2-1 - column3: value3-1 - column1: value1-2 column2: value2-2 - column3: value3-2 - - column1: '' - column2: null - column3: value3-3 + outputs: [write] write: service: '@CleverAge\ProcessBundle\Task\File\JsonStream\JsonStreamWriterTask' options: - file_path: '%kernel.project_dir%/var/data/json_stream_writer_{date_time}.csv' - spl_file_object_flags: [] + file_path: '%kernel.project_dir%/var/data/json_stream_writer_{date_time}.json' json_flags: - - !php/const JSON_PRETTY_PRINT + - !php/const JSON_THROW_ON_ERROR - !php/const JSON_UNESCAPED_SLASHES + - !php/const JSON_UNESCAPED_UNICODE ``` + +Notes +----- + +* `{date}` is replaced by `Ymd`, `{date_time}` by `Ymd_His`, `{timestamp}` by the Unix timestamp and `{unique_token}` + by a `uniqid()` value. +* The parent directory of the file must exist. +* Using `JSON_PRETTY_PRINT` produces multi-line documents: the file can no longer be read by + [JsonStreamReaderTask](json_stream_reader_task.md). diff --git a/docs/reference/tasks/line_reader_task.md b/docs/reference/tasks/line_reader_task.md index 8c5f2e4e..09c63472 100644 --- a/docs/reference/tasks/line_reader_task.md +++ b/docs/reference/tasks/line_reader_task.md @@ -1,7 +1,7 @@ LineReaderTask -============= +============== -Reads a file and iterate on each line, returning content as string. Skips empty lines. +Reads a file, whose path is set in the options, and iterates over its lines, outputting each line as a string. Task reference -------------- @@ -17,25 +17,38 @@ Input is ignored Possible outputs ---------------- -`string`: foreach line, it will return content as string. -Underlying method is [SplFileObject](https://www.php.net/manual/en/class.splfileobject.php). +`string`: for each line, its raw content. The line break is not removed. +Underlying class is [SplFileObject](https://www.php.net/manual/en/class.splfileobject.php), with the `READ_AHEAD` and +`SKIP_EMPTY` flags. Options ------- -| Code | Type | Required | Default | Description | -|------------|----------|:---------:|----------|------------------------------------------| -| `filename` | `string` | **X** | | Path of the file to read from (absolute) | +| Code | Type | Required | Default | Description | +|------------|----------|:--------:|---------|------------------------------------------------------------------------------------------------------------------| +| `filename` | `string` | **X** | | Path of the file to read. An `\UnexpectedValueException` is thrown if the file does not exist or is not readable | -Example -------- +Examples +-------- + +* Read a file and remove the trailing line break of each line ```yaml # Task configuration level -code: +entry: service: '@CleverAge\ProcessBundle\Task\File\LineReaderTask' options: - filename: 'path/to/file.txt' + filename: '%kernel.project_dir%/var/data/sample.txt' + outputs: [trim] +trim: + service: '@CleverAge\ProcessBundle\Task\TransformerTask' + options: + transformers: + trim: ~ + outputs: [debug] ``` +Notes +----- +* See also [InputLineReaderTask](input_line_reader_task.md) to read a file path given as input. diff --git a/docs/reference/tasks/logger_task.md b/docs/reference/tasks/logger_task.md index ffcc51ac..814346b7 100644 --- a/docs/reference/tasks/logger_task.md +++ b/docs/reference/tasks/logger_task.md @@ -1,7 +1,7 @@ LoggerTask -============= +========== -Log a specific message with context. +Logs a message with values read from the process state, then forwards the input unchanged. Task reference -------------- @@ -11,31 +11,47 @@ Task reference Accepted inputs --------------- -`any` +`mixed` Possible outputs ---------------- -`any` : forwarded input +`mixed`: the input, unchanged. Options ------- -| Code | Type | Required | Default | Description | -|-------------|--------------------|:---------:|-------------------|---------------------------------| -| `level` | `string` | **X** | `debug` | Use `Psr\Log\LogLevel` values | -| `message` | `string` | | `Log state input` | | -| `context` | `array` | | `['input']` | | -| `reference` | `string` or `null` | | `null` | Override `context['reference']` | +| Code | Type | Required | Default | Description | +|-------------|----------------|:--------:|-------------------|-----------------------------------------------------------------------------------------------------------------------------| +| `level` | `string` | | `debug` | Log level (`Psr\Log\LogLevel` values) | +| `message` | `string` | | `Log state input` | Log message | +| `context` | `array` | | `['input']` | List of property paths read on the `ProcessState` (e.g. `input`, `context`) and added to the log context under the same key | +| `reference` | `string\|null` | | `null` | If set, added to the log context as `reference` | -Example -------- +Examples +-------- + +* Log a warning with the current input ```yaml # Task configuration level -code: +log: service: '@CleverAge\ProcessBundle\Task\Reporting\LoggerTask' options: level: warning message: DEMO LOGGER + outputs: [next_task] +``` + +* Log the input and the process context with a reference + +```yaml +# Task configuration level +log: + service: '@CleverAge\ProcessBundle\Task\Reporting\LoggerTask' + options: + level: info + message: Transformed + context: [input, context] + reference: '{{ file }}' ``` diff --git a/docs/reference/tasks/normalizer_task.md b/docs/reference/tasks/normalizer_task.md index ebd1b2d8..988d8b6e 100644 --- a/docs/reference/tasks/normalizer_task.md +++ b/docs/reference/tasks/normalizer_task.md @@ -1,7 +1,7 @@ NormalizerTask ============== -Normalize data from the input and pass it to the output +Normalizes the input (usually an object) using the Symfony Serializer `NormalizerInterface`. Task reference -------------- @@ -11,18 +11,34 @@ Task reference Accepted inputs --------------- -Any normalizable object. +`mixed`: any value supported by the normalizers for the given `format`. If no normalizer supports it, an +`\UnexpectedValueException` is thrown. Possible outputs ---------------- -A normalized value as an array. +`array|string|int|float|bool|\ArrayObject|null`: result of `NormalizerInterface::normalize()` (usually an `array`). Options ------- -| Code | Type | Required | Default | Description | -| ---- | ---- | :------: | ------- | ----------- | -| `format` | `string` | **X** | | Format for normalization ("json", "xml", ... an empty string should also work) | -| `context` | `array` | | `[]` | Will be passed directly to the third parameter of the normalize method | - +| Code | Type | Required | Default | Description | +|-----------|----------|:--------:|---------|----------------------------------------------------------------| +| `format` | `string` | **X** | | Format passed to the normalizer (`json`, `xml`, ...) | +| `context` | `array` | | `[]` | Normalization context, passed as 3rd argument of `normalize()` | + +Examples +-------- + +* Normalize entities read from Doctrine into arrays, restricted to a serialization group + +```yaml +# Task configuration level +normalize: + service: '@CleverAge\ProcessBundle\Task\Serialization\NormalizerTask' + options: + format: json + context: + groups: [export] + outputs: [write] +``` diff --git a/docs/reference/tasks/object_updater_task.md b/docs/reference/tasks/object_updater_task.md new file mode 100644 index 00000000..537215af --- /dev/null +++ b/docs/reference/tasks/object_updater_task.md @@ -0,0 +1,49 @@ +ObjectUpdaterTask +================= + +Takes an array containing an object and a value, sets the value on the object at the configured property path, then +outputs the updated object. + +Task reference +-------------- + +* **Service**: `CleverAge\ProcessBundle\Task\ObjectUpdaterTask` + +Accepted inputs +--------------- + +`array` with two keys: + +* `object`: the object (or array) to update +* `value`: the value to set + +An `\UnexpectedValueException` is thrown if one of these keys is missing. + +Possible outputs +---------------- + +The `object` of the input, updated with the `value`. + +Options +------- + +| Code | Type | Required | Default | Description | +|-----------------|----------|:--------:|---------|-------------------------------------------------------------| +| `property_path` | `string` | **X** | | Property path of the `object` where the `value` will be set | + +Examples +-------- + +* Update the `name` of an object + +```yaml +# Task configuration level +update_object: + service: '@CleverAge\ProcessBundle\Task\ObjectUpdaterTask' + options: + property_path: name + outputs: [save] +``` + +With the input `['object' => $myEntity, 'value' => 'New Name']`, the property accessor sets `name` to `New Name` on +`$myEntity` (e.g. through `setName()`), and `$myEntity` is output. diff --git a/docs/reference/tasks/process_executor_task.md b/docs/reference/tasks/process_executor_task.md new file mode 100644 index 00000000..a7a5517f --- /dev/null +++ b/docs/reference/tasks/process_executor_task.md @@ -0,0 +1,45 @@ +ProcessExecutorTask +=================== + +Executes another process synchronously (in the same PHP process) for each input, passing the input to the +sub-process entry point. The output of the sub-process end point becomes the task output. This allows composing +processes. + +Task reference +-------------- + +* **Service**: `CleverAge\ProcessBundle\Task\Process\ProcessExecutorTask` + +Accepted inputs +--------------- + +`mixed`: passed as input to the sub-process (its `entry_point` task). + +Possible outputs +---------------- + +`mixed`: output of the sub-process `end_point` task, or `null` if the sub-process has no end point. + +Options +------- + +| Code | Type | Required | Default | Description | +|-----------|----------|:--------:|---------|---------------------------------------------------------------------------------------------------| +| `process` | `string` | **X** | | Code of the process to execute. An `InvalidConfigurationException` is thrown if it does not exist | +| `context` | `array` | | `[]` | Context of the sub-process (the context of the current process is **not** passed automatically) | + +Examples +-------- + +* Execute a sub-process for each input, forwarding a context value + +```yaml +# Task configuration level +run_subprocess: + service: '@CleverAge\ProcessBundle\Task\Process\ProcessExecutorTask' + options: + process: app.import_product + context: + source: '{{ source }}' + outputs: [next_task] +``` diff --git a/docs/reference/tasks/process_launcher_task.md b/docs/reference/tasks/process_launcher_task.md new file mode 100644 index 00000000..a0bef99c --- /dev/null +++ b/docs/reference/tasks/process_launcher_task.md @@ -0,0 +1,72 @@ +ProcessLauncherTask +=================== + +Launches a process in a separate system process (`bin/console cleverage:process:execute`) for each input received, +allowing parallelization. The task keeps a pool of at most `max_processes` running sub-processes and waits for a free +slot before launching a new one. + +Task reference +-------------- + +* **Service**: `CleverAge\ProcessBundle\Task\Process\ProcessLauncherTask` +* **Iterable task** +* **Flushable task** + +Accepted inputs +--------------- + +`scalar|\Stringable|null`: the input is cast to string and passed to the sub-process through stdin +(`--input-from-stdin` option of the command). + +Possible outputs +---------------- + +Nothing is output when an input is received (the task is skipped). When `json_buffering` is enabled, the sub-process +end point output (only if it is an array) is written to a JSON stream file (`var/cdm_buffer_*.json-stream`) and, once the sub-process is +finished, the task outputs the path of this file (`string`), during the next iterations or when flushed. Without +`json_buffering`, the task outputs nothing. + +If a sub-process exits with a non-zero code, its error output is logged as critical, all running sub-processes are +stopped and a `\RuntimeException` is thrown. + +Options +------- + +| Code | Type | Required | Default | Description | +|-------------------------------|--------------|:--------:|---------|-----------------------------------------------------------------------------------------------------| +| `process` | `string` | **X** | | Code of the process to launch. An `InvalidConfigurationException` is thrown if it does not exist | +| `max_processes` | `int` | | `3` | Maximum number of sub-processes running at the same time | +| `sleep_interval` | `int\|float` | | `1` | Time (in seconds) to wait between two checks when the pool is full | +| `sleep_interval_after_launch` | `int\|float` | | `1` | Time (in seconds) to wait after launching a sub-process | +| `sleep_on_finalize_interval` | `int\|float` | | `1` | Time (in seconds) to wait when the task has no finished result to output at the end of an iteration | +| `context` | `array` | | `[]` | Context values passed to each sub-process (as `--context=key:value`, values must be scalars) | +| `json_buffering` | `bool` | | `false` | Store each sub-process output in a JSON stream file and output its path | +| `process_options` | `array` | | `[]` | **Deprecated**: any non-empty value throws an `\InvalidArgumentException` (see Notes) | + +Examples +-------- + +* Import each file of a folder in parallel, 5 at a time + +```yaml +# Task configuration level +parallel_import: + service: '@CleverAge\ProcessBundle\Task\Process\ProcessLauncherTask' + options: + process: app.import_file + max_processes: 5 + sleep_interval: 0.5 + sleep_interval_after_launch: 0.1 + context: + mode: parallel +``` + +Notes +----- + +* Sub-processes are run with the same Symfony environment as the current one (`--env`). +* The incremental error output (stderr) of the running sub-processes is echoed directly. +* **Known issue**: the normalizer of the deprecated `process_options` option declares a scalar return type + (`int|float|string|bool|null`) but returns the option value, which is always an array (`[]` by default). Resolving + the options therefore raises a `\TypeError`, which is caught at initialization (logged as critical) and makes the + process fail as soon as this task is reached. diff --git a/docs/reference/tasks/property_getter_task.md b/docs/reference/tasks/property_getter_task.md index 9955bc54..1c15ecdf 100644 --- a/docs/reference/tasks/property_getter_task.md +++ b/docs/reference/tasks/property_getter_task.md @@ -1,9 +1,10 @@ PropertyGetterTask ================== -Accepts an array or an object as an input and read a value from a property path. +Reads a value from the input (array or object) using a property path, and outputs it. -See [PropertyAccess Component Reference](https://symfony.com/doc/current/components/property_access.html) for details on property path syntax and behavior. +See the [PropertyAccess component documentation](https://symfony.com/doc/current/components/property_access.html) +for the property path syntax. Task reference -------------- @@ -13,17 +14,44 @@ Task reference Accepted inputs --------------- -`array` or `object` that can be accessed by the property accessor +`array` or `object` readable by the property accessor at the given `property` path. Possible outputs ---------------- -Value of the property extracted from input. +`mixed`: the value read at the `property` path. + +If the value cannot be read, the exception is set on the state (with the `property` added to the error context) and +handled according to the task `error_strategy`. Options ------- -| Code | Type | Required | Default | Description | -| ---- | ---- | :------: | ------- | ----------- | -| `property` | `string` | **X** | | Property path to read from input | - +| Code | Type | Required | Default | Description | +|------------|----------|:--------:|---------|--------------------------------------| +| `property` | `string` | **X** | | Property path to read from the input | + +Examples +-------- + +* Extract the `path` of each file listed by a Flysystem task + +```yaml +# Task configuration level +get_file_path: + service: '@CleverAge\ProcessBundle\Task\PropertyGetterTask' + options: + property: 'path' + outputs: [remove_input] +``` + +* Read a key from an array input + +```yaml +# Task configuration level +get_sku: + service: '@CleverAge\ProcessBundle\Task\PropertyGetterTask' + options: + property: '[sku]' + outputs: [next_task] +``` diff --git a/docs/reference/tasks/property_setter_task.md b/docs/reference/tasks/property_setter_task.md index 1584d059..10d88144 100644 --- a/docs/reference/tasks/property_setter_task.md +++ b/docs/reference/tasks/property_setter_task.md @@ -1,9 +1,10 @@ PropertySetterTask ================== -Accepts an array or an object as an input and sets values before returning it as the output. +Sets static values on the input (array or object) using property paths, then outputs the modified input. -See [PropertyAccess Component Reference](https://symfony.com/doc/current/components/property_access.html) for details on property path syntax and behavior. +See the [PropertyAccess component documentation](https://symfony.com/doc/current/components/property_access.html) +for the property path syntax. Task reference -------------- @@ -13,17 +14,49 @@ Task reference Accepted inputs --------------- -`array` or `object` that can be accessed by the property accessor +`array` or `object` writable by the property accessor at the given property paths. Possible outputs ---------------- -Same `array` or `object`, with the property changed +The input, with the configured values set. + +If a value cannot be set, the exception is set on the state (with `property` and `value` added to the error context) +and handled according to the task `error_strategy`; the remaining values are not set. Note that only `string`, `int` +and `array` values can be added to the error context: for other value types (`bool`, `float`, `null`, objects), a +`\TypeError` is raised instead of the original exception (it is still handled according to `error_strategy`). Options ------- -| Code | Type | Required | Default | Description | -| ---- | ---- | :------: | ------- | ----------- | -| `values` | `array` | **X** | | List of property path => value to set in the input | - +| Code | Type | Required | Default | Description | +|----------|---------|:--------:|---------|-----------------------------------------------------| +| `values` | `array` | **X** | | Map of `property path => value` to set on the input | + +Examples +-------- + +* Change the first name of an entity + +```yaml +# Task configuration level +modify: + service: '@CleverAge\ProcessBundle\Task\PropertySetterTask' + options: + values: + firstname: Gérard + outputs: [dump_modified] +``` + +* Set keys on an array input + +```yaml +# Task configuration level +set_defaults: + service: '@CleverAge\ProcessBundle\Task\PropertySetterTask' + options: + values: + '[status]': imported + '[source]': '{{ source }}' + outputs: [next_task] +``` diff --git a/docs/reference/tasks/row_aggregator_task.md b/docs/reference/tasks/row_aggregator_task.md new file mode 100644 index 00000000..ecd675b1 --- /dev/null +++ b/docs/reference/tasks/row_aggregator_task.md @@ -0,0 +1,55 @@ +RowAggregatorTask +================= + +Groups input rows sharing the same value for the `aggregate_by` column. Each group is made of the first received row +(without the `aggregate_columns`), plus a sub-array under `aggregation_key` listing the `aggregate_columns` values of +every row of the group. The groups are output once all previous tasks are resolved. + +Task reference +-------------- + +* **Service**: `CleverAge\ProcessBundle\Task\RowAggregatorTask` +* **Blocking task** + +Accepted inputs +--------------- + +`array`: an associative array containing the `aggregate_by` key and all the `aggregate_columns` keys, otherwise an +`InvalidProcessConfigurationException` is thrown + +Possible outputs +---------------- + +`array`: list of groups (indexed numerically, in order of first appearance) + +Options +------- + +| Code | Type | Required | Default | Description | +|---------------------|----------|:--------:|---------|--------------------------------------------------------------| +| `aggregate_by` | `string` | **X** | | Column used to group rows together | +| `aggregate_columns` | `array` | **X** | | List of columns to collect into the aggregation sub-array | +| `aggregation_key` | `string` | **X** | | Key of the sub-array containing the aggregated column values | + +Examples +-------- + +* Group order lines by order + +```yaml +# Task configuration level +aggregate_rows: + service: '@CleverAge\ProcessBundle\Task\RowAggregatorTask' + options: + aggregate_by: order_id + aggregate_columns: [product, qty] + aggregation_key: lines +``` + +With inputs `{order_id: 1, customer: X, product: A, qty: 2}`, `{order_id: 1, customer: X, product: B, qty: 3}` and +`{order_id: 2, customer: Y, product: C, qty: 1}`, the output is: + +```yaml +- { order_id: 1, customer: X, lines: [{ product: A, qty: 2 }, { product: B, qty: 3 }] } +- { order_id: 2, customer: Y, lines: [{ product: C, qty: 1 }] } +``` diff --git a/docs/reference/tasks/serializer_task.md b/docs/reference/tasks/serializer_task.md new file mode 100644 index 00000000..aa589256 --- /dev/null +++ b/docs/reference/tasks/serializer_task.md @@ -0,0 +1,43 @@ +SerializerTask +============== + +Serializes the input into a string of the given format using the Symfony Serializer `SerializerInterface`. + +Task reference +-------------- + +* **Service**: `CleverAge\ProcessBundle\Task\Serialization\SerializerTask` + +Accepted inputs +--------------- + +`mixed`: any data supported by the serializer for the given `format`. + +Possible outputs +---------------- + +`string`: the serialized representation of the input. + +Options +------- + +| Code | Type | Required | Default | Description | +|-----------|----------|:--------:|---------|----------------------------------------------------------------| +| `format` | `string` | **X** | | Serialization format (`json`, `xml`, `csv`, ...) | +| `context` | `array` | | `[]` | Serialization context, passed as 3rd argument of `serialize()` | + +Examples +-------- + +* Serialize the input as pretty-printed JSON + +```yaml +# Task configuration level +serialize: + service: '@CleverAge\ProcessBundle\Task\Serialization\SerializerTask' + options: + format: json + context: + json_encode_options: !php/const JSON_PRETTY_PRINT + outputs: [write] +``` diff --git a/docs/reference/tasks/simple_batch_task.md b/docs/reference/tasks/simple_batch_task.md new file mode 100644 index 00000000..4602a73b --- /dev/null +++ b/docs/reference/tasks/simple_batch_task.md @@ -0,0 +1,48 @@ +SimpleBatchTask +=============== + +Buffers inputs and outputs them as an array every `batch_count` inputs (the task is skipped the rest of the time). +Remaining items are output when the task is flushed, at the end of the upstream iteration. + +Task reference +-------------- + +* **Service**: `CleverAge\ProcessBundle\Task\SimpleBatchTask` +* **Flushable task** + +Accepted inputs +--------------- + +`any` + +Possible outputs +---------------- + +`array`: list of buffered inputs, with at most `batch_count` elements. On flush, the task is skipped if the buffer is +empty. + +Options +------- + +| Code | Type | Required | Default | Description | +|---------------|-------------|:--------:|---------|-------------------------------------------------------------------------| +| `batch_count` | `int\|null` | | `10` | Batch size; if `null`, all inputs are buffered and only output on flush | + +Examples +-------- + +* Group iterated values by 2: outputs `[1, 2]`, then `[3]` on flush + +```yaml +# Task configuration level +data: + service: '@CleverAge\ProcessBundle\Task\ConstantIterableOutputTask' + options: + output: [1, 2, 3] + outputs: [batch] +batch: + service: '@CleverAge\ProcessBundle\Task\SimpleBatchTask' + options: + batch_count: 2 + outputs: [next_task] +``` diff --git a/docs/reference/tasks/skip_empty_task.md b/docs/reference/tasks/skip_empty_task.md new file mode 100644 index 00000000..6003c3e2 --- /dev/null +++ b/docs/reference/tasks/skip_empty_task.md @@ -0,0 +1,40 @@ +SkipEmptyTask +============= + +Passes the input to the output, but skips it if it is empty (PHP `empty()`: `null`, `false`, `0`, `'0'`, `''`, `[]`). +Useful after an aggregator task to avoid processing empty results. + +Task reference +-------------- + +* **Service**: `CleverAge\ProcessBundle\Task\SkipEmptyTask` + +Accepted inputs +--------------- + +`any` + +Possible outputs +---------------- + +`any`: the input, unchanged, if it is not empty (the task is skipped otherwise) + +Options +------- + +This task has no option. + +Examples +-------- + +* Only continue when the merged result is not empty + +```yaml +# Task configuration level +merge: + service: '@CleverAge\ProcessBundle\Task\ArrayMergeTask' + outputs: [skip_if_empty] +skip_if_empty: + service: '@CleverAge\ProcessBundle\Task\SkipEmptyTask' + outputs: [next_task] +``` diff --git a/docs/reference/tasks/split_join_line_task.md b/docs/reference/tasks/split_join_line_task.md new file mode 100644 index 00000000..ab5e5d2b --- /dev/null +++ b/docs/reference/tasks/split_join_line_task.md @@ -0,0 +1,55 @@ +SplitJoinLineTask +================= + +Splits a single line (array) into multiple lines: each configured column is exploded with a split character, and each +resulting value produces a new output line where it is stored in a single "join" column. + +Task reference +-------------- + +* **Service**: `CleverAge\ProcessBundle\Task\SplitJoinLineTask` +* **Iterable task** + +Accepted inputs +--------------- + +`array`: a line containing all the `split_columns` keys. An `\UnexpectedValueException` is thrown if one is missing. + +Possible outputs +---------------- + +`array`: one line per split value, iterated in the order of `split_columns`. Each line contains all the original columns +except the `split_columns`, plus the `join_column` holding the split value (as a string). + +Options +------- + +| Code | Type | Required | Default | Description | +|-------------------|----------|:--------:|---------|------------------------------------------------------| +| `split_columns` | `array` | **X** | | List of the columns whose values will be split | +| `join_column` | `string` | **X** | | Name of the output column receiving each split value | +| `split_character` | `string` | | `,` | Delimiter used to explode the columns values | + +Examples +-------- + +* Split two columns into a single `value` column + +```yaml +# Task configuration level +split_line: + service: '@CleverAge\ProcessBundle\Task\SplitJoinLineTask' + options: + split_columns: [category, tag] + join_column: value + split_character: ',' + outputs: [next_task] +``` + +With the input `{category: "A,B,C", tag: "x,y", name: "Item1"}`, the task iterates over 5 lines: + +- `{name: "Item1", value: "A"}` +- `{name: "Item1", value: "B"}` +- `{name: "Item1", value: "C"}` +- `{name: "Item1", value: "x"}` +- `{name: "Item1", value: "y"}` diff --git a/docs/reference/tasks/stat_counter_task.md b/docs/reference/tasks/stat_counter_task.md new file mode 100644 index 00000000..eed8a84f --- /dev/null +++ b/docs/reference/tasks/stat_counter_task.md @@ -0,0 +1,37 @@ +StatCounterTask +=============== + +Counts the number of times the task is executed and logs the total (`info` level) when the process ends. + +Task reference +-------------- + +* **Service**: `CleverAge\ProcessBundle\Task\Reporting\StatCounterTask` + +Accepted inputs +--------------- + +Input is ignored. + +Possible outputs +---------------- + +`null`: the task does not set any output. It is meant to be used as the last task of a branch. + +At finalization, the message `Processed item count: ` is logged. + +Options +------- + +This task has no option. + +Examples +-------- + +* Count the lines written by a process + +```yaml +# Task configuration level +count: + service: '@CleverAge\ProcessBundle\Task\Reporting\StatCounterTask' +``` diff --git a/docs/reference/tasks/stop_task.md b/docs/reference/tasks/stop_task.md new file mode 100644 index 00000000..1cb9bd5c --- /dev/null +++ b/docs/reference/tasks/stop_task.md @@ -0,0 +1,41 @@ +StopTask +======== + +Immediately stops the process and marks its history as failed. Unlike [DieTask](die_task.md), the stop is handled by +the process manager (no following task is executed, including remaining iterations). Useful to halt execution in an +error branch. + +Task reference +-------------- + +* **Service**: `CleverAge\ProcessBundle\Task\StopTask` + +Accepted inputs +--------------- + +Input is ignored + +Possible outputs +---------------- + +None, the process is stopped + +Options +------- + +This task has no option. + +Examples +-------- + +* Stop the process on the first validation error + +```yaml +# Task configuration level +validate: + service: '@CleverAge\ProcessBundle\Task\Validation\ValidatorTask' + error_strategy: skip + error_outputs: [abort] +abort: + service: '@CleverAge\ProcessBundle\Task\StopTask' +``` diff --git a/docs/reference/tasks/stopwatch_task.md b/docs/reference/tasks/stopwatch_task.md index 4953239b..05728a2e 100644 --- a/docs/reference/tasks/stopwatch_task.md +++ b/docs/reference/tasks/stopwatch_task.md @@ -1,7 +1,8 @@ StopwatchTask ============= -Log all the __root__ events of the Stopwatch component. +Logs (at `info` level, on the `cleverage_process_task` channel) every event of the `__root__` section of the Symfony +[Stopwatch component](https://symfony.com/doc/current/components/stopwatch.html). Useful to profile a process. Task reference -------------- @@ -11,23 +12,25 @@ Task reference Accepted inputs --------------- -`any` +Input is ignored Possible outputs ---------------- -None +`null`: no output is set Options ------- -None +This task has no option. -Example -------- +Examples +-------- + +* Log stopwatch events ```yaml # Task configuration level -code: +stopwatch: service: '@CleverAge\ProcessBundle\Task\Debug\StopwatchTask' ``` diff --git a/docs/reference/tasks/transformer_task.md b/docs/reference/tasks/transformer_task.md index efc98bcd..55720cae 100644 --- a/docs/reference/tasks/transformer_task.md +++ b/docs/reference/tasks/transformer_task.md @@ -1,9 +1,8 @@ TransformerTask =============== -Pass an input into a chain of transformers. - -See transformers references. +Passes the input through a chain of transformers and outputs the result. This is the main way to use transformers +inside a process. Task reference -------------- @@ -13,28 +12,60 @@ Task reference Accepted inputs --------------- -`any`: it should match the 1st expected input of the transform chain +`mixed`: it must match the expected input of the first transformer of the chain. Possible outputs ---------------- -`any`: result of the transform chain +`mixed`: result of the last transformer of the chain. With an empty `transformers` list, the input is output unchanged. + +If a transformer fails, the resulting `TransformerException` is set on the state (with the original error message +added to the error context as `error`) and handled according to the task `error_strategy`. Options ------- -| Code | Type | Required | Default | Description | -|----------------|---------|:---------:|----------|------------------------------------------------------------------------------| -| `transformers` | `array` | **X** | | List of transformers, see [TransformerTrait](../traits/transformer_trait.md) | +| Code | Type | Required | Default | Description | +|----------------|---------|:--------:|---------|------------------------------------------------------------------------------------------------------| +| `transformers` | `array` | | `[]` | Ordered map of `transformer code => options`, see [TransformerTrait](../traits/transformer_trait.md) | -Example -------- +Examples +-------- + +* Decode a JSON string into an associative array ```yaml # Task configuration level -code: +json_decode: service: '@CleverAge\ProcessBundle\Task\TransformerTask' options: transformers: - slugify: ~ + callback: + callback: json_decode + right_parameters: [true] + outputs: [dto] ``` + +* Map an array to a new structure, chaining the same transformer twice with the `#` suffix + +```yaml +# Task configuration level +transform: + service: '@CleverAge\ProcessBundle\Task\TransformerTask' + options: + transformers: + mapping: + mapping: + name: + code: '[firstname]' + transformers: + trim: ~ + callback#1: + callback: array_filter + callback#2: + callback: array_reverse + outputs: [load] +``` + +See [MappingTransformer](../transformers/mapping_transformer.md) and +[generic transformers](../03-generic_transformers_definition.md). diff --git a/docs/reference/tasks/validator_task.md b/docs/reference/tasks/validator_task.md new file mode 100644 index 00000000..c3a43a1a --- /dev/null +++ b/docs/reference/tasks/validator_task.md @@ -0,0 +1,77 @@ +ValidatorTask +============= + +Validates the input with the Symfony Validator component and outputs it unchanged when it is valid. + +Task reference +-------------- + +* **Service**: `CleverAge\ProcessBundle\Task\Validation\ValidatorTask` + +Accepted inputs +--------------- + +`mixed`: any value to validate (object, array, scalar). + +Possible outputs +---------------- + +`mixed`: the input, unchanged, when there is no violation. + +When violations are found, each one is logged (if `log_errors` is enabled) with the `property`, `violation_code` and +`invalid_value` in the log context, then: + +* if `error_output_violations` is `true`: the `ConstraintViolationListInterface` is sent to the error output and the + task is skipped (nothing is sent to the outputs) +* otherwise: an `\UnexpectedValueException` is thrown (`N constraint violations detected on validation`), handled + according to the task `error_strategy` + +Options +------- + +| Code | Type | Required | Default | Description | +|---------------------------|----------------|:--------:|------------|------------------------------------------------------------------------------------------------------------------------------------------| +| `log_errors` | `string\|bool` | | `critical` | PSR log level (`Psr\Log\LogLevel` values) used to log each violation. `true` means `critical`, `false` disables logging | +| `groups` | `array\|null` | | `null` | Validation groups, passed to `ValidatorInterface::validate()` | +| `constraints` | `array\|null` | | `null` | Constraints to validate against, using the same syntax as Symfony's YAML validation mapping. If `null`, the input class metadata is used | +| `error_output_violations` | `bool` | | `false` | Send the violations to the error output and skip the task instead of throwing an exception | + +Constraints are built by `CleverAge\ProcessBundle\Validator\ConstraintLoader`: each constraint is a single-key map +`ConstraintName: options`. The name is either a short name of a Symfony built-in constraint (`NotBlank`, `Collection`, +...) or a FQCN. Options can contain nested constraints. + +Examples +-------- + +* Validate an entity with its own metadata (attributes, YAML mapping...) for the `import` group + +```yaml +# Task configuration level +validate: + service: '@CleverAge\ProcessBundle\Task\Validation\ValidatorTask' + options: + groups: [import] + outputs: [save] +``` + +* Validate an array with inline constraints and forward violations to an error branch + +```yaml +# Task configuration level +validate: + service: '@CleverAge\ProcessBundle\Task\Validation\ValidatorTask' + options: + log_errors: warning + error_output_violations: true + constraints: + - Collection: + allowExtraFields: true + fields: + sku: + - NotBlank: ~ + price: + - Type: numeric + - PositiveOrZero: ~ + outputs: [save] + error_outputs: [log_violations] +``` diff --git a/docs/reference/tasks/xml_reader_task.md b/docs/reference/tasks/xml_reader_task.md index 4383f491..37b96901 100644 --- a/docs/reference/tasks/xml_reader_task.md +++ b/docs/reference/tasks/xml_reader_task.md @@ -1,8 +1,8 @@ XmlReaderTask ============= -Open and read an XML file. -Requires `php-xml`. +Reads an XML file, whose path is set in the options, and outputs it as a `\DOMDocument`. +Requires the `dom` PHP extension. Task reference -------------- @@ -12,28 +12,37 @@ Task reference Accepted inputs --------------- -No input accepted. +Input is ignored (a warning is logged if an input is given). Possible outputs ---------------- -A `\DOMDocument` built from the file. +`\DOMDocument`: document loaded from the file content with +[DOMDocument::loadXML](https://www.php.net/manual/en/domdocument.loadxml.php). Options ------- -| Code | Type | Required | Default | Description | -| ---- | ---- | :------: | ------- | ----------- | -| `file_path` | `string` | **X** | | Path of the file to read from (relative to symfony root or absolute) | -| `mode` | `string` | | `rb` | File open mode (see [fopen mode parameter](https://secure.php.net/manual/en/function.fopen.php)) | +| Code | Type | Required | Default | Description | +|-------------|----------|:--------:|---------|-----------------------------------------------------------------------------------------------| +| `file_path` | `string` | **X** | | Path of the file to read | +| `mode` | `string` | | `rb` | File open mode (see [fopen mode parameter](https://www.php.net/manual/en/function.fopen.php)) | Examples -------- ```yaml # Task configuration level -my_xml_reader: - service: '@CleverAge\ProcessBundle\Task\File\Xml\XmlReaderTask' - options: - file_path: '%kernel.project_dir%/var/data/file.xml' +read_xml: + service: '@CleverAge\ProcessBundle\Task\File\Xml\XmlReaderTask' + options: + file_path: '%kernel.project_dir%/var/data/file.xml' + outputs: [transform] ``` + +Notes +----- + +* The result of `loadXML()` is not checked: an invalid XML content raises libxml warnings (which may be converted to + exceptions by the Symfony error handler) and produces an empty or partial document. +* An empty file raises a `\ValueError` (the file content is read with `fread()` using the file size as length). diff --git a/docs/reference/tasks/xml_writer_task.md b/docs/reference/tasks/xml_writer_task.md index e8678317..33e8fc28 100644 --- a/docs/reference/tasks/xml_writer_task.md +++ b/docs/reference/tasks/xml_writer_task.md @@ -1,8 +1,8 @@ XmlWriterTask ============= -Open and write an XML file. -Requires `php-xml`. +Writes the `\DOMDocument` received as input to an XML file, whose path is set in the options. +Requires the `dom` PHP extension. Task reference -------------- @@ -12,28 +12,34 @@ Task reference Accepted inputs --------------- -A `\DOMDocument` to dump into the file. +`\DOMDocument`: document to dump with [DOMDocument::saveXML](https://www.php.net/manual/en/domdocument.savexml.php). +An `\UnexpectedValueException` is thrown for any other type. Possible outputs ---------------- -Resulting file path. +`string`: path of the written file (value of the `file_path` option). Options ------- -| Code | Type | Required | Default | Description | -| ---- | ---- | :------: | ------- | ----------- | -| `file_path` | `string` | **X** | | Path of the file to write into (relative to symfony root or absolute) | -| `mode` | `string` | | `rb` | File open mode (see [fopen mode parameter](https://secure.php.net/manual/en/function.fopen.php)) | +| Code | Type | Required | Default | Description | +|-------------|----------|:--------:|---------|-----------------------------------------------------------------------------------------------| +| `file_path` | `string` | **X** | | Path of the file to write to | +| `mode` | `string` | | `wb` | File open mode (see [fopen mode parameter](https://www.php.net/manual/en/function.fopen.php)) | Examples -------- ```yaml # Task configuration level -my_xml_reader: - service: '@CleverAge\ProcessBundle\Task\File\Xml\XmlWriterTask' - options: - file_path: '%kernel.project_dir%/var/data/file.xml' +write_xml: + service: '@CleverAge\ProcessBundle\Task\File\Xml\XmlWriterTask' + options: + file_path: '%kernel.project_dir%/var/data/file.xml' ``` + +Notes +----- + +* The file is opened on each execution: with the default `wb` mode, each input overwrites the file. diff --git a/docs/reference/tasks/yaml_reader_task.md b/docs/reference/tasks/yaml_reader_task.md new file mode 100644 index 00000000..cab89566 --- /dev/null +++ b/docs/reference/tasks/yaml_reader_task.md @@ -0,0 +1,47 @@ +YamlReaderTask +============== + +Parses a YAML file, whose path is set in the options, and iterates over its root elements. + +Task reference +-------------- + +* **Service**: `CleverAge\ProcessBundle\Task\File\Yaml\YamlReaderTask` +* **Iterable task** + +Accepted inputs +--------------- + +Input is ignored + +Possible outputs +---------------- + +`mixed`: the value of each root element of the parsed file, one per iteration (keys are not output). +Underlying method is Symfony `Yaml::parseFile()`. + +Options +------- + +| Code | Type | Required | Default | Description | +|-------------|----------|:--------:|---------|----------------------------------------------------------------------------------------------------------------------| +| `file_path` | `string` | **X** | | Path of the YAML file to read. An `\UnexpectedValueException` is thrown at initialization if the file does not exist | + +Examples +-------- + +```yaml +# Task configuration level +read_yaml: + service: '@CleverAge\ProcessBundle\Task\File\Yaml\YamlReaderTask' + options: + file_path: '%kernel.project_dir%/var/data/data.yaml' + outputs: [process_item] +``` + +Notes +----- + +* The root of the file must be a mapping or a sequence, otherwise an `\InvalidArgumentException` is thrown (e.g. for an + empty file). An empty root mapping or sequence (`{}` or `[]`) raises a `\TypeError`. +* The current root key is added to the error context of the process as `iterator_key`. diff --git a/docs/reference/tasks/yaml_writer_task.md b/docs/reference/tasks/yaml_writer_task.md new file mode 100644 index 00000000..c3e6f122 --- /dev/null +++ b/docs/reference/tasks/yaml_writer_task.md @@ -0,0 +1,45 @@ +YamlWriterTask +============== + +Dumps the input as YAML to a file, whose path is set in the options. The file is overwritten on each execution. + +Task reference +-------------- + +* **Service**: `CleverAge\ProcessBundle\Task\File\Yaml\YamlWriterTask` + +Accepted inputs +--------------- + +`mixed`: data to dump, usually an `array`. Underlying method is Symfony `Yaml::dump()`. + +Possible outputs +---------------- + +`string`: path of the written file (value of the `file_path` option). + +Options +------- + +| Code | Type | Required | Default | Description | +|-------------|----------|:--------:|---------|------------------------------------------------------------------------| +| `file_path` | `string` | **X** | | Path of the file to write to. Its parent directory must exist | +| `inline` | `int` | | `4` | Level at which the dumper switches to inline YAML (see `Yaml::dump()`) | + +Examples +-------- + +```yaml +# Task configuration level +write_yaml: + service: '@CleverAge\ProcessBundle\Task\File\Yaml\YamlWriterTask' + options: + file_path: '%kernel.project_dir%/var/data/export.yaml' + inline: 3 +``` + +Notes +----- + +* This task is not blocking: when receiving several inputs, each one overwrites the file. Aggregate the data first + (e.g. with [AggregateIterableTask](aggregate_iterable_task.md)) if needed. diff --git a/docs/reference/traits/condition_trait.md b/docs/reference/traits/condition_trait.md index 3a88903d..ec2bfd09 100644 --- a/docs/reference/traits/condition_trait.md +++ b/docs/reference/traits/condition_trait.md @@ -1,17 +1,88 @@ ConditionTrait ============== -Provide generic matching rules +Provide a configurable set of matching rules, used by tasks and transformers to test an input (e.g. to filter it). ## Reference * Namespace: `CleverAge\ProcessBundle\Transformer\ConditionTrait` -* Options algorithm: _TODO_ +* Options algorithm: + - each condition option is a list of `property path => expected value` (or just `property path` for `empty` and + `not_empty`, the value being ignored) + - the property path is read on the input with the Symfony + [PropertyAccessor](https://symfony.com/doc/current/components/property_access.html): use `[key]` for arrays, + `property` or `a.b` for objects. An unreadable path gives `null`, without error. The empty path `''` targets the + whole input + - all conditions must match (logical **AND**, there is no **OR**): the checks are run in the order `match`, `empty`, + `match_regexp`, `not_match`, `not_empty`, `not_match_regexp`, and stop at the first failure + - with no condition at all, the input always matches + +## Options + +| Code | Type | Required | Default | Description | +|--------------------|---------|:--------:|---------|---------------------------------------------------------------------------------------------------------------------| +| `match` | `array` | | `[]` | `path => value`: the value at `path` must be strictly equal (`===`) to `value` | +| `not_match` | `array` | | `[]` | `path => value`: the value at `path` must not be strictly equal (`!==`) to `value` | +| `empty` | `array` | | `[]` | `path => ~`: the value at `path` must be empty (PHP `empty()`: `null`, `''`, `'0'`, `0`, `false`, `[]`, or missing) | +| `not_empty` | `array` | | `[]` | `path => ~`: the value at `path` must not be empty | +| `match_regexp` | `array` | | `[]` | `path => pattern`: the value at `path`, cast to string, must match the regular expression | +| `not_match_regexp` | `array` | | `[]` | `path => pattern`: the value at `path`, cast to string, must not match the regular expression | + +An invalid regular expression makes both `match_regexp` and `not_match_regexp` fail. ## Usage -_TODO_ +* Call `ConditionTrait::configureConditionOptions` to add the condition options at the root of your `OptionsResolver`, + or `ConditionTrait::configureWrappedConditionOptions` to add them under a single option (e.g. `condition`) +* Set the `$accessor` property with a `PropertyAccessorInterface` (e.g. in the constructor) +* Call `ConditionTrait::checkCondition` with the input and the resolved conditions; it returns `true` if all + conditions match + +The input must be an `array` or an `object` as soon as a condition is defined (a scalar input raises a `TypeError`). + +## Examples + +* Strict equality: the `[status]` key must be `active` (string) and `[stock]` must not be `0` (integer) + +```yaml +match: + '[status]': active +not_match: + '[stock]': 0 +``` + +* Emptiness: `[email]` must be filled and `[deleted_at]` must be empty or missing + +```yaml +not_empty: + '[email]': ~ +empty: + '[deleted_at]': ~ +``` + +* Regular expressions, on an object input + +```yaml +match_regexp: + sku: '/^[A-Z]{3}-\d+$/' +not_match_regexp: + customer.email: '/@example\.com$/' +``` + +* Wrapped in a `condition` option (e.g. [ArrayFilterTransformer](../transformers/array_filter_transformer.md)) + +```yaml +# Transformer options level +array_filter: + condition: + match: + '[type]': product +``` ## Implementors -_TODO_ +* [FilterTask](../tasks/filter_task.md): condition options at the root of the task options +* [ColumnAggregatorTask](../tasks/column_aggregator_task.md): `condition` option, checked against + `{input_column_value: , input: }` (so use paths like `[input_column_value]` or `[input][key]`) +* [ArrayFilterTransformer](../transformers/array_filter_transformer.md): `condition` option, checked against each item +* [UnsetTransformer](../transformers/unset_transformer.md): `condition` option, checked against the input array diff --git a/docs/reference/traits/transformer_trait.md b/docs/reference/traits/transformer_trait.md index ad07adc0..2bb02ebc 100644 --- a/docs/reference/traits/transformer_trait.md +++ b/docs/reference/traits/transformer_trait.md @@ -1,30 +1,46 @@ TransformerTrait ================ -Allow to hold a list of sub-transformers and recursively configure their options. +Allow to hold a list of sub-transformers, configure their options at initialization, and apply them in sequence. ## Reference * Namespace: `CleverAge\ProcessBundle\Transformer\TransformerTrait` -* Options algorithm: - - for each element: the key maps to a transformer code, and the value are resolved by the matching transformer - - note that the key may be followed by `#` and any digit, to allow multiple transformer of the same type. Example: +* Options algorithm: + - the option (`transformers` by default) is an ordered list of `transformer code => transformer options` + - options must be an `array` or `null` (`~`); anything else throws an `InvalidArgumentException` + - options are resolved once, when the parent options are resolved, with the `configureOptions` method of the + matching transformer; a transformer that is not configurable must not receive options + - an unknown transformer code throws a `MissingTransformerException` + - at runtime, transformers are applied in the declared order, each one receiving the output of the previous one + - any error thrown by a transformer is wrapped in a `TransformerException` ("Transformation '' have failed: + "), the original exception being available as previous exception + - as YAML keys must be unique, a suffix starting with `#` can be added to the code to use the same transformer + several times. The convention is `#` followed by digits; the part before the first `#` is used as the transformer + code if it is registered. Example: + ```yaml transformers: - transformer_code#1: - some_options: ~ - transformer_code#2: - some_options: ~ + callback#1: + callback: array_filter + callback#2: + callback: array_reverse ``` ## Usage -* Call `TransformerTrait::configureTransformersOptions` with your own `OptionResolver`. You can change `$optionName` if you want a custom option name -* Call `TransformerTrait::applyTransformers` with the resolved transformer options (i.e. `$options["transformers"]`) and the value you want to pass +* Set the `$transformerRegistry` property with the `TransformerRegistry` service (e.g. in the constructor) +* Call `TransformerTrait::configureTransformersOptions` with your own `OptionsResolver`. You can change `$optionName` + if you want a custom option name +* Call `TransformerTrait::applyTransformers` with the resolved transformer option (i.e. `$options['transformers']`) + and the value you want to transform ## Implementors -* [TransformerTask](../tasks/transformer_task.md) -* [MappingTransformer](../transformers/mapping_transformer.md) -* [RulesTransformer](../transformers/rules_transformer.md) -* [Generic transformers](../03-generic_transformers_definition.md) +* [TransformerTask](../tasks/transformer_task.md): `transformers` option +* [ArrayMapTransformer](../transformers/array_map_transformer.md): `transformers` option, applied on each item +* [CachedTransformer](../transformers/cached_transformer.md): `transformers` and `key_transformers` options +* [MappingTransformer](../transformers/mapping_transformer.md): `transformers` option of each mapped property +* [RulesTransformer](../transformers/rules_transformer.md): `transformers` option of each rule +* [GenericTransformer](../transformers/generic_transformer.md), see + [Generic transformers definition](../03-generic_transformers_definition.md) diff --git a/docs/reference/transformers/_template.md b/docs/reference/transformers/_template.md index d5e98a70..74c4b578 100644 --- a/docs/reference/transformers/_template.md +++ b/docs/reference/transformers/_template.md @@ -1,43 +1,42 @@ TransformerName =============== -_Describe main goal an use cases of the transformer_ +_Describe the main goal and use cases of the transformer._ Transformer reference --------------------- -* **Service**: `ClassName` +* **Service**: `Fully\Qualified\ClassName` * **Transformer code**: `code` Accepted inputs --------------- -_Description of allowed types_ +_Description of allowed types._ Possible outputs ---------------- -_Description of possible types_ +_Description of possible types._ Options ------- -| Code | Type | Required | Default | Description | -| ---- | ---- | :------: | ------- | ----------- | -| `code` | `type` | **X** _or nothing_ | `default value` _if available_ | _description_ | +| Code | Type | Required | Default | Description | +|--------|--------|:--------:|-----------------|---------------| +| `code` | `type` | **X** | `default value` | _description_ | + +_If the transformer has no option, replace the table with "This transformer has no option."._ Examples -------- -_YAML samples and explanations_ - * Example 1 - details - - details - + ```yaml # Transformer options level code: - option1: a - option2: b + option1: a + option2: b ``` diff --git a/docs/reference/transformers/array_element_transformer.md b/docs/reference/transformers/array_element_transformer.md new file mode 100644 index 00000000..f9950b1f --- /dev/null +++ b/docs/reference/transformers/array_element_transformer.md @@ -0,0 +1,52 @@ +ArrayElementTransformer +======================= + +Return the element at a given position of an array, using `array_slice()`. The position is based on the order of the +elements, not on their keys, so it also works with associative arrays. Negative indexes are counted from the end. + +Transformer reference +--------------------- + +* **Service**: `CleverAge\ProcessBundle\Transformer\Array\ArrayElementTransformer` +* **Transformer code**: `array_element` + +Accepted inputs +--------------- + +`array` + +Possible outputs +---------------- + +`any`: the element found at the given position + +Options +------- + +| Code | Type | Required | Default | Description | +|---------|-------|:--------:|---------|-----------------------------------------------------------------------------| +| `index` | `int` | **X** | | Position of the element (0-based, a negative value starts from the end) | + +Examples +-------- + +* Get the 2nd element: `['foo', 'bar', 'baz']` becomes `'bar'` + +```yaml +# Transformer options level +array_element: + index: 1 +``` + +* Get the penultimate element: `['foo', 'bar', 'baz']` becomes `'bar'` + +```yaml +# Transformer options level +array_element: + index: -2 +``` + +Notes +----- + +There is no bound check: an index outside the array triggers an "Undefined array key" warning. diff --git a/docs/reference/transformers/array_filter_transformer.md b/docs/reference/transformers/array_filter_transformer.md index f91ad117..13746eef 100644 --- a/docs/reference/transformers/array_filter_transformer.md +++ b/docs/reference/transformers/array_filter_transformer.md @@ -1,27 +1,68 @@ ArrayFilterTransformer ====================== -Filter data from an iterable value. Should match mostly native [array_filter](https://secure.php.net/manual/fr/function.array-filter.php) function behavior, as such array keys are preserved. +Filter the elements of an iterable value, keeping only those matching a set of conditions. It mimics the native +[array_filter](https://www.php.net/manual/en/function.array-filter.php) function behavior: keys are preserved. -Task reference --------------- +Transformer reference +--------------------- -* **Service**: `CleverAge\ProcessBundle\Transformer\ArrayFilterTransformer` +* **Service**: `CleverAge\ProcessBundle\Transformer\Array\ArrayFilterTransformer` * **Transformer code**: `array_filter` Accepted inputs --------------- -`array` or `\Iterable` +`iterable` (`array` or `\Traversable`). Any other value throws an `\UnexpectedValueException`. Possible outputs ---------------- -`array` containing only filtered data +`array`: the elements matching the condition, with their original keys Options ------- -| Code | Type | Required | Default | Description | -| ---- | ---- | :------: | ------- | ----------- | -| `condition` | `array` | | `[]` | See [ConditionTrait](../traits/condition_trait.md) | +| Code | Type | Required | Default | Description | +|-------------|---------|:--------:|---------|-----------------------------------------------------------------------------------------------| +| `condition` | `array` | | `[]` | Conditions each element must match, see [ConditionTrait](../traits/condition_trait.md) | + +The `condition` option accepts the following keys, each one being a map of `property_path: value`. Properties are read +from each element with the Symfony PropertyAccessor (an empty path `''` targets the element itself); an unreadable +property is considered `null`. + +| Code | Type | Required | Default | Description | +|--------------------|---------|:--------:|---------|--------------------------------------------------------------------| +| `match` | `array` | | `[]` | Property must be strictly equal (`===`) to the value | +| `not_match` | `array` | | `[]` | Property must not be strictly equal (`!==`) to the value | +| `empty` | `array` | | `[]` | Property must be empty (`empty()`), the value is ignored | +| `not_empty` | `array` | | `[]` | Property must not be empty (`empty()`), the value is ignored | +| `match_regexp` | `array` | | `[]` | Property must match the regular expression given as value | +| `not_match_regexp` | `array` | | `[]` | Property must not match the regular expression given as value | + +With an empty `condition`, every element is kept. + +Examples +-------- + +* Keep only European countries from a list of objects + +```yaml +# Transformer options level +array_filter: + condition: + match: + sContinentCode: 'EU' +``` + +* Keep only array elements with a non-empty `[email]` and a `[sku]` starting with `A` + +```yaml +# Transformer options level +array_filter: + condition: + not_empty: + '[email]': ~ + match_regexp: + '[sku]': '/^A/' +``` diff --git a/docs/reference/transformers/array_first_transformer.md b/docs/reference/transformers/array_first_transformer.md new file mode 100644 index 00000000..949a1dd9 --- /dev/null +++ b/docs/reference/transformers/array_first_transformer.md @@ -0,0 +1,46 @@ +ArrayFirstTransformer +===================== + +Return the first element of an array, using `reset()`. + +Transformer reference +--------------------- + +* **Service**: `CleverAge\ProcessBundle\Transformer\Array\ArrayFirstTransformer` +* **Transformer code**: `array_first` + +Accepted inputs +--------------- + +`array`. With the default options, any non-iterable value is accepted and returned unchanged. + +Possible outputs +---------------- + +* `any`: the first element of the array +* `false` if the array is empty +* the input value itself if it is not iterable and `allow_not_iterable` is `false` + +Options +------- + +| Code | Type | Required | Default | Description | +|----------------------|--------|:--------:|---------|----------------------------------------------------------------------------------| +| `allow_not_iterable` | `bool` | | `false` | When `false`, a non-iterable input is returned unchanged (see [Notes](#notes)) | + +Examples +-------- + +* `['foo', 'bar', 'baz']` becomes `'foo'` + +```yaml +# Transformer options level +array_first: ~ +``` + +Notes +----- + +The `allow_not_iterable` option behaves counter-intuitively: when set to `true`, the non-iterable check is skipped and +`reset()` is called on the raw value, which throws a `\TypeError` for scalar or `null` inputs (objects are accepted by +`reset()`, which then returns their first public property). Keep the default value. diff --git a/docs/reference/transformers/array_last_transformer.md b/docs/reference/transformers/array_last_transformer.md new file mode 100644 index 00000000..af7cc71d --- /dev/null +++ b/docs/reference/transformers/array_last_transformer.md @@ -0,0 +1,40 @@ +ArrayLastTransformer +==================== + +Return the last element of an array, using `array_slice()`. + +Transformer reference +--------------------- + +* **Service**: `CleverAge\ProcessBundle\Transformer\Array\ArrayLastTransformer` +* **Transformer code**: `array_last` + +Accepted inputs +--------------- + +`array` + +Possible outputs +---------------- + +`any`: the last element of the array + +Options +------- + +This transformer has no option. + +Examples +-------- + +* `['foo', 'bar', 'baz']` becomes `'baz'` + +```yaml +# Transformer options level +array_last: ~ +``` + +Notes +----- + +An empty array triggers an "Undefined array key" warning. diff --git a/docs/reference/transformers/array_map_transformer.md b/docs/reference/transformers/array_map_transformer.md index 32f9bd01..a907d7dd 100644 --- a/docs/reference/transformers/array_map_transformer.md +++ b/docs/reference/transformers/array_map_transformer.md @@ -1,10 +1,10 @@ ArrayMapTransformer -========================= +=================== -Applies transformers to each element of an array. +Apply a chain of transformers to each element of an iterable value. Keys are preserved. -Task reference --------------- +Transformer reference +--------------------- * **Service**: `CleverAge\ProcessBundle\Transformer\Array\ArrayMapTransformer` * **Transformer code**: `array_map` @@ -12,25 +12,28 @@ Task reference Accepted inputs --------------- -`array` +`array` or `\Traversable`. Any other value throws an `\UnexpectedValueException`. Possible outputs ---------------- -`string` +`array`: the transformed elements, with their original keys Options ------- -| Code | Type | Required | Default | Description | -|----------------|---------|:--------:|---------|------------------------------------------------------------------------------| -| `transformers` | `array` | **X** | | List of transformers, see [TransformerTrait](../traits/transformer_trait.md) | -| `skip_null` | `bool` | | `false` | If true continue without applying other transformers on null values | +| Code | Type | Required | Default | Description | +|----------------|---------|:--------:|---------|-----------------------------------------------------------------------------------------------| +| `transformers` | `array` | **X** | | Transformers applied to each element, see [TransformerTrait](../traits/transformer_trait.md) | +| `skip_null` | `bool` | | `false` | If `true`, elements whose transformed value is `null` are removed from the result | +When a sub-transformer fails, the thrown `TransformerException` references the key of the failing element. Examples -------- +* Cast each element to string, then uppercase it + ```yaml # Transformer mapping level array_map: @@ -43,5 +46,22 @@ array_map: transformers: cast: type: 'string' - uppercase: ~ + callback: + callback: strtoupper +``` + +* Convert each `stdClass` item into an array and remap it + +```yaml +# Transformer options level +array_map: + transformers: + cast: + type: 'array' + mapping: + mapping: + isoCode: + code: '[sISOCode]' + name: + code: '[sName]' ``` diff --git a/docs/reference/transformers/array_unset_transformer.md b/docs/reference/transformers/array_unset_transformer.md new file mode 100644 index 00000000..23f72f53 --- /dev/null +++ b/docs/reference/transformers/array_unset_transformer.md @@ -0,0 +1,38 @@ +ArrayUnsetTransformer +===================== + +Remove a key from an array. + +Transformer reference +--------------------- + +* **Service**: `CleverAge\ProcessBundle\Transformer\Array\ArrayUnsetTransformer` +* **Transformer code**: `array_unset` + +Accepted inputs +--------------- + +`array`. Any other value throws an `\UnexpectedValueException`. + +Possible outputs +---------------- + +`array`: the input array without the given key (unchanged if the key does not exist) + +Options +------- + +| Code | Type | Required | Default | Description | +|-------|---------------|:--------:|---------|--------------------| +| `key` | `string\|int` | **X** | | The key to remove | + +Examples +-------- + +* `{id: 1, temporary_field: 'foo'}` becomes `{id: 1}` + +```yaml +# Transformer options level +array_unset: + key: temporary_field +``` diff --git a/docs/reference/transformers/cached_transformer.md b/docs/reference/transformers/cached_transformer.md new file mode 100644 index 00000000..ad14de6a --- /dev/null +++ b/docs/reference/transformers/cached_transformer.md @@ -0,0 +1,60 @@ +CachedTransformer +================= + +Wrap a chain of transformers with a PSR-6 cache layer. A cache key is built from the input value; if the cache +contains an item for this key it is returned directly, otherwise the wrapped transformers are applied and the result +is stored in cache. + +Transformer reference +--------------------- + +* **Service**: `CleverAge\ProcessBundle\Transformer\CachedTransformer` +* **Transformer code**: `cached` + +Accepted inputs +--------------- + +`string`: the value is used both to build the cache key (after `key_transformers`) and as input of `transformers`. +Any other type raises a `TypeError`. + +Possible outputs +---------------- + +`any`: result of the `transformers` chain, or the cached value on cache hit. + +Options +------- + +| Code | Type | Required | Default | Description | +|--------------------|-----------------------------------|:--------:|---------|---------------------------------------------------------------------------------------------------------------------------------| +| `cache_key` | `string` | **X** | | Root of the cache key. The final key is `\|` | +| `ttl` | `string\|DateTimeInterface\|null` | | `null` | Expiration date of the cache items. A string is converted with `new \DateTime($value)` (e.g. `+1 hour`). `null` means no expiry | +| `transformers` | `array` | | `[]` | Transformers applied on cache miss, see [TransformerTrait](../traits/transformer_trait.md) | +| `key_transformers` | `array` | | `[]` | Transformers applied on the input to compute the key value, see [TransformerTrait](../traits/transformer_trait.md) | + +Examples +-------- + +* Cache a costly transformation for one hour, using a slugified input as key + +```yaml +# Transformer options level +cached: + cache_key: my_prefix + ttl: '+1 hour' + key_transformers: + slugify: ~ + transformers: + callback: + callback: md5 +``` + +Notes +----- + +* The cache pool is the autowired `Psr\Cache\CacheItemPoolInterface` service (`cache.app` in a standard Symfony + application). Items are saved with `saveDeferred()`, a warning is logged if the save fails. +* A string `ttl` is converted to a date when the options are resolved (i.e. once, when the transformer is configured), + not each time an item is saved: all items share the same absolute expiration date. +* If the key value is not a string after `key_transformers`, or if the cache pool raises a PSR-6 + `InvalidArgumentException` (logged as a warning), the transformers are applied without cache. diff --git a/docs/reference/transformers/callback_transformer.md b/docs/reference/transformers/callback_transformer.md new file mode 100644 index 00000000..e61a52ac --- /dev/null +++ b/docs/reference/transformers/callback_transformer.md @@ -0,0 +1,60 @@ +CallbackTransformer +=================== + +Call a PHP callable (function or static method) with the input value. The value is inserted between configurable left +and right parameters: `callback(...left_parameters, $value, ...right_parameters)`. + +Transformer reference +--------------------- + +* **Service**: `CleverAge\ProcessBundle\Transformer\CallbackTransformer` +* **Transformer code**: `callback` + +Accepted inputs +--------------- + +`any`: whatever the callback accepts. + +Possible outputs +---------------- + +`any`: the return value of the callback. + +Options +------- + +| Code | Type | Required | Default | Description | +|-------------------------|-----------------|:--------:|---------|-----------------------------------------------------------------------------------------------------------| +| `callback` | `string\|array` | **X** | | A valid PHP callable (e.g. `strtoupper`, `['App\MyClass', 'myStaticMethod']`), checked with `is_callable` | +| `left_parameters` | `array` | | `[]` | Parameters passed before the value | +| `right_parameters` | `array` | | `[]` | Parameters passed after the value | +| `additional_parameters` | `array` | | `[]` | **Deprecated**: use `right_parameters` instead. Only used when `right_parameters` is empty | + +Examples +-------- + +* Simple PHP function + +```yaml +# Transformer options level +callback: + callback: strtoupper +``` + +* Function with extra parameters: `json_decode($value, true)` + +```yaml +# Transformer options level +callback: + callback: json_decode + right_parameters: [true] +``` + +* Value in second position: `explode(',', $value)` + +```yaml +# Transformer options level +callback: + callback: explode + left_parameters: [','] +``` diff --git a/docs/reference/transformers/cast_transformer.md b/docs/reference/transformers/cast_transformer.md new file mode 100644 index 00000000..0fb60c4f --- /dev/null +++ b/docs/reference/transformers/cast_transformer.md @@ -0,0 +1,52 @@ +CastTransformer +=============== + +Cast the input value to another PHP type using [`settype()`](https://www.php.net/manual/en/function.settype.php). + +Transformer reference +--------------------- + +* **Service**: `CleverAge\ProcessBundle\Transformer\CastTransformer` +* **Transformer code**: `cast` + +Accepted inputs +--------------- + +`any` + +Possible outputs +---------------- + +`any`: the input value converted to the configured type. + +Options +------- + +| Code | Type | Required | Default | Description | +|--------|----------|:--------:|---------|----------------------------------------------------------------------------------------------------------------------------------------------| +| `type` | `string` | **X** | | Target type, any value accepted by `settype()` (`bool`, `boolean`, `int`, `integer`, `float`, `double`, `string`, `array`, `object`, `null`) | + +Examples +-------- + +* Cast a string to an integer + +```yaml +# Transformer options level +cast: + type: int +``` + +* Convert a `stdClass` (e.g. a SOAP response item) to an array + +```yaml +# Transformer options level +cast: + type: array +``` + +Notes +----- + +The `type` value is not validated at configuration time: an invalid type throws a `ValueError` on transformation. See +[TypeSetterTransformer](type_setter_transformer.md) for a variant validating the type upfront. diff --git a/docs/reference/transformers/constant_transformer.md b/docs/reference/transformers/constant_transformer.md new file mode 100644 index 00000000..9f20e3bf --- /dev/null +++ b/docs/reference/transformers/constant_transformer.md @@ -0,0 +1,38 @@ +ConstantTransformer +=================== + +Always return the configured value, whatever the input. Inside a [MappingTransformer](mapping_transformer.md), the +`constant` property option usually does the same job; this transformer is useful elsewhere (e.g. in a +[TransformerTask](../tasks/transformer_task.md) or in a transformer chain). + +Transformer reference +--------------------- + +* **Service**: `CleverAge\ProcessBundle\Transformer\ConstantTransformer` +* **Transformer code**: `constant` + +Accepted inputs +--------------- + +`any`: the input is ignored. + +Possible outputs +---------------- + +`any`: the configured constant. + +Options +------- + +| Code | Type | Required | Default | Description | +|------------|-------|:--------:|---------|---------------------| +| `constant` | `any` | **X** | | The value to return | + +Examples +-------- + +```yaml +# Transformer options level +constant: + constant: default_value +``` diff --git a/docs/reference/transformers/convert_value_transformer.md b/docs/reference/transformers/convert_value_transformer.md new file mode 100644 index 00000000..eeb6711a --- /dev/null +++ b/docs/reference/transformers/convert_value_transformer.md @@ -0,0 +1,58 @@ +ConvertValueTransformer +======================= + +Convert a value into another one using a conversion map (lookup table). + +Transformer reference +--------------------- + +* **Service**: `CleverAge\ProcessBundle\Transformer\ConvertValueTransformer` +* **Transformer code**: `convert_value` + +Accepted inputs +--------------- + +* `string` or `int`: used as a key of `map` +* `null`: always returns `null`, without lookup +* other scalars or objects: only if `auto_cast` is `true` (cast to string), otherwise an `UnexpectedValueException` is + thrown. Arrays are always rejected. + +Possible outputs +---------------- + +`any`: the value matching the input in `map`, the input itself (`keep_missing`) or `null` (`ignore_missing`). + +Options +------- + +| Code | Type | Required | Default | Description | +|------------------|---------|:--------:|---------|------------------------------------------------------------------------------------------------------------| +| `map` | `array` | **X** | | Conversion table: input value as key, output value as value | +| `ignore_missing` | `bool` | | `false` | If `true`, return `null` when the value is not in `map`, instead of throwing an `UnexpectedValueException` | +| `keep_missing` | `bool` | | `false` | If `true`, return the input when the value is not in `map` (takes precedence on `ignore_missing`) | +| `auto_cast` | `bool` | | `false` | If `true`, cast input values that are not valid array keys (`float`, `bool`, `Stringable`...) to string | + +Examples +-------- + +* Simple conversion, unknown values become `null` + +```yaml +# Transformer options level +convert_value: + map: + TEXTE: text + NUMERIQUE: number + DATE: date + ignore_missing: true +``` + +* Keep the original value when it is not in the map + +```yaml +# Transformer options level +convert_value: + map: + old_code: new_code + keep_missing: true +``` diff --git a/docs/reference/transformers/date_format.md b/docs/reference/transformers/date_format.md deleted file mode 100644 index e19e2590..00000000 --- a/docs/reference/transformers/date_format.md +++ /dev/null @@ -1,39 +0,0 @@ -DateFormatTransformer -===================== - -Transforms a `\DateTime` into a formatted `string`. It will throw an error if the date cannot be parsed. - -Task reference --------------- - -* **Service**: `CleverAge\ProcessBundle\Transformer\DateFormatTransformer` -* **Transformer code**: `date_format` - -Accepted inputs ---------------- - -`\DateTime` - -Possible outputs ----------------- - -`string` - -Options -------- - -| Code | Type | Required | Default | Description | -| ---- | ---- | :------: | ------- | ----------- | -| `format` | `string` | **X** | | See [PHP date formats](https://www.php.net/manual/fr/function.date.php) for supported values | - -Examples --------- - -* Example : this will output a string like "2019-12-02" - -```yaml -# Transformer options level -transformers: - date_format: - format: Y-m-d -``` diff --git a/docs/reference/transformers/date_format_transformer.md b/docs/reference/transformers/date_format_transformer.md new file mode 100644 index 00000000..9380f6e3 --- /dev/null +++ b/docs/reference/transformers/date_format_transformer.md @@ -0,0 +1,56 @@ +DateFormatTransformer +===================== + +Format a date object into a `string`, using `\DateTimeInterface::format()`. + +Transformer reference +--------------------- + +* **Service**: `CleverAge\ProcessBundle\Transformer\Date\DateFormatTransformer` +* **Transformer code**: `date_format` + +Accepted inputs +--------------- + +* `\DateTimeInterface` (`\DateTime` or `\DateTimeImmutable`) +* Any falsy value (`null`, `''`, `false`…), returned unchanged + +Any other value (including a date string) throws an `\UnexpectedValueException`. Use +[DateParserTransformer](date_parser_transformer.md) first to convert a string into a date. + +Possible outputs +---------------- + +* `string`: the formatted date +* the input value itself if it is falsy + +Options +------- + +| Code | Type | Required | Default | Description | +|----------|----------|:--------:|---------|------------------------------------------------------------------------------------------------------| +| `format` | `string` | **X** | | Output format, see [PHP date formats](https://www.php.net/manual/en/datetime.format.php) | + +Examples +-------- + +* Output a string like `2019-12-02` + +```yaml +# Transformer options level +date_format: + format: Y-m-d +``` + +* Parse a French date and convert it to ISO 8601 + +```yaml +# Transformer mapping level +created_at: + code: '[date]' + transformers: + date_parser: + format: d/m/Y + date_format: + format: 'Y-m-d\TH:i:sP' +``` diff --git a/docs/reference/transformers/date_parser.md b/docs/reference/transformers/date_parser.md deleted file mode 100644 index aa4e796b..00000000 --- a/docs/reference/transformers/date_parser.md +++ /dev/null @@ -1,39 +0,0 @@ -DateParserTransformer -===================== - -Read a `string` to deduce the matching `\DateTime`. It will throw an error if the date cannot be read. - -Task reference --------------- - -* **Service**: `CleverAge\ProcessBundle\Transformer\DateParserTransformer` -* **Transformer code**: `date_parser` - -Accepted inputs ---------------- - -`string` - -Possible outputs ----------------- - -`\DateTime` - -Options -------- - -| Code | Type | Required | Default | Description | -| ---- | ---- | :------: | ------- | ----------- | -| `format` | `string` | **X** | | See [PHP date formats](https://www.php.net/manual/fr/function.date.php) for supported values | - -Examples --------- - -* Example : this will correctly read the string "2019-12-02" - -```yaml -# Transformer options level -transformers: - date_parser: - format: Y-m-d -``` diff --git a/docs/reference/transformers/date_parser_transformer.md b/docs/reference/transformers/date_parser_transformer.md new file mode 100644 index 00000000..fd584bb6 --- /dev/null +++ b/docs/reference/transformers/date_parser_transformer.md @@ -0,0 +1,59 @@ +DateParserTransformer +===================== + +Parse a string into a `\DateTime`, using `\DateTime::createFromFormat()`. + +Transformer reference +--------------------- + +* **Service**: `CleverAge\ProcessBundle\Transformer\Date\DateParserTransformer` +* **Transformer code**: `date_parser` + +Accepted inputs +--------------- + +* `string`: a date matching the configured `format` +* `\DateTime`, returned unchanged +* Any falsy value (`null`, `''`, `false`…), returned unchanged + +A string that cannot be parsed with the given format throws an `\UnexpectedValueException`. + +Possible outputs +---------------- + +* `\DateTime` +* the input value itself if it is falsy + +Options +------- + +| Code | Type | Required | Default | Description | +|----------|----------|:--------:|---------|--------------------------------------------------------------------------------------------------------------------| +| `format` | `string` | **X** | | Input format, see [DateTime::createFromFormat](https://www.php.net/manual/en/datetime.createfromformat.php) | + +Examples +-------- + +* Read the string `2019-12-02` + +```yaml +# Transformer options level +date_parser: + format: Y-m-d +``` + +* Read the string `02/12/2019 14:30` + +```yaml +# Transformer options level +date_parser: + format: 'd/m/Y H:i' +``` + +Notes +----- + +Fields missing from `format` are taken from the current time (e.g. with `Y-m-d`, the time part is the current time). Use +the `!` or `|` format characters to reset them, e.g. `'!Y-m-d'`. + +A `\DateTimeImmutable` input is not returned unchanged: it is passed to `createFromFormat()` and throws a `\TypeError`. diff --git a/docs/reference/transformers/debug_transformer.md b/docs/reference/transformers/debug_transformer.md new file mode 100644 index 00000000..cfe3654b --- /dev/null +++ b/docs/reference/transformers/debug_transformer.md @@ -0,0 +1,46 @@ +DebugTransformer +================ + +Dump the value with Symfony `VarDumper` (if the component is installed) and return it unchanged. Useful to inspect a +transformer chain. + +Transformer reference +--------------------- + +* **Service**: `CleverAge\ProcessBundle\Transformer\DebugTransformer` +* **Transformer code**: `dump` + +Accepted inputs +--------------- + +`any` + +Possible outputs +---------------- + +`any`: the input, unchanged. + +Options +------- + +This transformer has no option. + +Examples +-------- + +```yaml +# Transformer options level +dump: ~ +``` + +* Dump a value in the middle of a chain + +```yaml +# Transformer mapping level +name: + code: '[name]' + transformers: + trim: ~ + dump: ~ + slugify: ~ +``` diff --git a/docs/reference/transformers/default_transformer.md b/docs/reference/transformers/default_transformer.md new file mode 100644 index 00000000..55011a56 --- /dev/null +++ b/docs/reference/transformers/default_transformer.md @@ -0,0 +1,57 @@ +DefaultTransformer +================== + +Return a default value when the input is falsy (PHP `!$value`: `null`, `false`, `0`, `0.0`, `''`, `'0'`, `[]`), +otherwise return the input unchanged. + +Transformer reference +--------------------- + +* **Service**: `CleverAge\ProcessBundle\Transformer\DefaultTransformer` +* **Transformer code**: `default` + +Accepted inputs +--------------- + +`any` + +Possible outputs +---------------- + +`any`: the input if truthy, the configured `value` otherwise. + +Options +------- + +| Code | Type | Required | Default | Description | +|---------|-------|:--------:|---------|----------------------------------------| +| `value` | `any` | **X** | | Value returned when the input is falsy | + +Examples +-------- + +```yaml +# Transformer options level +default: + value: N/A +``` + +* Fallback after a conversion + +```yaml +# Transformer mapping level +type: + code: '[Type]' + transformers: + convert_value: + ignore_missing: true + map: + TEXTE: text + default: + value: unknown +``` + +Notes +----- + +As the check is loose, legitimate values such as `0` or `'0'` are also replaced by the default value. diff --git a/docs/reference/transformers/denormalize_transformer.md b/docs/reference/transformers/denormalize_transformer.md new file mode 100644 index 00000000..5883ff15 --- /dev/null +++ b/docs/reference/transformers/denormalize_transformer.md @@ -0,0 +1,53 @@ +DenormalizeTransformer +====================== + +Denormalize the input into an instance of the given class, using the Symfony +[Serializer](https://symfony.com/doc/current/components/serializer.html) denormalizer. + +See also [DenormalizerTask](../tasks/denormalizer_task.md) for the task equivalent. + +Transformer reference +--------------------- + +* **Service**: `CleverAge\ProcessBundle\Transformer\Serialization\DenormalizeTransformer` +* **Transformer code**: `denormalize` + +Accepted inputs +--------------- + +`any`: data supported by the configured denormalizers (usually an `array`) + +Possible outputs +---------------- + +`any`: the denormalized value, usually an instance of `class` + +Options +------- + +| Code | Type | Required | Default | Description | +|-----------|----------------|:--------:|---------|----------------------------------------------------------------------------------------------| +| `class` | `string` | **X** | | Target type: a fully qualified class name, or e.g. `App\Entity\Author[]` for a collection | +| `format` | `string\|null` | | `null` | Format passed to the denormalizer | +| `context` | `array` | | `[]` | Denormalization context (groups, `object_to_populate`…) | + +Examples +-------- + +* Denormalize an array into an `Author` entity + +```yaml +# Transformer options level +denormalize: + class: 'App\Entity\Author' +``` + +* Denormalize using serialization groups + +```yaml +# Transformer options level +denormalize: + class: 'App\Entity\Book' + context: + groups: [import] +``` diff --git a/docs/reference/transformers/evaluator_transformer.md b/docs/reference/transformers/evaluator_transformer.md new file mode 100644 index 00000000..6cd64fb5 --- /dev/null +++ b/docs/reference/transformers/evaluator_transformer.md @@ -0,0 +1,59 @@ +EvaluatorTransformer +==================== + +Evaluate a Symfony [ExpressionLanguage](https://symfony.com/doc/current/components/expression_language.html) +expression, using the input array as the expression variables. + +Transformer reference +--------------------- + +* **Service**: `CleverAge\ProcessBundle\Transformer\EvaluatorTransformer` +* **Transformer code**: `evaluator` + +Accepted inputs +--------------- + +`array`: variable name => value, injected in the expression. Any other type raises a `TypeError`. + +Possible outputs +---------------- + +`any`: the result of the expression. + +Options +------- + +| Code | Type | Required | Default | Description | +|--------------|----------------------------|:--------:|---------|--------------------------------------------------------------------------------------------------------------------------------| +| `expression` | `string\|ParsedExpression` | **X** | | The expression to evaluate | +| `variables` | `array\|null` | | `null` | List of variable names. If set, the expression is parsed once when options are resolved; if `null`, it is parsed on evaluation | + +Examples +-------- + +```yaml +# Transformer options level +evaluator: + expression: 'price * quantity' + variables: [price, quantity] +``` + +* Using a mapping to build the variables + +```yaml +# Transformer mapping level +total: + code: + price: '[unit_price]' + quantity: '[qty]' + transformers: + evaluator: + expression: 'price * quantity' +``` + +Notes +----- + +This transformer uses its own `ExpressionLanguage` instance, not the bundle's `cleverage_process.expression_language` +service: the extra functions registered on that service (e.g. `preg_match`) are not available here, unlike in +[RulesTransformer](rules_transformer.md) and [ExpressionLanguageMapTransformer](expression_language_map_transformer.md). diff --git a/docs/reference/transformers/explode_transformer.md b/docs/reference/transformers/explode_transformer.md new file mode 100644 index 00000000..926bfe89 --- /dev/null +++ b/docs/reference/transformers/explode_transformer.md @@ -0,0 +1,52 @@ +ExplodeTransformer +================== + +Split a string into an array, using PHP's [explode](https://www.php.net/manual/en/function.explode.php) function. + +Transformer reference +--------------------- + +* **Service**: `CleverAge\ProcessBundle\Transformer\String\ExplodeTransformer` +* **Transformer code**: `explode` + +Accepted inputs +--------------- + +Any value that can be cast to `string`, or `null`. + +Possible outputs +---------------- + +`array`: the list of parts; an empty array if the input is `null` or an empty string + +Options +------- + +| Code | Type | Required | Default | Description | +|-------------|----------|:--------:|---------|---------------------------------------------| +| `delimiter` | `string` | **X** | | The boundary string, must not be empty | + +Examples +-------- + +* `'a,b,c'` becomes `['a', 'b', 'c']` + +```yaml +# Transformer options level +explode: + delimiter: ',' +``` + +* Split then trim each part: `'a, b ,c'` becomes `['a', 'b', 'c']` + +```yaml +# Transformer mapping level +tags: + code: '[tags]' + transformers: + explode: + delimiter: ',' + array_map: + transformers: + trim: ~ +``` diff --git a/docs/reference/transformers/expression_language_map_transformer.md b/docs/reference/transformers/expression_language_map_transformer.md new file mode 100644 index 00000000..ab806b7f --- /dev/null +++ b/docs/reference/transformers/expression_language_map_transformer.md @@ -0,0 +1,63 @@ +ExpressionLanguageMapTransformer +================================ + +Return a value computed from the first matching rule of a list of +[ExpressionLanguage](https://symfony.com/doc/current/components/expression_language.html) `condition` / `output` +pairs. Behaves like a `switch/case` based on expressions. The input is available in expressions as the `data` +variable. + +Transformer reference +--------------------- + +* **Service**: `CleverAge\ProcessBundle\Transformer\ExpressionLanguageMapTransformer` +* **Transformer code**: `expression_language_map` + +Accepted inputs +--------------- + +`any`: exposed as `data` in expressions. + +Possible outputs +---------------- + +`any`: the evaluated `output` of the first rule whose `condition` is truthy, or the input (`keep_missing`), or `null` +(`ignore_missing`). + +Options +------- + +| Code | Type | Required | Default | Description | +|------------------|---------|:--------:|---------|--------------------------------------------------------------------------------------------------| +| `map` | `array` | **X** | | Ordered list of rules, see below | +| `ignore_missing` | `bool` | | `false` | If `true`, return `null` when no rule matches, instead of throwing an `UnexpectedValueException` | +| `keep_missing` | `bool` | | `false` | If `true`, return the input when no rule matches (takes precedence on `ignore_missing`) | + +Each entry of `map` has the following options: + +| Code | Type | Required | Default | Description | +|-------------|----------|:--------:|---------|-----------------------------------------------------------| +| `condition` | `string` | **X** | | Expression using `data`; the rule matches if it is truthy | +| `output` | `string` | **X** | | Expression using `data`, evaluated and returned on match | + +Examples +-------- + +```yaml +# Transformer options level +expression_language_map: + map: + - condition: 'data > 100' + output: '"high"' + - condition: 'data > 50' + output: '"medium"' + - condition: 'data >= 0' + output: '"low"' + ignore_missing: true +``` + +Notes +----- + +* Both `condition` and `output` are expressions: string literals must be quoted inside the expression (`'"high"'`). +* Expressions are parsed once, when options are resolved, with the bundle's `cleverage_process.expression_language` + service (which also exposes the PHP `preg_match` function). diff --git a/docs/reference/transformers/generic_transformer.md b/docs/reference/transformers/generic_transformer.md new file mode 100644 index 00000000..135967d7 --- /dev/null +++ b/docs/reference/transformers/generic_transformer.md @@ -0,0 +1,106 @@ +GenericTransformer +================== + +Class behind the transformers declared in the `clever_age_process.generic_transformers` configuration (see +[Generic transformers definition](../03-generic_transformers_definition.md)). It is not a service by itself: one +instance is created per declared generic transformer, and it simply applies a preconfigured chain of transformers, +optionally parametrized by `contextual_options`. + +For each entry of `generic_transformers`, the bundle extension (`CleverAgeProcessExtension`) registers a private +service `CleverAge\ProcessBundle\Transformer\GenericTransformer\`, tagged `cleverage.transformer`, and calls +`initialize(, )` on it. The instance is then added to the transformer registry like any other +transformer, under the configured code (which must not collide with an existing transformer code). + +When the generic transformer is used, its options are the declared contextual options. The `{{ option_code }}` +placeholders in the preconfigured `transformers` (keys and values, recursively) are replaced by the option values, +then the resulting transformer chain is resolved and applied, as with [TransformerTrait](../traits/transformer_trait.md). +A placeholder that is the whole string is replaced by the raw value (keeping its type, e.g. `int` or `null`); otherwise +the value is inserted in the string. + +Transformer reference +--------------------- + +* **Service**: `CleverAge\ProcessBundle\Transformer\GenericTransformer\` (one service per generic transformer) +* **Transformer code**: ``: the key used under `clever_age_process.generic_transformers` + +Accepted inputs +--------------- + +`any`: depends on the preconfigured transformers. + +Possible outputs +---------------- + +`any`: result of the preconfigured transformer chain. + +Options +------- + +The definition (`clever_age_process.generic_transformers.`) accepts: + +| Code | Type | Required | Default | Description | +|----------------------|---------|:--------:|---------|-------------------------------------------------------------------------------------------------------------------------| +| `contextual_options` | `array` | | `[]` | List of `option code => option definition` (see below); each one becomes an option of the transformer | +| `transformers` | `array` | | `[]` | Transformer chain, see [TransformerTrait](../traits/transformer_trait.md). May contain `{{ option_code }}` placeholders | + +Each contextual option definition (`~` is allowed to use all defaults) accepts: + +| Code | Type | Required | Default | Description | +|-------------------|--------|:--------:|---------|----------------------------------------------------------------------------------| +| `required` | `bool` | | `true` | The option must be provided when using the transformer (unless it has a default) | +| `default` | `any` | | `null` | If not `null`, default value of the option | +| `default_is_null` | `bool` | | `false` | Use `null` as default value (not possible with `default`) | + +When using the generic transformer, only its contextual options are accepted; passing `transformers` throws an +`InvalidArgumentException`. + +Examples +-------- + +* Definition without option, and usage + +```yaml +# Bundle configuration level (config/packages/*.yaml) +clever_age_process: + generic_transformers: + uppercase: + transformers: + callback: + callback: strtoupper +``` + +```yaml +# Transformer options level +uppercase: ~ +``` + +* Definition with contextual options, and usage: `substr($value, 2, null)` + +```yaml +# Bundle configuration level (config/packages/*.yaml) +clever_age_process: + generic_transformers: + substr: + contextual_options: + offset: + required: true + length: + default_is_null: true + transformers: + callback: + callback: substr + right_parameters: ['{{ offset }}', '{{ length }}'] +``` + +```yaml +# Transformer options level +substr: + offset: 2 +``` + +Notes +----- + +* Contextual options are transformer options, unrelated to the process context (`-c key:value`). +* An option declared with `required: false` and no default is neither required nor defined, so it cannot be used: + always give such an option a `default` or `default_is_null: true`. diff --git a/docs/reference/transformers/hash_transformer.md b/docs/reference/transformers/hash_transformer.md new file mode 100644 index 00000000..f0f85ed3 --- /dev/null +++ b/docs/reference/transformers/hash_transformer.md @@ -0,0 +1,54 @@ +HashTransformer +=============== + +Generate a hash of the input value, using PHP's [hash](https://www.php.net/manual/en/function.hash.php) function. + +Transformer reference +--------------------- + +* **Service**: `CleverAge\ProcessBundle\Transformer\String\HashTransformer` +* **Transformer code**: `hash` + +Accepted inputs +--------------- + +Any value that can be cast to `string`. + +Possible outputs +---------------- + +`string`: the hash, as lowercase hexits by default, or raw binary data if `raw_output` is `true` + +Options +------- + +| Code | Type | Required | Default | Description | +|--------------|----------|:--------:|---------|-----------------------------------------------------------------------------------------------| +| `algo` | `string` | **X** | | Hashing algorithm (e.g. `md5`, `sha1`, `sha256`, `crc32b`), must be one of `hash_algos()` | +| `raw_output` | `bool` | | `false` | If `true`, output raw binary data instead of lowercase hexits | + +Examples +-------- + +* `'foo'` becomes `'2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae'` + +```yaml +# Transformer options level +hash: + algo: sha256 +``` + +* Compute a checksum of several fields + +```yaml +# Transformer mapping level +checksum: + code: + - '[sku]' + - '[price]' + transformers: + implode: + separator: '|' + hash: + algo: md5 +``` diff --git a/docs/reference/transformers/implode_transformer.md b/docs/reference/transformers/implode_transformer.md index fb5d27c7..725493d1 100644 --- a/docs/reference/transformers/implode_transformer.md +++ b/docs/reference/transformers/implode_transformer.md @@ -1,12 +1,11 @@ ImplodeTransformer -========================= +================== -Join array elements with a string +Join the elements of an array into a string, using PHP's [implode](https://www.php.net/manual/en/function.implode.php) +function. -This transformer uses the php internal function: https://www.php.net/manual/en/function.implode.php - -Task reference --------------- +Transformer reference +--------------------- * **Service**: `CleverAge\ProcessBundle\Transformer\String\ImplodeTransformer` * **Transformer code**: `implode` @@ -14,7 +13,7 @@ Task reference Accepted inputs --------------- -`array` +`array` of values that can be cast to `string`. Any other input throws an `\UnexpectedValueException`. Possible outputs ---------------- @@ -24,20 +23,31 @@ Possible outputs Options ------- -| Code | Type | Required | Default | Description | -|-------------|----------|:--------:|---------|-------------| -| `separator` | `string` | **X** | `|` | | +| Code | Type | Required | Default | Description | +|-------------|----------|:--------:|---------|------------------------------------------| +| `separator` | `string` | | `'\|'` | String inserted between each element | Examples -------- +* `['1', '2', '3']` becomes `'1|2|3'` + +```yaml +# Transformer options level +implode: ~ +``` + +* Concatenate then slugify several fields + ```yaml # Transformer mapping level -sprintf_multiple: +slug: code: + - '[id]' - '[firstname]' - '[lastname]' transformers: implode: separator: '-' + slugify: ~ ``` diff --git a/docs/reference/transformers/instantiate_transformer.md b/docs/reference/transformers/instantiate_transformer.md new file mode 100644 index 00000000..b6ce1c5d --- /dev/null +++ b/docs/reference/transformers/instantiate_transformer.md @@ -0,0 +1,53 @@ +InstantiateTransformer +====================== + +Create a new instance of the configured class, using the input array values as constructor arguments +(`\ReflectionClass::newInstanceArgs()`). + +Transformer reference +--------------------- + +* **Service**: `CleverAge\ProcessBundle\Transformer\Object\InstantiateTransformer` +* **Transformer code**: `instantiate` + +Accepted inputs +--------------- + +`array`: the constructor arguments. A list is passed positionally; string keys are used as named arguments. Any other +value throws an `\UnexpectedValueException`. + +Possible outputs +---------------- + +`object`: a new instance of the configured class + +Options +------- + +| Code | Type | Required | Default | Description | +|---------|----------|:--------:|---------|--------------------------------------------| +| `class` | `string` | **X** | | Fully qualified class name to instantiate | + +Examples +-------- + +* Build an `App\Dto\Price` object whose constructor is `__construct(float $amount, string $currency)` + +```yaml +# Transformer mapping level +price: + code: + - '[amount]' + - '[currency]' + transformers: + instantiate: + class: 'App\Dto\Price' +``` + +* Same, input `{amount: 12.5, currency: 'EUR'}` being passed as named arguments + +```yaml +# Transformer options level +instantiate: + class: 'App\Dto\Price' +``` diff --git a/docs/reference/transformers/mapping_transformer.md b/docs/reference/transformers/mapping_transformer.md index 4971f66b..1b19750f 100644 --- a/docs/reference/transformers/mapping_transformer.md +++ b/docs/reference/transformers/mapping_transformer.md @@ -1,17 +1,18 @@ MappingTransformer ================== -Transform a set of properties into a (possibly) new output. +Build a (possibly) new array or object from the properties of the input. -Basically, the algorithm is: -* determine destination (from `initial_value` or `keep_input`) -* foreach property - - get value(s) from the source(s) (from `code`, `constant` or `set_null`) - - use additional transformers on the value - - merge the property and its value into the destination (with `merge_callback`, the property accessor, or as a simple array index) +The algorithm is: -Task reference --------------- +* determine the destination (`initial_value`, or the input itself with `keep_input`) +* for each target property of `mapping`: + - get the source value (from `constant`, `set_null`, or the `code` property path(s)) + - apply the property `transformers` on this value + - write the result into the destination (with `merge_callback`, the property accessor, or as a simple array key) + +Transformer reference +--------------------- * **Service**: `CleverAge\ProcessBundle\Transformer\MappingTransformer` * **Transformer code**: `mapping` @@ -19,99 +20,98 @@ Task reference Accepted inputs --------------- -`array` or `object` that can be accessed by the property accessor +`array` or `object` readable by the Symfony [PropertyAccessor](https://symfony.com/doc/current/components/property_access.html). Possible outputs ---------------- -`array` or `object` (the "destination") containing the property manipulated by the transformer +`array` or `object`: the destination, filled with the mapped properties. Options ------- -| Code | Type | Required | Default | Description | -|------------------|----------------------|:---------:|---------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `mapping` | `array` | **X** | | List of property => sub-mapping options. The property code can be a single string to be used as an array index, or a writable property path | -| `ignore_missing` | `bool` | | `false` | Ignore property accessor errors for the whole mapping | -| `keep_input` | `bool` | | `false` | Use input as the mapping destination (takes precedence on `initial_value`). Keep in mind that due to PHP behavior, arrays are cloned while objects are passed by reference | -| `initial_value` | `any` | | `[]` | Set the mapping destination | -| `merge_callback` | `callable` or `null` | | `null` | Allow to change how a property can be set in the destination | +| Code | Type | Required | Default | Description | +|------------------|------------------|:--------:|---------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `mapping` | `array` | **X** | | List of `target property => property options` (see below). The target is a writable property path of the destination, or a plain array key | +| `ignore_missing` | `bool` | | `false` | Ignore property accessor read errors for the whole mapping (the property is then skipped) | +| `keep_input` | `bool` | | `false` | Use the input as the destination. Cannot be combined with a non-empty `initial_value`. Due to PHP behavior, arrays are copied while objects are modified in place | +| `initial_value` | `any` | | `[]` | The destination to fill | +| `merge_callback` | `callable\|null` | | `null` | Custom callable used to write each value, called with `($destination, $targetProperty, $value)` | -Foreach property there is the following options. +Each property of `mapping` has the following options (`~` is allowed to use all defaults): -| Code | Type | Required | Default | Description | -|------------------|-------------------------------|:---------:|---------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `code` | `string` or `array` or `null` | | `null` | A property path, or a list of property path. By default it would be the same as the destination property. Will be used as a source. The special value '.' access the whole object. | -| `constant` | `any` | | `null` | If not `null`, will be directly used as a source (takes precedence on `code`) | -| `set_null` | `bool` | | `false` | If `true`, `null` will be directly used as a source (takes precedence on `code`) | -| `ignore_missing` | `bool` | | `false` | Ignore property accessor errors for this source | -| `transformers` | `array` | | `[]` | List of sub-transformers, see [TransformerTrait](../traits/transformer_trait.md) | +| Code | Type | Required | Default | Description | +|------------------|-----------------------|:--------:|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------------| +| `code` | `string\|array\|null` | | `null` | Source property path, or list of `key => property path` to build an array. Defaults to the target property. The special value `.` returns the whole input | +| `constant` | `any` | | `null` | If not `null`, used as the source value (takes precedence on `set_null` and `code`) | +| `set_null` | `bool` | | `false` | If `true`, `null` is used as the source value (takes precedence on `code`) | +| `ignore_missing` | `bool` | | `false` | Ignore property accessor read errors for this property (with a list of paths, only the missing keys are skipped) | +| `transformers` | `array` | | `[]` | Transformers applied on the source value, see [TransformerTrait](../traits/transformer_trait.md) | Examples -------- -* Simple transformation, will output an array with keys "code", "label", "type", "reference", "required" and "slug" - - required input: an array with keys "Code", "label", "Type", "Name" and "ID" - - output: an array with keys "code", "label", "type", "reference", "required" and "slug" +* Simple mapping + - input: an array with keys `Code`, `label`, `Type`, `Name` and `ID` + - output: an array with keys `code`, `label`, `type`, `reference`, `required` and `slug` ```yaml # Transformer options level mapping: mapping: - code: # Simple mapping from "Code" to "code" + code: # Simple mapping from "Code" to "code" code: '[Code]' - "[label]": ~ # Value from "label" will be kept with the same name - type: # Get value from "type" and map values (with a default) + '[label]': ~ # Value of "label" kept under the same key + type: # Convert values, with a fallback code: '[Type]' transformers: convert_value: ignore_missing: true map: - TEXTE: text - NUMERIQUE: number - LISTE_DEROULANTE: simpleselect - CHOIX_MULTIPLES: multiselect - DATE: date - default: - value: unknown - reference: # "null" column + TEXTE: text + NUMERIQUE: number + DATE: date + default: + value: unknown + reference: # null value set_null: true - required: # "true" column + required: # constant value constant: true - slug: # Get multiple sources, slugify them, and merge them + slug: # Multiple sources, slugified and imploded code: name: '[Name]' - id: '[ID]' + id: '[ID]' transformers: - array_map: + array_map: transformers: slugify: ~ implode: separator: '_' ``` -* Mapping in depth, using objects - - required input: an object with an iterable property "productItems", containing objects with property "longName" - - output: an array with key "items", containing a list of array with key "name" +* Nested mapping, using objects + - input: an object with an iterable property `productItems`, containing objects with a property `longName` + - output: an array with key `items`, containing a list of arrays with key `name` ```yaml # Transformer options level -mapping: # Transformer code - mapping: # MappingTransformer options - items: # property code - code: 'productItems' # property options - transformers: - array_map: # Transformer code - transformers: # ArrayMapTransformer options - mapping: # Transformer code - mapping: # MappingTransformer options - name: # property code - code: 'longName' # property options +mapping: + mapping: + items: + code: productItems + transformers: + array_map: + transformers: + mapping: + mapping: + name: + code: longName ``` -* Advanced property setter - - required input: an object with a property "address", containing an object with properties "postCode" and "customer", itself containing an object with property "hasFlag" - - output: same object, with a modified "address.customer.hasFlag" +* Update an object in place + - input: an object with a property `address`, containing properties `postCode` and `customer` (itself having a + property `hasFlag`) + - output: the same object, with `address.customer.hasFlag` updated ```yaml # Transformer options level @@ -128,3 +128,13 @@ mapping: default: value: false ``` + +Notes +----- + +* Array values must be read with the index notation (`[key]`). By default, the Symfony PropertyAccessor returns `null` + instead of throwing for a missing array index (`framework.property_access.throw_exception_on_invalid_index`), so + `ignore_missing` mostly matters for objects. +* When a sub-transformer fails, the thrown `TransformerException` reports the target property. +* `merge_callback` receives the destination by value: to modify an array destination, the callable must take its + first argument by reference. diff --git a/docs/reference/transformers/multi_replace_transformer.md b/docs/reference/transformers/multi_replace_transformer.md index 4a3f035e..cead0494 100644 --- a/docs/reference/transformers/multi_replace_transformer.md +++ b/docs/reference/transformers/multi_replace_transformer.md @@ -1,12 +1,11 @@ MultiReplaceTransformer -========================= +======================= -Quickly replace a list of values in a string. +Replace a list of substrings in a string, using PHP +[`str_replace()`](https://www.php.net/manual/en/function.str-replace.php) once per entry of the replacement map. -This transformer uses the php internal function: https://www.php.net/manual/en/function.str-replace.php - -Task reference --------------- +Transformer reference +--------------------- * **Service**: `CleverAge\ProcessBundle\Transformer\MultiReplaceTransformer` * **Transformer code**: `multi_replace` @@ -19,26 +18,33 @@ Any value that can be cast to string. Possible outputs ---------------- -`string` +`string`: the input with all replacements applied. If `replace_mapping` is empty, the input is returned unchanged +(not cast). Options ------- -| Code | Type | Required | Default | Description | -|-------------------|---------|:--------:|---------|-----------------------------------| -| `replace_mapping` | `array` | **X** | | $search as key, $replace as value | +| Code | Type | Required | Default | Description | +|-------------------|---------|:--------:|---------|------------------------------------------------------------------------------------------| +| `replace_mapping` | `array` | **X** | | Searched string as key, replacement as value. Entries are applied sequentially, in order | Examples -------- ```yaml -# Transformer mapping level +# Transformer options level multi_replace: - code: - - '[firstname]' + replace_mapping: + ' ': '!' + 'name': '' +``` + +```yaml +# Transformer mapping level +firstname: + code: '[firstname]' transformers: multi_replace: - replace_mapping: - ' ': '!' - 'name': '' + replace_mapping: + ' ': '-' ``` diff --git a/docs/reference/transformers/normalize_transformer.md b/docs/reference/transformers/normalize_transformer.md new file mode 100644 index 00000000..5389f3db --- /dev/null +++ b/docs/reference/transformers/normalize_transformer.md @@ -0,0 +1,50 @@ +NormalizeTransformer +==================== + +Normalize the input (typically an object) into an array or a scalar, using the Symfony +[Serializer](https://symfony.com/doc/current/components/serializer.html) normalizer. + +See also [NormalizerTask](../tasks/normalizer_task.md) for the task equivalent. + +Transformer reference +--------------------- + +* **Service**: `CleverAge\ProcessBundle\Transformer\Serialization\NormalizeTransformer` +* **Transformer code**: `normalize` + +Accepted inputs +--------------- + +`any`: data supported by the configured normalizers (typically an `object`) + +Possible outputs +---------------- + +`array`, `string`, `int`, `float`, `bool`, `\ArrayObject` or `null`: the normalized representation + +Options +------- + +| Code | Type | Required | Default | Description | +|-----------|----------------|:--------:|---------|-----------------------------------------------------| +| `format` | `string\|null` | | `null` | Format passed to the normalizer | +| `context` | `array` | | `[]` | Normalization context (groups, attributes…) | + +Examples +-------- + +* Normalize an object with default options + +```yaml +# Transformer options level +normalize: ~ +``` + +* Normalize an object using serialization groups + +```yaml +# Transformer options level +normalize: + context: + groups: [export] +``` diff --git a/docs/reference/transformers/preg_filter_transformer.md b/docs/reference/transformers/preg_filter_transformer.md new file mode 100644 index 00000000..29a7f70a --- /dev/null +++ b/docs/reference/transformers/preg_filter_transformer.md @@ -0,0 +1,55 @@ +PregFilterTransformer +===================== + +Perform a regular expression search and replace on the input using PHP +[`preg_filter()`](https://www.php.net/manual/en/function.preg-filter.php): unlike `preg_replace()`, `null` is returned +when the pattern does not match. + +Transformer reference +--------------------- + +* **Service**: `CleverAge\ProcessBundle\Transformer\PregFilterTransformer` +* **Transformer code**: `preg_filter` + +Accepted inputs +--------------- + +Any value that can be cast to string. + +Possible outputs +---------------- + +`string|null`: the replaced string, or `null` if no pattern matches (or on regex error). + +Options +------- + +| Code | Type | Required | Default | Description | +|---------------|-----------------|:--------:|---------|-------------------------------------------------------------------------------------------------------| +| `pattern` | `string\|array` | **X** | | Pattern, or list of patterns, to search | +| `replacement` | `string\|array` | **X** | | Replacement string. The value is cast to string, so an array is not supported in practice (see Notes) | + +Examples +-------- + +```yaml +# Transformer options level +preg_filter: + pattern: '/[^a-z0-9]/' + replacement: '' +``` + +* Reformat a date, `null` if the input does not match + +```yaml +# Transformer options level +preg_filter: + pattern: '/^(\d{2})\/(\d{2})\/(\d{4})$/' + replacement: '$3-$2-$1' +``` + +Notes +----- + +Although `replacement` accepts an `array`, it is cast to string before calling `preg_filter()`, which results in the +literal `Array` (and a PHP warning). diff --git a/docs/reference/transformers/preg_match_transformer.md b/docs/reference/transformers/preg_match_transformer.md index 83270645..b9cb6146 100644 --- a/docs/reference/transformers/preg_match_transformer.md +++ b/docs/reference/transformers/preg_match_transformer.md @@ -1,12 +1,11 @@ PregMatchTransformer -========================= +==================== -Perform a regular expression match +Perform a regular expression match, using PHP's [preg_match](https://www.php.net/manual/en/function.preg-match.php) +or [preg_match_all](https://www.php.net/manual/en/function.preg-match-all.php) function, and return the matches. -This transformer uses the php internal function: https://www.php.net/manual/en/function.preg-match.php - -Task reference --------------- +Transformer reference +--------------------- * **Service**: `CleverAge\ProcessBundle\Transformer\String\PregMatchTransformer` * **Transformer code**: `preg_match` @@ -14,40 +13,55 @@ Task reference Accepted inputs --------------- -`string` +Any value that can be cast to `string`, or `null`. Possible outputs ---------------- -`array` or `null` +* `array`: the `$matches` array filled by `preg_match` / `preg_match_all` (an empty array when nothing matches with + `preg_match`) +* `null` if the input is `null` or an empty string Options ------- -| Code | Type | Required | Default | Description | -|------------|-----------|:--------:|---------|------------------------------------------| -| `pattern` | `string` | **X** | | | -| `flags` | `int` | | 0 | | -| `offset` | `int` | | 0 | | -| `mode_all` | `boolean` | | false | Use preg_match_all instead of preg_match | +| Code | Type | Required | Default | Description | +|------------|----------|:--------:|---------|------------------------------------------------------------------------------------------------------| +| `pattern` | `string` | **X** | | The regular expression, with delimiters | +| `flags` | `int` | | `0` | `PREG_*` flags passed to the function (e.g. `!php/const PREG_OFFSET_CAPTURE`) | +| `offset` | `int` | | `0` | Offset (in bytes) from which to start the search | +| `mode_all` | `bool` | | `false` | If `true`, use `preg_match_all` instead of `preg_match` | Examples -------- +* `'foobarbaz'` becomes `['foobarbaz', 'foo', 'bar', 'baz']` + +```yaml +# Transformer options level +preg_match: + pattern: '/(foo)(bar)(baz)/' +``` + +* Capture offsets and keep only the 2nd group: `'foobarbaz'` becomes `['bar', 3]` + +```yaml +# Transformer mapping level +second_group: + code: '[text]' + transformers: + preg_match: + pattern: '/(foo)(bar)(baz)/' + flags: !php/const PREG_OFFSET_CAPTURE + property_accessor: + property_path: '[2]' +``` + +* Get all numbers of a string: `'a1b22c333'` becomes `[['1', '22', '333']]` + ```yaml # Transformer options level -entry: - service: '@CleverAge\ProcessBundle\Task\ConstantIterableOutputTask' - outputs: [ preg_match ] - options: - output: 'foobarbaz' preg_match: - service: '@CleverAge\ProcessBundle\Task\TransformerTask' - options: - transformers: - preg_match: - pattern: '/(foo)(bar)(baz)/' - flags: !php/const PREG_OFFSET_CAPTURE - property_accessor: - property_path: '[2]' + pattern: '/\d+/' + mode_all: true ``` diff --git a/docs/reference/transformers/property_accessor_transformer.md b/docs/reference/transformers/property_accessor_transformer.md new file mode 100644 index 00000000..ac39a560 --- /dev/null +++ b/docs/reference/transformers/property_accessor_transformer.md @@ -0,0 +1,65 @@ +PropertyAccessorTransformer +=========================== + +Read a value from the input using the Symfony [PropertyAccessor](https://symfony.com/doc/current/components/property_access.html) +and return it. + +Transformer reference +--------------------- + +* **Service**: `CleverAge\ProcessBundle\Transformer\Object\PropertyAccessorTransformer` +* **Transformer code**: `property_accessor` + +Accepted inputs +--------------- + +`array` or `object` readable by the PropertyAccessor. `null` is accepted only when `ignore_null` is `true`. + +Possible outputs +---------------- + +* `any`: the value found at the property path +* `null` if the input is `null` and `ignore_null` is `true`, or if the path is not readable and `ignore_missing` is + `true` + +Options +------- + +| Code | Type | Required | Default | Description | +|------------------|----------|:--------:|---------|------------------------------------------------------------------------------------------| +| `property_path` | `string` | **X** | | Property path to read (e.g. `[key]` for arrays, `property` for objects, `[a][b]`, `a.b`) | +| `ignore_null` | `bool` | | `false` | If `true`, return `null` when the input is `null` instead of failing | +| `ignore_missing` | `bool` | | `false` | If `true`, return `null` when the property path is not readable instead of failing | + +Examples +-------- + +* Read a nested property of an object + +```yaml +# Transformer options level +property_accessor: + property_path: 'FullCountryInfoAllCountriesResult.tCountryInfo' +``` + +* Read the 3rd element of a `preg_match` result + +```yaml +# Transformer mapping level +third_element: + code: '[text]' + transformers: + preg_match: + pattern: '/(foo)(bar)(baz)/' + property_accessor: + property_path: '[2]' +``` + +* Read an optional nested array key + +```yaml +# Transformer options level +property_accessor: + property_path: '[address][city]' + ignore_missing: true +``` diff --git a/docs/reference/transformers/recursive_property_setter_transformer.md b/docs/reference/transformers/recursive_property_setter_transformer.md new file mode 100644 index 00000000..cad98ddc --- /dev/null +++ b/docs/reference/transformers/recursive_property_setter_transformer.md @@ -0,0 +1,61 @@ +RecursivePropertySetterTransformer +================================== + +Read an iterable from the input, then set one or more properties on each of its items, using values read from the +input itself. Typically used to propagate a parent value (an id, a code…) to each child of a collection. + +Transformer reference +--------------------- + +* **Service**: `CleverAge\ProcessBundle\Transformer\Object\RecursivePropertySetterTransformer` +* **Transformer code**: `recursive_property_setter` + +Accepted inputs +--------------- + +`array` or `object` readable by the Symfony PropertyAccessor, containing an iterable at the `iterator` path. `null` is +accepted only when `ignore_null` is `true`. + +Possible outputs +---------------- + +* `iterable`: the collection read at the `iterator` path, with the properties set on each item (the rest of the input + is not returned) +* `null` if the input is `null` and `ignore_null` is `true`, or if the `iterator` path is not readable and + `ignore_missing` is `true` + +Options +------- + +| Code | Type | Required | Default | Description | +|------------------|----------|:--------:|---------|---------------------------------------------------------------------------------------------------------------| +| `iterator` | `string` | **X** | | Property path of the collection in the input; a non-iterable value throws a `TransformerException` | +| `set_properties` | `array` | **X** | | Map of `item_property_path: input_property_path`; each value is read from the input and written on each item | +| `ignore_null` | `bool` | | `false` | If `true`, a `null` input returns `null` and `null` source values are allowed (else a `TransformerException`) | +| `ignore_missing` | `bool` | | `false` | If `true`, an unreadable `iterator` path returns `null` and an unreadable source value is set as `null` | + +Examples +-------- + +* Propagate the parent id and name to each item + +```yaml +# Transformer options level +recursive_property_setter: + iterator: '[items]' + set_properties: + '[parentId]': '[id]' + '[parentName]': '[name]' +``` + +With input `{id: 1, name: 'Parent', items: [{label: 'A'}, {label: 'B'}]}`, the output is +`[{label: 'A', parentId: 1, parentName: 'Parent'}, {label: 'B', parentId: 1, parentName: 'Parent'}]`. + +Notes +----- + +Keys of `set_properties` are property paths written with the PropertyAccessor on each item: use the `[key]` notation +for array items and the `property` notation for objects. For `\stdClass` items, a property that cannot be written is +added to the object. + +Object items are modified in place, so the objects of the input are modified too. diff --git a/docs/reference/transformers/rules_transformer.md b/docs/reference/transformers/rules_transformer.md index 71ecc40d..45ceb5bd 100644 --- a/docs/reference/transformers/rules_transformer.md +++ b/docs/reference/transformers/rules_transformer.md @@ -1,20 +1,16 @@ RulesTransformer ================ -Uses a set of rules to apply some set of transformers on a value. Basically behaves like a `if/elseif/else` block. +Use an ordered set of rules to conditionally transform a value. Behaves like an `if / elseif / else` block: the first +rule whose condition matches is applied, and the input is returned unchanged if no rule matches. -By default a rule uses a variable named `value` containing anything you passed in input (`array`, `string`, ...). But this -can be overridden using options `use_value_as_variables` as `true` and setting `expression_variables` to a static list of -input variables. +Conditions are [ExpressionLanguage](https://symfony.com/doc/current/components/expression_language.html) expressions. +By default, the input is available as the `value` variable. With `use_value_as_variables: true`, the input (which +must then be an array) is used as the set of variables; `expression_variables` must then list these variable names. +`expression_variables` can also be set to `null` to disable parsing at initialization (more flexible, but slower). -Note that `expression_variables` can also be set to `null` for more flexibility, but this disable initial parsing and decrease -performances. - -See [The ExpressionLanguage Component Reference](https://symfony.com/doc/current/components/expression_language.html) for -more information. - -Task reference --------------- +Transformer reference +--------------------- * **Service**: `CleverAge\ProcessBundle\Transformer\RulesTransformer` * **Transformer code**: `rules` @@ -22,59 +18,55 @@ Task reference Accepted inputs --------------- -`any` or an `array` of `variable code => value` injectable into an expression +`any`, or an `array` of `variable name => value` when `use_value_as_variables` is `true`. Possible outputs ---------------- -`any` resulting from a transformation set. - -Without any matching rules, the value itself is returned. +`any`: the result of the matching rule (`null`, a constant, or the result of its transformers), or the input itself +if no rule matches. Options ------- -| Code | Type | Required | Default | Description | -|--------------------------|-------------------|:--------:|-----------|---------------------------------------------------------------------| -| `rules_set` | `array` | **X** | | Ordered list of rules, see bellow for details | -| `use_value_as_variables` | `bool` | | `false` | Use given value as an array of variable to inject in expression | -| `expression_variables` | `array` or `null` | | `[value]` | Name of variables injected in the expression at initialization time | +| Code | Type | Required | Default | Description | +|--------------------------|---------------|:--------:|-----------|--------------------------------------------------------------------------------------------------------| +| `rules_set` | `array` | **X** | | Ordered list of rules, see below | +| `use_value_as_variables` | `bool` | | `false` | Use the input array as the expression variables, instead of a single `value` variable | +| `expression_variables` | `array\|null` | | `[value]` | Variable names used to parse conditions when options are resolved. `null` defers parsing to evaluation | -Foreach rule there is the following options. +Each rule of `rules_set` has the following options: -| Code | Type | Required | Default | Description | -|----------------|--------------------|:---------:|---------|----------------------------------------------------------------------------------------------------------------------------------------------| -| `condition` | `string` or `null` | | `null` | An expression used to match a value | -| `default` | `bool` | | `false` | Mark this rule as a default rule. The given rule must be the last, cannot have a condition, and there cannot have 2 default in the same time | -| `transformers` | `array` | | `[]` | List of sub-transformers, see [TransformerTrait](../traits/transformer_trait.md) | -| `constant` | `any` | | `null` | If not `null`, given value will be directly output (takes precedence on `transformers`) | -| `set_null` | `bool` | | `false` | If `true`, `null` will be directly output (takes precedence on `constant`) | +| Code | Type | Required | Default | Description | +|----------------|----------------|:--------:|---------|---------------------------------------------------------------------------------------------------------------------------------------| +| `condition` | `string\|null` | | `null` | Expression; the rule matches if it is truthy | +| `default` | `bool` | | `false` | Mark the rule as the default one (always matches). It cannot have a `condition`, no conditional rule may follow it, only one allowed | +| `set_null` | `bool` | | `false` | If `true`, return `null` (takes precedence on `constant` and `transformers`) | +| `constant` | `any` | | `null` | If not `null`, return this value (takes precedence on `transformers`) | +| `transformers` | `array` | | `[]` | Transformers applied on the input, see [TransformerTrait](../traits/transformer_trait.md). With no transformer, the input is returned | + +A rule without `condition` and without `default: true` never matches. Examples -------- -* Simple rules with default value - - input value is an array containing an `order` object and a `customer` object - - output will be either a value from customer, or a numeric constant, or null +* Rules on the `value` variable, with a default rule ```yaml # Transformer options level rules: rules_set: - - condition: 'value["order"].origin === "marketplace"' + - condition: 'value["order"]["origin"] === "marketplace"' transformers: property_accessor: - property_path: '[customer].id' - - condition: 'value["order"].origin === "e-commerce"' - constant: 1234 + property_path: '[customer][id]' + - condition: 'value["order"]["origin"] === "e-commerce"' + constant: value1234 - default: true set_null: true ``` -* Use value as variables - - same example as above - - can be useful for more verbose expression - - transformers still get the input as the initial array +* Same rules, using the input keys as variables (transformers still receive the whole input) ```yaml # Transformer options level @@ -82,12 +74,18 @@ rules: use_value_as_variables: true expression_variables: [order, customer] rules_set: - - condition: 'order.origin === "marketplace"' + - condition: 'order["origin"] === "marketplace"' transformers: property_accessor: - property_path: '[customer].id' - - condition: 'order.origin === "e-commerce"' - constant: 1234 + property_path: '[customer][id]' + - condition: 'order["origin"] === "e-commerce"' + constant: variable1234 - default: true set_null: true ``` + +Notes +----- + +Conditions are parsed with the bundle's `cleverage_process.expression_language` service, which also exposes the PHP +`preg_match` function. diff --git a/docs/reference/transformers/slugify_transformer.md b/docs/reference/transformers/slugify_transformer.md index 3276f6de..8476aa6a 100644 --- a/docs/reference/transformers/slugify_transformer.md +++ b/docs/reference/transformers/slugify_transformer.md @@ -1,12 +1,15 @@ SlugifyTransformer -========================= +================== -Strip whitespace (or other characters) from the beginning and end of a string +Convert a string into a slug. The value is transliterated with a +[\Transliterator](https://www.php.net/manual/en/class.transliterator.php) (by default, accents are removed), HTML +tags are stripped, the result is trimmed and lowercased, every sequence of characters matching `replace` is replaced by +`separator`, and leading and trailing separators are removed. -This transformer uses the php internal function: https://www.php.net/manual/en/class.transliterator.php +Requires the `intl` PHP extension. -Task reference --------------- +Transformer reference +--------------------- * **Service**: `CleverAge\ProcessBundle\Transformer\String\SlugifyTransformer` * **Transformer code**: `slugify` @@ -14,7 +17,7 @@ Task reference Accepted inputs --------------- -Any value that can be cast to string. +`string` Possible outputs ---------------- @@ -24,20 +27,27 @@ Possible outputs Options ------- -| Code | Type | Required | Default | Description | -|------------------|----------|:---------:|----------------------------------------|--------------------------------| -| `transliterator` | `string` | | `NFD; [:Nonspacing Mark:] Remove; NFC` | Used to create \Transliterator | -| `replace` | `string` | | `/[^a-z0-9]+/` | Used on preg_replace | -| `separator` | `string` | | `_` | Used on preg_replace | +| Code | Type | Required | Default | Description | +|------------------|----------|:--------:|------------------------------------------|-----------------------------------------------------------------------------------------| +| `transliterator` | `string` | | `'NFD; [:Nonspacing Mark:] Remove; NFC'` | Transliterator identifier, passed to `\Transliterator::create()` | +| `replace` | `string` | | `'/[^a-z0-9]+/'` | Regular expression of the characters to replace (applied on the lowercased string) | +| `separator` | `string` | | `'_'` | Replacement string, also trimmed from both ends of the result | Examples -------- +* `'Hélène Dupont'` becomes `'helene_dupont'` + +```yaml +# Transformer options level +slugify: ~ +``` + +* Use a dash as separator, and also transliterate non-latin characters: `'Привет мир'` becomes `'privet-mir'` + ```yaml -# Transformer mapping level -slug: - code: - - '[firstname]' - transformers: - slugify: ~ +# Transformer options level +slugify: + transliterator: 'Any-Latin; Latin-ASCII' + separator: '-' ``` diff --git a/docs/reference/transformers/sprintf_transformer.md b/docs/reference/transformers/sprintf_transformer.md index e8f15f00..81ec6b4c 100644 --- a/docs/reference/transformers/sprintf_transformer.md +++ b/docs/reference/transformers/sprintf_transformer.md @@ -1,12 +1,10 @@ SprintfTransformer -========================= +================== -Return a formatted string. +Return a formatted string, using PHP's [vsprintf](https://www.php.net/manual/en/function.vsprintf.php) function. -This transformer uses the php internal function: https://www.php.net/manual/en/function.vsprintf.php - -Task reference --------------- +Transformer reference +--------------------- * **Service**: `CleverAge\ProcessBundle\Transformer\String\SprintfTransformer` * **Transformer code**: `sprintf` @@ -14,7 +12,8 @@ Task reference Accepted inputs --------------- -Any value that can be cast to `string` | `int` | `float` or `array` +* `array`: each element is used as an argument of the format, in order +* any other value (scalar, `null`, `\Stringable`): used as the single argument of the format Possible outputs ---------------- @@ -24,20 +23,36 @@ Possible outputs Options ------- -| Code | Type | Required | Default | Description | -|----------|----------|:--------:|---------|----------------------------------------------------------------------------------------------------------------------| -| `format` | `string` | **X** | `%s` | The format string is composed of zero or more directives. Escape % with another %% due to ParameterBag restrictions. | +| Code | Type | Required | Default | Description | +|----------|----------|:--------:|---------|------------------------------------------------------------------------------------------------------| +| `format` | `string` | | `'%s'` | The [format string](https://www.php.net/manual/en/function.sprintf.php); see [Notes](#notes) for `%` | Examples -------- +* `'bar'` becomes `'foo bar'` + +```yaml +# Transformer options level +sprintf: + format: 'foo %%s' +``` + +* Format one value + ```yaml # Transformer mapping level sprintf_one: - code: '[firstname]' + code: '[id]' transformers: sprintf: format: 'one/%%d' +``` + +* Format several values + +```yaml +# Transformer mapping level sprintf_multiple: code: - '[firstname]' @@ -46,3 +61,9 @@ sprintf_multiple: sprintf: format: 'multiple/%%s/%%s' ``` + +Notes +----- + +In Symfony YAML configuration files, `%` is used for container parameters: escape it as `%%` (`'%%s'` is resolved as +`'%s'`). A format with more placeholders than arguments throws a `\ValueError`. diff --git a/docs/reference/transformers/trim_transformer.md b/docs/reference/transformers/trim_transformer.md index 85a09053..3b24732c 100644 --- a/docs/reference/transformers/trim_transformer.md +++ b/docs/reference/transformers/trim_transformer.md @@ -1,12 +1,11 @@ TrimTransformer -========================= +=============== -Strip whitespace (or other characters) from the beginning and end of a string +Strip whitespace (or other characters) from the beginning and end of a string, using PHP's +[trim](https://www.php.net/manual/en/function.trim.php) function. -This transformer uses the php internal function: https://www.php.net/manual/en/function.trim.php - -Task reference --------------- +Transformer reference +--------------------- * **Service**: `CleverAge\ProcessBundle\Transformer\String\TrimTransformer` * **Transformer code**: `trim` @@ -14,19 +13,35 @@ Task reference Accepted inputs --------------- -Any value that can be cast to string and null. +Any value that can be cast to `string`, or `null`. Possible outputs ---------------- -Depending on the input : -- `null` if the input is null -- `string` if the input is not null +* `string`: the trimmed value +* `null` if the input is `null` Options ------- -| Code | Type | Required | Default | Description | -| ---- | ---- | :------: |-----------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `charlist` | `string` | | ***" \t\n\r\0\x0B"*** | List of characters to trim | +| Code | Type | Required | Default | Description | +|------------|----------|:--------:|---------------------|----------------------------------------------------------------| +| `charlist` | `string` | | `" \t\n\r\0\x0B"` | Characters to strip (ranges like `a..z` are supported) | + +Examples +-------- + +* `' trim me '` becomes `'trim me'` + +```yaml +# Transformer options level +trim: ~ +``` + +* `'-trim me-'` becomes `'trim me'` +```yaml +# Transformer options level +trim: + charlist: '-' +``` diff --git a/docs/reference/transformers/type_setter_transformer.md b/docs/reference/transformers/type_setter_transformer.md new file mode 100644 index 00000000..b83fa713 --- /dev/null +++ b/docs/reference/transformers/type_setter_transformer.md @@ -0,0 +1,42 @@ +TypeSetterTransformer +===================== + +Change the type of the input value using [`settype()`](https://www.php.net/manual/en/function.settype.php). Unlike +[CastTransformer](cast_transformer.md), the target type is validated when options are resolved. + +Transformer reference +--------------------- + +* **Service**: `CleverAge\ProcessBundle\Transformer\TypeSetterTransformer` +* **Transformer code**: `type_setter` + +Accepted inputs +--------------- + +`any` + +Possible outputs +---------------- + +`any`: the input value converted to the configured type. + +Options +------- + +| Code | Type | Required | Default | Description | +|--------|----------|:--------:|---------|-----------------------------------------------------------------------------------------------------------------| +| `type` | `string` | **X** | | Target type, one of `boolean`, `bool`, `integer`, `int`, `float`, `double`, `string`, `array`, `object`, `null` | + +Examples +-------- + +```yaml +# Transformer options level +type_setter: + type: string +``` + +Notes +----- + +A `TransformerException` is thrown if `settype()` returns `false`. diff --git a/docs/reference/transformers/unset_transformer.md b/docs/reference/transformers/unset_transformer.md new file mode 100644 index 00000000..168cda1a --- /dev/null +++ b/docs/reference/transformers/unset_transformer.md @@ -0,0 +1,51 @@ +UnsetTransformer +================ + +Remove a key from the input array, optionally only when a condition is met. + +Transformer reference +--------------------- + +* **Service**: `CleverAge\ProcessBundle\Transformer\UnsetTransformer` +* **Transformer code**: `unset` + +Accepted inputs +--------------- + +`array` containing the `property` key. An `UnexpectedValueException` is thrown if the input is not an array, or if the +key does not exist (even when the condition is not met). + +Possible outputs +---------------- + +`array`: the input, without the `property` key if the condition is met. + +Options +------- + +| Code | Type | Required | Default | Description | +|-------------|----------|:--------:|---------|-----------------------------------------------------------------------------------------------------------------------------------------| +| `property` | `string` | **X** | | Array key to remove (a plain key, not a property path) | +| `condition` | `array` | | `[]` | Conditions checked against the input before removing the key, see [ConditionTrait](../traits/condition_trait.md). Always met when empty | + +Examples +-------- + +* Unconditionally remove a key + +```yaml +# Transformer options level +unset: + property: internal_id +``` + +* Remove the key only if another value matches + +```yaml +# Transformer options level +unset: + property: debug_info + condition: + match: + '[environment]': production +``` diff --git a/docs/reference/transformers/wrapper_transformer.md b/docs/reference/transformers/wrapper_transformer.md new file mode 100644 index 00000000..6cc98082 --- /dev/null +++ b/docs/reference/transformers/wrapper_transformer.md @@ -0,0 +1,45 @@ +WrapperTransformer +================== + +Wrap the input value into a single-element array, under a configurable key. + +Transformer reference +--------------------- + +* **Service**: `CleverAge\ProcessBundle\Transformer\WrapperTransformer` +* **Transformer code**: `wrapper` + +Accepted inputs +--------------- + +`any` + +Possible outputs +---------------- + +`array`: `[ => ]` + +Options +------- + +| Code | Type | Required | Default | Description | +|---------------|---------------|:--------:|---------|----------------------------| +| `wrapper_key` | `string\|int` | | `0` | Key used to wrap the value | + +Examples +-------- + +* The input `hello` becomes `{ data: hello }` + +```yaml +# Transformer options level +wrapper: + wrapper_key: data +``` + +* The input `hello` becomes `[hello]` + +```yaml +# Transformer options level +wrapper: ~ +``` diff --git a/docs/reference/transformers/xpath_evaluator.md b/docs/reference/transformers/xpath_evaluator.md deleted file mode 100644 index 43fe7a4e..00000000 --- a/docs/reference/transformers/xpath_evaluator.md +++ /dev/null @@ -1,116 +0,0 @@ -XpathEvaluatorTransformer -========================= - -Manipulate a DOMNode to extract some information using xpath. -Requires `php-xml`. - -**Important** : due to the [behavior of `\DOMXpath::query`](https://www.php.net/manual/en/domxpath.query.php), if you want -to make a query on a sub element of the full `\DOMDocument` you need to start your query with a `.` to specify the current node. - -Task reference --------------- - -* **Service**: `CleverAge\ProcessBundle\Transformer\Xml\XpathEvaluatorTransformer` -* **Transformer code**: `xpath_evaluator` - -Accepted inputs ---------------- - -`\DOMNode` only. - -Possible outputs ----------------- - -Depending on the options : -- `string` -- `\DOMNode` -- `null` -- an `array` of one of the type above - -Options -------- - -| Code | Type | Required | Default | Description | -| ---- | ---- | :------: | ------- | ----------- | -| `query` | `string` or `array` | **X** | | One or multiple Xpath queries. Using an array, you can either have a simple list of subqueries, or override some root-level query options | -| `single_result` | `boolean` | | `true` | Force the result to match a single value | -| `ignore_missing` | `boolean` | | `true` | Only used with `single_result`, avoid errors if the query doesn't match anything | -| `unwrap_value` | `boolean` | | `true` | Return the textual content of the node, only works if the result is a `\DOMText` (you might need to use the `text()` xpath selector) or a `\DOMAttr` | - -Subqueries, in their complex form, have the following options : - -| Code | Type | Required | Default | Description | -| ---- | ---- | :------: | ------- | ----------- | -| `subquery` | `string` | **X** | | An Xpath query, no additional sublevel is allowed | -| `single_result` | `boolean` | | _Root-level value for `single_result`_ | Force the result to match a single value | -| `ignore_missing` | `boolean` | | _Root-level value for `ignore_missing`_ | Only used with `single_result`, avoid errors if the query doesn't match anything | -| `unwrap_value` | `boolean` | | _Root-level value for `unwrap_value`_ | Return the textual content of the node, only works if the result is a `\DOMText` (you might need to use the `text()` xpath selector) or a `\DOMAttr` | - - -Examples --------- - -All examples assume this XML -```xml - - - ok1 - ok2 - ok3 - - - ok4 - ok5 - ok6 - - -``` - -* Example 1 : get a single value - -```yaml -# Transformer options level -xpath_evaluator: - query: '/a/b/c[0]/text()' -``` - -* Example 2 : get a multiple values - -```yaml -# Transformer options level -xpath_evaluator: - query: '/a/b/c/text()' - single_result: false -``` - -```yaml -# Transformer options level -xpath_evaluator: - query: - - '/a/d/e/text()' - - '/a/d/f/text()' - - '/a/d/g/text()' -``` - -* Example 3 : get a `\DOMNode` - -```yaml -# Transformer options level -xpath_evaluator: - query: '/a/b' - unwrap_value: false -``` - -* Example 4 : subquery with partially overridden options - -```yaml -# Transformer options level -xpath_evaluator: - query: - all_c_values: - subquery: '/a/b/c/text()' - single_result: false - e_value: '/a/d/e/text()' - f_value: - subquery: '/a/d/f/text()' -``` diff --git a/docs/reference/transformers/xpath_evaluator_transformer.md b/docs/reference/transformers/xpath_evaluator_transformer.md new file mode 100644 index 00000000..d87b7a0a --- /dev/null +++ b/docs/reference/transformers/xpath_evaluator_transformer.md @@ -0,0 +1,133 @@ +XpathEvaluatorTransformer +========================= + +Evaluate one or several XPath queries on a `\DOMNode` and return the matching values or nodes. Requires the `dom` PHP +extension. + +Queries are evaluated with [\DOMXPath::query](https://www.php.net/manual/en/domxpath.query.php), using the input node +as context node: to query relatively to a sub element of the document, start the query with `.` (e.g. `./c/text()`); +an absolute query (starting with `/`) always searches the whole document. + +Transformer reference +--------------------- + +* **Service**: `CleverAge\ProcessBundle\Transformer\Xml\XpathEvaluatorTransformer` +* **Transformer code**: `xpath_evaluator` + +Accepted inputs +--------------- + +`\DOMNode` (including `\DOMDocument`). Any other value throws an `\UnexpectedValueException`. + +Possible outputs +---------------- + +For a single query, depending on the options: + +* `string`: text of the matching `\DOMText` or `\DOMAttr` (`single_result` and `unwrap_value` are `true`) +* `\DOMNode`: the matching node (`single_result` is `true`, `unwrap_value` is `false`) +* `null`: nothing matched (`single_result` and `ignore_missing` are `true`) +* `array` of the above when `single_result` is `false` + +When `query` is an array, the output is an `array` with the same keys, each value being the result of the matching +subquery. + +Options +------- + +| Code | Type | Required | Default | Description | +|------------------|-----------------|:--------:|---------|------------------------------------------------------------------------------------------------------------------------------------------| +| `query` | `string\|array` | **X** | | An XPath query, or an array of subqueries (see below) | +| `single_result` | `bool` | | `true` | Return a single value instead of a list; more than one result throws an `\UnexpectedValueException` | +| `ignore_missing` | `bool` | | `true` | Only used with `single_result`: return `null` when nothing matches, instead of throwing an `\UnexpectedValueException` | +| `unwrap_value` | `bool` | | `true` | Return the text value of each result; only `\DOMText` (use the `text()` selector) and `\DOMAttr` results are supported, others throw | + +Each element of an array `query` is either a string (the subquery itself, using the root level options), or an array +with the following options: + +| Code | Type | Required | Default | Description | +|------------------|----------|:--------:|--------------------------------|---------------------------------------------------------------------| +| `subquery` | `string` | **X** | | An XPath query (no further nesting is allowed) | +| `single_result` | `bool` | | _root level `single_result`_ | Same as the root level option, for this subquery only | +| `ignore_missing` | `bool` | | _root level `ignore_missing`_ | Same as the root level option, for this subquery only | +| `unwrap_value` | `bool` | | _root level `unwrap_value`_ | Same as the root level option, for this subquery only | + +Examples +-------- + +All examples use this XML document as input: + +```xml + + + ok1 + ok2 + ok3 + + + ok4 + ok5 + ok6 + + +``` + +* Get a single value: `'ok1'` (XPath positions start at 1) + +```yaml +# Transformer options level +xpath_evaluator: + query: '/a/b/c[1]/text()' +``` + +* Get an attribute value: `'g1'` + +```yaml +# Transformer options level +xpath_evaluator: + query: '/a/d/g/@id' +``` + +* Get multiple values: `['ok1', 'ok2', 'ok3']` + +```yaml +# Transformer options level +xpath_evaluator: + query: '/a/b/c/text()' + single_result: false +``` + +* Get a list of subqueries results: `['ok4', 'ok5', 'ok6']` + +```yaml +# Transformer options level +xpath_evaluator: + query: + - '/a/d/e/text()' + - '/a/d/f/text()' + - '/a/d/g/text()' +``` + +* Get a `\DOMNode` (the `` element) + +```yaml +# Transformer options level +xpath_evaluator: + query: '/a/b' + unwrap_value: false +``` + +* Named subqueries, partially overriding root level options: + `{all_c_values: ['ok1', 'ok2', 'ok3'], e_value: 'ok4', f_value: 'ok5'}` + +```yaml +# Transformer options level +xpath_evaluator: + query: + all_c_values: + subquery: '/a/b/c/text()' + single_result: false + e_value: '/a/d/e/text()' + f_value: + subquery: '/a/d/f/text()' +```