diff --git a/src/.vuepress/public/img/featured-function-01-en.png b/src/.vuepress/public/img/featured-function-01-en.png new file mode 100644 index 000000000..ccb63b679 Binary files /dev/null and b/src/.vuepress/public/img/featured-function-01-en.png differ diff --git a/src/.vuepress/public/img/featured-function-01.png b/src/.vuepress/public/img/featured-function-01.png new file mode 100644 index 000000000..62720d0e2 Binary files /dev/null and b/src/.vuepress/public/img/featured-function-01.png differ diff --git a/src/.vuepress/public/img/featured-function-02-en.png b/src/.vuepress/public/img/featured-function-02-en.png new file mode 100644 index 000000000..83a28c76c Binary files /dev/null and b/src/.vuepress/public/img/featured-function-02-en.png differ diff --git a/src/.vuepress/public/img/featured-function-02.png b/src/.vuepress/public/img/featured-function-02.png new file mode 100644 index 000000000..8f7acb5bd Binary files /dev/null and b/src/.vuepress/public/img/featured-function-02.png differ diff --git a/src/.vuepress/public/img/featured-function-03-en.png b/src/.vuepress/public/img/featured-function-03-en.png new file mode 100644 index 000000000..f7f0071bf Binary files /dev/null and b/src/.vuepress/public/img/featured-function-03-en.png differ diff --git a/src/.vuepress/public/img/featured-function-03.png b/src/.vuepress/public/img/featured-function-03.png new file mode 100644 index 000000000..a0aaacb83 Binary files /dev/null and b/src/.vuepress/public/img/featured-function-03.png differ diff --git a/src/.vuepress/public/img/featured-function-04-en.png b/src/.vuepress/public/img/featured-function-04-en.png new file mode 100644 index 000000000..e8bf68272 Binary files /dev/null and b/src/.vuepress/public/img/featured-function-04-en.png differ diff --git a/src/.vuepress/public/img/featured-function-04.png b/src/.vuepress/public/img/featured-function-04.png new file mode 100644 index 000000000..97dd57d14 Binary files /dev/null and b/src/.vuepress/public/img/featured-function-04.png differ diff --git a/src/UserGuide/Master/Table/SQL-Manual/Featured-Functions_apache.md b/src/UserGuide/Master/Table/SQL-Manual/Featured-Functions_apache.md index d0b771fee..500cbfba4 100644 --- a/src/UserGuide/Master/Table/SQL-Manual/Featured-Functions_apache.md +++ b/src/UserGuide/Master/Table/SQL-Manual/Featured-Functions_apache.md @@ -210,6 +210,202 @@ GROUP BY 1, device_id; +---------+---------+--------+ ``` +### 1.3 `M4` Function + +#### 1.3.1 Description + +`M4` is a built-in polymorphic table function in the table model for downsampling time-series data. For each window, it extracts the first, last, minimum (bottom), and maximum (top) points to reduce the number of returned points while preserving the shape of the trend. `M4` takes a table argument and returns a result table, so it must be called in the `FROM` clause. + +- First point: the point with the earliest timestamp and a non-`NULL` value in the window. +- Last point: the point with the latest timestamp and a non-`NULL` value in the window. +- Minimum and maximum points: the points with the smallest and largest values in the window. If multiple points have the same value, the first point encountered in the input scan order is retained. +- Both time windows and point-count windows are supported. A duration literal for `SIZE`, such as `1h` or `10m`, selects time-window mode; a positive integer, such as `100`, selects point-count-window mode. + +> Supported since V2.0.12 + +#### 1.3.2 Function Definition + +```sql +SELECT * +FROM M4( + DATA => input_table PARTITION BY partition_column ORDER BY time_column, + TIMECOL => 'time_column', + [ORIGIN => origin_timestamp,] + SIZE => window_duration | window_point_count, + [SLIDE => window_duration | window_point_count] +); +``` + +`DATA` is a set-semantic table argument. `ORDER BY` is required and must contain only the time column specified by `TIMECOL`, in ascending order. `PARTITION BY` is optional; if omitted, the entire input table is treated as one partition. Arguments can be passed by name or in declaration order. Named arguments are recommended. + +#### 1.3.3 Parameters + +|Parameter|Type|Required|Description| +|---|---|---|---| +|`DATA`|Table argument (`SET SEMANTIC`)|Yes|Input table or subquery. Supports `PARTITION BY` and `ORDER BY`.| +|`TIMECOL`|String|Yes|Name of the time column. The column must be of type `TIMESTAMP`, and `ORDER BY` must contain only this column.| +|`ORIGIN`|Timestamp|No|Alignment origin for time windows. Defaults to `0`. Not used in point-count-window mode.| +|`SIZE`|Duration or `INT64`|Yes|Duration of a time window or number of points in a point-count window. Must be positive.| +|`SLIDE`|Same type as `SIZE`|No|Window sliding step. Defaults to `SIZE` and must use the same window mode as `SIZE`.| + +**Calculation Rules** + +- The column specified by `TIMECOL` is excluded from M4 calculations. +- Columns specified by `PARTITION BY` are excluded from calculations but retained in the output. +- All remaining columns of comparable types participate in M4 calculations. +- After excluding the time and partition columns, analysis fails if no comparable column remains or if any remaining column has a non-comparable type. +- Records with a `NULL` value in a participating column are excluded from that column's `first`, `last`, `bottom`, and `top` calculations. +- For each participating column, the `first`, `last`, `bottom`, and `top` points are calculated, deduplicated, and output in ascending timestamp order. + +**Returned Results** + +- Time-window mode always generates `window_start` and `window_end`, both of type `TIMESTAMP`, to represent the interval `[window_start, window_end)`. +- Point-count-window mode always generates `window_index`, of type `INT64`. Window indices are calculated independently within each `PARTITION BY` partition, starting at `0`. +- Partition columns retain their original names and types and appear in declaration order. `OBJECT` columns cannot be partition columns. `BLOB` partition columns can be retained in the output but do not participate in M4 calculations. +- For each participating column `c`, two output columns are generated: `c_time` for the point's timestamp and `c` for its value. Their types match the corresponding input types. Results for multiple columns are expanded in input-column order. + +The number of output rows in a window equals the largest number of feature points among all participating columns. For columns with fewer feature points, the remaining `c_time` and `c` cells are filled with `NULL`. + +#### 1.3.4 Usage Examples + +The following examples use `table1` from [Sample Data](../Reference/Sample-Data.md). Because `table1` contains additional columns besides the time and partition columns, a subquery selects only `time`, `device_id`, `temperature`, and `humidity` for M4 calculations. + +**Example 1: Time Windows** + +For `device_id = '101'`, divide the data into 3-minute windows. `SIZE => 3m` selects time-window mode with a duration of 3 minutes. Each half-open interval `[window_start, window_end)` extracts the first, last, minimum, and maximum points for `temperature` and `humidity`, then outputs the points in ascending timestamp order. + +```sql +SELECT window_start, + window_end, + device_id, + temperature_time, + temperature, + humidity_time, + humidity +FROM M4( + DATA => ( + SELECT time, device_id, temperature, humidity + FROM table1 + WHERE device_id = '101' + AND time >= 2024-11-27 16:38:00 + AND time <= 2024-11-27 16:44:00 + ) PARTITION BY device_id ORDER BY time, + TIMECOL => 'time', + SIZE => 3m +); +``` + +Result: + +```text ++-----------------------------+-----------------------------+---------+-----------------------------+-----------+-----------------------------+--------+ +| window_start| window_end|device_id| temperature_time|temperature| humidity_time|humidity| ++-----------------------------+-----------------------------+---------+-----------------------------+-----------+-----------------------------+--------+ +|2024-11-27T16:36:00.000+08:00|2024-11-27T16:39:00.000+08:00| 101| null| null|2024-11-27T16:38:00.000+08:00| 35.1| +|2024-11-27T16:39:00.000+08:00|2024-11-27T16:42:00.000+08:00| 101|2024-11-27T16:39:00.000+08:00| 85.0|2024-11-27T16:39:00.000+08:00| 35.3| +|2024-11-27T16:39:00.000+08:00|2024-11-27T16:42:00.000+08:00| 101|2024-11-27T16:41:00.000+08:00| 85.0| null| null| +|2024-11-27T16:42:00.000+08:00|2024-11-27T16:45:00.000+08:00| 101| null| null|2024-11-27T16:42:00.000+08:00| 35.2| ++-----------------------------+-----------------------------+---------+-----------------------------+-----------+-----------------------------+--------+ +``` + +**Example 2: Point-Count Windows** + +For `device_id = '101'`, divide the data into windows of 3 points. `SIZE => 3` selects point-count-window mode, with 3 input rows per window. `window_index` identifies each window and starts at `0` within each partition. + +```sql +SELECT window_index, + device_id, + temperature_time, + temperature, + humidity_time, + humidity +FROM M4( + DATA => ( + SELECT time, device_id, temperature, humidity + FROM table1 + WHERE device_id = '101' + AND time >= 2024-11-27 16:38:00 + AND time <= 2024-11-27 16:44:00 + ) PARTITION BY device_id ORDER BY time, + TIMECOL => 'time', + SIZE => 3 +); +``` + +Result: + +```text ++------------+---------+-----------------------------+-----------+-----------------------------+--------+ +|window_index|device_id| temperature_time|temperature| humidity_time|humidity| ++------------+---------+-----------------------------+-----------+-----------------------------+--------+ +| 0| 101|2024-11-27T16:39:00.000+08:00| 85.0|2024-11-27T16:38:00.000+08:00| 35.1| +| 0| 101|2024-11-27T16:40:00.000+08:00| 85.0|2024-11-27T16:39:00.000+08:00| 35.3| +| 1| 101|2024-11-27T16:41:00.000+08:00| 85.0|2024-11-27T16:42:00.000+08:00| 35.2| ++------------+---------+-----------------------------+-----------+-----------------------------+--------+ +``` + +To specify the sliding step, also set `SLIDE`: + +```sql +SELECT window_index, + device_id, + temperature_time, + temperature, + humidity_time, + humidity +FROM M4( + DATA => ( + SELECT time, device_id, temperature, humidity + FROM table1 + WHERE device_id = '101' + AND time >= 2024-11-27 16:38:00 + AND time <= 2024-11-27 16:44:00 + ) PARTITION BY device_id ORDER BY time, + TIMECOL => 'time', + SIZE => 3, + SLIDE => 2 +); +``` + +This call uses a window size of 3 points and advances the window by 2 points each time. + +Result: + +```text ++------------+---------+-----------------------------+-----------+-----------------------------+--------+ +|window_index|device_id| temperature_time|temperature| humidity_time|humidity| ++------------+---------+-----------------------------+-----------+-----------------------------+--------+ +| 0| 101|2024-11-27T16:39:00.000+08:00| 85.0|2024-11-27T16:38:00.000+08:00| 35.1| +| 0| 101|2024-11-27T16:40:00.000+08:00| 85.0|2024-11-27T16:39:00.000+08:00| 35.3| +| 1| 101|2024-11-27T16:40:00.000+08:00| 85.0|2024-11-27T16:42:00.000+08:00| 35.2| +| 1| 101|2024-11-27T16:41:00.000+08:00| 85.0| null| null| +| 2| 101| null| null|2024-11-27T16:42:00.000+08:00| 35.2| ++------------+---------+-----------------------------+-----------+-----------------------------+--------+ +``` + +**Example 3: Invalid Sort Column** + +`M4` requires `ORDER BY` on the `DATA` argument, and the sort column must match the time column specified by `TIMECOL`. The following example incorrectly sorts by `temperature`: + +```sql +SELECT * +FROM M4( + DATA => ( + SELECT time, device_id, temperature, humidity + FROM table1 + WHERE device_id = '101' + ) PARTITION BY device_id ORDER BY temperature, + TIMECOL => 'time', + SIZE => 3m +); +``` + +Error: + +```text +Msg: org.apache.iotdb.jdbc.IoTDBSQLException: 701: The ORDER BY clause of the DATA argument must contain exactly the time column specified by the TIMECOL argument. +``` + ## 2. `DIFF` Function ### 2.1 **Description:** @@ -521,27 +717,47 @@ IoTDB> SELECT first(time) as window_start, last(time) as window_end, stock_id, a #### Function Description -The CAPACITY function groups data into fixed-size windows, where each window contains up to SIZE rows. +`CAPACITY` divides data into windows by point count (row count), with at most `SIZE` rows per window. Since V2.0.12, the optional `SLIDE` argument specifies the number of rows between the start positions of adjacent windows. #### Function Definition ```sql -CAPACITY(data [PARTITION BY(pkeys, ...)] [ORDER BY(okeys, ...)], size) +CAPACITY( + DATA => + [PARTITION BY [, ...]] + [ORDER BY [, ...]], + SIZE => + [, SLIDE => ] +) ``` #### Parameter Description -| Parameter | Type | Attributes | Description | -| ----------- | -------- | ---------------------------- | -------------------------------------- | -| DATA | Table | SET SEMANTIC, PASS THROUGH | Input table with partition/sort keys | -| SIZE | Scalar | Long integer | Window size (row count) | +|Parameter|Type|Required|Description| +|---|---|---|---| +|`DATA`|Table|Yes|Input table or subquery. Supports `PARTITION BY` and `ORDER BY` and passes through input columns.| +|`SIZE`|`INT64` scalar|Yes|Window capacity. Must be positive and specifies the maximum number of rows in a window.| +|`SLIDE`|`INT64` scalar|No|Window sliding step. Must be positive when specified. Defaults to `SIZE` when omitted.| + +- With `PARTITION BY`, row numbers and window indices are calculated independently within each partition, starting at `0`. +- With `ORDER BY`, row numbers follow the specified sort order. +- Window indices start at `0`. Window `k` covers the row-number interval `[k * SLIDE, k * SLIDE + SIZE)`. +- Window behavior depends on the relationship between `SLIDE` and `SIZE`: + +|Relationship|Window Type|Behavior| +|---|---|---| +|`SLIDE < SIZE`|Overlapping windows|Adjacent windows overlap. An input row can belong to multiple windows and appear multiple times with different `window_index` values.| +|`SLIDE = SIZE`|Non-overlapping windows|Adjacent windows are contiguous. This matches the behavior of `CAPACITY(DATA => ..., SIZE => n)` in earlier versions.| +|`SLIDE > SIZE`|Gapped windows|Gaps exist between adjacent windows. Input rows that fall into a gap are omitted from the output.| #### Returned Results -The CAPACITY function returns: +`CAPACITY` returns the following columns: -* `window_index`: Window identifier -* Pass-through columns: All input columns from DATA +- `window_index`: window index of type `INT64`, starting at `0`. +- Columns passed through from `DATA`, with their original names and types. + +You can query the result directly or aggregate it by `window_index` and business grouping columns. #### Usage Example @@ -558,7 +774,7 @@ IoTDB> SELECT * FROM CAPACITY(DATA => bid PARTITION BY stock_id ORDER BY time, S | 1|2021-01-01T09:09:00.000+08:00| AAPL|102.0| +------------+-----------------------------+--------+-----+ --- Equivalent to tree model's GROUP BY COUNT when combined with GROUP BY +-- Combine with GROUP BY, equivalent to GROUP BY COUNT in the tree model IoTDB> SELECT first(time) as start_time, last(time) as end_time, stock_id, avg(price) as avg FROM CAPACITY(DATA => bid PARTITION BY stock_id ORDER BY time, SIZE => 2) GROUP BY window_index, stock_id; +-----------------------------+-----------------------------+--------+-----+ | start_time| end_time|stock_id| avg| @@ -568,6 +784,23 @@ IoTDB> SELECT first(time) as start_time, last(time) as end_time, stock_id, avg(p |2021-01-01T09:05:00.000+08:00|2021-01-01T09:07:00.000+08:00| AAPL|101.5| |2021-01-01T09:09:00.000+08:00|2021-01-01T09:09:00.000+08:00| AAPL|102.0| +-----------------------------+-----------------------------+--------+-----+ + +-- Use SLIDE to generate overlapping windows (SLIDE < SIZE) +IoTDB> SELECT * FROM CAPACITY(DATA => bid PARTITION BY stock_id ORDER BY time, SIZE => 2, SLIDE => 1); ++------------+-----------------------------+--------+-----+ +|window_index| time|stock_id|price| ++------------+-----------------------------+--------+-----+ +| 0|2021-01-01T09:05:00.000+08:00| AAPL|100.0| +| 0|2021-01-01T09:07:00.000+08:00| AAPL|103.0| +| 1|2021-01-01T09:07:00.000+08:00| AAPL|103.0| +| 1|2021-01-01T09:09:00.000+08:00| AAPL|102.0| +| 2|2021-01-01T09:09:00.000+08:00| AAPL|102.0| +| 0|2021-01-01T09:06:00.000+08:00| TESL|200.0| +| 0|2021-01-01T09:07:00.000+08:00| TESL|202.0| +| 1|2021-01-01T09:07:00.000+08:00| TESL|202.0| +| 1|2021-01-01T09:15:00.000+08:00| TESL|195.0| +| 2|2021-01-01T09:15:00.000+08:00| TESL|195.0| ++------------+-----------------------------+--------+-----+ ``` ### 3.5 TUMBLE @@ -693,3 +926,835 @@ IoTDB> SELECT window_start, window_end, stock_id, avg(price) as avg FROM CUMULAT |2021-01-01T09:00:00.000+08:00|2021-01-01T09:10:00.000+08:00| AAPL|101.66666666666667| +-----------------------------+-----------------------------+--------+------------------+ ``` + + +## 4. `read_tsfile` Table Function + +### 4.1 Description + +`read_tsfile` reads table-model data from external TsFile files or directories and returns a temporary table source. You can use `SELECT`, `WHERE`, `ORDER BY`, `LIMIT`, `OFFSET`, and aggregation as you would with an ordinary table for read-only analysis and temporary processing. Results are not written back to the external TsFiles, and internal IoTDB data is not modified. This function does not support writing, deleting, modifying, or registering external TsFiles. + +> Supported since V2.0.12 + +### 4.2 Function Definition + +```sql +SELECT * +FROM TABLE( + read_tsfile( + TABLE_NAME => '', + PATHS => '' + ) +); +``` + +### 4.3 Parameters + +|Parameter|Required|Description| +|---|---|---| +|`PATHS`|Yes|Path to a TsFile or directory. Pass a single path as a string. For multiple paths, separate them with commas within the same string, for example `'/data/a.tsfile,/data/b.tsfile,/data/history'`.
- For a file path, the system attempts to open the file directly.
- For a directory, the system recursively scans files with the `.tsfile` suffix and identifies valid TsFiles by the magic numbers at the beginning and end of each file. Files that are unrecognized, incomplete, not in TsFile format, or not TsFile V4 are excluded from the query.
Paths must be visible to the DataNode process executing the query, and that process must have read permission. Specifying any DataNode data directory causes the query to fail.| +|`TABLE_NAME`|No|Name of the table to read. When specified, only valid files containing the target table's schema are read. If none of the valid files contains that schema, the query fails.
You can omit this argument when the table name can be inferred uniquely: every valid file to be read must contain exactly one table, and the inferred table name must be the same in all files. The query fails if a valid file contains multiple tables or if the unique table names differ between valid files.| +|`DEVICE_METADATA_INFO_SWAP_THRESHOLD`|No|When both the number of files and the number of devices are large, some metadata may be written to temporary swap files to limit memory use. This `Long` value sets the size threshold that forces a swap. The default is 8 MB.| + +**Schema Merging Rules** + +When reading multiple TsFiles, the system determines the target table and merges its schemas. The merged schema is used only as the output schema for the current query and is not registered in database metadata. The rules are: + +- TAG columns are merged in order and must have consistent prefixes. Different TAG names at the same position cause an error; additional TAG columns may be appended at later positions. +- The time-column name must be consistent across files; conflicting names cause an error. +- FIELD column order does not matter. Files can contain different FIELD columns, which are merged incrementally. FIELD columns with the same name must have compatible data types; otherwise, an error is reported. +- Columns with the same name in different TsFiles must have the same column category; otherwise, an error is reported. + +**Returned Results** + +- Returns the data of the specified table. +- Output columns come from the merged schema of the target table in the external TsFiles. +- The time column uses the time-column name recorded in the TsFiles. +- TAG columns represent device identifiers. +- FIELD columns contain measurement data. + +### 4.4 Usage Examples + +Assume `/data/backup/1.tsfile` exists and contains the metadata and data of `table1` from [Sample Data](../Reference/Sample-Data.md). + +**Example 1: Infer the Table Name** + +This example requires `/data/backup/1.tsfile` to contain exactly one table. + +```sql +SELECT * +FROM TABLE( + read_tsfile( + PATHS => '/data/backup/1.tsfile' + ) +) +LIMIT 10; +``` + +Result: + +```text ++-----------------------------+------+--------+---------+--------+-----------+-----------+--------+------+-----------------------------+ +| time|region|plant_id|device_id|model_id|maintenance|temperature|humidity|status| arrival_time| ++-----------------------------+------+--------+---------+--------+-----------+-----------+--------+------+-----------------------------+ +|2024-11-29T10:00:00.000+08:00| 上海| 3001| 101| D| 360| 85.0| null| null|2024-11-29T10:00:13.000+08:00| +|2024-11-29T11:00:00.000+08:00| 上海| 3002| 100| E| 180| null| 45.1| true| null| +|2024-11-29T18:30:00.000+08:00| 上海| 3002| 100| E| 180| 90.0| 35.4| true|2024-11-29T18:30:15.000+08:00| +|2024-11-30T09:30:00.000+08:00| 上海| 3002| 101| F| 360| 90.0| 35.2| true| null| +|2024-11-30T14:30:00.000+08:00| 上海| 3002| 101| F| 360| 90.0| 34.8| true|2024-11-30T14:30:17.000+08:00| +|2024-11-26T13:37:00.000+08:00| 北京| 1001| 100| A| 180| 90.0| 35.1| true|2024-11-26T13:37:34.000+08:00| +|2024-11-26T13:38:00.000+08:00| 北京| 1001| 100| A| 180| 90.0| 35.1| true|2024-11-26T13:38:25.000+08:00| +|2024-11-28T08:00:00.000+08:00| 上海| 3001| 100| C| 90| 85.0| null| null|2024-11-28T08:00:09.000+08:00| +|2024-11-28T09:00:00.000+08:00| 上海| 3001| 100| C| 90| null| 40.9| true| null| +|2024-11-28T10:00:00.000+08:00| 上海| 3001| 100| C| 90| 85.0| 35.2| null|2024-11-28T10:00:11.000+08:00| ++-----------------------------+------+--------+---------+--------+-----------+-----------+--------+------+-----------------------------+ +Total line number = 10 +It costs 0.159s +``` + +**Example 2: Read TsFiles from a Directory** + +```sql +SELECT time, device_id, temperature +FROM TABLE( + read_tsfile( + TABLE_NAME => 'table1', + PATHS => '/data/backup' + ) +) +WHERE time >= 2024-11-28 00:00:00 AND time < 2024-11-29 00:00:00 ; +``` + +Result: + +```text ++-----------------------------+---------+-----------+ +| time|device_id|temperature| ++-----------------------------+---------+-----------+ +|2024-11-28T08:00:00.000+08:00| 100| 85.0| +|2024-11-28T09:00:00.000+08:00| 100| null| +|2024-11-28T10:00:00.000+08:00| 100| 85.0| +|2024-11-28T11:00:00.000+08:00| 100| 88.0| ++-----------------------------+---------+-----------+ +Total line number = 4 +It costs 0.116s +``` + +**Example 3: Aggregate by TAG** + +```sql +SELECT device_id, count(temperature) +FROM TABLE( + read_tsfile( + TABLE_NAME => 'table1', + PATHS => '/data/backup' + ) +) +GROUP BY device_id; +``` + +Result: + +```text ++---------+-----+ +|device_id|_col1| ++---------+-----+ +| 100| 6| +| 101| 6| ++---------+-----+ +Total line number = 2 +It costs 0.049s +``` + +## 5. Time-Series Trend Matching Table Function + +### 5.1 Description + +The `pattern_match` table function searches a historical time series for subsequences that match an input pattern sequence and returns all matches satisfying the specified threshold, result bounds, and other conditions. + +> Supported since V2.0.12 + +### 5.2 Function Definition + +```sql +pattern_match( + data => {table_name | subquery} PARTITION BY partition_col ORDER BY time_col, + [time_col => 'time_column_name',] + data_col => 'data_column_name', + pattern => 'pattern_sequence', + smooth => numeric_value, + threshold => numeric_value, + [width => numeric_value,] + [height => numeric_value,] + [smooth_on_pattern => {true | false}] +) +``` + +### 5.3 Parameters + +|Parameter|Type|Required|Default|Description| +|---|---|---|---|---| +|`data`|Table name or subquery|Yes|-|Table or subquery containing the time series. **The time column must be ordered.** Supports `PARTITION BY` for partitioning and `ORDER BY` for sorting by time.| +|`time_col`|String|No|`'time'`|Name of the time column. **Only the TIMESTAMP type is supported.**| +|`data_col`|String|Yes|-|Name of the series column. Supported types: `INT32`, `INT64`, `DOUBLE`, and `FLOAT`.| +|`pattern`|String|Yes|-|Numeric sequence to match, supporting ordinary trend sequences and regular-expression rules:
1. Basic format: comma-separated integers or floating-point values, such as `'1.0,2.0,3.0'`.
2. Regular-expression extensions: `{segment}+` repeats a segment at least once, and `{segment}*` repeats it zero or more times. `{`, `}*`, and `}+` are separate units.
3. The leftmost point of each segment is shared with the preceding segment.| +|`smooth`|Non-negative real number|Yes|-|Tolerance for fluctuations (smoothing level). Larger values tolerate greater fluctuations but may oversmooth normal data. A recommended starting value is 1% of the value range; adjust it to the series.| +|`threshold`|Non-negative real number|Yes|-|Similarity threshold. Only matches with a similarity value below this threshold are returned. Smaller values require a closer match. Start at `1` and increase as needed.| +|`width`|Non-negative real number|No|Unlimited|Maximum width of a match. Only subsequences with a timestamp span below this value are returned.| +|`height`|Non-negative real number|No|Unlimited|Maximum height of a match. Only subsequences with a height below this value are returned.| +|`smooth_on_pattern`|Boolean|No|`false`|Whether to smooth the pattern:
1. `false`: treats the pattern as a trend shape and smooths only the original series.
2. `true`: applies the same smoothing and noise reduction to both the pattern and the original series.| + +**Returned Results** + +Only rows belonging to matching segments are retained, in ascending `match_index` order. Unmatched rows are discarded. + +|Column|Type|Description| +|---|---|---| +|`match_index`|`INT32`|Non-negative integer identifying the matched segment.| +|`similarity`|`DOUBLE`|Non-negative similarity value for the segment and pattern. Smaller values indicate closer matches.| +|All columns from `data`|Same as input|Original data from all columns of the input `data` argument.| + +### 5.4 Usage Examples + +The following examples demonstrate basic pattern matching, parameter adjustment, regular-expression patterns, and smoothing. + +#### 5.4.1 Prepare Test Data + +Create a table and insert data: + +```sql +create table test(value DOUBLE FIELD, value1 DOUBLE FIELD); +INSERT INTO test VALUES (1, 1, 0),(2, 2, 0),(3, 1, 0),(4, 1, 0),(5, 1, 0),(6, 1.2, 0),(7, 1, 0),(8, 1, 0),(9, 2, 0),(10, 3, 0),(11, 4, 0),(12, 3, 0),(13, 2, 0),(14, 3, 0),(15, 4, 0),(16, 2, 0),(17, 1, 0),(18, 1, 0),(19, 1, 0),(20, 1, 0),(21, 1, 0),(41, 1, 0),(42, 2, 0),(43, 3, 0),(44, 4, 0),(45, 3, 0),(46, 2, 0),(47, 2, 0),(48, 2, 0),(49, 2, 0),(50, 2, 0),(51, 2, 0),(52, 2, 0); +``` + +Query the data: + +```sql +select * from test; +``` + +The data is shown below: + +![](/img/featured-function-01-en.png) + +#### 5.4.2 Basic Trend Matching + +Use an ordinary sequence as the pattern to find subsequences with a similar trend. + +```sql +select * from pattern_match( + data => test ORDER BY time, + time_col => 'time', + data_col => 'value', + pattern => '1.0,2.0,3.0,2.0,1.0,1.0,1.0,1.0,1.0', + smooth => 0.5, + threshold => 10.0, + width => 1000.0, + height => 500.0, + smooth_on_pattern=> false +); +``` + +The matching results are shown below: + +![](/img/featured-function-02-en.png) + +The query returns 3 matching segments. A smaller similarity value indicates a closer match. + +```text ++-----------+------------------+-----------------------------+-----+------+ +|match_index| similarity| time|value|value1| ++-----------+------------------+-----------------------------+-----+------+ +| 0| 4.637180787192477|1970-01-01T08:00:00.001+08:00| 1.0| 0.0| +| 0| 4.637180787192477|1970-01-01T08:00:00.002+08:00| 2.0| 0.0| +| 0| 4.637180787192477|1970-01-01T08:00:00.003+08:00| 1.0| 0.0| +| 0| 4.637180787192477|1970-01-01T08:00:00.004+08:00| 1.0| 0.0| +| 0| 4.637180787192477|1970-01-01T08:00:00.005+08:00| 1.0| 0.0| +| 0| 4.637180787192477|1970-01-01T08:00:00.006+08:00| 1.2| 0.0| +| 0| 4.637180787192477|1970-01-01T08:00:00.007+08:00| 1.0| 0.0| +| 0| 4.637180787192477|1970-01-01T08:00:00.008+08:00| 1.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.013+08:00| 2.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.014+08:00| 3.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.015+08:00| 4.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.016+08:00| 2.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.017+08:00| 1.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.018+08:00| 1.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.019+08:00| 1.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.020+08:00| 1.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.021+08:00| 1.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.041+08:00| 1.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.042+08:00| 2.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.043+08:00| 3.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.044+08:00| 4.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.045+08:00| 3.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.046+08:00| 2.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.047+08:00| 2.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.048+08:00| 2.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.049+08:00| 2.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.050+08:00| 2.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.051+08:00| 2.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.052+08:00| 2.0| 0.0| ++-----------+------------------+-----------------------------+-----+------+ +Total line number = 29 +It costs 0.126s +``` + +#### 5.4.3 Adjust Matching Conditions + +Each example below starts from the query in Section 5.4.2 and changes only the indicated argument. + +|Purpose|Parameter|Original Value|New Value|Effect| +|---|---|---|---|---| +|Require a closer match|`threshold`|`10.0`|`1.0`|Retains only results with similarity below `1.0`, reducing the number of matching segments from 3 to 2.| +|Limit the time span|`width`|`1000.0`|`10.0`|Returns only subsequences with a timestamp span below `10.0`.| +|Limit the value range|`height`|`500.0`|`2.0`|Returns only subsequences with a value range below `2.0`.| + +#### 5.4.4 Regular-Expression Patterns + +Regular-expression patterns can match repeated trend segments. + +|Repetition Symbol|Meaning| +|---|---| +|`+`|The preceding subsequence occurs at least once.| +|`*`|The preceding subsequence occurs zero or more times.| + +The following example uses `+` to match one or more occurrences of `2.0,3.0,2.0`: + +```sql +select * from pattern_match( + data => test ORDER BY time, + time_col => 'time', + data_col => 'value', + pattern => '1.0,{2.0,3.0,2.0}+,1.0,1.0,1.0,1.0,1.0', + smooth => 0.5, + threshold => 100.0, + width => 1000.0, + height => 500.0, + smooth_on_pattern => false +); +``` + +To allow zero or more occurrences of `2.0,3.0,2.0`, replace `+` with `*`: + +```sql +pattern => '1.0,{2.0,3.0,2.0}*,1.0,1.0,1.0,1.0,1.0' +``` + +Regular-expression patterns are useful for periodic fluctuations, repeated peaks, and trend matches of variable length. + +#### 5.4.5 Pattern Smoothing + +When the pattern contains small fluctuations, use `smooth_on_pattern` to control whether the pattern is also smoothed. + +The following examples show the results of two configurations: + +|Configuration|`threshold`|`smooth_on_pattern`|Matching Behavior| +|---|---|---|---| +|Smooth both the original series and the pattern|`1.0`|`true`|Matches the `1.0,1.2,1.0` fluctuation in the original data.| +|Smooth only the original series|`10.0`|`false`|Returns other trend matches but excludes the corresponding `1.0,1.2,1.0` fluctuation.| + +Smooth both the original series and the pattern: + +```sql +SELECT * +FROM pattern_match( + data => test ORDER BY time, + time_col => 'time', + data_col => 'value', + pattern => '1.0,1.2,1.0', + smooth => 0.5, + threshold => 1.0, + width => 1000.0, + height => 500.0, + smooth_on_pattern => true +); +``` + +With the same smoothing rules applied to both sequences, the query matches the `1.0,1.2,1.0` fluctuation at time points 5, 6, and 7. + +The results are shown below: + +![](/img/featured-function-03-en.png) + +Smooth only the original series: + +```sql +SELECT * +FROM pattern_match( + data => test ORDER BY time, + time_col => 'time', + data_col => 'value', + pattern => '1.0,1.2,1.0', + smooth => 0.5, + threshold => 10.0, + width => 1000.0, + height => 500.0, + smooth_on_pattern => false +); +``` + +With smoothing applied only to the original series, the query still returns other trend matches, but the `1.0,1.2,1.0` fluctuation at time points 5, 6, and 7 is not matched. + +The results are shown below: + +![](/img/featured-function-04-en.png) + +## 6. Frequency-Domain Analysis Functions + +### 6.1 FFT + +#### 6.1.1 Description + +`FFT` is a built-in table function in the table model. It performs a one-dimensional fast Fourier transform on one or more real-valued numeric series in each input partition. `ORDER BY` must specify the time column in ascending order. You can also use `PARTITION BY` to group data by device or TAG. Each partition generates its own frequency bins, and all numeric columns in the partition share the same frequency axis. + +> Supported since V2.0.12 + +#### 6.1.2 Function Definition + +Named-argument syntax: + +```sql +SELECT * +FROM FFT( + DATA => {table_name | subquery} + [PARTITION BY partition_column [, ...]] + ORDER BY time_column, + [SAMPLE_INTERVAL => duration_literal,] + [N => positive_integer,] + [NORM => {'backward' | 'forward' | 'ortho'},] + [TIMECOL => 'time_column'] +); +``` + +Positional-argument syntax: + +```sql +SELECT * +FROM TABLE( + FFT( + TABLE(table_name) + [PARTITION BY partition_column [, ...]] + ORDER BY time_column + [, duration_literal + [, positive_integer + [, norm + [, time_column_name]]]] + ) +); +``` + +The canonical argument order is `DATA, SAMPLE_INTERVAL, N, NORM, TIMECOL`. **Named arguments** are recommended to make complex calls easier to read. + +#### 6.1.3 Parameters + +|Parameter|Required|Type|Default|Description| +|---|---|---|---|---| +|`DATA`|Yes|Table argument|None|Input data for FFT. With `PARTITION BY`, each partition is calculated independently; without it, all input rows form one partition. All `INT32`, `INT64`, `FLOAT`, and `DOUBLE` columns except the time and partition columns participate in the calculation. Other column types are ignored. At least one numeric column is required, and participating columns cannot contain `NULL`. To transform only selected numeric columns, project them in an input subquery.| +|`SAMPLE_INTERVAL`|No|Positive duration literal|Inferred from the time column|Sampling interval used to calculate the frequency axis, such as `1ms`, `10ms`, or `1s`. If omitted, it is inferred per partition as `(last timestamp - first timestamp) / (input row count - 1)`. A single-row partition requires an explicit interval; an empty partition returns 0 rows.| +|`N`|No|Positive integer|Input row count of the current partition|FFT length. If smaller than the input row count, only the first `N` rows in timestamp order are transformed. If larger, the series is padded with zeros at the end. Each partition returns `N` rows. The maximum `N` is `65536`, and `N × 2 × number of participating numeric columns ≤ 16777216` must hold.| +|`NORM`|No|String|`backward`|Normalization mode, case-insensitive. `backward`: no normalization; `forward`: divide by `N`; `ortho`: divide by `√N`.| +|`TIMECOL`|No|String|`time`|Name of the time column, which must have type `TIMESTAMP`. `DATA` must be sorted only by this column in ascending order; `DESC` is unsupported. Timestamps within each partition must be strictly increasing, with no duplicates.| + +- FFT is generally intended for evenly sampled data. This function does not automatically detect or correct irregular sampling. Interpolate or resample the data before calculation to ensure accurate spectra. If the sampling interval is known, explicitly set `SAMPLE_INTERVAL`. +- When `N` is smaller than the input row count, the function still validates all input rows in the partition, although only the first `N` rows are transformed. + +**Returned Results** + +`FFT` returns the full two-sided spectrum for each partition, with columns in the following order: + +```text +partition_columns... +frequency_index +frequency +_real +_imag +... +``` + +|Output Column|Type|Description| +|---|---|---| +|Columns specified by `PARTITION BY`|Same as input|Retain their original names and types and appear first in the result. Omitted when `PARTITION BY` is not used.| +|`frequency_index`|`INT64`|Frequency-bin index, from `0` to `N-1`.| +|`frequency`|`DOUBLE`|Frequency of the current bin, in Hz.| +|`_real`|`DOUBLE`|Real part of the FFT for the corresponding numeric input column.| +|`_imag`|`DOUBLE`|Imaginary part of the FFT for the corresponding numeric input column.| + +For transform length `N` and sampling interval `d` in seconds, `frequency` is calculated as follows: + +|`frequency_index` Range|`frequency`| +|---|---| +|`frequency_index < ceil(N / 2)`|`frequency_index / (N × d)`| +|Otherwise|`(frequency_index - N) / (N × d)`| + +Frequencies are ordered as zero, positive, then negative frequencies, in Hz. All numeric columns within a partition share the same frequency axis. + +#### 6.1.4 Notes + +- The function name `FFT` is case-insensitive. Each partition performs a separate one-dimensional FFT. +- The current version supports only real-valued input. Complex input, `DFT`, and `IFFT` are unsupported. +- The current version returns only the real and imaginary parts of the full spectrum. One-sided spectra, amplitude output, and phase output are unsupported. +- The `VALUE`, `SAMPLE_RATE`, and `SPECTRUM` arguments are unsupported in the current version. + +#### 6.1.5 Usage Examples + +These examples use `table1`. Because `temperature` contains `NULL` values, filter them out in the input subquery. + +**Example 1: FFT by Device** + +```sql +SELECT * +FROM FFT( + DATA => ( + SELECT time, device_id, temperature + FROM table1 + WHERE plant_id = '1001' + AND temperature IS NOT NULL + ) PARTITION BY device_id ORDER BY time +) +ORDER BY device_id, frequency_index; +``` + +The function transforms `temperature` separately for each device. With `N` and `SAMPLE_INTERVAL` omitted, the transform length is the input row count of each partition, and the sampling interval is inferred from the time column. In this example, the sampling interval is 1 minute. + +```text ++---------+---------------+---------------------+----------------------+---------------------+ +|device_id|frequency_index| frequency| temperature_real| temperature_imag| ++---------+---------------+---------------------+----------------------+---------------------+ +| 100| 0| 0.0| 180.0| 0.0| +| 100| 1|-0.008333333333333333| 0.0| 0.0| +| 101| 0| 0.0| 255.0| -6.67572021484375E-6| +| 101| 1| 0.005555555555555556|-4.6126147026370745E-6|-5.266308107820805E-6| +| 101| 2|-0.005555555555555556| -1.9073486328125E-6| 3.303624680484063E-6| ++---------+---------------+---------------------+----------------------+---------------------+ +``` + +**Example 2: Specify Transform Length and Normalization** + +```sql +SELECT * +FROM FFT( + DATA => ( + SELECT time, device_id, temperature + FROM table1 + WHERE plant_id = '1001' + AND temperature IS NOT NULL + ) PARTITION BY device_id ORDER BY time, + SAMPLE_INTERVAL => 1m, + N => 4, + NORM => 'ortho' +) +ORDER BY device_id, frequency_index; +``` + +The transform length is set to 4 for each device. Series with fewer than 4 input rows are padded with zeros at the end. FFT results are normalized by `1/√N`. Each device returns 4 spectrum rows. + +```text ++---------+---------------+---------------------+----------------------+----------------+ +|device_id|frequency_index| frequency| temperature_real|temperature_imag| ++---------+---------------+---------------------+----------------------+----------------+ +| 100| 0| 0.0| 90.0| 0.0| +| 100| 1| 0.004166666666666667| 45.0| -45.0| +| 100| 2|-0.008333333333333333| 0.0| 0.0| +| 100| 3|-0.004166666666666667| 45.0| 45.0| +| 101| 0| 0.0| 127.5| 0.0| +| 101| 1| 0.004166666666666667| 2.602374475716698E-15| -42.5| +| 101| 2|-0.008333333333333333| 42.5| 0.0| +| 101| 3|-0.004166666666666667|-2.602374475716698E-15| 42.5| ++---------+---------------+---------------------+----------------------+----------------+ +``` + +**Example 3: Numeric Column Contains `NULL`** + +```sql +SELECT * +FROM FFT( + DATA => ( + SELECT time, device_id, temperature + FROM table1 + WHERE plant_id = '1001' + ) PARTITION BY device_id ORDER BY time +); +``` + +The query fails because numeric columns participating in FFT cannot contain `NULL`: + +```text +Msg: org.apache.iotdb.jdbc.IoTDBSQLException: 701: FFT does not support null values in column [temperature]. +``` + +Filter out null values with `IS NOT NULL` in the input subquery before running FFT. + +### 6.2 lowPass + +#### 6.2.1 Description + +`lowPass` is a low-pass filter that applies frequency-domain filtering to numeric columns in the input table. It retains low-frequency components and attenuates high-frequency components, making it useful for signal smoothing and high-frequency noise removal. The normalized passband frequency `wpass`, in the range `(0, 1)`, specifies the upper frequency bound of the passband. + +> Supported since V2.0.12 + +#### 6.2.2 Function Definition + +Named-argument syntax: + +```sql +SELECT * +FROM lowPass( + DATA => input_table PARTITION BY partition_column ORDER BY time_column, + TIMECOL => 'time_column', + WPASS => passband_frequency +); +``` + +Positional-argument syntax: + +```sql +SELECT * +FROM lowPass( + input_table PARTITION BY partition_column ORDER BY time_column, + 'time_column', + passband_frequency +); +``` + +#### 6.2.3 Parameters + +|Parameter|Type|Required|Description| +|---|---|---|---| +|`DATA`|Table argument (`SET SEMANTIC`)|Yes|Input table or subquery. `ORDER BY` must contain only the time column in **ascending order**. `PARTITION BY` is optional; if omitted, the entire table is treated as one partition.| +|`TIMECOL`|String|Yes|Name of the time column. Must identify exactly one column of type `TIMESTAMP`.| +|`WPASS`|`DOUBLE`|Yes|Normalized passband frequency in `(0, 1)`. Components below this frequency are retained, and those above it are attenuated.| + +**Calculation Rules** + +- Columns specified by `TIMECOL` and `PARTITION BY` are excluded from calculations but retained in the output. +- Apart from the time and partition columns, only `INT32`, `INT64`, `FLOAT`, and `DOUBLE` columns can participate. Analysis fails if no eligible column remains or if any other column type is present. +- `NULL` values in a participating column are excluded from that column's filtering calculation. + +**Returned Results** + +- The output includes the partition columns, if specified, and the time column, with their original names and types. +- Each participating numeric column produces a filtered result column of type `DOUBLE`. Each result column has the same number of points as its corresponding input column, output in ascending timestamp order. + +#### 6.2.4 Usage Example + +The following example uses `table1`. + +```sql +SELECT * +FROM lowPass( + DATA => ( + SELECT time, device_id, temperature + FROM table1 + WHERE temperature is not null + ) PARTITION BY device_id ORDER BY time, + TIMECOL => 'time', + WPASS => 0.3 +); +``` + +The example partitions data by device, applies low-pass filtering to `temperature`, and returns `device_id`, `time`, and the filtered numeric column. + +```text ++---------+-----------------------------+--------------------+ +|device_id| time|lowpass(temperature)| ++---------+-----------------------------+--------------------+ +| 100|2024-11-26T13:37:00.000+08:00| 88.0| +| 100|2024-11-26T13:38:00.000+08:00| 88.0| +| 100|2024-11-28T08:00:00.000+08:00| 87.99999999999999| +| 100|2024-11-28T10:00:00.000+08:00| 87.99999999999999| +| 100|2024-11-28T11:00:00.000+08:00| 88.00000000000001| +| 100|2024-11-29T18:30:00.000+08:00| 88.0| +| 101|2024-11-27T16:39:00.000+08:00| 86.66666666666667| +| 101|2024-11-27T16:40:00.000+08:00| 86.66666666666667| +| 101|2024-11-27T16:41:00.000+08:00| 86.66666666666664| +| 101|2024-11-29T10:00:00.000+08:00| 86.66666666666664| +| 101|2024-11-30T09:30:00.000+08:00| 86.66666666666667| +| 101|2024-11-30T14:30:00.000+08:00| 86.66666666666664| ++---------+-----------------------------+--------------------+ +``` + +### 6.3 highPass + +#### 6.3.1 Description + +`highPass` is a high-pass filter that applies frequency-domain filtering to numeric columns in the input table. It retains high-frequency components and attenuates low-frequency components, making it useful for extracting rapid changes, abrupt transitions, and high-frequency noise. The normalized passband frequency `wpass`, in the range `(0, 1)`, specifies the lower frequency bound of the passband. + +> Supported since V2.0.12 + +#### 6.3.2 Function Definition + +Named-argument syntax: + +```sql +SELECT * +FROM highPass( + DATA => input_table PARTITION BY partition_column ORDER BY time_column, + TIMECOL => 'time_column', + WPASS => passband_frequency +); +``` + +Positional-argument syntax: + +```sql +SELECT * +FROM highPass( + input_table PARTITION BY partition_column ORDER BY time_column, + 'time_column', + passband_frequency +); +``` + +#### 6.3.3 Parameters + +|Parameter|Type|Required|Description| +|---|---|---|---| +|`DATA`|Table argument (`SET SEMANTIC`)|Yes|Input table or subquery. `ORDER BY` must contain only the time column in ascending order. `PARTITION BY` is optional; if omitted, the entire table is treated as one partition.| +|`TIMECOL`|String|Yes|Name of the time column. Must identify exactly one column of type `TIMESTAMP`.| +|`WPASS`|`DOUBLE`|Yes|Normalized passband frequency in `(0, 1)`. Components above this frequency are retained, and those below it are attenuated.| + +**Input Columns and Calculation Rules** + +- Columns specified by `TIMECOL` and `PARTITION BY` are excluded from calculations but retained in the output. +- Apart from the time and partition columns, only `INT32`, `INT64`, `FLOAT`, and `DOUBLE` columns can participate. Analysis fails if no eligible column remains or if any other column type is present. +- `NULL` values in a participating column are excluded from that column's filtering calculation. + +**Returned Results** + +- The output includes the partition columns, if specified, and the time column, with their original names and types. +- Each participating numeric column produces a filtered result column of type `DOUBLE`. Each result column has the same number of points as its corresponding input column, output in ascending timestamp order. + +#### 6.3.4 Usage Example + +```sql +SELECT * +FROM highPass( + DATA => ( + SELECT time, device_id, temperature + FROM table1 + WHERE temperature is not null + ) PARTITION BY device_id ORDER BY time, + TIMECOL => 'time', + WPASS => 0.3 +); +``` + +The example partitions data by device, applies high-pass filtering to `temperature`, and returns `device_id`, `time`, and the filtered numeric column. + +```text ++---------+-----------------------------+----------------------+ +|device_id| time| highpass(temperature)| ++---------+-----------------------------+----------------------+ +| 100|2024-11-26T13:37:00.000+08:00| 2.000000000000029| +| 100|2024-11-26T13:38:00.000+08:00| 2.000000000000004| +| 100|2024-11-28T08:00:00.000+08:00| -2.9999999999999702| +| 100|2024-11-28T10:00:00.000+08:00| -3.000000000000013| +| 100|2024-11-28T11:00:00.000+08:00|-4.182831670630972E-14| +| 100|2024-11-29T18:30:00.000+08:00| 1.9999999999999953| +| 101|2024-11-27T16:39:00.000+08:00| -1.666666666666642| +| 101|2024-11-27T16:40:00.000+08:00| -1.6666666666666519| +| 101|2024-11-27T16:41:00.000+08:00| -1.666666666666645| +| 101|2024-11-29T10:00:00.000+08:00| -1.666666666666701| +| 101|2024-11-30T09:30:00.000+08:00| 3.3333333333333015| +| 101|2024-11-30T14:30:00.000+08:00| 3.333333333333338| ++---------+-----------------------------+----------------------+ +``` + +## 7. Time-Series Correlation Analysis Functions + +### 7.1 Xcorr + +#### 7.1.1 Description + +`Xcorr` performs cross-correlation analysis on two numeric series to measure their shape similarity at different time lags. It shifts one series relative to the other, multiplies corresponding points, and sums the products to produce a similarity curve. This can reveal delays and similar patterns between two signals. + +> Supported since V2.0.12 + +#### 7.1.2 Function Definition + +Named-argument syntax: + +```sql +SELECT * +FROM Xcorr( + DATA => ( + SELECT time_column, signal_1, signal_2, partition_column + FROM input_table + ) PARTITION BY partition_column ORDER BY time_column, + TIMECOL => 'time_column' +); +``` + +Positional-argument syntax: + +```sql +SELECT * +FROM Xcorr( + ( SELECT time_column, signal_1, signal_2, partition_column + FROM input_table + ) PARTITION BY partition_column ORDER BY time_column, + 'time_column' +); +``` + +#### 7.1.3 Parameters + +|Parameter|Type|Required|Description| +|---|---|---|---| +|`DATA`|Table argument (`SET SEMANTIC`)|Yes|Input table or subquery. `ORDER BY` must specify the time column in ascending order. `PARTITION BY` is optional. Exactly two numeric columns must remain after excluding the time and partition columns.| +|`TIMECOL`|String|Yes|Name of the time column. Must identify exactly one column of type `TIMESTAMP`.| + +**Calculation Rules** + +- The time and partition columns are excluded from cross-correlation calculations. +- Exactly two columns must remain after those columns are excluded. Both must be of type `INT32`, `INT64`, `FLOAT`, or `DOUBLE`; otherwise, analysis fails. +- `NULL` values in either participating column are treated as `0` at that position. +- Timestamps must be strictly increasing within each partition. +- Normalization is applied according to the function definition. + +**Returned Results** + +- Partition columns specified by `PARTITION BY`, if any, retain their original names and types. +- One result column of type `DOUBLE` is generated for the two participating columns. + +#### 7.1.4 Usage Example + +The following example uses `table1` from the SQL manual's sample data. A subquery selects the two numeric series, `temperature` and `humidity`, for cross-correlation. + +```sql +SELECT * +FROM Xcorr( + DATA => ( + SELECT time, device_id, temperature, humidity + FROM table1 + WHERE temperature IS NOT NULL + AND humidity IS NOT NULL + ) + PARTITION BY device_id + ORDER BY time, + TIMECOL => 'time' +); +``` + +The example partitions data by device, cross-correlates `temperature` and `humidity`, and returns `device_id` and the correlation result column. + +```text ++---------+----------------------------+ +|device_id|xcorr(temperature, humidity)| ++---------+----------------------------+ +| 100| 3186.0001373291016| +| 100| 3622.5| +| 100| 3412.0000203450522| +| 100| 3318.9249839782715| +| 100| 3292.9599586486815| +| 100| 3324.7749156951904| +| 100| 3080.0999348958335| +| 100| 3123.8998641967773| +| 100| 3158.9998626708984| +| 101| 2957.9999351501465| +| 101| 3061.9999980926514| +| 101| 3100.166645050049| +| 101| 3172.5| +| 101| 3176.999931335449| ++---------+----------------------------+ +``` diff --git a/src/UserGuide/latest-Table/SQL-Manual/Featured-Functions_apache.md b/src/UserGuide/latest-Table/SQL-Manual/Featured-Functions_apache.md index fd91d58ef..500cbfba4 100644 --- a/src/UserGuide/latest-Table/SQL-Manual/Featured-Functions_apache.md +++ b/src/UserGuide/latest-Table/SQL-Manual/Featured-Functions_apache.md @@ -210,6 +210,202 @@ GROUP BY 1, device_id; +---------+---------+--------+ ``` +### 1.3 `M4` Function + +#### 1.3.1 Description + +`M4` is a built-in polymorphic table function in the table model for downsampling time-series data. For each window, it extracts the first, last, minimum (bottom), and maximum (top) points to reduce the number of returned points while preserving the shape of the trend. `M4` takes a table argument and returns a result table, so it must be called in the `FROM` clause. + +- First point: the point with the earliest timestamp and a non-`NULL` value in the window. +- Last point: the point with the latest timestamp and a non-`NULL` value in the window. +- Minimum and maximum points: the points with the smallest and largest values in the window. If multiple points have the same value, the first point encountered in the input scan order is retained. +- Both time windows and point-count windows are supported. A duration literal for `SIZE`, such as `1h` or `10m`, selects time-window mode; a positive integer, such as `100`, selects point-count-window mode. + +> Supported since V2.0.12 + +#### 1.3.2 Function Definition + +```sql +SELECT * +FROM M4( + DATA => input_table PARTITION BY partition_column ORDER BY time_column, + TIMECOL => 'time_column', + [ORIGIN => origin_timestamp,] + SIZE => window_duration | window_point_count, + [SLIDE => window_duration | window_point_count] +); +``` + +`DATA` is a set-semantic table argument. `ORDER BY` is required and must contain only the time column specified by `TIMECOL`, in ascending order. `PARTITION BY` is optional; if omitted, the entire input table is treated as one partition. Arguments can be passed by name or in declaration order. Named arguments are recommended. + +#### 1.3.3 Parameters + +|Parameter|Type|Required|Description| +|---|---|---|---| +|`DATA`|Table argument (`SET SEMANTIC`)|Yes|Input table or subquery. Supports `PARTITION BY` and `ORDER BY`.| +|`TIMECOL`|String|Yes|Name of the time column. The column must be of type `TIMESTAMP`, and `ORDER BY` must contain only this column.| +|`ORIGIN`|Timestamp|No|Alignment origin for time windows. Defaults to `0`. Not used in point-count-window mode.| +|`SIZE`|Duration or `INT64`|Yes|Duration of a time window or number of points in a point-count window. Must be positive.| +|`SLIDE`|Same type as `SIZE`|No|Window sliding step. Defaults to `SIZE` and must use the same window mode as `SIZE`.| + +**Calculation Rules** + +- The column specified by `TIMECOL` is excluded from M4 calculations. +- Columns specified by `PARTITION BY` are excluded from calculations but retained in the output. +- All remaining columns of comparable types participate in M4 calculations. +- After excluding the time and partition columns, analysis fails if no comparable column remains or if any remaining column has a non-comparable type. +- Records with a `NULL` value in a participating column are excluded from that column's `first`, `last`, `bottom`, and `top` calculations. +- For each participating column, the `first`, `last`, `bottom`, and `top` points are calculated, deduplicated, and output in ascending timestamp order. + +**Returned Results** + +- Time-window mode always generates `window_start` and `window_end`, both of type `TIMESTAMP`, to represent the interval `[window_start, window_end)`. +- Point-count-window mode always generates `window_index`, of type `INT64`. Window indices are calculated independently within each `PARTITION BY` partition, starting at `0`. +- Partition columns retain their original names and types and appear in declaration order. `OBJECT` columns cannot be partition columns. `BLOB` partition columns can be retained in the output but do not participate in M4 calculations. +- For each participating column `c`, two output columns are generated: `c_time` for the point's timestamp and `c` for its value. Their types match the corresponding input types. Results for multiple columns are expanded in input-column order. + +The number of output rows in a window equals the largest number of feature points among all participating columns. For columns with fewer feature points, the remaining `c_time` and `c` cells are filled with `NULL`. + +#### 1.3.4 Usage Examples + +The following examples use `table1` from [Sample Data](../Reference/Sample-Data.md). Because `table1` contains additional columns besides the time and partition columns, a subquery selects only `time`, `device_id`, `temperature`, and `humidity` for M4 calculations. + +**Example 1: Time Windows** + +For `device_id = '101'`, divide the data into 3-minute windows. `SIZE => 3m` selects time-window mode with a duration of 3 minutes. Each half-open interval `[window_start, window_end)` extracts the first, last, minimum, and maximum points for `temperature` and `humidity`, then outputs the points in ascending timestamp order. + +```sql +SELECT window_start, + window_end, + device_id, + temperature_time, + temperature, + humidity_time, + humidity +FROM M4( + DATA => ( + SELECT time, device_id, temperature, humidity + FROM table1 + WHERE device_id = '101' + AND time >= 2024-11-27 16:38:00 + AND time <= 2024-11-27 16:44:00 + ) PARTITION BY device_id ORDER BY time, + TIMECOL => 'time', + SIZE => 3m +); +``` + +Result: + +```text ++-----------------------------+-----------------------------+---------+-----------------------------+-----------+-----------------------------+--------+ +| window_start| window_end|device_id| temperature_time|temperature| humidity_time|humidity| ++-----------------------------+-----------------------------+---------+-----------------------------+-----------+-----------------------------+--------+ +|2024-11-27T16:36:00.000+08:00|2024-11-27T16:39:00.000+08:00| 101| null| null|2024-11-27T16:38:00.000+08:00| 35.1| +|2024-11-27T16:39:00.000+08:00|2024-11-27T16:42:00.000+08:00| 101|2024-11-27T16:39:00.000+08:00| 85.0|2024-11-27T16:39:00.000+08:00| 35.3| +|2024-11-27T16:39:00.000+08:00|2024-11-27T16:42:00.000+08:00| 101|2024-11-27T16:41:00.000+08:00| 85.0| null| null| +|2024-11-27T16:42:00.000+08:00|2024-11-27T16:45:00.000+08:00| 101| null| null|2024-11-27T16:42:00.000+08:00| 35.2| ++-----------------------------+-----------------------------+---------+-----------------------------+-----------+-----------------------------+--------+ +``` + +**Example 2: Point-Count Windows** + +For `device_id = '101'`, divide the data into windows of 3 points. `SIZE => 3` selects point-count-window mode, with 3 input rows per window. `window_index` identifies each window and starts at `0` within each partition. + +```sql +SELECT window_index, + device_id, + temperature_time, + temperature, + humidity_time, + humidity +FROM M4( + DATA => ( + SELECT time, device_id, temperature, humidity + FROM table1 + WHERE device_id = '101' + AND time >= 2024-11-27 16:38:00 + AND time <= 2024-11-27 16:44:00 + ) PARTITION BY device_id ORDER BY time, + TIMECOL => 'time', + SIZE => 3 +); +``` + +Result: + +```text ++------------+---------+-----------------------------+-----------+-----------------------------+--------+ +|window_index|device_id| temperature_time|temperature| humidity_time|humidity| ++------------+---------+-----------------------------+-----------+-----------------------------+--------+ +| 0| 101|2024-11-27T16:39:00.000+08:00| 85.0|2024-11-27T16:38:00.000+08:00| 35.1| +| 0| 101|2024-11-27T16:40:00.000+08:00| 85.0|2024-11-27T16:39:00.000+08:00| 35.3| +| 1| 101|2024-11-27T16:41:00.000+08:00| 85.0|2024-11-27T16:42:00.000+08:00| 35.2| ++------------+---------+-----------------------------+-----------+-----------------------------+--------+ +``` + +To specify the sliding step, also set `SLIDE`: + +```sql +SELECT window_index, + device_id, + temperature_time, + temperature, + humidity_time, + humidity +FROM M4( + DATA => ( + SELECT time, device_id, temperature, humidity + FROM table1 + WHERE device_id = '101' + AND time >= 2024-11-27 16:38:00 + AND time <= 2024-11-27 16:44:00 + ) PARTITION BY device_id ORDER BY time, + TIMECOL => 'time', + SIZE => 3, + SLIDE => 2 +); +``` + +This call uses a window size of 3 points and advances the window by 2 points each time. + +Result: + +```text ++------------+---------+-----------------------------+-----------+-----------------------------+--------+ +|window_index|device_id| temperature_time|temperature| humidity_time|humidity| ++------------+---------+-----------------------------+-----------+-----------------------------+--------+ +| 0| 101|2024-11-27T16:39:00.000+08:00| 85.0|2024-11-27T16:38:00.000+08:00| 35.1| +| 0| 101|2024-11-27T16:40:00.000+08:00| 85.0|2024-11-27T16:39:00.000+08:00| 35.3| +| 1| 101|2024-11-27T16:40:00.000+08:00| 85.0|2024-11-27T16:42:00.000+08:00| 35.2| +| 1| 101|2024-11-27T16:41:00.000+08:00| 85.0| null| null| +| 2| 101| null| null|2024-11-27T16:42:00.000+08:00| 35.2| ++------------+---------+-----------------------------+-----------+-----------------------------+--------+ +``` + +**Example 3: Invalid Sort Column** + +`M4` requires `ORDER BY` on the `DATA` argument, and the sort column must match the time column specified by `TIMECOL`. The following example incorrectly sorts by `temperature`: + +```sql +SELECT * +FROM M4( + DATA => ( + SELECT time, device_id, temperature, humidity + FROM table1 + WHERE device_id = '101' + ) PARTITION BY device_id ORDER BY temperature, + TIMECOL => 'time', + SIZE => 3m +); +``` + +Error: + +```text +Msg: org.apache.iotdb.jdbc.IoTDBSQLException: 701: The ORDER BY clause of the DATA argument must contain exactly the time column specified by the TIMECOL argument. +``` + ## 2. `DIFF` Function ### 2.1 **Description:** @@ -521,27 +717,47 @@ IoTDB> SELECT first(time) as window_start, last(time) as window_end, stock_id, a #### Function Description -The CAPACITY function groups data into fixed-size windows, where each window contains up to SIZE rows. +`CAPACITY` divides data into windows by point count (row count), with at most `SIZE` rows per window. Since V2.0.12, the optional `SLIDE` argument specifies the number of rows between the start positions of adjacent windows. #### Function Definition ```sql -CAPACITY(data [PARTITION BY(pkeys, ...)] [ORDER BY(okeys, ...)], size) +CAPACITY( + DATA => + [PARTITION BY [, ...]] + [ORDER BY [, ...]], + SIZE => + [, SLIDE => ] +) ``` #### Parameter Description -| Parameter | Type | Attributes | Description | -| ----------- | -------- | ---------------------------- | -------------------------------------- | -| DATA | Table | SET SEMANTIC, PASS THROUGH | Input table with partition/sort keys | -| SIZE | Scalar | Long integer | Window size (row count) | +|Parameter|Type|Required|Description| +|---|---|---|---| +|`DATA`|Table|Yes|Input table or subquery. Supports `PARTITION BY` and `ORDER BY` and passes through input columns.| +|`SIZE`|`INT64` scalar|Yes|Window capacity. Must be positive and specifies the maximum number of rows in a window.| +|`SLIDE`|`INT64` scalar|No|Window sliding step. Must be positive when specified. Defaults to `SIZE` when omitted.| + +- With `PARTITION BY`, row numbers and window indices are calculated independently within each partition, starting at `0`. +- With `ORDER BY`, row numbers follow the specified sort order. +- Window indices start at `0`. Window `k` covers the row-number interval `[k * SLIDE, k * SLIDE + SIZE)`. +- Window behavior depends on the relationship between `SLIDE` and `SIZE`: + +|Relationship|Window Type|Behavior| +|---|---|---| +|`SLIDE < SIZE`|Overlapping windows|Adjacent windows overlap. An input row can belong to multiple windows and appear multiple times with different `window_index` values.| +|`SLIDE = SIZE`|Non-overlapping windows|Adjacent windows are contiguous. This matches the behavior of `CAPACITY(DATA => ..., SIZE => n)` in earlier versions.| +|`SLIDE > SIZE`|Gapped windows|Gaps exist between adjacent windows. Input rows that fall into a gap are omitted from the output.| #### Returned Results -The CAPACITY function returns: +`CAPACITY` returns the following columns: -* `window_index`: Window identifier -* Pass-through columns: All input columns from DATA +- `window_index`: window index of type `INT64`, starting at `0`. +- Columns passed through from `DATA`, with their original names and types. + +You can query the result directly or aggregate it by `window_index` and business grouping columns. #### Usage Example @@ -558,7 +774,7 @@ IoTDB> SELECT * FROM CAPACITY(DATA => bid PARTITION BY stock_id ORDER BY time, S | 1|2021-01-01T09:09:00.000+08:00| AAPL|102.0| +------------+-----------------------------+--------+-----+ --- Equivalent to tree model's GROUP BY COUNT when combined with GROUP BY +-- Combine with GROUP BY, equivalent to GROUP BY COUNT in the tree model IoTDB> SELECT first(time) as start_time, last(time) as end_time, stock_id, avg(price) as avg FROM CAPACITY(DATA => bid PARTITION BY stock_id ORDER BY time, SIZE => 2) GROUP BY window_index, stock_id; +-----------------------------+-----------------------------+--------+-----+ | start_time| end_time|stock_id| avg| @@ -568,6 +784,23 @@ IoTDB> SELECT first(time) as start_time, last(time) as end_time, stock_id, avg(p |2021-01-01T09:05:00.000+08:00|2021-01-01T09:07:00.000+08:00| AAPL|101.5| |2021-01-01T09:09:00.000+08:00|2021-01-01T09:09:00.000+08:00| AAPL|102.0| +-----------------------------+-----------------------------+--------+-----+ + +-- Use SLIDE to generate overlapping windows (SLIDE < SIZE) +IoTDB> SELECT * FROM CAPACITY(DATA => bid PARTITION BY stock_id ORDER BY time, SIZE => 2, SLIDE => 1); ++------------+-----------------------------+--------+-----+ +|window_index| time|stock_id|price| ++------------+-----------------------------+--------+-----+ +| 0|2021-01-01T09:05:00.000+08:00| AAPL|100.0| +| 0|2021-01-01T09:07:00.000+08:00| AAPL|103.0| +| 1|2021-01-01T09:07:00.000+08:00| AAPL|103.0| +| 1|2021-01-01T09:09:00.000+08:00| AAPL|102.0| +| 2|2021-01-01T09:09:00.000+08:00| AAPL|102.0| +| 0|2021-01-01T09:06:00.000+08:00| TESL|200.0| +| 0|2021-01-01T09:07:00.000+08:00| TESL|202.0| +| 1|2021-01-01T09:07:00.000+08:00| TESL|202.0| +| 1|2021-01-01T09:15:00.000+08:00| TESL|195.0| +| 2|2021-01-01T09:15:00.000+08:00| TESL|195.0| ++------------+-----------------------------+--------+-----+ ``` ### 3.5 TUMBLE @@ -694,3 +927,834 @@ IoTDB> SELECT window_start, window_end, stock_id, avg(price) as avg FROM CUMULAT +-----------------------------+-----------------------------+--------+------------------+ ``` + +## 4. `read_tsfile` Table Function + +### 4.1 Description + +`read_tsfile` reads table-model data from external TsFile files or directories and returns a temporary table source. You can use `SELECT`, `WHERE`, `ORDER BY`, `LIMIT`, `OFFSET`, and aggregation as you would with an ordinary table for read-only analysis and temporary processing. Results are not written back to the external TsFiles, and internal IoTDB data is not modified. This function does not support writing, deleting, modifying, or registering external TsFiles. + +> Supported since V2.0.12 + +### 4.2 Function Definition + +```sql +SELECT * +FROM TABLE( + read_tsfile( + TABLE_NAME => '', + PATHS => '' + ) +); +``` + +### 4.3 Parameters + +|Parameter|Required|Description| +|---|---|---| +|`PATHS`|Yes|Path to a TsFile or directory. Pass a single path as a string. For multiple paths, separate them with commas within the same string, for example `'/data/a.tsfile,/data/b.tsfile,/data/history'`.
- For a file path, the system attempts to open the file directly.
- For a directory, the system recursively scans files with the `.tsfile` suffix and identifies valid TsFiles by the magic numbers at the beginning and end of each file. Files that are unrecognized, incomplete, not in TsFile format, or not TsFile V4 are excluded from the query.
Paths must be visible to the DataNode process executing the query, and that process must have read permission. Specifying any DataNode data directory causes the query to fail.| +|`TABLE_NAME`|No|Name of the table to read. When specified, only valid files containing the target table's schema are read. If none of the valid files contains that schema, the query fails.
You can omit this argument when the table name can be inferred uniquely: every valid file to be read must contain exactly one table, and the inferred table name must be the same in all files. The query fails if a valid file contains multiple tables or if the unique table names differ between valid files.| +|`DEVICE_METADATA_INFO_SWAP_THRESHOLD`|No|When both the number of files and the number of devices are large, some metadata may be written to temporary swap files to limit memory use. This `Long` value sets the size threshold that forces a swap. The default is 8 MB.| + +**Schema Merging Rules** + +When reading multiple TsFiles, the system determines the target table and merges its schemas. The merged schema is used only as the output schema for the current query and is not registered in database metadata. The rules are: + +- TAG columns are merged in order and must have consistent prefixes. Different TAG names at the same position cause an error; additional TAG columns may be appended at later positions. +- The time-column name must be consistent across files; conflicting names cause an error. +- FIELD column order does not matter. Files can contain different FIELD columns, which are merged incrementally. FIELD columns with the same name must have compatible data types; otherwise, an error is reported. +- Columns with the same name in different TsFiles must have the same column category; otherwise, an error is reported. + +**Returned Results** + +- Returns the data of the specified table. +- Output columns come from the merged schema of the target table in the external TsFiles. +- The time column uses the time-column name recorded in the TsFiles. +- TAG columns represent device identifiers. +- FIELD columns contain measurement data. + +### 4.4 Usage Examples + +Assume `/data/backup/1.tsfile` exists and contains the metadata and data of `table1` from [Sample Data](../Reference/Sample-Data.md). + +**Example 1: Infer the Table Name** + +This example requires `/data/backup/1.tsfile` to contain exactly one table. + +```sql +SELECT * +FROM TABLE( + read_tsfile( + PATHS => '/data/backup/1.tsfile' + ) +) +LIMIT 10; +``` + +Result: + +```text ++-----------------------------+------+--------+---------+--------+-----------+-----------+--------+------+-----------------------------+ +| time|region|plant_id|device_id|model_id|maintenance|temperature|humidity|status| arrival_time| ++-----------------------------+------+--------+---------+--------+-----------+-----------+--------+------+-----------------------------+ +|2024-11-29T10:00:00.000+08:00| 上海| 3001| 101| D| 360| 85.0| null| null|2024-11-29T10:00:13.000+08:00| +|2024-11-29T11:00:00.000+08:00| 上海| 3002| 100| E| 180| null| 45.1| true| null| +|2024-11-29T18:30:00.000+08:00| 上海| 3002| 100| E| 180| 90.0| 35.4| true|2024-11-29T18:30:15.000+08:00| +|2024-11-30T09:30:00.000+08:00| 上海| 3002| 101| F| 360| 90.0| 35.2| true| null| +|2024-11-30T14:30:00.000+08:00| 上海| 3002| 101| F| 360| 90.0| 34.8| true|2024-11-30T14:30:17.000+08:00| +|2024-11-26T13:37:00.000+08:00| 北京| 1001| 100| A| 180| 90.0| 35.1| true|2024-11-26T13:37:34.000+08:00| +|2024-11-26T13:38:00.000+08:00| 北京| 1001| 100| A| 180| 90.0| 35.1| true|2024-11-26T13:38:25.000+08:00| +|2024-11-28T08:00:00.000+08:00| 上海| 3001| 100| C| 90| 85.0| null| null|2024-11-28T08:00:09.000+08:00| +|2024-11-28T09:00:00.000+08:00| 上海| 3001| 100| C| 90| null| 40.9| true| null| +|2024-11-28T10:00:00.000+08:00| 上海| 3001| 100| C| 90| 85.0| 35.2| null|2024-11-28T10:00:11.000+08:00| ++-----------------------------+------+--------+---------+--------+-----------+-----------+--------+------+-----------------------------+ +Total line number = 10 +It costs 0.159s +``` + +**Example 2: Read TsFiles from a Directory** + +```sql +SELECT time, device_id, temperature +FROM TABLE( + read_tsfile( + TABLE_NAME => 'table1', + PATHS => '/data/backup' + ) +) +WHERE time >= 2024-11-28 00:00:00 AND time < 2024-11-29 00:00:00 ; +``` + +Result: + +```text ++-----------------------------+---------+-----------+ +| time|device_id|temperature| ++-----------------------------+---------+-----------+ +|2024-11-28T08:00:00.000+08:00| 100| 85.0| +|2024-11-28T09:00:00.000+08:00| 100| null| +|2024-11-28T10:00:00.000+08:00| 100| 85.0| +|2024-11-28T11:00:00.000+08:00| 100| 88.0| ++-----------------------------+---------+-----------+ +Total line number = 4 +It costs 0.116s +``` + +**Example 3: Aggregate by TAG** + +```sql +SELECT device_id, count(temperature) +FROM TABLE( + read_tsfile( + TABLE_NAME => 'table1', + PATHS => '/data/backup' + ) +) +GROUP BY device_id; +``` + +Result: + +```text ++---------+-----+ +|device_id|_col1| ++---------+-----+ +| 100| 6| +| 101| 6| ++---------+-----+ +Total line number = 2 +It costs 0.049s +``` + +## 5. Time-Series Trend Matching Table Function + +### 5.1 Description + +The `pattern_match` table function searches a historical time series for subsequences that match an input pattern sequence and returns all matches satisfying the specified threshold, result bounds, and other conditions. + +> Supported since V2.0.12 + +### 5.2 Function Definition + +```sql +pattern_match( + data => {table_name | subquery} PARTITION BY partition_col ORDER BY time_col, + [time_col => 'time_column_name',] + data_col => 'data_column_name', + pattern => 'pattern_sequence', + smooth => numeric_value, + threshold => numeric_value, + [width => numeric_value,] + [height => numeric_value,] + [smooth_on_pattern => {true | false}] +) +``` + +### 5.3 Parameters + +|Parameter|Type|Required|Default|Description| +|---|---|---|---|---| +|`data`|Table name or subquery|Yes|-|Table or subquery containing the time series. **The time column must be ordered.** Supports `PARTITION BY` for partitioning and `ORDER BY` for sorting by time.| +|`time_col`|String|No|`'time'`|Name of the time column. **Only the TIMESTAMP type is supported.**| +|`data_col`|String|Yes|-|Name of the series column. Supported types: `INT32`, `INT64`, `DOUBLE`, and `FLOAT`.| +|`pattern`|String|Yes|-|Numeric sequence to match, supporting ordinary trend sequences and regular-expression rules:
1. Basic format: comma-separated integers or floating-point values, such as `'1.0,2.0,3.0'`.
2. Regular-expression extensions: `{segment}+` repeats a segment at least once, and `{segment}*` repeats it zero or more times. `{`, `}*`, and `}+` are separate units.
3. The leftmost point of each segment is shared with the preceding segment.| +|`smooth`|Non-negative real number|Yes|-|Tolerance for fluctuations (smoothing level). Larger values tolerate greater fluctuations but may oversmooth normal data. A recommended starting value is 1% of the value range; adjust it to the series.| +|`threshold`|Non-negative real number|Yes|-|Similarity threshold. Only matches with a similarity value below this threshold are returned. Smaller values require a closer match. Start at `1` and increase as needed.| +|`width`|Non-negative real number|No|Unlimited|Maximum width of a match. Only subsequences with a timestamp span below this value are returned.| +|`height`|Non-negative real number|No|Unlimited|Maximum height of a match. Only subsequences with a height below this value are returned.| +|`smooth_on_pattern`|Boolean|No|`false`|Whether to smooth the pattern:
1. `false`: treats the pattern as a trend shape and smooths only the original series.
2. `true`: applies the same smoothing and noise reduction to both the pattern and the original series.| + +**Returned Results** + +Only rows belonging to matching segments are retained, in ascending `match_index` order. Unmatched rows are discarded. + +|Column|Type|Description| +|---|---|---| +|`match_index`|`INT32`|Non-negative integer identifying the matched segment.| +|`similarity`|`DOUBLE`|Non-negative similarity value for the segment and pattern. Smaller values indicate closer matches.| +|All columns from `data`|Same as input|Original data from all columns of the input `data` argument.| + +### 5.4 Usage Examples + +The following examples demonstrate basic pattern matching, parameter adjustment, regular-expression patterns, and smoothing. + +#### 5.4.1 Prepare Test Data + +Create a table and insert data: + +```sql +create table test(value DOUBLE FIELD, value1 DOUBLE FIELD); +INSERT INTO test VALUES (1, 1, 0),(2, 2, 0),(3, 1, 0),(4, 1, 0),(5, 1, 0),(6, 1.2, 0),(7, 1, 0),(8, 1, 0),(9, 2, 0),(10, 3, 0),(11, 4, 0),(12, 3, 0),(13, 2, 0),(14, 3, 0),(15, 4, 0),(16, 2, 0),(17, 1, 0),(18, 1, 0),(19, 1, 0),(20, 1, 0),(21, 1, 0),(41, 1, 0),(42, 2, 0),(43, 3, 0),(44, 4, 0),(45, 3, 0),(46, 2, 0),(47, 2, 0),(48, 2, 0),(49, 2, 0),(50, 2, 0),(51, 2, 0),(52, 2, 0); +``` + +Query the data: + +```sql +select * from test; +``` + +The data is shown below: + +![](/img/featured-function-01-en.png) + +#### 5.4.2 Basic Trend Matching + +Use an ordinary sequence as the pattern to find subsequences with a similar trend. + +```sql +select * from pattern_match( + data => test ORDER BY time, + time_col => 'time', + data_col => 'value', + pattern => '1.0,2.0,3.0,2.0,1.0,1.0,1.0,1.0,1.0', + smooth => 0.5, + threshold => 10.0, + width => 1000.0, + height => 500.0, + smooth_on_pattern=> false +); +``` + +The matching results are shown below: + +![](/img/featured-function-02-en.png) + +The query returns 3 matching segments. A smaller similarity value indicates a closer match. + +```text ++-----------+------------------+-----------------------------+-----+------+ +|match_index| similarity| time|value|value1| ++-----------+------------------+-----------------------------+-----+------+ +| 0| 4.637180787192477|1970-01-01T08:00:00.001+08:00| 1.0| 0.0| +| 0| 4.637180787192477|1970-01-01T08:00:00.002+08:00| 2.0| 0.0| +| 0| 4.637180787192477|1970-01-01T08:00:00.003+08:00| 1.0| 0.0| +| 0| 4.637180787192477|1970-01-01T08:00:00.004+08:00| 1.0| 0.0| +| 0| 4.637180787192477|1970-01-01T08:00:00.005+08:00| 1.0| 0.0| +| 0| 4.637180787192477|1970-01-01T08:00:00.006+08:00| 1.2| 0.0| +| 0| 4.637180787192477|1970-01-01T08:00:00.007+08:00| 1.0| 0.0| +| 0| 4.637180787192477|1970-01-01T08:00:00.008+08:00| 1.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.013+08:00| 2.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.014+08:00| 3.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.015+08:00| 4.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.016+08:00| 2.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.017+08:00| 1.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.018+08:00| 1.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.019+08:00| 1.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.020+08:00| 1.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.021+08:00| 1.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.041+08:00| 1.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.042+08:00| 2.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.043+08:00| 3.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.044+08:00| 4.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.045+08:00| 3.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.046+08:00| 2.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.047+08:00| 2.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.048+08:00| 2.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.049+08:00| 2.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.050+08:00| 2.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.051+08:00| 2.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.052+08:00| 2.0| 0.0| ++-----------+------------------+-----------------------------+-----+------+ +Total line number = 29 +It costs 0.126s +``` + +#### 5.4.3 Adjust Matching Conditions + +Each example below starts from the query in Section 5.4.2 and changes only the indicated argument. + +|Purpose|Parameter|Original Value|New Value|Effect| +|---|---|---|---|---| +|Require a closer match|`threshold`|`10.0`|`1.0`|Retains only results with similarity below `1.0`, reducing the number of matching segments from 3 to 2.| +|Limit the time span|`width`|`1000.0`|`10.0`|Returns only subsequences with a timestamp span below `10.0`.| +|Limit the value range|`height`|`500.0`|`2.0`|Returns only subsequences with a value range below `2.0`.| + +#### 5.4.4 Regular-Expression Patterns + +Regular-expression patterns can match repeated trend segments. + +|Repetition Symbol|Meaning| +|---|---| +|`+`|The preceding subsequence occurs at least once.| +|`*`|The preceding subsequence occurs zero or more times.| + +The following example uses `+` to match one or more occurrences of `2.0,3.0,2.0`: + +```sql +select * from pattern_match( + data => test ORDER BY time, + time_col => 'time', + data_col => 'value', + pattern => '1.0,{2.0,3.0,2.0}+,1.0,1.0,1.0,1.0,1.0', + smooth => 0.5, + threshold => 100.0, + width => 1000.0, + height => 500.0, + smooth_on_pattern => false +); +``` + +To allow zero or more occurrences of `2.0,3.0,2.0`, replace `+` with `*`: + +```sql +pattern => '1.0,{2.0,3.0,2.0}*,1.0,1.0,1.0,1.0,1.0' +``` + +Regular-expression patterns are useful for periodic fluctuations, repeated peaks, and trend matches of variable length. + +#### 5.4.5 Pattern Smoothing + +When the pattern contains small fluctuations, use `smooth_on_pattern` to control whether the pattern is also smoothed. + +The following examples show the results of two configurations: + +|Configuration|`threshold`|`smooth_on_pattern`|Matching Behavior| +|---|---|---|---| +|Smooth both the original series and the pattern|`1.0`|`true`|Matches the `1.0,1.2,1.0` fluctuation in the original data.| +|Smooth only the original series|`10.0`|`false`|Returns other trend matches but excludes the corresponding `1.0,1.2,1.0` fluctuation.| + +Smooth both the original series and the pattern: + +```sql +SELECT * +FROM pattern_match( + data => test ORDER BY time, + time_col => 'time', + data_col => 'value', + pattern => '1.0,1.2,1.0', + smooth => 0.5, + threshold => 1.0, + width => 1000.0, + height => 500.0, + smooth_on_pattern => true +); +``` + +With the same smoothing rules applied to both sequences, the query matches the `1.0,1.2,1.0` fluctuation at time points 5, 6, and 7. + +The results are shown below: + +![](/img/featured-function-03-en.png) + +Smooth only the original series: + +```sql +SELECT * +FROM pattern_match( + data => test ORDER BY time, + time_col => 'time', + data_col => 'value', + pattern => '1.0,1.2,1.0', + smooth => 0.5, + threshold => 10.0, + width => 1000.0, + height => 500.0, + smooth_on_pattern => false +); +``` + +With smoothing applied only to the original series, the query still returns other trend matches, but the `1.0,1.2,1.0` fluctuation at time points 5, 6, and 7 is not matched. + +The results are shown below: + +![](/img/featured-function-04-en.png) + +## 6. Frequency-Domain Analysis Functions + +### 6.1 FFT + +#### 6.1.1 Description + +`FFT` is a built-in table function in the table model. It performs a one-dimensional fast Fourier transform on one or more real-valued numeric series in each input partition. `ORDER BY` must specify the time column in ascending order. You can also use `PARTITION BY` to group data by device or TAG. Each partition generates its own frequency bins, and all numeric columns in the partition share the same frequency axis. + +> Supported since V2.0.12 + +#### 6.1.2 Function Definition + +Named-argument syntax: + +```sql +SELECT * +FROM FFT( + DATA => {table_name | subquery} + [PARTITION BY partition_column [, ...]] + ORDER BY time_column, + [SAMPLE_INTERVAL => duration_literal,] + [N => positive_integer,] + [NORM => {'backward' | 'forward' | 'ortho'},] + [TIMECOL => 'time_column'] +); +``` + +Positional-argument syntax: + +```sql +SELECT * +FROM TABLE( + FFT( + TABLE(table_name) + [PARTITION BY partition_column [, ...]] + ORDER BY time_column + [, duration_literal + [, positive_integer + [, norm + [, time_column_name]]]] + ) +); +``` + +The canonical argument order is `DATA, SAMPLE_INTERVAL, N, NORM, TIMECOL`. **Named arguments** are recommended to make complex calls easier to read. + +#### 6.1.3 Parameters + +|Parameter|Required|Type|Default|Description| +|---|---|---|---|---| +|`DATA`|Yes|Table argument|None|Input data for FFT. With `PARTITION BY`, each partition is calculated independently; without it, all input rows form one partition. All `INT32`, `INT64`, `FLOAT`, and `DOUBLE` columns except the time and partition columns participate in the calculation. Other column types are ignored. At least one numeric column is required, and participating columns cannot contain `NULL`. To transform only selected numeric columns, project them in an input subquery.| +|`SAMPLE_INTERVAL`|No|Positive duration literal|Inferred from the time column|Sampling interval used to calculate the frequency axis, such as `1ms`, `10ms`, or `1s`. If omitted, it is inferred per partition as `(last timestamp - first timestamp) / (input row count - 1)`. A single-row partition requires an explicit interval; an empty partition returns 0 rows.| +|`N`|No|Positive integer|Input row count of the current partition|FFT length. If smaller than the input row count, only the first `N` rows in timestamp order are transformed. If larger, the series is padded with zeros at the end. Each partition returns `N` rows. The maximum `N` is `65536`, and `N × 2 × number of participating numeric columns ≤ 16777216` must hold.| +|`NORM`|No|String|`backward`|Normalization mode, case-insensitive. `backward`: no normalization; `forward`: divide by `N`; `ortho`: divide by `√N`.| +|`TIMECOL`|No|String|`time`|Name of the time column, which must have type `TIMESTAMP`. `DATA` must be sorted only by this column in ascending order; `DESC` is unsupported. Timestamps within each partition must be strictly increasing, with no duplicates.| + +- FFT is generally intended for evenly sampled data. This function does not automatically detect or correct irregular sampling. Interpolate or resample the data before calculation to ensure accurate spectra. If the sampling interval is known, explicitly set `SAMPLE_INTERVAL`. +- When `N` is smaller than the input row count, the function still validates all input rows in the partition, although only the first `N` rows are transformed. + +**Returned Results** + +`FFT` returns the full two-sided spectrum for each partition, with columns in the following order: + +```text +partition_columns... +frequency_index +frequency +_real +_imag +... +``` + +|Output Column|Type|Description| +|---|---|---| +|Columns specified by `PARTITION BY`|Same as input|Retain their original names and types and appear first in the result. Omitted when `PARTITION BY` is not used.| +|`frequency_index`|`INT64`|Frequency-bin index, from `0` to `N-1`.| +|`frequency`|`DOUBLE`|Frequency of the current bin, in Hz.| +|`_real`|`DOUBLE`|Real part of the FFT for the corresponding numeric input column.| +|`_imag`|`DOUBLE`|Imaginary part of the FFT for the corresponding numeric input column.| + +For transform length `N` and sampling interval `d` in seconds, `frequency` is calculated as follows: + +|`frequency_index` Range|`frequency`| +|---|---| +|`frequency_index < ceil(N / 2)`|`frequency_index / (N × d)`| +|Otherwise|`(frequency_index - N) / (N × d)`| + +Frequencies are ordered as zero, positive, then negative frequencies, in Hz. All numeric columns within a partition share the same frequency axis. + +#### 6.1.4 Notes + +- The function name `FFT` is case-insensitive. Each partition performs a separate one-dimensional FFT. +- The current version supports only real-valued input. Complex input, `DFT`, and `IFFT` are unsupported. +- The current version returns only the real and imaginary parts of the full spectrum. One-sided spectra, amplitude output, and phase output are unsupported. +- The `VALUE`, `SAMPLE_RATE`, and `SPECTRUM` arguments are unsupported in the current version. + +#### 6.1.5 Usage Examples + +These examples use `table1`. Because `temperature` contains `NULL` values, filter them out in the input subquery. + +**Example 1: FFT by Device** + +```sql +SELECT * +FROM FFT( + DATA => ( + SELECT time, device_id, temperature + FROM table1 + WHERE plant_id = '1001' + AND temperature IS NOT NULL + ) PARTITION BY device_id ORDER BY time +) +ORDER BY device_id, frequency_index; +``` + +The function transforms `temperature` separately for each device. With `N` and `SAMPLE_INTERVAL` omitted, the transform length is the input row count of each partition, and the sampling interval is inferred from the time column. In this example, the sampling interval is 1 minute. + +```text ++---------+---------------+---------------------+----------------------+---------------------+ +|device_id|frequency_index| frequency| temperature_real| temperature_imag| ++---------+---------------+---------------------+----------------------+---------------------+ +| 100| 0| 0.0| 180.0| 0.0| +| 100| 1|-0.008333333333333333| 0.0| 0.0| +| 101| 0| 0.0| 255.0| -6.67572021484375E-6| +| 101| 1| 0.005555555555555556|-4.6126147026370745E-6|-5.266308107820805E-6| +| 101| 2|-0.005555555555555556| -1.9073486328125E-6| 3.303624680484063E-6| ++---------+---------------+---------------------+----------------------+---------------------+ +``` + +**Example 2: Specify Transform Length and Normalization** + +```sql +SELECT * +FROM FFT( + DATA => ( + SELECT time, device_id, temperature + FROM table1 + WHERE plant_id = '1001' + AND temperature IS NOT NULL + ) PARTITION BY device_id ORDER BY time, + SAMPLE_INTERVAL => 1m, + N => 4, + NORM => 'ortho' +) +ORDER BY device_id, frequency_index; +``` + +The transform length is set to 4 for each device. Series with fewer than 4 input rows are padded with zeros at the end. FFT results are normalized by `1/√N`. Each device returns 4 spectrum rows. + +```text ++---------+---------------+---------------------+----------------------+----------------+ +|device_id|frequency_index| frequency| temperature_real|temperature_imag| ++---------+---------------+---------------------+----------------------+----------------+ +| 100| 0| 0.0| 90.0| 0.0| +| 100| 1| 0.004166666666666667| 45.0| -45.0| +| 100| 2|-0.008333333333333333| 0.0| 0.0| +| 100| 3|-0.004166666666666667| 45.0| 45.0| +| 101| 0| 0.0| 127.5| 0.0| +| 101| 1| 0.004166666666666667| 2.602374475716698E-15| -42.5| +| 101| 2|-0.008333333333333333| 42.5| 0.0| +| 101| 3|-0.004166666666666667|-2.602374475716698E-15| 42.5| ++---------+---------------+---------------------+----------------------+----------------+ +``` + +**Example 3: Numeric Column Contains `NULL`** + +```sql +SELECT * +FROM FFT( + DATA => ( + SELECT time, device_id, temperature + FROM table1 + WHERE plant_id = '1001' + ) PARTITION BY device_id ORDER BY time +); +``` + +The query fails because numeric columns participating in FFT cannot contain `NULL`: + +```text +Msg: org.apache.iotdb.jdbc.IoTDBSQLException: 701: FFT does not support null values in column [temperature]. +``` + +Filter out null values with `IS NOT NULL` in the input subquery before running FFT. + +### 6.2 lowPass + +#### 6.2.1 Description + +`lowPass` is a low-pass filter that applies frequency-domain filtering to numeric columns in the input table. It retains low-frequency components and attenuates high-frequency components, making it useful for signal smoothing and high-frequency noise removal. The normalized passband frequency `wpass`, in the range `(0, 1)`, specifies the upper frequency bound of the passband. + +> Supported since V2.0.12 + +#### 6.2.2 Function Definition + +Named-argument syntax: + +```sql +SELECT * +FROM lowPass( + DATA => input_table PARTITION BY partition_column ORDER BY time_column, + TIMECOL => 'time_column', + WPASS => passband_frequency +); +``` + +Positional-argument syntax: + +```sql +SELECT * +FROM lowPass( + input_table PARTITION BY partition_column ORDER BY time_column, + 'time_column', + passband_frequency +); +``` + +#### 6.2.3 Parameters + +|Parameter|Type|Required|Description| +|---|---|---|---| +|`DATA`|Table argument (`SET SEMANTIC`)|Yes|Input table or subquery. `ORDER BY` must contain only the time column in **ascending order**. `PARTITION BY` is optional; if omitted, the entire table is treated as one partition.| +|`TIMECOL`|String|Yes|Name of the time column. Must identify exactly one column of type `TIMESTAMP`.| +|`WPASS`|`DOUBLE`|Yes|Normalized passband frequency in `(0, 1)`. Components below this frequency are retained, and those above it are attenuated.| + +**Calculation Rules** + +- Columns specified by `TIMECOL` and `PARTITION BY` are excluded from calculations but retained in the output. +- Apart from the time and partition columns, only `INT32`, `INT64`, `FLOAT`, and `DOUBLE` columns can participate. Analysis fails if no eligible column remains or if any other column type is present. +- `NULL` values in a participating column are excluded from that column's filtering calculation. + +**Returned Results** + +- The output includes the partition columns, if specified, and the time column, with their original names and types. +- Each participating numeric column produces a filtered result column of type `DOUBLE`. Each result column has the same number of points as its corresponding input column, output in ascending timestamp order. + +#### 6.2.4 Usage Example + +The following example uses `table1`. + +```sql +SELECT * +FROM lowPass( + DATA => ( + SELECT time, device_id, temperature + FROM table1 + WHERE temperature is not null + ) PARTITION BY device_id ORDER BY time, + TIMECOL => 'time', + WPASS => 0.3 +); +``` + +The example partitions data by device, applies low-pass filtering to `temperature`, and returns `device_id`, `time`, and the filtered numeric column. + +```text ++---------+-----------------------------+--------------------+ +|device_id| time|lowpass(temperature)| ++---------+-----------------------------+--------------------+ +| 100|2024-11-26T13:37:00.000+08:00| 88.0| +| 100|2024-11-26T13:38:00.000+08:00| 88.0| +| 100|2024-11-28T08:00:00.000+08:00| 87.99999999999999| +| 100|2024-11-28T10:00:00.000+08:00| 87.99999999999999| +| 100|2024-11-28T11:00:00.000+08:00| 88.00000000000001| +| 100|2024-11-29T18:30:00.000+08:00| 88.0| +| 101|2024-11-27T16:39:00.000+08:00| 86.66666666666667| +| 101|2024-11-27T16:40:00.000+08:00| 86.66666666666667| +| 101|2024-11-27T16:41:00.000+08:00| 86.66666666666664| +| 101|2024-11-29T10:00:00.000+08:00| 86.66666666666664| +| 101|2024-11-30T09:30:00.000+08:00| 86.66666666666667| +| 101|2024-11-30T14:30:00.000+08:00| 86.66666666666664| ++---------+-----------------------------+--------------------+ +``` + +### 6.3 highPass + +#### 6.3.1 Description + +`highPass` is a high-pass filter that applies frequency-domain filtering to numeric columns in the input table. It retains high-frequency components and attenuates low-frequency components, making it useful for extracting rapid changes, abrupt transitions, and high-frequency noise. The normalized passband frequency `wpass`, in the range `(0, 1)`, specifies the lower frequency bound of the passband. + +> Supported since V2.0.12 + +#### 6.3.2 Function Definition + +Named-argument syntax: + +```sql +SELECT * +FROM highPass( + DATA => input_table PARTITION BY partition_column ORDER BY time_column, + TIMECOL => 'time_column', + WPASS => passband_frequency +); +``` + +Positional-argument syntax: + +```sql +SELECT * +FROM highPass( + input_table PARTITION BY partition_column ORDER BY time_column, + 'time_column', + passband_frequency +); +``` + +#### 6.3.3 Parameters + +|Parameter|Type|Required|Description| +|---|---|---|---| +|`DATA`|Table argument (`SET SEMANTIC`)|Yes|Input table or subquery. `ORDER BY` must contain only the time column in ascending order. `PARTITION BY` is optional; if omitted, the entire table is treated as one partition.| +|`TIMECOL`|String|Yes|Name of the time column. Must identify exactly one column of type `TIMESTAMP`.| +|`WPASS`|`DOUBLE`|Yes|Normalized passband frequency in `(0, 1)`. Components above this frequency are retained, and those below it are attenuated.| + +**Input Columns and Calculation Rules** + +- Columns specified by `TIMECOL` and `PARTITION BY` are excluded from calculations but retained in the output. +- Apart from the time and partition columns, only `INT32`, `INT64`, `FLOAT`, and `DOUBLE` columns can participate. Analysis fails if no eligible column remains or if any other column type is present. +- `NULL` values in a participating column are excluded from that column's filtering calculation. + +**Returned Results** + +- The output includes the partition columns, if specified, and the time column, with their original names and types. +- Each participating numeric column produces a filtered result column of type `DOUBLE`. Each result column has the same number of points as its corresponding input column, output in ascending timestamp order. + +#### 6.3.4 Usage Example + +```sql +SELECT * +FROM highPass( + DATA => ( + SELECT time, device_id, temperature + FROM table1 + WHERE temperature is not null + ) PARTITION BY device_id ORDER BY time, + TIMECOL => 'time', + WPASS => 0.3 +); +``` + +The example partitions data by device, applies high-pass filtering to `temperature`, and returns `device_id`, `time`, and the filtered numeric column. + +```text ++---------+-----------------------------+----------------------+ +|device_id| time| highpass(temperature)| ++---------+-----------------------------+----------------------+ +| 100|2024-11-26T13:37:00.000+08:00| 2.000000000000029| +| 100|2024-11-26T13:38:00.000+08:00| 2.000000000000004| +| 100|2024-11-28T08:00:00.000+08:00| -2.9999999999999702| +| 100|2024-11-28T10:00:00.000+08:00| -3.000000000000013| +| 100|2024-11-28T11:00:00.000+08:00|-4.182831670630972E-14| +| 100|2024-11-29T18:30:00.000+08:00| 1.9999999999999953| +| 101|2024-11-27T16:39:00.000+08:00| -1.666666666666642| +| 101|2024-11-27T16:40:00.000+08:00| -1.6666666666666519| +| 101|2024-11-27T16:41:00.000+08:00| -1.666666666666645| +| 101|2024-11-29T10:00:00.000+08:00| -1.666666666666701| +| 101|2024-11-30T09:30:00.000+08:00| 3.3333333333333015| +| 101|2024-11-30T14:30:00.000+08:00| 3.333333333333338| ++---------+-----------------------------+----------------------+ +``` + +## 7. Time-Series Correlation Analysis Functions + +### 7.1 Xcorr + +#### 7.1.1 Description + +`Xcorr` performs cross-correlation analysis on two numeric series to measure their shape similarity at different time lags. It shifts one series relative to the other, multiplies corresponding points, and sums the products to produce a similarity curve. This can reveal delays and similar patterns between two signals. + +> Supported since V2.0.12 + +#### 7.1.2 Function Definition + +Named-argument syntax: + +```sql +SELECT * +FROM Xcorr( + DATA => ( + SELECT time_column, signal_1, signal_2, partition_column + FROM input_table + ) PARTITION BY partition_column ORDER BY time_column, + TIMECOL => 'time_column' +); +``` + +Positional-argument syntax: + +```sql +SELECT * +FROM Xcorr( + ( SELECT time_column, signal_1, signal_2, partition_column + FROM input_table + ) PARTITION BY partition_column ORDER BY time_column, + 'time_column' +); +``` + +#### 7.1.3 Parameters + +|Parameter|Type|Required|Description| +|---|---|---|---| +|`DATA`|Table argument (`SET SEMANTIC`)|Yes|Input table or subquery. `ORDER BY` must specify the time column in ascending order. `PARTITION BY` is optional. Exactly two numeric columns must remain after excluding the time and partition columns.| +|`TIMECOL`|String|Yes|Name of the time column. Must identify exactly one column of type `TIMESTAMP`.| + +**Calculation Rules** + +- The time and partition columns are excluded from cross-correlation calculations. +- Exactly two columns must remain after those columns are excluded. Both must be of type `INT32`, `INT64`, `FLOAT`, or `DOUBLE`; otherwise, analysis fails. +- `NULL` values in either participating column are treated as `0` at that position. +- Timestamps must be strictly increasing within each partition. +- Normalization is applied according to the function definition. + +**Returned Results** + +- Partition columns specified by `PARTITION BY`, if any, retain their original names and types. +- One result column of type `DOUBLE` is generated for the two participating columns. + +#### 7.1.4 Usage Example + +The following example uses `table1` from the SQL manual's sample data. A subquery selects the two numeric series, `temperature` and `humidity`, for cross-correlation. + +```sql +SELECT * +FROM Xcorr( + DATA => ( + SELECT time, device_id, temperature, humidity + FROM table1 + WHERE temperature IS NOT NULL + AND humidity IS NOT NULL + ) + PARTITION BY device_id + ORDER BY time, + TIMECOL => 'time' +); +``` + +The example partitions data by device, cross-correlates `temperature` and `humidity`, and returns `device_id` and the correlation result column. + +```text ++---------+----------------------------+ +|device_id|xcorr(temperature, humidity)| ++---------+----------------------------+ +| 100| 3186.0001373291016| +| 100| 3622.5| +| 100| 3412.0000203450522| +| 100| 3318.9249839782715| +| 100| 3292.9599586486815| +| 100| 3324.7749156951904| +| 100| 3080.0999348958335| +| 100| 3123.8998641967773| +| 100| 3158.9998626708984| +| 101| 2957.9999351501465| +| 101| 3061.9999980926514| +| 101| 3100.166645050049| +| 101| 3172.5| +| 101| 3176.999931335449| ++---------+----------------------------+ +``` diff --git a/src/zh/UserGuide/Master/Table/SQL-Manual/Featured-Functions_apache.md b/src/zh/UserGuide/Master/Table/SQL-Manual/Featured-Functions_apache.md index 20abca659..105c30828 100644 --- a/src/zh/UserGuide/Master/Table/SQL-Manual/Featured-Functions_apache.md +++ b/src/zh/UserGuide/Master/Table/SQL-Manual/Featured-Functions_apache.md @@ -213,6 +213,213 @@ GROUP BY 1, device_id; +---------+---------+--------+ ``` +### 1.3 `M4` 函数 + +#### 1.3.1 功能描述 + +`M4` 是表模型内置的多态表值函数，用于对时序数据进行降采样。函数在每个窗口内提取首点（First）、尾点（Last）、最小值点（Bottom）和最大值点（Top），在减少返回数据量的同时尽量保持趋势图形状。与标量函数或聚合函数不同，`M4` 接收一个表参数并返回结果表，必须在 `FROM` 子句中调用。 + +- 首点：窗口内时间戳最小且值不为 `NULL` 的点。 + +- 尾点：窗口内时间戳最大且值不为 `NULL` 的点。 + +- 最小值点和最大值点：窗口内值最小或最大的点；值相同时保留按输入扫描顺序遇到的第一个点。 + +- 支持时间窗口和点数窗口两种模式。`SIZE` 使用时间长度字面量（例如 `1h`、`10m`）时为时间窗口；使用正整数（例如 `100`）时为点数窗口。 + +> V2.0.12 版本起支持 + +#### 1.3.2 函数定义 + +```sql +SELECT * +FROM M4( + DATA => input_table PARTITION BY partition_column ORDER BY time_column, + TIMECOL => 'time_column', + [ORIGIN => origin_timestamp,] + SIZE => window_duration | window_point_count, + [SLIDE => window_duration | window_point_count] +); +``` + +`DATA` 是集合语义表参数。使用时必须指定 `ORDER BY`，且排序列只能是 `TIMECOL` 指定的时间列并按升序排列；`PARTITION BY` 可选，不指定时整张输入表作为一个分区处理。参数可按名称或按声明顺序传递，建议使用命名参数。 + +#### 1.3.3 参数说明 + +|参数|类型|必填|说明| +|---|---|---|---| +|`DATA`|表参数（SET SEMANTIC）|是|输入表或子查询。可指定 `PARTITION BY` 和 `ORDER BY`。| +|`TIMECOL`|字符串|是|时间列名。该列必须为 `TIMESTAMP` 类型，且 `ORDER BY` 只能包含该列。| +|`ORIGIN`|时间戳|否|时间窗口的对齐原点，默认值为 `0`。点数窗口模式不使用该参数。| +|`SIZE`|时间长度或 `INT64`|是|时间窗口的窗口长度，或点数窗口的点数大小；必须为正数。| +|`SLIDE`|与 `SIZE` 同类型|否|窗口滑动步长，默认与 `SIZE` 相同。必须与 `SIZE` 使用相同窗口模式。| + +**计算规则** + +- `TIMECOL` 指定列不参与 M4 计算。 + +- `PARTITION BY` 指定列不参与计算，但保留在输出结果中。 + +- 除上述列外，所有可比较类型列均参与 M4 计算。 + +- 排除上述列后，若无可比较类型列，或存在不可比较类型列，分析阶段直接报错。 + +- 参与列值为 `NULL` 的记录，不参与该列的 `first`、`last`、`bottom`、`top` 计算。 + +- 各参与列先计算并去重 `first`、`last`、`bottom`、`top` 特征点，再按时间戳升序输出。 + +**返回结果** + +- 时间窗口模式：固定生成 `window_start`、`window_end`（时间戳类型），表示窗口范围 `[window_start, window_end)`。 + +- 点数窗口模式：固定生成 `window_index`（长整数类型），表示窗口序号；在每个 `PARTITION BY` 分区内独立计算，从 `0` 开始递增。 + +- PARTITION BY 列：按原列名、原类型保留输出，多个列按声明顺序排列。`OBJECT` 类型不支持作为分区列；`BLOB` 类型可保留输出，但不参与 M4 计算。 + +- 动态生成列：每个参与 M4 计算的列 `c` 生成 `c_time`（特征点时间戳）和 `c`（特征点值）两列，类型与输入列保持一致。多列结果按输入列顺序展开。 + +补充说明：窗口输出行数取所有参与列特征点数量的最大值。特征点较少的列，其剩余行的 `c_time` 和 `c` 填充 `NULL`。 + +#### 1.3.4 使用示例 + +以下示例均使用[示例数据](../Reference/Sample-Data.md)中的 `table1`。由于 `table1` 中除时间列和分区列外还包含其他字段，因此示例通过子查询仅选择 `time`、`device_id`、`temperature` 和 `humidity` 参与 M4 计算。 + +- 示例 1：时间窗口模式 + +以 `device_id = '101'` 的数据为例，每 3 分钟划分一个窗口。`SIZE => 3m` 表示使用时间窗口模式，窗口长度为 3 分钟。窗口采用半开区间 `[window_start, window_end)`，每个窗口分别提取 `temperature` 和 `humidity` 列的首点、尾点、最小值点和最大值点，并按照特征点时间戳升序输出。 + +```sql +SELECT window_start, + window_end, + device_id, + temperature_time, + temperature, + humidity_time, + humidity +FROM M4( + DATA => ( + SELECT time, device_id, temperature, humidity + FROM table1 + WHERE device_id = '101' + AND time >= 2024-11-27 16:38:00 + AND time <= 2024-11-27 16:44:00 + ) PARTITION BY device_id ORDER BY time, + TIMECOL => 'time', + SIZE => 3m +); +``` + +结果： + +```text ++-----------------------------+-----------------------------+---------+-----------------------------+-----------+-----------------------------+--------+ +| window_start| window_end|device_id| temperature_time|temperature| humidity_time|humidity| ++-----------------------------+-----------------------------+---------+-----------------------------+-----------+-----------------------------+--------+ +|2024-11-27T16:36:00.000+08:00|2024-11-27T16:39:00.000+08:00| 101| null| null|2024-11-27T16:38:00.000+08:00| 35.1| +|2024-11-27T16:39:00.000+08:00|2024-11-27T16:42:00.000+08:00| 101|2024-11-27T16:39:00.000+08:00| 85.0|2024-11-27T16:39:00.000+08:00| 35.3| +|2024-11-27T16:39:00.000+08:00|2024-11-27T16:42:00.000+08:00| 101|2024-11-27T16:41:00.000+08:00| 85.0| null| null| +|2024-11-27T16:42:00.000+08:00|2024-11-27T16:45:00.000+08:00| 101| null| null|2024-11-27T16:42:00.000+08:00| 35.2| ++-----------------------------+-----------------------------+---------+-----------------------------+-----------+-----------------------------+--------+ +``` + +- 示例 2：点数窗口模式 + +以 `device_id = '101'` 的数据为例，每 3 个数据点划分一个窗口。`SIZE => 3` 表示使用点数窗口模式，每 3 行数据划分一个窗口。窗口序号通过 `window_index` 标识，并在每个分区内从 `0` 开始计算。 + +```sql +SELECT window_index, + device_id, + temperature_time, + temperature, + humidity_time, + humidity +FROM M4( + DATA => ( + SELECT time, device_id, temperature, humidity + FROM table1 + WHERE device_id = '101' + AND time >= 2024-11-27 16:38:00 + AND time <= 2024-11-27 16:44:00 + ) PARTITION BY device_id ORDER BY time, + TIMECOL => 'time', + SIZE => 3 +); +``` + +结果： + +```text ++------------+---------+-----------------------------+-----------+-----------------------------+--------+ +|window_index|device_id| temperature_time|temperature| humidity_time|humidity| ++------------+---------+-----------------------------+-----------+-----------------------------+--------+ +| 0| 101|2024-11-27T16:39:00.000+08:00| 85.0|2024-11-27T16:38:00.000+08:00| 35.1| +| 0| 101|2024-11-27T16:40:00.000+08:00| 85.0|2024-11-27T16:39:00.000+08:00| 35.3| +| 1| 101|2024-11-27T16:41:00.000+08:00| 85.0|2024-11-27T16:42:00.000+08:00| 35.2| ++------------+---------+-----------------------------+-----------+-----------------------------+--------+ +``` + +如果需要设置窗口滑动步长，可以同时指定 `SLIDE`： + +```sql +SELECT window_index, + device_id, + temperature_time, + temperature, + humidity_time, + humidity +FROM M4( + DATA => ( + SELECT time, device_id, temperature, humidity + FROM table1 + WHERE device_id = '101' + AND time >= 2024-11-27 16:38:00 + AND time <= 2024-11-27 16:44:00 + ) PARTITION BY device_id ORDER BY time, + TIMECOL => 'time', + SIZE => 3, + SLIDE => 2 +); +``` + +上述调用表示窗口大小为 3 个数据点，窗口每次向后滑动 2 个数据点。 + +结果： + +```text ++------------+---------+-----------------------------+-----------+-----------------------------+--------+ +|window_index|device_id| temperature_time|temperature| humidity_time|humidity| ++------------+---------+-----------------------------+-----------+-----------------------------+--------+ +| 0| 101|2024-11-27T16:39:00.000+08:00| 85.0|2024-11-27T16:38:00.000+08:00| 35.1| +| 0| 101|2024-11-27T16:40:00.000+08:00| 85.0|2024-11-27T16:39:00.000+08:00| 35.3| +| 1| 101|2024-11-27T16:40:00.000+08:00| 85.0|2024-11-27T16:42:00.000+08:00| 35.2| +| 1| 101|2024-11-27T16:41:00.000+08:00| 85.0| null| null| +| 2| 101| null| null|2024-11-27T16:42:00.000+08:00| 35.2| ++------------+---------+-----------------------------+-----------+-----------------------------+--------+ +``` + +- 示例 3：非法示例——`ORDER BY` 未指定时间列 + +`M4` 要求 `DATA` 参数必须指定 `ORDER BY`，且排序列必须与 `TIMECOL` 参数指定的时间列一致。下面的示例错误地使用 `temperature` 作为排序列： + +```sql +SELECT * +FROM M4( + DATA => ( + SELECT time, device_id, temperature, humidity + FROM table1 + WHERE device_id = '101' + ) PARTITION BY device_id ORDER BY temperature, + TIMECOL => 'time', + SIZE => 3m +); +``` + +报错： + +```text +Msg: org.apache.iotdb.jdbc.IoTDBSQLException: 701: The ORDER BY clause of the DATA argument must contain exactly the time column specified by the TIMECOL argument. +``` + ## 2. DIFF函数 ### 2.1 功能概述 @@ -522,27 +729,51 @@ IoTDB> SELECT first(time) as window_start, last(time) as window_end, stock_id, a #### 3.4.1 功能描述 -CAPACITY 函数用于按数据点数（行数）分窗，每个窗口最多有 SIZE 行数据。 +CAPACITY 函数用于按数据点数（行数）分窗，每个窗口最多有 SIZE 行数据。自 V2.0.12 起支持通过参数 SLIDE 指定相邻窗口起点之间间隔的行数。 #### 3.4.2 函数定义 ```sql -CAPACITY(data [PARTITION BY(pkeys, ...)] [ORDER BY(okeys, ...)], size) +CAPACITY( + DATA => + [PARTITION BY [, ...]] + [ORDER BY [, ...]], + SIZE => + [, SLIDE => ] +) ``` #### 3.4.3 参数说明 -| 参数名 | 参数类型 | 参数属性 | 描述 | -| -------- | ---------- | -------------------------- | ---------------------------------------- | -| DATA | 表参数 | SET SEMANTICPASS THROUGH | 输入表通过 pkeys、okeys 指定分区和排序 | -| SIZE | 标量参数 | 长整数类型 | 窗口大小 | +|参数|类型|是否必填|说明| +|---|---|---|---| +|DATA|Table|是|输入表或子查询，支持 PARTITION BY 和 ORDER BY，透传输入列。| +|SIZE|INT64 Scalar|是|窗口容量，必须为正数，表示单个窗口最多覆盖的行数。| +|SLIDE|INT64 Scalar|否|窗口滑动步长，显式传入时必须为正数；省略时默认等于 SIZE。| + +- 指定 PARTITION BY 时，每个分区独立从 0 开始计算行号和窗口编号； + +- 指定 ORDER BY 时，按排序结果确定行号顺序。 + +- 窗口编号从 0 开始，第 k 个窗口覆盖的行号范围为 `[k * SLIDE, k * SLIDE + SIZE)`。 + +- 不同参数关系对应的窗口行为如下： + +|参数关系|窗口类型|行为| +|---|---|---| +|`SLIDE < SIZE`|重叠窗口|相邻窗口有重叠，同一输入行可能属于多个窗口，并以不同的 `window_index` 输出多行。| +|`SLIDE = SIZE`|非重叠窗口|相邻窗口首尾相接，行为与历史版本的 CAPACITY(DATA => ..., SIZE => n) 一致。| +|`SLIDE > SIZE`|间隔窗口|相邻窗口之间存在空档，落在空档内的输入行不输出。| #### 3.4.4 返回结果 CAPACITY 函数的返回结果列包含： -* window\_index: 窗口编号 -* 映射列：DATA 参数的所有输入列 +- window_index: 窗口编号，为 INT64 类型，从 0 开始。 + +- DATA 参数透传列：保持输入表对应列的名称和类型。 + +返回结果可直接查询，也可继续基于 window_index 和业务分组列进行聚合。 #### 3.4.5 使用示例 @@ -569,6 +800,23 @@ IoTDB> SELECT first(time) as start_time, last(time) as end_time, stock_id, avg(p |2021-01-01T09:05:00.000+08:00|2021-01-01T09:07:00.000+08:00| AAPL|101.5| |2021-01-01T09:09:00.000+08:00|2021-01-01T09:09:00.000+08:00| AAPL|102.0| +-----------------------------+-----------------------------+--------+-----+ + +-- 使用 SLIDE 参数生成重叠窗口（SLIDE < SIZE） +IoTDB> SELECT * FROM CAPACITY(DATA => bid PARTITION BY stock_id ORDER BY time, SIZE => 2, SLIDE => 1); ++------------+-----------------------------+--------+-----+ +|window_index| time|stock_id|price| ++------------+-----------------------------+--------+-----+ +| 0|2021-01-01T09:05:00.000+08:00| AAPL|100.0| +| 0|2021-01-01T09:07:00.000+08:00| AAPL|103.0| +| 1|2021-01-01T09:07:00.000+08:00| AAPL|103.0| +| 1|2021-01-01T09:09:00.000+08:00| AAPL|102.0| +| 2|2021-01-01T09:09:00.000+08:00| AAPL|102.0| +| 0|2021-01-01T09:06:00.000+08:00| TESL|200.0| +| 0|2021-01-01T09:07:00.000+08:00| TESL|202.0| +| 1|2021-01-01T09:07:00.000+08:00| TESL|202.0| +| 1|2021-01-01T09:15:00.000+08:00| TESL|195.0| +| 2|2021-01-01T09:15:00.000+08:00| TESL|195.0| ++------------+-----------------------------+--------+-----+ ``` ### 3.5 TUMBLE @@ -695,3 +943,856 @@ IoTDB> SELECT window_start, window_end, stock_id, avg(price) as avg FROM CUMULAT +-----------------------------+-----------------------------+--------+------------------+ ``` + +## 4. `read_tsfile` 表值函数 + +### 4.1 功能概述 + +用于从外部 TsFile 文件或目录读取表模型数据。`read_tsfile` 返回一个临时表数据源，查询时可像普通表一样使用 `SELECT`、`WHERE`、`ORDER BY`、`LIMIT`、`OFFSET`、聚合等能力，对外部 TsFile 数据进行只读分析和临时处理，不会将处理结果写回外部 TsFile，也不会修改 IoTDB 内部数据，不支持通过该入口写入、删除、修改或注册外部 TsFile。 + +> V2.0.12 版本起支持 + +### 4.2 函数定义 + +```sql +SELECT * +FROM TABLE( + read_tsfile( + TABLE_NAME => '', + PATHS => '' + ) +); +``` + +### 4.3 参数说明 + +|参数|必填|说明| +|---|---|---| +|`PATHS`|是|TsFile 文件路径或目录路径。单路径直接传入字符串；多路径在同一个字符串中使用英文逗号分隔，例如 `'/data/a.tsfile,/data/b.tsfile,/data/history'`。
- 指定文件时，系统会直接尝试打开该文件；
- 指定目录时，系统会递归遍历目录下以 `.tsfile` 为后缀的文件，并根据文件首尾的 MAGIC NUMBER 识别有效 TsFile。文件内容不符合 TsFile 格式、无法识别、不完整或版本不是 TsFile V4 的文件不参与本次查询。
注意：指定的路径必须对执行查询的 DataNode 进程可见，并且 DataNode 进程需要具备相应的文件读取权限。禁止指定 DataNode 的任意数据目录，否则查询会报错。| +|`TABLE_NAME`|否|需要读取的表名。填写该参数时，系统只读取包含目标表 Schema 的有效文件；如果所有有效文件中都不存在目标表 Schema，查询报错。
当外部 TsFile 中可以唯一推断出待读取的表名时，可以省略该参数；省略时要求每个参与读取的有效文件中都有且只有一个表，并且所有有效文件推断出的表名一致。如果某个有效文件包含多个表，或者不同有效文件中的唯一表名不一致，查询报错。| +|`DEVICE_METADATA_INFO_SWAP_THRESHOLD` |否|查询时如果文件数和设备数量都很多，为了避免占用过多内存可能会将一些信息写成交换文件，这个参数可指定一个 Long 值，当达到此大小时强制触发交换临时文件，默认值为 8MB| + +**Schema 合并规则** + +读取多个 TsFile 时，系统会在确定目标表后，合并该表的 Schema，并将合并后的 Schema 作为本次查询的输出 Schema，且仅用于本次查询，不会注册到数据库元数据中。其合并规则如下： + +- tag 列按顺序合并。存在冲突时要求前缀一致；同一位置 tag 名称不同时报错；允许后续位置追加 tag 列。 + +- time 列名称必须一致，存在冲突时报错。 + +- field 列对顺序无要求，不同文件可以包含不同 field 列，系统会进行增量合并；同名 field 列的数据类型必须兼容，否则报错。 + +- 不同 TsFile 中同名列的列类型必须一致，否则报错。 + +**返回结果** + +- 返回指定表的数据结果集。 + +- 输出列来自外部 TsFile 中目标表的合并 Schema。 + +- `time` 列为时间列，列名取决于 TsFile 中记录的时间列名称。 + +- tag 列表示设备标识信息。 + +- field 列表示实际测点数据。 + +### 4.4 使用示例 + +假设在`/data/backup/` 路径下有文件 `1.tsfile`， 该文件中的元数据和数据可参考 [示例数据](../Reference/Sample-Data.md) 中的 `table1` + +1. 自动推断表名读取 TsFile。 + +该场景要求 `/data/backup/1.tsfile` 中有且只有一个表。 + +```sql +SELECT * +FROM TABLE( + read_tsfile( + PATHS => '/data/backup/1.tsfile' + ) +) +LIMIT 10; +``` + +返回结果 + +```text ++-----------------------------+------+--------+---------+--------+-----------+-----------+--------+------+-----------------------------+ +| time|region|plant_id|device_id|model_id|maintenance|temperature|humidity|status| arrival_time| ++-----------------------------+------+--------+---------+--------+-----------+-----------+--------+------+-----------------------------+ +|2024-11-29T10:00:00.000+08:00| 上海| 3001| 101| D| 360| 85.0| null| null|2024-11-29T10:00:13.000+08:00| +|2024-11-29T11:00:00.000+08:00| 上海| 3002| 100| E| 180| null| 45.1| true| null| +|2024-11-29T18:30:00.000+08:00| 上海| 3002| 100| E| 180| 90.0| 35.4| true|2024-11-29T18:30:15.000+08:00| +|2024-11-30T09:30:00.000+08:00| 上海| 3002| 101| F| 360| 90.0| 35.2| true| null| +|2024-11-30T14:30:00.000+08:00| 上海| 3002| 101| F| 360| 90.0| 34.8| true|2024-11-30T14:30:17.000+08:00| +|2024-11-26T13:37:00.000+08:00| 北京| 1001| 100| A| 180| 90.0| 35.1| true|2024-11-26T13:37:34.000+08:00| +|2024-11-26T13:38:00.000+08:00| 北京| 1001| 100| A| 180| 90.0| 35.1| true|2024-11-26T13:38:25.000+08:00| +|2024-11-28T08:00:00.000+08:00| 上海| 3001| 100| C| 90| 85.0| null| null|2024-11-28T08:00:09.000+08:00| +|2024-11-28T09:00:00.000+08:00| 上海| 3001| 100| C| 90| null| 40.9| true| null| +|2024-11-28T10:00:00.000+08:00| 上海| 3001| 100| C| 90| 85.0| 35.2| null|2024-11-28T10:00:11.000+08:00| ++-----------------------------+------+--------+---------+--------+-----------+-----------+--------+------+-----------------------------+ +Total line number = 10 +It costs 0.159s +``` + +2. 读取目录下的 TsFile。 + +```sql +SELECT time, device_id, temperature +FROM TABLE( + read_tsfile( + TABLE_NAME => 'table1', + PATHS => '/data/backup' + ) +) +WHERE time >= 2024-11-28 00:00:00 AND time < 2024-11-29 00:00:00 ; +``` + +返回结果 + +```text ++-----------------------------+---------+-----------+ +| time|device_id|temperature| ++-----------------------------+---------+-----------+ +|2024-11-28T08:00:00.000+08:00| 100| 85.0| +|2024-11-28T09:00:00.000+08:00| 100| null| +|2024-11-28T10:00:00.000+08:00| 100| 85.0| +|2024-11-28T11:00:00.000+08:00| 100| 88.0| ++-----------------------------+---------+-----------+ +Total line number = 4 +It costs 0.116s +``` + +3. 按 tag 聚合。 + +```sql +SELECT device_id, count(temperature) +FROM TABLE( + read_tsfile( + TABLE_NAME => 'table1', + PATHS => '/data/backup' + ) +) +GROUP BY device_id; +``` + +返回结果 + +```text ++---------+-----+ +|device_id|_col1| ++---------+-----+ +| 100| 6| +| 101| 6| ++---------+-----+ +Total line number = 2 +It costs 0.049s +``` + +## 5. 时间序列趋势匹配表值函数 + +### 5.1 功能描述 + +`pattern_match` 表值函数支持基于输入的模式（pattern）序列，在指定的历史时间序列中，查找并返回所有符合阈值、结果边界等预设判定条件的子序列。 + +> V2.0.12 版本起支持 + +### 5.2 函数定义 + +```sql +pattern_match( + data => {table_name | subquery} PARTITION BY partition_col ORDER BY time_col, + [time_col => 'time_column_name',] + data_col => 'data_column_name', + pattern => 'pattern_sequence', + smooth => numeric_value, + threshold => numeric_value, + [width => numeric_value,] + [height => numeric_value,] + [smooth_on_pattern => {true | false}] +) +``` + +### 5.3 参数说明 + +|参数|类型|必填|默认值|说明| +|---|---|---|---|---| +|`data`|表名/子查询
|是|-|时间序列所在的表名或子查询结果，**必须保证时间列有序**；支持 `PARTITION BY` 按字段分区、`ORDER BY` 按时间排序| +|`time_col`|字符串|否|`'time'`|时间列名称，**仅支持时间戳类型**| +|`data_col`|字符串|是|-|序列名称，支持的数据类型：int、long、double、float| +|`pattern`|字符串|是|-|匹配模式序列，格式为数值列表（支持普通趋势序列 + 正则规则）：
1. 基础格式：以逗号分隔的整型 / 浮点型数值（如 '1.0,2.0,3.0'）；
2. 正则扩展：支持 `{数值段}+`（至少 1 次重复）、`{数值段}*`（至少 0 次重复），其中 `{`，` }* `，`}+` 为独立单位；
3. 特殊规则：每个段的最左侧点与上一个段共用| +|`smooth`|非负实数|是||数据波动容忍幅度（平滑程度）：
1. 数值越大，容忍的波动越大，但易过度平滑正常数据；
2. 推荐值：值域的 1%，可根据序列特性调整| +|`threshold`|非负实数|是||相似度阈值：仅返回相似度小于该值的结果，数值越小匹配越精确；建议从 1 开始向上调整| +|`width`|非负实数|否|无限大|匹配结果的宽度限制：仅返回长度（时间戳差值）小于该值的子序列| +|`height`|非负实数|否|无限大|匹配结果的高度限制：仅返回高度小于该值的子序列| +|`smooth_on_pattern`|布尔值
|否|`false`|是否对 pattern 做平滑处理：
1. false：pattern 为形状趋势，仅对原始序列做平滑；
2. true：对 pattern 和原始序列做同等平滑降噪| + +**返回结果** + +输出结果中仅保留与 pattern 匹配的数据段对应的行，且输出行按 `match_index` 升序排列。未匹配的行全部丢弃。 + +|列名|类型|说明| +|---|---|---| +|`match_index`|int32|非负整数，表示匹配到的第 N 个符合 pattern 的数据段| +|`similarity`|double|非负实数，表示该数据段与 pattern 的匹配相似度（值越小，匹配越精确）| +|data 参数所有列|与原列一致|包含输入 `data` 参数中所有列的原始数据| + +### 5.4 使用示例 + +以下示例演示普通模板匹配、参数调整、正则模板匹配以及平滑处理。 + +#### 5.4.1 准备测试数据 + +创建表并写入数据： + +```sql +create table test(value DOUBLE FIELD, value1 DOUBLE FIELD); +INSERT INTO test VALUES (1, 1, 0),(2, 2, 0),(3, 1, 0),(4, 1, 0),(5, 1, 0),(6, 1.2, 0),(7, 1, 0),(8, 1, 0),(9, 2, 0),(10, 3, 0),(11, 4, 0),(12, 3, 0),(13, 2, 0),(14, 3, 0),(15, 4, 0),(16, 2, 0),(17, 1, 0),(18, 1, 0),(19, 1, 0),(20, 1, 0),(21, 1, 0),(41, 1, 0),(42, 2, 0),(43, 3, 0),(44, 4, 0),(45, 3, 0),(46, 2, 0),(47, 2, 0),(48, 2, 0),(49, 2, 0),(50, 2, 0),(51, 2, 0),(52, 2, 0); +``` + +可通过如下语句查询数据： + +```sql +select * from test; +``` + +图示效果如下： + +![](/img/featured-function-01.png) + +#### 5.4.2 基础趋势匹配 + +使用普通序列作为模板，查找数据中趋势相似的子序列。 + +```sql +select * from pattern_match( + data => test ORDER BY time, + time_col => 'time', + data_col => 'value', + pattern => '1.0,2.0,3.0,2.0,1.0,1.0,1.0,1.0,1.0', + smooth => 0.5, + threshold => 10.0, + width => 1000.0, + height => 500.0, + smooth_on_pattern=> false +); +``` + +图示效果 + +![](/img/featured-function-02.png) + +查询返回 3 组趋势匹配结果。相似度越小，表示匹配程度越高。 + +```text ++-----------+------------------+-----------------------------+-----+------+ +|match_index| similarity| time|value|value1| ++-----------+------------------+-----------------------------+-----+------+ +| 0| 4.637180787192477|1970-01-01T08:00:00.001+08:00| 1.0| 0.0| +| 0| 4.637180787192477|1970-01-01T08:00:00.002+08:00| 2.0| 0.0| +| 0| 4.637180787192477|1970-01-01T08:00:00.003+08:00| 1.0| 0.0| +| 0| 4.637180787192477|1970-01-01T08:00:00.004+08:00| 1.0| 0.0| +| 0| 4.637180787192477|1970-01-01T08:00:00.005+08:00| 1.0| 0.0| +| 0| 4.637180787192477|1970-01-01T08:00:00.006+08:00| 1.2| 0.0| +| 0| 4.637180787192477|1970-01-01T08:00:00.007+08:00| 1.0| 0.0| +| 0| 4.637180787192477|1970-01-01T08:00:00.008+08:00| 1.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.013+08:00| 2.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.014+08:00| 3.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.015+08:00| 4.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.016+08:00| 2.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.017+08:00| 1.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.018+08:00| 1.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.019+08:00| 1.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.020+08:00| 1.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.021+08:00| 1.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.041+08:00| 1.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.042+08:00| 2.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.043+08:00| 3.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.044+08:00| 4.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.045+08:00| 3.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.046+08:00| 2.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.047+08:00| 2.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.048+08:00| 2.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.049+08:00| 2.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.050+08:00| 2.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.051+08:00| 2.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.052+08:00| 2.0| 0.0| ++-----------+------------------+-----------------------------+-----+------+ +Total line number = 29 +It costs 0.126s +``` + +#### 5.4.3 调整匹配条件 + +以下示例均以 7.4.2 的基础查询为准，仅修改对应参数。 + +|调整目的|参数|基础值|调整值|调整效果| +|---|---|---|---|---| +|提高匹配精度|`threshold`|`10.0`|`1.0`|仅保留相似度小于 `1.0` 的结果，匹配结果由 3 组减少为 2 组| +|限制匹配时间跨度|`width`|`1000.0`|`10.0`|仅返回时间跨度小于 `10.0` 的子序列| +|限制数值波动范围|`height`|`500.0`|`2.0`|仅返回数值波动范围小于 `2.0` 的子序列| + +#### 5.4.4 正则模板匹配 + +正则模板可以用于匹配重复出现的趋势片段。 + +|重复符号|含义| +|---|---| +|`+`|前面的子序列至少出现 1 次| +|`*`|前面的子序列可以出现 0 次或多次| + +以下示例使用 `+` 匹配至少出现一次的子序列 `2.0,3.0,2.0`： + +```sql +select * from pattern_match( + data => test ORDER BY time, + time_col => 'time', + data_col => 'value', + pattern => '1.0,{2.0,3.0,2.0}+,1.0,1.0,1.0,1.0,1.0', + smooth => 0.5, + threshold => 100.0, + width => 1000.0, + height => 500.0, + smooth_on_pattern => false +); +``` + +如果允许子序列 `2.0,3.0,2.0` 出现零次或多次，可以将 `+` 修改为 `*`： + +```sql +pattern => '1.0,{2.0,3.0,2.0}*,1.0,1.0,1.0,1.0,1.0' +``` + +正则模板适用于周期性波动、重复峰值以及长度不固定的趋势匹配场景。 + +#### 5.4.5 模板平滑处理 + +当模板序列包含小幅波动时，可以通过 `smooth_on_pattern` 控制是否对模板序列进行平滑处理。 + +以下示例展示两种不同参数配置下的匹配结果： + +|配置|`threshold`|`smooth_on_pattern`|匹配效果| +|---|---|---|---| +|同时平滑原始序列和模板序列|`1.0`|`true`|可以匹配原始数据中的 `1.0,1.2,1.0` 波动片段| +|仅平滑原始序列|`10.0`|`false`|仍可返回其他趋势匹配结果，但不包含对应的 `1.0,1.2,1.0` 波动片段| + +同时平滑原始序列和模板序列： + +```sql +SELECT * +FROM pattern_match( + data => test ORDER BY time, + time_col => 'time', + data_col => 'value', + pattern => '1.0,1.2,1.0', + smooth => 0.5, + threshold => 1.0, + width => 1000.0, + height => 500.0, + smooth_on_pattern => true +); +``` + +在该配置下，原始序列和模板序列采用相同的平滑规则，可以匹配包含时间点 5、6、7 对应的 `1.0,1.2,1.0` 波动片段。 + +图示效果如下： + +![](/img/featured-function-03.png) + +仅平滑原始序列： + +```sql +SELECT * +FROM pattern_match( + data => test ORDER BY time, + time_col => 'time', + data_col => 'value', + pattern => '1.0,1.2,1.0', + smooth => 0.5, + threshold => 10.0, + width => 1000.0, + height => 500.0, + smooth_on_pattern => false +); +``` + +在该配置下，仅对原始序列进行平滑处理。查询仍可返回其他趋势匹配结果，但时间点 5、6、7 对应的 `1.0,1.2,1.0` 波动片段未被匹配。 + +图示效果如下： + +![](/img/featured-function-04.png) + +## 6. 频域分析函数 + +### 6.1 FFT + +#### 6.1.1 功能概述 + +表模型内置表值函数 `FFT` 用于对输入表各分区内的一个或多个实值数值序列执行一维快速傅里叶变换。输入数据必须通过 `ORDER BY` 指定时间列并按升序排列，也可通过 `PARTITION BY` 按设备或 Tag 进行分组。每个分区独立生成一组频率 bin，同一分区内的多个数值列共享该频率轴。 + +> V2.0.12 版本起支持 + +#### 6.1.2 函数定义 + +- 命名参数语法如下： + +```sql +SELECT * +FROM FFT( + DATA => {table_name | subquery} + [PARTITION BY partition_column [, ...]] + ORDER BY time_column, + [SAMPLE_INTERVAL => duration_literal,] + [N => positive_integer,] + [NORM => {'backward' | 'forward' | 'ortho'},] + [TIMECOL => 'time_column'] +); +``` + +- 位置参数语法如下： + +```sql +SELECT * +FROM TABLE( + FFT( + TABLE(table_name) + [PARTITION BY partition_column [, ...]] + ORDER BY time_column + [, duration_literal + [, positive_integer + [, norm + [, time_column_name]]]] + ) +); +``` + +参数的规范顺序为 `DATA, SAMPLE_INTERVAL, N, NORM, TIMECOL`。推荐使用**命名参数**，以提高复杂调用的可读性。 + +#### 6.1.3 参数说明 + +|参数|是否必填|类型|默认值|说明| +|---|---|---|---|---| +|`DATA`|是|表参数|无|指定 FFT 的输入数据。可以使用 `PARTITION BY` 分区，每个分区独立计算；省略时，全部数据作为一个分区。除时间列和分区列外，所有 `INT32`、`INT64`、`FLOAT` 和 `DOUBLE` 列均参与计算，其他类型的列自动忽略。输入中至少需要一个数值列，且参与计算的列不能包含 `NULL`。如需只计算部分数值列，可在输入子查询中提前选择。| +|`SAMPLE_INTERVAL`|否|正数时间间隔字面量|根据时间列推断|指定采样间隔，例如 `1ms`、`10ms` 或 `1s`，用于计算频率轴。未指定时，每个分区按照 `(最后一个时间戳 - 第一个时间戳) / (输入行数 - 1)` 推断。单行分区无法自动推断，必须显式指定；空分区直接返回 0 行。| +|`N`|否|正整数|当前分区的输入行数|指定 FFT 变换长度。小于输入行数时，使用按时间排序后的前 `N` 行；大于输入行数时，在序列尾部补 0。每个分区输出 `N` 行结果。`N` 最大为 `65536`，且需满足 `N × 2 × 参与计算的数值列数 ≤ 16777216`。| +|`NORM`|否|字符串|`backward`|指定 FFT 结果的归一化方式，大小写不敏感。`backward`：不进行归一化；`forward`：将结果除以变换长度 `N`；`ortho`：将结果除以 `√N`。| +|`TIMECOL`|否|字符串|`time`|指定时间列名称，对应列必须为 `TIMESTAMP` 类型。`DATA` 必须仅按照该列升序排列，不支持 `DESC`；每个分区内的时间戳必须严格递增，不允许重复。| + +- FFT 通常适用于等间隔采样的数据。本函数不会自动检测或修正不等间隔的数据。为确保频谱结果准确，请在计算前对数据进行插值或重采样；如果采样间隔已知，建议通过 `SAMPLE_INTERVAL` 显式指定。 + +- 当 `N` 小于输入行数时，函数仍会检查分区内的全部输入数据，但只使用前 `N` 行进行 FFT 计算。 + +**返回结果** + +`FFT` 返回每个分区的完整双边频谱，输出列顺序如下： + +```text +分区列... +frequency_index +frequency +<数值列名>_real +<数值列名>_imag +... +``` + +|输出列|类型|说明| +|---|---|---| +|`PARTITION BY` 指定的列|与输入类型一致|保留原列名和类型，位于结果集最前面。未使用 `PARTITION BY` 时不输出。| +|`frequency_index`|`INT64`|频率点索引，范围为 `0` 至 `N-1`。| +|`frequency`|`DOUBLE`|当前频率点对应的频率，单位为 Hz。| +|`<数值列名>_real`|`DOUBLE`|对应输入数值列的 FFT 实部。| +|`<数值列名>_imag`|`DOUBLE`|对应输入数值列的 FFT 虚部。| + +对于变换长度 `N` 和采样间隔 `d`（单位为秒），`frequency` 的计算方式如下： + +|**frequency_index 范围**|**frequency**| +|---|---| +|`frequency_index < ceil(N / 2)`|`frequency_index / (N × d)`| +|其他情况|`(frequency_index - N) / (N × d)`| + +频率按照零频率、正频率、负频率的顺序排列，单位为 Hz。同一分区内，所有数值列共享相同的频率轴。 + +#### 6.1.4 注意事项 + +- `FFT` 函数名不区分大小写。每个分区独立执行一维 FFT。 + +- 当前版本仅支持实数输入，不支持复数输入、`DFT` 或 `IFFT`。 + +- 当前版本仅输出完整频谱的实部和虚部，不支持单边频谱、振幅或相位输出。 + +- 当前版本不支持 `VALUE`、`SAMPLE_RATE` 和 `SPECTRUM` 参数。 + +#### 6.1.5 使用示例 + +以下示例直接使用 `table1` 中的数据。由于 `temperature` 包含 `NULL`，需要在输入子查询中提前过滤。 + +- 示例 1：按设备执行 FFT + +```sql +SELECT * +FROM FFT( + DATA => ( + SELECT time, device_id, temperature + FROM table1 + WHERE plant_id = '1001' + AND temperature IS NOT NULL + ) PARTITION BY device_id ORDER BY time +) +ORDER BY device_id, frequency_index; +``` + +函数分别对每个设备的 `temperature` 执行 FFT。未指定 `N` 和 `SAMPLE_INTERVAL` 时，函数根据各分区的输入行数确定变换长度，并根据时间列推断采样间隔。本例中的采样间隔为 1 分钟。 + +```text ++---------+---------------+---------------------+----------------------+---------------------+ +|device_id|frequency_index| frequency| temperature_real| temperature_imag| ++---------+---------------+---------------------+----------------------+---------------------+ +| 100| 0| 0.0| 180.0| 0.0| +| 100| 1|-0.008333333333333333| 0.0| 0.0| +| 101| 0| 0.0| 255.0| -6.67572021484375E-6| +| 101| 1| 0.005555555555555556|-4.6126147026370745E-6|-5.266308107820805E-6| +| 101| 2|-0.005555555555555556| -1.9073486328125E-6| 3.303624680484063E-6| ++---------+---------------+---------------------+----------------------+---------------------+ +``` + +- 示例 2：指定变换长度和归一化方式 + +```sql +SELECT * +FROM FFT( + DATA => ( + SELECT time, device_id, temperature + FROM table1 + WHERE plant_id = '1001' + AND temperature IS NOT NULL + ) PARTITION BY device_id ORDER BY time, + SAMPLE_INTERVAL => 1m, + N => 4, + NORM => 'ortho' +) +ORDER BY device_id, frequency_index; +``` + +函数将每个设备的变换长度设置为 4。输入数据不足 4 行时，在数值序列尾部补 0；FFT 结果按照 `1/√N` 进行归一化。每个设备均返回 4 行频谱结果。 + +```text ++---------+---------------+---------------------+----------------------+----------------+ +|device_id|frequency_index| frequency| temperature_real|temperature_imag| ++---------+---------------+---------------------+----------------------+----------------+ +| 100| 0| 0.0| 90.0| 0.0| +| 100| 1| 0.004166666666666667| 45.0| -45.0| +| 100| 2|-0.008333333333333333| 0.0| 0.0| +| 100| 3|-0.004166666666666667| 45.0| 45.0| +| 101| 0| 0.0| 127.5| 0.0| +| 101| 1| 0.004166666666666667| 2.602374475716698E-15| -42.5| +| 101| 2|-0.008333333333333333| 42.5| 0.0| +| 101| 3|-0.004166666666666667|-2.602374475716698E-15| 42.5| ++---------+---------------+---------------------+----------------------+----------------+ +``` + +- 示例 3：数值列包含 `NULL` + +```sql +SELECT * +FROM FFT( + DATA => ( + SELECT time, device_id, temperature + FROM table1 + WHERE plant_id = '1001' + ) PARTITION BY device_id ORDER BY time +); +``` + +由于参与 FFT 计算的数值列不能包含 `NULL`，查询执行失败： + +```text +Msg: org.apache.iotdb.jdbc.IoTDBSQLException: 701: FFT does not support null values in column [temperature]. +``` + +可以在输入子查询中使用 `IS NOT NULL` 过滤空值后再执行 FFT。 + +### 6.2 lowPass + +#### 6.2.1 功能描述 + +`lowPass` 是低通滤波函数，对输入表中的数值列执行频域滤波，保留低频成分并衰减高频成分，适用于信号平滑和高频噪声去除。参数 `wpass` 为归一化通带频率，取值范围为 `(0, 1)`，用于指定允许通过的频率上限。 + +> V2.0.12 版本起支持 + +#### 6.2.2 函数定义 + +- 命名参数语法如下： + +```sql +SELECT * +FROM lowPass( + DATA => input_table PARTITION BY partition_column ORDER BY time_column, + TIMECOL => 'time_column', + WPASS => passband_frequency +); +``` + +- 位置参数语法如下： + +```sql +SELECT * +FROM lowPass( + input_table PARTITION BY partition_column ORDER BY time_column, + 'time_column', + passband_frequency +); +``` + +#### 6.2.3 参数说明 + +|参数|类型|必填|说明| +|---|---|---|---| +|`DATA`|表参数（`SET SEMANTIC`）|是|输入表或子查询。`ORDER BY` 必须仅指定时间列并按**升序**排列；`PARTITION BY` 可选，不指定时整张表作为一个分区。| +|`TIMECOL`|字符串|是|时间列名称，必须为时间戳类型，且只能指定一个时间列。| +|`WPASS`|`DOUBLE`|是|归一化通带频率，取值范围为 `(0, 1)`；低于该频率的成分保留，高于该频率的成分衰减。| + +**计算规则** + +- `TIMECOL` 和 `PARTITION BY` 指定的列不参与计算，但会保留在输出中。 + +- 除时间列和分区列外，仅 `INT32`、`INT64`、`FLOAT`、`DOUBLE` 列参与计算；不存在可计算列或存在其他类型列时，分析阶段报错。 + +- 参与计算列中的 `NULL` 值不参与该列的滤波计算。 + +**返回结果** + +- 输出分区列（如指定）和时间列，列名及类型保持不变。 + +- 每个参与计算的数值列输出一个滤波结果列，类型统一为 `DOUBLE`。每个结果列的测点数与对应输入列一致，结果按时间戳升序输出。 + +#### 6.2.4 使用示例 + +以下示例直接使用 `table1` 中的数据。 + +```sql +SELECT * +FROM lowPass( + DATA => ( + SELECT time, device_id, temperature + FROM table1 + WHERE temperature is not null + ) PARTITION BY device_id ORDER BY time, + TIMECOL => 'time', + WPASS => 0.3 +); +``` + +该示例按设备分区，对 temperature 列执行低通滤波，输出 `device_id`、`time` 以及滤波后的数值列。 + +```text ++---------+-----------------------------+--------------------+ +|device_id| time|lowpass(temperature)| ++---------+-----------------------------+--------------------+ +| 100|2024-11-26T13:37:00.000+08:00| 88.0| +| 100|2024-11-26T13:38:00.000+08:00| 88.0| +| 100|2024-11-28T08:00:00.000+08:00| 87.99999999999999| +| 100|2024-11-28T10:00:00.000+08:00| 87.99999999999999| +| 100|2024-11-28T11:00:00.000+08:00| 88.00000000000001| +| 100|2024-11-29T18:30:00.000+08:00| 88.0| +| 101|2024-11-27T16:39:00.000+08:00| 86.66666666666667| +| 101|2024-11-27T16:40:00.000+08:00| 86.66666666666667| +| 101|2024-11-27T16:41:00.000+08:00| 86.66666666666664| +| 101|2024-11-29T10:00:00.000+08:00| 86.66666666666664| +| 101|2024-11-30T09:30:00.000+08:00| 86.66666666666667| +| 101|2024-11-30T14:30:00.000+08:00| 86.66666666666664| ++---------+-----------------------------+--------------------+ +``` + +### 6.3 highPass + +#### 6.3.1 功能描述 + +`highPass` 是高通滤波函数，对输入表中的数值列执行频域滤波，保留高频成分并衰减低频成分，适用于提取快速变化、跳变和高频噪声等信号成分。参数 `wpass` 为归一化通带频率，取值范围为 `(0, 1)`，用于指定允许通过的频率下限。 + +> V2.0.12 版本起支持 + +#### 6.3.2 函数定义 + +- 命名参数语法如下： + +```sql +SELECT * +FROM highPass( + DATA => input_table PARTITION BY partition_column ORDER BY time_column, + TIMECOL => 'time_column', + WPASS => passband_frequency +); +``` + +- 位置参数语法如下： + +```sql +SELECT * +FROM highPass( + input_table PARTITION BY partition_column ORDER BY time_column, + 'time_column', + passband_frequency +); +``` + +#### 6.3.3 参数说明 + +|参数|类型|必填|说明| +|---|---|---|---| +|`DATA`|表参数（`SET SEMANTIC`）|是|输入表或子查询。`ORDER BY` 必须仅指定时间列并按升序排列；`PARTITION BY` 可选，不指定时整张表作为一个分区。| +|`TIMECOL`|字符串|是|时间列名称，必须为时间戳类型，且只能指定一个时间列。| +|`WPASS`|`DOUBLE`|是|归一化通带频率，取值范围为 `(0, 1)`；高于该频率的成分保留，低于该频率的成分衰减。| + +**输入列和计算规则** + +- `TIMECOL` 和 `PARTITION BY` 指定的列不参与计算，但会保留在输出中。 + +- 除时间列和分区列外，仅 `INT32`、`INT64`、`FLOAT`、`DOUBLE` 列参与计算；不存在可计算列或存在其他类型列时，分析阶段报错。 + +- 参与计算列中的 `NULL` 值不参与该列的滤波计算。 + +**返回结果** + +- 输出分区列（如指定）和时间列，列名及类型保持不变。 + +- 每个参与计算的数值列输出一个滤波结果列，类型统一为 `DOUBLE`。每个结果列的测点数与对应输入列一致，结果按时间戳升序输出。 + +#### 6.3.4 使用示例 + +```sql +SELECT * +FROM highPass( + DATA => ( + SELECT time, device_id, temperature + FROM table1 + WHERE temperature is not null + ) PARTITION BY device_id ORDER BY time, + TIMECOL => 'time', + WPASS => 0.3 +); +``` + +该示例按设备分区，对 temperature 列执行高通滤波，输出 `device_id`、`time` 以及滤波后的数值列。 + +```text ++---------+-----------------------------+----------------------+ +|device_id| time| highpass(temperature)| ++---------+-----------------------------+----------------------+ +| 100|2024-11-26T13:37:00.000+08:00| 2.000000000000029| +| 100|2024-11-26T13:38:00.000+08:00| 2.000000000000004| +| 100|2024-11-28T08:00:00.000+08:00| -2.9999999999999702| +| 100|2024-11-28T10:00:00.000+08:00| -3.000000000000013| +| 100|2024-11-28T11:00:00.000+08:00|-4.182831670630972E-14| +| 100|2024-11-29T18:30:00.000+08:00| 1.9999999999999953| +| 101|2024-11-27T16:39:00.000+08:00| -1.666666666666642| +| 101|2024-11-27T16:40:00.000+08:00| -1.6666666666666519| +| 101|2024-11-27T16:41:00.000+08:00| -1.666666666666645| +| 101|2024-11-29T10:00:00.000+08:00| -1.666666666666701| +| 101|2024-11-30T09:30:00.000+08:00| 3.3333333333333015| +| 101|2024-11-30T14:30:00.000+08:00| 3.333333333333338| ++---------+-----------------------------+----------------------+ +``` + +## 7. 时序相关性分析函数 + +### 7.1 Xcorr + +#### 7.1.1 功能描述 + +`Xcorr` 是相关性分析函数，对输入的两列数值序列进行互相关分析，度量两条序列在不同时间滞后（`lag`）下的形状相似度。函数将一条序列逐步平移，与另一条序列逐点相乘并求和，输出相似度曲线，可用于发现两路信号之间的延迟关系和相似模式。 + +> V2.0.12 版本起支持 + +#### 7.1.2 函数定义 + +- 命名参数语法如下： + +```sql +SELECT * +FROM Xcorr( + DATA => ( + SELECT time_column, signal_1, signal_2, partition_column + FROM input_table + ) PARTITION BY partition_column ORDER BY time_column, + TIMECOL => 'time_column' +); +``` + +- 位置参数语法如下： + +```sql +SELECT * +FROM Xcorr( + ( SELECT time_column, signal_1, signal_2, partition_column + FROM input_table + ) PARTITION BY partition_column ORDER BY time_column, + 'time_column' +); +``` + +#### 7.1.3 参数说明 + +|参数|类型|必填|说明| +|---|---|---|---| +|`DATA`|表参数（`SET SEMANTIC`）|是|输入表或子查询。`ORDER BY` 必须指定时间列并按升序排列；`PARTITION BY` 可选。去除时间列和分区列后，必须恰好剩余两列数值列。| +|`TIMECOL`|字符串|是|时间列名称，必须为时间戳类型，且只能指定一个时间列。| + +**计算规则** + +- 时间列和分区列不参与互相关计算。 + +- 去除上述列后，必须恰好存在两列，且两列类型均为 `INT32`、`INT64`、`FLOAT` 或 `DOUBLE`；否则分析阶段报错。 + +- 参与计算的两列中出现 `NULL` 时，该位置按 `0` 参与计算。 + +- 每个分区内时间列必须严格升序。 + +- 计算时按函数定义进行归一化处理。 + +**返回结果** + +- 输出 `PARTITION BY` 指定的分区列（如指定），列名及类型保持不变。 + +- 对参与计算的两列，生成一个结果列，类型为double。 + +#### 7.1.4 使用示例 + +以下示例使用 SQL 手册示例数据中的 `table1`，并通过子查询投影两列数值序列 `temperature` 和 `humidity` 参与互相关计算。 + +```sql +SELECT * +FROM Xcorr( + DATA => ( + SELECT time, device_id, temperature, humidity + FROM table1 + WHERE temperature IS NOT NULL + AND humidity IS NOT NULL + ) + PARTITION BY device_id + ORDER BY time, + TIMECOL => 'time' +); +``` + +该示例按设备分区，对 temperature 和 humidity 两列进行互相关分析，输出 `device_id` 以及相关性分析结果列。 + +```text ++---------+----------------------------+ +|device_id|xcorr(temperature, humidity)| ++---------+----------------------------+ +| 100| 3186.0001373291016| +| 100| 3622.5| +| 100| 3412.0000203450522| +| 100| 3318.9249839782715| +| 100| 3292.9599586486815| +| 100| 3324.7749156951904| +| 100| 3080.0999348958335| +| 100| 3123.8998641967773| +| 100| 3158.9998626708984| +| 101| 2957.9999351501465| +| 101| 3061.9999980926514| +| 101| 3100.166645050049| +| 101| 3172.5| +| 101| 3176.999931335449| ++---------+----------------------------+ +``` diff --git a/src/zh/UserGuide/latest-Table/SQL-Manual/Featured-Functions_apache.md b/src/zh/UserGuide/latest-Table/SQL-Manual/Featured-Functions_apache.md index 20abca659..105c30828 100644 --- a/src/zh/UserGuide/latest-Table/SQL-Manual/Featured-Functions_apache.md +++ b/src/zh/UserGuide/latest-Table/SQL-Manual/Featured-Functions_apache.md @@ -213,6 +213,213 @@ GROUP BY 1, device_id; +---------+---------+--------+ ``` +### 1.3 `M4` 函数 + +#### 1.3.1 功能描述 + +`M4` 是表模型内置的多态表值函数，用于对时序数据进行降采样。函数在每个窗口内提取首点（First）、尾点（Last）、最小值点（Bottom）和最大值点（Top），在减少返回数据量的同时尽量保持趋势图形状。与标量函数或聚合函数不同，`M4` 接收一个表参数并返回结果表，必须在 `FROM` 子句中调用。 + +- 首点：窗口内时间戳最小且值不为 `NULL` 的点。 + +- 尾点：窗口内时间戳最大且值不为 `NULL` 的点。 + +- 最小值点和最大值点：窗口内值最小或最大的点；值相同时保留按输入扫描顺序遇到的第一个点。 + +- 支持时间窗口和点数窗口两种模式。`SIZE` 使用时间长度字面量（例如 `1h`、`10m`）时为时间窗口；使用正整数（例如 `100`）时为点数窗口。 + +> V2.0.12 版本起支持 + +#### 1.3.2 函数定义 + +```sql +SELECT * +FROM M4( + DATA => input_table PARTITION BY partition_column ORDER BY time_column, + TIMECOL => 'time_column', + [ORIGIN => origin_timestamp,] + SIZE => window_duration | window_point_count, + [SLIDE => window_duration | window_point_count] +); +``` + +`DATA` 是集合语义表参数。使用时必须指定 `ORDER BY`，且排序列只能是 `TIMECOL` 指定的时间列并按升序排列；`PARTITION BY` 可选，不指定时整张输入表作为一个分区处理。参数可按名称或按声明顺序传递，建议使用命名参数。 + +#### 1.3.3 参数说明 + +|参数|类型|必填|说明| +|---|---|---|---| +|`DATA`|表参数（SET SEMANTIC）|是|输入表或子查询。可指定 `PARTITION BY` 和 `ORDER BY`。| +|`TIMECOL`|字符串|是|时间列名。该列必须为 `TIMESTAMP` 类型，且 `ORDER BY` 只能包含该列。| +|`ORIGIN`|时间戳|否|时间窗口的对齐原点，默认值为 `0`。点数窗口模式不使用该参数。| +|`SIZE`|时间长度或 `INT64`|是|时间窗口的窗口长度，或点数窗口的点数大小；必须为正数。| +|`SLIDE`|与 `SIZE` 同类型|否|窗口滑动步长，默认与 `SIZE` 相同。必须与 `SIZE` 使用相同窗口模式。| + +**计算规则** + +- `TIMECOL` 指定列不参与 M4 计算。 + +- `PARTITION BY` 指定列不参与计算，但保留在输出结果中。 + +- 除上述列外，所有可比较类型列均参与 M4 计算。 + +- 排除上述列后，若无可比较类型列，或存在不可比较类型列，分析阶段直接报错。 + +- 参与列值为 `NULL` 的记录，不参与该列的 `first`、`last`、`bottom`、`top` 计算。 + +- 各参与列先计算并去重 `first`、`last`、`bottom`、`top` 特征点，再按时间戳升序输出。 + +**返回结果** + +- 时间窗口模式：固定生成 `window_start`、`window_end`（时间戳类型），表示窗口范围 `[window_start, window_end)`。 + +- 点数窗口模式：固定生成 `window_index`（长整数类型），表示窗口序号；在每个 `PARTITION BY` 分区内独立计算，从 `0` 开始递增。 + +- PARTITION BY 列：按原列名、原类型保留输出，多个列按声明顺序排列。`OBJECT` 类型不支持作为分区列；`BLOB` 类型可保留输出，但不参与 M4 计算。 + +- 动态生成列：每个参与 M4 计算的列 `c` 生成 `c_time`（特征点时间戳）和 `c`（特征点值）两列，类型与输入列保持一致。多列结果按输入列顺序展开。 + +补充说明：窗口输出行数取所有参与列特征点数量的最大值。特征点较少的列，其剩余行的 `c_time` 和 `c` 填充 `NULL`。 + +#### 1.3.4 使用示例 + +以下示例均使用[示例数据](../Reference/Sample-Data.md)中的 `table1`。由于 `table1` 中除时间列和分区列外还包含其他字段，因此示例通过子查询仅选择 `time`、`device_id`、`temperature` 和 `humidity` 参与 M4 计算。 + +- 示例 1：时间窗口模式 + +以 `device_id = '101'` 的数据为例，每 3 分钟划分一个窗口。`SIZE => 3m` 表示使用时间窗口模式，窗口长度为 3 分钟。窗口采用半开区间 `[window_start, window_end)`，每个窗口分别提取 `temperature` 和 `humidity` 列的首点、尾点、最小值点和最大值点，并按照特征点时间戳升序输出。 + +```sql +SELECT window_start, + window_end, + device_id, + temperature_time, + temperature, + humidity_time, + humidity +FROM M4( + DATA => ( + SELECT time, device_id, temperature, humidity + FROM table1 + WHERE device_id = '101' + AND time >= 2024-11-27 16:38:00 + AND time <= 2024-11-27 16:44:00 + ) PARTITION BY device_id ORDER BY time, + TIMECOL => 'time', + SIZE => 3m +); +``` + +结果： + +```text ++-----------------------------+-----------------------------+---------+-----------------------------+-----------+-----------------------------+--------+ +| window_start| window_end|device_id| temperature_time|temperature| humidity_time|humidity| ++-----------------------------+-----------------------------+---------+-----------------------------+-----------+-----------------------------+--------+ +|2024-11-27T16:36:00.000+08:00|2024-11-27T16:39:00.000+08:00| 101| null| null|2024-11-27T16:38:00.000+08:00| 35.1| +|2024-11-27T16:39:00.000+08:00|2024-11-27T16:42:00.000+08:00| 101|2024-11-27T16:39:00.000+08:00| 85.0|2024-11-27T16:39:00.000+08:00| 35.3| +|2024-11-27T16:39:00.000+08:00|2024-11-27T16:42:00.000+08:00| 101|2024-11-27T16:41:00.000+08:00| 85.0| null| null| +|2024-11-27T16:42:00.000+08:00|2024-11-27T16:45:00.000+08:00| 101| null| null|2024-11-27T16:42:00.000+08:00| 35.2| ++-----------------------------+-----------------------------+---------+-----------------------------+-----------+-----------------------------+--------+ +``` + +- 示例 2：点数窗口模式 + +以 `device_id = '101'` 的数据为例，每 3 个数据点划分一个窗口。`SIZE => 3` 表示使用点数窗口模式，每 3 行数据划分一个窗口。窗口序号通过 `window_index` 标识，并在每个分区内从 `0` 开始计算。 + +```sql +SELECT window_index, + device_id, + temperature_time, + temperature, + humidity_time, + humidity +FROM M4( + DATA => ( + SELECT time, device_id, temperature, humidity + FROM table1 + WHERE device_id = '101' + AND time >= 2024-11-27 16:38:00 + AND time <= 2024-11-27 16:44:00 + ) PARTITION BY device_id ORDER BY time, + TIMECOL => 'time', + SIZE => 3 +); +``` + +结果： + +```text ++------------+---------+-----------------------------+-----------+-----------------------------+--------+ +|window_index|device_id| temperature_time|temperature| humidity_time|humidity| ++------------+---------+-----------------------------+-----------+-----------------------------+--------+ +| 0| 101|2024-11-27T16:39:00.000+08:00| 85.0|2024-11-27T16:38:00.000+08:00| 35.1| +| 0| 101|2024-11-27T16:40:00.000+08:00| 85.0|2024-11-27T16:39:00.000+08:00| 35.3| +| 1| 101|2024-11-27T16:41:00.000+08:00| 85.0|2024-11-27T16:42:00.000+08:00| 35.2| ++------------+---------+-----------------------------+-----------+-----------------------------+--------+ +``` + +如果需要设置窗口滑动步长，可以同时指定 `SLIDE`： + +```sql +SELECT window_index, + device_id, + temperature_time, + temperature, + humidity_time, + humidity +FROM M4( + DATA => ( + SELECT time, device_id, temperature, humidity + FROM table1 + WHERE device_id = '101' + AND time >= 2024-11-27 16:38:00 + AND time <= 2024-11-27 16:44:00 + ) PARTITION BY device_id ORDER BY time, + TIMECOL => 'time', + SIZE => 3, + SLIDE => 2 +); +``` + +上述调用表示窗口大小为 3 个数据点，窗口每次向后滑动 2 个数据点。 + +结果： + +```text ++------------+---------+-----------------------------+-----------+-----------------------------+--------+ +|window_index|device_id| temperature_time|temperature| humidity_time|humidity| ++------------+---------+-----------------------------+-----------+-----------------------------+--------+ +| 0| 101|2024-11-27T16:39:00.000+08:00| 85.0|2024-11-27T16:38:00.000+08:00| 35.1| +| 0| 101|2024-11-27T16:40:00.000+08:00| 85.0|2024-11-27T16:39:00.000+08:00| 35.3| +| 1| 101|2024-11-27T16:40:00.000+08:00| 85.0|2024-11-27T16:42:00.000+08:00| 35.2| +| 1| 101|2024-11-27T16:41:00.000+08:00| 85.0| null| null| +| 2| 101| null| null|2024-11-27T16:42:00.000+08:00| 35.2| ++------------+---------+-----------------------------+-----------+-----------------------------+--------+ +``` + +- 示例 3：非法示例——`ORDER BY` 未指定时间列 + +`M4` 要求 `DATA` 参数必须指定 `ORDER BY`，且排序列必须与 `TIMECOL` 参数指定的时间列一致。下面的示例错误地使用 `temperature` 作为排序列： + +```sql +SELECT * +FROM M4( + DATA => ( + SELECT time, device_id, temperature, humidity + FROM table1 + WHERE device_id = '101' + ) PARTITION BY device_id ORDER BY temperature, + TIMECOL => 'time', + SIZE => 3m +); +``` + +报错： + +```text +Msg: org.apache.iotdb.jdbc.IoTDBSQLException: 701: The ORDER BY clause of the DATA argument must contain exactly the time column specified by the TIMECOL argument. +``` + ## 2. DIFF函数 ### 2.1 功能概述 @@ -522,27 +729,51 @@ IoTDB> SELECT first(time) as window_start, last(time) as window_end, stock_id, a #### 3.4.1 功能描述 -CAPACITY 函数用于按数据点数（行数）分窗，每个窗口最多有 SIZE 行数据。 +CAPACITY 函数用于按数据点数（行数）分窗，每个窗口最多有 SIZE 行数据。自 V2.0.12 起支持通过参数 SLIDE 指定相邻窗口起点之间间隔的行数。 #### 3.4.2 函数定义 ```sql -CAPACITY(data [PARTITION BY(pkeys, ...)] [ORDER BY(okeys, ...)], size) +CAPACITY( + DATA => + [PARTITION BY [, ...]] + [ORDER BY [, ...]], + SIZE => + [, SLIDE => ] +) ``` #### 3.4.3 参数说明 -| 参数名 | 参数类型 | 参数属性 | 描述 | -| -------- | ---------- | -------------------------- | ---------------------------------------- | -| DATA | 表参数 | SET SEMANTICPASS THROUGH | 输入表通过 pkeys、okeys 指定分区和排序 | -| SIZE | 标量参数 | 长整数类型 | 窗口大小 | +|参数|类型|是否必填|说明| +|---|---|---|---| +|DATA|Table|是|输入表或子查询，支持 PARTITION BY 和 ORDER BY，透传输入列。| +|SIZE|INT64 Scalar|是|窗口容量，必须为正数，表示单个窗口最多覆盖的行数。| +|SLIDE|INT64 Scalar|否|窗口滑动步长，显式传入时必须为正数；省略时默认等于 SIZE。| + +- 指定 PARTITION BY 时，每个分区独立从 0 开始计算行号和窗口编号； + +- 指定 ORDER BY 时，按排序结果确定行号顺序。 + +- 窗口编号从 0 开始，第 k 个窗口覆盖的行号范围为 `[k * SLIDE, k * SLIDE + SIZE)`。 + +- 不同参数关系对应的窗口行为如下： + +|参数关系|窗口类型|行为| +|---|---|---| +|`SLIDE < SIZE`|重叠窗口|相邻窗口有重叠，同一输入行可能属于多个窗口，并以不同的 `window_index` 输出多行。| +|`SLIDE = SIZE`|非重叠窗口|相邻窗口首尾相接，行为与历史版本的 CAPACITY(DATA => ..., SIZE => n) 一致。| +|`SLIDE > SIZE`|间隔窗口|相邻窗口之间存在空档，落在空档内的输入行不输出。| #### 3.4.4 返回结果 CAPACITY 函数的返回结果列包含： -* window\_index: 窗口编号 -* 映射列：DATA 参数的所有输入列 +- window_index: 窗口编号，为 INT64 类型，从 0 开始。 + +- DATA 参数透传列：保持输入表对应列的名称和类型。 + +返回结果可直接查询，也可继续基于 window_index 和业务分组列进行聚合。 #### 3.4.5 使用示例 @@ -569,6 +800,23 @@ IoTDB> SELECT first(time) as start_time, last(time) as end_time, stock_id, avg(p |2021-01-01T09:05:00.000+08:00|2021-01-01T09:07:00.000+08:00| AAPL|101.5| |2021-01-01T09:09:00.000+08:00|2021-01-01T09:09:00.000+08:00| AAPL|102.0| +-----------------------------+-----------------------------+--------+-----+ + +-- 使用 SLIDE 参数生成重叠窗口（SLIDE < SIZE） +IoTDB> SELECT * FROM CAPACITY(DATA => bid PARTITION BY stock_id ORDER BY time, SIZE => 2, SLIDE => 1); ++------------+-----------------------------+--------+-----+ +|window_index| time|stock_id|price| ++------------+-----------------------------+--------+-----+ +| 0|2021-01-01T09:05:00.000+08:00| AAPL|100.0| +| 0|2021-01-01T09:07:00.000+08:00| AAPL|103.0| +| 1|2021-01-01T09:07:00.000+08:00| AAPL|103.0| +| 1|2021-01-01T09:09:00.000+08:00| AAPL|102.0| +| 2|2021-01-01T09:09:00.000+08:00| AAPL|102.0| +| 0|2021-01-01T09:06:00.000+08:00| TESL|200.0| +| 0|2021-01-01T09:07:00.000+08:00| TESL|202.0| +| 1|2021-01-01T09:07:00.000+08:00| TESL|202.0| +| 1|2021-01-01T09:15:00.000+08:00| TESL|195.0| +| 2|2021-01-01T09:15:00.000+08:00| TESL|195.0| ++------------+-----------------------------+--------+-----+ ``` ### 3.5 TUMBLE @@ -695,3 +943,856 @@ IoTDB> SELECT window_start, window_end, stock_id, avg(price) as avg FROM CUMULAT +-----------------------------+-----------------------------+--------+------------------+ ``` + +## 4. `read_tsfile` 表值函数 + +### 4.1 功能概述 + +用于从外部 TsFile 文件或目录读取表模型数据。`read_tsfile` 返回一个临时表数据源，查询时可像普通表一样使用 `SELECT`、`WHERE`、`ORDER BY`、`LIMIT`、`OFFSET`、聚合等能力，对外部 TsFile 数据进行只读分析和临时处理，不会将处理结果写回外部 TsFile，也不会修改 IoTDB 内部数据，不支持通过该入口写入、删除、修改或注册外部 TsFile。 + +> V2.0.12 版本起支持 + +### 4.2 函数定义 + +```sql +SELECT * +FROM TABLE( + read_tsfile( + TABLE_NAME => '', + PATHS => '' + ) +); +``` + +### 4.3 参数说明 + +|参数|必填|说明| +|---|---|---| +|`PATHS`|是|TsFile 文件路径或目录路径。单路径直接传入字符串；多路径在同一个字符串中使用英文逗号分隔，例如 `'/data/a.tsfile,/data/b.tsfile,/data/history'`。
- 指定文件时，系统会直接尝试打开该文件；
- 指定目录时，系统会递归遍历目录下以 `.tsfile` 为后缀的文件，并根据文件首尾的 MAGIC NUMBER 识别有效 TsFile。文件内容不符合 TsFile 格式、无法识别、不完整或版本不是 TsFile V4 的文件不参与本次查询。
注意：指定的路径必须对执行查询的 DataNode 进程可见，并且 DataNode 进程需要具备相应的文件读取权限。禁止指定 DataNode 的任意数据目录，否则查询会报错。| +|`TABLE_NAME`|否|需要读取的表名。填写该参数时，系统只读取包含目标表 Schema 的有效文件；如果所有有效文件中都不存在目标表 Schema，查询报错。
当外部 TsFile 中可以唯一推断出待读取的表名时，可以省略该参数；省略时要求每个参与读取的有效文件中都有且只有一个表，并且所有有效文件推断出的表名一致。如果某个有效文件包含多个表，或者不同有效文件中的唯一表名不一致，查询报错。| +|`DEVICE_METADATA_INFO_SWAP_THRESHOLD` |否|查询时如果文件数和设备数量都很多，为了避免占用过多内存可能会将一些信息写成交换文件，这个参数可指定一个 Long 值，当达到此大小时强制触发交换临时文件，默认值为 8MB| + +**Schema 合并规则** + +读取多个 TsFile 时，系统会在确定目标表后，合并该表的 Schema，并将合并后的 Schema 作为本次查询的输出 Schema，且仅用于本次查询，不会注册到数据库元数据中。其合并规则如下： + +- tag 列按顺序合并。存在冲突时要求前缀一致；同一位置 tag 名称不同时报错；允许后续位置追加 tag 列。 + +- time 列名称必须一致，存在冲突时报错。 + +- field 列对顺序无要求，不同文件可以包含不同 field 列，系统会进行增量合并；同名 field 列的数据类型必须兼容，否则报错。 + +- 不同 TsFile 中同名列的列类型必须一致，否则报错。 + +**返回结果** + +- 返回指定表的数据结果集。 + +- 输出列来自外部 TsFile 中目标表的合并 Schema。 + +- `time` 列为时间列，列名取决于 TsFile 中记录的时间列名称。 + +- tag 列表示设备标识信息。 + +- field 列表示实际测点数据。 + +### 4.4 使用示例 + +假设在`/data/backup/` 路径下有文件 `1.tsfile`， 该文件中的元数据和数据可参考 [示例数据](../Reference/Sample-Data.md) 中的 `table1` + +1. 自动推断表名读取 TsFile。 + +该场景要求 `/data/backup/1.tsfile` 中有且只有一个表。 + +```sql +SELECT * +FROM TABLE( + read_tsfile( + PATHS => '/data/backup/1.tsfile' + ) +) +LIMIT 10; +``` + +返回结果 + +```text ++-----------------------------+------+--------+---------+--------+-----------+-----------+--------+------+-----------------------------+ +| time|region|plant_id|device_id|model_id|maintenance|temperature|humidity|status| arrival_time| ++-----------------------------+------+--------+---------+--------+-----------+-----------+--------+------+-----------------------------+ +|2024-11-29T10:00:00.000+08:00| 上海| 3001| 101| D| 360| 85.0| null| null|2024-11-29T10:00:13.000+08:00| +|2024-11-29T11:00:00.000+08:00| 上海| 3002| 100| E| 180| null| 45.1| true| null| +|2024-11-29T18:30:00.000+08:00| 上海| 3002| 100| E| 180| 90.0| 35.4| true|2024-11-29T18:30:15.000+08:00| +|2024-11-30T09:30:00.000+08:00| 上海| 3002| 101| F| 360| 90.0| 35.2| true| null| +|2024-11-30T14:30:00.000+08:00| 上海| 3002| 101| F| 360| 90.0| 34.8| true|2024-11-30T14:30:17.000+08:00| +|2024-11-26T13:37:00.000+08:00| 北京| 1001| 100| A| 180| 90.0| 35.1| true|2024-11-26T13:37:34.000+08:00| +|2024-11-26T13:38:00.000+08:00| 北京| 1001| 100| A| 180| 90.0| 35.1| true|2024-11-26T13:38:25.000+08:00| +|2024-11-28T08:00:00.000+08:00| 上海| 3001| 100| C| 90| 85.0| null| null|2024-11-28T08:00:09.000+08:00| +|2024-11-28T09:00:00.000+08:00| 上海| 3001| 100| C| 90| null| 40.9| true| null| +|2024-11-28T10:00:00.000+08:00| 上海| 3001| 100| C| 90| 85.0| 35.2| null|2024-11-28T10:00:11.000+08:00| ++-----------------------------+------+--------+---------+--------+-----------+-----------+--------+------+-----------------------------+ +Total line number = 10 +It costs 0.159s +``` + +2. 读取目录下的 TsFile。 + +```sql +SELECT time, device_id, temperature +FROM TABLE( + read_tsfile( + TABLE_NAME => 'table1', + PATHS => '/data/backup' + ) +) +WHERE time >= 2024-11-28 00:00:00 AND time < 2024-11-29 00:00:00 ; +``` + +返回结果 + +```text ++-----------------------------+---------+-----------+ +| time|device_id|temperature| ++-----------------------------+---------+-----------+ +|2024-11-28T08:00:00.000+08:00| 100| 85.0| +|2024-11-28T09:00:00.000+08:00| 100| null| +|2024-11-28T10:00:00.000+08:00| 100| 85.0| +|2024-11-28T11:00:00.000+08:00| 100| 88.0| ++-----------------------------+---------+-----------+ +Total line number = 4 +It costs 0.116s +``` + +3. 按 tag 聚合。 + +```sql +SELECT device_id, count(temperature) +FROM TABLE( + read_tsfile( + TABLE_NAME => 'table1', + PATHS => '/data/backup' + ) +) +GROUP BY device_id; +``` + +返回结果 + +```text ++---------+-----+ +|device_id|_col1| ++---------+-----+ +| 100| 6| +| 101| 6| ++---------+-----+ +Total line number = 2 +It costs 0.049s +``` + +## 5. 时间序列趋势匹配表值函数 + +### 5.1 功能描述 + +`pattern_match` 表值函数支持基于输入的模式（pattern）序列，在指定的历史时间序列中，查找并返回所有符合阈值、结果边界等预设判定条件的子序列。 + +> V2.0.12 版本起支持 + +### 5.2 函数定义 + +```sql +pattern_match( + data => {table_name | subquery} PARTITION BY partition_col ORDER BY time_col, + [time_col => 'time_column_name',] + data_col => 'data_column_name', + pattern => 'pattern_sequence', + smooth => numeric_value, + threshold => numeric_value, + [width => numeric_value,] + [height => numeric_value,] + [smooth_on_pattern => {true | false}] +) +``` + +### 5.3 参数说明 + +|参数|类型|必填|默认值|说明| +|---|---|---|---|---| +|`data`|表名/子查询
|是|-|时间序列所在的表名或子查询结果，**必须保证时间列有序**；支持 `PARTITION BY` 按字段分区、`ORDER BY` 按时间排序| +|`time_col`|字符串|否|`'time'`|时间列名称，**仅支持时间戳类型**| +|`data_col`|字符串|是|-|序列名称，支持的数据类型：int、long、double、float| +|`pattern`|字符串|是|-|匹配模式序列，格式为数值列表（支持普通趋势序列 + 正则规则）：
1. 基础格式：以逗号分隔的整型 / 浮点型数值（如 '1.0,2.0,3.0'）；
2. 正则扩展：支持 `{数值段}+`（至少 1 次重复）、`{数值段}*`（至少 0 次重复），其中 `{`，` }* `，`}+` 为独立单位；
3. 特殊规则：每个段的最左侧点与上一个段共用| +|`smooth`|非负实数|是||数据波动容忍幅度（平滑程度）：
1. 数值越大，容忍的波动越大，但易过度平滑正常数据；
2. 推荐值：值域的 1%，可根据序列特性调整| +|`threshold`|非负实数|是||相似度阈值：仅返回相似度小于该值的结果，数值越小匹配越精确；建议从 1 开始向上调整| +|`width`|非负实数|否|无限大|匹配结果的宽度限制：仅返回长度（时间戳差值）小于该值的子序列| +|`height`|非负实数|否|无限大|匹配结果的高度限制：仅返回高度小于该值的子序列| +|`smooth_on_pattern`|布尔值
|否|`false`|是否对 pattern 做平滑处理：
1. false：pattern 为形状趋势，仅对原始序列做平滑；
2. true：对 pattern 和原始序列做同等平滑降噪| + +**返回结果** + +输出结果中仅保留与 pattern 匹配的数据段对应的行，且输出行按 `match_index` 升序排列。未匹配的行全部丢弃。 + +|列名|类型|说明| +|---|---|---| +|`match_index`|int32|非负整数，表示匹配到的第 N 个符合 pattern 的数据段| +|`similarity`|double|非负实数，表示该数据段与 pattern 的匹配相似度（值越小，匹配越精确）| +|data 参数所有列|与原列一致|包含输入 `data` 参数中所有列的原始数据| + +### 5.4 使用示例 + +以下示例演示普通模板匹配、参数调整、正则模板匹配以及平滑处理。 + +#### 5.4.1 准备测试数据 + +创建表并写入数据： + +```sql +create table test(value DOUBLE FIELD, value1 DOUBLE FIELD); +INSERT INTO test VALUES (1, 1, 0),(2, 2, 0),(3, 1, 0),(4, 1, 0),(5, 1, 0),(6, 1.2, 0),(7, 1, 0),(8, 1, 0),(9, 2, 0),(10, 3, 0),(11, 4, 0),(12, 3, 0),(13, 2, 0),(14, 3, 0),(15, 4, 0),(16, 2, 0),(17, 1, 0),(18, 1, 0),(19, 1, 0),(20, 1, 0),(21, 1, 0),(41, 1, 0),(42, 2, 0),(43, 3, 0),(44, 4, 0),(45, 3, 0),(46, 2, 0),(47, 2, 0),(48, 2, 0),(49, 2, 0),(50, 2, 0),(51, 2, 0),(52, 2, 0); +``` + +可通过如下语句查询数据： + +```sql +select * from test; +``` + +图示效果如下： + +![](/img/featured-function-01.png) + +#### 5.4.2 基础趋势匹配 + +使用普通序列作为模板，查找数据中趋势相似的子序列。 + +```sql +select * from pattern_match( + data => test ORDER BY time, + time_col => 'time', + data_col => 'value', + pattern => '1.0,2.0,3.0,2.0,1.0,1.0,1.0,1.0,1.0', + smooth => 0.5, + threshold => 10.0, + width => 1000.0, + height => 500.0, + smooth_on_pattern=> false +); +``` + +图示效果 + +![](/img/featured-function-02.png) + +查询返回 3 组趋势匹配结果。相似度越小，表示匹配程度越高。 + +```text ++-----------+------------------+-----------------------------+-----+------+ +|match_index| similarity| time|value|value1| ++-----------+------------------+-----------------------------+-----+------+ +| 0| 4.637180787192477|1970-01-01T08:00:00.001+08:00| 1.0| 0.0| +| 0| 4.637180787192477|1970-01-01T08:00:00.002+08:00| 2.0| 0.0| +| 0| 4.637180787192477|1970-01-01T08:00:00.003+08:00| 1.0| 0.0| +| 0| 4.637180787192477|1970-01-01T08:00:00.004+08:00| 1.0| 0.0| +| 0| 4.637180787192477|1970-01-01T08:00:00.005+08:00| 1.0| 0.0| +| 0| 4.637180787192477|1970-01-01T08:00:00.006+08:00| 1.2| 0.0| +| 0| 4.637180787192477|1970-01-01T08:00:00.007+08:00| 1.0| 0.0| +| 0| 4.637180787192477|1970-01-01T08:00:00.008+08:00| 1.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.013+08:00| 2.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.014+08:00| 3.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.015+08:00| 4.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.016+08:00| 2.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.017+08:00| 1.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.018+08:00| 1.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.019+08:00| 1.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.020+08:00| 1.0| 0.0| +| 1|0.6465473220616865|1970-01-01T08:00:00.021+08:00| 1.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.041+08:00| 1.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.042+08:00| 2.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.043+08:00| 3.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.044+08:00| 4.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.045+08:00| 3.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.046+08:00| 2.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.047+08:00| 2.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.048+08:00| 2.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.049+08:00| 2.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.050+08:00| 2.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.051+08:00| 2.0| 0.0| +| 2| 0.679768727042932|1970-01-01T08:00:00.052+08:00| 2.0| 0.0| ++-----------+------------------+-----------------------------+-----+------+ +Total line number = 29 +It costs 0.126s +``` + +#### 5.4.3 调整匹配条件 + +以下示例均以 7.4.2 的基础查询为准，仅修改对应参数。 + +|调整目的|参数|基础值|调整值|调整效果| +|---|---|---|---|---| +|提高匹配精度|`threshold`|`10.0`|`1.0`|仅保留相似度小于 `1.0` 的结果，匹配结果由 3 组减少为 2 组| +|限制匹配时间跨度|`width`|`1000.0`|`10.0`|仅返回时间跨度小于 `10.0` 的子序列| +|限制数值波动范围|`height`|`500.0`|`2.0`|仅返回数值波动范围小于 `2.0` 的子序列| + +#### 5.4.4 正则模板匹配 + +正则模板可以用于匹配重复出现的趋势片段。 + +|重复符号|含义| +|---|---| +|`+`|前面的子序列至少出现 1 次| +|`*`|前面的子序列可以出现 0 次或多次| + +以下示例使用 `+` 匹配至少出现一次的子序列 `2.0,3.0,2.0`： + +```sql +select * from pattern_match( + data => test ORDER BY time, + time_col => 'time', + data_col => 'value', + pattern => '1.0,{2.0,3.0,2.0}+,1.0,1.0,1.0,1.0,1.0', + smooth => 0.5, + threshold => 100.0, + width => 1000.0, + height => 500.0, + smooth_on_pattern => false +); +``` + +如果允许子序列 `2.0,3.0,2.0` 出现零次或多次，可以将 `+` 修改为 `*`： + +```sql +pattern => '1.0,{2.0,3.0,2.0}*,1.0,1.0,1.0,1.0,1.0' +``` + +正则模板适用于周期性波动、重复峰值以及长度不固定的趋势匹配场景。 + +#### 5.4.5 模板平滑处理 + +当模板序列包含小幅波动时，可以通过 `smooth_on_pattern` 控制是否对模板序列进行平滑处理。 + +以下示例展示两种不同参数配置下的匹配结果： + +|配置|`threshold`|`smooth_on_pattern`|匹配效果| +|---|---|---|---| +|同时平滑原始序列和模板序列|`1.0`|`true`|可以匹配原始数据中的 `1.0,1.2,1.0` 波动片段| +|仅平滑原始序列|`10.0`|`false`|仍可返回其他趋势匹配结果，但不包含对应的 `1.0,1.2,1.0` 波动片段| + +同时平滑原始序列和模板序列： + +```sql +SELECT * +FROM pattern_match( + data => test ORDER BY time, + time_col => 'time', + data_col => 'value', + pattern => '1.0,1.2,1.0', + smooth => 0.5, + threshold => 1.0, + width => 1000.0, + height => 500.0, + smooth_on_pattern => true +); +``` + +在该配置下，原始序列和模板序列采用相同的平滑规则，可以匹配包含时间点 5、6、7 对应的 `1.0,1.2,1.0` 波动片段。 + +图示效果如下： + +![](/img/featured-function-03.png) + +仅平滑原始序列： + +```sql +SELECT * +FROM pattern_match( + data => test ORDER BY time, + time_col => 'time', + data_col => 'value', + pattern => '1.0,1.2,1.0', + smooth => 0.5, + threshold => 10.0, + width => 1000.0, + height => 500.0, + smooth_on_pattern => false +); +``` + +在该配置下，仅对原始序列进行平滑处理。查询仍可返回其他趋势匹配结果，但时间点 5、6、7 对应的 `1.0,1.2,1.0` 波动片段未被匹配。 + +图示效果如下： + +![](/img/featured-function-04.png) + +## 6. 频域分析函数 + +### 6.1 FFT + +#### 6.1.1 功能概述 + +表模型内置表值函数 `FFT` 用于对输入表各分区内的一个或多个实值数值序列执行一维快速傅里叶变换。输入数据必须通过 `ORDER BY` 指定时间列并按升序排列，也可通过 `PARTITION BY` 按设备或 Tag 进行分组。每个分区独立生成一组频率 bin，同一分区内的多个数值列共享该频率轴。 + +> V2.0.12 版本起支持 + +#### 6.1.2 函数定义 + +- 命名参数语法如下： + +```sql +SELECT * +FROM FFT( + DATA => {table_name | subquery} + [PARTITION BY partition_column [, ...]] + ORDER BY time_column, + [SAMPLE_INTERVAL => duration_literal,] + [N => positive_integer,] + [NORM => {'backward' | 'forward' | 'ortho'},] + [TIMECOL => 'time_column'] +); +``` + +- 位置参数语法如下： + +```sql +SELECT * +FROM TABLE( + FFT( + TABLE(table_name) + [PARTITION BY partition_column [, ...]] + ORDER BY time_column + [, duration_literal + [, positive_integer + [, norm + [, time_column_name]]]] + ) +); +``` + +参数的规范顺序为 `DATA, SAMPLE_INTERVAL, N, NORM, TIMECOL`。推荐使用**命名参数**，以提高复杂调用的可读性。 + +#### 6.1.3 参数说明 + +|参数|是否必填|类型|默认值|说明| +|---|---|---|---|---| +|`DATA`|是|表参数|无|指定 FFT 的输入数据。可以使用 `PARTITION BY` 分区，每个分区独立计算；省略时，全部数据作为一个分区。除时间列和分区列外，所有 `INT32`、`INT64`、`FLOAT` 和 `DOUBLE` 列均参与计算，其他类型的列自动忽略。输入中至少需要一个数值列，且参与计算的列不能包含 `NULL`。如需只计算部分数值列，可在输入子查询中提前选择。| +|`SAMPLE_INTERVAL`|否|正数时间间隔字面量|根据时间列推断|指定采样间隔，例如 `1ms`、`10ms` 或 `1s`，用于计算频率轴。未指定时，每个分区按照 `(最后一个时间戳 - 第一个时间戳) / (输入行数 - 1)` 推断。单行分区无法自动推断，必须显式指定；空分区直接返回 0 行。| +|`N`|否|正整数|当前分区的输入行数|指定 FFT 变换长度。小于输入行数时，使用按时间排序后的前 `N` 行；大于输入行数时，在序列尾部补 0。每个分区输出 `N` 行结果。`N` 最大为 `65536`，且需满足 `N × 2 × 参与计算的数值列数 ≤ 16777216`。| +|`NORM`|否|字符串|`backward`|指定 FFT 结果的归一化方式，大小写不敏感。`backward`：不进行归一化；`forward`：将结果除以变换长度 `N`；`ortho`：将结果除以 `√N`。| +|`TIMECOL`|否|字符串|`time`|指定时间列名称，对应列必须为 `TIMESTAMP` 类型。`DATA` 必须仅按照该列升序排列，不支持 `DESC`；每个分区内的时间戳必须严格递增，不允许重复。| + +- FFT 通常适用于等间隔采样的数据。本函数不会自动检测或修正不等间隔的数据。为确保频谱结果准确，请在计算前对数据进行插值或重采样；如果采样间隔已知，建议通过 `SAMPLE_INTERVAL` 显式指定。 + +- 当 `N` 小于输入行数时，函数仍会检查分区内的全部输入数据，但只使用前 `N` 行进行 FFT 计算。 + +**返回结果** + +`FFT` 返回每个分区的完整双边频谱，输出列顺序如下： + +```text +分区列... +frequency_index +frequency +<数值列名>_real +<数值列名>_imag +... +``` + +|输出列|类型|说明| +|---|---|---| +|`PARTITION BY` 指定的列|与输入类型一致|保留原列名和类型，位于结果集最前面。未使用 `PARTITION BY` 时不输出。| +|`frequency_index`|`INT64`|频率点索引，范围为 `0` 至 `N-1`。| +|`frequency`|`DOUBLE`|当前频率点对应的频率，单位为 Hz。| +|`<数值列名>_real`|`DOUBLE`|对应输入数值列的 FFT 实部。| +|`<数值列名>_imag`|`DOUBLE`|对应输入数值列的 FFT 虚部。| + +对于变换长度 `N` 和采样间隔 `d`（单位为秒），`frequency` 的计算方式如下： + +|**frequency_index 范围**|**frequency**| +|---|---| +|`frequency_index < ceil(N / 2)`|`frequency_index / (N × d)`| +|其他情况|`(frequency_index - N) / (N × d)`| + +频率按照零频率、正频率、负频率的顺序排列，单位为 Hz。同一分区内，所有数值列共享相同的频率轴。 + +#### 6.1.4 注意事项 + +- `FFT` 函数名不区分大小写。每个分区独立执行一维 FFT。 + +- 当前版本仅支持实数输入，不支持复数输入、`DFT` 或 `IFFT`。 + +- 当前版本仅输出完整频谱的实部和虚部，不支持单边频谱、振幅或相位输出。 + +- 当前版本不支持 `VALUE`、`SAMPLE_RATE` 和 `SPECTRUM` 参数。 + +#### 6.1.5 使用示例 + +以下示例直接使用 `table1` 中的数据。由于 `temperature` 包含 `NULL`，需要在输入子查询中提前过滤。 + +- 示例 1：按设备执行 FFT + +```sql +SELECT * +FROM FFT( + DATA => ( + SELECT time, device_id, temperature + FROM table1 + WHERE plant_id = '1001' + AND temperature IS NOT NULL + ) PARTITION BY device_id ORDER BY time +) +ORDER BY device_id, frequency_index; +``` + +函数分别对每个设备的 `temperature` 执行 FFT。未指定 `N` 和 `SAMPLE_INTERVAL` 时，函数根据各分区的输入行数确定变换长度，并根据时间列推断采样间隔。本例中的采样间隔为 1 分钟。 + +```text ++---------+---------------+---------------------+----------------------+---------------------+ +|device_id|frequency_index| frequency| temperature_real| temperature_imag| ++---------+---------------+---------------------+----------------------+---------------------+ +| 100| 0| 0.0| 180.0| 0.0| +| 100| 1|-0.008333333333333333| 0.0| 0.0| +| 101| 0| 0.0| 255.0| -6.67572021484375E-6| +| 101| 1| 0.005555555555555556|-4.6126147026370745E-6|-5.266308107820805E-6| +| 101| 2|-0.005555555555555556| -1.9073486328125E-6| 3.303624680484063E-6| ++---------+---------------+---------------------+----------------------+---------------------+ +``` + +- 示例 2：指定变换长度和归一化方式 + +```sql +SELECT * +FROM FFT( + DATA => ( + SELECT time, device_id, temperature + FROM table1 + WHERE plant_id = '1001' + AND temperature IS NOT NULL + ) PARTITION BY device_id ORDER BY time, + SAMPLE_INTERVAL => 1m, + N => 4, + NORM => 'ortho' +) +ORDER BY device_id, frequency_index; +``` + +函数将每个设备的变换长度设置为 4。输入数据不足 4 行时，在数值序列尾部补 0；FFT 结果按照 `1/√N` 进行归一化。每个设备均返回 4 行频谱结果。 + +```text ++---------+---------------+---------------------+----------------------+----------------+ +|device_id|frequency_index| frequency| temperature_real|temperature_imag| ++---------+---------------+---------------------+----------------------+----------------+ +| 100| 0| 0.0| 90.0| 0.0| +| 100| 1| 0.004166666666666667| 45.0| -45.0| +| 100| 2|-0.008333333333333333| 0.0| 0.0| +| 100| 3|-0.004166666666666667| 45.0| 45.0| +| 101| 0| 0.0| 127.5| 0.0| +| 101| 1| 0.004166666666666667| 2.602374475716698E-15| -42.5| +| 101| 2|-0.008333333333333333| 42.5| 0.0| +| 101| 3|-0.004166666666666667|-2.602374475716698E-15| 42.5| ++---------+---------------+---------------------+----------------------+----------------+ +``` + +- 示例 3：数值列包含 `NULL` + +```sql +SELECT * +FROM FFT( + DATA => ( + SELECT time, device_id, temperature + FROM table1 + WHERE plant_id = '1001' + ) PARTITION BY device_id ORDER BY time +); +``` + +由于参与 FFT 计算的数值列不能包含 `NULL`，查询执行失败： + +```text +Msg: org.apache.iotdb.jdbc.IoTDBSQLException: 701: FFT does not support null values in column [temperature]. +``` + +可以在输入子查询中使用 `IS NOT NULL` 过滤空值后再执行 FFT。 + +### 6.2 lowPass + +#### 6.2.1 功能描述 + +`lowPass` 是低通滤波函数，对输入表中的数值列执行频域滤波，保留低频成分并衰减高频成分，适用于信号平滑和高频噪声去除。参数 `wpass` 为归一化通带频率，取值范围为 `(0, 1)`，用于指定允许通过的频率上限。 + +> V2.0.12 版本起支持 + +#### 6.2.2 函数定义 + +- 命名参数语法如下： + +```sql +SELECT * +FROM lowPass( + DATA => input_table PARTITION BY partition_column ORDER BY time_column, + TIMECOL => 'time_column', + WPASS => passband_frequency +); +``` + +- 位置参数语法如下： + +```sql +SELECT * +FROM lowPass( + input_table PARTITION BY partition_column ORDER BY time_column, + 'time_column', + passband_frequency +); +``` + +#### 6.2.3 参数说明 + +|参数|类型|必填|说明| +|---|---|---|---| +|`DATA`|表参数（`SET SEMANTIC`）|是|输入表或子查询。`ORDER BY` 必须仅指定时间列并按**升序**排列；`PARTITION BY` 可选，不指定时整张表作为一个分区。| +|`TIMECOL`|字符串|是|时间列名称，必须为时间戳类型，且只能指定一个时间列。| +|`WPASS`|`DOUBLE`|是|归一化通带频率，取值范围为 `(0, 1)`；低于该频率的成分保留，高于该频率的成分衰减。| + +**计算规则** + +- `TIMECOL` 和 `PARTITION BY` 指定的列不参与计算，但会保留在输出中。 + +- 除时间列和分区列外，仅 `INT32`、`INT64`、`FLOAT`、`DOUBLE` 列参与计算；不存在可计算列或存在其他类型列时，分析阶段报错。 + +- 参与计算列中的 `NULL` 值不参与该列的滤波计算。 + +**返回结果** + +- 输出分区列（如指定）和时间列，列名及类型保持不变。 + +- 每个参与计算的数值列输出一个滤波结果列，类型统一为 `DOUBLE`。每个结果列的测点数与对应输入列一致，结果按时间戳升序输出。 + +#### 6.2.4 使用示例 + +以下示例直接使用 `table1` 中的数据。 + +```sql +SELECT * +FROM lowPass( + DATA => ( + SELECT time, device_id, temperature + FROM table1 + WHERE temperature is not null + ) PARTITION BY device_id ORDER BY time, + TIMECOL => 'time', + WPASS => 0.3 +); +``` + +该示例按设备分区，对 temperature 列执行低通滤波，输出 `device_id`、`time` 以及滤波后的数值列。 + +```text ++---------+-----------------------------+--------------------+ +|device_id| time|lowpass(temperature)| ++---------+-----------------------------+--------------------+ +| 100|2024-11-26T13:37:00.000+08:00| 88.0| +| 100|2024-11-26T13:38:00.000+08:00| 88.0| +| 100|2024-11-28T08:00:00.000+08:00| 87.99999999999999| +| 100|2024-11-28T10:00:00.000+08:00| 87.99999999999999| +| 100|2024-11-28T11:00:00.000+08:00| 88.00000000000001| +| 100|2024-11-29T18:30:00.000+08:00| 88.0| +| 101|2024-11-27T16:39:00.000+08:00| 86.66666666666667| +| 101|2024-11-27T16:40:00.000+08:00| 86.66666666666667| +| 101|2024-11-27T16:41:00.000+08:00| 86.66666666666664| +| 101|2024-11-29T10:00:00.000+08:00| 86.66666666666664| +| 101|2024-11-30T09:30:00.000+08:00| 86.66666666666667| +| 101|2024-11-30T14:30:00.000+08:00| 86.66666666666664| ++---------+-----------------------------+--------------------+ +``` + +### 6.3 highPass + +#### 6.3.1 功能描述 + +`highPass` 是高通滤波函数，对输入表中的数值列执行频域滤波，保留高频成分并衰减低频成分，适用于提取快速变化、跳变和高频噪声等信号成分。参数 `wpass` 为归一化通带频率，取值范围为 `(0, 1)`，用于指定允许通过的频率下限。 + +> V2.0.12 版本起支持 + +#### 6.3.2 函数定义 + +- 命名参数语法如下： + +```sql +SELECT * +FROM highPass( + DATA => input_table PARTITION BY partition_column ORDER BY time_column, + TIMECOL => 'time_column', + WPASS => passband_frequency +); +``` + +- 位置参数语法如下： + +```sql +SELECT * +FROM highPass( + input_table PARTITION BY partition_column ORDER BY time_column, + 'time_column', + passband_frequency +); +``` + +#### 6.3.3 参数说明 + +|参数|类型|必填|说明| +|---|---|---|---| +|`DATA`|表参数（`SET SEMANTIC`）|是|输入表或子查询。`ORDER BY` 必须仅指定时间列并按升序排列；`PARTITION BY` 可选，不指定时整张表作为一个分区。| +|`TIMECOL`|字符串|是|时间列名称，必须为时间戳类型，且只能指定一个时间列。| +|`WPASS`|`DOUBLE`|是|归一化通带频率，取值范围为 `(0, 1)`；高于该频率的成分保留，低于该频率的成分衰减。| + +**输入列和计算规则** + +- `TIMECOL` 和 `PARTITION BY` 指定的列不参与计算，但会保留在输出中。 + +- 除时间列和分区列外，仅 `INT32`、`INT64`、`FLOAT`、`DOUBLE` 列参与计算；不存在可计算列或存在其他类型列时，分析阶段报错。 + +- 参与计算列中的 `NULL` 值不参与该列的滤波计算。 + +**返回结果** + +- 输出分区列（如指定）和时间列，列名及类型保持不变。 + +- 每个参与计算的数值列输出一个滤波结果列，类型统一为 `DOUBLE`。每个结果列的测点数与对应输入列一致，结果按时间戳升序输出。 + +#### 6.3.4 使用示例 + +```sql +SELECT * +FROM highPass( + DATA => ( + SELECT time, device_id, temperature + FROM table1 + WHERE temperature is not null + ) PARTITION BY device_id ORDER BY time, + TIMECOL => 'time', + WPASS => 0.3 +); +``` + +该示例按设备分区，对 temperature 列执行高通滤波，输出 `device_id`、`time` 以及滤波后的数值列。 + +```text ++---------+-----------------------------+----------------------+ +|device_id| time| highpass(temperature)| ++---------+-----------------------------+----------------------+ +| 100|2024-11-26T13:37:00.000+08:00| 2.000000000000029| +| 100|2024-11-26T13:38:00.000+08:00| 2.000000000000004| +| 100|2024-11-28T08:00:00.000+08:00| -2.9999999999999702| +| 100|2024-11-28T10:00:00.000+08:00| -3.000000000000013| +| 100|2024-11-28T11:00:00.000+08:00|-4.182831670630972E-14| +| 100|2024-11-29T18:30:00.000+08:00| 1.9999999999999953| +| 101|2024-11-27T16:39:00.000+08:00| -1.666666666666642| +| 101|2024-11-27T16:40:00.000+08:00| -1.6666666666666519| +| 101|2024-11-27T16:41:00.000+08:00| -1.666666666666645| +| 101|2024-11-29T10:00:00.000+08:00| -1.666666666666701| +| 101|2024-11-30T09:30:00.000+08:00| 3.3333333333333015| +| 101|2024-11-30T14:30:00.000+08:00| 3.333333333333338| ++---------+-----------------------------+----------------------+ +``` + +## 7. 时序相关性分析函数 + +### 7.1 Xcorr + +#### 7.1.1 功能描述 + +`Xcorr` 是相关性分析函数，对输入的两列数值序列进行互相关分析，度量两条序列在不同时间滞后（`lag`）下的形状相似度。函数将一条序列逐步平移，与另一条序列逐点相乘并求和，输出相似度曲线，可用于发现两路信号之间的延迟关系和相似模式。 + +> V2.0.12 版本起支持 + +#### 7.1.2 函数定义 + +- 命名参数语法如下： + +```sql +SELECT * +FROM Xcorr( + DATA => ( + SELECT time_column, signal_1, signal_2, partition_column + FROM input_table + ) PARTITION BY partition_column ORDER BY time_column, + TIMECOL => 'time_column' +); +``` + +- 位置参数语法如下： + +```sql +SELECT * +FROM Xcorr( + ( SELECT time_column, signal_1, signal_2, partition_column + FROM input_table + ) PARTITION BY partition_column ORDER BY time_column, + 'time_column' +); +``` + +#### 7.1.3 参数说明 + +|参数|类型|必填|说明| +|---|---|---|---| +|`DATA`|表参数（`SET SEMANTIC`）|是|输入表或子查询。`ORDER BY` 必须指定时间列并按升序排列；`PARTITION BY` 可选。去除时间列和分区列后，必须恰好剩余两列数值列。| +|`TIMECOL`|字符串|是|时间列名称，必须为时间戳类型，且只能指定一个时间列。| + +**计算规则** + +- 时间列和分区列不参与互相关计算。 + +- 去除上述列后，必须恰好存在两列，且两列类型均为 `INT32`、`INT64`、`FLOAT` 或 `DOUBLE`；否则分析阶段报错。 + +- 参与计算的两列中出现 `NULL` 时，该位置按 `0` 参与计算。 + +- 每个分区内时间列必须严格升序。 + +- 计算时按函数定义进行归一化处理。 + +**返回结果** + +- 输出 `PARTITION BY` 指定的分区列（如指定），列名及类型保持不变。 + +- 对参与计算的两列，生成一个结果列，类型为double。 + +#### 7.1.4 使用示例 + +以下示例使用 SQL 手册示例数据中的 `table1`，并通过子查询投影两列数值序列 `temperature` 和 `humidity` 参与互相关计算。 + +```sql +SELECT * +FROM Xcorr( + DATA => ( + SELECT time, device_id, temperature, humidity + FROM table1 + WHERE temperature IS NOT NULL + AND humidity IS NOT NULL + ) + PARTITION BY device_id + ORDER BY time, + TIMECOL => 'time' +); +``` + +该示例按设备分区，对 temperature 和 humidity 两列进行互相关分析，输出 `device_id` 以及相关性分析结果列。 + +```text ++---------+----------------------------+ +|device_id|xcorr(temperature, humidity)| ++---------+----------------------------+ +| 100| 3186.0001373291016| +| 100| 3622.5| +| 100| 3412.0000203450522| +| 100| 3318.9249839782715| +| 100| 3292.9599586486815| +| 100| 3324.7749156951904| +| 100| 3080.0999348958335| +| 100| 3123.8998641967773| +| 100| 3158.9998626708984| +| 101| 2957.9999351501465| +| 101| 3061.9999980926514| +| 101| 3100.166645050049| +| 101| 3172.5| +| 101| 3176.999931335449| ++---------+----------------------------+ +```