Functions Importer RESTImporter

The RESTImporter allows to import data from JSON files fetched over REST Interfaces. It supports fetching from following requests, so that request one can be answered with paths in the JSON that will then be fetched with request two.

Configuration

The configuration manages data access and transfer from source JSON to SmartData JSON datasets.

Parameter	Description	Default	Optional
rest.baseurl	Base url of the REST API	-	no
rest.auth	Type of authorisation "none", "basic" or "bearer"	none	yes
rest.user	Username for basic authenitfication	-	yes
rest.pwd	Password for basic authenitfication	-	yes
rest.steps	Array of REST steps (see below)	-	no
rest.delay	Delay for executeing steps in ms	0	yes

REST steps configuration

The REST steps define the request queue executed for import.

Steps configuration:

Parameter	Description	Default	Optional
url	API URL (Can contain placeholders, see below)	-	no
extract	JSON Path to the value to extract	-	yes
as	Name of the variable where to store extract value	-	yes
download	true or false - if true this file will be parsed	-	yes
targetdir	Target where to store downloaded files	-	yes
param.*	Parameters to send with url call	-	yes

Placeholders are noted in brackets. Each extracted value can be accessed in following steps by its "as"-name. General available placeholders:

• {base} for the base directory path

Example configuration:

{
	"server":"http://localhost:8080", 
	"ignoreSSL": true,
	"usethreading": false,
	"smartdata":"SmartDataHELIOS", 
	"storage":"smartmonitoring", 
	"collection":"wematics_test",
	"save.blocksize": 5,
	"resultReciptants1": "florian.fehring@hsbi.de",
	
	"importer":"RESTImporter",
	"rest.baseurl": "https://api.wematics.cloud/api/cameras/2cab16245f07326b/rgb",
	"rest.auth": "bearer",
	"rest.token": "mytoken",
	"rest.steps": [
          {
            "url": "{base}/dates/",
            "extract": "available_dates[]",
            "as": "date"
          },
          {
            "url": "{base}/{date}",
            "extract": "files[].name",
            "as": "name"
          },
          {
            "url": "{base}/{date}/files/{name}",
            "download": true,
            "targetdir": "source/wematics/{date}",
            "param.date_from": "2025-07-07",
            "param.maxsize": 23
          }
        ],
	"rest.delay": 1000,
	
	
	"sourcedir": "/files",
	"parser":"FileAnalyserParser",
	"debugmode": true
}

Useing date placeholders

Any source URL can contain placeholders for a date.

Placeholder	Description
{yyyy}	The actual year
{MM}	The actual month
{dd}	The actual day (number)
{yyyy-X}	Actual year minus X years
{MM-X}	Actual month minus X month
{dd-X}	Actual day minus X days
{yyyy+X}	Actual year plus X years
{MM+X}	Actual month plus X month
{dd+X}	Actual day plus X days

Example with date placeholders

{
"rest.steps": [
          {
            "url": "{base}/{yyyy}-{MM}-{dd-1}/files/ghi_{yyyy}-{MM}-{dd-1}.csv",
            "download": true,
            "targetdir": "../docroot/files_helios/images/asi_wematics/paris/{yyyy}-{MM}-{dd-1}"
          }
        ], 
}