Managing Source Datasets | Lucanet Knowledge Base

In Data Hub, a source dataset specifies which data to import from a source system. Each source dataset belongs to a connection, and a connection can have one or more source datasets. On the Integrations page, each source dataset is displayed as a child row of the connection it uses.

To create a source dataset in Data Hub:

On the Integrations page, click the icon in the row of the connection you want to use to import data and select Create new dataset.

Specify the options of the source dataset.

Click Create. The source dataset is created and displayed under the selected connection on the Integrations page.

The following options are available for a source dataset in Data Hub:

Option	Description
Name	Name that identifies the source dataset on the Integrations page. Required.
Connection	Connection to the system the data is imported from. Required. Note: The connection you create the source dataset from is preselected. If necessary, you can select a different connection from the drop-down list. The Configuration area then displays the options of the selected source system. After the source dataset is created, the connection can no longer be changed.
Namespace	Technical identifier for the source dataset, used to reference its data tables in SQL queries against the Data Lake (for example, namespace.table_name). The namespace can have up to 255 characters and must be SQL-safe, with no spaces or unsupported special characters. Required.
Variables	Optional placeholders in the configuration whose values are selected every time an import is started. For more information, see Variables below.
Configuration	Data to import from the source system. Required. For more information, see Source Dataset Configuration below.

In Data Hub, a variable is a placeholder in the configuration of a source dataset. Instead of a fixed value, you can enter the variable in the configuration and select a value for the variable every time you start an import. This way, a single source dataset can import the data of a specific period or a specific company, without a separate source dataset for every combination.

The Variables area is displayed as follows, for example:

Shows the 'Variables' area of a source dataset in Data Hub with the columns 'Variable name', 'Display name', 'Data type', and 'Selection type', and the variables 'PERIOD' (display name 'Period') and 'YEAR' (display name 'Fiscal year'), both with the data type 'String' and the selection type 'Single value'. The 'PERIOD' variable shows '3 values', and the 'Add variable' option is displayed below the table. Highlighted in red is the 'Configure values' option of the 'YEAR' variable. — Variables area of a source dataset

You can define the variables either while you create the source dataset or later, using the Edit option. Click Add variable and specify the following for each variable:

Option	Description
Variable name	Name of the variable, used to reference the variable in the configuration of the source dataset
Display name	Name that is displayed for the variable when an import is started, for example `Period`
Data type	Data type of the values. Currently, only String is available.
Selection type	Number of values that can be selected for one import: • Single value: Exactly one value can be selected. • Set of values: Several values can be selected. The import extracts the data for each of the selected values separately.

To define the values that can be selected for a variable in Data Hub:

In the row of the variable, click Configure values.

Click Add value and specify the following:

Label

: Text that is displayed for the value when an import is started, for example

January

Value

: Value that replaces the variable in the configuration when the import is executed, for example

01

Click Save.

The allowed values of a variable are displayed as follows, for example:

Shows the 'Configure allowed values for Period' dialog in Data Hub with the labels 'January', 'February', and 'March' and the values '01', '02', and '03', and the 'Add value', 'Cancel', and 'Save' options. — Allowed values of the variable 'Period'

To remove a variable from the source dataset, click the icon in the row of the variable.

In the options of the Configuration area, you can enter a variable in a text field or in a table cell. Write the variable name in double curly brackets, for example {{PERIOD}}.

Every variable of a source dataset requires a value before an import can be executed. For more information, see Importing Data.

In Data Hub, the configuration of a source dataset defines which data is imported. The connection template of the source system determines which tabs the Configuration area displays and which options each tab contains. The configuration of an SAP S/4HANA source dataset therefore differs from the configuration of a Microsoft Dynamics 365 source dataset, and the options can change with the version of the connection template.

The Configuration area of an SAP S/4HANA source dataset is displayed as follows, for example:

Shows the 'Configuration' area of an SAP S/4HANA source dataset in Data Hub with the tabs 'General configuration', 'Custom RFC', and 'EXECUTE_SELECT query', and the 'Batch size', 'Page size', and 'Result table name' fields of the selected 'General configuration' tab. — Configuration of an SAP S/4HANA source dataset

The following tabs are available in the Configuration area of an SAP S/4HANA source dataset:

Tab	Description
General configuration	Technical settings of the import
Custom RFC	Settings for importing the result of a function module that is implemented in the SAP system
EXECUTE_SELECT query	Settings for importing the result of an SQL query against the tables of the SAP database

The options of the configuration can contain variables. For more information, see Variables.

You can manage an existing source dataset from the icon in the row of the source dataset on the Integrations page in Data Hub. The following options are available:

Option	Description
Start import	Option to start an import for the source dataset. For more information, see Importing Data.
View execution details	Option to display the execution history of the source dataset. For more information, see Importing Data.
Edit	Option to display and change the settings of the source dataset
Deactivate / Activate	Option to set the source dataset to Inactive or back to Active. An inactive source dataset cannot be imported.
Delete	Option to delete the source dataset