Azure Synapse Connector 1.0.0.0
The Azure Synapse Connector lets you find, import, and run Azure Synapse pipelines.
Prerequisites
- RunMyJobs 9.2.9 or later.
- Connection component 1.0.0.4 or later. Note that the Connections component will be installed or updated automatically if necessary when you install this extension.
- Azure Connections Extension 1.0.0.3 or later.
Contents of the Component
The Azure Synapse Connector consists of the following objects:
| Object Type | Name |
|---|---|
| Application | GLOBAL.Redwood.REDWOOD.AzureSynapse |
| Process Definition | REDWOOD.Redwood_Synapse_ShowPipelines |
| Process Definition | REDWOOD.Redwood_Synapse_RunPipeline |
| Process Definition | REDWOOD.Redwood_Synapse_RunPipeline_Template |
| Process Definition | REDWOOD.Redwood_Synapse_ImportPipeline |
| Process Definition Type | REDWOOD.Redwood_AzureSynapse |
| Library | REDWOOD.Redwood_AzureSynapse |
Redwood_Synapse_ShowPipelines
Lists all existing pipelines in Azure Synapse.
Parameters
| Tab | Name | Description | Documentation | Data Type | Direction |
|---|---|---|---|---|---|
| Parameters | connection
|
Connection |
|
String | In |
| Parameters | workspaceName
|
Workspace Name | List all Azure Synapse pipelines in the specified workspace. | String | In |
| Parameters | filter
|
Pipeline Name Filter | This filter can be used to limit the number of Azure Synapse pipelines returned to those with names that match the filter. Wildcards * and ? are supported. If omitted, all pipelines from the workspace will be returned. | String | In |
| Parameters | listing
|
Pipeline Listing | The listing of all available Azure Synapse pipelines that match the input filter (or all of them if no input filter was provided). | Table | Out |
Redwood_Synapse_RunPipeline
Runs a pipeline in Azure Synapse.
Parameters
| Tab | Name | Description | Documentation | Data Type | Direction |
|---|---|---|---|---|---|
| Parameters | connection
|
Connection |
|
String | In |
| Parameters | workspaceName
|
Workspace Name | The run pipeline in the specified workspace. | String | In |
| Parameters | pipelineName
|
Pipeline to run | The Azure Synapse pipeline to execute. | String | In |
| Parameters | parameters
|
Pipeline Parameters | Key-value pairs of parameters and values to run the pipeline with. | String | In |
| Parameters | runId
|
The Run ID of the Azure Synapse pipeline | The Azure Synapse Run ID. | String | Out |
Redwood_Synapse_RunPipeline_Template
Template definition to run an Pipeline in Azure Synapse.
Parameters
| Tab | Name | Description | Documentation | Data Type | Direction |
|---|---|---|---|---|---|
| Parameters | connection
|
Connection |
|
String | In |
| Parameters | workspaceName
|
Workspace Name | The workspace that contains the pipeline to run. | String | In |
| Parameters | pipelineName
|
Pipeline to run | The Azure Synapse pipeline to execute. | String | In |
| Parameters | runId
|
Run ID | The Run ID of the Azure Synapse pipeline. | String | Out |
Redwood_Synapse_ImportPipeline
Imports a pipeline from Azure Synapse.
Parameters
| Tab | Name | Description | Documentation | Data Type | Direction | Default Expression | Values |
|---|---|---|---|---|---|---|---|
| Parameters | connection
|
Connection |
|
String | In |
|
|
| Parameters | workspaceName
|
Workspace Name | Imports all Azure Synapse pipelines in the specified workspace. | String | In |
|
|
| Parameters | filter
|
Pipeline Name Filter | This filter can be used to limit the number of Azure Synapse pipelines returned to those with names that match the filter. Wildcards * and ? are supported. If omitted, all pipelines from the workspace will be returned. | String | In |
|
|
| Generation Settings | overwrite
|
Overwrite Existing Definition | When set to Y, if a definition with the same business key already exists, it will be overwritten by the import. When set to N, the import will be skipped if the definition already exists. | String | In | N | Y=Yes, N=No |
| Generation Settings | targetPartition
|
Partition | The Partition to create the new definitions in. | String | In |
|
QueryFilter:User.Redwood System.Partition.Partition%2e;all |
| Generation Settings | targetApplication
|
Application | The Application to create the new definitions in. | String | In |
|
QueryFilter:User.Redwood System.Application.Application%2e;all |
| Generation Settings | defaultQueue
|
Default Queue | The default Queue to assign to the generated definitions. | String | In |
|
QueryFilter:User.Redwood System.Queue.Queue%2e;all |
| Generation Settings | targetPrefix
|
Definition Name Prefix | The prefix to add onto the name of the imported Azure Synapse pipeline name to create the definition name. | String | In |
|
|
Setup
To install the Azure Synapse Connector and create a connection to Azure AD:
-
Locate the Azure Synapse component in the Catalog and install it.
Finding Azure Synapse Pipelines
To retrieve the list of Azure Synapse pipelines available for scheduling, submit the Redwood_Synapse_ShowPipelines Process Definition. Use your Azure AD Connection and specify the Workspace Name.
Once the process has finished, click listing.rtx in the process' Detail View. Sample output is shown below.
Running an Azure Synapse Pipeline
To run an Azure Synapse pipeline, submit the Redwood_Synapse_RunPipeline Process Definition. Specify the Connection and Workspace Name, enter the name of the Pipeline to run, and optionally specify Pipeline Parameters.
If you specify Pipeline Parameters, use a key=value format for each parameter, as demonstrated in the following screen shot.
For information on specifying Pipeline Parameters, see "Pipeline Parameters," below.
Running an Azure Synapse Pipeline with a Template
To create a customized Process Definition, optionally with default values:
-
Right-click the Redwood_Synapse_RunPipeline_Template Process Definition and choose New (from Template) from the context menu. The New Process Definition pop-up window displays.
-
In the Definition tab, delete the default Application value (if any) and substitute your own Application name if desired.
-
In the Parameters tab, enter any Default Expressions you want to use.
-
When specifying the Connection value, use the format
EXTCONNECTION:<partition>.<connection name>. -
If you want to add Pipeline Parameters, put them in a parameter group named
Pipeline Parameters, as shown in the following screen shot.
-
-
Save the new Process Definition.
Importing an Azure Synapse Pipeline
To import an Azure Synapse pipeline as a RunMyJobs Process Definition, submit Redwood_Synapse_ImportPipeline. Enter the name of the pipeline in the Pipeline Name Filter field.
When the process completes, you should have a new Process Definition that is named after the imported pipeline. It will be automatically populated with any parameter names and default values from Azure.
Pipeline Parameters
When specifying Pipeline Parameters, use the formats described in this section.
Redwood_Synapse_RunPipeline
To override the value of input parameters when submitting an Azure Synapse pipeline, add one or more entries to the Pipeline Parameters array parameter. Each entry should be provided in key=value format, where key is the name of the pipeline parameter, and value is the value to pass to that parameter. Observe the rules below for passing values for different types of parameters.
| Pipeline Parameter Type | Examples | Notes |
|---|---|---|
| String |
|
The value for string parameters should be enclosed in double quotes. |
| Secure String | N/A | Not supported on Redwood_Synapse_RunPipeline definition. Use a Process Definition created from Redwood_Synapse_RunPipeline_Template instead. |
| Number | ParameterName=12345
|
The value for number parameters should not be quoted. |
| Boolean | ParameterName=true
|
The value for boolean parameters should not be quoted. Allowed values are true and false. |
| Array | ParameterName=["string", 1, true]
|
Use JSON array syntax. Strings must be quoted. |
| Object |
|
Use JSON syntax. Strings must be quoted. |
Redwood_Synapse_RunPipeline_Template
To override the value of input parameters when submitting an Azure Synapse pipeline, add one or more Process Definition parameters to the Process Definition created from the template. Such parameters must be added to a parameter group named Pipeline Parameters. The name of each Process Definition parameter name should exactly match the name of the Pipeline Parameter it is supplying a value for. The value of the Process Definition parameter should be the value of the Pipeline Parameter. Observe the rules below for passing values for different types of parameters.
| Pipeline Parameter Type | Process Definition Parameter Properties | Example Value | Notes |
|---|---|---|---|
| String | Parameter Type: String |
|
The value for string parameters should be enclosed in double quotes. |
| Secure String |
Parameter Type: String Expand the parameter options at the bottom and check Password. |
SecureValue
|
To pass a value to a SecureString parameter, mark the Process Definition parameter as Password and supply the secure value in the parameter value. |
| Number | Parameter Type: Number | 12345
|
The Process Definition parameter type should be set to ‘Number’ to supply a value to a number parameter. |
| Boolean | Parameter Type: Boolean | true
|
Use a String type Process Definition parameter with the value true or false to supply the value to a Boolean Pipeline Parameter. |
| Array | Parameter Type: String | ["string", 1, true]
|
Use JSON array syntax in the value of a String type Process Definition parameter to supply the value for an Array type Pipeline Parameter. String values inside the array must be quoted. |
| Object | Parameter Type: String |
|
Use JSON syntax in the value of a String type Process Definition parameter to supply the value for an Object type Pipeline Parameter. String values must be quoted. |