Configuring ADLS Smart Connector for DQ Sync

This topic walks you through configuring the Azure Data Lake Storage (ADLS) smart connector and sync DQ Scores from DQLabs for the assets in your Metadata Manager.

This process involves the following:

• Configuring ADLS Connection in DQLabs
• Configuring the ADLS Smart Connector
• Enabling erwin DI - DQLabs Sync

Configuring ADLS Connection in DQLabs

To profile data and view DQ Scores for the metadata in the ADLS database, you need to create a new catalog manually and connect to the ADLS database.

Before you connect to the ADLS database, configure the ADLS smart connector in DQLabs. For more information, refer to the Azure Data Lake Connection for DQLabs configuration guide.

To connect to ADLS and profile data, follow these steps:

1. Log in to the DQLabs instance.
2. Click to create a new catalog.
3. From the Select a Datasource Type page, select AzureDataLake.
   The Configure Azure DataLake Connection page appears.
   
4. Enter the connection details for the ADLS database.
5. Click Validate.

This displays the list of datasets, based on the data in the ADLS along with the dataset hierarchy.


6. From the datasets list, select the required datasets and then, click Connect.
   After, profiling data, the DQ Scores will be available for the datasets.

Once the ADLS connection is configured, ensure that you integrate erwin DI and DQLabs to retrieve DQ Scores from DQLabs. For more information about integration, refer to the Configuring Data Profiling topic.

Configuring the ADLS Smart Connector

Once you have created the catalog and profiled data from DQLabs, you need to configure the ADLS smart connector in erwin DI.

Before running the smart connector, refer to the ADLS Smart Connector document and contact services team.

To configure the ADLS smart connector, follow these steps:

1. Go to Application Menu > Miscellaneous > Automation Framework.

The following page appears.


2. Under the Connector tab, click .

The Import Code Automation Connector page appears.



3. Click Browse, and select the ADLS smart connector (.acp file).
4. Click .
   The connector will be imported and added to the Connectors list.
5. Double-click the connector, and click to run the smart connector.
   

   Ensure that you configure the ADLS smart connector before you can run it. Contact the services team to configure the smart connector to import metadata.

   
   
   Once the connector executes successfully, the connector page displays a success message. This creates metadata based on the ADLS database in the Metadata Manager.

Enabling erwin DI - DQLabs Sync

After the ADLS connector configuration in DQLabs and Automation Framework module, you can select a data source (catalog in DQLabs created using an ADLS smart connector) for the system to sync DQ Scores from DQLabs.

To select a data source for DQ Score sync, follow these steps:

1. Go to Application Menu > Metadata Manager, and locate the new system created using the ADLS smart connector.
2. Right-click the system, and click Edit System.
   The Edit System page appears.
   
3. Click .
   The Select Data Sources page appears and displays the data sources (ADLS Catalogs) created manually in DQLabs. To create a catalog and ADLS connection in DQLabs, refer to the Configure ADLS Connection in DQLabs section.
   
   Use the search box to filter data sources using their name, or type.
4. Select the required data source from the list, and click .
   This page displays data sources from DQLabs only when the Enable DQ For Smart Connector option is enabled and Organization Name field is updated in the configuration page.
   

   Selecting a data source here ensures that, the DQ Scores from DQLabs are synced to the system.

If you are running erwin DI locally, to get DQ Scores for the ADLS environments, follow the below steps:

1. In your local machine, go to C:\Program Files\Apache Software Foundation\Tomcat 9.0_Tomcat9.0.73\webapps\erwinDISuite\WEB-INF\configuration\properties.
2. Locate and open the Application.properties file.
3. Enter external - path=External Path.

Once the above configurations are complete, perform a sync job to sync the DQ Scores from DQLabs to erwin DI, and the scores are displayed in the Metadata Manager. For more information on scheduling a sync job, refer to the Scheduling Jobs topic.