In the example picture two linked services created: REST and Azure Storage They act just like connection managers in SSIS.
The next abstraction is a Dataset. The Dataset is a named view of data that simply points or references the data that has to bed used in activities as inputs and outputs. SSIS has analogs in its dataflow - various sources and destinations.
And the final piece – Copy activity. It is a generic task that copies data among various data stores located on-premises and in the cloud.
To make a Copy activity working all three types of objects have to be created one by one, just as described in the following few steps:
Go to Connections page (1) and click on “+ New” to create two linked services:
Linked Service 1, refers to a Data Source:
Linked Service 2, refers to a Data Sink:
In this step two datasets to be created. One per corresponding linked service. Therefore, source and sink datasets. To create them, click on three dots that stands next to a Datasets in the left pane and then choose “New dataset” to add source and sink objects:
After setting up all dependent objects it is time to perform a test execution to validate that we are on the right track. Just like on a screenshot below, a button Debug to be pressed. This will trigger the execution of the pipeline:
When the execution of a Copy activity is finished, click on details button (3). This will open a modal window with a detailed information of a copy activity:
The window shows a flow of the data and it shows one issue: the activity read 1.8 MB from a REST, but written a few bytes only.
Storage Explorer confirms - the new empty file was created:
Such an output result can be explained by the fact that Mapping and Pagination are still not configured in a Copy activity. These two actions are going to be covered in the next blog post.
The second piece of the pipeline – a Copy activity was implemented. It establishes connections with a REST data source, and it tries to write data to Azure storage.
The REST data source outputs data in a JSON format, however, we specified to write data to the sink as a “Delimited Text”, therefore Mapping and Pagination also need to be implemented and it is covered in a next blog post - Azure Data Factory and REST APIs - Mapping and Pagination
Many thanks for reading.