Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,692 questions
0 answers
How to use next link for pagination in adf (I have to use pagination for web activity)
Hi, I am trying to build up an adf pipeline to pull data from NetSuite. The result of the api call of NetSuite is like this in postman As we can see that this returns next link with the data and now we have to us this next link for pagination in adf. …
asked 2024-05-01T17:29:46.6833333+00:00
Amar Agnihotri
891
Reputation points
commented 2024-05-20T05:18:18.4566667+00:00
AnnuKumari-MSFT
31,561
Reputation points Microsoft Employee
1 answer
check if the value is a number and output in iif logic of derived transformation
I am checking if the input value is a number then output as '100' else 'invalid' in derived transformation in dataflow. In the documentation, I found 'isLong' function. But, how do we use this function in an iif…
asked 2024-05-18T23:01:01.27+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 23%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPS%3C/text%3E%3C/svg%3E)
Peruka, Sikander reddy
25
Reputation points
commented 2024-05-20T05:10:06.4333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
10,405
Reputation points Microsoft Vendor
2 answers
One of the answers was accepted by the question author.
Azure Data Flow Sink - Connection string is not valid or missing Server/User/Database properties
Could you please assist here to resolve this issue below ? We are trying to read Delta table and store it as table in Azure SQL Database as using ADF Data Flow. Source Delta table via inline dataset from ADLS Gen2 its working fine and slink is getting…
asked 2024-05-13T06:36:57.6633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(3.2, 84%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKA%3C/text%3E%3C/svg%3E)
Kashif Ahmed
40
Reputation points
accepted 2024-05-20T05:09:42.0966667+00:00
Kashif Ahmed
40
Reputation points
1 answer
ADF Pipeline runs successfully but data doesn't show up
Hi team Using Azure Data Factory pipeline using two web activity Source web is reading data from log analytics workspace w1 table t1 and I am passing the result of source web activity as request body of destination web activity then my destination web…
asked 2024-05-16T17:50:19.24+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 39%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERS%3C/text%3E%3C/svg%3E)
Rishabh Srivastava
60
Reputation points Microsoft Employee
commented 2024-05-20T04:34:51.2666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
6,550
Reputation points Microsoft Vendor
1 answer
‘Failed to execute script. Exception: 'Odbc Operation Failed.’ Error in Script activity in ADF
Hi Team, We have few pipelines in ADF where we are fetching the Data from multiple API endpoints using Copy Activity. We are connecting to API by leveraging ‘Rest API’ linked service and Integration Runtime(IR) as ‘AutoResolveIntegrationRuntime’. We are…
asked 2024-05-16T10:51:36.78+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 31%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
Harshvardhan Medikonda
5
Reputation points
commented 2024-05-20T03:33:42.0733333+00:00
phemanth
6,550
Reputation points Microsoft Vendor
0 answers
Data Factory Copy task getting stuck on a specific record count when copying
Hi All, I am copying data from a SAP Hana table on Prem to parquet file in ADLS, the table is only 30 000 000 big but when the copy task get to 12 000 000 odd records it seems to hang or stops copying. I have checked the source data and it is fine no…
asked 2024-05-18T08:07:27.7766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 62%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Stephen Smith
0
Reputation points
edited the question 2024-05-20T03:15:11.5633333+00:00
vipullag-MSFT
24,711
Reputation points
1 answer
Azure Data Factory still generating deleted log even with "After completition" set to "No Action"
Hi, I’m facing the issue below. I have a Data Flow that inserts a filter into an Azure SQL DB. I’ve set it to “Delete source files” after completion, but it’s still generating a log in the Blob storage, which I don’t want. So, I changed it to “No action…
asked 2024-05-19T13:19:22.4433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 94%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAE%3C/text%3E%3C/svg%3E)
Azevedo Emanoel
1
Reputation point
answered 2024-05-19T23:14:04.8066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(176, 14.000000000000002%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVC%3C/text%3E%3C/svg%3E)
Vlad Costa
0
Reputation points
0 answers
Azure ADF doesn't recognize changes in pipeline activity
Hi, I'm facing an issue with ADF when trying to change my pipelines. It doesn't accept my changes when it executes using the activity runtime, it only applies the changes in debug mode. Before the changes, my pipeline looks like this with the third Data…
asked 2024-05-19T10:24:41.03+00:00
Azevedo Emanoel
1
Reputation point
asked 2024-05-19T10:24:41.03+00:00
Azevedo Emanoel
1
Reputation point
0 answers
Azure Data Factory is not firing the trigger when I create a blob as output from ETL tool
Hi, I'm using a ETL tool to save files into a blob container and it is not firing the trigger that I have created to the process. When I manually upload the file to the storage, it works fine. Can someone help?
asked 2024-04-19T14:57:13.3966667+00:00
Azevedo Emanoel
1
Reputation point
commented 2024-05-19T10:17:04.2333333+00:00
Azevedo Emanoel
1
Reputation point
0 answers
How to resolve invalid property name error in ADF copy activity?
ErrorCode=AdlsGen2OperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ADLS Gen2 operation failed for: Operation returned an invalid status code 'BadRequest'. Account: 'rmsdatalakestracc1mqa'. FileSystem:…
asked 2024-05-19T06:23:54.9633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(256, 35%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMY%3C/text%3E%3C/svg%3E)
Mansi Yadav
0
Reputation points
asked 2024-05-19T06:23:54.9633333+00:00
Mansi Yadav
0
Reputation points
1 answer
One of the answers was accepted by the question author.
Trouble connecting Azure SQL database to Azure Data Factory
I keep getting an error message saying "Cannot connect to SQL Database." I'm using Server: 'redacted', Database: 'redacted', User: 'redacted'. I've checked the linked service configuration, and made sure the SQL Database firewall allows access…
asked 2024-05-17T19:57:11.5033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 19%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
Jae
20
Reputation points
edited the question 2024-05-19T04:28:43.5033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
BhargavaGunnam-MSFT
27,156
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
about the relationship and data type in power bi
in power bi is this true, always ensure that both of the columns that you are using to participate in a relationship are sharing the same data type. Your model will never work if you try to build a relationship between two columns, where one column has a…
asked 2023-09-12T05:19:00.8533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(160, 67%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENS%3C/text%3E%3C/svg%3E)
Nanthakumar S
30
Reputation points
commented 2024-05-17T22:53:51.3933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(9.6, 37%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERP%3C/text%3E%3C/svg%3E)
Ram Prakash Patel
0
Reputation points
1 answer
list of pipeline names
How do I easily get a downloaded list of all pipeline names?
asked 2024-05-17T20:33:42.42+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 34%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EFD%3C/text%3E%3C/svg%3E)
Feuerborn, Diana
0
Reputation points
answered 2024-05-17T21:21:05.8566667+00:00
BhargavaGunnam-MSFT
27,156
Reputation points Microsoft Employee
1 answer
Why there is no upsert key for cosmos Db container
Hello, Using the copy activity- Azure SQL to cosmos DB there is no upsert key. Is this expected behavior or this is a product limitation? Appriciate your help!
asked 2024-05-17T18:43:35.1466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(19.2, 26%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
Michael M
65
Reputation points
answered 2024-05-17T19:44:46.78+00:00
BhargavaGunnam-MSFT
27,156
Reputation points Microsoft Employee
1 answer
Using cached data from one data flow activity as the source for another data flow activity
On this reference page... https://learn.microsoft.com/en-us/azure/data-factory/data-flow-sink ...MS says, "Write to activity output The cached sink can optionally write your output data to the input of the next pipeline activity. This will allow you…
asked 2024-05-15T14:57:15.3366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(60.8, 54%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDL%3C/text%3E%3C/svg%3E)
David Lang
20
Reputation points
commented 2024-05-17T19:31:12.6933333+00:00
David Lang
20
Reputation points
2 answers
How to copy the data from Amazon S3 to Azure SQL database incrementally.
Hello team, I wanted to wanted to copy data from Amazon S3(JSON format) to Azure SQL database. I wanted to learn the mechanism for incrementally updating the data from Amazon S3 to Azure SQL. Also, the environment has only SHIR deployed, so data flows…
asked 2024-05-01T20:06:01.4533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(131.20000000000002, 0%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESC%3C/text%3E%3C/svg%3E)
Shweta Chavan
40
Reputation points
commented 2024-05-17T18:52:24.04+00:00
Oury Ba-MSFT
16,731
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
ADF - CDC - enable and disable CDC tables during deployment
Hi, I have setup ADF pipeline for data sync of tables between Azure SQL Database to SQL Managed Instance databases. I am using Data Flow and CDC to track changes and sink activity to replicate it to target. However, when we deploy any table changes (add…
asked 2024-05-08T04:29:23.9233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(166.4, 83%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Ashwani SRIVASTAVA
60
Reputation points
edited the question 2024-05-17T16:53:36.08+00:00
BhargavaGunnam-MSFT
27,156
Reputation points Microsoft Employee
3 answers
data factory if condition false, exit pipeline execution
In if component under false condition , how to exit from pipeline execution? Through Getmetadata component, i am search the file and if it's true, it has to progress with rest of pipeline and under false i.e no file available it has exist/stop…
asked 2021-12-22T17:05:29.133+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 85%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
a1990
11
Reputation points
commented 2024-05-17T14:35:09.0033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(9.6, 37%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDH%3C/text%3E%3C/svg%3E)
David Hernandez
0
Reputation points
1 answer
Can't able to connect the VM installed postgresql from ADF
Hi Team, We are unable to access the PostgreSQL from the ADF linked server. it's a azure PostgreSQL flexible server, and we have enabled the "Allow public services" setting within Azure for this server. However, we are still unable to create a…
asked 2024-05-14T06:10:02.6633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(121.6, 79%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Aravind
20
Reputation points
commented 2024-05-17T13:24:29.33+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 4%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Sina Salam
3,901
Reputation points
1 answer
One of the answers was accepted by the question author.
Connect SharePoint Online List with Synapse Integrated Runtime Managed Virtual Network
Getting an error when attempting to create a Linked Service for SharePoint Online List in Synapse. I've followed the instructions and prerequisites on MS documentation, but getting the following error when testing the connection: An error occurred while…
asked 2024-05-16T14:01:45.88+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 56.00000000000001%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EC2%3C/text%3E%3C/svg%3E)
CarmeloLoPresti-2973
61
Reputation points
accepted 2024-05-17T12:50:39.8966667+00:00
CarmeloLoPresti-2973
61
Reputation points