Data Factory error handling with finally
- Chen Hirsh
- Feb 6, 2024
- 1 min read
Today I stumbled upon a weird behavior in Azure Data Factory (ADF) error handling.
ADF lets us add error handling in the flow control, In this example, I'm trying to copy some data, and if that fails go to on failure branch (red line). If the activity succeeded, go to on success branch (green line)
These work great (If you can call a failure great...).
Let's take another step. What if I want to run another activity at the end, no matter if the copy succeeded or failed?
This is called Finally in programming languages. After try and catch, the code under finally will run anyway.
Look at the example below. I added another activity at the end and connected it to both previous activities.
But nothing happens. The last activity does not run.
Why doesn't the last activity run? Well, it's waiting for the completion of the 2 activities it's connected to, and since the on-success one didn't run, the last activity will not start either.
What can we do?
Option 1 - duplicate the activity
Create 2 versions of the same activity, and connect one to the on-success path and one to the on-failure path.
Option 2 - use "on skip"
Connect the last activity to both previous activities with the "on skip" path. Then, even when one path completes, the other is skipped and the activity is triggered.
Lost half a day's work during a system crash, and I thought everything was backed up automatically. Turned out the sync hadn’t run properly in over a week. I needed to recover some version history or at least grab a recent file, but my local backups were out of date. Started digging through online tools again, and switching back to dropbox made the most sense after everything. It handles version tracking better, doesn’t choke on big files, and so far it’s caught every change without a hitch. Lesson learned — never trust one system alone.