Databricks job clusters are stricter than interactive clusters

While converting a list to a dataframe, I got a type error, but only when scheduling the notebook on a job cluster, not when running it directly on an interactive cluster.

How to get around that and why does it happen?

See the full post on my blog - https://chenhirsh.com/databricks-job-clusters-are-stricter-than-interactive-clusters/