Using the Snowflake Python connector in a YAML pipeline

Introduction

If you want to use the snowflake python connector in your pipeline you have to install the connector software on a agent. If you have a private agent it's only necessary once but on a public agent you need to install the software every time. If you don't do that, you will get an error in python :

ImportError: No module named snowflake.connector

##[error]The process '/usr/bin/python' failed with exit code 1

Installation

Installation of the Python connector in a YAML pipeline is fairly easy. There are a couple of steps :

• Upgrade pip to the latest version.
• Download the python connector.
• Installation of the python connector.

Here is the script:

- script : | 
    python --version
    python -m pip install --upgrade pip
    python -m pip install -r https://raw.githubusercontent.com/snowflakedb/snowflake-connector-python/v2.3.7/tested_requirements/requirements_36.reqs
    python -m pip install snowflake-connector-python==2.1.3
  displayName: 'Install Snowflake connector'

This will download the software on the machine and you can use the connector in your python script.

Final thoughts

It was a bit of a surprise how simple it was to include this in my YAML pipeline. Now I can use the Snowflake python connector in my python scripts. Maybe that is for a future blogpost!

Hennie

DAT208x : Introduction to Python for Data Science

Introduction

I'm participating in the Microsoft Professional Program AI by Micrsosoft. I've already done the programs Data science and Big data. These are also part of the Microsoft Professional Programs. I've experienced them as an easy to follow instruction based courses. There are more of these programs available that are interesting for data enthusiasts. Think about Internet of Things and Data analysis.

The great thing about these programs is that these programs consists of high quality instructor led courses, broken in easy to digest videos, exercises, labs and quizzes on the Edx site. So every minute spare time you have you can follow a couple of videos.

The program is broken in the following courses :

• Introduction to Artificial Intelligence (AI)
• Introduction to Python for Data Science
• Essential Mathematics for Artificial Intelligence
• Ethics and Law in Data and Analytics
• Data Science Essentials
• Build Machine Learning Models
• Build Deep Learning Models
• Build Reinforcement Learning Models
• Develop Applied AI Solutions
• Microsoft Professional Capstone : Artificial Intelligence

DAT208x : Introduction to Python for Data Science

This course is an introduction of Python in combination with data science. There are other Python courses available but they do not always focus on data science. This course it is. The course is a collaboration between Edx and DataCamp and I have to say that the interaction between the two sites works great.

The course is divided in the following sections:

• Python Basics
• List - A Data Structure
• Functions and Packages
• Numpy
• Plotting with Matplotlib
• Control Flow and Pandas
• Final Lab
• Final Exam

Final thoughts

The whole scripting is executing in a controlled environment of DataCamp. They did a great job building an integrated learning environment. Every section has one or more labs and they are graded in Edx.

The Final Lab is lot of work and covers more than the material in the sections and in the videos. This took me quite some time finding out how and what. Google is your friend here. The Final Exam contains 50 questions and must be finished within 4 hours. You have limited time per question.

Hennie