Schedule machine learning pipeline jobs

• The Azure CLI and ml extension installed by following the instructions in Install, set up, and use the CLI (v2).
• Knowledge of how to create Azure Machine Learning YAML pipelines. For information, see Create and run machine learning pipelines using components with the Azure Machine Learning CLI.

• The Azure Machine Learning SDK v2 for Python installed.
• Knowledge of how to create Python pipelines for Azure Machine Learning. For information, see Create and run machine learning pipelines using components with the Azure Machine Learning SDK.

• Knowledge of how to create pipelines in Azure Machine Learning studio. For information, see Create and run machine learning pipelines using components with the Azure Machine Learning studio.

The code examples in this article are from Working with Schedule in Azure Machine Learning CLI 2.0.

The code examples in this article are excerpts from the Working with Schedule Azure Machine Learning notebook. Run the notebook to create and manage the schedules as described.

To create a schedule for an Azure Machine Learning pipeline job in the studio UI, open the pipeline job detail page.

The following YAML code defines a recurring schedule for a pipeline job. The required type parameter specifies that the trigger type is recurrence.

$schema: https://azuremlschemas.azureedge.net/latest/schedule.schema.json
name: simple_recurrence_job_schedule
display_name: Simple recurrence job schedule
description: a simple hourly recurrence job schedule

trigger:
  type: recurrence
  frequency: day #can be minute, hour, day, week, month
  interval: 1 #every day
  schedule:
    hours: [4,5,10,11,12]
    minutes: [0,30]
  start_time: "2022-07-10T10:00:00" # optional - default will be schedule creation time
  time_zone: "Pacific Standard Time" # optional - default will be UTC

create_job: ./simple-pipeline-job.yml
# create_job: azureml:simple-pipeline-job

You must or can provide the following schedule parameters:

The following code uses RecurrenceTrigger to provide a better coding experience.

schedule_name = "simple_sdk_create_schedule_recurrence"

schedule_start_time = datetime.utcnow()
recurrence_trigger = RecurrenceTrigger(
    frequency="day",
    interval=1,
    schedule=RecurrencePattern(hours=10, minutes=[0, 1]),
    start_time=schedule_start_time,
    time_zone=TimeZone.UTC,
)

job_schedule = JobSchedule(
    name=schedule_name, trigger=recurrence_trigger, create_job=pipeline_job
)

You must or can provide the following schedule parameters:

To open the schedule creation wizard, select Schedule > Create new schedule at the top of the page.

Parameters

• frequency (required) is the time unit on which basis the schedule fires. Can be minutes, hours, days, weeks, or months.
• interval (required) is the number of time units between schedule recurrences.
• schedule (optional) defines the recurrence pattern, which can contain hours, minutes, and weekdays. If omitted, jobs trigger according to the logic of start_time, frequency, and interval.
  • When frequency is day, the pattern can specify hours and minutes.
  • When frequency is week or month, the pattern can specify hours, minutes, and weekdays.
    • hours is an integer or list from 0 to 23.
    • minutes is an integer or list from 0 to 59.
    • weekdays is a string or list from monday to sunday.
• start_time (optional) is the start date and time with timezone. If omitted, the default is equal to schedule creation time. If the start time is in the past, the first job runs at the next calculated run time.
• end_time (optional) is the end date and time with timezone. If omitted, the schedule remains active until manually disabled.
• time_zone (optional) specifies the time zone of the recurrence schedule. If omitted, the default is Coordinated Universal Time (UTC). For more information about timezone values, see the appendix for timezone values.

To define a recurrence-based schedule, on the Basic settings screen, define the following properties. Only the Name property requires you to enter a value. If you don't specify values for the other properties, the default schedule is a Recurrence pattern that starts at schedule creation and runs every Monday through Friday at 4:00 PM UTC.

• Name: Unique identifier of the schedule within the workspace.
• Description: Schedule description.
• Trigger: Recurrence pattern of the schedule, including the following properties:
  • Time zone: Time zone to use for the trigger time, UTC by default.
  • Select Recurrence and specify a recurring pattern of minutes, hours, days, weeks, or months.
  • Start: Date the schedule becomes active, by default the date created.
  • End: Date when the schedule becomes inactive. By default the value is none, and the schedule remains active until manually disabled.
  • Tags: Tags on the schedule.

After you create the schedule YAML, use the following command to create the schedule via CLI:

# This action creates related resources for a schedule. It takes dozens of seconds to complete.
az ml schedule create --file simple-pipeline-job.yml --no-wait

The following Python code creates the schedule you defined:

job_schedule = ml_client.schedules.begin_create_or_update(
    schedule=job_schedule
).result()
print(job_schedule)

After you configure the basic settings, select Review + Create, review the settings, and then select Review + Create again to create the schedule.

The following YAML code defines a recurring schedule for a pipeline job. The required type parameter specifies that the trigger type is cron.

$schema: https://azuremlschemas.azureedge.net/latest/schedule.schema.json
name: simple_cron_job_schedule
display_name: Simple cron job schedule
description: a simple hourly cron job schedule

trigger:
  type: cron
  expression: "0 * * * *"
  start_time: "2022-07-10T10:00:00" # optional - default will be schedule creation time
  time_zone: "Pacific Standard Time" # optional - default will be UTC

# create_job: azureml:simple-pipeline-job
create_job: ./simple-pipeline-job.yml

You must or can provide the following schedule parameters:

The following code uses CronTrigger to provide a better coding experience.

schedule_name = "simple_sdk_create_schedule_cron"

schedule_start_time = datetime.utcnow()
cron_trigger = CronTrigger(
    expression="15 10 * * *",
    start_time=schedule_start_time,  # start time
    time_zone="Eastern Standard Time",  # time zone of expression
)

job_schedule = JobSchedule(
    name=schedule_name, trigger=cron_trigger, create_job=pipeline_job
)

You must or can provide the following schedule parameters:

To define a cron-based schedule, select Cron expression instead of Recurrence on the Basic settings screen for the schedule.

Parameters

• expression (required) is a standard crontab expression that expresses a recurring schedule.
• start_time (optional) is the schedule start date and time with timezone. For example, start_time: "2022-05-10T10:15:00-04:00" means the schedule starts from 10:15:00 AM on May 10, 2022 in UTC-4 timezone. If omitted, the default is equal to schedule creation time. If the start time is in the past, the first job runs at the next calculated run time.
• end_time (optional) is the end date and time with timezone. If omitted, the schedule remains active until manually disabled.
• time_zone (optional) specifies the time zone of the recurrence schedule. If omitted, the default is UTC.

Only the Name property requires you to enter a value. If you don't specify a cron expression, the default cron expression creates a schedule that runs daily at 4:00 PM UTC.

• Name: Unique identifier of the schedule within the workspace.
• Description: Description of the schedule.
• Trigger: Recurrence pattern of the schedule, including the following properties:
  • Time zone: Time zone to use for the trigger time, Coordinated Universal Time (UTC) by default.
  • Select Cron expression and provide a standard crontab expression that expresses a recurring schedule.
  • Start: Date the schedule becomes active, by default the date created.
  • End: Date the schedule becomes inactive. By default the value is none, and a schedule remains active until you manually disable it.
  • Tags: Tags on the schedule.

After you create the schedule YAML, use the following command to create the schedule via CLI:

# This action creates related resources for a schedule. It takes dozens of seconds to complete.
az ml schedule create --file simple-pipeline-job.yml --no-wait

The following Python code creates the schedule you defined:

job_schedule = ml_client.schedules.begin_create_or_update(
    schedule=job_schedule
).result()
print(job_schedule)

After you configure the basic settings, select Review + Create, review the settings, and then select Review + Create again to create the schedule.

When you define a schedule, you can change the settings, inputs, or outputs to use when running the pipeline job. You can also change the experiment_name of the triggered job.

The following schedule definition changes the settings of an existing job.

$schema: https://azuremlschemas.azureedge.net/latest/schedule.schema.json
name: cron_with_settings_job_schedule
display_name: Simple cron job schedule
description: a simple hourly cron job schedule

trigger:
  type: cron
  expression: "0 * * * *"
  start_time: "2022-07-10T10:00:00" # optional - default will be schedule creation time
  time_zone: "Pacific Standard Time" # optional - default will be UTC

create_job: 
  type: pipeline
  job: ./simple-pipeline-job.yml
  # job: azureml:simple-pipeline-job
  # runtime settings
  settings:
    #default_compute: azureml:cpu-cluster
    continue_on_step_failure: true
  inputs:
    hello_string_top_level_input: ${{name}} 
  tags: 
    schedule: cron_with_settings_schedule

When you define a schedule, you can change the settings, inputs, or outputs to use when running the pipeline job. You can also change the experiment_name of the triggered job.

The following schedule definition changes the settings of a pipeline per job.

# set run time settings
pipeline_job = pipeline_with_components_from_yaml(
    training_input=Input(type="uri_folder", path=parent_dir + "/data/"),
    test_input=Input(type="uri_folder", path=parent_dir + "/data/"),
    training_max_epochs=20,
    training_learning_rate=1.8,
    learning_rate_schedule="time-based",
)

# set pipeline level compute
pipeline_job.settings.default_compute = "cpu-cluster"

In the studio UI, you can use Advanced settings in the schedule creation wizard to modify inputs, outputs, and runtime settings for a pipeline job. You can't change the experiment_name in the studio UI.

1. In Job inputs & outputs, you can modify inputs and outputs for future jobs triggered by the schedule. You can use macro expressions for the inputs and outputs paths.

   

2. In Job runtime settings, you can modify compute and other runtime settings for jobs triggered by the schedule.

   

3. Select Review + Create to review the schedule settings you configured, and then select Review + Create again to create the schedule.

   

az ml schedule list

schedules = ml_client.schedules.list()
[s.name for s in schedules]

Under Jobs in the studio, select the All schedules tab to see a list of all the v2 job schedules created by SDK, CLI, or studio UI in this workspace.

az ml schedule show -n simple_cron_job_schedule

created_schedule = ml_client.schedules.get(name=schedule_name)
[created_schedule.name]

Select a schedule name to show the schedule detail page, which contains the following tabs:

• Overview provides basic information about the schedule.

  

• Job definition defines the job triggered by the schedule.

  

• Jobs history provides a list of all jobs triggered by the schedule.

  

az ml schedule update -n simple_cron_job_schedule  --set description="new description" --no-wait

# Update trigger expression
job_schedule.trigger.expression = "10 10 * * 1"
job_schedule = ml_client.schedules.begin_create_or_update(
    schedule=job_schedule
).result()
print(job_schedule)

In the schedule detail page, you can select Update settings to update the schedule settings, including job input/output and runtime settings.

Update a new version pipeline to an existing schedule

Once you set up a schedule to do regular retraining or batch inference on production, you might continue fine tuning or optimizing the model. When you have a new version pipeline job with optimized performance, you can update the new version pipeline to run on an existing schedule.

1. In the new version pipeline job detail page, select Schedule > Update to existing schedule.

   

2. Select an existing schedule to update its job definition.

   

   Important

   Make sure you select the correct schedule you want to update.

3. Optionally, select Next to modify the job inputs/outputs and runtime settings for the future jobs triggered by the schedule.

4. Select Review + Update to review the schedule settings, and then select Review + Update again to finish the update.

After the update completes, you can view the new job definition in the schedule detail page. The schedule now triggers the new job.

az ml schedule disable -n simple_cron_job_schedule --no-wait

The following code returns False:

job_schedule = ml_client.schedules.begin_disable(name=schedule_name).result()
job_schedule.is_enabled

You can disable schedules from the All schedules tab or disable the current schedule from the schedule detail page.

az ml schedule enable -n simple_cron_job_schedule --no-wait

The following code returns True:

job_schedule = ml_client.schedules.begin_enable(name=schedule_name).result()
job_schedule.is_enabled

You can enable schedules from the All schedules tab or enable the current schedule from the schedule detail page.

az ml schedule delete -n simple_cron_job_schedule

# Only disabled schedules can be deleted
ml_client.schedules.begin_disable(name=schedule_name).result()
ml_client.schedules.begin_delete(name=schedule_name).result()

You can delete schedules from the All schedules tab or delete the current schedule from the schedule detail page.