quickstarts/Tuning.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tce3stUlHN0L"
      },
      "source": [
        "##### Copyright 2024 Google LLC."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "tuOe1ymfHZPu"
      },
      "outputs": [],
      "source": [
        "# @title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yeadDkMiISin"
      },
      "source": [
        "# Gemini API: Tuning Quickstart with Python"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lEXQ3OwKIa-O"
      },
      "source": [
        "<table align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/google-gemini/cookbook/blob/main/quickstarts/Tuning.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jp_CKyzxUqx6"
      },
      "source": [
        "In this notebook, you'll learn how to get started with model tuning."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4x-2x8A_vi9g"
      },
      "source": [
        "## What is model tuning?\n",
        "\n",
        "Prompt design strategies such as few shot prompting may not always produce the results you need. Use model tuning to improve a model's performance on specific tasks or help the model adhere to specific output requirements when instructions aren't sufficient and you have a set of examples that demonstrate the outputs you want.\n",
        "\n",
        "The goal of model tuning is to further improve the performance of the model for your specific task. Model tuning works by providing the model with a training dataset containing many examples of the task. For niche tasks, you can get significant improvements in model performance by tuning the model on a modest number of examples.\n",
        "\n",
        "Your training data should be structured as examples with prompt inputs and expected response outputs. The goal is to teach the model to mimic the wanted behavior or task, by giving it many examples illustrating that behavior or task.\n",
        "\n",
        "You can also tune models using example data directly in Google AI Studio."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SWxKvwd-MSIV"
      },
      "source": [
        "## OAuth Authentication"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JjS8Zy1ojIgc"
      },
      "source": [
        "Unlike the other quickstarts which use API keys, model tuning uses OAuth.\n",
        "\n",
        "This tutorial assumes you have completed the [OAuth Quickstart](https://github.com/google-gemini/cookbook/blob/main/quickstarts/Authentication_with_OAuth.ipynb) and you have your client secret saved as `CLIENT_SECRET` in the Colab secrets manager.\n",
        "\n",
        "> Important: **Don't just click the link this command prints**. That will fail. Follow the instructions and copy the `gcloud` command it prints to your local machine and run it there, then paste the output from your local machine back\n",
        "here."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9FUwyB_MJ0-2"
      },
      "outputs": [],
      "source": [
        "from google.colab import userdata\n",
        "import pathlib\n",
        "pathlib.Path('client_secret.json').write_text(userdata.get('CLIENT_SECRET'))\n",
        "\n",
        "# Use `--no-browser` in colab\n",
        "!gcloud auth application-default login --no-browser --client-id-file client_secret.json --scopes='https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/generative-language.tuning'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cbcf72bcb56d"
      },
      "outputs": [],
      "source": [
        "!pip install -q -U google-generativeai"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8enrppafJPCX"
      },
      "outputs": [],
      "source": [
        "import google.generativeai as genai"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P-MYZECwlRCq"
      },
      "source": [
        "You can check your existing tuned models with the `genai.list_tuned_model` method."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XyWzoYFxU4r6"
      },
      "outputs": [],
      "source": [
        "for i, m in zip(range(5), genai.list_tuned_models()):\n",
        "  print(m.name)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BhkXRzciv3Dp"
      },
      "source": [
        "## Create tuned model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OO8VZYAinLWc"
      },
      "source": [
        "To create a tuned model, you need to pass your dataset to the model in the `genai.create_tuned_model` method. You can do this be directly defining the input and output values in the call or importing from a file into a dataframe to pass to the method.\n",
        "\n",
        "For this example, you will tune a model to generate the next number in the sequence. For example, if the input is `1`, the model should output `2`. If the input is `one hundred`, the output should be `one hundred one`.\n",
        "\n",
        "**Note**: In general, you need between 100 and 500 examples to significantly change the behavior of the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "w-EBSe9wTbLB"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "Model(name='models/gemini-1.0-pro-001',\n",
              "      base_model_id='',\n",
              "      version='001',\n",
              "      display_name='Gemini 1.0 Pro 001 (Tuning)',\n",
              "      description=('The best model for scaling across a wide range of tasks. This is a stable '\n",
              "                   'model that supports tuning.'),\n",
              "      input_token_limit=30720,\n",
              "      output_token_limit=2048,\n",
              "      supported_generation_methods=['generateContent', 'countTokens', 'createTunedModel'],\n",
              "      temperature=0.9,\n",
              "      top_p=1.0,\n",
              "      top_k=1)"
            ]
          },
          "execution_count": 23,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "base_model = [\n",
        "    m for m in genai.list_models()\n",
        "    if \"createTunedModel\" in m.supported_generation_methods][0]\n",
        "base_model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "baHjHh1oTTTC"
      },
      "outputs": [],
      "source": [
        "import random\n",
        "\n",
        "name = f'generate-num-{random.randint(0,10000)}'\n",
        "operation = genai.create_tuned_model(\n",
        "    # You can use a tuned model here too. Set `source_model=\"tunedModels/...\"`\n",
        "    source_model=base_model.name,\n",
        "    training_data=[\n",
        "        {\n",
        "             'text_input': '1',\n",
        "             'output': '2',\n",
        "        },{\n",
        "             'text_input': '3',\n",
        "             'output': '4',\n",
        "        },{\n",
        "             'text_input': '-3',\n",
        "             'output': '-2',\n",
        "        },{\n",
        "             'text_input': 'twenty two',\n",
        "             'output': 'twenty three',\n",
        "        },{\n",
        "             'text_input': 'two hundred',\n",
        "             'output': 'two hundred one',\n",
        "        },{\n",
        "             'text_input': 'ninety nine',\n",
        "             'output': 'one hundred',\n",
        "        },{\n",
        "             'text_input': '8',\n",
        "             'output': '9',\n",
        "        },{\n",
        "             'text_input': '-98',\n",
        "             'output': '-97',\n",
        "        },{\n",
        "             'text_input': '1,000',\n",
        "             'output': '1,001',\n",
        "        },{\n",
        "             'text_input': '10,100,000',\n",
        "             'output': '10,100,001',\n",
        "        },{\n",
        "             'text_input': 'thirteen',\n",
        "             'output': 'fourteen',\n",
        "        },{\n",
        "             'text_input': 'eighty',\n",
        "             'output': 'eighty one',\n",
        "        },{\n",
        "             'text_input': 'one',\n",
        "             'output': 'two',\n",
        "        },{\n",
        "             'text_input': 'three',\n",
        "             'output': 'four',\n",
        "        },{\n",
        "             'text_input': 'seven',\n",
        "             'output': 'eight',\n",
        "        }\n",
        "    ],\n",
        "    id = name,\n",
        "    epoch_count = 100,\n",
        "    batch_size=4,\n",
        "    learning_rate=0.001,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-As7ayWDK1w8"
      },
      "source": [
        "Your tuned model is immediately added to the list of tuned models, but its status is set to \"creating\" while the model is tuned."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "su64KgY4Uztj"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "TunedModel(name='tunedModels/generate-num-5392',\n",
              "           source_model='models/gemini-1.0-pro-001',\n",
              "           base_model='models/gemini-1.0-pro-001',\n",
              "           display_name='',\n",
              "           description='',\n",
              "           temperature=0.9,\n",
              "           top_p=1.0,\n",
              "           top_k=1,\n",
              "           state=<State.CREATING: 1>,\n",
              "           create_time=datetime.datetime(2024, 3, 16, 0, 41, 42, 702621, tzinfo=datetime.timezone.utc),\n",
              "           update_time=datetime.datetime(2024, 3, 16, 0, 41, 42, 702621, tzinfo=datetime.timezone.utc),\n",
              "           tuning_task=TuningTask(start_time=datetime.datetime(2024, 3, 16, 0, 41, 43, 81144, tzinfo=datetime.timezone.utc),\n",
              "                                  complete_time=None,\n",
              "                                  snapshots=[],\n",
              "                                  hyperparameters=Hyperparameters(epoch_count=100,\n",
              "                                                                  batch_size=4,\n",
              "                                                                  learning_rate=0.001)))"
            ]
          },
          "execution_count": 7,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "model = genai.get_tuned_model(f'tunedModels/{name}')\n",
        "\n",
        "model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EUodUwZkKPi-"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<State.CREATING: 1>"
            ]
          },
          "execution_count": 25,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "model.state"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pi8X5vkQv-3_"
      },
      "source": [
        "### Check tuning progress"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tWI-vAh4LJIz"
      },
      "source": [
        "Use `metadata` to check the state:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "g08vqtxYLMxT"
      },
      "outputs": [],
      "source": [
        "operation.metadata"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3lQ6gSMgK-kz"
      },
      "source": [
        "Wait for the training to finish using `operation.result()`, or `operation.wait_bar()`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SOUowIv1HgSE"
      },
      "outputs": [],
      "source": [
        "import time\n",
        "\n",
        "for status in operation.wait_bar():\n",
        "  time.sleep(30)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4cg868HzqOx5"
      },
      "source": [
        "You can cancel your tuning job any time using the `cancel()` method. Uncomment the line below and run the code cell to cancel your job before it finishes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oQuJ70_hqJi9"
      },
      "outputs": [],
      "source": [
        "# operation.cancel()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lqiL0TWDqAPn"
      },
      "source": [
        "Once the tuning is complete, you can view the loss curve from the tuning results. The [loss curve](https://generativeai.devsite.corp.google.com/guide/model_tuning_guidance#recommended_configurations) shows how much the model's predictions deviate from the ideal outputs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bIiG57xWLhP7"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<Axes: xlabel='epoch', ylabel='mean_loss'>"
            ]
          },
          "execution_count": 10,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjMAAAGwCAYAAABcnuQpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA5kElEQVR4nO3de3RU9b3//9dn77kkgSTcJIAGQdGicqkWpYhWWWCttXhpK9rSlmpPe6xYBDxV0GKrVGM9R5e1erTt6ql11Wsv2lYtalFEPYhcxANeuFR+wlcFrEACucxl78/vj5lMEggIM3tmMuH5WGtWyZ5J8mavtrzW+/Pen4+x1loBAACUKKfYBQAAAOSCMAMAAEoaYQYAAJQ0wgwAAChphBkAAFDSCDMAAKCkEWYAAEBJCxW7gHzzfV8ffPCBKisrZYwpdjkAAOAAWGu1a9cuDRo0SI6z/95Ltw8zH3zwgWpra4tdBgAAyMLmzZt1xBFH7Pcz3T7MVFZWSkrdjKqqqiJXAwAADkRDQ4Nqa2sz/47vT7cPM61LS1VVVYQZAABKzIGMiDAADAAAShphBgAAlDTCDAAAKGmEGQAAUNIIMwAAoKQRZgAAQEkjzAAAgJJGmAEAACWNMAMAAEoaYQYAAJQ0wgwAAChphBkAAFDSCDMAAKCkEWZy4PlWCc8vdhkAABzSCDM52NrQovc+bix2GQAAHNIIMznwrdXuWLLYZQAAcEgjzOSoJeHL822xywAA4JBFmMlR0vMVTzI3AwBAsRBmcpQgzAAAUFSEmRwlPKuY5xW7DAAADlmEmRzRmQEAoLgIMznyfKuWBJ0ZAACKhTCTIyupMUaYAQCgWAgzAWiKe7KWx7MBACgGwkwAkr6vOMcaAABQFISZHLnGMAQMAEAREWZyFHIdJT1LmAEAoEgIMzkySg0Bs8wEAEBxFDXMLF68WJMnT9agQYNkjNETTzyReS+RSOjaa6/VyJEj1aNHDw0aNEjf+ta39MEHHxSv4P2gMwMAQHEUNcw0NjZq9OjRuueee/Z6r6mpSStXrtS8efO0cuVK/fnPf9batWt13nnnFaHS/Qs5Rk1xHs8GAKAYQsX85eecc47OOeecTt+rrq7Wc8891+Ha3XffrVNOOUWbNm3S4MGDC1HiAQm7jhpjyWKXAQDAIamoYeZg1dfXyxijXr167fMzsVhMsVgs83VDQ0Pe6wo7jmJJX0nPV8hlDAkAgEIqmX95W1padO211+prX/uaqqqq9vm5uro6VVdXZ161tbV5ry3kph/PZggYAICCK4kwk0gkNGXKFFlrde+99+73s3PnzlV9fX3mtXnz5rzXF+bxbAAAiqbLLzO1Bpn33ntPzz///H67MpIUjUYVjUYLVF2K6xh51tKZAQCgCLp0mGkNMuvXr9cLL7ygvn37FrukfbM8ng0AQDEUNczs3r1bGzZsyHy9ceNGrVq1Sn369NHAgQP11a9+VStXrtSTTz4pz/O0ZcsWSVKfPn0UiUSKVbYkyVqrpnhSzelHsh1j1JLg8WwAAArN2CIe97xo0SJNmDBhr+vTpk3TT37yEw0dOrTT73vhhRd05plnHtDvaGhoUHV1terr6z9xiepg3LVwve54bp1OH9ZP13xhuD7eHVPvHhGNru0V2O8AAOBQdTD/fhe1M3PmmWdqf1mqiDnrE1WVpW5d62Z5IddRU9yTtVbGmGKWBgDAIaUknmbqiqorwpKkpnhqs7xw+vHsGHMzAAAUFGEmS9XlqTDTmO7MhF1HCd9XgieaAAAoKMJMllrDTGtnJuQY9poBAKAICDNZagszqc5M65wMe80AAFBYhJksVaXDTHPck99uUJnODAAAhUWYyVJrZ8aq3RNNjuH0bAAACowwk6VoyFU0lLp9u2OtTzQ5mYFgAABQGISZHFSm95rZ3dIWZmJJX0nmZgAAKBjCTA4qy9KPZ8fa9ppJJn2GgAEAKCDCTA4ynZlY6+PZjpK+VcLrujsXAwDQ3RBmcrBnmHGM5Fsr3yfMAABQKISZHPSMdgwzxhhZa+V14TOlAADobggzOWidmWkdAJYkY0RnBgCAAiLM5GDPZSZJsjJKEmYAACgYwkwOOgszspJHmAEAoGAIMznILDO1DzPpIWAAAFAYhJkcdNaZcWSUYJ8ZAAAKhjCTg9Yw0/48JseIfWYAACggwkwOKqN7LzM5Dp0ZAAAKiTCTg6p2nRmbnpNxjVHSJ8wAAFAohJkctC4z+VZqTqROy3Yco0SSZSYAAAqFMJODaNhVyDGS2jbOc4zk+2ycBwBAoRBmclQRcSW1P5/JyBNHGgAAUCiEmRz1iO552KSR71s2zgMAoEAIMznaqzPjpE/OpjMDAEBBEGZyVBHppDNj6cwAAFAohJkcZTozLe2XmVJDwAAAIP8IMzkwxqhHtOMyk+sYeT4DwAAAFAphJgeOkcojnZycbTg5GwCAQiHM5MAxRj3Sy0ztz2cy4uRsAAAKhTCTA8eYvQaAJcmKAWAAAAqFMJMDx+z9aHYrwgwAAIVBmMlBapkp3ZlpaRdmrGGZCQCAAiHM5MBx9n6aqfV6wuPZbAAACoEwkwPHSD3K2mZmbLob4xgpniTMAABQCISZHDjGqCq9zORbqTnhSZJcY5TwWGYCAKAQCDM5cIxRNOzIdYykjkcasMwEAEBhEGZy4DiS4zjqmT45uzHmpa8bJXmaCQCAgiDM5MAxJjU3s+fJ2UbyffaaAQCgEAgzOUiFGaMeUU7OBgCgWAgzOXBM6rDJ1o3zGlvaHTZpLXvNAABQAISZHJj0MtOeRxo4xsj67AIMAEAhFDXMLF68WJMnT9agQYNkjNETTzzR4X1rrW644QYNHDhQ5eXlmjRpktavX1+cYvchHDKdzsx4svLozAAAkHdFDTONjY0aPXq07rnnnk7fv+2223TXXXfpvvvu09KlS9WjRw+dffbZamlpKXCl++YaR+V7hhkn1Znx6cwAAJB3oWL+8nPOOUfnnHNOp+9Za3XnnXfqRz/6kc4//3xJ0gMPPKCamho98cQTuuSSSwpZ6j65TufLTAwAAwBQGF12Zmbjxo3asmWLJk2alLlWXV2tsWPHasmSJfv8vlgspoaGhg6vfAq5zr5PzmaZCQCAvOuyYWbLli2SpJqamg7Xa2pqMu91pq6uTtXV1ZlXbW1tXusMOaZtmamlY5jx2QQYAIC867JhJltz585VfX195rV58+a8/r6Q46g83ElnxtCZAQCgELpsmBkwYIAkaevWrR2ub926NfNeZ6LRqKqqqjq88slpv89MuzDDADAAAIXRZcPM0KFDNWDAAC1cuDBzraGhQUuXLtW4ceOKWFlHzh4DwDbdjTFGSnLYJAAAeVfUp5l2796tDRs2ZL7euHGjVq1apT59+mjw4MGaOXOmfvrTn+qYY47R0KFDNW/ePA0aNEgXXHBB8Yreg2OMKqKpTJj0rWJJX2VhV47hsEkAAAqhqGFm+fLlmjBhQubr2bNnS5KmTZum+++/X9dcc40aGxv1ve99Tzt37tRpp52mBQsWqKysrFgl78UxRmUhN3WEgW+1O5bMhJkEnRkAAPKuqGHmzDPPzCzLdMYYo5tuukk33XRTAas6OI6RZIx6RkOqb05od0tS/XpG5Rgp4dGZAQAg37rszEypcBwjWbv3kQYOnRkAAAqBMJMjxxgZY9SzbO9dgBOe3W/nCQAA5I4wkyPHpIJLj2jHMOOmjzRgBhgAgPwizOTIcYxkpB57ns/kiPOZAAAoAMJMjhxj5BipR3SPmRlj5PtWPstMAADkFWEmR5llpnRnppGTswEAKCjCTI5SA8AddwFOXZc8KzbOAwAgzwgzOUotM5m2ZaaWtkezOZ8JAID8I8zkyDGpzf0qOltmkuXkbAAA8owwkyPHGLkyqoikbuXudidnS3RmAADIN8JMjhzHyDh7z8y0ojMDAEB+EWYCEHKMyvc4zqAVTzMBAJBfhJkAhBxHFeFUZybhWcWSXuY9n+OZAADIK8JMABxHioZSm+dJ7Z5oklGSNAMAQF4RZgIQcox8aa/zmRwjTs4GACDPCDMBCDlG1ko99wwzjiHMAACQZ4SZAIRcR9a2dWba7zXTbnwGAADkAWEmAK6T2iCvrTOTSjCOkRLMzAAAkFeEmQA4xkgdwkz7zoyVZa8ZAADyhjATAMcxkkwmzDS2m5nh5GwAAPKLMBOA1key9+7MpDbNYxdgAADyhzATAMcYGdlOHs1OdWYYmwEAIH8IMwFwjJHtZJnJbV1mojMDAEDeEGYCkFlmKuukM+NzPhMAAPlEmAmAY4wcGfVoPWyypW1mJrXMRJgBACBfCDMBcIyRMVLFHidnG2NkWWYCACCvCDMBcBzJOFJFJD0zE09m3jNGdGYAAMgjwkwAUstMbZ2ZloSvZPpMJiujJGEGAIC8IcwEwDFGjjEqC7uZa61LTbIMAAMAkE+EmQAYk5qPMUZtQ8CtYSY9BAwAAPKDMBMA1zFyjDqcnJ15PFtGCY9d8wAAyBfCTABSTzMZWat2G+e1OznbozMDAEC+EGYC4Ji2owv2PJ/JpK8DAID8IMwEwBiT2SBv7/OZGAAGACCfCDMBCbnpZaayTjozjMwAAJA3hJmAhBwn1ZlJb5zXeqSBkeRZ0gwAAPlCmAmI60hWbZ2Z1l2AjZFYZQIAIH8IMwFxHUe+324AuKVtmYmZGQAA8ocwE5Cwa+Sr/aPZbctM1kqWJ5oAAMgLwkxAQo4j28mj2U765GyaMwAA5AdhJiBOetO8HtFOjjMQnRkAAPKFMBMQY1L/uecyk5NeZ6IzAwBAfnTpMON5nubNm6ehQ4eqvLxcRx99tObPn98luxyuYyTTtszUGPfk+VZGRr6VrLpezQAAdAehYhewPz/72c9077336ne/+51OOOEELV++XJdeeqmqq6s1Y8aMYpfXgWOMJJPZAViSmuJJhd3ULE0XzF8AAHQLXTrM/O///q/OP/98nXvuuZKkIUOG6OGHH9Zrr71W5Mr25qSXmcKuo2jIUSzpa3csqT49IrIS5zMBAJAnXXqZ6dRTT9XChQu1bt06SdIbb7yhl19+Weecc84+vycWi6mhoaHDqxAcx6i1/dJ+rxkjQ2cGAIA86tKdmTlz5qihoUHDhw+X67ryPE8333yzpk6dus/vqaur04033ljAKlNal5mkVJj5uDGuxrgnY0RnBgCAPOrSnZnHHntMDz74oB566CGtXLlSv/vd7/Rf//Vf+t3vfrfP75k7d67q6+szr82bNxekVsekhoB9azscNumY1AAwTzMBAJAfXboz88Mf/lBz5szRJZdcIkkaOXKk3nvvPdXV1WnatGmdfk80GlU0Gi1kmZJSxxbIqNMjDWTTLwAAELgu3ZlpamqS43Qs0XVd+X7XO4XadYyc9JJS5uTsWFKtozQsMwEAkB9dujMzefJk3XzzzRo8eLBOOOEEvf7667rjjjt02WWXFbu0vThGcrT3MpMxqWtEGQAA8qNLh5lf/OIXmjdvnq644gpt27ZNgwYN0r//+7/rhhtuKHZpe3GMkXFSXZg9dwGWLJ0ZAADypEuHmcrKSt1555268847i13KJzJG6WFfm9k4r/V8ptbuDAAACF6XnpkpJa4x6dCivU7OliwDwAAA5AlhJiCOMXKUOh27554nZ6fPZwIAAMEjzATEcYwcJxVaeuwxM8OmeQAA5A9hJkCZTfP2WmZilQkAgHwJJMw0NDToiSee0Ntvvx3EjytZIcfs9TSTtVZGqc30AABA8LIKM1OmTNHdd98tSWpubtaYMWM0ZcoUjRo1Sn/6058CLbCUuE7qUMnWfWZ8KzUnPA6ZBAAgj7IKM4sXL9bpp58uSXr88cdlrdXOnTt111136ac//WmgBZaSUHpmJuI6CjmpQyd3tyRlDDMzAADkS1Zhpr6+Xn369JEkLViwQF/5yldUUVGhc889V+vXrw+0wFLSOjNjjOm4C7CMPJaZAADIi6zCTG1trZYsWaLGxkYtWLBAn//85yVJO3bsUFlZWaAFlpKQ42SWlDrsAkxnBgCAvMlqB+CZM2dq6tSp6tmzp4488kideeaZklLLTyNHjgyyvpISctt2+m3/RFONRGcGAIA8ySrMXHHFFTrllFO0efNmnXXWWZmTrY866qhDembGMUYyqdDS4UgDQ5gBACBfsj6bacyYMRozZowkyfM8rV69Wqeeeqp69+4dWHGlxnGMpNTgb/vOjGOYmQEAIF+ympmZOXOmfvOb30hKBZkzzjhDJ510kmpra7Vo0aIg6ysp6QeYJLUPM15qnxmyDAAAeZFVmPnjH/+o0aNHS5L+9re/aePGjXrnnXc0a9YsXX/99YEWWEocY9SaZ9oPAHNqNgAA+ZNVmPnXv/6lAQMGSJKefvppXXTRRTr22GN12WWXafXq1YEWWEqMaTu2oP0yk2FmBgCAvMkqzNTU1Oitt96S53lasGCBzjrrLElSU1OTXNcNtMBS4pq2daYe7U7OZmYGAID8yWoA+NJLL9WUKVM0cOBAGWM0adIkSdLSpUs1fPjwQAssJa3LTLb9YZMtqcMmyTIAAORHVmHmJz/5iUaMGKHNmzfroosuUjQalSS5rqs5c+YEWmApcYyRY1JHGnR8mkmSTR026bSfEgYAADnL+tHsr371q3tdmzZtWk7FlDrjpF5+u8MmG+Op4wx8WdGcAQAgeFnNzEjSiy++qMmTJ2vYsGEaNmyYzjvvPL300ktB1lZyXGPkSLK23aZ5LUlJVtZanmgCACAPsgozv//97zVp0iRVVFRoxowZmjFjhsrLyzVx4kQ99NBDQddYMhxjMo9hty4zJX2rhO/LivOZAADIh6yWmW6++WbddtttmjVrVubajBkzdMcdd2j+/Pn6+te/HliBpcSYVKCxVioPu3JMavC3KeapPOqKLAMAQPCy6sy8++67mjx58l7XzzvvPG3cuDHnokqV65h0gLEyxmSWmhrjnmRFmAEAIA+yCjO1tbVauHDhXtf/8Y9/qLa2NueiSpWbeZqp48nZTfGkrBUjwAAA5EFWy0xXX321ZsyYoVWrVunUU0+VJL3yyiu6//779fOf/zzQAkuJ4xiFXKNYwpfUPsx48mXZawYAgDzIKsx8//vf14ABA3T77bfrsccekyQdd9xxevTRR3X++ecHWmCpiYZcNcU8SW1PNDXGkgwAAwCQJ1nvM3PhhRfqwgsvDLKWbiHsGnl7LDM1xjxZn5kZAADyIet9ZtC5SMiRn1plagszrTMzpBkAAAJ3wJ2Z3r17y5gD24p/+/btWRdU6iKuu9cAcGqZiZkZAADy4YDDzJ133pnHMroP1zWSSaWWHu2XmURnBgCAfDjgMJPNuUu33nqrLr/8cvXq1eugv7dUhRyj1NnZUkXElSQ1JzwZcXI2AAD5kNeZmVtuueWQW3JyHZPpwJSFU7e3JZHuzLDPDAAAgctrmDkUl1VCjpFjHPnWKhpKdWZakr6MeJoJAIB84GmmgLmOkeNKnm8VTXdmYsnUvjPsMwMAQPAIMwELOY5cY+T5VmXpzkws4cvK0pkBACAPCDMBcx0j10mdz1QWTi8zJTwZGTozAADkAWEmYK5jMp2Z1mWmlqRHZwYAgDzJa5g5/fTTVV5ens9f0eW0dmb2XGaSDGEGAIA8yPpsJt/3tWHDBm3btk1+6/79aZ/73OckSU8//XRu1ZWoaMhVS8LPPJqd9K0837LMBABAHmQVZl599VV9/etf13vvvbfX49fGGHmeF0hxpSoSSnVmKsvabm8i6ctj1zwAAAKXVZi5/PLLNWbMGD311FMaOHDgAZ/ZdKiIhBx51qb3nEnt/Bv3fDozAADkQVZhZv369frjH/+oYcOGBV1Pt9B62KQxRmVhV01xT/GkrySdGQAAApfVAPDYsWO1YcOGoGvp1Pvvv69vfOMb6tu3r8rLyzVy5EgtX768IL87W67b1qmKhlK3OOExMwMAQD5k1Zn5wQ9+oKuvvlpbtmzRyJEjFQ6HO7w/atSoQIrbsWOHxo8frwkTJujvf/+7DjvsMK1fv169e/cO5Ofni2taj5pUeq+ZhBJJj5kZAADyIKsw85WvfEWSdNlll2WuGZM6YDHIAeCf/exnqq2t1W9/+9vMtaFDhwbys/Op/WGTrZ2ZWNLn0WwAAPIgqzCzcePGoOvo1F//+ledffbZuuiii/Tiiy/q8MMP1xVXXKHvfve7+/yeWCymWCyW+bqhoaEQpXbQ/rDJ1l2A456v5B6PsAMAgNxlFWaOPPLIoOvo1Lvvvqt7771Xs2fP1nXXXadly5ZpxowZikQimjZtWqffU1dXpxtvvLEg9e2L66YOm/T9dmEm6YtVJgAAgpf1pnmS9NZbb2nTpk2Kx+Mdrp933nk5FdXK932NGTNGt9xyiyTpxBNP1Jo1a3TfffftM8zMnTtXs2fPznzd0NCg2traQOo5UKHWIw2sbVtm8thnBgCAfMgqzLz77ru68MILtXr16sysjKTMfjNBzcwMHDhQxx9/fIdrxx13nP70pz/t83ui0aii0Wggvz9brmPktJ6c3dqZSaRmZlrnigAAQDCyejT7qquu0tChQ7Vt2zZVVFTozTff1OLFizVmzBgtWrQosOLGjx+vtWvXdri2bt26gi1zZSvkOKmTs32prF1nxloOmwQAIGhZhZklS5bopptuUr9+/eQ4jhzH0Wmnnaa6ujrNmDEjsOJmzZqlV199Vbfccos2bNighx56SL/61a80ffr0wH5HPmQOm7RW0fYzMxJ7zQAAELCswozneaqsrJQk9evXTx988IGk1GDwnp2UXJx88sl6/PHH9fDDD2vEiBGaP3++7rzzTk2dOjWw35Evrecztc7MxJO+ZC1DwAAABCyrmZkRI0bojTfe0NChQzV27FjddtttikQi+tWvfqWjjjoq0AK/9KUv6Utf+lKgP7MQIq4jz09mZmZiSU++laxIMwAABCmrMPOjH/1IjY2NkqSbbrpJX/rSl3T66aerb9++evTRRwMtsFRFQ648a1UWbr9pHjMzAAAELaswc/bZZ2f+PGzYML3zzjvavn27evfuzZM6aZFQatO8aKi1M+PLipkZAACCltXMTKsNGzbomWeeUXNzs/r06RNUTd2C6xjJqm2ZKeHRmQEAIA+yCjMff/yxJk6cqGOPPVZf/OIX9eGHH0qSvvOd7+jqq68OtMBSFXIcyajjMpPozAAAELSswsysWbMUDoe1adMmVVRUZK5ffPHFWrBgQWDFlTLXSS23laWXmVoSrQPAAAAgSFnNzDz77LN65plndMQRR3S4fswxx+i9994LpLBSF3KMjFKzM1KqMyMrWc6aBAAgUFl1ZhobGzt0ZFpt37696EcJdBWua+Q4RhE31aFJPc3EMhMAAEHLKsycfvrpeuCBBzJfG2Pk+75uu+02TZgwIbDiSplrjELGKJzuzKSWmdhlBgCAoGW1zHTbbbdp4sSJWr58ueLxuK655hq9+eab2r59u1555ZWgayxJrpPqzISc9p0Zn84MAAABy6ozM2LECK1du1annXaazj//fDU2NurLX/6yXn/9dR199NFB11iSQunzmcKOm7mW8FlmAgAgaFl1ZiSprKxMZ511lkaPHi3fT021Llu2TJJ03nnnBVNdCWs9bDLktm0iGE8meZwJAICAZRVmFixYoG9+85vavn277B6dBmOMPM8LpLhSZoxRJGQUT6TOaYp7vmJJDpoEACBoWS0z/eAHP9CUKVP0wQcfyPf9Di+CTJuw48izVtEOG+eRZgAACFJWYWbr1q2aPXu2ampqgq6nW4mGHHm+bXdytk9nBgCAgGUVZr761a9q0aJFAZfS/UTDbirMpB/Pjid8+aQZAAACldXMzN13362LLrpIL730kkaOHKlwONzh/RkzZgRSXKlz0ieIRzOdGZbgAAAIWlZh5uGHH9azzz6rsrIyLVq0SMa0PbFjjCHMpIVckzpssrUz47HPDAAAQcsqzFx//fW68cYbNWfOHDlOVitVh4TMYZPpzkw8yTITAABByyqJxONxXXzxxQSZTxBynA6HTcY9K4/ODAAAgcoqjUybNk2PPvpo0LV0O65j5BijaKi1M+PJozMDAECgslpm8jxPt912m5555hmNGjVqrwHgO+64I5DiSl3qSIPUI9pSapnJ84tcFAAA3UxWYWb16tU68cQTJUlr1qzp8F77YeBDXethkxG3/QAwaQYAgCBlFWZeeOGFoOvollKdGUfhdGcmlqAzAwBA0JjgzSPXMXKNaTcAzKPZAAAEjTCTR8YYhV2jSPrk7FjSZwAYAICAEWbyLBJyFG6dmUl6dGYAAAgYYSbPou3CTCxpmZkBACBghJk8i7iOwullpnjSk6zYBRgAgAARZvIs5DoKh9pmZnxZEWUAAAgOYSbPXKdtB+BY0peVmJsBACBAhJk8CzkmswNwLOHJWiuyDAAAwSHM5Fn7HYBbkql9ZujMAAAQHMJMnrnGqCycWmbyfKuk59OZAQAgQISZPHOctjAjSS1xRoABAAgSYSbPXMcoEjJy0udvtiQ98WQ2AADBIczkmWuMQo6jaLo7E/N8WdaZAAAIDGEmzxwnvdSUfqKpJe7TmQEAIECEmTxzjZHT7uTsWNKjMwMAQIAIM3nmOqkw07pxXkuCp5kAAAgSYSbPjDEKux07M+wzAwBAcAgzBRB2HUXdtl2AmZkBACA4hJkCCLXrzMQ9yz4zAAAEqKTCzK233ipjjGbOnFnsUg5KxHXawkySmRkAAIJUMmFm2bJl+uUvf6lRo0YVu5SDFnadzPlMMcIMAACBKokws3v3bk2dOlW//vWv1bt372KXc9BSuwC3dmYYAAYAIEglEWamT5+uc889V5MmTfrEz8ZiMTU0NHR4FZvTIcz4hBkAAAIUKnYBn+SRRx7RypUrtWzZsgP6fF1dnW688cY8V3VwQu12AGaZCQCAYHXpzszmzZt11VVX6cEHH1RZWdkBfc/cuXNVX1+feW3evDnPVX4yxxiFCTMAAORFl+7MrFixQtu2bdNJJ52UueZ5nhYvXqy7775bsVhMrut2+J5oNKpoNFroUvfLdYyi6TpjLDMBABCoLh1mJk6cqNWrV3e4dumll2r48OG69tpr9woyXZVrjMoibTMzHrvmAQAQmC4dZiorKzVixIgO13r06KG+ffvudb0rcxypLJwKXnGPzgwAAEHq0jMz3YXrGJWHW5eZPHmEGQAAAtOlOzOdWbRoUbFLOGiOMW2dGZaZAAAIFJ2ZAnAdo/JI2wAwYQYAgOAQZgrANUblobbODKtMAAAEhzBTAI5jVBFte5op6XtFrggAgO6DMFMgPcvCkiQrqTnuF7cYAAC6EcJMgVSE2/bEaYrTmQEAICiEmQIpj7gKu0aS1JLwZBmcAQAgEISZAgm7jqLpIeBUmClyQQAAdBOEmQIJOY4i6cMmmxPsAgwAQFAIMwXiOFK09eTshCeiDAAAwSDMFEjHzoxHZwYAgIAQZgqkQ2eGjfMAAAgMYaZAXGMynZlYkgFgAACCQpgpENcx7Z5mYgAYAICgEGYKxHGMysJtnRnCDAAAwSDMFIhrOnZmiDIAAASDMFMgbvvOTMKX5xFnAAAIAmGmQFzHqCzdmYklPSV9wgwAAEEgzBSIa4zKwq1hxlfS5+RsAACCQJgpEMcxKou0DQAnWWYCACAQhJkC6hEJSZISnlXCozMDAEAQCDMF1COSWmaKJ321JLwiVwMAQPdAmCmgHtFUZyae9BVL0pkBACAIhJkC6pkOMzGPMAMAQFAIMwXUvjOT9K2SzM0AAJAzwkwBVaRnZmIJT57vs9cMAAABIMwUUM+y9DJT0pfnW8IMAAABIMwUUOuj2UnfppaaWGYCACBnhJkCap2ZkaTmeJLODAAAASDMFFBZyJFjUn9OeJZdgAEACABhpoBc11E0fdhkS8JnF2AAAAJAmCkg1zGZIeDdsSQzMwAABIAwU0CuMepVHpYk7WpJqIWN8wAAyBlhpoBcx6hXRSrMNDQnFSfMAACQM8JMAbmOUZ8eEUlSfUuCwyYBAAgAYaaAHCP1rkiFmYbmhJK+lcfj2QAA5IQwU0DGGPXrmQozO5sS8nzLE00AAOSIMFNgh1WWSZJ2Nifk+T6dGQAAckSYKbDDKlOdmR1NcXm+2DgPAIAcEWYKrCbdmdnVklRL0lPSZ5kJAIBcEGYKrE+PiELpMw12pYeAAQBA9ggzBRZyHVWnN87b2ZxgABgAgBwRZgrMdYyq0mGmvjnBADAAADnq8mGmrq5OJ598siorK9W/f39dcMEFWrt2bbHLytqeuwCzcR4AALnp8mHmxRdf1PTp0/Xqq6/queeeUyKR0Oc//3k1NjYWu7SsOMZklpl2tSQU40gDAAByEip2AZ9kwYIFHb6+//771b9/f61YsUKf+9znilRV9lzHqLqsbWaGMAMAQG66fJjZU319vSSpT58+nb4fi8UUi8UyXzc0NBSkrgPlGqPePdqONEgkffm+lZN+wgkAABycLr/M1J7v+5o5c6bGjx+vESNGdPqZuro6VVdXZ161tbUFrnL/HEfqXdHWmfGsVYK9ZgAAyFpJhZnp06drzZo1euSRR/b5mblz56q+vj7z2rx5cwEr/GSu09aZ2dEYl+dx2CQAALkomWWmK6+8Uk8++aQWL16sI444Yp+fi0ajikajBazs4LiOUZ/Wk7NbkmrxPCU40gAAgKx1+c6MtVZXXnmlHn/8cT3//PMaOnRosUvKiWuMqivCck1qRmbnbvaaAQAgF12+MzN9+nQ99NBD+stf/qLKykpt2bJFklRdXa3y8vIiV3fwQq6jsrCr6oqwtjfGVd+SUJJdgAEAyFqX78zce++9qq+v15lnnqmBAwdmXo8++mixS8taedhp93h2XAk6MwAAZK3Ld2as7X7/0PeIhFRVnrr19c1JeczMAACQtS7fmemOIullJim110wsyZEGAABkizBTBBG3bZmJMAMAQG4IM0UQcZ3MXjP1zQm1JBgABgAgW4SZIoiEHPVtF2YSSdstZ4MAACgEwkwRhF2jvj1TG/vtbErIsz4b5wEAkCXCTBGEXEc1VakwU9+SOjmbjfMAAMgOYaZIBlSVyRjJWmlHc5zDJgEAyBJhpkiqysOqSj/RtGN3nL1mAADIEmGmSMKuo+ryVJipb07QmQEAIEuEmSKJhBz1SoeZHU0JJenMAACQFcJMkYRdR70q2jozDAADAJAdwkyRREOO+qT3mmloTrILMAAAWSLMFEnYbds4r6Eloe2NcTXGkkWuCgCA0kOYKRLXMTosvdfM7lhS2xvjevvDBgINAAAHiTBTRAOryyVJOxrjGlhdro93x/XWhw3aTaABAOCAEWaK6IheFZKkHU1xWSsNqC7T9sa43vqgXrtaEkWuDgCA0kCYKaL+1REZSb5Nzc04xmhAVZl2NiX01gcNakkwFAwAwCchzBRRRTikyrKQpNRSk6RMoKlvTrDcBADAASDMFFEk5Kg6vdfM9nSYkSRjjHxr6cwAAHAACDNFFHYd9a5IPZ69vSne4T3XcdQcJ8wAAPBJCDNFFHEd9U2HmR2N8b3e2xVjCBgAgE9CmCkixzHqV5naa+bjTsJMc9xX0uMASgAA9ocwU2RH9E7tNbN+6+4O18Mho0TSVyxJmAEAYH8IM0U2YXh/hRyjDR/t1vqtuzLXI66juOczBAwAwCcgzBRZTVWZThrcW5L09zVbMteNMbISnRkAAD4BYabIwq6jMz51mCTpxfUfaXdL294yjhGdGQAAPgFhpsgiIUfDB/TU4D4Viid9Pb92W+a9sONoVwsb5wEAsD+EmSILu0aRkKtJx/WXJC1Y86Gsten3HDXFPPm+LWaJAAB0aYSZIou4jkKOo88e1VdlYUebdzRrzfv1qfdCjuKex9wMAAD7QZgpMmOMKiKuIq6jM49NdWeeTg8Ch11H8STHGgAAsD+EmS6gR9RVwrf64sgBkqQl736sHY1xuY6RlaUzAwDAfhBmuoBoyJXnWw3t11PDB1TK862efXurJKUfz6YzAwDAvhBmuoCe0ZBCjlHS83XOiIGSpGfe3CLPt+knmjijCQCAfSHMdAFV5WH1LAtpVyyp04b1U2U0pI92xfT6ph2KhBztjnmZJ5wAAEBHhJkuwHWMBlaXqSnuKRJyNGF4ahD4ube3KuI6iiV5ogkAgH0hzHQRvSoiioSMYklPZx1XI0l6beN2NSc8JZJWsQRhBgCAzhBmuoiqspCqy8Pa3ZLUkH49dEz/nkr6VovXfSTPWoaAAQDYB8JMF2GM0YDqcrWkl5POOj7VnXn27a2y1meZCQCAfSDMdCG9ysMqCztqjnv63DGHKRJytHl7kzZ/3KzdMc5oAgCgM4SZLqRHNKQ+PSLaFUuoRzSk8Uf3lSS98s+PCTMAAOwDYaaL6V9ZpoRnZa3VWcendgReunG7djbFmZsBAKAThJkupldFWD2irhrjnkYMqtLA6jI1Jzz97z8/Zm4GAIBOlESYueeeezRkyBCVlZVp7Nixeu2114pdUt6UhV0d1jOqxlhSxhhNSj+m/dK6j3g8GwCATnT5MPPoo49q9uzZ+vGPf6yVK1dq9OjROvvss7Vt27Zil5Y3fXtG5Vkrz7eaOLy/HCNt+KhR67ftKnZpAAB0OcZ28X3yx44dq5NPPll33323JMn3fdXW1uoHP/iB5syZ84nf39DQoOrqatXX16uqqirf5QYi4fl6Y/NOfdwYU9hxdfcL67Vy004dVhnVyMOrdWxNpT41oKcO71WhsGsUdh2FXUch18gxRkaSMZKRkTHB1RXkz2p1oP/ty8fvBgAEozIaVnVFONCfeTD/focC/c0Bi8fjWrFihebOnZu55jiOJk2apCVLlnT6PbFYTLFYLPN1Q0ND3usMWth1NOLwau1oimtrfYsmDD9Mr2/aqY92xfT8O9v0/DvdtysFACg9//65ozT3i8cV7fd36TDzr3/9S57nqaampsP1mpoavfPOO51+T11dnW688cZClJdXZWFXA6vLNaCqTEMP66lxR/XV6v/XoP/v40a9v7NZ/29Hs+qbE/L81HJU0veV9K2sTXU7rFItD2s772q074h06dYc8sdKKlTH60B+V1er50B/zoE4kL97toL62fmqMZ9/96CUQo1diZGcPe5ZyC1u+7xLh5lszJ07V7Nnz8583dDQoNra2iJWlBtjjKrLwxozpK/GDOnb4T3fT0UWa1P/6du2MNPKHsD/CvcVePb8DAAAYddRJNS1Rm67dJjp16+fXNfV1q1bO1zfunWrBgwY0On3RKNRRaPRQpRXdE4mGjNQAgA4dHWtaLWHSCSiz3zmM1q4cGHmmu/7WrhwocaNG1fEygAAQFfRpTszkjR79mxNmzZNY8aM0SmnnKI777xTjY2NuvTSS4tdGgAA6AK6fJi5+OKL9dFHH+mGG27Qli1b9OlPf1oLFizYaygYAAAcmrr8PjO5KsV9ZgAAONQdzL/fXXpmBgAA4JMQZgAAQEkjzAAAgJJGmAEAACWNMAMAAEoaYQYAAJQ0wgwAAChphBkAAFDSCDMAAKCkdfnjDHLVusFxQ0NDkSsBAAAHqvXf7QM5qKDbh5ldu3ZJkmpra4tcCQAAOFi7du1SdXX1fj/T7c9m8n1fH3zwgSorK2WMCfRnNzQ0qLa2Vps3b+bcpzziPhcG97kwuM+FwX0ujHzeZ2utdu3apUGDBslx9j8V0+07M47j6Igjjsjr76iqquJ/LAXAfS4M7nNhcJ8Lg/tcGPm6z5/UkWnFADAAAChphBkAAFDSCDM5iEaj+vGPf6xoNFrsUro17nNhcJ8Lg/tcGNznwugq97nbDwADAIDujc4MAAAoaYQZAABQ0ggzAACgpBFmAABASSPMZOmee+7RkCFDVFZWprFjx+q1114rdkklra6uTieffLIqKyvVv39/XXDBBVq7dm2Hz7S0tGj69Onq27evevbsqa985SvaunVrkSruHm699VYZYzRz5szMNe5zMN5//3194xvfUN++fVVeXq6RI0dq+fLlmfettbrhhhs0cOBAlZeXa9KkSVq/fn0RKy5Nnudp3rx5Gjp0qMrLy3X00Udr/vz5Hc7z4V4fvMWLF2vy5MkaNGiQjDF64oknOrx/IPd0+/btmjp1qqqqqtSrVy995zvf0e7du/NTsMVBe+SRR2wkErH/8z//Y99880373e9+1/bq1ctu3bq12KWVrLPPPtv+9re/tWvWrLGrVq2yX/ziF+3gwYPt7t27M5+5/PLLbW1trV24cKFdvny5/exnP2tPPfXUIlZd2l577TU7ZMgQO2rUKHvVVVdlrnOfc7d9+3Z75JFH2m9/+9t26dKl9t1337XPPPOM3bBhQ+Yzt956q62urrZPPPGEfeONN+x5551nhw4dapubm4tYeem5+eabbd++fe2TTz5pN27caP/whz/Ynj172p///OeZz3CvD97TTz9tr7/+evvnP//ZSrKPP/54h/cP5J5+4QtfsKNHj7avvvqqfemll+ywYcPs1772tbzUS5jJwimnnGKnT5+e+drzPDto0CBbV1dXxKq6l23btllJ9sUXX7TWWrtz504bDoftH/7wh8xn3n77bSvJLlmypFhllqxdu3bZY445xj733HP2jDPOyIQZ7nMwrr32Wnvaaaft833f9+2AAQPsf/7nf2au7dy500ajUfvwww8XosRu49xzz7WXXXZZh2tf/vKX7dSpU6213Osg7BlmDuSevvXWW1aSXbZsWeYzf//7360xxr7//vuB18gy00GKx+NasWKFJk2alLnmOI4mTZqkJUuWFLGy7qW+vl6S1KdPH0nSihUrlEgkOtz34cOHa/Dgwdz3LEyfPl3nnntuh/spcZ+D8te//lVjxozRRRddpP79++vEE0/Ur3/968z7Gzdu1JYtWzrc5+rqao0dO5b7fJBOPfVULVy4UOvWrZMkvfHGG3r55Zd1zjnnSOJe58OB3NMlS5aoV69eGjNmTOYzkyZNkuM4Wrp0aeA1dfuDJoP2r3/9S57nqaampsP1mpoavfPOO0WqqnvxfV8zZ87U+PHjNWLECEnSli1bFIlE1KtXrw6framp0ZYtW4pQZel65JFHtHLlSi1btmyv97jPwXj33Xd17733avbs2bruuuu0bNkyzZgxQ5FIRNOmTcvcy87+f4T7fHDmzJmjhoYGDR8+XK7ryvM83XzzzZo6daokca/z4EDu6ZYtW9S/f/8O74dCIfXp0ycv950wgy5n+vTpWrNmjV5++eVil9LtbN68WVdddZWee+45lZWVFbucbsv3fY0ZM0a33HKLJOnEE0/UmjVrdN9992natGlFrq57eeyxx/Tggw/qoYce0gknnKBVq1Zp5syZGjRoEPf6EMIy00Hq16+fXNfd6+mOrVu3asCAAUWqqvu48sor9eSTT+qFF17QEUcckbk+YMAAxeNx7dy5s8Pnue8HZ8WKFdq2bZtOOukkhUIhhUIhvfjii7rrrrsUCoVUU1PDfQ7AwIEDdfzxx3e4dtxxx2nTpk2SlLmX/P9I7n74wx9qzpw5uuSSSzRy5Eh985vf1KxZs1RXVyeJe50PB3JPBwwYoG3btnV4P5lMavv27Xm574SZgxSJRPSZz3xGCxcuzFzzfV8LFy7UuHHjilhZabPW6sorr9Tjjz+u559/XkOHDu3w/mc+8xmFw+EO933t2rXatGkT9/0gTJw4UatXr9aqVasyrzFjxmjq1KmZP3Ofczd+/Pi9thZYt26djjzySEnS0KFDNWDAgA73uaGhQUuXLuU+H6SmpiY5Tsd/ylzXle/7krjX+XAg93TcuHHauXOnVqxYkfnM888/L9/3NXbs2OCLCnyk+BDwyCOP2Gg0au+//3771ltv2e9973u2V69edsuWLcUurWR9//vft9XV1XbRokX2ww8/zLyampoyn7n88svt4MGD7fPPP2+XL19ux40bZ8eNG1fEqruH9k8zWct9DsJrr71mQ6GQvfnmm+369evtgw8+aCsqKuzvf//7zGduvfVW26tXL/uXv/zF/t///Z89//zzeVw4C9OmTbOHH3545tHsP//5z7Zfv372mmuuyXyGe33wdu3aZV9//XX7+uuvW0n2jjvusK+//rp97733rLUHdk+/8IUv2BNPPNEuXbrUvvzyy/aYY47h0eyu5he/+IUdPHiwjUQi9pRTTrGvvvpqsUsqaZI6ff32t7/NfKa5udleccUVtnfv3raiosJeeOGF9sMPPyxe0d3EnmGG+xyMv/3tb3bEiBE2Go3a4cOH21/96lcd3vd9386bN8/W1NTYaDRqJ06caNeuXVukaktXQ0ODveqqq+zgwYNtWVmZPeqoo+z1119vY7FY5jPc64P3wgsvdPr/ydOmTbPWHtg9/fjjj+3XvvY127NnT1tVVWUvvfRSu2vXrrzUa6xtt00iAABAiWFmBgAAlDTCDAAAKGmEGQAAUNIIMwAAoKQRZgAAQEkjzAAAgJJGmAEAACWNMAMAAEoaYQbAIWfRokUyxux1oCaA0kSYAQAAJY0wAwAAShphBkDB+b6vuro6DR06VOXl5Ro9erT++Mc/SmpbAnrqqac0atQolZWV6bOf/azWrFnT4Wf86U9/0gknnKBoNKohQ4bo9ttv7/B+LBbTtddeq9raWkWjUQ0bNky/+c1vOnxmxYoVGjNmjCoqKnTqqadq7dq1+f2LA8gLwgyAgqurq9MDDzyg++67T2+++aZmzZqlb3zjG3rxxRczn/nhD3+o22+/XcuWLdNhhx2myZMnK5FISEqFkClTpuiSSy7R6tWr9ZOf/ETz5s3T/fffn/n+b33rW3r44Yd111136e2339Yvf/lL9ezZs0Md119/vW6//XYtX75coVBIl112WUH+/gCCxanZAAoqFoupT58++sc//qFx48Zlrv/bv/2bmpqa9L3vfU8TJkzQI488oosvvliStH37dh1xxBG6//77NWXKFE2dOlUfffSRnn322cz3X3PNNXrqqaf05ptvat26dfrUpz6l5557TpMmTdqrhkWLFmnChAn6xz/+oYkTJ0qSnn76aZ177rlqbm5WWVlZnu8CgCDRmQFQUBs2bFBTU5POOuss9ezZM/N64IEH9M9//jPzufZBp0+fPvrUpz6lt99+W5L09ttva/z48R1+7vjx47V+/Xp5nqdVq1bJdV2dccYZ+61l1KhRmT8PHDhQkrRt27ac/44ACitU7AIAHFp2794tSXrqqad0+OGHd3gvGo12CDTZKi8vP6DPhcPhzJ+NMZJS8zwASgudGQAFdfzxxysajWrTpk0aNmxYh1dtbW3mc6+++mrmzzt27NC6det03HHHSZKOO+44vfLKKx1+7iuvvKJjjz1Wrutq5MiR8n2/wwwOgO6LzgyAgqqsrNR//Md/aNasWfJ9X6eddprq6+v1yiuvqKqqSkceeaQk6aabblLfvn1VU1Oj66+/Xv369dMFF1wgSbr66qt18skna/78+br44ou1ZMkS3X333frv//5vSdKQIUM0bdo0XXbZZbrrrrs0evRovffee9q2bZumTJlSrL86gDwhzAAouPnz5+uwww5TXV2d3n33XfXq1UsnnXSSrrvuuswyz6233qqrrrpK69ev16c//Wn97W9/UyQSkSSddNJJeuyxx3TDDTdo/vz5GjhwoG666SZ9+9vfzvyOe++9V9ddd52uuOIKffzxxxo8eLCuu+66Yvx1AeQZTzMB6FJanzTasWOHevXqVexyAJQAZmYAAEBJI8wAAICSxjITAAAoaXRmAABASSPMAACAkkaYAQAAJY0wAwAAShphBgAAlDTCDAAAKGmEGQAAUNIIMwAAoKT9/23/rtFeQg6nAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "import pandas as pd\n",
        "import seaborn as sns\n",
        "\n",
        "model = operation.result()\n",
        "\n",
        "snapshots = pd.DataFrame(model.tuning_task.snapshots)\n",
        "\n",
        "sns.lineplot(data=snapshots, x = 'epoch', y='mean_loss')\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rkoQTXb1vSBC"
      },
      "source": [
        "## Evaluate your model\n",
        "\n",
        "You can use the `genai.generate_text` method and specify the name of your model to test your model performance."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zO0YcuSyxydZ"
      },
      "outputs": [],
      "source": [
        "model = genai.GenerativeModel(model_name=f'tunedModels/{name}')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UwGrrj6hS_x2"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'56'"
            ]
          },
          "execution_count": 12,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "result = model.generate_content('55')\n",
        "result.text"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YSNB2zjTx5SZ"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'123456'"
            ]
          },
          "execution_count": 13,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "result = model.generate_content('123455')\n",
        "result.text"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Y2YVO-m0Ut9H"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'five'"
            ]
          },
          "execution_count": 14,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "result = model.generate_content('four')\n",
        "result.text"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "h2MkTR0uTb6U"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'cinq'"
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "result = model.generate_content('quatre') # French 4\n",
        "result.text                               # French 5 is \"cinq\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OruCW1zETsZw"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'IV'"
            ]
          },
          "execution_count": 16,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "result = model.generate_content('III')    # Roman numeral 3\n",
        "result.text                               # Roman numeral 4 is IV"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "thDdSuUDUJOx"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'八'"
            ]
          },
          "execution_count": 17,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "result = model.generate_content('七')  # Japanese 7\n",
        "result.text                            # Japanese 8 is 八!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HpIA1IFevQQR"
      },
      "source": [
        "It really seems to have picked up the task despite the limited examples, but \"next\" is a simple concept, see the [tuning guide](https://ai.google.dev/docs/model_tuning_guidance) for more guidance on improving performance."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nmuQCbTYwIOx"
      },
      "source": [
        "## Update the description\n",
        "\n",
        "You can update the description of your tuned model any time using the `genai.update_tuned_model` method."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9gAVuXT_wG3x"
      },
      "outputs": [],
      "source": [
        "genai.update_tuned_model(f'tunedModels/{name}', {\"description\":\"This is my model.\"});"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d-c3YerBxVYs"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'This is my model.'"
            ]
          },
          "execution_count": 20,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "model = genai.get_tuned_model(f'tunedModels/{name}')\n",
        "\n",
        "model.description"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i_TpwvBB4bQ7"
      },
      "source": [
        "## Delete the model\n",
        "\n",
        "You can clean up your tuned model list by deleting models you no longer need. Use the `genai.delete_tuned_model` method to delete a model. If you canceled any tuning jobs, you may want to delete those as their performance may be unpredictable."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cepfaUCvVGCo"
      },
      "outputs": [],
      "source": [
        "genai.delete_tuned_model(f'tunedModels/{name}')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ljEssIshYDEr"
      },
      "source": [
        "The model no longer exists:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kN_bkut_4ayL"
      },
      "outputs": [],
      "source": [
        "try:\n",
        "  m = genai.get_tuned_model(f'tunedModels/{name}')\n",
        "  print(m)\n",
        "except Exception as e:\n",
        "  print(f\"{type(e)}: {e}\")"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "Tuning.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}