{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "Tce3stUlHN0L" }, "source": [ "##### Copyright 2024 Google LLC." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cellView": "form", "id": "tuOe1ymfHZPu" }, "outputs": [], "source": [ "# @title Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ] }, { "cell_type": "markdown", "metadata": { "id": "yeadDkMiISin" }, "source": [ "# Gemini API: Tuning Quickstart with Python" ] }, { "cell_type": "markdown", "metadata": { "id": "lEXQ3OwKIa-O" }, "source": [ "\n", " \n", "
\n", " Run in Google Colab\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "Jp_CKyzxUqx6" }, "source": [ "In this notebook, you'll learn how to get started with model tuning." ] }, { "cell_type": "markdown", "metadata": { "id": "4x-2x8A_vi9g" }, "source": [ "## What is model tuning?\n", "\n", "Prompt design strategies such as few shot prompting may not always produce the results you need. Use model tuning to improve a model's performance on specific tasks or help the model adhere to specific output requirements when instructions aren't sufficient and you have a set of examples that demonstrate the outputs you want.\n", "\n", "The goal of model tuning is to further improve the performance of the model for your specific task. Model tuning works by providing the model with a training dataset containing many examples of the task. For niche tasks, you can get significant improvements in model performance by tuning the model on a modest number of examples.\n", "\n", "Your training data should be structured as examples with prompt inputs and expected response outputs. The goal is to teach the model to mimic the wanted behavior or task, by giving it many examples illustrating that behavior or task.\n", "\n", "You can also tune models using example data directly in Google AI Studio." ] }, { "cell_type": "markdown", "metadata": { "id": "SWxKvwd-MSIV" }, "source": [ "## OAuth Authentication" ] }, { "cell_type": "markdown", "metadata": { "id": "JjS8Zy1ojIgc" }, "source": [ "Unlike the other quickstarts which use API keys, model tuning uses OAuth.\n", "\n", "This tutorial assumes you have completed the [OAuth Quickstart](https://github.com/google-gemini/cookbook/blob/main/quickstarts/Authentication_with_OAuth.ipynb) and you have your client secret saved as `CLIENT_SECRET` in the Colab secrets manager.\n", "\n", "> Important: **Don't just click the link this command prints**. That will fail. Follow the instructions and copy the `gcloud` command it prints to your local machine and run it there, then paste the output from your local machine back\n", "here." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "9FUwyB_MJ0-2" }, "outputs": [], "source": [ "from google.colab import userdata\n", "import pathlib\n", "pathlib.Path('client_secret.json').write_text(userdata.get('CLIENT_SECRET'))\n", "\n", "# Use `--no-browser` in colab\n", "!gcloud auth application-default login --no-browser --client-id-file client_secret.json --scopes='https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/generative-language.tuning'" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "cbcf72bcb56d" }, "outputs": [], "source": [ "!pip install -q -U google-generativeai" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "8enrppafJPCX" }, "outputs": [], "source": [ "import google.generativeai as genai" ] }, { "cell_type": "markdown", "metadata": { "id": "P-MYZECwlRCq" }, "source": [ "You can check your existing tuned models with the `genai.list_tuned_model` method." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "XyWzoYFxU4r6" }, "outputs": [], "source": [ "for i, m in zip(range(5), genai.list_tuned_models()):\n", " print(m.name)" ] }, { "cell_type": "markdown", "metadata": { "id": "BhkXRzciv3Dp" }, "source": [ "## Create tuned model" ] }, { "cell_type": "markdown", "metadata": { "id": "OO8VZYAinLWc" }, "source": [ "To create a tuned model, you need to pass your dataset to the model in the `genai.create_tuned_model` method. You can do this be directly defining the input and output values in the call or importing from a file into a dataframe to pass to the method.\n", "\n", "For this example, you will tune a model to generate the next number in the sequence. For example, if the input is `1`, the model should output `2`. If the input is `one hundred`, the output should be `one hundred one`.\n", "\n", "**Note**: In general, you need between 100 and 500 examples to significantly change the behavior of the model." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "w-EBSe9wTbLB" }, "outputs": [ { "data": { "text/plain": [ "Model(name='models/gemini-1.0-pro-001',\n", " base_model_id='',\n", " version='001',\n", " display_name='Gemini 1.0 Pro 001 (Tuning)',\n", " description=('The best model for scaling across a wide range of tasks. This is a stable '\n", " 'model that supports tuning.'),\n", " input_token_limit=30720,\n", " output_token_limit=2048,\n", " supported_generation_methods=['generateContent', 'countTokens', 'createTunedModel'],\n", " temperature=0.9,\n", " top_p=1.0,\n", " top_k=1)" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "base_model = [\n", " m for m in genai.list_models()\n", " if \"createTunedModel\" in m.supported_generation_methods][0]\n", "base_model" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "baHjHh1oTTTC" }, "outputs": [], "source": [ "import random\n", "\n", "name = f'generate-num-{random.randint(0,10000)}'\n", "operation = genai.create_tuned_model(\n", " # You can use a tuned model here too. Set `source_model=\"tunedModels/...\"`\n", " source_model=base_model.name,\n", " training_data=[\n", " {\n", " 'text_input': '1',\n", " 'output': '2',\n", " },{\n", " 'text_input': '3',\n", " 'output': '4',\n", " },{\n", " 'text_input': '-3',\n", " 'output': '-2',\n", " },{\n", " 'text_input': 'twenty two',\n", " 'output': 'twenty three',\n", " },{\n", " 'text_input': 'two hundred',\n", " 'output': 'two hundred one',\n", " },{\n", " 'text_input': 'ninety nine',\n", " 'output': 'one hundred',\n", " },{\n", " 'text_input': '8',\n", " 'output': '9',\n", " },{\n", " 'text_input': '-98',\n", " 'output': '-97',\n", " },{\n", " 'text_input': '1,000',\n", " 'output': '1,001',\n", " },{\n", " 'text_input': '10,100,000',\n", " 'output': '10,100,001',\n", " },{\n", " 'text_input': 'thirteen',\n", " 'output': 'fourteen',\n", " },{\n", " 'text_input': 'eighty',\n", " 'output': 'eighty one',\n", " },{\n", " 'text_input': 'one',\n", " 'output': 'two',\n", " },{\n", " 'text_input': 'three',\n", " 'output': 'four',\n", " },{\n", " 'text_input': 'seven',\n", " 'output': 'eight',\n", " }\n", " ],\n", " id = name,\n", " epoch_count = 100,\n", " batch_size=4,\n", " learning_rate=0.001,\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "-As7ayWDK1w8" }, "source": [ "Your tuned model is immediately added to the list of tuned models, but its status is set to \"creating\" while the model is tuned." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "su64KgY4Uztj" }, "outputs": [ { "data": { "text/plain": [ "TunedModel(name='tunedModels/generate-num-5392',\n", " source_model='models/gemini-1.0-pro-001',\n", " base_model='models/gemini-1.0-pro-001',\n", " display_name='',\n", " description='',\n", " temperature=0.9,\n", " top_p=1.0,\n", " top_k=1,\n", " state=,\n", " create_time=datetime.datetime(2024, 3, 16, 0, 41, 42, 702621, tzinfo=datetime.timezone.utc),\n", " update_time=datetime.datetime(2024, 3, 16, 0, 41, 42, 702621, tzinfo=datetime.timezone.utc),\n", " tuning_task=TuningTask(start_time=datetime.datetime(2024, 3, 16, 0, 41, 43, 81144, tzinfo=datetime.timezone.utc),\n", " complete_time=None,\n", " snapshots=[],\n", " hyperparameters=Hyperparameters(epoch_count=100,\n", " batch_size=4,\n", " learning_rate=0.001)))" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model = genai.get_tuned_model(f'tunedModels/{name}')\n", "\n", "model" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "EUodUwZkKPi-" }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.state" ] }, { "cell_type": "markdown", "metadata": { "id": "Pi8X5vkQv-3_" }, "source": [ "### Check tuning progress" ] }, { "cell_type": "markdown", "metadata": { "id": "tWI-vAh4LJIz" }, "source": [ "Use `metadata` to check the state:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "g08vqtxYLMxT" }, "outputs": [], "source": [ "operation.metadata" ] }, { "cell_type": "markdown", "metadata": { "id": "3lQ6gSMgK-kz" }, "source": [ "Wait for the training to finish using `operation.result()`, or `operation.wait_bar()`" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "SOUowIv1HgSE" }, "outputs": [], "source": [ "import time\n", "\n", "for status in operation.wait_bar():\n", " time.sleep(30)" ] }, { "cell_type": "markdown", "metadata": { "id": "4cg868HzqOx5" }, "source": [ "You can cancel your tuning job any time using the `cancel()` method. Uncomment the line below and run the code cell to cancel your job before it finishes." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "oQuJ70_hqJi9" }, "outputs": [], "source": [ "# operation.cancel()" ] }, { "cell_type": "markdown", "metadata": { "id": "lqiL0TWDqAPn" }, "source": [ "Once the tuning is complete, you can view the loss curve from the tuning results. The [loss curve](https://generativeai.devsite.corp.google.com/guide/model_tuning_guidance#recommended_configurations) shows how much the model's predictions deviate from the ideal outputs." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "bIiG57xWLhP7" }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjMAAAGwCAYAAABcnuQpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA5kElEQVR4nO3de3RU9b3//9dn77kkgSTcJIAGQdGicqkWpYhWWWCttXhpK9rSlmpPe6xYBDxV0GKrVGM9R5e1erTt6ql11Wsv2lYtalFEPYhcxANeuFR+wlcFrEACucxl78/vj5lMEggIM3tmMuH5WGtWyZ5J8mavtrzW+/Pen4+x1loBAACUKKfYBQAAAOSCMAMAAEoaYQYAAJQ0wgwAAChphBkAAFDSCDMAAKCkEWYAAEBJCxW7gHzzfV8ffPCBKisrZYwpdjkAAOAAWGu1a9cuDRo0SI6z/95Ltw8zH3zwgWpra4tdBgAAyMLmzZt1xBFH7Pcz3T7MVFZWSkrdjKqqqiJXAwAADkRDQ4Nqa2sz/47vT7cPM61LS1VVVYQZAABKzIGMiDAADAAAShphBgAAlDTCDAAAKGmEGQAAUNIIMwAAoKQRZgAAQEkjzAAAgJJGmAEAACWNMAMAAEoaYQYAAJQ0wgwAAChphBkAAFDSCDMAAKCkEWZy4PlWCc8vdhkAABzSCDM52NrQovc+bix2GQAAHNIIMznwrdXuWLLYZQAAcEgjzOSoJeHL822xywAA4JBFmMlR0vMVTzI3AwBAsRBmcpQgzAAAUFSEmRwlPKuY5xW7DAAADlmEmRzRmQEAoLgIMznyfKuWBJ0ZAACKhTCTIyupMUaYAQCgWAgzAWiKe7KWx7MBACgGwkwAkr6vOMcaAABQFISZHLnGMAQMAEAREWZyFHIdJT1LmAEAoEgIMzkySg0Bs8wEAEBxFDXMLF68WJMnT9agQYNkjNETTzyReS+RSOjaa6/VyJEj1aNHDw0aNEjf+ta39MEHHxSv4P2gMwMAQHEUNcw0NjZq9OjRuueee/Z6r6mpSStXrtS8efO0cuVK/fnPf9batWt13nnnFaHS/Qs5Rk1xHs8GAKAYQsX85eecc47OOeecTt+rrq7Wc8891+Ha3XffrVNOOUWbNm3S4MGDC1HiAQm7jhpjyWKXAQDAIamoYeZg1dfXyxijXr167fMzsVhMsVgs83VDQ0Pe6wo7jmJJX0nPV8hlDAkAgEIqmX95W1padO211+prX/uaqqqq9vm5uro6VVdXZ161tbV5ry3kph/PZggYAICCK4kwk0gkNGXKFFlrde+99+73s3PnzlV9fX3mtXnz5rzXF+bxbAAAiqbLLzO1Bpn33ntPzz///H67MpIUjUYVjUYLVF2K6xh51tKZAQCgCLp0mGkNMuvXr9cLL7ygvn37FrukfbM8ng0AQDEUNczs3r1bGzZsyHy9ceNGrVq1Sn369NHAgQP11a9+VStXrtSTTz4pz/O0ZcsWSVKfPn0UiUSKVbYkyVqrpnhSzelHsh1j1JLg8WwAAArN2CIe97xo0SJNmDBhr+vTpk3TT37yEw0dOrTT73vhhRd05plnHtDvaGhoUHV1terr6z9xiepg3LVwve54bp1OH9ZP13xhuD7eHVPvHhGNru0V2O8AAOBQdTD/fhe1M3PmmWdqf1mqiDnrE1WVpW5d62Z5IddRU9yTtVbGmGKWBgDAIaUknmbqiqorwpKkpnhqs7xw+vHsGHMzAAAUFGEmS9XlqTDTmO7MhF1HCd9XgieaAAAoKMJMllrDTGtnJuQY9poBAKAICDNZagszqc5M65wMe80AAFBYhJksVaXDTHPck99uUJnODAAAhUWYyVJrZ8aq3RNNjuH0bAAACowwk6VoyFU0lLp9u2OtTzQ5mYFgAABQGISZHFSm95rZ3dIWZmJJX0nmZgAAKBjCTA4qy9KPZ8fa9ppJJn2GgAEAKCDCTA4ynZlY6+PZjpK+VcLrujsXAwDQ3RBmcrBnmHGM5Fsr3yfMAABQKISZHPSMdgwzxhhZa+V14TOlAADobggzOWidmWkdAJYkY0RnBgCAAiLM5GDPZSZJsjJKEmYAACgYwkwOOgszspJHmAEAoGAIMznILDO1DzPpIWAAAFAYhJkcdNaZcWSUYJ8ZAAAKhjCTg9Yw0/48JseIfWYAACggwkwOKqN7LzM5Dp0ZAAAKiTCTg6p2nRmbnpNxjVHSJ8wAAFAohJkctC4z+VZqTqROy3Yco0SSZSYAAAqFMJODaNhVyDGS2jbOc4zk+2ycBwBAoRBmclQRcSW1P5/JyBNHGgAAUCiEmRz1iO552KSR71s2zgMAoEAIMznaqzPjpE/OpjMDAEBBEGZyVBHppDNj6cwAAFAohJkcZTozLe2XmVJDwAAAIP8IMzkwxqhHtOMyk+sYeT4DwAAAFAphJgeOkcojnZycbTg5GwCAQiHM5MAxRj3Sy0ztz2cy4uRsAAAKhTCTA8eYvQaAJcmKAWAAAAqFMJMDx+z9aHYrwgwAAIVBmMlBapkp3ZlpaRdmrGGZCQCAAiHM5MBx9n6aqfV6wuPZbAAACoEwkwPHSD3K2mZmbLob4xgpniTMAABQCISZHDjGqCq9zORbqTnhSZJcY5TwWGYCAKAQCDM5cIxRNOzIdYykjkcasMwEAEBhEGZy4DiS4zjqmT45uzHmpa8bJXmaCQCAgiDM5MAxJjU3s+fJ2UbyffaaAQCgEAgzOUiFGaMeUU7OBgCgWAgzOXBM6rDJ1o3zGlvaHTZpLXvNAABQAISZHJj0MtOeRxo4xsj67AIMAEAhFDXMLF68WJMnT9agQYNkjNETTzzR4X1rrW644QYNHDhQ5eXlmjRpktavX1+cYvchHDKdzsx4svLozAAAkHdFDTONjY0aPXq07rnnnk7fv+2223TXXXfpvvvu09KlS9WjRw+dffbZamlpKXCl++YaR+V7hhkn1Znx6cwAAJB3oWL+8nPOOUfnnHNOp+9Za3XnnXfqRz/6kc4//3xJ0gMPPKCamho98cQTuuSSSwpZ6j65TufLTAwAAwBQGF12Zmbjxo3asmWLJk2alLlWXV2tsWPHasmSJfv8vlgspoaGhg6vfAq5zr5PzmaZCQCAvOuyYWbLli2SpJqamg7Xa2pqMu91pq6uTtXV1ZlXbW1tXusMOaZtmamlY5jx2QQYAIC867JhJltz585VfX195rV58+a8/r6Q46g83ElnxtCZAQCgELpsmBkwYIAkaevWrR2ub926NfNeZ6LRqKqqqjq88slpv89MuzDDADAAAIXRZcPM0KFDNWDAAC1cuDBzraGhQUuXLtW4ceOKWFlHzh4DwDbdjTFGSnLYJAAAeVfUp5l2796tDRs2ZL7euHGjVq1apT59+mjw4MGaOXOmfvrTn+qYY47R0KFDNW/ePA0aNEgXXHBB8Yreg2OMKqKpTJj0rWJJX2VhV47hsEkAAAqhqGFm+fLlmjBhQubr2bNnS5KmTZum+++/X9dcc40aGxv1ve99Tzt37tRpp52mBQsWqKysrFgl78UxRmUhN3WEgW+1O5bMhJkEnRkAAPKuqGHmzDPPzCzLdMYYo5tuukk33XRTAas6OI6RZIx6RkOqb05od0tS/XpG5Rgp4dGZAQAg37rszEypcBwjWbv3kQYOnRkAAAqBMJMjxxgZY9SzbO9dgBOe3W/nCQAA5I4wkyPHpIJLj2jHMOOmjzRgBhgAgPwizOTIcYxkpB57ns/kiPOZAAAoAMJMjhxj5BipR3SPmRlj5PtWPstMAADkFWEmR5llpnRnppGTswEAKCjCTI5SA8AddwFOXZc8KzbOAwAgzwgzOUotM5m2ZaaWtkezOZ8JAID8I8zkyDGpzf0qOltmkuXkbAAA8owwkyPHGLkyqoikbuXudidnS3RmAADIN8JMjhzHyDh7z8y0ojMDAEB+EWYCEHKMyvc4zqAVTzMBAJBfhJkAhBxHFeFUZybhWcWSXuY9n+OZAADIK8JMABxHioZSm+dJ7Z5oklGSNAMAQF4RZgIQcox8aa/zmRwjTs4GACDPCDMBCDlG1ko99wwzjiHMAACQZ4SZAIRcR9a2dWba7zXTbnwGAADkAWEmAK6T2iCvrTOTSjCOkRLMzAAAkFeEmQA4xkgdwkz7zoyVZa8ZAADyhjATAMcxkkwmzDS2m5nh5GwAAPKLMBOA1key9+7MpDbNYxdgAADyhzATAMcYGdlOHs1OdWYYmwEAIH8IMwFwjJHtZJnJbV1mojMDAEDeEGYCkFlmKuukM+NzPhMAAPlEmAmAY4wcGfVoPWyypW1mJrXMRJgBACBfCDMBcIyRMVLFHidnG2NkWWYCACCvCDMBcBzJOFJFJD0zE09m3jNGdGYAAMgjwkwAUstMbZ2ZloSvZPpMJiujJGEGAIC8IcwEwDFGjjEqC7uZa61LTbIMAAMAkE+EmQAYk5qPMUZtQ8CtYSY9BAwAAPKDMBMA1zFyjDqcnJ15PFtGCY9d8wAAyBfCTABSTzMZWat2G+e1OznbozMDAEC+EGYC4Ji2owv2PJ/JpK8DAID8IMwEwBiT2SBv7/OZGAAGACCfCDMBCbnpZaayTjozjMwAAJA3hJmAhBwn1ZlJb5zXeqSBkeRZ0gwAAPlCmAmI60hWbZ2Z1l2AjZFYZQIAIH8IMwFxHUe+324AuKVtmYmZGQAA8ocwE5Cwa+Sr/aPZbctM1kqWJ5oAAMgLwkxAQo4j28mj2U765GyaMwAA5AdhJiBOetO8HtFOjjMQnRkAAPKFMBMQY1L/uecyk5NeZ6IzAwBAfnTpMON5nubNm6ehQ4eqvLxcRx99tObPn98luxyuYyTTtszUGPfk+VZGRr6VrLpezQAAdAehYhewPz/72c9077336ne/+51OOOEELV++XJdeeqmqq6s1Y8aMYpfXgWOMJJPZAViSmuJJhd3ULE0XzF8AAHQLXTrM/O///q/OP/98nXvuuZKkIUOG6OGHH9Zrr71W5Mr25qSXmcKuo2jIUSzpa3csqT49IrIS5zMBAJAnXXqZ6dRTT9XChQu1bt06SdIbb7yhl19+Weecc84+vycWi6mhoaHDqxAcx6i1/dJ+rxkjQ2cGAIA86tKdmTlz5qihoUHDhw+X67ryPE8333yzpk6dus/vqaur04033ljAKlNal5mkVJj5uDGuxrgnY0RnBgCAPOrSnZnHHntMDz74oB566CGtXLlSv/vd7/Rf//Vf+t3vfrfP75k7d67q6+szr82bNxekVsekhoB9azscNumY1AAwTzMBAJAfXboz88Mf/lBz5szRJZdcIkkaOXKk3nvvPdXV1WnatGmdfk80GlU0Gi1kmZJSxxbIqNMjDWTTLwAAELgu3ZlpamqS43Qs0XVd+X7XO4XadYyc9JJS5uTsWFKtozQsMwEAkB9dujMzefJk3XzzzRo8eLBOOOEEvf7667rjjjt02WWXFbu0vThGcrT3MpMxqWtEGQAA8qNLh5lf/OIXmjdvnq644gpt27ZNgwYN0r//+7/rhhtuKHZpe3GMkXFSXZg9dwGWLJ0ZAADypEuHmcrKSt1555268847i13KJzJG6WFfm9k4r/V8ptbuDAAACF6XnpkpJa4x6dCivU7OliwDwAAA5AlhJiCOMXKUOh27554nZ6fPZwIAAMEjzATEcYwcJxVaeuwxM8OmeQAA5A9hJkCZTfP2WmZilQkAgHwJJMw0NDToiSee0Ntvvx3EjytZIcfs9TSTtVZGqc30AABA8LIKM1OmTNHdd98tSWpubtaYMWM0ZcoUjRo1Sn/6058CLbCUuE7qUMnWfWZ8KzUnPA6ZBAAgj7IKM4sXL9bpp58uSXr88cdlrdXOnTt111136ac//WmgBZaSUHpmJuI6CjmpQyd3tyRlDDMzAADkS1Zhpr6+Xn369JEkLViwQF/5yldUUVGhc889V+vXrw+0wFLSOjNjjOm4C7CMPJaZAADIi6zCTG1trZYsWaLGxkYtWLBAn//85yVJO3bsUFlZWaAFlpKQ42SWlDrsAkxnBgCAvMlqB+CZM2dq6tSp6tmzp4488kideeaZklLLTyNHjgyyvpISctt2+m3/RFONRGcGAIA8ySrMXHHFFTrllFO0efNmnXXWWZmTrY866qhDembGMUYyqdDS4UgDQ5gBACBfsj6bacyYMRozZowkyfM8rV69Wqeeeqp69+4dWHGlxnGMpNTgb/vOjGOYmQEAIF+ympmZOXOmfvOb30hKBZkzzjhDJ510kmpra7Vo0aIg6ysp6QeYJLUPM15qnxmyDAAAeZFVmPnjH/+o0aNHS5L+9re/aePGjXrnnXc0a9YsXX/99YEWWEocY9SaZ9oPAHNqNgAA+ZNVmPnXv/6lAQMGSJKefvppXXTRRTr22GN12WWXafXq1YEWWEqMaTu2oP0yk2FmBgCAvMkqzNTU1Oitt96S53lasGCBzjrrLElSU1OTXNcNtMBS4pq2daYe7U7OZmYGAID8yWoA+NJLL9WUKVM0cOBAGWM0adIkSdLSpUs1fPjwQAssJa3LTLb9YZMtqcMmyTIAAORHVmHmJz/5iUaMGKHNmzfroosuUjQalSS5rqs5c+YEWmApcYyRY1JHGnR8mkmSTR026bSfEgYAADnL+tHsr371q3tdmzZtWk7FlDrjpF5+u8MmG+Op4wx8WdGcAQAgeFnNzEjSiy++qMmTJ2vYsGEaNmyYzjvvPL300ktB1lZyXGPkSLK23aZ5LUlJVtZanmgCACAPsgozv//97zVp0iRVVFRoxowZmjFjhsrLyzVx4kQ99NBDQddYMhxjMo9hty4zJX2rhO/LivOZAADIh6yWmW6++WbddtttmjVrVubajBkzdMcdd2j+/Pn6+te/HliBpcSYVKCxVioPu3JMavC3KeapPOqKLAMAQPCy6sy8++67mjx58l7XzzvvPG3cuDHnokqV65h0gLEyxmSWmhrjnmRFmAEAIA+yCjO1tbVauHDhXtf/8Y9/qLa2NueiSpWbeZqp48nZTfGkrBUjwAAA5EFWy0xXX321ZsyYoVWrVunUU0+VJL3yyiu6//779fOf/zzQAkuJ4xiFXKNYwpfUPsx48mXZawYAgDzIKsx8//vf14ABA3T77bfrsccekyQdd9xxevTRR3X++ecHWmCpiYZcNcU8SW1PNDXGkgwAAwCQJ1nvM3PhhRfqwgsvDLKWbiHsGnl7LDM1xjxZn5kZAADyIet9ZtC5SMiRn1plagszrTMzpBkAAAJ3wJ2Z3r17y5gD24p/+/btWRdU6iKuu9cAcGqZiZkZAADy4YDDzJ133pnHMroP1zWSSaWWHu2XmURnBgCAfDjgMJPNuUu33nqrLr/8cvXq1eugv7dUhRyj1NnZUkXElSQ1JzwZcXI2AAD5kNeZmVtuueWQW3JyHZPpwJSFU7e3JZHuzLDPDAAAgctrmDkUl1VCjpFjHPnWKhpKdWZakr6MeJoJAIB84GmmgLmOkeNKnm8VTXdmYsnUvjPsMwMAQPAIMwELOY5cY+T5VmXpzkws4cvK0pkBACAPCDMBcx0j10mdz1QWTi8zJTwZGTozAADkAWEmYK5jMp2Z1mWmlqRHZwYAgDzJa5g5/fTTVV5ens9f0eW0dmb2XGaSDGEGAIA8yPpsJt/3tWHDBm3btk1+6/79aZ/73OckSU8//XRu1ZWoaMhVS8LPPJqd9K0837LMBABAHmQVZl599VV9/etf13vvvbfX49fGGHmeF0hxpSoSSnVmKsvabm8i6ctj1zwAAAKXVZi5/PLLNWbMGD311FMaOHDgAZ/ZdKiIhBx51qb3nEnt/Bv3fDozAADkQVZhZv369frjH/+oYcOGBV1Pt9B62KQxRmVhV01xT/GkrySdGQAAApfVAPDYsWO1YcOGoGvp1Pvvv69vfOMb6tu3r8rLyzVy5EgtX768IL87W67b1qmKhlK3OOExMwMAQD5k1Zn5wQ9+oKuvvlpbtmzRyJEjFQ6HO7w/atSoQIrbsWOHxo8frwkTJujvf/+7DjvsMK1fv169e/cO5Ofni2taj5pUeq+ZhBJJj5kZAADyIKsw85WvfEWSdNlll2WuGZM6YDHIAeCf/exnqq2t1W9/+9vMtaFDhwbys/Op/WGTrZ2ZWNLn0WwAAPIgqzCzcePGoOvo1F//+ledffbZuuiii/Tiiy/q8MMP1xVXXKHvfve7+/yeWCymWCyW+bqhoaEQpXbQ/rDJ1l2A456v5B6PsAMAgNxlFWaOPPLIoOvo1Lvvvqt7771Xs2fP1nXXXadly5ZpxowZikQimjZtWqffU1dXpxtvvLEg9e2L66YOm/T9dmEm6YtVJgAAgpf1pnmS9NZbb2nTpk2Kx+Mdrp933nk5FdXK932NGTNGt9xyiyTpxBNP1Jo1a3TfffftM8zMnTtXs2fPznzd0NCg2traQOo5UKHWIw2sbVtm8thnBgCAfMgqzLz77ru68MILtXr16sysjKTMfjNBzcwMHDhQxx9/fIdrxx13nP70pz/t83ui0aii0Wggvz9brmPktJ6c3dqZSaRmZlrnigAAQDCyejT7qquu0tChQ7Vt2zZVVFTozTff1OLFizVmzBgtWrQosOLGjx+vtWvXdri2bt26gi1zZSvkOKmTs32prF1nxloOmwQAIGhZhZklS5bopptuUr9+/eQ4jhzH0Wmnnaa6ujrNmDEjsOJmzZqlV199Vbfccos2bNighx56SL/61a80ffr0wH5HPmQOm7RW0fYzMxJ7zQAAELCswozneaqsrJQk9evXTx988IGk1GDwnp2UXJx88sl6/PHH9fDDD2vEiBGaP3++7rzzTk2dOjWw35Evrecztc7MxJO+ZC1DwAAABCyrmZkRI0bojTfe0NChQzV27FjddtttikQi+tWvfqWjjjoq0AK/9KUv6Utf+lKgP7MQIq4jz09mZmZiSU++laxIMwAABCmrMPOjH/1IjY2NkqSbbrpJX/rSl3T66aerb9++evTRRwMtsFRFQ648a1UWbr9pHjMzAAAELaswc/bZZ2f+PGzYML3zzjvavn27evfuzZM6aZFQatO8aKi1M+PLipkZAACCltXMTKsNGzbomWeeUXNzs/r06RNUTd2C6xjJqm2ZKeHRmQEAIA+yCjMff/yxJk6cqGOPPVZf/OIX9eGHH0qSvvOd7+jqq68OtMBSFXIcyajjMpPozAAAELSswsysWbMUDoe1adMmVVRUZK5ffPHFWrBgQWDFlTLXSS23laWXmVoSrQPAAAAgSFnNzDz77LN65plndMQRR3S4fswxx+i9994LpLBSF3KMjFKzM1KqMyMrWc6aBAAgUFl1ZhobGzt0ZFpt37696EcJdBWua+Q4RhE31aFJPc3EMhMAAEHLKsycfvrpeuCBBzJfG2Pk+75uu+02TZgwIbDiSplrjELGKJzuzKSWmdhlBgCAoGW1zHTbbbdp4sSJWr58ueLxuK655hq9+eab2r59u1555ZWgayxJrpPqzISc9p0Zn84MAAABy6ozM2LECK1du1annXaazj//fDU2NurLX/6yXn/9dR199NFB11iSQunzmcKOm7mW8FlmAgAgaFl1ZiSprKxMZ511lkaPHi3fT021Llu2TJJ03nnnBVNdCWs9bDLktm0iGE8meZwJAICAZRVmFixYoG9+85vavn277B6dBmOMPM8LpLhSZoxRJGQUT6TOaYp7vmJJDpoEACBoWS0z/eAHP9CUKVP0wQcfyPf9Di+CTJuw48izVtEOG+eRZgAACFJWYWbr1q2aPXu2ampqgq6nW4mGHHm+bXdytk9nBgCAgGUVZr761a9q0aJFAZfS/UTDbirMpB/Pjid8+aQZAAACldXMzN13362LLrpIL730kkaOHKlwONzh/RkzZgRSXKlz0ieIRzOdGZbgAAAIWlZh5uGHH9azzz6rsrIyLVq0SMa0PbFjjCHMpIVckzpssrUz47HPDAAAQcsqzFx//fW68cYbNWfOHDlOVitVh4TMYZPpzkw8yTITAABByyqJxONxXXzxxQSZTxBynA6HTcY9K4/ODAAAgcoqjUybNk2PPvpo0LV0O65j5BijaKi1M+PJozMDAECgslpm8jxPt912m5555hmNGjVqrwHgO+64I5DiSl3qSIPUI9pSapnJ84tcFAAA3UxWYWb16tU68cQTJUlr1qzp8F77YeBDXethkxG3/QAwaQYAgCBlFWZeeOGFoOvollKdGUfhdGcmlqAzAwBA0JjgzSPXMXKNaTcAzKPZAAAEjTCTR8YYhV2jSPrk7FjSZwAYAICAEWbyLBJyFG6dmUl6dGYAAAgYYSbPou3CTCxpmZkBACBghJk8i7iOwullpnjSk6zYBRgAgAARZvIs5DoKh9pmZnxZEWUAAAgOYSbPXKdtB+BY0peVmJsBACBAhJk8CzkmswNwLOHJWiuyDAAAwSHM5Fn7HYBbkql9ZujMAAAQHMJMnrnGqCycWmbyfKuk59OZAQAgQISZPHOctjAjSS1xRoABAAgSYSbPXMcoEjJy0udvtiQ98WQ2AADBIczkmWuMQo6jaLo7E/N8WdaZAAAIDGEmzxwnvdSUfqKpJe7TmQEAIECEmTxzjZHT7uTsWNKjMwMAQIAIM3nmOqkw07pxXkuCp5kAAAgSYSbPjDEKux07M+wzAwBAcAgzBRB2HUXdtl2AmZkBACA4hJkCCLXrzMQ9yz4zAAAEqKTCzK233ipjjGbOnFnsUg5KxHXawkySmRkAAIJUMmFm2bJl+uUvf6lRo0YVu5SDFnadzPlMMcIMAACBKokws3v3bk2dOlW//vWv1bt372KXc9BSuwC3dmYYAAYAIEglEWamT5+uc889V5MmTfrEz8ZiMTU0NHR4FZvTIcz4hBkAAAIUKnYBn+SRRx7RypUrtWzZsgP6fF1dnW688cY8V3VwQu12AGaZCQCAYHXpzszmzZt11VVX6cEHH1RZWdkBfc/cuXNVX1+feW3evDnPVX4yxxiFCTMAAORFl+7MrFixQtu2bdNJJ52UueZ5nhYvXqy7775bsVhMrut2+J5oNKpoNFroUvfLdYyi6TpjLDMBABCoLh1mJk6cqNWrV3e4dumll2r48OG69tpr9woyXZVrjMoibTMzHrvmAQAQmC4dZiorKzVixIgO13r06KG+ffvudb0rcxypLJwKXnGPzgwAAEHq0jMz3YXrGJWHW5eZPHmEGQAAAtOlOzOdWbRoUbFLOGiOMW2dGZaZAAAIFJ2ZAnAdo/JI2wAwYQYAgOAQZgrANUblobbODKtMAAAEhzBTAI5jVBFte5op6XtFrggAgO6DMFMgPcvCkiQrqTnuF7cYAAC6EcJMgVSE2/bEaYrTmQEAICiEmQIpj7gKu0aS1JLwZBmcAQAgEISZAgm7jqLpIeBUmClyQQAAdBOEmQIJOY4i6cMmmxPsAgwAQFAIMwXiOFK09eTshCeiDAAAwSDMFEjHzoxHZwYAgIAQZgqkQ2eGjfMAAAgMYaZAXGMynZlYkgFgAACCQpgpENcx7Z5mYgAYAICgEGYKxHGMysJtnRnCDAAAwSDMFIhrOnZmiDIAAASDMFMgbvvOTMKX5xFnAAAIAmGmQFzHqCzdmYklPSV9wgwAAEEgzBSIa4zKwq1hxlfS5+RsAACCQJgpEMcxKou0DQAnWWYCACAQhJkC6hEJSZISnlXCozMDAEAQCDMF1COSWmaKJ321JLwiVwMAQPdAmCmgHtFUZyae9BVL0pkBACAIhJkC6pkOMzGPMAMAQFAIMwXUvjOT9K2SzM0AAJAzwkwBVaRnZmIJT57vs9cMAAABIMwUUM+y9DJT0pfnW8IMAAABIMwUUOuj2UnfppaaWGYCACBnhJkCap2ZkaTmeJLODAAAASDMFFBZyJFjUn9OeJZdgAEACABhpoBc11E0fdhkS8JnF2AAAAJAmCkg1zGZIeDdsSQzMwAABIAwU0CuMepVHpYk7WpJqIWN8wAAyBlhpoBcx6hXRSrMNDQnFSfMAACQM8JMAbmOUZ8eEUlSfUuCwyYBAAgAYaaAHCP1rkiFmYbmhJK+lcfj2QAA5IQwU0DGGPXrmQozO5sS8nzLE00AAOSIMFNgh1WWSZJ2Nifk+T6dGQAAckSYKbDDKlOdmR1NcXm+2DgPAIAcEWYKrCbdmdnVklRL0lPSZ5kJAIBcEGYKrE+PiELpMw12pYeAAQBA9ggzBRZyHVWnN87b2ZxgABgAgBwRZgrMdYyq0mGmvjnBADAAADnq8mGmrq5OJ598siorK9W/f39dcMEFWrt2bbHLytqeuwCzcR4AALnp8mHmxRdf1PTp0/Xqq6/queeeUyKR0Oc//3k1NjYWu7SsOMZklpl2tSQU40gDAAByEip2AZ9kwYIFHb6+//771b9/f61YsUKf+9znilRV9lzHqLqsbWaGMAMAQG66fJjZU319vSSpT58+nb4fi8UUi8UyXzc0NBSkrgPlGqPePdqONEgkffm+lZN+wgkAABycLr/M1J7v+5o5c6bGjx+vESNGdPqZuro6VVdXZ161tbUFrnL/HEfqXdHWmfGsVYK9ZgAAyFpJhZnp06drzZo1euSRR/b5mblz56q+vj7z2rx5cwEr/GSu09aZ2dEYl+dx2CQAALkomWWmK6+8Uk8++aQWL16sI444Yp+fi0ajikajBazs4LiOUZ/Wk7NbkmrxPCU40gAAgKx1+c6MtVZXXnmlHn/8cT3//PMaOnRosUvKiWuMqivCck1qRmbnbvaaAQAgF12+MzN9+nQ99NBD+stf/qLKykpt2bJFklRdXa3y8vIiV3fwQq6jsrCr6oqwtjfGVd+SUJJdgAEAyFqX78zce++9qq+v15lnnqmBAwdmXo8++mixS8taedhp93h2XAk6MwAAZK3Ld2as7X7/0PeIhFRVnrr19c1JeczMAACQtS7fmemOIullJim110wsyZEGAABkizBTBBG3bZmJMAMAQG4IM0UQcZ3MXjP1zQm1JBgABgAgW4SZIoiEHPVtF2YSSdstZ4MAACgEwkwRhF2jvj1TG/vtbErIsz4b5wEAkCXCTBGEXEc1VakwU9+SOjmbjfMAAMgOYaZIBlSVyRjJWmlHc5zDJgEAyBJhpkiqysOqSj/RtGN3nL1mAADIEmGmSMKuo+ryVJipb07QmQEAIEuEmSKJhBz1SoeZHU0JJenMAACQFcJMkYRdR70q2jozDAADAJAdwkyRREOO+qT3mmloTrILMAAAWSLMFEnYbds4r6Eloe2NcTXGkkWuCgCA0kOYKRLXMTosvdfM7lhS2xvjevvDBgINAAAHiTBTRAOryyVJOxrjGlhdro93x/XWhw3aTaABAOCAEWaK6IheFZKkHU1xWSsNqC7T9sa43vqgXrtaEkWuDgCA0kCYKaL+1REZSb5Nzc04xmhAVZl2NiX01gcNakkwFAwAwCchzBRRRTikyrKQpNRSk6RMoKlvTrDcBADAASDMFFEk5Kg6vdfM9nSYkSRjjHxr6cwAAHAACDNFFHYd9a5IPZ69vSne4T3XcdQcJ8wAAPBJCDNFFHEd9U2HmR2N8b3e2xVjCBgAgE9CmCkixzHqV5naa+bjTsJMc9xX0uMASgAA9ocwU2RH9E7tNbN+6+4O18Mho0TSVyxJmAEAYH8IM0U2YXh/hRyjDR/t1vqtuzLXI66juOczBAwAwCcgzBRZTVWZThrcW5L09zVbMteNMbISnRkAAD4BYabIwq6jMz51mCTpxfUfaXdL294yjhGdGQAAPgFhpsgiIUfDB/TU4D4Viid9Pb92W+a9sONoVwsb5wEAsD+EmSILu0aRkKtJx/WXJC1Y86Gsten3HDXFPPm+LWaJAAB0aYSZIou4jkKOo88e1VdlYUebdzRrzfv1qfdCjuKex9wMAAD7QZgpMmOMKiKuIq6jM49NdWeeTg8Ch11H8STHGgAAsD+EmS6gR9RVwrf64sgBkqQl736sHY1xuY6RlaUzAwDAfhBmuoBoyJXnWw3t11PDB1TK862efXurJKUfz6YzAwDAvhBmuoCe0ZBCjlHS83XOiIGSpGfe3CLPt+knmjijCQCAfSHMdAFV5WH1LAtpVyyp04b1U2U0pI92xfT6ph2KhBztjnmZJ5wAAEBHhJkuwHWMBlaXqSnuKRJyNGF4ahD4ube3KuI6iiV5ogkAgH0hzHQRvSoiioSMYklPZx1XI0l6beN2NSc8JZJWsQRhBgCAzhBmuoiqspCqy8Pa3ZLUkH49dEz/nkr6VovXfSTPWoaAAQDYB8JMF2GM0YDqcrWkl5POOj7VnXn27a2y1meZCQCAfSDMdCG9ysMqCztqjnv63DGHKRJytHl7kzZ/3KzdMc5oAgCgM4SZLqRHNKQ+PSLaFUuoRzSk8Uf3lSS98s+PCTMAAOwDYaaL6V9ZpoRnZa3VWcendgReunG7djbFmZsBAKAThJkupldFWD2irhrjnkYMqtLA6jI1Jzz97z8/Zm4GAIBOlESYueeeezRkyBCVlZVp7Nixeu2114pdUt6UhV0d1jOqxlhSxhhNSj+m/dK6j3g8GwCATnT5MPPoo49q9uzZ+vGPf6yVK1dq9OjROvvss7Vt27Zil5Y3fXtG5Vkrz7eaOLy/HCNt+KhR67ftKnZpAAB0OcZ28X3yx44dq5NPPll33323JMn3fdXW1uoHP/iB5syZ84nf39DQoOrqatXX16uqqirf5QYi4fl6Y/NOfdwYU9hxdfcL67Vy004dVhnVyMOrdWxNpT41oKcO71WhsGsUdh2FXUch18gxRkaSMZKRkTHB1RXkz2p1oP/ty8fvBgAEozIaVnVFONCfeTD/focC/c0Bi8fjWrFihebOnZu55jiOJk2apCVLlnT6PbFYTLFYLPN1Q0ND3usMWth1NOLwau1oimtrfYsmDD9Mr2/aqY92xfT8O9v0/DvdtysFACg9//65ozT3i8cV7fd36TDzr3/9S57nqaampsP1mpoavfPOO51+T11dnW688cZClJdXZWFXA6vLNaCqTEMP66lxR/XV6v/XoP/v40a9v7NZ/29Hs+qbE/L81HJU0veV9K2sTXU7rFItD2s772q074h06dYc8sdKKlTH60B+V1er50B/zoE4kL97toL62fmqMZ9/96CUQo1diZGcPe5ZyC1u+7xLh5lszJ07V7Nnz8583dDQoNra2iJWlBtjjKrLwxozpK/GDOnb4T3fT0UWa1P/6du2MNPKHsD/CvcVePb8DAAAYddRJNS1Rm67dJjp16+fXNfV1q1bO1zfunWrBgwY0On3RKNRRaPRQpRXdE4mGjNQAgA4dHWtaLWHSCSiz3zmM1q4cGHmmu/7WrhwocaNG1fEygAAQFfRpTszkjR79mxNmzZNY8aM0SmnnKI777xTjY2NuvTSS4tdGgAA6AK6fJi5+OKL9dFHH+mGG27Qli1b9OlPf1oLFizYaygYAAAcmrr8PjO5KsV9ZgAAONQdzL/fXXpmBgAA4JMQZgAAQEkjzAAAgJJGmAEAACWNMAMAAEoaYQYAAJQ0wgwAAChphBkAAFDSCDMAAKCkdfnjDHLVusFxQ0NDkSsBAAAHqvXf7QM5qKDbh5ldu3ZJkmpra4tcCQAAOFi7du1SdXX1fj/T7c9m8n1fH3zwgSorK2WMCfRnNzQ0qLa2Vps3b+bcpzziPhcG97kwuM+FwX0ujHzeZ2utdu3apUGDBslx9j8V0+07M47j6Igjjsjr76iqquJ/LAXAfS4M7nNhcJ8Lg/tcGPm6z5/UkWnFADAAAChphBkAAFDSCDM5iEaj+vGPf6xoNFrsUro17nNhcJ8Lg/tcGNznwugq97nbDwADAIDujc4MAAAoaYQZAABQ0ggzAACgpBFmAABASSPMZOmee+7RkCFDVFZWprFjx+q1114rdkklra6uTieffLIqKyvVv39/XXDBBVq7dm2Hz7S0tGj69Onq27evevbsqa985SvaunVrkSruHm699VYZYzRz5szMNe5zMN5//3194xvfUN++fVVeXq6RI0dq+fLlmfettbrhhhs0cOBAlZeXa9KkSVq/fn0RKy5Nnudp3rx5Gjp0qMrLy3X00Udr/vz5Hc7z4V4fvMWLF2vy5MkaNGiQjDF64oknOrx/IPd0+/btmjp1qqqqqtSrVy995zvf0e7du/NTsMVBe+SRR2wkErH/8z//Y99880373e9+1/bq1ctu3bq12KWVrLPPPtv+9re/tWvWrLGrVq2yX/ziF+3gwYPt7t27M5+5/PLLbW1trV24cKFdvny5/exnP2tPPfXUIlZd2l577TU7ZMgQO2rUKHvVVVdlrnOfc7d9+3Z75JFH2m9/+9t26dKl9t1337XPPPOM3bBhQ+Yzt956q62urrZPPPGEfeONN+x5551nhw4dapubm4tYeem5+eabbd++fe2TTz5pN27caP/whz/Ynj172p///OeZz3CvD97TTz9tr7/+evvnP//ZSrKPP/54h/cP5J5+4QtfsKNHj7avvvqqfemll+ywYcPs1772tbzUS5jJwimnnGKnT5+e+drzPDto0CBbV1dXxKq6l23btllJ9sUXX7TWWrtz504bDoftH/7wh8xn3n77bSvJLlmypFhllqxdu3bZY445xj733HP2jDPOyIQZ7nMwrr32Wnvaaaft833f9+2AAQPsf/7nf2au7dy500ajUfvwww8XosRu49xzz7WXXXZZh2tf/vKX7dSpU6213Osg7BlmDuSevvXWW1aSXbZsWeYzf//7360xxr7//vuB18gy00GKx+NasWKFJk2alLnmOI4mTZqkJUuWFLGy7qW+vl6S1KdPH0nSihUrlEgkOtz34cOHa/Dgwdz3LEyfPl3nnntuh/spcZ+D8te//lVjxozRRRddpP79++vEE0/Ur3/968z7Gzdu1JYtWzrc5+rqao0dO5b7fJBOPfVULVy4UOvWrZMkvfHGG3r55Zd1zjnnSOJe58OB3NMlS5aoV69eGjNmTOYzkyZNkuM4Wrp0aeA1dfuDJoP2r3/9S57nqaampsP1mpoavfPOO0WqqnvxfV8zZ87U+PHjNWLECEnSli1bFIlE1KtXrw6framp0ZYtW4pQZel65JFHtHLlSi1btmyv97jPwXj33Xd17733avbs2bruuuu0bNkyzZgxQ5FIRNOmTcvcy87+f4T7fHDmzJmjhoYGDR8+XK7ryvM83XzzzZo6daokca/z4EDu6ZYtW9S/f/8O74dCIfXp0ycv950wgy5n+vTpWrNmjV5++eVil9LtbN68WVdddZWee+45lZWVFbucbsv3fY0ZM0a33HKLJOnEE0/UmjVrdN9992natGlFrq57eeyxx/Tggw/qoYce0gknnKBVq1Zp5syZGjRoEPf6EMIy00Hq16+fXNfd6+mOrVu3asCAAUWqqvu48sor9eSTT+qFF17QEUcckbk+YMAAxeNx7dy5s8Pnue8HZ8WKFdq2bZtOOukkhUIhhUIhvfjii7rrrrsUCoVUU1PDfQ7AwIEDdfzxx3e4dtxxx2nTpk2SlLmX/P9I7n74wx9qzpw5uuSSSzRy5Eh985vf1KxZs1RXVyeJe50PB3JPBwwYoG3btnV4P5lMavv27Xm574SZgxSJRPSZz3xGCxcuzFzzfV8LFy7UuHHjilhZabPW6sorr9Tjjz+u559/XkOHDu3w/mc+8xmFw+EO933t2rXatGkT9/0gTJw4UatXr9aqVasyrzFjxmjq1KmZP3Ofczd+/Pi9thZYt26djjzySEnS0KFDNWDAgA73uaGhQUuXLuU+H6SmpiY5Tsd/ylzXle/7krjX+XAg93TcuHHauXOnVqxYkfnM888/L9/3NXbs2OCLCnyk+BDwyCOP2Gg0au+//3771ltv2e9973u2V69edsuWLcUurWR9//vft9XV1XbRokX2ww8/zLyampoyn7n88svt4MGD7fPPP2+XL19ux40bZ8eNG1fEqruH9k8zWct9DsJrr71mQ6GQvfnmm+369evtgw8+aCsqKuzvf//7zGduvfVW26tXL/uXv/zF/t///Z89//zzeVw4C9OmTbOHH3545tHsP//5z7Zfv372mmuuyXyGe33wdu3aZV9//XX7+uuvW0n2jjvusK+//rp97733rLUHdk+/8IUv2BNPPNEuXbrUvvzyy/aYY47h0eyu5he/+IUdPHiwjUQi9pRTTrGvvvpqsUsqaZI6ff32t7/NfKa5udleccUVtnfv3raiosJeeOGF9sMPPyxe0d3EnmGG+xyMv/3tb3bEiBE2Go3a4cOH21/96lcd3vd9386bN8/W1NTYaDRqJ06caNeuXVukaktXQ0ODveqqq+zgwYNtWVmZPeqoo+z1119vY7FY5jPc64P3wgsvdPr/ydOmTbPWHtg9/fjjj+3XvvY127NnT1tVVWUvvfRSu2vXrrzUa6xtt00iAABAiWFmBgAAlDTCDAAAKGmEGQAAUNIIMwAAoKQRZgAAQEkjzAAAgJJGmAEAACWNMAMAAEoaYQbAIWfRokUyxux1oCaA0kSYAQAAJY0wAwAAShphBkDB+b6vuro6DR06VOXl5Ro9erT++Mc/SmpbAnrqqac0atQolZWV6bOf/azWrFnT4Wf86U9/0gknnKBoNKohQ4bo9ttv7/B+LBbTtddeq9raWkWjUQ0bNky/+c1vOnxmxYoVGjNmjCoqKnTqqadq7dq1+f2LA8gLwgyAgqurq9MDDzyg++67T2+++aZmzZqlb3zjG3rxxRczn/nhD3+o22+/XcuWLdNhhx2myZMnK5FISEqFkClTpuiSSy7R6tWr9ZOf/ETz5s3T/fffn/n+b33rW3r44Yd111136e2339Yvf/lL9ezZs0Md119/vW6//XYtX75coVBIl112WUH+/gCCxanZAAoqFoupT58++sc//qFx48Zlrv/bv/2bmpqa9L3vfU8TJkzQI488oosvvliStH37dh1xxBG6//77NWXKFE2dOlUfffSRnn322cz3X3PNNXrqqaf05ptvat26dfrUpz6l5557TpMmTdqrhkWLFmnChAn6xz/+oYkTJ0qSnn76aZ177rlqbm5WWVlZnu8CgCDRmQFQUBs2bFBTU5POOuss9ezZM/N64IEH9M9//jPzufZBp0+fPvrUpz6lt99+W5L09ttva/z48R1+7vjx47V+/Xp5nqdVq1bJdV2dccYZ+61l1KhRmT8PHDhQkrRt27ac/44ACitU7AIAHFp2794tSXrqqad0+OGHd3gvGo12CDTZKi8vP6DPhcPhzJ+NMZJS8zwASgudGQAFdfzxxysajWrTpk0aNmxYh1dtbW3mc6+++mrmzzt27NC6det03HHHSZKOO+44vfLKKx1+7iuvvKJjjz1Wrutq5MiR8n2/wwwOgO6LzgyAgqqsrNR//Md/aNasWfJ9X6eddprq6+v1yiuvqKqqSkceeaQk6aabblLfvn1VU1Oj66+/Xv369dMFF1wgSbr66qt18skna/78+br44ou1ZMkS3X333frv//5vSdKQIUM0bdo0XXbZZbrrrrs0evRovffee9q2bZumTJlSrL86gDwhzAAouPnz5+uwww5TXV2d3n33XfXq1UsnnXSSrrvuuswyz6233qqrrrpK69ev16c//Wn97W9/UyQSkSSddNJJeuyxx3TDDTdo/vz5GjhwoG666SZ9+9vfzvyOe++9V9ddd52uuOIKffzxxxo8eLCuu+66Yvx1AeQZTzMB6FJanzTasWOHevXqVexyAJQAZmYAAEBJI8wAAICSxjITAAAoaXRmAABASSPMAACAkkaYAQAAJY0wAwAAShphBgAAlDTCDAAAKGmEGQAAUNIIMwAAoKT9/23/rtFeQg6nAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import pandas as pd\n", "import seaborn as sns\n", "\n", "model = operation.result()\n", "\n", "snapshots = pd.DataFrame(model.tuning_task.snapshots)\n", "\n", "sns.lineplot(data=snapshots, x = 'epoch', y='mean_loss')\n" ] }, { "cell_type": "markdown", "metadata": { "id": "rkoQTXb1vSBC" }, "source": [ "## Evaluate your model\n", "\n", "You can use the `genai.generate_text` method and specify the name of your model to test your model performance." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "zO0YcuSyxydZ" }, "outputs": [], "source": [ "model = genai.GenerativeModel(model_name=f'tunedModels/{name}')" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "UwGrrj6hS_x2" }, "outputs": [ { "data": { "application/vnd.google.colaboratory.intrinsic+json": { "type": "string" }, "text/plain": [ "'56'" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "result = model.generate_content('55')\n", "result.text" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "YSNB2zjTx5SZ" }, "outputs": [ { "data": { "application/vnd.google.colaboratory.intrinsic+json": { "type": "string" }, "text/plain": [ "'123456'" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "result = model.generate_content('123455')\n", "result.text" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Y2YVO-m0Ut9H" }, "outputs": [ { "data": { "application/vnd.google.colaboratory.intrinsic+json": { "type": "string" }, "text/plain": [ "'five'" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "result = model.generate_content('four')\n", "result.text" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "h2MkTR0uTb6U" }, "outputs": [ { "data": { "application/vnd.google.colaboratory.intrinsic+json": { "type": "string" }, "text/plain": [ "'cinq'" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "result = model.generate_content('quatre') # French 4\n", "result.text # French 5 is \"cinq\"" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "OruCW1zETsZw" }, "outputs": [ { "data": { "application/vnd.google.colaboratory.intrinsic+json": { "type": "string" }, "text/plain": [ "'IV'" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "result = model.generate_content('III') # Roman numeral 3\n", "result.text # Roman numeral 4 is IV" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "thDdSuUDUJOx" }, "outputs": [ { "data": { "application/vnd.google.colaboratory.intrinsic+json": { "type": "string" }, "text/plain": [ "'八'" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "result = model.generate_content('七') # Japanese 7\n", "result.text # Japanese 8 is 八!" ] }, { "cell_type": "markdown", "metadata": { "id": "HpIA1IFevQQR" }, "source": [ "It really seems to have picked up the task despite the limited examples, but \"next\" is a simple concept, see the [tuning guide](https://ai.google.dev/docs/model_tuning_guidance) for more guidance on improving performance." ] }, { "cell_type": "markdown", "metadata": { "id": "nmuQCbTYwIOx" }, "source": [ "## Update the description\n", "\n", "You can update the description of your tuned model any time using the `genai.update_tuned_model` method." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "9gAVuXT_wG3x" }, "outputs": [], "source": [ "genai.update_tuned_model(f'tunedModels/{name}', {\"description\":\"This is my model.\"});" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "d-c3YerBxVYs" }, "outputs": [ { "data": { "application/vnd.google.colaboratory.intrinsic+json": { "type": "string" }, "text/plain": [ "'This is my model.'" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model = genai.get_tuned_model(f'tunedModels/{name}')\n", "\n", "model.description" ] }, { "cell_type": "markdown", "metadata": { "id": "i_TpwvBB4bQ7" }, "source": [ "## Delete the model\n", "\n", "You can clean up your tuned model list by deleting models you no longer need. Use the `genai.delete_tuned_model` method to delete a model. If you canceled any tuning jobs, you may want to delete those as their performance may be unpredictable." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "cepfaUCvVGCo" }, "outputs": [], "source": [ "genai.delete_tuned_model(f'tunedModels/{name}')" ] }, { "cell_type": "markdown", "metadata": { "id": "ljEssIshYDEr" }, "source": [ "The model no longer exists:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "kN_bkut_4ayL" }, "outputs": [], "source": [ "try:\n", " m = genai.get_tuned_model(f'tunedModels/{name}')\n", " print(m)\n", "except Exception as e:\n", " print(f\"{type(e)}: {e}\")" ] } ], "metadata": { "colab": { "name": "Tuning.ipynb", "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "name": "python3" } }, "nbformat": 4, "nbformat_minor": 0 }