We are thrilled to announce the launch of our exclusive Certified User Group on the Databricks Community! This group is dedicated to individuals who have achieved Databricks certification and are eager to collaborate, brainstorm, and share insights w...
- 334 Views
- 0 replies
- 1 kudos
Thursday
We are thrilled to announce some fantastic new features that have gone live to enhance your experience in our Community! Your feedback has been invaluable in driving these improvements, and we can't wait for you to check them out. 1. Progress Bar Und...
- 145 Views
- 0 replies
- 3 kudos
Wednesday
Discover the latest Databricks Data Intelligent Platform with our product tours, showcasing the latest innovations in just a few clicks. Join us for an immersive experience where you'll explore the cutting-edge features and capabilities of our platfo...
- 178 Views
- 0 replies
- 2 kudos
Wednesday
Meet G Samhitha Priyadarshini, our featured member for this month! She has been our top contributor and possesses extensive knowledge in Databricks. She is also a Certified Lounge User Group owner. Here are some of the questions we asked her during t...
- 365 Views
- 5 replies
- 2 kudos
Thursday
Over 12,000 companies have adopted data intelligence. Join a tour stop near you to discover how Databricks can unlock your data and AI too. Find your city now! Explore technologies Explore the latest advancements in the Databricks Data Intellige...
- 656 Views
- 0 replies
- 4 kudos
2 weeks ago
We are thrilled to introduce DatabricksTV – your go-to hub for all things Databricks! DatabricksTV is packed with insightful videos from a diverse range of creators, offering you the latest tips, tutorials, and deep dives into the Databricks platfor...
- 119 Views
- 0 replies
- 3 kudos
Tuesday
Community Activity
- 64 Views
- 3 replies
- 0 kudos
Clarification on databricks-labs-ucx sync-workspace-info Command for Syncing Workspaces
I am currently working on syncing workspaces in Databricks Accounts console, and I came across the command databricks-labs-ucx sync-workspace-info. I am wondering if this command is suitable for both UCX (Unity Catalog) assessments and other non-UCX ...
- 64 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi there, you can use UCX sync to get an inventory of everything in your workspaces and track it as you upgrade to avoid cross workspace duplication where there's not a 1:1 mapping. It's designed for Unity Catalog upgrades, so whilst it's battle test...
- 0 kudos
- 20 Views
- 0 replies
- 0 kudos
csv file processing
whats best possible solution to process csv file in databricks.Please consider scalability,optimization, qa give m best solution...
- 20 Views
- 0 replies
- 0 kudos
- 22 Views
- 0 replies
- 0 kudos
Create delta files from Unity Catalog Objects
Hello,I have tables created on unity catalog that point to the raw area , from these tables I need to create a data model (facts and dimensions) that will aggregate this data, transform certain things. Then I need to store in the Azure Datalake in de...
- 22 Views
- 0 replies
- 0 kudos
- 21 Views
- 0 replies
- 0 kudos
Can I load the files based on the data in my table as variable without iterating through each row?
Hi,I have created this table which contains the data that I need for my source path and target table. source_path: /data/customer/sid={sid}/abc=1/attr_provider={attr_prov}/source_data_provider_code={src_prov}/So basically, the value of each row are c...
- 21 Views
- 0 replies
- 0 kudos
- 31 Views
- 0 replies
- 0 kudos
Erorr connecting to Databricks from ADF Delta Lake - Error message: Client Secret is invalid
Hi,The client secret recently expired and I updated the configuration with the new secret. From then on, I'm not able to access the data from ADF using the delta lake component, and here is the error message I keep getting:HTTP Error 401; url='https:...
- 31 Views
- 0 replies
- 0 kudos
- 25 Views
- 0 replies
- 0 kudos
Glue database and saveAsTable
Hello all,I am saving my data frame as a Delta Table to S3 and AWS Glue using pyspark and `saveAsTable`, so far I can do this but something curious happens when I try to change the `path` (as an option or as an argument of `saveAsTable`).The location...
- 25 Views
- 0 replies
- 0 kudos
- 27 Views
- 0 replies
- 0 kudos
Pass variable from one notebook to another
I have a main notebook where I am setting a python variable to the name of the catalog I want to work in. I then call another notebook, using %run, which runs an insert into using a SQL command where I want to specify the catalog using the catalog v...
- 27 Views
- 0 replies
- 0 kudos
- 26 Views
- 0 replies
- 0 kudos
Not able to write in the schema stored in the external location
I have three tables and I trying to write it in the bronze schema which is store in the external location. I am able to write two of those but for third one I am getting this error:Not sure why is the case, I am exactly doing the same.
- 26 Views
- 0 replies
- 0 kudos
- 48 Views
- 1 replies
- 0 kudos
How to search the run id of an experiment run created in another notebook?
Hello,I have created an experiment using with mlflow.start_run(run_name='experment_1'):in a notebook say 'notebook_1'. In the 'Experiments' tab if I click on 'notebook_1', I am able to see 'experiment_1'. Now I am trying to search the experiment in ...
- 48 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi,I believe the run name is an attribute, not a tag.Try:my_run = mlflow.search_runs(search_all_experiments=True,filter_string="attributes.run_name='experiment_1'")To search for runs in specific notebooks, you can add "tags.environment='notebook_name...
- 0 kudos
- 102 Views
- 2 replies
- 1 kudos
spark.sql() is suddenly giving an error "Unable to instantiate org.apache.hadoop.hive.metastore.Hive
spark.sql() is suddenly giving an error "Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient" on databricks jobs and python scripts that worked last month. No local changes on my end.What could be the cause of this and what sh...
- 102 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi @copper-carrot, Have there been any recent updates to the Databricks environment or the underlying infrastructure? Sometimes, updates to the Databricks runtime or libraries can mess with compatibility. Also, check if any of your project's dependen...
- 1 kudos
- 112 Views
- 3 replies
- 1 kudos
System Tables
Hi,I've recently enabled the billing schema in system tables, but still not able to view it in catalog. I am both an account admin and workspace admin, as well as an owner of the workspace in Azure. A colleague (who used to be Azure AD admin but no l...
- 112 Views
- 3 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi @millie_dbrx_use, The account admin can see the billing schema. To see the schema you must be either account or metastore admin or have someone grant you the rights to the schema via Unity Catalog.
- 1 kudos
- 111 Views
- 6 replies
- 1 kudos
Unable to view Manage Account Option in Databricks UI
Hi All, I want to remove Unity Catalog from Admin Console so that I need to navigate to manage account option but the option is not available in my Databricks workspace. Please help me to sort this issue by removing Unity Catalog.
- 111 Views
- 6 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi @faaiqsaheed12, Thanks for reaching out! Please review the responses and let us know which best addresses your question. Your feedback is valuable to us and the community. If the response resolves your issue, kindly mark it as the accepted solut...
- 1 kudos
- 82 Views
- 4 replies
- 2 kudos
can we parameterize the tags in the job compute
I want to monitor the cost better for the databricks job computes.I am using tags in the cluster to monitor cost.The tag values is static as of now.can we parameterize the compute the job cluster so that I can pass the tag values during the runtime a...
- 82 Views
- 4 replies
- 2 kudos
Latest Reply
- 2 kudos
Hi @narenderkumar53, Thanks for reaching out! Please review the response and let us know if it answers your question. Your feedback is valuable to us and the community. If the response resolves your issue, kindly mark it as the accepted solution. Thi...
- 2 kudos
- 72 Views
- 4 replies
- 0 kudos
Promoting assets from Dev workspace to Prod
What is the best way to promote Databricks objects from one workspace to another? For example, DABs are great for code and models. However, SQL alerts do not seem to fall into DABs. The API allows for retrieving queries and creating them via API in t...
- 72 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @StephanieAlba, Please review the responses and let us know which best addresses your question. If the response resolves your issue, kindly mark it as the accepted solution.
- 0 kudos
- 88 Views
- 2 replies
- 1 kudos
How to list my groups in unitity catalog via an API
I'm trying to list my groups in unity catalog via an API call, but it keeps giving the error "Bad Target"My GET request is:-https://adb-3676685197829242.2.azuredatabricks.net/api/2.1/accounts/cfb89269-0d9e-4235-b7985-455481b501a/scim/v2/GroupsI'm abl...
- 88 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi @Marco37, The SCIM API endpoint you provided is for managing users and groups at the account level, not specifically for the Unity Catalog. Unfortunately, there isn’t a direct API endpoint for listing groups in the Unity Catalog. However, you can ...
- 1 kudos
Announcements
Latest from our Blog
Simplifying Multi-Model LLM Development: A Developer’s Guide to LiteLLM and Databricks
LLMs on Databricks are now available to call via LiteLLM. LiteLLM is a library that provides a python client and an OpenAI-compatible proxy for accessing 100+ LLMs with the same input/output formats, ...
Eight Weeks, Two Developers, One Production AI Conference Assistant
Introduction: The Origins of Brickbot If you attended the Databricks Data and AI Summit (DAIS) this year, you may have noticed a new addition to the event app: Brickbot, the summit AI assistant. Brick...
You should know these 4 things about setting up Unity Catalog!
Over the past year we’ve seen huge adoption of Unity Catalog, and we've noticed certain considerations that are key to all successful deployments. This blog focuses on how to set up a key artifact in ...
Databricks SSO with Auth0 using SAML 2.0
With the recent changes to Databricks login in order to increase our customers' security posture, some customers might be scrambling to setup SSO on their Databricks account and workspaces. Many of t...
