cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcing the Certified User Group on Databricks Community

We are thrilled to announce the launch of our exclusive Certified User Group on the Databricks Community! This group is dedicated to individuals who have achieved Databricks certification and are eager to collaborate, brainstorm, and share insights w...

  • 322 Views
  • 0 replies
  • 1 kudos
Thursday
New Enhancements to Elevate Your Community Experience

We are thrilled to announce some fantastic new features that have gone live to enhance your experience in our Community! Your feedback has been invaluable in driving these improvements, and we can't wait for you to check them out. 1. Progress Bar Und...

  • 137 Views
  • 0 replies
  • 3 kudos
Wednesday
📢 New Launch: Databricks Product Tours

Discover the latest Databricks Data Intelligent Platform with our product tours, showcasing the latest innovations in just a few clicks. Join us for an immersive experience where you'll explore the cutting-edge features and capabilities of our platfo...

  • 173 Views
  • 0 replies
  • 2 kudos
Wednesday
Featured Member Interview - July 2024 - G Samhitha Priyadarshini

Meet G Samhitha Priyadarshini, our featured member for this month! She has been our top contributor and possesses extensive knowledge in Databricks. She is also a Certified Lounge User Group owner. Here are some of the questions we asked her during t...

  • 342 Views
  • 5 replies
  • 2 kudos
Thursday
Data + AI World Tour 2024

Over 12,000 companies have adopted data intelligence. Join a tour stop near you to discover how Databricks can unlock your data and AI too. Find your city now!  Explore technologies Explore the latest advancements in the Databricks Data Intellige...

  • 643 Views
  • 0 replies
  • 4 kudos
2 weeks ago
Check Out the Latest Videos on DatabricksTV

We are thrilled to introduce DatabricksTV – your go-to hub for all things Databricks! DatabricksTV is packed with insightful videos from a diverse range of creators, offering you the latest tips, tutorials, and deep dives into the Databricks platfor...

  • 114 Views
  • 0 replies
  • 3 kudos
Tuesday

Community Activity

shadowinc
by > New Contributor III
  • 468 Views
  • 5 replies
  • 3 kudos

Databricks SQL endpoint as Linked Service in Azure Data Factory

We have a special endpoint that grants access to delta tables and we want to know if we can use SQL endpoints as a linked service in ADF.If yes then which ADF-linked service would be suitable for this?Appreciate your support on this. 

Data Engineering
SQl endpoint
  • 468 Views
  • 5 replies
  • 3 kudos
Latest Reply
RobsonNLPT
Contributor
  • 3 kudos

Adf does not support adb sql endpoint as linked server. 

  • 3 kudos
4 More Replies
Anonymous3
by > New Contributor
  • 20 Views
  • 1 replies
  • 0 kudos

How to migrate Legacy Dashboard from one workspace to another workspace?

Is it possible to migrate Databricks Legacy Dashboards along with associated components such as queries and datasets from one workspace to another? I have attempted to export the Legacy Dashboards along with the queries and datasets, but upon importi...

  • 20 Views
  • 1 replies
  • 0 kudos
Latest Reply
holly
Valued Contributor
  • 0 kudos

Hello, to migrate between workspaces you'll need a different strategy based on the asset type Legacy dashboards you have completed, but for anyone reading you can use the 'export' functionalityQueries: you can either put them together in a folder and...

  • 0 kudos
fonini
by > New Contributor
  • 91 Views
  • 1 replies
  • 0 kudos

dbutils secrets break matplotlib in recent Databricks runtimes

Using a Databricks cluster running Databricks Runtime version 15.4 LTS Beta, and with no init-scripts, the following code works fine:import numpy as npimport pandas as pddf = pd.DataFrame(np.random.randn(2000, 10))df.plot(subplots=True, figsize=(10, ...

Administration & Architecture
Bug
dbutils
matplotlib
  • 91 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @fonini, This issue doesn't occur in older runtimes, such as 7.3, which is why your legacy clusters are unaffected. To work around this, you can try re-importing matplotlib after calling dbutils.secrets.get, or use a different runtime version.

  • 0 kudos
darkolexis
by > New Contributor
  • 71 Views
  • 2 replies
  • 1 kudos

Service Principal types in Azure Databricks

In Azure Databricks, we can create two types of Service Principals, namely:1. Databricks Managed SP2. Microsoft Entra ID Managed SP What is the difference between two, other than one being specific to single workspace, and another being usable from m...

  • 71 Views
  • 2 replies
  • 1 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @darkolexis , Thanks for reaching out! Please review the response and let us know if it addresses your question. Your feedback is valuable to us and the community.   If the response resolves your issue, kindly mark it as the accepted solution. Thi...

  • 1 kudos
1 More Replies
Sharif
by > New Contributor
  • 131 Views
  • 8 replies
  • 3 kudos

Resolved! Cannot Create databricks account

I am trying to create databricks community account but after providing all the information and completing the puzzle it is showing me an error occurred: I also recorded the the network request for the error:Header:Request URL:https://www.databricks.c...

Sharif_0-1722566342399.png
  • 131 Views
  • 8 replies
  • 3 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 3 kudos

Hi @Sharif, Thank you for contacting Databricks Community Discussion Forum.   Please note that for any issues related to the Databricks Community Edition product, you can find helpful resources here. If you encounter any difficulties beyond what's co...

  • 3 kudos
7 More Replies
Kjetil
by > New Contributor III
  • 20 Views
  • 0 replies
  • 0 kudos

Feature Store - lookback_window does not work with primary keys of "date" type

I just discovered what I believe is a bug in Feature Store. The expected value (of the "value" column) is 'NULL' but the actual value is "a". If I instead change the format to timestamp of the "date" column (i.e. removes the .date() in the generation...

  • 20 Views
  • 0 replies
  • 0 kudos
Trilleo
by > New Contributor II
  • 53 Views
  • 1 replies
  • 0 kudos

STATEMENT_TIMEOUT on a specific SQL Warehouse

Hi, I would like to se STATEMENT_TIMEOUT for a specific SQL warehouse and not on a global level.How would I do that?P.s. I would like to avoid it on a session level, just one-time configuration for a given SQL warehouse. 

  • 53 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Trilleo, To set the STATEMENT_TIMEOUT for a specific SQL warehouse in Databricks, you can use the ALTER WAREHOUSE command. This allows you to configure the timeout at the warehouse level without affecting the global or session settings. 

  • 0 kudos
TheManOfSteele
by > New Contributor II
  • 72 Views
  • 1 replies
  • 0 kudos

Azure Pipeline Release Bundle Validate Failure

Hello,I am trying to create a CI/CD pipeline. After the build pipeline, I am trying to create a release to databricks that runs a notebook. I am trying to run this as a service principal. During the bundle validate step I am getting this error. "Erro...

TheManOfSteele_0-1722516587122.png
  • 72 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @TheManOfSteele, It sounds like you're facing an authentication issue with a service principal in your Databricks CI/CD pipeline. To resolve this, ensure that your service principal is correctly configured with the necessary permissions, and doubl...

  • 0 kudos
sunnypol
by > New Contributor
  • 54 Views
  • 1 replies
  • 0 kudos

SQL Warehouse: INVALID_PARAMETER_VALUE when starting

Hi Team,Suddenly our SQL warehouse stopped running, can you please help us how can we check why it stopped.  

  • 54 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @sunnypol, start by checking the Data Access Configuration properties in the Workspace Settings to ensure that the necessary secrets exist and that the cluster's owner has the proper read permissions. Verify that the user has read access to any se...

  • 0 kudos
hpant1
by > New Contributor
  • 17 Views
  • 0 replies
  • 0 kudos

What is more optimized way of writing delta table in a workflow, "append" or "overwrite"?

What is more optimized way of writing delta table in a workflow which is running every hour, "append" or "overwrite"?

  • 17 Views
  • 0 replies
  • 0 kudos
ahmed_zarar
by > New Contributor
  • 94 Views
  • 2 replies
  • 2 kudos

Process single data set with different JSON schema rows using Pyspark in databricks

 Hi,i am getting data from event hub and stored in delta table as a row table, i data i received in json , the problem i data i have different schema in each row but i code i use it take first row a json schema i am stuck how to do please any one gui...

ahmed_zarar_0-1722683168135.png
  • 94 Views
  • 2 replies
  • 2 kudos
Latest Reply
ahmed_zarar
New Contributor
  • 2 kudos

Thank you , I got it.

  • 2 kudos
1 More Replies
ag2all
by > New Contributor
  • 76 Views
  • 2 replies
  • 0 kudos

Databricks + Apache Iceberg = advantageous or wasted effort due to duplicate functionality ?

Trying to design a Lakehouse. Spark is at the base layer. Now wondering if adding Apache Iceberg sitting below Spark will be of help, or, not ? Preferring Iceberg for its auto indexing, ACID query facilities over big hetergenous datasets. Wonder if i...

  • 76 Views
  • 2 replies
  • 0 kudos
Latest Reply
holly
Valued Contributor
  • 0 kudos

Hello, if you're planning on building your own open source stack of spark+iceberg, it can be a good choice.  If you're on Databricks, however, you're going to miss out a *lot* on delta features that are baked into the platform. Specifically compute +...

  • 0 kudos
1 More Replies
slechtd
by > New Contributor
  • 164 Views
  • 6 replies
  • 0 kudos

Cannot sign-in at accounts.cloud.databricks.com

Hi,I have registered for Community Edition and can access it with no problems trough: https://community.cloud.databricks.com/login.htmlNow, I'm interested in completing the free "lakehouse fundamentals" training here and taking the quiz to get the ba...

  • 164 Views
  • 6 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @slechtd, To expedite your request, please list your concerns on our ticketing portal. Our support staff would be able to act faster on the resolution (our standard resolution time is 24-48 hours).  

  • 0 kudos
5 More Replies
Prashanth24
by > New Contributor
  • 19 Views
  • 0 replies
  • 0 kudos

Error connecting Databricks Notebook using managed identity from Azure Data Factory

I am trying to connect Databricks Notebook using managed identity authentication type from Azure Data Factory. Below are the settings done. Error message is appended at the bottom of this message. With the same settings but with different authenticat...

  • 19 Views
  • 0 replies
  • 0 kudos
tobi
by > New Contributor
  • 110 Views
  • 4 replies
  • 0 kudos

Deleted workspace

Hello guys, I have a question. We have databricks on gcp, we forgot to pay for subscription and they removed our workspace. I had code notebooks on that workspace. If there is any way to reproduce this code? Or maybe it’s automatically saved this cod...

  • 110 Views
  • 4 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @tobi,  If you have set up automated backups or if your notebooks are synced with a GCS bucket, you might find your code there. Check your GCS buckets for any saved versions of your notebooks. Also, If you had a disaster recovery plan in place, yo...

  • 0 kudos
3 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Polls
Are you a member of a Regional User Group?

Are you a member of a Regional User Group?

Top Kudoed Authors

Latest from our Blog

Databricks SSO with Auth0 using SAML 2.0

With the recent changes to Databricks login in order to increase our customers' security posture, some customers might be scrambling to setup SSO on their Databricks account and workspaces.  Many of t...

569Views 0kudos