Skip to main content

11 posts tagged with "Data-Platform"

View All Tags

Controlling Databricks Costs on Azure, Part 2: Building the Cost Dashboard

· 11 min read
Shpend Kelmendi
Software Engineer & Architect

In Part 1, I compared unexpected cloud costs to a surprise restaurant bill. An itemized bill helps you understand what you ordered and spot a bottle of wine that might belong to another table. A cost dashboard does something similar: it breaks down spending so you can investigate unexpected charges.

That visibility is the starting point for meaningful alerts and budgets. In this post, we'll build a dashboard and explore:

  • Which dashboarding options exist on Azure and inside Databricks, and where each one fits
  • How to move the built-in usage dashboard from ClickOps to Terraform
  • How we bring Azure infrastructure costs into the same dashboard
  • Which Azure cost export type to choose: actual, amortized or FOCUS
  • Who should get CAN_RUN on the dashboard, and when that is risky

Controlling Databricks Costs on Azure

· 9 min read
Shpend Kelmendi
Software Engineer & Architect

No one likes surprises on the bill. Whether it is at a restaurant with your date or in a company setting, an unexpectedly high invoice can quickly put us in unnecessary trouble. Explaining to leadership why costs suddenly went up can be okay when it was a conscious decision, but not when something was simply overlooked. That is exactly one of the biggest challenges for our Data Platform Engineering team: cloud and Databricks offer huge opportunities, but costs can rise sharply if you do not pay close attention. What is often missing is the clarity and early steering mechanisms needed to identify the main cost drivers and react in time. Our goal is to enable data teams to unlock the full potential of the platform without unpleasant end-of-month surprises. For us, success means no unexpected costs and cost awareness across all users. After evaluating multiple options, we chose a solution that gives us this freedom without giving up control.

TL;DR

There’s no silver bullet. We combined following components to achieve transparency, responsiveness, and control:

  • Databricks dashboards for visibility into cost drivers → what costs
  • Databricks SQL alerts for proactive notifications when spend deviates → when to act
  • Budget Control for per-user and per-user-group governance → enforce limits