4 posts tagged with "Data-Platform"

This is the final part of the series. Part 1 compared the alternatives and explained our combined approach. Part 2 built the cost dashboard, and Part 3 added proactive SQL alerts. One piece is still missing: enforcement. Dashboards show and alerts warn. Neither of them stops anything. That is the job of Budget Control.

In this post, you'll learn:

• What Budget Control (Public Preview) can and cannot do today
• How we set up budgets per user and user group, and how we derived role-based limits
• How spend blocking works (currently Genie only) and how our override process looks
• How we manage the ClickOps setup as documented technical debt until Terraform support catches up

In Part 1 I laid out our three building blocks for Databricks cost control: dashboards for visibility, SQL alerts for proactive management, and Budget Control for governance. Part 2 covered the dashboard. Now for the part that actually wakes us up when something goes wrong: SQL alerts.

In this post, you'll learn:

• Why a great dashboard still leaves you reactive
• The exact query behind our "yesterday above the 30-day median" alert, including segmentation by sandbox vs. production
• How to provision queries and alerts with Terraform
• How we design thresholds and keep alert fatigue under control

In Part 1, I compared unexpected cloud costs to a surprise restaurant bill. An itemized bill helps you understand what you ordered and spot a bottle of wine that might belong to another table. A cost dashboard does something similar: it breaks down spending so you can investigate unexpected charges.

That visibility is the starting point for meaningful alerts and budgets. In this post, we'll build a dashboard and explore:

• Which dashboarding options exist on Azure and inside Databricks, and where each one fits
• How to move the built-in usage dashboard from ClickOps to Terraform
• How we bring Azure infrastructure costs into the same dashboard
• Which Azure cost export type to choose: actual, amortized or FOCUS
• Who should get CAN_RUN on the dashboard, and when that is risky