Skip to main content

3 posts tagged with "tagging"

View All Tags

Azure tagging strategy, part 3: cost per data product on Databricks and Fabric

· 12 min read
Shpend Kelmendi
Software Engineer & Architect

The invoice says "Azure Databricks: CHF X". Which product caused it? Azure tags alone can't answer that, because Databricks and Microsoft Fabric each add their own tag system. This is part 3 of 3. It shows which tags reach which billing data, how we get one CHF figure per data product, and where Fabric tags fit. It uses the data-product tag and the Terraform module from part 2.

TL;DR
  • Databricks and Fabric bring a second tag system. Databricks compute tags reach billing data. Fabric tags don't reach Azure billing; they help with discovery and feed your own cost allocation.
  • The concept is the same for both: export the Azure cost data, load it into tables, attribute the cost lines to products, report. Export and ingestion are already described in my Databricks cost control series.
  • Look at your own cost export first. If the cost lines carry data-product, Cost Management answers. If not, you need usage data from the platform: DBUs (Databricks Units) for Databricks, capacity unit consumption for Fabric.
  • Databricks: add VM cost from Cost Management and the Databricks charges split by DBU share, per workspace. The shares always add up to the invoice, so check the assumptions against independent data.
  • Fabric: the capacity is the smallest unit in the Azure bill. Separate capacities per cost center are the simplest split.
  • Report shared cost as its own line. Allocate it only when someone needs full cost per product.

Azure tagging strategy, part 2: from alert to ticket with Terraform and ITSM tags

· 6 min read
Shpend Kelmendi
Software Engineer & Architect

When something breaks on a data platform, the first question is which service is affected and who owns it. This is part 2 of 3. It shows how we answer that with tags: three itsm-* tags that mirror the ITSM (IT service management) service tree, plus env, managed-by and data-product, all generated by one Terraform module. It builds on the decisions from part 1.

TL;DR
  • Keep the set small, mandatory and enforced. Every tag needs a reader. Ours are six, each with a named reader: three itsm-* tags route incidents, data-product attributes cost, env splits prod from non-prod cost, and managed-by exposes click-ops. All are generated and audited.
  • Generate tags in a central Terraform module, including managed-by for everything IaC creates. Azure Policy audits; it isn't the source.
  • Mirror your ITSM service tree in tags, so they route incidents. Don't reuse ITSM tags for cost: a shared Databricks workspace has one ITSM service but many products.
  • One product, one spelling, everywhere. The module emits the data-product slug once; Azure resources and Databricks jobs reuse the same key and value.

Azure tagging strategy, part 1: what to tag, what not, and where tags can live

· 8 min read
Shpend Kelmendi
Software Engineer & Architect

Tags without a strategy grow wild: inconsistent spellings, no cost attribution and slow incident routing. This is part 1 of 3. It follows one question: which question does a tag answer, and where can it live? It covers Azure's limits, the control plane vs. data plane boundary, what not to tag, and which implementation option fits. Part 2 shows how we generate the tags with Terraform and use them for incident routing. Part 3 covers Databricks, Microsoft Fabric and cost per data product.

TL;DR
  • Tag what causes cost or incidents. Azure tags are control-plane metadata on Azure Resource Manager (ARM) resources. Tag data-plane objects such as Key Vault secrets only when a defined process reads that metadata.
  • Every tag needs a named reader. A tag that no report, policy or process reads has no value and nobody will maintain it.
  • Identity goes in the name, mutable context in tags. Changing a tag is an in-place update; a rename means recreating the resource.
  • Let IaC write the tags and Azure Policy audit them. A Modify policy and Terraform overwrite each other's changes.