Skip to main content

Azure tagging strategy, part 2: from alert to ticket with Terraform and ITSM tags

· 6 min read
Shpend Kelmendi
Software Engineer & Architect

When something breaks on a data platform, the first question is which service is affected and who owns it. This is part 2 of 3. It shows how we answer that with tags: three itsm-* tags that mirror the ITSM (IT service management) service tree, plus env, managed-by and data-product, all generated by one Terraform module. It builds on the decisions from part 1.

TL;DR
  • Keep the set small, mandatory and enforced. Every tag needs a reader. Ours are six, each with a named reader: three itsm-* tags route incidents, data-product attributes cost, env splits prod from non-prod cost, and managed-by exposes click-ops. All are generated and audited.
  • Generate tags in a central Terraform module, including managed-by for everything IaC creates. Azure Policy audits; it isn't the source.
  • Mirror your ITSM service tree in tags, so they route incidents. Don't reuse ITSM tags for cost: a shared Databricks workspace has one ITSM service but many products.
  • One product, one spelling, everywhere. The module emits the data-product slug once; Azure resources and Databricks jobs reuse the same key and value.
Series: Azure tagging strategy for a data platform
  1. Part 1: what to tag, what not, and where tags can live
  2. Part 2: from alert to ticket with Terraform and ITSM tags (this post)
  3. Part 3: cost per data product on Databricks and Fabric

Goal: from alert to the right ticket queue​

Our main driver wasn't cost, it was operations. When something breaks, the ITSM tool must immediately show which service is affected, so that the ticket lands in the right queue with the right priority. ITSM tools model services as a tree, and we mirror that tree in three tags:

  • itsm-level-1: the platform per environment.
  • itsm-level-2: one per data product. We derive it from the product, not the organisational unit: products are stable and own their incidents, org charts change.
  • itsm-level-3: one per product and building block (storage, Databricks, Event Hub, ...).

These three tags answer "who gets the ticket?". env and managed-by answer lifecycle questions, and each has its own reader (see the table in Part 1).

The environment appears in five places: the name, env, and at the end of each itsm-* value. That isn't the Part 1 problem. Part 1 had three people typing three spellings. Here, one input (environment) produces all five values in one module, so they can't drift apart. Each copy also has a different reader. The itsm-* values must match the ITSM tool's configuration item names exactly, and DEV, INT and PRD are separate items there. env is the short, stable value that Cost Management groups by.

"Who pays?" needs a sixth tag, data-product, and it must stay separate from the ITSM tags. They describe who operates a resource, and that isn't always who uses it. Our shared Databricks workspace is operated by the platform team, so its itsm-level-2 names the platform, not a product. Those tags land on every cluster VM. If we grouped cost by itsm-level-2, all Databricks compute would be booked to the platform, whichever product's job ran.

data-product is a lowercase slug (sales-forecast) that the module derives from component. Every consumer takes it from the module output: Azure resources, Databricks job tags and compute policies. Cost Management and system.billing.usage therefore use the same key and value, so they can be joined. They don't measure the same thing, though: one shows invoiced CHF, the other shows DBUs. From DBUs to CHF adds them up.

Shared platform resources, the Databricks workspace among them, get no data-product tag. On the workspace it would land on every VM next to the job's own data-product, and Databricks doesn't document which value wins. What stays untagged in Cost Management is the shared platform cost, an explicit bucket instead of a wrong attribution.

No process of ours reads a cost center from Azure, so by our own rule there is no cost-center tag.

Anything a product team passes through additional_tags is optional and unenforced. It's allowed, but it isn't part of the strategy.

We follow the IaC + Policy (audit) option: every platform resource is deployed through Terraform and CI, so every tag change is a reviewed pull request and visible in the plan.

A central, reusable Terraform module​

The module creates no resources and needs no providers. It's a pure function from a few inputs to complete tag maps, so it's trivial to test and safe to call anywhere.

modules/platform-tags/variables.tf
variable "business_service" {
type = string
nullable = false
validation {
condition = contains(["Contoso Data Platform", "Contoso Integration Platform"], trimspace(var.business_service))
error_message = "business_service must be an approved ITSM business service."
}
}

variable "component" {
type = string
description = "ITSM service component: the data product, or the platform component for shared resources."
nullable = false
validation {
condition = can(regex("^[A-Za-z0-9 ]+$", trimspace(var.component)))
error_message = "component must contain only letters, numbers and spaces."
}
}

variable "shared" {
type = bool
description = "True for platform resources used by several data products. Omits data-product."
default = false
nullable = false
}

variable "environment" {
type = string
nullable = false
validation {
condition = contains(["dev", "int", "prd"], lower(trimspace(var.environment)))
error_message = "environment must be DEV, INT, or PRD."
}
}

variable "additional_tags" {
type = map(string)
default = {}
nullable = false
validation {
# Azure tag names are case-insensitive, so compare lower-case.
condition = alltrue([
for k in keys(var.additional_tags) :
!contains(["itsm-level-1", "itsm-level-2", "itsm-level-3", "managed-by", "env", "data-product"], lower(k))
])
error_message = "additional_tags must not override centrally managed tag keys."
}
}

Pitfalls from practice​

  1. The ITSM tree must exist first. A tag pointing to an entry that doesn't exist in the ITSM tool is worse than no tag, because tickets silently fall through.
  2. Validate inputs aggressively. A typo in business_service produces a valid tag that matches nothing. The allow-list catches it at terraform plan time.
  3. Reserve central keys. Without the additional_tags guard, a product team overrides itsm-level-3 "just for this one resource", and the hierarchy breaks.
  4. Map children to parents. Action groups, alert rules and access connectors shouldn't create their own itsm-level-3 values. Explicit parent mappings stop the ITSM tree from exploding.
  5. No ITSM tags on resource groups. They aren't operated assets, and tagging them creates noise in ITSM discovery. The Indexed policy mode ignores them anyway.
  6. Don't reuse operations tags for cost. itsm-level-2 says who operates a resource, data-product says who uses it. They differ for every shared resource.
  7. Spell the product once. If one team types Sales Forecast and another salesforecast, Cost Management shows two products. Derive the slug in the module and pass the output on; never type it by hand.

Next: cost per data product​

Part 3 covers Databricks and Microsoft Fabric, which bring their own tag systems, and shows how to turn data-product into one CHF figure per product. Continue with part 3.

Further Readings​