Ghost Regions to Alerts Storm: Building Azure Observability When Your VM Won’t Deploy (Part 2)

If you can’t deploy compute, you pivot to telemetry. Part 1 was a multi-region ghost hunt for a corner of Azure that would let my free subscription build a VM. Part 2 starts with a blank slate, because Azure Monitor, Log Analytics and Kusto Query Language (KQL) don’t need a running virtual machine.

I used my blocked deployment days to build a centralized logging pipeline, write custom search queries, configure a log alert and design an interactive workbook.

Then came the humbling twist: once the pipeline went live, I discovered that almost every error in my logs came from my own troubleshooting.

As always, everything documented here comes from commands I ran and output I read.

1. The strategy

Rather than waiting for the VM restriction to clear, I built a small observability pipeline for the administrative actions on my subscription:

Azure actions -> Activity Log -> Diagnostic setting -> Log Analytics workspace
                                                         |-- KQL queries
                                                         |-- Alert rule -> email
                                                         |-- Workbook

Provision a Log Analytics workspace: a central log store with an ingestion cap.
Configure a diagnostic setting: stream the subscription’s Activity Log to the workspace.
Write KQL triage queries: extract operational failures.
Build an alert: email me when a failure appears.
Build a workbook: chart Activity Log operations and failures over time.

2. Bootstrapping observability: registering providers and capping ingestion

Before creating the workspace, I needed two more resource providers registered: Microsoft.OperationalInsights and Microsoft.Insights. I registered the first one and created the workspace:

# Register the Log Analytics provider (--wait blocks until registration completes)
az provider register --namespace Microsoft.OperationalInsights --wait

# Create the workspace in UK South with 30-day retention
az monitor log-analytics workspace create \
  --resource-group rg-azure-ops-lab \
  --workspace-name law-lab \
  --location uksouth \
  --retention-time 30 \
  --tags purpose=lab owner=gloria

# Set a low daily ingestion cap as a safeguard against unexpected ingestion charges
az monitor log-analytics workspace update \
  --resource-group rg-azure-ops-lab \
  --workspace-name law-lab \
  --quota 0.1

The diagnostic setting trap

The next step failed. When I tried to send the subscription’s Activity Log to the new workspace, Azure returned the error code InvalidAuthenticationToken. That sounds like a login problem, but the descriptive message pointed to a missing resource provider: Microsoft.Insights.

Registering it fixed the problem:

# Register the provider behind diagnostic settings and alerts
az provider register --namespace Microsoft.Insights --wait

# Stream the subscription's Administrative events to Log Analytics
az monitor diagnostic-settings subscription create \
  --name ds-activity-to-law \
  --location uksouth \
  --workspace "$(az monitor log-analytics workspace show -g rg-azure-ops-lab -n law-lab --query id -o tsv)" \
  --logs '[{"category":"Administrative","enabled":true}]'

Takeaway: Read the full message of an Azure CLI error, not just the error code.

3. The missing logs mystery: CLI versus Log Analytics

After setting up the pipeline, I ran my first verification query in Log Analytics:

AzureActivity | take 5

My first verification query returned only two automatic Advisor events.

It returned only two rows, both automatic Azure Advisor events. None of my own deployment or configuration events were there.

To check whether Azure had recorded my actions at all, I read the subscription’s Activity Log directly from the CLI, bypassing Log Analytics:

az monitor activity-log list --offset 3h \
  --query "[].{time:eventTimestamp, op:operationName.value, status:status.value}" \
  --output table

The CLI listed my actions, even though the workspace had none of them.

The events were there: my NSG tests, the failed West Europe VNet writes and the diagnostic-setting writes, each with Started, Accepted and Succeeded or Failed steps. Azure had recorded my actions, but those events were not appearing in the workspace.

I re-opened the diagnostic setting in the portal, ticked all the log categories, and saved. After that, new events began arriving in the workspace. I could not establish why the earlier events were missing. The change was followed by new events arriving, but I did not confirm whether the original gap was caused by the setting configuration, a propagation delay or another issue.

Takeaway: The events from before I re-saved the setting never appeared in the workspace, even though Azure had recorded them. Check that events are arriving right after you create a diagnostic setting, not days later.

4. KQL realities: exact strings, pipelines and surprising patterns

Lesson 1: check the exact values before you filter

My first triage query returned nothing, even though I knew there were errors:

// Wrong: I assumed the status was "Failed"
AzureActivity
| where ActivityStatusValue == "Failed"

Running distinct on the column showed what it really contains:

AzureActivity
| distinct ActivityStatusValue

The values were Start, Accept, Success and Failure. KQL string comparisons are exact, so the filter matched nothing and gave no warning.

The column holds “Failure”, not “Failed”.

The corrected triage query:

AzureActivity
| where TimeGenerated > ago(2h)
| where ActivityStatusValue == "Failure"
| project TimeGenerated, OperationNameValue, ResourceGroup
| order by TimeGenerated desc

The corrected query found the failed VNet write and the policy deny.

Lesson 2: read KQL as a pipeline

Each pipe (|) hands the result to the next step:

where filters rows.
project picks columns.
summarize groups and counts.
order by sorts the result.

Lesson 3: group the data to find the pattern

To see where my failures came from, I grouped them by resource and operation:

AzureActivity
| where TimeGenerated > ago(24h)
| where ActivityStatusValue == "Failure"
| summarize failures=count() by _ResourceId, OperationNameValue
| order by failures desc

Failures grouped by resource and operation.

When I checked on 23 October, the log had recorded 26 failures over 24 hours:

16: MICROSOFT.INSIGHTS/ACTIONGROUPS/CREATENOTIFICATIONS, my failed action-group test calls, which free subscriptions don’t support.
5: MICROSOFT.NETWORK/VIRTUALNETWORKS/WRITE, the West Europe attempts.
5: MICROSOFT.AUTHORIZATION/POLICIES/DENY/ACTION, the policy deny that came with them.

Most of the errors in my observability workspace came from my own testing.

5. The Alert Storm: when your own testing triggers the alert

Using the failure query, I built a log search alert rule in Azure Monitor:

Query: AzureActivity | where ActivityStatusValue == "Failure"
Condition: measure Table rows, aggregation Count, greater than 0
Schedule: evaluated every 15 minutes over a 15-minute window
Action group: ag-lab-email, with an email notification
Rule: alert-activity-failures, severity 3 (Informational)

Operational Pitfalls Encountered

Action Group Structure: Email receivers must be defined on the Notifications tab of an Action Group, not the Actions tab (which handles webhooks, Automation Runbooks, and Logic Apps).
Verification Requirement: Azure requires one-time passcode verification for email receivers. An unverified address won’t receive notifications.
Free Tier Action Group Limitations: The portal's "Test" button for Action Groups returned Unknown, and executing az monitor action-group test-notifications create failed with "Free subscription not supported". However, real alerts still worked.
Alert Aggregation Split: The rule split alerts by Resource ID, generating individual notifications per failing resource rather than a single aggregated digest. So I got a separate notification for each failing resource, not one combined message. The Metric value in each email was the failure count for one resource in one window.

One alert email: the Metric value and the resource it applies to.

The workspace’s own Alerts page looked empty. It appears to list only alerts whose affected resource is the workspace. Mine were attached to the action groups and the VNet, which is my best explanation, though I didn’t confirm it. The subscription-level Monitor → Alerts page showed all of them.

The subscription-level Alerts page showed all the fired alerts.

The alert flurry

Later, a burst of testing produced 25 alert emails, and I disabled the rule. I think my region-validation runs caused it, since the workbook later showed 21 failed VALIDATE operations, but I didn’t trace it.

The burst of alert emails in my inbox.

To eliminate alert noise during active development, I hope to test these next time:

Filter out known test activity where appropriate. For example, where OperationNameValue !contains "VALIDATE" could reduce noise from my validation runs, but I would need to make sure it does not hide validation failures that matter.
Remove granular per-resource splits for high-volume logs.
Configure alert suppression / repeat notification throttling.

6. Dashboarding the mess: a workbook

To maintain continuous visibility across the environment, I created an Azure Workbook linked to the Log Analytics workspace.

A workbook is a saved page of queries and charts that you can reopen and share. I built one, wb-azure-ops-lab, with a title, a time-range picker and two panels, both reading from law-lab:

// Panel 1: operations per hour, split by status (bar chart)
AzureActivity
| summarize operations=count() by bin(TimeGenerated, 1h), ActivityStatusValue

// Panel 2: failures by operation (grid)
AzureActivity
| where ActivityStatusValue == "Failure"
| summarize failures=count() by OperationNameValue
| order by failures desc

The workbook: a time-range picker and operations per hour, split by status.

Workbook configuration pitfalls

Log Engine Selection: Ensure visual panels use Logs (Analytics) rather than Logs (Basic). The Basic option showed a warning about extra cost, and I didn’t need it for this table.
Time Range Parameter Binding: Link every query panel to the workbook’s top-level Time Range parameter. A panel set to a fixed value, such as Last 7 days, or left on Set in query, will ignore changes made through the workbook’s interactive time-range picker.
Empty Visualization Misconceptions:Check the workbook’s default time range before assuming an empty chart means the query is broken. My workbook initially showed no data because the default range was 24 hours, while my newest events were from the previous day. I confirmed the data was present by checking the newest event timestamp:

AzureActivity
| summarize newest=max(TimeGenerated)

7-day failure analysis

Over the 7-day window I checked, the grid listed five failing operations:

21: MICROSOFT.RESOURCES/DEPLOYMENTS/VALIDATE/ACTION. The name matches my --validate region tests, so I believe these are my own scouting runs.
16: MICROSOFT.INSIGHTS/ACTIONGROUPS/CREATENOTIFICATIONS, my action-group test calls.
6: MICROSOFT.AUTHORIZATION/POLICIES/DENY/ACTION, the policy deny from my West Europe attempts.
6: MICROSOFT.NETWORK/VIRTUALNETWORKS/WRITE, the failed VNet writes from the same attempts.
1: MICROSOFT.RESOURCES/DEPLOYMENTS/WRITE. Failed resource deployment attempt. I didn’t trace which attempt this was.

The failures-by-operation grid over a 7-day window.

The counts come from a sliding 7-day window, so they will change as old events drop out.

Summary and key takeaways

Even with VM compute blocked, building a telemetry pipeline gave me real operational experience:

Verify that logs are flowing. Creating a diagnostic setting doesn’t prove events arrive. I checked with a raw CLI call and a row count.
Check exact values before you filter. A query on "Failed" returned nothing, because the column holds "Failure".
Think about alert noise early. Decide which of your own test activity should alert you before you connect an action group to an alert rule.

This exercise reminded me that monitoring is not just about collecting logs or sending alerts. It is about making sure the right events reach the right place, asking useful questions of the data, and making alerts meaningful enough to act on. I still have work to do on alert tuning and cost verification, but I now have a clearer understanding of how these Azure services fit together.

Next step: In Part 3, I want to use Azure RBAC to control who can view or edit wb-azure-ops-lab, following least privilege. I haven’t started yet, and I don’t know whether a free subscription will let me create a test identity to assign roles to, so I will document whatever I find.

How do you manage alert noise and filter out your own debugging in your environment? I’d love to hear your preferred KQL techniques.

Connect with me:

LinkedIn: linkedin.com/in/gloriaboakye
Medium: medium.com/@gloriaboakye
GitHub: github.com/gloriaboakye