Overview

As I mentioned on my previous post, we have process and service metadata flowing into Zabbix per-host. The pain point: every new process or service still requires editing the template by hand.

This guide assumes you have a working agent install with custom UserParameters (covered in the process logging post) and basic familiarity with Zabbix templates.

Low-Level Discovery (LLD) solves this. The agent returns a JSON array of "things it found", and Zabbix uses that array to clone a set of prototypes items, triggers, graphs once per discovered entry.

The flow:

1. Write a script that returns a JSON array of objects with {#MACRO} keys.
2. Register it as a UserParameter (or external check).
3. Create a Discovery rule on the template pointing at that key.
4. Create Item / Trigger / Graph prototypes that reference the macros.
5. Wait one discovery cycle. Zabbix creates one item per discovered entry.

The contract is the JSON shape. Once Zabbix sees it, the rest is just templating.

The JSON Contract

Zabbix expects an array under any key, each element being an object whose keys start with # and are written {#UPPERCASE} when used in prototypes. The simplest possible payload:

{
  "data": [
    { "{#NAME}": "sshd",  "{#PID}": 1234 },
    { "{#NAME}": "nginx", "{#PID}": 5678 }
  ]
}

Modern Zabbix (5.0+) also accepts the array directly without a data wrapper. Both still work pick one and stay consistent.

Windows: Discovering Services with Powershell

Let's auto-discover every Windows service whose name starts with MyApp and create per-service items for state, start mode, and uptime.

1. The Discovery Script

Save as C:\Program Files\Zabbix Agent\scripts\Discover-Services.ps1:

<#
.SYNOPSIS
    Returns a Zabbix LLD JSON payload of services matching a name prefix.
#>
[CmdletBinding()]
[OutputType([string])]
param(
    [Parameter(Mandatory, Position = 0)]
    [string]$Prefix
)

$ErrorActionPreference = 'Stop'

$services = Get-CimInstance Win32_Service -Filter "Name LIKE '${Prefix}%'"

$entries = foreach ($s in $services) {
    [ordered]@{
        '{#SERVICE.NAME}'    = $s.Name
        '{#SERVICE.DISPLAY}' = $s.DisplayName
        '{#SERVICE.PATH}'    = $s.PathName
        '{#SERVICE.USER}'    = $s.StartName
    }
}

@{ data = @($entries) } | ConvertTo-Json -Depth 4 -Compress

Test it from the host:

& 'C:\Program Files\Zabbix Agent\scripts\Discover-Services.ps1' 'MyApp'

You should get something like:

{"data":[{"{#SERVICE.NAME}":"MyAppWeb","{#SERVICE.DISPLAY}":"MyApp Web","{#SERVICE.PATH}":"\"C:\\MyApp\\web.exe\"","{#SERVICE.USER}":"NT AUTHORITY\\NetworkService"}]}

2. Register the UserParameter

Add to C:\Program Files\Zabbix Agent\zabbix_agentd.d\lld.conf:

UserParameter=discovery.services[*],powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Program Files\Zabbix Agent\scripts\Discover-Services.ps1" "$1"

Restart the agent:

Restart-Service 'Zabbix Agent'

Verify from the server:

zabbix_get -s 10.0.0.50 -k 'discovery.services[MyApp]'

Linux: Discovering Mounted Filesystems

The classic LLD example. Built-in vfs.fs.discovery exists, but rolling our own gives full control over the macros.

1. The Discovery Script

Save as /etc/zabbix/scripts/discover-fs.sh and chmod +x:

#!/usr/bin/env bash
# Emits LLD JSON for non-pseudo filesystems.
set -euo pipefail

first=1
printf '{"data":['
while read -r src target fstype _; do
    case "$fstype" in
        ext4|xfs|btrfs|zfs) ;;
        *) continue ;;
    esac
    [[ $first -eq 0 ]] && printf ','
    first=0
    printf '{"{#FSNAME}":"%s","{#FSTYPE}":"%s","{#FSDEV}":"%s"}' \
           "$target" "$fstype" "$src"
done < /proc/mounts
printf ']}\n'

2. Register the UserParameter

Add to /etc/zabbix/zabbix_agentd.d/lld.conf:

UserParameter=discovery.fs,/etc/zabbix/scripts/discover-fs.sh

Restart the agent:

sudo systemctl restart zabbix-agent

Test:

zabbix_get -s 10.0.0.51 -k 'discovery.fs'

Wire It Up in the Frontend

1. Create the Discovery Rule

In your template, go to Discovery rules -> Create discovery rule:

• Name: MyApp services
• Type: Zabbix agent
• Key: discovery.services[MyApp]
• Update interval: 1h (LLD does not need to run every minute)
• Keep lost resources period: 7d

Setting "keep lost resources" to a few days means a service that briefly disappears doesn't immediately wipe its history. The right value depends on how transient your discoveries are.

2. Item Prototypes

Under the discovery rule -> Item prototypes -> Create item prototype:

• Name: Service [{#SERVICE.DISPLAY}] state
• Key: service.info[{#SERVICE.NAME},state]
• Type of information: Numeric (unsigned)
• Value mapping: Windows service state

Repeat for any other per-service metric you want (start mode, PID, uptime).

3. Trigger Prototypes

• Name: Service {#SERVICE.DISPLAY} on {HOST.NAME} is not running
• Expression: last(/Template/service.info[{#SERVICE.NAME},state]) <> 0 # template could be Windows by Zabbix Agent last(/Windows by Zabbix agent/service.info[{#SERVICE.NAME},state])<>0
• Severity: High

4. Graph Prototypes (Optional)

Useful when the per-discovery item is numeric (CPU, memory, uptime). Create a graph prototype that plots one metric across all discovered services.

Filtering Discovered Entries

Sometimes the script returns more than you want (e.g. all filesystems including /snap/...). Two ways to filter:

A. In the script cleanest. Cull at the source as the bash example does.

B. In the discovery rule under Filters, add {#FSNAME} does not match @System mountpoints for Linux (a built-in regex). Multiple filters AND together.

Prefer filtering in the script. It keeps the JSON small (less data on the wire), and the rule remains reusable across hosts that need different filters via macros.

Macros and Overrides

Two more things make LLD genuinely powerful:

• User macros in keys discovery.services[{$MYAPP.PREFIX}] lets each host override the prefix via {$MYAPP.PREFIX} without cloning the template.
• Overrides under the discovery rule, you can match on a discovered macro and conditionally disable items, change update intervals, or link extra templates. Great for "treat the prod-* services as critical, everything else as standard".

Test the Whole Loop

After saving the template:

1. Wait one Update interval (or hit Execute now on the discovery rule).
2. Go to the host -> Latest data. The discovered items should appear with names like Service [MyApp Web] state.
3. Stop one of the services. Within a couple of update cycles, the trigger fires.
4. Add a new service matching the prefix on the host. After the next discovery cycle, items appear automatically.

What to Do Next

LLD turns "edit the template every time something changes" into "deploy the host and walk away". You decide what to monitor; Zabbix figures out which instances exist. The discipline that holds the pattern up is filtering aggressively (an unfiltered LLD rule on a Kubernetes node can create thousands of items per hour), reviewing item prototypes carefully (one prototype change rewrites the entire fleet), and using lld_macro_paths so JSON producers don't have to flatten themselves.

Three concrete moves to land your first LLD rule cleanly:

1. Start with a tightly-filtered rule, not a wildcard. Discover only services matching a known prefix, only filesystems mounted under /data, only network interfaces matching eth*. Loosen the filter once the rule is producing exactly what you expect.
2. Set a discovery interval measured in hours, not minutes. LLD is for things that change rarely. A 1-hour interval on a service-discovery rule keeps churn bounded and protects you against a misbehaving JSON producer creating thousands of items in a single afternoon.
3. Build one custom LLD rule with a UserParameter. The built-in rules cover obvious cases; the real win is wrapping your own JSON producer in a UserParameter so domain-specific objects (queues, tenants, partitions, websites) become first-class discoverable items.

Pairs naturally with the PSK and TLS post (so all this discovered metadata isn't crossing the wire in plain text) and the process logging post (the canonical example of LLD against a custom JSON producer).