cenko.tech

drift-sentinel: a nightly check for router config drift

A Python bot that diffs each router's running config against an approved baseline in git, and opens a GitHub issue when they stop matching.

published
read time
10 min
tags

Someone logs into a router at 11 p.m. to give a vendor “temporary” access. The change works, everyone goes home, and six months later the vendor account, the open RDP port and telnet on the VTY lines are all still there. Nobody decided to keep them. They just stayed.

drift-sentinel is a small Python tool that catches this. Every night it pulls each router’s running config, removes the lines that change on their own (timestamps, byte counts, NTP drift), and diffs the result against an approved baseline stored in git. If a router has drifted, it opens a GitHub issue with the diff. When the router matches its baseline again, it closes the issue.

                 nightly (GitHub Actions cron / systemd timer)
                                   │
  routers ──ssh──► running config ─┤
                                   ├─► normalize ─► diff ─► GitHub issue per router
  approved/*.cfg (git) ────────────┘                    ├─► Slack / Discord webhook
                                                        └─► Markdown + JSON report

What it finds#

The repo ships with three demo routers whose “running” configs are files in examples/running/, so the nightly workflow has something to check without any real hardware. One of them, edge-rtr-01, has had exactly the vendor visit described above. This is the output from the repo’s nightly workflow run:

drift-sentinel: 3 device(s) checked: 1 drifted, 2 clean, 0 error(s)

DRIFT  edge-rtr-01  (+7 / -6)

--- approved/edge-rtr-01
+++ running/edge-rtr-01
@@ -12,6 +12,7 @@
 ip domain name example.net
 ip ssh version 2
 username netops privilege 15 secret 9 <redacted>
+username vendor-tmp privilege 15 secret 9 <redacted>
 interface Loopback0
  description Router-ID
  ip address 10.255.0.2 255.255.255.255
@@ -27,15 +28,15 @@
  ip nat inside
  no shutdown
 interface GigabitEthernet0/0/2
- description UNUSED
- no ip address
- shutdown
+ description temp - vendor access
+ ip address 192.0.2.1 255.255.255.0
+ no shutdown
 router bgp 64512
  bgp router-id 10.255.0.2
  bgp log-neighbor-changes
  neighbor 203.0.113.1 remote-as 64496
  neighbor 203.0.113.1 description ISP-A
- neighbor 203.0.113.1 password 7 <redacted>
+ neighbor 203.0.113.1 password 7 <redacted>
  address-family ipv4
   network 198.51.100.0 mask 255.255.255.0
   neighbor 203.0.113.1 activate
@@ -47,16 +48,16 @@
  10 permit 10.0.0.0 0.255.255.255
 ip access-list extended INTERNET-IN
  10 permit tcp any any established
+ 15 permit tcp any any eq 3389
  20 permit udp any eq domain any
  30 permit icmp any any echo-reply
  40 permit tcp host 203.0.113.1 host 203.0.113.2 eq bgp
  100 deny ip any any log
 ip prefix-list ISP-A-OUT seq 10 permit 198.51.100.0/24
-logging host 10.99.0.20
 tacacs server TAC1
  address ipv4 10.99.0.10
  key 7 <redacted>
 line vty 0 4
- transport input ssh
+ transport input ssh telnet
 ntp server 10.99.0.1
 end

OK     core-rtr-01
OK     branch-rtr-01
github: opened issue #1 for edge-rtr-01

A new admin account, an unused port brought up with an address, RDP allowed in from the internet, the syslog server removed, telnet enabled, and a changed BGP password. The BGP line looks the same on both sides because the password is redacted in the output, but the real values were compared, so the change still counts. More on that below.

The two other routers are reported clean, even though their files differ from the baselines. That is the normalization step doing its job.

Removing the noise#

If you diff two copies of show running-config taken a minute apart on a Cisco router, they are rarely identical. The header tells you how many bytes the config is and when it last changed, and ntp clock-period moves on its own. The running config of the demo core-rtr-01 has 24 lines its baseline doesn’t, and none of them are configuration:

Building configuration...

Current configuration : 1874 bytes
!
! Last configuration change at 01:58:12 UTC Sat Oct 17 2026 by netops
! NVRAM config last updated at 01:58:14 UTC Sat Oct 17 2026 by netops
!
...
ntp clock-period 17179738

A drift checker that alerts on this every night gets muted within a week. So before anything is compared, both sides go through normalize(), which turns a config into a list of the lines that matter:

def normalize(text: str, platform: str = "generic", ignore: Iterable[str] = ()) -> list[str]:
    patterns = compile_patterns(platform, ignore)
    drop_bangs = platform in BANG_COMMENTS
    lines = []
    for raw in text.replace("\r\n", "\n").replace("\r", "\n").split("\n"):
        line = raw.rstrip()
        if not line.strip():
            continue
        if drop_bangs and line.strip() == "!":
            continue
        if any(p.search(line) for p in patterns):
            continue
        lines.append(line)
    return lines

Line endings, trailing whitespace and blank lines go first. Then the bare ! separators, but only on platforms where ! is a comment: Cisco IOS, XE, XR, NX-OS, ASA and Arista EOS. Last are the volatile lines for that platform, which live in a table of regexes. The IOS entry, for example:

"cisco_ios": [
    r"^Building configuration",
    r"^Current configuration\s*:",
    r"^! Last configuration change at",
    r"^! NVRAM config last updated at",
    r"^! No configuration change since last restart",
    r"^ntp clock-period",
    r"^!Time:",
],

Junos gets the same treatment for ## Last commit: and the {master:0} prompt. It is also collected with show configuration | display set instead of the default curly-brace format, so every change is a single self-contained line in the diff, rather than a line buried three levels deep in braces.

Anything else that changes on its own in your network can be ignored with your own regexes in inventory.yml, for all devices or for one:

settings:
  ignore:
    - '^ntp clock-period'
devices:
  - name: lab-rtr
    host: 10.9.9.9
    ignore: ['^\s+description ']

The patterns are matched against the line with its indentation intact, so ^\s+description ignores interface descriptions without touching a top-level line that happens to start with the same word.

The baseline is a pull request#

Approved configs are plain files in approved/<name>.cfg, committed to the repo. That was the main design decision, and most of the rest follows from it. Approving a change means changing a file in git, so it gets the same review as code: someone opens a pull request, someone else reads the diff, and the merge is the approval. The git log is a history of every config change anyone signed off on.

When a drift turns out to be wanted, you don’t edit the baseline by hand. An Approve running config workflow takes a device name, logs into the router, captures its current config and opens the pull request for you:

drift-sentinel approve edge-rtr-01      # or: drift-sentinel approve --all

The baseline is written in the normalized form, so the pull request is not full of timestamps. After the merge, the next nightly run finds the router clean and closes its issue.

When the drift is a mistake, you fix the router instead, and the issue closes itself the same way. Either way, there is no “acknowledge” button. The issue stays open until the router and the baseline agree.

One issue per router, updated rather than repeated#

The nightly job runs every night, and the drift on edge-rtr-01 will still be there tomorrow. Opening a new issue each time would be useless, and so would commenting “still drifted” 30 times a month. Each run has to find the issue it opened before, and decide whether anything is new.

The issue body starts with an HTML comment that GitHub does not render:

<!-- drift-sentinel device="edge-rtr-01" fingerprint="3abad2910fa6" -->

The device name ties the issue to a router, so the bot never relies on the title and you can rename the issue freely. The fingerprint is the first 12 hex characters of a SHA-256 of the diff. On each run the bot lists open issues with the config-drift label, reads the marker out of each body, and compares:

if r.status == DRIFT:
    if issue is None:
        # open "Config drift: <name>"
    elif issue["fingerprint"] != r.fingerprint:
        # rewrite the body with the new diff and add a comment
    else:
        # same drift as last night: do nothing
elif r.status == CLEAN and issue is not None:
    # comment and close
# ERROR: leave any issue untouched

The second nightly run on the demo repo, a minute after the first, shows the quiet path:

github: issue #1 for edge-rtr-01 already up to date

So you get one notification when a router drifts, one when the drift changes, and one when it is resolved. GitHub’s normal email and app notifications handle the delivery. The issue you open in the morning always shows the current diff, not the one from the night it was first spotted.

The last branch matters as much as the others. If a router cannot be reached, the bot does not know whether it is clean or not, so it does not touch the issue. A timeout should not close a real drift issue, and it should not open a false one. An unreachable device is reported as an error instead, and the run exits with code 2 and turns red.

Secrets: compared, never shown#

Router configs are full of secrets: enable secrets, user hashes, BGP and OSPF keys, TACACS keys, SNMP communities, IPsec pre-shared keys. A drift alert that pastes them into a GitHub issue, Slack or a CI log is a new problem. But if you redact them before comparing, a changed BGP password is no longer drift, and that is exactly the kind of change you want to hear about.

drift-sentinel does both, in that order. The configs are compared with the real values, then every line of the diff is passed through redact() on the way out:

return DriftResult(device, DRIFT, "\n".join(redact(line) for line in lines), added, removed)

That is why the BGP neighbor line appears twice above with <redacted> on both sides. The values were 094F471A1A0A and 13061E010803, and the output only says that they differ. The same redacted diff is what goes into the issue, the webhook, the Markdown report and the JSON report.

redact() is a list of regexes. Each one captures a prefix such as secret 9 or snmp-server community and replaces the value after it:

r"(\b(?:secret|password)\s+(?:(?:\d|sha256|sha512)\s+)?)" + _VALUE,
r"((?<!-)\bkey(?:-string)?\s+\d\s+)" + _VALUE,
r"(\bsnmp(?:-server)?\s+community\s+)" + _VALUE,
r"(\bpre-shared-key\s+(?:(?:local|remote)\s+)?(?:\d\s+)?)" + _VALUE,

The hard part was the false positives. service password-encryption, security passwords min-length 8, key chain OSPF and authentication md5 key-chain HSRP all contain words that look like a secret is coming, and none of them have one. The tests check those lines are left alone, next to 16 real secret lines from IOS, NX-OS and Junos that must be redacted.

Two trade-offs come out of this.

First, the fingerprint is taken from the redacted diff. If a password changes again while its drift issue is already open, the redacted diff looks the same, so the issue is not updated. It is still open and still shows the password line as changed, so nothing goes unreported, but you will not get a second notification for it.

Second, comparing real values means the baselines in git contain real secrets. If you don’t want that, set secrets: mask. Both sides are then redacted before the comparison, and approve writes redacted baselines. A new or removed secret line is still drift, but a changed value is not. The README is honest that redaction is pattern-based and best effort, and that you should read your baselines before pushing them anywhere public.

Getting the configs#

Three collectors decide where a running config comes from:

  • ssh logs in with netmiko, so any netmiko device_type works. Credentials come from environment variables, which in Actions are repository secrets. Devices are checked in parallel, 8 at a time by default.
  • file reads a config that something else has saved. The demo uses this, and it is also the way in if you already back up configs with Oxidized, RANCID or NAPALM.
  • command runs a shell command and treats its stdout as the config, with {name} and {host} substituted and shell-quoted: ssh backup@nms cat /configs/{name}.

The catch with GitHub Actions is that hosted runners are on the public internet and your management network is not. For real routers you register a self-hosted runner on a jump host and change runs-on. If you would rather not use Actions at all, the repo has a systemd service and timer for 02:17 every night. Run that way, it can still open GitHub issues using a fine-grained token with issue access.

Try it#

You need Python 3.10 or newer.

git clone https://github.com/0xgitpushpray/drift-sentinel.git
cd drift-sentinel
pip install .
drift-sentinel check

That runs against the three demo routers and prints the edge-rtr-01 diff above. Only PyYAML is required. Add ".[ssh]" to pull in netmiko when you point it at real routers.

The README walks through adding your own devices, recording the first baselines and setting up Slack or Discord. The open config-drift issue on the repo is the one the bot opened for the demo.

Tests#

65 tests with pytest, none of them on the network. The normalize tests feed in real-looking IOS and Junos headers with CRLF line endings. The GitHub tests use a fake client that records every API call, and check that a repeated drift makes no writes at all, a changed drift edits and comments, a clean router closes its issue, and an error leaves it alone. CI runs them on Python 3.10 and 3.13, then runs the demo with --fail-on-drift and checks that it exits with 1, so a change that stops the tool from noticing edge-rtr-01 fails the build.

cd ~/blog