Why I Built an AI Agent to Troubleshoot my Network

TLDR

I built the Network Copilot Troubleshooting Agent to eliminate repetitive manual network triage. Instead of engineers SSHing into devices, running the same checklist, parsing CLI output, and correlating failures by hand, the agent automates the full workflow: command execution, parsing, evidence collection, failure-domain classification, and confidence scoring. It turns many 15–20 minute troubleshooting sessions into roughly 2-minute structured diagnoses.

Introduction

I got tired of SSHing into devices and running the same 9 commands every time a ticket came in. So I built something that does it for me. Here is how it works.

The Problem: Why Are Engineers Expensive Data Collectors?

Every network incident in a data center follows the same routine. Ticket comes in, you SSH in, run the checklist interfaces, VLANs, MAC table, ARP, IP routes, BGP, syslogs correlate everything in your head, find the break, fix it. Repeat 5 times a day.

The commands are not the hard part. The problem is you are doing it by hand every single time. Different engineers run them in different orders, miss things, and spend 15-20 minutes on something that has a 2-minute answer.

Shell 
 Real ticket example: 
"Server 10.10.10.15 cannot reach its gateway. Drops started at 14:23. Please investigate."
 This ticket deserves a 2-minute answer, not a 20-minute session.

Manual Troubleshooting vs. Network Copilot AI Agent:

• SSH login + navigation: ~1 min → automated
• Run command chain: 5–8 min → automated
• Correlate CLI outputs: 5–10 min → automated
• Decide failure domain: 2–3 min → automated
• Total per incident: 15–20 min → ~2 min

90% time saved per incident. Across 5 incidents a day, that is 1.5+ hours back. And zero variance between engineers.

The Agent: Meet the Network Copilot AI Agent

The Network Copilot Troubleshooting Agent is built specifically for datacenter network triage. You give it a device name and a destination IP. It runs the full diagnostic workflow, figures out where in the stack the problem is, and gives you a structured report with a confidence score. No flags, no syntax to remember, no SSH.

It is not a chatbot that tells you which commands to run. It actually runs them, parses the output, and tells you what is broken and how confident it is.

Shell
How you interact with it:
"troubleshoot why 10.10.20.5 is unreachable from leaf-01"

Three ways to use it:

Single command queries

1. show interface Eth1/1 on leaf-01
2. show ip route 10.1.1.1 on spine-01
3. show bgp summary on leaf-02

Focused checks

1. Check VLAN 200 on leaf-01
2. Verify ARP for 10.10.10.1
3. validate port-channel 10

Full triage (the main one)

1. troubleshoot 10.10.20.5 from leaf-01
2. datacenter triage for server 10.20.30.40
3. Run a full diagnostic via Eth1/10

How Is the Agent Architecture Built?

Each layer does one thing. The AI layer never touches SSH. The SSH layer never touches diagnosis. This separation is what makes the whole thing testable and easy to extend.

Layer 1 - AGENT (agents/main_agent.py)

Takes user input, determines the mode to run, and returns structured output. No SSH, no parsing here. The system prompt is kept short on purpose - if you give a reasoning model too much context, it burns the token budget before it can think.

Layer 2 - WORKFLOW (pkg_tools/diagnostic_workflow.py)

The actual triage logic. Runs all checks in sequence, collects everything into an evidence object, then classifies the failure domain from the full picture. Adds a confidence score to every result.

Layer 3 - TOOLS (interface_tools.py, l2_tools.py, etc.)

One file per domain, one tool per command. Each returns the same schema: { success, device, command, parsed, raw_output }. 30 tools total, all registered in main_agent.py so the LLM can call any of them directly for targeted queries.

Layer 4 - PARSER (parsers.py, report_tools.py)

Raw CLI output goes in, clean JSON comes out. This is what hides NX-OS vs Catalyst vs Arista formatting differences from everything above it. Handles interface state, VLANs, SVI state, trunk membership, MAC tables, ARP, routes, BGP sessions, port-channels, ACL hits, syslogs, and CDP neighbors.

Layer 5 - TRANSPORT (ssh_client.py, device_inventory.py, command_library.py)

Paramiko SSH connection pooling, so we are not doing a full handshake on every command. All command strings live in command_library.py, mapped per OS. Supports NX-OS, Catalyst, Arista, SONiC.

Why Does the Workflow Collect First and Classify After?

Most tools like this get it wrong in the same way. They run checks one by one and stop at the first failure. ARP missing? Report L3_ARP, done. The problem is that ARP missing has 5 different root causes, and you need the full picture to tell them apart.

The agent collects everything before it classifies anything:

checks["ping"]	_ping_check(device_name, destination_ip)	# reachability
checks["interfaces_status"]	_interfaces_status(device_name)	# L1
checks["interface"]	_interface_detail(device_name, intf)	# L1 specific
checks["vlan_brief"]	_vlan_brief(device_name)	# L2
checks["vlan"]	_vlan_presence(device_name, vlan_id)	# L2
checks["trunk"]	_trunk_check(device_name, vlan_id)	# L2 trunk
checks["svi"]	_svi_check(device_name, vlan_id)	# L3 gateway
checks["mac"]	_mac_table(device_name)	# L2
checks["arp"]	_arp_check(device_name, target_ip)	# L3
checks["route"]	_route_check(device_name, dest_ip)	# L3
checks["bgp"]	_bgp_summary(device_name)	# Control plane
checks["port_channel"]	_port_channel(device_name)	# Aggregation
checks["acl"]	_acl_check(device_name)	# Policy
checks["logs"]	_log_check(device_name)	# Events

Four things that make this actually reliable:

• Collect everything, classify once.
  The classifier runs after all checks are done. One failed check does not decide the result.
• A timeout is not the same as a missing entry.
  If SSH times out on the ARP check, that gets stored as { "success": False, "error": "..." } The classifier knows the difference between "ARP timed out" and "ARP is genuinely missing." If 3 or more commands fail, it returns UNKNOWN with low confidence and tells you the device might be unreachable rather than quietly blaming ARP.
• Only report what you confirmed.
  The agent will not infer a VLAN from an IP address. It only reports VLANs that actually showed up in show vlan brief. If the VLAN ID is unknown, it says so and gives you the command to find it.
• Every result has a confidence score.
  High = multiple checks agree, act on it.
  Medium = primary check fired, but some data is missing; verify before touching config.
  Low = either everything passed, or most commands failed, dig deeper.

This is what makes the output usable without babysitting.

What Is the 9-Step Automated Workflow?

Step 1 - Reachability
Ping first. If the destination responds, the agent stops immediately and tells you the problem is not at the network layer. No point running 8 more checks.
Function: _ping_check()

Step 2 - Interface health
Admin/oper state, CRC errors, drops. Catches physical issues before going deeper.
Functions: show_interface_detail(), show_interface_counters_errors()

Step 3 - Layer-2 forwarding
VLAN in the database, VLAN allowed on trunk, MAC learning active. All three, not just one.
Functions: check_vlan_presence(), check_vlan_on_trunk(), show_mac_address_table()

Step 4 - SVI/Gateway state
Checks show ip interface brief to confirm the Vlan<id> SVI is up. An SVI can be down or missing while the VLAN and physical interface look fine. If the gateway interface is not up, ARP will never work, regardless of anything else.
Function: check_svi_state()

Step 5 - ARP
Checks the ARP table only after confirming the SVI is up. Reports L3_ARP only if the command actually ran and the entry is genuinely absent.
Functions: check_arp_for_ip(), check_gateway_arp()

Step 6 - Routing
Route table lookup for the destination. L3_ROUTING if nothing matches.
Function: show_ip_route_prefix()

Step 7 - BGP
Session state and prefix counts. CONTROL_PLANE_BGP if the session is not established.
Functions: show_ip_bgp_summary(), check_bgp_neighbor_state()

Step 8 - Logs
Last 150 syslog lines parsed for interface flaps and hardware errors. Honestly, this one finds the root cause faster than the other 7 steps combined, more often than I expected.
Functions: show_logging_tail(), find_recent_errors()

Step 9 - ACL
Hit counters on deny rules. If there is an ACL blocking the path, it will show up here.
Functions: show_access_lists(), find_acl_hits()

check_arp_for_ip show_ip_arp show_ip_route show_ip_route_prefix, and show_ip_interface_brief all take an optional vrf= parameter. In multi-tenant datacenters, querying the global table returns nothing, and you spend 10 minutes confused. Pass the VRF name, and you get the right answer.

What Failure Domains Does the Agent Identify?

The whole point of the agent is not collecting data; it is telling you where the problem is. Every run produces one of these:

• L1_PHYSICAL - interface or cable fault
• L2_VLAN - VLAN missing or inactive
• L2_TRUNK - VLAN not allowed on trunk
• L2_MAC_LEARNING - no dynamic MAC entries
• L3_ARP - gateway not resolved, SVI is up
• L3_ROUTING - no route to destination
• CONTROL_PLANE_BGP - BGP session not established
• AGGREGATION_PORT_CHANNEL - LAG member down
• POLICY_ACL - ACL deny with active hits
• SYSTEM_EVENT - hardware error or flap in logs
• UNKNOWN - all checks passed or device unreachable

What Does the Output Actually Look Like?

Shell
Input:
"troubleshoot why 10.10.20.5 is unreachable from leaf-01"

Network Copilot Agent - Diagnosis Report - leaf-01

Device: leaf-01
Destination: 10.10.20.5
Failure Domain: L2_TRUNK
Confidence: high
Probable Cause: VLAN 200 is in the VLAN database but not allowed and active on trunk interfaces. Traffic is dropping at L2.

Evidence:
- Ping to 10.10.20.5: unreachable
- Interface Ethernet1/10: up/up, 0 CRC errors
- VLANs on device (show vlan brief): [100, 200, 300]
- VLAN 200: present, active
- Trunk check: VLAN 200 not in the allowed+active column
- SVI Vlan200: up/up, 10.10.20.1/24
- MAC learning: active
- ARP: not checked (L2 failure upstream)
- Route to 10.10.20.5: present
- BGP: established
- Port-channel: healthy
- ACL denies: none
- Logs: clean

Next Actions:
1. switchport trunk allowed vlan add 200
2. Show interface trunk to verify
3. Check the upstream device as well

Conclusion

The commands still exist. Engineers just should not have to run them manually.

The agent does the SSH, the parsing, the correlation, and the scoring. You get the answer. That is the whole point.

https://github.com/AvizNetworks/ncp-sdk-agents