Prometheus Endpoint

Cable Validation Tool - Prometheus Metrics Endpoint

Overview

The Cable Validation Tool (CVT) now provides a Prometheus-compatible metrics endpoint that exposes real-time cable health and performance data for monitoring and alerting. This endpoint enables integration with modern monitoring stacks like Prometheus, Grafana, and other observability tools.

Features

🎯 Key Capabilities

• Real-time Metrics: Live cable validation data from network switches and hosts

• Multi-format Support: Prometheus, JSON, and CSV output formats

• Rich Labeling: Complete network topology context with peer relationships

• High Performance: Multi-level caching optimized for frequent scraping

• Production Ready: Handles 100K+ ports with memory-adaptive optimizations

📊 Metrics Categories

• Power Metrics: RX/TX optical power per lane (up to 8 lanes per port)

• BER Metrics: Effective and Raw Bit Error Rates

• Temperature Metrics: Module temperature and thresholds

• Counter Metrics: Transceiver reinsert/swap events

• Validation Metrics: Port validation status with issue descriptions

• Threshold Metrics: Power and temperature alarm thresholds

• Timestamp Metrics: Data collection and report timestamps

API Endpoints

Base URL

Available Endpoints

Authentication

• No authentication required for metrics endpoints

• HTTPS enforced for security

• Bypasses session handling for automated scraping

Sample Output

Prometheus Format

JSON Format

Metrics Reference

Power Metrics

BER Metrics

Temperature Metrics

Status Metrics

Counter Metrics

Validation Metrics

Validation Status Labels

The validation_status metric includes dynamic labels for each type of validation issue:

• WrongNeighbor: "Check cable connection to correct switch"

• MediaUnplugged: "Insert; Reseat or Replace Cable/Transceiver"

• AnomalousPort: "Temperature exceeds threshold"

• FlappingLink: "Reseat transceiver; Check Fiber"

• UnknownNeighbor: "Verify neighbor device connectivity"

• WrongPort: "Check port mapping in topology"

• ExtraCable: "Remove unexpected cable connection"

• UnreachableDevice: "Check device connectivity and power"

• LinkDown_NoSignal: "Check physical connection"

• ErrDisable_Flap: "Port disabled due to flapping"

• AdminDown: "Port administratively disabled"

• ErrDisable_Rx: "RX error disable condition"

• NegotiationFail: "Check autonegotiation settings"

• NicNameMismatch: "Verify NIC provisioning"

• ModulePnMismatch: "Replace with compatible module"

Note: Commas in descriptions are automatically converted to semicolons to maintain Prometheus label format compatibility.

Examples:

Labels

Topology Labels (All Metrics)

• node: Switch or host name

• port: Port identifier

• peer_node: Connected peer node name

• peer_port: Connected peer port identifier

• node_type: Node type (Switch, Host, etc.)

• su_number: Scalable Unit identifier

• data_hall: Data hall location

Cable Labels (Status Metrics Only)

• cable_sn: Cable serial number

• cable_pn: Cable part number

• protocol: Cable protocol (400G, InfiniBand, etc.)

• port_status: Port status (Up, Down, etc.)

• plugged: Module plugged status

Performance Characteristics

Update Frequency

• Agent Data: Updated every 10 minutes (configurable)

• Metrics Cache: Invalidated on data changes

• Prometheus Scraping: Recommended 15-30 second intervals

Performance Metrics

Optimization Features

• Multi-level Caching: Port, collection, and label caching

• Memory Adaptive: Automatically adjusts for large deployments

• Smart Change Detection: Only updates when cable/module data changes

• Zero Value Handling: Includes all values for complete visibility

Troubleshooting

Common Issues

1. No Metrics Data

Symptoms: Empty response or no metrics Causes:

• CVT service not running

• No topology loaded

• No advanced stats collection

Solutions:

2. Missing Port Data

Symptoms: Some ports not appearing in metrics Causes:

• Port not in loaded topology

• Agent not deployed on switch

• Advanced stats not collected

Solutions:

• Verify topology includes all expected ports

• Deploy agents on missing switches

• Check agent connectivity and data collection

3. Stale Timestamps

Symptoms: Old timestamps in metrics Causes:

• Agent not sending updates

• Network connectivity issues

Solutions:

• Check agent logs for errors

• Verify network connectivity to switches

• Restart agents if necessary

4. Missing Validation Data

Symptoms: validation_status metrics missing or always 0 Causes:

• Validation reports not being generated

• Agent data filtering (switch not in topology)

• Report processing errors

Solutions:

• Verify validation is started on agents

• Check switch IP exists in topology

• Review agent and collector logs for errors

5. Inconsistent Issue Counts

Symptoms: validation_status count doesn't match expected issues Causes:

• Issues filtered by port

• Report data synchronization issues

• Processing errors

Solutions:

• Check that report data includes port-specific issues

• Verify advanced stats and reports arrive together

• Review validation report structure

Performance Tuning

Environment Variables

TBD: not supported yet.

Memory Optimization

For deployments > 50K ports:

• Collection-level caching automatically disabled

• Port-level caching automatically disabled

• Real-time generation used (acceptable 200-500ms response time)

Security Considerations

Access Control

• HTTPS Required: All access must be over HTTPS

• No Authentication: Designed for automated monitoring tools

• Network Restrictions: Consider IP-based access control

Sensitive Data

• Network Topology: Metrics expose network structure

• Cable Information: Serial numbers and part numbers included

• Performance Data: Could reveal network capacity information

Recommended Security

Integration Examples

Python Client Example

Validation Monitoring Examples

Architecture

Data Flow

Enhanced Data Processing

1. Agent Data Validation: Switch IP validated against topology before processing

2. Synchronized Processing: Advanced stats and validation reports processed together

3. Optimized Issue Processing: Report data pre-processed to group issues by port (O(n+m) complexity)

4. Independent Validation Cache: PortValidationStatus class with hash-based change detection

5. Robust Syndrome Handling: Automatic fallback for unknown syndromes with developer warnings

6. Smart Data Quality: NA values properly excluded, counters preserve semantics

Caching Strategy

1. Port-level Cache: Individual port metrics cached until data changes

2. Collection-level Cache: Aggregated output cached for fast retrieval

3. Label Cache: Stable topology/cable labels cached separately

4. Validation Cache: Independent cache for validation status with hash-based change detection

5. Metadata Cache: Static TYPE/HELP comments cached permanently

Performance Optimizations

• Push-based Updates: Metrics updated when advanced stats arrive

• Smart Change Detection: Only cable/module changes invalidate caches

• Memory Adaptive: Caching disabled automatically for large deployments

• String Manipulation: Efficient JSON aggregation using string operations

• Validation Processing: O(n+m) complexity with report preprocessing

• Hash-based Cache: Validation cache only invalidated when issue content changes

• Agent Data Filtering: Invalid switch IPs filtered early to prevent unnecessary processing

Monitoring Best Practices

Prometheus Configuration

• Scrape Interval: 15-30 seconds (matches CVT data update frequency)

• Timeout: 10 seconds (allows for cache generation)

• Retention: Configure based on historical analysis needs

Alerting Guidelines

• BER Thresholds: Alert when effective_ber > 1e-12

• Temperature Limits: Alert when module_temperature approaches temperature_high_th

• Validation Issues: Alert when validation_status > 0

• Critical Issues: Alert on specific syndromes (MediaUnplugged, UnreachableDevice, ModulePnMismatch)

• Infrastructure Issues: Alert on LinkDown_NoSignal, ErrDisable conditions

• Counter Anomalies: Alert on rapid increases in transceiver_reinsert_cnt

Sample Alerting Rules

Version Information

Release Notes

• Version: 1.1.0

• Release Date: October 2025

• Compatibility: CVT 1.7.0 and later

• Dependencies: Requires advanced stats collection enabled

New in Version 1.1.0

• ✅ Validation Metrics Integration: Port validation status with actionable issue descriptions

• ✅ Synchronized Data Processing: Advanced stats and validation reports processed together

• ✅ Performance Optimizations: O(n+m) validation processing, hash-based change detection

• ✅ Enhanced Security: Agent data validation prevents processing from unknown switches

• ✅ Improved Data Quality: None-based initialization for gauges, proper counter semantics

• ✅ Better Caching: Independent validation cache with content-based invalidation

• ✅ Comprehensive Syndrome Coverage: 15+ validation issue types with fallback handling

• ✅ Real-world Validation: Successfully tested with production data and unplugged ports

API Stability

• Metric Names: Stable (no breaking changes planned)

• Label Names: Stable (additions possible, no removals)

• Output Format: Prometheus standard compliance maintained

• Endpoint URLs: Stable API contract

Data Quality Improvements

Enhanced Counter Semantics

• Gauges default to None: Missing sensor data excluded instead of showing false zeros

• Counters preserve values: No unexpected resets when data temporarily unavailable

• Proper NA handling: Invalid data marked as NA and excluded from metrics

• Temperature accuracy: Fixed zero temperature issue by using actual amber timestamps

Validation Integration Benefits

• Synchronized processing: Performance metrics and validation issues always in sync

• Rich context: Issue descriptions provide actionable corrective actions

• Efficient processing: O(n+m) complexity prevents performance degradation

• Smart caching: Validation cache independent of performance metrics cache

Support

Troubleshooting

1. Check CVT Service: Ensure Cable Validation service is running

2. Verify Topology: Confirm network topology is loaded

3. Agent Status: Check that agents are deployed and collecting data

4. Network Connectivity: Verify switch/host accessibility

Performance Monitoring

Contact Information

• Development Team: Cable Validation Engineering

• Documentation: [Internal Wiki Link]

• Support: [Support Channel/Email]

This endpoint provides comprehensive cable validation metrics for modern monitoring and observability workflows, enabling proactive network health management and automated alerting.

Last updated: June 03, 2026